Crawler: Scrappy Optimization

I am a senior Python Developer. I have more than 4 years of experience working under different employers and have proved them the best to my capacity. I have good proficiency in the all the areas of Python. Areas include Distributed Computing, Data Mining, Imaging, Network Programming, Offensive Security tools Development, Web Application Development and many more.

hello i can optimize your scraper code , will be using mysql thanks

Hi. I have a powerful experience in python and help you with that. I can help you using scrapy, beautifulsoup, selenium and so on. Please see the attachment. This is the sample source for scraping products of e-commerce site and scraping doctor personal info which is developed using scrapy framework and beautifulsoup I have enough skills to help you. Please hire me, then i'm ready to help you. Thanks. Regards.

Hi, Can you show code of ypur scrapy project? Do you rerequest pages with timeout? Best regards, Ilshat

A proposal has not yet been provided

My name is Duane and I am an application developer with 7 years’ experience in the IT industry. I am the project manager for a small software company Water Web Design LLC. The technologies we use are Html 5, CSS 3, C, C++, PHP, MySql, Javascript, ajax, Java and Python. We develop websites, mobile applications and desktop applications. i am ready to start working on this project and would like to make this a long term business relationship.

Hi, I am very pleased to help you. I'm the right person for your crawler:-) I'm experienced in python, and advanced in SQL optimization. Prevent random timeouts on links that are really working. For example, so links that are going to linkedin is making the crawler timed out. Solution: Add more request when time-out Minimize cpu usage & make the crawler more efficient Solution: add gevent(non-blocking asyn i/o) support and multithreading Make the crawler efficiently insert to the database. You are free to use whether mysql or mongodb if you think what is the best. Solution: Optimize SQL statements:-) Use different method of matching the url uniqueness (crawler options). Do not use mysql on this cause it is so expensive. One option is to use reddis. Solution: I have created a distributed crawler using redis. So just porting my code to your crawler will be ok Use different database for storing crawler jobs. Ex. reddis Solution: just porting my redis part will be ok. I'll keep in touch with you if I am awarded, thank you:-)

My expertise include Data Mining, Image Processing, Web and Desktop Application Development, Network tools and many others. Try me! You will know. :)

Hey there! Looking forward to be hired! I have many years of experience in python. Let me know if you'r interested.