Closed

Scrapy + Redis / Cassandra -- Price Monitoring Bot / Crawler / Scraping / Data Mining

Hello,

We are looking to utilize Scrapy (Scrapy+Redis or other option) to run on our internal servers to handling data mining for the purposes of price monitoring, alerting and adjustments.

We need a freelancer who is skilled with scrapy already building the environment from the ground up (its ok to use templates and such).

It needs to have the following functions at minimum:

1) Scrapy running from scaleable server environment

2) Documented method for business users to add websites/fields to be scraped

3) Store Data

4) Data Reporting

5) Anti-Blacklisting (IP Rotation, Proxies, Check [url removed, login to view], Throttling, Spoofing, Headless Browser)

6) Quality Assurance / Compliance

The project must be well documented in a understandable way in case you leave the project at some point. We also need you to stay on board long term to help maintain this environment.

We will provide all the infrastructure and associated items.

Payment:

50% milestone funded upfront, second 50% funded when working model shown. 100% released once you finish and it is fully functional with all items in scope of this project complete.

Any items not specifically mentioned in the original project are not valid.

If you bid on this project, your bid is final price for initial project.

Skills: Data Mining, Elasticsearch, Python, Software Architecture, Web Scraping

See more: Data Extraction Web Scraping Data Mining, web scraping data mining, web scraping data mining mapping, message scraping data mining, data scraping data mining project, bot crawler search, yellow pages bot crawler, bot crawler web, bot crawler work, bot crawler engine script, airline price monitoring email alert, airline price monitoring, project profile market price monitoring cell, spider bot crawler maker, screen scraping data mining

About the Employer:
( 83 reviews ) Marietta, United States

Project ID: #15890077

8 freelancers are bidding on average $455 for this job

mantislin

Hi sir, This is Lin and I am scraping expert, i have checked all details for your project. can we discuss more info then i can provide example data for you? Please message me then we can discuss more ASAP. Tha More

$333 USD in 6 days
(233 Reviews)
7.4
bestit4u

Hi,sir I am very interested in your project, because I have much experience in such projects. Relevant Skills and Experience I have expert and smart technic with web scraping (1000 over projects for scraping), automat More

$333 USD in 3 days
(64 Reviews)
6.4
ITCristRo

Hi! I am a professional expert. I have developed many similar projects successfully. If you want to fast and good results, you can hire me. Relevant Skills and Experience Data Mining, Elasticsearch, Python, Software A More

$2000 USD in 30 days
(8 Reviews)
4.5
DataHome

Can do anything on scrapping any site including login and captcha Let us discuss further Waiting.. Amaan Relevant Skills and Experience 10 years history Proposed Milestones $35 USD - milestone let us discuss for fai More

$35 USD in 3 days
(7 Reviews)
3.4
anatolygenay123

Hello. I am a Python expert. I'd like to discuss about your project more details. thanks. Relevant Skills and Experience Data Mining, Elasticsearch, Python, Software Architecture, Web Scraping Proposed Milestones $33 More

$333 USD in 3 days
(5 Reviews)
2.6
htmn

IT engineer, 6 years of experience in web crawling/ web scraping, I have such project ready,I can show you a sample. Please contact me for details: website to scrape, deadlines, price.. Relevant Skills and Experience More

$250 USD in 30 days
(1 Review)
2.4
$155 USD in 3 days
(0 Reviews)
0.0
lafard

There's not much to say, but I know I'll do a very good job Relevant Skills and Experience python

$200 USD in 10 days
(0 Reviews)
0.0