Python 3 Scrapy Web Scraper behind Flask (or equivalent) Rest API
$30-250 USD
Cancelled
Posted over 6 years ago
$30-250 USD
Paid on delivery
I created a scrapy web crawler that can be invoked a by script or through the command to scrape between 2-30 pages and turn a json file. I'm trying to put that behind a simple Rest API using Flask and I can't get any of these suggestions to work:
[login to view URL]
The API works once, and then I get a reactor error. I need to both read the results from a JSON file or object that I can pass into another function, before inserting the results into a mongoDB database.
I need the code to be simple so I understand it (and scalable) so many people can use the API and utilize different threads/queue. The objective of this project is just to put the scraper behind simple rest API that can be secured, so that it can be called in a async non blocking manner (multiple times) and return the scraped json to python variable. Its not a requirement to use Flask as long as the approach is simple and uses well known and maintained libraries. I also need to pass arguments to the spider that will be in the POST json body.