Create a scraping engine that automatically runs and deposits data into a database

I need to build a scraping engine that periodically scrapes certain data from 6 different websites and compiles that data into a database that can be displayed on my website. You will have to build the scraping script and set up a database for it to write the data to, as well as calibrate the script to run successfully on 6 different websites. You do *not* have to do any front-end development for displaying the database information on my website. However, ultimately I will be generating static HTML pages on my website based on data in the database, so keep that in mind. The websites each contain about 100-1000 items. The details of each of these items need to be scraped. Here is an example of what the websites may look like: ([login to view URL]) For each item, there are about 10 different specification/category labels that need to be captured, and some have a small image or two as well. There is data on both the Item List and Item Details pages that should be captured (some of it is redundant and present on both pages). I will show you how to label and categorize each of the specific data fields available on each website. I have a VPS hosting account through Inmotion, and I can also supply an Amazon Web Services account if it's necessary. Whichever you think is a better method. Requirements of scraping engine: - The script automatically runs every 6 hours, and writes the data to a database. - Can navigate through non-static pagination (example: [login to view URL]) - Can deal with variations among sites (some sites have pagination, some have none. Some have Item Details pages, some have none). Since there is only a small number of websites to be scraped (6 total), you can design the script independently for each of the 6 sites, rather than a "one-size-fits-all" script, whichever is easier for you. - Cache invalidation for when data is changed or removed from the source websites The estimated budget for this project is $200 - $300.