Build a large Image Scraper

Stagecoach Software Pty Ltd is developing a website to contain statistical profiles of a particular industry in Australia. Our stack is Ruby on Rails 4.2, Ruby 2.2, Passenger 4 and MySQL 5.5. We run a 3 server configuration on Engine Yard with dedicated instances for our db, web app and utilities. Our database receives about 30,000 new rows per day through scraping operations (22 different scrapers using nokogiri). Presently our app instance only provides admin operations; the consumer front end is under development. Currently our team comprises 3 backend ruby developers and 2 front end developers. We have about 16 M products in our database. Each product has a distinct id number. Our front end website must be able to display an image of each product. There is a website that contains images of a larger range of products but which includes the ones in our database. The ones that match our database are identifiable on that site based on the descriptions in our database. We need copies of these images downloaded to our AWS S3 bucket. Each image must be given a filename that connects it to the relevant product id in our database. The image table in our database must be updated to reflect the details necessary to allow the images to be uploaded to our website. In other words a scraper is needed that will search the image website based on the product descriptions in our database, download matching images and update our database to record the details of the images downloaded and facilitate retrieval. Development and testing must be done locally. Then code must be uploaded to our github repo and approved by the lead developer before deployment. This is one of a pipeline of projects in this enterprise. Further ruby projects will be available if the above tasks are carried out well. Applications from a freelancer experienced with Engine Yard are welcomed.