I need a crawler
which does the following:
-crawls scriptlance, rentacoder, elance, sologig, odesk, ifreelance etc. and stores in a db [project name, project description, project start date, project end date, project category for all projects (open, frozen, closed, cancelled), username and bids + the bid amounts (if no bid amount, nothing)]
-the intervals at which the crawl will happen, can be specified for each site and changed via a config file.
-the project category can also be specified (because some sites have random categories ... I can build the initial set of categories to scour or so)
-if the site offers RSS, we can also use RSS to gather this data.
-(optional) the crawler also crawls for the unique users, their profiles (as plain page dump) and stores their feedback ratings and their usernames + a link to their profile pages.
crawler can be perl, php whatever language you like.
it needs to not fail.
we'll set the cron and let it go.
Please keep the bidding at a reasonable price.