I need a web crawler

I need a webcrawler that can find websites that are using or have Flash elements on their website.

Furthermore it needs to be able to differentiate the language of the website. I am ONLY interested in finding German and English language websites!

The project literally requires that I find ONLY websites in German and English language. That could be a whole number of different domain endings such as .de .at .ch for German language and .com .[login to view URL] .[login to view URL] [login to view URL] and so forth for English. But I definitely don't want websites that are in any other language (french, spanish, italian, polish, russian, japanese, chinese etc).

The crawler needs to generate a list of domains and their e-mail addresses for THAT language.

The crawler needs to have a GUI where the above parameters most be adjustable. It also must have the possibility to check against a blacklist, so if he hits a known domain or email, it gets skipped from the beginning.

There is a second stage where the crawler is supposed to find all websites that do not have an SSL certificate.

And a third stage to find those websites that use cookies, but do not have a cookie banner.

If the crawler can do all 3 of the above things, that much the better. But the first stage is to find websites with Flash elements.

I need a minimum of 2 million addresses per language

I am looking forward to your offers!

Skills: Web Scraping, PHP, Python

About the Employer:
( 1 review ) Kensington, South Africa

Project ID: #20352419

53 freelancers are bidding on average $1234 for this job


Hi, I can design and develop a web crawler for you. I am familiar with related libraries and frameworks such as Tensorflow, Keras, Theano, Torch Scipy, Scikit-learn. and I am skilled in Python, R, Lua, Go etc. So thi More

$3000 USD in 35 days
(166 Reviews)

I have checked your requirements where you mentioned that you need a webcrawler that can find websites that are using or have Flash elements on their website. Please invite me in chat so we can discuss in detail.

$1500 USD in 20 days
(143 Reviews)

Hello, As per need, you need a web crawler, we have shared our company portfolio please go through it: We have a team of professional Full Stack developers with an experience of more than 8+ years in creating and cus More

$1500 USD in 20 days
(186 Reviews)

Hello, Dear How are you? I have checked your project description and am ready for discussing with you about project as Web scraping developer. I have experienced in PHP , Python, Web Scraping. I will work very hard and More

$1125 USD in 20 days
(104 Reviews)

Hi, I have read your post and understood your requirement. Looking for the freelancer to work on your next project? Or just need some issues/bugs/fixes ASAP? I have 11+ years of experience and I'm here for you! My sk More

$750 USD in 7 days
(104 Reviews)

Hello, I read your description and got what to do. I am looking forward to discussing more for your project. I am ready now, please message me to start your project. I have 7+ years of experience in this field I am More

$1125 USD in 7 days
(20 Reviews)

Greetings! My name is Neha. We would love to have the opportunity to discuss your project with you. We can read through the job details extremely carefully and absolutely sure that we can do the project very well. More

$1000 USD in 12 days
(335 Reviews)

Hello, How are you? Yes i am expert in web scraping . i have more than 6 year experienced in php. i can do this scraping in php, You can check my work:- [login to view URL] Thanks Shweta

$1500 USD in 7 days
(114 Reviews)

With Freelancer Preferred badge bound to give 100% Quality (Let's Chat) Hello, A great team is here for your service!! I read the project description properly, and I agreed to fulfill 100% of all your requirements. More

$1125 USD in 7 days
(151 Reviews)

Hi, I'm experienced web scraping developer with 5 years of hands-on experience. I have created several scraping projects and even developed automatic scraping bots. I'm good at python programming language and used scra More

$1500 USD in 20 days
(39 Reviews)

Hello BrandBuilder63, I have checked your project(I need a web crawler) requirements. We are one of the best programming team on freelancer. Awarded PREFERRED FREELANCER BADGE by We have more than 127 More

$1625 USD in 7 days
(83 Reviews)

Greetings! Seeing is believing - please check my recent reviews and portfolios first. I am proficient in Web scrapping and Data mining, and I have already finished several smiliar projects before. I am confident in m More

$1125 USD in 7 days
(37 Reviews)

Hi. I am very interested in your project, because I have much experience in such projects. I have good skills with the program language including C/C++, C#, java, php,, python, VB.NET. So I have expert and smar More

$1125 USD in 7 days
(112 Reviews)

Hello I am a talented Web developer. To detect Flash elements, I can scrape Web pages from each website using selenium. In this way, we can also detect languages. The second and third stage is a bit difficult but I ca More

$1000 USD in 7 days
(28 Reviews)

Hello there, As you have posted jobs "I need a web crawler". We are a team of 60+ members in PHP, CodeIgniter, WordPress, Magento, Big commerce, Interspire, , Node Js, AngularJs, and Mobile Apps having experience ove More

$1500 USD in 70 days
(22 Reviews)

Dear Sir, Requirements : - CMS AS per your need or we can make custom CMS too - Design : HTML5, CSS 3, Bootstrap - Jquery - CMS will Custom CMS - Validation Java Script - Google Map API - API ( FACEBOOK ) https:/ More

$1500 USD in 40 days
(24 Reviews)

Highly experienced web scraper developer. My web scrapers scrape more than 10 Million pages each day without getting blocked. Please check my reviews here to know more about the quality you can expect : [login to view URL] More

$1500 USD in 10 days
(30 Reviews)

Hello, I love to create Web crawler that can find out German and English language websites! using flash on their websites. I see web crawler should generate a list of domains and their e-mail addresses for THAT langua More

$1000 USD in 7 days
(251 Reviews)

Hi. I am Guicai Y. I have read your post content and I am able to help you. I am web expert using python. I have made more project using it for client. And otherwise I am web scraping expert too. I have more knowledge More

$1125 USD in 7 days
(45 Reviews)

Hello, Greeting for the day! I recently came across your job post and feel that I can assist you perfectly. I am a senior web developer with an experience of 8+ years, working proficiently on various domains which in More

$900 USD in 7 days
(37 Reviews)