Web crawler / scrapers for bloomberg

Detected by the web site as a robot

Need to fix below to crawl the web successfully

$ch = curl_init();

$headers = array();

$headers[] = "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:62.0) Gecko/20100101 Firefox/62.0";

$headers[] = "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";

$headers[] = "Accept-Language: zh-TW,zh-HK;q=0.8,en-US;q=0.5,en;q=0.3";

$headers[] = "Dnt: 1";

$headers[] = "Connection: keep-alive";

$headers[] = "Upgrade-Insecure-Requests: 1";

$headers[] = "Te: Trailers";

curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);

curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);

curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "GET");

curl_setopt($ch, CURLOPT_ENCODING, 'gzip, deflate');

curl_setopt($ch, CURLOPT_URL, "[login to view URL]:IND");

curl_setopt($ch, CURLOPT_MAXREDIRS, 10);

curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 100);

curl_setopt($ch, CURLOPT_REFERER, "[login to view URL]:IND");

curl_setopt($ch, CURLOPT_SSL_VERIFYHOST,2);

curl_setopt($ch, CURLOPT_SSL_VERIFYPEER,false);

curl_setopt($ch, CURLOPT_FAILONERROR, false);

curl_setopt($ch, CURLOPT_HEADER, 1);

curl_setopt($ch, CURLOPT_COOKIE, $cookie);

curl_setopt($ch, CURLOPT_COOKIEJAR, COOKIE);


$result = curl_exec($ch);

if (curl_errno($ch)) {

echo 'Error:' . curl_error($ch);


curl_close ($ch);

Skills: PHP, Web Scraping

See more: java web crawler image capture, web crawler cpu usage, write web crawler, web scraping, web scraping tutorial, web crawler using beautifulsoup, beautifulsoup bloomberg, web scraping python beautifulsoup, beautifulsoup tutorial python 3, building a web scraper in python, scraping bloomberg data, web crawler names, web crawler system requirement, web crawler demo gathering information function, java spring web crawler, web crawler traffic stats, auto daily web crawler, web crawler sale, web crawler asp spider, financial news web crawler

About the Employer:
( 4 reviews ) hong kong, Hong Kong

Project ID: #17927169

9 freelancers are bidding on average $234 for this job


Hi sir, This is Lin and I am scraping expert, please check my reviews then you will know. Can we discuss more details about this? Thanks, Lin

$240 HKD in 2 days
(389 Reviews)

I can update your PHP scraping script so it will be not detected as a robot. I can start right now and complete it in next 2 hours.

$240 HKD in 1 day
(428 Reviews)

Hi, dear. Nice to meet you. i've read your post carefully. I'm php expert. Please discuss more details by chatting. Regards. Gao M.

$240 HKD in 1 day
(35 Reviews)

Hi, I have checked the requirements. Will do this script using PHP selenium web driver. I'm available to start now. Please share additional details. Regards, Mohan

$240 HKD in 1 day
(87 Reviews)

I see the code is only for loading. What exactly need to be scrapped and where to store this? Want to discuss more details on the project

$240 HKD in 1 day
(77 Reviews)

Hi, do you know how Bloomberg website detected your web crawler as a robot when you're presenting it as Firefox 62.0. They use JavaScript based pings to monitor their website traffic. When we fetch the page using cURL More

$500 HKD in 3 days
(28 Reviews)

this sounds pretty straightforward. i have done a lot of webscraping work. please have a look over my feedback and get in touch if you have any questions. thanks, simon

$120 HKD in 1 day
(16 Reviews)

Hello my friend, I HAVE FIXED YOUR CODE! Send me a message to send you the result of fixed script! I can help you in any problem you have.

$90 HKD in 1 day
(10 Reviews)

Hello! I feel your job is an interesting one, so I would like to offer you my services. I have more than 8 years of experience in web-development. I have studied your problem carefully and I am ready to start working More

$200 HKD in 1 day
(0 Reviews)