Closed

Web crawler / scrapers for bloomberg

Detected by the web site as a robot

Need to fix below to crawl the web successfully

$ch = curl_init();

$headers = array();

$headers[] = "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:62.0) Gecko/20100101 Firefox/62.0";

$headers[] = "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";

$headers[] = "Accept-Language: zh-TW,zh-HK;q=0.8,en-US;q=0.5,en;q=0.3";

$headers[] = "Dnt: 1";

$headers[] = "Connection: keep-alive";

$headers[] = "Upgrade-Insecure-Requests: 1";

$headers[] = "Te: Trailers";

curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);

curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);

curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "GET");

curl_setopt($ch, CURLOPT_ENCODING, 'gzip, deflate');

curl_setopt($ch, CURLOPT_URL, "[login to view URL]:IND");

curl_setopt($ch, CURLOPT_MAXREDIRS, 10);

curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 100);

curl_setopt($ch, CURLOPT_REFERER, "[login to view URL]:IND");

curl_setopt($ch, CURLOPT_SSL_VERIFYHOST,2);

curl_setopt($ch, CURLOPT_SSL_VERIFYPEER,false);

curl_setopt($ch, CURLOPT_FAILONERROR, false);

curl_setopt($ch, CURLOPT_HEADER, 1);

curl_setopt($ch, CURLOPT_COOKIE, $cookie);

curl_setopt($ch, CURLOPT_COOKIEJAR, COOKIE);

curl_setopt($ch, CURLOPT_COOKIEFILE, COOKIE);

$result = curl_exec($ch);

if (curl_errno($ch)) {

echo 'Error:' . curl_error($ch);

}

curl_close ($ch);

Skills: PHP, Web Scraping

See more: java web crawler image capture, web crawler cpu usage, write web crawler, web scraping, web scraping tutorial, web crawler using beautifulsoup, beautifulsoup bloomberg, web scraping python beautifulsoup, beautifulsoup tutorial python 3, building a web scraper in python, scraping bloomberg data, web crawler names, web crawler system requirement, web crawler demo gathering information function, java spring web crawler, web crawler traffic stats, auto daily web crawler, web crawler sale, web crawler asp spider, financial news web crawler

About the Employer:
( 4 reviews ) hong kong, Hong Kong

Project ID: #17927169

10 freelancers are bidding on average $221 for this job

mantislin

Hi sir, This is Lin and I am scraping expert, please check my reviews then you will know. Can we discuss more details about this? Thanks, Lin

$240 HKD in 2 days
(377 Reviews)
7.7
gangabass

I can update your PHP scraping script so it will be not detected as a robot. I can start right now and complete it in next 2 hours.

$240 HKD in 1 day
(396 Reviews)
7.2
cuebiztech

I see the code is only for loading. What exactly need to be scrapped and where to store this? Want to discuss more details on the project

$240 HKD in 1 day
(77 Reviews)
5.7
MohanKumar28

Hi, I have checked the requirements. Will do this script using PHP selenium web driver. I'm available to start now. Please share additional details. Regards, Mohan

$240 HKD in 1 day
(59 Reviews)
5.3
case1352

this sounds pretty straightforward. i have done a lot of webscraping work. please have a look over my feedback and get in touch if you have any questions. thanks, simon

$120 HKD in 1 day
(16 Reviews)
4.6
leemilun

Hi, dear. Nice to meet you. i've read your post carefully. I'm php expert. Please discuss more details by chatting. Regards. Gao M.

$240 HKD in 1 day
(17 Reviews)
4.5
kanwalrafique

Hi, do you know how Bloomberg website detected your web crawler as a robot when you're presenting it as Firefox 62.0. They use JavaScript based pings to monitor their website traffic. When we fetch the page using cURL More

$500 HKD in 3 days
(20 Reviews)
4.5
kchoudhary95

ok It is easy I will give you that solution if you are hire me. I am php developer. I have experience in php.

$95 HKD in 1 day
(1 Review)
1.3
saeedfalsafin

Hello my friend, I HAVE FIXED YOUR CODE! Send me a message to send you the result of fixed script! I can help you in any problem you have.

$90 HKD in 1 day
(1 Review)
0.8
jsphp

Hello! I feel your job is an interesting one, so I would like to offer you my services. I have more than 8 years of experience in web-development. I have studied your problem carefully and I am ready to start working More

$200 HKD in 1 day
(0 Reviews)
0.0