In Progress

Laravel: Write a script to crawl my website to find and list links

I need a freelancer to write a script that will crawl the entire blog section of my website and retrieve all of the links (broken and unbroken), then dump those links (along with the page origin and the request status) into a log file.

The page origin path must be relative to my domain. The page origin, link url, and request status must be on one line, seperated by a comma, and then a space ( ", ") Like this:

PAGE ORIGIN, LINK URL, REQUEST STATUS

======

for example:

/blog/my-awesome-blog-post, [url removed, login to view], 200

/blog/another-blog-post, [url removed, login to view], 400

/blog/another-blog-post, [url removed, login to view], 500

/blog/what-a-great-blog-post, [url removed, login to view], 301

The crawler should crawl all links at [url removed, login to view]*

Also, the crawler should EXCLUDE my search route: [url removed, login to view]*

Skills: Data Processing, Laravel, PHP, Shell Script, Web Scraping

See more: list links website, website crawl script, movie links website script, software write mq4, software write chip epson, useful software write book, software write web specs, software write edid, write list links joomla, software write websites idea, software write book images, software write books, software write protection, free software write book, software write book, write script searching website, write script search website, software write technical manual

About the Employer:
( 4 reviews ) Atlanta, United States

Project ID: #11707744

Awarded to:

sszelag

Hi, Can you give me a link to your blog? I'd like to know how much data needs to be extracted. Thanks

$80 USD in 1 day
(9 Reviews)
4.8

3 freelancers are bidding on average $88 for this job

Verz1Lka

Hello! I'm web-scraping and web-automation expert and i can done your project. I use python language and scrapy framework. My scripts works on windows, mac or linux, but linux is preferably. I can schedule scripts on More

$83 USD in 3 days
(69 Reviews)
6.2
raheelhassan14

Nice to see your project. I have read the details. .Ready to start my work. Looking forward. Thank You!!

$100 USD in 2 days
(25 Reviews)
5.8