# Code Golf: Make a Web Crawler

A Web Crawler is a program that starts with a small list of websites and gets the source code for all of them. It then finds all the links in those source codes, gets the website it links to then puts them in a list. It then starts again but with the new list of websites.The main reason for a Web Crawler is to find all (or most) of the websites in the internet. Google use this to power their search engine.Wikipedia Page about Web Crawlers

Your task is to create a Web Crawler in any language you wish.

Rules

• Your first list can contain any amount of websites <= 5 and any websites you wish.

• Your web crawler can ignore any websites with the HTTP GET request or any other pages other than the home page, e.g. www.somewebsite.com/?anything=1 and www.somewebsite.com/secondpage can be ignored, we are only trying to find the website/domain, e.g. www.somewebsite.com

• You can use any method you wish to store the list of websites currently being "crawled".

• Although this is code golf, the highest voted answer/solution will win, and don't forget to have fun!

• Participants are cautioned that you can make site owners and/or your ISP very mad by doing this badly. – dmckee Mar 28 '12 at 14:53
• "Although this is code golf, the highest voted answer/solution will win, and don't forget to have fun!" No. [code-golf] defines the winning critera. If you want a different objective winning critera you can set that, but you should be aware that many user don't like "most votes" as a criteria. – dmckee Mar 28 '12 at 14:55
• To clarify dmckee's comment, any considerate web crawler should find and honor the robots.txt document for any domain it crawls. One of my college courses used this as an assignment and consistantly got the campus network blacklsted from several major sites a few times a year. See robotstxt.org – captncraig Mar 28 '12 at 18:37
• Oh, I never actually thought about that. So do you think this question should be deleted? – 3aw5TZetdf Mar 28 '12 at 19:40
• @CeilingSpy: What's about updating it? Or flag it for closing by a mod, and discuss it in chat or at the meta sandbox, which is meant for that, until clarification. It can be reopened later. – user unknown Mar 29 '12 at 1:46

Javascript, 491

Golfed

var C=function(c){w=c.length;for(i=0;i<w;i++){var b=new XMLHttpRequest;b.setRequestHeader("Access-Control-Allow-Origin","*");b.setRequestHeader("X-Requested-With","XMLHttpRequest");b.open("GET",c[i],!0);b.send();if(200===b.status){a=b.responseText.match(/\<[(a|img)]+\s[^\>]+[(http|https)]+:\/\/([\w\.])+([.])+.+?[^.]+?./gi);u=c[i].substr(7);re=RegExp(u,"i");for(j=0;j<a.length;j++)if(a[j].match(re)&&(b=a[j].match(/[(http|https)]+:\/\/([\w\.])+([.])/gi))&&!c.indexOf(b))c.push(a[j]),w++}}};


Un-Golfed

var C = function(s){
w = s.length;
for(i = 0; i < w; i++){
var r = new XMLHttpRequest();
r.open('GET', s[i], true); r.send();
if(r.status === 200) {
a = r.responseText.match(/\<[(a|img)]+\s[^\>]+[(http|https)]+:\/\/([\w\.])+([.])+.+?[^.]+?./gi);
u = s[i].substr(7),
re = new RegExp(u,'i');
for(j = 0; j < a.length; j++){
if(!a[j].match(re)) continue;
var uri = a[j].match(/[(http|https)]+:\/\/([\w\.])+([.])/gi);
if(uri && !s.indexOf(uri)) {
s.push(a[j]); w++;
}
}
}
}
}


Method

var c = new C(['http://www.cnn.com', 'http://www.usatoday.com', 'http://www.huffingtonpost.com', 'http://www.drudgereport.com/', 'http://news.cnet.com']);

• How did you run the above script? I've tried Firefox and Node.js, both fail for different reasons. – Annan Nov 9 '15 at 16:51