A Web Crawler is a program that starts with a small list of websites and gets the source code for all of them. It then finds all the links in those source codes, gets the website it links to then puts them in a list. It then starts again but with the new list of websites.The main reason for a Web Crawler is to find all (or most) of the websites in the internet. Google use this to power their search engine.Wikipedia Page about Web Crawlers

Your task is to create a Web Crawler in any language you wish.


  • Your first list can contain any amount of websites <= 5 and any websites you wish.

  • Your web crawler can ignore any websites with the HTTP GET request or any other pages other than the home page, e.g. www.somewebsite.com/?anything=1 and www.somewebsite.com/secondpage can be ignored, we are only trying to find the website/domain, e.g. www.somewebsite.com

  • You can use any method you wish to store the list of websites currently being "crawled".

  • Although this is code golf, the highest voted answer/solution will win, and don't forget to have fun!

  • 4
    \$\begingroup\$ Participants are cautioned that you can make site owners and/or your ISP very mad by doing this badly. \$\endgroup\$ Mar 28, 2012 at 14:53
  • 5
    \$\begingroup\$ "Although this is code golf, the highest voted answer/solution will win, and don't forget to have fun!" No. [code-golf] defines the winning critera. If you want a different objective winning critera you can set that, but you should be aware that many user don't like "most votes" as a criteria. \$\endgroup\$ Mar 28, 2012 at 14:55
  • 2
    \$\begingroup\$ To clarify dmckee's comment, any considerate web crawler should find and honor the robots.txt document for any domain it crawls. One of my college courses used this as an assignment and consistantly got the campus network blacklsted from several major sites a few times a year. See robotstxt.org \$\endgroup\$
    – captncraig
    Mar 28, 2012 at 18:37
  • \$\begingroup\$ Oh, I never actually thought about that. So do you think this question should be deleted? \$\endgroup\$
    – 3aw5TZetdf
    Mar 28, 2012 at 19:40
  • 1
    \$\begingroup\$ @CeilingSpy: What's about updating it? Or flag it for closing by a mod, and discuss it in chat or at the meta sandbox, which is meant for that, until clarification. It can be reopened later. \$\endgroup\$ Mar 29, 2012 at 1:46

1 Answer 1


Javascript, 491


var C=function(c){w=c.length;for(i=0;i<w;i++){var b=new XMLHttpRequest;b.setRequestHeader("Access-Control-Allow-Origin","*");b.setRequestHeader("X-Requested-With","XMLHttpRequest");b.open("GET",c[i],!0);b.send();if(200===b.status){a=b.responseText.match(/\<[(a|img)]+\s[^\>]+[(http|https)]+:\/\/([\w\.])+([.])+.+?[^.]+?./gi);u=c[i].substr(7);re=RegExp(u,"i");for(j=0;j<a.length;j++)if(a[j].match(re)&&(b=a[j].match(/[(http|https)]+:\/\/([\w\.])+([.])/gi))&&!c.indexOf(b))c.push(a[j]),w++}}};


var C = function(s){
  w = s.length;
  for(i = 0; i < w; i++){
    var r = new XMLHttpRequest();
    r.setRequestHeader("Access-Control-Allow-Origin", "*")
    r.setRequestHeader("X-Requested-With", "XMLHttpRequest");    
    r.open('GET', s[i], true); r.send();
    if(r.status === 200) {
      a = r.responseText.match(/\<[(a|img)]+\s[^\>]+[(http|https)]+:\/\/([\w\.])+([.])+.+?[^.]+?./gi);
      u = s[i].substr(7),
      re = new RegExp(u,'i');
      for(j = 0; j < a.length; j++){
        if(!a[j].match(re)) continue;
        var uri = a[j].match(/[(http|https)]+:\/\/([\w\.])+([.])/gi);
        if(uri && !s.indexOf(uri)) {
          s.push(a[j]); w++;


var c = new C(['http://www.cnn.com', 'http://www.usatoday.com', 'http://www.huffingtonpost.com', 'http://www.drudgereport.com/', 'http://news.cnet.com']);
  • \$\begingroup\$ How did you run the above script? I've tried Firefox and Node.js, both fail for different reasons. \$\endgroup\$
    – AnnanFay
    Nov 9, 2015 at 16:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.