161
\$\begingroup\$

Stroustrup has recently posted a series of posts debunking popular myths about C++. The fifth myth is: “C++ is for large, complicated, programs only”. To debunk it, he wrote a simple C++ program downloading a web page and extracting links from it. Here it is:

#include <string>
#include <set>
#include <iostream>
#include <sstream>
#include <regex>
#include <boost/asio.hpp>

using namespace std;

set<string> get_strings(istream& is, regex pat)
{
    set<string> res;
    smatch m;
    for (string s; getline(is, s);)  // read a line
        if (regex_search(s, m, pat))
            res.insert(m[0]);              // save match in set
    return res;
}

void connect_to_file(iostream& s, const string& server, const string& file)
// open a connection to server and open an attach file to s
// skip headers
{
    if (!s)
        throw runtime_error{ "can't connect\n" };

    // Request to read the file from the server:
    s << "GET " << "http://" + server + "/" + file << " HTTP/1.0\r\n";
    s << "Host: " << server << "\r\n";
    s << "Accept: */*\r\n";
    s << "Connection: close\r\n\r\n";

    // Check that the response is OK:
    string http_version;
    unsigned int status_code;
    s >> http_version >> status_code;

    string status_message;
    getline(s, status_message);
    if (!s || http_version.substr(0, 5) != "HTTP/")
        throw runtime_error{ "Invalid response\n" };

    if (status_code != 200)
        throw runtime_error{ "Response returned with status code" };

    // Discard the response headers, which are terminated by a blank line:
    string header;
    while (getline(s, header) && header != "\r")
        ;
}

int main()
{
    try {
        string server = "www.stroustrup.com";
        boost::asio::ip::tcp::iostream s{ server, "http" };  // make a connection
        connect_to_file(s, server, "C++.html");    // check and open file

        regex pat{ R"((http://)?www([./#\+-]\w*)+)" }; // URL
        for (auto x : get_strings(s, pat))    // look for URLs
            cout << x << '\n';
    }
    catch (std::exception& e) {
        std::cout << "Exception: " << e.what() << "\n";
        return 1;
    }
}

Let's show Stroustrup what small and readable program actually is.

  1. Download http://www.stroustrup.com/C++.html
  2. List all links:

    http://www-h.eng.cam.ac.uk/help/tpl/languages/C++.html
    http://www.accu.org
    http://www.artima.co/cppsource
    http://www.boost.org
    ...
    

You can use any language, but no third-party libraries are allowed.

Winner

C++ answer won by votes, but it relies on a semi-third-party library (which is disallowed by rules), and, along with another close competitor Bash, relies on a hacked together HTTP client (it won't work with HTTPS, gzip, redirects etc.). So Wolfram is a clear winner. Another solution which comes close in terms of size and readability is PowerShell (with improvement from comments), but it hasn't received much attention. Mainstream languages (Python, C#) came pretty close too.

\$\endgroup\$
  • 43
    \$\begingroup\$ To each his own, I've been called worse. If the OP's goal wasn't to try and somehow prove that Stroustrup is wrong, then I'd agree with your assessment. But the entire premise of the question is to show how "your favorite language" can do the same thing as this 50 lines of C++ in much less lines of code. The problem is that none of the examples do the same thing. In particular, none of the answers perform any error checking, none of the answers provide reusable functions, most of the answers don't provide a complete program. The Stroustrup example provides all of that. \$\endgroup\$ – Dunk Jan 8 '15 at 15:40
  • 19
    \$\begingroup\$ What's sad is his web page isn't even valid UTF-8. Now I've gotta work around that, despite his server advertising Content-Type: text/html; charset=UTF-8... I'm gonna email him. \$\endgroup\$ – Cornstalks Jan 8 '15 at 16:33
  • 27
    \$\begingroup\$ @Dunk The other examples don't provide reusable functions because they accomplish the entire functionality of those functions in a single line and it makes no sense to make that a whole function on its own, and the C++ example doesn't perform any error checking that isn't handled natively in almost an identical manner, and the phrase "complete program" is almost meaningless. \$\endgroup\$ – Jason Jan 8 '15 at 21:09
  • 16
    \$\begingroup\$ "You can use any language, but no third-party libraries are allowed." I don't think that's a fair requirement considering boost/asio is used up there which is a third-party library. I mean how will languages that don't include url/tcp fetching as part of its standard library compete? \$\endgroup\$ – greatwolf Jan 9 '15 at 6:43
  • 11
    \$\begingroup\$ He's... parsing... html... with... regex... twitch \$\endgroup\$ – Riot Jan 15 '15 at 5:53

57 Answers 57

115
\$\begingroup\$

Wolfram

This feels like complete cheating

Import["http://www.stroustrup.com/C++.html", "Hyperlinks"]

So just add some honest parsing on top

Cases[
 Import["http://www.stroustrup.com/C++.html", "XMLObject"],
 XMLElement["a", {___, "href" -> link_, ___}, ___] :> 
  link /; StringMatchQ[link, RegularExpression["((http://)?www([./#\\+-]\\w*)+)"]]
, Infinity]
\$\endgroup\$
  • 49
    \$\begingroup\$ Nope, I don't see any cheating here. This challenge is about bringing out the best of your language. And that first line is the epitome of "small and readable". \$\endgroup\$ – Martin Ender Jan 7 '15 at 19:01
  • \$\begingroup\$ An answer that can ignore the silly arguments about catching ftp links. Brilliant. \$\endgroup\$ – Seth Battin Jan 8 '15 at 17:11
  • \$\begingroup\$ Came here to offer this exact solution, pleased to see others have appreciated it as well. \$\endgroup\$ – Michael Stern Jan 11 '15 at 15:28
  • \$\begingroup\$ @MartinBüttner In that case you might want to consider downvoting meta.codegolf.stackexchange.com/a/1078/12130 \$\endgroup\$ – David Mulder Jan 12 '15 at 15:56
  • 6
    \$\begingroup\$ @DavidMulder Technically, the loophole is currently not valid, since the vote breakdown is +41/-21 (and the loophole question states that loopholes are accepted if there are at least twice as many upvotes as downvotes). A close call, admittedly, but still. ;) Furthermore, this is a popularity contest, not a code golf, and in particular, it's a pop-con about showing how easily this can be done in a given language, which is why I think the loophole doesn't really apply to this challenge anyway (since the challenge basically asks for it). \$\endgroup\$ – Martin Ender Jan 12 '15 at 16:06
115
\$\begingroup\$

C++

#include <boost/asio.hpp>
#include <regex>
#include <iostream>
int main() {
    std::string server = "www.stroustrup.com";
    std::string request = "GET http://" + server + "/C++.html HTTP/1.0\r\nHost: " + server + "\r\n\r\n";
    boost::asio::ip::tcp::iostream s{server, "http"};
    s << request;
    std::regex pat{R"((http://)?www([./#\+-]\w*)+)"};
    std::smatch m;
    for (std::string l; getline(s, l);)
        if (std::regex_search(l, m, pat))
            std::cout << m[0] << "\n";
}

The main shortcoming is the awkward nature of boost::asio, I'm sure it can be even shorter with a better library.

\$\endgroup\$
  • 166
    \$\begingroup\$ Funny how "no third-party libraries" means Python may still import urllib2, C3 may still be using System.Net, Haskel may still import Network.HTTP, but a C++ coder must make excuses for #include <boost/asio.hpp> as if having a metric crapton of specialized, purpose-built C++ (and C!) libraries available to chose from is something to be ashamed of just because the committee didn't bother to force-feed you a specific one... \$\endgroup\$ – DevSolar Jan 8 '15 at 14:18
  • 19
    \$\begingroup\$ @DevSolar almost went for creating a 2nd account to give you another upvote for that comment \$\endgroup\$ – user Jan 8 '15 at 14:38
  • 15
    \$\begingroup\$ @DevSolar System.Net isn't forced, it's just a high-quality library following all .NET recommendations included with the language. There're alternative implementations, but having HTTP support in the standard library means writing simple apps is simple, means better interoperability between third-party libraries, means less dependencies, means easy implementation for facades etc. Imagine a world without std::string, imagine how everyone uses their own library, imagine all difficulties that come with it. \$\endgroup\$ – Athari Jan 9 '15 at 15:59
  • 17
    \$\begingroup\$ @DevSolar: urllib2 is not 3rd party. It is in stdlib like <iostream> in C++. urllib2 in Python is always available unlike <boost/asio.hpp> in C++. If we were allowed to use 3rd party modules; I would use lxml or BeautifulSoup in Python. \$\endgroup\$ – jfs Jan 11 '15 at 9:31
  • 22
    \$\begingroup\$ Also, I think the most important comment here is just that C++ doesn't standardize as much stuff in its standard libraries as other languages, but there still are widely-used robust portable libraries for a lot of the same tasks that are standard in languages like python, and some of these libs are almost a de-facto standard. And some of this is the result of C++ being able to target embedded systems with small binaries and small libraries. \$\endgroup\$ – Peter Cordes Jan 11 '15 at 14:06
85
\$\begingroup\$

Pure Bash on Linux/OS X (no external utilities)

HTTP client software is notoriously bloated. We don't want those kinds of dependencies. Instead we can push the appropriate headers down a TCP stream and read the result. No need to call archaic utilities like grep or sed to parse the result.

domain="www.stroustrup.com"
path="C++.html"
exec 3<> /dev/tcp/$domain/80
printf "GET /$path HTTP/1.1\r\nhost: %s\r\nConnection: close\r\n\r\n" "$domain" >&3
while read -u3; do
    if [[ "$REPLY" =~ http://[^\"]* ]]; then
        printf '%s\n' "$BASH_REMATCH"
    fi
done

Meh - I suppose it could be more readable...

\$\endgroup\$
  • 1
    \$\begingroup\$ Like this one using unix file handles for the pipes. \$\endgroup\$ – javadba Jan 8 '15 at 1:52
  • 2
    \$\begingroup\$ Wow, never thought one can do this without external utils. Although it seems my bash 3.2.17 on LFS is a tiny bit obsolete so that doesn't support mapfile :) \$\endgroup\$ – Ruslan Jan 8 '15 at 7:01
  • \$\begingroup\$ @Ruslan Yep, mapfile comes with bash 4.x. The same thing is totally doable with a while read loop as well. \$\endgroup\$ – Digital Trauma Jan 8 '15 at 17:06
  • 3
    \$\begingroup\$ @Ruslan I changed it to while read instead of mapfile. More portable and more readable, I think. \$\endgroup\$ – Digital Trauma Jan 8 '15 at 17:37
  • 1
    \$\begingroup\$ Works on OS X, too! \$\endgroup\$ – Alex Cohn Jan 11 '15 at 21:23
65
\$\begingroup\$

Python 2

import urllib2 as u, re
s = "http://www.stroustrup.com/C++.html"
w = u.urlopen(s)
h = w.read()
l = re.findall('"((http)s?://.*?)"', h)
print l

Lame, but works

\$\endgroup\$
  • 9
    \$\begingroup\$ Why not chain a lot of those calls? l = re.findall('"((http)s?://.*?)"', u.urlopen(s).read()) \$\endgroup\$ – Fake Name Jan 7 '15 at 9:57
  • 13
    \$\begingroup\$ It is short but it is not idiomatic (readability counts in Python) \$\endgroup\$ – jfs Jan 7 '15 at 12:50
  • 24
    \$\begingroup\$ Hmmm...if all my code ignored errors like this example then 75% to 90% of my work would already be done on every project I work on. \$\endgroup\$ – Dunk Jan 7 '15 at 15:37
  • 20
    \$\begingroup\$ @Dunk: Suppose the example did catch some exception (e.g. from urlopen()). What should it do with such an exception, other than crash and die? If it's going to crash and die anyway, why not just let Python handle the crashing-and-dying, and leave off the exception handling altogether? \$\endgroup\$ – Kevin Jan 8 '15 at 19:51
  • 8
    \$\begingroup\$ @Dunk: If I were using somebody else's Python code, I'd much rather they not catch urlopen errors than (say) catch them and call sys.exit("something's borked!"). If they do the latter, I have to catch SystemExit, which is never fun. \$\endgroup\$ – Kevin Jan 8 '15 at 21:13
55
\$\begingroup\$

C#

using System;
using System.Net;
using System.Text.RegularExpressions;

class Program {
    static void Main() {
        string html = new WebClient().DownloadString("http://www.stroustrup.com/C++.html");
        foreach (Match match in Regex.Matches(html, @"https?://[^""]+"))
            Console.WriteLine(match);
    }
}
\$\endgroup\$
  • 4
    \$\begingroup\$ You can use var html, and probably var match to shave off a few characters. \$\endgroup\$ – Superbest Jan 7 '15 at 20:43
  • 15
    \$\begingroup\$ @Superbest I can make names single-character and get rid of html variable altogether too, but it's not what I'm after. \$\endgroup\$ – Athari Jan 7 '15 at 22:36
  • 6
    \$\begingroup\$ @Superbest not code-golf. :D \$\endgroup\$ – Kroltan Jan 7 '15 at 23:26
  • 5
    \$\begingroup\$ Well, it improves readability, too. Is there ever a reason not to use var when it won't impact code semantics? \$\endgroup\$ – Superbest Jan 8 '15 at 20:56
  • 6
    \$\begingroup\$ @Superbest: "it improves readability" is subjective. Personally, I think explicitly stating the type of the variable improves readability (usually, like in this code here). I don't want to debate this, though; I just want to point out that alternative views exist. \$\endgroup\$ – Cornstalks Jan 8 '15 at 23:55
54
\$\begingroup\$

"No third-party" is a fallacy

I think the "no third-party" assumption is a fallacy. And is a specific fallacy that afflicts C++ developers, since it's so hard to make reusable code in C++. When you are developing anything at all, even if it's a small script, you will always make use of whatever pieces of reusable code are available to you.

The thing is, in languages like Perl, Python, Ruby (to name a few), reusing someone else's code is not only easy, but it is how most people actually write code most of the time.

C++, with its nearly impossible-to-maintain-compatible-ABI-requirements makes that a much tougher job, you end up with a project like Boost, which is a monstrous repository of code and very little composability outside it.

A CPAN example

Just for the fun of it, here goes a CPAN-based example, with proper parsing of the html, instead of trying to use regex to parse html

#!/usr/bin/perl
use HTML::LinkExtor;
sub callback {
   my ($tag, %links) = @_;
   print map { "$_\n" } values %links
}
$p = HTML::LinkExtor->new(\&callback, "http://www.stroustrup.com/C++.html");
\$\endgroup\$
  • 6
    \$\begingroup\$ Upvote for addressing the point of the 3rd party libs, but: crap, making reusable code in C++ is as easy cheesy as in other language. Using and especially finding reusable code may be a tad harder, but the only thing that's seriously problematic is reusing compiled artifacts, but that's often a non-issue in interpreted languages like Perl, etc. \$\endgroup\$ – Martin Ba Jan 8 '15 at 9:47
  • 4
    \$\begingroup\$ To stretch an analogy, Boost is more like CPAN - pick and choose. You don't call CPAN a "monstrous repository of code" just because there's such a lot of stuff in there you don't use? \$\endgroup\$ – Martin Ba Jan 8 '15 at 9:49
  • 22
    \$\begingroup\$ CPAN is a 'monstrous repository of code', by any reasonable definition of those four words. \$\endgroup\$ – jwg Jan 8 '15 at 14:39
  • 3
    \$\begingroup\$ @MartinBa I disagree, C++ being a compiled language, requiring every executable to rebuild its full stack of dependencies because it's hard to maintain ABI compatibility seriously hinders code reusability. In order to produce a reusable library in C++ you have to go through really long lengths in order to make sure you don't force yourself into ABI-incompatible changes all the time. \$\endgroup\$ – Daniel Ruoso Jan 8 '15 at 16:53
  • 6
    \$\begingroup\$ @MartinBa because having to rebuild the whole universe everytime you want to implement a simple task is unbearable. \$\endgroup\$ – Daniel Ruoso Jan 8 '15 at 20:59
47
\$\begingroup\$

UNIX shell

lynx -dump http://www.stroustrup.com/C++.html | grep -o '\w*://.*'

Also finds an ftp:// link :)

Another way, without relying on :// syntax:

lynx -dump -listonly http://www.stroustrup.com/C++.html | sed -n 's/^[ 0-9.]\+//p'
\$\endgroup\$
  • 38
    \$\begingroup\$ I can't work out whether to +1 because using a web browser to download a web page is the right tool for the job or to -1 because the challenge is to write a program to do blahblahblah and you just called a program to do the blahing. \$\endgroup\$ – David Richerby Jan 7 '15 at 11:44
  • 2
    \$\begingroup\$ I think it's better to replace lynx with curl or wget. They are more commonly used to download a webpage. \$\endgroup\$ – Pavel Strakhov Jan 7 '15 at 14:03
  • 4
    \$\begingroup\$ @PavelStrakhov I chose lynx exactly because it can dump the links without me doing anything special :) \$\endgroup\$ – Ruslan Jan 7 '15 at 14:06
  • 2
    \$\begingroup\$ @SteveJessop by "special" I mean actually parsing or regexing or whatever. With lynx I just grep out the list of links (which curl and wget don't list) and remove the numbering. You may consider it cheating or whatever, but I thought it's fun to {use the tool which almost perfectly does what is required}, just fine-tuning the output. \$\endgroup\$ – Ruslan Jan 7 '15 at 15:37
  • 7
    \$\begingroup\$ "but no third-party libraries are allowed". I contend that lynx is functionally equivalent to a third-party library in this scenario. \$\endgroup\$ – Digital Trauma Jan 7 '15 at 20:32
44
\$\begingroup\$

CSS 3

* {
  margin: 0;
  padding: 0;
}
*:not(a) {
  font: 0/0 monospace;
  color: transparent;
  background: transparent !important;
}
a {
  content: "";
}
a[href*="://"]::after {
  content: attr(href);
  float: left;
  clear: left;
  display: block;
  font: 12px monospace;
  color: black;
}

This code can be used as a user style to display only absolute links on a page in an unformatted list. It may not work correctly if your browser enforces minimum font size.

It works correctly with http://www.stroustrup.com/C++.html (note !important on background). In order to work on other pages with more styles, it must be extended (reset more properties, mark properties as important etc.).

Alternative version which includes relative links except intrapage links starting with hashes (it relies on a hard-coded absolute link, unfortunately):

* {
  margin: 0;
  padding: 0;
}
*:not(a) {
  font: 0/0 monospace;
  color: transparent;
  background: transparent !important;
  float: none !important;
  width: auto !important;
  border: none !important;
}
a {
  content: "";
}
a::after {
  display: none;
}
a:not([href^="#"])::after {
  content: attr(href);
  float: left;
  clear: left;
  display: block;
  font: 12px monospace;
  color: black;
}
a:not([href*="://"])::after {
  content: "http://www.stroustrup.com/" attr(href);
}
\$\endgroup\$
  • 16
    \$\begingroup\$ This is the worst thing I've ever seen. +1 \$\endgroup\$ – Emmett R. Jan 13 '15 at 18:11
  • 1
    \$\begingroup\$ This is beautiful and completely horrifying. +1 \$\endgroup\$ – ricdesi Jun 1 '16 at 15:59
36
\$\begingroup\$

Clojure

(->> (slurp "http://www.stroustrup.com")
     (re-seq #"(?:http://)?www(?:[./#\+-]\w*)+"))
\$\endgroup\$
  • 27
    \$\begingroup\$ Slurp?! I need to learn Clojure. \$\endgroup\$ – 11684 Jan 8 '15 at 10:01
  • 10
    \$\begingroup\$ @11684 - Clojure also has standard functions named spit, zipper, and lazy-cat... :-) \$\endgroup\$ – Bob Jarvis Jan 9 '15 at 19:13
  • 2
    \$\begingroup\$ Wow, I think that's gonna be a late New Year's Resolution. @BobJarvis \$\endgroup\$ – 11684 Jan 9 '15 at 21:36
30
\$\begingroup\$

Emacs Lisp

(with-current-buffer (url-retrieve-synchronously "http://www.stroustrup.com/C++.html")
  (while (re-search-forward "https?://[^\\\"]*")
    (print (match-string 0))))
\$\endgroup\$
  • 2
    \$\begingroup\$ I'm a little dissapointed, given how compact and eminently readable this code is, that it does not have more votes. Well done. \$\endgroup\$ – Spacemoose Jan 16 '15 at 11:40
28
\$\begingroup\$

Scala

"""\"(https?://.*?)\"""".r.findAllIn(scala.io.Source.fromURL("http://www.stroustrup.com/C++.html").mkString).foreach(println)
\$\endgroup\$
  • 8
    \$\begingroup\$ pack everything in one line - C++ can do it too \$\endgroup\$ – quetzalcoatl Jan 8 '15 at 9:57
  • \$\begingroup\$ What about ftp://ftp.research.att.com/pub/c++std/WP/CD2? \$\endgroup\$ – Tobias Kienzler Jan 8 '15 at 12:19
  • 22
    \$\begingroup\$ @quetzalcoatl - This is one expression, not just one line. You can just delete all of the line breaks from the C++ code, but that's not the same thing as doing the whole task in a single expression. \$\endgroup\$ – DaoWen Jan 8 '15 at 23:56
  • 4
    \$\begingroup\$ @DaoWen: Sorry, but starting expressions-vs-line is just going silly. Add some functors and C++ you can do it too. But that's just the question of what libs are considered to be "granted" and have "zero code inside". It doesn't change the fact that packing it into a line hurst readability. One can keep it still as a single expression and just reformat it into a few lines to gain much and loose nothing other than .. line count. That's my point. Silly packing - C++ can do it too. If someone wants to get out of the "silly packing" box, then should format the code for readability, not linecount. \$\endgroup\$ – quetzalcoatl Jan 9 '15 at 11:27
  • 3
    \$\begingroup\$ @quetzalcoatl Tobias didn't put the link there for us to follow it. He was asking the writer of this answer why it wasn't in his results. \$\endgroup\$ – JLRishe Jan 9 '15 at 21:19
25
\$\begingroup\$

PHP 5

<?php
preg_match_all('/"(https?:\/\/.*?)"/',file_get_contents('http://www.stroustrup.com/C++.html'),$m);
print_r($m[1]);
\$\endgroup\$
  • 5
    \$\begingroup\$ Suggested edits: '/"((http)s?://.*?)"/''|"((http)s?://.*?)"|' (currently an error); remove array_unshift($m); (currently an error, you likely meant array_shift instead); print_r($m);print_r($m[1]); (only output the urls). \$\endgroup\$ – primo Jan 7 '15 at 10:12
  • \$\begingroup\$ fixed, thanks for your input \$\endgroup\$ – David Xu Jan 7 '15 at 17:13
  • \$\begingroup\$ @DavidXu Except you didn't fix it...? \$\endgroup\$ – Shahar Jan 7 '15 at 18:00
  • \$\begingroup\$ now its fixed.! \$\endgroup\$ – David Xu Jan 7 '15 at 19:17
25
\$\begingroup\$

PowerShell

Text search for all fully-qualified URLs (including JavaScript, CSS, etc.):

[string[]][regex]::Matches((iwr "http://www.stroustrup.com/C++.html"), '\w+://[^"]+')

Or to get links in anchor tags only (includes relative URLs):

(iwr "http://www.stroustrup.com/C++.html").Links | %{ $_.href }

Shorter versions from comments:

(iwr "http://www.stroustrup.com/C++.html").Links.href
(iwr "http://www.stroustrup.com/C++.html").Links.href-match":"
\$\endgroup\$
  • 6
    \$\begingroup\$ If anyone wonders, iwr is an alias for Invoke-WebRequest (PS3+). \$\endgroup\$ – Athari Jan 7 '15 at 23:59
  • 8
    \$\begingroup\$ You could abuse PowerShell's eagerness to flatten collections and do: (iwr "http://www.stroustrup.com/C++.html").Links.href (or (iwr "http://www.stroustrup.com/C++.html").Links.href-match":" for only absolute URI's) \$\endgroup\$ – Mathias R. Jessen Jan 8 '15 at 22:11
  • 1
    \$\begingroup\$ That's pretty handy! \$\endgroup\$ – Justin Dunlap Jan 9 '15 at 6:20
22
\$\begingroup\$

D

import std.net.curl, std.stdio;
import std.algorithm, std.regex;

void main() {
foreach(_;byLine("http://www.stroustrup.com/C++.html")
    .map!((a)=>a.matchAll(regex(`<a.*?href="(.*)"`)))
    .filter!("a")){ writeln(_.front[1]); }
}
\$\endgroup\$
  • \$\begingroup\$ To make the list similar to the original example, you could pipe the program's output through | sort | uniq or instead add import std.array and change the line .filter!("a")){ writeln(_.front[1]); } into this: .filter!("a").map!(a => a.front[1]).array.sort.uniq){ writeln(_); }. Note, however, that I have only tried this code and not proved it to be correct or "idiomatic". :) \$\endgroup\$ – Frg Jan 10 '15 at 11:58
22
\$\begingroup\$

Node.js

var http = require('http');

http.get('http://www.stroustrup.com/C++.html', function (res) {
    var data = '';
    res.on('data', function (d) {
        data += d;
    }).on('end', function () {
        console.log(data.match(/"https?:\/\/.*?"/g));
    }).setEncoding('utf8');
});
\$\endgroup\$
  • 3
    \$\begingroup\$ I wonder if require('http').get works. If it does then we can ditch the var statement and shorten another line. \$\endgroup\$ – Unihedron Jan 7 '15 at 17:46
  • \$\begingroup\$ @Unihedro It does. \$\endgroup\$ – TimWolla Jan 7 '15 at 18:22
  • 9
    \$\begingroup\$ @Unihedro It does, but this isn't a golfing contest. \$\endgroup\$ – c.P.u1 Jan 7 '15 at 19:43
  • \$\begingroup\$ You don’t need to use any capturing groups. \$\endgroup\$ – Ry- Jan 15 '15 at 0:25
  • \$\begingroup\$ I think it's JavaScript rather than a framework name. \$\endgroup\$ – mr5 Jan 17 '15 at 16:09
20
\$\begingroup\$

Ruby

require 'net/http'
result = Net::HTTP.get(URI.parse('http://www.stroustrup.com/C++.html'))
result.scan(/"((http)s?://.*?)"/)
\$\endgroup\$
  • 1
    \$\begingroup\$ Your regex will fail, you need to use %r{"(https?://[^"]+)"}. Also you can use Net::HTTP.get('www.stroustrup.com', '/C++.html') to shorten up request (and keep it readable). So whole code can be in one line (keeping it readable): puts Net::HTTP.get("www.stroustrup.com", "/C++.html").scan(%r{"(https?://[^"]+)"}). Run it with ruby -rnet/http and you don't even need require 'net/http' line. \$\endgroup\$ – Hauleth Jan 12 '15 at 22:47
20
\$\begingroup\$

Haskell

Some troubles with "\w" in Text.Regex.Posix

import Network.HTTP
import Text.Regex.Posix
pattern = "((http://)?www([./#\\+-][a-zA-Z]*)+)"
site = "http://www.stroustrup.com/C++.html"

main = do
    file <- getResponseBody =<< simpleHTTP (getRequest site)
    let result = getAllTextMatches $ file =~ pattern
    putStr $ unlines result -- looks nicer
\$\endgroup\$
  • \$\begingroup\$ Why is the type of result specified explicitly? It should be fully constrained by its use in unlines. \$\endgroup\$ – John Dvorak Jan 7 '15 at 16:25
  • 1
    \$\begingroup\$ This does stretch the rules a bit, seeing as neither Network.HTTP nor TextRegex.Posix are in the base package. (Though they are in the Haskell Platform, and of course on Hackage, so...) \$\endgroup\$ – ceased to turn counterclockwis Jan 7 '15 at 20:49
  • 1
    \$\begingroup\$ @JanDvorak, I start to write in ghci (probably i should post it unchanged). But your note is relevant, thanks. \$\endgroup\$ – vlastachu Jan 9 '15 at 12:22
  • \$\begingroup\$ @leftaroundabout, did not know. It looks like I could not have done, if had used the base package. \$\endgroup\$ – vlastachu Jan 9 '15 at 12:35
  • \$\begingroup\$ network isn't in base either, so save for rolling your own socket bindings there's no practical way to do it with just base. \$\endgroup\$ – Lambda Fairy Jan 13 '15 at 10:09
18
\$\begingroup\$

PHP

As far as I can tell, most modern PHP installations come with DOM processing, so here's one that actually traverses the anchors inside the HTML:

foreach (@DOMDocument::loadHTMLFile('http://stroustrup.com/C++.html')->getElementsByTagName('a') as $a) {
    if (in_array(parse_url($url = $a->getAttribute('href'), PHP_URL_SCHEME), ['http', 'https'], true)) {
        echo $url, PHP_EOL;
    }
}

The inner loop could be shortened to:

preg_match('~^https?://~', $url = $a->getAttribute('href')) && printf("%s\n", $url);
\$\endgroup\$
  • \$\begingroup\$ Actually wanted to come up w this (as my first answer here). You did it first, so here's your +1 (for not using an error prone Regex)! Hint: You could use a lame 1 instead of the true for the in_array strict search. You can as well omit the brackets. I'm not completely sure, but iirc you could as well drop the http and only leave the :// (go w/o the scheme). . \$\endgroup\$ – kaiser Jan 10 '15 at 12:27
  • \$\begingroup\$ And: Another possibility would be to drop the if ( ) {} in favor of in_array() and print $url.PHP_EOL. But yeah, you would get another +1 (if I could) for best readability :) \$\endgroup\$ – kaiser Jan 10 '15 at 12:30
  • \$\begingroup\$ Just tried your example and got an error for strict standards (PHP 5.4). Seems like in the source, there's somewhere a corrupted or wrongly formatted link with a missing semicolon. You could turn off error reporting by using @\DOMDocument. Just tried that and can confirm it works. \$\endgroup\$ – kaiser Jan 10 '15 at 14:10
  • \$\begingroup\$ Nah, it's the documentation that's wrong; technically you're not supposed to call ::loadHTMLFile() statically, and adding @ only hides that artefact. \$\endgroup\$ – Jack Jan 10 '15 at 14:12
  • 2
    \$\begingroup\$ This is definitely one of the most "correct" solutions, one of the only ones I could see in use in production. nice job \$\endgroup\$ – Jordon Biondo Jan 13 '15 at 14:29
14
\$\begingroup\$

Unix Shell

wget -q -O - http://www.stroustrup.com/C++.html | sed -n '/http:/s/.*href="\([^"]*\)".*/\1/p' | sort

Though i have to admit this doesn't work if there's more than one link on a line.

\$\endgroup\$
  • 1
    \$\begingroup\$ curl http://www.stroustrup.com/C++.html saves a few characters. \$\endgroup\$ – l0b0 Jan 7 '15 at 14:56
  • 7
    \$\begingroup\$ "but no third-party libraries are allowed". I guess since wget is GNU (as is bash), you could argue that it is not third-party. But curl definitely is third-party. \$\endgroup\$ – Digital Trauma Jan 7 '15 at 20:34
  • \$\begingroup\$ What about ftp://ftp.research.att.com/pub/c++std/WP/CD2 and https://www.youtube.com/watch?v=jDqQudbtuqo&feature=youtu.be? \$\endgroup\$ – Tobias Kienzler Jan 8 '15 at 12:21
  • 4
    \$\begingroup\$ @TobiasKienzler I guess Stroustrup's original code doesn't find them either \$\endgroup\$ – Ruslan Jan 8 '15 at 17:05
14
\$\begingroup\$

Java

import java.util.regex.*;
class M{
    public static void main(String[]v)throws Throwable{
        Matcher m = Pattern.compile( "\"((http)s?://.*?)\"" )
            .matcher(
                 new Scanner(
                         new URL( "http://www.stroustrup.com/C++.html" )
                             .openStream(),
                         "UTF-8")
                     .useDelimiter("\\A")
                     .next());
        while(m.find())
            System.out.println(m.group());
    }
}
\$\endgroup\$
  • 3
    \$\begingroup\$ Could you properly format code in your answers? It isn't competition for the least readable code. You can format it to avoid horizontal scrollbars at least. \$\endgroup\$ – Athari Jan 7 '15 at 9:07
  • \$\begingroup\$ If you use a Scanner you can make it processing the regex pattern for links directly and iterate over the Scanner’s results. \$\endgroup\$ – Holger Jan 7 '15 at 9:11
  • 5
    \$\begingroup\$ Yep .. that's java for you. Using it for code golf is a brave undertaking. \$\endgroup\$ – javadba Jan 8 '15 at 1:53
  • 4
    \$\begingroup\$ Never thought I'd see a java solution that's actually shorter than C++! \$\endgroup\$ – slebetman Jan 8 '15 at 2:59
  • 2
    \$\begingroup\$ Correction to my last comment: I must admit this is pretty much the shortest and cleanest code that can be written in Java. I've tried a SAX parser approach, which could be made even shorter with lambdas, but the web page is not XHTML and the parser throws exceptions. Regex is the only way to go. \$\endgroup\$ – Mister Smith Jan 9 '15 at 15:59
11
\$\begingroup\$

Groovy

"http://www.stroustrup.com/C++.html".toURL().text.findAll(/https?:\/\/[^"]+/).each{println it}
\$\endgroup\$
  • \$\begingroup\$ Could be improved by using ?. operator to avoid NPEs? \$\endgroup\$ – Chris K Jan 8 '15 at 14:31
  • 2
    \$\begingroup\$ @ChrisKaminski and be the first (beside Bjarne) around here to check for errors? never! beside that: i only see IO related exceptions here. where do you see a NPE? \$\endgroup\$ – cfrick Jan 8 '15 at 14:53
  • \$\begingroup\$ findAll() could return null, no? Or will it return an empty list? Still a bit new to Groovy. EDIT: nm, looks like findAll() returns an empty list. Those Groovy guys were so smart. :-) \$\endgroup\$ – Chris K Jan 8 '15 at 15:11
11
\$\begingroup\$

SQL (SQL Anywhere 16)

Define a stored procedure to fetch the web page

CREATE OR REPLACE PROCEDURE CPPWebPage()
URL 'http://www.stroustrup.com/C++.html'
TYPE 'HTTP';

Produce the result set using a single query

SELECT REGEXP_SUBSTR(Value,'"https?://[^""]+"',1,row_num) AS Link  
FROM (SELECT Value FROM CPPWebPage() WITH (Attribute LONG VARCHAR, Value LONG VARCHAR) 
      WHERE Attribute = 'Body') WebPage, 
      sa_rowgenerator( 1, 256 ) 
WHERE Link IS NOT NULL;

Limitations: This produces up to 256 links. If more links exist, then bump up the 256 to an appropriate value.

\$\endgroup\$
  • 2
    \$\begingroup\$ I didn't believe there would be golf in SQL... until now. \$\endgroup\$ – vaxquis Jan 11 '15 at 12:05
  • \$\begingroup\$ I get it ... "links". :-) \$\endgroup\$ – Jack at SAP Canada Jan 15 '15 at 15:31
10
\$\begingroup\$

CoffeeScript / NodeJS

require('http').get 'http://www.stroustrup.com/C++.html', (r) ->
    dt = '';
    r.on 'data', (d) -> dt += d
    r.on 'end' , (d) -> console.log dt.match /"((http)s?:\/\/.*?)"/g
\$\endgroup\$
  • 1
    \$\begingroup\$ I guess this is CoffeeScript/Node? I guess you should specify that... \$\endgroup\$ – John Dvorak Jan 7 '15 at 16:27
  • \$\begingroup\$ Wow. That's very readable. \$\endgroup\$ – slebetman Jan 8 '15 at 3:00
  • \$\begingroup\$ @slebetman it definitely is small though \$\endgroup\$ – John Dvorak Jan 8 '15 at 7:06
  • \$\begingroup\$ @slebetman Yeah CoffeeScript is so much more readable than JavaScript :) I was glad to get rid of all the curly braces }:) \$\endgroup\$ – RobAu Jan 8 '15 at 7:54
9
\$\begingroup\$

Perl

use LWP;
use feature 'say';

my $agent = new LWP::UserAgent();
my $response = $agent->get('http://www.stroustrup.com/C++.html');

say for $response->content =~ m<"(https?://.+?)">g;
\$\endgroup\$
  • 1
    \$\begingroup\$ The code would be more clear if you avoided the field-separator and record-separator variables and just did: print map { "$_\n" } $response->content =~ m<"(https?://.+?)">g; \$\endgroup\$ – Daniel Ruoso Jan 7 '15 at 16:55
  • \$\begingroup\$ @DanielRuoso agreed. \$\endgroup\$ – primo Jan 8 '15 at 6:50
  • \$\begingroup\$ or even use v5.10; and say for $response->content... \$\endgroup\$ – Mark Reed Jan 8 '15 at 18:41
  • \$\begingroup\$ To each his own, I suppose. Some of the backported perl6 features have been problematic (smart matching, I'm looking at you), but say is quite useful, and in my mind clearer here. (Also, there have been rather a lot of completely-unrelated-to-perl6ism improvements to perl5 in the last 13 years; it might be worth checking out.) \$\endgroup\$ – Mark Reed Jan 8 '15 at 20:15
  • \$\begingroup\$ @MarkReed I agree that say is probably more readable in this case, particularly for those less familiar with perl. \$\endgroup\$ – primo Jan 10 '15 at 5:26
9
\$\begingroup\$

R

html<-paste(readLines("http://www.stroustrup.com/C++.html"),collapse="\n")
regmatches(html,gregexpr("http[^([:blank:]|\\\"|<|&|#\n\r)]+",html))

...although R is written mainly in C... so probably a few lines of C code behind those 2 lines of R code.

\$\endgroup\$
  • 2
    \$\begingroup\$ That (or something similar) is true for pretty much all the answers here. \$\endgroup\$ – JLRishe Jan 8 '15 at 18:39
8
\$\begingroup\$

Objective-C

NSString *s;
for (id m in [[NSRegularExpression regularExpressionWithPattern:@"\"((http)s?://.*?)\"" options:0 error:nil] matchesInString:(s=[NSString stringWithContentsOfURL:[NSURL URLWithString:@"http://www.stroustrup.com/C++.html"]])]){
    NSLog(@"%@",[s substringWithRange:[m range]]);
}
\$\endgroup\$
  • 3
    \$\begingroup\$ What? Please write the Swift version. That square bracket nonsense is hurting my eyes :) \$\endgroup\$ – Mister Smith Jan 8 '15 at 21:09
  • 2
    \$\begingroup\$ Hurray for []! Also, we should totally add a Smalltalk version ;) \$\endgroup\$ – Bersaelor Jan 9 '15 at 9:52
  • \$\begingroup\$ @MisterSmith Swift answer now available here. \$\endgroup\$ – JAL Jun 1 '16 at 21:23
7
\$\begingroup\$

Tcl

package require http
set html [http::data [http::geturl http://www.stroustrup.com/C++.html]]
puts [join [regexp -inline -all {(?:http://)?www(?:[./#\+-]\w*)+} $html] \n]
\$\endgroup\$
  • \$\begingroup\$ You can get away by doing http::data inside the puts. No need to create a temporary variable. And I'd also format it by putting newlines and indenting at every [. But that's a style choice. \$\endgroup\$ – slebetman Jan 8 '15 at 2:58
7
\$\begingroup\$

Go

package main

import (
    "fmt"
    "io/ioutil"
    "net/http"
    "os"
    "regexp"
)

func main() {
    resp, err := http.Get("http://www.stroustrup.com/C++.html")
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
    defer resp.Body.Close()
    data, _ := ioutil.ReadAll(resp.Body)
    results := regexp.MustCompile(`https?://[^""]+`).FindAll(data, -1)
    for _, row := range results {
        fmt.Println(string(row))
    }
}

P.S. this code reads entire source into memory, so consider using regexp.FindReaderIndex to search in stream, that'll make the app bulletproof.

\$\endgroup\$
6
\$\begingroup\$

CJam

CJam does not have regex so I had to use a different approach in this one:

"http://www.stroustrup.com/C++.html"g''/'"*'"/(;2%{_"http://"#!\"https://"#!e|},N*

I first convert all ' to ", then I split on all ", take every alternative string and then finally filter that list for strings starting with http:// or https://. After that, simply print each filtered string on a new line.

Try it using the Java interpreter like

java -jar cjam-0.6.2.jar file.cjam

where file.cjam has the contents of the code above.

\$\endgroup\$
  • 9
    \$\begingroup\$ Don't know about the readable part... didn't know Cjam has web functionality \$\endgroup\$ – Def Jan 7 '15 at 12:16
  • \$\begingroup\$ If you want to golf it... ''/'"f/:+ for ''/'"*'"/'"f/0f=. \$\endgroup\$ – jimmy23013 Jan 7 '15 at 14:08
  • \$\begingroup\$ ...wait why is '"f/0f= there? Is that supposed to do something (2% for instance)? \$\endgroup\$ – jimmy23013 Jan 7 '15 at 14:14
6
\$\begingroup\$

F#

This code could be far shorter but I would write something like this if I ever expected to have to read or use this code again so it has many unnecessary type annotations. It demonstrates the use of an active pattern MatchValue to enable pattern-matching against the standard CLR type Match

open System.Net

let (|MatchValue|) (reMatch: Match) : string = reMatch.Value

let getHtml (uri : string) : string = 
    use webClient = WebClient() in
        let html : string = webClient.DownloadString(uri)
        html

let getLinks (uri : string) : string list =
    let html : string = getHtml uri
    let matches : MatchCollection = Regex.Matches(html, @"https?://[^""]+") 
    let links = [ for MatchValue reMatch in matches do yield reMatch ]
    links

let links = getLinks "http://www.stroustrup.com/C++.html" 
for link in links do
    Console.WriteLine(link)

Edit I made getLinks its own function

\$\endgroup\$
  • \$\begingroup\$ I really like how you used type annotations. I think naming values to describe what you return is ok, but name of the function is expressive enough: getHTML and html value, getLinks and links value. Last two lines may be links |> Seq.iter (printfn "%s") \$\endgroup\$ – MichalMa Jan 11 '15 at 10:36
  • \$\begingroup\$ @MichalMa I agree that the name of the function is expressive enough on its own, the html and links variables are there for pragmatic reasons: so there is somewhere to set a breakpoint. I used the for loop instead of List.iter just because I like the way it reads more, although in a repl I probably would have used List.iter. \$\endgroup\$ – SourceSimian Jan 12 '15 at 4:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.