161
\$\begingroup\$

Stroustrup has recently posted a series of posts debunking popular myths about C++. The fifth myth is: “C++ is for large, complicated, programs only”. To debunk it, he wrote a simple C++ program downloading a web page and extracting links from it. Here it is:

#include <string>
#include <set>
#include <iostream>
#include <sstream>
#include <regex>
#include <boost/asio.hpp>

using namespace std;

set<string> get_strings(istream& is, regex pat)
{
    set<string> res;
    smatch m;
    for (string s; getline(is, s);)  // read a line
        if (regex_search(s, m, pat))
            res.insert(m[0]);              // save match in set
    return res;
}

void connect_to_file(iostream& s, const string& server, const string& file)
// open a connection to server and open an attach file to s
// skip headers
{
    if (!s)
        throw runtime_error{ "can't connect\n" };

    // Request to read the file from the server:
    s << "GET " << "http://" + server + "/" + file << " HTTP/1.0\r\n";
    s << "Host: " << server << "\r\n";
    s << "Accept: */*\r\n";
    s << "Connection: close\r\n\r\n";

    // Check that the response is OK:
    string http_version;
    unsigned int status_code;
    s >> http_version >> status_code;

    string status_message;
    getline(s, status_message);
    if (!s || http_version.substr(0, 5) != "HTTP/")
        throw runtime_error{ "Invalid response\n" };

    if (status_code != 200)
        throw runtime_error{ "Response returned with status code" };

    // Discard the response headers, which are terminated by a blank line:
    string header;
    while (getline(s, header) && header != "\r")
        ;
}

int main()
{
    try {
        string server = "www.stroustrup.com";
        boost::asio::ip::tcp::iostream s{ server, "http" };  // make a connection
        connect_to_file(s, server, "C++.html");    // check and open file

        regex pat{ R"((http://)?www([./#\+-]\w*)+)" }; // URL
        for (auto x : get_strings(s, pat))    // look for URLs
            cout << x << '\n';
    }
    catch (std::exception& e) {
        std::cout << "Exception: " << e.what() << "\n";
        return 1;
    }
}

Let's show Stroustrup what small and readable program actually is.

  1. Download http://www.stroustrup.com/C++.html
  2. List all links:

    http://www-h.eng.cam.ac.uk/help/tpl/languages/C++.html
    http://www.accu.org
    http://www.artima.co/cppsource
    http://www.boost.org
    ...
    

You can use any language, but no third-party libraries are allowed.

Winner

C++ answer won by votes, but it relies on a semi-third-party library (which is disallowed by rules), and, along with another close competitor Bash, relies on a hacked together HTTP client (it won't work with HTTPS, gzip, redirects etc.). So Wolfram is a clear winner. Another solution which comes close in terms of size and readability is PowerShell (with improvement from comments), but it hasn't received much attention. Mainstream languages (Python, C#) came pretty close too.

\$\endgroup\$
  • 43
    \$\begingroup\$ To each his own, I've been called worse. If the OP's goal wasn't to try and somehow prove that Stroustrup is wrong, then I'd agree with your assessment. But the entire premise of the question is to show how "your favorite language" can do the same thing as this 50 lines of C++ in much less lines of code. The problem is that none of the examples do the same thing. In particular, none of the answers perform any error checking, none of the answers provide reusable functions, most of the answers don't provide a complete program. The Stroustrup example provides all of that. \$\endgroup\$ – Dunk Jan 8 '15 at 15:40
  • 19
    \$\begingroup\$ What's sad is his web page isn't even valid UTF-8. Now I've gotta work around that, despite his server advertising Content-Type: text/html; charset=UTF-8... I'm gonna email him. \$\endgroup\$ – Cornstalks Jan 8 '15 at 16:33
  • 27
    \$\begingroup\$ @Dunk The other examples don't provide reusable functions because they accomplish the entire functionality of those functions in a single line and it makes no sense to make that a whole function on its own, and the C++ example doesn't perform any error checking that isn't handled natively in almost an identical manner, and the phrase "complete program" is almost meaningless. \$\endgroup\$ – Jason Jan 8 '15 at 21:09
  • 16
    \$\begingroup\$ "You can use any language, but no third-party libraries are allowed." I don't think that's a fair requirement considering boost/asio is used up there which is a third-party library. I mean how will languages that don't include url/tcp fetching as part of its standard library compete? \$\endgroup\$ – greatwolf Jan 9 '15 at 6:43
  • 11
    \$\begingroup\$ He's... parsing... html... with... regex... twitch \$\endgroup\$ – Riot Jan 15 '15 at 5:53

57 Answers 57

6
\$\begingroup\$

Rebol

parse read http://www.stroustrup.com/C++.html [
    any [
        thru {<a href="} copy link to {"} (print to-string link)
    ]
]
\$\endgroup\$
5
\$\begingroup\$

Delphi

program Links;
{$APPTYPE CONSOLE}
{$R *.res}
uses
  System.SysUtils, idHTTP, RegularExpressions;
var
  client: TidHTTP;
  match : TMatch;
begin
  client := TidHTTP.Create;
  try
    match := TRegEx.Create('<a(.*)href="(.*)">(.*)<\/a>', [roIgnoreCase, roMultiline])
      .match(client.Get('http://www.stroustrup.com/C++.html'));
    with match do
      while Success do
      begin
        if Groups.Count >= 3 then
          if copy(lowercase(Groups[2].Value), 1, 4) = 'http' then
              writeln(Groups[2].Value);
        match := NextMatch;
      end;
  finally
    client.Free;
  end;
end.

This would work in (I suppose) Delphi XE and later. It requires no component other than those that are already installed in a default setup (namedly indy and regular expressions). Even tho delphi is quite close to C++ in general structure, I guess this task turned out to be a bit shorter.

\$\endgroup\$
4
\$\begingroup\$

JS/jQuery

$.get('http://www.stroustrup.com/C++.html',{},function(s){
    $(s).find('a').each(function() {
        console.log($(this).attr('href'))
    })
})
\$\endgroup\$
  • 8
    \$\begingroup\$ jQuery is a third-party library. You should use plain JavaScript. \$\endgroup\$ – Athari Jan 7 '15 at 8:56
  • 6
    \$\begingroup\$ @Athari if Mr. Stroustrup allows Boost for C++, jQuery should be fine for JS :P \$\endgroup\$ – Nick T Jan 7 '15 at 20:30
  • 6
    \$\begingroup\$ This will only work if executed from the stroustrup domain since cross domain XHR requests need to be handled specifically by the server. \$\endgroup\$ – pllee Jan 7 '15 at 21:59
  • 1
    \$\begingroup\$ This answer has examples of jquey makings both easier and harder. The XHR is drastically simpler, and getting the href attribute out of the dom object is needlessly more complicated. \$\endgroup\$ – Seth Battin Jan 8 '15 at 17:21
  • 2
    \$\begingroup\$ @JLRishe I'm not bothered by the few characters; it's still perfectly readable and short. But it's also generating a complex jquery object to wrap around an already complex DOM object, and then accessing a property through a string lookup. It could get there in one step rather than...lots of steps. Consistency is great, but that kind of line makes me think it was applied by reflex. Everything gets wrapped in $(...) all the time, whether it needs it or not, i.e. jQuery because jQuery. That's the aspect I don't like. \$\endgroup\$ – Seth Battin Jan 10 '15 at 22:25
4
\$\begingroup\$

Scala

object Downloader extends App {
    val s = io.Source.fromURL("http://www.stroustrup.com/C++.html", "iso-8859-1").mkString // load URL to String
    val regex = """((http://)?www([./#\+-]\w*)+)""".r                                      // create and compile regexp
    println(regex.findAllIn(s).mkString("\n"))                                             // print matches
}
\$\endgroup\$
4
\$\begingroup\$

F#

open System.Net
open System.Text.RegularExpressions

let html = (new WebClient()).DownloadString("http://www.stroustrup.com/C++.html") in
    Regex.Matches(html, @"https?://[^""]+") |> Seq.cast<Match> |> Seq.iter (printfn "%A")

Taken from the C# version.

\$\endgroup\$
4
\$\begingroup\$

Ruby

Another Ruby solution:

require 'open-uri'
open('http://www.stroustrup.com/C++.html', 'r:iso-8859-1:utf-8') do |f|
  puts f.read.scan(%r{"(https?://www[^"]*)"}).sort
end
\$\endgroup\$
4
\$\begingroup\$

SmallTalk (Pharo 3)

Hurray for []! Also, we should totally add a Smalltalk version ;)

@Bersaelor at Objective-C answer.

I know basics of Smalltalk - syntax of language and some tutorials. I decide that it's good place for practice. I have already installed Pharo 3.0.

but no third-party libraries are allowed

Okay. On downloaded image I found Zinc-HTTP and regex packages. Probably I should read about "third-party" meaning.

So, code:

(ZnClient new get: 'http://www.stroustrup.com/C++.html')
   regex: '((http\://)?www([./#+-]\w*)+)' matchesDo: [ :x | Transcript show: String cr, x ].

Code with exception handlings:

|response|
[  
response := (ZnClient new url: 'http://www.stroustrup.com/C++.html'; get; response).
response isSuccess 
  ifFalse: [ Transcript show: 'Bad status : ', (response status asString) ]
  ifTrue:  [ response contents regex: '((http\://)?www([./#+-]\w*)+)' matchesDo: 
           [ :x | Transcript show: String cr, x ]]
]
    on: NameLookupFailure do: [ Transcript show: 'Connection problem...' ].

If you try to check me out you will get an exception. Something like UTF8EncoderException: errorIllegalLeadingByte. The first thing I thought that the package pretty outdated. But then realized that it downloads other sites well. Then I thought it is not always well copes with utf8. In debug received byte = 150 (1001 0110) - it's bad if it is the first byte in sequence. I spent a some time on the localization error (moved up in call stack and found parsed line). So:

Ehh

Lang.Next'14 Keynote: What � if anything � have we learned from C++? A 68 minute talk incl. Q&A.

You should see here squares or diamonds with question marks, depending on your browser. Trap from Stroustrup. At best, I had to write an exception handler in the package at the point where the line is created. But i just comment exception call and return ? character.

remove exception call

Also in code which calc the sequence length I return 1 in place of exception call (seems browsers does same).

P.S. some notes about Pharo (if anyone interested):

  • Too much GUI
  • Frendly for beginners.
  • Nice package searching
  • Nice highlighting and code autocompletion (not nice enough for serios IDE like IDEA but better than what I saw a few years ago in the same Pharo). Autocompletion sometimes trying deduce type - very thankless job.
  • Very chaotic GUI. At IDE we used to see tiled windows system, but there is only floating litle windows.
  • Where is imports, namespaces? There are many entities already. Is never a collision occurs?
\$\endgroup\$
4
\$\begingroup\$

Vimscript

function! Cpp()
    " grab the page in a new buffer in a new window in a new tab page
    tabedit http://www.stroustrup.com/C++.html

    " delete all lines that don't contain at least one 'http' hyperlink
    v/"http/d

    " only keep the hyperlink on every line
    %s/^.\+="\([^"]\+\)".\+$/\1
endfunction
\$\endgroup\$
3
\$\begingroup\$

Python 2

I don't like using regex on HTML for established reasons, so here's an ungolfed HTMLParser approach:

from HTMLParser import HTMLParser
import urllib2 as u

class LinkFinder(HTMLParser):
    def handle_starttag(self, tag, attrs):
        if tag=='a':
            attrdct = dict(attrs)  # attrs is a list of ('key', 'value') tuples
            if 'href' in attrdct:
                href = attrdct['href']
                print href

url = "http://www.stroustrup.com/C++.html"
contents = u.urlopen(url).read()
LinkFinder().feed(contents)

Note this also gives local links such as index.html and anchors à la #learning. If you only want absolute links, prepend print href by

                if ':/' in href:  # also handles ftp, https etc.
                    print href

while for only omitting the anchors, use

                if !href.startswith('#'):
                    print href
\$\endgroup\$
3
\$\begingroup\$

VBScript in Windows Script Host

That is, if this is stored in links.vbs file, run it via cscript /nologo links.vbs.

sub writeline( s ): WScript.StdOut.WriteLine s : end sub
function re( s ): set re = new RegExp: re.pattern = s: re.global = true: end function

set http = createobject( "Msxml2.XMLHTTP" )
http.open "GET", "http://www.stroustrup.com/C++.html", false: http.send
set links = re( "\w+://[^\""]+" ).execute( http.responseText )
for each link in links: writeline( link ): next

Addendum:

While the above lists all full links, which seems to be the goal, Stroustrup’s code additionally pares it down to unique links, and here’s a version that does that:

sub writeline( s ): WScript.StdOut.WriteLine s : end sub
function re( s ): set re = new RegExp: re.pattern = s: re.global = true: end function

set http = createobject( "Msxml2.XMLHTTP" )
http.open "GET", "http://www.stroustrup.com/C++.html", false: http.send
set links = re( "\w+://[^\""]+" ).execute( http.responseText )

set unique_links = createobject( "Scripting.Dictionary" )
on error resume next
for each link in links: unique_links.add ucase(link), link & "": next
for each link in unique_links.items(): writeline( link ): next

This reduces the number of output lines from 81 to 77.

\$\endgroup\$
3
\$\begingroup\$

MATLAB

Is quite straightforward with urlread and regexp:

url = 'http://www.stroustrup.com/C++.html';
links = regexp(urlread(url), '<a href="http://([^"]*\.*)">', 'tokens');
\$\endgroup\$
3
\$\begingroup\$

Lua

Here's a lua solution complete with error checking and duplicate URL elimination like Stroustrups's C++ version.

Just made it in under 10 lines

local http, urlunique = require 'socket.http', {}

local body, resp, _, respmsg = http.request "http://www.stroustrup.com/C++.html"
assert(resp == 200, respmsg or resp)
for each in body:gmatch 'https?://[^%s<>"]+' do
  if not urlunique[each] then
    urlunique[each] = true
    print(each)
    end end

Here's another version using string.gsub fitting in just 6 lines!

local http, urlunique = require 'socket.http', {}

local body, resp, _, respmsg = http.request "http://www.stroustrup.com/C++.html"
assert(resp == 200, respmsg or resp)
body:gsub ('https?://[^%s<>"]+', function(r) urlunique[r] = true end)
for url in pairs(urlunique) do print(url) end
\$\endgroup\$
3
\$\begingroup\$

Rust

Here is Rust solution:

extern crate reqwest;
extern crate select;
extern crate regex;

use select::document::Document;
use select::predicate::Name;
use regex::Regex;

fn main() {
    scrape_links("http://www.stroustrup.com/C++.html");
}

fn scrape_links(url: &str) {
    let resp = reqwest::get(url).unwrap();
    assert!(resp.status().is_success());
    let re = Regex::new(r"((http://)?www([./#\+-]\w*)+)").unwrap();
    Document::from_read(resp)
        .unwrap()
        .find(Name("a"))
        .filter_map(|n| n.attr("href"))
        .filter(|text| re.is_match(text))
        .for_each(|x| println!("{}", x));
}
\$\endgroup\$
2
\$\begingroup\$

R

I'm super new to regex so I gave this my best shot... any improvements appreciated!

grep("(http)s?://.*?", readLines("http://www.stroustrup.com/C++.html"), value = T)

readLines() just dumps the HTML source into a character vector. I then used grep() to find the URLs. The problem I ran into was that HTML element tags as well as corresponding link text were included in the output. substring() could be used to trim some of them I guess but it wouldn't work in all cases. If anyone knows a better way please let me know - especially if I could use a better regex.

\$\endgroup\$
  • 2
    \$\begingroup\$ regex is not only not mandatory, it is actually a rather bad method for parsing HTML. Anyway, your code is obviously much more concise than the C++ one :) \$\endgroup\$ – Tobias Kienzler Jan 8 '15 at 13:18
2
\$\begingroup\$

Python 2.7

Some readable Python code.

import urllib2
import re
page = urllib2.urlopen("http://www.stroustrup.com/C++.html").read()
for link in re.findall('"(http[s]?://.*?)"', page):
    print link
\$\endgroup\$
2
\$\begingroup\$

(C++)--

aka C

I'm suprised no one has done C yet. The code is nice and clean.

linkfetch.c:

#include "inet_utils.h"
main(){
  char* SITE="www.stroustrup.com";
  char* PAGE="C++.html";
  char* REGX="((http://)?www([./#\\+-]\\w*)+)";
  int s = connect_to(SITE);
  FILE* f = fetch_page(s,SITE,PAGE);
  if (f) list_matches(f,REGX);
  else return printf("Can't fetch page %s/%s\n",SITE,PAGE);
  fclose(f); close(s);
}

That is, clean assuming you also write these simple utiliies:

inet_utils.h

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/socket.h>
#include <netdb.h>
#include <regex.h>

#define SZ 1024 //good default buffer size
#define ERROR(s) (puts(s)&&0)
#define VP(e) ((void*)(long)(e))

inline int connect_to(char* w){
  //:returns socket connected to website w.
  struct sockaddr_in a;
  int s=socket(AF_INET, SOCK_STREAM, 0);                       // make socket
  struct hostent* h = gethostbyname(w);                        // lookup host
  if (!h) return ERROR("No Such Host");                        // check err
  a.sin_family=AF_INET;                                        // set ip address
  a.sin_port=htons(80);                                        // port 80
  memcpy(&a.sin_addr.s_addr, h->h_addr, h->h_length);          // of host
  if (connect(s,(struct sockaddr*)&a,sizeof(a))<0)             // connect
    return ERROR("can't connect");                             // handle error
  return s;                                                    // return socket
}

inline FILE* fetch_page(int s, char*w, char* p){
  //:returns open file handle for page p from site w connected to socket s

  FILE*f=fopen("/tmp/wcache","w+");                   // create cache file
  size_t n; char*b=malloc(SZ);                        // allocate temp buffer
  if (!s||!f||!b) return VP(ERROR("Resource Error")); // check for errors
  sprintf(b,                                          // compose request
          "GET / HTTP/1.0\r\nHost:%s\r\nAccept:*/*\r\nConnection:close\r\n\r\n",
          w);
  send(s,b,strlen(b),0);                         // send request
  while ((n=recv(s,b,SZ,0))>0)                   // receive response
    fwrite(b,1,n,f);                             // write it to cache file
  fseek(f,n=0,SEEK_SET);                         // read from beginnng
  fgets(b,SZ,f);                                 // look at first line
  if (!f||strncmp(strtok(b," "),"HTTP/",5))      // is it http?
    return VP(ERROR("Invalid Response"));        // error if not
  if (atoi(strtok(0," "))!=200)                  // check good status code
    return VP(ERROR("Bad Status Code"));         // error if not
  while (getline(&b,&n,f)>=0 && *b!='\r');       // skip headers upto blank line
  free(b);                                       // cleanup
  return f;                                      // return open handle
}

inline void list_matches(FILE* f, char* regx){
  //prints all strings from f which match regx
  regex_t r;
  size_t n = SZ; char*b=malloc(n);               // temp buffer
  if (regcomp(&r,regx,REG_NOSUB|REG_EXTENDED))   // compile regex
    puts("invalid regex");                       // handle error
  else while (getline(&b,&n,f)>0)                // fetch line
         if (!regexec(&r,b,0,0,0))               // check match
           puts(b);                              // show match
  regfree(&r); free(b);                          // cleanup
}

It compiles without warnings on out-of-the-box gcc (version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) )

\$\endgroup\$
  • \$\begingroup\$ Whether the code readable is highly questionble, considering weird formatting. Comments are nice, but how about properly formatting the code? You don't normally put multiple declarations with assignments, if and return on a single line, do you? \$\endgroup\$ – Athari Jan 16 '15 at 2:11
  • \$\begingroup\$ Clean and readable special purpose C usually comes by hiding ugly general purpose APIs and memory management details in "here be dragons" wrapper functions. I could have prettified the 2nd part, but it still wouldn't be nice. So I went for condensed instead. \$\endgroup\$ – AShelly Jan 16 '15 at 14:19
  • \$\begingroup\$ This is ridiculous. Tell Linus about C code being unreadable by design. Properly formatted code is much easier to understand, there's nothing "magic" in your mess besides crazy formatting. pastebin.com/WHrqJzXb \$\endgroup\$ – Athari Jan 16 '15 at 15:51
  • \$\begingroup\$ Fine, here's some formatting. I didn't say C is unreadable by design, I said general purpose APIs are usually less readable than special purpose ones. And I point to the gymnastics needed to populate the address for connecting to a web host as an example. \$\endgroup\$ – AShelly Jan 16 '15 at 16:15
2
\$\begingroup\$

Swift (2.2)

What? Please write the Swift version. That square bracket nonsense is hurting my eyes :)

Mister Smith on the Objective-C answer.

let x = try!NSString(contentsOfURL:NSURL(string:"http://www.stroustrup.com/C++.html")!,encoding:4)
for y in try!NSRegularExpression(pattern:"\"((http)s?://.*?)\"",options:[]).matchesInString(x as String,options:[],range:NSMakeRange(0,x.length)){print(x.substringWithRange(y.range))}

Ungolfed:

let url = NSURL(string: "http://www.stroustrup.com/C++.html")!
let html = try! NSString(contentsOfURL: url, encoding: NSUTF8StringEncoding)
let regex = try! NSRegularExpression(pattern: "\"((http)s?://.*?)\"", options: [])
let results = regex.matchesInString(html as String, options: [], range: NSMakeRange(0, html.length))
for result in results {
    print(html.substringWithRange(result.range))
}

Assumes Foundation has been implicitly imported.

I can't access the original page for some reason, so this was tested with the Google Cached version: http://webcache.googleusercontent.com/search?q=cache:USk4BseSofcJ:www.stroustrup.com/C%2B%2B.html+&cd=1&hl=en&ct=clnk&gl=us

282 bytes. Slightly shorter than the 292 byte Objective-C answer. I'm falling back onto Foundation APIs, so there may be room for improvement by using pure Swift types. The Cocoa APIs have also changed since the Objective-C answer was posted.

stringWithContentsOfURL: has been deprecated on NSString in favor of stringWithContentsOfURL:encoding:error:. We lose some bytes on the encoding parameter, but gain some back because the ErrorPointer is no longer passed in with Swift. The function now throws its NSError so instead, I'm using try! to force the execution of the NSString and NSRegularExpression initializers. I also save some bytes by passing the raw value 4 as the value of the encoding parameter instead of the constant NSUTF8StringEncoding. 19 bytes saved. But I lose some bytes by having to pass in an empty array ([]) instead of 0 to represent no options. 2 bytes lost there. I also lose two bytes for every variable declaration since Swift requires whitespace characters on either side of the = character.

I lose 10 bytes by having to cast the NSString as a Swift String when calling matchesInString. This is required because I'm using the NSString method contentsOfURL to get the web page HTML, but the NSRegularExpression method matchesInString takes in a Swift String as a parameter. The implicit conversion between NSString and String isn't available here, so I am forced to use as to explicitly convert the types.

Interestingly enough matchesInString has not been completely converted to use Swift types. It still requires its range parameter to be an NSRange struct instead of a Swift Range<String>. I have to fall back and use NSMakeRange to create the range of the string. I could save 4 bytes by using x.characters.indices of type Range<String.CharacterView.Index> instead, but Swift Range structs are not compatible with Foundation NSRange structs. Additionally, if x were a Swift String, I might be able to save a few bytes by replacing substringWithRange with a subscript on String. I haven't found a great way to do that yet, as creating two Index structs is currently longer than using substringWithRange.

\$\endgroup\$
  • \$\begingroup\$ Welcome to Programming Puzzles and Code Golf. Great answer, well explained, I'm not sure you need any help at all. Quick tip though: the convention on this site is to put your byte count in the title, just after the language name. \$\endgroup\$ – wizzwizz4 Jun 1 '16 at 15:52
  • \$\begingroup\$ Oh, I've just noticed: This isn't a code-golf: its objective is to "show Stroustrup what small and readable code is". If you could provide an ungolfed (readable) version, that would be very helpful. \$\endgroup\$ – wizzwizz4 Jun 1 '16 at 15:53
  • \$\begingroup\$ @wizzwizz4 Thank you for your comments. I didn't see the byte count next to the language name in the other answers, so I just added mine below. I've also added an ungolfed version to my answer. \$\endgroup\$ – JAL Jun 1 '16 at 15:56
  • \$\begingroup\$ The reason you didn't see a byte count is because this is not a code-golf challenge. \$\endgroup\$ – wizzwizz4 Jun 1 '16 at 15:57
  • 1
    \$\begingroup\$ Everybody is, even me. Even Dennis! \$\endgroup\$ – wizzwizz4 Jun 1 '16 at 15:58
1
\$\begingroup\$

ColdFusion

(using the same regex that Stroustrup uses)

<cfhttp url="http://www.stroustrup.com/C++.html" result="response" />
<cfif response.statusCode does not contain "200">
    <cfset writeOutput("Error getting the page: #response.statusCode#") />
<cfelse>
    <cftry>
        <cfset htmlLinks = REMatchNoCase("((http://)?www([./#\+-]w*)+)",response.fileContent) />
        <cfdump var="#htmlLinks#" />
    <cfcatch>
        <cfset writeOutput("There was a problem: #cfcatch.message# #cfcatch.detail#") />
    </cfcatch>
    </cftry>
</cfelse>
\$\endgroup\$
  • 1
    \$\begingroup\$ @Athari how do you do the code highlighting? \$\endgroup\$ – Matt Gutting Jan 9 '15 at 18:06
  • \$\begingroup\$ See Markdown help - Syntax highlighting for code. The syntax is usually <!-- language: lang-$LANGUAGE_NAME$ --> before the code block. If it doesn't work (support for many languages is missing), I look what CSS class is applied on StackOverflow to code blocks in questions for that language. \$\endgroup\$ – Athari Jan 9 '15 at 18:13
1
\$\begingroup\$

XQuery

HTML should be something similar to XML, so why not use langauges designed for this job?

If the page would've been "real" XHTML, we could run a query as beautiful as

doc("http://www.stroustrup.com/C++.html")//a/@href/data()

As this is crappy, broken HTML, let's use the BaseX-specific HTML parser (BaseX is an XQuery implementation):

html:parse(fetch:binary("http://www.stroustrup.com/C++.html"))//a/@href/data()

If limiting to URLs starting with http: is a must, lets do it:

html:parse(fetch:binary("http://www.stroustrup.com/C++.html"))//a/@href[starts-with(., 'http:')]/data()

Disclaimer: I am somewhat affiliated with the BaseX team as I wrote some code during my thesis. This would've been the tool of my choice for that kind of task, anyway. Other XQuery implementations provide similar HTML parsing capabilities, but I don't know their XQuery extensions by heart.

\$\endgroup\$
1
\$\begingroup\$

Bash + AWK

wget -q -O http://www.stroustrup.com/C++.html \
|awk '/((http:\/\/)?www([./#\+-]\w*)+)/ {print gensub(/.*((http:\/\/)?www([./#\+-]\w*)+).*/,"\\1","g")}'

I know it probably misses a few URLs, but I chosen to use the same regex than the original Stroustrup's code, so this should returns the same output than the original piece of code.

It may be possible to add some CR to make it more visible but I don't have a Linux available ATM for testing it works... (tested on Windows)

Here is a "clean" version of the AWK part:

/((http:\/\/)?www([./#\+-]\w*)+)/ {
    print gensub(/.*((http:\/\/)?www([./#\+-]\w*)+).*/,"\\1","g")
}
\$\endgroup\$
1
\$\begingroup\$

PHP 4.3+ / 5.0+

I know there are 2 different answers regarding PHP, but I'm going to show here a similar aproach, using nothing but standard functions.

For this, you will need to have the following on a file named php.ini ON THE SAME DIRECTORY:

allow_url_fopen= On
allow_url_include= On

THAT PART IS IMPORTANT!
In case you can't change (didn't worked with XAMPP), you have a default php.ini file on the PHP installation folder.
Changing the values will solve it.
Remember to restart apache after.

Since this isn't , I made my code somewhat readable.

Here it is:

ob_start(); //creates an output buffer

//now we 'include' the file, which will output the source code.
include 'http://www.stroustrup.com/C++.html';

$html = ob_get_clean(); //stores the output buffer and closes it

$offset = 0; //initial offset to search
$links = array(); //will contain all links

//while a link is found
while($pos = strpos($html, 'href="http', $offset))
{
    //look for the closing "
    $end = strpos($html, '"', $pos + 7);
    //take it from the string, store it into the array
    $links[] = substr($html, $pos + 6, ($end - $pos) - 6);
    //increase the offset, so it doesn't find the same link again
    $offset = $end + 1;
}

print_r($links); //spits it out, with the output buffer closed

I've added some comments to try to explain the code.

No regex used or DOM parsers: only pure hard-cold string manipulation.

For this to work in other pages, you must be sure that the values of the property href are between "", or it will fail.

\$\endgroup\$
1
\$\begingroup\$

F#

do
  use client = new System.Net.WebClient()
  let html = client.DownloadString "http://www.stroustrup.com/C++.html"
  System.Text.RegularExpressions.Regex.Matches(html, @"https?://[^""]+")
  |> Seq.cast<System.Text.RegularExpressions.Match>
  |> Seq.iter (printfn "%O")
\$\endgroup\$
0
\$\begingroup\$

CSS - idea

In Firefox, for example, for any page you are on, you can go to Tools | Web Developer | Style Editor, and use CSS to display anchors only:

* {display:none;}
a {display:block;}

However, the above will not work because display of parent elements overrides children.

Still working on a CSS solution, but suggestions welcome!

\$\endgroup\$
  • 1
    \$\begingroup\$ Maybe using positioning? Push everything off the page to the left, then push links back on to the right? \$\endgroup\$ – Izkata Jan 8 '15 at 22:38
  • \$\begingroup\$ @Izkata - looks like I've been scooped by Athari who provided a CSS solution! \$\endgroup\$ – user15259 Jan 9 '15 at 15:11
  • \$\begingroup\$ You could use "* {font-size:0pt} a {font-size:8pt}" to display only the links, but you won't see the URLs \$\endgroup\$ – LeFauve Jan 11 '15 at 3:25
  • \$\begingroup\$ It seems Athari nailed it :) \$\endgroup\$ – LeFauve Jan 11 '15 at 3:26
0
\$\begingroup\$

C++/U++

Here is the U++ (which is 'another' C++ library, slightly different approach from boost) version:

#include <Core/Core.h>
#include <plugin/pcre/Pcre.h>

using namespace Upp;

CONSOLE_APP_MAIN
{
    String s = ToCharset(CHARSET_UTF8, HttpRequest("http://www.stroustrup.com/C++.html").Execute(),
                         CHARSET_ISO8859_1);
    RegExp x("href *= *\"(.*?)\"");
    while(x.GlobalMatch(s))
        Cout() << x.GetStrings()[0] << "\n";
}

(BTW, funny how most versions here are IMO not implementing the original specification: extract all links. There are many links on that page that do not start with http...)

\$\endgroup\$
  • \$\begingroup\$ Kudos, I just posted the same below (deleted now), and equally tried to reply to the original post ( BTW: ultimatepp.org/forums/… ) but it won't let me to post links as a first message. \$\endgroup\$ – user1284631 Jan 9 '15 at 13:49
0
\$\begingroup\$

Javascript / DOM

var r = new XMLHttpRequest();
r.open('GET', 'http://www.stroustrup.com/C++.html', false);
r.send(null);
var div = document.createElement('div');
div.innerHTML = r.response;
[].slice.call(div.getElementsByTagName('a')).forEach( function(a) {
  if (a.href && a.getAttribute('href').charAt(0) != '#') console.log(a.href);
} );

(Tested in the Google Chrome javascript console.)

Of course, CORS blocks this by default - but its easy to disable CORS in Google Chrome for development purposes.

\$\endgroup\$
  • \$\begingroup\$ r.response is a string. You have to set r.responseType = "document" first. But then, you get an error because the request is synchronous. \$\endgroup\$ – gilly3 Jan 15 '15 at 19:07
  • \$\begingroup\$ Thanks @gilly3. I've modified it to take the text response and put it in a div's innerHTML instead. When running the browser with CORS disabled, this definitely now works. \$\endgroup\$ – Usas Jan 16 '15 at 4:11
0
\$\begingroup\$

Lua

More concise, without checking for errors or unique URLs: In readable...

http = require 'socket.http'
page = http.request 'http://www.stroustrup.com/C++.html'
page:gsub('http://www[%w./#\\+-]+', print)

...and less-readable...

require'socket.http'.request'http://www.stroustrup.com/C++.html':gsub('http://www[%w./#\\+-]+',print)
\$\endgroup\$
-1
\$\begingroup\$

Bash

 GET http://www.stroustrup.com/C++.html | grep -o "https\?://[^ \"]\+

needs libwww-perl and grep packages :)

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.