This page describes some of the work I did in the early days of the World Wide Web (1993-1995). In many of these cases the emphasis was on providing services to make existing data collections available to web users.

The Robots Exclusion Protocol

I developed the `robots.txt` "Robots Exclusion Protocol", currently used by over 500 million websites. See and Robots.txt is 25 years old for the history.

The LWP libwww-perl library

Back in 1994/1995 I restructured Roy Fielding's libwww-perl as a Perl 5 object-oriented library, collaborated on it with Gisle Aas (see history), before he took over ongoing development and maintenance. The library is still around (and popular) and there is a book.


I contributed in the HTTP 1.1 specification development, and am quite chuffed to see my name in RFC2616. But I can't actually recall what I did to warrant that, I don't think it was anything major.