Tuesday, July 21, 2009

open-uri in ruby doesn't support multiline headers

Working with recipes in Automateit, I've released that the download() method fails when I try to download and install jetty from its repository server.
After some frustration, and some time tcpdumping and diving in the ruby sources, I realised that open-uri is raising an exception because the jetty mirror I'm using is sending a multi line header response. This is the server's response with the problematic header:

$ wget -S http://eclipse.ialto.org/jetty/7.0.0.M4/dist/jetty-distribution-7.0.0.M4.zip
HTTP/1.0 200 OK
Date: Tue, 21 Jul 2009 10:01:39 GMT
Server: Eclipse mirror service provided by <a style='font-weight: bold; color: [...]

Length: 7735180 (7.4M) [application/zip]
Saving to: `jetty-distribution-7.0.0.M4.zip'

At the beginning I suspected that this was an illegal response, but it is not. RFC 2616 in the message headers specification says that 'values for headers can contain new lines if they begin with a space or tabulator character'.

As you can see in this piece of code, the problem is that ruby reads the response line by line, and checks whether each line is valid, even though it has to read blocks ending with a new-line and a non space character, and do the check.

file: net/http.rb
lines: 2032-2040

def each_response_header(sock)
while true
line = sock.readuntil("\n", true).sub(/\s+\z/, '')
break if line.empty?
m = /\A([^:]+):\s*/.match(line) or
puts HTTPBadResponse, 'wrong header line format'
yield m[1], m.post_match if (m)

I've opened a bug with this problem in the ruby ticket system. Now it is time to know how long these tickets take to be fixed :-)