librelist archives

CTL characters allowed in header field values?

From:

Eric Wong

Date:

2011-07-13 @ 01:42

This affects Mongrel 1, but it appears to affect Mongrel2 as well from
reading the code and I'm posting here because the Mongrel 1 dev list is
very idle.

I'm wondering if there was any particular reason CTL chars never got
rejected in HTTP header field values for Mongrel.  My understanding of
RFC2616 says they should be rejected (except "\t" which is LWS).  I've
been seeing headers with \x00 hit my server, so at least some (bad)
clients are sending those requests to my server.

Anyways my patch for Unicorn (which was forked from Mongrel 1 long ago)
is below:

diff --git a/ext/unicorn_http/unicorn_http_common.rl 
b/ext/unicorn_http/unicorn_http_common.rl
index cf93fec..cc1d455 100644
--- a/ext/unicorn_http/unicorn_http_common.rl
+++ b/ext/unicorn_http/unicorn_http_common.rl
@@ -20,6 +20,7 @@
   pchar = (uchar | ":" | "@" | "&" | "=" | "+");
   tspecials = ("(" | ")" | "<" | ">" | "@" | "," | ";" | ":" | "\\" | 
"\"" | "/" | "[" | "]" | "?" | "=" | "{" | "}" | " " | "\t");
   lws = (" " | "\t");
+  content = ((any -- CTL) | lws);
 
 # elements
   token = (ascii -- (CTL | tspecials));
@@ -50,9 +51,9 @@
 
   field_name = ( token -- ":" )+ >start_field $snake_upcase_field %write_field;
 
-  field_value = any* >start_value %write_value;
+  field_value = content* >start_value %write_value;
 
-  value_cont = lws+ any* >start_value %write_cont_value;
+  value_cont = lws+ content* >start_value %write_cont_value;
 
   message_header = ((field_name ":" lws* field_value)|value_cont) :> CRLF;
   chunk_ext_val = token*;
---
Thanks for reading!
-- 
Eric Wong

On Wed, Jul 13, 2011 at 01:42:27AM +0000, Eric Wong wrote:
> This affects Mongrel 1, but it appears to affect Mongrel2 as well from
> reading the code and I'm posting here because the Mongrel 1 dev list is
> very idle.
> 
> I'm wondering if there was any particular reason CTL chars never got
> rejected in HTTP header field values for Mongrel.  My understanding of
> RFC2616 says they should be rejected (except "\t" which is LWS).  I've
> been seeing headers with \x00 hit my server, so at least some (bad)
> clients are sending those requests to my server.

You know, I vaguely remember that some broken client was doing that and
I had to allow any.  I'll include these changes and see what comes of
it, because frankly anyone doing that now should just go to hell.

Do you have any idea what clients were doing that?

-- 
Zed A. Shaw
http://zedshaw.com/

"Zed A. Shaw" <zedshaw@zedshaw.com> wrote:
> On Wed, Jul 13, 2011 at 01:42:27AM +0000, Eric Wong wrote:
> > This affects Mongrel 1, but it appears to affect Mongrel2 as well from
> > reading the code and I'm posting here because the Mongrel 1 dev list is
> > very idle.
> > 
> > I'm wondering if there was any particular reason CTL chars never got
> > rejected in HTTP header field values for Mongrel.  My understanding of
> > RFC2616 says they should be rejected (except "\t" which is LWS).  I've
> > been seeing headers with \x00 hit my server, so at least some (bad)
> > clients are sending those requests to my server.
> 
> You know, I vaguely remember that some broken client was doing that and
> I had to allow any.  I'll include these changes and see what comes of
> it, because frankly anyone doing that now should just go to hell.

Agreed :)

> Do you have any idea what clients were doing that?

I've only seen them come from one IP with the same User-Agent string, so
I think it's just somebody who wrote a bot and faked the UA (of an
otherwise common browser) incorrectly.

I can't reveal more without breaking confidentiality agreements.

-- 
Eric Wong