librelist archives

home // archives //

« back to archive

Chunked transfer encoding

Chunked transfer encoding

From:
tsuraan
Date:
2010-12-27 @ 04:59 
I saw in an email from 2009 that the mongrel2 http parser has some
degree of support for chunked post, so, not knowing what to expect, I
tried it out.  The error I'm getting (in the terminal running
mongrel2) is this:

[ERROR] (src/connection.c:638: errno: Resource temporarily
unavailable) Error parsing request.

On the curl side, I'm getting "Bad Request" as a response.  My
configuration looks like this:

servers = [Server(
  uuid="f400bf85-4538-4f7a-8908-67e313d515c2",
  access_log="./logs/access.log",
  error_log="./logs/error.log",
  default_host="localhost",
  name="bob"
  chroot="/",
  pid_file="./some.pid",
  port=8080,
  hosts = [
    Host(name="localhost",
         routes={ '/' : Handler(send_spec="tcp://127.0.0.1:9998",
                                send_ident="sender",
                                recv_spec="tcp://127.0.0.0:9999",
                                recv_ident="sender")})])]

settings={'upload.temp_store' : './tmp/foo.XXXXXXXX'}

And my curl invocation (from within the mongrel2 source dir) looks like this:

curl --header 'Transfer-Encoding: chunked' -F frm=@io.c
http://localhost:8080/upload

I didn't really expect this to work, because it looks like the
Upload_file function requires a content_len which isn't specified in
chunked post, but I didn't think I'd get an error in the actual http
parsing code.  Can anybody give me a hint about what I'm going wrong?
I'd be happy to provide any other details that might be relevant.

Re: [mongrel2] Chunked transfer encoding

From:
Zed A. Shaw
Date:
2010-12-27 @ 05:49 
On Sun, Dec 26, 2010 at 10:59:48PM -0600, tsuraan wrote:
> I saw in an email from 2009 that the mongrel2 http parser has some
> degree of support for chunked post, so, not knowing what to expect, I
> tried it out.  The error I'm getting (in the terminal running
> mongrel2) is this:
> 
> [ERROR] (src/connection.c:638: errno: Resource temporarily
> unavailable) Error parsing request.

Sorry, it doesn't support chunked-encoding from the client.
Chunked-encoding from the client is error prone, not part of the standard,
and totally useless since the client knows what the size of its data
is from the start.  Allowing chunked-ecoding from the client would allow
for a range of attacks where a client could eat up resources by sending
an endless stream of chunks.

It supports it from the proxy side though.

-- 
Zed A. Shaw
http://zedshaw.com/

Re: [mongrel2] Chunked transfer encoding

From:
tsuraan
Date:
2010-12-27 @ 15:23 
> Sorry, it doesn't support chunked-encoding from the client.
> Chunked-encoding from the client is error prone, not part of the standard,
> and totally useless since the client knows what the size of its data
> is from the start.  Allowing chunked-ecoding from the client would allow
> for a range of attacks where a client could eat up resources by sending
> an endless stream of chunks.

Is that an actual attack?  The difference between chunked and normal
upload is that with normal uploads, the client tells the server the
size before it starts sending chunks, but once it's said a size, it
can send chunks of data as quickly or slowly as it wants to.  With
chunked upload, the client doesn't tell the server in advance how much
data it's going to send, but once the server's seen too much data, it
can sever the connection, right?  How would it be any different from a
client figuring out the max upload size and just slowly (I assume
that's the key to the attach) sending it that much data to tie up
resources?

The reason that I'd like chunked post is that some browsers really
don't function correctly without it.  When trying to upload a large
file (gigabytes), firefox and explorer (not sure about any of the
other browsers) use massive amounts of RAM and CPU before they start
uploading if chunked post is not available.  I have no idea why this
happens, but chunked post support hides the issue.  It I wanted to
patch in my own support and make it available as a discouraged option,
where would I start?  I can't find anything in the http11_parser.rl
that says anything at all about chunking, but from tracing the code,
it looks like the error is coming from the http parser.

Re: [mongrel2] Chunked transfer encoding

From:
Zed A. Shaw
Date:
2010-12-27 @ 17:52 
On Mon, Dec 27, 2010 at 09:23:06AM -0600, tsuraan wrote:
> > Sorry, it doesn't support chunked-encoding from the client.
> > Chunked-encoding from the client is error prone, not part of the standard,
> > and totally useless since the client knows what the size of its data
> > is from the start.  Allowing chunked-ecoding from the client would allow
> > for a range of attacks where a client could eat up resources by sending
> > an endless stream of chunks.
> 
> Is that an actual attack?  The difference between chunked and normal
> upload is that with normal uploads, the client tells the server the
> size before it starts sending chunks, but once it's said a size, it
> can send chunks of data as quickly or slowly as it wants to.  With
> chunked upload, the client doesn't tell the server in advance how much
> data it's going to send, but once the server's seen too much data, it
> can sever the connection, right?  How would it be any different from a
> client figuring out the max upload size and just slowly (I assume
> that's the key to the attach) sending it that much data to tie up
> resources?

http://ha.ckers.org/slowloris/

There ya go, and that's just with headers.  With CE the client can just
send and send and send, and also do a slowloris using headers in the
chunks.  It's endless.

There's also an attack where using client-side CE through badly written
proxies allows for poisoning the connection.

Clients should be stating how much they're going to send and send only
that much.  They can't be trusted and shouldn't be allowed to endlessly
send ever growing requests, streams of request bodies, and arbitrary
headers at any point in the stream.

> The reason that I'd like chunked post is that some browsers really
> don't function correctly without it.  When trying to upload a large
> file (gigabytes), firefox and explorer (not sure about any of the
> other browsers) use massive amounts of RAM and CPU before they start
> uploading if chunked post is not available.

What?  Then that is a bug you should file with the browsers, and
actually I don't think this is true.  First, I've used browsers to
upload DVD sized files without this happening.  Second, you say they
don't do this *if* the server supports CE client side, but how does the
browser know that?  There's no mechanism to ask a server this before
sending.  Also, how are you possibly sending these giant POSTs since
they'd take forever and the browser has the worst UI for upload
progress.

I think you're mistaken, but if you can work up a test I'll gladly
investigate.

> I have no idea why this happens, but chunked post support hides the
> issue.  It I wanted to patch in my own support and make it available
> as a discouraged option, where would I start?  I can't find anything
> in the http11_parser.rl that says anything at all about chunking, but
> from tracing the code, it looks like the error is coming from the http
> parser.

This won't happen, and I think you're going in the wrong direction.
I've not seen this happen on other sites or other projects, so there's
something you have to be doing wrong.  Until you find out why, or try to
use PUT instead, I wouldn't waste time trying to get mongrel2 to support
CE just yet.

-- 
Zed A. Shaw
http://zedshaw.com/

Re: [mongrel2] Chunked transfer encoding

From:
Zed A. Shaw
Date:
2010-12-27 @ 18:01 
On Mon, Dec 27, 2010 at 09:52:45AM -0800, Zed A. Shaw wrote:
> This won't happen, and I think you're going in the wrong direction.
> I've not seen this happen on other sites or other projects, so there's
> something you have to be doing wrong.  Until you find out why, or try to
> use PUT instead, I wouldn't waste time trying to get mongrel2 to support
> CE just yet.

But, I should clarify, I'm not totally against this, just against doing
it without making sure it really is totally necessary.

So, if you wanted to do it, go look at the
src/http11/httpclient_parser.rl as there's chunked encoding parsing in
there.  Then look at how that's handled and probably it'd involve a
similar bit of parsing juggling but on the other side.

Also we hang out on irc.freenode.org#mongrel2 and if you pop in there I
can help get it going.  But first make really sure it's necessary, since
what you're experiencing doesn't match what I've experienced.

-- 
Zed A. Shaw
http://zedshaw.com/

Re: Chunked transfer encoding

From:
tsuraan
Date:
2010-12-27 @ 05:20 
> I saw in an email from 2009 that the mongrel2 http parser has some
> degree of support for chunked post, so, not knowing what to expect, I
> tried it out.

Come to think of it, I don't think mongrel2 existed in september 2009,
but I believe the discussion was about the same http parser that
mongrel2 uses (link is
<http://www.mail-archive.com/mongrel-development@rubyforge.org/msg00148.html>).
 I hope the rest of my question is a bit better-founded.

Archives based on code by Armin Ronacher //