librelist archives

Re: [mongrel2] Handler task DoS

From:

Aliaksandr Valialkin

Date:

2011-11-17 @ 19:32

On Thu, Nov 17, 2011 at 6:27 PM, Jason Miller <jason@milr.com> wrote:
> Hi Aliaksandr,
>
> I'm testing a simple solution that uses a buffered connection between a
> delivering task and the handler task.  Cleanup of the connection and its
> related tasks at the end has a lot of corner cases so I'm not going to
> commit this until I'm done.
>
> If the handler task ever detects a full buffer then it just kills the
> connection.  The buffer size will be tunable and will be related to the
> number of messages rather than the raw amount of data (I'm just sending
> pointers to dynamically allocated bstrings).  I haven't decided if
> sending a message to the handler when the buffer starts to get full is a
> good idea or not; it's probably just better to let any handlers that
> will be sending large quantities of data to listen to the control port
> to make sure things aren't getting out of hand, since I assume the
> "average" handler won't send too much.  Then the only DoS issue is the
> memory allocated for outstanding sends.

Sounds good.
Don't forget about streaming handlers, which can send a lot of
response chunks. Since the connection speed between handler and
Mongrel2 is usually much higher than the connection speed between
Mongrel2 and the client, these responses will quickly queue up in
Mongrel2 process until the corresponding client connection is killed
by your patch. This will result in frequent and random connection
drops for clients with slow (relatively slow) links. So some sort of
rate limiting between Mongrel2 and handler is required for proper
streaming support.

>
> -Jason
>
>
> On 20:57 Tue 15 Nov     , Aliaksandr Valialkin wrote:
>> Hi all,
>>
>> It looks like Handler task, which is responsible for routing responses
>> from request handlers to clients, is vulnerable to the following
>> simple and cheap DoS attack:
>> 1. Create a socket with small receive buffer size (for example, Linux
>> allows setting SO_RCVBUF to 256 bytes according to man pages -
>> http://linux.die.net/man/7/socket ).
>> 2. Send a request to Mongrel2 and wait until it closes the connection,
>> but don't fetch the response back.
>> 3. Return to 1.
>>
>> This will block the corresponding Handler task if the response
>> returned from request handler doesn't fit default send buffer of the
>> socket on the server plus our small receive buffer on the client. The
>> Handler task will be blocked if the response size is larger than 16KiB
>> for default SO_SNDBUF value on Linux (see
>> http://linux.die.net/man/7/tcp ). Here is the corresponding call stack
>> for the latest Mongrel2 tree on github, which results in Handler task
>> block: 
https://github.com/zedshaw/mongrel2/tree/68681d051aa7ce23ac9e9ac5b41dddf164e8be0e
>> :
>>
>> Handler_task() -> handler_process_request() -> deliver_payload() ->
>> Connection_deliver_raw() -> IOBuf_send() -> plaintext_send() ->
>> fdsend() -> fdwait() -> _wait() -> taskswitch() -> contextswitch() ->
>> swapcontext() .
>>
>> Other clients won't receive responses from the corresponding request
>> handler until the Handler task is unblocked. This results in a DoS.
>> The Handler task can be unblocked only on two events:
>> 1) After our malicious client starts receiving response data or closes
>> the connection. This is obviously impossible :)
>> 2) After the corresponding socket will be forcibly closed by Mongrel2
>> itself due to timeout. I didn't investigate the code for the presence
>> of this timeout, but even if it presents it won't help much, since
>> timeouts smaller than a second will result in random connection
>> interrupts for legitimate clients, while larger timeouts don't really
>> defend from this attack.
>>
>> This attack also can result in Mongrel2's out of memory condition or,
>> more likely, swap trashing on x64 under high load and/or large
>> response sizes, because ZMQ will put pending responses received from
>> the request handler into unbound SUBSCRIBE queue. This can be somewhat
>> mitigated by using ZMQ_HWM and ZMQ_SWAP options (
>> http://api.zeromq.org/2-1:zmq-setsockopt ), but this isn't a real
>> solution.
>>
>> The following workaround can be used in order to mitigate this attack:
>> Use a separate task for each client connection. This task will be
>> responsible for sending responses back to the client. Let's call it
>> 'sender task'. The Handler task in this case must route responses from
>> request handler to the corresponding sender task. It's possible
>> delegating Handler task's job to ZMQ via creating separate SUBSCRIBE
>> socket for each client connection, which will filter not only by
>> request handler's uuid, but also by client's connection id. This
>> workaround converts our attack into less severe Slowloris-like attack
>> ( http://en.wikipedia.org/wiki/Slowloris ), which can block a lot of
>> sender tasks together with associated pending responses until they are
>> timed out. Since tasks are cheaper than threads, this Slowloris
>> shouldn't be a big problem if responses are small. But this workaround
>> doesn't rescue from memory outage or swap trashing for large responses
>> such as audio or video streams. In this case proper rate limiting
>> should be implemented, so Mongrel2 could notify request handler to
>> pause streaming for the given client after the corresponding pending
>> response queue size becomes larger than the given threshold.
>>
>> --
>> Best Regards,
>>
>> Aliaksandr
>>
>
>



-- 
Best Regards,

Aliaksandr

On 21:32 Thu 17 Nov     , Aliaksandr Valialkin wrote:
> Don't forget about streaming handlers, which can send a lot of
> response chunks. Since the connection speed between handler and
> Mongrel2 is usually much higher than the connection speed between
> Mongrel2 and the client, these responses will quickly queue up in
> Mongrel2 process until the corresponding client connection is killed
> by your patch. This will result in frequent and random connection
> drops for clients with slow (relatively slow) links. So some sort of
> rate limiting between Mongrel2 and handler is required for proper
> streaming support.
I disagree.

Simple handlers that send fewer messages to each connection than we
allow to be buffered are a non-issue (obviously).  Long running
streaming handlers are different.

First of all consider fixed-rate streaming:
Just implement a handler that sends fixed-sized messages at a fixed
rate.  With a sufficient buffer size, any clients that fill up are
incapable of maintaining the fixed rate, so dropping them isn't a
problem.  You probably want to detect when these drops are happening
though, as perhaps your server's bandwidth is insufficiently
provisioned.

Now consider non-fixed rate streaming.

There are 2 cases to consider here:
1) Sending huge amounts of data as fast as possible (e.g. downloading
large dynamically generated content)

This can be accomplished by monitoring the control port.  The handler
should back-off when sending to clients whose buffers are filling.  For
just sending huge-amounts of data as fast as possible, this is easily
accomplished by on/off logic and some hysteresis, for example, stop
sending when the buffer is nearly full, and start sending again when the
buffer is nearly empty.

2) Adaptive bitrate compression schemes for streaming media
This is a very special case, I don't know enough about the domain, but I
imagine a more complicated version of #1 (it's no longer off/on but
higher lower, so a PID might work) could work.

I really don't see a need to implement something on the mongrel2 side
when backends already can self rate-limit.  Keeping the buffer
information as pull rather than push (in the colloquial, not the ZMQ
sense) doesn't significantly complicate matters for handlers that care
about it, while keeping it simple for the common case.

-Jason

In any case, I mentioned this on IRC, but I committed my fix for this
issue to the develop branch.  Hopefully somoene else can take a look at
it and comment.  The next step is to do a proof-of-concept example of
streaming large amounts of data without using too much ram.
On 14:20 Thu 17 Nov     , Jason Miller wrote:
> On 21:32 Thu 17 Nov     , Aliaksandr Valialkin wrote:
> > Don't forget about streaming handlers, which can send a lot of
> > response chunks. Since the connection speed between handler and
> > Mongrel2 is usually much higher than the connection speed between
> > Mongrel2 and the client, these responses will quickly queue up in
> > Mongrel2 process until the corresponding client connection is killed
> > by your patch. This will result in frequent and random connection
> > drops for clients with slow (relatively slow) links. So some sort of
> > rate limiting between Mongrel2 and handler is required for proper
> > streaming support.
> I disagree.
> 
> Simple handlers that send fewer messages to each connection than we
> allow to be buffered are a non-issue (obviously).  Long running
> streaming handlers are different.
> 
> First of all consider fixed-rate streaming:
> Just implement a handler that sends fixed-sized messages at a fixed
> rate.  With a sufficient buffer size, any clients that fill up are
> incapable of maintaining the fixed rate, so dropping them isn't a
> problem.  You probably want to detect when these drops are happening
> though, as perhaps your server's bandwidth is insufficiently
> provisioned.
> 
> Now consider non-fixed rate streaming.
> 
> There are 2 cases to consider here:
> 1) Sending huge amounts of data as fast as possible (e.g. downloading
> large dynamically generated content)
> 
> This can be accomplished by monitoring the control port.  The handler
> should back-off when sending to clients whose buffers are filling.  For
> just sending huge-amounts of data as fast as possible, this is easily
> accomplished by on/off logic and some hysteresis, for example, stop
> sending when the buffer is nearly full, and start sending again when the
> buffer is nearly empty.
> 
> 2) Adaptive bitrate compression schemes for streaming media
> This is a very special case, I don't know enough about the domain, but I
> imagine a more complicated version of #1 (it's no longer off/on but
> higher lower, so a PID might work) could work.
> 
> I really don't see a need to implement something on the mongrel2 side
> when backends already can self rate-limit.  Keeping the buffer
> information as pull rather than push (in the colloquial, not the ZMQ
> sense) doesn't significantly complicate matters for handlers that care
> about it, while keeping it simple for the common case.
> 
> -Jason
> 
>