librelist archives

« back to archive

Detect and deal with rogue abusive web clients/bots?

Detect and deal with rogue abusive web clients/bots?

From:
Roman Chyla
Date:
2012-10-01 @ 12:02
Hi !

Do you have some recommendations how to best deal with web clients
that abuse a website?

I.e. in situations when many users are hidden behind one proxy, one of
them may be a robot with thousands of requests/sec - we would need to
detect such a client (based on a combination of IP address and a
cookie). Are there some solutions in Flask, or would you prefer some
thing in front of the Flask app?

Thanks,

  Roman

Re: [flask] Detect and deal with rogue abusive web clients/bots?

From:
Sergio Pelissari
Date:
2012-10-01 @ 13:19
Hello,

Generally on this request i would recommend a iptables rules. I had the
same problem and solved with 4 lines of iptables, theese lines control the
burst for a unique source if its exceed i move this access to a queue and
wait about 5 seconds and resume the request. Now at the Flask
side unfortunately i can't help.

## the rules ##  IMPORTANT you have to set your values based on your site
traffic.

add syncookies flag to 1

net.ipv4.tcp_syncookies=1

# create new chains
iptables -N syn-flood

# limits incoming packets
iptables -A syn-flood -m limit --limit 10/second --limit-burst 50 -j RETURN

# log attacks
iptables -A syn-flood -j LOG --log-prefix "SYN flood: "

# silently drop the rest
iptables -A syn-flood -j DROP

On Mon, Oct 1, 2012 at 9:02 AM, Roman Chyla <roman.chyla@gmail.com> wrote:

> Hi !
>
> Do you have some recommendations how to best deal with web clients
> that abuse a website?
>
> I.e. in situations when many users are hidden behind one proxy, one of
> them may be a robot with thousands of requests/sec - we would need to
> detect such a client (based on a combination of IP address and a
> cookie). Are there some solutions in Flask, or would you prefer some
> thing in front of the Flask app?
>
> Thanks,
>
>   Roman
>

Re: [flask] Detect and deal with rogue abusive web clients/bots?

From:
Roman Chyla
Date:
2012-10-02 @ 11:02
Hello Sergio,

Thanks for the info, this is useful. If there were a few clients (ie.
a university proxy with 1000 users, with 10 bots and each bot
establishes only 2 connections and starts harvesting) would this help
against them?

Thanks,

  roman

On Mon, Oct 1, 2012 at 3:19 PM, Sergio Pelissari <sonared@gmail.com> wrote:
> Hello,
>
> Generally on this request i would recommend a iptables rules. I had the same
> problem and solved with 4 lines of iptables, theese lines control the burst
> for a unique source if its exceed i move this access to a queue and wait
> about 5 seconds and resume the request. Now at the Flask side unfortunately
> i can't help.
>
> ## the rules ##  IMPORTANT you have to set your values based on your site
> traffic.
>
> add syncookies flag to 1
>
> net.ipv4.tcp_syncookies=1
>
> # create new chains
> iptables -N syn-flood
>
> # limits incoming packets
> iptables -A syn-flood -m limit --limit 10/second --limit-burst 50 -j RETURN
>
> # log attacks
> iptables -A syn-flood -j LOG --log-prefix "SYN flood: "
>
> # silently drop the rest
> iptables -A syn-flood -j DROP
>
> On Mon, Oct 1, 2012 at 9:02 AM, Roman Chyla <roman.chyla@gmail.com> wrote:
>>
>> Hi !
>>
>> Do you have some recommendations how to best deal with web clients
>> that abuse a website?
>>
>> I.e. in situations when many users are hidden behind one proxy, one of
>> them may be a robot with thousands of requests/sec - we would need to
>> detect such a client (based on a combination of IP address and a
>> cookie). Are there some solutions in Flask, or would you prefer some
>> thing in front of the Flask app?
>>
>> Thanks,
>>
>>   Roman
>
>

Re: [flask] Detect and deal with rogue abusive web clients/bots?

From:
Sergio Pelissari
Date:
2012-10-02 @ 13:40
Well its depend how the bot was written...

If the bot send a large amount of requests to map your application, it will
work....

If the bot knows the amount of requests he can make it will decrease the
requests to map your application, so basically if the bot scan your app in
5 minutes without these rules, with the rules the time will be increased to
many hours depending your app size.

But you can save some bandwidth with these bot request and move it to a
controlled queue.

On Tue, Oct 2, 2012 at 8:02 AM, Roman Chyla <roman.chyla@gmail.com> wrote:

> Hello Sergio,
>
> Thanks for the info, this is useful. If there were a few clients (ie.
> a university proxy with 1000 users, with 10 bots and each bot
> establishes only 2 connections and starts harvesting) would this help
> against them?
>
> Thanks,
>
>   roman
>
> On Mon, Oct 1, 2012 at 3:19 PM, Sergio Pelissari <sonared@gmail.com>
> wrote:
> > Hello,
> >
> > Generally on this request i would recommend a iptables rules. I had the
> same
> > problem and solved with 4 lines of iptables, theese lines control the
> burst
> > for a unique source if its exceed i move this access to a queue and wait
> > about 5 seconds and resume the request. Now at the Flask side
> unfortunately
> > i can't help.
> >
> > ## the rules ##  IMPORTANT you have to set your values based on your site
> > traffic.
> >
> > add syncookies flag to 1
> >
> > net.ipv4.tcp_syncookies=1
> >
> > # create new chains
> > iptables -N syn-flood
> >
> > # limits incoming packets
> > iptables -A syn-flood -m limit --limit 10/second --limit-burst 50 -j
> RETURN
> >
> > # log attacks
> > iptables -A syn-flood -j LOG --log-prefix "SYN flood: "
> >
> > # silently drop the rest
> > iptables -A syn-flood -j DROP
> >
> > On Mon, Oct 1, 2012 at 9:02 AM, Roman Chyla <roman.chyla@gmail.com>
> wrote:
> >>
> >> Hi !
> >>
> >> Do you have some recommendations how to best deal with web clients
> >> that abuse a website?
> >>
> >> I.e. in situations when many users are hidden behind one proxy, one of
> >> them may be a robot with thousands of requests/sec - we would need to
> >> detect such a client (based on a combination of IP address and a
> >> cookie). Are there some solutions in Flask, or would you prefer some
> >> thing in front of the Flask app?
> >>
> >> Thanks,
> >>
> >>   Roman
> >
> >
>

Re: [flask] Detect and deal with rogue abusive web clients/bots?

From:
Filipe Cifali
Date:
2012-10-02 @ 14:42
Using a cache-proxy like Varnish can be a solution for High flood bots.

Low flood bots can be catch with Deny from env=BadUserAgent sometimes.
(Apache?)

Fortunately, most bot users are newbies to DOS/DDOS techniques and grab
pre-made scripts.

A botnet can be dangerous if well implemented...

BTW, what OS / Webserver are you using?

2012/10/2 Sergio Pelissari <sonared@gmail.com>

> Well its depend how the bot was written...
>
> If the bot send a large amount of requests to map your application, it
> will work....
>
> If the bot knows the amount of requests he can make it will decrease the
> requests to map your application, so basically if the bot scan your app in
> 5 minutes without these rules, with the rules the time will be increased to
> many hours depending your app size.
>
> But you can save some bandwidth with these bot request and move it to a
> controlled queue.
>
>
> On Tue, Oct 2, 2012 at 8:02 AM, Roman Chyla <roman.chyla@gmail.com> wrote:
>
>> Hello Sergio,
>>
>> Thanks for the info, this is useful. If there were a few clients (ie.
>> a university proxy with 1000 users, with 10 bots and each bot
>> establishes only 2 connections and starts harvesting) would this help
>> against them?
>>
>> Thanks,
>>
>>   roman
>>
>> On Mon, Oct 1, 2012 at 3:19 PM, Sergio Pelissari <sonared@gmail.com>
>> wrote:
>> > Hello,
>> >
>> > Generally on this request i would recommend a iptables rules. I had the
>> same
>> > problem and solved with 4 lines of iptables, theese lines control the
>> burst
>> > for a unique source if its exceed i move this access to a queue and wait
>> > about 5 seconds and resume the request. Now at the Flask side
>> unfortunately
>> > i can't help.
>> >
>> > ## the rules ##  IMPORTANT you have to set your values based on your
>> site
>> > traffic.
>> >
>> > add syncookies flag to 1
>> >
>> > net.ipv4.tcp_syncookies=1
>> >
>> > # create new chains
>> > iptables -N syn-flood
>> >
>> > # limits incoming packets
>> > iptables -A syn-flood -m limit --limit 10/second --limit-burst 50 -j
>> RETURN
>> >
>> > # log attacks
>> > iptables -A syn-flood -j LOG --log-prefix "SYN flood: "
>> >
>> > # silently drop the rest
>> > iptables -A syn-flood -j DROP
>> >
>> > On Mon, Oct 1, 2012 at 9:02 AM, Roman Chyla <roman.chyla@gmail.com>
>> wrote:
>> >>
>> >> Hi !
>> >>
>> >> Do you have some recommendations how to best deal with web clients
>> >> that abuse a website?
>> >>
>> >> I.e. in situations when many users are hidden behind one proxy, one of
>> >> them may be a robot with thousands of requests/sec - we would need to
>> >> detect such a client (based on a combination of IP address and a
>> >> cookie). Are there some solutions in Flask, or would you prefer some
>> >> thing in front of the Flask app?
>> >>
>> >> Thanks,
>> >>
>> >>   Roman
>> >
>> >
>>
>
>


-- 
[]'s

Filipe Cifali Stangler

Re: [flask] Detect and deal with rogue abusive web clients/bots?

From:
Roman Chyla
Date:
2012-10-03 @ 17:23
We are sing Centos and Apache. Thanks  for comments to both of you.

I have also found this module for Apache which looks promising:
http://opensource.adnovum.ch/mod_qos/

roman

On Tue, Oct 2, 2012 at 4:42 PM, Filipe Cifali <cifali.filipe@gmail.com> wrote:
> Using a cache-proxy like Varnish can be a solution for High flood bots.
>
> Low flood bots can be catch with Deny from env=BadUserAgent sometimes.
> (Apache?)
>
> Fortunately, most bot users are newbies to DOS/DDOS techniques and grab
> pre-made scripts.
>
> A botnet can be dangerous if well implemented...
>
> BTW, what OS / Webserver are you using?
>
>
> 2012/10/2 Sergio Pelissari <sonared@gmail.com>
>>
>> Well its depend how the bot was written...
>>
>> If the bot send a large amount of requests to map your application, it
>> will work....
>>
>> If the bot knows the amount of requests he can make it will decrease the
>> requests to map your application, so basically if the bot scan your app in 5
>> minutes without these rules, with the rules the time will be increased to
>> many hours depending your app size.
>>
>> But you can save some bandwidth with these bot request and move it to a
>> controlled queue.
>>
>>
>> On Tue, Oct 2, 2012 at 8:02 AM, Roman Chyla <roman.chyla@gmail.com> wrote:
>>>
>>> Hello Sergio,
>>>
>>> Thanks for the info, this is useful. If there were a few clients (ie.
>>> a university proxy with 1000 users, with 10 bots and each bot
>>> establishes only 2 connections and starts harvesting) would this help
>>> against them?
>>>
>>> Thanks,
>>>
>>>   roman
>>>
>>> On Mon, Oct 1, 2012 at 3:19 PM, Sergio Pelissari <sonared@gmail.com>
>>> wrote:
>>> > Hello,
>>> >
>>> > Generally on this request i would recommend a iptables rules. I had the
>>> > same
>>> > problem and solved with 4 lines of iptables, theese lines control the
>>> > burst
>>> > for a unique source if its exceed i move this access to a queue and
>>> > wait
>>> > about 5 seconds and resume the request. Now at the Flask side
>>> > unfortunately
>>> > i can't help.
>>> >
>>> > ## the rules ##  IMPORTANT you have to set your values based on your
>>> > site
>>> > traffic.
>>> >
>>> > add syncookies flag to 1
>>> >
>>> > net.ipv4.tcp_syncookies=1
>>> >
>>> > # create new chains
>>> > iptables -N syn-flood
>>> >
>>> > # limits incoming packets
>>> > iptables -A syn-flood -m limit --limit 10/second --limit-burst 50 -j
>>> > RETURN
>>> >
>>> > # log attacks
>>> > iptables -A syn-flood -j LOG --log-prefix "SYN flood: "
>>> >
>>> > # silently drop the rest
>>> > iptables -A syn-flood -j DROP
>>> >
>>> > On Mon, Oct 1, 2012 at 9:02 AM, Roman Chyla <roman.chyla@gmail.com>
>>> > wrote:
>>> >>
>>> >> Hi !
>>> >>
>>> >> Do you have some recommendations how to best deal with web clients
>>> >> that abuse a website?
>>> >>
>>> >> I.e. in situations when many users are hidden behind one proxy, one of
>>> >> them may be a robot with thousands of requests/sec - we would need to
>>> >> detect such a client (based on a combination of IP address and a
>>> >> cookie). Are there some solutions in Flask, or would you prefer some
>>> >> thing in front of the Flask app?
>>> >>
>>> >> Thanks,
>>> >>
>>> >>   Roman
>>> >
>>> >
>>
>>
>
>
>
> --
> []'s
>
> Filipe Cifali Stangler
>