librelist archives

« back to archive

Requests module performance

Requests module performance

From:
Andres Riancho
Date:
2012-12-29 @ 01:55
List,

    I've been looking for a replacement for urllib2 for a while now
and I'm almost decided to use the requests module, but something that
I've been unable to get from the documentation (might start testing
tomorrow) is a performance comparison between requests and other
libraries that do the same thing.

   The questions are:
- Are there any performance measurements for requests that I've not found?
- Is speed one of your objectives?
- Have you considered merging with something like geventhttpclient?
(although I understand that it adds many dependencies, some of them in
C)

[0] https://github.com/gwik/geventhttpclient/

Regards,
--
Andrés Riancho
Project Leader at w3af - http://w3af.org/
Web Application Attack and Audit Framework
Twitter: @w3af
GPG: 0x93C344F3

Re: [requests] Requests module performance

From:
Ian Cordasco
Date:
2012-12-30 @ 01:57
Hi Andrés,

I think (if I remember correctly) that Kenneth (or someone else) has
claimed a significant speed increase over using urllib/urllib2 but I
don't have a citation or a good enough recollection to be entirely
certain.

If you're looking for asynchronous requests, you should check out
grequests and erequests. The former uses gevent and the latter
eventlet.

I'm currently not able to, but later this week or early next, I'd be
interested in collaborating on some performance testing if you would
like to as well. Feel free to reply here or off the list directly to
this address.

On Fri, Dec 28, 2012 at 8:55 PM, Andres Riancho
<andres.riancho@gmail.com> wrote:
> List,
>
>     I've been looking for a replacement for urllib2 for a while now
> and I'm almost decided to use the requests module, but something that
> I've been unable to get from the documentation (might start testing
> tomorrow) is a performance comparison between requests and other
> libraries that do the same thing.
>
>    The questions are:
> - Are there any performance measurements for requests that I've not found?
> - Is speed one of your objectives?
> - Have you considered merging with something like geventhttpclient?
> (although I understand that it adds many dependencies, some of them in
> C)
>
> [0] https://github.com/gwik/geventhttpclient/
>
> Regards,
> --
> Andrés Riancho
> Project Leader at w3af - http://w3af.org/
> Web Application Attack and Audit Framework
> Twitter: @w3af
> GPG: 0x93C344F3

Re: [requests] Requests module performance

From:
Andres Riancho
Date:
2012-12-30 @ 20:05
Ian,

    grequests [0], which is also a work from kennethreitz, looks very
simple and easy to use and maintain but while testing I found the
following:

Traceback (most recent call last):
  File "test.py", line 12, in <module>
    grequests.map(rs)
  File "/tmp/grequests/grequests.py", line 79, in map
    requests = list(requests)
  File "test.py", line 11, in <genexpr>
    rs = (grequests.get(u) for u in urls)
  File "/tmp/grequests/grequests.py", line 44, in wrapped
    return f(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line
44, in request
    return session.request(method=method, url=url, **kwargs)
TypeError: request() got an unexpected keyword argument 'return_response'

    Looks like the requests API changed and grequests wasn't updated?

[0] https://github.com/kennethreitz/grequests/blob/master/grequests.py

Regards,

On Sat, Dec 29, 2012 at 10:57 PM, Ian Cordasco
<graffatcolmingov@gmail.com> wrote:
> Hi Andrés,
>
> I think (if I remember correctly) that Kenneth (or someone else) has
> claimed a significant speed increase over using urllib/urllib2 but I
> don't have a citation or a good enough recollection to be entirely
> certain.
>
> If you're looking for asynchronous requests, you should check out
> grequests and erequests. The former uses gevent and the latter
> eventlet.
>
> I'm currently not able to, but later this week or early next, I'd be
> interested in collaborating on some performance testing if you would
> like to as well. Feel free to reply here or off the list directly to
> this address.
>
> On Fri, Dec 28, 2012 at 8:55 PM, Andres Riancho
> <andres.riancho@gmail.com> wrote:
>> List,
>>
>>     I've been looking for a replacement for urllib2 for a while now
>> and I'm almost decided to use the requests module, but something that
>> I've been unable to get from the documentation (might start testing
>> tomorrow) is a performance comparison between requests and other
>> libraries that do the same thing.
>>
>>    The questions are:
>> - Are there any performance measurements for requests that I've not found?
>> - Is speed one of your objectives?
>> - Have you considered merging with something like geventhttpclient?
>> (although I understand that it adds many dependencies, some of them in
>> C)
>>
>> [0] https://github.com/gwik/geventhttpclient/
>>
>> Regards,
>> --
>> Andrés Riancho
>> Project Leader at w3af - http://w3af.org/
>> Web Application Attack and Audit Framework
>> Twitter: @w3af
>> GPG: 0x93C344F3



-- 
Andrés Riancho
Project Leader at w3af - http://w3af.org/
Web Application Attack and Audit Framework
Twitter: @w3af
GPG: 0x93C344F3

Re: [requests] Requests module performance

From:
Ian Cordasco
Date:
2012-12-30 @ 23:13
You're right. The requests API did change. I forked grequests and made
the edits that should be sufficient. I do not have gevent installed
nor do I have sufficient enough time to test it, so could you test it
from https://github.com/sigmavirus24/grequests?

Thanks,
Ian

On Sun, Dec 30, 2012 at 3:05 PM, Andres Riancho
<andres.riancho@gmail.com> wrote:
> Ian,
>
>     grequests [0], which is also a work from kennethreitz, looks very
> simple and easy to use and maintain but while testing I found the
> following:
>
> Traceback (most recent call last):
>   File "test.py", line 12, in <module>
>     grequests.map(rs)
>   File "/tmp/grequests/grequests.py", line 79, in map
>     requests = list(requests)
>   File "test.py", line 11, in <genexpr>
>     rs = (grequests.get(u) for u in urls)
>   File "/tmp/grequests/grequests.py", line 44, in wrapped
>     return f(*args, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 55, in get
>     return request('get', url, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line
> 44, in request
>     return session.request(method=method, url=url, **kwargs)
> TypeError: request() got an unexpected keyword argument 'return_response'
>
>     Looks like the requests API changed and grequests wasn't updated?
>
> [0] https://github.com/kennethreitz/grequests/blob/master/grequests.py
>
> Regards,
>
> On Sat, Dec 29, 2012 at 10:57 PM, Ian Cordasco
> <graffatcolmingov@gmail.com> wrote:
>> Hi Andrés,
>>
>> I think (if I remember correctly) that Kenneth (or someone else) has
>> claimed a significant speed increase over using urllib/urllib2 but I
>> don't have a citation or a good enough recollection to be entirely
>> certain.
>>
>> If you're looking for asynchronous requests, you should check out
>> grequests and erequests. The former uses gevent and the latter
>> eventlet.
>>
>> I'm currently not able to, but later this week or early next, I'd be
>> interested in collaborating on some performance testing if you would
>> like to as well. Feel free to reply here or off the list directly to
>> this address.
>>
>> On Fri, Dec 28, 2012 at 8:55 PM, Andres Riancho
>> <andres.riancho@gmail.com> wrote:
>>> List,
>>>
>>>     I've been looking for a replacement for urllib2 for a while now
>>> and I'm almost decided to use the requests module, but something that
>>> I've been unable to get from the documentation (might start testing
>>> tomorrow) is a performance comparison between requests and other
>>> libraries that do the same thing.
>>>
>>>    The questions are:
>>> - Are there any performance measurements for requests that I've not found?
>>> - Is speed one of your objectives?
>>> - Have you considered merging with something like geventhttpclient?
>>> (although I understand that it adds many dependencies, some of them in
>>> C)
>>>
>>> [0] https://github.com/gwik/geventhttpclient/
>>>
>>> Regards,
>>> --
>>> Andrés Riancho
>>> Project Leader at w3af - http://w3af.org/
>>> Web Application Attack and Audit Framework
>>> Twitter: @w3af
>>> GPG: 0x93C344F3
>
>
>
> --
> Andrés Riancho
> Project Leader at w3af - http://w3af.org/
> Web Application Attack and Audit Framework
> Twitter: @w3af
> GPG: 0x93C344F3

Re: [requests] Requests module performance

From:
Andres Riancho
Date:
2013-01-02 @ 14:20
After installing python-gevent and requests-1.0.4.tar.gz:

Traceback (most recent call last):
  File "example.py", line 12, in <module>
    print grequests.map(rs)
  File "/tmp/grequests/grequests.py", line 76, in map
    jobs = [send(r, pool, stream=stream) for r in requests]
  File "/tmp/grequests/grequests.py", line 50, in send
    return gevent.spawn(r.send, stream=stream)
AttributeError: 'Response' object has no attribute 'send'

On Sun, Dec 30, 2012 at 8:13 PM, Ian Cordasco
<graffatcolmingov@gmail.com> wrote:
> You're right. The requests API did change. I forked grequests and made
> the edits that should be sufficient. I do not have gevent installed
> nor do I have sufficient enough time to test it, so could you test it
> from https://github.com/sigmavirus24/grequests?
>
> Thanks,
> Ian
>
> On Sun, Dec 30, 2012 at 3:05 PM, Andres Riancho
> <andres.riancho@gmail.com> wrote:
>> Ian,
>>
>>     grequests [0], which is also a work from kennethreitz, looks very
>> simple and easy to use and maintain but while testing I found the
>> following:
>>
>> Traceback (most recent call last):
>>   File "test.py", line 12, in <module>
>>     grequests.map(rs)
>>   File "/tmp/grequests/grequests.py", line 79, in map
>>     requests = list(requests)
>>   File "test.py", line 11, in <genexpr>
>>     rs = (grequests.get(u) for u in urls)
>>   File "/tmp/grequests/grequests.py", line 44, in wrapped
>>     return f(*args, **kwargs)
>>   File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 
55, in get
>>     return request('get', url, **kwargs)
>>   File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line
>> 44, in request
>>     return session.request(method=method, url=url, **kwargs)
>> TypeError: request() got an unexpected keyword argument 'return_response'
>>
>>     Looks like the requests API changed and grequests wasn't updated?
>>
>> [0] https://github.com/kennethreitz/grequests/blob/master/grequests.py
>>
>> Regards,
>>
>> On Sat, Dec 29, 2012 at 10:57 PM, Ian Cordasco
>> <graffatcolmingov@gmail.com> wrote:
>>> Hi Andrés,
>>>
>>> I think (if I remember correctly) that Kenneth (or someone else) has
>>> claimed a significant speed increase over using urllib/urllib2 but I
>>> don't have a citation or a good enough recollection to be entirely
>>> certain.
>>>
>>> If you're looking for asynchronous requests, you should check out
>>> grequests and erequests. The former uses gevent and the latter
>>> eventlet.
>>>
>>> I'm currently not able to, but later this week or early next, I'd be
>>> interested in collaborating on some performance testing if you would
>>> like to as well. Feel free to reply here or off the list directly to
>>> this address.
>>>
>>> On Fri, Dec 28, 2012 at 8:55 PM, Andres Riancho
>>> <andres.riancho@gmail.com> wrote:
>>>> List,
>>>>
>>>>     I've been looking for a replacement for urllib2 for a while now
>>>> and I'm almost decided to use the requests module, but something that
>>>> I've been unable to get from the documentation (might start testing
>>>> tomorrow) is a performance comparison between requests and other
>>>> libraries that do the same thing.
>>>>
>>>>    The questions are:
>>>> - Are there any performance measurements for requests that I've not found?
>>>> - Is speed one of your objectives?
>>>> - Have you considered merging with something like geventhttpclient?
>>>> (although I understand that it adds many dependencies, some of them in
>>>> C)
>>>>
>>>> [0] https://github.com/gwik/geventhttpclient/
>>>>
>>>> Regards,
>>>> --
>>>> Andrés Riancho
>>>> Project Leader at w3af - http://w3af.org/
>>>> Web Application Attack and Audit Framework
>>>> Twitter: @w3af
>>>> GPG: 0x93C344F3
>>
>>
>>
>> --
>> Andrés Riancho
>> Project Leader at w3af - http://w3af.org/
>> Web Application Attack and Audit Framework
>> Twitter: @w3af
>> GPG: 0x93C344F3



-- 
Andrés Riancho
Project Leader at w3af - http://w3af.org/
Web Application Attack and Audit Framework
Twitter: @w3af
GPG: 0x93C344F3