librelist archives

« back to archive

Grequests leads to gevent threading error

Grequests leads to gevent threading error

From:
Dan Frank
Date:
2012-11-12 @ 15:11
Hi everyone,

I've been trying to use the grequests (gevent-patched requests) module with
Flask, but I encountered some setbacks. When running with the development
server, I repeatedly saw this traceback upon *loading* the grequests module
(not even using it):

```
Exception happened during processing of request from ('127.0.0.1', 63396)
Traceback (most recent call last):
  File

"/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py",
line 284, in _handle_request_noblock
    self.process_request(request, client_address)
  File

"/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py",
line 310, in process_request
    self.finish_request(request, client_address)
  File

"/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py",
line 323, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File

"/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py",
line 638, in __init__
    self.handle()
  File

"/Users/dan/Dropbox/Sites/musichackday/env/lib/python2.7/site-packages/werkzeug/serving.py",
line 182, in handle
    rv = BaseHTTPRequestHandler.handle(self)
  File

"/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/BaseHTTPServer.py",
line 340, in handle
    self.handle_one_request()
  File

"/Users/dan/Dropbox/Sites/musichackday/env/lib/python2.7/site-packages/werkzeug/serving.py",
line 213, in handle_one_request
    self.raw_requestline = self.rfile.readline()
  File

"/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py",
line 447, in readline
    data = self._sock.recv(self._rbufsize)
  File

"/Users/dan/Dropbox/Sites/musichackday/env/lib/python2.7/site-packages/gevent/socket.py",
line 432, in recv
    wait_read(sock.fileno(), timeout=self.timeout, event=self._read_event)
  File

"/Users/dan/Dropbox/Sites/musichackday/env/lib/python2.7/site-packages/gevent/socket.py",
line 169, in wait_read
    switch_result = get_hub().switch()
  File

"/Users/dan/Dropbox/Sites/musichackday/env/lib/python2.7/site-packages/gevent/hub.py",
line 135, in get_hub
    raise NotImplementedError('gevent is only usable from a single thread')
NotImplementedError: gevent is only usable from a single thread
```

I should clarify that when I say "repeatedly" I mean that in one pageload,
I would see between one and, say, eight of those blobs pop out on the
command line. Sometimes the page would load successfully, sometimes not.
Running with foreman with gunicorn as the server seemed to work just fine.

Now, of course I see that the source of the problem seems to be around the
creation of multiple threads, which I certainly am not doing in my
application. So I suspect that is something happening under the hood in
flask.

I was able to circumvent it by using the gevent wsgi
server<http://flask.pocoo.org/docs/deploying/others/#gevent>,
but never quite understood what the real source of the problem was. In the
future I'd be interested in being able to use the regular development
server for all the other conveniences it offers; would anyone on this list
be able to offer some guidance on what is going on and what if anything I
can do to make the two play nicely? Many thanks!

df

Re: [flask] Grequests leads to gevent threading error

From:
kracekumar ramaraju
Date:
2012-11-12 @ 15:36
When you use `app.run()` the werkzeug starts with single thread.

In your code , add

```
from gevent import monkey
monkey.patch_all()
```
This will patch all your socket, thread code.

if you are using gevent code add above code, but not needed for grequests.



Then try to use grequest/gevent directly, depending on the operation error
may differ.

If you use `app.run` remember werkzeug is running in single thread, have
look into code
https://github.com/mitsuhiko/flask/blob/master/flask/app.py#L712

You can pass threads/process options here.

Since gevent monkey patches all the socket library and other one, when
using grequest/gevent in dev/prod use gevent-wsgi.


On Mon, Nov 12, 2012 at 8:41 PM, Dan Frank <danielhfrank@gmail.com> wrote:

> Hi everyone,
>
> I've been trying to use the grequests (gevent-patched requests) module
> with Flask, but I encountered some setbacks. When running with the
> development server, I repeatedly saw this traceback upon *loading* the
> grequests module (not even using it):
>
> ```
> Exception happened during processing of request from ('127.0.0.1', 63396)
> Traceback (most recent call last):
>   File
> 
"/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py",
> line 284, in _handle_request_noblock
>     self.process_request(request, client_address)
>   File
> 
"/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py",
> line 310, in process_request
>     self.finish_request(request, client_address)
>   File
> 
"/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py",
> line 323, in finish_request
>     self.RequestHandlerClass(request, client_address, self)
>   File
> 
"/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py",
> line 638, in __init__
>     self.handle()
>   File
> 
"/Users/dan/Dropbox/Sites/musichackday/env/lib/python2.7/site-packages/werkzeug/serving.py",
> line 182, in handle
>     rv = BaseHTTPRequestHandler.handle(self)
>   File
> 
"/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/BaseHTTPServer.py",
> line 340, in handle
>     self.handle_one_request()
>   File
> 
"/Users/dan/Dropbox/Sites/musichackday/env/lib/python2.7/site-packages/werkzeug/serving.py",
> line 213, in handle_one_request
>     self.raw_requestline = self.rfile.readline()
>   File
> 
"/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py",
> line 447, in readline
>     data = self._sock.recv(self._rbufsize)
>   File
> 
"/Users/dan/Dropbox/Sites/musichackday/env/lib/python2.7/site-packages/gevent/socket.py",
> line 432, in recv
>     wait_read(sock.fileno(), timeout=self.timeout, event=self._read_event)
>    File
> 
"/Users/dan/Dropbox/Sites/musichackday/env/lib/python2.7/site-packages/gevent/socket.py",
> line 169, in wait_read
>     switch_result = get_hub().switch()
>   File
> 
"/Users/dan/Dropbox/Sites/musichackday/env/lib/python2.7/site-packages/gevent/hub.py",
> line 135, in get_hub
>     raise NotImplementedError('gevent is only usable from a single thread')
> NotImplementedError: gevent is only usable from a single thread
> ```
>
> I should clarify that when I say "repeatedly" I mean that in one pageload,
> I would see between one and, say, eight of those blobs pop out on the
> command line. Sometimes the page would load successfully, sometimes not.
> Running with foreman with gunicorn as the server seemed to work just fine.
>
> Now, of course I see that the source of the problem seems to be around the
> creation of multiple threads, which I certainly am not doing in my
> application. So I suspect that is something happening under the hood in
> flask.
>
> I was able to circumvent it by using the gevent wsgi 
server<http://flask.pocoo.org/docs/deploying/others/#gevent>,
> but never quite understood what the real source of the problem was. In the
> future I'd be interested in being able to use the regular development
> server for all the other conveniences it offers; would anyone on this list
> be able to offer some guidance on what is going on and what if anything I
> can do to make the two play nicely? Many thanks!
>
> df
>



-- 
*
Thanks & Regards

"Talk is cheap, show me the code" -- Linus Torvalds
kracekumar
www.kracekumar.com
*