Hello all, I'm writing a web site that will allow a user to query a remote web service for some XML data. The remote service is usually very quick to respond, but I can see it not being a good idea to make a direct query from the Flask view code. If there were several hundred users doing this all at once that might cause some issues. So, what would be the best way to allow this to happen? I've seen some sites implement some sort of a queue feature. It looks like your request for the XML data gets put into a queue and the requests isn't made until it's your turn. I have also looked into Eventlet, specifically their feed scraper example: http://eventlet.net/doc/examples.html#feed-scraper. I'm just not sure if Eventlet would actually help or not. I'm using Flask, Gunicorn + gevent workers. Thank you, Ryan
>From what I can tell the User in your cases would expect an answer
immediately so the only thing you can really do is caching.
Performing tasks in the background or asynchronously is only interesting
if you don't need a response as fast as possible but sometime in the
future, if the one getting the response didn't do anything directly to
provoke the notification or if the code you use for dispatching those
requests has multiple possible execution paths which led to different
responses.
This is usually the case for notifications where you send one to another
application using a queue and the other applications checks the queue
for new notifications, processes them and sends them to the user. The
rendering of templates and sending of emails is not something you have
to do to give the user an immediate response if he did at all something
which provokes a response, in a forum it doesn't really matter if it
takes 5 seconds or a minute for a user to get a notification about a new
post in a thread he is following.
I've never used it but would something like celery work for you? http://celeryproject.org/ On Sep 6, 2010 2:09 PM, "Ryan Cole" <betawarz@gmail.com> wrote: Hello all, I'm writing a web site that will allow a user to query a remote web service for some XML data. The remote service is usually very quick to respond, but I can see it not being a good idea to make a direct query from the Flask view code. If there were several hundred users doing this all at once that might cause some issues. So, what would be the best way to allow this to happen? I've seen some sites implement some sort of a queue feature. It looks like your request for the XML data gets put into a queue and the requests isn't made until it's your turn. I have also looked into Eventlet, specifically their feed scraper example: http://eventlet.net/doc/examples.html#feed-scraper. I'm just not sure if Eventlet would actually help or not. I'm using Flask, Gunicorn + gevent workers. Thank you, Ryan
Hrm, that does look promising. Would something like eventlets not be helpful? I have seen some example things use them, but when I took a glance I couldn't see how they would. Am I correct, though, in thinking that it's not a good idea to make a direct request for the remote XML data? Ryan On Mon, Sep 6, 2010 at 1:15 PM, Tim Riley <riltim@gmail.com> wrote: > I've never used it but would something like celery work for you? > > http://celeryproject.org/ > > On Sep 6, 2010 2:09 PM, "Ryan Cole" <betawarz@gmail.com> wrote: > > Hello all, > > I'm writing a web site that will allow a user to query a remote web service > for some XML data. The remote service is usually very quick to respond, but > I can see it not being a good idea to make a direct query from the Flask > view code. If there were several hundred users doing this all at once that > might cause some issues. > > So, what would be the best way to allow this to happen? I've seen some > sites implement some sort of a queue feature. It looks like your request for > the XML data gets put into a queue and the requests isn't made until it's > your turn. I have also looked into Eventlet, specifically their feed scraper > example: http://eventlet.net/doc/examples.html#feed-scraper. I'm just not > sure if Eventlet would actually help or not. > > I'm using Flask, Gunicorn + gevent workers. > > Thank you, > Ryan > >