librelist archives

« back to archive

Flask & queue design pattern

Flask & queue design pattern

From:
Date:
2013-04-12 @ 22:35
Hi,

I'd like to get some advice on how to implement a queue with a Flask app.

I have a Flask app running. I'd like to send it a particular kind of 
request that leads to a complex calculation, one that may take several 
minutes or more. I envision that the request will come in, immediately 
return a token to the user, and the user will occasionally ping the server
with the token to see if the result is ready. On the back end of this, I'd
like to limit the number of calculations that can be made at one time so 
as not to kill the server, and then to place the rest in a queue.

What would people recommend as a design pattern for this? I need some 
process to control the queue (e.g. limit to x threads, limit x requests 
from a single IP), as well as start processes and get their result.

I've considered placing the incoming queries into an SQLite database. This
has the disadvantage that I can't really have concurrent access to it from
multiple flask instances, but the advantage that it creates one less (big)
dependency on the application. And really, I one need one process to 
read/write to it at any time.

Any suggestions welcome!

Cheers,
Demitri

Re: [flask] Flask & queue design pattern

From:
Eric Scrivner
Date:
2013-04-12 @ 22:45
Hey Demetri,

At SHIFT we have this problem almost all the time. We've tried several 
approaches over the years (Including an SQL/Threads approach). These were 
all way over-complicated. What we've ended up using is Celery 
(http://www.celeryproject.org/) with RabbitMQ or, if you want even easier 
setup, Redis. This has worked out fantastically and allows you to spawn a 
job (creating a job object in the database with a UUID), and then allowing
your frontend to poll a route with that ID to find out when the job is 
done and retrieve the data. WebSockets are also another great way to push 
job results to your application when they complete.  Hope this helps you 
get started.

Best,

Eric Scrivner | Senior Software Developer | SHIFT
1447 Cloverfield Blvd, Suite 101
Santa Monica, CA 90404
(315) 730 9628
eric@shift.com







On Apr 12, 2013, at 3:35 PM, thatsanicehatyouhave@me.com wrote:

> Hi,
> 
> I'd like to get some advice on how to implement a queue with a Flask app.
> 
> I have a Flask app running. I'd like to send it a particular kind of 
request that leads to a complex calculation, one that may take several 
minutes or more. I envision that the request will come in, immediately 
return a token to the user, and the user will occasionally ping the server
with the token to see if the result is ready. On the back end of this, I'd
like to limit the number of calculations that can be made at one time so 
as not to kill the server, and then to place the rest in a queue.
> 
> What would people recommend as a design pattern for this? I need some 
process to control the queue (e.g. limit to x threads, limit x requests 
from a single IP), as well as start processes and get their result.
> 
> I've considered placing the incoming queries into an SQLite database. 
This has the disadvantage that I can't really have concurrent access to it
from multiple flask instances, but the advantage that it creates one less 
(big) dependency on the application. And really, I one need one process to
read/write to it at any time.
> 
> Any suggestions welcome!
> 
> Cheers,
> Demitri

Re: [flask] Flask & queue design pattern

From:
Joe Esposito
Date:
2013-04-13 @ 01:06
Look into Redis. Celery was *really* too bulky for my taste. Redis is
amazing for queues. Lightweight and fast.

Perhaps this can help, as an example on how to use
Redis<https://github.com/joeyespo/hello-redis-tasks>
.
Whether or not it does, definitely check out Redis.

Joe

On Fri, Apr 12, 2013 at 6:45 PM, Eric Scrivner <eric@grapheffect.com> wrote:

> Hey Demetri,
>
> At SHIFT we have this problem almost all the time. We've tried several
> approaches over the years (Including an SQL/Threads approach). These were
> all way over-complicated. What we've ended up using is Celery (
> http://www.celeryproject.org/) with RabbitMQ or, if you want even easier
> setup, Redis. This has worked out fantastically and allows you to spawn a
> job (creating a job object in the database with a UUID), and then allowing
> your frontend to poll a route with that ID to find out when the job is done
> and retrieve the data. WebSockets are also another great way to push job
> results to your application when they complete.  Hope this helps you get
> started.
>
> Best,
>
>   *Eric Scrivner | Senior Software Developer | SHIFT*
> 1447 Cloverfield Blvd, Suite 101
> Santa Monica, CA 90404
> (315) 730 9628
> eric@shift.com
>
>
>
>
>
>
>
> On Apr 12, 2013, at 3:35 PM, thatsanicehatyouhave@me.com wrote:
>
> Hi,
>
> I'd like to get some advice on how to implement a queue with a Flask app.
>
> I have a Flask app running. I'd like to send it a particular kind of
> request that leads to a complex calculation, one that may take several
> minutes or more. I envision that the request will come in, immediately
> return a token to the user, and the user will occasionally ping the server
> with the token to see if the result is ready. On the back end of this, I'd
> like to limit the number of calculations that can be made at one time so as
> not to kill the server, and then to place the rest in a queue.
>
> What would people recommend as a design pattern for this? I need some
> process to control the queue (e.g. limit to x threads, limit x requests
> from a single IP), as well as start processes and get thei r result.
>
>
> I've considered placing the incoming queries into an SQLite database. This
> has the disadvantage that I can't really have concurrent access to it from
> multiple flask instances, but the advantage that it creates one less (big)
> dependency on the application. And really, I one need one process to
> read/write to it at any time.
>
> Any suggestions welcome!
>
> Cheers,
> Demitri
>
>
>

Re: [flask] Flask & queue design pattern

From:
Steven Kryskalla
Date:
2013-04-13 @ 08:03
On Fri, Apr 12, 2013 at 6:06 PM, Joe Esposito <espo58@gmail.com> wrote:
> Look into Redis. Celery was really too bulky for my taste. Redis is amazing
> for queues. Lightweight and fast.

Another vote for redis, with either of these for running the jobs:

https://github.com/binarydud/pyres
http://python-rq.org/

There is also a flask snippet here for creating a simple job queue
with flask + redis:

http://flask.pocoo.org/snippets/73/

Celery is good, as others have mentioned, but it is more complex and
brings in more dependencies.

There is another option, if all you need is simple background
processing and your app is running on a single machine. It's not much
work to write your own job queue & worker using the filesystem to
store the jobs:

https://gist.github.com/lost-theory/5377499

If you want to rate limit based on IP, I think you will need to write
your own code, even if you used something like celery. Here's another
snippet from Armin on how to do that with redis :)

http://flask.pocoo.org/snippets/70/

-Steve

Re: [flask] Flask & queue design pattern

From:
Andrea D'Amore
Date:
2013-04-14 @ 07:18
On Sat, Apr 13, 2013 at 10:03 AM, Steven Kryskalla <skryskalla@gmail.com> wrote:
> Another vote for redis

This is the third vote for Redis. Out of curiosity, what is the
advantage of using Redis compared to SQLite  in such a case?

--
Andrea

Re: [flask] Flask & queue design pattern

From:
Steven Kryskalla
Date:
2013-04-14 @ 08:42
On Sun, Apr 14, 2013 at 12:18 AM, Andrea D'Amore <and.damore@gmail.com> wrote:
> On Sat, Apr 13, 2013 at 10:03 AM, Steven Kryskalla <skryskalla@gmail.com> wrote:
>> Another vote for redis
>
> This is the third vote for Redis. Out of curiosity, what is the
> advantage of using Redis compared to SQLite  in such a case?

On a single machine with a low number of processes writing to the
database, sqlite will work great, and is nice because it doesn't
require any extra dependencies. Some advantages of redis are:

* wider variety of data structures and operations (e.g. rpop/lpush to
create a queue)
* can handle many concurrent writers; sqlite can handle only one
writer at a time
* redis is accessed over the network, so you are able to scale out to
multiple machines (multiple servers for web requests vs. background
processing)
* nice tools like pyres / rq / celery which use redis as their data store

Re: [flask] Flask & queue design pattern

From:
Joe Esposito
Date:
2013-04-13 @ 13:58
@Steven that's awesome, didn't know about Pyres.

Another option while I was thinking about it is to use
APScheduler<http://pythonhosted.org/APScheduler/>.
Then you can use Redis or another data store for scheduling jobs, if you're
already using one.
It's thread-safe, so you could set it up to run from your main app when
debugging locally, or running from one or more "worker.py" scripts on
production. This is similar to Steven's example, but more cross-platform.

Hope this helps.
-Joe

On Sat, Apr 13, 2013 at 4:03 AM, Steven Kryskalla <skryskalla@gmail.com>wrote:

> On Fri, Apr 12, 2013 at 6:06 PM, Joe Esposito <espo58@gmail.com> wrote:
> > Look into Redis. Celery was really too bulky for my taste. Redis is
> amazing
> > for queues. Lightweight and fast.
>
> Another vote for redis, with either of these for running the jobs:
>
> https://github.com/binarydud/pyres
> http://python-rq.org/
>
> There is also a flask snippet here for creating a simple job queue
> with flask + redis:
>
> http://flask.pocoo.org/snippets/73/
>
> Celery is good, as others have mentioned, but it is more complex and
> brings in more dependencies.
>
> There is another option, if all you need is simple background
> processing and your app is running on a single machine. It's not much
> work to write your own job queue & worker using the filesystem to
> store the jobs:
>
> https://gist.github.com/lost-theory/5377499
>
> If you want to rate limit based on IP, I think you will need to write
> your own code, even if you used something like celery. Here's another
> snippet from Armin on how to do that with redis :)
>
> http://flask.pocoo.org/snippets/70/
>
> -Steve
>

Re: [flask] Flask & queue design pattern

From:
Date:
2013-04-13 @ 20:24
Hi All,

Thank you very much for all the responses. I hadn't come across any of 
those suggestions, so I will get to investigating!

Cheers,
Demitri

Re: [flask] Flask & queue design pattern

From:
cbrueggenolte
Date:
2013-04-13 @ 22:05
Thanks Steven,

This is also exactly something I can use to pull rss feeds.
Nice.

Am Samstag, 13. April 2013 schrieb Steven Kryskalla :

> On Fri, Apr 12, 2013 at 6:06 PM, Joe Esposito <espo58@gmail.com<javascript:;>>
> wrote:
> > Look into Redis. Celery was really too bulky for my taste. Redis is
> amazing
> > for queues. Lightweight and fast.
>
> Another vote for redis, with either of these for running the jobs:
>
> https://github.com/binarydud/pyres
> http://python-rq.org/
>
> There is also a flask snippet here for creating a simple job queue
> with flask + redis:
>
> http://flask.pocoo.org/snippets/73/
>
> Celery is good, as others have mentioned, but it is more complex and
> brings in more dependencies.
>
> There is another option, if all you need is simple background
> processing and your app is running on a single machine. It's not much
> work to write your own job queue & worker using the filesystem to
> store the jobs:
>
> https://gist.github.com/lost-theory/5377499
>
> If you want to rate limit based on IP, I think you will need to write
> your own code, even if you used something like celery. Here's another
> snippet from Armin on how to do that with redis :)
>
> http://flask.pocoo.org/snippets/70/
>
> -Steve
>


-- 
--
Carsten Brueggenolte
http://cbrueggenolte.de

Re: [flask] Flask & queue design pattern

From:
David J
Date:
2013-04-12 @ 22:40
Sounds like you need celery.
On Apr 12, 2013 6:37 PM, <thatsanicehatyouhave@me.com> wrote:

> Hi,
>
> I'd like to get some advice on how to implement a queue with a Flask app.
>
> I have a Flask app running. I'd like to send it a particular kind of
> request that leads to a complex calculation, one that may take several
> minutes or more. I envision that the request will come in, immediately
> return a token to the user, and the user will occasionally ping the server
> with the token to see if the result is ready. On the back end of this, I'd
> like to limit the number of calculations that can be made at one time so as
> not to kill the server, and then to place the rest in a queue.
>
> What would people recommend as a design pattern for this? I need some
> process to control the queue (e.g. limit to x threads, limit x requests
> from a single IP), as well as start processes and get their result.
>
> I've considered placing the incoming queries into an SQLite database. This
> has the disadvantage that I can't really have concurrent access to it from
> multiple flask instances, but the advantage that it creates one less (big)
> dependency on the application. And really, I one need one process to
> read/write to it at any time.
>
> Any suggestions welcome!
>
> Cheers,
> Demitri
>