librelist archives

« back to archive

using celery with flask-sqlalchemy

using celery with flask-sqlalchemy

From:
Bastian Hoyer
Date:
2012-04-13 @ 12:19
Hi,

I'm currently working on a flask app that uses celery and
flask-sqlalchemy. I'm wonder what the correct way is to use the
database objects within the workers.

If I understand everything correctly I need a request context to make
sure the database session is correctly set up and destroyed?

Currently I'm doing something like:

@celery.task
def mytask(id):
    with celery.flask_app.test_request_context():
        obj = MyModel.query.get(id)

is that correct ? And with 0.9 there will be an app context I could
use instead ?


Thanks,

Bastian

Re: [flask] using celery with flask-sqlalchemy

From:
Ron DuPlain
Date:
2012-04-16 @ 19:28
Hi Bastian,

On Fri, Apr 13, 2012 at 8:19 AM, Bastian Hoyer <dafire@gmail.com> wrote:
> I'm currently working on a flask app that uses celery and
> flask-sqlalchemy. I'm wonder what the correct way is to use the
> database objects within the workers.

In general for Celery, you access the database directly from the
workers, which means localhost access for single host deployments, or
network access for distributed applications.  You can serialize
instances of your SQLAlchemy models across tasks, but you should only
do that if you have a specific reason for doing so.


> If I understand everything correctly I need a request context to make
> sure the database session is correctly set up and destroyed?

In Flask-SQLAlchemy, yes.  Here, you can use Flask-SQLAlchemy and
test_request_context (old way) or app_context (new way).  Or you can
use raw SQLAlchemy and manage the db session yourself.

Hope this helps,

Ron


> Currently I'm doing something like:
>
> @celery.task
> def mytask(id):
>    with celery.flask_app.test_request_context():
>        obj = MyModel.query.get(id)
>
> is that correct ? And with 0.9 there will be an app context I could
> use instead ?
>
>
> Thanks,
>
> Bastian

Re: [flask] using celery with flask-sqlalchemy

From:
dalan.miller@gmail.com
Date:
2012-04-16 @ 20:24
So as a noobie to Celery and RabbitMQ (or other message queue servers) if
you want your web app to do intensive things in the background, is Celery
and RabbitMQ the usual solution?

Daniel

On Mon, Apr 16, 2012 at 12:28 PM, Ron DuPlain <ron.duplain@gmail.com> wrote:

> Hi Bastian,
>
> On Fri, Apr 13, 2012 at 8:19 AM, Bastian Hoyer <dafire@gmail.com> wrote:
> > I'm currently working on a flask app that uses celery and
> > flask-sqlalchemy. I'm wonder what the correct way is to use the
> > database objects within the workers.
>
> In general for Celery, you access the database directly from the
> workers, which means localhost access for single host deployments, or
> network access for distributed applications.  You can serialize
> instances of your SQLAlchemy models across tasks, but you should only
> do that if you have a specific reason for doing so.
>
>
> > If I understand everything correctly I need a request context to make
> > sure the database session is correctly set up and destroyed?
>
> In Flask-SQLAlchemy, yes.  Here, you can use Flask-SQLAlchemy and
> test_request_context (old way) or app_context (new way).  Or you can
> use raw SQLAlchemy and manage the db session yourself.
>
> Hope this helps,
>
> Ron
>
>
> > Currently I'm doing something like:
> >
> > @celery.task
> > def mytask(id):
> >    with celery.flask_app.test_request_context():
> >        obj = MyModel.query.get(id)
> >
> > is that correct ? And with 0.9 there will be an app context I could
> > use instead ?
> >
> >
> > Thanks,
> >
> > Bastian
>

Re: [flask] using celery with flask-sqlalchemy

From:
Joe Esposito
Date:
2012-04-16 @ 20:33
Celery and RabbitMQ were pretty heavy-weight for me personally, so I
started using Redis to coordinate background tasks like in this snippet:
http://flask.pocoo.org/snippets/73/

I have an example project with that pattern, if you're interested. It's
marginally simpler than Celery+RabbitMQ to dive into:
https://github.com/joeyespo/hello-redis-tasks

On Mon, Apr 16, 2012 at 4:24 PM, dalan.miller@gmail.com <
dalan.miller@gmail.com> wrote:

> So as a noobie to Celery and RabbitMQ (or other message queue servers) if
> you want your web app to do intensive things in the background, is Celery
> and RabbitMQ the usual solution?
>
> Daniel
>
>
> On Mon, Apr 16, 2012 at 12:28 PM, Ron DuPlain <ron.duplain@gmail.com>wrote:
>
>> Hi Bastian,
>>
>> On Fri, Apr 13, 2012 at 8:19 AM, Bastian Hoyer <dafire@gmail.com> wrote:
>> > I'm currently working on a flask app that uses celery and
>> > flask-sqlalchemy. I'm wonder what the correct way is to use the
>> > database objects within the workers.
>>
>> In general for Celery, you access the database directly from the
>> workers, which means localhost access for single host deployments, or
>> network access for distributed applications.  You can serialize
>> instances of your SQLAlchemy models across tasks, but you should only
>> do that if you have a specific reason for doing so.
>>
>>
>> > If I understand everything correctly I need a request context to make
>> > sure the database session is correctly set up and destroyed?
>>
>> In Flask-SQLAlchemy, yes.  Here, you can use Flask-SQLAlchemy and
>> test_request_context (old way) or app_context (new way).  Or you can
>> use raw SQLAlchemy and manage the db session yourself.
>>
>> Hope this helps,
>>
>> Ron
>>
>>
>> > Currently I'm doing something like:
>> >
>> > @celery.task
>> > def mytask(id):
>> >    with celery.flask_app.test_request_context():
>> >        obj = MyModel.query.get(id)
>> >
>> > is that correct ? And with 0.9 there will be an app context I could
>> > use instead ?
>> >
>> >
>> > Thanks,
>> >
>> > Bastian
>>
>
>

Re: [flask] using celery with flask-sqlalchemy

From:
Bastian Hoyer
Date:
2012-04-16 @ 21:51
Thanks for your answers. I'm also using redis as the broker backend
with celery. I'm not serializing the models, just passing  the primary
keys to the tasks.

I'm also looking into using redis for a celerycam backend for a simple
task status display.

Bastian

Re: [flask] using celery with flask-sqlalchemy

From:
Jevgenij Tsoi
Date:
2012-04-17 @ 09:36
Has anyone tried beanstalkd ( http://kr.github.com/beanstalkd/ ) or gearman
( http://gearman.org )
I have looked at Celery and it seems really heavy for what is needed. (The
choice or RabbitMQ)

/JT

On Mon, Apr 16, 2012 at 11:51 PM, Bastian Hoyer <dafire@gmail.com> wrote:

> Thanks for your answers. I'm also using redis as the broker backend
> with celery. I'm not serializing the models, just passing  the primary
> keys to the tasks.
>
> I'm also looking into using redis for a celerycam backend for a simple
> task status display.
>
> Bastian
>

Re: [flask] using celery with flask-sqlalchemy

From:
Honza Javorek
Date:
2012-04-22 @ 23:05
Don't forget this: http://nvie.com/posts/introducing-rq/ A brand new
simple Redis queue based on the snippet previously mentioned by Joe.

H

On Tue, Apr 17, 2012 at 11:36, Jevgenij Tsoi <jevgenij@virtusize.com> wrote:
> Has anyone tried beanstalkd ( http://kr.github.com/beanstalkd/ ) or gearman
> ( http://gearman.org )
> I have looked at Celery and it seems really heavy for what is needed. (The
> choice or RabbitMQ)
>
> /JT
>
>
> On Mon, Apr 16, 2012 at 11:51 PM, Bastian Hoyer <dafire@gmail.com> wrote:
>>
>> Thanks for your answers. I'm also using redis as the broker backend
>> with celery. I'm not serializing the models, just passing  the primary
>> keys to the tasks.
>>
>> I'm also looking into using redis for a celerycam backend for a simple
>> task status display.
>>
>> Bastian
>
>