librelist archives

« back to archive

Flask/Flask-SQLAlchemy with Celery worker gives DetachedInstanceError's ?

Flask/Flask-SQLAlchemy with Celery worker gives DetachedInstanceError's ?

From:
Dries De Smet
Date:
2014-12-12 @ 07:11
Hello,

I've been implementing the sql alchemy pattern as described in the flask
recipe for celery (http://flask.pocoo.org/docs/patterns/celery/) but while
the task executes fine for the *first* time, the*second* run (with the same
celery worker), SQLAlchemy (Using Flask-SQLAchemy) gives
"DetachedInstanceErrors" when querying the database.

Celery is run with:

celery -A staffing.tasks.celery worker[2014-12-11 17:43:36,392:
WARNING/MainProcess] celery@localhost.localdomain ready.

This is the output after running the first task: [2014-12-11 17:46:45,854:
WARNING/Worker-1] Import done. Took 11322.968 ms This is the output after
running it again:

[2014-12-11 17:46:55,722: ERROR/MainProcess] Task
staffing.tasks.import_tics_file[2086dd42-e492-4fa2-931d-cf9e5f97046f]
raised unexpected: DetachedInstanceError('Instance

*Code can be found at:* https://gist.github.com/Busata/a788720c7f7b128cf66a

I had to add the "blueprints_enabled" to avoid circular imports when
importing the task in one of the routes.

I'm using Redis as message broker, and updated flask/flask-sqlalchemy to
their latest versions.

(My question was posted on SO as well.)

Re: [flask] Flask/Flask-SQLAlchemy with Celery worker gives DetachedInstanceError's ?

From:
Matthias Urlichs
Date:
2014-12-12 @ 07:28
Hi,

Dries De Smet:
> I've been implementing the sql alchemy pattern as described in the flask
> recipe for celery (http://flask.pocoo.org/docs/patterns/celery/) but while
> the task executes fine for the *first* time, the*second* run (with the same
> celery worker), SQLAlchemy (Using Flask-SQLAchemy) gives
> "DetachedInstanceErrors" when querying the database.
> 
Do you keep any objects you loaded from the database around?

Don't. SQLAlchemy objects are tied to the session you get them from, which
Flask ties to the HTTP request. While you can refresh them, you'll run into
very interesting bugs when you get more than one concurrent request.

-- 
-- Matthias Urlichs

Re: [flask] Flask/Flask-SQLAlchemy with Celery worker gives DetachedInstanceError's ?

From:
Dries De Smet
Date:
2014-12-12 @ 07:42
Wow! Yes, I cached employees in a dictionary and since that dictionary was
on the module level, it did not get cleared on the second run. Thanks a
lot, that fixed it!

On 12 December 2014 at 08:28, Matthias Urlichs <matthias@urlichs.de> wrote:
>
> Hi,
>
> Dries De Smet:
> > I've been implementing the sql alchemy pattern as described in the flask
> > recipe for celery (http://flask.pocoo.org/docs/patterns/celery/) but
> while
> > the task executes fine for the *first* time, the*second* run (with the
> same
> > celery worker), SQLAlchemy (Using Flask-SQLAchemy) gives
> > "DetachedInstanceErrors" when querying the database.
> >
> Do you keep any objects you loaded from the database around?
>
> Don't. SQLAlchemy objects are tied to the session you get them from, which
> Flask ties to the HTTP request. While you can refresh them, you'll run into
> very interesting bugs when you get more than one concurrent request.
>
> --
> -- Matthias Urlichs
>