librelist archives

« back to archive

Re: [flask] flask-celery deprecated?

Re: [flask] flask-celery deprecated?

From:
Ask Solem
Date:
2012-07-26 @ 15:12
Hey Jonathan,

Sorry for the late reply but I just found your message
in a Google search.

>With my current config, Celery 3.0 is temporarily broken. I'm finding that
>it's taking a spaghetti of dependencies to get it right. As such, it
>certainly feels to me like there is a continued need for the extension. In
>my mind a Flask-Celery extension would (at least) do the following:

What sort of dependencies?  The only dependency you should need
is the 'celery' package itself.

>1) Accept an instance of a Flask application in either an extension
>constructor or init_app() method (i.e. understands how to operate with a
>deferred configuration)

What sort of things would you like Celery do do with the Flask application?
Is there anything it needs beyond the configuration?

You can tell Celery to use a Flask apps configuration easily:

    app = Flask()
    celery = Celery()
    celery.add_defaults(app.config)

add_defaults will be part of 3.0.4, it's different from conf.update in that
it won't make a copy of your configuration, and it won't have to pickle it
when starting new child processes.  For version before you have to do:

    celery.conf.defaults.insert(0, app.config)
    celery.conf._order.insert(1, app.config)

or just:

    celery.conf.update(app.config)  # makes a copy

In this case the config will be read directly, but you
can pass a callable to have it loaded lazily:

   celery.add_defaults(lambda: flask_app.config)

>2) Support a common Flask/Celery configuration (i.e. no celeryconfig.py)

Celery 3.0 does not require a celeryconfig, as noted above you can
use the Flask configuration, use a separate config module for Celery,
or specify the celery configuration directly in the code.

>3) Operate well when instantiated from an Application

Could you elaborate on this?

>Factory<http://flask.pocoo.org/docs/patterns/appfactories/>
>4) Continue to provide working Flask-Script commands that can be used
>during development

But with the new 'celery' umbrella command why would you need it?

All you need to do is specify the celery instance that should be used:

   celery -A myapp worker -l info

(-A myapp is a shortcut to --app=myapp.celery, see 
http://docs.celeryproject.org/en/latest/getting-started/next-steps.html#about-the-app-argument)

You can also easily start the umbrella command programmatically:

    celery = Celery('myapp')

    if __name__ == '__main__':
        celery.start()

where 'myapp' is the name of the module, necessary to convert the __main__ prefix
when generating task names.

Flask-Celery already consisted of very few lines of code when used with 
Celery 2.5,
where most of the code was there to convert optparse options into argparse
options,
in Celery 3.1 we hope to move to argparse, and there's an issue open for it too:
https://github.com/celery/celery/issues/793



I believe that everyone sharing the same API is a big step forward, with many
advantages like having the same documentation and issue tracker.
Of course there will be some issues to solve at first, but I believe
we should be able to fix them and improve the core API so that everyone
is happy.

And if not, and we find out that it was better after all,
we can always go back to using extensions...

-- 
Ask Solem
twitter.com/asksol | +44 (0)7713357179

Re: [flask] flask-celery deprecated?

From:
Jonathan Zempel
Date:
2012-07-29 @ 21:22
Ask,

Thanks for getting back to me on this. I think this makes more sense if I
describe a scenario and then produce the issue with code.

The scenario is a large Flask application that sends email notifications
based certain events. The application uses the Flask-Mail extension to send
email. The application also uses Celery to send email asynchronously so
that there is no delayed response to the client. The dependencies are
simply Flask, Flask-Mail, and Celery 3.0.

I added a gist that outlines the basic contours of this scenario:
https://gist.github.com/3201722. I've used (very basic) documented patterns
for application factories<http://flask.pocoo.org/docs/patterns/appfactories/>,
blueprints <http://flask.pocoo.org/docs/blueprints/#blueprints>, and dedicated
celery 
modules<http://docs.celeryproject.org/en/latest/getting-started/next-steps.html#project-layout>.
If you download the gist, you can run Flask using "python run.py" and
Celery using "celery worker -A application". Ping the only available route
via "http://127.0.0.1:5000/"

The primary issue is that Flask-Mail no longer works. (I have a similar
setup in production using Flask-Celery with Celery 2.5.5 and it does work.)
The exception traceback is:

Traceback (most recent call last):
  File

"/Users/jzempel/.virtualenvs/temp/lib/python2.7/site-packages/celery/task/trace.py",
line 212, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/Users/jzempel/Sites/temp/application/tasks.py", line 8, in
handle_notification
    message = Message("Test")
  File

"/Users/jzempel/.virtualenvs/temp/lib/python2.7/site-packages/flaskext/mail/message.py",
line 58, in __init__
    app = _request_ctx_stack.top.app
AttributeError: 'NoneType' object has no attribute 'app'

So it appears that Celery 3.0 (unlike Flask-Celery & Celery 2.5.5) does not
allow task access to the application context. It seems like this would be a
common requirement, especially when an app wants to depend on Flask
extensions in an asynchronous process.

Here is a second gist, modified in the way that I wish a Flask-Celery 3.0
extension would work with Flask: https://gist.github.com/3201883. The main
differences are: 1) no more celery.py, 2) Celery is instantiated in
extensions.py, 3) Celery is initialized (implicitly depending on current
application config) in the "create_app" application factory. And (somehow?)
the ability to reference the application context in an async celery task is
restored.

Does this help clarify what I'm seeing and why I feel there is still an
important niche for Flask-Celery?

Jonathan.

On Thu, Jul 26, 2012 at 8:12 AM, Ask Solem <ask@celeryproject.org> wrote:

> Hey Jonathan,
>
> Sorry for the late reply but I just found your message
> in a Google search.
>
> >With my current config, Celery 3.0 is temporarily broken. I'm finding that
> >it's taking a spaghetti of dependencies to get it right. As such, it
> >certainly feels to me like there is a continued need for the extension. In
> >my mind a Flask-Celery extension would (at least) do the following:
>
> What sort of dependencies?  The only dependency you should need
> is the 'celery' package itself.
>
> >1) Accept an instance of a Flask application in either an extension
> >constructor or init_app() method (i.e. understands how to operate with a
> >deferred configuration)
>
> What sort of things would you like Celery do do with the Flask application?
> Is there anything it needs beyond the configuration?
>
> You can tell Celery to use a Flask apps configuration easily:
>
>     app = Flask()
>     celery = Celery()
>     celery.add_defaults(app.config)
>
> add_defaults will be part of 3.0.4, it's different from conf.update in that
> it won't make a copy of your configuration, and it won't have to pickle it
> when starting new child processes.  For version before you have to do:
>
>     celery.conf.defaults.insert(0, app.config)
>     celery.conf._order.insert(1, app.config)
>
> or just:
>
>     celery.conf.update(app.config)  # makes a copy
>
> In this case the config will be read directly, but you
> can pass a callable to have it loaded lazily:
>
>    celery.add_defaults(lambda: flask_app.config)
>
> >2) Support a common Flask/Celery configuration (i.e. no celeryconfig.py)
>
> Celery 3.0 does not require a celeryconfig, as noted above you can
> use the Flask configuration, use a separate config module for Celery,
> or specify the celery configuration directly in the code.
>
> >3) Operate well when instantiated from an Application
>
> Could you elaborate on this?
>
> >Factory<http://flask.pocoo.org/docs/patterns/appfactories/>
> >4) Continue to provide working Flask-Script commands that can be used
> >during development
>
> But with the new 'celery' umbrella command why would you need it?
>
> All you need to do is specify the celery instance that should be used:
>
>    celery -A myapp worker -l info
>
> (-A myapp is a shortcut to --app=myapp.celery, see
> 
http://docs.celeryproject.org/en/latest/getting-started/next-steps.html#about-the-app-argument
> )
>
> You can also easily start the umbrella command programmatically:
>
>     celery = Celery('myapp')
>
>     if __name__ == '__main__':
>         celery.start()
>
> where 'myapp' is the name of the module, necessary to convert the __main__
> prefix
> when generating task names.
>
> Flask-Celery already consisted of very few lines of code when used with
> Celery 2.5,
> where most of the code was there to convert optparse options into argparse
> options,
> in Celery 3.1 we hope to move to argparse, and there's an issue open for
> it too:
> https://github.com/celery/celery/issues/793
>
>
>
> I believe that everyone sharing the same API is a big step forward, with
> many
> advantages like having the same documentation and issue tracker.
> Of course there will be some issues to solve at first, but I believe
> we should be able to fix them and improve the core API so that everyone
> is happy.
>
> And if not, and we find out that it was better after all,
> we can always go back to using extensions...
>
> --
> Ask Solem
> twitter.com/asksol | +44 (0)7713357179
>
>