librelist archives

« back to archive

Flask Application with multiprocessing and uwsgi

Flask Application with multiprocessing and uwsgi

From:
Olav Grønås Gjerde
Date:
2013-01-28 @ 16:13
I've developed a flask backend application that is syncing files and
folders to an elasticsearch database.

I've implemented the following logic for each request:
First check if a process with the following name already exists in
multiprocessing.Manager().dict()
If not:
    create a new task which is added to multiproccesing.JoinableQueue
    Then create a worker which is an instance of -> class Worker(Process)
    And finally start the process that check if there are any jobs in
queue and execute them one by one.

Full implementation is available here:
https://github.com/olavgg/py-sth/blob/master/client/services/task_service.py

This works fine in development mode. But if I try to start this with
uwsgi, the process will hang when executing the JoinableQueue.get()
method (line 95)

Am I doing this completely wrong? Are there any workarounds to handle
expensive functions?

Re: [flask] Flask Application with multiprocessing and uwsgi

From:
Trey Long
Date:
2013-01-28 @ 16:34
My advice would to be to use a separate process with communication between
the two or a task management system like Celery to do this for you.
http://celeryproject.org/

It seems like a less than stellar idea to try to attach this process into 
your web stack.


On Jan 28, 2013, at 11:13 AM, "Olav Grønås Gjerde" <olavgg@gmail.com> wrote:

> I've developed a flask backend application that is syncing files and
> folders to an elasticsearch database.
> 
> I've implemented the following logic for each request:
> First check if a process with the following name already exists in
> multiprocessing.Manager().dict()
> If not:
>    create a new task which is added to multiproccesing.JoinableQueue
>    Then create a worker which is an instance of -> class Worker(Process)
>    And finally start the process that check if there are any jobs in
> queue and execute them one by one.
> 
> Full implementation is available here:
> https://github.com/olavgg/py-sth/blob/master/client/services/task_service.py
> 
> This works fine in development mode. But if I try to start this with
> uwsgi, the process will hang when executing the JoinableQueue.get()
> method (line 95)
> 
> Am I doing this completely wrong? Are there any workarounds to handle
> expensive functions?