librelist archives

Re: [flask]Why multiprocessing does not work in the flask view function?

From:

刘一新

Date:

2011-09-30 @ 14:36

Yes, now I can view the simulation results and it looks amazing to me :)
However, I encountered another problem when I want to terminate the
subprocess.
For the flask builtin server, it's ok to terminate the subprocess in the
view function by call
p.terminate()
p.join()

But when I dispatch the app through apache+mod_wsgi, the main flask app
crashed after I browse the view which includes the termination. The
subprocess is not terminated.

Have any idea? You can check this issue described on
bitbucket.org/liuyxpp/ngpy

Regards,

Eshin

2011/9/30 Joe Esposito <espo58@gmail.com>

> Ok then yes, if you're utilizing multiple processors then multiprocessing
> is perfect.
>
> Does viewing the page start the simulation as you expected now?
>
> On Fri, Sep 30, 2011 at 2:29 AM, 刘一新 <liuyxpp@gmail.com> wrote:
>
>> Hi, everyone again.
>>
>> Thanks, Joe.
>>
>> I prefer multiprocessing over threads because I want to utilize my
>> multiple processors in my lab's computing cluster. According to the python
>> docs, it state that "the multiprocessing module allows the programmer to
>> fully leverage multiple processors on a given machine".
>>
>> My background tasks are both time-consuming and computing intensive. In
>> particular, they are Monte-Carlo simulations.
>>
>> Now I succeeded in managing my MC simulation tasks via web using
>> multiprocessing and flask. Thanks again for the help. Soon I will make this
>> app public at Bitbucket (https://bitbucket.org/liuyxpp/ngpy). Maybe
>> someone will be interested in it. I also hope someone can help me improve
>> the code because I am really naive at WEB programming.
>>
>> Eshin
>>
>> 2011/9/29 Joe Esposito <espo58@gmail.com>
>>
>>> Why multiprocessing and not a thread or external task daemon?
>>>
>>> I ask because the purpose of multiprocessing is to get around the GIL,
>>> not for general background work where a thread could work.
>>> Take a look at this snippet <http://flask.pocoo.org/snippets/73/>, which
>>> uses Redis. Is that what you're looking for?
>>>
>>> To answer your question though, you need to call p.start() before
>>> returning to start the subprocess.
>>>
>>
>


-- 
*Dr. Yi-Xin Liu*
*Department of Macromolecular Science*
*Fudan University*
*Room 415, Yuejing Building *
*Handan Rd. 220, **Shanghai, China*
*Tel +86-021-65642863*
*Mobile +86-13916819745*
http://www.mendeley.com/profiles/yi-xin-liu/

Hi,

On 2011-09-30 4:36 PM, 刘一新 wrote:
> Have any idea? You can check this issue described on
> bitbucket.org/liuyxpp/ngpy <http://bitbucket.org/liuyxpp/ngpy>
WSGI does not support spawning one process from another.  That might 
work it it might not work.  What I would recommend doing is using redis 
as a queue to message from your web app to the queue worker to create a 
new process.


Regards,
Armin

Thanks, Armin.
I am sorry for my poor knowledge of multiple process programming. I have
several questions about your "Basic Message Queue with Redis", some may seem
silly to you.

1. Is the dumps and loads in your code json.dumps and json.loads?
    when I experiment it in python shell, it gives
    <function add at 0x140f320> is not JSON serializable

2. Do I need to run the queue runner before do any queue task?
i. e. run
    >python /path/to/run_queue_daemon.py
run_queue_daemon.py is the script containing queue_daemon(app)

3. In your queue_daemon, all functions will be run in the same process as
the script run_queue_daemon.py. Am I right?

4. If I want to run each function in a separate process, I shall run them
using multiprocessing module in queue_daemon(). Am I right?

And an out of the scope question:
*. How can I deploy the process to another node in my cluster?
My cluster has a main node 'console', and other nodes are named 'c0101',
'c0102', 'c0103', etc.

I'm sorry for these basic questions. Thanks again.

Eshin @ Fudan Univ.


2011/9/30 Armin Ronacher <armin.ronacher@active-4.com>

> Hi,
>
> On 2011-09-30 4:36 PM, 刘一新 wrote:
> > Have any idea? You can check this issue described on
> > bitbucket.org/liuyxpp/ngpy <http://bitbucket.org/liuyxpp/ngpy>
> WSGI does not support spawning one process from another.  That might
> work it it might not work.  What I would recommend doing is using redis
> as a queue to message from your web app to the queue worker to create a
> new process.
>
>
> Regards,
> Armin
>



-- 
*Dr. Yi-Xin Liu*
*Department of Macromolecular Science*
*Fudan University*
*Room 415, Yuejing Building *
*Handan Rd. 220, **Shanghai, China*
*Tel +86-021-65642863*
*Mobile +86-13916819745*
http://www.mendeley.com/profiles/yi-xin-liu/

Hi,

On 2011-10-01 7:01 AM, 刘一新 wrote:
> 1. Is the dumps and loads in your code json.dumps and json.loads?
> when I experiment it in python shell, it gives
> <function add at 0x140f320> is not JSON serializable
Yep.  It's pickle.loads/dumps.  I updated the snippet.  You cannot 
pickle functions that are not in real python modules so it fails from 
the Python shell.

> 2. Do I need to run the queue runner before do any queue task?
> i. e. run
>  >python /path/to/run_queue_daemon.py
> run_queue_daemon.py is the script containing queue_daemon(app)
The queue runner indeed has to run separately somewhere.

> 3. In your queue_daemon, all functions will be run in the same process
> as the script run_queue_daemon.py. Am I right?
Yes.  So the trick is to make the function execute in a separate process 
with multiprocessing.  You will have to adapt the snippet.

> 4. If I want to run each function in a separate process, I shall run
> them using multiprocessing module in queue_daemon(). Am I right?
Yup.

> And an out of the scope question:
> *. How can I deploy the process to another node in my cluster?
> My cluster has a main node 'console', and other nodes are named 'c0101',
> 'c0102', 'c0103', etc.
Personally, I would not use multiprocessing at all.  Since you can start 
the queue_daemon multiple times you just need to start it up 20 times on 
different machines.  Make sure all connect to the same redis machine. 
Multiprocessing is not really suitable for web applications.


Regards,
Armin

On Oct 1, 2011, at 8:01 AM, 刘一新 wrote:

> Thanks, Armin.
> I am sorry for my poor knowledge of multiple process programming. I have
several questions about your "Basic Message Queue with Redis", some may 
seem silly to you.  
> 
> 1. Is the dumps and loads in your code json.dumps and json.loads?
>     when I experiment it in python shell, it gives
>     <function add at 0x140f320> is not JSON serializable

Then perhaps you should change the messages to be strings instead of 
functions, it's easier to debug and more robust. Failing that, use 
"pickle":

  from cPickle import loads, dumps

> 2. Do I need to run the queue runner before do any queue task?
> i. e. run
>     >python /path/to/run_queue_daemon.py
> run_queue_daemon.py is the script containing queue_daemon(app)
> 
> 3. In your queue_daemon, all functions will be run in the same process 
as the script run_queue_daemon.py. Am I right?
> 
> 4. If I want to run each function in a separate process, I shall run 
them using multiprocessing module in queue_daemon(). Am I right?

Yes, for all 3 questions above.

> And an out of the scope question:
> *. How can I deploy the process to another node in my cluster?
> My cluster has a main node 'console', and other nodes are named 'c0101',
'c0102', 'c0103', etc.

You can move "run_queue_daemon.py" to another node, all you need to change
is the connection to Redis. If you move the queue runner and Redis to node
c0103, in the flask application you need to write:

  redis = Redis(host='c0103')

> I'm sorry for these basic questions. Thanks again.

Don't apologise for asking questions, that's why this list exists, and 
other people will learn from your questions and answers :)

Cheers,
-- Alex