librelist archives

« back to archive

Fwd: Help with memory leak

Fwd: Help with memory leak

From:
mail@totokaka.io
Date:
2013-12-25 @ 18:01
Hello everyone,

I've just recently started using Flask(cuple of months ago), and I love it!
I've made a little web app that takes server info in the URL, connects to
the
server and replies with an image displaying the status for the server. This
is all
for Minecraft servers.

The app works very well, but I have a memory leak somewhere, which I can't
seem to
trace down. It's happening both on Flask's test server and with uwsgi.

The source code is here: https://github.com/totokaka/dmcb
 I use Pillow for imaging and dnspython for resolving SRV records. In
addition I use Flask-Cache, buildt from git, for caching the requests. To
reproduce the memory leak:

    git clone https://github.com/totokaka/dmcb.git
    cd dmcb
    virtualenv --python=python3 venv
    source ./venv/bin/activate
    pip install -r requirements.txt
    python start.py
    [New terminal/shell/instance]
    while true; do curl localhost:5000/test/eu.shotbow.net/banner.png >
/dev/null; done

On my system this leaks aprox. 1 Mb/s.

I'd be very happy if someone could assist me in finding where the leak is,
and if someone could comment on how I do things, best practises and etc.

Thank you, and merry Christmas!

*------totokaka*
*email:* mail@totokaka.io

Re: [flask] Fwd: Help with memory leak

From:
Daniel Fairhead (OMNIvision)
Date:
2013-12-26 @ 20:01
Hi totokaka,

One thing you could try, and I could be barking up totally the wrong 
tree here, is adding in an explicit call to the python garbage collector 
( import gc; gc.collect() ), say after requests:

     import gc

     @app.after_request
     def per_request_callbacks(response):
         gc.collect()
         return response

I have no idea at all if this would help at all, or if it's a good idea. 
  Tell us what happens... :-)

Another idea could be deleting the PIL image after you've written it to
the BytesIO object  (so adding in a 'del image' in after line 101 in 
generator.py).

Separating out your generator module like that and having such simple 
views is great.  If you run call the generator.banner(...) from a 
standalone python script repeatedly, do you still get the memory leak?

Welcome to flask, by the way, I hope we can help you!

Dan

O, n 12/25/2013 06:01 PM, mail@totokaka.io wrote:
> Hello everyone,
>
> I've just recently started using Flask(cuple of months ago), and I love it!
> I've made a little web app that takes server info in the URL, connects
> to the
> server and replies with an image displaying the status for the server.
> This is all
> for Minecraft servers.
>
> The app works very well, but I have a memory leak somewhere, which I
> can't seem to
> trace down. It's happening both on Flask's test server and with uwsgi.
>
> The source code is here:https://github.com/totokaka/dmcb
> <https://github.com/totokaka/dmcb>
>   I use Pillow for imaging and dnspython for resolving SRV records. In
> addition I use Flask-Cache, buildt from git, for caching the requests.
> To reproduce the memory leak:
>
>      git clone https://github.com/totokaka/dmcb.git
>      cd dmcb
>      virtualenv --python=python3 venv
>      source ./venv/bin/activate
>      pip install -r requirements.txt
>      python start.py
>      [New terminal/shell/instance]
>      while true; do curl localhost:5000/test/eu.shotbow.net/banner.png
> <http://eu.shotbow.net/banner.png> > /dev/null; done
>
> On my system this leaks aprox. 1 Mb/s.
>
> I'd be very happy if someone could assist me in finding where the leak
> is, and if someone could comment on how I do things, best practises and etc.
>
> Thank you, and merry Christmas!
> *------
> totokaka*
> *email:* mail@totokaka.io <mailto:mail@totokaka.io>
>

Re: [flask] Fwd: Help with memory leak

From:
Paul Hoogendijk
Date:
2013-12-26 @ 21:43
Hi,

When I run into situations like these I look for the knife. I cut my
program in half, so comment out the half, and see if the problem persist.
And so on. Primarily you have to find out if they leak is caused by
something you have introduced by the tools you use, and then particular the
sequences in which you use those troops.

If this doesn't help try to run each separate component in high volume to
skip components.

Hopefully this helps a little.
On Dec 25, 2013 7:03 PM, "mail@totokaka.io" <mail@totokaka.io> wrote:

> Hello everyone,
>
> I've just recently started using Flask(cuple of months ago), and I love it!
> I've made a little web app that takes server info in the URL, connects to
> the
> server and replies with an image displaying the status for the server.
> This is all
> for Minecraft servers.
>
> The app works very well, but I have a memory leak somewhere, which I can't
> seem to
> trace down. It's happening both on Flask's test server and with uwsgi.
>
> The source code is here: https://github.com/totokaka/dmcb
>  I use Pillow for imaging and dnspython for resolving SRV records. In
> addition I use Flask-Cache, buildt from git, for caching the requests. To
> reproduce the memory leak:
>
>     git clone https://github.com/totokaka/dmcb.git
>     cd dmcb
>     virtualenv --python=python3 venv
>     source ./venv/bin/activate
>     pip install -r requirements.txt
>     python start.py
>     [New terminal/shell/instance]
>     while true; do curl localhost:5000/test/eu.shotbow.net/banner.png >
> /dev/null; done
>
> On my system this leaks aprox. 1 Mb/s.
>
> I'd be very happy if someone could assist me in finding where the leak is,
> and if someone could comment on how I do things, best practises and etc.
>
> Thank you, and merry Christmas!
>
> *------totokaka*
> *email:* mail@totokaka.io
>
>

Re: [flask] Fwd: Help with memory leak

From:
Julien Meyer
Date:
2013-12-27 @ 05:49
Hi

I too do something like Paul:

I place some simple timers inside the code before/after major
functions/operations and then study the logs it returns.

Hi,

When I run into situations like these I look for the knife. I cut my
program in half, so comment out the half, and see if the problem persist.
And so on. Primarily you have to find out if they leak is caused by
something you have introduced by the tools you use, and then particular the
sequences in which you use those troops.

If this doesn't help try to run each separate component in high volume to
skip components.

Hopefully this helps a little.
On Dec 25, 2013 7:03 PM, "mail@totokaka.io" <mail@totokaka.io> wrote:

> Hello everyone,
>
> I've just recently started using Flask(cuple of months ago), and I love it!
> I've made a little web app that takes server info in the URL, connects to
> the
> server and replies with an image displaying the status for the server.
> This is all
> for Minecraft servers.
>
> The app works very well, but I have a memory leak somewhere, which I can't
> seem to
> trace down. It's happening both on Flask's test server and with uwsgi.
>
> The source code is here: https://github.com/totokaka/dmcb
>  I use Pillow for imaging and dnspython for resolving SRV records. In
> addition I use Flask-Cache, buildt from git, for caching the requests. To
> reproduce the memory leak:
>
>     git clone https://github.com/totokaka/dmcb.git
>     cd dmcb
>     virtualenv --python=python3 venv
>     source ./venv/bin/activate
>     pip install -r requirements.txt
>     python start.py
>     [New terminal/shell/instance]
>     while true; do curl localhost:5000/test/eu.shotbow.net/banner.png >
> /dev/null; done
>
> On my system this leaks aprox. 1 Mb/s.
>
> I'd be very happy if someone could assist me in finding where the leak is,
> and if someone could comment on how I do things, best practises and etc.
>
> Thank you, and merry Christmas!
>
> *------totokaka*
> *email:* mail@totokaka.io
>
>

Re: [flask] Fwd: Help with memory leak

From:
mail@totokaka.io
Date:
2013-12-27 @ 11:18
Thanks for all your help.
The first thing I did for testing this after I heard from you was to
extract everything related to Flask out of the generator module, and just
have a memoized wrapper method in the views module.
Then I removed the networking and generated pictures at a high frequency
-->leaking like 30 mb/s. Then I just took my font module, and it seems like
it might be that which leaks. Calling the collec module on the garbage
collector doesn't help, it doesn't find anything to throw away. I haven't
taken time to test the font module to find the leak yet, but I will do
later.


*------totokaka*
*email:* mail@totokaka.io


2013/12/27 Julien Meyer <jmeyer@itpros.fr>

> Hi
>
> I too do something like Paul:
>
> I place some simple timers inside the code before/after major
> functions/operations and then study the logs it returns.
>
> Hi,
>
> When I run into situations like these I look for the knife. I cut my
> program in half, so comment out the half, and see if the problem persist.
> And so on. Primarily you have to find out if they leak is caused by
> something you have introduced by the tools you use, and then particular the
> sequences in which you use those troops.
>
> If this doesn't help try to run each separate component in high volume to
> skip components.
>
> Hopefully this helps a little.
> On Dec 25, 2013 7:03 PM, "mail@totokaka.io" <mail@totokaka.io> wrote:
>
>> Hello everyone,
>>
>> I've just recently started using Flask(cuple of months ago), and I love
>> it!
>> I've made a little web app that takes server info in the URL, connects to
>> the
>> server and replies with an image displaying the status for the server.
>> This is all
>> for Minecraft servers.
>>
>> The app works very well, but I have a memory leak somewhere, which I
>> can't seem to
>> trace down. It's happening both on Flask's test server and with uwsgi.
>>
>> The source code is here: https://github.com/totokaka/dmcb
>>  I use Pillow for imaging and dnspython for resolving SRV records. In
>> addition I use Flask-Cache, buildt from git, for caching the requests. To
>> reproduce the memory leak:
>>
>>     git clone https://github.com/totokaka/dmcb.git
>>     cd dmcb
>>     virtualenv --python=python3 venv
>>     source ./venv/bin/activate
>>     pip install -r requirements.txt
>>     python start.py
>>     [New terminal/shell/instance]
>>     while true; do curl localhost:5000/test/eu.shotbow.net/banner.png >
>> /dev/null; done
>>
>> On my system this leaks aprox. 1 Mb/s.
>>
>> I'd be very happy if someone could assist me in finding where the leak
>> is, and if someone could comment on how I do things, best practises and etc.
>>
>> Thank you, and merry Christmas!
>>
>> *------totokaka*
>> *email:* mail@totokaka.io
>>
>>