librelist archives

« back to archive

flask app profiling (visualization?)

flask app profiling (visualization?)

From:
Matthew Hoopes
Date:
2012-07-16 @ 21:02
Hi,

I'm trying to profile my flask application using the werkzeug
ProfilerMiddleware like so:

from werkzeug.contrib.profiler import ProfilerMiddleware
f = open('/tmp/profiler.log', 'a')
app.wsgi_app = ProfilerMiddleware(app.wsgi_app, f)

On each request, I get profiling information, but I was wondering if anyone
knew a good way to visualize this data? Is there a way to import it into
kcachegrind (the only one i've used before)?

I've done a good bit of googling, but haven't found anything. If there are
any links at which I can RTFM, I'd be much obliged.

Thanks very much in advance!

- matt

Re: [flask] flask app profiling (visualization?)

From:
Steven Kryskalla
Date:
2012-07-16 @ 21:05
On Mon, Jul 16, 2012 at 2:02 PM, Matthew Hoopes
<matthew.hoopes@gmail.com> wrote:
> On each request, I get profiling information, but I was wondering if anyone
> knew a good way to visualize this data? Is there a way to import it into
> kcachegrind (the only one i've used before)?

Have a look at linesman:

http://pypi.python.org/pypi/linesman/

I have used it in the past and the graph output was really useful for
visualizing the slow parts of my app.

-steve

Re: [flask] flask app profiling (visualization?)

From:
Simon Sapin
Date:
2012-07-16 @ 22:56
Le 16/07/2012 23:02, Matthew Hoopes a écrit :
> Hi,
>
> I'm trying to profile my flask application using the werkzeug
> ProfilerMiddleware like so:
>
> from werkzeug.contrib.profiler import ProfilerMiddleware
> f = open('/tmp/profiler.log', 'a')
> app.wsgi_app = ProfilerMiddleware(app.wsgi_app, f)
>
> On each request, I get profiling information, but I was wondering if
> anyone knew a good way to visualize this data? Is there a way to import
> it into kcachegrind (the only one i've used before)?
>
> I've done a good bit of googling, but haven't found anything. If there
> are any links at which I can RTFM, I'd be much obliged.
>
> Thanks very much in advance!

I use gprof2dot together with xdot to visualize cProfile data.

http://code.google.com/p/jrfonseca/wiki/Gprof2Dot
http://code.google.com/p/jrfonseca/wiki/XDot

My usual workflow is:

python -m cProfile -o profile foo.py
gprof2dot -f pstats --skew=.1 profile | xdot

If course the first part does not apply for web stuff. But since 
ProfilerMiddleware is based on cProfile the latter step should work 
unchanged.

-- 
Simon Sapin

Re: [flask] flask app profiling (visualization?)

From:
Matthew Hoopes
Date:
2012-07-17 @ 02:49
On Mon, Jul 16, 2012 at 6:56 PM, Simon Sapin <simon.sapin@exyr.org> wrote:

> Le 16/07/2012 23:02, Matthew Hoopes a écrit :
> > Hi,
> >
> > I'm trying to profile my flask application using the werkzeug
> > ProfilerMiddleware like so:
> >
> > from werkzeug.contrib.profiler import ProfilerMiddleware
> > f = open('/tmp/profiler.log', 'a')
> > app.wsgi_app = ProfilerMiddleware(app.wsgi_app, f)
> >
> > On each request, I get profiling information, but I was wondering if
> > anyone knew a good way to visualize this data? Is there a way to import
> > it into kcachegrind (the only one i've used before)?
> >
> > I've done a good bit of googling, but haven't found anything. If there
> > are any links at which I can RTFM, I'd be much obliged.
> >
> > Thanks very much in advance!
>
> I use gprof2dot together with xdot to visualize cProfile data.
>
> http://code.google.com/p/jrfonseca/wiki/Gprof2Dot
> http://code.google.com/p/jrfonseca/wiki/XDot
>
> My usual workflow is:
>
> python -m cProfile -o profile foo.py
> gprof2dot -f pstats --skew=.1 profile | xdot
>
> If course the first part does not apply for web stuff. But since
> ProfilerMiddleware is based on cProfile the latter step should work
> unchanged.
>
> --
> Simon Sapin
>
>
>
Thanks to both of you for the response. (It appears linesman is a little
out of date - got "DeprecationWarning: The attribute None is deprecated:
disabled starting WebOb 1.2, use params instead" when I went to
/__profiler__)

Simon, thanks, I feel like I'm close to understanding this, but google is
being surprisingly little help today.

The relevant part of the werkzeug middleware appears to be (in
werkzeug/contrib/profiler.py):

        self._stream.write('-' * 80)
        self._stream.write('\nPATH: %r\n' % environ.get('PATH_INFO'))
        stats.print_stats(*self._restrictions)
        self._stream.write('-' * 80 + '\n\n')

Is this a standard pstats format? I can't find the definition for it
anywhere. When I run the "./gprof2dot.py -f pstats profiler.log" command,
it appears to give me just a single point:

digraph {
        graph [ranksep=0.25, fontname=Arial, nodesep=0.125];
        node [fontname=Arial, style=filled, height=0, width=0, shape=box,
fontcolor=white];
        edge [fontname=Arial];
        0 [color="#ff0000", fontcolor="#ffffff", fontsize="10.00",
label="profile:0:profiler\n100.00%\n(100.00%)\n0×"];
}

I removed the 80 dash lines, and PATH line from the profiler.log, but then
I got an error, so obviously the provided format is correct.

Am I doing anything obviously wrong here? I feel like I'm missing something
small...

Thanks again,

- matt

Re: [flask] flask app profiling (visualization?)

From:
Audrius Kažukauskas
Date:
2012-07-19 @ 14:40
Hey, Matthew,

On Mon, 2012-07-16 at 22:49:51 -0400, Matthew Hoopes wrote:
> The relevant part of the werkzeug middleware appears to be (in
> werkzeug/contrib/profiler.py):
> 
>         self._stream.write('-' * 80)
>         self._stream.write('\nPATH: %r\n' % environ.get('PATH_INFO'))
>         stats.print_stats(*self._restrictions)
>         self._stream.write('-' * 80 + '\n\n')

I managed to make gprof2dot work by changing stats.print_stats call to
stats.dump_stats[0], which writes binary profiling data directly to the
file. That's what

  $ python -m cProfile -o output.profile app.py

is using and this output format is expected by gprof2dot -f pstats.

This change required to adapt ProfilerMiddleware class by removing
self._stream.write calls and other now unneeded stuff.

One problem still remains, as I am calling dump_stats with the same
filename for all HTTP requests, so it overwrites previous profiling
data.  This could be fixed by writing to a separate file for each
request.

After these changes I got nicely looking graph of all function calls,
which is not as easy to understand as I hoped it would be.  I think I'm
going to try RunSnakeRun[1] next, it takes a different visualization
approach.

Thanks everybody for valuable info!

[0] http://docs.python.org/library/profile.html#pstats.Stats.dump_stats
[1] http://www.vrplumber.com/programming/runsnakerun/

-- 
Audrius Kažukauskas
http://neutrino.lt/

Re: [flask] flask app profiling (visualization?)

From:
Simon Sapin
Date:
2012-07-19 @ 15:00
Le 19/07/2012 16:40, Audrius Kažukauskas a écrit :
> One problem still remains, as I am calling dump_stats with the same
> filename for all HTTP requests, so it overwrites previous profiling
> data.  This could be fixed by writing to a separate file for each
> request.

I don’t know if this would produce a valid file, but have you tried to 
just concatenate all files? Or just append to the same file to begin with.

-- 
Simon Sapin

Re: [flask] flask app profiling (visualization?)

From:
Audrius Kažukauskas
Date:
2012-07-19 @ 15:55
On Thu, 2012-07-19 at 17:00:01 +0200, Simon Sapin wrote:
> Le 19/07/2012 16:40, Audrius Kažukauskas a écrit :
> > One problem still remains, as I am calling dump_stats with the same
> > filename for all HTTP requests, so it overwrites previous profiling
> > data.  This could be fixed by writing to a separate file for each
> > request.
> 
> I don’t know if this would produce a valid file, but have you tried to 
> just concatenate all files? Or just append to the same file to begin with.

No, I haven't tried that, all I have done was a quick hack just to see
how gprof2dot and xdot visualization would look like.  Not sure if
appending to the same file is possible, as Stats.dump_stats takes
filename as its only argument, there's no way to pass file object with
append mode to it.

I tried to concatenate profile data file with itself:

  $ cat profile profile > profile2

gprof2dot is able to parse the result, but it seems that only data from
the first profile is being used.

-- 
Audrius Kažukauskas
http://neutrino.lt/

Re: [flask] flask app profiling (visualization?)

From:
Audrius Kažukauskas
Date:
2012-07-19 @ 16:25
On Thu, 2012-07-19 at 18:55:35 +0300, Audrius Kažukauskas wrote:
> No, I haven't tried that, all I have done was a quick hack just to see
> how gprof2dot and xdot visualization would look like.  Not sure if
> appending to the same file is possible, as Stats.dump_stats takes
> filename as its only argument, there's no way to pass file object with
> append mode to it.
> 
> I tried to concatenate profile data file with itself:
> 
>   $ cat profile profile > profile2
> 
> gprof2dot is able to parse the result, but it seems that only data from
> the first profile is being used.

I found another profiling middleware:

  http://pypi.python.org/pypi/repoze.profile

This one accumulates profiling data from all requests and writes it to a
file (using the binary pstats format).  It also provides web GUI to view
this data in the browser.

-- 
Audrius Kažukauskas
http://neutrino.lt/

Re: [flask] flask app profiling (visualization?)

From:
Matthew Hoopes
Date:
2012-07-19 @ 17:36
On Thu, Jul 19, 2012 at 12:25 PM, Audrius Kažukauskas
<audrius@neutrino.lt>wrote:

> On Thu, 2012-07-19 at 18:55:35 +0300, Audrius Kažukauskas wrote:
> > No, I haven't tried that, all I have done was a quick hack just to see
> > how gprof2dot and xdot visualization would look like.  Not sure if
> > appending to the same file is possible, as Stats.dump_stats takes
> > filename as its only argument, there's no way to pass file object with
> > append mode to it.
> >
> > I tried to concatenate profile data file with itself:
> >
> >   $ cat profile profile > profile2
> >
> > gprof2dot is able to parse the result, but it seems that only data from
> > the first profile is being used.
>
> I found another profiling middleware:
>
>   http://pypi.python.org/pypi/repoze.profile
>
> This one accumulates profiling data from all requests and writes it to a
> file (using the binary pstats format).  It also provides web GUI to view
> this data in the browser.
>
> --
> Audrius Kažukauskas
> http://neutrino.lt/
>

Nice!

Thanks, Audrius, I'll dig into this as soon as I can find the time

- matt