librelist archives

Flask extensions approval process

From:

Dan Jacob

Date:

2010-07-15 @ 09:43

Would it be a good idea to have some form of approval process for
"official" extensions ? Of course anyone can release an extension -
whether on github/bitbucket or PyPi - but the Extensions page on
flask.pocoo.org is where newcomers to the framework will go first, so
it's important that these extensions have a high quality bar.

Having released some of those extensions myself, I can't speak for
their quality as it's hard to be objective about your own code, but I
would be more confident if there was some kind of formal process other
than throw the thing up on Bitbucket, maybe get some people on the IRC
to critique it, then wait for Armin to add it to the list - and he
might not have time to review the code.

Do we need a checklist for having extensions approved, or maybe
something more formal, like a test suite that checks all extensions to
make sure your extension doesn't break anything ? Is there some way of
having a group process so that Armin doesn't have to do all the work ?
This process should also include a proof-reading of the documentation,
as well as code review.

I don't want an over-bureaucratic solution, and maybe everyone is
happy about the current situation - which is quite healthy IMHO. It's
just if the framework grows (which it appears to be doing) and we have
more contributors it might be more difficult to maintain the current
system.

Hi,

Strong +1 on that proposal. I right now only list extensions after a quick
and informal review, but there are currently no real guidelines. That 
should be improved.

Also with thqt the list should explain that those are approved but that 
further extensions are available on github/pypi etc.

Regards,
Armin

(sent from a handheld device)

On 15.07.2010, at 11:43, Dan Jacob <danjac354@gmail.com> wrote:

> Would it be a good idea to have some form of approval process for
> "official" extensions ? Of course anyone can release an extension -
> whether on github/bitbucket or PyPi - but the Extensions page on
> flask.pocoo.org is where newcomers to the framework will go first, so
> it's important that these extensions have a high quality bar.
> 
> Having released some of those extensions myself, I can't speak for
> their quality as it's hard to be objective about your own code, but I
> would be more confident if there was some kind of formal process other
> than throw the thing up on Bitbucket, maybe get some people on the IRC
> to critique it, then wait for Armin to add it to the list - and he
> might not have time to review the code.
> 
> Do we need a checklist for having extensions approved, or maybe
> something more formal, like a test suite that checks all extensions to
> make sure your extension doesn't break anything ? Is there some way of
> having a group process so that Armin doesn't have to do all the work ?
> This process should also include a proof-reading of the documentation,
> as well as code review.
> 
> I don't want an over-bureaucratic solution, and maybe everyone is
> happy about the current situation - which is quite healthy IMHO. It's
> just if the framework grows (which it appears to be doing) and we have
> more contributors it might be more difficult to maintain the current
> system.

Here are some ideas for the checklist, to get the ball rolling:

1. Code must be PEP 8 compliant.
2. Code must follow extensions layout (uses flaskext.* package etc)
3. All dependencies must be included in setup.py if possible
4. Test coverage must be at least 80% (or whatever seems like a reasonable %)
5. Documentation must follow Flask style
6. Any particular requirements are made clear (e.g. requires Python
2.7, not tested on Windows, etc)
7. If applicable, executable code examples should be provided.

The process should include:

1. Integration tests
2. Code review
3. Documentation proof-reading

The maintainer should also provide a link for reporting bugs - e.g.
"submit tickets to BitBucket repository" and other contact details.

Another issue is with clashing extensions - for example, two people
want to release a Flask-Twitter extension. There should really only be
one extension, as multiple extensions all doing basically the same
thing can be confusing (see the multitude of Rails gems and Django
apps for pagination for example). If someone comes up with a much
better Flask-Twitter of course it might replace the current one, but
there should be only one canonical version.

On 15 July 2010 11:19, Armin Ronacher <armin.ronacher@active-4.com> wrote:
> Hi,
>
> Strong +1 on that proposal. I right now only list extensions after a 
quick and informal review, but there are currently no real guidelines. 
That should be improved.
>
> Also with thqt the list should explain that those are approved but that 
further extensions are available on github/pypi etc.
>
> Regards,
> Armin
>
> (sent from a handheld device)
>
> On 15.07.2010, at 11:43, Dan Jacob <danjac354@gmail.com> wrote:
>
>> Would it be a good idea to have some form of approval process for
>> "official" extensions ? Of course anyone can release an extension -
>> whether on github/bitbucket or PyPi - but the Extensions page on
>> flask.pocoo.org is where newcomers to the framework will go first, so
>> it's important that these extensions have a high quality bar.
>>
>> Having released some of those extensions myself, I can't speak for
>> their quality as it's hard to be objective about your own code, but I
>> would be more confident if there was some kind of formal process other
>> than throw the thing up on Bitbucket, maybe get some people on the IRC
>> to critique it, then wait for Armin to add it to the list - and he
>> might not have time to review the code.
>>
>> Do we need a checklist for having extensions approved, or maybe
>> something more formal, like a test suite that checks all extensions to
>> make sure your extension doesn't break anything ? Is there some way of
>> having a group process so that Armin doesn't have to do all the work ?
>> This process should also include a proof-reading of the documentation,
>> as well as code review.
>>
>> I don't want an over-bureaucratic solution, and maybe everyone is
>> happy about the current situation - which is quite healthy IMHO. It's
>> just if the framework grows (which it appears to be doing) and we have
>> more contributors it might be more difficult to maintain the current
>> system.
>

On 07/15/2010 12:37 PM, Dan Jacob wrote:
> Here are some ideas for the checklist, to get the ball rolling:
> 
> 1. Code must be PEP 8 compliant.
We can use pylint and pyflakes
> 2. Code must follow extensions layout (uses flaskext.* package etc)

Hi,

On 7/15/10 12:37 PM, Dan Jacob wrote:
> Here are some ideas for the checklist, to get the ball rolling:
>
> 1. Code must be PEP 8 compliant.
I would go as far as writing down the Pocoo Styleguide in detail 
somewhere.  It mostly follows PEP 8 but has some specific guidelines on 
docstrings.

> 2. Code must follow extensions layout (uses flaskext.* package etc)
And it should only have one module or one package in there.  The current 
offender is Flask-WTF which has both wtf.py in there as well as 
recaptcha/.  wtf should become a package and recaptcha a subpackage of it :)

> 3. All dependencies must be included in setup.py if possible
Totally.  Not only that, the setup.py should also contain a egg-info 
link to the development version so that installing with ==dev works.

> 4. Test coverage must be at least 80% (or whatever seems like a reasonable %)
While I would agree with that, it is hard to test a few things.  The 
OAuth and OpenID packages are currently tested by hand.  This is not 
optimal and should be improved, but this is actually quite a tricky 
thing.  Maybe twill and Flask-Testing could help out there.

> 5. Documentation must follow Flask style
Agreed.

> 6. Any particular requirements are made clear (e.g. requires Python
> 2.7, not tested on Windows, etc)
Agreed.

> 7. If applicable, executable code examples should be provided.
I would say one example should be the requirement or the tests should be 
written in a way that usage becomes obvious.

> The process should include:
>
> 1. Integration tests
> 2. Code review
> 3. Documentation proof-reading
>
> The maintainer should also provide a link for reporting bugs - e.g.
> "submit tickets to BitBucket repository" and other contact details.
I can agree with that.

> Another issue is with clashing extensions - for example, two people
> want to release a Flask-Twitter extension. There should really only be
> one extension, as multiple extensions all doing basically the same
> thing can be confusing (see the multitude of Rails gems and Django
> apps for pagination for example). If someone comes up with a much
> better Flask-Twitter of course it might replace the current one, but
> there should be only one canonical version.
The extension development page should outline that early feedback with 
the community is a good idea.  Right now we have one Flask-Markdown 
package in the making and at least two other people announced interest 
in writing their own one.

Another requirement I would like is that the testsuite is either powered 
by unittest or that it has a hook so that it can be combined with a 
unittest runner.  I hope that this works with nose, have not tried.  The 
reason behind that is that this way the Flask could get an extra 
"setup.py test_extensions" command that pulls in all extensions into a 
virtualenv and tests them all.


Regards,
Armin

On 07/15/2010 08:45 AM, Armin Ronacher wrote:
> On 7/15/10 12:37 PM, Dan Jacob wrote:
>> 1. Code must be PEP 8 compliant.
> I would go as far as writing down the Pocoo Styleguide in detail
> somewhere.  It mostly follows PEP 8 but has some specific guidelines on
> docstrings.

I saw the Pocoo style guide on Github. How strictly would you have to 
adhere to the style guide to be listed? AFAIK, all my code is PEP 8 
compliant, but I tend to use different docstring conventions, i.e.

"""
This function does stuff.
"""

as well as a different source file header. (Text is aligned flush with 
the left margin, = is used instead of ~ in the heading borders.) Also, I 
sometimes use CONSTANT_STYLE_NAMING for precompiled regexen.

I know that's probably not enough to disclaim being registered, but you 
might want to specify something like, "Using the Pocoo Style Guide is 
recommended, but it is not required, although complying with PEP 8 is."
-- 
Regards, Matthew "LeafStorm" Frazier
http://leafstorm.us/

Hi,

On 7/15/10 7:16 PM, LeafStorm wrote:
> I saw the Pocoo style guide on Github. How strictly would you have to
> adhere to the style guide to be listed?
I don't think it makes any sense to be a dick about code styles in 
non-Flask core components.  PEP 8 should be a requirement, the Pocoo 
styleguide a recommendation.

However reStructuredText as well as a Sphinx documentation should be a 
requirement to support linking between documentations.


Regards,
Armin

That makes perfect sense, especially being strict with the documentation
requirements.

On Thu, Jul 15, 2010 at 1:27 PM, Armin Ronacher <armin.ronacher@active-4.com
> wrote:

> Hi,
>
> On 7/15/10 7:16 PM, LeafStorm wrote:
> > I saw the Pocoo style guide on Github. How strictly would you have to
> > adhere to the style guide to be listed?
> I don't think it makes any sense to be a dick about code styles in
> non-Flask core components.  PEP 8 should be a requirement, the Pocoo
> styleguide a recommendation.
>
> However reStructuredText as well as a Sphinx documentation should be a
> requirement to support linking between documentations.
>
>
> Regards,
> Armin
>



-- 
bo

That's what I thought. I was just confirming, because some people could 
interpret it the strict way.

On 07/15/2010 01:27 PM, Armin Ronacher wrote:
> Hi,
>
> On 7/15/10 7:16 PM, LeafStorm wrote:
>> I saw the Pocoo style guide on Github. How strictly would you have to
>> adhere to the style guide to be listed?
> I don't think it makes any sense to be a dick about code styles in
> non-Flask core components.  PEP 8 should be a requirement, the Pocoo
> styleguide a recommendation.
>
> However reStructuredText as well as a Sphinx documentation should be a
> requirement to support linking between documentations.
>
>
> Regards,
> Armin

-- 
Regards, Matthew "LeafStorm" Frazier
http://leafstorm.us/

>> 1. Code must be PEP 8 compliant.
> I would go as far as writing down the Pocoo Styleguide in detail
> somewhere.  It mostly follows PEP 8 but has some specific guidelines on
> docstrings.
>

Please do, that would be most helpful.

>> 2. Code must follow extensions layout (uses flaskext.* package etc)
> And it should only have one module or one package in there.  The current
> offender is Flask-WTF which has both wtf.py in there as well as
> recaptcha/.  wtf should become a package and recaptcha a subpackage of it :)
>

Yes, that's mine :-) It's on the todo list.

>> 3. All dependencies must be included in setup.py if possible
> Totally.  Not only that, the setup.py should also contain a egg-info
> link to the development version so that installing with ==dev works.
>

Agreed.

>> 4. Test coverage must be at least 80% (or whatever seems like a reasonable %)
> While I would agree with that, it is hard to test a few things.  The
> OAuth and OpenID packages are currently tested by hand.  This is not
> optimal and should be improved, but this is actually quite a tricky
> thing.  Maybe twill and Flask-Testing could help out there.

True. I suppose test coverage is something that comes under code
review, as it depends on the project at hand.

>> The maintainer should also provide a link for reporting bugs - e.g.
>> "submit tickets to BitBucket repository" and other contact details.
> I can agree with that.

There should be a set place for it, e.g. in the README and docs.

>
>> Another issue is with clashing extensions - for example, two people
>> want to release a Flask-Twitter extension. There should really only be
>> one extension, as multiple extensions all doing basically the same
>> thing can be confusing (see the multitude of Rails gems and Django
>> apps for pagination for example). If someone comes up with a much
>> better Flask-Twitter of course it might replace the current one, but
>> there should be only one canonical version.

> The extension development page should outline that early feedback with
> the community is a good idea.  Right now we have one Flask-Markdown
> package in the making and at least two other people announced interest
> in writing their own one.
>

Maybe we could outline the etiquette for this, e.g. if you have an
idea for an extension, put it to the mailing list first, as someone
else could be working on the same thing.

> Another requirement I would like is that the testsuite is either powered
> by unittest or that it has a hook so that it can be combined with a
> unittest runner.  I hope that this works with nose, have not tried.  The
> reason behind that is that this way the Flask could get an extra
> "setup.py test_extensions" command that pulls in all extensions into a
> virtualenv and tests them all.

That would be ideal, as automated as possible.

> Another requirement I would like is that the testsuite is either powered
> by unittest or that it has a hook so that it can be combined with a
> unittest runner.  I hope that this works with nose, have not tried.  The
> reason behind that is that this way the Flask could get an extra
> "setup.py test_extensions" command that pulls in all extensions into a
> virtualenv and tests them all.

Please don't forget py.test

Ali

Hi,

On 7/15/10 2:48 PM, Ali Afshar wrote:
> Please don't forget py.test
py.test can't be integrated into unittest as far as I know.


Regards,
Armin

On Thu, Jul 15, 2010 at 7:12 AM, Armin Ronacher <armin.ronacher@active-4.com
> wrote:

> Hi,
>
> On 7/15/10 2:48 PM, Ali Afshar wrote:
> > Please don't forget py.test
> py.test can't be integrated into unittest as far as I know.
>

It has a plug-in to collect and run unittest cases:
http://codespeak.net/py/dist/test/plugin/unittest.html

 Regards,
> Armin
>


=====
--Ryan E. Freckleton

Hi,

On 7/15/10 3:14 PM, Ryan Freckleton wrote:
> It has a plug-in to collect and run unittest cases:
> http://codespeak.net/py/dist/test/plugin/unittest.html
I need it the other way round.


Regards,
Armin

On Thu, Jul 15, 2010 at 7:33 AM, Armin Ronacher <armin.ronacher@active-4.com
> wrote:

> Hi,
>
> On 7/15/10 3:14 PM, Ryan Freckleton wrote:
> > It has a plug-in to collect and run unittest cases:
> > http://codespeak.net/py/dist/test/plugin/unittest.html
> I need it the other way round.
>


So, we want to have a set of tests written in py.test (or nose) style and
run them using a unittest runner?

I think there's a way to get nose/py.test to execute with a setup.py test
command...We may want to ask on the "Testing in python" list
http://lists.idyll.org/listinfo/testing-in-python .

Regards,
> Armin
>

=====
--Ryan E. Freckleton

Hi,

On 7/15/10 3:42 PM, Ryan Freckleton wrote:
> So, we want to have a set of tests written in py.test (or nose) style
> and run them using a unittest runner?
The Flask tests and the tests of most extensions are written in 
unittest.  The idea would be to combine all the tests into one suite 
before running them.  A "setup.py test" would not achieve that, only 
true unittest classes could help there.

In case Holger is at europython I will ask him there if he knows a nice 
solution.  In any case we could switch to py.test as test runner, but I 
have to carefully evaluate that because I just recently switched away 
from py.test for various reasons.


Regards,
Armin

On Thu, Jul 15, 2010 at 9:45 AM, Armin Ronacher
<armin.ronacher@active-4.com> wrote:
> On 7/15/10 3:42 PM, Ryan Freckleton wrote:
>> So, we want to have a set of tests written in py.test (or nose) style
>> and run them using a unittest runner?
> The Flask tests and the tests of most extensions are written in
> unittest.  The idea would be to combine all the tests into one suite
> before running them.  A "setup.py test" would not achieve that, only
> true unittest classes could help there.

Are you just looking for setup.py integration?

python setup.py nosetests
http://somethingaboutorange.com/mrl/projects/nose/0.11.1/api/commands.html

This would also test your .rst doctests.

-Ron

On 07/15/2010 11:43 AM, Dan Jacob wrote:
> Would it be a good idea to have some form of approval process for
> "official" extensions ? Of course anyone can release an extension -
> whether on github/bitbucket or PyPi - but the Extensions page on
> flask.pocoo.org is where newcomers to the framework will go first, so
> it's important that these extensions have a high quality bar.
+1 because if we check the modules of Django, some modules are very poor 
or outdated :/
> 
> Having released some of those extensions myself, I can't speak for
> their quality as it's hard to be objective about your own code, but I
> would be more confident if there was some kind of formal process other
> than throw the thing up on Bitbucket, maybe get some people on the IRC
> to critique it, then wait for Armin to add it to the list - and he
> might not have time to review the code.
> 
> Do we need a checklist for having extensions approved, or maybe
> something more formal, like a test suite that checks all extensions to
> make sure your extension doesn't break anything ? Is there some way of
> having a group process so that Armin doesn't have to do all the work ?
> This process should also include a proof-reading of the documentation,
> as well as code review.
if you add a checklist for the approval, don't forget to continue the 
maintenance of these modules.

+ add the number of the version, may be that a module for Flask 0.X won't 
run for Flask 1.X
> 
> I don't want an over-bureaucratic solution, and maybe everyone is
> happy about the current situation - which is quite healthy IMHO. It's
> just if the framework grows (which it appears to be doing) and we have
> more contributors it might be more difficult to maintain the current
> system.

> if you add a checklist for the approval, don't forget to continue the 
maintenance of these modules.
>

Any bugs would of course be up to the maintainer to take care of,
however major version changes might need approval.

> + add the number of the version, may be that a module for Flask 0.X 
won't run for Flask 1.X

I like the idea of a test suite that runs all extension unit tests
along with Flask, so maintainers can be alerted of breaking changes in
Flask versions ahead of a release. Of course, that requires a good set
of unit tests....