I'm making some large changes to Flask-Testing. You can see the code
up on the alfajor-refactoring branch on
bitbucket.org/danjac/flask-testing, but bear in mind it's very early
stage, and needs refactoring, docs and more tests.
The current TestCase and TwillTestCase will remain for the time being
for backwards compatiibility, although I plan to deprecate
TwillTestCase. Twill functionality will eventually be provided through
alfajor through a plugin.
The basic ideas are:
1. Integration with alfajor. This project is still very young, and
needs above all some documentation. Eventually however functional
testing will be optionally provided through alfajor.
2. Move to a decorator-based API. Flask-Testing will use a pattern
familiar in Flask and other extensions:
from flaskext.testing import Testing
from myproject import app
testing = Testing(app)
@testing.with_context
def test_x():
assert 1 < 2
The @with_context decorator runs the test under
app.test_request_context, so current_app etc are available, as well as
test context variables (see below).
You can add common setup/teardown functionality through decorators:
@testing.setup
def create_db():
db.create_all()
@testing.teardown
def destroy_db():
db.drop_all()
db.session.remove()
3. Local proxies will be used for managing test-local context, in a
similar way as Flask request, g, etc:
- client - Flask test client (eventually alfajor APIClient)
- browser - alfajor WebBrowser instance, if available
- fixtures - a place to stick various test data, like Flask "g" object
- templates - list of templates/contexts used in a request, if blinker installed
The fixtures object (for example) will be available to your @setup and
@teardown functions.
4. A module system like Flask Module, to allow you to create groups of
tests with related conditions:
users = testing.register_module()
@users.setup
def create_user_data():
# testing setup functions already called
# so database initialized
user = User(name="Joe")
db.session.add(user)
db.session.commit()
fixtures.user = user # only appears in this module
@users.with_context
def test_user_name():
assert fixtures.user.name == "Joe"
5. Methods in TestCase such as assert_200 or assert_redirects will now
be available as functions. In addition nose.tools functions such as
assert_equals or ok_ will be added to the namespace.
As this is all very early stage I'd like to request any suggestions or
feature requests from the mailing list.
These changes will appear in Flask-Testing 0.4.
I have some ideas for a testing API. I've been considering writing a
testing library not specific to Flask, but I don't know if it'll happen.
- I think setup/teardown is an outdated pattern.
- Name sensitivity smells like string programming to me and is ugly
- Decorators are pretty :)
- Context managers are ze future
- Not in love with custom assert functions, plain assert is prettier but
less helpful errors... Maybe some frame hacking can make it right.
Instead of setup/teardown I suggest context managers and/or decorators,
yes, decorators. As they wrap functions they can add setup and teardown
code before and after.
Instead of name sensitivity I suggest function attributes and a
convenience decorator.
@contextmanager
def db_connection(db):
db.connect()
yield
db.disconnect()
# A custom decorator can be used to control a test
# like with setup/teardown methods
def db_test(f):
@test # like nose' @istest but I prefer this name
@wraps(f) # Could have a @test_wrapping(f) convenience
def wrapper(fixture): # Or fixture could be a context-local
# But Armin seems to disapprove
# We can modify the fixture here
# But we'll just wrap the test in a DB connection
with db_connection(db):
f(fixture)
return wrapper
@db_test
def function_testing_something(fixture):
"""Human readable description of what this tests"""
assert fixture.something == other
# Test not using a setup
@test
def testing_something_else(fixture):
assert fixture.other == something
# The above runs without a DB connection
# But we can freely use the context manager wherever we wish
with db_connection(db):
assert app.db_dependent is not None
I also have this idea that maybe tests should be a sub-package inside
your project package, maybe with __main__ so that this runs the tests:
python -m myproject.tests
My general feel is that Python concepts and constructs should be reused
where applicable, and referred to as such, in example a context manager
is a context manager not a "fixture setup" etc; context managers
shouldn't be something required, just a pattern that comes naturally.
The same is true for the use of decorators. A test is just a very stupid
function that might raise some exceptions; Python has powerful
constructs to let us build on stupid functions.
Similarly, expected exceptions would naturally be tested with context
managers:
# Provided by library
@contextmanager
def raising(exception):
try:
yield
except exception:
pass
else:
raise AssertionError('did not raise %s' % exception.__name__)
On 30 July 2010 23:13, Dag Odenhall <dag.odenhall@gmail.com> wrote: > I have some ideas for a testing API. I've been considering writing a > testing library not specific to Flask, but I don't know if it'll happen. > If you can come up with a better way of doing tests, that would be great. However I'm looking not to reinvent any wheels but really to find an easier way to integrate Flask with existing tools. > - I think setup/teardown is an outdated pattern. That's interesting; why ? > - Name sensitivity smells like string programming to me and is ugly No argument there, a case of convention over configuration that is a bit of (ugly) magic. > - Decorators are pretty :) Again, no disagreement. > - Context managers are ze future Maybe. But see my comment below. > - Not in love with custom assert functions, plain assert is prettier but > less helpful errors... Maybe some frame hacking can make it right. Custom assert functions are just helpers. You can write assert response.status_code === 200 or assert_200(response). The former is more explicit, the latter saves typing. Do whatever suits your style best. Personally I use assert instead of assert_equals et all but use more complex cases like assert_redirects etc simply as a timesaver. > > Instead of setup/teardown I suggest context managers and/or decorators, > yes, decorators. As they wrap functions they can add setup and teardown > code before and after. > > Instead of name sensitivity I suggest function attributes and a > convenience decorator. > I agree with you up to a point, and in general I like the decorator style, which seems to be an emerging pattern in Flask. I've adopted this style in Flask-Script for example. Where I might disagree is when it comes to unit tests. To me, tests should be simple and dumb. My code might be clever, and complex, but the tests covering that code should be obvious and simple. I can see what's been addJaed in setUp and what's been reset in tearDown. I can do all the basic setup (e.g. creating/dropping database tables) in a base class, inherit from that, and just get on with it. Test classes are just that - simple and dumb. Decorators are neat, and more Pythonic, but they make the test code less obvious, and that's not what I want in my tests. I agree that unittest.TestCase (which flaskext.testing.TestCase) is Java-ish, and a bit old-fashioned and ugly, but it's clear how it works (outside of the name sensitivity) and it just does work. If things go wrong and I get errors or failures I can worry about the code I'm testing, not worry about whether I've set up my decorators correctly. I can drop fixtures in setup and be done with it, I don't need to think about using some magic proxy solution or a non-DRY approach of having to call the same fixture loading function in a dozen places. That all said, I appreciate a lot of people, like yourself, have a different point of view, and one I respect. The two biggest testing libraries in Python outside the stdlib are nose and py.test, and I'd like Flask-Testing to be able to support these, decorators and all; the question is what the best way to do it is. If another library comes along, like yours for example, or unittest2, I'd like to support that too. The trick is to be able to do that in such a way as not to rely on breaking changes to these libraries. I'm open to suggestions here from those who use nose/py.test and others on what might be the best way for Flask-Testing to support the function/decorator style.
> > - I think setup/teardown is an outdated pattern. > > That's interesting; why ? It's kinda *the* definition of a context manager; we have more specific ways to deal with that. > > - Name sensitivity smells like string programming to me and is ugly > > No argument there, a case of convention over configuration that is a > bit of (ugly) magic. So to me seems the best way to mark something as a test is with a decorator, and if we already need a decorator it follows naturally that special purpose decorators is used for specific types of tests. > > > - Decorators are pretty :) > > Again, no disagreement. > > > - Context managers are ze future > > Maybe. But see my comment below. They're also reusable which setup/teardown isn't as easily. > > > - Not in love with custom assert functions, plain assert is prettier but > > less helpful errors... Maybe some frame hacking can make it right. > > Custom assert functions are just helpers. You can write assert > response.status_code === 200 or assert_200(response). The former is > more explicit, the latter saves typing. Do whatever suits your style > best. Personally I use assert instead of assert_equals et all but use > more complex cases like assert_redirects etc simply as a timesaver. assert status(200) assert redirects('/') > > > > Instead of setup/teardown I suggest context managers and/or decorators, > > yes, decorators. As they wrap functions they can add setup and teardown > > code before and after. > > > > Instead of name sensitivity I suggest function attributes and a > > convenience decorator. > > > > I agree with you up to a point, and in general I like the decorator > style, which seems to be an emerging pattern in Flask. I've adopted > this style in Flask-Script for example. > > Where I might disagree is when it comes to unit tests. > > To me, tests should be simple and dumb. My code might be clever, and > complex, but the tests covering that code should be obvious and > simple. I can see what's been addJaed in setUp and what's been reset > in tearDown. I can do all the basic setup (e.g. creating/dropping > database tables) in a base class, inherit from that, and just get on > with it. > > Test classes are just that - simple and dumb. Decorators are neat, and > more Pythonic, but they make the test code less obvious, and that's > not what I want in my tests. I must protest; classes are way more complex than a simple function, and much less readable. It is also see what happens before and after a test in the decorator. Can also nest decorators for multiple setups. Decorators being decorative, it's very readable. Backporting Python 3's @contextmanager and we can also make decorators that way: # Well, needs to set the "this is a test" attribute too @contextmanager def db_test(): db.connect() with closing(db): yield @db_test def connects_to_db(): assert g.db Note also how we got a try-finally for free here with closing(); can you even do that with teardown? > I agree that unittest.TestCase (which flaskext.testing.TestCase) is > Java-ish, and a bit old-fashioned and ugly, but it's clear how it > works (outside of the name sensitivity) and it just does work. If > things go wrong and I get errors or failures I can worry about the > code I'm testing, not worry about whether I've set up my decorators > correctly. I can drop fixtures in setup and be done with it, I don't > need to think about using some magic proxy solution or a non-DRY > approach of having to call the same fixture loading function in a > dozen places. A major obstacle for me in learning testing has been the lack of clarity regarding how the * you use that crazy, restrictive API. I think it's "clear" to you because you're used to it. Don't you prefer Flask-style @app.route over Rails-style controllers? Controllers never made sense to me and I felt locked inside an non-sensible API; I much prefer the Flasky *lack* of an API - it's just Python, I know Python, I'm in charge. My suggested use of decorators and context managers are mainly an example of how I'd use my suggested test runner; at the core a test is simply a callable whose __test__ (or similar) attribute is True. The library might provide some convenience helpers, but they really are just helpers. You could easily do classes if you prefer: class TestCase(object): __test__ = True def __call__(self): for test in self.tests: self.setup() getattr(self, test)() self.teardown() class TestDB(TestCase): tests = ('test_connection',) def setup(self): db.connect() def teardown(self): db.disconnect() def test_connection(self): assert g.db A (much) more solid version of TestCase could be provided by the library. This exact API is just something I made up on the spot and might not be optimal. Actually needs a metaclass to work, or we can add to the definition of a test that if it issubclass object it needs to be instantiated. Whatever you go with, maybe you could build the core on my ideas and provide higher level APIs on top of it. > > That all said, I appreciate a lot of people, like yourself, have a > different point of view, and one I respect. The two biggest testing > libraries in Python outside the stdlib are nose and py.test, and I'd > like Flask-Testing to be able to support these, decorators and all; > the question is what the best way to do it is. If another library > comes along, like yours for example, or unittest2, I'd like to support > that too. The trick is to be able to do that in such a way as not to > rely on breaking changes to these libraries. > > I'm open to suggestions here from those who use nose/py.test and > others on what might be the best way for Flask-Testing to support the > function/decorator style.
> > They're also reusable which setup/teardown isn't as easily. Sorry, how not ? A setUp method can be inherited by a subclass - how is it not reusable ? > >> >> > - Not in love with custom assert functions, plain assert is prettier but >> > less helpful errors... Maybe some frame hacking can make it right. >> >> Custom assert functions are just helpers. You can write assert >> response.status_code === 200 or assert_200(response). The former is >> more explicit, the latter saves typing. Do whatever suits your style >> best. Personally I use assert instead of assert_equals et all but use >> more complex cases like assert_redirects etc simply as a timesaver. > > assert status(200) > assert redirects('/') > How is the response passed in there ? Is that an omission, or are we just arguing about syntactic sugar ? >> > >> > Instead of setup/teardown I suggest context managers and/or decorators, >> > yes, decorators. As they wrap functions they can add setup and teardown >> > code before and after. >> > And how is this easier/clearer ? > I must protest; classes are way more complex than a simple function, and > much less readable. > Depends on the function, and depends on the class. That's a huge over-simplification. > It is also see what happens before and after a test in the decorator. > Can also nest decorators for multiple setups. Decorators being > decorative, it's very readable. > Again, depends. Nesting n decorators may not be readable at all, and may not be easy to debug. > Note also how we got a try-finally for free here with closing(); can you > even do that with teardown? Actually I do that inside of Flask-Testing. > A major obstacle for me in learning testing has been the lack of clarity > regarding how the * you use that crazy, restrictive API. I think it's > "clear" to you because you're used to it. Don't you prefer Flask-style > @app.route over Rails-style controllers? Controllers never made sense to > me and I felt locked inside an non-sensible API; I much prefer the > Flasky *lack* of an API - it's just Python, I know Python, I'm in > charge. And unittest is Python; it's in the standard library. Granted it is overdue for overhaul and replacement, but it's well-known, stable and just works. Back to what I said in previous emails, I don't want to reinvent the wheel. Flask-Testing exists to make it easier to do unit tests in Flask, not to reinvent the way you do testing. What it lacks right now is support for the decorator-based testing libraries such as py.test and nose, and I would like to provide support for those - even if I disagree with that way, I respect a lot of people do not. If you have a better idea of how to do Python testing in general - great ! Go and write a new testing library. I'd then like to extend Flask-Testing support for that as well.
> > > > They're also reusable which setup/teardown isn't as easily. > > Sorry, how not ? A setUp method can be inherited by a subclass - how > is it not reusable ? > > > > >> > >> > - Not in love with custom assert functions, plain assert is prettier but > >> > less helpful errors... Maybe some frame hacking can make it right. > >> > >> Custom assert functions are just helpers. You can write assert > >> response.status_code === 200 or assert_200(response). The former is > >> more explicit, the latter saves typing. Do whatever suits your style > >> best. Personally I use assert instead of assert_equals et all but use > >> more complex cases like assert_redirects etc simply as a timesaver. > > > > assert status(200) > > assert redirects('/') > > > > How is the response passed in there ? Is that an omission, or are we > just arguing about syntactic sugar ? I was thinking it's a context local, but I was confusing it with request. The point is that I'd prefer a predicate that says True or False, that you can then apply to an assertion. OTOH I'm not sure I'm convinced assert needs to be a keyword to begin with, so there's two sides to this coin. But now that it is, it sticks out more, means less parenthesis and is more generalized, "consistent" if you will. Without an assert keyword I might instead argue that tests should raise different descriptive exceptions, not simply AssertionError or similar. I just have this feeling that if a skilled Python programmer invented testing *today*, from scratch, for the first time; how we thought about it would be very different from what we do today. What we do today is based on ideas from long ago and from Java etc. We stopped doing that in other areas, why not in testing? I respect the argument that the tests code should be simple so that bugs in the tests themselves are less likely. But OTOH I personally find unittest style API more complex than my proposal; as you've seen I've displayed numerous misconceptions and misunderstandings (such as inability to do try..finally) and this is due to the "unnatural" way tests are created. > >> > > >> > Instead of setup/teardown I suggest context managers and/or decorators, > >> > yes, decorators. As they wrap functions they can add setup and teardown > >> > code before and after. > >> > > > And how is this easier/clearer ? I think it's more conventional and consistent with Python idioms and as such come more naturally to (some) programmers. But as I said, at the core a test should be so simple that no idiom or API is enforced - this is my main point of advocacy. The library can provide some suggestive APIs but it should be possible, easy and natural to "roll your own". > > I must protest; classes are way more complex than a simple function, and > > much less readable. > > > > Depends on the function, and depends on the class. That's a huge > over-simplification. Probably true. For certain cases, classes are better than some alternatives. I'm not convinced testing is one of those cases. > > It is also see what happens before and after a test in the decorator. > > Can also nest decorators for multiple setups. Decorators being > > decorative, it's very readable. > > > > Again, depends. Nesting n decorators may not be readable at all, and > may not be easy to debug. I disagree; it's very obvious what a decorator does if you read it. Not any less than what inheriting from a class does with classes. That the setup and teardown methods are special is much more magic and less obvious than what goes on in a decorator; have to follow the inheritance chain possibly up to the topmost parent, and it's not necessarily obvious that those methods are called or how. In a decorator, it's obvious because code simply surrounds a call to f() or whatever, similarly yield in a @contextmanager. And again: I'm just advocating that a test library should *allow* these patterns in not defining what a test is and how it's discovered too strictly or specifically. A simple definition like "callable(o) and o.__test__ == True" allows developers to do what works for them and their needs, including classes and setup/teardown methods. > > > Note also how we got a try-finally for free here with closing(); can you > > even do that with teardown? > > Actually I do that inside of Flask-Testing. See how the unittest API confuses me. ;) I'm not saying it confuses everyone, but does some; I'm merely arguing for a lack of API enforcement at the core. Flexibility. > > A major obstacle for me in learning testing has been the lack of clarity > > regarding how the * you use that crazy, restrictive API. I think it's > > "clear" to you because you're used to it. Don't you prefer Flask-style > > @app.route over Rails-style controllers? Controllers never made sense to > > me and I felt locked inside an non-sensible API; I much prefer the > > Flasky *lack* of an API - it's just Python, I know Python, I'm in > > charge. > > And unittest is Python; it's in the standard library. Granted it is > overdue for overhaul and replacement, but it's well-known, stable and > just works. > > Back to what I said in previous emails, I don't want to reinvent the > wheel. Flask-Testing exists to make it easier to do unit tests in > Flask, not to reinvent the way you do testing. What it lacks right now > is support for the decorator-based testing libraries such as py.test > and nose, and I would like to provide support for those - even if I > disagree with that way, I respect a lot of people do not. Yea, sorry; I'm rambling about my ideas and forgetting where I'm discussing them. I guess I like to try my ideas in theory on people before I go build something no one will use, or before I make mistakes others could warn me about in advance. I'm defending my idea more than I'm trying to argue that you should do your project exactly like I'm suggesting. :) But I'm defending it *because* I expect it might be controversial and I'd like to hear arguments teaching me why, if there's something to the controversy. Maybe my idea sucks, it happens, and there are many people out there (here) smarter/more experienced than me. :) > If you have a better idea of how to do Python testing in general - > great ! Go and write a new testing library. I'd then like to extend > Flask-Testing support for that as well. I might. I'll need to contemplate how important it is to me personally, and maybe hear if anyone besides me is interested/agree with my ideas at some level. Currently I'm able to do most of what I'm suggesting with nose, not by using nose "how you're supposed to" but by using nose to implement my pattern; I do it with Flask-Genshi for example. > <danjac> donrI: wrote a reply to on testing in mailing list. I like your ideas, but it's not what I'm trying to do with Flask-Testing. Hope I didn't come across too negative. No worries; I'll say the same: I get excited about my ideas and feel a need to defend them when really I'm not trying to tell you to do it like that but rather defending my intelligence in coming up with the (controversial) idea. ;) So I argue passionately, but really I'm just curious what people think of the idea. Wrong place for this though, sorry...
I would like to use Alfajor for testing - would you recommend Flask Testing as is, or is support a work in progress / some way out? Thanks, Jim. On 1 August 2010 22:51, Dag Odenhall <dag.odenhall@gmail.com> wrote: > > > > > > They're also reusable which setup/teardown isn't as easily. > > > > Sorry, how not ? A setUp method can be inherited by a subclass - how > > is it not reusable ? > > > > > > > >> > > >> > - Not in love with custom assert functions, plain assert is prettier > but > > >> > less helpful errors... Maybe some frame hacking can make it right. > > >> > > >> Custom assert functions are just helpers. You can write assert > > >> response.status_code === 200 or assert_200(response). The former is > > >> more explicit, the latter saves typing. Do whatever suits your style > > >> best. Personally I use assert instead of assert_equals et all but use > > >> more complex cases like assert_redirects etc simply as a timesaver. > > > > > > assert status(200) > > > assert redirects('/') > > > > > > > How is the response passed in there ? Is that an omission, or are we > > just arguing about syntactic sugar ? > > I was thinking it's a context local, but I was confusing it with > request. The point is that I'd prefer a predicate that says True or > False, that you can then apply to an assertion. > > OTOH I'm not sure I'm convinced assert needs to be a keyword to begin > with, so there's two sides to this coin. But now that it is, it sticks > out more, means less parenthesis and is more generalized, "consistent" > if you will. > > Without an assert keyword I might instead argue that tests should raise > different descriptive exceptions, not simply AssertionError or similar. > > I just have this feeling that if a skilled Python programmer invented > testing *today*, from scratch, for the first time; how we thought about > it would be very different from what we do today. What we do today is > based on ideas from long ago and from Java etc. We stopped doing that in > other areas, why not in testing? > > I respect the argument that the tests code should be simple so that bugs > in the tests themselves are less likely. But OTOH I personally find > unittest style API more complex than my proposal; as you've seen I've > displayed numerous misconceptions and misunderstandings (such as > inability to do try..finally) and this is due to the "unnatural" way > tests are created. > > > >> > > > >> > Instead of setup/teardown I suggest context managers and/or > decorators, > > >> > yes, decorators. As they wrap functions they can add setup and > teardown > > >> > code before and after. > > >> > > > > > And how is this easier/clearer ? > > I think it's more conventional and consistent with Python idioms and as > such come more naturally to (some) programmers. But as I said, at the > core a test should be so simple that no idiom or API is enforced - this > is my main point of advocacy. The library can provide some suggestive > APIs but it should be possible, easy and natural to "roll your own". > > > > I must protest; classes are way more complex than a simple function, > and > > > much less readable. > > > > > > > Depends on the function, and depends on the class. That's a huge > > over-simplification. > > Probably true. For certain cases, classes are better than some > alternatives. I'm not convinced testing is one of those cases. > > > > It is also see what happens before and after a test in the decorator. > > > Can also nest decorators for multiple setups. Decorators being > > > decorative, it's very readable. > > > > > > > Again, depends. Nesting n decorators may not be readable at all, and > > may not be easy to debug. > > I disagree; it's very obvious what a decorator does if you read it. Not > any less than what inheriting from a class does with classes. That the > setup and teardown methods are special is much more magic and less > obvious than what goes on in a decorator; have to follow the inheritance > chain possibly up to the topmost parent, and it's not necessarily > obvious that those methods are called or how. In a decorator, it's > obvious because code simply surrounds a call to f() or whatever, > similarly yield in a @contextmanager. > > And again: I'm just advocating that a test library should *allow* these > patterns in not defining what a test is and how it's discovered too > strictly or specifically. A simple definition like "callable(o) and > o.__test__ == True" allows developers to do what works for them and > their needs, including classes and setup/teardown methods. > > > > > > Note also how we got a try-finally for free here with closing(); can > you > > > even do that with teardown? > > > > Actually I do that inside of Flask-Testing. > > See how the unittest API confuses me. ;) > > I'm not saying it confuses everyone, but does some; I'm merely arguing > for a lack of API enforcement at the core. Flexibility. > > > > A major obstacle for me in learning testing has been the lack of > clarity > > > regarding how the * you use that crazy, restrictive API. I think it's > > > "clear" to you because you're used to it. Don't you prefer Flask-style > > > @app.route over Rails-style controllers? Controllers never made sense > to > > > me and I felt locked inside an non-sensible API; I much prefer the > > > Flasky *lack* of an API - it's just Python, I know Python, I'm in > > > charge. > > > > And unittest is Python; it's in the standard library. Granted it is > > overdue for overhaul and replacement, but it's well-known, stable and > > just works. > > > > Back to what I said in previous emails, I don't want to reinvent the > > wheel. Flask-Testing exists to make it easier to do unit tests in > > Flask, not to reinvent the way you do testing. What it lacks right now > > is support for the decorator-based testing libraries such as py.test > > and nose, and I would like to provide support for those - even if I > > disagree with that way, I respect a lot of people do not. > > Yea, sorry; I'm rambling about my ideas and forgetting where I'm > discussing them. I guess I like to try my ideas in theory on people > before I go build something no one will use, or before I make mistakes > others could warn me about in advance. I'm defending my idea more than > I'm trying to argue that you should do your project exactly like I'm > suggesting. :) > > But I'm defending it *because* I expect it might be controversial and > I'd like to hear arguments teaching me why, if there's something to the > controversy. Maybe my idea sucks, it happens, and there are many people > out there (here) smarter/more experienced than me. :) > > > If you have a better idea of how to do Python testing in general - > > great ! Go and write a new testing library. I'd then like to extend > > Flask-Testing support for that as well. > > I might. I'll need to contemplate how important it is to me personally, > and maybe hear if anyone besides me is interested/agree with my ideas at > some level. > > Currently I'm able to do most of what I'm suggesting with nose, not by > using nose "how you're supposed to" but by using nose to implement my > pattern; I do it with Flask-Genshi for example. > > > <danjac> donrI: wrote a reply to on testing in mailing list. I like > your ideas, but it's not what I'm trying to do with Flask-Testing. Hope > I didn't come across too negative. > > No worries; I'll say the same: I get excited about my ideas and feel a > need to defend them when really I'm not trying to tell you to do it like > that but rather defending my intelligence in coming up with the > (controversial) idea. ;) So I argue passionately, but really I'm just > curious what people think of the idea. Wrong place for this though, > sorry... > >
I intend to support Alfajor, however there are some reservations:
1) dependency on an alfajor.ini config. I'd prefer a solution that
allows pure-Python configuration so it can be tied in with Flask
configuration and setup.
2. lack of any proper documentation - I'd like to be able to refer to
Alfajor docs in the Flask-Testing documentation, but at present there
is just a bare minimum.
I understand the Alfajor devs are improving things and I fully intend
to support Alfajor when it's in a more user-friendly state.
That said, if you want to use Alfajor right now, here is some sample code
from alfajor import APIClient, WebBrowser
from flaskext.testing import TestCase
class MyTestCase(TestCase):
def _pre_setup(self):
super(MyTestCase, self)._pre_setup()
# replace Flask test client with Aflajor's
self.client = APIClient()
self.client.configure_in_scope()
self.browser = WebBrowser()
self.browser.configure_in_scope()
You might need to pass in certain args to configure_in_scope() calls
to get it working.
On 12 August 2010 13:33, JimG <j.gumbley@gmail.com> wrote:
> I would like to use Alfajor for testing - would you recommend Flask Testing
> as is, or is support a work in progress / some way out?
> Thanks, Jim.
>
> On 1 August 2010 22:51, Dag Odenhall <dag.odenhall@gmail.com> wrote:
>>
>> > >
>> > > They're also reusable which setup/teardown isn't as easily.
>> >
>> > Sorry, how not ? A setUp method can be inherited by a subclass - how
>> > is it not reusable ?
>> >
>> > >
>> > >>
>> > >> > - Not in love with custom assert functions, plain assert is
>> > >> > prettier but
>> > >> > less helpful errors... Maybe some frame hacking can make it right.
>> > >>
>> > >> Custom assert functions are just helpers. You can write assert
>> > >> response.status_code === 200 or assert_200(response). The former is
>> > >> more explicit, the latter saves typing. Do whatever suits your style
>> > >> best. Personally I use assert instead of assert_equals et all but use
>> > >> more complex cases like assert_redirects etc simply as a timesaver.
>> > >
>> > > assert status(200)
>> > > assert redirects('/')
>> > >
>> >
>> > How is the response passed in there ? Is that an omission, or are we
>> > just arguing about syntactic sugar ?
>>
>> I was thinking it's a context local, but I was confusing it with
>> request. The point is that I'd prefer a predicate that says True or
>> False, that you can then apply to an assertion.
>>
>> OTOH I'm not sure I'm convinced assert needs to be a keyword to begin
>> with, so there's two sides to this coin. But now that it is, it sticks
>> out more, means less parenthesis and is more generalized, "consistent"
>> if you will.
>>
>> Without an assert keyword I might instead argue that tests should raise
>> different descriptive exceptions, not simply AssertionError or similar.
>>
>> I just have this feeling that if a skilled Python programmer invented
>> testing *today*, from scratch, for the first time; how we thought about
>> it would be very different from what we do today. What we do today is
>> based on ideas from long ago and from Java etc. We stopped doing that in
>> other areas, why not in testing?
>>
>> I respect the argument that the tests code should be simple so that bugs
>> in the tests themselves are less likely. But OTOH I personally find
>> unittest style API more complex than my proposal; as you've seen I've
>> displayed numerous misconceptions and misunderstandings (such as
>> inability to do try..finally) and this is due to the "unnatural" way
>> tests are created.
>>
>> > >> >
>> > >> > Instead of setup/teardown I suggest context managers and/or
>> > >> > decorators,
>> > >> > yes, decorators. As they wrap functions they can add setup and
>> > >> > teardown
>> > >> > code before and after.
>> > >> >
>> >
>> > And how is this easier/clearer ?
>>
>> I think it's more conventional and consistent with Python idioms and as
>> such come more naturally to (some) programmers. But as I said, at the
>> core a test should be so simple that no idiom or API is enforced - this
>> is my main point of advocacy. The library can provide some suggestive
>> APIs but it should be possible, easy and natural to "roll your own".
>>
>> > > I must protest; classes are way more complex than a simple function,
>> > > and
>> > > much less readable.
>> > >
>> >
>> > Depends on the function, and depends on the class. That's a huge
>> > over-simplification.
>>
>> Probably true. For certain cases, classes are better than some
>> alternatives. I'm not convinced testing is one of those cases.
>>
>> > > It is also see what happens before and after a test in the decorator.
>> > > Can also nest decorators for multiple setups. Decorators being
>> > > decorative, it's very readable.
>> > >
>> >
>> > Again, depends. Nesting n decorators may not be readable at all, and
>> > may not be easy to debug.
>>
>> I disagree; it's very obvious what a decorator does if you read it. Not
>> any less than what inheriting from a class does with classes. That the
>> setup and teardown methods are special is much more magic and less
>> obvious than what goes on in a decorator; have to follow the inheritance
>> chain possibly up to the topmost parent, and it's not necessarily
>> obvious that those methods are called or how. In a decorator, it's
>> obvious because code simply surrounds a call to f() or whatever,
>> similarly yield in a @contextmanager.
>>
>> And again: I'm just advocating that a test library should *allow* these
>> patterns in not defining what a test is and how it's discovered too
>> strictly or specifically. A simple definition like "callable(o) and
>> o.__test__ == True" allows developers to do what works for them and
>> their needs, including classes and setup/teardown methods.
>>
>> >
>> > > Note also how we got a try-finally for free here with closing(); can
>> > > you
>> > > even do that with teardown?
>> >
>> > Actually I do that inside of Flask-Testing.
>>
>> See how the unittest API confuses me. ;)
>>
>> I'm not saying it confuses everyone, but does some; I'm merely arguing
>> for a lack of API enforcement at the core. Flexibility.
>>
>> > > A major obstacle for me in learning testing has been the lack of
>> > > clarity
>> > > regarding how the * you use that crazy, restrictive API. I think it's
>> > > "clear" to you because you're used to it. Don't you prefer Flask-style
>> > > @app.route over Rails-style controllers? Controllers never made sense
>> > > to
>> > > me and I felt locked inside an non-sensible API; I much prefer the
>> > > Flasky *lack* of an API - it's just Python, I know Python, I'm in
>> > > charge.
>> >
>> > And unittest is Python; it's in the standard library. Granted it is
>> > overdue for overhaul and replacement, but it's well-known, stable and
>> > just works.
>> >
>> > Back to what I said in previous emails, I don't want to reinvent the
>> > wheel. Flask-Testing exists to make it easier to do unit tests in
>> > Flask, not to reinvent the way you do testing. What it lacks right now
>> > is support for the decorator-based testing libraries such as py.test
>> > and nose, and I would like to provide support for those - even if I
>> > disagree with that way, I respect a lot of people do not.
>>
>> Yea, sorry; I'm rambling about my ideas and forgetting where I'm
>> discussing them. I guess I like to try my ideas in theory on people
>> before I go build something no one will use, or before I make mistakes
>> others could warn me about in advance. I'm defending my idea more than
>> I'm trying to argue that you should do your project exactly like I'm
>> suggesting. :)
>>
>> But I'm defending it *because* I expect it might be controversial and
>> I'd like to hear arguments teaching me why, if there's something to the
>> controversy. Maybe my idea sucks, it happens, and there are many people
>> out there (here) smarter/more experienced than me. :)
>>
>> > If you have a better idea of how to do Python testing in general -
>> > great ! Go and write a new testing library. I'd then like to extend
>> > Flask-Testing support for that as well.
>>
>> I might. I'll need to contemplate how important it is to me personally,
>> and maybe hear if anyone besides me is interested/agree with my ideas at
>> some level.
>>
>> Currently I'm able to do most of what I'm suggesting with nose, not by
>> using nose "how you're supposed to" but by using nose to implement my
>> pattern; I do it with Flask-Genshi for example.
>>
>> > <danjac> donrI: wrote a reply to on testing in mailing list. I like
>> your ideas, but it's not what I'm trying to do with Flask-Testing. Hope
>> I didn't come across too negative.
>>
>> No worries; I'll say the same: I get excited about my ideas and feel a
>> need to defend them when really I'm not trying to tell you to do it like
>> that but rather defending my intelligence in coming up with the
>> (controversial) idea. ;) So I argue passionately, but really I'm just
>> curious what people think of the idea. Wrong place for this though,
>> sorry...
>>
>
>
Hi, On 7/30/10 10:03 AM, Dan Jacob wrote: > The current TestCase and TwillTestCase will remain for the time being > for backwards compatiibility, although I plan to deprecate > TwillTestCase. Twill functionality will eventually be provided through > alfajor through a plugin. I'm fine with deprecating TwillTestCase, but TestCase should stay there. Many people love unittest (including me) and I don't see a reason to remove it. > 1. Integration with alfajor. This project is still very young, and > needs above all some documentation. Eventually however functional > testing will be optionally provided through alfajor. +1 > 2. Move to a decorator-based API. Flask-Testing will use a pattern > familiar in Flask and other extensions: -1 That is something nose/py.test already do. Furthermore I would love to see context locals not being used for tests because they are a pain to debug there. I am already unhappy with the request stuff, but that is not going away because there is no other way to handle that. Instead I would like to see a support for nose and py.test that provide the same functionality as the test case classes in functions for these systems. This might be the time to figure out of nose and py.test have some common ground so that you don't have to implement those functions for both. An @with_context decorator makes sense, but that decorator then has to support both test case classes and functions in a module for nose. With py.test you should probably ask Holger or Ronny for feedback. Regards, Armin
On 30 July 2010 09:46, Armin Ronacher <armin.ronacher@active-4.com> wrote: > Hi, > > On 7/30/10 10:03 AM, Dan Jacob wrote: >> The current TestCase and TwillTestCase will remain for the time being >> for backwards compatiibility, although I plan to deprecate >> TwillTestCase. Twill functionality will eventually be provided through >> alfajor through a plugin. > I'm fine with deprecating TwillTestCase, but TestCase should stay there. > Many people love unittest (including me) and I don't see a reason to > remove it. > No plans on removing TestCase at this time. >> 1. Integration with alfajor. This project is still very young, and >> needs above all some documentation. Eventually however functional >> testing will be optionally provided through alfajor. > +1 > It's really the lack of docs that are holding this up - if I want to refer users to alfajor docs in the main Flask-Testing documentation. The other issue is the dependency on an alfajor.ini file - this needs to be handled gracefully in the setup. >> 2. Move to a decorator-based API. Flask-Testing will use a pattern >> familiar in Flask and other extensions: > -1 > > That is something nose/py.test already do. Furthermore I would love to > see context locals not being used for tests because they are a pain to > debug there. I am already unhappy with the request stuff, but that is > not going away because there is no other way to handle that. > I agree with your concerns. However there needs to be a way to manage context for individual tests to maintain DRY. Perhaps a better idea would be to add the context variables in through a single TestContext object. For example: @setup def create_data(ctx): ctx.fixtures.user_name = "Joe" @with_context def test_user_name(ctx): assert ctx.fixtures.user_name == "Joe" > Instead I would like to see a support for nose and py.test that provide > the same functionality as the test case classes in functions for these > systems. This might be the time to figure out of nose and py.test have > some common ground so that you don't have to implement those functions > for both. > > An @with_context decorator makes sense, but that decorator then has to > support both test case classes and functions in a module for nose. With > py.test you should probably ask Holger or Ronny for feedback. > That's fine, the decorator would probably use existing nose or py.test stuff like @with_setup. There are really just two concerns: 1. Being able to transparently execute tests in the test request environment (as TestCase currently does) 2. Being able to manage fixtures & common setup/teardown operations in a DRY fashion.
On Jul 30, 2010, at 2:01 AM, Dan Jacob wrote: > On 30 July 2010 09:46, Armin Ronacher <armin.ronacher@active-4.com> wrote: >> Hi, >> >> On 7/30/10 10:03 AM, Dan Jacob wrote: >>> 1. Integration with alfajor. This project is still very young, and >>> needs above all some documentation. Eventually however functional >>> testing will be optionally provided through alfajor. >> +1 >> > > It's really the lack of docs that are holding this up - if I want to > refer users to alfajor docs in the main Flask-Testing documentation. > > The other issue is the dependency on an alfajor.ini file - this needs > to be handled gracefully in the setup. > I've got some time today where I can start working on documenting alfajor's use. In addition, I have an idea for making configuration controllable without the .ini file, but I'll need to talk that over with jek first. --Dan
That would be great. After some discussion on IRC today, I'll put the decorator stuff on the back burner for now until I have some clearer proposals for how it may work with existing tools. However alfajor integration is a priority and will be added to the TestCase as soon as possible. Most likely it will degrade gracefully, i.e. if you don't have alfajor installed/configured it will fall back to standard Flask test client. On 30 July 2010 15:22, Dan Colish <dcolish@gmail.com> wrote: > > On Jul 30, 2010, at 2:01 AM, Dan Jacob wrote: > >> On 30 July 2010 09:46, Armin Ronacher <armin.ronacher@active-4.com> wrote: >>> Hi, >>> >>> On 7/30/10 10:03 AM, Dan Jacob wrote: >>>> 1. Integration with alfajor. This project is still very young, and >>>> needs above all some documentation. Eventually however functional >>>> testing will be optionally provided through alfajor. >>> +1 >>> >> >> It's really the lack of docs that are holding this up - if I want to >> refer users to alfajor docs in the main Flask-Testing documentation. >> >> The other issue is the dependency on an alfajor.ini file - this needs >> to be handled gracefully in the setup. >> > > I've got some time today where I can start working on documenting alfajor's use. In addition, I have an idea for making configuration controllable without the .ini file, but I'll need to talk that over with jek first. > > > --Dan >