librelist archives

home // archives //

« back to archive

Flask, WTForms, and duplicate IDs

Flask, WTForms, and duplicate IDs

From:
Daniele Nicolodi
Date:
2011-06-17 @ 15:55 
Hello,

I'm sorry, the subject of this mail is kind of vague, but I'm unable to
summarize my problem any better. In my application I use WTForms and I
would like to validate user chosen object names for being unique.

Storing the data into an relation database it is easy to add such
constrain at the storage level, the creation of the new object with a
duplicate ID would thus fail at insert time, and this is the only form
of validation I would trust.

What's the preferred pattern to transform a duplicate key error into a
validation error?

Currently I do the following (I hope the code is clear enough)::


@app.route('/create', methods=('GET', 'POST'))
@require('admin')
def create():
    form = IUser()
    if request.method == 'POST' and form.validate():
        user = User()
        form.update(user)
	try:
            user.create()
	except DuplicateKeyError: # I wish MySQLdb would have
                                  # something like this
	    from.username.errors.append('Duplicated user name.')
	else:
            flash('User "%s" created.' % form.data['username'])
            return redirect(url_for('manage.users.index'))
    return render_template('users/create.html', form=form)


but it feels suboptimal. Suggestions?

Thank you. Cheers,
-- 
Daniele

Re: [flask] Flask, WTForms, and duplicate IDs

From:
Alex
Date:
2011-06-17 @ 18:48 
On Fri, Jun 17, 2011 at 5:55 PM, Daniele Nicolodi <daniele@grinta.net> wrote:

> What's the preferred pattern to transform a duplicate key error into a
> validation error?

Here how I do it:

First I create a custum WTForm validator

class Unique(object):
    """ Validator that checks field uniqueness """
    def __init__(self, model, field, message=None):
        self.model = model
        self.field = field
        if not message:
            message = u'This element already exists'
        self.message = message

    def __call__(self, form, field):
        check = self.model.query.filter(self.field == field.data).first()
        if check:
            raise ValidationError(self.message)

Then in the form class:

class MyForm(Form):
    """Form that manage what you like"""
    my_field = TextField('URL', [validators.Required(),
validators.URL(), Unique(Site, Site.URL)])

Re: [flask] Flask, WTForms, and duplicate IDs

From:
Daniele Nicolodi
Date:
2011-06-17 @ 17:01 
Hello,

Thank you all for the answers to my question. I reply to my own mail
just because all the answers are many and I would have to answer almost
all of them in the same way.

I'm well aware of WTForms customs validators, and I use them in my
application, but they do not solve my problem in the way I would like to
solve it.

My point is that I would like the validation of key uniqueness and
creation of new object to be a single atomic operation, to catch the
rare case when an object with the same id is created between validation
and actual creation of the object (and to spare an SQL query, but this
is definitely a minor concern).

Cheers,
-- 
Daniele

Re: [flask] Flask, WTForms, and duplicate IDs

From:
Sean Chittenden
Date:
2011-06-17 @ 18:02 
> My point is that I would like the validation of key uniqueness and
> creation of new object to be a single atomic operation, to catch the
> rare case when an object with the same id is created between validation
> and actual creation of the object (and to spare an SQL query, but this
> is definitely a minor concern).

Or you can use PostgreSQL and a stored procedure to handle this logic. In SQL:

> SELECT result, "column", message FROM aaa.register(email := 
'user@example.com', password := 'myfailpw', ip_address := '11.22.33.44') 
AS (result BOOL, "column" TEXT, message TEXT);

Backed by the function:


https://github.com/sean-/flask-skeleton/blob/master/sql/initialize/300_funcs.sql.in#L273

PostgreSQL will return a tuple (success BOOL, column TEXT, message TEXT) 
which makes it easy to set very tailored error messages:


https://github.com/sean-/flask-skeleton/blob/master/skeleton/modules/aaa/views.py#L203


Here's the complete code used to call this:


https://github.com/sean-/flask-skeleton/blob/master/skeleton/modules/aaa/views.py#L184

I mixed and matched a PL function for handling the creation of a user for 
the sake of demoing, but normally I'd handle all of this in a single PL 
call to the database. -sc


--
Sean Chittenden
sean@chittenden.org

Re: [flask] Flask, WTForms, and duplicate IDs

From:
Adam Oakman
Date:
2011-06-17 @ 17:19 
Hmm I work with MSSQL a lot more than MySQL and I don't know if this  
is even possible in MySQL. At the db level the only authoritative way  
of doing that would be to perform everything in a database transaction  
that kept a table write lock on your usernames. But that could be an  
horribly expensive blocking operation depending on your load.

On Jun 17, 2011, at 12:01 PM, Daniele Nicolodi wrote:

> Hello,
>
> Thank you all for the answers to my question. I reply to my own mail
> just because all the answers are many and I would have to answer  
> almost
> all of them in the same way.
>
> I'm well aware of WTForms customs validators, and I use them in my
> application, but they do not solve my problem in the way I would  
> like to
> solve it.
>
> My point is that I would like the validation of key uniqueness and
> creation of new object to be a single atomic operation, to catch the
> rare case when an object with the same id is created between  
> validation
> and actual creation of the object (and to spare an SQL query, but this
> is definitely a minor concern).
>
> Cheers,
> -- 
> Daniele

Re: [flask] Flask, WTForms, and duplicate IDs

From:
Daniele Nicolodi
Date:
2011-06-17 @ 17:25 
On 17/06/11 19:19, Adam Oakman wrote:
> Hmm I work with MSSQL a lot more than MySQL and I don't know if this  
> is even possible in MySQL. At the db level the only authoritative way  
> of doing that would be to perform everything in a database transaction  
> that kept a table write lock on your usernames. But that could be an  
> horribly expensive blocking operation depending on your load.

That's definitely possible at the database level: just try the insert
and catch any integrity check error. That's what the code in my original
email does. The only drawback is that you have some kind of validation
code outside the form definition. It looks a bit ugly (but not much
uglier that having to specify different validators for the upgrade and
create cases).

Cheers,
-- 
Daniele

Re: [flask] Flask, WTForms, and duplicate IDs

From:
Adam Oakman
Date:
2011-06-17 @ 18:49 
Yeah I agree its a bit ugly.

Just thinking out loud. I assume username is your primary key on the  
table. You might add a datetime column to the table named something  
like pending_expiry. When you want to check if the username is in use  
you just insert into users (username, now + 30 mins). If the username  
is already in use it will throw an error and you can report back. If  
its not you have effectively reserved the username from other signups,  
without any lengthy locks. When ready to save the object for real you  
just remove that expiry. You would need to have a clean up method to  
remove the accounts that no one finished the signup process for, but  
that should be reasonably trivial.


On Jun 17, 2011, at 12:25 PM, Daniele Nicolodi wrote:

> On 17/06/11 19:19, Adam Oakman wrote:
>> Hmm I work with MSSQL a lot more than MySQL and I don't know if this
>> is even possible in MySQL. At the db level the only authoritative way
>> of doing that would be to perform everything in a database  
>> transaction
>> that kept a table write lock on your usernames. But that could be an
>> horribly expensive blocking operation depending on your load.
>
> That's definitely possible at the database level: just try the insert
> and catch any integrity check error. That's what the code in my  
> original
> email does. The only drawback is that you have some kind of validation
> code outside the form definition. It looks a bit ugly (but not much
> uglier that having to specify different validators for the upgrade and
> create cases).
>
> Cheers,
> -- 
> Daniele

Re: [flask] Flask, WTForms, and duplicate IDs

From:
Sean Chittenden
Date:
2011-06-17 @ 19:03 
> Just thinking out loud. I assume username is your primary key on the  
> table. You might add a datetime column to the table named something  
> like pending_expiry. When you want to check if the username is in use  
> you just insert into users (username, now + 30 mins). If the username  
> is already in use it will throw an error and you can report back. If  
> its not you have effectively reserved the username from other signups,  
> without any lengthy locks. When ready to save the object for real you  
> just remove that expiry. You would need to have a clean up method to  
> remove the accounts that no one finished the signup process for, but  
> that should be reasonably trivial.

Asynchronous operations need to have their state maintained someplace.... 
databases are fantastic at doing this very easily.

CREATE TABLE "user" (
  id SERIAL NOT NULL,
  username TEXT NOT NULL,
  active BOOL NOT NULL DEFAULT TRUE,
  PRIMARY KEY(id), -- This really should be "(username) WHERE active = 
TRUE" and id is the surrogate key that gets used throughout the database
);
-- User a WHERE clause on your UNIQUE INDEX
CREATE UNIQUE INDEX user_username_lower_udx ON "user"(LOWER(username)) 
WHERE active = TRUE;

CREATE TABLE user_confirmation (
  user_id INT NOT NULL,
  confirmation_sent_utc TIMESTAMP WITH TIME ZONE NOT NULL,
  confirmed_utc TIMESTAMP WITH TIME ZONE,
  confirmed BOOL,
  FOREIGN KEY(user_id) REFERENCES "user"(id)
);
-- Create an INDEX that you can index_scan easily to determine what 
usernames need to be expired
CREATE INDEX user_confirmation_sent_idx ON 
user_confirmation(confirmation_sent_utc) WHERE confirmed IS NULL OR 
confirmed IS FALSE;

Then run a cron job that does:

UPDATE "user" SET active = FALSE WHERE id IN(SELECT uc.user_id FROM 
user_confirmation AS uc WHERE uc.confirmed IS NULL OR uc.confirmed = FALSE
OR uc.confirmation_sent_utc + '30 minutes'::INTERVAL < NOW());

Easy mode. -sc


--
Sean Chittenden
sean@chittenden.org

Re: [flask] Flask, WTForms, and duplicate IDs

From:
Adam Oakman
Date:
2011-06-17 @ 16:11 
I do it all with WTForms and a custom validator, I'm using GAE for my  
data store. Probably not perfect but it works and is quite simple.

def unique_user_name(form, field):
     users = db.Query(Users).filter('user_name =', field.data)
     if users.count() == 1:
         raise validators.ValidationError('That user name is already  
in use, please select another.')

class RegisterForm(Form):
     user_name = TextField(u'Username*',  
validators=[validators.Required(), validators.length(max=200),  
unique_user_name])
     password = PasswordField(u'Password*',  
validators=[validators.Required(), validators.EqualTo('confirm',  
message='Passwords must match')])
     confirm  = PasswordField(u'Confirm Password*')

On Jun 17, 2011, at 10:55 AM, Daniele Nicolodi wrote:

> Hello,
>
> I'm sorry, the subject of this mail is kind of vague, but I'm unable  
> to
> summarize my problem any better. In my application I use WTForms and I
> would like to validate user chosen object names for being unique.
>
> Storing the data into an relation database it is easy to add such
> constrain at the storage level, the creation of the new object with a
> duplicate ID would thus fail at insert time, and this is the only form
> of validation I would trust.
>
> What's the preferred pattern to transform a duplicate key error into a
> validation error?
>
> Currently I do the following (I hope the code is clear enough)::
>
>
> @app.route('/create', methods=('GET', 'POST'))
> @require('admin')
> def create():
>    form = IUser()
>    if request.method == 'POST' and form.validate():
>        user = User()
>        form.update(user)
> 	try:
>            user.create()
> 	except DuplicateKeyError: # I wish MySQLdb would have
>                                  # something like this
> 	    from.username.errors.append('Duplicated user name.')
> 	else:
>            flash('User "%s" created.' % form.data['username'])
>            return redirect(url_for('manage.users.index'))
>    return render_template('users/create.html', form=form)
>
>
> but it feels suboptimal. Suggestions?
>
> Thank you. Cheers,
> -- 
> Daniele

Re: [flask] Flask, WTForms, and duplicate IDs

From:
Jesse
Date:
2011-06-17 @ 16:24 
Here's my version, just for comparison's sake :)

class UniqueValidator(object):
    def __init__(self, attribute, message=None):
        self.attribute = attribute
        if not message:
            message = u'This field\'s value must be unique.'
        self.message = message

    def __call__(self, form, field):
        if 
self.attribute.parententity.class_.query.filter(self.attribute==field.data).count()
> 0:
            raise wtf.ValidationError(self.message)

class RegisterForm(wtf.Form):
    username = wtf.TextField('Username', validators=[
        wtf.Required(),
        UniqueValidator(User.username),
        ...
        ])

On Fri, Jun 17, 2011 at 9:11 AM, Adam Oakman <adam.oakman@gmail.com> wrote:
> I do it all with WTForms and a custom validator, I'm using GAE for my
> data store. Probably not perfect but it works and is quite simple.
>
> def unique_user_name(form, field):
>     users = db.Query(Users).filter('user_name =', field.data)
>     if users.count() == 1:
>         raise validators.ValidationError('That user name is already
> in use, please select another.')
>
> class RegisterForm(Form):
>     user_name = TextField(u'Username*',
> validators=[validators.Required(), validators.length(max=200),
> unique_user_name])
>     password = PasswordField(u'Password*',
> validators=[validators.Required(), validators.EqualTo('confirm',
> message='Passwords must match')])
>     confirm  = PasswordField(u'Confirm Password*')
>
> On Jun 17, 2011, at 10:55 AM, Daniele Nicolodi wrote:
>
>> Hello,
>>
>> I'm sorry, the subject of this mail is kind of vague, but I'm unable
>> to
>> summarize my problem any better. In my application I use WTForms and I
>> would like to validate user chosen object names for being unique.
>>
>> Storing the data into an relation database it is easy to add such
>> constrain at the storage level, the creation of the new object with a
>> duplicate ID would thus fail at insert time, and this is the only form
>> of validation I would trust.
>>
>> What's the preferred pattern to transform a duplicate key error into a
>> validation error?
>>
>> Currently I do the following (I hope the code is clear enough)::
>>
>>
>> @app.route('/create', methods=('GET', 'POST'))
>> @require('admin')
>> def create():
>>    form = IUser()
>>    if request.method == 'POST' and form.validate():
>>        user = User()
>>        form.update(user)
>>       try:
>>            user.create()
>>       except DuplicateKeyError: # I wish MySQLdb would have
>>                                  # something like this
>>           from.username.errors.append('Duplicated user name.')
>>       else:
>>            flash('User "%s" created.' % form.data['username'])
>>            return redirect(url_for('manage.users.index'))
>>    return render_template('users/create.html', form=form)
>>
>>
>> but it feels suboptimal. Suggestions?
>>
>> Thank you. Cheers,
>> --
>> Daniele
>
>

Re: [flask] Flask, WTForms, and duplicate IDs

From:
Amirouche Boubekki
Date:
2011-06-17 @ 16:05 
Héllo Daniele,

If I understand well you want username to be unique in the application,
check out this code for an example of this feature (with mongoalchemy tough)
``unique_login`` and  ``unique_email`` [1]. I think the code is explicit
enough but I may add more explanation if you want.


[1]
https://bitbucket.org/abki/andalucia/src/72f835438458/andalucia/forms.py#cl-42

2011/6/17 Daniele Nicolodi <daniele@grinta.net>

> Hello,
>
> I'm sorry, the subject of this mail is kind of vague, but I'm unable to
> summarize my problem any better. In my application I use WTForms and I
> would like to validate user chosen object names for being unique.
>
> Storing the data into an relation database it is easy to add such
> constrain at the storage level, the creation of the new object with a
> duplicate ID would thus fail at insert time, and this is the only form
> of validation I would trust.
>
> What's the preferred pattern to transform a duplicate key error into a
> validation error?
>
> Currently I do the following (I hope the code is clear enough)::
>
>
> @app.route('/create', methods=('GET', 'POST'))
> @require('admin')
> def create():
>    form = IUser()
>    if request.method == 'POST' and form.validate():
>        user = User()
>        form.update(user)
>        try:
>            user.create()
>        except DuplicateKeyError: # I wish MySQLdb would have
>                                  # something like this
>            from.username.errors.append('Duplicated user name.')
>        else:
>            flash('User "%s" created.' % form.data['username'])
>            return redirect(url_for('manage.users.index'))
>    return render_template('users/create.html', form=form)
>
>
> but it feels suboptimal. Suggestions?
>
> Thank you. Cheers,
> --
> Daniele
>

Re: [flask] Flask, WTForms, and duplicate IDs

From:
Alfred Hall
Date:
2011-06-17 @ 16:05 
Here is how I do it:

class ProjectForm(Form):
    id = IntegerField("Id")

    name = TextField("Name", validators=[
          required(message="You must provide project name")
    ])

    description = TextAreaField("Description", validators=[
          required(message="You must provide project name")
    ])

    def validate_name(form, field):
        # If we're editing and not changing the name we don't need to validate
        # any further.
        if form.id.data:
            obj = Project.query.get(form.id.data)
            if obj.name == field.data:
                return

        count = Project.query.filter_by(name=field.data).count()
        if count > 0:
            raise ValidationError(u'Name must be unique')


@project.route('/edit/<int:id>/', methods=['POST'])
@admin_permission.require(401)
def edit(id):
    project = Project.query.get_or_404(id)

    # Do validation of new data.
    form = ProjectForm(id=id)
    if not form.validate_on_submit():
        return jsonify(success=False, errors=form.errors,
                       csrf=form.csrf.data,
                       message="Validation failed")

    # Do the update of the object.
    form.populate_obj(project)
    db.session.add(project)
    db.session.commit()
    return jsonify(success=True, message="Project updated",
                   csrf=form.csrf.data)


@project.route('/create/', methods=['POST'])
@admin_permission.require(401)
def create():
    form = ProjectForm()
    if not form.validate_on_submit():
        return jsonify(success=False, errors=form.errors,
                       csrf=form.csrf.data,
                       message="Validation failed")

    project = Project()
    form.populate_obj(project)

    db.session.add(project)
    db.session.commit()
    return jsonify(success=True, message="Project created",
                   csrf=form.csrf.data)


Then will catch 99% and then if someone changes it between the checks
you will get a 500 this is how you tackle that in an errorhandler:
    @app.errorhandler(500)
    def server_error(error):
        if request.is_xhr:
            return jsonify(error='Sorry, an error has occurred')
        return render_template("errors/500.html", error=error)

I want all my validation logic in the forms and avoid all try/except
in the views if necessary.


On Fri, Jun 17, 2011 at 4:55 PM, Daniele Nicolodi <daniele@grinta.net> wrote:
> Hello,
>
> I'm sorry, the subject of this mail is kind of vague, but I'm unable to
> summarize my problem any better. In my application I use WTForms and I
> would like to validate user chosen object names for being unique.
>
> Storing the data into an relation database it is easy to add such
> constrain at the storage level, the creation of the new object with a
> duplicate ID would thus fail at insert time, and this is the only form
> of validation I would trust.
>
> What's the preferred pattern to transform a duplicate key error into a
> validation error?
>
> Currently I do the following (I hope the code is clear enough)::
>
>
> @app.route('/create', methods=('GET', 'POST'))
> @require('admin')
> def create():
>    form = IUser()
>    if request.method == 'POST' and form.validate():
>        user = User()
>        form.update(user)
>        try:
>            user.create()
>        except DuplicateKeyError: # I wish MySQLdb would have
>                                  # something like this
>            from.username.errors.append('Duplicated user name.')
>        else:
>            flash('User "%s" created.' % form.data['username'])
>            return redirect(url_for('manage.users.index'))
>    return render_template('users/create.html', form=form)
>
>
> but it feels suboptimal. Suggestions?
>
> Thank you. Cheers,
> --
> Daniele
>

Re: [flask] Flask, WTForms, and duplicate IDs

From:
Daniele Nicolodi
Date:
2011-06-17 @ 16:54 
On 17/06/11 18:05, Alfred Hall wrote:
> Then will catch 99% and then if someone changes it between the checks
> you will get a 500 this is how you tackle that in an errorhandler:

> I want all my validation logic in the forms and avoid all try/except
> in the views if necessary.

Hello Alfred,

thank you for sharing for your code, this is close to my initial
solution, but I'm worried exactly about the 1% cases that this solution
misses. I do not like my solution exactly for the points you raise.

Cheers,
-- 
Daniele

Re: [flask] Flask, WTForms, and duplicate IDs

From:
Ron DuPlain
Date:
2011-06-17 @ 16:03 
Hi Daniele,

On Fri, Jun 17, 2011 at 11:55 AM, Daniele Nicolodi <daniele@grinta.net> wrote:
> I'm sorry, the subject of this mail is kind of vague, but I'm unable to
> summarize my problem any better. In my application I use WTForms and I
> would like to validate user chosen object names for being unique.
>
> Storing the data into an relation database it is easy to add such
> constrain at the storage level, the creation of the new object with a
> duplicate ID would thus fail at insert time, and this is the only form
> of validation I would trust.
>
> What's the preferred pattern to transform a duplicate key error into a
> validation error?

Check WTForms custom validators:
http://wtforms.simplecodes.com/docs/0.6.1/validators.html#custom-validators

You can perform a query that raises ValidationError on duplicate keys.

-Ron

Archives based on code by Armin Ronacher //