librelist archives

« back to archive

Problem testing POST with unicode data

Problem testing POST with unicode data

From:
Karsten Hoffrath
Date:
2012-05-01 @ 18:02
Hey all.

I'm playing with flask to build a REST-API. While testing a view i came
across the following problem:

When i'm testing the app with the flask test-client i can't POST
JSON-objects which contains umlauts. When i start the server and do a
POST with curl the server/view works as expected.

The error message is:

Error
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 327, in run
    testMethod()
  File
"/home/khoffrath/projects/flask_post_unicode/projects/test_u_p_s.py",
line 42, in test_post_unicode_invalid
    rv = self.app.post('/', data=u"{'tag': u'Äüößäname'}")
  File
"/media/os/virtualenv/local/lib/python2.7/site-packages/werkzeug/test.py",
line 745, in post
    return self.open(*args, **kw)
  File
"/media/os/virtualenv/local/lib/python2.7/site-packages/flask/testing.py",
line 97, in open
    builder = make_test_environ_builder(self.application, *args, **kwargs)
  File
"/media/os/virtualenv/local/lib/python2.7/site-packages/flask/testing.py",
line 28, in make_test_environ_builder
    return EnvironBuilder(path, base_url, *args, **kwargs)
  File
"/media/os/virtualenv/local/lib/python2.7/site-packages/werkzeug/test.py",
line 305, in __init__
    self.input_stream = StringIO(data)
UnicodeEncodeError: 'ascii' codec can't encode characters in position
10-14: ordinal not in range(128)

My environment:
- Python 2.7.3
- flask 0.8
- Werkzeug 0.8.3
- simplejson 2.5.0

My demo files are here: https://gist.github.com/2570059

Can someone shed some light on this for me?


Best Regards
Karsten

Re: [flask] Problem testing POST with unicode data

From:
Simon Sapin
Date:
2012-05-02 @ 06:35
Le 01/05/2012 20:02, Karsten Hoffrath a écrit :
> I'm playing with flask to build a REST-API. While testing a view i came
> across the following problem:
>
> When i'm testing the app with the flask test-client i can't POST
> JSON-objects which contains umlauts. When i start the server and do a
> POST with curl the server/view works as expected.

Hi,

HTTP uses bytes, not Unicode. Usually you’ll need some metadata in the 
headers to indicate the character encoding. For example:
Content-Type: text/html; charset=utf8

For JSON however, it is safe to just use UTF-8.
(See http://www.ietf.org/rfc/rfc4627.txt section 3)

Actually you shouldn’t even worry about that and let the JSON encoder do 
its job:

 >>> from flask import json
 >>> json.dumps({'tag': u'Äüößäname'}) # use a dict, not a single string
'{"tag": "\\u00c4\\u00fc\\u00f6\\u00df\\u00e4name"}'

Regards,
-- 
Simon Sapin

Re: [flask] Problem testing POST with unicode data

From:
Karsten Hoffrath
Date:
2012-05-02 @ 14:45
Hi,

i had an error in my JSON encoder encapsulation (ensure_ascii was set to 
false) which
basically caused this error. Your JSON example pushed me into the right 
direction :)

Thanks,
Karsten

Am 02.05.2012 08:35, schrieb Simon Sapin:
> Le 01/05/2012 20:02, Karsten Hoffrath a écrit :
>> I'm playing with flask to build a REST-API. While testing a view i came
>> across the following problem:
>>
>> When i'm testing the app with the flask test-client i can't POST
>> JSON-objects which contains umlauts. When i start the server and do a
>> POST with curl the server/view works as expected.
> 
> Hi,
> 
> HTTP uses bytes, not Unicode. Usually you’ll need some metadata in the 
> headers to indicate the character encoding. For example:
> Content-Type: text/html; charset=utf8
> 
> For JSON however, it is safe to just use UTF-8.
> (See http://www.ietf.org/rfc/rfc4627.txt section 3)
> 
> Actually you shouldn’t even worry about that and let the JSON encoder do 
> its job:
> 
>  >>> from flask import json
>  >>> json.dumps({'tag': u'Äüößäname'}) # use a dict, not a single string
> '{"tag": "\\u00c4\\u00fc\\u00f6\\u00df\\u00e4name"}'
> 
> Regards,

Re: [flask] Problem testing POST with unicode data

From:
Joe Esposito
Date:
2012-05-02 @ 15:09
Since we're on the topic of encodings, this is an *excellent* presentation
by Ned Batchelder at PyCon 2012 on Unicode that was shared at the last
PghPy meetup. Was very helpful, so figured I'd pass it along.

http://nedbatchelder.com/text/unipain.html

On Wed, May 2, 2012 at 10:45 AM, Karsten Hoffrath <maillists@khoffrath.de>wrote:

> Hi,
>
> i had an error in my JSON encoder encapsulation (ensure_ascii was set to
> false) which
> basically caused this error. Your JSON example pushed me into the right
> direction :)
>
> Thanks,
> Karsten
>
> Am 02.05.2012 08:35, schrieb Simon Sapin:
> > Le 01/05/2012 20:02, Karsten Hoffrath a écrit :
> >> I'm playing with flask to build a REST-API. While testing a view i came
> >> across the following problem:
> >>
> >> When i'm testing the app with the flask test-client i can't POST
> >> JSON-objects which contains umlauts. When i start the server and do a
> >> POST with curl the server/view works as expected.
> >
> > Hi,
> >
> > HTTP uses bytes, not Unicode. Usually you’ll need some metadata in the
> > headers to indicate the character encoding. For example:
> > Content-Type: text/html; charset=utf8
> >
> > For JSON however, it is safe to just use UTF-8.
> > (See http://www.ietf.org/rfc/rfc4627.txt section 3)
> >
> > Actually you shouldn’t even worry about that and let the JSON encoder do
> > its job:
> >
> >  >>> from flask import json
> >  >>> json.dumps({'tag': u'Äüößäname'}) # use a dict, not a single string
> > '{"tag": "\\u00c4\\u00fc\\u00f6\\u00df\\u00e4name"}'
> >
> > Regards,
>

Re: [flask] Problem testing POST with unicode data

From:
Karsten Hoffrath
Date:
2012-05-02 @ 18:16
Thanks, bookmarked!

On 02.05.2012 17:09, Joe Esposito wrote:
> Since we're on the topic of encodings, this is
> an /excellent/ presentation by Ned Batchelder at PyCon 2012 on Unicode
> that was shared at the last PghPy meetup. Was very helpful, so figured
> I'd pass it along.
> 
> http://nedbatchelder.com/text/unipain.html
> 
> On Wed, May 2, 2012 at 10:45 AM, Karsten Hoffrath
> <maillists@khoffrath.de <mailto:maillists@khoffrath.de>> wrote:
> 
>     Hi,
> 
>     i had an error in my JSON encoder encapsulation (ensure_ascii was
>     set to false) which
>     basically caused this error. Your JSON example pushed me into the
>     right direction :)
> 
>     Thanks,
>     Karsten
> 
>     Am 02.05.2012 08:35, schrieb Simon Sapin:
>     > Le 01/05/2012 20:02, Karsten Hoffrath a écrit :
>     >> I'm playing with flask to build a REST-API. While testing a view
>     i came
>     >> across the following problem:
>     >>
>     >> When i'm testing the app with the flask test-client i can't POST
>     >> JSON-objects which contains umlauts. When i start the server and do a
>     >> POST with curl the server/view works as expected.
>     >
>     > Hi,
>     >
>     > HTTP uses bytes, not Unicode. Usually you’ll need some metadata in the
>     > headers to indicate the character encoding. For example:
>     > Content-Type: text/html; charset=utf8
>     >
>     > For JSON however, it is safe to just use UTF-8.
>     > (See http://www.ietf.org/rfc/rfc4627.txt section 3)
>     >
>     > Actually you shouldn’t even worry about that and let the JSON
>     encoder do
>     > its job:
>     >
>     >  >>> from flask import json
>     >  >>> json.dumps({'tag': u'Äüößäname'}) # use a dict, not a single
>     string
>     > '{"tag": "\\u00c4\\u00fc\\u00f6\\u00df\\u00e4name"}'
>     >
>     > Regards,
> 
>