librelist archives

« back to archive

flask utf-8 postgresql 9.1 database UnicodeDecodeError

flask utf-8 postgresql 9.1 database UnicodeDecodeError

From:
Igor TAmara
Date:
2012-06-02 @ 14:12
 Hi, I'm starting with Flask and I got the exception :

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1:
ordinal not in range(128)

, I have a postgresql database like this

osm              | osm             | UTF8         | es_CO.utf8 | es_CO.utf8

My function is:

  @app.route('/')
  def show_entries():
      g.db.execute("SELECT count(*) AS cant, upper(substr(tags ->
'name',0,(strpos(tags -> 'name',' ')))) AS initial  FROM ways WHERE tags ?&
Array['highway','name'] GROUP BY initial ORDER BY cant DESC, initial")
      #cur = g.db.execute("select 1,2")
      entries = [dict(name=row[0], cant=row[1]) for row in g.db.fetchall()]
      return render_template('show_entries.html', entries=entries)

The template is

{% extends "layout.html" %}
{% block body %}
<table class=entries>
<tr><th>Tipo</th><th>Cantidad</th>
  {% for entry in entries %}
    <tr><td>{{ entry.name }}</td><td>{{ entry.cant }}</td></tr>
  {% else %}
<!-- Nothing -->
  {% endfor %}
  </ul>
{% endblock %}

If I run something like

  cur=c.cursor()

  cur.execute("SELECT count(*) AS cant, upper(substr(tags ->
'name',0,(strpos(tags -> 'name',' ')))) AS initial  FROM ways WHERE tags ?&
Array['highway','name'] GROUP BY initial ORDER BY cant DESC, initial")
  for i in cur.fetchall():
      print i[1]

It shows no problems, so the problem must be with render_template, what do
I have to do to avoid the problem?

<ikks> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position
1: ordinal not in range(128)
<ikks> http://dpaste.com/754095/
<ikks> osm              | osm             | UTF8         | es_CO.utf8 |
es_CO.utf8
<ikks> In the table I  have words like vía intersección
<ikks> How do I avoid this encoding problem?
<ikks> http://dpaste.com/754099/
<ikks> The trace : http://dpaste.com/754100/
<ikks> The only choice is to work with SqlAlchemy?
<ikks> Or is there a workaround?

Re: [flask] flask utf-8 postgresql 9.1 database UnicodeDecodeError

From:
Audrius Kažukauskas
Date:
2012-06-02 @ 15:48
On Sat, 2012-06-02 at 09:12:50 -0500, Igor TAmara wrote:
>  Hi, I'm starting with Flask and I got the exception :
> 
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1:
> ordinal not in range(128)
> 
> , I have a postgresql database like this
> 
> osm              | osm             | UTF8         | es_CO.utf8 | es_CO.utf8
> 
> My function is:
> 
>   @app.route('/')
>   def show_entries():
>       g.db.execute("SELECT count(*) AS cant, upper(substr(tags ->
> 'name',0,(strpos(tags -> 'name',' ')))) AS initial  FROM ways WHERE tags ?&
> Array['highway','name'] GROUP BY initial ORDER BY cant DESC, initial")
>       #cur = g.db.execute("select 1,2")
>       entries = [dict(name=row[0], cant=row[1]) for row in g.db.fetchall()]
>       return render_template('show_entries.html', entries=entries)
> 
> The template is
> 
> {% extends "layout.html" %}
> {% block body %}
> <table class=entries>
> <tr><th>Tipo</th><th>Cantidad</th>
>   {% for entry in entries %}
>     <tr><td>{{ entry.name }}</td><td>{{ entry.cant }}</td></tr>
>   {% else %}
> <!-- Nothing -->
>   {% endfor %}
>   </ul>
> {% endblock %}

I believe the problem is that Jinja2 is expecting unicode objects[0],
while your results from the database are UTF-8 encoded str objects.  You
need to decode them to unicode before passing them to the template,
e.g.:

  entries = [{
      'name': row[0].decode('utf-8'),
      'cant': row[1],
  } for row in g.db.fetchall()]

[0] http://jinja.pocoo.org/docs/api/#unicode

-- 
Audrius Kažukauskas
http://neutrino.lt/

Re: [flask] flask utf-8 postgresql 9.1 database UnicodeDecodeError

From:
Audrius Kažukauskas
Date:
2012-06-02 @ 16:14
On Sat, 2012-06-02 at 18:48:53 +0300, Audrius Kažukauskas wrote:
> I believe the problem is that Jinja2 is expecting unicode objects[0],
> while your results from the database are UTF-8 encoded str objects.  You
> need to decode them to unicode before passing them to the template,
> e.g.:
> 
>   entries = [{
>       'name': row[0].decode('utf-8'),
>       'cant': row[1],
>   } for row in g.db.fetchall()]
> 
> [0] http://jinja.pocoo.org/docs/api/#unicode

Or even better, configure psycopg2 to return unicode objects instead:

  import psycopg2
  import psycopg2.extensions
  psycopg2.extensions.register_type(psycopg2.extensions.UNICODE)

See http://initd.org/psycopg/docs/usage.html#unicode-handling for the
details.

-- 
Audrius Kažukauskas
http://neutrino.lt/