Re: [flask] werkzeug.security.gen_salt shouldn't use string.letters
- From:
- Michael Fogleman
- Date:
- 2011-08-02 @ 16:42
My database is UTF-8. You can see that in my original post.
How do I check/set the encoding for Apache?
I played around at a Python prompt and noticed this, but not sure if
it's relevant:
>>> import locale
>>> import string
>>> string.letters
'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ'
>>> locale.setlocale(locale.LC_ALL, "")
'English_United States.1252'
>>> string.letters
'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz\x83\x8a\x8c\x8e\x9a\x9c\x9e\x9f\xaa\xb5\xba\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8\xe9\xea\xeb\xec\xed\xee\xef\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff'
Michael
On Tue, Aug 2, 2011 at 12:12 PM, Sean Chittenden <sean@chittenden.org> wrote:
>> ascii_lowercase and ascii_uppercase should be used instead.
>> string.letters is locale-dependent. I'm getting SQL errors because the
>> salt contains weird characters.
>>
>> 'sha1$\\xee\\xb5y3o\\xc6\\xc5Q$5fdd6f832af523189ba4f5871f47ed44ed5ae7af'
>>
>> DataError: (DataError) invalid byte sequence for encoding "UTF8": 0xeeb579
>>
>> I'm only running into this issue on my production Apache server. Any
>> ideas why it's using a different locale or how I can fix it?
>
>
> The following isn't entirely safe (though I think it beats the
alternative). I run around with the following environment variables set:
>
> LANG=en_US.UTF-8
> LC_TYPE=en_US.utf-8
>
> to help ensure that everything that is capable, uses UTF-8. If you've
had the misfortune of cleaning up a MySQL install where the database is
initialized with ISO-8895-1 and ${LANGUAGE} is set to use UTF-8, but isn't
checking the encoding and begins inserting garbage data, you'll know what
I'm referring to. Debugging and correcting that mess after the fact is
anything but pleasant.
>
> Bottom line, there are still a number of programs, bindings, databases,
etc., that default to US-ASCII instead of UTF-8. I'm not convinced this
will solve your problem, but I'd guess something in your stack is ASCII
instead of UTF-8. Things you will have to answer for yourself:
>
> *) What's the encoding of the data in your salt?
> *) What encoding do you want your database to be?
> *) What encoding is your web server defaulting to?
>
> My guess is you've initialized your database using US-ASCII and your
python database library is checking for invalid code points and is raising
the exception shown as a result. Good luck & happy hunting. -sc
>
>
> --
> Sean Chittenden
> sean@chittenden.org
>
>
Re: [flask] werkzeug.security.gen_salt shouldn't use string.letters
- From:
- Sean Chittenden
- Date:
- 2011-08-02 @ 16:56
> My database is UTF-8. You can see that in my original post.
Your database may be, but is your database client library? I've debugged
this in the past in PHP where the DB was initialized UTF-8, the client
programming language was UTF-8, but the encoding for over-the-wire
transmission was US-ASCII. There are other similar weird situations that
you can get yourself in to unless things are consistently set from
end-to-end.
Stepping back for a second, why do you have high-byte data in your salt? I
thought the representation of standard sha1 salts was hex encoded data.
Did you forget to hex encode your salt?
>>> '\xee\xb5y3o\xc6\xc5Q'.encode('hex')
'eeb579336fc6c551'
> How do I check/set the encoding for Apache?
Check apache's environment variables.
> I played around at a Python prompt and noticed this, but not sure if
> it's relevant:
>
>>>> import locale
>>>> import string
>>>> string.letters
> 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ'
>>>> locale.setlocale(locale.LC_ALL, "")
> 'English_United States.1252'
>>>> string.letters
>
'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz\x83\x8a\x8c\x8e\x9a\x9c\x9e\x9f\xaa\xb5\xba\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8\xe9\xea\xeb\xec\xed\xee\xef\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff'
Check your environment variables for the startup of Apache (and whatever's
starting your python app). I'm just guessing at this point, however. Good
luck. -sc
--
Sean Chittenden
sean@chittenden.org