librelist archives

« back to archive

Not able to pass a URL parameter with // in it

Not able to pass a URL parameter with // in it

From:
Mark Haase
Date:
2012-08-24 @ 22:08
I have a URL endpoint that receives the title of a document as it's
parameter, and some of these document titles have a URL embedded in them.
For example, one of the titles is "IRS Internal Revenue Manual (IRM) -- (
http://www.irs.gov/irm/)". url_for() generates a URL that looks like this:


http://localhost/cce_resource/IRS%20Internal%20Revenue%20Manual%20%28IRM%29%20--%20%28http://www.irs.gov/irm/%29

.. and my route is defined as:

My route is defined as follows:

@app.route('/cce_resource/<path:id>')
def cce_resource(id):
    ...do stuff...

This leads to "id" being set to "IRS Internal Revenue Manual (IRM) --
(http:/www.irs.gov/irm/)". Notice that there is only one slash after
"http:", not 2 slashes as I expected.

I'm not sure why this is. This is running through Apache and mod_wsgi. I've
tried both embedded mode and daemon mode, and I've also tried all 3
settings of the Apache AllowEncodeSlashes directive, and none of those
fixes this. This *does* work as expected on the debug server ( app.run() )
so it's something related to the interaction between Apache and Flask, but
I can't figure out what.

It seems to me like the safe thing to do is to URL encode the slashes in
the parameter when I generate the URL, but for some reason, url_for does
not escape forward slashes. Before I go and hack everything to pieces, I'd
like to find out if I'm missing something obvious.

Thoughts?

-- 
Mark E. Haase

Re: [flask] Not able to pass a URL parameter with // in it

From:
Raphael Slinckx
Date:
2012-08-25 @ 11:11
On Sat, Aug 25, 2012 at 12:08 AM, Mark Haase <mark.haase@lunarline.com> wrote:
> I have a URL endpoint that receives the title of a document as it's
> parameter, and some of these document titles have a URL embedded in them.
> For example, one of the titles is "IRS Internal Revenue Manual (IRM) --
> (http://www.irs.gov/irm/)". url_for() generates a URL that looks like this:
>
> 
http://localhost/cce_resource/IRS%20Internal%20Revenue%20Manual%20%28IRM%29%20--%20%28http://www.irs.gov/irm/%29

I think url_for doesn't encode /, much like werkerug.Href does,
leading to this result which i think is wrong, slashes should be
transformed to %2f

Could this considered  bug, or it's a feature ?

Re: [flask] Not able to pass a URL parameter with // in it

From:
Jan Riechers
Date:
2012-08-25 @ 12:29
On 25.08.2012 14:11, Raphael Slinckx wrote:
> On Sat, Aug 25, 2012 at 12:08 AM, Mark Haase <mark.haase@lunarline.com> wrote:
>> I have a URL endpoint that receives the title of a document as it's
>> parameter, and some of these document titles have a URL embedded in them.
>> For example, one of the titles is "IRS Internal Revenue Manual (IRM) --
>> (http://www.irs.gov/irm/)". url_for() generates a URL that looks like this:
>>
>> 
http://localhost/cce_resource/IRS%20Internal%20Revenue%20Manual%20%28IRM%29%20--%20%28http://www.irs.gov/irm/%29
>
> I think url_for doesn't encode /, much like werkerug.Href does,
> leading to this result which i think is wrong, slashes should be
> transformed to %2f
>
> Could this considered  bug, or it's a feature ?
>
>

Hi,

the conversion of " " (space character) to the url variant %20 and 
similar is due to url_quote to make a url safe.

This happens in Werkzeug's routing.py - I provided some workaround 
(through an additional parameter and some code change in routing.py) 
some time ago, but it has not happen to be in cooperated into Werkzeug.

Here is what you can do, if you want to have "unsafe" urls by providing 
a additional parameter in url_for:

The patch is unoffical(!):

Function to patch:

def build(self, values, append_unknown=True): (Line 701)

and the following code parts have to be inserted/adapted (721)
by doing so, you can add the parameter "doEncode=False" alike:
url_for('your_target_url', cmd=cmdString, doEncode=False)

Result is, you get your target url with the "cmd" alike 
:"http://localhost/?yourAddionalParameter"

Code below, sorry for the bad formatting ahead.

if append_unknown:
             query_vars = MultiDict(values)
             try:
                 doEncode = query_vars['doEncode']
                 if doEncode != False:
                     raise

                 del query_vars['doEncode']
                 commands = '?'
                 for cmd in query_vars:
                     commands += query_vars[cmd]
                 url += commands
             except:
                 for key in processed:
                     if key in query_vars:
                         del query_vars[key]

                 if query_vars:
                     url += '?' + url_encode(query_vars, self.map.charset,
                                             sort=self.map.sort_parameters,
                                             key=self.map.sort_key)

Re: [flask] Not able to pass a URL parameter with // in it

From:
Mark Haase
Date:
2012-08-27 @ 14:05
I'm not really sure if it's a bug... it was surprising behavior to me, but
it's clearly intentional behavior.

To work around this it's actually quite easy (once you realize what the
problem is). Werkzeug supports custom "converters" which are used for
converting URL path elements into function arguments (and vice-versa). So I
wrote up a quick converter that *does* encode forward slashes:

from werkzeug.urls import url_quote, url_unquote
from werkzeug.routing import BaseConverter

class UrlConverter(BaseConverter):
    """
    This converter allows you to pass URLs as parameters, even though
Apache converts
    multiple slashes (e.g. http://) into a single slash (e.g. http:/).
    """

    def to_python(self, value):
        """ Convert a URL into an argument to a Python function by URL
decoding it. """

        return url_unquote(value)

    def to_url(self, value):
        """ Convert a URL into a URL query parameter by URL encoding it. """

        return url_quote(value, self.map.charset, safe='')


To use this, you need to register it somewhere in your bootstrap:

app.url_map.converters["url"] = route_converters.UrlConverter


And then finally when you register a route, use "url" as the type:

@app.route('/cce_resource/<url:id>')
def cce_resource(id):
    """ do stuff """


Jan, thanks for the explanation of the mechanism. It took a while to get to
the bottom of it, but once I found it, the Werkzeug documentation is pretty
clear on how to build a custom converter:

http://werkzeug.pocoo.org/docs/routing/#custom-converters

On Sat, Aug 25, 2012 at 7:11 AM, Raphael Slinckx <
r.slinckx@whatever-company.com> wrote:

> I think url_for doesn't encode /, much like werkerug.Href does,
> leading to this result which i think is wrong, slashes should be
> transformed to %2f
>
> Could this considered  bug, or it's a feature ?


-- 
Mark E. Haase
CISSP
Sr. Security Software Engineer
www.lunarline.com
3300 N Fairfax Drive, Suite 308, Arlington, VA 22201
571-334-8408

"Solutions Built on Security" TM
Lunarline, Inc. is an ISO 9001 and CMMI Level 2 Certified SDVOSB
Information Assurance\ Cyber Security Services Company.

Re: Not able to pass a URL parameter with // in it

From:
Mark Haase
Date:
2012-08-24 @ 22:33
Ahh, please ignore. I realize this is something Apache does to be "helpful".

http://www.webmasterworld.com/apache/3978441.htm

I guess it's off topic, then. Still, if anybody has some advice that is not
germane to Flask, please contact me off list.

Cheers,

On Fri, Aug 24, 2012 at 6:08 PM, Mark Haase <mark.haase@lunarline.com>wrote:

> I have a URL endpoint that receives the title of a document as it's
> parameter, and some of these document titles have a URL embedded in them.
> For example, one of the titles is "IRS Internal Revenue Manual (IRM) -- (
> http://www.irs.gov/irm/)". url_for() generates a URL that looks like this:
>
>
> 
http://localhost/cce_resource/IRS%20Internal%20Revenue%20Manual%20%28IRM%29%20--%20%28http://www.irs.gov/irm/%29
>
> .. and my route is defined as:
>
> My route is defined as follows:
>
> @app.route('/cce_resource/<path:id>')
> def cce_resource(id):
>     ...do stuff...
>
> This leads to "id" being set to "IRS Internal Revenue Manual (IRM) --
> (http:/www.irs.gov/irm/)". Notice that there is only one slash after
> "http:", not 2 slashes as I expected.
>
> I'm not sure why this is. This is running through Apache and mod_wsgi.
> I've tried both embedded mode and daemon mode, and I've also tried all 3
> settings of the Apache AllowEncodeSlashes directive, and none of those
> fixes this. This *does* work as expected on the debug server ( app.run() )
> so it's something related to the interaction between Apache and Flask, but
> I can't figure out what.
>
> It seems to me like the safe thing to do is to URL encode the slashes in
> the parameter when I generate the URL, but for some reason, url_for does
> not escape forward slashes. Before I go and hack everything to pieces, I'd
> like to find out if I'm missing something obvious.
>
> Thoughts?
>
> --
> Mark E. Haase
>
>


-- 
Mark E. Haase