librelist archives

« back to archive

Assets management with Photon, ideas

Assets management with Photon, ideas

From:
Loic d'Anterroches
Date:
2011-10-19 @ 12:57
Hello,

at the moment, the way we go is: the files are in a folder, we just
through a nicely crafted view refer to them. Basically, like Apache, we
define the "document root" and then the match in the view gives the path
from the document root. Nothing really innovative but robust.

The only problem we have is then the clean management of all the
expiration headers, the ETag, etc. As we "package" the assets in the
.phar, we basically preprocess them, but if you use the
\photon\middleware\Gzip, you can see that we gzip at each request. Waste
of CPU cycles and added latency... not good.

What a good asset manager should do is:

- Prevent the need to gzip on the fly (we can store the gzip version too).
- Generate clean urls, like '/media/css/style-crc32.css' where crc32 is
generated at build time. This way we can set the expiry header to far in
the future.
- Minimize the .css/.js/.png before packing them.
- Push the files on your preferred CDN, this way you can refer to them
in your code "normally" when develop in local, but it links to the CDN
in production.
- And more...

In PHP you have the Assetic library which is a port of the webassets
Python library[0]. The problem is that it is 4000 lines of code
(sloccount) whithout the tests and Photon is 2600 lines (from the code
coverage analysis). So, this is clear that they are not going to play
well together.

So, my goal is to start with a very simple system.

1. keep the current media view, which allow a predictive way to access a
file. For example, in your html code you can write manually
example.com/media/folder/funnypic.jpg and it will be served. Maybe not
with all the optimizations, but simple if you write an article in your
Photon powered blog and want to link to a file.

2. A special view with automatic cache and compression handling.

The requests will have the format:

    example.com/media/folder/funnypic.a8e94c62.jpg

here a8e94c62 is basically dechex(crc32('content of the file')). This
means that the link is valid as long as the content of funnypic.jpg does
not change.

In your code, you would get the link by using something like:

    $url = Asset::url('asset_view', '/folder/funnypic.jpg');

This call would basically generate the crc32 of the file, insert it
before the extension and then pass it to URL::forView('asset_view',
$path); to generate the full url.

In the templates, a simple:

    {asset '/folder/funnypic.jpg'}

would do the same.

The limitation would be that you have only one "smart" asset view. But I
think that it is a good start to keep it simple. The goal is really here
to keep things simple.

3. Hooks to package the assets in the .phar

Here the hooks would be very simple, basically, you could define in your
configuration file a mapping like:

    array('.css' => 'sass,mini',
          '.js' => 'closure,mini');

where sass, mini, closure would describe a set of filters to be run on
the content of the files before packaging. The idea is to have this
independent of all the caching/gzip code to again keep the code simple
and small.

So yes, a bit long, but the packaging as .phar is opening a lot of
opportunities to do things efficiently by doing the hard work once and
reuse it later.

Your feedback is welcomed,
loïc



[0]: http://elsdoerfer.name/files/docs/webassets/

Re: [photon.users] Assets management with Photon, ideas

From:
Nicolas
Date:
2011-10-19 @ 15:32
Hello,

The idea of preparing optimized versions of our assets while we package our
application is great. This will be a huge improvement to what most
developers usually do when they deploy. And it is automated, so I love it :)

Now there is something I just thought about : What if some of the assets are
provided at run-time ? Let's say for instance that I'm building an app that
lets people upload photos to share them with their friends. How should I
handle the uploaded files with Photon ?

Nicolas





On Wed, Oct 19, 2011 at 2:57 PM, Loic d'Anterroches <loic@ceondo.com> wrote:

> Hello,
>
> at the moment, the way we go is: the files are in a folder, we just
> through a nicely crafted view refer to them. Basically, like Apache, we
> define the "document root" and then the match in the view gives the path
> from the document root. Nothing really innovative but robust.
>
> The only problem we have is then the clean management of all the
> expiration headers, the ETag, etc. As we "package" the assets in the
> .phar, we basically preprocess them, but if you use the
> \photon\middleware\Gzip, you can see that we gzip at each request. Waste
> of CPU cycles and added latency... not good.
>
> What a good asset manager should do is:
>
> - Prevent the need to gzip on the fly (we can store the gzip version too).
> - Generate clean urls, like '/media/css/style-crc32.css' where crc32 is
> generated at build time. This way we can set the expiry header to far in
> the future.
> - Minimize the .css/.js/.png before packing them.
> - Push the files on your preferred CDN, this way you can refer to them
> in your code "normally" when develop in local, but it links to the CDN
> in production.
> - And more...
>
> In PHP you have the Assetic library which is a port of the webassets
> Python library[0]. The problem is that it is 4000 lines of code
> (sloccount) whithout the tests and Photon is 2600 lines (from the code
> coverage analysis). So, this is clear that they are not going to play
> well together.
>
> So, my goal is to start with a very simple system.
>
> 1. keep the current media view, which allow a predictive way to access a
> file. For example, in your html code you can write manually
> example.com/media/folder/funnypic.jpg and it will be served. Maybe not
> with all the optimizations, but simple if you write an article in your
> Photon powered blog and want to link to a file.
>
> 2. A special view with automatic cache and compression handling.
>
> The requests will have the format:
>
>    example.com/media/folder/funnypic.a8e94c62.jpg
>
> here a8e94c62 is basically dechex(crc32('content of the file')). This
> means that the link is valid as long as the content of funnypic.jpg does
> not change.
>
> In your code, you would get the link by using something like:
>
>    $url = Asset::url('asset_view', '/folder/funnypic.jpg');
>
> This call would basically generate the crc32 of the file, insert it
> before the extension and then pass it to URL::forView('asset_view',
> $path); to generate the full url.
>
> In the templates, a simple:
>
>    {asset '/folder/funnypic.jpg'}
>
> would do the same.
>
> The limitation would be that you have only one "smart" asset view. But I
> think that it is a good start to keep it simple. The goal is really here
> to keep things simple.
>
> 3. Hooks to package the assets in the .phar
>
> Here the hooks would be very simple, basically, you could define in your
> configuration file a mapping like:
>
>    array('.css' => 'sass,mini',
>          '.js' => 'closure,mini');
>
> where sass, mini, closure would describe a set of filters to be run on
> the content of the files before packaging. The idea is to have this
> independent of all the caching/gzip code to again keep the code simple
> and small.
>
> So yes, a bit long, but the packaging as .phar is opening a lot of
> opportunities to do things efficiently by doing the hard work once and
> reuse it later.
>
> Your feedback is welcomed,
> loïc
>
>
>
> [0]: http://elsdoerfer.name/files/docs/webassets/
>
>

Re: [photon.users] Assets management with Photon, ideas

From:
Loic d'Anterroches
Date:
2011-10-19 @ 17:23
Hello,

> The idea of preparing optimized versions of our assets while we package
> our application is great. This will be a huge improvement to what most
> developers usually do when they deploy. And it is automated, so I love it :)

Yes, automation is what I want, basically you should not have to think
to do the right way.

> Now there is something I just thought about : What if some of the assets
> are provided at run-time ? Let's say for instance that I'm building an
> app that lets people upload photos to share them with their friends. How
> should I handle the uploaded files with Photon ?

Ok, this is not what I consider part of the application asset, but more
user provided. The more I code web application, the more my motto
becomes: "do not be file system dependent".

What I do is very simple:

- push everything into MongoDB directly embedded in the document or with
filefs.
- push into a 3rd party provider like Amazon S3.

This allows me to deploy the application anywhere on my cloud. I either
access the files directly on S3 or in my MongoDB replica set.

Of course, when you store, you can apply some optimizations but this
will be application specific.

This is my approach and this is not necessarily the best, but I found
that having basically everything in MongoDB is saving a great deal of
time because I can backup everything in a simple way etc.

I hope it can help you.

loïc


> 
> On Wed, Oct 19, 2011 at 2:57 PM, Loic d'Anterroches <loic@ceondo.com
> <mailto:loic@ceondo.com>> wrote:
> 
>     Hello,
> 
>     at the moment, the way we go is: the files are in a folder, we just
>     through a nicely crafted view refer to them. Basically, like Apache, we
>     define the "document root" and then the match in the view gives the path
>     from the document root. Nothing really innovative but robust.
> 
>     The only problem we have is then the clean management of all the
>     expiration headers, the ETag, etc. As we "package" the assets in the
>     .phar, we basically preprocess them, but if you use the
>     \photon\middleware\Gzip, you can see that we gzip at each request. Waste
>     of CPU cycles and added latency... not good.
> 
>     What a good asset manager should do is:
> 
>     - Prevent the need to gzip on the fly (we can store the gzip version
>     too).
>     - Generate clean urls, like '/media/css/style-crc32.css' where crc32 is
>     generated at build time. This way we can set the expiry header to far in
>     the future.
>     - Minimize the .css/.js/.png before packing them.
>     - Push the files on your preferred CDN, this way you can refer to them
>     in your code "normally" when develop in local, but it links to the CDN
>     in production.
>     - And more...
> 
>     In PHP you have the Assetic library which is a port of the webassets
>     Python library[0]. The problem is that it is 4000 lines of code
>     (sloccount) whithout the tests and Photon is 2600 lines (from the code
>     coverage analysis). So, this is clear that they are not going to play
>     well together.
> 
>     So, my goal is to start with a very simple system.
> 
>     1. keep the current media view, which allow a predictive way to access a
>     file. For example, in your html code you can write manually
>     example.com/media/folder/funnypic.jpg
>     <http://example.com/media/folder/funnypic.jpg> and it will be
>     served. Maybe not
>     with all the optimizations, but simple if you write an article in your
>     Photon powered blog and want to link to a file.
> 
>     2. A special view with automatic cache and compression handling.
> 
>     The requests will have the format:
> 
>        example.com/media/folder/funnypic.a8e94c62.jpg
>     <http://example.com/media/folder/funnypic.a8e94c62.jpg>
> 
>     here a8e94c62 is basically dechex(crc32('content of the file')). This
>     means that the link is valid as long as the content of funnypic.jpg does
>     not change.
> 
>     In your code, you would get the link by using something like:
> 
>        $url = Asset::url('asset_view', '/folder/funnypic.jpg');
> 
>     This call would basically generate the crc32 of the file, insert it
>     before the extension and then pass it to URL::forView('asset_view',
>     $path); to generate the full url.
> 
>     In the templates, a simple:
> 
>        {asset '/folder/funnypic.jpg'}
> 
>     would do the same.
> 
>     The limitation would be that you have only one "smart" asset view. But I
>     think that it is a good start to keep it simple. The goal is really here
>     to keep things simple.
> 
>     3. Hooks to package the assets in the .phar
> 
>     Here the hooks would be very simple, basically, you could define in your
>     configuration file a mapping like:
> 
>        array('.css' => 'sass,mini',
>              '.js' => 'closure,mini');
> 
>     where sass, mini, closure would describe a set of filters to be run on
>     the content of the files before packaging. The idea is to have this
>     independent of all the caching/gzip code to again keep the code simple
>     and small.
> 
>     So yes, a bit long, but the packaging as .phar is opening a lot of
>     opportunities to do things efficiently by doing the hard work once and
>     reuse it later.
> 
>     Your feedback is welcomed,
>     loïc
> 
> 
> 
>     [0]: http://elsdoerfer.name/files/docs/webassets/
> 
> 

Re: [photon.users] Assets management with Photon, ideas

From:
Loic d'Anterroches
Date:
2011-10-19 @ 17:35
Hello,

after a small discussion on IRC, about the packaging of several css/js
in one and delivering an optimized version. The idea is coming from
Mehdi (piouPiouM).

In a template you write:

 {assetpack 'css_homepage'}

In the configuration file you put something like:

 array('asset_packs' =>
   array('css_homepage' =>
       array('/folder/style.css',
             '/folder/homepage.css',
             '/folder/foobar.css'),
         'js_homepage' => function() {return array('/foo.js')}
     ));

the path to the css would be relative to a 'media_path' or something
like that. Instead of an array, one could also put a callable generating
the list of files.

The good point is that the list is easily known at packaging time, this
means that one can really at packaging time convert this to the
corresponding optimized version with crc32 etc.

Of course if one have several applications into one project you may need
.css/.js from many of them in your home page. So you would have to merge
the pack. I need to see how to do that nicely to avoid ending up with
crazy configuration files.

loïc





On 2011-10-19 14:57, Loic d'Anterroches wrote:
> Hello,
> 
> at the moment, the way we go is: the files are in a folder, we just
> through a nicely crafted view refer to them. Basically, like Apache, we
> define the "document root" and then the match in the view gives the path
> from the document root. Nothing really innovative but robust.
> 
> The only problem we have is then the clean management of all the
> expiration headers, the ETag, etc. As we "package" the assets in the
> .phar, we basically preprocess them, but if you use the
> \photon\middleware\Gzip, you can see that we gzip at each request. Waste
> of CPU cycles and added latency... not good.
> 
> What a good asset manager should do is:
> 
> - Prevent the need to gzip on the fly (we can store the gzip version too).
> - Generate clean urls, like '/media/css/style-crc32.css' where crc32 is
> generated at build time. This way we can set the expiry header to far in
> the future.
> - Minimize the .css/.js/.png before packing them.
> - Push the files on your preferred CDN, this way you can refer to them
> in your code "normally" when develop in local, but it links to the CDN
> in production.
> - And more...
> 
> In PHP you have the Assetic library which is a port of the webassets
> Python library[0]. The problem is that it is 4000 lines of code
> (sloccount) whithout the tests and Photon is 2600 lines (from the code
> coverage analysis). So, this is clear that they are not going to play
> well together.
> 
> So, my goal is to start with a very simple system.
> 
> 1. keep the current media view, which allow a predictive way to access a
> file. For example, in your html code you can write manually
> example.com/media/folder/funnypic.jpg and it will be served. Maybe not
> with all the optimizations, but simple if you write an article in your
> Photon powered blog and want to link to a file.
> 
> 2. A special view with automatic cache and compression handling.
> 
> The requests will have the format:
> 
>     example.com/media/folder/funnypic.a8e94c62.jpg
> 
> here a8e94c62 is basically dechex(crc32('content of the file')). This
> means that the link is valid as long as the content of funnypic.jpg does
> not change.
> 
> In your code, you would get the link by using something like:
> 
>     $url = Asset::url('asset_view', '/folder/funnypic.jpg');
> 
> This call would basically generate the crc32 of the file, insert it
> before the extension and then pass it to URL::forView('asset_view',
> $path); to generate the full url.
> 
> In the templates, a simple:
> 
>     {asset '/folder/funnypic.jpg'}
> 
> would do the same.
> 
> The limitation would be that you have only one "smart" asset view. But I
> think that it is a good start to keep it simple. The goal is really here
> to keep things simple.
> 
> 3. Hooks to package the assets in the .phar
> 
> Here the hooks would be very simple, basically, you could define in your
> configuration file a mapping like:
> 
>     array('.css' => 'sass,mini',
>           '.js' => 'closure,mini');
> 
> where sass, mini, closure would describe a set of filters to be run on
> the content of the files before packaging. The idea is to have this
> independent of all the caching/gzip code to again keep the code simple
> and small.
> 
> So yes, a bit long, but the packaging as .phar is opening a lot of
> opportunities to do things efficiently by doing the hard work once and
> reuse it later.
> 
> Your feedback is welcomed,
> loïc
> 
> 
> 
> [0]: http://elsdoerfer.name/files/docs/webassets/
>