librelist archives

Re: [nesta] external-by-default and caching

From:

Graham Ashton

Date:

2011-03-11 @ 15:07

On 11 Mar 2011, at 04:07, Mat Schaffer wrote:

> Hello, I wanted to reply to 
https://twitter.com/nestacms/status/45100653789458434 but 140 characters 
just didn't seem like enough, so here goes.

Thanks for going into detail.

> Layout templates weren't in my project by default. Usually the thing I 
like about minimalistic website engines is that they don't hide any html 
from me.

Nesta used to work like that before it was a gem; all you did was clone 
the repo, edit the code, then deploy it. It worked well for a while, but 
the more sites you make, and the more changes I made to the default 
layout, the more painful it becomes to pull in bug fixes. HTML and CSS 
merge conflicts became quite common.

How to re-skin it also came up yesterday; it needs to be easier to import 
the default HTML/CSS into your project, and then edit it.

> After importing the views, it still wasn't obvious how I'd change or 
nest layouts if I wanted to.


Interesting. Did you notice that the layouts are in the views folder too? 
I'm wondering if it seemed as though they were all partial templates?

I'm thinking I should add an option to the nesta command to make this 
smoother. Perhaps something like this:

    $ nesta theme:override default

or

    $ nesta theme:override sometheme

(where you'd already have themes/sometheme installed)

When its copied the files in it could print a short help message telling 
you how to proceed.

> Since nesta was generating pages in real time, I hoped that there would 
be etags or last-modified headers on the requests. But at least running it
locally I didn't see any. I think you mentioned that you can pre-cache or 
something. How does that work?

I deployed the first version of Nesta behind Nginx, which handled cache 
related headers for me. I just wanted Nesta to write HTML to disk.

There's a cache option in config/config.yml that you can set to true or 
false. It controls whether or not Nesta will try and write an HTML file in
the public directory for each request.

Now that we're all using Heroku, HTTP caching comes up a lot more often 
(though I don't think many of us actually need it, which is why I've not 
rushed to implement it). The big question is, how should you configure it?
And should max-age be the same on every page? (I've concluded it 
shouldn't)

Matt Yoho sent me a great patch with plenty of configurable options. I 
didn't merge it as I was concerned about how to label caching related 
settings (not all Nesta's users are Ruby programmers, so it needs to make 
sense), and I wasn't convinced that we need as much control as his patch 
provided.

This plan is the best I've come up with:

https://github.com/gma/nesta/issues#issue/29

It's fairly high on the priority list (though I intend to sort out the 
plugin API first), but I'm still undecided about creating a metadata key 
called "Max age". It doesn't seem very descriptive to me.

And do you all feel that we really need etags, or does setting max-age cover it?

See this issue for some history:

https://github.com/gma/nesta/issues/closed#issue/9

Cheers,
Graham

On Fri, Mar 11, 2011 at 10:07 AM, Graham Ashton <graham@effectif.com> wrote:

>  > After importing the views, it still wasn't obvious how I'd change or
> nest layouts if I wanted to.
>
> Interesting. Did you notice that the layouts are in the views folder too?
> I'm wondering if it seemed as though they were all partial templates?
>

I saw the layout there, but I didn't see how nesta was deciding to use that
file, or how I could make it use a different layout file for different
pages.


> I deployed the first version of Nesta behind Nginx, which handled cache
> related headers for me. I just wanted Nesta to write HTML to disk.
>
> There's a cache option in config/config.yml that you can set to true or
> false. It controls whether or not Nesta will try and write an HTML file in
> the public directory for each request.
>
> Now that we're all using Heroku, HTTP caching comes up a lot more often
> (though I don't think many of us actually need it, which is why I've not
> rushed to implement it). The big question is, how should you configure it?
> And should max-age be the same on every page? (I've concluded it shouldn't)

Matt Yoho sent me a great patch with plenty of configurable options. I
> didn't merge it as I was concerned about how to label caching related
> settings (not all Nesta's users are Ruby programmers, so it needs to make
> sense), and I wasn't convinced that we need as much control as his patch
> provided.
>
> This plan is the best I've come up with:
>
> https://github.com/gma/nesta/issues#issue/29
>
> It's fairly high on the priority list (though I intend to sort out the
> plugin API first), but I'm still undecided about creating a metadata key
> called "Max age". It doesn't seem very descriptive to me.
>
> And do you all feel that we really need etags, or does setting max-age
> cover it?
>
> See this issue for some history:
>
> https://github.com/gma/nesta/issues/closed#issue/9



Yeah, I'm a heroku guy myself. I agree that max-age is a little risky. But I
think at least ETag and Last-Modified would go a long way. You'd still have
to serve each page once, but the subsequent calls through varnish would be
conditional which would improve your chances of remaining online during a
traffic spike. Especially if the Last-Modified check was built to return
very quickly (perhaps the app's start time in 'prod' mode or something to
that effect).

I may see what I can do with the static generation option though.

Thanks,
Mat

On 11 Mar 2011, at 20:22, Mat Schaffer wrote:

> Yeah, I'm a heroku guy myself. I agree that max-age is a little risky. 
But I think at least ETag and Last-Modified would go a long way. You'd 
still have to serve each page once, but the subsequent calls through 
varnish would be conditional which would improve your chances of remaining
online during a traffic spike. Especially if the Last-Modified check was 
built to return very quickly (perhaps the app's start time in 'prod' mode 
or something to that effect).

The reason I was going for max-age is really all down to what Heroku put 
in their docs:

http://devcenter.heroku.com/articles/http-caching

"The simplest way to cache a page is to set a header on output which cues 
the cache that this page can be served, unmodified, to everyone visiting 
that same URL. The Cache-Control header also specifies how long the cached
copy should be retained."

In the example in their docs they just set max-age. I can't see how the 
etag would improve on that unless your content could vary between requests
(which probably isn't that common in Nesta). I'm no HTTP caching wizzard 
though, so feel free to keep me on the right track...

On Sat, Mar 12, 2011 at 5:12 AM, Graham Ashton <graham@effectif.com> wrote:

> On 11 Mar 2011, at 20:22, Mat Schaffer wrote:
>
> > Yeah, I'm a heroku guy myself. I agree that max-age is a little risky.
> But I think at least ETag and Last-Modified would go a long way. You'd still
> have to serve each page once, but the subsequent calls through varnish would
> be conditional which would improve your chances of remaining online during a
> traffic spike. Especially if the Last-Modified check was built to return
> very quickly (perhaps the app's start time in 'prod' mode or something to
> that effect).
>
> The reason I was going for max-age is really all down to what Heroku put in
> their docs:
>
> http://devcenter.heroku.com/articles/http-caching
>
> "The simplest way to cache a page is to set a header on output which cues
> the cache that this page can be served, unmodified, to everyone visiting
> that same URL. The Cache-Control header also specifies how long the cached
> copy should be retained."
>
> In the example in their docs they just set max-age. I can't see how the
> etag would improve on that unless your content could vary between requests
> (which probably isn't that common in Nesta). I'm no HTTP caching wizzard
> though, so feel free to keep me on the right track...


http://railslab.newrelic.com/2009/02/25/episode-10-client-side-caching and
http://railslab.newrelic.com/2009/02/26/episode-11-advanced-http-cachingcover
pretty much everything I know :)

My understanding is that if nesta had etag and last-modified support built
in, it would allow the cache (varnish for heroku) to do conditional GETs. If
every page returned a last-modified time of when the server started, you
could avoid ever having to really generate a page. Nesta would still have to
handle the conditional get requests, but if you do this and short-circuit
the template rendering you should be able to respond very quickly.

Max-age of course can be better because even the conditional GETs never
happen. But finding a satisfactory max-age can be tricky.
-Mat