I have been working Brubeck and got it to a point where it feels like a full framework now. I added support for Jinja2 and Tornado template systems. I'm really enjoying the discussion on tnetstrings and thought perhaps everyone would have good opinions on testing. I tried testing Brubeck last night with siege and my first test suggests it holds up with Tornado. https://gist.github.com/882555 But I ran this on my iMac at home. I want to do something more like what we'd see in a web env. I think I will get two boxes with Linode and use one to peg the other. I will probably increment the blast rate until I have an idea of when one of them starts to fail. What else would you guys do?
Hello, On 2011-03-23 18:57, James Dennis wrote: > I have been working Brubeck and got it to a point where it feels like a > full framework now. I added support for Jinja2 and Tornado template systems. > > I'm really enjoying the discussion on tnetstrings and thought perhaps > everyone would have good opinions on testing. > > I tried testing Brubeck last night with siege and my first test suggests > it holds up with Tornado. > > https://gist.github.com/882555 > > But I ran this on my iMac at home. I want to do something more like what > we'd see in a web env. I think I will get two boxes with Linode and use > one to peg the other. I will probably increment the blast rate until I > have an idea of when one of them starts to fail. > > What else would you guys do? Testing is a pain. A big pain. You can always hammer with siege but the real important point is going from siege to real world usage. When you use siege, you end up most of the time, if you have good links, to just be CPU limited. In practice, you are never CPU limited because your application will run your business logic (DB access, disc access, other services). So, when you hammer your system, you end up having everything cached in RAM, if your DB queries are not stupid they will hit indexes, etc. But when your application start to be under load, you start to suffer because you have disc access everywhere (so a lot of seeks without cache) and this really start to "hit" you. So, if your system can on your development machine be as good as nginx/tornado/etc. you are already good, very good. For EC2, you can always test with it, the problem is that you have no idea who is sharing the hardware node at the same time you do the test, this means you need to run a lot of tests at different times, with different instances, to at the end get results which have statistical significance. So, my approach would be to do comparative testing against what you are running now and trying to reproduce the edge cases of your current stack. What happens when you are IO bound, etc... what is really interesting is also not the raw number of requests per second, but the deviation from the average. req/s is just a mean and you know nearly nothing when you know it. A good read, because fast and easy: http://zedshaw.com/essays/programmer_stats.html If you are speaking statistics fluently, please ignore everything I wrote, you know most likely more than me. loïc -- Indefero - Project management and code hosting - http://www.indefero.net Photon - High Performance PHP Framework - http://photon-project.com Céondo Ltd - Web + Science = Fun - http://www.ceondo.com
Siege let's you build up a list of requests to try and allows you to add specific POST or HEADER data too. I'm starting with slamming, but I agree that a more comprehensive test would require activity more typical of big scale site's performance. Thanks for the reminder about Zed's post on stats. My single gist doesn't account for variance in anyway, but it gives me hope that Brubeck can help people get off Tornado. I'll report back after more testing. On Wed, Mar 23, 2011 at 3:12 PM, Loic d'Anterroches <loic@ceondo.com> wrote: > Hello, > > On 2011-03-23 18:57, James Dennis wrote: > > I have been working Brubeck and got it to a point where it feels like a > > full framework now. I added support for Jinja2 and Tornado template > systems. > > > > I'm really enjoying the discussion on tnetstrings and thought perhaps > > everyone would have good opinions on testing. > > > > I tried testing Brubeck last night with siege and my first test suggests > > it holds up with Tornado. > > > > https://gist.github.com/882555 > > > > But I ran this on my iMac at home. I want to do something more like what > > we'd see in a web env. I think I will get two boxes with Linode and use > > one to peg the other. I will probably increment the blast rate until I > > have an idea of when one of them starts to fail. > > > > What else would you guys do? > > Testing is a pain. A big pain. You can always hammer with siege but the > real important point is going from siege to real world usage. When you > use siege, you end up most of the time, if you have good links, to just > be CPU limited. In practice, you are never CPU limited because your > application will run your business logic (DB access, disc access, other > services). So, when you hammer your system, you end up having everything > cached in RAM, if your DB queries are not stupid they will hit indexes, > etc. > > But when your application start to be under load, you start to suffer > because you have disc access everywhere (so a lot of seeks without > cache) and this really start to "hit" you. So, if your system can on > your development machine be as good as nginx/tornado/etc. you are > already good, very good. > > For EC2, you can always test with it, the problem is that you have no > idea who is sharing the hardware node at the same time you do the test, > this means you need to run a lot of tests at different times, with > different instances, to at the end get results which have statistical > significance. > > So, my approach would be to do comparative testing against what you are > running now and trying to reproduce the edge cases of your current > stack. What happens when you are IO bound, etc... what is really > interesting is also not the raw number of requests per second, but the > deviation from the average. req/s is just a mean and you know nearly > nothing when you know it. > > A good read, because fast and easy: > > http://zedshaw.com/essays/programmer_stats.html > > If you are speaking statistics fluently, please ignore everything I > wrote, you know most likely more than me. > > loïc > > -- > Indefero - Project management and code hosting - http://www.indefero.net > Photon - High Performance PHP Framework - http://photon-project.com > Céondo Ltd - Web + Science = Fun - http://www.ceondo.com > >
On Wed, Mar 23, 2011 at 10:57 AM, James Dennis <jdennis@gmail.com> wrote: > But I ran this on my iMac at home. I want to do something more like what > we'd see in a web env. I think I will get two boxes with Linode and use one > to peg the other. I will probably increment the blast rate until I have an > idea of when one of them starts to fail. > What else would you guys do? Seems like this is just the sort of thing EC2 is meant for.
Linode makes a similar offering, but I am going to go with EC2 anyway. I think the performance variance will be interesting! On Wed, Mar 23, 2011 at 2:02 PM, Jon Rosebaugh <chairos@gmail.com> wrote: > On Wed, Mar 23, 2011 at 10:57 AM, James Dennis <jdennis@gmail.com> wrote: > > But I ran this on my iMac at home. I want to do something more like what > > we'd see in a web env. I think I will get two boxes with Linode and use > one > > to peg the other. I will probably increment the blast rate until I have > an > > idea of when one of them starts to fail. > > What else would you guys do? > > Seems like this is just the sort of thing EC2 is meant for. >