librelist archives

« back to archive

Why one process per handler without threads is good

Why one process per handler without threads is good

From:
Loic d'Anterroches
Date:
2011-02-27 @ 12:54
Hello,

interesting scaling issues/lessons presentation at reddit:
http://thinkvitamin.com/code/steve-huffman-on-lessons-learned-at-reddit/

Here are two quotes:

"The other lesson we learned is don’t use threads for anything. We’re in
Python, so Python threads are just the kiss of death, the recipe for
slow. We just run everything in multiple processes. Once you solve that
problem of having different jobs and different processes, for us it was
things like spam and thumbnails, and query caching sort of stuff; it
allows you to put it on different machines."

"Do the minimum amount of work on the backend to finish the request and
then tell the user you’re done. If you need to do something else, do it
while they’re not waiting for you. Dump it in a queue. When a user votes
on Reddit, that updates all the listings that link is in, it updates a
bunch of users’ karma, it updates his liked page, it updates all sorts
of stuff"

The 7 lessons are:

1: Crash Often
2: Separation of Services
3: Open Schema
4: Keep it Stateless
5: Memcache
6: Store Redundant Data
7: Work Offline

loïc

--
Indefero - Project management and code hosting - http://www.indefero.net
Photon - High Performance PHP Framework - http://photon-project.com
Céondo Ltd - Web + Science = Fun - http://www.ceondo.com