librelist archives

« back to archive

[ANN] Joblib news: mailing list and cache size limit

[ANN] Joblib news: mailing list and cache size limit

From:
Gael Varoquaux
Date:
2010-08-23 @ 14:31
Hi,

You are getting this e-mail because at some point you showed some
interest in joblib. If you are no longer interested, do not worry, this
will be the last e-mail sent to you personally (follow ups on the mailing
list).

Mailing list and 'organisation' github account
===============================================

As joblib matures, it evolves to more of a shared project than of a
personal project.

* There is now a mailing list for joblib:
    http://librelist.com/browser/joblib/

  To subscribe, simply send a mail to joblib@librelist.com, and the system
  will subscribe you after a confirmation e-mail.

  Please use the mailing list, as it enables open discussions and is
  indexed by google. I suspect that it will be of very low traffic for a
  while.

* The main joblib code is hosted in a github shared repository that does
  not belong specifically to me:

    http://github.com/joblib/joblib

Latest hot feature: cache size limits
======================================

I have just finished coding something that I had been wanting for quite a
while: the option to set a let to the size of the cache directory on the
disk. Now you can avoid having joblib fill in your disk, simple by
passing a "limit='1G'" keyword argument to the Memory class.

I had to modify a lot of code to get there, and this feature is very much
in beta. However, I am using it for my day-to-day work, and, if you are
interested, you can join me in stress-testing it. For this, you need to
get the latest code from github:

    git clone git://github.com/joblib/joblib.git

I will make a release with this feature, but only after a little while,
once I have used and abused it a lot.

Code that I added for this feature did result in a slight increase in
overhead from joblib. In the future, their will be options to have fast
memory, that does not store to disk but only keeps in a RAM, and even
later two levels of cache, to get the best of both worlds. However, this
requires a refactor, and I'd like to let the dust settle a bit in order
to clean up the current state of the code.

Happy joblibbing!

Gaƫl