librelist archives

« back to archive

writing numpy array into pickle file

writing numpy array into pickle file

From:
Peter Prettenhofer
Date:
2014-02-03 @ 00:39
Hi list,

I'm trying to write a single file storage for joblib. My goal is to write a
custom pickler that writes every ndarray directly into the file generated
by the pickler -- it should support chunked writes and compression.
Basically, the output should be the same as the default pickler but the
"data" state of the array should be written in chunks to limit memory
consumption -- ideally, I'd create a wrapper for the array that would have
its data state compressed.
I'm grateful for any pointers how to best accomplish this.

thanks,
 Peter

-- 
Peter Prettenhofer

Re: [joblib] writing numpy array into pickle file

From:
Olivier Grisel
Date:
2014-02-03 @ 12:57
For reference:

I had started to investigate how to do that in the past as well but
fall short of finding a fully working solution. I think this requires
some work refactoring the joblib custom numpy pickler to not rely on
the the __setstate__ method as per the discussions in:

https://github.com/joblib/joblib/pull/115
https://github.com/ogrisel/joblib/commit/569dca33f7ab5409fba288b6e02e9e4cd090d228


-- 
Olivier