writing numpy array into pickle file
- Peter Prettenhofer
- 2014-02-03 @ 00:39
I'm trying to write a single file storage for joblib. My goal is to write a
custom pickler that writes every ndarray directly into the file generated
by the pickler -- it should support chunked writes and compression.
Basically, the output should be the same as the default pickler but the
"data" state of the array should be written in chunks to limit memory
consumption -- ideally, I'd create a wrapper for the array that would have
its data state compressed.
I'm grateful for any pointers how to best accomplish this.
Re: [joblib] writing numpy array into pickle file
- Olivier Grisel
- 2014-02-03 @ 12:57
I had started to investigate how to do that in the past as well but
fall short of finding a fully working solution. I think this requires
some work refactoring the joblib custom numpy pickler to not rely on
the the __setstate__ method as per the discussions in: