librelist archives

« back to archive

concerning implementation using numpy.memmap

concerning implementation using numpy.memmap

From:
chunyang xiao
Date:
2015-01-20 @ 10:54
Hello,

From the documentation, I see that shared data is realized through
numpy.memmap.


However, I see from numpy documentation that "Memory-mapped arrays use the
Python memory-map object which (prior to Python 2.5) does not allow files
to be larger than a certain size depending on the platform. This size is
always < 2GB even on 64-bit systems."

So how to work round this subject if I have such a large file to load into
memory please?

Thank you for your help

Re: [joblib] concerning implementation using numpy.memmap

From:
Gael Varoquaux
Date:
2015-01-20 @ 11:12
> So how to work round this subject if I have such a large file to load into
> memory please?

Use a Python version greater than 2.4. 2.5 has been out since quite a
while.

Re: [joblib] concerning implementation using numpy.memmap

From:
Olivier Grisel
Date:
2015-01-20 @ 12:05
I have no problem creating a 10GB memmap'ed file:

a = np.memmap('/tmp/big.bin', shape=1e10, dtype=np.uint8, mode='w+')
a[:] = 42

-- 
Olivier