librelist archives

« back to archive

Knowing in advance if memoised function will result in a cache miss or hit

Knowing in advance if memoised function will result in a cache miss or hit

From:
Arnaud Joly
Date:
2014-08-08 @ 11:02
Hi,


Is it  possible to know if a function call have already been memoised?
Otherwise said is it possible to know in advance if a joblib memorised function
will result in a cache hit or a cache miss?

I would like to use such feature for the following scenario:
	1. Being able to launch computation based on what has already been done
	    previously. The main purpose would be to have a dedicated job
	   on slurm / sge cluster for each memorised function call. I would only 
	   launch new slurm / sge jobs if the function is not memorized.


	2. make heavy computation on a fast remote machine (with many cores)
	-> cache the result of the function call
	-> fetch the cache directory
	 -> analyse the resulting data only for available cache output functions.
	

Best regards,
Arnaud Joly