librelist archives

« back to archive

Locking datastore refactor

Locking datastore refactor

From:
Dag Sverre Seljebotn
Date:
2011-03-28 @ 09:39
I've been working on a locking data store very slowly in spare moments. 
I found that the only realistic option was to prepare for a simple 
refactoring, because I could never write such code without very isolated 
unit tests.

https://github.com/dagss/joblib/blob/locking/joblib/store.py
https://github.com/dagss/joblib/blob/locking/joblib/test/test_store.py

I haven't integrated it with Memory.py yet, but that should be a very 
quick job. I know, waterfall development and all that, but I felt I 
could justify it in this rather simple case.

Comments welcome on the data store API before I actually put the 
necessary hour in to refactor Memory.py

Todo:
  - Memory.clear() -> data store (which is still missing)
  - Actually refactor Memory.py.
  - Write stress-testing script to provoke deadlock situations and make 
sure they're all catered for (they should be)
     - Spawns thousands of processes with thousands of threads, which 
gets SIGKILL-ed (or segfaults itself...) randomly, and also monkeypatch 
"lockf" to make it randomly return a lock without locking :-)

Dag Sverre