[mnet-devel] Ickle me, pickle me, tickle me too...

[Zooko]

You can inherit from pyutil.LazySaver, and tell it how often to save to disk 
(e.g. "every 60 seconds"), and call "self._lazy_save()" whenever something has 
changed (i.e., a block added or deleted).  It will then pickle-and-dump-to-
disk every 60 seconds but only if something changed in the previous 60 seconds.

LazySaver achieves overwriting the previous state by dint of opening the file 
in "w" mode and them using "dumps()".  This means the pickle gets built into a 
Python string in RAM before begin written and then freed.

As to speed of different dumpers, I did some benchmarking, and I just cvs 
committed the benchmarking code to mnet_new/mnetlib/scripts/benchpickle.py.  
Appended is sample output of this script on my machine, with the default 
settings for "how long to run the thing while timing it", and with me using 
the machine (including XEmacs) while it ran.

I suggest that someone run it on an unloaded machine and post the results!

N is the number of items in the dict to be dumped.  "ave rate:" is the 
average "items dumped per second".

The fastest way to dump would be a custom dumper that does a 4-byte (signed) 
length prepend to a string, but that isn't implemented currently.  The 
"myownspecialdumper" algorithm is a failed attempt to do that.

It would be easy to implement.  (Use the Python struct module, or be macho and 
write an extension module in C.  Hey -- I bet Faried would write such a module 
for us!  :-)  For bonus points spew the result straight to/from file instead of 
working with a string in RAM.)

--Z



-------------------------------------------------------
This SF.Net email sponsored by: Free pre-built ASP.NET sites including
Data Reports, E-commerce, Portals, and Forums are available now.
Download today and enter to win an XBOX or Visual Studio .NET.
http://aspnet.click-url.com/go/psa00100003ave/direct;at.aspnet_072303_01/01
_______________________________________________
mnet-devel mailing list
mnet-devel at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mnet-devel