[Mnet-devel] erasure code optimization project

[Zooko]

Here is a new darcs repository (as always, if you don't have darcs, you 
can fetch the files with wget or with a web browser):

http://mnetproject.org/repos/ecopt

Here is the contents of the README file:

This is an experiment in optimizing erasure codes.  The motivation is 
to learn
about erasure codes and hopefully come up with a faster implementation 
for the
use of Mnet [1].

I took erasure code implementations from Rizzo [2] and from Plank [3].  
I intend
to compare them and to try optimizing both of them, then use the 
fastest one in
Mnet.

The current status of this project is that I've written a driver script 
(copied
from Plank's code) to invoke Rizzo's code on a file and write the 
results into a
file.

To run this preliminary benchmark yourself:

1. build:
cd plankerasureimpl && make && cd rizzoerasureimpl && make

2. get a test file: e.g.
python -c "open('r1','w').write(open('/dev/urandom','r').read(2**17))"

3. run one of the encoders and see how long it takes:
time ./plankerasureimpl/rs_encode_file r1 84 170 x

4. make a temporary directory to hold a subset of the blocks, cd into 
that
    directory, and use the "randsel.py" script to make symlinks to a 
random
    subset of the blocks:
mkdir testrun && cd testrun && ../randsel.py x 84

5. run the appropriate decoder and see how long it takes:
time ../plankerasureimpl/rs_decode_file x > r1.decoder 2>decode.log

Now do likewise with the other implementation.  Now optimize one of 
them and
send me the patch!

I intend to do that last step myself, but first I'm going to help an 
Mnet newbie
set up a private Mnet network, and update the Mnet docs so that the 
next such
newbie won't need my help.

Regards,

Zooko

zooko at zooko.com

[1] http://mnetproject.org/
[2] http://info.iet.unipi.it/~luigi/fec.html
[3] http://www.cs.utk.edu/~plank/plank/papers/CS-03-510.html