[mnet-devel] Re: [freenet-dev] (no subject)

[Zooko]

Ian: thanks for your message!

 Ian Clarke wrote:
>
> Thanks for your comments - they are extremely interesting and suggest
> that there is increasing convergence between the two projects.

Yep.  Since Mnet has (temporarily) lost economic mechanisms, since Freenet is 
adding erasure coding, and since both projects are currently developing new 
routing schemes, the two designs have grown closer.

However I think (and hope) that this similarity will be temporary -- the new 
routing schemes will probably be substantially different, and each project 
will continue to add new features that the other project doesn't.


> I look
> forward to continuing to work together to our mutual benefit, however
> before continuing, I do want to say that as the former Nigerian Minister
> for Penis Enlargement I think it is quite a shame that this email won't
> make it to the Mnet mailing list.

Heh heh heh.  Fortunately, I found your letter in my "special offers" folder 
while I was searching for a way to buy a green card application.


> Anyway, one high-level point that I think is worth making at the outset
> is that it is important to draw a distinction between NGrouting, which
> is more of a general principal, and the specifics of our current
> implementation.

Ah!  That *is* interesting.  The abstraction of NGrouting requires only that 
the decision of which node to send to is based on past history, including both 
performance and keyspace.

I think you also want a sort of "convergence" property -- you want it to be 
the case that even if node A and node B use *different* NGrouting 
implementations to evaluate node C, that they tend to come to similar 
conclusions about which part of the keyspace node C specializes in.  This 
would seem to be a natural result of almost any reasonable NGrouting scheme, 
wouldn't it?

Hm.  The longer I think about it, the more ways I come up with for two 
different NGrouting schemes to develop divergent views of which peers handle 
which keyspace.  That would be bad.

Perhaps the convergence property should be written down in the NGrouting 
doc -- it is very important for the behavior of a large-scale network.


> Well, I am not exactly sure what you mean by this - but perhaps your
> question will be answered by the fact that we plan to introduce some
> random salt in the decision as to where a request should be routed, and
> also the fact that the more DNFs a node returns - the worse future
> estimates for its routing time will be.

M-hm.  Interesting.  Thanks for the answer to my question.


> > So Mnet combines all kinds of failure (stored in hit rate) and
> > performance (stored in average turnaround time) into a single scalar:
> > (1/latency) * hitrate.
> 
> Very interesting indeed, and quite a different way to look at it.  If
> you were to describe what this number is an estimate of - how would you
> characterize it?  Something like "the average amount of data retrieved
> assuming continuous requests for data"?  I realize that this isn't it -
> but something similar?

No I think that's it -- average amount of data retrieved per time, assuming 
continuous requests for data.  I think it is really just the inverse of your 
"expected time to get the data by hook or by crook".  Using the inverse notion 
of rate instead of latency makes it easy to factor in failures by simply 
multiplying in the failure probability.

There is definitely some information lost by this though.  Suppose peer A and 
peer B have identical chance of returning data versus returning DNF, and it 
takes them an identical amount of time to return the data if they return it, 
but peer A returns DNFs twice as fast as peer B.  Well, peer A is definitely 
better, and the more frequent DNFs are the better peer A becomes, but the Mnet 
measure gives peer A and peer B identical scores.

Does the Freenet measure distinguish between peer A and peer B?  I'm not sure.


> Yes, comparisons of Freenet, particularly post-NGrouting, with DHT
> routing algorithms are *very* interesting to me - since they represent
> different approaches, which rely on a similar vague principal (get
> closer to the data at each hop), but where Freenet's approach is more
> "organic" or "heuristic", whereas DHTs are quite rigorous.  This gives
> DHTs the benefit of being able to make performance guarantees, which are
> essential in many situations, but my suspicion is that a more organic
> approach might be better-suited to tolerate the unreliability and
> unpredictability of the Internet (particularly when relying on peers
> that can vanish at any moment).

Yep.  I wish there were some really good simulations of NGrouting -- it seems 
hard to answer these questions analytically.


> Well, I am not so sure - we can observe the accuracy of the estimates
> produced by this NGrouting implementation for a single node, and based
> on that accuracy we should be able to make inferences about the
> network's performance as a whole.

Hm.  Better performance from the perspective of a single node *does* suggest 
better performance of the network as a whole, but only assuming fairly 
convergent routing.  That is: if people inserting data into the network are 
choosing different nodes to store the data than the people requesting the data 
from the network choose to query, then the network as a whole will suffer even 
though from any individual node's perspective his own strategy is locally 
optimal.  A similar argument goes for two different queriers using different 
routing, because if their routes are divergent they lose caching, and maybe 
they even get DNFs.


> In the case where a node received most of its requests within 0.004 of
> its keyspace - then isn't it desirable that most of the points be within
> that segment to more accurately represent the variations in request
> times within that segment of keyspace for which the rest of the network
> obviously considers this node to be an expert?

Yes, but wouldn't it take an infinitely long time for the points to get 
dragged into a small space?  I guess I misunderstood how the running average 
is computed.


> Perhaps we are working to different definitions of "scale-free".

Hm.  Well, the constant number "10" interacts with the size of the network, at 
least at *first*, if not indefinitely.  If you start up a fresh new network of 
5 Freenet nodes, then the computation will tend to influence around four of 
the points at first, whereas if you start up a fresh new network of 50,000 
Freenet nodes, then the computation will tend to influence only two of the 
nodes at first.  It isn't clear to me how long this difference would persist 
as the Freenet network matures.

Regards,

Zooko

http://zooko.com/



-------------------------------------------------------
This SF.Net email sponsored by: Free pre-built ASP.NET sites including
Data Reports, E-commerce, Portals, and Forums are available now.
Download today and enter to win an XBOX or Visual Studio .NET.
http://aspnet.click-url.com/go/psa00100003ave/direct;at.aspnet_072303_01/01
_______________________________________________
mnet-devel mailing list
mnet-devel at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mnet-devel