[mnet-devel] Boy, what we have here is a failure to communicate...

[Zooko O'Whielacronx]

Jim:

Thanks for your message!  It's a very good thing to bring up.

I do think that EGTP is a good idea, and we will ultimately need it even after 
we have ent.

However, I have gotten confused while trying to implement a new emergent 
network with EGTP, so my current plan is to finish implementing the emergent 
network on a separate branch with the least-effort underlying layer (namely 
twisted.PB with the easiest twisted transport underneath, which is almost 
certainly TCP).

(Myers wrote PBChord in only a few days, so this gives me confidence that 
PBent won't take too long.)

After that, I intend to sort of look around and see what is the best way to 
integrate PBent with the rest of the system.  (Heh heh.  "P-bent".)

In the long run, the most appealing thing would be for Twisted to use EGTP so 
that the ent source code could be written in PB's idiom (see example in the 
P.S.) while retaining EGTP's secure comms semantics.

However, there are a few problems that would have to be solved, the biggest 
being that PB assumes an underlying connection.  It's not insoluble, but it's 
definitely a real effort and we would require help from the Twisted folks to 
do it right.

One confusing thing about implementing (but not using) EGTP is that it is 
recursive.  EGTP uses some underlay (e.g. TCP) for transport, but it also uses 
itself for transport sometimes (e.g. Relay).  Another recursion is that EGTP 
can be used to map from a public key to a node, but EGTP also needs to *use* 
some map from public key to node (e.g. MetaTracker, ent, cp2pc) in order to do 
its job.

I'm not sure what I think about these issues yet.

One reassuring thing is that there are three simple abstractions that seem to 
cover everything we need to use or that we provide (except for PB, which is 
the problem with PB).

1.  LookupManager

2.  DiscoveryManager

If you don't know what those two are and how they are different (Artimage), 
then please read these docs:

http://cvs.sourceforge.net/viewcvs.py/mnet/mnet_new/egtp/interfaces.py?view=markup

If you've read the docs and you *still* don't know, then please tell me and 
I will change the docs!  

3.  Routing ("send()")

Routing is where you say "Send this message to this node.".  EGTP offers it.  
It is almost like lookup, since you could implement "send(node, msg)" with 
"socket.connect(lookup(node.id + "'s IP addr")); socket.write(msg)".

That is pretty much what route() does in a MetaTracker net (although actually 
it has an if-then for whether you are using relay or TCP).

However, that would be a very inefficient way to route on a DHT, because the 
lookup() operation itself probably results in a message being sent to the 
node, so it would be a lot more efficient if you just sent the message instead 
of sending "Hey there: give me your IP address", waiting for him to write 
back, then opening a connection and sending your message!


It's too bad about PB not fitting in nicely here.  I want PB for three reasons:

1.  Less code for me to maintain in the upper layer.  The current EGTP 
abstraction requires that the caller marshal and unmarshal arguments and map 
from message-type-strings to methods.  PB does both of those things for you.  
I've been struggling with this while writing ent.  I've spent lots of time 
fixing stupid typos in the repetitive "marshal, map, unmap, unmarshal, 
type-check" code.

2.  Less code for me to maintain in the unit tests.  Actually the biggest 
problems I've been having are in SimulEnt -- a simulated network for unit 
testing.  Myers says that in addition to the above-mentioned features, PB's 
integration with twisted.trial and twisted.reactor make such testing easier.

3.  Less code for me to maintain in the lower layer.  There are lots and lots 
of Twisted hackers who are happy to deal with debugging, optimizing, and 
porting all the stuff from PB on down to reactor.  They do good work!

4.  PB offers a few added features such as object unmarshalling so that when 
my method gets invoked remotely, the argument I have to deal with is a 
full-fledged Python object instead of primitive data.


Okay, I'm signing off for now, and appended is a concrete example from our 
actual source code.

Regards,

Zooko

P.S.

Here's a concrete example.  We currently have code like this in ent.py:

def __init__(self, egtpnode):
    self.egtpnode.set_handler_func("hello", self.remote_hello)

def remote_hello(self, msg):
    _assert(isinstance(msg, dict), "hello msg is required to be a dict.", peer=peer, msg=msg)
    _assert(isinstance(msg.get('nonce'), int), "hello msg is required to contain a nonce which is an int.", peer=peer, msg=msg)
    nonce = msg.get('nonce')
    self.check_nonce(nonce)
    _assert(isinstance(msg.get('commstrat'), dict), "hello msg is required to contain a dict which is a dict.", peer=peer, msg=msg)
    commstrat = CommStrat.dict_to_strategy(msg.get('commstrat'), self.egtpnode.mtm)
    peer = self.peerset.new_peer(commstrat)
    self._make_link_to(peer)

With PB feature #1, we can change it to:

def remote_hello(self, nonce, commstratasdict):
    _assert(isinstance(nonce, int), "nonce is required to be an int.", nonce)
    _assert(isinstance(commstratasdict, dict), "commstrat is required to be a dict.", commstratasdict)
    self.check_nonce(nonce)
    commstrat = CommStrat.dict_to_strategy(commstratasdict, self.egtpnode.mtm)
    peer = self.peerset.new_peer(commstrat)
    self._make_link_to(peer)
    
With PB feature #4, we can change it to:

def remote_hello(self, nonce, peer):
    self.check_nonce(nonce)
    self._make_link_to(peer)



-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
mnet-devel mailing list
mnet-devel at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mnet-devel