Proving Server Faults: RPCs for Distributed Systems in Byzantine Networks

Distributed systems are often designed to recover from downed nodes. Unfortunately, it is challenging to create recovery mechanisms that work in Byzantine networks, where the attacker controls some of the nodes and links. Often times an adversarial node can lie about an honest node being offline, and there is no way to verify this claim or detect the liar. To resolve this challenge, we design rRPC, a robust remote procedure call library for distributed systems running in Byzantine networks. rRPC ensures that either the call succeeds or the online caller/callee can create a third-party verifiable proof that the other party is faulty. A distributed system can use these proofs to identify and remove a faulty node automatically. We implement a prototype of rRPC, and use 20 - 100 EC2 VMs to evaluate its performance as a standalone library and in the context of a distributed mix-net system. Our results quantitatively show that rRPC's overhead is low and induces about 1% increase in latency in the common case.