Gossip

As explained in the Sync document, any blockchain implementation needs communication with peers in order to be truly useful. Apart from the lower level details described in that document, there is the added dimension of how peers are selected for communication, how connections to those peers are maintained over time. This is the purpose of gossiping functionality.

Initial Configuration

The initial set of peers is defined by configuration when starting the node. This is a predefined set of peers that are automatically connected to upon startup.

Peers

A peer is identified by a URI consisting of the protocol 'aenode://', the public key, an '@' character, the hostname or IP number, a ':' character and the Noise port number the node is using. If the address contains a hostname, it will be resolved to an IP. Example:

aenode://pp_HryRGHJ7Ct3trkktVyVBgfhHL1J4EYSD9cScuMZDV61eSHrCZ@mynode.example.com:3015

A node is uniquely identified by its public key, and it can only have one IP and port at the same time. This means one IP can have several instances of æternity node started at the same time listening on different ports if they have different public keys. This key corresponds to the private key that is used for the Noise protocol listener associated with the IP and port.

Apart from being used in encryption via the Noise protocol, the public key is also used to determine which node should keep its initiated connection open in case two connections are opened (see Gossiping of New Peers).

Gossiping of New Peers

Whenever a ping message is exchanged between peers, either a ping request or ping response, the peers also attach a subset of neighboring peers they know of. The node that generates the message populates the neighbor list with 32 peers at most that are randomly selected from its verified peer pool (see Peers Maintenance).

When a ping request is received from another peer, that peer is first checked for acceptance. This includes verifying

It is not the local node itself
The peer is not on the blocked list
It doesn't already have an existing connection

(3) refers to the case where two peers have learned about each other separately and are both trying to initiate a connection. The connections are both initiated via Noise, but once they are connected each node checks if they already have a connection to that same peer. If they do, they will keep it if their public key is larger than their peers, and drop it if not. This ordering of keys is arbitrary but achieves the effect of only ever being true at one node or the other. Thus, only one connection is kept.

Once the peer is accepted, all the neighbors not already verified are added to the unverified pool. If this is the first ping from an inbound connection from an unverified peer, the peer itself is added to the unverified pool too (see Peers Maintenance for more details).

Peers Maintenance

The objectives of peer maintenance are:

Prevent peer poisoning (Sybil/eclipse attack)
Limit the number of active peer connections
Cache the known peers between restart to make peer poisoning harder.

To prevent an attacker from poisoning the list of peers, isolating the node from the rest of the network, we make it impossible to predict the set of peer's IP/port the attacker should control to fill enough of the list for the attack to be statistically successful. In addition, we randomize the peer eviction to prevent an attacker from predicting it and replace all good peers for their own compromised ones.

This is achieved by first categorizing peers in two groups:

The unverified pool contains all the peers received through gossip. It ensures no Byzantine node can fill it completely by limiting the subset of the pool it can affect; it prevents attackers from predicting the set of IP/port they must use to successfully eclipse the node.
The verified pool contains the peers the node connected to explicitly after passing through the filter of the unverified pool. It randomizes the peer distribution to the eye of attackers, preventing them from predicting the eviction algorithm.

Peers are considered verified when the node has been able to connect to them using the Noise protocol.

Both groups clusterize peers into multiple buckets; they select which one an added peer belongs to based on a combination of the address of the node the information is coming from, the address of the peer to be added and a secret generated by the node for the purpose of randomizing the selection process. See Unverified Pool and Verified Pool for more details on how the peers are added to the pools.

The peers received from configuration are added directly to the verified pool and are marked as "trusted", meaning that they will never get downgraded to the unverified pool even if the node cannot connect to them.

Peer Groups

Peers are grouped by the 16 most significant bits of a peer IP (\16 mask). This group is used when selecting a bucket and establishing the node outbound connections to prevent nodes from being connected only to local nodes; this make block forwarding more effective. NOTE: If all the nodes are in the same group, they may not reach their maximum number of outbound connection and this should probably be disabled.

Connections

There are two pools of connections, inbound connection and outbound connections.

Both inbound and outbound connections are used for mempool, gossip and sync protocol; but only outbound connections are used for relaying new blocks to reduce the network load.

There is a hard limit on the maximum number of outbound connection (default to 10).

There is a soft limit on the maximum of inbound connections (default to 100). When it is reached, the node will still accept inbound connections, but they will be closed right after responding to the first ping. This is a soft limit to allow nodes to join the cluster even if the trusted nodes inbound limit has been reached. In addition, this prevents a node, in particular a trusted node, to have so much inbound connections that it cannot reach its number of outbound connections; a guaranteed number of outbound connection is crucial for proper propagation of new blocks. For all nodes to reach their established number of outbound connections, the soft maximum limit for inbound connections must be smaller than the maximum number of nodes minus the wanted number of outgoing connections.

The node will first burst-connect to the trusted peers and then periodically connects to more peers until it reaches the maximum number of outbound connections; the delay between connections to new peers should be small enough so the nodes has enough outbound connections to work properly, but large enough so it has enough peers to choose from. This is done by having a variable delay between connections of 2^(OUTBOUND_CONNECTION_COUNT - 1) seconds when OUTBOUND_CONNECTION_COUNT is greater than 0 with a maximum of 30 seconds; this allow the node to reach 5 outbound connection under 15 seconds and connect to the last peer when having received around 20 gossip messages.

The node iteratively picks a random peer (not yet connected) from either the verified pool or the unverified pool (1.0 probability by default, so it always tries to pick a peer from the verified pool first) that is from a different group (See Peer Groups) than any actual outbound connection. In case there are no more peers available in the selected pool the other one will be tried. If the connection succeeds, an unverified peer is upgraded to the verified pool and removed from the unverified pool. If the connection fails, the peer retry counter and last retry time are updated.

The node will periodically check a random peer from the unverified pool without sending a ping message, only establishing a TCP connection. This ensures the verified pool contains enough reachable peers to reduce the chance of a Sybil attack where most of the good peers are unreachable augmenting the probability of only hostile node getting selected.

If a peer changes its IP address but not its key, the new address will never be updated through gossip. This is to prevent a hostile node from gossiping bad addresses for known good nodes and making them unreachable. The peer will be removed after the normal retry policy is exhausted; then the new address will be added through the usual gossip exchanges.

The peer retry counter and last retry time (initialized to '0' and 'infinity'), are used to filter out peers when picking them from the pools providing exponential backoff. After N failed attempts, a verified peer that is not marked as trusted is downgraded to the unverified pool and the counter and time is reset; unverified peers are simply removed.

Connected peers will periodically be pinged, and their connection state monitored. The ping interval is configurable and defaults to once every 120 seconds. If a ping fails, this is logged and the peer is scheduled for another ping in the normal interval.

Connection Cleanup

Connections are monitored and cleaned when inactive to prevent an attacker from isolating the node by blocking chain traffic. If there is actually no traffic, it will just accelerate the connection rotation.

All connections without any chain-related activity (not counting gossip) for more than 180 seconds will get disconnected. (Not yet implemented)
All inbound connections that don't send a ping after 30 seconds will be disconnected.
All recent inbound connections (less than 90 seconds old) without any chain-related activity (not counting gossip) will be closed. (Not yet implemented)
When the maximum number of outbound connections has been reached, a random peer is checked every minute (only Noise handcheck).

Unverified Peers

The unverified pool is composed of 1024 buckets of up to 64 peers, resulting in a maximum capacity of 65536 peers.

To prevent a rogue node to fill it with compromised peers, it uses the address group of the node gossiping the peers to select a subset of the buckets; then the added peer address group is used to select a smaller subset and then the rest of the address is finally used to select the bucket it belongs to. A secret known only by the node is used to randomize the selection process so it cannot be predicted by the attacker.

If a peer is not already in the verified pool, the steps to add it to the unverified pool are:

If there is already 8 references of the peer in the pool, no more references are added; if there is N references, a random probability check of '1/2^N' is performed to decide if another reference to the peer should be added.
A subset of 64 buckets are selected based on the secret and the group of the node that gossiped the peer (IP '/16' mask); this limits the part of the pool that can be changed by any given rogue node IP.
From this subset, 4 buckets are selected based on the secret and the IP of the peer to be added; this randomizes the peer distribution preventing the prediction of the way peers will be evicted.
From these 4 buckets, a single one is selected randomly; this reduces the collisions between peers that share the same IP.
If the bucket already contains a reference to the peer, it is not added again.
If the bucket the peer has to be added to is full, one existing peer has to be evicted. This is done by first cleaning all the peers that weren't gossiped for a certain amount of time, then if the bucket is still full, by selecting a random peer with a bias favoring peers that were added the longest time ago.
The eventually evicted peer is completely removed from the pool.

Only the IPs of the peers are used to select a subset of the pool because IP is an expensive resource while ports are comparatively cheap.

See Bucket Selection for more details on how the buckets are selected from the secret and other discriminators.

Verified Peers

The verified pool is composed of 256 buckets of up to 32 peers, resulting in a maximum capacity of 8192 peers.

To prevent attackers from predicting how good peers are evicted and replace them by compromised peers, the eviction is done per-bucket; the buckets are selected from the peer address and a secret to randomize the selection process.

When a peer is verified the first time by connecting to it using the Noise protocol, it is removed from the unverified pool and the steps to add it to the verified pool are:

A subset of 8 buckets are selected based on the secret and the address group of the peer to be added.
From this subset, a single bucket is selected based on the secret and the rest of the IP of the peer to be added.
If the bucket is full, one peer has to be evicted. This is done by first cleaning the peers that weren't gossiped for a long time, and if the bucket is still full, by selecting a random one with a bias toward the ones that are not connected and the last connection was the longest time ago. Trusted peers and connected peers are never evicted.
The eventually evicted peer is added to the unverified pool; its retry counter and last retry time are reset.

See Bucket Selection for more details on how the buckets are selected from the secret and other discriminators.

Technical Details

Constants

List of constants with their current default values:

Maximum number of outbound connections: 10
Soft maximum number of inbound connections: 100
Number of buckets in the verified pool: 2^8 (256)
Number of peer per verified buckets: 2^5 (32)
Number of buckets in the unverified pool: 2^10 (1024)
Number of peer per unverified buckets: 2^6 (64)
Maximum number of duplicated peers in the unverified pool: 8
Probability of adding a Nth duplicated reference of an existing peer to the unverified pool: 1/2^N
Period of verified peer random peer check when the maximum number of verified connections has been reached: 60 seconds
Period of new peer connection up to the maximum number of verified connections: min(30, 2^(OUTBOUND_CONNECTION_COUNT - 1)) seconds.
Gossip ping frequency: 120 seconds
Maximum time to get a ping from an inbound connection: 30 seconds
Maximum time without activity (besides gossip) for inbound connections less than 90 seconds old: 30 seconds
Maximum time without activity (besides gossip) for all connections: 180 seconds
Number of unverified buckets selected based on the source address group: 64
Number of unverified sub-buckets selected based on the peer address group: 4
Number of verified buckets selected based on peer address group: 8

With these default values:

The maximum cumulated number of distinct peers in the pools is from 16384 to 73728 depending on peer duplication in the unverified pool.
The maximum number of peers that can be added through gossip by peers sharing the same address group is from 512 to 4096 depending on peer duplication in the unverified pool.
The maximum number of neighboring peers sharing the same IP a single node can add through gossip is from 23 to 184 depending on peer duplication in the unverified pool.
The minimum time to reach 10 outbound connections is 2 minutes and 31 seconds.

Bucket Selection

The bucket selection is done by hashing the secret with the discriminator and use the result as an integer; this integer modulo is used to restrict the subset.

For the unverified bucket selection:

<<N1:160>> = crypto:hash(sha, <<Secret/binary, PeerGroup/binary>>).
<<N2:160>> = crypto:hash(sha, <<Secret/binary, PeerAddress/binary>>).
<<N3:160>> = crypto:hash(sha, <<Secret/binary, SourceGroup/binary, (N1 rem 16):8, (N2 rem 4):8>>).
BucketIdx = N3 rem 1024.

For the verified bucket selection:

<<N1:160>> = crypto:hash(sha, <<Secret/binary, PeerAddress/binary>>).
<<N2:160>> = crypto:hash(sha, <<Secret/binary, PeerGroup/binary, (N1 rem 8):8>>).
BucketIdx = N2 rem 256.

Persistence

(Not yet implemented)

Persistence of the known peers is important in preventing Sybil/eclipse attacks.

If the node rebuild its list of peers from scratch every time it is restarted, an attacker could use any other attack vector to crash the node or wait for a scheduled restart and be the first to spam it with compromised addresses.

To support persistence, both verified and unverified pools are dumped to disk periodically and loaded on startup.