# Network Monitoring

This document describes how to configure network monitoring within the node provided by `aemon` application.

## Configuration

By default monitoring is disabled. To turn it on `monitoring.publisher.pubkey` and `monitoring.active = true` need to be setup. To start posting transaction setup `monitoring.publisher.autostart` to `true`.

To post transactions account's public and private key is needed. For online processing only public key needs to be configured. Offline post-processing can be done by filtering transaction by publisher public key.

## Metrics configuration

Monitoring uses statsd backend provided by `apps/aecore/src/aec_metrics.erl`. See metrics `metrics.*` configuration keys in `apps/aeutils/priv/aeternity_config_schema.json`

## Metrics

Each metric uses `ae.epoch.aemon.` prefix.

| Name                               | Type      | Description                                               |
| ---------------------------------- | --------- | --------------------------------------------------------- |
| `confirmation.delay`               | histogram | Number of keyblock created before signing transaction     |
| `forks.micro.count`                | counter   | Count of observed micro-forks                             |
| `forks.micro.height`               | histogram | Height difference of observed micro-forks                 |
| `gen_stats.microblocks.total`      | histogram | Number of microblocks in a generation                     |
| `gen_stats.tx.monitoring`          | histogram | Number of monitoring transactions in a generation         |
| `gen_stats.tx.total`               | histogram | Number of transactions in a generation                    |
| `publisher.balance`                | gauge     | Publisher balance                                         |
| `publisher.post_tx.max_adjustment` | counter   | Transaction posting error:                                |
| `publisher.post_tx.nonce_too_high` | counter   | Transaction posting error:                                |
| `publisher.post_tx.nonce_too_low`  | counter   | Transaction posting error:                                |
| `publisher.post_tx.success`        | counter   | Successful transaction posts                              |
| `publisher.queue.size`             | histogram | Number of transactions posted but not signed on chain     |
| `publisher.queue.ttl_expired`      | histogram | Number of transactions with expired ttl                   |
| `block.propagation_time.key`       | histogram | Time until key-blocks reached this node in milliseconds   |
| `block.propagation_time.micro`     | histogram | Time until micro-blocks reached this node in milliseconds |
| `block.time_since_prev.key`        | histogram | Time between key-blocks in milliseconds                   |
| `block.time_since_prev.micro`      | histogram | Time between micro-blocks in milliseconds                 |
| `block.tx.total.micro`             | histogram | Number of transactions in a microblock                    |
| `block.gas.total.micro`            | histogram | Gas used per microblock                                   |
| `block.gas.per_tx.micro`           | histogram | Gas used per transaction in a microblock                  |
| `block.size.per_tx.micro`          | histogram | Size of transactions in a microblock in bytes             |
| `chain.top.difficulty`             | gauge     | Difficulty of the top block                               |

## How to read metrics

#### `confirmation.delay`

represents network latency. A high number might imply a busy network or unfair leaders.

#### `forks.micro` & `gen_stats.microblocks.total`

`forks.micro` represents the length of microfork. It shows a minimum number observed by monitoring, not an exact one. Based on behaviour observed in mainnet in the first half of 2019, roughly 33% of microblocks transactions are rewritten to the next generation. Use `gen_stats.microblocks.total` as a reference.

#### `gen_stats.microblocks.total` and `gen_stats.tx.{monitoring,total}`

Statistic metrics can be used to measure network saturation

#### `publisher.post_tx.*`

Metrics can be used to monitor mempool's transaction propagation. When `publisher.post_tx.nonce_too_high` is preset you might want to check `mempool.nonce_offset` configuration/

#### `publisher.queue.*`

For further network transaction propagation investigation. All transactions accepted by mempool are tracked by `*.size`. Over time it should correlate with `gen_stats.tx.monitoring`.

`*.ttl_expired` might imply low transaction fee or busy network.