Troubleshooting

This page helps operators find the cause of a ciphernode problem from what they see. Each section starts with the symptom, explains what it means, and gives the checks in order. At the end of a section, you know if the node needs an action from you or if it recovers without one.

Symptoms

Read the logs

The CLI sets the log level with flags. See Global options.

By default the node logs libp2p gossipsub only at ERROR, because its routine warnings can fill gigabytes during one E3. RUST_LOG directives apply on top of the flag level and change only the targets they name. For example, this also shows gossipsub warnings:

RUST_LOG=libp2p_gossipsub=warn interfold start -v

The node writes readable log lines to standard output. Your supervisor collects them, for example the terminal, journalctl, or docker logs. The Docker image starts the node with -v.

The node also writes each log entry as JSON to ciphernode.jsonl in the node data directory, the directory that contains db. The file rotates at 25 MiB and keeps four old files, ciphernode.jsonl.1 to ciphernode.jsonl.4. The Logs view of the local dashboard shows the same entries.

Not selected for committees

Symptom: the node logs This node was NOT selected for sortition for each E3.

This line means that the node has no ticket to submit for the E3. The node is not eligible, all its tickets are in use, or its bond owner is not ranked high enough. The last cause is normal: the committee has a fixed number of seats.

Node not in finalized committee is a different case. The node submitted a ticket, but the best tickets of other bond owners took the seats. This result is normal.

Ticket submission fails

Symptom: the node logs This node was SELECTED for sortition, and then Failed to submit ticket: … at the ERROR level.

The message includes the decoded contract error.

At deployment, sortitionSubmissionWindow was 600 s on mainnet and 180 s on Sepolia. The operator wallet must have ETH for gas. A slow or rate-limited RPC endpoint delays the transaction. See RPC endpoints.

When the ticket is already on-chain, the node logs Ticket already recorded on chain; skipping submission. No action is necessary.

DHT put record fails

Symptom: the node logs these lines during a DKG:

ERROR DHT put record failed: the quorum failed; needed 1 peers
ERROR Operation failed after 2 attempts: DHT put record failed: PutRecordError(QuorumFailed { key: …, success: [], quorum: 1 })

Before the second line, the node also logs Attempt 1/2 failed, retrying in 1000ms: … at the WARN level.

What the error means

During a DKG, the node publishes each DKG document that it makes: its threshold shares, its encryption key, and its decryption key share. It stores the document in its own DHT store and gossips a notification to the committee. Separately, it uploads the document to the Kademlia peers closest to the document's key. The upload uses Quorum::One, so at least one remote peer must acknowledge it. These errors come from the upload; a failing upload does not hold back the notification.

An inbound DHT put cannot replace the node's own published record. For a stored replica, the node keeps the later expiry. A record without an expiry keeps its unlimited lifetime.

• The retry sequence. Each upload makes 2 attempts, 1 s apart.
• success: []. Remote peers receive records with record filtering on (StoreInserts::FilterBoth). With this setting, a remote peer sends a response even when its filter rejects the record. An empty success list therefore means that no peer acknowledged the request at all.
• QuorumFailed, not Timeout. The Kademlia query timeout is 60 s. A QuorumFailed result before the timeout means that the query had no reachable peers to send the record to.

The node therefore had no admitted and reachable Kademlia peers at that time.

What happens next

The node keeps announcing the document from its own store: again 30 s after the first notification, then with the wait doubling up to 5 minutes. It retries a failed upload 15 s later, also doubling the wait up to 5 minutes, and uploads each document again every 30 minutes. It continues until the document expires or the DKG ends. The DKG ends when the chain reports the E3 stage KeyPublished, CiphertextReady, Complete, or Failed. The node then ends a put that is uploading its record. Kademlia reports that end as a quorum failure, and the node logs it at the DEBUG level as DHT put record cancelled, not as this error. A put that still looks up its closest peers runs on, and requests already handed to a connection still go out, so such a put can still log this error.

• While the uploads fail, a peer whose DHT lookup reaches this node fetches the document from it. A lookup asks the about 20 peers closest to the key, so in a larger network some peers can fetch the document only after an upload succeeds.
• If the connectivity comes back in time, an upload succeeds and the node recovers without action.
• If the connectivity does not come back in time, the peers that cannot reach this node do not receive its documents for that E3.

Causes

1. No connectivity. A firewall or NAT blocks the QUIC port, or the configured peers are down. After 3 failed dials in a row, the node logs Evicting unreachable peer <peer> after 3 consecutive failures. The node then keeps that peer out of its Kademlia routing table for 30 minutes.
2. A network identity mismatch. The Identify protocol string is interfold/<network_id>/protocol/<protocol_version>/deployments/<hash>/1. The hash is a SHA-256 over the chain ID and the Interfold contract address of every enabled chain. A different protocol_version, one more enabled chain, or a different interfold address changes the string. Every peer then rejects the node, and the node logs Rejected an incompatible Interfold peer with the reason.
3. Short churn. Peers restarted at the same time. The errors stop after the peers reconnect.

Checks

curl -s http://127.0.0.1:<port>/api/snapshot | jq '.network'

No connected peers

Symptom: the Overview of the local dashboard shows 0 connected peers, or the log shows peer errors.

The node uses QUIC over UDP. The node admits a peer only after the Identify check passes. The check compares the network identity and the supported Kademlia and sync protocols.

Identify does not cache peer addresses. The node keeps admitted peer addresses in Kademlia and filters loopback addresses when it has a listener on a non-loopback interface. Each compatible Identify exchange refreshes these addresses, so a connected peer can advertise a new address for later reconnection. The node keeps unique filtered addresses within both limits: 8 addresses per peer and 2 KiB of encoded addresses, including peer IDs. Advertised live endpoints take slots first. The remaining slots fill in advertised order. Each update removes superseded Identify addresses. Live connection endpoints remain until they close. A withdrawn endpoint is removed after its last connection closes.

The dashboard API shows configured_peers, connected_peers, and gossip_subscribed_peers. A peer that is connected but not subscribed to the gossip topic cannot take part in committee work. For the full check sequence, follow the DHT checks.

The node does not follow the chain

Symptom: E3 requests appear on-chain, but the node logs nothing about them.

Startup fails

Out of memory

Symptom: the operating system kills the node or its prover processes during a DKG. The kernel log shows an out-of-memory kill.

DKG proofs use much memory. multithread_concurrent_jobs sets the number of proof and TrBFV jobs that run at the same time. The default is 2, and the node can use fewer jobs when the host or container memory is small. multithread_reserve_threads (default 2) keeps CPU threads free for the network and the RPC connection. At startup, the node logs the values in a line that starts with Ciphernode multithread:.

To reduce the peak memory, set a lower multithread_concurrent_jobs under node, or add memory. See Compute and startup limits and Requirements.

Still stuck

• Read Tickets and sortition for the selection parameters.
• Ask in the Interfold technical chat (opens in a new tab) or the community Telegram group (opens in a new tab).
• If you suspect a bug, open an issue on GitHub (opens in a new tab). Include the release, the log lines, and the output of interfold node validate.

Next steps