Operations

What a running twin survives, what it loses, and how to read the timing data the API returns. This page is the short version. The full treatment lives in the repository's failure model and operation timelines documents.

What survives a failure

The cluster is the source of truth for the topology. The backend keeps only what the cluster cannot hold, in a single local SQLite file plus a files volume, and nothing runs between operations.

State	Lives in	Backend restart	Backend volume lost
Topology (nodes, links, CRD-declared addresses)	Kubernetes: StatefulSets, Meshnet CRDs, ConfigMaps, Secrets	kept	kept
Configuration applied to nodes	the pods, plus one history row per action	kept	pods keep it, history gone
Operation history (what replay uses)	SQLite	kept	lost
Node positions, sensitive file flags	SQLite	kept	lost
File manager files	files volume	kept	lost, the mounted ConfigMaps stay
Admin password, sessions, API tokens	SQLite	kept	password comes back from the environment, tokens must be recreated

In practice:

• Backend restart or replacement. Nothing is lost. Stale operation locks are wiped at startup and the backend picks up whatever is deployed from the cluster. A replacement on another machine needs only the kubeconfig.
• Interrupted operation. A deploy that dies half way leaves its resources behind, and the next deploy is refused with 409. Clear the topology and repeat it. A modify or restart that dies mid way is repaired by the heal pass of the next modify, or by a Restart of the affected pod.
• Pod recreated by KubeNDT (the Restart action, a modify that restarts a neighbour, the heal pass). Handled: its history is replayed and the neighbours' operations on the recreated interfaces are re-applied.
• Pod recreated behind KubeNDT's back (kubectl delete pod, an eviction, a worker going down). Not handled, by design: nothing watches the cluster between operations. The pod comes back with the CRD-declared addresses only. One Restart of that pod, through the API or the dashboard, runs the full recovery.
• Meshnet not running. Deploys and modifies are refused with 412, see the Meshnet dependency.

What stays manual: one Restart per externally recreated pod, anything applied inside a pod by hand, the internal state of the workloads themselves (routing tables learnt by protocols, sessions, database contents), and backups of the two volumes. KubeNDT does not replicate its database.

Operation timelines

Deploy, modify and restart responses carry a timeline block next to the human-readable took_time. It answers one question: of the time an operation took, how much was KubeNDT and how much was Kubernetes and the node images underneath.

"timeline": {
  "request_started_at": "2026-09-20T12:47:47.120Z",
  "backend_ms": {"prepare": 1180, "wait_ready": 5300, "replay": 40, "peer_replay": 1800, "total": 6760},
  "pods": [
    {
      "pod": "r1-0",
      "kubernetes": {
        "created": "2026-09-20T12:47:51Z",
        "scheduled": "2026-09-20T12:47:51Z",
        "sandbox_ready": "2026-09-20T12:47:52Z",
        "container_started": "2026-09-20T12:47:52Z",
        "ready": "2026-09-20T12:47:53Z"
      },
      "observed_ms": {"delete_issued": 1180, "old_pod_gone": 4210, "ready_seen": 6480}
    }
  ]
}

Two clocks, two resolutions:

• kubernetes.* are copied from the Pod object exactly as Kubernetes wrote them, at 1 s resolution. They are the same values kubectl get pod -o yaml shows: the backend does not time these phases, it reports what Kubernetes recorded. sandbox_ready needs Kubernetes 1.29 or newer.
• observed_ms are milliseconds since request_started_at on the backend clock, marking when its pod watch saw each transition. A step that had already happened when the watch started is omitted rather than stamped with a wrong time.

backend_ms lists the backend's own phases, and which ones appear depends on the operation: validation and resource_creation on a deploy, prepare on a modify or restart, then wait_ready, replay (the recreated pods' own history), peer_replay (the re-application on their neighbours), heal and total.

Pods progress in parallel, so total is a critical path, not the sum of the per-pod phases, and the backend phases do not add up to it either. Treat the block as timestamps to reason about, not as a stacked bar to sum. Ready also means something different per node type: the image up for a plain container, the guest's HTTP API answering for VyOS, its daemons answering for FRR, ovsdb-server up for Open vSwitch. Compare readiness figures with that in mind.