Health probes

Two probes, served on both planes, answering two different questions. Pointing an orchestrator at the wrong one is the most expensive mistake available here, so they are kept apart on purpose.

/healthz is liveness

It answers UP unconditionally, with the version.

That is the correct answer rather than a lazy one. Liveness decides whether to kill the process. A liveness probe that failed because the database was unreachable would turn a database blip into a restart of every node at once - each one killed for a fault none of them has, and none of them repairs by dying. What belongs to a dependency belongs to readiness.

{"status":"UP","version":"dev"}

It is also the startup probe. The store is opened and migrated before the ports open, so a gateway that is migrating answers nothing yet: the startup probe gives it up to two minutes before liveness starts counting. Without it, a long migration after an upgrade would be killed halfway and start over, in a loop.

/readyz is readiness

It decides whether to send traffic, so it asks the two questions that make a node useless:

• does the store answer? A ping, bounded to two seconds. A readiness check that hangs is a readiness check that says nothing, and the orchestrator's own timeout then decides on no information at all.
• has the router compiled its table at least once? A node that has accepted connections but never compiled answers 404 to everything, which a rolling update happily reads as a healthy instance and feeds live traffic.

On failure it is a 503 with the reason, because an operator reading a probe failure needs to know which of the two it was.

{"status":"DOWN","reason":"the store is not answering","version":"dev"}

{"status":"DOWN","reason":"the routing table has not been compiled yet","version":"dev"}

A node whose database has gone still holds its compiled table and still answers requests, while it can no longer resolve a session, read a setting or reload a route. That is exactly the state that used to declare itself ready.

Both escape the HTTPS redirect

Where the plain port redirects to HTTPS, these two paths are exempt: a 308 reads as "not ready" to most probes, and a gateway would be taken out of rotation for being correctly configured.

From inside the container

The image has no shell and no wget (it is distroless): a Docker, compose or Swarm healthcheck asks the gateway's own binary, which exits 0 on a 2xx and 1 otherwise.

healthcheck:
  test: ["CMD", "/meerkat", "health", "http://127.0.0.1:8080/readyz"]

Without an address it probes http://127.0.0.1:8080/readyz. The Helm chart's Kubernetes probes are the same command, run with exec: /meerkat health http://127.0.0.1:8080/healthz for startup and liveness, /readyz for readiness. An httpGet would come from the kubelet, outside the container, and a plain port switched off on Infra > Domains & TLS listens on the loopback only: the pod would stop being ready for a setting that works (see Switching the plain port off).

In a cluster

The load balancer probes /readyz and never /healthz. There is no session affinity to ask for - sessions live in the database, and the five-second cache is invalidated by the change bus - and no primary node to aim at, since the advisory lock guards whatever must happen only once.