core: back off and give up on crash-looping services
Generated service units took systemd's default start limit (5 starts in 10s), which can never trip when each failed attempt outlives the window. openclaw failed ~5s in and restarted every ~5.5s, so the counter reset on every cycle: 260k restarts at ~7.4s CPU each, over a core burned continuously for days, with the unit stuck in `activating (auto-restart)` rather than anything a status check would flag. Service units now get exponential restart backoff (RestartSteps=8, RestartMaxDelaySec=300) and a start limit sized against it (10 starts per hour, in [Unit] where systemd reads it). A broken service retries with a growing delay for ~17min — wide enough to ride out a boot-time dependency race — then lands in `failed` where it's visible and stops consuming CPU. All four values are SystemdSpec fields, overridable per deployment. Timer-driven oneshot jobs are unaffected; they never restart. systemd older than 254 ignores RestartSteps/RestartMaxDelaySec with a warning and keeps the flat RestartSec, so the start limit still applies there.
This commit is contained in:
@@ -155,6 +155,19 @@ class SystemdSpec(BaseModel):
|
||||
wanted_by: list[str] = Field(default_factory=lambda: ["default.target"])
|
||||
restart: RestartPolicy = RestartPolicy.ON_FAILURE
|
||||
restart_sec: int = 2
|
||||
# Exponential restart backoff: the delay grows from ``restart_sec`` to
|
||||
# ``restart_max_delay_sec`` over ``restart_steps`` restarts, so a service that
|
||||
# can't start stops hammering the box (systemd >= 254; older systemd ignores
|
||||
# both keys with a warning and keeps the flat ``restart_sec``).
|
||||
restart_steps: int = 8
|
||||
restart_max_delay_sec: int = 300
|
||||
# Give up eventually: ``start_limit_burst`` starts inside
|
||||
# ``start_limit_interval_sec`` puts the unit in `failed` instead of looping
|
||||
# forever. Sized against the backoff above — with the defaults, ~10 restarts
|
||||
# span ~17min, long enough to ride out a boot-time dependency race and short
|
||||
# enough that a genuinely broken service surfaces as failed the same hour.
|
||||
start_limit_burst: int = 10
|
||||
start_limit_interval_sec: int = 3600
|
||||
no_new_privileges: bool = True
|
||||
readiness: ReadinessHttpGet | None = None
|
||||
exec_reload: str | None = None
|
||||
@@ -530,7 +543,9 @@ class CaddyDeployment(DeploymentBase):
|
||||
@model_validator(mode="after")
|
||||
def _validate_reach(self) -> CaddyDeployment:
|
||||
if self.reach == Reach.OFF:
|
||||
raise ValueError("a static (caddy) deployment is always served; reach must be internal|public")
|
||||
raise ValueError(
|
||||
"a static (caddy) deployment is always served; reach must be internal|public"
|
||||
)
|
||||
_validate_public_host(self.public_host, self.reach)
|
||||
return self
|
||||
|
||||
|
||||
Reference in New Issue
Block a user