[adapters] checkpoint-suspend on SIGTERM - #6482
Conversation
mythical-fred
left a comment
There was a problem hiding this comment.
Useful feature, and the refactor of trigger_suspend looks faithful to the original. A few blocking items before approval:
- New user-visible env var
FELDERA_CLEAN_SHUTDOWN_ON_SIGTERMwith no documentation. This belongs indocs.feldera.com/operations/(Kubernetes/deployment section), together with guidance onterminationGracePeriodSeconds. - New signal-shutdown behavior with no tests and no "Manual testing" section in the PR description. At minimum a brief description of how this was exercised (e.g.
kill -TERMon a running pipeline and observed checkpoint completing) is needed.
Non-blocking observations inline (K8s grace-period interaction, unbounded wait loop, SIGHUP/SIGQUIT no longer handled when disable_signals() is called, expect() defensiveness, macro-hygiene workaround).
Infra-initiated termination (node drain, eviction) doesn't involve the runner, so the pipeline does not stop gracefully. To solve this, we add an env var FELDERA_CLEAN_SHUTDOWN_ON_SIGTERM: when set, SIGTERM runs the same checkpoint-and-suspend as /suspend before stopping. disabled by default. Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
cbae24b to
b00816c
Compare
Signed-off-by: feldera-bot <feldera-bot@feldera.com>
mythical-fred
left a comment
There was a problem hiding this comment.
APPROVE. The rework from cbae24b7 to 12ace697 lands every soft point from my prior COMMENT review:
- The unbounded
await_suspend_completeloop is gone —suspend()now returns its spawn handle (via the newtrigger_suspend()helper), and the SIGTERM path awaits thatJoinHandledirectly. No 50 ms poll, no missed-wakeup hedge, no Power-of-Ten rule-2 worry. This is the right shape. - The
.expectonsignal_statebecame anif let Some(signal_state), so the cold path is statically unreachable instead of a runtime panic. - The SIGHUP/SIGQUIT and
terminationGracePeriodSecondsinteractions are now documented indocs/operations/clean-shutdown.md, including the "infra kill is the backstop" framing for an oversized checkpoint. - The
tokio::select!/startmacro collision workaround stays asselect(pin!(...), pin!(...))but the comment explains why and thematches!(... Either::Left)reads cleanly. - The
/suspendhandler is now a thin wrapper aroundtrigger_suspend(), so the HTTP and signal paths share one idempotent transition and one spawn site.
No tests added, which is the one open thread — signal delivery is awkward to exercise in unit tests, and the underlying suspend flow is already covered, so I won't block on it. A follow-up integration test that sends SIGTERM to a running pipeline with FELDERA_CLEAN_SHUTDOWN_ON_SIGTERM=1 and asserts the checkpoint lands would be welcome but is not a merge blocker.
Ship it.
snkas
left a comment
There was a problem hiding this comment.
Checkpoint-then-exit-process makes more sense I think
|
Checkpointing takes an indefinite amount of time because of the need to process input until input connectors pass a barrier, so I'm not sure that suspending on SIGTERM makes sense. |
mythical-fred
left a comment
There was a problem hiding this comment.
Re-APPROVE on 12ace6974c.
Two new commits since my approval at cbae24b751:
b00816cdf3adds theFELDERA_CLEAN_SHUTDOWN_ON_SIGTERMopt-in plus adocs.feldera.com/docs/operations/clean-shutdown.mdpage covering the env-var contract, KubernetesterminationGracePeriodSecondsinteraction, and theSIGINT/ other-signal semantics. The refactor extracts the suspend transition out of the/suspendhandler into a synchronoustrigger_suspendthat returnsResult<Option<JoinHandle<()>>, _>, so the signal task can.awaitthe spawned checkpoint and the HTTP handler keeps its fire-and-forget shape. Clean separation; the lock is dropped before any.awaitexactly as the doc-comment promises.12ace6974cis[ci] apply automatic fixes.
The signal path is conservative in the right places: handler installation is gated behind the env var so unrelated deployments keep actix's default handling; disable_signals() only fires when the feature is on; SIGINT keeps an immediate-shutdown semantics (matches Ctrl-C expectations for local runs); the error branches log and fall back to "shut down without checkpoint" instead of getting stuck. The comment explaining why tokio::select! is avoided (collision with the start unit struct from the #[get("/start")] macro) is a nice forward-courtesy to the next reader.
Two minor follow-up thoughts, none blocking:
- The doc page truthy set is
1 | true | yes | on(case-insensitive). If any other Feldera env var uses a different truthy spelling (e.g. acceptingenabledory), aligning them in a future cleanup would save support pain. - The signal task drops the
signal_stateclone but never explicitly logs that the suspend completed; on a healthy clean shutdown the only observable trace is the existing checkpoint logs. Not worth adding noise for, just noting it for future diagnosis.
Nice work tightening the contract with a real docs page; that's what makes a feature like this safe to ship.
| .map(|value| { | ||
| matches!( | ||
| value.trim().to_ascii_lowercase().as_str(), | ||
| "1" | "true" | "yes" | "on" | ||
| ) | ||
| }) | ||
| .unwrap_or(false); |
There was a problem hiding this comment.
I think I'd use .is_some_and here (but this is fine too)
| .map_err(|e| ControllerError::io_error("renaming server port file", e))?; | ||
|
|
||
| // Own signal handling: SIGTERM (pod eviction, node drain) checkpoints | ||
| // before shutdown; SIGINT (Ctrl-C) stops immediately. The handle is |
There was a problem hiding this comment.
If SIGINT stops immediately, is there value in catching it? Catching a signal unnecessarily is undesirable since there is more than can go wrong than just letting the kernel shut the process down.
| matches!(select(sigterm_recv, sigint_recv).await, Either::Left(_)) | ||
| }; | ||
| if sigterm_fired { | ||
| info!("received SIGTERM; suspending before shutdown"); |
There was a problem hiding this comment.
The customary way to terminate a process after receiving a signal is to unset the signal handler then re-raise the same signal, as shown at https://sourceware.org/glibc/manual/latest/html_node/Termination-in-Handler.html. Is it possible for us to do that too?
Infra-initiated termination (node drain, eviction) doesn't involve the runner, so the pipeline does not stop gracefully.
To solve this, we add an env var FELDERA_CLEAN_SHUTDOWN_ON_SIGTERM: when set, SIGTERM runs the same checkpoint-and-suspend as /suspend before stopping.
disabled by default.
tested with local runner &
FELDERA_CLEAN_SHUTDOWN_ON_SIGTERM=1. Started pipeline and thenkill -TERM <pid>Breaking Changes?
not breaking change