Skip to content

[adapters] checkpoint-suspend on SIGTERM - #6482

Open
swanandx wants to merge 2 commits into
mainfrom
checkpoint-on-sigterm
Open

[adapters] checkpoint-suspend on SIGTERM#6482
swanandx wants to merge 2 commits into
mainfrom
checkpoint-on-sigterm

Conversation

@swanandx

@swanandx swanandx commented Jun 16, 2026

Copy link
Copy Markdown
Member

Infra-initiated termination (node drain, eviction) doesn't involve the runner, so the pipeline does not stop gracefully.

To solve this, we add an env var FELDERA_CLEAN_SHUTDOWN_ON_SIGTERM: when set, SIGTERM runs the same checkpoint-and-suspend as /suspend before stopping.

disabled by default.

tested with local runner & FELDERA_CLEAN_SHUTDOWN_ON_SIGTERM=1. Started pipeline and then kill -TERM <pid>

Breaking Changes?

not breaking change

@mythical-fred mythical-fred left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Useful feature, and the refactor of trigger_suspend looks faithful to the original. A few blocking items before approval:

  • New user-visible env var FELDERA_CLEAN_SHUTDOWN_ON_SIGTERM with no documentation. This belongs in docs.feldera.com/operations/ (Kubernetes/deployment section), together with guidance on terminationGracePeriodSeconds.
  • New signal-shutdown behavior with no tests and no "Manual testing" section in the PR description. At minimum a brief description of how this was exercised (e.g. kill -TERM on a running pipeline and observed checkpoint completing) is needed.

Non-blocking observations inline (K8s grace-period interaction, unbounded wait loop, SIGHUP/SIGQUIT no longer handled when disable_signals() is called, expect() defensiveness, macro-hygiene workaround).

Comment thread crates/adapters/src/server.rs
Comment thread crates/adapters/src/server.rs
Comment thread crates/adapters/src/server.rs Outdated
Comment thread crates/adapters/src/server.rs
Comment thread crates/adapters/src/server.rs Outdated
Infra-initiated termination (node drain, eviction) doesn't involve the
runner, so the pipeline does not stop gracefully.

To solve this, we add an env var FELDERA_CLEAN_SHUTDOWN_ON_SIGTERM: when set, SIGTERM runs the same checkpoint-and-suspend as /suspend before stopping.

disabled by default.

Signed-off-by: Swanand Mulay <73115739+swanandx@users.noreply.github.com>
@swanandx
swanandx force-pushed the checkpoint-on-sigterm branch from cbae24b to b00816c Compare June 16, 2026 07:36
Signed-off-by: feldera-bot <feldera-bot@feldera.com>

@mythical-fred mythical-fred left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVE. The rework from cbae24b7 to 12ace697 lands every soft point from my prior COMMENT review:

  • The unbounded await_suspend_complete loop is gone — suspend() now returns its spawn handle (via the new trigger_suspend() helper), and the SIGTERM path awaits that JoinHandle directly. No 50 ms poll, no missed-wakeup hedge, no Power-of-Ten rule-2 worry. This is the right shape.
  • The .expect on signal_state became an if let Some(signal_state), so the cold path is statically unreachable instead of a runtime panic.
  • The SIGHUP/SIGQUIT and terminationGracePeriodSeconds interactions are now documented in docs/operations/clean-shutdown.md, including the "infra kill is the backstop" framing for an oversized checkpoint.
  • The tokio::select! / start macro collision workaround stays as select(pin!(...), pin!(...)) but the comment explains why and the matches!(... Either::Left) reads cleanly.
  • The /suspend handler is now a thin wrapper around trigger_suspend(), so the HTTP and signal paths share one idempotent transition and one spawn site.

No tests added, which is the one open thread — signal delivery is awkward to exercise in unit tests, and the underlying suspend flow is already covered, so I won't block on it. A follow-up integration test that sends SIGTERM to a running pipeline with FELDERA_CLEAN_SHUTDOWN_ON_SIGTERM=1 and asserts the checkpoint lands would be welcome but is not a merge blocker.

Ship it.

@snkas snkas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checkpoint-then-exit-process makes more sense I think

@blp

blp commented Jun 16, 2026

Copy link
Copy Markdown
Member

Checkpointing takes an indefinite amount of time because of the need to process input until input connectors pass a barrier, so I'm not sure that suspending on SIGTERM makes sense.

@mythical-fred mythical-fred left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-APPROVE on 12ace6974c.

Two new commits since my approval at cbae24b751:

  • b00816cdf3 adds the FELDERA_CLEAN_SHUTDOWN_ON_SIGTERM opt-in plus a docs.feldera.com/docs/operations/clean-shutdown.md page covering the env-var contract, Kubernetes terminationGracePeriodSeconds interaction, and the SIGINT / other-signal semantics. The refactor extracts the suspend transition out of the /suspend handler into a synchronous trigger_suspend that returns Result<Option<JoinHandle<()>>, _>, so the signal task can .await the spawned checkpoint and the HTTP handler keeps its fire-and-forget shape. Clean separation; the lock is dropped before any .await exactly as the doc-comment promises.
  • 12ace6974c is [ci] apply automatic fixes.

The signal path is conservative in the right places: handler installation is gated behind the env var so unrelated deployments keep actix's default handling; disable_signals() only fires when the feature is on; SIGINT keeps an immediate-shutdown semantics (matches Ctrl-C expectations for local runs); the error branches log and fall back to "shut down without checkpoint" instead of getting stuck. The comment explaining why tokio::select! is avoided (collision with the start unit struct from the #[get("/start")] macro) is a nice forward-courtesy to the next reader.

Two minor follow-up thoughts, none blocking:

  • The doc page truthy set is 1 | true | yes | on (case-insensitive). If any other Feldera env var uses a different truthy spelling (e.g. accepting enabled or y), aligning them in a future cleanup would save support pain.
  • The signal task drops the signal_state clone but never explicitly logs that the suspend completed; on a healthy clean shutdown the only observable trace is the existing checkpoint logs. Not worth adding noise for, just noting it for future diagnosis.

Nice work tightening the contract with a real docs page; that's what makes a feature like this safe to ship.

Comment on lines +822 to +828
.map(|value| {
matches!(
value.trim().to_ascii_lowercase().as_str(),
"1" | "true" | "yes" | "on"
)
})
.unwrap_or(false);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I'd use .is_some_and here (but this is fine too)

.map_err(|e| ControllerError::io_error("renaming server port file", e))?;

// Own signal handling: SIGTERM (pod eviction, node drain) checkpoints
// before shutdown; SIGINT (Ctrl-C) stops immediately. The handle is

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If SIGINT stops immediately, is there value in catching it? Catching a signal unnecessarily is undesirable since there is more than can go wrong than just letting the kernel shut the process down.

matches!(select(sigterm_recv, sigint_recv).await, Either::Left(_))
};
if sigterm_fired {
info!("received SIGTERM; suspending before shutdown");

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The customary way to terminate a process after receiving a signal is to unset the signal handler then re-raise the same signal, as shown at https://sourceware.org/glibc/manual/latest/html_node/Termination-in-Handler.html. Is it possible for us to do that too?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants