rust_compiler cleanup: newly-discovered stale artifacts never get persisted when the state file write fails #6791

Summary

The cleanup routine that's supposed to free up space in the Rust compilation working dir (pipeline_manager::compiler::rust_compiler) tracks its own progress in a state file, target_cleanup.json, sitting on the same volume it's cleaning. If the disk fills up before a cleanup cycle gets to run, the routine can't write that file anymore, so it can never start freeing anything. It deadlocks on the exact problem it exists to solve.

• Once it hit 100%, every cleanup cycle (runs roughly every 2 min) logged this:
  ERROR pipeline_manager::compiler::rust_compiler: Unable to deserialize cleanup state due to: EOF while parsing a value at line 1 column 0
  ERROR pipeline_manager::compiler::rust_compiler: Unable to write cleanup state to file due to: flushing file '/home/ubuntu/working-dir/rust-compilation/target_cleanup.json': No space left on device (os error 28)

And real pipelines started failing to compile:
ERROR pipeline_manager::compiler::sql_compiler: SQL compilation failed due to system error (program version: 1): creating directory '/home/ubuntu/working-dir/sql-compilation/pipeline-...': No space left on device (os error 28) pipeline-name=fraud-detection

Mitigation status

• Immediate fix: freed space manually on the pod to break the deadlock.
• Enabled the existing rust_compiler_full_cleanup unstable feature (auto-wipes target/ at 90% usage, no state file involved so it can't deadlock) on staging, it ran clean, no errors. Same one-line change opened for prod: https://github.com/feldera/cloud-deployments/pull/339
• This issue tracks the deeper bug below, which the feature flag works around but doesn't fix.

Root cause

The cleanup routine can only make progress by writing its state file first, but writing needs free space, which is precisely what's gone when cleanup is actually needed. Circular dependency.

The fix should be:

1. Must fix: flip the order so it deletes stale stuff first and only writes the state file after, as a best-effort step. Deleting never needs free space, only the bookkeeping write does, so a failed write shouldn't be able to block the actual cleanup.
2. Should go with #1: add a dumb, stateless fallback for when things are already critical, if usage is near 100%, just delete the oldest file(s) by mtime in target/deps or pipeline-binaries directly. No state needed there, so it physically can't deadlock the same way again.
3. Make the state file write atomic (temp file, then rename). Right now a failed write truncates it to 0 bytes and wipes out whatever good state was there before, that's why the deserialize errors kept repeating every cycle.
4. Trigger cleanup proactively once usage crosses some threshold (say 85%) instead of waiting on a fixed timer, so it has room to actually work before things get dire.