Skip to content

[DBSP] New Recursion API Proposal - #6727

Open
lstwn wants to merge 1 commit into
feldera:mainfrom
lstwn:new-recursion-api-proposal
Open

[DBSP] New Recursion API Proposal#6727
lstwn wants to merge 1 commit into
feldera:mainfrom
lstwn:new-recursion-api-proposal

Conversation

@lstwn

@lstwn lstwn commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

This PR is an idea for #6666 and a follow-up to #6653 (it reuses the benchmarks created in there). I like the proposed API for especially these reasons:

  1. It subsumes both ChildCircuit::recursive and ChildCircuit::recursive_dynamic under one API ChildCircuit::recursion by offering a user-supplied init closure which sets up the shape of and the recursive streams themselves. No need to provide an extra arity which must agree with the length of the Vec returned from the computation closure passed to recursive_dynamic. Required traits are implemented for a single stream, tuples of streams, and a Vec<Stream> (just like before).
  2. It uses the builder pattern to optionally disable distinct which is a real performance gain (see transitive_closure_acyclic benchmark below).
  3. It uses the builder pattern to optionally enforce a recursion depth limit (bound). Upon hitting the bound, the recursive computation is short-circuited and a partial result is returned. Otherwise, the fixed point computation runs as without the bound.
  4. It can optionally report back a RecursionReport which makes the information of how many iterations the recursion took as well as if the bound (3) was sharp available to users. I believe this is important to know!

The changes are covered by documentation as well as unit tests, and no performance regressions happened, as the criterion-backed benchmarks confirm. Most importantly, users can easily omit a distinct from a recursive computation without having to fiddle with more complicated, lower-level APIs, and enjoy a speedup of ~1.6 for such computations.

Open questions if you want to proceed with this approach:

  1. Shall we flag the ChildCircuit::recursive and ChildCircuit::recursive_dynamic APIs as deprecated?
  2. As mark_distinct does not work with sharding (see conversation in Implement and document thread-safe use of the iterate API #6653), shall it be removed from the public API?
  3. The RecursionBuilder::run method may find a better home in the Circuit trait and could be made more compositional. Maybe the builder pattern could be used there to create custom iterate_* methods to selectively enable or disable the following behavior:
iterate_ Effect
_with_bound Adds an iteration cap to the termination check
_with_report. Adds RecursionReport stream
_with_consensus Adds a Consensus over the termination check
_until_fixedpoint Add the child.check_fixedpoint to the termination check

This is just an idea and some food for thought. For now, I'm quite happy with what is already in here.

  1. Naming and docs. This work is the result of some extensive AI back-and-forth and manual adjustments. If you like this approach and want to proceed with it, I'll do a detailed line-by-line review again.

Benches

Performance is on-par with the manual circuit without distinct but faster than the old API:
transitive_closure_acyclic

Performance is on-par with the old API and the manual circuit with distinct required:
graph_coloring

Performance is on-par with the old API and the manual circuit after aggregation:
transitive_closure_cyclic

@lstwn
lstwn force-pushed the new-recursion-api-proposal branch from 576a879 to af50650 Compare July 25, 2026 11:33

@mythical-fred mythical-fred left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a genuinely nicer API than what it subsumes. The arity-from-shape trick (recursive_var calls in init determine the arity) removes the footgun that recursive_dynamic has today, the builder chain is honest about which knobs are optional, and the type-state on Reporting vs NoReport keeps the return shape of finish unambiguous. The reachability tests exercise the three shapes (single, Vec, tuple) against the same fixture as the old APIs, and the bound + reset regression test (with_bound_counter_resets_across_transactions) is the right shape. Nice work.

I want to answer the four open questions in the PR body up front, and leave the rest inline.

1. Deprecate recursive and recursive_dynamic? Yes, once this lands. Attach #[deprecated(note = "use ChildCircuit::recursion")] and let them coexist for one release so downstream callers can migrate, then remove. Two APIs for the same thing is worse than either alone; having both indefinitely will train people to reach for whichever they see first.

2. Remove mark_distinct from the public API? Yes. A public method that silently misbehaves under sharding is a correctness trap — future users will find it, use it, hit the wrong answer in production, and spend a day tracing it. Either fix the sharding interaction or remove the entry point (deprecate + private module). Do not leave broken API in the public surface.

3. Move run to the Circuit trait / offer builder-composed iterate_* variants? Not in this PR. The current shape is coherent and covers the cases users actually have. The four-knob matrix (_with_bound, _with_report, _with_consensus, _until_fixedpoint) is a classic second-system trap — you would be designing for hypothetical users of iterate who don't yet exist. Ship this as-is; wait for a real second use case before generalising. If it turns out you want the general form, it's an additive extension.

4. Naming and docs. Overall solid — the doc examples are well-chosen and the RecursionBuilder rustdoc explains the mental model. A few specific things below. The one non-cosmetic worry is the # Safety heading on Consensus: rustdoc reserves that section for unsafe items, and Consensus is safe Rust with logical invariants. Rename to # Invariants or fold into prose.

None of these block; posting as COMMENT because the PR is explicitly an RFC and I want the four answers on the record. Once the naming/marker questions are settled I'll do a proper line-by-line pass.

One meta point: the commit history is going to need a git rebase -i before this merges — the "Prototype…" tip on top of #6653's history means the same "Reset iterate_* APIs…" commit needs to survive the squash cleanly. Standard hygiene, same as last time.

Comment thread crates/dbsp/src/circuit/runtime.rs
Comment thread crates/dbsp/src/operator/recursive.rs Outdated
Comment thread crates/dbsp/src/operator/recursive.rs Outdated
Comment thread crates/dbsp/src/operator/recursive.rs Outdated
Comment thread crates/dbsp/src/operator/recursive.rs Outdated
Comment thread crates/dbsp/src/operator/recursive.rs
Comment thread crates/dbsp/src/circuit/circuit_builder.rs Outdated
Comment thread crates/dbsp/src/operator/recursive.rs Outdated
@lstwn
lstwn force-pushed the new-recursion-api-proposal branch from af50650 to e12f858 Compare July 27, 2026 09:33
@lstwn
lstwn force-pushed the new-recursion-api-proposal branch from e12f858 to 87449dc Compare July 27, 2026 13:26
@ryzhyk

ryzhyk commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

The other PR has merged. @lstwn , please rebase and let me know when you think this is ready for review.

@lstwn
lstwn force-pushed the new-recursion-api-proposal branch from 87449dc to a3e1e8c Compare July 30, 2026 15:19
@lstwn

lstwn commented Jul 30, 2026

Copy link
Copy Markdown
Contributor Author

The other PR has merged. @lstwn , please rebase and let me know when you think this is ready for review.

Thanks. I've rebased and addressed the comments from the AI, except for any deprecation notices, as I leave that up to you to decide. Otherwise, it's ready for a review!

Comment thread crates/dbsp/src/operator/recursive.rs Outdated
/// mutually-recursive-streams-of-different-types case handled by
/// [`recursive`](ChildCircuit::recursive) over a tuple.
#[allow(clippy::unused_unit)]
#[impl_for_tuples(2, 14)]

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My thinking here was that it's stupid to have a single element tuple because there is the single stream impl. But maybe a one element tuple should be allowed here as well in case you want to use this API in Feldera's SQL-to-DBSP compiler?!

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't like special cases.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The impl_for_tuples is now for any tuple of size 1-14. Only the empty tuple is not allowed but that doesn't make any sense and would break the report() implementation, which needs at least one element to generate the report stream.

@mythical-fred mythical-fred left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rebase + AI-feedback pass looks good. Walking my earlier eight nits against the current tip:

  1. Consensus # Safety heading — resolved upstream via #6653 (now merged in through rebase). ✓
  2. RecursionReport::converged() -> Option<u64> naming — split into converged() -> bool, truncated() -> bool, and converged_iterations() -> Option<u64>. Reads correctly at call sites now. ✓
  3. RecursiveVar::stream() unused pub method — dropped. ✓
  4. Reporting slot allocated under NoReport — genuinely fixed by pushing the recorder into the ReportMode type-state: NoReport::record is a no-op, NoReport::build_report returns None, and only Reporting owns the Rc<Cell<RecursionReport>>. Zero-alloc in the default path. ✓
  5. with_bound(0) corner — solved at the type level with NonZeroU64, which is nicer than any runtime assertion. ✓
  6. Tuple report is_none() walk — replaced with a direct call on the first element. ✓
  7. Counter-reset comment — now cites the regression test by name (with_bound_counter_resets_across_transactions) and adds a debug_assert! on the stop invariant. ✓ (tiny nit below)
  8. is_fixedpoint vs check_fixedpoint — resolved via #6653. ✓
  9. Sealed markers — NoReport/Reporting are now unconstructible outside the module (private fields) and gated by a sealed::Sealed supertrait on ReportMode. Exactly right. ✓

Deferred item — deprecation of recursive/recursive_dynamic — is explicitly punted to the maintainers, which is fine; that was a soft direction question, not a blocker.

One micro-nit, non-blocking:

  • debug_assert!(stop == true); — clippy's bool_comparison will grumble; prefer debug_assert!(stop); (or debug_assert!(stop, "iteration.set(0) must only fire on the last epoch iteration") to preserve intent).

Otherwise the design and implementation are in good shape. Nice cleanup.


Reviewed by mythical-fred (an AI persona) at tip a3e1e8c. My reviews are advisory input for humans, not authoritative approval.

…ound) and an optional distinct along with some optional reporting of convergence and number of iterations of the recursion

Signed-off-by: Leo Stewen <lstwn@mailbox.org>
@lstwn
lstwn force-pushed the new-recursion-api-proposal branch from a3e1e8c to 049821e Compare July 31, 2026 08:51

@mythical-fred mythical-fred left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — the three new benchmarks exercise RecursionBuilder across all three shapes (single var in acyclic, tuple in graph coloring, single var + Min in cyclic), assert cross-variant agreement against the existing recursive/iterate implementations, and the debug_assert!(stop) clippy nit is fixed.

@ryzhyk

ryzhyk commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

The other PR has merged. @lstwn , please rebase and let me know when you think this is ready for review.

Thanks. I've rebased and addressed the comments from the AI, except for any deprecation notices, as I leave that up to you to decide. Otherwise, it's ready for a review!

Apologies about the delay. I need to clean my plate a bit so I can review this carefully. Recursion is subtle.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants