[DBSP] New Recursion API Proposal - #6727
Conversation
576a879 to
af50650
Compare
mythical-fred
left a comment
There was a problem hiding this comment.
This is a genuinely nicer API than what it subsumes. The arity-from-shape trick (recursive_var calls in init determine the arity) removes the footgun that recursive_dynamic has today, the builder chain is honest about which knobs are optional, and the type-state on Reporting vs NoReport keeps the return shape of finish unambiguous. The reachability tests exercise the three shapes (single, Vec, tuple) against the same fixture as the old APIs, and the bound + reset regression test (with_bound_counter_resets_across_transactions) is the right shape. Nice work.
I want to answer the four open questions in the PR body up front, and leave the rest inline.
1. Deprecate recursive and recursive_dynamic? Yes, once this lands. Attach #[deprecated(note = "use ChildCircuit::recursion")] and let them coexist for one release so downstream callers can migrate, then remove. Two APIs for the same thing is worse than either alone; having both indefinitely will train people to reach for whichever they see first.
2. Remove mark_distinct from the public API? Yes. A public method that silently misbehaves under sharding is a correctness trap — future users will find it, use it, hit the wrong answer in production, and spend a day tracing it. Either fix the sharding interaction or remove the entry point (deprecate + private module). Do not leave broken API in the public surface.
3. Move run to the Circuit trait / offer builder-composed iterate_* variants? Not in this PR. The current shape is coherent and covers the cases users actually have. The four-knob matrix (_with_bound, _with_report, _with_consensus, _until_fixedpoint) is a classic second-system trap — you would be designing for hypothetical users of iterate who don't yet exist. Ship this as-is; wait for a real second use case before generalising. If it turns out you want the general form, it's an additive extension.
4. Naming and docs. Overall solid — the doc examples are well-chosen and the RecursionBuilder rustdoc explains the mental model. A few specific things below. The one non-cosmetic worry is the # Safety heading on Consensus: rustdoc reserves that section for unsafe items, and Consensus is safe Rust with logical invariants. Rename to # Invariants or fold into prose.
None of these block; posting as COMMENT because the PR is explicitly an RFC and I want the four answers on the record. Once the naming/marker questions are settled I'll do a proper line-by-line pass.
One meta point: the commit history is going to need a git rebase -i before this merges — the "Prototype…" tip on top of #6653's history means the same "Reset iterate_* APIs…" commit needs to survive the squash cleanly. Standard hygiene, same as last time.
af50650 to
e12f858
Compare
e12f858 to
87449dc
Compare
|
The other PR has merged. @lstwn , please rebase and let me know when you think this is ready for review. |
87449dc to
a3e1e8c
Compare
Thanks. I've rebased and addressed the comments from the AI, except for any deprecation notices, as I leave that up to you to decide. Otherwise, it's ready for a review! |
| /// mutually-recursive-streams-of-different-types case handled by | ||
| /// [`recursive`](ChildCircuit::recursive) over a tuple. | ||
| #[allow(clippy::unused_unit)] | ||
| #[impl_for_tuples(2, 14)] |
There was a problem hiding this comment.
My thinking here was that it's stupid to have a single element tuple because there is the single stream impl. But maybe a one element tuple should be allowed here as well in case you want to use this API in Feldera's SQL-to-DBSP compiler?!
There was a problem hiding this comment.
I don't like special cases.
There was a problem hiding this comment.
The impl_for_tuples is now for any tuple of size 1-14. Only the empty tuple is not allowed but that doesn't make any sense and would break the report() implementation, which needs at least one element to generate the report stream.
mythical-fred
left a comment
There was a problem hiding this comment.
Rebase + AI-feedback pass looks good. Walking my earlier eight nits against the current tip:
Consensus# Safetyheading — resolved upstream via #6653 (now merged in through rebase). ✓RecursionReport::converged() -> Option<u64>naming — split intoconverged() -> bool,truncated() -> bool, andconverged_iterations() -> Option<u64>. Reads correctly at call sites now. ✓RecursiveVar::stream()unused pub method — dropped. ✓- Reporting slot allocated under
NoReport— genuinely fixed by pushing the recorder into theReportModetype-state:NoReport::recordis a no-op,NoReport::build_reportreturnsNone, and onlyReportingowns theRc<Cell<RecursionReport>>. Zero-alloc in the default path. ✓ with_bound(0)corner — solved at the type level withNonZeroU64, which is nicer than any runtime assertion. ✓- Tuple
reportis_none()walk — replaced with a direct call on the first element. ✓ - Counter-reset comment — now cites the regression test by name (
with_bound_counter_resets_across_transactions) and adds adebug_assert!on thestopinvariant. ✓ (tiny nit below) is_fixedpointvscheck_fixedpoint— resolved via #6653. ✓- Sealed markers —
NoReport/Reportingare now unconstructible outside the module (private fields) and gated by asealed::Sealedsupertrait onReportMode. Exactly right. ✓
Deferred item — deprecation of recursive/recursive_dynamic — is explicitly punted to the maintainers, which is fine; that was a soft direction question, not a blocker.
One micro-nit, non-blocking:
debug_assert!(stop == true);— clippy'sbool_comparisonwill grumble; preferdebug_assert!(stop);(ordebug_assert!(stop, "iteration.set(0) must only fire on the last epoch iteration")to preserve intent).
Otherwise the design and implementation are in good shape. Nice cleanup.
Reviewed by mythical-fred (an AI persona) at tip a3e1e8c. My reviews are advisory input for humans, not authoritative approval.
…ound) and an optional distinct along with some optional reporting of convergence and number of iterations of the recursion Signed-off-by: Leo Stewen <lstwn@mailbox.org>
a3e1e8c to
049821e
Compare
mythical-fred
left a comment
There was a problem hiding this comment.
LGTM — the three new benchmarks exercise RecursionBuilder across all three shapes (single var in acyclic, tuple in graph coloring, single var + Min in cyclic), assert cross-variant agreement against the existing recursive/iterate implementations, and the debug_assert!(stop) clippy nit is fixed.
Apologies about the delay. I need to clean my plate a bit so I can review this carefully. Recursion is subtle. |
This PR is an idea for #6666 and a follow-up to #6653 (it reuses the benchmarks created in there). I like the proposed API for especially these reasons:
ChildCircuit::recursiveandChildCircuit::recursive_dynamicunder one APIChildCircuit::recursionby offering a user-suppliedinitclosure which sets up the shape of and the recursive streams themselves. No need to provide an extraaritywhich must agree with the length of theVecreturned from the computation closure passed torecursive_dynamic. Required traits are implemented for a single stream, tuples of streams, and aVec<Stream>(just like before).distinctwhich is a real performance gain (seetransitive_closure_acyclicbenchmark below).RecursionReportwhich makes the information of how many iterations the recursion took as well as if the bound (3) was sharp available to users. I believe this is important to know!The changes are covered by documentation as well as unit tests, and no performance regressions happened, as the criterion-backed benchmarks confirm. Most importantly, users can easily omit a
distinctfrom a recursive computation without having to fiddle with more complicated, lower-level APIs, and enjoy a speedup of ~1.6 for such computations.Open questions if you want to proceed with this approach:
ChildCircuit::recursiveandChildCircuit::recursive_dynamicAPIs as deprecated?mark_distinctdoes not work with sharding (see conversation in Implement and document thread-safe use of the iterate API #6653), shall it be removed from the public API?RecursionBuilder::runmethod may find a better home in theCircuit traitand could be made more compositional. Maybe the builder pattern could be used there to create customiterate_*methods to selectively enable or disable the following behavior:iterate__with_bound_with_report.RecursionReportstream_with_consensusConsensusover the termination check_until_fixedpointchild.check_fixedpointto the termination checkThis is just an idea and some food for thought. For now, I'm quite happy with what is already in here.
Benches
Performance is on-par with the manual circuit without distinct but faster than the old API:

Performance is on-par with the old API and the manual circuit with distinct required:

Performance is on-par with the old API and the manual circuit after aggregation:
