Skip to content

Add a non-incremental / batch oracle to the indexed left-join property tests - #6855

Open
nmarasoiu wants to merge 3 commits into
feldera:mainfrom
nmarasoiu:outer_join_new_batch_path_oracle
Open

Add a non-incremental / batch oracle to the indexed left-join property tests#6855
nmarasoiu wants to merge 3 commits into
feldera:mainfrom
nmarasoiu:outer_join_new_batch_path_oracle

Conversation

@nmarasoiu

@nmarasoiu nmarasoiu commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Add a non-incremental / batch oracle to the indexed left-join property tests.

The oracle integrates both inputs, computes matched rows with stream_join, derives right-key presence with stream_aggregate, and subtracts unmatched candidates whose keys occur on the right. It supports signed Z-set weights, including keys represented only by negative weights.

The existing reference implementation remains in place and checks left join through a composition of incremental operators. The new oracle recomputes from accumulated batches, independently covering the incrementalization behavior used by both ordinary and balanced left joins in per-step and transaction modes.

Describe Manual Test Plan

Checklist

  • Unit tests added/updated
  • Integration tests added/updated
  • Documentation updated
  • Changelog updated

Breaking Changes?

Mark if you think the answer is yes for any of these components:

Describe Incompatible Changes

…tests. The oracle integrates both inputs, computes matched rows with stream_join, derives right-key presence with stream_aggregate, and subtracts unmatched candidates whose keys occur on the right. It supports signed Z-set weights, including keys represented only by negative weights.

The existing reference implementation remains in place and checks left join through a composition of incremental operators. The new oracle recomputes from accumulated batches, independently covering the incrementalization behavior used by both ordinary and balanced left joins in per-step and transaction modes.
@nmarasoiu
nmarasoiu marked this pull request as draft August 13, 2026 17:17
The indexed left-join property tests drew weights from
generate_join_test_data(10, 5, 1, 50). Weights come from the half-open
range -max_weight..max_weight, so max_weight = 1 yielded only {-1, 0}.
Zero-weight tuples are no-ops, which made every non-trivial tuple a
deletion: the non-incremental batch oracle never saw an insert, and
multiplicity above 1 never arose. Raise max_weight to 3, matching every
other call site in the crate, so the tests cover insertions, deletions,
multiplicity, and mixed cancellation.

Also document the oracle's matched + unmatched - suppressed algebra, and
in particular why the unit weight of right_presence makes the
cancellation exact for signed weights. Replace plus(&suppressed.neg())
with minus(&suppressed), name the third closure clone suppressed_f for
symmetry with matched_f and unmatched_f, and give the assertions messages
that identify which oracle disagreed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@nmarasoiu
nmarasoiu marked this pull request as ready for review August 13, 2026 17:35
@mihaibudiu

Copy link
Copy Markdown
Contributor

I think this could use a shorter commit message.
You are just adding a new way to compute the reference results for a left-join if I understand right.

@mihaibudiu

Copy link
Copy Markdown
Contributor

What in particular caused this PR? What do you find insufficient in the existing tests?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants