Skip to content

[SQL] Support for new Calcite UNNEST operator - #6599

Draft
mihaibudiu wants to merge 1 commit into
feldera:mainfrom
mihaibudiu:unnest
Draft

[SQL] Support for new Calcite UNNEST operator#6599
mihaibudiu wants to merge 1 commit into
feldera:mainfrom
mihaibudiu:unnest

Conversation

@mihaibudiu

@mihaibudiu mihaibudiu commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Describe Manual Test Plan

I will mark this as draft, it needs apache/calcite#5031 to be merged first
apache/calcite#5086 is a fragment of this PR which could be useful too.

Checklist

  • Unit tests added/updated

@mihaibudiu
mihaibudiu marked this pull request as draft July 6, 2026 23:31

@mythical-fred mythical-fred left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Draft first-pass review — high-level only.

Nice cleanup: replacing the hand-rolled pattern match in visitCorrelate with a proper Calcite pipeline (Project extraction → CORRELATE_UNCOLLECT_MERGE → single visitUncollect) is the right direction and removes a lot of fragile RelNode sniffing.

A few high-level notes/questions:

  1. CorrelateProjectExtractor.java looks like a straight port of a Calcite class (naming, RelHomogeneousShuttle, @Override public RelNode style, doc format). If it's vendored from apache/calcite#5031 (or a companion CALCITE JIRA) pending release, a one-line pointer at the class-level Javadoc would save the next reader some archaeology. Also worth noting whether we plan to delete this file once we upgrade to a Calcite version that ships it.

  2. visitCorrelate now unconditionally throws decorrelateError. The invariant is now "by the time we compile, every LogicalCorrelate must have been merged with its Uncollect or otherwise decorrelated." The comment on visit() — "Give up if the plan is not decorrelated" — is a bit terse for a fairly load-bearing invariant. Consider spelling it out (what the optimizer step is responsible for producing, and that a surviving Correlate means the new merge rule didn't fire).

  3. compileModifyTable LogicalUnion fix looks like a genuine correctness bug fix — the old loop silently dropped LogicalValues children of a mixed union, so INSERT ... VALUES (expr_row), (lit_row) would lose the literal rows. Nice catch (and testUnnestCoalesce exercises exactly this shape). Semantically unrelated to UNNEST though; if it's easy to split, it would make bisection nicer, but not blocking on a draft.

  4. UNNEST semantics look right for the cases in scope:

    • Arrays and maps both flow through ICollectionType uniformly.
    • WITH ORDINALITY, the "Calcite forgets to mark inner fields nullable" workaround, and the ordinality index type are preserved verbatim from the old path.
    • getCollectionFieldIndices().cardinality() != 1 is a stricter and clearer guard than the old inputRowType.size() > 1 (which conflated passthrough fields with collection fields).
    • uncollect.isOuter is the new gate for LEFT JOIN UNNEST — correctly still unimplemented.
    • shuffleSize = type.size() with an IdShuffle is fine given the merged Uncollect already carries the full output row type.
  5. Test coverage for the new shape is present (nested field UNNEST, nested + COALESCE, ordinality, mixed VALUES rows) and the pre-existing testUnnest1/testDoubleUnnest1 cases remain, so the merged pipeline is exercised on both old and new plan shapes. Good.

  6. containsUncollect full-plan scan before every step is a minor efficiency wart — a Hep rule set could be made no-op automatically — but negligible on realistic plans. Not worth touching unless it shows up in profiling.

Multisets — not exercised explicitly in the new tests as far as I can see; they go through the same ICollectionType path so should be fine, but a one-liner test wouldn't hurt once the Calcite dep lands.

Marking as COMMENT — happy to do a proper pass once apache/calcite#5031 is merged and this leaves draft.

Signed-off-by: Mihai Budiu <mbudiu@feldera.com>
@mihaibudiu

Copy link
Copy Markdown
Contributor Author

I have reworked apache/calcite#5031. This PR needs that one to be merged into Calcite, but this code verifies that all tests pass if linked with a version of Calcite containing that change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants