[SQL] Support for new Calcite UNNEST operator - #6599
Conversation
mythical-fred
left a comment
There was a problem hiding this comment.
Draft first-pass review — high-level only.
Nice cleanup: replacing the hand-rolled pattern match in visitCorrelate with a proper Calcite pipeline (Project extraction → CORRELATE_UNCOLLECT_MERGE → single visitUncollect) is the right direction and removes a lot of fragile RelNode sniffing.
A few high-level notes/questions:
-
CorrelateProjectExtractor.javalooks like a straight port of a Calcite class (naming,RelHomogeneousShuttle,@Override public RelNodestyle, doc format). If it's vendored from apache/calcite#5031 (or a companion CALCITE JIRA) pending release, a one-line pointer at the class-level Javadoc would save the next reader some archaeology. Also worth noting whether we plan to delete this file once we upgrade to a Calcite version that ships it. -
visitCorrelatenow unconditionally throwsdecorrelateError. The invariant is now "by the time we compile, everyLogicalCorrelatemust have been merged with itsUncollector otherwise decorrelated." The comment onvisit()— "Give up if the plan is not decorrelated" — is a bit terse for a fairly load-bearing invariant. Consider spelling it out (what the optimizer step is responsible for producing, and that a surviving Correlate means the new merge rule didn't fire). -
compileModifyTableLogicalUnion fix looks like a genuine correctness bug fix — the old loop silently droppedLogicalValueschildren of a mixed union, soINSERT ... VALUES (expr_row), (lit_row)would lose the literal rows. Nice catch (andtestUnnestCoalesceexercises exactly this shape). Semantically unrelated to UNNEST though; if it's easy to split, it would make bisection nicer, but not blocking on a draft. -
UNNEST semantics look right for the cases in scope:
- Arrays and maps both flow through
ICollectionTypeuniformly. WITH ORDINALITY, the "Calcite forgets to mark inner fields nullable" workaround, and the ordinality index type are preserved verbatim from the old path.getCollectionFieldIndices().cardinality() != 1is a stricter and clearer guard than the oldinputRowType.size() > 1(which conflated passthrough fields with collection fields).uncollect.isOuteris the new gate for LEFT JOIN UNNEST — correctly still unimplemented.shuffleSize = type.size()with anIdShuffleis fine given the merged Uncollect already carries the full output row type.
- Arrays and maps both flow through
-
Test coverage for the new shape is present (nested field UNNEST, nested +
COALESCE, ordinality, mixed VALUES rows) and the pre-existingtestUnnest1/testDoubleUnnest1cases remain, so the merged pipeline is exercised on both old and new plan shapes. Good. -
containsUncollectfull-plan scan before every step is a minor efficiency wart — a Hep rule set could be made no-op automatically — but negligible on realistic plans. Not worth touching unless it shows up in profiling.
Multisets — not exercised explicitly in the new tests as far as I can see; they go through the same ICollectionType path so should be fine, but a one-liner test wouldn't hurt once the Calcite dep lands.
Marking as COMMENT — happy to do a proper pass once apache/calcite#5031 is merged and this leaves draft.
Signed-off-by: Mihai Budiu <mbudiu@feldera.com>
|
I have reworked apache/calcite#5031. This PR needs that one to be merged into Calcite, but this code verifies that all tests pass if linked with a version of Calcite containing that change. |
Describe Manual Test Plan
I will mark this as draft, it needs apache/calcite#5031 to be merged first
apache/calcite#5086 is a fragment of this PR which could be useful too.
Checklist