Skip to content

[dbsp] Merge backpressure uses loose batch count, not total batch count. #6866

Description

@ryzhyk

The backpressure mechanism is currently triggered by the number of loose batches in the spine. This ignores the potentially large number of batches being merged, which are still part of the spine until the merges finish. Arguably, total batch count is a better trigger.

Metadata

Metadata

Assignees

Labels

DBSP coreRelated to the core DBSP librarystoragePersistence for internal state in DBSP operators

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions