NGA-TRAN commented on issue #24438:
URL: https://github.com/apache/datafusion/issues/24438#issuecomment-5414148135
Thanks, @alamb for outlining the complication concerns and asking for
concrete use cases.
**Use case:**
We’re entering an AI‑driven era with extremely high‑volume telemetry and
analysis workloads. On the same dataset, we now face two extremes:
1. Interactive queries — require low latency. We partition data finely so
queries run in parallel. Both DataFusion and Distributed DataFusion handle this
well, and we scale perfectly.
2. Analytic queries — must finish within acceptable latency on very large
datasets. Here, the number of partitions far exceeds the number of cores (even
across many workers), so we must merge partitions. We’re fine with sequential
execution within a merged partition as long as it completes in reasonable time.
But merging drops the sort order, which causes two major issues:
i .Loss of streaming aggregation, hurting downstream efficiency and
latency.
ii. For high‑cardinality workloads, hash tables become huge, increasing
resource pressure and risking OOMs under concurrency.
We now see the **core idea**: the external system can indicate that data is
unsorted but non‑overlapping. This is **similar to other metadata properties**
(e.g., sort order) that come from the external system.
**On complexity:**
Yes, it’s complicated — and we’re happy to design and support this (thanks
@gene-bordegaray and @xavlee and future Datadog contributors) with input from
the DataFusion community. We see this as an opportunity to push DataFusion to
the next level for AI workloads, with Datadog acting as a pioneer in providing
real use cases.
Let us know your thoughts, and whether you’d like us to bring in other users
of range partitioning to contribute.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
