yashmayya opened a new pull request, #19166: URL: https://github.com/apache/pinot/pull/19166
## Why A colocated join fails today when a partition of one of its tables holds no segments: ``` Failed to find any segment for table: myTable_OFFLINE, partition: 3 ``` This is easy to hit and hard to avoid: - A partition key with skew leaves some partitions with no rows. - Retention removes every segment of a partition. - A stream partition exists but has never committed a segment. - One logical partition space is split across several tables, so each table populates a subset. - A table declares more partitions than it currently uses, to leave room to grow. The assignment cannot simply skip the empty partition. Worker ids come from a counter over the partitions that hold data, so skipping one shifts every later partition down a slot. Two tables that each drop a different empty partition then end up with equal worker counts, get wired one-to-one, and join mismatched partitions. That loses rows and reports no error, which is why the assignment refuses to continue at all. ## What changes - The stages tied together by direct exchanges form a group, and the group shares **one ordered list of partition classes**. - A class leaves the list only when **no member of the group** holds data in it. - A class the group keeps, but one member holds no data for, gets a worker with **no segments**. That worker sits on a server borrowed from a member that does hold the class, so the exchange stays in process. - Both sides of every direct exchange assert that they agree on the list. A disagreement fails at plan time instead of returning wrong rows. - The broker now publishes the partitions whose only segments are new and have no online replica. Those hold data that no server can serve as a whole, so they keep failing. A side effect is less fan-out. A table that declares 8 partitions and populates 3 runs 3 workers, so the broker sends fewer requests and waits on fewer servers. ## What now fails on purpose - A hybrid table with segments whose partition metadata does not match the table config now fails planning. It used to answer and drop those segments in silence. The most common cause is a partition config added after some segments were built. - A table spread over more than one cluster reports no partition info, so a partition-aware plan is not attempted on a partial view. No server holds another cluster's segments, so one cluster's view would make a partition served elsewhere look empty. ## Broker pruning comes next Broker pruning is off for a colocated join today, and this change is what makes it possible. - Pruning drops partitions that hold no matching segment. That is the same operation as dropping a class that holds no data. - The group already shares one class list, so a pruned class can leave the list the same way an empty one does, and both sides stay in step. - The gain is fewer servers per query, which protects tail latency on large clusters. A follow-up PR adds it. ## Testing - 22 files. Planner, runtime, broker and core suites all pass. - An end-to-end test joins two tables that declare 8 partitions and populate 3 and 4 of them. It asserts the rows, the worker count per leaf, the segments each leaf reads, and that each send goes to one receiver with the same worker id. That last check proves the plan did not fall back to a shuffle. - One test drives the broker metadata directly and proves the new signal does not depend on map iteration order. - One test asserts the aggregation merge identity over every registered aggregation, because a worker with no segments contributes one identity row. ## Labels `bugfix`, `backward-incompat`, `release-notes` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
