rseetham opened a new issue, #18739: URL: https://github.com/apache/pinot/issues/18739
Multitopic realtime ingestion as it currently exists has some issues. In this [PR](https://github.com/apache/pinot/pull/18433), @Jackie-Jiang said The multi-topic partition id overloading existing partition id has caused so many issues: We are missing checks here and there Stream with custom partition id fails (See https://github.com/apache/pinot/issues/17260) I think it is the time to revisit this design decision and avoid the troubles going forward. I can see 2 possible solutions: Make partition_id part within the table name x_y where x is the topic id and y is the partition id. For single topic name, it remains the same Leverage logical table for multiple topics, each topic a separate physical table This is the PEP request to make those changes. I've detailed both suggestions in this doc: https://docs.google.com/document/d/1dY6ZvaG_-FbTAQ5e1lxB4uvi8VGbR_WfCfCl5zD76A4/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
