Re: [I] Upsert table backfill enhancement: support externally partitioned data [pinot]

2024-04-24 Thread via GitHub
rohityadav1993 commented on issue #12987: URL: https://github.com/apache/pinot/issues/12987#issuecomment-2075075108 Another approach I believe can be utlized is defining a naming convention for uploaded segment similar to LLC. The segment name can capture the partition id. We already have

Re: [I] Upsert table backfill enhancement: support externally partitioned data [pinot]

2024-04-23 Thread via GitHub
tibrewalpratik17 commented on issue #12987: URL: https://github.com/apache/pinot/issues/12987#issuecomment-2071920309 > Provide partition id externally: Option 1: Provide partition id as http headers during segment upload Option 2: Provide partition id as part of uploaded segment

Re: [I] Upsert table backfill enhancement: support externally partitioned data [pinot]

2024-04-22 Thread via GitHub
Jackie-Jiang commented on issue #12987: URL: https://github.com/apache/pinot/issues/12987#issuecomment-2071013541 For real-time ingested data, the partition must match the upstream partition id to ensure the upsert assumption of all data of the same partition served by the same server, and

[I] Upsert table backfill enhancement: support externally partitioned data [pinot]

2024-04-22 Thread via GitHub
rohityadav1993 opened a new issue, #12987: URL: https://github.com/apache/pinot/issues/12987 ## Problem #6567 allows uploading a batch generated segment to Pinot upsert realtime table. Partitioned data is handled by defining the partition column in