#general
@kaushikf9t: @kaushikf9t has joined the channel
@ragiroux: @ragiroux has joined the channel
@ramabaratam: @ramabaratam has joined the channel
@karinwolok1: To all the Pinot power players here! Wanted to make you aware of an awesome event coming up next week:
@sheetalarun.kadam2: @sheetalarun.kadam2 has joined the channel
#random
@kaushikf9t: @kaushikf9t has joined the channel
@ragiroux: @ragiroux has joined the channel
@ramabaratam: @ramabaratam has joined the channel
@sheetalarun.kadam2: @sheetalarun.kadam2 has joined the channel
#feat-presto-connector
@ming.liu: @ming.liu has joined the channel
#troubleshooting
@kaushikf9t: @kaushikf9t has joined the channel
@shaileshjha061: HI @mayanks @dlavoie @npawar I got few questions on Pinot Restore Mechanism... *Can offline tables be converted back to realtime* after pushing the segments backup from offline table .. and the backup of realtime table that is stored in pinot .. can it be used to create an offline table means *is it possible by any means to restore the realtime table data if gcp service account is changed and vpc is changed?* Kindly help me out in figuring out restoration on realtime tables. Thanks in Advance. *CC: @nadeemsadim* *@mohamed.sultan* *@mags.carlin*
@patidar.rahul8392: @shaileshjha061 For this use can u can try hybrid table.it's a combination of realtime and offline table, and then u can push the segment in hybrid table.
@shaileshjha061: @patidar.rahul8392 Can you suggest or share docs link? To push the segment back in hybrid table of offline table. CC: @nadeemsadim
@patidar.rahul8392:
@kaushikf9t: Just started exploring Apache Pinot on setting it up on AWS using the documentation here -
@kaushikf9t: When I do a kubectl get all -n pinot-quickstart, I see this has brought up classic load balancers to expose both the broker and the controller on tcp ports, and when I make a curl/browser request to the DNSs, I expect these to show up the UI for the broker and the Swagger UI for the controller, but the request times out eventually without bringing up the UI. I am a beginner in AWS Networking as well, but the security groups created by these setup instructions which I have followed exactly allows TCP requests from 0.0.0.0/0. Any inputs on bringing the UI up for the broker and server are much appreciated!
@ragiroux: @ragiroux has joined the channel
@joshhighley: I have a realtime table consuming messages from a 3 partition Kafka topic. Possibly due to some network issues over the weekend, all 3 consumers are repeating the same error messages about a bad offset: ```2021/06/07 15:24:27.918 INFO [Fetcher] [agent_daily__2__2__20210605T0819Z] [Consumer clientId=consumer-71, groupId=] Fetch offset 22 is out of range for partition agent_daily-2, resetting offset 2021/06/07 15:24:27.919 INFO [Fetcher] [agent_daily__2__2__20210605T0819Z] [Consumer clientId=consumer-71, groupId=] Resetting offset for partition agent_daily-2 to offset 5. 2021/06/07 15:24:27.938 INFO [Fetcher] [agent_daily__1__2__20210605T0819Z] [Consumer clientId=consumer-73, groupId=] Fetch offset 20 is out of range for partition agent_daily-1, resetting offset 2021/06/07 15:24:27.939 INFO [Fetcher] [agent_daily__1__2__20210605T0819Z] [Consumer clientId=consumer-73, groupId=] Resetting offset for partition agent_daily-1 to offset 0. 2021/06/07 15:24:27.942 INFO [Fetcher] [agent_daily__0__2__20210605T0819Z] [Consumer clientId=consumer-72, groupId=] Fetch offset 24 is out of range for partition agent_daily-0, resetting offset 2021/06/07 15:24:27.943 INFO [Fetcher] [agent_daily__0__2__20210605T0819Z] [Consumer clientId=consumer-72, groupId=] Resetting offset for partition agent_daily-0 to offset 1. 2021/06/07 15:24:33.018 INFO [Fetcher] [agent_daily__2__2__20210605T0819Z] [Consumer clientId=consumer-71, groupId=] Fetch offset 22 is out of range for partition agent_daily-2, resetting offset 2021/06/07 15:24:33.018 INFO [Fetcher] [agent_daily__2__2__20210605T0819Z] [Consumer clientId=consumer-71, groupId=] Resetting offset for partition agent_daily-2 to offset 5. 2021/06/07 15:24:33.038 INFO [Fetcher] [agent_daily__1__2__20210605T0819Z] [Consumer clientId=consumer-73, groupId=] Fetch offset 20 is out of range for partition agent_daily-1, resetting offset 2021/06/07 15:24:33.039 INFO [Fetcher] [agent_daily__1__2__20210605T0819Z] [Consumer clientId=consumer-73, groupId=] Resetting offset for partition agent_daily-1 to offset 0. 2021/06/07 15:24:33.042 INFO [Fetcher] [agent_daily__0__2__20210605T0819Z] [Consumer clientId=consumer-72, groupId=] Fetch offset 24 is out of range for partition agent_daily-0, resetting offset 2021/06/07 15:24:33.043 INFO [Fetcher] [agent_daily__0__2__20210605T0819Z] [Consumer clientId=consumer-72, groupId=] Resetting offset for partition agent_daily-0 to offset 1.``` The 'reset' offsets of 5, 0, and 1 are correct: I created a new 'test' table for the same topic and it used those offsets with no issue. I've tried disabling/enabling the table but it resumes those error messages. Is there some other way to reset the table consumers?
@mayanks: Sounds like an issue on Kafka side might have made Pinot's offset state inconsistent?
@mayanks: If so, then the only way I can think of is to delete and recreate the table in Pinot.
@joshhighley: but I can create a new table against the same topic, and not have the issue. Is there any other way to reset Pinot's offset state? Luckily this is a dev environment -- deleting and re-creating the table isn't really an option in production
@joshhighley: ...Pinot seems to know what the offset should be, since it logs it. It doesn't seem to use that offset, though
@mayanks: If the offsets are corrupted on the kafka side, then it is a kafka side issue that needs to be prevented in production?
@mayanks: Would be good to understand the root cause here first, if we want to avoid it in production.
@mayanks: If it turns out to be a pinot side issue, then yes definitely we should have a way to recover gracefully.
@mayanks: BTW, the log message is from kafka consumer.
@joshhighley: I can create a new table subscribed to the same topic and receive messages while the existing table still does not. If there was a Kafka issue at some point then it appears corrected
@mayanks: What I am saying is that Pinot stores offsets as it commits segments. If a disruptive change on kafka side makes these offsets inconsistent with what Pinot had seen earlier, then the kafka consumer inside of Pinot will run into this state.
@mayanks: When you start a new table, the new table is seeing all consistent offsets (as it doesn't have anything saved).
@mayanks: Can you try to restart the Pinot servers and see if that helps?
@mayanks: My suspicion is that after Pinot committed segments and saved the offsets for them, there was a disruptive change in kafka side due to which these saved offsets are out-of-sync with Kafka.
@ssubrama: See what the segment metadata says regarding the offset where the segment is supposed to start consuming. Pinot uses that offset. Also, have you what is your kafka retention time? Maybe increasing it a little will solve your problem
@ramabaratam: @ramabaratam has joined the channel
@nadeemsadim: Hi .. Any docs/video for pushing the tar backup on gcs to hybrid/offline table
@mayanks: Have you looked at:
@sheetalarun.kadam2: @sheetalarun.kadam2 has joined the channel
#pinot-dev
@ming.liu: @ming.liu has joined the channel
#getting-started
@kaushikf9t: @kaushikf9t has joined the channel
@kaushikf9t: Just started exploring Apache Pinot on setting it up on AWS using the documentation here -
@kaushikf9t: When I do a kubectl get all -n pinot-quickstart, I see this has brought up classic load balancers to expose both the broker and the controller on tcp ports, and when I make a curl/browser request to the DNSs, I expect these to show up the UI for the broker and the Swagger UI for the controller, but the request times out eventually without bringing up the UI. I am a beginner in AWS Networking as well, but the security groups created by these setup instructions which I have followed exactly allows TCP requests from 0.0.0.0/0. Any inputs on bringing the UI up for the broker and server are much appreciated!
@ming.liu: @ming.liu has joined the channel
#pinot-docsrus
@mayanks: @jlli Could you please help with converting the google doc design doc for segment sorting/partitioning into docs?
@jlli: @jlli has joined the channel
@nadeemsadim: Any docs/video for pushing the tar backup on gcs to hybrid/offline table
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
