#general
@adireddijagadesh: @adireddijagadesh has joined the channel
@lrhadoop143: @lrhadoop143 has joined the channel
@tahir.fayyaz: @tahir.fayyaz has joined the channel
@karinwolok1: :wave: Heyyyy to all the new :wine_glass: members! :smile: Please tell us who you are and what brought you here! @adireddijagadesh @lrhadoop143 @tahir.fayyaz @iamluckysharma.0910 @nanda.yugandhar @son.nguyen.nam @tgauchoux @senthissenthh @alinoorrahman @ssainz @cebofil371 @arunsundar4298 @ssandeepadas007 @barana @rjpatrick @meetdesai74 @moeryomenko @kskarthik.columbia @rajesh.narayan @gsisodiya @nageshblore @rahul.oracle.db @yifanzhao0210 @nakkul @pradeepks2003 @salkadam @arun11299 @rupesh_raghavan @i.young @zainamro1 @yuchaoran2011 @cechovsky.jozef @bvarunpy @becca.silverman @leb9882 @sanipindi @raj.swarnim @lxy1995seu @syed.hadi @ebuth @sirsh @mattk @zineb.raiiss @frank_gao2 @suresh.intuit @weixiang.sun @chengweili402 @alec @pascal.jerome
@sirsh: Hi all!! I am (slowly) evaluating Pinot and hope to spend much more time with it in the coming weeks. I came across Pinot after looking for an alternative to Druid that might be easier to maintain but offer some of the same types of OLAP over either Kafka data or data stored on an S3 data lake (via presto). I am looking for a tool that augments what Snowflake provides (in the space of ad hoc on higher latency queries.) I hope to sell my organization on the power of going beyond the "snowflake model" to understand assembly "flows" in manufacturing. K8s is important to me so a big part of my evaluation will be how i make this work within my infrastructure (deployed via Argo-CD along with all our other great open source apps) Look forward to learning with you all!
@adireddijagadesh: Hi All, I am here to contribute and raise PR’s. Previously contributed to Kafka and Druid.
@alec: My firm is in the very early stages of exploring Pinot and I am just here to be able to search around for FAQs I have and keep an eye on the roadmap. I work with Tiger Zhao.
@sanipindi: Hello, I am here to explore Pinot in our organization for user facing analytics in aws cloud infra. hugely interested in the operability and maintainability aspects of it in production workloads.
@dadelcas: Hi, I'm experiencing a strange issue today. My realtime table has bucketTimePeriod of 1d and bucketTimePeriod of 2d. My offline workflows are not running and I can see a message in the logs that says "Window data overflows into CONSUMING segments for partition of segments..." then "Found no eligible segments for task: RealtimeToOfflineSegmentsTask with window [1555200000 - 1641600000]. Skipping task generation...". Note these timestamps seem to be in seconds instead of milliseconds. I can see the segment.end.time and segment.start.time values are in seconds which I'm not sure of whether this was the case before. Looking through the code I can see TimeUtils compute the window using milliseconds so this is why the window spans 2 years instead of 2 days. I'm trying to figure out why this is happening now, any help is appreciated
@mayanks: Do you mind pasting your table and task configs? I am guessing some issue with time unit. cc: @npawar
@dadelcas: It seems the schema was defined in microseconds and the table in milliseconds. Is there an easy way to fix this issue?
@npawar: we only read what is set in schema. the “timeColumnUnit” in table config is ignored (also deprecated). So schema was MICROSECONDS and data had values in millis?
@npawar: it would be cleanest to just start the realtime table from scratch. Because any segments that have completed so far, will have incorrect time start/end
@dadelcas: Yup, the data contains milliseconds
@dadelcas: I wouldn't like to loose the test data I've got because I still don't have a proper backfilling process in place. I could write a script to update the znodes but that's a bit of effort I rather spend on working on backfilling
@dadelcas: Thanks both
@camerronadams: @camerronadams has joined the channel
#random
@adireddijagadesh: @adireddijagadesh has joined the channel
@lrhadoop143: @lrhadoop143 has joined the channel
@tahir.fayyaz: @tahir.fayyaz has joined the channel
@camerronadams: @camerronadams has joined the channel
#troubleshooting
@msoni6226: Hi All, I am running a hybrid table setup. If I delete a table for some reason and recreate the table with same name, should the Minion be re-started to make the realtime to offline working for this newly created table with same name?
@mayanks: As long as the new table config has the realtime to offline configs in it, it will work.
@msoni6226: Yes the REALTIME table has the realtime to offline configuration defined. So, if we re-create table with same name, Minion re-start is not required. Right?
@mayanks: No, minions scan all table configs to identify jobs to schedule.
@msoni6226: Thanks Mayank for confirming it
@vibhor.jain: Hi Team, We are planning to add a de-duplication flow for our solution. As part of that, we are doing 2 things: 1. Enable UPSERT in realtime table. - Flink has the key set. - Primary key defined in schema and realtime table using it. - UPSERT working fine. 2. For realtime to offline flow via minion, we found that these duplicates were coming in OFFLINE table. - so we tried with mergeType: dedup (earlier it was concat) Now, the realtime to offline flow has stopped working (no data in OFFLINE table, minion is up and running) Queries: -------- 1. Is our dedup flow proper? UPSERT for realtime and mergeType: dedup for realtime to offline flow? 2. Any pointers around why this realtime to offline flow stopped working after adding these configs?
@mayanks: Upsert feature currently works for realtime only tables, afaik
@mayanks: cc @jackie.jxt @yupeng
@vibhor.jain: Hi @mayanks, we are using UPSERT (i.e. "primaryKeyColumns": [] defined in schema) for realtime table and its working fine. Are we saying hybrid table flow is not supported here via mergeType: dedup?
@mayanks: Hi @vibhor.jain I am saying UPSERT does not support hybrid table at the moment, it is being worked on afaik.
@vibhor.jain: mergeType: "dedup" should work there? Although we are doing a full row comparison, that should be ok for deduplication?
@mayanks: However, if let's say an upsert comes for the same primary key, I don't think the code will take care of what is already moved to offline, right?
@vibhor.jain: Hi @mayanks We are not looking for UPSERT across realtime and offline table. UPSERT is limited to realtime table and its working as expected for us. Now, when we move this realtime data to offline, these duplicate records are showing up again (UPSERT keeps all copies). So, we tried mergeType config with value "dedup" to handle duplicates here and we see the realtime to offline flow has suddenly stopped working.
@mayanks: Ok, may be check the dedup config, and also see if errors in minion log
@npawar: you might also find logs about realtimeToOffline job in controller. First some lines should appear in controller about task scheduling, then you will see corresponding lines in minion logs about task execution
@jackie.jxt: Realtime to offline flow will move records to the offline table (managed separately from the realtime table), and UPSERT cannot be applied to the offline side. Dedup won't help with this setup
@jackie.jxt: Please check the log on controller about the realtimeToOffline task as Neha suggested. And also check if there are time gaps. IIRC there is a known issue of realtimeToOffline task stuck when there are time gaps in the data
@adireddijagadesh: @adireddijagadesh has joined the channel
@lrhadoop143: @lrhadoop143 has joined the channel
@valentin: Hello, I’m looking to enable segment replication on my cluster and I found 2 way of do it in the documentation: •
@npawar: You can start off with the first way. Replica groups is an advanced optimization when you want to reduce server fan-out and scale throughout (as mentioned in that doc)
@tahir.fayyaz: @tahir.fayyaz has joined the channel
@bajpai.arpita746462: Hi All, I am trying to to move data from REALTIME to OFFLINE table through minion flow but I am getting the below error: Starting PinotTaskManager with running frequency of 1200 seconds. Start running task: PinotTaskManager Trying to schedule task type: RealtimeToOfflineSegmentsTask, isLeader: true Start generating task configs for table: dataanalytics_REALTIME for task: RealtimeToOfflineSegmentsTask Caught exception while running task: PinotTaskManager java.lang.IllegalStateException: null at
@npawar: can you check the segment metadata of the completed segments? it looks like they have incorrect start/end times.
@anu110195: How to do query on pinot's broker with increased timeout ?
@dadelcas: Try passing `timeoutMs` as part of the query options
@adireddijagadesh:
@andruszd: When i use the pinot api to check that the tables are all ok , using ('ingestionState') == "UNHEALTHY": I get the following error messages Error Message for stream4_metric: Did not get any response from servers for segment: stream4_metric__0__22__20210930T1201Z Segment Error for:-> stream4_metric__0__22__20210930T1201Z: {"code":404,"error":"[] not found."} and Error Message for stream5_events: Did not get any response from servers for segment: stream5_events__0__0__20210819T1031Z Segment Error for:-> stream5_events__0__0__20210819T1031Z: {"code":404,"error":"[] not found."} So when i go to the server(x2) the steam4 file is totally missing and stream5 is in a _tmp directory ls -l stream4_metric__0__22__20210930T1201Z ls: cannot access stream4_metric__0__22__20210930T1201Z: No such file or directory There are no files in _tmp and for the stream4 file it is in the _tmp directory , multiple times and as you can see has data .. ls *stream5_events__0__0__20210819T1031Z* tmp-stream5_events__0__0__20210819T1031Z-1629822465115: tmp-c1054878-57f2-47f7-b7c2-085b5dfec02f tmp-stream5_events__0__0__20210819T1031Z-1630594394292: tmp-fdb518aa-929b-4eb3-9ecf-a4386ae9d084 tmp-stream5_events__0__0__20210819T1031Z-1630661411117: tmp-5f8ed825-d323-4344-b433-5287a7a32d4e tmp-stream5_events__0__0__20210819T1031Z-1632927825055: tmp-0f94cfe9-f024-4deb-9f72-8435b5315ee5 tmp-stream5_events__0__0__20210819T1031Z-1633016079847: tmp-df43885d-83c5-41fb-9eb1-0cfd1bfc6be9 /data/pinot/data/index/stream5_events_REALTIME/_tmp/tmp-stream5_events__0__0__20210819T1031Z-1629822465115/tmp-c1054878-57f2-47f7-b7c2-085b5dfec02f tmp-c1054878-57f2-47f7-b7c2-085b5dfec02f]$ ls -l total 24176 -rw-r--r--. 1 pinot pinot 20 Aug 24 17:27 device_id.dict -rw-r--r--. 1 pinot pinot 541 Aug 24 17:27
@camerronadams: @camerronadams has joined the channel
@gabuglc: Hello guys, how do i create a multitenant cluster for my production environment. It gives me the following error ```Executing command: AddTenant -controllerProtocol http -controllerHost localhost -controllerPort 9001 -name krealoBrokerTenant -role BROKER -instanceCount 3 -offlineInstanceCount 0 -realTimeInstanceCount 3 -exec {"_code":500,"_error":"Failed to create tenant"} {"_code":500,"_error":"Failed to create tenant"}```
@mayanks: Do you have enough untagged broker participants?
@mayanks: Agree though, that the response should be user friendly
@gabuglc: Just have 1
@gabuglc: same error ```Executing command: AddTenant -controllerProtocol http -controllerHost 0.0.0.0 -controllerPort 9001 -name krealoBrokerTenant -role BROKER -instanceCount 1 -offlineInstanceCount 0 -realTimeInstanceCount 1 -exec {"_code":500,"_error":"Failed to create tenant"} {"_code":500,"_error":"Failed to create tenant"}root@pinot-controller:/opt/pinot#```
@mayanks: You have 1 untagged, but you are asking for 3, I think that is one issue
@gabuglc: yeah modified for 1 here, still same error ```Executing command: AddTenant -controllerProtocol http -controllerHost 0.0.0.0 -controllerPort 9001 -name krealoBrokerTenant -role BROKER -instanceCount 1 -offlineInstanceCount 0 -realTimeInstanceCount 1 -exec {"_code":500,"_error":"Failed to create tenant"} {"_code":500,"_error":"Failed to create tenant"}root@pinot-controller:/opt/pinot#```
#onboarding
@son.nguyen.nam: @son.nguyen.nam has joined the channel
#pinot-dev
@adireddijagadesh: @adireddijagadesh has joined the channel
#getting-started
@adireddijagadesh: @adireddijagadesh has joined the channel
#releases
@son.nguyen.nam: @son.nguyen.nam has joined the channel
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@pinot.apache.org For additional commands, e-mail: dev-h...@pinot.apache.org
