#general
@tymm: hi. reported and estimated size. what do they mean?
@wooodini: @wooodini has joined the channel
@karinwolok1: Heyyyy all! Thanks for being part of the real-time movement! :smile:
@daniel: @karinwolok1 thanks for the warm welcome, I'm Daniel from Berlin and I'm building the world's first data engineering bootcamp.
@karinwolok1: That's incredible!!! Congrats!!
@justin.smalkowski: @justin.smalkowski has joined the channel
@bharath.sankara.v: @here - I am evaluating Pinot for my organization.. We will be ingesting Real-time data from Kafka. What schema formats does Pinot support to ingest from Kafka? We have protobuf serialized data, is there a support for that with pinot? Thanks in advance for your help! (I am sure I will have more questions)
@wrbriggs: I have only used JSON and Avro - it looks like there is an open issue for supporting protobuf:
@wrbriggs: However, it does look like there was Protobuf support added in 0.6.0:
@npawar: this protobuf support is only for Offline currently
@npawar: from kafka we can only do json/avro
@bharath.sankara.v: Thanks @wrbriggs @npawar
@bharath.sankara.v: is there a plan to support protos for Pinot realtime ingestion from Kafka @npawar?
@npawar: yes, we plan to
@npawar: how soon do you need it?
@bharath.sankara.v: any timeline cadence? Just need to factor that into our design... Thanks again
@npawar: @g.kishore ^
@bharath.sankara.v: we are actively developing - so sooner I would say..
@npawar: also @kharekartik ^ who would likely be working on it
@bharath.sankara.v: @g.kishore @kharekartik - any timeline for protobuf support for pinot?
@thomas.may: @thomas.may has joined the channel
@edan: @edan has joined the channel
@gokulrk2696: @gokulrk2696 has joined the channel
@aviv4339: @aviv4339 has joined the channel
@pankaj: @pankaj has joined the channel
@bowenli86: @bowenli86 has joined the channel
#random
@wooodini: @wooodini has joined the channel
@justin.smalkowski: @justin.smalkowski has joined the channel
@thomas.may: @thomas.may has joined the channel
@edan: @edan has joined the channel
@gokulrk2696: @gokulrk2696 has joined the channel
@aviv4339: @aviv4339 has joined the channel
@pankaj: @pankaj has joined the channel
@bowenli86: @bowenli86 has joined the channel
#feat-presto-connector
@aviv4339: @aviv4339 has joined the channel
#troubleshooting
@wooodini: @wooodini has joined the channel
@justin.smalkowski: @justin.smalkowski has joined the channel
@harold: I have a pinot setup using helm charts. I have setup a realtime table. The server ran out of disk space and it was stuck at the last segment (not consuming any new data). I updated the disk size and restarted the pinot server, and in the web UI, the last segment status was shown as BAD. So, I deleted the segment from the UI, but it still hasn't start consuming new data. Is there a way around this?
@wrbriggs: Is it the server or the controller that ran out of space? I have seen this behavior when the server cannot push completed segments to the controller.
@harold: It's the server.
@fx19880617: after you restart the pinot server, have you tried to enter into pinot server and do df to see the disk size reflection?
@fx19880617: if disk is enough, then pinot server restart should recover the consuming
@harold: I increased it to 20G and df shows that it has increased. /dev/sdb ext4 19G 3.4G 16G 18% /var/pinot/server/data
@harold: But I don't see new segments getting created/data consumed.
@harold: So, originally after restart the status in the UI for segment 0_12 shows as BAD. I tried reloading segment, it didn't help. Then, I tried deleting it. so segment. 0_12 is now gone, but it didn't seem to have recovered consuming
@harold: segment 0_0 to 0_11 shows status as DONE
@fx19880617: hmmm, will it be recovered if the consuming segment got deleted ? @npawar
@npawar: yes, validation manager will create the new CONSUMING segment
@npawar: validation manager runs every 15 minutes i think. So it should’ve recovered already
@harold: It seems that it hasn't recovered. Any steps that I need to verifY?
@npawar: what does this API return for that partition: ```curl -i -X GET "http://<controller host>:<controller port>/tables/<your table name>/consumingSegmentsInfo"```
@harold: seems to be empty
@harold: Pinot-Controller-Host: pinot-controller-0.pinot-controller-headless.pinot-quickstart.svc.cluster.local Pinot-Controller-Version: Unknown Access-Control-Allow-Origin: * Content-Type: application/json Content-Length: 33 {"_segmentToConsumingInfoMap":{}}
@npawar: and can you share the ideal state for the table?
@harold: I'm still new to Pinot. What do you mean by ideal state?
@npawar: try this ```curl -i -X GET "http://<controller host>:<controller port>/tables/<your table name>/idealstate"```
@harold: got it.
@harold: ```HTTP/1.1 200 OK Pinot-Controller-Host: pinot-controller-0.pinot-controller-headless.pinot-quickstart.svc.cluster.local Pinot-Controller-Version: Unknown Access-Control-Allow-Origin: * Content-Type: application/json Content-Length: 1615 {"OFFLINE":null,"REALTIME":{"prometheus__0__0__20210130T1836Z":{"Server_pinot-server-0.pinot-server-headless.pinot-quickstart.svc.cluster.local_8098":"ONLINE"},"prometheus__0__10__20210201T1730Z":{"Server_pinot-server-0.pinot-server-headless.pinot-quickstart.svc.cluster.local_8098":"ONLINE"},"prometheus__0__11__20210201T1751Z":{"Server_pinot-server-0.pinot-server-headless.pinot-quickstart.svc.cluster.local_8098":"ONLINE"},"prometheus__0__1__20210131T0716Z":{"Server_pinot-server-0.pinot-server-headless.pinot-quickstart.svc.cluster.local_8098":"ONLINE"},"prometheus__0__2__20210131T0737Z":{"Server_pinot-server-0.pinot-server-headless.pinot-quickstart.svc.cluster.local_8098":"ONLINE"},"prometheus__0__3__20210131T0759Z":{"Server_pinot-server-0.pinot-server-headless.pinot-quickstart.svc.cluster.local_8098":"ONLINE"},"prometheus__0__4__20210131T0821Z":{"Server_pinot-server-0.pinot-server-headless.pinot-quickstart.svc.cluster.local_8098":"ONLINE"},"prometheus__0__5__20210131T0842Z":{"Server_pinot-server-0.pinot-server-headless.pinot-quickstart.svc.cluster.local_8098":"ONLINE"},"prometheus__0__6__20210131T0904Z":{"Server_pinot-server-0.pinot-server-headless.pinot-quickstart.svc.cluster.local_8098":"ONLINE"},"prometheus__0__7__20210131T0926Z":{"Server_pinot-server-0.pinot-server-headless.pinot-quickstart.svc.cluster.local_8098":"ONLINE"},"prometheus__0__8__20210131T0948Z":{"Server_pinot-server-0.pinot-server-headless.pinot-quickstart.svc.cluster.local_8098":"ONLINE"},"prometheus__0__9__20210131T1009Z":{"Server_pinot-server-0.pinot-server-headless.pinot-quickstart.svc.cluster.local_8098":"ONLINE"}}}```
@harold: It was at 0__12 before, but then the server ran out of disk space. after resizing the pvc size and restarting pinot, the state of the segment went BAD. So I deleted it and restarted the server again
@npawar: do you have access to the cluster manager UI? can you go to the Zookeeper browser tab, and check if you see the segment 0__12 under PROPERTY_STORE/SEGMENTS/tableName
@harold: it doesn't have it.
@harold: I do see 0__12 in the local disk of the server though. in /var/pinot/server/data/index/<table name>/_tmp
@npawar: hmm, deleting the segment removed the metadata, so it got into a state where the partition is unable to recover.. i dont know if we can do anything other than manually add the metadata back. @ssubrama any ideas how to recover from this?
@ssubrama: Best to let the validation manager do its job. Check to see if you have disabled it. The controller logs should indicate somethig about realtime validationmanager
@npawar: this case cannot be solved by validation manager
@ssubrama: why not?
@npawar: just traced it. we will log `"Got unexpected status: {} in segment ZK metadata for segment: {}"` and do nothing
@ssubrama: The segment has disappeared from idealstate as well as metadata right?
@ssubrama: is that for segment 12 or 11?
@npawar: segment 0 to 11 have status DONE and ONLINE
@npawar: segment 12 is nowhere
@ssubrama: the log message shiould indicate the segment name. what is it saying/
@ssubrama: Also, it may be easier and faster to just drop the table and recreate it
@npawar: i dont know what it is saying. But i’m sure we are not handling this. Check it :slightly_smiling_face:
@npawar: line 1024 PinotLLCRealtimeSegmentManager
@ssubrama: You mean to say we print `{}` and not the actual segment name in the log message?
@npawar: that is not what i’m saying
@npawar: we dont handle this case. we will not recover - is what i’m saying
@ssubrama: Are you saying we dont handle the case where the last segment is missing from ideal state and metadata?
@ssubrama: That definitely worked, so it is a recent bug ontroduced.
@ssubrama: we do have test cases to cover that afaik
@npawar: looks like it was always like this
@npawar: anyway, @pabraham.usa is it possible to recreate the table? we can fix this on our end
@ssubrama: How does this test pass?
@ssubrama: @harold can you restart the controller? Maybe you have disabled the ReatlimeValidationManager?
@ssubrama: `controller.realtime.segment.validation.frequencyInSeconds`
@harold: I already tried restarting both controller and server earlier. It didn't work.
@harold: I just stepped out. Let me check that when I get back
@npawar: it is not the same case @ssubrama. That test will delete the CONSUMING segment and also make the previous segment CONSUMING. It simulates failure in step 2 exactly.
@npawar: but in this case previous segment is ONLINE
@npawar: why dont you trace it in the validator
@krishna080: apart from the problem at hand wrt BAD segment, how is the space management done for the real time segments backed by persistent volume ? Or is it the ingesting consumers' responsibility ?
@npawar: could you elaborate what about space mamagement you wanted to know?
@krishna080: are there any checks/balances to make sure that have enough space of the volume before accepting the next segment ?
@npawar: no such checks afaik. The ingestion assumes sufficient resources. @fx19880617 anything you’d like to add here to address this concern?
@pabraham.usa: @npawar, I was planning to use quota.storage part of tableConfig to specify a fixed size per Pinot server to protect disk. Do you think this will work?
@npawar: i believe the storage quota only applies to offline segment pushes.
@pabraham.usa: ok Thanks for confirming
@npawar: @mayanks what happens in LI production environments? any practices followed to prevent realtime tables from going over desired storage. Or just monitoring?
@ssubrama: @krishna080 you can use the RealtimeProvisoiningHelper to decide on storate requirements.
@ssubrama: @npawar we use RealtimeProvisioningHelper for memory requirements. We never exceed disk requirements since the realtime tables we have are small in retention.
@thomas.may: @thomas.may has joined the channel
@edan: @edan has joined the channel
@gokulrk2696: @gokulrk2696 has joined the channel
@aviv4339: @aviv4339 has joined the channel
@pabraham.usa: Hello is there a way to backup and recover all segments? In situations like, server is not recovering and you are forced to recreate the table but don’t want the indexing to happen on previously indexed data but resume from the crash point.
@g.kishore: Segments are always backed up in the segment store automatically
@pabraham.usa: Is it possible to set Efs/nfs as segment store. It will be very helpful if you could share some docs related to this.
@g.kishore: yes. you can have NFS as segment store. @mayanks do you have a diagram on how it works at LinkedIn
@mayanks: Yes NFS can be one of the deep-stores.
@mayanks: IIRC it is the default PinotFs implementation. So simply setting the controller dataDir to point to a NFS mount point should set you up @pabraham.usa
@pabraham.usa: Thanks @mayanks, I tried that but I cannot see all the segments coming to controller folder. Controller segment folder size is very less compared to server data folder
@pabraham.usa: The file colums.psf is missing from all the segments in the controller data folder
@pabraham.usa: So controller is only saving segment index without data . I am guessing there could be some configuration setting somewhere that enables the controller to store the data as well?
@pabraham.usa: Actually it is there, sorry my bad
@mayanks: So it works?
@pabraham.usa: All segments are in controller data folder zipped. But I don't know how to fetch it back to pinot server when required.
@pabraham.usa: Wondering reload all segment api will pull segments from controller if it is missing from the server ?
@mayanks: Yes
@pabraham.usa: Thanks will try it
@mayanks: Yes, loading of data from deep-store/NFS to servers is transparent to users (you should not need to worry about that).
@pabraham.usa: Great design there, thanks for sharing
@elon.azoulay: Hi, we have a hybrid table that is not serving realtime data (unless _REALTIME is explicitly used) and noticed that the time boundary is given in ms but the time column is a datetime field with input in ms from another field (base column in kafka) and output granularity in seconds. Anyone else experience this? Is there a way to set the time boundary?
@mayanks: Yes, likely the root cause is incorrect time boundary.
@elon.azoulay: Nice! Is there a way we can reset it?
@mayanks: Incorrect time boundary is because of incorrect setup/config, fixing that should solve the problem. Afaik, there isn't a way to explicitly set time boundary
@elon.azoulay: Ah ok, checking - you mean table config or server config? Or both? Checking table configs and comparing
@elon.azoulay: for offline vs realtime
@mayanks: Table config
@mayanks: They should have the same unit
@mayanks: Also, there might be a min granularity (seconds?) needed for hybrid tables. @jackie.jxt?
@elon.azoulay: The segment configs use the same column - it's a datetime field
@elon.azoulay: both realtime and offline:
@elon.azoulay: ```"OFFLINE": { "tableName": "enriched_orders_OFFLINE", "tableType": "OFFLINE", "segmentsConfig": { "schemaName": "enriched_orders", "segmentPushFrequency": "daily", "segmentPushType": "APPEND", "timeColumnName": "order_timestamp_seconds", "retentionTimeUnit": "DAYS", "retentionTimeValue": "365", "replication": "3", "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy" }, ...```
@elon.azoulay: ```"REALTIME": { "tableName": "enriched_orders_REALTIME", "tableType": "REALTIME", "segmentsConfig": { "schemaName": "enriched_orders", "segmentPushFrequency": "daily", "segmentPushType": "APPEND", "timeColumnName": "order_timestamp_seconds", "retentionTimeUnit": "DAYS", "retentionTimeValue": "100", "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy", "replicasPerPartition": "3" }, ...```
@elon.azoulay: and here is the fieldspec from the schema:
@elon.azoulay: ```"dateTimeFieldSpecs": [ { "name": "order_timestamp_seconds", "dataType": "LONG", "defaultNullValue": 0, "transformFunction": "toEpochSeconds(order_timestamp_ms)", "format": "1:MILLISECONDS:EPOCH", "granularity": "1:SECONDS" }```
@elon.azoulay: note that the source column `order_timestamp_ms` is not included in the table.
@mayanks: Hmm, why do you see time boundary in ms?
@elon.azoulay: Not sure:) I just did the curl from the broker
@elon.azoulay: and the value if converted to seconds is expected: it's the newest offline timestamp - 24 hours
@mayanks: Can you select min time from RT and max time for OFFLINE?
@elon.azoulay: there are realtime records newer than that
@elon.azoulay: Yep, already did:
@mayanks: what's the unit there? seconds or ms?
@elon.azoulay: ```--offline: min: 1601424056 max: 1612051039 --realtime: min: 1607816472 max: 1612317600 --time boundary in ms: 1611964639000```
@mayanks: Interesting
@elon.azoulay: the unit for the time column should be seconds - that's what it is when we select
@elon.azoulay: maybe the code expects that granularity unit == format unit?
@npawar: ```"transformFunction": "toEpochSeconds(order_timestamp_ms)", "format": "1:MILLISECONDS:EPOCH", "granularity": "1:SECONDS"``` this is incorrect ^. toEpochSeconds will convert it to millis/1000, but the format here says MILLISECONDS
@elon.azoulay: but granularity is seconds - it seems correct when we do selects on the datetime field (both realtime and offline)
@elon.azoulay: I thought format should be what the input is and granularity is what the output is, right?
@npawar: not really, they both represent what the final value in the time column should be
@npawar: granularity is not used anywhere atm
@npawar: in your case, millis will get converted using function millis/1000, so your format should be 1:SECONDS:EPOCH
@elon.azoulay: Oh wow, I didn't know that, thanks:) I tried using millis/1000 and got an error, but when I use the epoch* functions it works
@npawar: if you want to keep it in millis, but simply round to nearest seconds, use “round(order_timestamp, 1000)”
@npawar: if you do this ^, then your current datetimespec is correct
@elon.azoulay: thx!
@elon.azoulay: I think they want it in seconds - so should I just change format to `1:SECONDS:EPOCH`?
@npawar: yeah
@npawar: is this a new table? i’m wondering how any segments got created. the segment creation shoudve failed due to inconsistent value and spec
@elon.azoulay: No, it has tons of offline and realtime segments
@elon.azoulay: and they are correctly named as well
@elon.azoulay: not sure if that matters
@elon.azoulay: so right now we can consider granularity to be meaningless? i.e. unused anywhere?
@npawar: yes
@elon.azoulay: thanks for clearing that up @npawar!
@elon.azoulay: If I change the table spec now would it cause any issues?
@npawar: not quite sure about that..
@elon.azoulay: ok, we'll try it in staging :slightly_smiling_face: And let you know. Thanks again!
@pankaj: @pankaj has joined the channel
@bowenli86: @bowenli86 has joined the channel
#feat-geo-spatial-index
@aviv4339: @aviv4339 has joined the channel
#announcements
@edan: @edan has joined the channel
#presto-pinot-connector
@aviv4339: @aviv4339 has joined the channel
#getting-started
@victoranand: @victoranand has joined the channel
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
