#general
@santhosh: Hi. We have a usecase to store the incoming user events. We have multiple dimensions where we want to query on. We want to use S3 as deep storage. We also have requirements like, the last half hour data will be queried on frequently like a hot shard. Can we use pinot for this use case? How can we model data optimally for this kind of use case? Do we have data retention support where data older than that can be removed after some time?
@mayanks: Yes, this seems like a good fit for Pinot. and yes Pinot has retention support.
@ravi.ranjan: @ravi.ranjan has joined the channel
@jmeyer: Hello :slightly_smiling_face: Is there an "official" / commonly used Grafana Dashboard for Pinot ? (I've found this one ->
@kchavda: @kchavda has joined the channel
@karinwolok1: Welcome new Apache Pinot community members! :wave: We're so happy you're here and joining the movement of speeeeeed for user-facing analytics. :wine_glass: Please take a moment, if you haven't already - to introduce yourself and tell us what brought you here! :smiley: @kchavda @ravi.ranjan @sheetalarun.kadam2 @tenghuanhe @santhosh @fritzbx @kaushikf9t @ragiroux @ramabaratam @cgoddard @bryagd @mercyshans @kkmagic99 @saurabhd336 @bcbazevedo @ming.liu @kanleecarro @hari.prasanna @richard.hallier @bowenwan @sharma.vinit @anusha.munukuntla @kylebanker @hongtaozhang @krishnalumia535 @kaustabhganguly
@jai.patel856: For an upsert table I have the order columns: timeColumnName set to my updated_at timestamp. It used to be created_at when I was using an offline-only table. I believe this is the correct change. My question is for the sortedcolumn index, do I need to change it too? For my use case I generally still want to be sorting on created_at. But does upsert required the sortedcolumn be the same as the timecolumn?
@jai.patel856: @g.kishore @yupeng
@jackie.jxt: No, you can keep the current sort column
@jai.patel856: sweet
#random
@ravi.ranjan: @ravi.ranjan has joined the channel
@kchavda: @kchavda has joined the channel
#troubleshooting
@patidar.rahul8392: Hi, I have one hybrid table in pinot in tablename_OFFLINE table I have 250000 records and tablename_realtime I have total 2000 records. When I am doing select count from table it showing 252000 that is correct count. But when I am doing select * from tablename in superset for the same query. It's below error message.
@patidar.rahul8392:
@patidar.rahul8392: @fx19880617 @ken @jackie.jxt kindly suggest is there any row limit in pinot.?
@patidar.rahul8392: This is the complete log
@jackie.jxt: Pinot does not have this limit. Seems thrown from presto connector @fx19880617
@fx19880617: the log says the limit is 50k for select * query:
@fx19880617: you can increase it from presto config
@fx19880617: default is 50k
@ravi.ranjan: @ravi.ranjan has joined the channel
@patidar.rahul8392: In prestro config file I am not giving any limit .I have only these properties.
@patidar.rahul8392: If I am doing select from any other tables .i.e. hive or druid I am able to select data .but in case of pinot in throwing this error. Some problem with this prestro-pinot connector.kindly suggest if is there any why to select data directly from pinot without prestro connector or any way to increase this limit. @fx19880617 @jackie.jxt
@fx19880617: check pinot catalog config : ```pinot.limit-large-for-segment=2147483647```
@fx19880617: wait, are you using prestodb or trino?
@fx19880617: for trino,set ```pinot.max-rows-per-split-for-segment-queries=2147483647```
@patidar.rahul8392: It's prestrodb @fx19880617 version 350
@fx19880617: if 350, then it’s trino
@patidar.rahul8392: Ohh ok in logs everywhere I am getting prestrosql so I thought it's prestrodb. Ok let me try with trino property then. Thanks alot @fx19880617
@patidar.rahul8392: @fx19880617 thanks alot it worked. One update had to decrease the max row limit from Int max size value i.e. 2147483647 otherwise it was taking the minimum value of Int i.e. -2147483648 and again showing limit error.
@patidar.rahul8392: In case of max value of integer , it was showing this issue
@fx19880617: ok
@fx19880617: then you can make it a bit smaller I think
@fx19880617: I think 2Billion is fine as well
@fx19880617: As long as it’s bigger than your max docs of one single segment
@patidar.rahul8392: Ok @fx19880617
@pedro.cls93: Hello, does Pinot support some auto-scaling of some sort to deal with increasingly larger and heavier workloads? I have a single real-time table consuming events from kafka (this is a wide table but not many fields, currently there are 39, mostly strings, one of which has a max length of 2147483647 (INTEGER.MAX_VALUE) since it holds a json blob). My pinot cluster is deployed in Kubernetes (hosted in azure) 2 pinot server instances with 5GB heap + 3GB for direct memory, 100GB persistance volume (segment deepstorage is configured) with a k8s memory limit of 10G. 1 controller instance with 1GB heap for JVM, k8s memory limit 2G. 1 broker instance with 4GB heap, k8s memory limit 5G. My servers are crashing with segment faults & OOM, as follows: Server 1: ```# # A fatal error has been detected by the Java Runtime Environment: # # SIGBUS (0x7) at pc=0x00007f4b79052422, pid=8, tid=0x00007f4ae8739700 # # JRE version: OpenJDK Runtime Environment (8.0_292-b10) (build 1.8.0_292-b10) # Java VM: OpenJDK 64-Bit Server VM (25.292-b10 mixed mode linux-amd64 compressed oops) # Problematic frame: # v ~StubRoutines::jbyte_disjoint_arraycopy # # Core dump written. Default location: /opt/pinot/core or core.8 # [thread 139959708407552 also had an error] # An error report file with more information is saved as: # /opt/pinot/hs_err_pid8.log # # If you would like to submit a bug report, please visit: #
@mayanks: Pinot does support scaling, but it is not fully auto at the moment. You can add server nodes and invoke rebalance api
@mayanks: You seem to be running into `java.lang.OutOfMemoryError: Java heap space`
@pedro.cls93: I'm injecting 33M records into the table, this is something like 15GB in parquet. However the 100GB of disk in server 1 + 76GB in server 2 are used up. With 4.6M records, pinot reports 56GB in uncompressed table data. Is this normal?
@mayanks: 56GB of heap?
@pedro.cls93: in `Reported Size` as reported by the table summary UI in Pinot. I don't know if it is heap or disk.
@mayanks: I am guessing you used raw index for the string columns?
@pedro.cls93: The field with maxLength: 2147483647 is json indexed. Everything else is the default
@pedro.cls93: This is the table indexing config: ```"tableIndexConfig": { "invertedIndexColumns": [], "rangeIndexColumns": [], "jsonIndexColumns": [ "inputForUiControls" ], "autoGeneratedInvertedIndex": false, "createInvertedIndexDuringSegmentGeneration": false, "bloomFilterColumns": [], "loadMode": "MMAP", "streamConfigs": { "streamType": "kafka", "stream.kafka.topic.name": "test.data.hitexecutionview_3mo", "stream.kafka.broker.list": "dckafka.dc-kafka.svc.cluster.local:9092", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.consumer.prop.auto.offset.reset": "smallest", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder", "realtime.segment.flush.threshold.rows": "1600000" }, "noDictionaryColumns": [], "onHeapDictionaryColumns": [], "varLengthDictionaryColumns": [], "enableDefaultStarTree": false, "enableDynamicStarTreeCreation": false, "segmentPartitionConfig": { "columnPartitionMap": { "externalHitExecutionId": { "functionName": "Murmur", "numPartitions": 16 } } }```
@mayanks: By default string columns are padded to make them equal length in the dictionary. If you did not explicitly set no-dict column, then my guess is that is where the space is being spent. Do you have access to `metadata.properties` file inside the segment directory on the server? If so, can you share?
@mayanks: yeah `"noDictionaryColumns": [],`
@mayanks: can you share metadata.properties file?
@pedro.cls93: let me check
@pedro.cls93: I can't seem to find the metadata.properties file, where would it be found by default?
@mayanks: One the server's dataDir (where it stores segments), go inside one of the segment dir
@pedro.cls93: any segment will do?
@mayanks: Sure. Try to pick biggest one
@pedro.cls93: Server 1 is crashed, can't recover it but in server 2 all metadata.properties files are 35k, here is an example of the content:
@mayanks: which column has the INT_MAX length?
@pedro.cls93: inputForUiControls
@mayanks: `column.inputForUiControls.lengthOfEachEntry = 33655`
@mayanks: Your segment has 200k rows. With this padded string, the dictionary for this column becomes 6,7GB
@pedro.cls93: This is the schema:
@pedro.cls93: Is that 6,7G per segment?
@mayanks: 6.7G for this one segment
@mayanks: what's the disk size you see for this segment (du -sh)
@pedro.cls93: 4.0G
@mayanks: hmm
@pedro.cls93: ```root@pinot-server-1:/var/pinot/server/data# du -sh ./index/HitExecutionView_3mo_REALTIME/HitExecutionView_3mo__5__3__20210608T1551Z/v3/ 4.0G ./index/HitExecutionView_3mo_REALTIME/HitExecutionView_3mo__5__3__20210608T1551Z/v3/```
@mayanks: There are a few things to do, since you are running out of heap
@pedro.cls93: Also space in server 0: ```Filesystem Size Used Avail Use% Mounted on /dev/sdb 93G 93G 99M 100% /var/pinot/server/data```
@mayanks: Where is the space being used? All in segments?
@pedro.cls93: Supposedly, yes, can't access the pod it's crashed
@mayanks: How did you get the metadata then?
@pedro.cls93: I accessed the other server, I have 2 (server 0 is crashed, server 1 is ok)
@pedro.cls93: not the same volumes
@ssubrama: from the stack, it seems that they are runin gout of heap during segment build
@mayanks: Closing the thread here - one string column had huge variations in length of entries and almost 99% of the 4GB segment size was dictionary padding. Converting it to no-dictionary column reduced the size by an order of magnitude.
@kchavda: @kchavda has joined the channel
#pinot-dev
@ragiroux: @ragiroux has joined the channel
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@pinot.apache.org For additional commands, e-mail: dev-h...@pinot.apache.org
