#general
@karthik.chopperla: @karthik.chopperla has joined the channel
@kautsshukla: Hi All, everyone my pinot server keep getting lost on restart, have kept JVM 16GB and table [“loadMode”: “MMAP”] . Screenshot
@santosh.reddy: @santosh.reddy has joined the channel
#random
@karthik.chopperla: @karthik.chopperla has joined the channel
@santosh.reddy: @santosh.reddy has joined the channel
#troubleshooting
@karthik.chopperla: @karthik.chopperla has joined the channel
@humengyuk18: I got the following exception when trying to consume from kafka using high level consumer, I checked the configured `stream.kafka.hlc.bootstrap.server`, `stream.kafka.hlc.zk.connect.string`, `stream.kafka.zk.broker.url` is correct, but it’s not consuming. ```2021/05/21 13:59:59.025 WARN [PinotRealtimeSegmentManager] [
@mayanks: Curious why you want to use high level consumer?
@humengyuk18: Because low level consumer will create one segment per partition, this will cause many small segments for small datasources, so I think high level consumer will be a better fit?
@mayanks: HLC is not scalable, it has the reverse problem where every node has to consume all
@mayanks: For small data source why do you have too many partitions?
@humengyuk18: We have a uniformed partition number for all topics, we don’t know for sure what the traffic will be before creating that topic.
@mayanks: Hmm. You can still use low level and have segment open for longer to avoid too many small segments
@humengyuk18: I see, thanks.
@kautsshukla:
@mayanks: Do you have inverted index or on-heap dictionary specified ?
@mayanks: In table config
@kautsshukla: @mayanks It looks like more than a GC issue, in logs everytime its lossing connection and reconnecting to zookeeper
@mayanks: That is because of GC pause, it times out to send heart beat to ZK
@kautsshukla: @mayanks Does it creates multiple connection can hit Zookeeper IOPS on call for every segemnt check.
@mayanks: No, should be single session per server
@kautsshukla: @mayanks ```"invertedIndexColumns": [ "userid", "sessionid", "eventlabel", "dp_created_at", "timestampist" ]```
@kautsshukla: ```"sortedColumn": [ "dp_created_at", "timestampist" ],```
@mayanks: Any On-heap dictionary?
@kautsshukla: ```"autoGeneratedInvertedIndex": true, "createInvertedIndexDuringSegmentGeneration": true, "enableDefaultStarTree": true, "enableDynamicStarTreeCreation": true,```
@mayanks: If not, I can’t think what is occupying heap. Metadata cannot take 64GB
@mayanks: Oh segment generation takes heap
@mayanks: Are too many segments being generated in parallel?
@kautsshukla: 50 segemnts in parallel
@mayanks: There you go
@kautsshukla: is this is the culprit
@kautsshukla: ```"createInvertedIndexDuringSegmentGeneration": true, ```
@mayanks: No
@mayanks: Periodically segments consumed in memory are flushed to disk. This goes through some heap usage. If 50 partitions go through the same at once it will run out of heap
@mayanks: How did you specify 50?
@kautsshukla: kafka topic has 50 partitions
@kautsshukla: I haven’t specified its already existed
@mayanks: Not talking about consumption. Periodically the consuming segment needs to be flushed to disk, this uses some heap. Typically if all partitions flush time disk at the same time then there will be heap pressure
@mayanks: There is a way to specify max parallel segment generation
@kautsshukla: @mayanks: how to specify it.
@mayanks: I’ll find. In the meanwhile can you grip the log for segment generation
@kautsshukla: what i have to grep ?
@mayanks: Try something like "grep -i created segment"
@kautsshukla: post MessageLatencyMonitor it always throw ERROR [SegmentBuildTimeLeaseExtender] [pool-5-thread-1] Failed to send lease extension
@mayanks: Yeah, then it is likely segment generation
@mayanks: grep -i "Driver, indexing time :"
@kautsshukla: no log with this
@mayanks: In server's data dir do ls -l and see if segments have timestamp that are near by
@mayanks: that will tell how many generated at the same time
@mayanks: There should have been logs
@mayanks: `grep -i "Trying to build segment"`
@mayanks: or `grep -i "Successfully built segment"`
@kautsshukla: last segment created at May 22 14:08 UTC
@mayanks: how many around that time?
@kautsshukla: no luck with this grep -i “Successfully built segment”
@kautsshukla: no logs on both the server with this greo
@kautsshukla: 7:38 pm IST
@mayanks: Hmm, what is uyour logging level? These are info messages and should be there for sure.
@mayanks: Also config `realtime.max.parallel.segment.builds` to specify how many segment generation in parallel
@kautsshukla: logging level is INFO only
@kautsshukla: realtime.max.parallel.segment.builds ? where I should mention this
@kautsshukla: Table config ?
@mayanks: In server config
@mayanks: Although, I'd think that the default should not be unlimited, so still unsure if this is the root cause
@mayanks: Can you try setting it to a small value like 4.
@mayanks: Also look at
@mayanks: If your current segment size is 1.8GB, reducing its to 112M would increase number of segments too much. May be 300MB or 500MB
@santosh.reddy: @santosh.reddy has joined the channel
@santosh.reddy: Hi I am trying to run pinot as a cluster mode, but is there any way to push all logs of controller, broker, zkp and server to /var/log/pinot ? and also want to apply log rotation of it, please help me with refer docs where i am running below commands
@santosh.reddy: ```#StartZookeeper bin/pinot-admin.sh StartZookeeper -zkPort 2181 2>&1 #start Controller bin/pinot-admin.sh StartController -zkAddress localhost:2181 -clusterName pinot-dev-cluster -controllerPort 9000 2>&1 # start Broker bin/pinot-admin.sh StartBroker \ -zkAddress localhost:2181 \ -clusterName pinot-dev-cluster \ -brokerPort 7000 #start Server bin/pinot-admin.sh StartServer -zkAddress localhost:2181 -clusterName pinot-dev-cluster -serverPort 8001 -serverAdminPort 8002```
@mayanks: You can use quick-start if you are just playing, it will have single log file if you use the latest code from master (0.7.1 does not have that fix).
@santosh.reddy: what is the recommended version to use
@mayanks: 0.7.1 is the official release. But if you are playing you can use master
@santosh.reddy: currently we are doing poc on it
@mayanks: Ok, then stick to official version.
@mayanks: You can play with log4j settings to direct your logs. But since you are starting separate jvm’s, you can’t use the same output file
#feat-partial-upsert
@qiaochu: hello @jackie.jxt i used the lastest master branch and run quick-start-streaming.sh demo, and observed the following error. is it possible there is a potential bug in the RealtimeQuickStart producer? ```➜ apache-pinot-incubating-0.8.0-SNAPSHOT-bin git:(master) bin/quick-start-streaming.sh ***** Starting Kafka ***** ***** Starting meetup data stream and publishing to Kafka ***** ***** Starting Zookeeper, controller, server and broker ***** May 21, 2021 7:06:37 PM org.glassfish.grizzly.http.server.NetworkListener start INFO: Started listener bound to [0.0.0.0:9000] May 21, 2021 7:06:37 PM org.glassfish.grizzly.http.server.HttpServer start INFO: [HttpServer] Started. May 21, 2021 7:06:46 PM org.glassfish.grizzly.http.server.NetworkListener start INFO: Started listener bound to [0.0.0.0:8000] May 21, 2021 7:06:46 PM org.glassfish.grizzly.http.server.HttpServer start INFO: [HttpServer-1] Started. May 21, 2021 7:06:53 PM org.glassfish.grizzly.http.server.NetworkListener start INFO: Started listener bound to [0.0.0.0:7500] May 21, 2021 7:06:53 PM org.glassfish.grizzly.http.server.HttpServer start INFO: [HttpServer-2] Started. ***** Bootstrap meetupRSVP table ***** ***** Waiting for 5 seconds for a few events to get populated ***** ***** Realtime quickstart setup complete ***** Total number of documents in the table Query : select count(*) from meetupRsvp limit 1 Exception caught: java.lang.NullPointerException: null at org.apache.pinot.tools.Quickstart.prettyPrintResponse(Quickstart.java:113) ~[pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-08b909c45e85f9bf8d8659561a2d13b4cc443ebc] at org.apache.pinot.tools.RealtimeQuickStart.execute(RealtimeQuickStart.java:111) ~[pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-08b909c45e85f9bf8d8659561a2d13b4cc443ebc] at org.apache.pinot.tools.admin.command.QuickStartCommand.execute(QuickStartCommand.java:147) ~[pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-08b909c45e85f9bf8d8659561a2d13b4cc443ebc] at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:166) [pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-08b909c45e85f9bf8d8659561a2d13b4cc443ebc] at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:186) [pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-08b909c45e85f9bf8d8659561a2d13b4cc443ebc] at org.apache.pinot.tools.RealtimeQuickStart.main(RealtimeQuickStart.java:50) [pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-08b909c45e85f9bf8d8659561a2d13b4cc443ebc]```
@qiaochu:
@yupeng: Is this from a fresh master checkout?
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@pinot.apache.org For additional commands, e-mail: dev-h...@pinot.apache.org
