vineethvp opened a new issue, #13335: URL: https://github.com/apache/pinot/issues/13335
Data is ingested from Kafka to realtime table with below config. It is having replication factor 2. **Table config** "instanceAssignmentConfigMap": { "CONSUMING": { "tagPoolConfig": { "tag": "DefaultTenant_REALTIME" }, "replicaGroupPartitionConfig": { "numInstances": 3 } } "tableIndexConfig": { "loadMode": "MMAP", "streamConfigs": { "streamType": "kafka", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "realtime.segment.flush.threshold.rows": "0", "realtime.segment.flush.threshold.time": "24h", "realtime.segment.flush.threshold.segment.size": "300M", "stream.kafka.consumer.prop.auto.offset.reset": "largest" } }, **Getting below error after inserting around 60M records, and server is crashing.** Consumed 5722 events from (rate:63.164402/s), currentOffset=389448757, numRowsConsumedSoFar=257283, numRowsIndexedSoFar=257283 Consumed 3216 events from (rate:37.62988/s), currentOffset=376002707, numRowsConsumedSoFar=252239, numRowsIndexedSoFar=252239 Consumed 4045 events from (rate:46.583138/s), currentOffset=376940305, numRowsConsumedSoFar=262888, numRowsIndexedSoFar=262888 Consumed 2880 events from (rate:40.99819/s), currentOffset=372932263, numRowsConsumedSoFar=246467, numRowsIndexedSoFar=246467 Consumed 3625 events from (rate:46.28802/s), currentOffset=381415112, numRowsConsumedSoFar=249777, numRowsIndexedSoFar=249777 Consumed 2898 events from (rate:35.928143/s), currentOffset=374420450, numRowsConsumedSoFar=252748, numRowsIndexedSoFar=252748 Slow query: request handler processing time: 6786, send response latency: 7946, total time to handle request: 14732 Consumed 3757 events from (rate:38.51003/s), currentOffset=383381895, numRowsConsumedSoFar=254725, numRowsIndexedSoFar=254725 Client session timed out, have not heard from server in 20317ms for session id 0x2000fcc5a450005 Session 0x2000fcc5a450005 for server pinot-zookeeper/172.20.29.227:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. org.apache.zookeeper.ClientCnxn$SessionTimeoutException: Client session timed out, have not heard from server in 20317ms for session id 0x2000fcc5a450005 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1242) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-1d1d25dc0f1fc1abb73d9516414168c82b116b58] zkclient 3, zookeeper state changed ( Disconnected ) [Consumer clientId=events_REALTIME-CENTRAL-STREAMING-NW-EVENTS-21, groupId=null] Error sending fetch request (sessionId=214284632, epoch=84) to node 1: org.apache.pinot.shaded.org.apache.kafka.common.errors.DisconnectException: null Opening socket connection to server pinot-zookeeper/172.20.29.227:2181. SASL config status: Will not attempt to authenticate using SASL (unknown error) Timed out while polling results block, numBlocksMerged: 0 (query: QueryContext{_tableName='events_REALTIME', _subquery=null, _selectExpressions=[channel_util, cpu_util, customer_id, data, data_source, device_id, device_type, event_type, fan_id, fan_status, is_stack_switch, mac_address, mem_util, member_id, noise_floor, power_supply_id, power_supply_status, radio_band, radio_type, sensor_id, sensor_temperature, sensor_temperature_trend, site_id, stack_id, status, sub_account_id, sub_site_id, timestamp, trend_value, upload_timestamp, upload_ts_millis, uptime, uptime_string], _distinct=false, _aliasList=[null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null], _filter=null, _groupByExpressions=null, _havingFilter=null, _orderByExpressions=null, _limit=10000, _offset=0, _queryOptions={timeoutMs=10000}, _expressionOverrideHints={}, _explain=false }) Consumed 500 events from (rate:7.955576/s), currentOffset=373866452, numRowsConsumedSoFar=257550, numRowsIndexedSoFar=257550 Consumed 1735 events from (rate:15.82827/s), currentOffset=376659150, numRowsConsumedSoFar=252384, numRowsIndexedSoFar=252384 Consumed 3230 events from (rate:27.967546/s), currentOffset=374117225, numRowsConsumedSoFar=248670, numRowsIndexedSoFar=248670 Socket connection established, initiating session, client: /10.3.145.54:33952, server: pinot-zookeeper/172.20.29.227:2181 Consumed 250 events from (rate:3.7271154/s), currentOffset=380505319, numRowsConsumedSoFar=237006, numRowsIndexedSoFar=237006 Exception in thread "events__18__5__20240607T0718Z" java.lang.OutOfMemoryError: Java heap space Exception in thread "events__29__5__20240607T0719Z" java.lang.OutOfMemoryError: Java heap space Consumed 1000 events from (rate:8.947665/s), currentOffset=383012407, numRowsConsumedSoFar=250268, numRowsIndexedSoFar=250268 Exception in thread "req-rsp-timeout-task" java.lang.OutOfMemoryError: Java heap space at jdk.httpserver/sun.net.httpserver.ServerImpl$ReqRspTimeoutTask.run(ServerImpl.java:1026) at java.base/java.util.TimerThread.mainLoop(Timer.java:556) at java.base/java.util.TimerThread.run(Timer.java:506) [Consumer clientId=events_REALTIME-CENTRAL-STREAMING-NW-EVENTS-21, groupId=null] Error sending fetch request (sessionId=214284632, epoch=INITIAL) to node 1: org.apache.pinot.shaded.org.apache.kafka.common.errors.TimeoutException: Failed to send request after 30000 ms. Session 0x2000fcc5a450005 for server pinot-zookeeper/172.20.29.227:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. org.apache.zookeeper.ClientCnxn$EndOfStreamException: Unable to read additional data from server sessionid 0x2000fcc5a450005, likely server has closed socket at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-1d1d25dc0f1fc1abb73d9516414168c82b116b58] at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-1d1d25dc0f1fc1abb73d9516414168c82b116b58] at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1274) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-1d1d25dc0f1fc1abb73d9516414168c82b116b58] **Server config** resources: requests: memory: "300Mi" cpu: "300m" limits: memory: "4000Mi" cpu: "4000m" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org