Martin, I don’t know a lot about Hive. The log message you’re noting there is definitely coming from Hive’s client, but I can’t say whether or not it’s anything that should actually be concerning in terms of Java memory/heap. Is the NiFi heap size actually set to a max of 130 GB? (That would be a really massive heap for NiFi). Are you seeing any full garbage collections occurring? This information would be available in the Cluster table if you’re running clustered, or else in the Summary table (hamburger menu in the top-right -> Summary -> System Diagnostics in the bottom-right).
Thanks -Mark On May 11, 2020, at 5:20 AM, Hengesbach, Martin <[email protected]<mailto:[email protected]>> wrote: Hi, I’m using the PutHive3Streaming processor to write round about 30,000 avro records into a Hive table (round about 150 GB). This works for a while but the used memory is increasing over the time. And after some thousands records, the memory is full. When I remove the PutHive3Streaming processor from my flow, everything works OK. So, the problem must be the PutHive3Streaming processor. As Record Reader I’m using the AvroReader, the NiFi version is 1.11.4, Hive is version 3.1.0. The Hive Table is created as follow: create table yyyy(id string, data string) clustered by (id) into 50 buckets stored as orc tblproperties("transactional"="true") Any ideas if I make something wrong or is there really a memory leak in the processor? Thanks in advance and best regards Martin 2020-05-08 09:37:22,076 INFO [Timer-Driven Process Thread-44] o.a.h.streaming.HiveStreamingConnection Creating metastore client for streaming-connection 2020-05-08 09:37:22,091 INFO [Timer-Driven Process Thread-44] o.a.h.hive.metastore.HiveMetaStoreClient Trying to connect to metastore with URI thrift://xxxxxx:9083 2020-05-08 09:37:22,092 INFO [Timer-Driven Process Thread-44] o.a.h.hive.metastore.HiveMetaStoreClient Opened a connection to metastore, current connections: 1 2020-05-08 09:37:22,092 INFO [Timer-Driven Process Thread-44] o.a.h.hive.metastore.HiveMetaStoreClient Connected to metastore. 2020-05-08 09:37:22,093 INFO [Timer-Driven Process Thread-44] o.a.h.h.m.RetryingMetaStoreClient RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.metastore.HiveMetaStoreClient ugi=nifi (auth:SIMPLE) retries=24 delay=5 lifetime=0 2020-05-08 09:37:22,093 INFO [Timer-Driven Process Thread-44] o.a.h.streaming.HiveStreamingConnection Creating metastore client for streaming-connection-heartbeat 2020-05-08 09:37:22,093 INFO [Timer-Driven Process Thread-44] o.a.h.hive.metastore.HiveMetaStoreClient Trying to connect to metastore with URI thrift://xxxx:9083 2020-05-08 09:37:22,094 INFO [Timer-Driven Process Thread-44] o.a.h.hive.metastore.HiveMetaStoreClient Opened a connection to metastore, current connections: 2 2020-05-08 09:37:22,095 INFO [Timer-Driven Process Thread-44] o.a.h.hive.metastore.HiveMetaStoreClient Connected to metastore. 2020-05-08 09:37:22,095 INFO [Timer-Driven Process Thread-44] o.a.h.h.m.RetryingMetaStoreClient RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.metastore.HiveMetaStoreClient ugi=nifi (auth:SIMPLE) retries=24 delay=5 lifetime=0 2020-05-08 09:37:22,220 INFO [Timer-Driven Process Thread-44] o.a.h.streaming.HiveStreamingConnection STREAMING CONNECTION INFO: { metastore-uri: thrift://xxxxxxxx:9083, database: default, table: yyyy, partitioned-table: false, dynamic-partitioning: false, username: nifi, secure-mode: false, record-writer: HiveRecordWriter, agent-info: NiFi PutHive3Streaming [01711000-51db-1b1c-227a-f276a5410458] thread 152[Timer-Driven Process Thread-44] } 2020-05-08 09:37:22,220 INFO [Timer-Driven Process Thread-44] o.a.h.streaming.HiveStreamingConnection Starting heartbeat thread with interval: 150000 ms initialDelay: 64376 ms for agentInfo: NiFi PutHive3Streaming [01711000-51db-1b1c-227a-f276a5410458] thread 152[Timer-Driven Process Thread-44] 2020-05-08 09:37:22,232 INFO [Timer-Driven Process Thread-44] o.a.hive.streaming.AbstractRecordWriter Created new filesystem instance: 484197097 2020-05-08 09:37:22,232 INFO [Timer-Driven Process Thread-44] o.a.hive.streaming.AbstractRecordWriter Memory monitorings settings - autoFlush: true memoryUsageThreshold: 0.7 ingestSizeThreshold: 0 2020-05-08 09:37:22,232 INFO [Timer-Driven Process Thread-44] o.a.hadoop.hive.common.HeapMemoryMonitor Setting collection usage threshold to 2147483647 2020-05-08 09:37:22,233 WARN [Timer-Driven Process Thread-44] o.a.hive.streaming.AbstractRecordWriter LOW MEMORY ALERT! Tenured gen memory is already low. Increase memory to improve performance. Used: 129.54GB Max: 130.00GB 2020-05-08 09:37:22,233 INFO [Timer-Driven Process Thread-44] o.a.h.streaming.HiveStreamingConnection Opened new transaction batch TxnId/WriteIds=[292449/7336...292449/7336] on connection = { metaStoreUri: thrift://xxxxxxxx:9083, database: default, table: yyyy }; TxnStatus[O] LastUsed txnid:0 2020-05-08 09:37:22,272 INFO [Timer-Driven Process Thread-44] org.apache.orc.impl.PhysicalFsWriter ORC writer created for path: hdfs://hadoop1/tmp/nifi/yyyy/delta_0007336_0007336/bucket_00024 with stripeSize: 8388608 blockSize: 268435456 compression: NONE bufferSize: 32768 2020-05-08 09:37:22,292 INFO [Timer-Driven Process Thread-44] org.apache.orc.impl.WriterImpl ORC writer created for path: hdfs://hadoop1/tmp/nifi/yyyy/delta_0007336_0007336/bucket_00024 with stripeSize: 8388608 blockSize: 268435456 compression: NONE bufferSize: 32768 2020-05-08 09:37:22,420 INFO [Timer-Driven Process Thread-44] o.a.hive.streaming.AbstractRecordWriter Stats before flush: [record-updaters: 50, partitions: 1, buffered-records: 1 total-records: 1 buffered-ingest-size: 0B, total-ingest-size: 0B tenured-memory-usage: used/max => 129.54GB/130.00GB] 2020-05-08 09:37:22,420 INFO [Timer-Driven Process Thread-44] o.a.hive.streaming.AbstractRecordWriter Flushing record updater for partitions: default.yyyy 2020-05-08 09:37:22,578 INFO [Timer-Driven Process Thread-44] o.a.hive.streaming.AbstractRecordWriter Stats after flush: [record-updaters: 50, partitions: 1, buffered-records: 0 total-records: 1 buffered-ingest-size: 0B, total-ingest-size: 0B tenured-memory-usage: used/max => 129.54GB/130.00GB] 2020-05-08 09:37:22,582 INFO [Timer-Driven Process Thread-44] o.a.hive.streaming.AbstractRecordWriter Stats before close: [record-updaters: 50, partitions: 1, buffered-records: 0 total-records: 1 buffered-ingest-size: 0B, total-ingest-size: 0B tenured-memory-usage: used/max => 129.54GB/130.00GB] 2020-05-08 09:37:22,582 INFO [Timer-Driven Process Thread-44] o.a.hive.streaming.AbstractRecordWriter Closing updater for partitions: default.yyyy 2020-05-08 09:37:22,600 INFO [Timer-Driven Process Thread-44] o.a.hive.streaming.AbstractRecordWriter Stats after close: [record-updaters: 0, partitions: 1, buffered-records: 0 total-records: 1 buffered-ingest-size: 0B, total-ingest-size: 0B tenured-memory-usage: used/max => 129.54GB/130.00GB] 2020-05-08 09:37:22,600 INFO [Timer-Driven Process Thread-44] o.a.h.hive.metastore.HiveMetaStoreClient Closed a connection to metastore, current connections: 1 2020-05-08 09:37:22,600 INFO [Timer-Driven Process Thread-44] o.a.h.hive.metastore.HiveMetaStoreClient Closed a connection to metastore, current connections: 0 2020-05-08 09:37:22,600 INFO [Timer-Driven Process Thread-44] o.a.h.streaming.HiveStreamingConnection Closed streaming connection. Agent: NiFi PutHive3Streaming [01711000-51db-1b1c-227a-f276a5410458] thread 152[Timer-Driven Process Thread-44] Stats: {records-written: 1, records-size: 0, committed-transactions: 1, aborted-transactions: 0, auto-flushes: 0, metastore-calls: 7 } At the beginning the memory starts with Used: 2.63GB Max: 130.00GB ________________________________ FIZ Karlsruhe - Leibniz-Institut für Informationsinfrastruktur GmbH. Sitz der Gesellschaft: Eggenstein-Leopoldshafen, Amtsgericht Mannheim HRB 101892. Geschäftsführerin: Sabine Brünger-Weilandt. Vorsitzende des Aufsichtsrats: MinDirig'in Dr. Angelika Willms-Herget. FIZ Karlsruhe ist zertifiziert mit dem Siegel "audit berufundfamilie".
