[ https://issues.apache.org/jira/browse/APEXMALHAR-2557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vlad Rozov updated APEXMALHAR-2557: ----------------------------------- Description: While using the TimeBasedDedupOperator for deduping, I see that operator keeps failing with below NullPointer exception. I also see that operator is always high on CPU usages. Almost reaching 100%. No matter the setting any values for vcores or container memory for operator {noformat} 2018-03-22 15:10:10,037 INFO stram.FSRecoveryHandler (FSRecoveryHandler.java:rotateLog(103)) - Creating hdfs://littleredns/user/SVDATHDP/datatorrent/apps/application_1519410901484_187748/recovery/log 2018-03-22 15:10:10,056 INFO stram.StreamingContainerParent (StreamingContainerParent.java:log(170)) - child msg: Stopped running due to an exception. java.lang.NullPointerException at org.apache.hadoop.io.file.tfile.TFile$Writer.append(TFile.java:387) at com.datatorrent.lib.fileaccess.TFileWriter.append(TFileWriter.java:66) at org.apache.apex.malhar.lib.state.managed.BucketsFileSystem.writeBucketData(BucketsFileSystem.java:179) at org.apache.apex.malhar.lib.state.managed.IncrementalCheckpointManager.transferWindowFiles(IncrementalCheckpointManager.java:139) at org.apache.apex.malhar.lib.state.managed.IncrementalCheckpointManager$1.run(IncrementalCheckpointManager.java:110) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) context: PTContainer[id=3(container_e3125_1519410901484_187748_01_000012),state=ACTIVE,operators=[PTOperator[id=3,name=dedupeOperator,state=ACTIVE]]] 2018-03-22 15:10:10,915 WARN stram.StreamingContainerManager (StreamingContainerManager.java:processOperatorFailure(1439)) - Operator failure: PTOperator[id=3,name=dedupeOperator,state=INACTIVE] count: 1 {noformat} was: While using the TimeBasedDedupOperator for deduping, I see that operator keeps failing with below NullPointer exception. I also see that operator is always high on CPU usages. Almost reaching 100%. No matter the setting any values for vcores or container memory for operator {noformat} 2018-03-22 15:10:10,037 INFO stram.FSRecoveryHandler (FSRecoveryHandler.java:rotateLog(103)) - Creating hdfs://littleredns/user/SVDATHDP/datatorrent/apps/application_1519410901484_187748/recovery/log 2018-03-22 15:10:10,056 INFO stram.StreamingContainerParent (StreamingContainerParent.java:log(170)) - child msg: Stopped running due to an exception. java.lang.NullPointerException at org.apache.hadoop.io.file.tfile.TFile$Writer.append(TFile.java:387) at com.datatorrent.lib.fileaccess.TFileWriter.append(TFileWriter.java:66) at org.apache.apex.malhar.lib.state.managed.BucketsFileSystem.writeBucketData(BucketsFileSystem.java:179) at org.apache.apex.malhar.lib.state.managed.IncrementalCheckpointManager.transferWindowFiles(IncrementalCheckpointManager.java:139) at org.apache.apex.malhar.lib.state.managed.IncrementalCheckpointManager$1.run(IncrementalCheckpointManager.java:110) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) context: PTContainer[id=3(container_e3125_1519410901484_187748_01_000012),state=ACTIVE,operators=[PTOperator[id=3,name=dedupeOperator,state=ACTIVE]]] 2018-03-22 15:10:10,915 WARN stram.StreamingContainerManager (StreamingContainerManager.java:processOperatorFailure(1439)) - Operator failure: PTOperator[id=3,name=dedupeOperator,state=INACTIVE] count: 1 {noformat} > TimeBasedDedupOperator fails with NullPointer > ----------------------------------------------- > > Key: APEXMALHAR-2557 > URL: https://issues.apache.org/jira/browse/APEXMALHAR-2557 > Project: Apache Apex Malhar > Issue Type: Bug > Affects Versions: 3.8.0 > Reporter: Vivek Bhide > Priority: Major > > While using the TimeBasedDedupOperator for deduping, I see that operator > keeps failing with below NullPointer exception. > I also see that operator is always high on CPU usages. Almost reaching 100%. > No matter the setting any values for vcores or container memory for operator > {noformat} > 2018-03-22 15:10:10,037 INFO stram.FSRecoveryHandler > (FSRecoveryHandler.java:rotateLog(103)) - Creating > hdfs://littleredns/user/SVDATHDP/datatorrent/apps/application_1519410901484_187748/recovery/log > > 2018-03-22 15:10:10,056 INFO stram.StreamingContainerParent > (StreamingContainerParent.java:log(170)) - child msg: Stopped running due to > an exception. java.lang.NullPointerException > at > org.apache.hadoop.io.file.tfile.TFile$Writer.append(TFile.java:387) > at > com.datatorrent.lib.fileaccess.TFileWriter.append(TFileWriter.java:66) > at > org.apache.apex.malhar.lib.state.managed.BucketsFileSystem.writeBucketData(BucketsFileSystem.java:179) > > at > org.apache.apex.malhar.lib.state.managed.IncrementalCheckpointManager.transferWindowFiles(IncrementalCheckpointManager.java:139) > > at > org.apache.apex.malhar.lib.state.managed.IncrementalCheckpointManager$1.run(IncrementalCheckpointManager.java:110) > > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > > at java.lang.Thread.run(Thread.java:745) > context: > PTContainer[id=3(container_e3125_1519410901484_187748_01_000012),state=ACTIVE,operators=[PTOperator[id=3,name=dedupeOperator,state=ACTIVE]]] > > 2018-03-22 15:10:10,915 WARN stram.StreamingContainerManager > (StreamingContainerManager.java:processOperatorFailure(1439)) - Operator > failure: PTOperator[id=3,name=dedupeOperator,state=INACTIVE] count: 1 > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)