Hi Bhupesh,
The exception occurred immediately. Rather operator didn't even initialized
completely and failed before that causing all further operators to stuck in
pending deploy state. Regarding properties file, as i said, there was no
configuration with numBucket in config file and only config for dedup
operator was below
<property>
<name>dt.application.DataUsageIngest.operator.dedupeOperator.port.input.attr.TUPLE_CLASS</name>
<value>com.tgt.dqs.datausageingest.object.DataSetAttributeWithChecksum</value>
</property>
<property>
<name>dt.application.DataUsageIngest.operator.dedupeOperator.prop.keyExpression</name>
<value>checksum</value>
</property>
Below are the full container logs for dedup operator
2017-06-08 17:52:03,491 INFO engine.StreamingContainer
(StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending request.
2017-06-08 17:52:03,996 INFO engine.StreamingContainer
(StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending request.
2017-06-08 17:52:04,500 INFO engine.StreamingContainer
(StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending request.
2017-06-08 17:52:05,004 INFO engine.StreamingContainer
(StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending request.
2017-06-08 17:52:05,512 INFO engine.StreamingContainer
(StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending request.
2017-06-08 17:52:06,016 INFO engine.StreamingContainer
(StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending request.
2017-06-08 17:52:06,519 INFO engine.StreamingContainer
(StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending request.
2017-06-08 17:52:07,023 INFO engine.StreamingContainer
(StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending request.
2017-06-08 17:52:07,527 INFO engine.StreamingContainer
(StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending request.
2017-06-08 17:52:08,030 INFO engine.StreamingContainer
(StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending request.
2017-06-08 17:52:08,535 INFO engine.StreamingContainer
(StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending request.
2017-06-08 17:52:09,039 INFO engine.StreamingContainer
(StreamingContainer.java:heartbeatLoop(711)) - Waiting for pending request.
2017-06-08 17:52:09,340 INFO server.Server (Server.java:onMessage(599)) -
Received subscriber request: SubscribeRequestTuple{version=1.0,
identifier=tcp://brdn1251.target.com:38212/5.unique.1,
windowId=ffffffffffffffff, type=uniquMessages/6.inputPort,
upstreamIdentifier=5.unique.1, mask=0, partitions=null, bufferSize=1024}
2017-06-08 17:52:09,558 INFO engine.StreamingContainer
(StreamingContainer.java:processHeartbeatResponse(825)) - Deploy request:
[OperatorDeployInfo[id=5,name=dedupeOperator,type=GENERIC,checkpoint={ffffffffffffffff,
0,
0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=checkDuplicates,sourceNodeId=3,sourcePortName=dedupePort,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=unique,streamId=uniquMessages,bufferServer=brdn1251.target.com]]]]
2017-06-08 17:52:09,672 INFO server.Server (Server.java:onMessage(555)) -
Received publisher request: PublishRequestTuple{version=1.0,
identifier=5.unique.1, windowId=ffffffffffffffff}
2017-06-08 17:52:10,128 ERROR engine.StreamingContainer
(StreamingContainer.java:run(1456)) - Operator set
[OperatorDeployInfo[id=5,name=dedupeOperator,type=GENERIC,checkpoint={ffffffffffffffff,
0,
0},inputs=[OperatorDeployInfo.InputDeployInfo[portName=input,streamId=checkDuplicates,sourceNodeId=3,sourcePortName=dedupePort,locality=<null>,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=unique,streamId=uniquMessages,bufferServer=brdn1251.target.com]]]]
stopped running due to an exception.
java.lang.IllegalArgumentException: bucket conflict -27877 27877
at
org.apache.apex.malhar.lib.state.managed.AbstractManagedStateImpl.handleBucketConflict(AbstractManagedStateImpl.java:320)
at
org.apache.apex.malhar.lib.state.managed.AbstractManagedStateImpl.prepareBucket(AbstractManagedStateImpl.java:269)
at
org.apache.apex.malhar.lib.state.managed.AbstractManagedStateImpl.getValueFromBucketAsync(AbstractManagedStateImpl.java:305)
at
org.apache.apex.malhar.lib.state.managed.ManagedTimeStateImpl.getAsync(ManagedTimeStateImpl.java:76)
at
org.apache.apex.malhar.lib.dedup.BoundedDedupOperator.getAsyncManagedState(BoundedDedupOperator.java:151)
at
org.apache.apex.malhar.lib.dedup.AbstractDeduper.processTuple(AbstractDeduper.java:163)
at
org.apache.apex.malhar.lib.dedup.BoundedDedupOperator$1.process(BoundedDedupOperator.java:91)
at com.datatorrent.api.DefaultInputPort.put(DefaultInputPort.java:79)
at
com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.sweep(BufferServerSubscriber.java:288)
at com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:269)
at
com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1428)
2017-06-08 17:52:11,691 INFO engine.StreamingContainer
(StreamingContainer.java:processHeartbeatResponse(808)) - Undeploy request:
[5]
2017-06-08 17:52:11,692 INFO engine.StreamingContainer
(StreamingContainer.java:undeploy(561)) - Undeploy complete.
2017-06-08 17:52:11,694 INFO server.Server (Server.java:run(414)) -
Removing ln
LogicalNode@3f062c75identifier=tcp://brdn1251.target.com:38212/5.unique.1,
upstream=5.unique.1, group=uniquMessages/6.inputPort, partitions=[],
iterator=com.datatorrent.bufferserver.internal.DataList$DataListIterator@7e693f46{da=com.datatorrent.bufferserver.internal.DataList$Block@137d497f{identifier=5.unique.1,
data=67108864, readingOffset=0, writingOffset=306,
starting_window=5939d50100000001, ending_window=5939d50100000032,
refCount=2, uniqueIdentifier=0, next=null, future=null}}} from dl
com.datatorrent.bufferserver.internal.DataList@799564b7 {5.unique.1}
Regards
Vivek
--
View this message in context:
http://apache-apex-users-list.78494.x6.nabble.com/BoundedDedupOperator-failing-with-java-lang-IllegalArgumentException-bucket-conflict-tp1698p1702.html
Sent from the Apache Apex Users list mailing list archive at Nabble.com.
