Hey Alex, Do you use ONE_TO_ONE or ONE_TO_MANY partition?
Regards, Siyuan On Wed, Aug 24, 2016 at 10:27 AM, McCullough, Alex < alex.mccullo...@capitalone.com> wrote: > Hey Siyuan, > > > > We are using 3.4.0 > > > > Thanks, > > Alex > > *From: *"hsy...@gmail.com" <hsy...@gmail.com> > *Reply-To: *"users@apex.apache.org" <users@apex.apache.org> > *Date: *Wednesday, August 24, 2016 at 12:47 PM > *To: *"users@apex.apache.org" <users@apex.apache.org> > *Subject: *Re: Malhar 0.8.1.1 Kafka Operator and chroot > > > > Hey McCullough, > > > > What malhar version do you use? > > > > Regards, > > Siyuan > > > > On Wed, Aug 24, 2016 at 9:07 AM, McCullough, Alex < > alex.mccullo...@capitalone.com> wrote: > > Hey All, > > > > We are using the 0.8.1 kafka operator and the ZK connection string has a > chroot on it. We get errors when launching and the app fails, is there a > proper way in apex to append a chroot? > > > > > > **the ip’s are masked with #, but that’s not how they appear in our code > obviously** > > > > When we add this to the property for ZK: > > > > > > <property> > > <name>dt.operator.kafkaInputOperator.prop.consumer.zookeeper</name> > > <value>10.##.##.#:2181,10.##.##.##:2181,10.##.##.##:2181,10. > ##.##.#:2181/kafka2</value> > > </property> > > > > > > > > We get this error (connecting to a cluster without chroot it works fine): > > > > > > 2016-08-24 11:55:13,448 [main-EventThread] INFO zkclient.ZkClient > processStateChanged - zookeeper state changed (SyncConnected) > > 2016-08-24 11:55:13,585 [ZkClient-EventThread-30-10.## > .##.#:2181,10.##.##.##:2181,10.##.##.#:2181/kafka2,10.##.##.##:2181] > INFO zkclient.ZkEventThread run - Terminate ZkClient event thread. > > 2016-08-24 11:55:13,596 [main] INFO zookeeper.ZooKeeper close - Session: > 0x4558654aacf4263 closed > > 2016-08-24 11:55:13,596 [main-EventThread] INFO zookeeper.ClientCnxn run > - EventThread shut down > > 2016-08-24 11:55:13,597 [main] INFO kafka.AbstractKafkaInputOperator > definePartitions - [ONE_TO_ONE]: Initializing partition(s) > > 2016-08-24 11:55:13,602 [main] INFO service.AbstractService noteFailure - > Service com.datatorrent.stram.StreamingAppMasterService failed in state > INITED; cause: java.lang.IllegalArgumentException: there has to be one > idempotent storage manager > > java.lang.IllegalArgumentException: there has to be one idempotent > storage manager > > at com.google.common.base.Preconditions.checkArgument( > Preconditions.java:93) > > at org.apache.apex.malhar.lib.wal.FSWindowDataManager.partitioned( > FSWindowDataManager.java:251) > > at com.datatorrent.contrib.kafka.AbstractKafkaInputOperator. > definePartitions(AbstractKafkaInputOperator.java:637) > > at com.datatorrent.stram.plan.physical.PhysicalPlan. > initPartitioning(PhysicalPlan.java:752) > > at com.datatorrent.stram.plan.physical.PhysicalPlan. > addLogicalOperator(PhysicalPlan.java:1676) > > at com.datatorrent.stram.plan.physical.PhysicalPlan.<init>( > PhysicalPlan.java:378) > > at com.datatorrent.stram.StreamingContainerManager.<init>( > StreamingContainerManager.java:418) > > at com.datatorrent.stram.StreamingContainerManager.getInstance( > StreamingContainerManager.java:3023) > > at com.datatorrent.stram.StreamingAppMasterService.serviceInit( > StreamingAppMasterService.java:551) > > at org.apache.hadoop.service.AbstractService.init( > AbstractService.java:163) > > at com.datatorrent.stram.StreamingAppMaster.main( > StreamingAppMaster.java:102) > > 2016-08-24 11:55:13,604 [main] WARN service.AbstractService stopQuietly - > When stopping the service com.datatorrent.stram.StreamingAppMasterService > : java.lang.NullPointerException > > java.lang.NullPointerException > > at com.datatorrent.stram.StreamingAppMasterService.serviceStop( > StreamingAppMasterService.java:629) > > at org.apache.hadoop.service.AbstractService.stop( > AbstractService.java:221) > > at org.apache.hadoop.service.ServiceOperations.stop( > ServiceOperations.java:52) > > at org.apache.hadoop.service.ServiceOperations.stopQuietly( > ServiceOperations.java:80) > > at org.apache.hadoop.service.AbstractService.init( > AbstractService.java:171) > > at com.datatorrent.stram.StreamingAppMaster.main( > StreamingAppMaster.java:102) > > 2016-08-24 11:55:13,605 [main] ERROR stram.StreamingAppMaster main - > Exiting Application Master > > java.lang.IllegalArgumentException: there has to be one idempotent > storage manager > > at com.google.common.base.Preconditions.checkArgument( > Preconditions.java:93) > > at org.apache.apex.malhar.lib.wal.FSWindowDataManager.partitioned( > FSWindowDataManager.java:251) > > at com.datatorrent.contrib.kafka.AbstractKafkaInputOperator. > definePartitions(AbstractKafkaInputOperator.java:637) > > at com.datatorrent.stram.plan.physical.PhysicalPlan. > initPartitioning(PhysicalPlan.java:752) > > at com.datatorrent.stram.plan.physical.PhysicalPlan. > addLogicalOperator(PhysicalPlan.java:1676) > > at com.datatorrent.stram.plan.physical.PhysicalPlan.<init>( > PhysicalPlan.java:378) > > at com.datatorrent.stram.StreamingContainerManager.<init>( > StreamingContainerManager.java:418) > > at com.datatorrent.stram.StreamingContainerManager.getInstance( > StreamingContainerManager.java:3023) > > at com.datatorrent.stram.StreamingAppMasterService.serviceInit( > StreamingAppMasterService.java:551) > > at org.apache.hadoop.service.AbstractService.init( > AbstractService.java:163) > > at com.datatorrent.stram.StreamingAppMaster.main( > StreamingAppMaster.java:102) > > > ------------------------------ > > The information contained in this e-mail is confidential and/or > proprietary to Capital One and/or its affiliates and may only be used > solely in performance of work or services for Capital One. The information > transmitted herewith is intended only for use by the individual or entity > to which it is addressed. If the reader of this message is not the intended > recipient, you are hereby notified that any review, retransmission, > dissemination, distribution, copying or other use of, or taking of any > action in reliance upon this information is strictly prohibited. If you > have received this communication in error, please contact the sender and > delete the material from your computer. > > > > ------------------------------ > > The information contained in this e-mail is confidential and/or > proprietary to Capital One and/or its affiliates and may only be used > solely in performance of work or services for Capital One. The information > transmitted herewith is intended only for use by the individual or entity > to which it is addressed. If the reader of this message is not the intended > recipient, you are hereby notified that any review, retransmission, > dissemination, distribution, copying or other use of, or taking of any > action in reliance upon this information is strictly prohibited. If you > have received this communication in error, please contact the sender and > delete the material from your computer. >
