@tushar
I have used apex-cli in one terminal and launched my application with below
command, in another terminal, i checked for application id but it is coming
zero active applications, I must be missing some thing here.
apex> launch -local /home/cloudera/myapexapp-1.0-SNAPSHOT.apa
1. Kafka2HDFS2
2. MyFirstApplication
Choose application: 1
Apex CLI 3.5.0 06.12.2016 @ 22:11:51 PST rev: 6de8828 branch:
6de8828e4f3d5734d0a6f9c1be0aa7057cb60ac8
apex> list-apps *
{"apps": []}
0 active, total 0 applications.
@chaitanya
hdfs and yarn are running, but the container folder is blank
[cloudera@quickstart bin]$ cd /var/log/hadoop-yarn/container
[cloudera@quickstart container]$ ls
[cloudera@quickstart container]$
On Fri, May 5, 2017 at 4:41 PM, Tushar Gosavi <[email protected]>
wrote:
> If log aggregations is enabled you can get the logs using yarn logs
> command. you can get the application id using apex-cli.
>
> - Tushar.
>
>
> On Fri, May 5, 2017 at 4:33 PM, Sushil Apex <[email protected]>
> wrote:
>
>> To add here
>> I am using Cloudera VM for hdfs and running Kafka from apache with single
>> machine zookeeper
>>
>> On Fri, May 5, 2017 at 4:32 PM, Sushil Apex <[email protected]>
>> wrote:
>>
>>> @vikram, I am using apex cli
>>> not sure where ts logs will be generated, checked /var/log/hadoop-yarn
>>> but did found any updated logs for apex
>>>
>>> I have changed the application name but that didn't helped
>>>
>>> On Fri, May 5, 2017 at 4:03 PM, vikram patil <[email protected]>
>>> wrote:
>>>
>>>> Can you share your Apex logs? Have you launched an application using
>>>> apex cli?
>>>>
>>>> Also, can you try once by changing name of your application and
>>>> relaunch again. Please kill or shut down your existing running
>>>> application before this.
>>>>
>>>> -Vikram
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, May 5, 2017 at 3:56 PM, Sushil Apex <[email protected]>
>>>> wrote:
>>>> > Tried with EARLIEST option also, no luck :(
>>>> >
>>>> > On Fri, May 5, 2017 at 3:49 PM, Sushil Apex <[email protected]>
>>>> wrote:
>>>> >>
>>>> >> @Vikram the folder /tmp/FromKafka is already created.
>>>> >> @Chaitanya yes I am pusing messages in kafka topic using
>>>> consoleProducer
>>>> >> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic
>>>> >> kafka2hdfs
>>>> >>
>>>> >> @Ajay
>>>> >> will try with the initial offset to EARLIEST.
>>>> >>
>>>> >> Thank you
>>>> >> Sushil
>>>> >>
>>>> >> On Fri, May 5, 2017 at 2:54 PM, AJAY GUPTA <[email protected]>
>>>> wrote:
>>>> >>>
>>>> >>> Also, you may be required to specify the initial offset to EARLIEST.
>>>> >>>
>>>> >>> Ajay
>>>> >>>
>>>> >>> On Fri, May 5, 2017 at 2:35 PM, vikram patil <[email protected]
>>>> >
>>>> >>> wrote:
>>>> >>>>
>>>> >>>> Hi Sushil,
>>>> >>>>
>>>> >>>> Have you provided configuration specifying hdfs directory and file
>>>> for
>>>> >>>> an application?
>>>> >>>> You may have to create /tmp/fromKafka directory in hdfs.
>>>> >>>> Thanks & Regards,
>>>> >>>> Vikram
>>>> >>>>
>>>> >>>> On Fri, May 5, 2017 at 2:30 PM, Sushil Apex <
>>>> [email protected]>
>>>> >>>> wrote:
>>>> >>>> >
>>>> >>>> > I am using Apex 3.5.0 and kafka 0.9
>>>> >>>> > malhar library used is 3.6.0
>>>> >>>> >
>>>> >>>> > I am following the example from
>>>> >>>> > https://github.com/DataTorrent/examples/blob/master/tutorial
>>>> s/kafka/src/main/java/com/example/myapexapp/KafkaApp.java
>>>> >>>> >
>>>> >>>> > I am putting messages in Kafka topic using consolekafkaProducer
>>>> >>>> > provided by Kafka, but I am not able to read these messages in
>>>> Apex
>>>> >>>> > DAG(created based on above link).
>>>> >>>> >
>>>> >>>> > I am running the apex apa file through apex-cli
>>>> >>>> >
>>>> >>>> > Apex CLI 3.5.0 06.12.2016 @ 22:11:51 PST rev: 6de8828 branch:
>>>> >>>> > 6de8828e4f3d5734d0a6f9c1be0aa7057cb60ac8
>>>> >>>> > apex> launch --local /home/cloudera/myapexapp-1.0-SNAPSHOT.apa
>>>> >>>> > 1. Kafka2HDFS
>>>> >>>> > 2. MyFirstApplication
>>>> >>>> > Choose application: 1
>>>> >>>> >
>>>> >>>> > and nothing happens after this, I can see the messages put in
>>>> >>>> > consoleConsumer in kafka logs
>>>> >>>> >
>>>> >>>> > Properties used are
>>>> >>>> >
>>>> >>>> > <property>
>>>> >>>> > <name>dt.operator.kafkaIn.prop.topics</name>
>>>> >>>> > <value>kafka2hdfs</value>
>>>> >>>> > </property>
>>>> >>>> >
>>>> >>>> > <property>
>>>> >>>> > <name>dt.operator.kafkaIn.prop.consumer.zookeeper</name>
>>>> >>>> > <value>localhost:2181</value>
>>>> >>>> > </property>
>>>> >>>> > <property>
>>>> >>>> > <name>dt.operator.kafkaIn.prop.clusters</name>
>>>> >>>> > <value>localhost:9092</value>
>>>> >>>> > </property>
>>>> >>>> > <property>
>>>> >>>> > <name>dt.operator.kafkaIn.prop.initialPartitionCount</name>
>>>> >>>> > <value>1</value>
>>>> >>>> > </property>
>>>> >>>> >
>>>> >>>> >
>>>> >>>> > Application Code
>>>> >>>> >
>>>> >>>> > KafkaSinglePortByteArrayInputOperator in
>>>> >>>> > = dag.addOperator("kafkaIn", new
>>>> >>>> > KafkaSinglePortByteArrayInputOperator());
>>>> >>>> >
>>>> >>>> > LineOutputOperator out = dag.addOperator("fileOut", new
>>>> >>>> > LineOutputOperator());
>>>> >>>> >
>>>> >>>> > dag.addStream("dataf", in.outputPort, out.input);
>>>> >>>> >
>>>> >>>> >
>>>> >>>> > I am not able to understand what config I am missing here?
>>>> >>>
>>>> >>>
>>>> >>
>>>> >
>>>>
>>>
>>>
>>
>
