date:20150726

Re: kafka producer failed

2015-07-26 Thread Yi Pan

Great to hear that! Cheers!

-Yi

On Sun, Jul 26, 2015 at 10:53 PM, Job-Selina Wu 
wrote:

> Hi, Yi:
>
> you are right. After I add the producer.close(); the bug is fixed now.
>
> Thanks a lot!
>
> Selina
>
> On Sun, Jul 26, 2015 at 10:28 PM, Yi Pan  wrote:
>
> > Hi, Selina,
> >
> > Did you forget to close your Kafka producer in your Http servlet? If you
> > create a new Kafka producer in each Http request and do not close the
> > producer, that might cause this problem.
> >
> > -Yi
> >
> > On Sun, Jul 26, 2015 at 9:25 PM, Job-Selina Wu 
> > wrote:
> >
> > > Hi, Yan:
> > >
> > > the kaka.log got *java.io.IOException: Too many open files in
> system*
> > >
> > > Some detail list below.
> > >
> > > Sincerely,
> > > Selina
> > >
> > > [2015-07-26 21:07:01,500] INFO Verifying properties
> > > (kafka.utils.VerifiableProperties)
> > > [2015-07-26 21:07:01,545] INFO Property broker.id is overridden to 0
> > > (kafka.utils.VerifiableProperties)
> > > [2015-07-26 21:07:01,545] INFO Property log.cleaner.enable is
> > > overridden to false (kafka.utils.VerifiableProperties)
> > > [2015-07-26 21:07:01,545] INFO Property log.dirs is overridden to
> > > /tmp/kafka-logs (kafka.utils.VerifiableProperties)
> > > [2015-07-26 21:07:01,545] INFO Property
> > > log.retention.check.interval.ms is overridden to 30
> > > (kafka.utils.VerifiableProperties)
> > > [2015-07-26 21:07:01,546] INFO Property log.retention.hours is
> > > overridden to 168 (kafka.utils.VerifiableProperties)
> > > [2015-07-26 21:07:01,546] INFO Property log.segment.bytes is
> > > overridden to 1073741824 (kafka.utils.VerifiableProperties)
> > > [2015-07-26 21:07:01,546] INFO Property num.io.threads is overridden
> > > to 8 (kafka.utils.VerifiableProperties)
> > > [2015-07-26 21:07:01,547] INFO Property num.network.threads is
> > > overridden to 3 (kafka.utils.VerifiableProperties)
> > > [2015-07-26 21:07:01,547] INFO Property num.partitions is overridden
> > > to 1 (kafka.utils.VerifiableProperties)
> > > [2015-07-26 21:07:01,547] INFO Property
> > > num.recovery.threads.per.data.dir is overridden to 1
> > > (kafka.utils.VerifiableProperties)
> > > [2015-07-26 21:07:01,547] INFO Property port is overridden to 9092
> > > (kafka.utils.VerifiableProperties)
> > > [2015-07-26 21:07:01,548] INFO Property socket.receive.buffer.bytes is
> > > overridden to 102400 (kafka.utils.VerifiableProperties)
> > > [2015-07-26 21:07:01,548] INFO Property socket.request.max.bytes is
> > > overridden to 104857600 (kafka.utils.VerifiableProperties)
> > > [2015-07-26 21:07:01,548] INFO Property socket.send.buffer.bytes is
> > > overridden to 102400 (kafka.utils.VerifiableProperties)
> > > [2015-07-26 21:07:01,548] INFO Property zookeeper.connect is
> > > overridden to localhost:2181 (kafka.utils.VerifiableProperties)
> > > [2015-07-26 21:07:01,548] INFO Property
> > > zookeeper.connection.timeout.ms is overridden to 6000
> > > (kafka.utils.VerifiableProperties)
> > > [2015-07-26 21:07:01,607] INFO [Kafka Server 0], starting
> > > (kafka.server.KafkaServer)
> > > [2015-07-26 21:07:01,608] INFO [Kafka Server 0], Connecting to
> > > zookeeper on localhost:2181 (kafka.server.KafkaServer)
> > > [2015-07-26 21:07:01,619] INFO Starting ZkClient event thread.
> > > (org.I0Itec.zkclient.ZkEventThread)
> > > [2015-07-26 21:07:01,631] INFO Client
> > > environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09
> > > GMT (org.apache.zookeeper.ZooKeeper)
> > > [2015-07-26 21:07:01,631] INFO Client
> > > environment:host.name=selinas-mbp.attlocal.net
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2015-07-26 21:07:01,631] INFO Client
> > > environment:java.version=1.8.0_45 (org.apache.zookeeper.ZooKeeper)
> > > [2015-07-26 21:07:01,631] INFO Client environment:java.vendor=Oracle
> > > Corporation (org.apache.zookeeper.ZooKeeper)
> > > [2015-07-26 21:07:01,631] INFO Client
> > >
> > >
> >
> environment:java.home=/Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home/jre
> > > (org.apache.zookeeper.ZooKeeper)
> > > [2015-07-26 21:07:01,631] INFO Client
> > >
> > >
> >
> environment:java.class.path=:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../core/build/dependant-libs-2.10.4*/*.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../examples/build/libs//kafka-examples*.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../contrib/hadoop-consumer/build/libs//kafka-hadoop-consumer*.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../contrib/hadoop-producer/build/libs//kafka-hadoop-producer*.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../clients/build/libs/kafka-clients*.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../libs/jopt-simple-3.2.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../libs/kafka-clients-0.8.2.1.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../libs/kafka_2.10-0.8.2.1-javadoc.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../libs/kafka_2.10-0.8.2.1-

Re: kafka producer failed

2015-07-26 Thread Job-Selina Wu

Hi, Yi:

you are right. After I add the producer.close(); the bug is fixed now.

Thanks a lot!

Selina

On Sun, Jul 26, 2015 at 10:28 PM, Yi Pan  wrote:

> Hi, Selina,
>
> Did you forget to close your Kafka producer in your Http servlet? If you
> create a new Kafka producer in each Http request and do not close the
> producer, that might cause this problem.
>
> -Yi
>
> On Sun, Jul 26, 2015 at 9:25 PM, Job-Selina Wu 
> wrote:
>
> > Hi, Yan:
> >
> > the kaka.log got *java.io.IOException: Too many open files in system*
> >
> > Some detail list below.
> >
> > Sincerely,
> > Selina
> >
> > [2015-07-26 21:07:01,500] INFO Verifying properties
> > (kafka.utils.VerifiableProperties)
> > [2015-07-26 21:07:01,545] INFO Property broker.id is overridden to 0
> > (kafka.utils.VerifiableProperties)
> > [2015-07-26 21:07:01,545] INFO Property log.cleaner.enable is
> > overridden to false (kafka.utils.VerifiableProperties)
> > [2015-07-26 21:07:01,545] INFO Property log.dirs is overridden to
> > /tmp/kafka-logs (kafka.utils.VerifiableProperties)
> > [2015-07-26 21:07:01,545] INFO Property
> > log.retention.check.interval.ms is overridden to 30
> > (kafka.utils.VerifiableProperties)
> > [2015-07-26 21:07:01,546] INFO Property log.retention.hours is
> > overridden to 168 (kafka.utils.VerifiableProperties)
> > [2015-07-26 21:07:01,546] INFO Property log.segment.bytes is
> > overridden to 1073741824 (kafka.utils.VerifiableProperties)
> > [2015-07-26 21:07:01,546] INFO Property num.io.threads is overridden
> > to 8 (kafka.utils.VerifiableProperties)
> > [2015-07-26 21:07:01,547] INFO Property num.network.threads is
> > overridden to 3 (kafka.utils.VerifiableProperties)
> > [2015-07-26 21:07:01,547] INFO Property num.partitions is overridden
> > to 1 (kafka.utils.VerifiableProperties)
> > [2015-07-26 21:07:01,547] INFO Property
> > num.recovery.threads.per.data.dir is overridden to 1
> > (kafka.utils.VerifiableProperties)
> > [2015-07-26 21:07:01,547] INFO Property port is overridden to 9092
> > (kafka.utils.VerifiableProperties)
> > [2015-07-26 21:07:01,548] INFO Property socket.receive.buffer.bytes is
> > overridden to 102400 (kafka.utils.VerifiableProperties)
> > [2015-07-26 21:07:01,548] INFO Property socket.request.max.bytes is
> > overridden to 104857600 (kafka.utils.VerifiableProperties)
> > [2015-07-26 21:07:01,548] INFO Property socket.send.buffer.bytes is
> > overridden to 102400 (kafka.utils.VerifiableProperties)
> > [2015-07-26 21:07:01,548] INFO Property zookeeper.connect is
> > overridden to localhost:2181 (kafka.utils.VerifiableProperties)
> > [2015-07-26 21:07:01,548] INFO Property
> > zookeeper.connection.timeout.ms is overridden to 6000
> > (kafka.utils.VerifiableProperties)
> > [2015-07-26 21:07:01,607] INFO [Kafka Server 0], starting
> > (kafka.server.KafkaServer)
> > [2015-07-26 21:07:01,608] INFO [Kafka Server 0], Connecting to
> > zookeeper on localhost:2181 (kafka.server.KafkaServer)
> > [2015-07-26 21:07:01,619] INFO Starting ZkClient event thread.
> > (org.I0Itec.zkclient.ZkEventThread)
> > [2015-07-26 21:07:01,631] INFO Client
> > environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09
> > GMT (org.apache.zookeeper.ZooKeeper)
> > [2015-07-26 21:07:01,631] INFO Client
> > environment:host.name=selinas-mbp.attlocal.net
> > (org.apache.zookeeper.ZooKeeper)
> > [2015-07-26 21:07:01,631] INFO Client
> > environment:java.version=1.8.0_45 (org.apache.zookeeper.ZooKeeper)
> > [2015-07-26 21:07:01,631] INFO Client environment:java.vendor=Oracle
> > Corporation (org.apache.zookeeper.ZooKeeper)
> > [2015-07-26 21:07:01,631] INFO Client
> >
> >
> environment:java.home=/Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home/jre
> > (org.apache.zookeeper.ZooKeeper)
> > [2015-07-26 21:07:01,631] INFO Client
> >
> >
> environment:java.class.path=:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../core/build/dependant-libs-2.10.4*/*.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../examples/build/libs//kafka-examples*.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../contrib/hadoop-consumer/build/libs//kafka-hadoop-consumer*.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../contrib/hadoop-producer/build/libs//kafka-hadoop-producer*.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../clients/build/libs/kafka-clients*.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../libs/jopt-simple-3.2.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../libs/kafka-clients-0.8.2.1.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../libs/kafka_2.10-0.8.2.1-javadoc.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../libs/kafka_2.10-0.8.2.1-scaladoc.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../libs/kafka_2.10-0.8.2.1-sources.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../libs/kafka_2.10-0.8.2.1-test.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../libs/kafka_2.10-0.8.

Re: kafka producer failed

2015-07-26 Thread Yi Pan

Hi, Selina,

Did you forget to close your Kafka producer in your Http servlet? If you
create a new Kafka producer in each Http request and do not close the
producer, that might cause this problem.

-Yi

On Sun, Jul 26, 2015 at 9:25 PM, Job-Selina Wu 
wrote:

> Hi, Yan:
>
> the kaka.log got *java.io.IOException: Too many open files in system*
>
> Some detail list below.
>
> Sincerely,
> Selina
>
> [2015-07-26 21:07:01,500] INFO Verifying properties
> (kafka.utils.VerifiableProperties)
> [2015-07-26 21:07:01,545] INFO Property broker.id is overridden to 0
> (kafka.utils.VerifiableProperties)
> [2015-07-26 21:07:01,545] INFO Property log.cleaner.enable is
> overridden to false (kafka.utils.VerifiableProperties)
> [2015-07-26 21:07:01,545] INFO Property log.dirs is overridden to
> /tmp/kafka-logs (kafka.utils.VerifiableProperties)
> [2015-07-26 21:07:01,545] INFO Property
> log.retention.check.interval.ms is overridden to 30
> (kafka.utils.VerifiableProperties)
> [2015-07-26 21:07:01,546] INFO Property log.retention.hours is
> overridden to 168 (kafka.utils.VerifiableProperties)
> [2015-07-26 21:07:01,546] INFO Property log.segment.bytes is
> overridden to 1073741824 (kafka.utils.VerifiableProperties)
> [2015-07-26 21:07:01,546] INFO Property num.io.threads is overridden
> to 8 (kafka.utils.VerifiableProperties)
> [2015-07-26 21:07:01,547] INFO Property num.network.threads is
> overridden to 3 (kafka.utils.VerifiableProperties)
> [2015-07-26 21:07:01,547] INFO Property num.partitions is overridden
> to 1 (kafka.utils.VerifiableProperties)
> [2015-07-26 21:07:01,547] INFO Property
> num.recovery.threads.per.data.dir is overridden to 1
> (kafka.utils.VerifiableProperties)
> [2015-07-26 21:07:01,547] INFO Property port is overridden to 9092
> (kafka.utils.VerifiableProperties)
> [2015-07-26 21:07:01,548] INFO Property socket.receive.buffer.bytes is
> overridden to 102400 (kafka.utils.VerifiableProperties)
> [2015-07-26 21:07:01,548] INFO Property socket.request.max.bytes is
> overridden to 104857600 (kafka.utils.VerifiableProperties)
> [2015-07-26 21:07:01,548] INFO Property socket.send.buffer.bytes is
> overridden to 102400 (kafka.utils.VerifiableProperties)
> [2015-07-26 21:07:01,548] INFO Property zookeeper.connect is
> overridden to localhost:2181 (kafka.utils.VerifiableProperties)
> [2015-07-26 21:07:01,548] INFO Property
> zookeeper.connection.timeout.ms is overridden to 6000
> (kafka.utils.VerifiableProperties)
> [2015-07-26 21:07:01,607] INFO [Kafka Server 0], starting
> (kafka.server.KafkaServer)
> [2015-07-26 21:07:01,608] INFO [Kafka Server 0], Connecting to
> zookeeper on localhost:2181 (kafka.server.KafkaServer)
> [2015-07-26 21:07:01,619] INFO Starting ZkClient event thread.
> (org.I0Itec.zkclient.ZkEventThread)
> [2015-07-26 21:07:01,631] INFO Client
> environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09
> GMT (org.apache.zookeeper.ZooKeeper)
> [2015-07-26 21:07:01,631] INFO Client
> environment:host.name=selinas-mbp.attlocal.net
> (org.apache.zookeeper.ZooKeeper)
> [2015-07-26 21:07:01,631] INFO Client
> environment:java.version=1.8.0_45 (org.apache.zookeeper.ZooKeeper)
> [2015-07-26 21:07:01,631] INFO Client environment:java.vendor=Oracle
> Corporation (org.apache.zookeeper.ZooKeeper)
> [2015-07-26 21:07:01,631] INFO Client
>
> environment:java.home=/Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home/jre
> (org.apache.zookeeper.ZooKeeper)
> [2015-07-26 21:07:01,631] INFO Client
>
> environment:java.class.path=:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../core/build/dependant-libs-2.10.4*/*.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../examples/build/libs//kafka-examples*.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../contrib/hadoop-consumer/build/libs//kafka-hadoop-consumer*.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../contrib/hadoop-producer/build/libs//kafka-hadoop-producer*.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../clients/build/libs/kafka-clients*.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../libs/jopt-simple-3.2.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../libs/kafka-clients-0.8.2.1.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../libs/kafka_2.10-0.8.2.1-javadoc.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../libs/kafka_2.10-0.8.2.1-scaladoc.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../libs/kafka_2.10-0.8.2.1-sources.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../libs/kafka_2.10-0.8.2.1-test.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../libs/kafka_2.10-0.8.2.1.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../libs/log4j-1.2.16.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../libs/lz4-1.2.0.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../libs/metrics-core-2.2.0.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../libs/scal

Re: kafka producer failed

2015-07-26 Thread Job-Selina Wu

Hi, Yan:

the kaka.log got *java.io.IOException: Too many open files in system*

Some detail list below.

Sincerely,
Selina

[2015-07-26 21:07:01,500] INFO Verifying properties
(kafka.utils.VerifiableProperties)
[2015-07-26 21:07:01,545] INFO Property broker.id is overridden to 0
(kafka.utils.VerifiableProperties)
[2015-07-26 21:07:01,545] INFO Property log.cleaner.enable is
overridden to false (kafka.utils.VerifiableProperties)
[2015-07-26 21:07:01,545] INFO Property log.dirs is overridden to
/tmp/kafka-logs (kafka.utils.VerifiableProperties)
[2015-07-26 21:07:01,545] INFO Property
log.retention.check.interval.ms is overridden to 30
(kafka.utils.VerifiableProperties)
[2015-07-26 21:07:01,546] INFO Property log.retention.hours is
overridden to 168 (kafka.utils.VerifiableProperties)
[2015-07-26 21:07:01,546] INFO Property log.segment.bytes is
overridden to 1073741824 (kafka.utils.VerifiableProperties)
[2015-07-26 21:07:01,546] INFO Property num.io.threads is overridden
to 8 (kafka.utils.VerifiableProperties)
[2015-07-26 21:07:01,547] INFO Property num.network.threads is
overridden to 3 (kafka.utils.VerifiableProperties)
[2015-07-26 21:07:01,547] INFO Property num.partitions is overridden
to 1 (kafka.utils.VerifiableProperties)
[2015-07-26 21:07:01,547] INFO Property
num.recovery.threads.per.data.dir is overridden to 1
(kafka.utils.VerifiableProperties)
[2015-07-26 21:07:01,547] INFO Property port is overridden to 9092
(kafka.utils.VerifiableProperties)
[2015-07-26 21:07:01,548] INFO Property socket.receive.buffer.bytes is
overridden to 102400 (kafka.utils.VerifiableProperties)
[2015-07-26 21:07:01,548] INFO Property socket.request.max.bytes is
overridden to 104857600 (kafka.utils.VerifiableProperties)
[2015-07-26 21:07:01,548] INFO Property socket.send.buffer.bytes is
overridden to 102400 (kafka.utils.VerifiableProperties)
[2015-07-26 21:07:01,548] INFO Property zookeeper.connect is
overridden to localhost:2181 (kafka.utils.VerifiableProperties)
[2015-07-26 21:07:01,548] INFO Property
zookeeper.connection.timeout.ms is overridden to 6000
(kafka.utils.VerifiableProperties)
[2015-07-26 21:07:01,607] INFO [Kafka Server 0], starting
(kafka.server.KafkaServer)
[2015-07-26 21:07:01,608] INFO [Kafka Server 0], Connecting to
zookeeper on localhost:2181 (kafka.server.KafkaServer)
[2015-07-26 21:07:01,619] INFO Starting ZkClient event thread.
(org.I0Itec.zkclient.ZkEventThread)
[2015-07-26 21:07:01,631] INFO Client
environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09
GMT (org.apache.zookeeper.ZooKeeper)
[2015-07-26 21:07:01,631] INFO Client
environment:host.name=selinas-mbp.attlocal.net
(org.apache.zookeeper.ZooKeeper)
[2015-07-26 21:07:01,631] INFO Client
environment:java.version=1.8.0_45 (org.apache.zookeeper.ZooKeeper)
[2015-07-26 21:07:01,631] INFO Client environment:java.vendor=Oracle
Corporation (org.apache.zookeeper.ZooKeeper)
[2015-07-26 21:07:01,631] INFO Client
environment:java.home=/Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home/jre
(org.apache.zookeeper.ZooKeeper)
[2015-07-26 21:07:01,631] INFO Client
environment:java.class.path=:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../core/build/dependant-libs-2.10.4*/*.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../examples/build/libs//kafka-examples*.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../contrib/hadoop-consumer/build/libs//kafka-hadoop-consumer*.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../contrib/hadoop-producer/build/libs//kafka-hadoop-producer*.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../clients/build/libs/kafka-clients*.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../libs/jopt-simple-3.2.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../libs/kafka-clients-0.8.2.1.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../libs/kafka_2.10-0.8.2.1-javadoc.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../libs/kafka_2.10-0.8.2.1-scaladoc.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../libs/kafka_2.10-0.8.2.1-sources.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../libs/kafka_2.10-0.8.2.1-test.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../libs/kafka_2.10-0.8.2.1.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../libs/log4j-1.2.16.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../libs/lz4-1.2.0.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../libs/metrics-core-2.2.0.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../libs/scala-library-2.10.4.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../libs/slf4j-api-1.7.6.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../libs/slf4j-log4j12-1.6.1.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../libs/snappy-java-1.1.1.6.jar:/Users/selina/IdeaProjects/samza-Demo/deploy/kafka/bin/../libs/zkclient-0.3.jar:/Users/selina/IdeaProjects/samza-De

Re: Review Request 35444: SAMZA-692: Scrooge (Scala) Thrift serde

2015-07-26 Thread Eli Reisman


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35444/
---

(Updated July 26, 2015, 11:29 p.m.)


Review request for samza.


Repository: samza


Description
---

SAMZA-692: Scrooge (Scala) Thrift serde


Diffs (updated)
-

  build.gradle 0852adc 
  gradle/dependency-versions-scala-2.10.gradle f12beaf 
  gradle/dependency-versions.gradle fb06e8e 
  samza-scrooge/src/main/resources/thrift/test.thrift PRE-CREATION 
  samza-scrooge/src/main/scala/org/apache/samza/scrooge/ThriftTest.scala 
PRE-CREATION 
  
samza-scrooge/src/main/scala/org/apache/samza/scrooge/serde/SamzaScroogeDeserializer.scala
 PRE-CREATION 
  
samza-scrooge/src/main/scala/org/apache/samza/scrooge/serde/ScroogeExampleSerdeFactory.scala
 PRE-CREATION 
  
samza-scrooge/src/main/scala/org/apache/samza/scrooge/serde/ScroogeSerde.scala 
PRE-CREATION 
  samza-scrooge/src/main/scala/org/apache/samza/scrooge/util/ReflectUtil.scala 
PRE-CREATION 
  
samza-scrooge/src/test/scala/org/apache/samza/scrooge/TestScroogeExampleSerde.scala
 PRE-CREATION 
  settings.gradle 19bff97 

Diff: https://reviews.apache.org/r/35444/diff/


Testing
---

New unit test. Passes "./gradlew test" and "./gradlew check"


Thanks,

Eli Reisman

Re: Review Request 35445: SAMZA-693: Very basic HDFS Producer service for Samza

2015-07-26 Thread Eli Reisman


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35445/
---

(Updated July 26, 2015, 10:47 p.m.)


Review request for samza.


Repository: samza


Description
---

SAMZA-693: Very basic HDFS Producer service for Samza


Diffs
-

  build.gradle 0852adc 
  samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/HdfsConfig.scala 
PRE-CREATION 
  samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/HdfsSystemAdmin.scala 
PRE-CREATION 
  
samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/HdfsSystemFactory.scala 
PRE-CREATION 
  
samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/HdfsSystemProducer.scala 
PRE-CREATION 
  
samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/HdfsSystemProducerMetrics.scala
 PRE-CREATION 
  
samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/writer/BinarySequenceFileHdfsWriter.scala
 PRE-CREATION 
  samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/writer/Bucketer.scala 
PRE-CREATION 
  
samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/writer/HdfsWriter.scala 
PRE-CREATION 
  
samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/writer/JobNameDateTimeBucketer.scala
 PRE-CREATION 
  
samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/writer/SequenceFileHdfsWriter.scala
 PRE-CREATION 
  
samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/writer/TextSequenceFileHdfsWriter.scala
 PRE-CREATION 
  samza-hdfs/src/test/resources/samza-hdfs-test-batch-job-text.properties 
PRE-CREATION 
  samza-hdfs/src/test/resources/samza-hdfs-test-batch-job.properties 
PRE-CREATION 
  samza-hdfs/src/test/resources/samza-hdfs-test-job-text.properties 
PRE-CREATION 
  samza-hdfs/src/test/resources/samza-hdfs-test-job.properties PRE-CREATION 
  
samza-hdfs/src/test/scala/org/apache/samza/system/hdfs/TestHdfsSystemProducerTestSuite.scala
 PRE-CREATION 
  settings.gradle 19bff97 

Diff: https://reviews.apache.org/r/35445/diff/


Testing (updated)
---

Updated: See JIRA SAMZA-693 for details, this latest update (693-4) addresses 
post-review issues and adds more pluggable design, several default writer 
implementations, and more (and more thorough) unit tests.

Passes 'gradle clean test'.


Thanks,

Eli Reisman

Re: Review Request 35445: SAMZA-693: Very basic HDFS Producer service for Samza

2015-07-26 Thread Eli Reisman


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35445/
---

(Updated July 26, 2015, 10:31 p.m.)


Review request for samza.


Repository: samza


Description
---

SAMZA-693: Very basic HDFS Producer service for Samza


Diffs (updated)
-

  build.gradle 0852adc 
  samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/HdfsConfig.scala 
PRE-CREATION 
  samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/HdfsSystemAdmin.scala 
PRE-CREATION 
  
samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/HdfsSystemFactory.scala 
PRE-CREATION 
  
samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/HdfsSystemProducer.scala 
PRE-CREATION 
  
samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/HdfsSystemProducerMetrics.scala
 PRE-CREATION 
  
samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/writer/BinarySequenceFileHdfsWriter.scala
 PRE-CREATION 
  samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/writer/Bucketer.scala 
PRE-CREATION 
  
samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/writer/HdfsWriter.scala 
PRE-CREATION 
  
samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/writer/JobNameDateTimeBucketer.scala
 PRE-CREATION 
  
samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/writer/SequenceFileHdfsWriter.scala
 PRE-CREATION 
  
samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/writer/TextSequenceFileHdfsWriter.scala
 PRE-CREATION 
  samza-hdfs/src/test/resources/samza-hdfs-test-batch-job-text.properties 
PRE-CREATION 
  samza-hdfs/src/test/resources/samza-hdfs-test-batch-job.properties 
PRE-CREATION 
  samza-hdfs/src/test/resources/samza-hdfs-test-job-text.properties 
PRE-CREATION 
  samza-hdfs/src/test/resources/samza-hdfs-test-job.properties PRE-CREATION 
  
samza-hdfs/src/test/scala/org/apache/samza/system/hdfs/TestHdfsSystemProducerTestSuite.scala
 PRE-CREATION 
  settings.gradle 19bff97 

Diff: https://reviews.apache.org/r/35445/diff/


Testing
---

New unit test, but it's fairly rudimentary. Passes "./gradlew test" and 
"./gradlew check"

This only supplies an HDFS Producer, and this producer only writes 
SequenceFiles of ByteWriteables so far. If the patch were accepted as-is, I'd 
suggest future tickets for a matching HDFS Consumer, and a pluggable set of 
output formats, configurable via HdfsConfig settings.

On the upside, this patch has been tested on a real cluster with real data, 
using several serdes, with good results.


Thanks,

Eli Reisman

Re: kafka producer failed

2015-07-26 Thread Yan Fang

You may check the Kafka.log to see what's inside 

Yan Fang

> On Jul 26, 2015, at 2:01 AM, Job-Selina Wu  wrote:
> 
> The exception is below:
> 
> kafka.common.FailedToSendMessageException: Failed to send messages after 3
> tries.
> kafka.common.FailedToSendMessageException: Failed to send messages after 3
> tries.
> at
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
> at kafka.producer.Producer.send(Producer.scala:77)
> at kafka.javaapi.producer.Producer.send(Producer.scala:33)
> at http.server.HttpDemoHandler.doDemo(HttpDemoHandler.java:71)
> at http.server.HttpDemoHandler.handle(HttpDemoHandler.java:32)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> at org.eclipse.jetty.server.Server.handle(Server.java:498)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:265)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:243)
> at
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:610)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:539)
> at java.lang.Thread.run(Thread.java:745)
> 
> On Sun, Jul 26, 2015 at 12:42 AM, Job-Selina Wu 
> wrote:
> 
>> Hi, Yan:
>> 
>>  My Http Server send message to Kafka.
>> 
>> The server.log at deploy/kafka/logs/server.log shown :
>> 
>> [2015-07-26 00:33:51,910] INFO Closing socket connection to /127.0.0.1. 
>> (kafka.network.Processor)
>> [2015-07-26 00:33:51,984] INFO Closing socket connection to /127.0.0.1. 
>> (kafka.network.Processor)
>> [2015-07-26 00:33:52,011] INFO Closing socket connection to /127.0.0.1. 
>> (kafka.network.Processor)
>> 
>> .
>> 
>> 
>> Your help is highly appreciated.
>> 
>> Sincerely,
>> 
>> Selina
>> 
>> 
>>> On Sun, Jul 26, 2015 at 12:01 AM, Yan Fang  wrote:
>>> 
>>> You are giving the Kafka code and the Samza log, which does not make sense
>>> actually...
>>> 
>>> Fang, Yan
>>> yanfang...@gmail.com
>>> 
>>> On Sat, Jul 25, 2015 at 10:31 PM, Job-Selina Wu 
>>> wrote:
>>> 
 Hi, Yi, Navina and Benjamin:
 
Thanks a lot to spending your time to help me this issue.
 
The configuration is below. Do you think it could be the
>>> configuration
 problem?
 I tried props.put("request.required.acks", "0"); and
 props.put("request.required.acks", "1"); both did not work.
 
 
 Properties props = new Properties();
 
private final Producer producer;
 
public KafkaProducer() {
//BOOTSTRAP.SERVERS
props.put("metadata.broker.list", "localhost:9092");
props.put("bootstrap.servers", "localhost:9092 ");
props.put("serializer.class", "kafka.serializer.StringEncoder");
props.put("partitioner.class", "com.kafka.SimplePartitioner");
props.put("request.required.acks", "0");
 
ProducerConfig config = new ProducerConfig(props);
 
producer = new Producer(config);
}
 
 --
 
 Exceptions at log are list below.
 
 Your help is highly appreciated.
 
 Sincerely,
 Selina Wu
 
 
 Exceptions at log
>>> deploy/yarn/logs/userlogs/application_1437886672214_0001/container_1437886672214_0001_01_01/samza-application-master.log
 
 2015-07-25 22:03:52 Shell [DEBUG] Failed to detect a valid hadoop home
 directory
 *java.io.IOException: HADOOP_HOME or hadoop.home.dir are not set*.
   at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:265)
   at org.apache.hadoop.util.Shell.(Shell.java:290)
   at org.apache.hadoop.util.StringUtils.(StringUtils.java:76)
   at
>>> org.apache.hadoop.yarn.conf.YarnConfiguration.(YarnConfiguration.java:517)
   at
 org.apache.samza.job.yarn.SamzaAppMaster$.main(SamzaAppMaster.scala:77)
   at
>>> org.apache.samza.job.yarn.SamzaAppMaster.main(SamzaAppMaster.scala)
 2015-07-25 22:03:52 Shell [DEBUG] setsid is not available on this
 machine. So not using it.
 2015-07-25 22:03:52 Shell [DEBUG] setsid exited with exit code 0
 2015-07-25 22:03:52 ClientHelper [INFO] trying to connect to RM
 127.0.0.1:8032
 2015-07-25 22:03:52 AbstractService [DEBUG] Service:
 org.apache.hadoop.yarn.client.api.impl.YarnClientImpl entered state
 INITED
 2015-07-25 22:03:52 RMProxy [INFO] Connecting to ResourceManager at
 /127.0.0.1:8032
 2015-07-25 22:03:52 MutableMetricsFactory [DEBUG] field
 org.apache.hadoop.metrics2.lib.MutableRate
 org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess
 with annotation
 @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=,
 always=false, type=DEFAULT, value=[Rate of successful kerberos logins
 and latency (milliseconds)], valueName=Time)
 2015-07-25 22:03:52 MutableMetricsFact

Re: Review Request 36768: SAMZA-740: Add ElasticsearchProducer example to samza-hello-samza

2015-07-26 Thread Navina Ramesh



> On July 24, 2015, 6:19 p.m., Navina Ramesh wrote:
> > It will also be nice to have a tutorial added to the website on how to 
> > setup and run the example Samza job that uses ElasticSearchProducer.
> 
> Stuart Davidson wrote:
> Happy to write something up. Do you want it on the hello-samza index.md, 
> or a seperate page?

You can create a web page and add it to the tutorial list here - 
http://samza.apache.org/learn/tutorials/latest/ Thanks!


- Navina


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36768/#review92953
---


On July 24, 2015, 8:31 a.m., Stuart Davidson wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36768/
> ---
> 
> (Updated July 24, 2015, 8:31 a.m.)
> 
> 
> Review request for samza.
> 
> 
> Repository: samza-hello-samza
> 
> 
> Description
> ---
> 
> I'd put together an example of how to use the new ElasticsearchProvider for 
> my work colleagues and I thought it'd be worth submitting it back to the 
> community. This also includes a script to start elasticsearch and kibana on 
> the host - be aware, there's a 64bit version depending on what machine you 
> want to run against.
> 
> Also note, this is against 0.10.0 of Samza which is not released yet. I am 
> making the assumption that the Elasticsearch jars are bundled as part of that 
> - if not, we'll need to add them to the POM here.
> 
> 
> Diffs
> -
> 
>   bin/grid-elastic PRE-CREATION 
>   pom.xml f9c4fa9 
>   src/main/assembly/src.xml f57fee2 
>   src/main/config/wikipedia-elastic.properties PRE-CREATION 
>   src/main/java/samza/examples/wikipedia/task/WikipediaElasticStreamTask.java 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/36768/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Stuart Davidson
> 
>

Re: Review Request 36768: SAMZA-740: Add ElasticsearchProducer example to samza-hello-samza

2015-07-26 Thread Stuart Davidson



> On July 24, 2015, 6:19 p.m., Navina Ramesh wrote:
> > It will also be nice to have a tutorial added to the website on how to 
> > setup and run the example Samza job that uses ElasticSearchProducer.

Happy to write something up. Do you want it on the hello-samza index.md, or a 
seperate page?


- Stuart


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36768/#review92953
---


On July 24, 2015, 8:31 a.m., Stuart Davidson wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36768/
> ---
> 
> (Updated July 24, 2015, 8:31 a.m.)
> 
> 
> Review request for samza.
> 
> 
> Repository: samza-hello-samza
> 
> 
> Description
> ---
> 
> I'd put together an example of how to use the new ElasticsearchProvider for 
> my work colleagues and I thought it'd be worth submitting it back to the 
> community. This also includes a script to start elasticsearch and kibana on 
> the host - be aware, there's a 64bit version depending on what machine you 
> want to run against.
> 
> Also note, this is against 0.10.0 of Samza which is not released yet. I am 
> making the assumption that the Elasticsearch jars are bundled as part of that 
> - if not, we'll need to add them to the POM here.
> 
> 
> Diffs
> -
> 
>   bin/grid-elastic PRE-CREATION 
>   pom.xml f9c4fa9 
>   src/main/assembly/src.xml f57fee2 
>   src/main/config/wikipedia-elastic.properties PRE-CREATION 
>   src/main/java/samza/examples/wikipedia/task/WikipediaElasticStreamTask.java 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/36768/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Stuart Davidson
> 
>

Re: Review Request 36768: SAMZA-740: Add ElasticsearchProducer example to samza-hello-samza

2015-07-26 Thread Stuart Davidson



> On July 24, 2015, 6:18 p.m., Navina Ramesh wrote:
> > bin/grid-elastic, line 1
> > 
> >
> > Can you make changes to bin/grid by adding options to install 
> > elastic_search and/or kibana? Code seems redundant.

I deliberately split it up because I didn't think that all people would want to 
know how to use Elasticsearch - thus, not everyone would want to install it. I 
really don't think we should combine the two - most of hello-samza talks about 
the core components of using Samza, whereas this example doesn't. 

Are you sure you want me to combine the two?


> On July 24, 2015, 6:18 p.m., Navina Ramesh wrote:
> > pom.xml, line 116
> > 
> >
> > You won't need this if you apply your patch to the latest branch in 
> > samza-hello-samza.

So do you want me to get rid of this change? Without 0.10.0 (and above), this 
patch won't work - thus the deliberate change to force the upgrade.


- Stuart


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36768/#review92951
---


On July 24, 2015, 8:31 a.m., Stuart Davidson wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36768/
> ---
> 
> (Updated July 24, 2015, 8:31 a.m.)
> 
> 
> Review request for samza.
> 
> 
> Repository: samza-hello-samza
> 
> 
> Description
> ---
> 
> I'd put together an example of how to use the new ElasticsearchProvider for 
> my work colleagues and I thought it'd be worth submitting it back to the 
> community. This also includes a script to start elasticsearch and kibana on 
> the host - be aware, there's a 64bit version depending on what machine you 
> want to run against.
> 
> Also note, this is against 0.10.0 of Samza which is not released yet. I am 
> making the assumption that the Elasticsearch jars are bundled as part of that 
> - if not, we'll need to add them to the POM here.
> 
> 
> Diffs
> -
> 
>   bin/grid-elastic PRE-CREATION 
>   pom.xml f9c4fa9 
>   src/main/assembly/src.xml f57fee2 
>   src/main/config/wikipedia-elastic.properties PRE-CREATION 
>   src/main/java/samza/examples/wikipedia/task/WikipediaElasticStreamTask.java 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/36768/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Stuart Davidson
> 
>

Re: kafka producer failed

2015-07-26 Thread Job-Selina Wu

The exception is below:

kafka.common.FailedToSendMessageException: Failed to send messages after 3
tries.
kafka.common.FailedToSendMessageException: Failed to send messages after 3
tries.
at
kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
at kafka.producer.Producer.send(Producer.scala:77)
at kafka.javaapi.producer.Producer.send(Producer.scala:33)
at http.server.HttpDemoHandler.doDemo(HttpDemoHandler.java:71)
at http.server.HttpDemoHandler.handle(HttpDemoHandler.java:32)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:498)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:265)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:243)
at
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:610)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:539)
at java.lang.Thread.run(Thread.java:745)

On Sun, Jul 26, 2015 at 12:42 AM, Job-Selina Wu 
wrote:

> Hi, Yan:
>
>   My Http Server send message to Kafka.
>
> The server.log at deploy/kafka/logs/server.log shown :
>
> [2015-07-26 00:33:51,910] INFO Closing socket connection to /127.0.0.1. 
> (kafka.network.Processor)
> [2015-07-26 00:33:51,984] INFO Closing socket connection to /127.0.0.1. 
> (kafka.network.Processor)
> [2015-07-26 00:33:52,011] INFO Closing socket connection to /127.0.0.1. 
> (kafka.network.Processor)
>
> .
>
>
> Your help is highly appreciated.
>
> Sincerely,
>
> Selina
>
>
> On Sun, Jul 26, 2015 at 12:01 AM, Yan Fang  wrote:
>
>> You are giving the Kafka code and the Samza log, which does not make sense
>> actually...
>>
>> Fang, Yan
>> yanfang...@gmail.com
>>
>> On Sat, Jul 25, 2015 at 10:31 PM, Job-Selina Wu 
>> wrote:
>>
>> > Hi, Yi, Navina and Benjamin:
>> >
>> > Thanks a lot to spending your time to help me this issue.
>> >
>> > The configuration is below. Do you think it could be the
>> configuration
>> > problem?
>> > I tried props.put("request.required.acks", "0"); and
>> >  props.put("request.required.acks", "1"); both did not work.
>> >
>> > 
>> > Properties props = new Properties();
>> >
>> > private final Producer producer;
>> >
>> > public KafkaProducer() {
>> > //BOOTSTRAP.SERVERS
>> > props.put("metadata.broker.list", "localhost:9092");
>> > props.put("bootstrap.servers", "localhost:9092 ");
>> > props.put("serializer.class", "kafka.serializer.StringEncoder");
>> > props.put("partitioner.class", "com.kafka.SimplePartitioner");
>> > props.put("request.required.acks", "0");
>> >
>> > ProducerConfig config = new ProducerConfig(props);
>> >
>> > producer = new Producer(config);
>> > }
>> >
>> > --
>> >
>> >  Exceptions at log are list below.
>> >
>> > Your help is highly appreciated.
>> >
>> > Sincerely,
>> > Selina Wu
>> >
>> >
>> > Exceptions at log
>> >
>> >
>> deploy/yarn/logs/userlogs/application_1437886672214_0001/container_1437886672214_0001_01_01/samza-application-master.log
>> >
>> > 2015-07-25 22:03:52 Shell [DEBUG] Failed to detect a valid hadoop home
>> > directory
>> > *java.io.IOException: HADOOP_HOME or hadoop.home.dir are not set*.
>> >at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:265)
>> >at org.apache.hadoop.util.Shell.(Shell.java:290)
>> >at org.apache.hadoop.util.StringUtils.(StringUtils.java:76)
>> >at
>> >
>> org.apache.hadoop.yarn.conf.YarnConfiguration.(YarnConfiguration.java:517)
>> >at
>> > org.apache.samza.job.yarn.SamzaAppMaster$.main(SamzaAppMaster.scala:77)
>> >at
>> org.apache.samza.job.yarn.SamzaAppMaster.main(SamzaAppMaster.scala)
>> > 2015-07-25 22:03:52 Shell [DEBUG] setsid is not available on this
>> > machine. So not using it.
>> > 2015-07-25 22:03:52 Shell [DEBUG] setsid exited with exit code 0
>> > 2015-07-25 22:03:52 ClientHelper [INFO] trying to connect to RM
>> > 127.0.0.1:8032
>> > 2015-07-25 22:03:52 AbstractService [DEBUG] Service:
>> > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl entered state
>> > INITED
>> > 2015-07-25 22:03:52 RMProxy [INFO] Connecting to ResourceManager at
>> > /127.0.0.1:8032
>> > 2015-07-25 22:03:52 MutableMetricsFactory [DEBUG] field
>> > org.apache.hadoop.metrics2.lib.MutableRate
>> > org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess
>> > with annotation
>> > @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=,
>> > always=false, type=DEFAULT, value=[Rate of successful kerberos logins
>> > and latency (milliseconds)], valueName=Time)
>> > 2015-07-25 22:03:52 MutableMetricsFactory [DEBUG] field
>> > org.apache.hadoop.metrics2.lib.MutableRate
>> > org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure
>> > with annotation
>> > @org.apache.hadoop.metrics2.annotatio

Re: kafka producer failed

2015-07-26 Thread Job-Selina Wu

Hi, Yan:

  My Http Server send message to Kafka.

The server.log at deploy/kafka/logs/server.log shown :

[2015-07-26 00:33:51,910] INFO Closing socket connection to
/127.0.0.1. (kafka.network.Processor)
[2015-07-26 00:33:51,984] INFO Closing socket connection to
/127.0.0.1. (kafka.network.Processor)
[2015-07-26 00:33:52,011] INFO Closing socket connection to
/127.0.0.1. (kafka.network.Processor)

.


Your help is highly appreciated.

Sincerely,

Selina


On Sun, Jul 26, 2015 at 12:01 AM, Yan Fang  wrote:

> You are giving the Kafka code and the Samza log, which does not make sense
> actually...
>
> Fang, Yan
> yanfang...@gmail.com
>
> On Sat, Jul 25, 2015 at 10:31 PM, Job-Selina Wu 
> wrote:
>
> > Hi, Yi, Navina and Benjamin:
> >
> > Thanks a lot to spending your time to help me this issue.
> >
> > The configuration is below. Do you think it could be the
> configuration
> > problem?
> > I tried props.put("request.required.acks", "0"); and
> >  props.put("request.required.acks", "1"); both did not work.
> >
> > 
> > Properties props = new Properties();
> >
> > private final Producer producer;
> >
> > public KafkaProducer() {
> > //BOOTSTRAP.SERVERS
> > props.put("metadata.broker.list", "localhost:9092");
> > props.put("bootstrap.servers", "localhost:9092 ");
> > props.put("serializer.class", "kafka.serializer.StringEncoder");
> > props.put("partitioner.class", "com.kafka.SimplePartitioner");
> > props.put("request.required.acks", "0");
> >
> > ProducerConfig config = new ProducerConfig(props);
> >
> > producer = new Producer(config);
> > }
> >
> > --
> >
> >  Exceptions at log are list below.
> >
> > Your help is highly appreciated.
> >
> > Sincerely,
> > Selina Wu
> >
> >
> > Exceptions at log
> >
> >
> deploy/yarn/logs/userlogs/application_1437886672214_0001/container_1437886672214_0001_01_01/samza-application-master.log
> >
> > 2015-07-25 22:03:52 Shell [DEBUG] Failed to detect a valid hadoop home
> > directory
> > *java.io.IOException: HADOOP_HOME or hadoop.home.dir are not set*.
> >at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:265)
> >at org.apache.hadoop.util.Shell.(Shell.java:290)
> >at org.apache.hadoop.util.StringUtils.(StringUtils.java:76)
> >at
> >
> org.apache.hadoop.yarn.conf.YarnConfiguration.(YarnConfiguration.java:517)
> >at
> > org.apache.samza.job.yarn.SamzaAppMaster$.main(SamzaAppMaster.scala:77)
> >at org.apache.samza.job.yarn.SamzaAppMaster.main(SamzaAppMaster.scala)
> > 2015-07-25 22:03:52 Shell [DEBUG] setsid is not available on this
> > machine. So not using it.
> > 2015-07-25 22:03:52 Shell [DEBUG] setsid exited with exit code 0
> > 2015-07-25 22:03:52 ClientHelper [INFO] trying to connect to RM
> > 127.0.0.1:8032
> > 2015-07-25 22:03:52 AbstractService [DEBUG] Service:
> > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl entered state
> > INITED
> > 2015-07-25 22:03:52 RMProxy [INFO] Connecting to ResourceManager at
> > /127.0.0.1:8032
> > 2015-07-25 22:03:52 MutableMetricsFactory [DEBUG] field
> > org.apache.hadoop.metrics2.lib.MutableRate
> > org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess
> > with annotation
> > @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=,
> > always=false, type=DEFAULT, value=[Rate of successful kerberos logins
> > and latency (milliseconds)], valueName=Time)
> > 2015-07-25 22:03:52 MutableMetricsFactory [DEBUG] field
> > org.apache.hadoop.metrics2.lib.MutableRate
> > org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure
> > with annotation
> > @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=,
> > always=false, type=DEFAULT, value=[Rate of failed kerberos logins and
> > latency (milliseconds)], valueName=Time)
> > 2015-07-25 22:03:52 MutableMetricsFactory [DEBUG] field
> > org.apache.hadoop.metrics2.lib.MutableRate
> > org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups
> > with annotation
> > @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=,
> > always=false, type=DEFAULT, value=[GetGroups], valueName=Time)
> > 2015-07-25 22:03:52 MetricsSystemImpl [DEBUG] UgiMetrics, User and
> > group related metrics
> > 2015-07-25 22:03:52 KerberosName [DEBUG] Kerberos krb5 configuration
> > not found, setting default realm to empty
> > 2015-07-25 22:03:52 Groups [DEBUG]  Creating new Groups object
> > 2015-07-25 22:03:52 NativeCodeLoader [DEBUG] Trying to load the
> > custom-built native-hadoop library...
> > 2015-07-25 22:03:52 NativeCodeLoader [DEBUG] Failed to load
> > native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in
> > java.library.path
> > 2015-07-25 22:03:52 NativeCodeLoader [DEBUG]
> >
> >
> java.library.path=/home//Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.
> > 2015-

Re: kafka producer failed

2015-07-26 Thread Yan Fang

You are giving the Kafka code and the Samza log, which does not make sense
actually...

Fang, Yan
yanfang...@gmail.com

On Sat, Jul 25, 2015 at 10:31 PM, Job-Selina Wu 
wrote:

> Hi, Yi, Navina and Benjamin:
>
> Thanks a lot to spending your time to help me this issue.
>
> The configuration is below. Do you think it could be the configuration
> problem?
> I tried props.put("request.required.acks", "0"); and
>  props.put("request.required.acks", "1"); both did not work.
>
> 
> Properties props = new Properties();
>
> private final Producer producer;
>
> public KafkaProducer() {
> //BOOTSTRAP.SERVERS
> props.put("metadata.broker.list", "localhost:9092");
> props.put("bootstrap.servers", "localhost:9092 ");
> props.put("serializer.class", "kafka.serializer.StringEncoder");
> props.put("partitioner.class", "com.kafka.SimplePartitioner");
> props.put("request.required.acks", "0");
>
> ProducerConfig config = new ProducerConfig(props);
>
> producer = new Producer(config);
> }
>
> --
>
>  Exceptions at log are list below.
>
> Your help is highly appreciated.
>
> Sincerely,
> Selina Wu
>
>
> Exceptions at log
>
> deploy/yarn/logs/userlogs/application_1437886672214_0001/container_1437886672214_0001_01_01/samza-application-master.log
>
> 2015-07-25 22:03:52 Shell [DEBUG] Failed to detect a valid hadoop home
> directory
> *java.io.IOException: HADOOP_HOME or hadoop.home.dir are not set*.
>at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:265)
>at org.apache.hadoop.util.Shell.(Shell.java:290)
>at org.apache.hadoop.util.StringUtils.(StringUtils.java:76)
>at
> org.apache.hadoop.yarn.conf.YarnConfiguration.(YarnConfiguration.java:517)
>at
> org.apache.samza.job.yarn.SamzaAppMaster$.main(SamzaAppMaster.scala:77)
>at org.apache.samza.job.yarn.SamzaAppMaster.main(SamzaAppMaster.scala)
> 2015-07-25 22:03:52 Shell [DEBUG] setsid is not available on this
> machine. So not using it.
> 2015-07-25 22:03:52 Shell [DEBUG] setsid exited with exit code 0
> 2015-07-25 22:03:52 ClientHelper [INFO] trying to connect to RM
> 127.0.0.1:8032
> 2015-07-25 22:03:52 AbstractService [DEBUG] Service:
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl entered state
> INITED
> 2015-07-25 22:03:52 RMProxy [INFO] Connecting to ResourceManager at
> /127.0.0.1:8032
> 2015-07-25 22:03:52 MutableMetricsFactory [DEBUG] field
> org.apache.hadoop.metrics2.lib.MutableRate
> org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess
> with annotation
> @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=,
> always=false, type=DEFAULT, value=[Rate of successful kerberos logins
> and latency (milliseconds)], valueName=Time)
> 2015-07-25 22:03:52 MutableMetricsFactory [DEBUG] field
> org.apache.hadoop.metrics2.lib.MutableRate
> org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure
> with annotation
> @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=,
> always=false, type=DEFAULT, value=[Rate of failed kerberos logins and
> latency (milliseconds)], valueName=Time)
> 2015-07-25 22:03:52 MutableMetricsFactory [DEBUG] field
> org.apache.hadoop.metrics2.lib.MutableRate
> org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups
> with annotation
> @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=,
> always=false, type=DEFAULT, value=[GetGroups], valueName=Time)
> 2015-07-25 22:03:52 MetricsSystemImpl [DEBUG] UgiMetrics, User and
> group related metrics
> 2015-07-25 22:03:52 KerberosName [DEBUG] Kerberos krb5 configuration
> not found, setting default realm to empty
> 2015-07-25 22:03:52 Groups [DEBUG]  Creating new Groups object
> 2015-07-25 22:03:52 NativeCodeLoader [DEBUG] Trying to load the
> custom-built native-hadoop library...
> 2015-07-25 22:03:52 NativeCodeLoader [DEBUG] Failed to load
> native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in
> java.library.path
> 2015-07-25 22:03:52 NativeCodeLoader [DEBUG]
>
> java.library.path=/home//Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.
> 2015-07-25 22:03:52 NativeCodeLoader [WARN] Unable to load
> native-hadoop library for your platform... using builtin-java classes
> where applicable
>
>
> 2015-07-25 22:03:53 KafkaCheckpointManager [WARN] While trying to
> validate topic __samza_checkpoint_ver_1_for_demo-parser7_1:
> *kafka.common.LeaderNotAvailableException. Retrying.*
> 2015-07-25 22:03:53 KafkaCheckpointManager [DEBUG] Exception detail:
> kafka.common.LeaderNotAvailableException
>at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.

Re: kafka producer failed

Re: kafka producer failed

Re: kafka producer failed

Re: kafka producer failed

Re: Review Request 35444: SAMZA-692: Scrooge (Scala) Thrift serde

Re: Review Request 35445: SAMZA-693: Very basic HDFS Producer service for Samza

Re: Review Request 35445: SAMZA-693: Very basic HDFS Producer service for Samza

Re: kafka producer failed

Re: Review Request 36768: SAMZA-740: Add ElasticsearchProducer example to samza-hello-samza

Re: Review Request 36768: SAMZA-740: Add ElasticsearchProducer example to samza-hello-samza

Re: Review Request 36768: SAMZA-740: Add ElasticsearchProducer example to samza-hello-samza

Re: kafka producer failed

Re: kafka producer failed

Re: kafka producer failed

14 matches

Site Navigation

Mail list logo

Footer information