Re: Connection timed out error while installing "Hello Samza"

[Yi Pan]

P.S., the attachment you post w/ the previous email does not show up
(Apache mailing list disallow attachments). If would be better if you send
the command line outputs in pure text or html format.

-Yi

On Thu, Sep 14, 2017 at 3:34 PM, Yi Pan  wrote:

> Hi, Anantharaman,
>
> A lot of firewall setup prevents "incoming connections", not "outgoing
> connections" from a box. I.e. if you are downloading files, you are
> establishing outgoing connections and it is fine. When you are hosting a
> server (i.e. ZooKeeper or Kafka), you are making an "incoming connections"
> to the box and will trigger the firewall rules.
>
> Just trying to pin down the problem, could you try on a Linux box w/o any
> firewall rules setup and see whether that works? At least, verify that
> localhost resolves to 127.0.0.1 and try telnet locahost 9092 or 2181?
>
> NOTE: Success in ping simply means that you can access via ICMP, not
> guarantee you can establish TCP connection to the ports via firewall rules.
>
> -Yi
>
> On Thu, Sep 14, 2017 at 12:33 PM, Anantharaman, Srinatha (Contractor) <
> srinatha_ananthara...@comcast.com> wrote:
>
>> That is a good catch Naveena the server which I am trying to install is
>> an Hadoop edge node, it has kafka broker and Zookeeper client already
>> installed for my Hadoop
>> Does it matter If I am installing it on a separate folder?. My guess Even
>> if there are conflicts on earlier installations it should throw some other
>> error but not the connection time out
>>
>> Coming to your question " it could either related resolving "localhost"
>> or firewall that prevents communication between ports" - how to prove it is
>> because of Firewall issue. I am able to clone Samza files from Apache git.
>> I can download any external files on this server
>>
>> ~Sri
>>
>> -Original Message-
>> From: Navina Ramesh [mailto:nram...@linkedin.com]
>> Sent: Thursday, September 14, 2017 2:51 PM
>> To: dev@samza.apache.org
>> Subject: Re: Connection timed out error while installing "Hello Samza"
>>
>> I wonder if this has anything to do with previous kafka / zookeeper
>> installed on your box. Just for sanity, try clearing /tmp/zookeeeper* and
>> /tmp/kafka* before re-trying those steps.
>>
>>
>> Same as Yi, I strongly suspect issues with your local laptop setup - it
>> could either related resolving "localhost" or firewall that prevents
>> communication between ports.
>>
>>
>> Navina
>>
>> 
>> From: Yi Pan 
>> Sent: Thursday, September 14, 2017 11:37:46 AM
>> To: dev@samza.apache.org
>> Subject: Re: Connection timed out error while installing "Hello Samza"
>>
>> Hi, Anantharaman,
>>
>> I just did the same steps as you described in your email and all passed
>> on my box. Hence, I strongly suspect that it is related to your local
>> laptop network setup.
>>
>> Could you post all the command line output when you ran the sequence of
>> commands?
>>
>> -Yi
>>
>> On Thu, Sep 14, 2017 at 11:21 AM, Yi Pan  wrote:
>>
>> > Hi, Ananarath,
>> >
>> > It is very strange that you are seeing this timeout exception that we
>> > do not see. I am trying to follow the exact steps you did to see
>> > whether there is anything broken. I will update you this afternoon.
>> >
>> > Meanwhile, could you check your hostname setup and firewall
>> > configuration to see whether your local laptop has blocked access via
>> > the public IP address to your laptop? Could you verify that your
>> > localhost is resolved to
>> > 127.0.0.1 and is accessible?
>> >
>> > -Yi
>> >
>> > On Thu, Sep 14, 2017 at 11:18 AM, Anantharaman, Srinatha (Contractor)
>> > < srinatha_ananthara...@comcast.com> wrote:
>> >
>> >> Yi,
>> >>
>> >> Is there any alternate way to install Samza  Or solution to the
>> >> connection time out error?
>> >>
>> >> Regards,
>> >> ~Sri
>> >>
>> >> From: Anantharaman, Srinatha (Contractor)
>> >> Sent: Wednesday, September 13, 2017 11:37 AM
>> >> To: dev@samza.apache.org
>> >> Subject: RE: Connection timed out error while installing "Hello Samza"
>> >>
>> >>
>> >> Yi,
>> >>
>> >>
>> >>
>> >> I am trying to build Samza locally by following the steps provided by
>> >> Navina.
>> >>
>> >> As per those steps kafka will be installed after Zookeeper, I am
>> >> getting Error while starting Zookeeper after it is installed
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> Steps Followed :
>> >>
>> >>
>> >>
>> >> Yes. You can clone apache/samza locally and build it with:
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> cd 
>> >>
>> >>
>> >>
>> >> gradle -b bootstrap.gradle
>> >>
>> >>
>> >>
>> >> ./gradlew clean build -x test
>> >>
>> >>
>> >>
>> >> ./gradlew publishToMavenLocal## This publishes a snapshot version
>> of
>> >> the latest apache/samza into your local maven repo
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> Then, head to hello-samza workspace and build again:
>> >>
>> >>
>> >>
>> >> cd 
>> >>
>> >>
>> >>
>> >> mvn clean package  ## This should create a build target
>> >>
>> >>
>> >>
>> >> ./bin/grid install zookeeper
>> >>
>> >>

Re: Connection timed out error while installing "Hello Samza"

[Yi Pan]

Hi, Anantharaman,

A lot of firewall setup prevents "incoming connections", not "outgoing
connections" from a box. I.e. if you are downloading files, you are
establishing outgoing connections and it is fine. When you are hosting a
server (i.e. ZooKeeper or Kafka), you are making an "incoming connections"
to the box and will trigger the firewall rules.

Just trying to pin down the problem, could you try on a Linux box w/o any
firewall rules setup and see whether that works? At least, verify that
localhost resolves to 127.0.0.1 and try telnet locahost 9092 or 2181?

NOTE: Success in ping simply means that you can access via ICMP, not
guarantee you can establish TCP connection to the ports via firewall rules.

-Yi

On Thu, Sep 14, 2017 at 12:33 PM, Anantharaman, Srinatha (Contractor) <
srinatha_ananthara...@comcast.com> wrote:

> That is a good catch Naveena the server which I am trying to install is an
> Hadoop edge node, it has kafka broker and Zookeeper client already
> installed for my Hadoop
> Does it matter If I am installing it on a separate folder?. My guess Even
> if there are conflicts on earlier installations it should throw some other
> error but not the connection time out
>
> Coming to your question " it could either related resolving "localhost" or
> firewall that prevents communication between ports" - how to prove it is
> because of Firewall issue. I am able to clone Samza files from Apache git.
> I can download any external files on this server
>
> ~Sri
>
> -Original Message-
> From: Navina Ramesh [mailto:nram...@linkedin.com]
> Sent: Thursday, September 14, 2017 2:51 PM
> To: dev@samza.apache.org
> Subject: Re: Connection timed out error while installing "Hello Samza"
>
> I wonder if this has anything to do with previous kafka / zookeeper
> installed on your box. Just for sanity, try clearing /tmp/zookeeeper* and
> /tmp/kafka* before re-trying those steps.
>
>
> Same as Yi, I strongly suspect issues with your local laptop setup - it
> could either related resolving "localhost" or firewall that prevents
> communication between ports.
>
>
> Navina
>
> 
> From: Yi Pan 
> Sent: Thursday, September 14, 2017 11:37:46 AM
> To: dev@samza.apache.org
> Subject: Re: Connection timed out error while installing "Hello Samza"
>
> Hi, Anantharaman,
>
> I just did the same steps as you described in your email and all passed on
> my box. Hence, I strongly suspect that it is related to your local laptop
> network setup.
>
> Could you post all the command line output when you ran the sequence of
> commands?
>
> -Yi
>
> On Thu, Sep 14, 2017 at 11:21 AM, Yi Pan  wrote:
>
> > Hi, Ananarath,
> >
> > It is very strange that you are seeing this timeout exception that we
> > do not see. I am trying to follow the exact steps you did to see
> > whether there is anything broken. I will update you this afternoon.
> >
> > Meanwhile, could you check your hostname setup and firewall
> > configuration to see whether your local laptop has blocked access via
> > the public IP address to your laptop? Could you verify that your
> > localhost is resolved to
> > 127.0.0.1 and is accessible?
> >
> > -Yi
> >
> > On Thu, Sep 14, 2017 at 11:18 AM, Anantharaman, Srinatha (Contractor)
> > < srinatha_ananthara...@comcast.com> wrote:
> >
> >> Yi,
> >>
> >> Is there any alternate way to install Samza  Or solution to the
> >> connection time out error?
> >>
> >> Regards,
> >> ~Sri
> >>
> >> From: Anantharaman, Srinatha (Contractor)
> >> Sent: Wednesday, September 13, 2017 11:37 AM
> >> To: dev@samza.apache.org
> >> Subject: RE: Connection timed out error while installing "Hello Samza"
> >>
> >>
> >> Yi,
> >>
> >>
> >>
> >> I am trying to build Samza locally by following the steps provided by
> >> Navina.
> >>
> >> As per those steps kafka will be installed after Zookeeper, I am
> >> getting Error while starting Zookeeper after it is installed
> >>
> >>
> >>
> >>
> >>
> >> Steps Followed :
> >>
> >>
> >>
> >> Yes. You can clone apache/samza locally and build it with:
> >>
> >>
> >>
> >>
> >>
> >> cd 
> >>
> >>
> >>
> >> gradle -b bootstrap.gradle
> >>
> >>
> >>
> >> ./gradlew clean build -x test
> >>
> >>
> >>
> >> ./gradlew publishToMavenLocal## This publishes a snapshot version of
> >> the latest apache/samza into your local maven repo
> >>
> >>
> >>
> >>
> >>
> >> Then, head to hello-samza workspace and build again:
> >>
> >>
> >>
> >> cd 
> >>
> >>
> >>
> >> mvn clean package  ## This should create a build target
> >>
> >>
> >>
> >> ./bin/grid install zookeeper
> >>
> >>
> >>
> >> ./bin/grid start zookeeper
> >>
> >>
> >>
> >> ./bin/grid install kafka
> >>
> >>
> >>
> >> ./bin/grid start kafka
> >>
> >>
> >>
> >> ./bin/grid install yarn
> >>
> >>
> >>
> >> ./bin/grid start yarn
> >>
> >>
> >>
> >>
> >>
> >> mkdir -p deploy/samza
> >>
> >>
> >>
> >> tar -xvf ./target/hello-samza-*-SNAPSHOT-dist.tar.gz -C deploy/samza
> >>
> >>
> >>
> >>
> >>
> >> NOTE : BTW From the above steps I could not exec

Re: Connection timed out error while installing "Hello Samza"

[Navina Ramesh]

if it is already installed in your system, I don't think you need re-install 
them again. However, do make sure that there are no ACLs enabled on ZK because 
kafka may not support it.


> My guess Even if there are conflicts on earlier installations it should throw 
> some other error but not the connection time out

I think your connection timeout was related to when you were trying to download 
samza from apache.


If you can re-use the already install kafka and zookeeper client, then install 
yarn, and follow the remaining steps to build and deploy samza. In any case, 
more logs will be useful for us to debug.


> it could either related resolving "localhost" or firewall that prevents 
> communication between ports"

I meant this could be the reason why ZK is not able to install and/or talk to 
the kafka broker.


It totally stumps me why you can't downloads samza from your server. Is it 
possible for you try on another clean host and see if it works or fails for you?


Navina


From: Anantharaman, Srinatha (Contractor) 
Sent: Thursday, September 14, 2017 12:33:26 PM
To: dev@samza.apache.org
Subject: RE: Connection timed out error while installing "Hello Samza"

That is a good catch Naveena the server which I am trying to install is an 
Hadoop edge node, it has kafka broker and Zookeeper client already installed 
for my Hadoop
Does it matter If I am installing it on a separate folder?. My guess Even if 
there are conflicts on earlier installations it should throw some other error 
but not the connection time out

Coming to your question " it could either related resolving "localhost" or 
firewall that prevents communication between ports" - how to prove it is 
because of Firewall issue. I am able to clone Samza files from Apache git. I 
can download any external files on this server

~Sri

-Original Message-
From: Navina Ramesh [mailto:nram...@linkedin.com]
Sent: Thursday, September 14, 2017 2:51 PM
To: dev@samza.apache.org
Subject: Re: Connection timed out error while installing "Hello Samza"

I wonder if this has anything to do with previous kafka / zookeeper installed 
on your box. Just for sanity, try clearing /tmp/zookeeeper* and /tmp/kafka* 
before re-trying those steps.


Same as Yi, I strongly suspect issues with your local laptop setup - it could 
either related resolving "localhost" or firewall that prevents communication 
between ports.


Navina


From: Yi Pan 
Sent: Thursday, September 14, 2017 11:37:46 AM
To: dev@samza.apache.org
Subject: Re: Connection timed out error while installing "Hello Samza"

Hi, Anantharaman,

I just did the same steps as you described in your email and all passed on my 
box. Hence, I strongly suspect that it is related to your local laptop network 
setup.

Could you post all the command line output when you ran the sequence of 
commands?

-Yi

On Thu, Sep 14, 2017 at 11:21 AM, Yi Pan  wrote:

> Hi, Ananarath,
>
> It is very strange that you are seeing this timeout exception that we
> do not see. I am trying to follow the exact steps you did to see
> whether there is anything broken. I will update you this afternoon.
>
> Meanwhile, could you check your hostname setup and firewall
> configuration to see whether your local laptop has blocked access via
> the public IP address to your laptop? Could you verify that your
> localhost is resolved to
> 127.0.0.1 and is accessible?
>
> -Yi
>
> On Thu, Sep 14, 2017 at 11:18 AM, Anantharaman, Srinatha (Contractor)
> < srinatha_ananthara...@comcast.com> wrote:
>
>> Yi,
>>
>> Is there any alternate way to install Samza  Or solution to the
>> connection time out error?
>>
>> Regards,
>> ~Sri
>>
>> From: Anantharaman, Srinatha (Contractor)
>> Sent: Wednesday, September 13, 2017 11:37 AM
>> To: dev@samza.apache.org
>> Subject: RE: Connection timed out error while installing "Hello Samza"
>>
>>
>> Yi,
>>
>>
>>
>> I am trying to build Samza locally by following the steps provided by
>> Navina.
>>
>> As per those steps kafka will be installed after Zookeeper, I am
>> getting Error while starting Zookeeper after it is installed
>>
>>
>>
>>
>>
>> Steps Followed :
>>
>>
>>
>> Yes. You can clone apache/samza locally and build it with:
>>
>>
>>
>>
>>
>> cd 
>>
>>
>>
>> gradle -b bootstrap.gradle
>>
>>
>>
>> ./gradlew clean build -x test
>>
>>
>>
>> ./gradlew publishToMavenLocal## This publishes a snapshot version of
>> the latest apache/samza into your local maven repo
>>
>>
>>
>>
>>
>> Then, head to hello-samza workspace and build again:
>>
>>
>>
>> cd 
>>
>>
>>
>> mvn clean package  ## This should create a build target
>>
>>
>>
>> ./bin/grid install zookeeper
>>
>>
>>
>> ./bin/grid start zookeeper
>>
>>
>>
>> ./bin/grid install kafka
>>
>>
>>
>> ./bin/grid start kafka
>>
>>
>>
>> ./bin/grid install yarn
>>
>>
>>
>> ./bin/grid start yarn
>>
>>
>>
>>
>>
>> mkdir -p deploy/samza
>>
>>
>>
>> tar -xvf ./target/hello-samza-*-SNAPSHOT-dist.tar.gz -C deploy/samza
>>
>>

RE: Connection timed out error while installing "Hello Samza"

[Anantharaman, Srinatha (Contractor)]

That is a good catch Naveena the server which I am trying to install is an 
Hadoop edge node, it has kafka broker and Zookeeper client already installed 
for my Hadoop
Does it matter If I am installing it on a separate folder?. My guess Even if 
there are conflicts on earlier installations it should throw some other error 
but not the connection time out

Coming to your question " it could either related resolving "localhost" or 
firewall that prevents communication between ports" - how to prove it is 
because of Firewall issue. I am able to clone Samza files from Apache git. I 
can download any external files on this server

~Sri

-Original Message-
From: Navina Ramesh [mailto:nram...@linkedin.com] 
Sent: Thursday, September 14, 2017 2:51 PM
To: dev@samza.apache.org
Subject: Re: Connection timed out error while installing "Hello Samza"

I wonder if this has anything to do with previous kafka / zookeeper installed 
on your box. Just for sanity, try clearing /tmp/zookeeeper* and /tmp/kafka* 
before re-trying those steps.


Same as Yi, I strongly suspect issues with your local laptop setup - it could 
either related resolving "localhost" or firewall that prevents communication 
between ports.


Navina


From: Yi Pan 
Sent: Thursday, September 14, 2017 11:37:46 AM
To: dev@samza.apache.org
Subject: Re: Connection timed out error while installing "Hello Samza"

Hi, Anantharaman,

I just did the same steps as you described in your email and all passed on my 
box. Hence, I strongly suspect that it is related to your local laptop network 
setup.

Could you post all the command line output when you ran the sequence of 
commands?

-Yi

On Thu, Sep 14, 2017 at 11:21 AM, Yi Pan  wrote:

> Hi, Ananarath,
>
> It is very strange that you are seeing this timeout exception that we 
> do not see. I am trying to follow the exact steps you did to see 
> whether there is anything broken. I will update you this afternoon.
>
> Meanwhile, could you check your hostname setup and firewall 
> configuration to see whether your local laptop has blocked access via 
> the public IP address to your laptop? Could you verify that your 
> localhost is resolved to
> 127.0.0.1 and is accessible?
>
> -Yi
>
> On Thu, Sep 14, 2017 at 11:18 AM, Anantharaman, Srinatha (Contractor) 
> < srinatha_ananthara...@comcast.com> wrote:
>
>> Yi,
>>
>> Is there any alternate way to install Samza  Or solution to the 
>> connection time out error?
>>
>> Regards,
>> ~Sri
>>
>> From: Anantharaman, Srinatha (Contractor)
>> Sent: Wednesday, September 13, 2017 11:37 AM
>> To: dev@samza.apache.org
>> Subject: RE: Connection timed out error while installing "Hello Samza"
>>
>>
>> Yi,
>>
>>
>>
>> I am trying to build Samza locally by following the steps provided by 
>> Navina.
>>
>> As per those steps kafka will be installed after Zookeeper, I am 
>> getting Error while starting Zookeeper after it is installed
>>
>>
>>
>>
>>
>> Steps Followed :
>>
>>
>>
>> Yes. You can clone apache/samza locally and build it with:
>>
>>
>>
>>
>>
>> cd 
>>
>>
>>
>> gradle -b bootstrap.gradle
>>
>>
>>
>> ./gradlew clean build -x test
>>
>>
>>
>> ./gradlew publishToMavenLocal## This publishes a snapshot version of
>> the latest apache/samza into your local maven repo
>>
>>
>>
>>
>>
>> Then, head to hello-samza workspace and build again:
>>
>>
>>
>> cd 
>>
>>
>>
>> mvn clean package  ## This should create a build target
>>
>>
>>
>> ./bin/grid install zookeeper
>>
>>
>>
>> ./bin/grid start zookeeper
>>
>>
>>
>> ./bin/grid install kafka
>>
>>
>>
>> ./bin/grid start kafka
>>
>>
>>
>> ./bin/grid install yarn
>>
>>
>>
>> ./bin/grid start yarn
>>
>>
>>
>>
>>
>> mkdir -p deploy/samza
>>
>>
>>
>> tar -xvf ./target/hello-samza-*-SNAPSHOT-dist.tar.gz -C deploy/samza
>>
>>
>>
>>
>>
>> NOTE : BTW From the above steps I could not execute " gradle -b 
>> bootstrap.gradle" since that command does not exists
>>
>>
>>
>> Regards,
>>
>> ~Sri
>>
>>
>>
>> -Original Message-
>> From: Yi Pan [mailto:nickpa...@gmail.com]
>> Sent: Tuesday, September 12, 2017 7:13 PM
>> To: dev@samza.apache.org
>> Subject: Re: Connection timed out error while installing "Hello Samza"
>>
>>
>>
>> Hi, Anantharam,
>>
>>
>>
>> Could you confirm at which step your setup failed? It seems that your 
>> zookeeper server is running. Could you check to see whether your 
>> Kafka broker is running? You can either do a telnet localhost 9092 or 
>> do a ps auxww | grep kafka to see whether you got any broker running.
>>
>>
>>
>> Sometimes, the Kafka service takes time to start on a single laptop. 
>> You can just try to run ./bin/grid start kafka
>>
>>
>>
>> again to see whether the service is up.
>>
>>
>>
>> Thanks!
>>
>>
>>
>> -Yi
>>
>>
>>
>> On Tue, Sep 12, 2017 at 11:38 AM, Anantharaman, Srinatha (Contractor) 
>> < srinatha_ananthara...@comcast.com> a...@comcast.com>> wrote:
>>
>>
>>
>> > Find below Zookeeper logs:
>>
>> >
>>
>> >

Re: Connection timed out error while installing "Hello Samza"

[Navina Ramesh]

I wonder if this has anything to do with previous kafka / zookeeper installed 
on your box. Just for sanity, try clearing /tmp/zookeeeper* and /tmp/kafka* 
before re-trying those steps.


Same as Yi, I strongly suspect issues with your local laptop setup - it could 
either related resolving "localhost" or firewall that prevents communication 
between ports.


Navina


From: Yi Pan 
Sent: Thursday, September 14, 2017 11:37:46 AM
To: dev@samza.apache.org
Subject: Re: Connection timed out error while installing "Hello Samza"

Hi, Anantharaman,

I just did the same steps as you described in your email and all passed on
my box. Hence, I strongly suspect that it is related to your local laptop
network setup.

Could you post all the command line output when you ran the sequence of
commands?

-Yi

On Thu, Sep 14, 2017 at 11:21 AM, Yi Pan  wrote:

> Hi, Ananarath,
>
> It is very strange that you are seeing this timeout exception that we do
> not see. I am trying to follow the exact steps you did to see whether there
> is anything broken. I will update you this afternoon.
>
> Meanwhile, could you check your hostname setup and firewall configuration
> to see whether your local laptop has blocked access via the public IP
> address to your laptop? Could you verify that your localhost is resolved to
> 127.0.0.1 and is accessible?
>
> -Yi
>
> On Thu, Sep 14, 2017 at 11:18 AM, Anantharaman, Srinatha (Contractor) <
> srinatha_ananthara...@comcast.com> wrote:
>
>> Yi,
>>
>> Is there any alternate way to install Samza  Or solution to the
>> connection time out error?
>>
>> Regards,
>> ~Sri
>>
>> From: Anantharaman, Srinatha (Contractor)
>> Sent: Wednesday, September 13, 2017 11:37 AM
>> To: dev@samza.apache.org
>> Subject: RE: Connection timed out error while installing "Hello Samza"
>>
>>
>> Yi,
>>
>>
>>
>> I am trying to build Samza locally by following the steps provided by
>> Navina.
>>
>> As per those steps kafka will be installed after Zookeeper, I am getting
>> Error while starting Zookeeper after it is installed
>>
>>
>>
>>
>>
>> Steps Followed :
>>
>>
>>
>> Yes. You can clone apache/samza locally and build it with:
>>
>>
>>
>>
>>
>> cd 
>>
>>
>>
>> gradle -b bootstrap.gradle
>>
>>
>>
>> ./gradlew clean build -x test
>>
>>
>>
>> ./gradlew publishToMavenLocal## This publishes a snapshot version of
>> the latest apache/samza into your local maven repo
>>
>>
>>
>>
>>
>> Then, head to hello-samza workspace and build again:
>>
>>
>>
>> cd 
>>
>>
>>
>> mvn clean package  ## This should create a build target
>>
>>
>>
>> ./bin/grid install zookeeper
>>
>>
>>
>> ./bin/grid start zookeeper
>>
>>
>>
>> ./bin/grid install kafka
>>
>>
>>
>> ./bin/grid start kafka
>>
>>
>>
>> ./bin/grid install yarn
>>
>>
>>
>> ./bin/grid start yarn
>>
>>
>>
>>
>>
>> mkdir -p deploy/samza
>>
>>
>>
>> tar -xvf ./target/hello-samza-*-SNAPSHOT-dist.tar.gz -C deploy/samza
>>
>>
>>
>>
>>
>> NOTE : BTW From the above steps I could not execute " gradle -b
>> bootstrap.gradle" since that command does not exists
>>
>>
>>
>> Regards,
>>
>> ~Sri
>>
>>
>>
>> -Original Message-
>> From: Yi Pan [mailto:nickpa...@gmail.com]
>> Sent: Tuesday, September 12, 2017 7:13 PM
>> To: dev@samza.apache.org
>> Subject: Re: Connection timed out error while installing "Hello Samza"
>>
>>
>>
>> Hi, Anantharam,
>>
>>
>>
>> Could you confirm at which step your setup failed? It seems that your
>> zookeeper server is running. Could you check to see whether your Kafka
>> broker is running? You can either do a telnet localhost 9092 or do a ps
>> auxww | grep kafka to see whether you got any broker running.
>>
>>
>>
>> Sometimes, the Kafka service takes time to start on a single laptop. You
>> can just try to run ./bin/grid start kafka
>>
>>
>>
>> again to see whether the service is up.
>>
>>
>>
>> Thanks!
>>
>>
>>
>> -Yi
>>
>>
>>
>> On Tue, Sep 12, 2017 at 11:38 AM, Anantharaman, Srinatha (Contractor) <
>> srinatha_ananthara...@comcast.com> a...@comcast.com>> wrote:
>>
>>
>>
>> > Find below Zookeeper logs:
>>
>> >
>>
>> > 2017-09-12 17:05:19,124 [myid:] - INFO  [main:QuorumPeerConfig@101] -
>>
>> > Reading configuration from: /app/home/eventsvc/samza-git/
>>
>> > hello-samza/deploy/zookeeper/bin/../conf/zoo.cfg
>>
>> > 2017-09-12 17:05:19,133 [myid:] - INFO
>>
>> > [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
>>
>> > 2017-09-12 17:05:19,134 [myid:] - INFO
>>
>> > [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
>>
>> > 2017-09-12 17:05:19,134 [myid:] - INFO
>>
>> > [main:DatadirCleanupManager@101]
>>
>> > - Purge task is not scheduled.
>>
>> > 2017-09-12 17:05:19,135 [myid:] - WARN  [main:QuorumPeerMain@113] -
>>
>> > Either no config or no quorum defined in config, running  in
>>
>> > standalone mode
>>
>> > 2017-09-12 17:05:19,161 [myid:] - INFO  [main:QuorumPeerConfig@101] -
>>
>> > Reading configuration from: /app/home/eventsvc/samza-git

RE: Connection timed out error while installing "Hello Samza"

[Anantharaman, Srinatha (Contractor)]

Yi,



Thank you for your kindly reply.



I am able to ping localhost and it is accessible.

BTW I am installing Samza on Linux server



Eagerly waiting for your help.





Regards,

~Sri





[cid:image001.png@01D32D68.6F1A2730]



-Original Message-
From: Yi Pan [mailto:nickpa...@gmail.com]
Sent: Thursday, September 14, 2017 2:22 PM
To: dev@samza.apache.org
Subject: Re: Connection timed out error while installing "Hello Samza"



Hi, Ananarath,



It is very strange that you are seeing this timeout exception that we do not 
see. I am trying to follow the exact steps you did to see whether there is 
anything broken. I will update you this afternoon.



Meanwhile, could you check your hostname setup and firewall configuration to 
see whether your local laptop has blocked access via the public IP address to 
your laptop? Could you verify that your localhost is resolved to

127.0.0.1 and is accessible?



-Yi



On Thu, Sep 14, 2017 at 11:18 AM, Anantharaman, Srinatha (Contractor) < 
srinatha_ananthara...@comcast.com> 
wrote:



> Yi,

>

> Is there any alternate way to install Samza  Or solution to the

> connection time out error?

>

> Regards,

> ~Sri

>

> From: Anantharaman, Srinatha (Contractor)

> Sent: Wednesday, September 13, 2017 11:37 AM

> To: dev@samza.apache.org

> Subject: RE: Connection timed out error while installing "Hello Samza"

>

>

> Yi,

>

>

>

> I am trying to build Samza locally by following the steps provided by

> Navina.

>

> As per those steps kafka will be installed after Zookeeper, I am

> getting Error while starting Zookeeper after it is installed

>

>

>

>

>

> Steps Followed :

>

>

>

> Yes. You can clone apache/samza locally and build it with:

>

>

>

>

>

> cd 

>

>

>

> gradle -b bootstrap.gradle

>

>

>

> ./gradlew clean build -x test

>

>

>

> ./gradlew publishToMavenLocal## This publishes a snapshot version of

> the latest apache/samza into your local maven repo

>

>

>

>

>

> Then, head to hello-samza workspace and build again:

>

>

>

> cd 

>

>

>

> mvn clean package  ## This should create a build target

>

>

>

> ./bin/grid install zookeeper

>

>

>

> ./bin/grid start zookeeper

>

>

>

> ./bin/grid install kafka

>

>

>

> ./bin/grid start kafka

>

>

>

> ./bin/grid install yarn

>

>

>

> ./bin/grid start yarn

>

>

>

>

>

> mkdir -p deploy/samza

>

>

>

> tar -xvf ./target/hello-samza-*-SNAPSHOT-dist.tar.gz -C deploy/samza

>

>

>

>

>

> NOTE : BTW From the above steps I could not execute " gradle -b

> bootstrap.gradle" since that command does not exists

>

>

>

> Regards,

>

> ~Sri

>

>

>

> -Original Message-

> From: Yi Pan [mailto:nickpa...@gmail.com]

> Sent: Tuesday, September 12, 2017 7:13 PM

> To: 
> dev@samza.apache.org>

> Subject: Re: Connection timed out error while installing "Hello Samza"

>

>

>

> Hi, Anantharam,

>

>

>

> Could you confirm at which step your setup failed? It seems that your

> zookeeper server is running. Could you check to see whether your Kafka

> broker is running? You can either do a telnet localhost 9092 or do a

> ps auxww | grep kafka to see whether you got any broker running.

>

>

>

> Sometimes, the Kafka service takes time to start on a single laptop.

> You can just try to run ./bin/grid start kafka

>

>

>

> again to see whether the service is up.

>

>

>

> Thanks!

>

>

>

> -Yi

>

>

>

> On Tue, Sep 12, 2017 at 11:38 AM, Anantharaman, Srinatha (Contractor)

> <

> srinatha_ananthara...@comcast.com

> .com>>

> wrote:

>

>

>

> > Find below Zookeeper logs:

>

> >

>

> > 2017-09-12 17:05:19,124 [myid:] - INFO  [main:QuorumPeerConfig@101]

> > -

>

> > Reading configuration from: /app/home/eventsvc/samza-git/

>

> > hello-samza/deploy/zookeeper/bin/../conf/zoo.cfg

>

> > 2017-09-12 17:05:19,133 [myid:] - INFO

>

> > [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3

>

> > 2017-09-12 17:05:19,134 [myid:] - INFO

>

> > [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0

>

> > 2017-09-12 17:05:19,134 [myid:] - INFO

>

> > [main:DatadirCleanupManager@101]

>

> > - Purge task is not scheduled.

>

> > 2017-09-12 17:05:19,135 [myid:] - WARN  [main:QuorumPeerMain@113] -

>

> > Either no config or no quorum defined in config, running  in

>

> > standalone mode

>

> > 2017-09-12 17:05:19,161 [myid:] - INFO  [main:QuorumPeerConfig@101]

> > -

>

> > Reading configuration from: /app/home/eventsvc/samza-git/

>

> > hello-samza/deploy/zookeeper/bin/../conf/zoo.cfg

>

> > 2017-09-12 17:05:19,162 [myid:] - INFO

> > [main:ZooKeeperServerMain@95]

>

> > - Starting server

>

> > 2017-09-12 17:05:19,175 [myid:] - INFO  [main

Re: Connection timed out error while installing "Hello Samza"

[Yi Pan]

Hi, Anantharaman,

I just did the same steps as you described in your email and all passed on
my box. Hence, I strongly suspect that it is related to your local laptop
network setup.

Could you post all the command line output when you ran the sequence of
commands?

-Yi

On Thu, Sep 14, 2017 at 11:21 AM, Yi Pan  wrote:

> Hi, Ananarath,
>
> It is very strange that you are seeing this timeout exception that we do
> not see. I am trying to follow the exact steps you did to see whether there
> is anything broken. I will update you this afternoon.
>
> Meanwhile, could you check your hostname setup and firewall configuration
> to see whether your local laptop has blocked access via the public IP
> address to your laptop? Could you verify that your localhost is resolved to
> 127.0.0.1 and is accessible?
>
> -Yi
>
> On Thu, Sep 14, 2017 at 11:18 AM, Anantharaman, Srinatha (Contractor) <
> srinatha_ananthara...@comcast.com> wrote:
>
>> Yi,
>>
>> Is there any alternate way to install Samza  Or solution to the
>> connection time out error?
>>
>> Regards,
>> ~Sri
>>
>> From: Anantharaman, Srinatha (Contractor)
>> Sent: Wednesday, September 13, 2017 11:37 AM
>> To: dev@samza.apache.org
>> Subject: RE: Connection timed out error while installing "Hello Samza"
>>
>>
>> Yi,
>>
>>
>>
>> I am trying to build Samza locally by following the steps provided by
>> Navina.
>>
>> As per those steps kafka will be installed after Zookeeper, I am getting
>> Error while starting Zookeeper after it is installed
>>
>>
>>
>>
>>
>> Steps Followed :
>>
>>
>>
>> Yes. You can clone apache/samza locally and build it with:
>>
>>
>>
>>
>>
>> cd 
>>
>>
>>
>> gradle -b bootstrap.gradle
>>
>>
>>
>> ./gradlew clean build -x test
>>
>>
>>
>> ./gradlew publishToMavenLocal## This publishes a snapshot version of
>> the latest apache/samza into your local maven repo
>>
>>
>>
>>
>>
>> Then, head to hello-samza workspace and build again:
>>
>>
>>
>> cd 
>>
>>
>>
>> mvn clean package  ## This should create a build target
>>
>>
>>
>> ./bin/grid install zookeeper
>>
>>
>>
>> ./bin/grid start zookeeper
>>
>>
>>
>> ./bin/grid install kafka
>>
>>
>>
>> ./bin/grid start kafka
>>
>>
>>
>> ./bin/grid install yarn
>>
>>
>>
>> ./bin/grid start yarn
>>
>>
>>
>>
>>
>> mkdir -p deploy/samza
>>
>>
>>
>> tar -xvf ./target/hello-samza-*-SNAPSHOT-dist.tar.gz -C deploy/samza
>>
>>
>>
>>
>>
>> NOTE : BTW From the above steps I could not execute " gradle -b
>> bootstrap.gradle" since that command does not exists
>>
>>
>>
>> Regards,
>>
>> ~Sri
>>
>>
>>
>> -Original Message-
>> From: Yi Pan [mailto:nickpa...@gmail.com]
>> Sent: Tuesday, September 12, 2017 7:13 PM
>> To: dev@samza.apache.org
>> Subject: Re: Connection timed out error while installing "Hello Samza"
>>
>>
>>
>> Hi, Anantharam,
>>
>>
>>
>> Could you confirm at which step your setup failed? It seems that your
>> zookeeper server is running. Could you check to see whether your Kafka
>> broker is running? You can either do a telnet localhost 9092 or do a ps
>> auxww | grep kafka to see whether you got any broker running.
>>
>>
>>
>> Sometimes, the Kafka service takes time to start on a single laptop. You
>> can just try to run ./bin/grid start kafka
>>
>>
>>
>> again to see whether the service is up.
>>
>>
>>
>> Thanks!
>>
>>
>>
>> -Yi
>>
>>
>>
>> On Tue, Sep 12, 2017 at 11:38 AM, Anantharaman, Srinatha (Contractor) <
>> srinatha_ananthara...@comcast.com> a...@comcast.com>> wrote:
>>
>>
>>
>> > Find below Zookeeper logs:
>>
>> >
>>
>> > 2017-09-12 17:05:19,124 [myid:] - INFO  [main:QuorumPeerConfig@101] -
>>
>> > Reading configuration from: /app/home/eventsvc/samza-git/
>>
>> > hello-samza/deploy/zookeeper/bin/../conf/zoo.cfg
>>
>> > 2017-09-12 17:05:19,133 [myid:] - INFO
>>
>> > [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
>>
>> > 2017-09-12 17:05:19,134 [myid:] - INFO
>>
>> > [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
>>
>> > 2017-09-12 17:05:19,134 [myid:] - INFO
>>
>> > [main:DatadirCleanupManager@101]
>>
>> > - Purge task is not scheduled.
>>
>> > 2017-09-12 17:05:19,135 [myid:] - WARN  [main:QuorumPeerMain@113] -
>>
>> > Either no config or no quorum defined in config, running  in
>>
>> > standalone mode
>>
>> > 2017-09-12 17:05:19,161 [myid:] - INFO  [main:QuorumPeerConfig@101] -
>>
>> > Reading configuration from: /app/home/eventsvc/samza-git/
>>
>> > hello-samza/deploy/zookeeper/bin/../conf/zoo.cfg
>>
>> > 2017-09-12 17:05:19,162 [myid:] - INFO  [main:ZooKeeperServerMain@95]
>>
>> > - Starting server
>>
>> > 2017-09-12 17:05:19,175 [myid:] - INFO  [main:Environment@98] - Server
>>
>> > environment:zookeeper.version=3.4.3-1240972, built on 02/06/2012 10:48
>>
>> > GMT
>>
>> > 2017-09-12 17:05:19,176 [myid:] - INFO  [main:Environment@98] - Server
>>
>> > environment:host.name=codehdplak-po-r19p.sys.comcast.net
>>
>> > 2017-09-12 17:05:19,176 [myid:] - INFO  [main:Environment@98] - Server
>>
>>

Re: Connection timed out error while installing "Hello Samza"

[Yi Pan]

Hi, Ananarath,

It is very strange that you are seeing this timeout exception that we do
not see. I am trying to follow the exact steps you did to see whether there
is anything broken. I will update you this afternoon.

Meanwhile, could you check your hostname setup and firewall configuration
to see whether your local laptop has blocked access via the public IP
address to your laptop? Could you verify that your localhost is resolved to
127.0.0.1 and is accessible?

-Yi

On Thu, Sep 14, 2017 at 11:18 AM, Anantharaman, Srinatha (Contractor) <
srinatha_ananthara...@comcast.com> wrote:

> Yi,
>
> Is there any alternate way to install Samza  Or solution to the connection
> time out error?
>
> Regards,
> ~Sri
>
> From: Anantharaman, Srinatha (Contractor)
> Sent: Wednesday, September 13, 2017 11:37 AM
> To: dev@samza.apache.org
> Subject: RE: Connection timed out error while installing "Hello Samza"
>
>
> Yi,
>
>
>
> I am trying to build Samza locally by following the steps provided by
> Navina.
>
> As per those steps kafka will be installed after Zookeeper, I am getting
> Error while starting Zookeeper after it is installed
>
>
>
>
>
> Steps Followed :
>
>
>
> Yes. You can clone apache/samza locally and build it with:
>
>
>
>
>
> cd 
>
>
>
> gradle -b bootstrap.gradle
>
>
>
> ./gradlew clean build -x test
>
>
>
> ./gradlew publishToMavenLocal## This publishes a snapshot version of
> the latest apache/samza into your local maven repo
>
>
>
>
>
> Then, head to hello-samza workspace and build again:
>
>
>
> cd 
>
>
>
> mvn clean package  ## This should create a build target
>
>
>
> ./bin/grid install zookeeper
>
>
>
> ./bin/grid start zookeeper
>
>
>
> ./bin/grid install kafka
>
>
>
> ./bin/grid start kafka
>
>
>
> ./bin/grid install yarn
>
>
>
> ./bin/grid start yarn
>
>
>
>
>
> mkdir -p deploy/samza
>
>
>
> tar -xvf ./target/hello-samza-*-SNAPSHOT-dist.tar.gz -C deploy/samza
>
>
>
>
>
> NOTE : BTW From the above steps I could not execute " gradle -b
> bootstrap.gradle" since that command does not exists
>
>
>
> Regards,
>
> ~Sri
>
>
>
> -Original Message-
> From: Yi Pan [mailto:nickpa...@gmail.com]
> Sent: Tuesday, September 12, 2017 7:13 PM
> To: dev@samza.apache.org
> Subject: Re: Connection timed out error while installing "Hello Samza"
>
>
>
> Hi, Anantharam,
>
>
>
> Could you confirm at which step your setup failed? It seems that your
> zookeeper server is running. Could you check to see whether your Kafka
> broker is running? You can either do a telnet localhost 9092 or do a ps
> auxww | grep kafka to see whether you got any broker running.
>
>
>
> Sometimes, the Kafka service takes time to start on a single laptop. You
> can just try to run ./bin/grid start kafka
>
>
>
> again to see whether the service is up.
>
>
>
> Thanks!
>
>
>
> -Yi
>
>
>
> On Tue, Sep 12, 2017 at 11:38 AM, Anantharaman, Srinatha (Contractor) <
> srinatha_ananthara...@comcast.com>
> wrote:
>
>
>
> > Find below Zookeeper logs:
>
> >
>
> > 2017-09-12 17:05:19,124 [myid:] - INFO  [main:QuorumPeerConfig@101] -
>
> > Reading configuration from: /app/home/eventsvc/samza-git/
>
> > hello-samza/deploy/zookeeper/bin/../conf/zoo.cfg
>
> > 2017-09-12 17:05:19,133 [myid:] - INFO
>
> > [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
>
> > 2017-09-12 17:05:19,134 [myid:] - INFO
>
> > [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
>
> > 2017-09-12 17:05:19,134 [myid:] - INFO
>
> > [main:DatadirCleanupManager@101]
>
> > - Purge task is not scheduled.
>
> > 2017-09-12 17:05:19,135 [myid:] - WARN  [main:QuorumPeerMain@113] -
>
> > Either no config or no quorum defined in config, running  in
>
> > standalone mode
>
> > 2017-09-12 17:05:19,161 [myid:] - INFO  [main:QuorumPeerConfig@101] -
>
> > Reading configuration from: /app/home/eventsvc/samza-git/
>
> > hello-samza/deploy/zookeeper/bin/../conf/zoo.cfg
>
> > 2017-09-12 17:05:19,162 [myid:] - INFO  [main:ZooKeeperServerMain@95]
>
> > - Starting server
>
> > 2017-09-12 17:05:19,175 [myid:] - INFO  [main:Environment@98] - Server
>
> > environment:zookeeper.version=3.4.3-1240972, built on 02/06/2012 10:48
>
> > GMT
>
> > 2017-09-12 17:05:19,176 [myid:] - INFO  [main:Environment@98] - Server
>
> > environment:host.name=codehdplak-po-r19p.sys.comcast.net
>
> > 2017-09-12 17:05:19,176 [myid:] - INFO  [main:Environment@98] - Server
>
> > environment:java.version=1.8.0_31
>
> > 2017-09-12 17:05:19,177 [myid:] - INFO  [main:Environment@98] - Server
>
> > environment:java.vendor=Oracle Corporation
>
> > 2017-09-12 17:05:19,177 [myid:] - INFO  [main:Environment@98] - Server
>
> > environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.
>
> > 8.0.31-2.b13.el7.x86_64/jre
>
> > 2017-09-12 17:05:19,178 [myid:] - INFO  [main:Environment@98] - Server
>
> > environment:java.class.path=/app/home/eventsvc/samza-git/
>
> > hello-samza/deploy/zookeeper/bin/../build/classes:/app/
>
> > home/eventsvc/sam

RE: Connection timed out error while installing "Hello Samza"

[Anantharaman, Srinatha (Contractor)]

Yi,

Is there any alternate way to install Samza  Or solution to the connection time 
out error?

Regards,
~Sri

From: Anantharaman, Srinatha (Contractor)
Sent: Wednesday, September 13, 2017 11:37 AM
To: dev@samza.apache.org
Subject: RE: Connection timed out error while installing "Hello Samza"


Yi,



I am trying to build Samza locally by following the steps provided by Navina.

As per those steps kafka will be installed after Zookeeper, I am getting  Error 
while starting Zookeeper after it is installed





Steps Followed :



Yes. You can clone apache/samza locally and build it with:





cd 



gradle -b bootstrap.gradle



./gradlew clean build -x test



./gradlew publishToMavenLocal## This publishes a snapshot version of the 
latest apache/samza into your local maven repo





Then, head to hello-samza workspace and build again:



cd 



mvn clean package  ## This should create a build target



./bin/grid install zookeeper



./bin/grid start zookeeper



./bin/grid install kafka



./bin/grid start kafka



./bin/grid install yarn



./bin/grid start yarn





mkdir -p deploy/samza



tar -xvf ./target/hello-samza-*-SNAPSHOT-dist.tar.gz -C deploy/samza





NOTE : BTW From the above steps I could not execute " gradle -b 
bootstrap.gradle" since that command does not exists



Regards,

~Sri



-Original Message-
From: Yi Pan [mailto:nickpa...@gmail.com]
Sent: Tuesday, September 12, 2017 7:13 PM
To: dev@samza.apache.org
Subject: Re: Connection timed out error while installing "Hello Samza"



Hi, Anantharam,



Could you confirm at which step your setup failed? It seems that your zookeeper 
server is running. Could you check to see whether your Kafka broker is running? 
You can either do a telnet localhost 9092 or do a ps auxww | grep kafka to see 
whether you got any broker running.



Sometimes, the Kafka service takes time to start on a single laptop. You can 
just try to run ./bin/grid start kafka



again to see whether the service is up.



Thanks!



-Yi



On Tue, Sep 12, 2017 at 11:38 AM, Anantharaman, Srinatha (Contractor) < 
srinatha_ananthara...@comcast.com> 
wrote:



> Find below Zookeeper logs:

>

> 2017-09-12 17:05:19,124 [myid:] - INFO  [main:QuorumPeerConfig@101] -

> Reading configuration from: /app/home/eventsvc/samza-git/

> hello-samza/deploy/zookeeper/bin/../conf/zoo.cfg

> 2017-09-12 17:05:19,133 [myid:] - INFO

> [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3

> 2017-09-12 17:05:19,134 [myid:] - INFO

> [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0

> 2017-09-12 17:05:19,134 [myid:] - INFO

> [main:DatadirCleanupManager@101]

> - Purge task is not scheduled.

> 2017-09-12 17:05:19,135 [myid:] - WARN  [main:QuorumPeerMain@113] -

> Either no config or no quorum defined in config, running  in

> standalone mode

> 2017-09-12 17:05:19,161 [myid:] - INFO  [main:QuorumPeerConfig@101] -

> Reading configuration from: /app/home/eventsvc/samza-git/

> hello-samza/deploy/zookeeper/bin/../conf/zoo.cfg

> 2017-09-12 17:05:19,162 [myid:] - INFO  [main:ZooKeeperServerMain@95]

> - Starting server

> 2017-09-12 17:05:19,175 [myid:] - INFO  [main:Environment@98] - Server

> environment:zookeeper.version=3.4.3-1240972, built on 02/06/2012 10:48

> GMT

> 2017-09-12 17:05:19,176 [myid:] - INFO  [main:Environment@98] - Server

> environment:host.name=codehdplak-po-r19p.sys.comcast.net

> 2017-09-12 17:05:19,176 [myid:] - INFO  [main:Environment@98] - Server

> environment:java.version=1.8.0_31

> 2017-09-12 17:05:19,177 [myid:] - INFO  [main:Environment@98] - Server

> environment:java.vendor=Oracle Corporation

> 2017-09-12 17:05:19,177 [myid:] - INFO  [main:Environment@98] - Server

> environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.

> 8.0.31-2.b13.el7.x86_64/jre

> 2017-09-12 17:05:19,178 [myid:] - INFO  [main:Environment@98] - Server

> environment:java.class.path=/app/home/eventsvc/samza-git/

> hello-samza/deploy/zookeeper/bin/../build/classes:/app/

> home/eventsvc/samza-git/hello-samza/deploy/zookeeper/bin/../

> build/lib/*.jar:/app/home/eventsvc/samza-git/hello-

> samza/deploy/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/

> app/home/eventsvc/samza-git/hello-samza/deploy/zookeeper/

> bin/../lib/slf4j-api-1.6.1.jar:/app/home/eventsvc/samza-

> git/hello-samza/deploy/zookeeper/bin/../lib/netty-3.

> 2.2.Final.jar:/app/home/eventsvc/samza-git/hello-

> samza/deploy/zookeeper/bin/../lib/log4j-1.2.15.jar:/app/

> home/eventsvc/samza-git/hello-samza/deploy/zookeeper/bin/../

> lib/jline-0.9.94.jar:/app/home/eventsvc/samza-git/hello-

> samza/deploy/zookeeper/bin/../zookeeper-3.4.3.jar:/app/home/

> eventsvc/samza-git/hello-samza/deploy/zookeeper/bin/../

> src/java/lib/*.jar:/app/home/eventsvc/samza-git/hello-

> samza/deploy/zookeeper/bin/../conf:

> 2017-09-12 17:05:19,178 [myid:] - INFO  [main:Environment@98] - Server

> environment:java.library.

Re: Kafka 0.10.2 compatibility with Samza 0.13.1

[Yi Pan]

Hi, XiaoChuan,

Great! So I assume that you were able to upgrade to Samza 0.13.1
successfully? If you need any further help, feel free to send us emails!

Thanks!

-Yi

On Thu, Sep 14, 2017 at 8:11 AM, XiaoChuan Yu  wrote:

> Hi Yi,
>
> Thanks for all the suggestions.
> Our team eventually decided to move off CDH Kafka(and YARN) due to some
> issues not relevant to Samza.
> For what its worth, I did test the a couple basic Hello Samza (Samza
> 0.13.1) jobs against a single node CDH Kafka version
> 0.10.2.0+kafka2.2.0+110 with success.
> The testing was done on AWS EC2 instances running RedHat 7.
> We also found out that running against CDH YARN required various dependency
> overrides such as the ones in Hello Samza's "cdh5" build profile
>  >.
>
> Thanks,
> Xiaochuan Yu
>
> On Fri, Sep 1, 2017 at 1:18 PM Yi Pan  wrote:
>
> > Hi, XiaoChuan,
> >
> > Just confirmed with Jiangjie Qin (our in-house Kafka PMC), Kafka client
> > 0.10.1.1 does not work with broker 0.10.0 due to new protocol version
> > introduced in 0.10.1.1. Hence, if you try to run Samza 0.13.1 against
> Kafka
> > broker 0.10.0 version, it won't work. The best option for that seems to
> be
> > swapping out the runtime Kafka client 0.10.1.1 version in Samza 0.13.1 to
> > Kafka client 0.10.0.1 (which was used in Samza 0.12). Since there is no
> > incompatible API calls to Kafka client lib between Samza 0.12.0 and Samza
> > 0.13.1, that should work and is worth trying out.
> >
> > Let us know if we can be further assistance here.
> >
> > Thanks!
> >
> > -Yi
> >
> > On Thu, Aug 31, 2017 at 4:14 PM, Yi Pan  wrote:
> >
> > > Hi, Xiaochuan,
> > >
> > > I am relaying your question to our in-house Kafka expert to confirm.
> But
> > > as far as I can tell from the description, the main breaking changes
> are
> > in
> > > the KStreams API. Samza only uses the standard Kafka consumer/producer
> > APIs
> > > now and should not be affected.
> > >
> > > I would recommend to give it a try in your question 1.
> > >
> > > As for your question 2, are you referring to a) binding a Kafka
> 0.10.0.0
> > > client library with Samza 0.13.1? Or b) running Samza 0.13.1 against
> > Kafka
> > > broker version 0.10.0.0?
> > >
> > > If you are referring to a), from the commit history, there is no source
> > > code level changes when we upgrade the dependency from Kafka 0.10.0.1
> to
> > > 0.10.1.1 in Samza 0.13.1. If you want to make sure that works, you can
> > try
> > > to downgrade the Kafka version in your build environment to 0.10.0.0
> and
> > > build your app to see whether it breaks the build. If the build passes,
> > the
> > > client should work w/ Samza 0.13.1, unless Kafka client has an API that
> > > changes runtime behavior between 0.10.0.1 and 0.10.1.1.
> > >
> > > If you are referring to b), it is not guaranteed that the higher
> version
> > > of Kafka client library will work with an older version of Kafka
> broker.
> > > However, Samza has been very conservative in the usage of new features
> in
> > > Kafka client library. Hence, you would have a higher chance of success
> to
> > > run Samza 0.13.1 against Kafka 0.10.0 brokers.
> > >
> > > Unfortunately, we have not test all the above combinations internally.
> I
> > > would wait for the confirmation from our Kafka expert before I can give
> > you
> > > a definite answer.
> > >
> > > Thanks!
> > >
> > > -Yi
> > >
> >
>

Re: Kafka 0.10.2 compatibility with Samza 0.13.1

[XiaoChuan Yu]

Hi Yi,

Thanks for all the suggestions.
Our team eventually decided to move off CDH Kafka(and YARN) due to some
issues not relevant to Samza.
For what its worth, I did test the a couple basic Hello Samza (Samza
0.13.1) jobs against a single node CDH Kafka version
0.10.2.0+kafka2.2.0+110 with success.
The testing was done on AWS EC2 instances running RedHat 7.
We also found out that running against CDH YARN required various dependency
overrides such as the ones in Hello Samza's "cdh5" build profile
.

Thanks,
Xiaochuan Yu

On Fri, Sep 1, 2017 at 1:18 PM Yi Pan  wrote:

> Hi, XiaoChuan,
>
> Just confirmed with Jiangjie Qin (our in-house Kafka PMC), Kafka client
> 0.10.1.1 does not work with broker 0.10.0 due to new protocol version
> introduced in 0.10.1.1. Hence, if you try to run Samza 0.13.1 against Kafka
> broker 0.10.0 version, it won't work. The best option for that seems to be
> swapping out the runtime Kafka client 0.10.1.1 version in Samza 0.13.1 to
> Kafka client 0.10.0.1 (which was used in Samza 0.12). Since there is no
> incompatible API calls to Kafka client lib between Samza 0.12.0 and Samza
> 0.13.1, that should work and is worth trying out.
>
> Let us know if we can be further assistance here.
>
> Thanks!
>
> -Yi
>
> On Thu, Aug 31, 2017 at 4:14 PM, Yi Pan  wrote:
>
> > Hi, Xiaochuan,
> >
> > I am relaying your question to our in-house Kafka expert to confirm. But
> > as far as I can tell from the description, the main breaking changes are
> in
> > the KStreams API. Samza only uses the standard Kafka consumer/producer
> APIs
> > now and should not be affected.
> >
> > I would recommend to give it a try in your question 1.
> >
> > As for your question 2, are you referring to a) binding a Kafka 0.10.0.0
> > client library with Samza 0.13.1? Or b) running Samza 0.13.1 against
> Kafka
> > broker version 0.10.0.0?
> >
> > If you are referring to a), from the commit history, there is no source
> > code level changes when we upgrade the dependency from Kafka 0.10.0.1 to
> > 0.10.1.1 in Samza 0.13.1. If you want to make sure that works, you can
> try
> > to downgrade the Kafka version in your build environment to 0.10.0.0 and
> > build your app to see whether it breaks the build. If the build passes,
> the
> > client should work w/ Samza 0.13.1, unless Kafka client has an API that
> > changes runtime behavior between 0.10.0.1 and 0.10.1.1.
> >
> > If you are referring to b), it is not guaranteed that the higher version
> > of Kafka client library will work with an older version of Kafka broker.
> > However, Samza has been very conservative in the usage of new features in
> > Kafka client library. Hence, you would have a higher chance of success to
> > run Samza 0.13.1 against Kafka 0.10.0 brokers.
> >
> > Unfortunately, we have not test all the above combinations internally. I
> > would wait for the confirmation from our Kafka expert before I can give
> you
> > a definite answer.
> >
> > Thanks!
> >
> > -Yi
> >
>

Re: [VOTE] SEP-8: Add in-memory system consumer & producer

[Yi Pan]

Hi, Bharath,

Overall looks good! I have the following comments:

i) Question on the Type of IME + data partition:

How do we enforce that user adds IME w/ the expected partition id to the
corresponding sub-collection?



ii) In the architecture graph, what's the difference between SSP queues and
Data source/sink? What is the layer exposed to the user (I.e. programmer)?



ii) Agree w/ the approach to use a customized queues managed by the admin.
However, the reason not to use BEM is not very clear. For the matter of
fact, BEM is just one optional base class for SystemConsumer implementation.
Not sure why we necessarily need to be limited by BEM.



iii) In the code examples,

A) what's the difference between durable state vs non-durable state in
highlevel API? I don't see any difference. Also, the SEP has clearly
described that the design is only for InMemory input/output/intermediate
streams. I noticed that you added changelog as inputs in low-level API. But
it is not clear how this changelog is defined and why it is an input to the
application???

B) the code example for checkpoint is empty and we have stated that we
won't support checkpoint in this SEP. Can we remove it?


Thanks!


-Yi

On Wed, Sep 6, 2017 at 2:06 PM, xinyu liu  wrote:

> +1 on the overall design. This will make testing a lot easier!
>
> Thanks,
> Xinyu
>
> On Wed, Sep 6, 2017 at 10:45 AM, Bharath Kumara Subramanian <
> codin.mart...@gmail.com> wrote:
>
> > Hi all,
> >
> > Can you please vote for SEP-8?
> > You can find the design document here
> >  action?pageId=71013043
> > >.
> >
> > Thanks,
> > Bharath
> >
>