Re: Message throughput per day - Kafka

[siva prasad]

Thanks for Responding.

I am not looking for throughput achieved by the broker.

Metrics is needed to see how the platform is used for business, to know
daily usage trends and variation after every release. It also helps to do
capacity planning.

I am not producer/consumer. My part is to host Kafka clusters and enable
all functionalities for clients to make use of it.

Cheers,
Siva

On Tue, Feb 20, 2018 at 12:55 PM, Sharninder  wrote:

> >
> > >
> > > I am wondering if there is way to get messages through put of Kafka
> > > brokers.
> > >
> > > Ex:
> > > 1) Number of messages sent per day to a Broker/Cluster
> > > 2) Number of messages consumed per day by a Broker/Cluster
> > >
> >
>
> I don't think it makes sense to have total metrics per day for
> brokers/producers. There are per second metrics such as MessagesInPerSec at
> brokers that you can use to gauge throughput. If you really want total
> metrics, measure at your client/producer end yourself.
>
> --
> Sharninder
>



-- 
Cheers,
Siva

Re: Message throughput per day - Kafka

[Sharninder]

>
> >
> > I am wondering if there is way to get messages through put of Kafka
> > brokers.
> >
> > Ex:
> > 1) Number of messages sent per day to a Broker/Cluster
> > 2) Number of messages consumed per day by a Broker/Cluster
> >
>

I don't think it makes sense to have total metrics per day for
brokers/producers. There are per second metrics such as MessagesInPerSec at
brokers that you can use to gauge throughput. If you really want total
metrics, measure at your client/producer end yourself.

--
Sharninder

Re: Message throughput per day - Kafka

[Sharath Gururaj]

There are no statistics on a per-day basis. Kafka exposes metrics for
producer throughput per sec. Both in terms of byte rate as well as number
of messages. you'll have to write some sort of a cron to periodically
sample it.

If you want exact numbers, then write a cron job to get the current offset
at the start of the day and current offset at the end of the day and
subtract it. you'll have to sum it up over all topicPartitions.

On Tue, Feb 20, 2018 at 12:37 PM, siva prasad 
wrote:

> Guys,
>
> You have any details on queried ?
>
> Cheers,
> Siva
>
> On Fri, Feb 16, 2018 at 4:56 PM, siva prasad 
> wrote:
>
> > Hey Guys,
> >
> >
> > I am wondering if there is way to get messages through put of Kafka
> > brokers.
> >
> > Ex:
> > 1) Number of messages sent per day to a Broker/Cluster
> > 2) Number of messages consumed per day by a Broker/Cluster
> >
> > It should be cumulative from all topics on a cluster. Also, count
> > shouldn't consider replication.
> >
> > Ex:
> > If a single message sent to a topic with replication factor of 3 then
> > message count should be 1.
> >
> > These statistics will be helpful to business.
> >
> > --
> > Cheers,
> > Siva
> >
>
>
>
> --
> Cheers,
> Siva
>

Re: Message throughput per day - Kafka

[siva prasad]

Guys,

You have any details on queried ?

Cheers,
Siva

On Fri, Feb 16, 2018 at 4:56 PM, siva prasad 
wrote:

> Hey Guys,
>
>
> I am wondering if there is way to get messages through put of Kafka
> brokers.
>
> Ex:
> 1) Number of messages sent per day to a Broker/Cluster
> 2) Number of messages consumed per day by a Broker/Cluster
>
> It should be cumulative from all topics on a cluster. Also, count
> shouldn't consider replication.
>
> Ex:
> If a single message sent to a topic with replication factor of 3 then
> message count should be 1.
>
> These statistics will be helpful to business.
>
> --
> Cheers,
> Siva
>



-- 
Cheers,
Siva

Re: Testing with MockConsumer

[Ted Yu]

For #3, a better example would be in ConsumerCoordinator (around line 632).

commitOffsetsAsync(allConsumedOffsets, new OffsetCommitCallback() {
@Override
public void onComplete(Map
offsets, Exception exception) {

FYI

On Mon, Feb 19, 2018 at 10:56 AM, Gabriel Giussi 
wrote:

> Hi Ted,
> my mistake was believe that commited offsets are used on the next poll, but
> is not the case
>  4018cf7d3a/clients/src/main/java/org/apache/kafka/clients/
> consumer/KafkaConsumer.java#L1202>
> .
>
> > The offsets committed using this API will be used on the first fetch
> after
> > every rebalance and also on startup
> >
>
> So, what to do after a failed commit depends on the nature of the exception
> I guess.
>
>- WakeupException: retry
>- Others: close consumer
>
> Thanks for your help to solve #2. I'm wondering about 1# and 3# yet.
>
> 2018-02-19 11:46 GMT-03:00 Ted Yu :
>
> > For #2, I think the assumption is that the records are processed by the
> > loop:
> >
> > https://github.com/apache/kafka/blob/73be1e1168f91ee2a9d68e1d1c75c1
> > 4018cf7d3a/clients/src/main/java/org/apache/kafka/clients/
> > consumer/MockConsumer.java#L164
> >
> >
> >
> > On Mon, Feb 19, 2018 at 4:39 AM, Gabriel Giussi  >
> > wrote:
> >
> > > Hi,
> > >
> > > I'm trying to use MockConsumer to test my application code but I've
> > faced a
> > > couple of limitations and I want to know if there are workarounds or
> > > something that I'm overlooking.
> > > Note: I'm using kafka-clients v 0.11.0.2
> > >
> > >
> > >1. Why the addRecord
> > > 73be1e1168f91ee2a9d68e1d1c75c1
> > > 4018cf7d3a/clients/src/main/java/org/apache/kafka/clients/
> > > consumer/MockConsumer.java#L179>
> > >requires that the consumer has assigned partitions? Given that this
> is
> > > just
> > >simulating records being produced or existing records.
> > >2. Why the poll
> > > 73be1e1168f91ee2a9d68e1d1c75c1
> > > 4018cf7d3a/clients/src/main/java/org/apache/kafka/clients/
> > > consumer/MockConsumer.java#L132>
> > >clear the map of records? It should not be cleared after commit?
> > >3. Why the commitAsync
> > > 73be1e1168f91ee2a9d68e1d1c75c1
> > > 4018cf7d3a/clients/src/main/java/org/apache/kafka/clients/
> > > consumer/MockConsumer.java#L198>
> > >doesn't check for an exception and always succeed?
> > >
> > > Due to items (2) and (3) I'm not be able to test scenarios where the
> > > commits fails and the consumer should poll again the same elements.
> > >
> > > If someone knows about other scenarios that can't be tested with
> > > MockConsumer, please let me know.
> > >
> > > Thanks.
> > >
> >
>

Re: KafkaUtils.createStream(..) is removed for API

[Cody Koeninger]

I can't speak for committers, but my guess is it's more likely for
DStreams in general to stop being supported before that particular
integration is removed.

On Sun, Feb 18, 2018 at 9:34 PM, naresh Goud  wrote:
> Thanks Ted.
>
> I see  createDirectStream is experimental as annotated with
> "org.apache.spark.annotation.Experimental".
>
> Is it possible to be this API will be removed in future?  because we wanted
> to use this API in one of our production jobs. afraid if it will not be
> supported in future.
>
> Thank you,
> Naresh
>
>
>
>
> On Sun, Feb 18, 2018 at 7:47 PM, Ted Yu  wrote:
>>
>> createStream() is still in
>> external/kafka-0-8/src/main/scala/org/apache/spark/streaming/kafka/KafkaUtils.scala
>> But it is not in
>> external/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/KafkaUtils.scala
>>
>> FYI
>>
>> On Sun, Feb 18, 2018 at 5:17 PM, naresh Goud 
>> wrote:
>>>
>>> Hello Team,
>>>
>>> I see "KafkaUtils.createStream() " method not available in spark 2.2.1.
>>>
>>> Can someone please confirm if these methods are removed?
>>>
>>> below is my pom.xml entries.
>>>
>>>
>>> 
>>>   2.11.8
>>>   2.11
>>> 
>>>
>>>
>>>   
>>>   org.apache.spark
>>>   spark-streaming_${scala.tools.version}
>>>   2.2.1
>>>   provided
>>>   
>>> 
>>>   org.apache.spark
>>>   spark-streaming-kafka-0-10_2.11
>>>   2.2.1
>>>   provided
>>> 
>>> 
>>>   org.apache.spark
>>>   spark-core_2.11
>>>   2.2.1
>>>   provided
>>> 
>>>   
>>>
>>>
>>>
>>>
>>>
>>> Thank you,
>>> Naresh
>>
>>
>

Re: timestamp-oriented API

[Matthias J. Sax]

The broker maintains a timestamp index and uses this timestamp index to
answer the "offsetForTimes" request.

The returned offset guarantees, that there is no record with a smaller
timestamp and smaller offset in the topic. Thus, if there are
out-of-order records in the topic, and you start reading from the
returned offset, you might see record with _smaller_ timestamps than the
specified in offsetForTimes().

Ie, it is guaranteed, that you receive _all_ records that have a
timestamp equal or larger than the specified one (however, it is not
guaranteed that you get _only_ record with equal or larger timestamps --
there might be later records with smaller timestamps, too).


-Matthias



On 2/19/18 10:00 AM, Steve Jang wrote:
> If you set *message.timestamp.type* (or *log.message.timestamp.type*) to be
> LogAppendTime, this would make sense.
> 
> I am new to Kafka, too, and if this was set to CreateTime, I don't know
> what the behavior would be.  There is *message.timestamp.difference.max.ms
> * setting too, so there seem to
> be certain "boundedness" of how much clock skew is allowed between the
> producer and the broker, so you could implement various types of policies
> (min, max, etc) for this API.
> 
> 
> On Mon, Feb 19, 2018 at 7:36 AM, Xavier Noria  wrote:
> 
>> In the mental model I am building of how Kafka works (new to this), the
>> broker keeps offsets by consumer group, and individual consumers basically
>> depend on the offset of the consumer group they join. Also consumer groups
>> may opt to start from the beginning.
>>
>> OK, in that mental model there is a linearization of messages per
>> partition. As the documentation says, there is a total order per partition,
>> and the order is based on the offset, unrelated to the timestamp.
>>
>> But I see the Java library has timestamp-oriented methods like:
>>
>>
>> https://kafka.apache.org/0102/javadoc/org/apache/kafka/
>> clients/consumer/Consumer.html#offsetsForTimes(java.util.Map)
>>
>> How does that make sense given the model described above? How is that
>> implemented? Does the broker has builtin support for this? What happens if
>> due to race conditions or machines with clocks out of sync you have
>> messages with timestamps interleaved?
>>
>> Could anyone concile that API with the intrinsec offset-based contract?
>>
> 
> 
> 



signature.asc
Description: OpenPGP digital signature

Re: Aggregation events Stream Kafka

[Matthias J. Sax]

Using Kafka's Streams API sound like a very good solution to  your
problem. I'd recommend to check out the docs and examples:

https://kafka.apache.org/10/documentation/streams/

https://github.com/confluentinc/kafka-streams-examples


-Matthias


On 2/19/18 1:19 AM, Maria Pilar wrote:
> Hi
> 
> I need to create aggretions events and publish them in other topic for a
> Stream Kafka API.
> 
> I usually i have done aggregates events with Apache Spark, however it
> requires include a new bussines layer into our E2e solution.
> 
> I have checked the possibility to use aggreate method with KTABLE.
> 
> Do you know any best solution for that?
> 
> Thanks
> 



signature.asc
Description: OpenPGP digital signature

connecting to 2 different clusters with different sasl realms in a single process

[Michal Hariš]

Hi all,

I have one cluster with kerberos authenticator and another with simple
authenticator. I need to be able to consume certain topics from the
kerberized cluster and produce into the cluster with the simple auth. The
ACLs on both cluster work well for the purpose but I can't see the way how
to setup a simple mirror maker - or any other process that can connect at
the same time to 2 separate kafka security realms. The problem is that one
jvm process can only have one property:

Djava.security.auth.login.config=/path/to/server/jaas/file.conf

The Jaas file could have multiple client identifiers but from what i've
read in the docs this is expected always to be KafkaClient.

Has anybody faced the same issue ? Is it somehow not correct to be
connecting to 2 different realms - this would mean that any global
replication would have to also have a single, global authentication realm.

Thanks,
Michal

Re: Testing with MockConsumer

[Gabriel Giussi]

Hi Ted,
my mistake was believe that commited offsets are used on the next poll, but
is not the case

.

> The offsets committed using this API will be used on the first fetch after
> every rebalance and also on startup
>

So, what to do after a failed commit depends on the nature of the exception
I guess.

   - WakeupException: retry
   - Others: close consumer

Thanks for your help to solve #2. I'm wondering about 1# and 3# yet.

2018-02-19 11:46 GMT-03:00 Ted Yu :

> For #2, I think the assumption is that the records are processed by the
> loop:
>
> https://github.com/apache/kafka/blob/73be1e1168f91ee2a9d68e1d1c75c1
> 4018cf7d3a/clients/src/main/java/org/apache/kafka/clients/
> consumer/MockConsumer.java#L164
>
>
>
> On Mon, Feb 19, 2018 at 4:39 AM, Gabriel Giussi 
> wrote:
>
> > Hi,
> >
> > I'm trying to use MockConsumer to test my application code but I've
> faced a
> > couple of limitations and I want to know if there are workarounds or
> > something that I'm overlooking.
> > Note: I'm using kafka-clients v 0.11.0.2
> >
> >
> >1. Why the addRecord
> > > 4018cf7d3a/clients/src/main/java/org/apache/kafka/clients/
> > consumer/MockConsumer.java#L179>
> >requires that the consumer has assigned partitions? Given that this is
> > just
> >simulating records being produced or existing records.
> >2. Why the poll
> > > 4018cf7d3a/clients/src/main/java/org/apache/kafka/clients/
> > consumer/MockConsumer.java#L132>
> >clear the map of records? It should not be cleared after commit?
> >3. Why the commitAsync
> > > 4018cf7d3a/clients/src/main/java/org/apache/kafka/clients/
> > consumer/MockConsumer.java#L198>
> >doesn't check for an exception and always succeed?
> >
> > Due to items (2) and (3) I'm not be able to test scenarios where the
> > commits fails and the consumer should poll again the same elements.
> >
> > If someone knows about other scenarios that can't be tested with
> > MockConsumer, please let me know.
> >
> > Thanks.
> >
>

Re: timestamp-oriented API

[Steve Jang]

If you set *message.timestamp.type* (or *log.message.timestamp.type*) to be
LogAppendTime, this would make sense.

I am new to Kafka, too, and if this was set to CreateTime, I don't know
what the behavior would be.  There is *message.timestamp.difference.max.ms
* setting too, so there seem to
be certain "boundedness" of how much clock skew is allowed between the
producer and the broker, so you could implement various types of policies
(min, max, etc) for this API.


On Mon, Feb 19, 2018 at 7:36 AM, Xavier Noria  wrote:

> In the mental model I am building of how Kafka works (new to this), the
> broker keeps offsets by consumer group, and individual consumers basically
> depend on the offset of the consumer group they join. Also consumer groups
> may opt to start from the beginning.
>
> OK, in that mental model there is a linearization of messages per
> partition. As the documentation says, there is a total order per partition,
> and the order is based on the offset, unrelated to the timestamp.
>
> But I see the Java library has timestamp-oriented methods like:
>
>
> https://kafka.apache.org/0102/javadoc/org/apache/kafka/
> clients/consumer/Consumer.html#offsetsForTimes(java.util.Map)
>
> How does that make sense given the model described above? How is that
> implemented? Does the broker has builtin support for this? What happens if
> due to race conditions or machines with clocks out of sync you have
> messages with timestamps interleaved?
>
> Could anyone concile that API with the intrinsec offset-based contract?
>



-- 




*Steve JangPRINCIPAL ENGINEER Mobile +1.206.384.2999  |
Support +1.800.340.9194
*

Kafka broker throwing ConfigException Invalid value configuration log.segment.bytes: Not a number of type INT

[Debraj Manna]

Cross-posting from stackoverflow


I have a single node kafka broker and single node zookeeper with the
server.properties like below

broker.id=0
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
log.retention.hours=168
log.retention.check.interval.ms=30
zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=6000
num.network.threads=20
log.roll.hours=24
log.retention.bytes=107374182400
listeners=PLAINTEXT://172.36.0.72:9092
advertised.listeners=PLAINTEXT://172.36.0.72:9092

The zookeeper.properties looks like below

dataDir=/tmp/zookeeper
clientPort=2181
maxClientCnxns=0

But whenever I am starting the broker I am getting exception like below

FATAL [2018-02-19 17:01:10,772] kafka.Kafka$:[Logging$class:fatal:113]
- [main] -
org.apache.kafka.common.config.ConfigException: Invalid value
10737418240 for configuration log.segment.bytes: Not a number of type
INT
at 
org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:670)
at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:418)
at 
org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:55)
at kafka.server.KafkaConfig.(KafkaConfig.scala:759)
at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:743)
at kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:740)
at 
kafka.server.KafkaServerStartable$.fromProps(KafkaServerStartable.scala:28)
at kafka.Kafka$.main(Kafka.scala:58)
at kafka.Kafka.main(Kafka.scala)

Kafka Server 0.10.1 running on Ubuntu 14.

Can someone let me know what is going wrong?

timestamp-oriented API

[Xavier Noria]

In the mental model I am building of how Kafka works (new to this), the
broker keeps offsets by consumer group, and individual consumers basically
depend on the offset of the consumer group they join. Also consumer groups
may opt to start from the beginning.

OK, in that mental model there is a linearization of messages per
partition. As the documentation says, there is a total order per partition,
and the order is based on the offset, unrelated to the timestamp.

But I see the Java library has timestamp-oriented methods like:


https://kafka.apache.org/0102/javadoc/org/apache/kafka/clients/consumer/Consumer.html#offsetsForTimes(java.util.Map)

How does that make sense given the model described above? How is that
implemented? Does the broker has builtin support for this? What happens if
due to race conditions or machines with clocks out of sync you have
messages with timestamps interleaved?

Could anyone concile that API with the intrinsec offset-based contract?

Re: Testing with MockConsumer

[Ted Yu]

For #2, I think the assumption is that the records are processed by the
loop:

https://github.com/apache/kafka/blob/73be1e1168f91ee2a9d68e1d1c75c14018cf7d3a/clients/src/main/java/org/apache/kafka/clients/consumer/MockConsumer.java#L164



On Mon, Feb 19, 2018 at 4:39 AM, Gabriel Giussi 
wrote:

> Hi,
>
> I'm trying to use MockConsumer to test my application code but I've faced a
> couple of limitations and I want to know if there are workarounds or
> something that I'm overlooking.
> Note: I'm using kafka-clients v 0.11.0.2
>
>
>1. Why the addRecord
> 4018cf7d3a/clients/src/main/java/org/apache/kafka/clients/
> consumer/MockConsumer.java#L179>
>requires that the consumer has assigned partitions? Given that this is
> just
>simulating records being produced or existing records.
>2. Why the poll
> 4018cf7d3a/clients/src/main/java/org/apache/kafka/clients/
> consumer/MockConsumer.java#L132>
>clear the map of records? It should not be cleared after commit?
>3. Why the commitAsync
> 4018cf7d3a/clients/src/main/java/org/apache/kafka/clients/
> consumer/MockConsumer.java#L198>
>doesn't check for an exception and always succeed?
>
> Due to items (2) and (3) I'm not be able to test scenarios where the
> commits fails and the consumer should poll again the same elements.
>
> If someone knows about other scenarios that can't be tested with
> MockConsumer, please let me know.
>
> Thanks.
>

RE: Kafka control messages

[adrien ruffie]

Hi Ben,


it's depend on your consumer group configuration.

If you have all consumer arein differents group (only one consumer for each 
consumer group), you can use 2) because all the consumer instances have 
different consumer group,

then the control records will be broadcasted to all your consumer processes.


But if you have several consumer in each consumer group, you must implement the 
1) solution, because records will effectively be load balanced over the 
consumer processes.

So if only one consumer by group will read it...


best regards,


Adrien


De : Young, Ben 
Envoyé : lundi 19 février 2018 13:53:05
À : users@kafka.apache.org
Objet : Kafka control messages

Hi,

We have a situation where we are streaming data in a Kafka topic to be 
processed. The topic has multiple partitions. From time to time we need to add 
control messages to the stream and this needs to be coherent with the data 
steam (e.g. "replay from point X to this new location"). Because of the way 
Kafka partitions it seems hard to get these control messages to every consumer, 
no matter how they have been assigned, in the right place in the queues. The 
way I see it there are two options:

1) Have the producer put these control messages on every partition. We could 
get the number of partitions and manually post the same message on every 
partition, which will guarantee it gets read in the right place.
2) Reserve a special partition (e.g. 0) to be a control one, and have all 
consumers subscribe to that one. I don't think this gives the guarantees we 
want though.

The other option is to use a separate queue for these, but then we'd need some 
special logic to ensure they have their effect at the right point in the queue.

Has anyone else implemented anything like this? Am I barking up the wrong tree?

Thanks,
Ben

Ben Young
Principal Software Engineer - Risk
Risk & Compliance
T:  +44 (0) 1865 578140
E: ben.yo...@fisglobal.com
FIS | Empowering the Financial World
FIS Systems Limited * Registered in England and Wales No. 1937159 * Registered 
Office: Level 39, 25 Canada Square, London E14 5LQ

The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.

Kafka control messages

[Young, Ben]

Hi,

We have a situation where we are streaming data in a Kafka topic to be 
processed. The topic has multiple partitions. From time to time we need to add 
control messages to the stream and this needs to be coherent with the data 
steam (e.g. "replay from point X to this new location"). Because of the way 
Kafka partitions it seems hard to get these control messages to every consumer, 
no matter how they have been assigned, in the right place in the queues. The 
way I see it there are two options:

1) Have the producer put these control messages on every partition. We could 
get the number of partitions and manually post the same message on every 
partition, which will guarantee it gets read in the right place.
2) Reserve a special partition (e.g. 0) to be a control one, and have all 
consumers subscribe to that one. I don't think this gives the guarantees we 
want though.

The other option is to use a separate queue for these, but then we'd need some 
special logic to ensure they have their effect at the right point in the queue.

Has anyone else implemented anything like this? Am I barking up the wrong tree?

Thanks,
Ben

Ben Young
Principal Software Engineer - Risk
Risk & Compliance
T:  +44 (0) 1865 578140
E: ben.yo...@fisglobal.com
FIS | Empowering the Financial World
FIS Systems Limited * Registered in England and Wales No. 1937159 * Registered 
Office: Level 39, 25 Canada Square, London E14 5LQ

The information contained in this message is proprietary and/or confidential. 
If you are not the intended recipient, please: (i) delete the message and all 
copies; (ii) do not disclose, distribute or use the message in any manner; and 
(iii) notify the sender immediately. In addition, please be aware that any 
message addressed to our domain is subject to archiving and review by persons 
other than the intended recipient. Thank you.

Testing with MockConsumer

[Gabriel Giussi]

Hi,

I'm trying to use MockConsumer to test my application code but I've faced a
couple of limitations and I want to know if there are workarounds or
something that I'm overlooking.
Note: I'm using kafka-clients v 0.11.0.2


   1. Why the addRecord
   

   requires that the consumer has assigned partitions? Given that this is just
   simulating records being produced or existing records.
   2. Why the poll
   

   clear the map of records? It should not be cleared after commit?
   3. Why the commitAsync
   

   doesn't check for an exception and always succeed?

Due to items (2) and (3) I'm not be able to test scenarios where the
commits fails and the consumer should poll again the same elements.

If someone knows about other scenarios that can't be tested with
MockConsumer, please let me know.

Thanks.

Re: Zookeeper Error

[M. Manna]

Just a heads up. Windows doesn’t cleanup logs. There’s a pull req pending
for issue #1194.

Regards,

On Mon, 19 Feb 2018 at 09:14, Maria Pilar  wrote:

> Now It´s working properly, I have changed the server.id in to the
> zookeeper. properties and I have created topics into mutinode.
>
> I´m using windows because it´s a simple proof of concept.
>
> Thanks
>
> On 18 February 2018 at 03:15, Ted Yu  wrote:
>
> > What are the entries in /etc/hosts w.r.t. localhost ?
> >
> > I wonder if the exception had something to do with ipv6.
> >
> > On Sat, Feb 17, 2018 at 5:49 PM, Maria Pilar 
> wrote:
> >
> > > When i try to create a topic in that multicluster,
> > >
> > > kafka-topics.bat --create --topic my-kafka-topic --zookeeper
> > locahost:2181
> > > --replication-factor 2 --partitions 3
> > >
> > >
> > > i have received the same error
> > >
> > > Exception in thread "main" org.I0Itec.zkclient.exception.ZkException:
> > > Unable to connect to locahost:2181
> > > at org.I0Itec.zkclient.ZkConnection.connect(
> > ZkConnection.java:72)
> > > at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1228)
> > > at org.I0Itec.zkclient.ZkClient.(ZkClient.java:157)
> > > at org.I0Itec.zkclient.ZkClient.(ZkClient.java:131)
> > > at
> > > kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:115)
> > > at kafka.utils.ZkUtils$.apply(ZkUtils.scala:97)
> > > at kafka.admin.TopicCommand$.main(TopicCommand.scala:56)
> > > at kafka.admin.TopicCommand.main(TopicCommand.scala)
> > > Caused by: java.net.UnknownHostException: locahost
> > > at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
> > > at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.
> > java:928)
> > > at
> > > java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
> > > at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
> > > at java.net.InetAddress.getAllByName(InetAddress.java:1192)
> > > at java.net.InetAddress.getAllByName(InetAddress.java:1126)
> > > at
> > > org.apache.zookeeper.client.StaticHostProvider.(
> > > StaticHostProvider.java:61)
> > > at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445)
> > > at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380)
> > > at org.I0Itec.zkclient.ZkConnection.connect(
> > ZkConnection.java:70)
> > > ... 7 more
> > >
> > > Thanks
> > >
> > > On 18 February 2018 at 02:45, Maria Pilar  wrote:
> > >
> > > > Hi
> > > >
> > > > I´m trying to configure a multinode cluster in kafka. I have
> configured
> > > > each server.properties according with the new properties for each
> > server.
> > > >
> > > > When i start each server, the zookeeper console shows that error.
> > > >
> > > >  INFO Got user-level KeeperException when processing
> > > > sessionid:0x161a690f731 type:create cxid:0xd8 zxid:0x11f
> txntype:-1
> > > > reqpath:n/a Error Path:/brokers/ids Error:KeeperErrorCode =
> NodeExists
> > > for
> > > > /brokers/ids (org.apache.zookeeper.server.PrepRequestProcessor)
> > > >
> > > > I have configured each broker.id
> > > >
> > > >
> > > > Anybody knows what is the error?
> > > >
> > > > Thanks
> > > >
> > >
> >
>

Aggregation events Stream Kafka

[Maria Pilar]

Hi

I need to create aggretions events and publish them in other topic for a
Stream Kafka API.

I usually i have done aggregates events with Apache Spark, however it
requires include a new bussines layer into our E2e solution.

I have checked the possibility to use aggreate method with KTABLE.

Do you know any best solution for that?

Thanks

Re: Zookeeper Error

[Maria Pilar]

Yes, I made a spelling mistake. I have changed it.

Thanks

On 18 February 2018 at 11:49, Gerd König 
wrote:

> Hi,
>
> in your message there is "locahost", but I am pretty sure you wanted to use
> "localhost", including the "l", right?
> This one will usually be resolvable, hence doesn't throw an
> unknownHostException
>
> Regards
>
> Maria Pilar  schrieb am So., 18. Feb. 2018, 02:49:
>
> > When i try to create a topic in that multicluster,
> >
> > kafka-topics.bat --create --topic my-kafka-topic --zookeeper
> locahost:2181
> > --replication-factor 2 --partitions 3
> >
> >
> > i have received the same error
> >
> > Exception in thread "main" org.I0Itec.zkclient.exception.ZkException:
> > Unable to connect to locahost:2181
> > at org.I0Itec.zkclient.ZkConnection.connect(
> ZkConnection.java:72)
> > at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1228)
> > at org.I0Itec.zkclient.ZkClient.(ZkClient.java:157)
> > at org.I0Itec.zkclient.ZkClient.(ZkClient.java:131)
> > at
> > kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:115)
> > at kafka.utils.ZkUtils$.apply(ZkUtils.scala:97)
> > at kafka.admin.TopicCommand$.main(TopicCommand.scala:56)
> > at kafka.admin.TopicCommand.main(TopicCommand.scala)
> > Caused by: java.net.UnknownHostException: locahost
> > at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
> > at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.
> java:928)
> > at
> > java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
> > at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
> > at java.net.InetAddress.getAllByName(InetAddress.java:1192)
> > at java.net.InetAddress.getAllByName(InetAddress.java:1126)
> > at
> >
> > org.apache.zookeeper.client.StaticHostProvider.(
> StaticHostProvider.java:61)
> > at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445)
> > at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380)
> > at org.I0Itec.zkclient.ZkConnection.connect(
> ZkConnection.java:70)
> > ... 7 more
> >
> > Thanks
> >
> > On 18 February 2018 at 02:45, Maria Pilar  wrote:
> >
> > > Hi
> > >
> > > I´m trying to configure a multinode cluster in kafka. I have configured
> > > each server.properties according with the new properties for each
> server.
> > >
> > > When i start each server, the zookeeper console shows that error.
> > >
> > >  INFO Got user-level KeeperException when processing
> > > sessionid:0x161a690f731 type:create cxid:0xd8 zxid:0x11f txntype:-1
> > > reqpath:n/a Error Path:/brokers/ids Error:KeeperErrorCode = NodeExists
> > for
> > > /brokers/ids (org.apache.zookeeper.server.PrepRequestProcessor)
> > >
> > > I have configured each broker.id
> > >
> > >
> > > Anybody knows what is the error?
> > >
> > > Thanks
> > >
> >
>

Re: Zookeeper Error

[Maria Pilar]

Now It´s working properly, I have changed the server.id in to the
zookeeper. properties and I have created topics into mutinode.

I´m using windows because it´s a simple proof of concept.

Thanks

On 18 February 2018 at 03:15, Ted Yu  wrote:

> What are the entries in /etc/hosts w.r.t. localhost ?
>
> I wonder if the exception had something to do with ipv6.
>
> On Sat, Feb 17, 2018 at 5:49 PM, Maria Pilar  wrote:
>
> > When i try to create a topic in that multicluster,
> >
> > kafka-topics.bat --create --topic my-kafka-topic --zookeeper
> locahost:2181
> > --replication-factor 2 --partitions 3
> >
> >
> > i have received the same error
> >
> > Exception in thread "main" org.I0Itec.zkclient.exception.ZkException:
> > Unable to connect to locahost:2181
> > at org.I0Itec.zkclient.ZkConnection.connect(
> ZkConnection.java:72)
> > at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1228)
> > at org.I0Itec.zkclient.ZkClient.(ZkClient.java:157)
> > at org.I0Itec.zkclient.ZkClient.(ZkClient.java:131)
> > at
> > kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:115)
> > at kafka.utils.ZkUtils$.apply(ZkUtils.scala:97)
> > at kafka.admin.TopicCommand$.main(TopicCommand.scala:56)
> > at kafka.admin.TopicCommand.main(TopicCommand.scala)
> > Caused by: java.net.UnknownHostException: locahost
> > at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
> > at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.
> java:928)
> > at
> > java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
> > at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
> > at java.net.InetAddress.getAllByName(InetAddress.java:1192)
> > at java.net.InetAddress.getAllByName(InetAddress.java:1126)
> > at
> > org.apache.zookeeper.client.StaticHostProvider.(
> > StaticHostProvider.java:61)
> > at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445)
> > at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380)
> > at org.I0Itec.zkclient.ZkConnection.connect(
> ZkConnection.java:70)
> > ... 7 more
> >
> > Thanks
> >
> > On 18 February 2018 at 02:45, Maria Pilar  wrote:
> >
> > > Hi
> > >
> > > I´m trying to configure a multinode cluster in kafka. I have configured
> > > each server.properties according with the new properties for each
> server.
> > >
> > > When i start each server, the zookeeper console shows that error.
> > >
> > >  INFO Got user-level KeeperException when processing
> > > sessionid:0x161a690f731 type:create cxid:0xd8 zxid:0x11f txntype:-1
> > > reqpath:n/a Error Path:/brokers/ids Error:KeeperErrorCode = NodeExists
> > for
> > > /brokers/ids (org.apache.zookeeper.server.PrepRequestProcessor)
> > >
> > > I have configured each broker.id
> > >
> > >
> > > Anybody knows what is the error?
> > >
> > > Thanks
> > >
> >
>