Re: FW: mirror and schema topics

[Ewen Cheslack-Postava]

Can you give more details about how you're setting up your mirror? It
sounds like you're simply missing the __schemas topic, but it's hard to
determine the problem without more details about your mirroring setup.

-Ewen

On Wed, Nov 30, 2016 at 12:03 PM, Berryman, Eric 
wrote:

>
> Hello!
>
> I'm trying to mirror a kafka cluster, then run connect on the mirror.
> It seems the schemas are not getting moved in the mirror though, so I get
> the following error.
> Is this a configuration problem?
>
> Thank you for the help!
>
>
> Mirror>curl -X GET http://localhost:8081/subjects
> []
>
> Machine1>curl -X GET http://localhost:8081/subjects
> ["cts_olog_logbooks-value","cts_olog_logs_logbooks-value","
> cts_olog_entries-value","cts_olog_bitemporal_log-value","cts
> _olog_logs-value"]
>
> Mirror:
> [2016-11-30 13:12:07,612] ERROR Task cts-olog-bi-jdbc-sink-0 threw an
> uncaught and unrecoverable exception (org.apache.kafka.connect.runt
> ime.WorkerTask:142)
> org.apache.kafka.connect.errors.DataException: Failed to deserialize data
> to Avro:
> at io.confluent.connect.avro.AvroConverter.toConnectData(AvroCo
> nverter.java:109)
> at org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessa
> ges(WorkerSinkTask.java:358)
> at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerS
> inkTask.java:239)
> at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(
> WorkerSinkTask.java:172)
> at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(
> WorkerSinkTask.java:143)
> at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask
> .java:140)
> at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.
> java:175)
> at java.util.concurrent.Executors$RunnableAdapter.call(
> Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> Executor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> lExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.kafka.common.errors.SerializationException: Error
> retrieving Avro schema for id 43
> Caused by: 
> io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException:
> Schema not found; error code: 40403
> at io.confluent.kafka.schemaregistry.client.rest.RestService.
> sendHttpRequest(RestService.java:170)
> at io.confluent.kafka.schemaregistry.client.rest.RestService.
> httpRequest(RestService.java:187)
> at io.confluent.kafka.schemaregistry.client.rest.RestService.
> getId(RestService.java:323)
> at io.confluent.kafka.schemaregistry.client.rest.RestService.
> getId(RestService.java:316)
> at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistr
> yClient.getSchemaByIdFromRegistry(CachedSchemaRegistryClient.java:63)
> at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistr
> yClient.getBySubjectAndID(CachedSchemaRegistryClient.java:118)
> at io.confluent.kafka.serializers.AbstractKafkaAvroDeserializer
> .deserialize(AbstractKafkaAvroDeserializer.java:121)
> at io.confluent.kafka.serializers.AbstractKafkaAvroDeserializer
> .deserializeWithSchemaAndVersion(AbstractKafkaAvroDeserializer.java:190)
> at io.confluent.connect.avro.AvroConverter$Deserializer.deseria
> lize(AvroConverter.java:130)
> at io.confluent.connect.avro.AvroConverter.toConnectData(AvroCo
> nverter.java:99)
> at org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessa
> ges(WorkerSinkTask.java:358)
> at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerS
> inkTask.java:239)
> at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(
> WorkerSinkTask.java:172)
> at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(
> WorkerSinkTask.java:143)
> at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask
> .java:140)
> at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.
> java:175)
> at java.util.concurrent.Executors$RunnableAdapter.call(
> Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> Executor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> lExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> [2016-11-30 13:12:07,637] ERROR Task is being killed and will not recover
> until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:143)
> [2016-11-30 13:12:07,637] INFO Stopping task (io.confluent.connect.jdbc.sin
> k.JdbcSinkTask:88)
>



-- 
Thanks,
Ewen

Re: Expected client producer/consumer CPU utilization when idle

[Ewen Cheslack-Postava]

If completely idle, producers shouldn't need to do anything beyond very
infrequent metadata updates (once ever few minutes). Consumers, however,
will have some ongoing work -- they will always issues fetch requests (to
get more data) and heartbeats (to indicate they are still alive). But these
shouldn't create a ton of overhead since each only needs a small amount of
work every few seconds if there's no data flowing.

Note that the setting you mention as not default (fetch.min.bytes) is, in
fact, the default value of 1.

If you're concerned about the CPU usage, I'd suggest running your processes
under a profiler for a bit to determine where the CPU time is going.

-Ewen



On Thu, Dec 1, 2016 at 6:20 AM, Niklas Ström  wrote:

> Hello all
>
> Can anyone say something about what CPU utilization we can expect for a
> producer/consumer process that is idle, i.e. not producing or consuming any
> messages? Should it be like 0%? What is your experience?
>
> We have a small program with a few kafka producers and consumers and we are
> concerned about the CPU utilization when it is idle. Currently it uses on
> average 6% CPU when idle and about 12% when under a low load of 4 messages
> per second. In a scaled down test program with only one producer and one
> consumer the corresponding figures are 2.6% and 4.8%.
>
> We want to have a lot of different small processes each producing and
> consuming a number of topics so we really want to minimize the CPU
> utilization of each process, they will not all do heavy work at the same
> time so occasional high loads for a process is not a problem, but if all
> processes are using the CPU when not really doing anything useful we might
> get into problem.
>
> Tests are run with one local kafka broker, on a machine with Intel i7 2
> Cores @ 3Ghz, 16 GB RAM, in a virtual environment with Ubuntu 14.04.4 LTS.
> Using java kafka client 0.10.0.0. Our only kafka configuration parameter
> that is not default is KAFKA_FETCH_MIN_BYTES that is 1 in order to reduce
> latency.
>
> So far we have not run so many processes at the same time, but we fear that
> if we try to run a couple of hundred of these processes we will get into
> problem.
>
> Would greatly appreciate any input. If not else, please tell me the CPU
> utilization of your producer/consumer processes when they are not really
> under load. Just so I can conclude if our program behaves as expected or if
> we have any configuration or environment issues
>
> Thanks
> Niklas Ström
>



-- 
Thanks,
Ewen

Re: ConnectStandalone with no starting connector properties

[Ewen Cheslack-Postava]

Micah,

Sure, we'd be happy to commit a patch that removes this restriction.

In practice, for most folks its a bit simpler to generate the config file
in whatever deployment system they are using than to start the standalone
server separately and make an HTTP request to trigger creation of the
connector (which needs to happen on ever restart of the process, even those
that might not be managed by your deployment system, e.g. due to unexpected
process failure that is recovered by something like your init system).
However, I don't see any reason we can't support the approach it sounds
like you're proposing.

-Ewen

On Fri, Dec 2, 2016 at 10:42 AM, Micah Whitacre 
wrote:

> I'm curious if there was an intentional reason that Kafka Connect
> standalone requires a connector properties on startup?[1]
>  ConnectDistributed only requires the worker properties.  ConnectStandalone
> however requires the worker properties and at least one connector
> properties.  Since connectors can still be managed for standalone using the
> REST API, I was wondering if that restriction could be lifted?  If so I'll
> log an enhancement JIRA but wanted to make sure I wasn't missing something
> obvious.
>
> Thanks,
> Micah
>
> [1] -
> https://github.com/apache/kafka/blob/93804d50ffc40fb6cdc61c073f4a62
> f0931f042d/connect/runtime/src/main/java/org/apache/kafka/connect/cli/
> ConnectStandalone.java#L60-L63
>



-- 
Thanks,
Ewen

Re: Kafka windowed table not aggregating correctly

[Matthias J. Sax]

To unsubscribe you need to sent an email to

  users-unsubscr...@kafka.apache.org


-Matthias

On 12/3/16 6:13 PM, williamtellme123 wrote:
> Unsubscribe
> 
> 
> Sent via the Samsung Galaxy S7, an AT 4G LTE smartphone
>  Original message From: Guozhang Wang  
> Date: 12/2/16  5:48 PM  (GMT-06:00) To: users@kafka.apache.org Subject: Re: 
> Kafka windowed table not aggregating correctly 
> Sachin,
> 
> One thing to note is that the retention of the windowed stores works by
> keeping multiple segments of the stores where each segments stores a time
> range which can potentially span multiple windows, if a new window needs to
> be created that is further from the oldest segment's time range + retention
> period (from your code it seems you do not override it from
> TimeWindows.of("stream-table",
> 10 * 1000L).advanceBy(5 * 1000L), via until(...)), so the default of one
> day is used.
> 
> So with WallclockTimeExtractor since it is using system time, it wont give
> you timestamps that span for more than a day during a short period of time,
> but if your own defined timestamps expand that value, then old segments
> will be dropped immediately and hence the aggregate values will be returned
> as a single value.
> 
> Guozhang
> 
> 
> On Fri, Dec 2, 2016 at 11:58 AM, Matthias J. Sax 
> wrote:
> 
>> The extractor is used in
>>
>> org.apache.kafka.streams.processor.internals.RecordQueue#addRawRecords()
>>
>> Let us know, if you could resolve the problem or need more help.
>>
>> -Matthias
>>
>> On 12/2/16 11:46 AM, Sachin Mittal wrote:
>>> https://github.com/SOHU-Co/kafka-node/ this is the node js client i am
>>> using. The version is 0.5x. Can you please tell me what code in streams
>>> calls the timestamp extractor. I can look there to see if there is any
>>> issue.
>>>
>>> Again issue happens only when producing the messages using producer that
>> is
>>> compatible with kafka version 0.8x. I see that this producer does not
>> send
>>> a record timestamp as this was introduced in version 0.10 only.
>>>
>>> Thanks
>>> Sachin
>>>
>>> On 3 Dec 2016 1:03 a.m., "Matthias J. Sax" 
>> wrote:
>>>
 I am not sure what is happening. That's why it would be good to have a
 toy example to reproduce the issue.

 What do you mean by "Kafka node version 0.5"?

 -Matthias

 On 12/2/16 11:30 AM, Sachin Mittal wrote:
> I can provide with the data but data does not seem to be the issue.
> If I submit the same data and use same timestamp extractor  using the
 java
> client with kafka version 0.10.0.1 aggregation works fine.
> I find the issue only when submitting the data with kafka node version
 0.5.
> It looks like the stream does not extract the time correctly in that
 case.
>
> Thanks
> Sachin
>
> On 2 Dec 2016 11:41 p.m., "Matthias J. Sax" 
 wrote:
>
>> Can you provide example input data (including timetamps) and result.
>> What is the expected result (ie, what aggregation do you apply)?
>>
>>
>> -Matthias
>>
>> On 12/2/16 7:43 AM, Sachin Mittal wrote:
>>> Hi,
>>> After much debugging I found an issue with timestamp extractor.
>>>
>>> If I use a custom timestamp extractor with following code:
>>>  public static class MessageTimestampExtractor implements
>>> TimestampExtractor {
>>>  public long extract(ConsumerRecord record) {
>>>  if (record.value() instanceof Message) {
>>>  return ((Message) record.value()).ts;
>>>  } else {
>>>  return record.timestamp();
>>>  }
>>>  }
>>>  }
>>>
>>> Here message has a long field ts which stores the timestamp, the
>>> aggregation does not work.
>>> Note I have checked and ts has valid timestamp values.
>>>
>>> However if I replace it with say WallclockTimestampExtractor
 aggregation
>> is
>>> working fine.
>>>
>>> I do not understand what could be the issue here.
>>>
>>> Also note I am using kafka streams version 0.10.0.1 and I am
>> publishing
>>> messages via
>>> https://github.com/SOHU-Co/kafka-node/ whose version is quite old
 0.5.x
>>>
>>> Let me know if there is some bug in time stamp extractions.
>>>
>>> Thanks
>>> Sachin
>>>
>>>
>>>
>>> On Mon, Nov 28, 2016 at 11:52 PM, Guozhang Wang 
>> wrote:
>>>
 Sachin,

 This is indeed a bit wired, and we'd like to try to re-produce your
>> issue
 locally. Do you have a sample input data for us to try out?

 Guozhang

 On Fri, Nov 25, 2016 at 10:12 PM, Sachin Mittal >>
 wrote:

> Hi,
> I fixed that sorted set issue but I am 

Re: Tracking when a batch of messages has arrived?

[Ali Akhtar]

I don't - it would require fetching all messages and iterating over them
just to count them, which is expensive. I know the counts after they have
been sent.

On Sun, Dec 4, 2016 at 9:34 PM, Marko Bonaći 
wrote:

> Do you know in advance (when sending the first message) how many messages
> that batch is going to have?
>
>
> Marko Bonaći
> Monitoring | Alerting | Anomaly Detection | Centralized Log Management
> Solr & Elasticsearch Support
> Sematext  | Contact
> 
>
> On Sat, Dec 3, 2016 at 1:01 AM, Ali Akhtar  wrote:
>
> > Hey Apurva,
> >
> > I am including the batch_id inside the messages.
> >
> > Could you give me an example of what you mean by custom control messages
> > with a control topic please?
> >
> >
> >
> > On Sat, Dec 3, 2016 at 12:35 AM, Apurva Mehta 
> wrote:
> >
> > > That should work, though it sounds like you may be interested in :
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 98+-+Exactly+Once+Delivery+and+Transactional+Messaging
> > >
> > > If you can include the 'batch_id' inside your messages, and define
> custom
> > > control messages with a control topic, then you would not need one
> topic
> > > per batch, and you would be very close to the essence of the above
> > > proposal.
> > >
> > > Thanks,
> > > Apurva
> > >
> > > On Fri, Dec 2, 2016 at 5:02 AM, Ali Akhtar 
> wrote:
> > >
> > > > Heya,
> > > >
> > > > I need to send a group of messages, which are all related, and then
> > > process
> > > > those messages, only when all of them have arrived.
> > > >
> > > > Here is how I'm planning to do this. Is this the right way, and can
> any
> > > > improvements be made to this?
> > > >
> > > > 1) Send a message to a topic called batch_start, with a batch id
> (which
> > > > will be a UUID)
> > > >
> > > > 2) Post the messages to a topic called batch_msgs_. Here
> > > batch_id
> > > > will be the batch id sent in batch_start.
> > > >
> > > > The number of messages sent will be recorded by the producer.
> > > >
> > > > 3) Send a message to batch_end with the batch id and the number of
> sent
> > > > messages.
> > > >
> > > > 4) On the consumer side, using Kafka Streaming, I would listen to
> > > > batch_end.
> > > >
> > > > 5) When the message there arrives, I will start another instance of
> > Kafka
> > > > Streaming, which will process the messages in batch_msgs_
> > > >
> > > > 6) Perhaps to be extra safe, whenever batch_end arrives, I will
> start a
> > > > throwaway consumer which will just count the number of messages in
> > > > batch_msgs_. If these don't match the # of messages
> specified
> > > in
> > > > the batch_end message, then it will assume that the batch hasn't yet
> > > > finished arriving, and it will wait for some time before retrying.
> Once
> > > the
> > > > correct # of messages have arrived, THEN it will trigger step 5
> above.
> > > >
> > > > Will the above method work, or should I make any changes to it?
> > > >
> > > > Is step 6 necessary?
> > > >
> > >
> >
>

Re: Tracking when a batch of messages has arrived?

[Marko Bonaći]

Do you know in advance (when sending the first message) how many messages
that batch is going to have?


Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext  | Contact


On Sat, Dec 3, 2016 at 1:01 AM, Ali Akhtar  wrote:

> Hey Apurva,
>
> I am including the batch_id inside the messages.
>
> Could you give me an example of what you mean by custom control messages
> with a control topic please?
>
>
>
> On Sat, Dec 3, 2016 at 12:35 AM, Apurva Mehta  wrote:
>
> > That should work, though it sounds like you may be interested in :
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 98+-+Exactly+Once+Delivery+and+Transactional+Messaging
> >
> > If you can include the 'batch_id' inside your messages, and define custom
> > control messages with a control topic, then you would not need one topic
> > per batch, and you would be very close to the essence of the above
> > proposal.
> >
> > Thanks,
> > Apurva
> >
> > On Fri, Dec 2, 2016 at 5:02 AM, Ali Akhtar  wrote:
> >
> > > Heya,
> > >
> > > I need to send a group of messages, which are all related, and then
> > process
> > > those messages, only when all of them have arrived.
> > >
> > > Here is how I'm planning to do this. Is this the right way, and can any
> > > improvements be made to this?
> > >
> > > 1) Send a message to a topic called batch_start, with a batch id (which
> > > will be a UUID)
> > >
> > > 2) Post the messages to a topic called batch_msgs_. Here
> > batch_id
> > > will be the batch id sent in batch_start.
> > >
> > > The number of messages sent will be recorded by the producer.
> > >
> > > 3) Send a message to batch_end with the batch id and the number of sent
> > > messages.
> > >
> > > 4) On the consumer side, using Kafka Streaming, I would listen to
> > > batch_end.
> > >
> > > 5) When the message there arrives, I will start another instance of
> Kafka
> > > Streaming, which will process the messages in batch_msgs_
> > >
> > > 6) Perhaps to be extra safe, whenever batch_end arrives, I will start a
> > > throwaway consumer which will just count the number of messages in
> > > batch_msgs_. If these don't match the # of messages specified
> > in
> > > the batch_end message, then it will assume that the batch hasn't yet
> > > finished arriving, and it will wait for some time before retrying. Once
> > the
> > > correct # of messages have arrived, THEN it will trigger step 5 above.
> > >
> > > Will the above method work, or should I make any changes to it?
> > >
> > > Is step 6 necessary?
> > >
> >
>

Re: Messages intermittently get lost

[Sudev A C]

Hi Zak,

Why don't you try using zookeeper four letter admin tools

?

echo stat | nc zookeeper-ip zookeeper-port
echo stat | nc localhost 2181

These commands will tell you the current status of zookeeper client.

https://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#The+Four+Letter+Words

Thanks
Sudev

On Thu, 1 Dec 2016 at 5:11 AM, Flavio Junqueira  wrote:

Hey Martin,

Do you want to create a jira for this and report the issue?

-Flavio

> On 30 Nov 2016, at 18:33, Martin Gainty  wrote:
>
> a shock when a zk script goes fubar
>
>
> zookeeper devs can we get some help for sh zkServer.sh status?
>
>
> Thanks!
>
> Martin
> __
>
>
>
> 
> From: Zac Harvey 
> Sent: Wednesday, November 30, 2016 10:25 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Hi Martin, makes sense.
>
>
> When I SSH into all 3 of my ZK nodes and run:
>
>
> sh zkServer.sh status
>
>
> All three of them give me the following output:
>
>
> JMX enabled by default
>
> zkServer.sh: 81: /opt/zookeeper/bin/zkEnv.sh: Syntax error: "("
unexpected (expecting "fi")
>
>
> Looks like a bug in the ZK shell script?
>
>
> Best,
>
> Zac
>
> 
> From: Martin Gainty 
> Sent: Tuesday, November 29, 2016 11:18:21 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Hi Zach
>
>
> we dont know whats causing this intermittent problem..so lets Divide and
Conquer each part of this problem individually starting at the source of
the data feeds
>
>
> Let us eliminate any potential problem with feeds from external sources
>
>
> Once you verify the zookeeper feeds are 100% reliable lets move onto Kafka
>
>
> Pingback when you have verifiable results from zookeeper feeds
>
>
> Thanks
>
> Martin
> __
>
>
>
> 
> From: Zac Harvey 
> Sent: Tuesday, November 29, 2016 10:46 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Does anybody have any idea why ZK might be to blame if messages sent by a
Kafka producer fail to be received by a Kafka consumer?
>
> 
> From: Zac Harvey 
> Sent: Monday, November 28, 2016 9:07:41 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Thanks Martin, I will look at those links.
>
>
> But you seem to be 100% confident that the problem is with
ZooKeeper...can I ask why? What is it about my problem description that
makes you think this is an issue with ZooKeeper?
>
> 
> From: Martin Gainty 
> Sent: Friday, November 25, 2016 1:46:28 PM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
>
>
> 
> From: Zac Harvey 
> Sent: Friday, November 25, 2016 6:17 AM
> To: users@kafka.apache.org
> Subject: Re: Messages intermittently get lost
>
> Hi Martin,
>
>
> My server.properties looks like this:
>
>
> listeners=PLAINTEXT://0.0.0.0:9092
>
> advertised.host.name=
>
> broker.id=2
>
> port=9092
>
> num.partitions=4
>
> zookeeper.connect=zkA:2181,zkB:2181,zkC:2181
>
> MG>can you check status for each ZK Node in the quorum?
>
> sh>$ZOOKEEPER_HOME/bin/zkServer.sh status
>
>
http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html
> ZooKeeper problems and solutions - IBM<
http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html
>
> www.ibm.com
> Use these solutions to resolve the problems that you might encounter with
Apache ZooKeeper.
>
>
>
> ZooKeeper problems and solutions - IBM<
http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html
>
> ZooKeeper problems and solutions - IBM<
http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html
>
> www.ibm.com
> Use these solutions to resolve the problems that you might encounter with
Apache ZooKeeper.
>
>
>
> www.ibm.com
> [
http://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/200px-IBM_logo.svg.png
]
>
> IBM - United States
> www.ibm.com
> For more than a century IBM has been dedicated to every client's success
and to creating innovations that matter for the world
>
>
>
> Use these solutions to resolve the problems that you might encounter with
Apache ZooKeeper.
>
>
>
>
> ZooKeeper problems and solutions - IBM<
http://www.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/com.ibm.streams.pd.doc/doc/containerstreamszookeeper.html
>
>