reseting dirty offset

2019-03-08 Thread adrien ruffie 
Hello all,

This morning I got the following error:

[2019-03-08 09:43:03,066] WARN Resetting first dirty offset to log start offset 
1 since the checkpointed offset 0 is invalid. 
(kafka.log.LogCleanerManager$)

If I correctly understand, the log cleaner try to return to a old offset which 
not present yet in the segment right ?

What I can solve the problem please ? It's very annoying problem for me because 
it's a production logs ...

Thank and best regards
Adrien

Re: KTable.suppress(Suppressed.untilWindowCloses) does not suppress some non-final results when the kafka streams process is restarted

2019-03-08 Thread Jonathan Santilli 
Sure John, I will document it.


Thanks a lot for your reply.
--
Jonathan



On Tue, Mar 5, 2019 at 7:38 PM John Roesler  wrote:

> Hi Jonathan,
>
> Just a quick update: I have not been able to reproduce the duplicates issue
> with the 2.2 RC, even with a topology very similar to the one you included
> in your stackoverflow post.
>
> I think we should treat this as a new bug. Would you mind opening a new
> Jira bug ticket with some steps to reproduce the problem, and also exactly
> the behavior you observe?
>
> Thanks,
> -John
>
> On Mon, Mar 4, 2019 at 10:41 PM John Roesler  wrote:
>
> > Hi Jonathan,
> >
> > Sorry to hear that the feature is causing you trouble as well, and that
> > the 2.2 release candidate didn't seem to fix it.
> >
> > I'll try and do a repro based on the code in your SO post tomorrow.
> >
> > I don't think it's related to the duplicates, but that shutdown error is
> > puzzling. Can you print the topology (with topology.describe() ) ? This
> > will tell us what is in task 1 (i.e., *1_*) of your program.
> >
> > Thanks,
> > -John
> >
> > On Fri, Mar 1, 2019 at 11:33 AM Jonathan Santilli <
> > jonathansanti...@gmail.com> wrote:
> >
> >> BTW, after stopping the app gracefully (Stream#close()), this error
> shows
> >> up repeatedly:
> >>
> >> 2019-03-01 17:18:07,819 WARN
> >> [XXX-116ba7c8-678e-47f7-9074-7d03627b1e1a-StreamThread-1]
> >> internals.ProcessorStateManager (ProcessorStateManager.java:327) - task
> >> [0_0] Failed to write offset checkpoint file to
> >> [/tmp/kafka-stream/XXX/0_0/.checkpoint]
> >>
> >> java.io.FileNotFoundException: /tmp/kafka-stream/XXX/0_0/.checkpoint.tmp
> >> (No such file or directory)
> >>
> >> at java.io.FileOutputStream.open0(Native Method) ~[?:1.8.0_191]
> >>
> >> at java.io.FileOutputStream.open(FileOutputStream.java:270)
> ~[?:1.8.0_191]
> >>
> >> at java.io.FileOutputStream.(FileOutputStream.java:213)
> >> ~[?:1.8.0_191]
> >>
> >> at java.io.FileOutputStream.(FileOutputStream.java:162)
> >> ~[?:1.8.0_191]
> >>
> >> at org.apache.kafka.streams.state.internals.OffsetCheckpoint.write(
> >> OffsetCheckpoint.java:79) ~[kafka-streams-2.2.0.jar:?]
> >>
> >> at
> >>
> >>
> org.apache.kafka.streams.processor.internals.ProcessorStateManager.checkpoint(
> >> ProcessorStateManager.java:325) [kafka-streams-2.2.0.jar:?]
> >>
> >> at org.apache.kafka.streams.processor.internals.StreamTask.suspend(
> >> StreamTask.java:599) [kafka-streams-2.2.0.jar:?]
> >>
> >> at org.apache.kafka.streams.processor.internals.StreamTask.close(
> >> StreamTask.java:721) [kafka-streams-2.2.0.jar:?]
> >>
> >> at org.apache.kafka.streams.processor.internals.AssignedTasks.close(
> >> AssignedTasks.java:337) [kafka-streams-2.2.0.jar:?]
> >>
> >> at org.apache.kafka.streams.processor.internals.TaskManager.shutdown(
> >> TaskManager.java:267) [kafka-streams-2.2.0.jar:?]
> >>
> >> at
> >>
> >>
> org.apache.kafka.streams.processor.internals.StreamThread.completeShutdown(
> >> StreamThread.java:1209) [kafka-streams-2.2.0.jar:?]
> >>
> >> at org.apache.kafka.streams.processor.internals.StreamThread.run(
> >> StreamThread.java:786) [kafka-streams-2.2.0.jar:?]
> >>
> >>
> >> However, I have checked and the folder created starts with: *1_*
> >>
> >> ls -lha /tmp/kafka-stream/XXX/1_1
> >> total 8
> >> drwxr-xr-x   5 a  b   160B  1 Mar 17:18 .
> >> drwxr-xr-x  34 a  b   1.1K  1 Mar 17:15 ..
> >> -rw-r--r--   1 a  b   2.9K  1 Mar 17:18 .checkpoint
> >> -rw-r--r--   1 a  b 0B  1 Mar 16:05 .lock
> >> drwxr-xr-x   3 a  b96B  1 Mar 16:43
> >> KSTREAM-REDUCE-STATE-STORE-05
> >>
> >>
> >>
> >> Cheers!
> >> --
> >> Jonathan
> >>
> >>
> >>
> >> On Fri, Mar 1, 2019 at 5:11 PM Jonathan Santilli <
> >> jonathansanti...@gmail.com>
> >> wrote:
> >>
> >> > Hello John, hope you are well.
> >> > I have tested the version 2.2 release candidate (although I know it
> has
> >> > been postponed).
> >> > I have been following this email thread because I think am
> experiencing
> >> > the same issue. I have reported in an email to this list and also all
> >> the
> >> > details are in OS (
> >> >
> >>
> https://stackoverflow.com/questions/54145281/why-do-the-offsets-of-the-consumer-group-app-id-of-my-kafka-streams-applicatio
> >> > ).
> >> >
> >> > After the test, the result is the same as before (at least for my
> case),
> >> > already processed records are passed again to the output topic causing
> >> the
> >> > data duplication:
> >> >
> >> > ...
> >> > 2019-03-01 16:55:23,808 INFO
> >> [XXX-116ba7c8-678e-47f7-9074-7d03627b1e1a-StreamThread-1]
> >> > internals.StoreChangelogReader (StoreChangelogReader.java:221) -
> >> > stream-thread
> [XXX-116ba7c8-678e-47f7-9074-7d03627b1e1a-StreamThread-1]
> >> No
> >> > checkpoint found for task 1_10 state store
> >> > KTABLE-SUPPRESS-STATE-STORE-11 changelog
> >> > XXX-KTABLE-SUPPRESS-STATE-STORE-11-changelog-10 with EOS
> turned
> >> on. *Reinitializing
> >> > the task and restore its state from the beginning.*
> >> >
> >> > ...
> >> >
> >> >
> >> >

question about kafka topic

2019-03-08 Thread Calvin Chen 
Hi,
I have a question about kafka topic, recently we face kafka client sending 
message to kafka topic issue, got error about offset, and client can not send 
message to kafka cluster.

My question is, since we configure kafka servers to one cluster, when cluster 
get message, will it try it best to deliver message to good host's partition? 
say, if cluster forward message to host A's topic partition, but it failed, 
then will cluster find another host's partition and redeliver the message? or 
just simply return error message to client?

Is it configurable? what is the parameter to configure it?

Thanks
-Calvin

Re: [VOTE] 2.2.0 RC1 [CANCELED]

2019-03-08 Thread Matthias J. Sax 
A blocker was found: https://issues.apache.org/jira/browse/KAFKA-8069

I am canceling this VOTE in favor of a new RC.


-Matthias

On 3/7/19 10:13 AM, Jonathan Santilli wrote:
> Hello Matthias, I have run the tests without issue.
> Also, I have performed the quick start successfully, thanks for the release.
> 
> 
> Cheers!
> --
> Jonathan
> 
> 
> 
> On Wed, Mar 6, 2019 at 8:58 PM Matthias J. Sax 
> wrote:
> 
>> Thanks for testing the RC Jakub!
>>
>> I agree, that both issue are no reason to cancel the current RC. I'll
>> make sure that the doc fixes go into 2.2 webpage when we do the release
>> though.
>>
>> Thanks for the patches!
>>
>>
>> -Matthias
>>
>>
>> On 3/6/19 11:58 AM, Jakub Scholz wrote:
>>> Thanks for driving the release Matthias. I noticed two very minor issues
>>> while testing the RC1 and opened PRs against master:
>>> * https://github.com/apache/kafka/pull/6379
>>> * https://github.com/apache/kafka/pull/6381
>>>
>>> I assume it might not be worth including these into 2.2 since it is
>> nothing
>>> serious. Otherwise the RC1 worked fine for me.
>>>
>>> Jakub
>>>
>>>
>>> On Fri, Mar 1, 2019 at 8:48 PM Matthias J. Sax 
>>> wrote:
>>>
 Hello Kafka users, developers and client-developers,

 This is the second candidate for release of Apache Kafka 2.2.0.

  - Added SSL support for custom principle name
  - Allow SASL connections to periodically re-authenticate
  - Improved consumer group management
- default group.id is `null` instead of empty string
  - Add --under-min-isr option to describe topics command
  - API improvement
- Producer: introduce close(Duration)
- AdminClient: introduce close(Duration)
- Kafka Streams: new flatTransform() operator in Streams DSL
- KafkaStreams (and other classed) now implement AutoClosable to
 support try-with-resource
- New Serdes and default method implementations
  - Kafka Streams exposed internal client.id via ThreadMetadata
  - Metric improvements:  All `-min`, `-avg` and `-max` metrics will now
 output `NaN` as default value

 Release notes for the 2.2.0 release:
 https://home.apache.org/~mjsax/kafka-2.2.0-rc1/RELEASE_NOTES.html

 *** Please download, test and vote by Thursday, March 7, 9am PST

 Kafka's KEYS file containing PGP keys we use to sign the release:
 https://kafka.apache.org/KEYS

 * Release artifacts to be voted upon (source and binary):
 https://home.apache.org/~mjsax/kafka-2.2.0-rc1/

 * Maven artifacts to be voted upon:
 https://repository.apache.org/content/groups/staging/org/apache/kafka/

 * Javadoc:
 https://home.apache.org/~mjsax/kafka-2.2.0-rc1/javadoc/

 * Tag to be voted upon (off 2.2 branch) is the 2.2.0 tag:
 https://github.com/apache/kafka/releases/tag/2.2.0-rc1

 * Documentation:
 https://kafka.apache.org/22/documentation.html

 * Protocol:
 https://kafka.apache.org/22/protocol.html

 * Jenkins builds for the 2.2 branch:
 Unit/integration tests: https://builds.apache.org/job/kafka-2.2-jdk8/

 * System tests for the 2.2 branch:
 https://jenkins.confluent.io/job/system-test-kafka/job/2.2/


 /**

 Thanks,


 -Matthias


>>>
>>
>>
> 



signature.asc
Description: OpenPGP digital signature

Re: Kafka 2.1.0 JMX mbean broken ?

2019-03-08 Thread Raghav 
Hello Allen, Ted, Mani, Gwen, James

https://www.mail-archive.com/dev@kafka.apache.org/msg86226.html

I read this email thread and I see that you guys were split in going ahead
with this change. This has broken our dashboard in 2.1 Kafka, which is ok
as long as we know how to fix it.

Can you please help us figure out the answer for the email below ? It will
be greatly appreciated. We just want to know how to find the version number
?

Many thanks.

R

On Fri, Dec 14, 2018 at 5:16 PM Raghav  wrote:

> I got it to work. I fired up a console and then saw what beans are
> registered there and then queries using the code. It works then.
>
> But the apiVersion is different and how do we know what apiVersion a
> producer is using ? Our dashboards cannot have a-priori knowledge of the
> version numbers, and wildcards don't work. See the screenshot below,
> apiVersion is 7. Where did this come from ? Can someone please help to
> understand.
>
> [image: jmx.png]
>
>
>
> On Fri, Dec 14, 2018 at 4:29 PM Raghav  wrote:
>
>> Is this a test case for this commit:
>> https://github.com/apache/kafka/pull/4506 ? I have tried using all
>> possible cases but cannot get it to work.
>>
>> I cannot get it to work using mbean reader via JMX. Can any one please
>> help? Or atleast confirm that it is broken. Thanks
>>
>> R
>>
>> On Fri, Dec 14, 2018 at 6:34 AM Raghav  wrote:
>>
>>> Thanks Ismael. How to query it in 2.1 ? I tried all possible ways
>>> including using version, but I am still getting the same exception message.
>>>
>>> Thanks for your help.
>>>
>>> On Thu, Dec 13, 2018 at 7:19 PM Ismael Juma  wrote:
>>>
 The metric was changed to include the API version. I believe this was in
 the upgrade notes for 2.0.0.

 Ismael

 On Thu, Dec 13, 2018, 3:35 PM Raghav >>>
 > Hi
 >
 > We are trying to move from Kafka 1.1.0 to Kafka 2.1.0. We used to
 monitor
 > our 3 node Kafka using JMX. Upon moving to 2.1.0, we have observed
 that the
 > *below* mentioned metric can't be retrie
 > and we get the below exception:
 >
 >
 *"kafka.network:type=RequestMetrics,name=RequestsPerSec,request=Produce"*
 >
 > javax.management.InstanceNotFoundException:
 > kafka.network:type=RequestMetrics,name=RequestsPerSec,request=Produce
 > at
 >
 >
 com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1095)
 > at
 >
 >
 com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBeanInfo(DefaultMBeanServerInterceptor.java:1375)
 > at
 >
 >
 com.sun.jmx.mbeanserver.JmxMBeanServer.getMBeanInfo(JmxMBeanServer.java:920)
 > at
 >
 >
 javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1462)
 > at
 >
 >
 javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
 > at
 >
 >
 javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309)
 > at
 >
 >
 javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401)
 > at
 >
 >
 javax.management.remote.rmi.RMIConnectionImpl.getMBeanInfo(RMIConnectionImpl.java:905)
 > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 > at
 >
 >
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 > at
 >
 >
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 > at java.lang.reflect.Method.invoke(Method.java:498)
 > at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:324)
 > at sun.rmi.transport.Transport$1.run(Transport.java:200)
 > at sun.rmi.transport.Transport$1.run(Transport.java:197)
 > at java.security.AccessController.doPrivileged(Native Method)
 > at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
 > at
 sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
 > at
 >
 >
 sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
 > at
 >
 >
 sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
 > at java.security.AccessController.doPrivileged(Native Method)
 > at
 >
 >
 sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
 > at
 >
 >
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 > at
 >
 >
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 > at java.lang.Thread.run(Thread.java:745)
 > at
 >
 >
 sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:276)
 > at
 >
 sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:253)
 > at sun.rmi.server.UnicastRef.invoke(Unica

Re: Kafka 2.1.0 JMX mbean broken ?

2019-03-08 Thread Manikumar 
Hi Raghav,

As you know, KIP-372 added "version" tag to RequestsPerSec metric to
monitor requests for each version.
As mentioned in the KIP, to get total count per request (across all
versions), we need to aggregate over all versions.
You may need to automate this by fetching all metrics using wildcard and
aggregating over all versions.
We may need to use JMX readers which support wildcard queries.

All possible protocol request versions are available here:
https://kafka.apache.org/protocol.html#The_Messages_Produce


On Sat, Mar 9, 2019 at 5:59 AM Raghav  wrote:

> Hello Allen, Ted, Mani, Gwen, James
>
> https://www.mail-archive.com/dev@kafka.apache.org/msg86226.html
>
> I read this email thread and I see that you guys were split in going ahead
> with this change. This has broken our dashboard in 2.1 Kafka, which is ok
> as long as we know how to fix it.
>
> Can you please help us figure out the answer for the email below ? It will
> be greatly appreciated. We just want to know how to find the version number
> ?
>
> Many thanks.
>
> R
>
> On Fri, Dec 14, 2018 at 5:16 PM Raghav  wrote:
>
>> I got it to work. I fired up a console and then saw what beans are
>> registered there and then queries using the code. It works then.
>>
>> But the apiVersion is different and how do we know what apiVersion a
>> producer is using ? Our dashboards cannot have a-priori knowledge of the
>> version numbers, and wildcards don't work. See the screenshot below,
>> apiVersion is 7. Where did this come from ? Can someone please help to
>> understand.
>>
>> [image: jmx.png]
>>
>>
>>
>> On Fri, Dec 14, 2018 at 4:29 PM Raghav  wrote:
>>
>>> Is this a test case for this commit:
>>> https://github.com/apache/kafka/pull/4506 ? I have tried using all
>>> possible cases but cannot get it to work.
>>>
>>> I cannot get it to work using mbean reader via JMX. Can any one please
>>> help? Or atleast confirm that it is broken. Thanks
>>>
>>> R
>>>
>>> On Fri, Dec 14, 2018 at 6:34 AM Raghav  wrote:
>>>
 Thanks Ismael. How to query it in 2.1 ? I tried all possible ways
 including using version, but I am still getting the same exception message.

 Thanks for your help.

 On Thu, Dec 13, 2018 at 7:19 PM Ismael Juma  wrote:

> The metric was changed to include the API version. I believe this was
> in
> the upgrade notes for 2.0.0.
>
> Ismael
>
> On Thu, Dec 13, 2018, 3:35 PM Raghav 
> > Hi
> >
> > We are trying to move from Kafka 1.1.0 to Kafka 2.1.0. We used to
> monitor
> > our 3 node Kafka using JMX. Upon moving to 2.1.0, we have observed
> that the
> > *below* mentioned metric can't be retrie
> > and we get the below exception:
> >
> >
> *"kafka.network:type=RequestMetrics,name=RequestsPerSec,request=Produce"*
> >
> > javax.management.InstanceNotFoundException:
> > kafka.network:type=RequestMetrics,name=RequestsPerSec,request=Produce
> > at
> >
> >
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1095)
> > at
> >
> >
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBeanInfo(DefaultMBeanServerInterceptor.java:1375)
> > at
> >
> >
> com.sun.jmx.mbeanserver.JmxMBeanServer.getMBeanInfo(JmxMBeanServer.java:920)
> > at
> >
> >
> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1462)
> > at
> >
> >
> javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
> > at
> >
> >
> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309)
> > at
> >
> >
> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401)
> > at
> >
> >
> javax.management.remote.rmi.RMIConnectionImpl.getMBeanInfo(RMIConnectionImpl.java:905)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:498)
> > at
> sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:324)
> > at sun.rmi.transport.Transport$1.run(Transport.java:200)
> > at sun.rmi.transport.Transport$1.run(Transport.java:197)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
> > at
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
> > at
> >
> >
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
> > at
> >
> >
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTran