[jira] [Created] (KAFKA-7751) Kryo convertor in kafka connect

[Manjeet Duhan (JIRA)]

Manjeet Duhan created KAFKA-7751:


 Summary: Kryo convertor in kafka connect
 Key: KAFKA-7751
 URL: https://issues.apache.org/jira/browse/KAFKA-7751
 Project: Kafka
  Issue Type: Improvement
Reporter: Manjeet Duhan


We had earlier SFDC logs project which was serializing complete request and 
response data in file store and our support team was deserializing it on demand 
whenever there was any issue in production. 

Approach :- We wanted to send data to elasticsearch via kafka connect to make 
it search optimistic.

Problem :- data size was high ~1GB so we wanted to save some space using any 
available compression Technics. At the same time , we wanted to utilize our 
confluent infrastructure.

 

Solution :- I have written Kryo serializer and converter for same to send and 
receive data into kafka connect and send it to elasticsearch. 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

[Eno Thereska]

Ryanne, thank you, this looks great and will be really appreciated by the
community.

My only comment at this point: is the REST api strictly necessary for this
KIP? Perhaps consider moving that to a different KIP since the main
contribution is more than sufficient. However that is just a suggestion.

Thanks
Eno

On Tue, Dec 18, 2018 at 4:46 AM Ryanne Dolan  wrote:

> > So, if we want to add it, it seems it would be useful to do it in a
> backward compatible way in the connect framework, rather than sth specific
> to MM
>
> Jun, that sgtm. The MirrorMaker driver I have right now creates multiple
> Herders (for multiple Kafka clusters) and exposes them through a high-level
> API. I think if we narrow the scope of this API a bit, we might have a
> better chance of eventually ending up with a single API for both MM and
> Connect.
>
> Specifically, I think MM2 can be stripped down to include only the
> endpoints in the existing Connect API, except grouped by Herder:
>
> GET //connectors
> PUT //connectors//config
> etc
>
> (i.e pretty much exactly what Sönke proposed earlier for Connect in
> general)
>
> This would accomplish what we need for MM2 without diverging much from the
> existing Connect API. Moreover, there would be a clear path to combining
> both APIs as Sönke suggested.
>
> What do you think?
>
> Ryanne
>
>
> On Mon, Dec 17, 2018 at 3:40 PM Jun Rao  wrote:
>
> > Hi, Sonke, Ryanne,
> >
> > Thanks for the explanation. To me, the single connect cluster model could
> > be useful for any connector, not just MM. So, if we want to add it, it
> > seems it would be useful to do it in a backward compatible way in the
> > connect framework, rather than sth specific to MM. I am not sure what the
> > best approach is. For example, one other option is KIP-296. If we feel
> > that's
> > adding too much work in this KIP, it might be ok to leave this part out
> in
> > this KIP.
> >
> > Jun
> >
> > On Fri, Dec 14, 2018 at 1:25 PM Ryanne Dolan 
> > wrote:
> >
> > > Thanks Sönke, you're spot-on. I don't want MM2 to wait for Connect
> > features
> > > that don't exist yet, especially if MM2 is the primary use case for
> them.
> > > Moreover, I think MM2 can drive and inform some of these features,
> which
> > > only makes sense if we adopt MM2 first.
> > >
> > > Ryanne
> > >
> > > On Fri, Dec 14, 2018, 9:03 AM Sönke Liebau
> > >  > >
> > > > Hi Jun,
> > > >
> > > > I believe Ryanne's idea is to run multiple workers per MM
> cluster-node,
> > > one
> > > > per target cluster. So in essence you'd specify three clusters in the
> > MM
> > > > config and MM would then instantiate one worker per cluster. Every MM
> > > > connector would then be deployed to the appropriate (internal) worker
> > > that
> > > > is configured for the cluster in question. Thus there would be no
> > changes
> > > > necessary to the Connect framework itself, everything would be
> handled
> > > by a
> > > > new layer around existing Connect code (probably a sibling
> > implementation
> > > > to the DistributedHerder if I understood him correctly). Ryanne,
> please
> > > > correct/expand if I misunderstood your intentions.
> > > >
> > > > To briefly summarize the discussion that Ryanne and I had around this
> > > > earlier, my opinion was that the extra layer could potentially be
> > avoided
> > > > by extending Connect instead, which would benefit all connectors.
> > > >
> > > > My proposal was to add a configuration option to the worker config
> that
> > > > allows defining "external clusters" which can then be referenced from
> > the
> > > > connector config.
> > > >
> > > > For example:
> > > >
> > > > # Core cluster config stays the same and is used for status, config
> and
> > > > offsets as usual
> > > > bootstrap.servers=localkafka1:9092,localkafka2:9092
> > > >
> > > > # Allow defining extra remote clusters
> > > >
> > > >
> > >
> >
> externalcluster.kafka_europe.bootstrap.servers=europekafka1:9092,europekafka2:9092
> > > > externalcluster.kafka_europe.security.protocol=SSL
> > > >
> > > >
> > >
> >
> externalcluster.kafka_europe.ssl.truststore.location=/var/private/ssl/kafka.client.truststore.jks
> > > > ...
> > > >
> > > >
> > >
> >
> externalcluster.kafka_asia.bootstrap.servers=asiakafka1:9092,asiakafka2:9092
> > > >
> > > >
> > > > When starting a connector you could now reference these
> pre-configured
> > > > clusters in the config:
> > > > {
> > > >   "name": "file-source",
> > > >   "config": {
> > > > "connector.class": "FileStreamSource",
> > > > "file": "/tmp/test.txt",
> > > > "topic": "connect-test",
> > > > "name": "file-source",
> > > > "cluster": "kafka_asia"
> > > >   }
> > > > }
> > > >
> > > > When omitting the "cluster" parameter current behavior of Connect
> > remains
> > > > unchanged. This way we could address multiple remote clusters from
> > > within a
> > > > single worker without adding the extra layer for MirrorMaker. I
> believe
> > > > that this could be done without major structural changes

Re: [DISCUSS] KIP-398: Support reading trust store from classpath

[Noa Resare]

I believe that I have addressed the concerns raised in this discussion. It 
seems reasonable to start a vote in about two days. Please raise any concerns 
you may have and I will be happy to attempt to address them.

/noa

> On 10 Dec 2018, at 10:53, Noa Resare  wrote:
> 
> Thank you for your comments, see replies inline.
> 
>> On 9 Dec 2018, at 01:33, Harsha  wrote:
>> 
>> Hi Noa,
>>  Based on KIP"s motivation section
>> "If we had the ability to load a trust store from the classpath as well as 
>> from a file, the trust store could be shipped in a jar that could be 
>> declared as a dependency and piggyback on the distribution infrastructure 
>> already in place."
>> 
>> It looks like you are making the assumption that distributing a jar is 
>> better than the file. I am not sure why one is better than the other. There 
>> are other use-cases where one can make a call local "daemon" over Unix 
>> socket to fetch a certificate as well.
> 
> It was not my intention to convey that loading the trust store from the 
> classpath is inherently better in all cases. The proposed change simply 
> brings more choice. That said, I do believe that maven central and the 
> transitive dependency features of maven, gradle and ivy makes for a smoother 
> user experience in many cases. Validating broker certificates against a 
> organisation wide private CA cert has benefits in that it means that the 
> person setting up the kafka cluster(s) doesn’t need to bother with purchasing 
> or obtaining publicly trusted certs for every broker while still providing 
> strong cryptographic validation that a client is connecting to a legitimate 
> endpoint. If there was a way to distribute a private trust store that is as 
> easy as declaring an additional maven style dependency, I imagine that this 
> would be a more appealing proposition than it is today. I would assume that 
> many organisations opt to disable strict host checking in certificates to 
> sidestep the CA cert distribution problem. I think it would be valuable to 
> make it slightly more easy to do the right thing.
> 
>> 
>> Just supporting a "classpath" option might work for a few users but it's not 
>> generic enough to support a wide variety of other infrastructures. My 
>> suggestion if the KIP motivation is to make the certificate/truststore 
>> available with different mechanisms, Lets make a interface that allow users 
>> to roll their own based on their infra and support File as the default 
>> mechanism so that we can support existing users.
> 
> I agree that this is a fairly small change that doesn’t aim to support all 
> possible mechanisms that one might conceive of. I believe that KIP-383: 
> Pluggable interface for SSL Factory 
> 
>  might be a good vehicle to provide this level of flexibility, but even if 
> that proposal is accepted I still think that there is room for this KIP to 
> provide an easy to use solution to this problem.
> 
> cheers
> noa
> 
>> -Harsha
>> 
>> On Sat, Dec 8, 2018, at 7:03 AM, Noa Resare wrote:
>>> 
>>> 
 On 6 Dec 2018, at 20:16, Rajini Sivaram  wrote:
 
 Hi Noa,
 
 Thanks for the KIP. A few comments/questions:
 
 1. If we support filenames starting with `classpath:` by requiring
 `file:`prefix,
 then we are presumably not supporting files starting `file:`. Not
 necessarily an issue, but we do need to document any restrictions.
>>> 
>>> I think that it would be trivial to support ‘file:’ as a prefix in a 
>>> filesystem path
>>> by just asking the user that really want that to add it twice:
>>> 
>>> The config value "file:file:my_weird_file_name" would map to the 
>>> filesystem path "file:my_weird_file_name”
>>> 
>>> 
 2. On the broker-side, trust stores are dynamically updatable. And we use
 file modification time to decide whether trust store needs to be reloaded.
 This is less of an issue once we implement
 https://cwiki.apache.org/confluence/display/KAFKA/KIP-339%3A+Create+a+new+IncrementalAlterConfigs+API,
 but at the moment, we are relying on actual files on the file system for
 which we can compare modification times.
>>> 
 3. On the client-side, trust stores are not currently updatable. And we
 don't have an API to make them updatable. By using class path, we preclude
 the use of file modification times in future to detect key or trust store
 updates for clients. It will be good to get feedback from the community on
 whether this is a reasonable longer-term restriction.
>>> 
>>> Interesting. I think that it is a reasonable graceful degradation to 
>>> simply not pick up on changed truststores
>>> read from the classpath as long as it is documented, but if we really 
>>> want we could save a checksum of
>>> the truststore, re-read and compare to determine any changes.
>>> 
 4. It will be good to get more feedback from the commun

Re: [DISCUSS] KIP-331 Add default implementation to close() and configure() for Serializer, Deserializer and Serde

[Saïd Bouras]

Hi everyone !

I saw this KIP and I comment the PR, but like *@Chia-Ping* mentioned, it's
better to talk here with community.

Well to summarize, I think that putting the methods *configure* and *close*
default could bring misunderstanding in the approach when implementing
customs serializers and deserializers. Now we know that we have to do some
configurations in theses methods and that it's the default behavior but by
hiding them, developers can think that it's not  "normal" to overrides this
methods.

My idea would be to extends *Serializer* and *Deserializer* interfaces to
specify that we do not need configuration in them, we can call them
*UnConfigureSerializer*/*UnconfigureDeserializer. S*erializers and
deserializers that do nothing in *configure* and *close* methods can
implement them directly and let the other implements the base interface.

That way, developers that begin to work with Kafka can see that we give
them two approach to create their custom serdes.

I don't know if I'm clear, tell me if I don't.
Cheers

On Wed, Jul 4, 2018 at 5:06 PM Chia-Ping Tsai  wrote:

> hi John
>
> > Just really scraping my mind for concerns to investigate... This change
> > won't break source compatibility, but will it affect binary
> compatibility?
> > For example, if I compile my application against Kafka 2.0, for example,
> > and then swap in the Kafka jar containing your change on my classpath at
> > run time, will it still work?
> > I think it should be binary compatible, assuming Oracle didn't do
> anything
> > crazy, since all the method references would be the same. It might we
> worth
> > an experiment, though.
>
> The side effect of this change is IncompatibleClassChangeError caused by
> the conflict of multi default implementations (see
> https://docs.oracle.com/javase/specs/jls/se8/html/jls-13.html#jls-13.5.6).
> But the change won't introduce the such error since there is no default
> impl before.
>
> I will start the Vote at 7/6.
>
> Cheers,
> chia-ping
>
> On 2018/07/03 18:17:28, John Roesler  wrote:
> > Hi again, Chia-Ping,
> >
> > Thanks! I took a look at it, and it seems like a good change to me.
> >
> > I don't know if it will generate much discussion, so I recommend just
> > letting it steep for another day or two and then moving on to a vote if
> no
> > one else chimes in.
> >
> > Just really scraping my mind for concerns to investigate... This change
> > won't break source compatibility, but will it affect binary
> compatibility?
> > For example, if I compile my application against Kafka 2.0, for example,
> > and then swap in the Kafka jar containing your change on my classpath at
> > run time, will it still work?
> >
> > I think it should be binary compatible, assuming Oracle didn't do
> anything
> > crazy, since all the method references would be the same. It might we
> worth
> > an experiment, though.
> >
> > Thanks,
> > -John
> >
> > On Mon, Jul 2, 2018 at 9:29 PM Chia-Ping Tsai 
> wrote:
> >
> > > > Can you provide a link, please?
> > >
> > > Pardon me. I put the page in the incorrect location.
> > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-331+Add+default+implementation+to+close%28%29+and+configure%28%29+for+Serializer%2C+Deserializer+and+Serde
> > >
> > > Cheers,
> > > Chia-Ping
> > >
> > > On 2018/07/02 19:45:19, John Roesler  wrote:
> > > > Hi Chia-Ping,
> > > >
> > > > I couldn't find KIP-331 in the list of KIPs (
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > > > ).
> > > >
> > > > Can you provide a link, please?
> > > >
> > > > Thanks,
> > > > -John
> > > >
> > > > On Sun, Jul 1, 2018 at 11:33 AM Chia-Ping Tsai 
> > > wrote:
> > > >
> > > > > hi folks,
> > > > >
> > > > > KIP-331 is waiting for any suggestions, feedback and reviews. The
> main
> > > > > purpose of the KIP-331 is to add empty implementations to the
> close()
> > > and
> > > > > configure() so as to user can write less code to develop custom
> > > Serialzier,
> > > > > Deserializer and Serde.
> > > > >
> > > > > Cheers,
> > > > > Chia-Ping
> > > > >
> > > >
> > >
> >
>
-- 

*Saïd BOURAS*

Re: [DISCUSS] KIP-271 Add NetworkClient redirector

[Sergey Korytnikov]

Dear devs,

It's been a while since I created KIP-271 and received just a little 
feedback. I'm going to start the voting process in a couple of days as I 
feel responsible for the KIP created. Community needs to decide whether 
the change is viable or not, especially, since kafka 1.0 when the change 
has been originally proposed, so that the change has to be retested with 
trunk head (> 2.1).


As part of the update from my side, I've added advertised.listeners 
proposal as rejected alternative to KIP wiki page.
I still believe Kafka Client should have the flexibility to override the 
routing rules in cases when the deployment environment with multiple 
Kafka broker instances does not allow to fine tune each replica.


Thanks,
Sergey

On 3/17/2018 9:12 AM, Sergey Korytnikov wrote:

Hi, Rajini,

Thank you! This is really great advice, I will review that and test it 
out in our scenarios.

Do you know any limitations when advertised.listeners may not work?

Having the option to redirecting broker calls on client side is still 
valuable for kafka server setup that cannot have advertised.listeners 
configured for any reason.


Thanks,
Sergey


On Fri, Mar 16, 2018 at 3:41 PM, Rajini Sivaram 
mailto:rajinisiva...@gmail.com>> wrote:


Hi Sergey,

Thanks for the KIP. Is there a reason why brokers cannot be
configured with
`advertised.listeners` that use hostnames (or addresses) that
clients can
connect to, instead of performing a redirection?

Regards,

Rajini

On Fri, Mar 16, 2018 at 1:24 PM, Sergey Korytnikov <
sergey.korytni...@gmail.com >
wrote:

> Hi devs,
>
> I have created new KIP-271 for discussion:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-

> 271%3A+Add+NetworkClient+redirector
> Kafka JIRA issue:
https://issues.apache.org/jira/browse/KAFKA-6669

>
> The fix has been implemented and tested with version 1.0 and
trunk snapshot
> of clients jar.
> Changes are available here:
> https://github.com/ssbiox/kafka/tree/trunk-redirector

> Diff of changes:
>
https://github.com/apache/kafka/compare/trunk...ssbiox:trunk-redirector

>
> Thank you,
> Sergey
>

[jira] [Created] (KAFKA-7752) zookeeper-security-migration.sh does not remove ACL on kafka-acl-extended

[Attila Sasvari (JIRA)]

Attila Sasvari created KAFKA-7752:
-

 Summary: zookeeper-security-migration.sh does not remove ACL on 
kafka-acl-extended
 Key: KAFKA-7752
 URL: https://issues.apache.org/jira/browse/KAFKA-7752
 Project: Kafka
  Issue Type: Bug
  Components: tools
Affects Versions: 2.0.0
Reporter: Attila Sasvari


Executed {{zookeeper-security-migration.sh --zookeeper.connect $(hostname 
-f):2181/kafka --zookeeper.acl secure}} to secure Kafka znodes and then 
{{zookeeper-security-migration.sh --zookeeper.connect $(hostname -f):2181/kafka 
--zookeeper.acl unsecure}} to unsecure those.

I noticed that the tool did not remove ACLs on certain nodes: 
{code}
] getAcl /kafka/kafka-acl-extended
'world,'anyone
: r
'sasl,'kafka
: cdrwa
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[DISCUSS] KIP-409: Allow creating under-replicated topics and partitions

[Mickael Maison]

Hi,

We have submitted a KIP to handle topics and partitions creation when
a cluster is not fully available:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-409%3A+Allow+creating+under-replicated+topics+and+partitions

As always, we welcome feedback and suggestions.

Thanks
Mickael and Edoardo

[jira] [Created] (KAFKA-7753) ValueTransformerWithKey should not require producing one value every message

[Mateusz Owczarek (JIRA)]

Mateusz Owczarek created KAFKA-7753:
---

 Summary: ValueTransformerWithKey should not require producing one 
value every message
 Key: KAFKA-7753
 URL: https://issues.apache.org/jira/browse/KAFKA-7753
 Project: Kafka
  Issue Type: Wish
  Components: streams
Reporter: Mateusz Owczarek


Hi, speaking about: 
[https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/kstream/ValueTransformerWithKey.java]
I have a quite simple case - I want to implement a Transformer which I will use 
later in my DSL-api-defined topology. Those are my requirements:
- no repartition topic should be created since I do not change the keys
- I can't forward messages on every message transformed. I have my internal 
state store which I'm using to ensure max. 1 message per window is sent.

Basic transformer gives me possibility to not send messages down the stream 
(null internal handler), but it forces repartitioning which I think is 
unnecessary in my case. WDYT?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Windows Testing

[Jacob Braaten]

Hello,

Quick question about your testing setup. I can't find where I read this
before and it may be incorrect so I wanted to check with you.

Do you guys do any stress or integration testing in windows environments?
I'm fairly certain I read somewhere that you only do testing in linux
environments, but I wanted to be sure.

Thanks,
Jake

Re: [VOTE] [REMINDER] KIP-383 Pluggable interface for SSL Factory

[Harsha]

Yes. +1 binding.

Thanks,
Harsha

On Mon, Dec 17, 2018, at 5:21 AM, Pellerin, Clement wrote:
> I'm new here. Is this vote binding or not?
> 
> -Original Message-
> From: Harsha [mailto:ka...@harsha.io] 
> Sent: Saturday, December 15, 2018 1:59 PM
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] [REMINDER] KIP-383 Pluggable interface for SSL Factory
> 
> Overall LGTM. +1.
> 
> Thanks,
> Harsha

Re: Windows Testing

[M. Manna]

There’s no specific test or setup for Windows. There are issues with file
cleanup on windows, so your best bet is to run it on containers or Linux.

Thanks,

On Tue, 18 Dec 2018 at 17:06, Jacob Braaten  wrote:

> Hello,
>
> Quick question about your testing setup. I can't find where I read this
> before and it may be incorrect so I wanted to check with you.
>
> Do you guys do any stress or integration testing in windows environments?
> I'm fairly certain I read somewhere that you only do testing in linux
> environments, but I wanted to be sure.
>
> Thanks,
> Jake
>

Re: [DISCUSS] KIP-404: Add Kafka Connect configuration parameter for disabling WADL output on OPTIONS request

[Oleksandr Diachenko]

Konstantine,

Just to clarify, this option(*rest.wadl.enable*), is proposed to be
introduced in the KIP,
it was never used anywhere else than in unit tests, as a part of PR for the
KIP.
So no public interface changes are needed if we proceed without exposing
the option.
WDYT?

Regards, Alex.

On Mon, Dec 17, 2018 at 9:19 AM Konstantine Karantasis <
konstant...@confluent.io> wrote:

> Thanks for considering removal Alex.
> I totally agree with your assessment. Still, I'd be in favor of making
> KIP-404 a small KIP that describes that this option is now being disabled.
> (If I'm not mistaken, one place I've noticed this feature being used is in
> Connect's unit tests for the rest interface).
>
> Konstantine
>
> On Fri, Dec 14, 2018 at 5:00 PM Oleksandr Diachenko <
> alex.diache...@confluent.io> wrote:
>
> > Konstantine and Jason,
> >
> > I do agree that this functionality was not documented, and most likely
> not
> > intended to be present.
> > Therefore we can consider it as not a part of the public interface, and
> > current behavior as not expected.
> > Hence, addressing the issue by just disabling the WADL output seems like
> a
> > viable solution to me.
> >
> > In order to proceed, do we need this KIP at all, or creating a new JIRA
> and
> > fixing it as a bug without
> > changes in public interfaces are sufficient?
> >
> > Regards, Alex.
> >
> > On Fri, Dec 14, 2018 at 4:18 PM Jason Gustafson 
> > wrote:
> >
> > > Hi Alex,
> > >
> > > I think WADL support was likely unintentional, so this could be treated
> > as
> > > more of a bug. Unless we think it's a good idea to support it going
> > > forward, I'd suggest going with the rejected alternative of just
> turning
> > it
> > > off. What do you think?
> > >
> > > Thanks,
> > > Jason
> > >
> > > On Fri, Dec 14, 2018 at 3:06 PM Oleksandr Diachenko <
> > > alex.diache...@confluent.io> wrote:
> > >
> > > > Thanks, everyone for taking the time to review the KIP.
> > > >
> > > > It looks like there are no major objections on it, so I will start
> > voting
> > > > thread.
> > > >
> > > > Regards, Alex.
> > > >
> > > >
> > > >
> > > > On Thu, Dec 13, 2018 at 3:50 PM Randall Hauch 
> > wrote:
> > > >
> > > > > Thanks, Alex. The KIP looks good to me.
> > > > >
> > > > > Randall
> > > > >
> > > > > On Wed, Dec 12, 2018 at 10:08 PM Guozhang Wang  >
> > > > wrote:
> > > > >
> > > > > > Alex,
> > > > > >
> > > > > > Thanks for putting up this KIP. The proposal lgtm.
> > > > > >
> > > > > > Guozhang
> > > > > >
> > > > > > On Wed, Dec 12, 2018 at 7:41 PM Oleksandr Diachenko <
> > > > > odiache...@apache.org
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I would like to start a discussing for the following KIP:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-404%3A+Add+Kafka+Connect+configuration+parameter+for+disabling+WADL+output+on+OPTIONS+request
> > > > > > > .
> > > > > > >
> > > > > > > The KIP proposes to add a configuration parameter for Connect
> > > Worker,
> > > > > > which
> > > > > > > would allow to not expose WADL information in Connect REST api
> > > > > responces.
> > > > > > >
> > > > > > > Feedback is appreciated, thanks in advance.
> > > > > > >
> > > > > > > Regards, Alex.
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > -- Guozhang
> > > > > >
> > > > >
> > > >
> > >
> >
>

transaction 2PC protocol

[Jose Raul Perez Rodriguez]

Hi all,

Reading this 
 
document about transactions in Kafka, specifically epigraphs 5.2 
WriteTxnMarkerRequest and 5.3 Writing the final Commit or Abort Message.


What I understand  from *5.2* is that the coordinator sends a "write txt 
market request" to each topic/partition leader in the transaction, then, 
each leader broker in those topic-partition write a Commit or Abort 
message to the log, then, using this information the Consumer for that 
particular topic-partition decide to read the messages (in case of 
Commit) or to drop the message (in case of Abort).


My doubt is; those messages that passed the "write txt market request" 
phase for a particular topic-partition (the Commit cases) are just 
delivered to the user?, or hided from the user until full transaction 
confirms it Committed?, in the case is delivered to the user, there 
could be inconsistent reads, because another topic-partition could fail 
and then the full transaction needs to abort. On the other hand, if 
those messages are not delivered to user until the Consumer reads a 
Commit message for the full transaction *(5.3*), here is my second 
question; how this works?, i.e how a Consumer is aware that a particular 
message belonging to a transaction can be delivered to the user, i.e the 
transaction that owns the succeed message?


I hope is clear this explanation, I have not found this question in any doc.

Thanks in advance,

Re: [DISCUSS] KIP-404: Add Kafka Connect configuration parameter for disabling WADL output on OPTIONS request

[Konstantine Karantasis]

My point was that, even though leaving this feature open via the REST
interface was not intended initially and therefore we can treat this as a
bug with respect to its deprecation,
I believe it'd be good to document its removal with KIP-404, since it's de
facto part of the public interface of the REST api.

But I don't feel too strong about whether we proceed with a KIP or just a
PR.

Best,
Konstantine.

On Tue, Dec 18, 2018 at 1:10 PM Oleksandr Diachenko <
alex.diache...@confluent.io> wrote:

> Konstantine,
>
> Just to clarify, this option(*rest.wadl.enable*), is proposed to be
> introduced in the KIP,
> it was never used anywhere else than in unit tests, as a part of PR for the
> KIP.
> So no public interface changes are needed if we proceed without exposing
> the option.
> WDYT?
>
> Regards, Alex.
>
> On Mon, Dec 17, 2018 at 9:19 AM Konstantine Karantasis <
> konstant...@confluent.io> wrote:
>
> > Thanks for considering removal Alex.
> > I totally agree with your assessment. Still, I'd be in favor of making
> > KIP-404 a small KIP that describes that this option is now being
> disabled.
> > (If I'm not mistaken, one place I've noticed this feature being used is
> in
> > Connect's unit tests for the rest interface).
> >
> > Konstantine
> >
> > On Fri, Dec 14, 2018 at 5:00 PM Oleksandr Diachenko <
> > alex.diache...@confluent.io> wrote:
> >
> > > Konstantine and Jason,
> > >
> > > I do agree that this functionality was not documented, and most likely
> > not
> > > intended to be present.
> > > Therefore we can consider it as not a part of the public interface, and
> > > current behavior as not expected.
> > > Hence, addressing the issue by just disabling the WADL output seems
> like
> > a
> > > viable solution to me.
> > >
> > > In order to proceed, do we need this KIP at all, or creating a new JIRA
> > and
> > > fixing it as a bug without
> > > changes in public interfaces are sufficient?
> > >
> > > Regards, Alex.
> > >
> > > On Fri, Dec 14, 2018 at 4:18 PM Jason Gustafson 
> > > wrote:
> > >
> > > > Hi Alex,
> > > >
> > > > I think WADL support was likely unintentional, so this could be
> treated
> > > as
> > > > more of a bug. Unless we think it's a good idea to support it going
> > > > forward, I'd suggest going with the rejected alternative of just
> > turning
> > > it
> > > > off. What do you think?
> > > >
> > > > Thanks,
> > > > Jason
> > > >
> > > > On Fri, Dec 14, 2018 at 3:06 PM Oleksandr Diachenko <
> > > > alex.diache...@confluent.io> wrote:
> > > >
> > > > > Thanks, everyone for taking the time to review the KIP.
> > > > >
> > > > > It looks like there are no major objections on it, so I will start
> > > voting
> > > > > thread.
> > > > >
> > > > > Regards, Alex.
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Dec 13, 2018 at 3:50 PM Randall Hauch 
> > > wrote:
> > > > >
> > > > > > Thanks, Alex. The KIP looks good to me.
> > > > > >
> > > > > > Randall
> > > > > >
> > > > > > On Wed, Dec 12, 2018 at 10:08 PM Guozhang Wang <
> wangg...@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Alex,
> > > > > > >
> > > > > > > Thanks for putting up this KIP. The proposal lgtm.
> > > > > > >
> > > > > > > Guozhang
> > > > > > >
> > > > > > > On Wed, Dec 12, 2018 at 7:41 PM Oleksandr Diachenko <
> > > > > > odiache...@apache.org
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > I would like to start a discussing for the following KIP:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-404%3A+Add+Kafka+Connect+configuration+parameter+for+disabling+WADL+output+on+OPTIONS+request
> > > > > > > > .
> > > > > > > >
> > > > > > > > The KIP proposes to add a configuration parameter for Connect
> > > > Worker,
> > > > > > > which
> > > > > > > > would allow to not expose WADL information in Connect REST
> api
> > > > > > responces.
> > > > > > > >
> > > > > > > > Feedback is appreciated, thanks in advance.
> > > > > > > >
> > > > > > > > Regards, Alex.
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > -- Guozhang
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-404: Add Kafka Connect configuration parameter for disabling WADL output on OPTIONS request

[Randall Hauch]

Konstatine, IIUC your suggestion is to change KIP-404 to document the
removal of the support for WADL since it was never intended to be exposed,
and to no longer make it an option. I agree we could make that change, but
I'm also wondering if there is precedence for this kind of a KIP. Anyone
know?

If there is no precedence, then I would be fine with simply addressing this
as a PR and backporting -- the WADL operation was never documented nor
described in any of the prior KIPs, and it seems like an implementation
detail that leaked out accidentally.

Randall

On Tue, Dec 18, 2018 at 4:15 PM Konstantine Karantasis <
konstant...@confluent.io> wrote:

> My point was that, even though leaving this feature open via the REST
> interface was not intended initially and therefore we can treat this as a
> bug with respect to its deprecation,
> I believe it'd be good to document its removal with KIP-404, since it's de
> facto part of the public interface of the REST api.
>
> But I don't feel too strong about whether we proceed with a KIP or just a
> PR.
>
> Best,
> Konstantine.
>
> On Tue, Dec 18, 2018 at 1:10 PM Oleksandr Diachenko <
> alex.diache...@confluent.io> wrote:
>
> > Konstantine,
> >
> > Just to clarify, this option(*rest.wadl.enable*), is proposed to be
> > introduced in the KIP,
> > it was never used anywhere else than in unit tests, as a part of PR for
> the
> > KIP.
> > So no public interface changes are needed if we proceed without exposing
> > the option.
> > WDYT?
> >
> > Regards, Alex.
> >
> > On Mon, Dec 17, 2018 at 9:19 AM Konstantine Karantasis <
> > konstant...@confluent.io> wrote:
> >
> > > Thanks for considering removal Alex.
> > > I totally agree with your assessment. Still, I'd be in favor of making
> > > KIP-404 a small KIP that describes that this option is now being
> > disabled.
> > > (If I'm not mistaken, one place I've noticed this feature being used is
> > in
> > > Connect's unit tests for the rest interface).
> > >
> > > Konstantine
> > >
> > > On Fri, Dec 14, 2018 at 5:00 PM Oleksandr Diachenko <
> > > alex.diache...@confluent.io> wrote:
> > >
> > > > Konstantine and Jason,
> > > >
> > > > I do agree that this functionality was not documented, and most
> likely
> > > not
> > > > intended to be present.
> > > > Therefore we can consider it as not a part of the public interface,
> and
> > > > current behavior as not expected.
> > > > Hence, addressing the issue by just disabling the WADL output seems
> > like
> > > a
> > > > viable solution to me.
> > > >
> > > > In order to proceed, do we need this KIP at all, or creating a new
> JIRA
> > > and
> > > > fixing it as a bug without
> > > > changes in public interfaces are sufficient?
> > > >
> > > > Regards, Alex.
> > > >
> > > > On Fri, Dec 14, 2018 at 4:18 PM Jason Gustafson 
> > > > wrote:
> > > >
> > > > > Hi Alex,
> > > > >
> > > > > I think WADL support was likely unintentional, so this could be
> > treated
> > > > as
> > > > > more of a bug. Unless we think it's a good idea to support it going
> > > > > forward, I'd suggest going with the rejected alternative of just
> > > turning
> > > > it
> > > > > off. What do you think?
> > > > >
> > > > > Thanks,
> > > > > Jason
> > > > >
> > > > > On Fri, Dec 14, 2018 at 3:06 PM Oleksandr Diachenko <
> > > > > alex.diache...@confluent.io> wrote:
> > > > >
> > > > > > Thanks, everyone for taking the time to review the KIP.
> > > > > >
> > > > > > It looks like there are no major objections on it, so I will
> start
> > > > voting
> > > > > > thread.
> > > > > >
> > > > > > Regards, Alex.
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, Dec 13, 2018 at 3:50 PM Randall Hauch 
> > > > wrote:
> > > > > >
> > > > > > > Thanks, Alex. The KIP looks good to me.
> > > > > > >
> > > > > > > Randall
> > > > > > >
> > > > > > > On Wed, Dec 12, 2018 at 10:08 PM Guozhang Wang <
> > wangg...@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Alex,
> > > > > > > >
> > > > > > > > Thanks for putting up this KIP. The proposal lgtm.
> > > > > > > >
> > > > > > > > Guozhang
> > > > > > > >
> > > > > > > > On Wed, Dec 12, 2018 at 7:41 PM Oleksandr Diachenko <
> > > > > > > odiache...@apache.org
> > > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > I would like to start a discussing for the following KIP:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-404%3A+Add+Kafka+Connect+configuration+parameter+for+disabling+WADL+output+on+OPTIONS+request
> > > > > > > > > .
> > > > > > > > >
> > > > > > > > > The KIP proposes to add a configuration parameter for
> Connect
> > > > > Worker,
> > > > > > > > which
> > > > > > > > > would allow to not expose WADL information in Connect REST
> > api
> > > > > > > responces.
> > > > > > > > >
> > > > > > > > > Feedback is appreciated, thanks in advance.
> > > > > > > > >
> > > > 

Monitoring Mirror Maker Lag & source

[Saravanan B]

Hi Team,

We have few kafka clusters replicating & we would like to monitor the lag
per topic on source, in mirror maker we have configured CONSUMER_GROUP_ID &
replication is working fine.

When i run "kafka-consumer-groups.sh  --list --bootstrap-server
bootstrap.kafka:9092" , i dnt find the consumer group listed in the source
kafka cluster.

Please let me know how we can get the lag per topic using describe consumer
group for mirror maker.

Thanks a lot in advance

[jira] [Created] (KAFKA-7754) zookeeper-security-migration.sh sets the root ZNode as world-readable

[Badai Aqrandista (JIRA)]

Badai Aqrandista created KAFKA-7754:
---

 Summary: zookeeper-security-migration.sh sets the root ZNode as 
world-readable
 Key: KAFKA-7754
 URL: https://issues.apache.org/jira/browse/KAFKA-7754
 Project: Kafka
  Issue Type: Bug
  Components: security
Affects Versions: 2.0.1
Reporter: Badai Aqrandista


If I start broker with {{zookeeper.set.acl=true}} from the first time I start 
the broker, the root ZNode is not set to be world-readable to allow other 
application to share the Zookeeper ensemble with chroot.

But if I run {{zookeeper-security-migration.sh}} with  {{–zookeeper.acl 
secure}}, the root ZNode becomes world-readable. Is this correct?

 
{noformat}
root@localhost:/# zookeeper-shell localhost:2181
Connecting to localhost:2181
Welcome to ZooKeeper!
JLine support is enabled
[zk: localhost:2181(CONNECTING) 0] 
WATCHER::

WatchedEvent state:SyncConnected type:None path:null

WATCHER::

WatchedEvent state:SaslAuthenticated type:None path:null

[zk: localhost:2181(CONNECTED) 0] getAcl /
'world,'anyone
: cdrwa
[zk: localhost:2181(CONNECTED) 1] getAcl /brokers
'world,'anyone
: r
'sasl,'kafkabroker
: cdrwa
[zk: localhost:2181(CONNECTED) 2] quit
Quitting...
root@localhost:/# zookeeper-security-migration --zookeeper.acl secure 
--zookeeper.connect localhost:2181
root@localhost:/# zookeeper-shell localhost:2181
Connecting to localhost:2181
Welcome to ZooKeeper!
JLine support is enabled
[zk: localhost:2181(CONNECTING) 0] 
WATCHER::

WatchedEvent state:SyncConnected type:None path:null

WATCHER::

WatchedEvent state:SaslAuthenticated type:None path:null

[zk: localhost:2181(CONNECTED) 0] getAcl /
'world,'anyone
: r
'sasl,'kafkabroker
: cdrwa
[zk: localhost:2181(CONNECTED) 1] getAcl /brokers
'world,'anyone
: r
'sasl,'kafkabroker
: cdrwa
[zk: localhost:2181(CONNECTED) 2] 
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-7755) Kubernetes - Kafka clients are resolving DNS entries only one time

[JIRA]

Loïc Monney created KAFKA-7755:
--

 Summary: Kubernetes - Kafka clients are resolving DNS entries only 
one time
 Key: KAFKA-7755
 URL: https://issues.apache.org/jira/browse/KAFKA-7755
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 2.1.0, 2.2.0, 2.1.1
 Environment: Kubernetes
Reporter: Loïc Monney


*Introduction*
 Since 2.1.0 Kafka clients are supporting multiple DNS resolved IP addresses if 
the first one fails. This change has been introduced by 
https://issues.apache.org/jira/browse/KAFKA-6863. However this DNS resolution 
is now performed only one time by the clients. This is not a problem if all 
brokers have fixed IP addresses, however this is definitely an issue when Kafka 
brokers are run on top of Kubernetes. Indeed, new Kubernetes pods will receive 
another IP address, so as soon as all brokers will have been restarted clients 
won't be able to reconnect to any broker.

*Impact*
 Everyone running Kafka 2.1 or later on top of Kubernetes is impacted when a 
rolling restart is performed.

*Root cause*
 Since https://issues.apache.org/jira/browse/KAFKA-6863 Kafka clients are 
resolving DNS entries only once.

*Proposed solution*
 In 
[https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/ClusterConnectionStates.java#L368]
 Kafka clients should perform the DNS resolution again when all IP addresses 
have been "used" (when _index_ is back to 0)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)