Re: [GitHub] [kafka-site] mjsax commented on pull request #156: MINOR increasing queued.max.requests for production recommendations in 0.9 and newer.

[M. Manna]

Unsubscribe


On Sun, 28 Mar 2021 at 20:44, GitBox  wrote:

>
> mjsax commented on pull request #156:
> URL: https://github.com/apache/kafka-site/pull/156#issuecomment-808948878
>
>
>Closing this PR as abandoned. (We can reopen if necessary.)
>
>
> --
> This is an automated message from the Apache Git Service.
> To respond to the message, please log on to GitHub and use the
> URL above to go to the specific comment.
>
> For queries about this service, please contact Infrastructure at:
> us...@infra.apache.org
>
>
>

Re: We want to remove Zookeeper from our Kafka cluster which version to use?

[M. Manna]

Manor,

If you check the KIP it’s not yet implemented.  Also, I believe we might
see some update on ASF Kafka site and/or Confluent soon about this. In
other words, you cannot just use it yet.

Last I remember, Collin was PMing KIP-500 - so I would let them comment on
this.

Regards,


On Tue, 19 Jan 2021 at 16:45, Manojbvn Barthipudi 
wrote:

> Hi All,
>
> We have a Kafka cluster [5-10 nodes] with multiple topics [with multi
> partitions] managed by the Zookeeper cluster. We want to remove the
> Zookeeper and manage the metadata using Kafka itself. I have gone through
> KIP 500 [Kafka] but nowhere it is mentioned which version of Kafka will
> this capability and by when it will be available? Please advise
>
> Thanks
> Manoj
>

Re: Why many "Load Bug xxx" JIRA bug by Tim?

[M. Manna]

I had to register this as spam and block them. I couldn’t disable it from
ASF JiRA.

 I’m also curious to know how/why such surge occurred.

Regards,

On Wed, 6 Jan 2021 at 03:45, Luke Chen  wrote:

> Hi,
> I received a lot of JIRA notification emails today, and they are all
> titled: "Load Bug xxx" by Tim.
> The bug content doesn't look like a real bug, they are like generated by
> automation.
> I'm wondering why that could happen?
> Do we have any way to delete them all?
>
> Thanks.
> Luke
>

Re: [jira] [Created] (KAFKA-10930) Load Bug rsTheretuv 18

[M. Manna]

Can this be stopped ? Some spamming going on ?

On Tue, 5 Jan 2021 at 18:27, Tim van der Kooi (Jira) 
wrote:

> Tim van der Kooi created KAFKA-10930:
> 
>
>  Summary: Load Bug rsTheretuv 18
>  Key: KAFKA-10930
>  URL: https://issues.apache.org/jira/browse/KAFKA-10930
>  Project: Kafka
>   Issue Type: Bug
> Reporter: Tim van der Kooi
>
>
> DEF Square new hors2es and put better end. Sincerity collected happiness
> do is contented. Sigh ever way now many. Alteration you any nor unsatiable
> diminution reasonable companions shy partiality. Leaf by left deal mile oh
> if easy. Added woman first get led joy not early jokes. On projection
> apartments unsatiable so if he entreaties appearance. Rose you wife how set
> lady half wish. Hard sing an in true felt. Welcomed stronger if steepest
> ecstatic an suitable finished of oh. Entered at excited at forming between
> so produce. Chicken unknown besides attacks gay compact out you. Continuing
> no simplicity no favourable on reasonably melancholy estimating. Own hence
> views two ask right whole ten seems. What near kept met call old west dine.
> Our announcing sufficient why pianoforte. He moonlight difficult engrossed
> an it sportsmen. Interested has all devonshire difficulty gay assistance
> joy. Unaffected at ye of compliment alteration to. Place voice no arise
> along to. Parlors waiting so against me no. Wishing calling are warrant
> settled was luckily. Express besides it present if at an opinion visitor.
> On no twenty spring of in esteem spirit likely estate. Continue new you
> declared differed learning bringing honoured. At mean mind so upon they
> rent am walk. Shortly am waiting inhabit smiling he chiefly of in. Lain
> tore time gone him his dear sure. Fat decisively estimating affronting
> assistance not. Resolve pursuit regular so calling me. West he plan girl
> been my then up no. He oppose at thrown desire of no. Announcing impression
> unaffected day his are unreserved indulgence. Him hard find read are you
> sang. Parlors visited noisier how explain pleased his see suppose. Do
> ashamed assured on related offence at equally totally. Use mile her whom
> they its. Kept hold an want as he bred of. Was dashwood landlord cheerful
> husbands two. Estate why theirs indeed him polite old settle though she. In
> as at regard easily narrow roused adieus. On projection apartments
> unsatiable so if he entreaties appearance. Rose you wife how set lady half
> wish. Hard sing an in true felt. Welcomed stronger if steepest ecstatic an
> suitable finished of oh. Entered at excited at forming between so produce.
> Chicken unknown besides attacks gay compact out you. Continuing no
> simplicity no favourable on reasonably melancholy estimating. Own hence
> views two ask right whole ten seems. What near kept met call old west dine.
> Our announcing sufficient why pianoforte. He moonlight difficult engrossed
> an it sportsmen. Interested has all devonshire difficulty gay assistance
> joy. Unaffected at ye of compliment alteration to. Place voice no arise
> along to. Parlors waiting so against me no. Wishing calling are warrant
> settled was luckily. Express besides it present if at an opinion visitor.
> On no twenty spring of in esteem spirit likely estate. Continue new you
> declared differed learning bringing honoured. At mean mind so upon they
> rent am walk. Shortly am waiting inhabit smiling he chiefly of in. Lain
> tore time gone him his dear sure. Fat decisively estimating affronting
> assistance not. Resolve pursuit regular so calling me. West he plan girl
> been my then up no. He oppose at thrown desire of no. Announcing impression
> unaffected day his are unreserved indulgence. Him hard find read are you
> sang. Parlors visited noisier how explain pleased his see suppose. Do
> ashamed assured on related offence at equally totally. Use mile her whom
> they its. Kept hold an want as he bred of. Was dashwood landlord cheerful
> husbands two. Estate why theirs indeed him polite old settle though she. In
> as at regard easily narrow roused adieus.From a Lorem Ipsum passage, and
> going through the cites of the word in classical literature, discovered the
> undoubtable source. a quis enim. Donec pede justo, fringilla vel, aliquet
> nec, arcu. In enim justo, rhoncus ut, imperdiet a, venenatis vitae, justo.
>
>
>
> --
> This message was sent by Atlassian Jira
> (v8.3.4#803005)
>

[jira] [Created] (KAFKA-10465) Potential Bug/Doc update in Transactional Producer and Isolation Level

[M. Manna (Jira)]

M. Manna created KAFKA-10465:


 Summary: Potential Bug/Doc update in Transactional Producer and 
Isolation Level
 Key: KAFKA-10465
 URL: https://issues.apache.org/jira/browse/KAFKA-10465
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 2.4.1
Reporter: M. Manna
 Attachments: ConsumerTestApp.java, Consumer_endOffsets_return_84.jpg, 
ProducerTestApp.java, Producer_Committed_Successfully_to_82.jpg

*Issue*

Difference between LSO and High Watermark offsets when a consumer with 
"read_committed" aren't probably explained in the correct place.

*Expected Behaviour*
According to documentation, the offset returned should be the one committed 
last (and successfully). 

*Observed (with steps)*

1. Start a local or test kafka cluster (2.4.1 or above)
2. Create a topic (I have used 3 replication-factor with 1 partition, but 1 and 
1 is good)
3. Use the attached producer app file and set debug pointer to be able to pause 
on print
4. Use the attached consumer app file to start a consumer and debug through 
steps.

It can be seen that the consumer is actually able to fetch an offset that's not 
committed by the producer yet. 

Just trying to raise this ticket to confirm whether:

1) this is well-documented anywhere (which I have missed) - Please refer to 
this documentation as a resolution

2) This is a bug - please confirm and provide a timeline when this can be fixed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Broker side round robin on topic partitions when receiving messages

[M. Manna]

Vinicius,

If you have an idea of what you want , and how it is going to improve
current Kafka offering perhaps you could submit a KiP ?

The guidelines are provided on Apache Kafka Wiki (Confluence).

Thanks,


On Sun, 14 Jun 2020 at 22:26, Vinicius Scheidegger <
vinicius.scheideg...@gmail.com> wrote:

> Hi Collin,
>
> Thanks for the reply. Actually the RoundRobinPartitioner won't do an equal
> distribution when working with multiple producers. One producer does not
> know the others. If you consider that producers are randomly producing
> messages, in the worst case scenario all producers can be synced and one
> could have as many messages in a single partition as the number of
> producers.
> It's easy to generate evidences of it.
>
> I have asked this question on the users mail list too (and on Slack and on
> Stackoverflow).
>
> Kafka currently does not have means to do a round robin across multiple
> producers or on the broker side.
>
> This means there is currently NO GUARANTEE of equal distribution across
> partitions as the partition election is decided by the producer.
>
> There result is an unbalanced consumption when working with consumer groups
> and the options are: creating a custom shared partitioner, relying on Kafka
> random partition or introducing a middle man between topics (all of them
> having big cons).
>
> I thought of asking here to see whether this is a topic that could concern
> other developers (and maybe understand whether this could be a KIP
> discussion)
>
> Maybe I'm missing something... I would like to know.
>
> According to my interpretation of the code (just read through some
> classes), but there is currently no way to do partition balancing on the
> broker - the producer sends messages directly to partition leaders so
> partition currently needs to be defined on the producer.
>
> I understand that in order to perform round robin across partitions of a
> topic when working with multiple producers, some development needs to be
> done. Am I right?
>
>
> Thanks
>
>
> On Fri, Jun 12, 2020, 10:57 PM Colin McCabe  wrote:
>
> > HI Vinicius,
> >
> > This question seems like a better fit for the user mailing list rather
> > than the developer mailing list.
> >
> > Anyway, if I understand correctly, you are asking if the producer can
> > choose to assign partitions in a round-robin fashion rather than based on
> > the key.  The answer is, you can, by using RoundRobinPartitioner. (again,
> > if I'm understanding the question correctly).
> >
> > best,
> > Colin
> >
> > On Tue, Jun 9, 2020, at 00:48, Vinicius Scheidegger wrote:
> > > Anyone?
> > >
> > > On Fri, Jun 5, 2020 at 2:42 PM Vinicius Scheidegger <
> > > vinicius.scheideg...@gmail.com> wrote:
> > >
> > > > Does anyone know how could I perform a load balance to distribute
> > equally
> > > > the messages to all consumers within the same consumer group having
> > > > multiple producers?
> > > >
> > > > Is this a conceptual flaw on Kafka, wasn't it thought for equal
> > > > distribution with multiple producers or am I missing something?
> > > > I've asked on Stack Overflow, on Kafka users mailing group, here (on
> > Kafka
> > > > Devs) and on Slack - and still have no definitive answer (actually
> > most of
> > > > the time I got no answer at all)
> > > >
> > > > Would something like this even be possible in the way Kafka is
> > currently
> > > > designed?
> > > > How does proposing for a KIP work?
> > > >
> > > > Thanks,
> > > >
> > > >
> > > >
> > > > On Thu, May 28, 2020, 3:44 PM Vinicius Scheidegger <
> > > > vinicius.scheideg...@gmail.com> wrote:
> > > >
> > > >> Hi,
> > > >>
> > > >> I'm trying to understand a little bit more about how Kafka works.
> > > >> I have a design with multiple producers writing to a single topic
> and
> > > >> multiple consumers in a single Consumer Group consuming message from
> > this
> > > >> topic.
> > > >>
> > > >> My idea is to distribute the messages from all producers equally.
> From
> > > >> reading the documentation I understood that the partition is always
> > > >> selected by the producer. Is that correct?
> > > >>
> > > >> I'd also like to know if there is an out of the box option to assign
> > the
> > > >> partition via a round robin *on the broker side *to guarantee equal
> > > >> distribution of the load - if possible to each consumer, but if not
> > > >> possible, at least to each partition.
> > > >>
> > > >> If my understanding is correct, it looks like in a multiple producer
> > > >> scenario there is lack of support from Kafka regarding load
> balancing
> > and
> > > >> customers have to either stick to the hash of the key (random
> > distribution,
> > > >> although it would guarantee same key goes to the same partition) or
> > they
> > > >> have to create their own logic on the producer side (i.e. by sharing
> > memory)
> > > >>
> > > >> Am I missing something?
> > > >>
> > > >> Thank you,
> > > >>
> > > >> Vinicius Scheidegger
> > > >>
> > > >
> > >
> >
>

Re: [VOTE] KIP-579: new exception on min.insync.replicas > replication.factor

[M. Manna]

+1 (binding).

Thanks for the KIP.

On Tue, 31 Mar 2020 at 14:17, Paolo Moriello 
wrote:

> Hello,
>
> Thanks to everybody who has given feedback. I've incorporated the
> suggestions and think that this is now ready for a vote.
>
> KIP 579:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-579%3A+new+exception+on+min.insync.replicas+%3E+replication.factor
>
> PR:
> https://github.com/apache/kafka/pull/8225
>
> Thanks,
> Paolo
>

Re: [DISCUSS] KIP-579: new exception on min.insync.replicas > replication.factor

[M. Manna]

Hey Paolo,

Overall LGTM. I only have one suggestion.

You are planning to call it as "INCONSISTENT_REPLICA_CONFIGURATION".

How about if we call it "INCONSISTENT_REPLICATION_FACTOR"? Replica
configuration might also mean that one of the configuration is not
consistent. But with "INCONSISTENT_REPLICATION_FACTOR" it's semantically
closer to your KIP motivation. Also, users will likely and inherently,
discover that it's ISR and replication factor that might've been wrong for
them.

Perhaps, someone else may have different ideas, but please feel free to
consider it.

Regards,
M. MAnna

On Mon, 30 Mar 2020 at 12:55, Paolo Moriello 
wrote:

> Hi,
>
> Any other feedback on this before we start the vote?
>
> Paolo
>
> On Fri, 13 Mar 2020 at 17:28, Paolo Moriello 
> wrote:
>
> > Hi Mickael,
> >
> > Thanks for your interest in this. The main motivation to NOT make topic
> > creation fail when this mismatch happens is because at the moment it is
> > possible to produce/consume on topics if acks is not set to all. I'm not
> > sure we want to disable this behavior (as we would by failing at topic
> > creation). That's why I decided to go for a softer approach, which at
> least
> > gives some more clarity to the users and avoids other issues mentioned in
> > the KIP.
> >
> > Let's see what others think!
> >
> > On Fri, 13 Mar 2020 at 17:16, Mickael Maison 
> > wrote:
> >
> >> Hi Paolo,
> >>
> >> Thanks for looking at this issue. This can indeed be a source of
> >> confusion.
> >>
> >> I'm wondering if we should prevent the creation of topics with
> >> min.insync.replicas > replication.factor?
> >> You listed that as a rejected alternative because it requires more
> >> changes. However, I can't think of any scenarios where a user would
> >> want to create such a topic. I'm guessing it's probably always by
> >> mistake.
> >>
> >> Let's see what other people think but I think it's worth checking what
> >> needs to be done if we wanted to prevent topics with bogus configs
> >>
> >> On Fri, Mar 13, 2020 at 3:28 PM Paolo Moriello
> >>  wrote:
> >> >
> >> > Hi,
> >> >
> >> > Following this Jira ticket (
> >> https://issues.apache.org/jira/browse/KAFKA-4680),
> >> > I've created a proposal (
> >> >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-579%3A+new+exception+on+min.insync.replicas+%3E+replication.factor
> >> )
> >> > to add a new exception/error to be used on min.insync.replicas >
> >> > replication.factor.
> >> >
> >> > The proposal aims to introduce a new exception specific for the
> >> > configuration mismatch above to be used when producers requires acks =
> >> all.
> >> > At the moment we are using NotEnoughReplicaException, which is a
> >> retriable
> >> > exception and is used to fail on insync replicas < min isr. Plan is to
> >> have
> >> > a new, non-retriable exception, to separate the two cases.
> >> >
> >> > I've also submitted a PR for the change mentioned above:
> >> > https://github.com/apache/kafka/pull/8225
> >> >
> >> > Please have a look and let me know what you think.
> >> >
> >> > Thanks,
> >> > Paolo
> >>
> >
>

Re: How to change/increase ISR

[M. Manna]

Hey Upendra,

https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools

The above should guide you through the reassignment of partitions/replicas.

Also, you should read about

offset.topic.num.partitions
offset.topic.replication.factor

I hope this helps you.

Regards,



On Thu, 30 Jan 2020 at 21:48, Upendra Yadav  wrote:

> Hi Team,
>
> Is there way to change ISR for existing topics.
> I want this for user topics as well as for __consumer_offset topic.
>
> By mistake, __consumer_offset topic was configured with 1 replication
> factor and 1 ISR.
>
> kafka broker and client version: 0.10.0.1
>
> Thanks,
> Upendra
>

Re: [DISCUSS] KIP-566: Add rebalance callbacks to ConsumerInterceptor

[M. Manna]

Hey Thomas,

On Thu, 23 Jan 2020 at 21:17, Thomas Becker  wrote:

> Hi folks,
> I'd like to open the discussion for KIP-566: Add rebalance callbacks to
> ConsumerInterceptor. We've been looking to implement some custom metrics
> via ConsumerInterceptor, and not knowing when partition ownership changes
> is a significant impediment. I'd appreciate your thoughts.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-566%3A+Add+rebalance+callbacks+to+ConsumerInterceptor
>
>
>
>  I had a quick read through the KIP. I don't see any obvious issues.
Sounds like a simple improvement. Perhaps, you could add your thoughts
about RebalanceListener API in the future e.g. when to unify the
functionality. If implemented, we can simply use one implementation for
both things.

I would be interested to hear others' comments about this.

Thanks,


>
> 
>
> This email and any attachments may contain confidential and privileged
> material for the sole use of the intended recipient. Any review, copying,
> or distribution of this email (or any attachments) by others is prohibited.
> If you are not the intended recipient, please contact the sender
> immediately and permanently delete this email and any attachments. No
> employee or agent of TiVo is authorized to conclude any binding agreement
> on behalf of TiVo by email. Binding agreements with TiVo may only be made
> by a signed written agreement.
>

Re: [VOTE] KIP-553: Disable all SSL protocols except TLSV1.2 by default.

[M. Manna]

+1 (binding). A simple, and yet powerful enforcement of TLS version.

Thanks for this KIP :)

On Tue, 21 Jan 2020 at 20:39, Mickael Maison 
wrote:

> +1 (binding)
> Thanks
>
> On Tue, Jan 21, 2020 at 7:58 PM Ron Dagostino  wrote:
> >
> > +1 (non-binding)
> >
> > Ron
> >
> > On Tue, Jan 21, 2020 at 11:29 AM Manikumar 
> wrote:
> > >
> > > +1 (binding).
> > >
> > > Thanks for the KIP.
> > >
> > >
> > > On Tue, Jan 21, 2020 at 9:56 PM Ted Yu  wrote:
> > >
> > > > +1
> > > >
> > > > On Tue, Jan 21, 2020 at 8:24 AM Rajini Sivaram <
> rajinisiva...@gmail.com>
> > > > wrote:
> > > >
> > > > > +1 (binding)
> > > > >
> > > > > Thanks for the KIP!
> > > > >
> > > > > Regards,
> > > > >
> > > > > Rajini
> > > > >
> > > > >
> > > > > On Tue, Jan 21, 2020 at 3:43 PM Николай Ижиков <
> nizhi...@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hello.
> > > > > >
> > > > > > I would like to start vote for KIP-553: Disable all SSL protocols
> > > > except
> > > > > > TLSV1.2 by default.
> > > > > >
> > > > > > KIP -
> > > > > >
> > > > >
> > > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=142641956
> > > > > > Discussion thread -
> > > > > >
> > > > >
> > > >
> https://lists.apache.org/thread.html/9c6201fe403a24f84fc3aa27f47dd06b718c1d80de0ee3412b9b877c%40%3Cdev.kafka.apache.org%3E
> > > > >
> > > >
>

Re: [VOTE] KIP-409: Allow creating under-replicated topics and partitions

[M. Manna]

MIckael,



On Thu, 16 Jan 2020 at 14:01, Mickael Maison 
wrote:

> Hi Manna,
>
> In your example, the topic 'dummy' is not under replicated. It just
> has 1 replica. A topic under replicated is a topic with less ISRs than
> replicas.
>
> Having under replicated topics is relatively common in a Kafka
> cluster, it happens everytime is broker is down. However Kafka does
> not permit it to happen at topic creation. Currently at creation,
> Kafka requires to have at least as many brokers as the replication
> factor. This KIP addresses this limitation.
>
> Regarding your 2nd point. When rack awareness is enabled, Kafka tries
> to distribute partitions across racks. When all brokers in a rack are
> down (ie: a zone is offline), you can end up with partitions not well
> distributed even with rack awareness. There are currently no easy way
> to track such partitions so I decided to not attempt addressing this
> issue in this KIP.
>
> I hope that answers your questions.
>

 It does and I appreciate you taking time and explaining this.

 +1 (binding) if I haven't already.

>
>
>
> On Wed, Jan 15, 2020 at 4:10 PM Kamal Chandraprakash
>  wrote:
> >
> > +1 (non-binding). Thanks for the KIP!
> >
> > On Mon, Jan 13, 2020 at 1:58 PM M. Manna  wrote:
> >
> > > Hi Mikael,
> > >
> > > Apologies for last minute question, as I just caught up with it.
> Thanks for
> > > your work on the KIP.
> > >
> > > Just trying to get your thoughts on one thing (I might have
> misunderstood
> > > it) - currently it's possible (even though I am strongly against it) to
> > > create Kafka topics which are under-replicated; despite all brokers
> being
> > > online. This the the output of an intentionally under-replicated topic
> > > "dummy" with p=6 and RF=1 (with a 3 node cluster)
> > >
> > >
> > > virtualadmin@kafka-broker-machine-1:/opt/kafka/bin$ ./kafka-topics.sh
> > > --create --topic dummy --partitions 6 --replication-factor 1
> > > --bootstrap-server localhost:9092
> > > virtualadmin@kafka-broker-machine-1:/opt/kafka/bin$ ./kafka-topics.sh
> > > --describe --topic dummy  --bootstrap-server localhost:9092
> > > Topic:dummy PartitionCount:6ReplicationFactor:1
> > >
> > >
> Configs:compression.type=gzip,min.insync.replicas=2,cleanup.policy=delete,segment.bytes=10485760,max.message.bytes=10642642,retention.bytes=20971520
> > > Topic: dummyPartition: 0Leader: 3   Replicas: 3
> > > Isr: 3
> > > Topic: dummyPartition: 1Leader: 1   Replicas: 1
> > > Isr: 1
> > > Topic: dummyPartition: 2Leader: 2   Replicas: 2
> > > Isr: 2
> > > Topic: dummyPartition: 3Leader: 3   Replicas: 3
> > > Isr: 3
> > > Topic: dummyPartition: 4Leader: 1   Replicas: 1
> > > Isr: 1
> > > Topic: dummyPartition: 5Leader: 2   Replicas: 2
> > > Isr: 2
> > >
> > >  This is with respect to the following statement on your KIP (i.e.
> > > under-replicated topic creation is also permitted when none is
> offline):
> > >
> > > *but note that this may already happen (without this KIP) when
> > > > topics/partitions are created while all brokers in a rack are offline
> > > (ie:
> > > > an availability zone is offline). Tracking topics/partitions not
> > > optimally
> > > > spread across all racks can be tackled in a follow up KIP.  *
> > >
> > >
> > >
> > >
> > > Did you mean to say that such under-replicated topics (including
> > > human-created ones) will be handled in a separete KIP ?
> > >
> > > Regards,
> > >
> > >
> > > On Mon, 13 Jan 2020 at 10:15, Mickael Maison  >
> > > wrote:
> > >
> > > > Hi all.
> > > >
> > > > With 2.5.0 approaching, bumping this thread once more as feedback or
> > > > votes would be nice.
> > > >
> > > > Thanks
> > > >
> > > > On Wed, Dec 18, 2019 at 1:59 PM Tom Bentley 
> wrote:
> > > > >
> > > > > +1 non-binding. Thanks!
> > > > >
> > > > > On Wed, Dec 18, 2019 at 1:05 PM Sönke Liebau
> > > > >  wrote:
> > > > >
> > > > > > Hi Mickael,
> > > > > >
> > > > > > thanks for your response! That all makes perfect sense and I
> ca

Re: [VOTE] KIP-399: Extend ProductionExceptionHandler to cover serialization exceptions

[M. Manna]

+1 (non-binding)

Thanks for this KIP

Regards,

On Wed, 15 Jan 2020 at 20:35, Mitchell  wrote:

> +1(non-binding)
>
> Very useful
> -mitch
>
> On Wed, Jan 15, 2020, 3:29 PM Anna McDonald 
> wrote:
>
> > Greetings,
> > I would like to propose a vote on KIP-399, extending the
> > ProductionExceptionHandler to cover serialization exceptions. This KIP
> > is aimed at improving the error-handling semantics in Kafka Streams
> > when Kafka Streams fails to serialize a message to the downstream
> > sink.
> >
> > KIP details located here:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-399%3A+Extend+ProductionExceptionHandler+to+cover+serialization+exceptions
> >
> > Discussion Thread:
> >
> >
> https://lists.apache.org/thread.html/rbbc887ca31d46f6e73ffc6e08df7e4bda69c89ff820986c30274e272%40%3Cdev.kafka.apache.org%3E
> >
> > Thanks,
> > anna
> >
>

Re: [VOTE] KIP-555: An admin tools proposal to accomplish the deprecation of zookeeper access that is direct

[M. Manna]

+1 (Binding) - a long awaited KIP to have a simpler partition reassignment
script (without ZK)>

Kudos to you Colin :)

On Wed, 15 Jan 2020 at 10:10, Viktor Somogyi-Vass 
wrote:

> +1 (non-binding)
>
> Viktor
>
> On Wed, Jan 15, 2020 at 10:27 AM Manikumar 
> wrote:
>
> > +1 (binding)
> >
> > Thanks for the KIP.
> >
> > On Wed, Jan 15, 2020 at 2:48 AM Gwen Shapira  wrote:
> >
> > > +1 (binding, re-vote)
> > >
> > > On Tue, Jan 14, 2020 at 11:23 AM Colin McCabe 
> > wrote:
> > > >
> > > > Hi all,
> > > >
> > > > I'm reposting this since I've been informed that gmail mashed the
> > > original VOTE thread into a different email thread.  Hopefully the
> > > different thread title will prevent it doing that in this case.
> > > >
> > > > I'd like to start the vote on KIP-555: Deprecate Direct Zookeeper
> > access
> > > in Kafka Administrative Tools.
> > > >
> > > > KIP:  https://cwiki.apache.org/confluence/x/Wg6dC
> > > >
> > > > Discussion thread:
> > >
> >
> https://lists.apache.org/thread.html/ra0e4338c596d037c406b52a719bf13f775b03797cd5ca8d03d7f71c4%40%3Cdev.kafka.apache.org%3E
> > > >
> > > > cheers,
> > > > Colin
> > >
> >
>

Re: [VOTE] KIP-409: Allow creating under-replicated topics and partitions

[M. Manna]

Hi Mikael,

Apologies for last minute question, as I just caught up with it. Thanks for
your work on the KIP.

Just trying to get your thoughts on one thing (I might have misunderstood
it) - currently it's possible (even though I am strongly against it) to
create Kafka topics which are under-replicated; despite all brokers being
online. This the the output of an intentionally under-replicated topic
"dummy" with p=6 and RF=1 (with a 3 node cluster)


virtualadmin@kafka-broker-machine-1:/opt/kafka/bin$ ./kafka-topics.sh
--create --topic dummy --partitions 6 --replication-factor 1
--bootstrap-server localhost:9092
virtualadmin@kafka-broker-machine-1:/opt/kafka/bin$ ./kafka-topics.sh
--describe --topic dummy  --bootstrap-server localhost:9092
Topic:dummy PartitionCount:6ReplicationFactor:1
Configs:compression.type=gzip,min.insync.replicas=2,cleanup.policy=delete,segment.bytes=10485760,max.message.bytes=10642642,retention.bytes=20971520
Topic: dummyPartition: 0Leader: 3   Replicas: 3
Isr: 3
Topic: dummyPartition: 1Leader: 1   Replicas: 1
Isr: 1
Topic: dummyPartition: 2Leader: 2   Replicas: 2
Isr: 2
Topic: dummyPartition: 3Leader: 3   Replicas: 3
Isr: 3
Topic: dummyPartition: 4Leader: 1   Replicas: 1
Isr: 1
Topic: dummyPartition: 5Leader: 2   Replicas: 2
Isr: 2

 This is with respect to the following statement on your KIP (i.e.
under-replicated topic creation is also permitted when none is offline):

*but note that this may already happen (without this KIP) when
> topics/partitions are created while all brokers in a rack are offline (ie:
> an availability zone is offline). Tracking topics/partitions not optimally
> spread across all racks can be tackled in a follow up KIP.  *




Did you mean to say that such under-replicated topics (including
human-created ones) will be handled in a separete KIP ?

Regards,


On Mon, 13 Jan 2020 at 10:15, Mickael Maison 
wrote:

> Hi all.
>
> With 2.5.0 approaching, bumping this thread once more as feedback or
> votes would be nice.
>
> Thanks
>
> On Wed, Dec 18, 2019 at 1:59 PM Tom Bentley  wrote:
> >
> > +1 non-binding. Thanks!
> >
> > On Wed, Dec 18, 2019 at 1:05 PM Sönke Liebau
> >  wrote:
> >
> > > Hi Mickael,
> > >
> > > thanks for your response! That all makes perfect sense and I cannot
> > > give any actual use cases for where what I asked about would be useful
> > > :)
> > > It was more the idle thought if this might be low hanging fruit while
> > > changing this anyway to avoid having to circle back later on and
> > > wanted to at least mention it.
> > >
> > > I am totally happy either way!
> > >
> > > Best regards,
> > > Sönke
> > >
> > > On Wed, 18 Dec 2019 at 11:20, Mickael Maison  >
> > > wrote:
> > > >
> > > > Thanks Sönke for the feedback.
> > > >
> > > > I debated this point quite a bit before deciding to base creation
> > > > around "min.insync.replicas".
> > > >
> > > > For me, the goal of this KIP is to enable administrators to provide
> > > > higher availability. In a 3 node cluster configured for high
> > > > availability (3 replicas, 2 min ISR), by enabling this feature,
> > > > clusters should be fully usable even when 1 broker is down. This
> > > > should cover all "normal" maintenance operations like a rolling
> > > > restart or just the recovery of a broker.
> > > >
> > > > At the moment, when creating a topic/partition, the assumption is
> that
> > > > the resource will be fully functioning. This KIP does not change this
> > > > assumption. If this is something someone wants, I think it should be
> > > > handled in a different KIP that targets that use case. By relying on
> > > > "min.insync.replicas", we don't break any assumptions the user has
> and
> > > > this should be fully transparent from the user point of view.
> > > >
> > > > About "min.insync.replicas", one caveat that is not explicit in the
> > > > KIP is that it's currently possible to create topics with less
> > > > replicas than this settings. For that reason, I think the
> > > > implementation will actually rely on min(replicas, min-isr) instead
> of
> > > > simply min.insync.replicas. I have updated the KIP to explicitly
> > > > mention this point.
> > > >
> > > > I hope that answers your question, let me know.
> > > > Thanks
> > > >
> > > >
> > > > On Mon, Dec 16, 2019 at 4:38 PM Sönke Liebau
> > > >  wrote:
> > > > >
> > > > > Hi Michael,
> > > > >
> > > > > that sounds like a useful addition! I can't help but wonder
> whether by
> > > > > leaving in the restriction that "min.insync.replicas" has to be
> > > > > satisfied we'll be back here in a years time because someone has a
> > > > > scenario where he or she wants to go below that :)
> > > > > I don't have a strong opinion either way to be honest, just a
> random
> > > > > thought when reading the KIP.
> > > > >
> > > > > Best regards,
> > > > > Sönke
> > > > >
> > > > > On Thu, 12 Dec 2019 at 22:4

Re: [VOTE] KIP-551: Expose disk read and write metrics

[M. Manna]

Hey Colin,

+1 - Really useful for folks managing a cluster by themselves.

M. MAnna

On Fri, 10 Jan 2020 at 22:35, Jose Garcia Sancio 
wrote:

> +1, LGTM.
>
> On Fri, Jan 10, 2020 at 2:19 PM Gwen Shapira  wrote:
>
> > +1, thanks for driving this
> >
> > On Fri, Jan 10, 2020 at 2:17 PM Colin McCabe  wrote:
> > >
> > > Hi all,
> > >
> > > I'd like to start the vote on KIP-551: Expose disk read and write
> > metrics.
> > >
> > > KIP:  https://cwiki.apache.org/confluence/x/sotSC
> > >
> > > Discussion thread:
> > >
> >
> https://lists.apache.org/thread.html/cfaac4426455406abe890464a7f4ae23a5c69a39afde66fe6eb3d696%40%3Cdev.kafka.apache.org%3E
> > >
> > > cheers,
> > > Colin
> >
>
>
> --
> -Jose
>

[jira] [Created] (KAFKA-9363) Admin Script "kafka-topics" doesn't confirm on successful creation

[M. Manna (Jira)]

M. Manna created KAFKA-9363:
---

 Summary: Admin Script "kafka-topics" doesn't confirm on successful 
creation
 Key: KAFKA-9363
 URL: https://issues.apache.org/jira/browse/KAFKA-9363
 Project: Kafka
  Issue Type: Bug
  Components: admin
Affects Versions: 2.4.0
        Reporter: M. Manna
        Assignee: M. Manna


When a topic is created from admin console, no confirmation is provided if 
--bootstrap-server is chosen. 

 

How to reproduce:

1) Get 2.4.0 distro

2) Download and extract code

3) Run "kafka-topics --create --topic "dummy" --partition 1 
--replication-factor 1 --bootstrap-server localhost:9092

4) Observe that no confirmation e.g. "Successfully created dummy" was provided.


We should, at least, provide a confirmation or restore the confirmation message 
which was annunciated before using --zookeeper argument. We all must use 
--describe flag to do a follow-up, but a confirmation message is a nice 
addition.

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-531: Drop support for Scala 2.11 in Kafka 2.5

[M. Manna]

+1 (non-binding).

Thanks,

On Tue, 17 Dec 2019 at 22:14, Ewen Cheslack-Postava 
wrote:

> +1 (binding), though there is definitely still some compatibility risk --
> regardless of updates, people are going to have apps that may need other
> updates but upgrading between scala versions is still too painful for too
> long to warrant the effort to update. But in that case they probably aren't
> updating Kafka libs either.
>
> One thing to consider would be also including a change so we stop relying
> on publicly exposing the scala versions. Easy initial step would be to add
> a shim module that just targets our default scala version as a dependency
> so users (especially those that don't care about scala otherwise) don't
> even need to think or know about scala versions. Larger step would be to
> eventually remove different scala versions as an option, but that probably
> requires documenting how to shade if you're using the core module for
> things like integration tests (possibly even providing a pre-shaded version
> for convenience). The latter is more than we probably want to do here and
> probably has to actually wait for major version, but former might be
> doable.
>
> -Ewen
>
> On Mon, Nov 18, 2019 at 9:06 AM Mickael Maison 
> wrote:
>
> > +1 (binding)
> > Thanks for the KIP
> >
> > On Mon, Nov 18, 2019 at 3:54 PM Sean Glover 
> > wrote:
> > >
> > > +1 (non-binding)
> > >
> > > Good idea.  It will streamline the Streams Scala DSL nicely.  It will
> > also
> > > affect 2.11 users of embedded-kafka.  Are there any other non-broker
> > > dependencies that could be affected?
> > >
> > > Sean
> > >
> > > On Mon, Nov 18, 2019 at 10:43 AM Ismael Juma 
> wrote:
> > >
> > > > Yes, everyone is encouraged to vote. Committer votes are binding, but
> > we
> > > > are interested in what the wider community thinks too.
> > > >
> > > > Ismael
> > > >
> > > > On Mon, Nov 18, 2019 at 5:40 AM Ivan Yurchenko <
> > ivan0yurche...@gmail.com>
> > > > wrote:
> > > >
> > > > > Do I understand correctly, that non-commiters can also vote,
> despite
> > > > their
> > > > > votes don't decide?
> > > > >
> > > > > If so, then +1 from me.
> > > > >
> > > > > Ivan
> > > > >
> > > > >
> > > > > On Mon, 18 Nov 2019 at 15:19, Ismael Juma 
> wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > People seemed supportive in general, so I'd like to start a vote
> on
> > > > > > KIP-531:
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-531%3A+Drop+support+for+Scala+2.11+in+Kafka+2.5
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > > Sean Glover
> > > Principal Engineer, Alpakka, Lightbend, Inc. 
> > > @seg1o , in/seanaglover
> > > 
> >
>

Re: Broker Interceptors

[M. Manna]

Hi Tom,

On Mon, 2 Dec 2019 at 09:41, Thomas Aley  wrote:

> Hi Kafka community,
>
> I am hoping to get some feedback and thoughts about broker interceptors.
>
> KIP-42 Added Producer and Consumer interceptors which have provided Kafka
> users the ability to collect client side metrics and trace the path of
> individual messages end-to-end.
>
> This KIP also mentioned "Adding message interceptor on the broker makes a
> lot of sense, and will add more detail to monitoring. However, the
> proposal is to do it later in a separate KIP".
>
> One of the motivations for leading with client interceptors was to gain
> experience and see how useable they are before tackling the server side
> implementation which would ultimately "allow us to have a more
> complete/detailed message monitoring".
>
> Broker interceptors could also provide more value than just more complete
> and detailed monitoring such as server side schema validation, so I am
> curious to learn if anyone in the community has progressed this work; has
> ideas about other potential server side interceptor uses or has actually
> implemented something similar.
>

 I personally feel that the cost here is the impact on performance. If I am
right, this interceptor is going to tap into nearly everything. If you have
strong guarantee (min.in.sync.replicas = N-1) then this may incur some
delay (and let's not forget inter broker comms protection by TLS config).
This may not be desirable for some systems. That said, it would be good to
know what others think about this.

Thanks,

>
> Regards,
>
> Tom Aley
> thomas.a...@ibm.com
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>
>

Re: Kafka cluster issues

[M. Manna]

Hi

On Thu, 28 Nov 2019 at 16:40, Uday Kiran A 
wrote:

> Hi Team,
>
>
>
> We have setup the open source kafka_2.11 setup on ubuntu with 6 nodes .
> we are facing issue with the subscribers(dotnet) .Messages are failed to
> deliver and messages are missing.
>
> Can you help us in verifying the configuration.
>
>
>

The above doesn't really tell us much about your problems. What did you do
and how did you arrive at the conclusion that messages are missing?

Thanks,

> Thanks and Regards,
> *Uday Kiran A*
> Asociate Architect  – Product Engineering
>
> *M: *+91 8143747274
> *[image: http://mystifly.com/images/loop.png]* 
>  [image:
> http://mystifly.com/images/facebook.png]
>   [image:
> http://mystifly.com/images/twitters.png] 
>   [image: http://mystifly.com/images/linkedin.png]
> 
>
>
> Disclaimer: The information contained in this e-mail is private &
> confidential and may also be legally privileged. The contents of this mail
> and any of its attachments are subject to change without notice. If you are
> not the intended recipient of this mail, please notify us, preferably by
> e-mail; and do not read, copy or disclose the contents of this message to
> anyone. Whilst we have taken reasonable precautions to ensure that any
> attachment to this e-mail has been swept for viruses, e-mail communications
> cannot be guaranteed to be secure or error free, as information can be
> corrupted, intercepted, lost or contain viruses. We do not accept liability
> for such matter or their consequences.
>

Re: Kafka SSH Tunnel Connection without editing hostfile

[M. Manna]

Hi
On Fri, 8 Nov 2019 at 11:13, Akshay Das  wrote:

> Hi Team,
>
> I'm trying to consume from a kafka cluster using java client, but the kafka
> server can only be accessed via jumphost/ssh tunnel. But even after
> creating ssh tunnel we are not able to read because once consumer fetches
> metadata it uses original hosts to connect to broker. Is it possible to
> stop this behaviour?  Also we don't want to edit the hosts file in the
> local machine.
>
> Thanks,
> Akshay Das
>

I think a similar issue was reported in SO a while back. In summary
metadata request OFF with correctly configured broker seems to be the
option (without /etc/host change).

https://stackoverflow.com/questions/45854168/consume-from-a-kafka-cluster-through-ssh-tunnel

Re: [VOTE] KIP-538: Add a metric tracking the number of open connections with a given SSL cipher type

[M. Manna]

+1 (non-bonding) it’s a helpful info.

Thanks for the KIP.

Regards,

On Tue, 22 Oct 2019 at 23:45, Jason Gustafson  wrote:

> +1 Thanks Colin.
>
> On Tue, Oct 22, 2019 at 12:42 AM Tom Bentley  wrote:
>
> > +1 (non-binding). Thanks!
> >
> > On Tue, Oct 22, 2019 at 1:03 AM Gwen Shapira  wrote:
> >
> > > +1
> > >
> > > Thanks for leading this :)
> > >
> > > On Mon, Oct 21, 2019 at 3:43 PM Colin McCabe 
> wrote:
> > > >
> > > > Hi all,
> > > >
> > > > I'd like to start the vote on KIP-538: Add a metric tracking the
> number
> > > of open connections with a given SSL cipher type.
> > > >
> > > > KIP: https://cwiki.apache.org/confluence/x/txPjBw
> > > >
> > > > Discussion thread:
> > >
> >
> https://lists.apache.org/thread.html/b1ee845ef3a3d4fe4d83da81d5444cf00e43840d3b2710487255e6eb@%3Cdev.kafka.apache.org%3E
> > > >
> > > > best,
> > > > Colin
> > >
> >
>

Re: [VOTE] KIP-537: Increase default zookeeper session timeout

[M. Manna]

+1 binding. This kip will possibly fix a lot of extra line of configuration.

Thanks,

On Tue, 22 Oct 2019 at 09:06, Stanislav Kozlovski 
wrote:

> Solid KIP. This also has the side-effect of bumping
> zookeeper.connection.timeout.ms to a more reasonable default (the 18s
> session timeout). Perhaps worth mentioning
>
> +1 (non-binding)
>
> On Tue, Oct 22, 2019 at 5:47 AM Ismael Juma  wrote:
>
> > +1 (binding)
> >
> > On Mon, Oct 21, 2019, 5:28 PM Jason Gustafson 
> wrote:
> >
> > > I'd like to start a vote for KIP-537:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-537%3A+Increase+default+zookeeper+session+timeout
> > > .
> > >
> > > +1 from me
> > >
> > > Thanks,
> > > Jason
> > >
> >
>
>
> --
> Best,
> Stanislav
>

Re: [DISCUSS] KIP-541: Create a fetch.max.bytes configuration for the broker

[M. Manna]

Thanks Collin.

I read this statement " Fetch request from replicas will also be affected
by the *fetch.max.bytes* limit."

Which made me think whether this was referring to replica fetcher byte
size. But thanks for clarifying.

Regards,

On Tue, 22 Oct 2019 at 00:26, Colin McCabe  wrote:

> On Mon, Oct 21, 2019, at 15:52, M. Manna wrote:
> > Hello Colin,
> >
> > The KIP looks concise. My comments are below.
> >
> > replica.fetch.max.bytes is relevant when there is replication involved,
> so
> > I am trying to understand how fetch.max.bytes for a broker will play a
> role
> > here. Apologies for any limited assumptions (always trying to catchup
> with
> > Kafka :).
>
> Hi M. Manna,
>
> Thanks for taking a look.
>
> replica.fetch.max.bytes is only used to control how big the fetches that
> the replicas make to other brokers are.  It does not act as an upper limit
> on the size of inbound fetches made by Kafka consumers.  It is only
> involved in the fetch requests that the broker itself initiates.
>
> >
> > Also, would you kindly suggest how (or if ) the traditional performance
> > tests are affected due to this change?
> > Regards,
> >
>
> There shouldn't be any effect at all, since the upper limit that we are
> setting is higher than the limit which the consumer sets by default.  The
> main goal here is to prevent clients from setting values which don't really
> make sense, not to find the optimum value.  The optimum value will depend
> somewhat on how fast the cluster's disks are, and other factors.
>
> best,
> Colin
>
> >
> > On Mon, 21 Oct 2019 at 22:57, Colin McCabe  wrote:
> >
> > > Hi all,
> > >
> > > I wrote a KIP about creating a fetch.max.bytes configuration for the
> > > broker.  Please take a look here:
> > > https://cwiki.apache.org/confluence/x/4g73Bw
> > >
> > > thanks,
> > > Colin
> > >
> >
>

Re: [DISCUSS] KIP-541: Create a fetch.max.bytes configuration for the broker

[M. Manna]

Hello Colin,

The KIP looks concise. My comments are below.

replica.fetch.max.bytes is relevant when there is replication involved, so
I am trying to understand how fetch.max.bytes for a broker will play a role
here. Apologies for any limited assumptions (always trying to catchup with
Kafka :).

Also, would you kindly suggest how (or if ) the traditional performance
tests are affected due to this change?
Regards,


On Mon, 21 Oct 2019 at 22:57, Colin McCabe  wrote:

> Hi all,
>
> I wrote a KIP about creating a fetch.max.bytes configuration for the
> broker.  Please take a look here:
> https://cwiki.apache.org/confluence/x/4g73Bw
>
> thanks,
> Colin
>

Re: How to stop partition rebalancing

[M. Manna]

Hi,

If you choose not to close consumer, but only poll as a background thread
it, the rebalance wouldn’t take place. This is only if you are sending
heartbeats (via Poll() ) within the timeout period. Also, if one of the pod
dies, rebalance has to happen. You’d want a fare share of work.

An example code is also provided on KafkaConsumer docs. You may want to
check that.

The other option is to use manual partition assignment, but that means you
need to manually assign and manage who gets the partition.

Thanks,


On Wed, 9 Oct 2019 at 20:21, Gopinath  wrote:

> Hi,
>
> I have 2 partition in a topic and 3 pods/instances of my microservice
> running in my k8s cluster. I wanted all 3 pods to pull message from 2
> partitions. (I'm using same group id for all 3 pods/instances).
>
> I have achieved this by closing consumer as soon as i pulled message from
> partition. kafkaconsumer.close(); in this way i see i could see 3 pods can
> able to pull message from this 2 partition.
>
> But  what i observed is, when ever i close suing kafkaconsumer.close(); i
> see rebalancing is happening in partition. I do not want rebalancing to
> happen as another pod is ready to pick message from this partition.
>
> Can you please suggest a way to stop rebalancing.
>

[jira] [Resolved] (KAFKA-4941) Better definition/introduction to the term brokers

[M. Manna (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

M. Manna resolved KAFKA-4941.
-
Resolution: Invalid

> Better definition/introduction to the term brokers
> --
>
> Key: KAFKA-4941
> URL: https://issues.apache.org/jira/browse/KAFKA-4941
> Project: Kafka
>  Issue Type: Improvement
>  Components: documentation
>Reporter: Yih Feng Low
>Priority: Trivial
>
> Hi,
> I just wanted to point out that in the documentation at 
> https://kafka.apache.org/documentation/, there are over 500 references to the 
> word "broker". However, the idea of what a broker is not clear. 
> The first mention of a Kafka broker comes from the sentence:
> ??Alternatively, instead of manually creating topics you can also configure 
> your brokers to auto-create topics when a non-existent topic is published 
> to.??
> Perhaps there could be a better discussion of what a broker is, similar to 
> the discussions on what a consumer/producer/partition etc. is?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-5712) Expose ProducerConfig in the KafkaProducer clients

[M. Manna (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

M. Manna resolved KAFKA-5712.
-
  Reviewer: Bill Bejeck
Resolution: Information Provided

> Expose ProducerConfig in the KafkaProducer clients
> --
>
> Key: KAFKA-5712
> URL: https://issues.apache.org/jira/browse/KAFKA-5712
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Reporter: Leo Xuzhang Lin
>Priority: Trivial
>
> There is no easy way to introspect into a Producer's configuration. 
> For important configurations such as `acks` and `retries`, it is useful to be 
> able to verify programmatically the value for those configurations. 
> Since the ProducerConfig object is read only, there seems to be no harm to 
> expose that in a getter.
> ProducerConfig value in KafkaProducer:
> https://github.com/apache/kafka/blob/0.11.0/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java#L246



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Kafka with kubernetes

[M. Manna]

Hi,

1) Assuming you are looking for helm chart - Confluent has it made
available here -  https://github.com/helm/charts/tree/master/incubator/kafka

2) Gwen Shapira published a very good white paper on Kafka with K8s -
https://www.confluent.io/resources/recommendations-for-deploying-apache-kafka-on-kubernetes?utm_source=blog&utm_medium=why-kafka-k8s&utm_campaign=co

3) "Preferred Architecture" - what did mean here exactly? IaaS provider
(e.g. Azure/AWS/GCP), or Platform specs ? Or, are you referring to
resource/platform setup (disk,RAM,prcoessor etc) ?

On Thu, 26 Sep 2019 at 03:52, MyLearningWay 
wrote:

> Hi Team,
>
> I am planning to implement Kafka and planning to deploy same in kubernetes.
>
> Could you please help me what should be preferred architecture to deploy on
> kubernetes clusters.
>
> Regards
> Shivam Shukla
>

Re: Kafka SSH Tunnel Connection without editing hostfile

[M. Manna]

why not try using internal vs external traffic

https://cwiki.apache.org/confluence/display/KAFKA/KIP-103%3A+Separation+of+Internal+and+External+traffic


if you set EXTERNAL enndpoints and map it to SSL - you clients should only
receive EXTERNAL endpoints for comms. Does this sound okay for you?

Thanks,

On Fri, 13 Sep 2019 at 06:41, Akshay Das  wrote:

> We cannot use external endpoints because of security reasons.
> Is there an option to tell zookeeper/broker not to send broker host detail
> metadata to its clients?
>
> On Thu, Sep 12, 2019 at 3:05 PM M. Manna  wrote:
>
>> Have you tried using EXTERNAL endpoints for your Kafka broker to separate
>> TLS from internal traffic? Also, have you checked zk admin whether the
>> broker metadata is exposing your TLS endpoints to clients ?
>>
>>
>> On Thu, 12 Sep 2019 at 10:23, Akshay Das 
>> wrote:
>>
>> > Hi Team,
>> >
>> > I'm trying to consume from a kafka cluster using java client, but the
>> kafka
>> > server can only be accessed via jumphost/ssh tunnel. But even after
>> > creating ssh tunnel we are not able to read because once conusmer
>> fetches
>> > metadata it uses original hosts to connect to broker. Is it possible to
>> > stop this behaviour?
>> >
>> > Thanks,
>> > Akshay Das
>> >
>>
>

Re: Kafka SSH Tunnel Connection without editing hostfile

[M. Manna]

Have you tried using EXTERNAL endpoints for your Kafka broker to separate
TLS from internal traffic? Also, have you checked zk admin whether the
broker metadata is exposing your TLS endpoints to clients ?


On Thu, 12 Sep 2019 at 10:23, Akshay Das 
wrote:

> Hi Team,
>
> I'm trying to consume from a kafka cluster using java client, but the kafka
> server can only be accessed via jumphost/ssh tunnel. But even after
> creating ssh tunnel we are not able to read because once conusmer fetches
> metadata it uses original hosts to connect to broker. Is it possible to
> stop this behaviour?
>
> Thanks,
> Akshay Das
>

Re: [VOTE] KIP-520: Augment Consumer.committed(partition) to allow multiple partitions

[M. Manna]

Gushing,

Good kip. +1 (binding) from me.

Thanks,
MAnna

On Thu, 12 Sep 2019 at 09:55, Bruno Cadonna  wrote:

> Guozhang, Thanks for the KIP.
>
> +1 (non-binding)
>
> Best,
> Bruno
>
> On Wed, Sep 11, 2019 at 9:17 AM Kamal Chandraprakash
>  wrote:
> >
> > Thanks for the KIP!
> >
> > LGTM, +1 (non-binding).
> >
> > On Wed, Sep 11, 2019 at 3:23 AM Matthias J. Sax 
> > wrote:
> >
> > > I don't have a strong preference. So I am also fine to deprecate the
> > > existing methods. Let's see what Jason thinks.
> > >
> > > Can you update the KIP to reflect the semantics of the return `Map`
> (ie,
> > > does only contain entries for partitions with committed offsets, and
> > > does not contain `null` values)?
> > >
> > >
> > > +1 (binding)
> > >
> > > -Matthias
> > >
> > >
> > >
> > >
> > > On 9/10/19 11:53 AM, Guozhang Wang wrote:
> > > > Hi Jason / Matthias,
> > > >
> > > > I understand your concerns now. Just to clarify my main motivation on
> > > > deprecating the old APIs is not only for aesthetics (confess I
> personally
> > > > do have preference on succinct APIs), but to urge people to use the
> > > batched
> > > > API for better latency when possible --- as stated in the KIP, my
> > > > observation is that in practice most callers are actually going to
> get
> > > > committed offsets for more than one partitions, and without
> deprecating
> > > the
> > > > old APIs it would be hard for them to realize that the new API does
> have
> > > a
> > > > benefit in performance.
> > > >
> > > > This is different from some of the existing APIs though -- e.g.,
> Matthias
> > > > mentioned about seek / seekToBeginning / seekToEnd, where only
> seekToXX
> > > > have plurals and seek only have singulars. We can, of course, make
> > > seekToXX
> > > > with plurals as well just like commitSync/Async, but since seeks are
> > > > non-blocking APIs (they rely on the follow-up polling call to talk
> to the
> > > > brokers) either calling it multiple times with one partition each
> v.s.
> > > > calling it one time with a plural makes little difference (still, I'd
> > > argue
> > > > that today we do not have a same-named function overloaded with both
> > > types
> > > > :P) On the other hand, committed, commitSync, offsetsForTimes etc
> > > blocking
> > > > calls are all in the form of plurals except
> > > >
> > > > * committed
> > > > * position
> > > > * partitionsFor
> > > >
> > > > My rationale was that 1) for consecutive calls of #position, mostly
> it
> > > > would only require a single round-trip to brokers since we are
> trying to
> > > > refresh fetching positions for all partitions anyways, and 2) for
> > > > #partitionsFor, theoretically we could also consider to ask for
> multiple
> > > > topics in one call since each blocking call potentially incurs one
> round
> > > > trip, but I did not include it in the scope of this KIP only because
> I
> > > have
> > > > not observed too many usage patterns that are commonly calling it
> > > > consecutively for multiple topics. At the moment, what I truly want
> to
> > > > "improve" on is the committed calls, as in many cases I've seen it
> being
> > > > called consecutively for multiple topic-partitions.
> > > >
> > > > Therefore, I'm still more inclined to deprecate the old APIs so that
> we
> > > can
> > > > enforce people to discover the new batching APIs for efficiency in
> this
> > > > KIP. But if you feel that this compatibility is very crucial to
> maintain
> > > I
> > > > could be convinced.
> > > >
> > > >
> > > > Guozhang
> > > >
> > > > On Tue, Sep 10, 2019 at 10:18 AM Matthias J. Sax <
> matth...@confluent.io>
> > > > wrote:
> > > >
> > > >> Thanks for the KIP Guozhang.
> > > >>
> > > >>> Another reason is that other functions of KafkaConsumers do not
> have
> > > >> those
> > > >>> overloaded functions to be consistent
> > > >>
> > > >> I tend to agree with Jason about keeping the existing methods. Your
> > > >> argument does not seem to hold. I just checked the `Consumer` API,
> and
> > > >> it's mix of overloads:
> > > >>
> > > >> Methods only talking `Collections`
> > > >>
> > > >>
> > > >>
> > >
> subscribe/assign/commitSync/commitAsyn/pause/resume/offsetsForTimes/beginningOffsets/endOffsets
> > > >>
> > > >> Method with overload taking `Collections` or as single value:
> > > >>
> > > >> seek/seekToBeginning/seekToEnd
> > > >>
> > > >> (those are strictly different methods, but they are semantically
> > > related)
> > > >>
> > > >> Only talking single value:
> > > >>
> > > >> position/committed/partitionsFor
> > > >>
> > > >>
> > > >> While you are strictly speaking correct, that there is no method
> with an
> > > >> overload for `Collection` and single value, the API mix seems to
> suggest
> > > >> that it might actually be worth to have corresponding overloads for
> all
> > > >> methods instead of sticking to `Collections` only.
> > > >>
> > > >>
> > > >>
> > > >> About the return map: I agree that not containing any entry in the
> map
> > > >> is better. I

Re: all tests passed but still got ‘Some checks were not successful’

[M. Manna]

Could you send a retest ? Add “Retest this please” comment. It’ll kick off.

On Sun, 18 Aug 2019 at 16:16, hong mao  wrote:

> Hi all,
> I submitted a PR and all testcases passed in Jenkins, but I still got a
> 'Some checks were not successful' tip. Could anybody give some advices?
> Here's the PR link https://github.com/apache/kafka/pull/7153
> [image: image.png]
>
> Thanks!
>

Re: [DISCUSS] KIP-487: Automatic Topic Creation on Producer

[M. Manna]

Hi,

If I may, perhaps you could simplify everything by using only
'auto.create.topics.enable' as a value along with true. In other words, the
public interfaces section should only have [true,auto.create.topics.enable,
false].

The reason for this is that auto.create.topics.enable is already known to
users as a "Server-SIde" config. So all you are saying is

a) To avoid day 1 impact, it will follow whatever auto.create.topics.enable
value is set.
b) False means - no client side topic creation
c) True means client side topic creation.

It saves creating 2 more new strings :). But not too expensive anyway.

Also, when you deprecate auto.create.topics.enable - you must provide
sufficient logic to ensure that things like rolling upgrade doesn't
temporarily break anything. I apologise if you have already accounted for
this, but wanted to mention since I didn't notice this on the KIP.

Let me know how this sounds.

Regards,

On Wed, 7 Aug 2019 at 19:10, Justine Olshan  wrote:

> Hi Harsha,
>
> I think my message may have gotten lost in all the others.
>
> Two of the goals of this KIP are to 1) allow auto-creation on specific
> clients when the broker default is false and 2) eventually replace the
> broker config.
>
> In order to accomplish these two goals, we need the producer to be able to
> create topics despite the broker config. (How can we replace a function
> when we rely on it?)
> I think at this point we have a fundamental disagreement in what we should
> allow the producer to do.
> In my previous message I mentioned a config that would allow for the broker
> to prevent producer auto-creation. (It would be disabled by default.) It
> would fix your issue for now, but could lead to more complications later.
>
> Thank you,
> Justine
>
>
> On Wed, Aug 7, 2019 at 10:56 AM Harsha Chintalapani 
> wrote:
>
> > On Wed, Aug 07, 2019 at 9:50 AM, Colin McCabe  wrote:
> >
> > > On Wed, Aug 7, 2019, at 09:24, Harsha Ch wrote:
> > >
> > > On Tue, Aug 06, 2019 at 11:46 PM, Colin McCabe < cmcc...@apache.org >
> > > wrote:
> > >
> > > On Tue, Aug 6, 2019, at 21:38, Harsha Ch wrote:
> > >
> > > Hi Colin,
> > > "Hmm... I'm not sure I follow. Users don't have to build their own
> > > tooling, right? They can use any of the shell scripts that we've
> shipped
> > in
> > > the last few releases. For example, if any of your users run it, this
> > shell
> > > script will delete all of the topics from your non-security-enabled
> > > cluster:
> > >
> > > ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh )
> > > --bootstrap-server localhost:9092 --list 2>/dev/null
> > > | xargs -l ./ bin/ kafka-topics. sh ( http://bin/kafka-topics.sh )
> > > --bootstrap-server localhost:9092 --delete
> > > --topic
> > >
> > > They will need to fill in the correct bootstrap servers list, of
> course,
> > > not localhost. This deletion script will work on some pretty old
> brokers,
> > > even back to the 0.10 releases. It seems a little odd to trust your
> users
> > > with this power, but not trust them to avoid changing a particular
> > > configuration key."
> > >
> > > The above will blocked by the server if we set delete.topic.enable to
> > > false and thats exactly what I am asking for.
> > >
> > > Hi Harsha,
> > >
> > > I was wondering if someone was going to bring up that configuration :)
> > >
> > > it's an interesting complication, but globally disabling topic deletion
> > is
> > > not very practical for most use-cases.
> > >
> > > In any case, there are plenty of other bad things that users with full
> > > permissions can do that aren't blocked by any server configuration. For
> > > example, they can delete every record in every topic. I can write a
> > script
> > > for that too, and there's no server configuration you can set to
> disable
> > > it. Or I could simply create hundreds of thousands of topics, until
> > cluster
> > > performance becomes unacceptable (this will be even more of a problem
> if
> > > someone configured delete.topic.enable as false). Or publish bad data
> to
> > > every topic, etc. etc.
> > >
> > > The point I'm trying to make here is that you can't rely on these kind
> of
> > > server-side configurations for security. At most, they're a way to set
> up
> > > certain very simple policies. But the policies are so simple that
> they're
> > > hardly ever useful any more.
> > >
> > > For example, if the problem you want to solve is that you want a user
> to
> > > only be able to create 50 topics and not delete anyone else's topics,
> you
> > > can solve that with a CreateTopicsPolicy that limits the number of
> > topics,
> > > and some ACLs. There's no combination of auto.create.topics.enable and
> > > delete.topic.enable that will help here.
> > >
> > > Hi Colin,
> > >
> > > Well you gave the example that a user can delete the topics
> > > just by running that script  :).
> > >
> > > I understand there are open APIs in Kafka and can lead to rogue clients
> > > taking advantage of it without proper securit

[jira] [Created] (KAFKA-8759) Message Order is reversed when client run behind a VPN

[M. Manna (JIRA)]

M. Manna created KAFKA-8759:
---

 Summary: Message Order is reversed when client run behind a VPN
 Key: KAFKA-8759
 URL: https://issues.apache.org/jira/browse/KAFKA-8759
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 2.2.0
Reporter: M. Manna


We have noticed this behaviour whilst testing console producer against a kafka 
service installed on GCP. We have been using a fork from confluent Helm Chart.

[https://github.com/helm/charts/tree/master/incubator/kafka]

FYI - we've used cp 5.3.0 with Apache Kafka 2.3.0

Our VPN connection throughput was 1 mbps. Upon connecting to VPN, we opened a 
console producer client (2.2.0) with the following command:


{code:java}
kafka-console-producer.bat --topic some_topic --broker-list 
gcp_broker1:19092,gcp_broker2:19092,gcp_broker3:19092{code}
 

Similarly, we ran a consumer with the following command before publishing 
messages
{code:java}
kafka-console-consumer.bat --topic some_topic --bootstrap-server 
gcp_broker1:19092,gcp_broker2:19092,gcp_broker3:19092{code}

For producer console, we did receive a carat (>) prompt for publishing, so we 
entered messages:
{code:java}
>one
>two
>three
>{code}
After a while, it responded with NETWORK_EXCEPTION


{code:java}
[2019-08-02 11:17:19,690] WARN [Producer clientId=console-producer] Got error 
produce response with correlation id 8 on topic-partition some_topic-0, 
retrying (2 attempts left). Error: NETWORK_EXCEPTION 
(org.apache.kafka.clients.producer.internals.Sender){code}

We then hit "Enter" and received a carat (>) back


{code:java}
[2019-08-02 11:17:19,690] WARN [Producer clientId=console-producer] Got error 
produce response with correlation id 8 on topic-partition some_topic-0, 
retrying (2 attempts left). Error: NETWORK_EXCEPTION 
(org.apache.kafka.clients.producer.internals.Sender)

>{code}
 

Immediately after that, on consumer window, we received the following:
{code:java}
three
two
one{code}
 

We ran the same exercise from a regular network (wifi/lan) and didn't see this 
issue (i.e. works as described on Quickstart). 

This is slightly concerning for us since tunneling into a VPN shouldn't have 
any impact (or, should it) how kafka message protocol works over tcp. It seems 
that Kafka couldn't guarantee order of messages when network latency is 
involved. 

FYI 
1) We tried on VPN with --request_timeout_ms 12 and still same results.
2) Our setup was 3 node (3 br, 3 zk) with every topic having 1 partition only 
(RF - 3).



 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: [VOTE] KIP-480 : Sticky Partitioner

[M. Manna]

+1(na)

On Sat, 13 Jul 2019 at 22:17, Stanislav Kozlovski 
wrote:

> +1 (non-binding)
>
> Thanks!
>
> On Fri, Jul 12, 2019 at 6:02 PM Gwen Shapira  wrote:
>
> > +1 (binding)
> >
> > Thank you for the KIP. This was long awaited.
> >
> > On Tue, Jul 9, 2019 at 5:15 PM Justine Olshan 
> > wrote:
> > >
> > > Hello all,
> > >
> > > I'd like to start the vote for KIP-480 : Sticky Partitioner.
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-480%3A+Sticky+Partitioner
> > >
> > > Thank you,
> > > Justine Olshan
> >
> >
> >
> > --
> > Gwen Shapira
> > Product Manager | Confluent
> > 650.450.2760 | @gwenshap
> > Follow us: Twitter | blog
> >
>
>
> --
> Best,
> Stanislav
>

Re: [DISCUSS] KIP-480 : Sticky Partitioner

[M. Manna]

Thanks for the comments Colin.

My only concern is that this KIP is addressing a good feature and having
that extended to RoundRobinPartitioner means 1 less KIP in the future.

Would it be appropriate to extend the support to RoundRobinPartitioner too?

Thanks,

On Tue, 9 Jul 2019 at 17:24, Colin McCabe  wrote:

> Hi M,
>
> The RoundRobinPartitioner added by KIP-369 doesn't interact with this
> KIP.  If you configure your producer to use RoundRobinPartitioner, then the
> DefaultPartitioner will not be used.  And the "sticky" behavior is
> implemented only in the DefaultPartitioner.
>
> regards,
> Colin
>
>
> On Tue, Jul 9, 2019, at 05:12, M. Manna wrote:
> > Hello Justine,
> >
> > I have one item I wanted to discuss.
> >
> > We are currently in review stage for KAFKA- where we can choose
> always
> > RoundRobin regardless of null/usable key.
> >
> > If I understood this KIP motivation correctly, you are still honouring
> how
> > the hashing of key works for DefaultPartitioner. Would you say that
> having
> > an always "Round-Robin" partitioning with "Sticky" assignment (efficient
> > batching of records for a partition) doesn't deviate from your original
> > intention?
> >
> > Thanks,
> >
> > On Tue, 9 Jul 2019 at 01:00, Justine Olshan 
> wrote:
> >
> > > Hello all,
> > >
> > > If there are no more comments or concerns, I would like to start the
> vote
> > > on this tomorrow afternoon.
> > >
> > > However, if there are still topics to discuss, feel free to bring them
> up
> > > now.
> > >
> > > Thank you,
> > > Justine
> > >
> > > On Tue, Jul 2, 2019 at 4:25 PM Justine Olshan 
> > > wrote:
> > >
> > > > Hello again,
> > > >
> > > > Another update to the interface has been made to the KIP.
> > > > Please let me know if you have any feedback!
> > > >
> > > > Thank you,
> > > > Justine
> > > >
> > > > On Fri, Jun 28, 2019 at 2:52 PM Justine Olshan  >
> > > > wrote:
> > > >
> > > >> Hi all,
> > > >> I made some changes to the KIP.
> > > >> The idea is to clean up the code, make behavior more explicit,
> provide
> > > >> more flexibility, and to keep default behavior the same.
> > > >>
> > > >> Now we will change the partition in onNewBatch, and specify the
> > > >> conditions for this function call (non-keyed values, no explicit
> > > >> partitions) in willCallOnNewBatch.
> > > >> This clears up some of the issues with the implementation. I'm
> happy to
> > > >> hear further opinions and discuss this change!
> > > >>
> > > >> Thank you,
> > > >> Justine
> > > >>
> > > >> On Thu, Jun 27, 2019 at 2:53 PM Colin McCabe 
> > > wrote:
> > > >>
> > > >>> On Thu, Jun 27, 2019, at 01:31, Ismael Juma wrote:
> > > >>> > Thanks for the KIP Justine. It looks pretty good. A few comments:
> > > >>> >
> > > >>> > 1. Should we favor partitions that are not under replicated?
> This is
> > > >>> > something that Netflix did too.
> > > >>>
> > > >>> This seems like it could lead to cascading failures, right?  If a
> > > >>> partition becomes under-replicated because there is too much
> traffic,
> > > the
> > > >>> producer stops sending to it, which puts even more load on the
> > > remaining
> > > >>> partitions, which are even more likely to fail then, etc.  It also
> will
> > > >>> create unbalanced load patterns on the consumers.
> > > >>>
> > > >>> >
> > > >>> > 2. If there's no measurable performance difference, I agree with
> > > >>> Stanislav
> > > >>> > that Optional would be better than Integer.
> > > >>> >
> > > >>> > 3. We should include the javadoc for the newly introduced method
> that
> > > >>> > specifies it and its parameters. In particular, it would good to
> > > >>> specify if
> > > >>> > it gets called when an explicit partition id has been provided.
> > > >>>
> > > >>> Agreed.
> > > >>>
> > > >>> best,
> > > >>> Colin
> > > >>>
> > > >>> >
> > > >>> > Ismael
> > > >>> >
> > > >>> > On Mon, Jun 24, 2019, 2:04 PM Justine Olshan <
> jols...@confluent.io>
> > > >>> wrote:
> > > >>> >
> > > >>> > > Hello,
> > > >>> > > This is the discussion thread for KIP-480: Sticky Partitioner.
> > > >>> > >
> > > >>> > >
> > > >>> > >
> > > >>>
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-480%3A+Sticky+Partitioner
> > > >>> > >
> > > >>> > > Thank you,
> > > >>> > > Justine Olshan
> > > >>> > >
> > > >>> >
> > > >>>
> > > >>
> > >
> >
>

Re: [DISCUSS] KIP-480 : Sticky Partitioner

[M. Manna]

Hello Justine,

I have one item I wanted to discuss.

We are currently in review stage for KAFKA- where we can choose always
RoundRobin regardless of null/usable key.

If I understood this KIP motivation correctly, you are still honouring how
the hashing of key works for DefaultPartitioner. Would you say that having
an always "Round-Robin" partitioning with "Sticky" assignment (efficient
batching of records for a partition) doesn't deviate from your original
intention?

Thanks,

On Tue, 9 Jul 2019 at 01:00, Justine Olshan  wrote:

> Hello all,
>
> If there are no more comments or concerns, I would like to start the vote
> on this tomorrow afternoon.
>
> However, if there are still topics to discuss, feel free to bring them up
> now.
>
> Thank you,
> Justine
>
> On Tue, Jul 2, 2019 at 4:25 PM Justine Olshan 
> wrote:
>
> > Hello again,
> >
> > Another update to the interface has been made to the KIP.
> > Please let me know if you have any feedback!
> >
> > Thank you,
> > Justine
> >
> > On Fri, Jun 28, 2019 at 2:52 PM Justine Olshan 
> > wrote:
> >
> >> Hi all,
> >> I made some changes to the KIP.
> >> The idea is to clean up the code, make behavior more explicit, provide
> >> more flexibility, and to keep default behavior the same.
> >>
> >> Now we will change the partition in onNewBatch, and specify the
> >> conditions for this function call (non-keyed values, no explicit
> >> partitions) in willCallOnNewBatch.
> >> This clears up some of the issues with the implementation. I'm happy to
> >> hear further opinions and discuss this change!
> >>
> >> Thank you,
> >> Justine
> >>
> >> On Thu, Jun 27, 2019 at 2:53 PM Colin McCabe 
> wrote:
> >>
> >>> On Thu, Jun 27, 2019, at 01:31, Ismael Juma wrote:
> >>> > Thanks for the KIP Justine. It looks pretty good. A few comments:
> >>> >
> >>> > 1. Should we favor partitions that are not under replicated? This is
> >>> > something that Netflix did too.
> >>>
> >>> This seems like it could lead to cascading failures, right?  If a
> >>> partition becomes under-replicated because there is too much traffic,
> the
> >>> producer stops sending to it, which puts even more load on the
> remaining
> >>> partitions, which are even more likely to fail then, etc.  It also will
> >>> create unbalanced load patterns on the consumers.
> >>>
> >>> >
> >>> > 2. If there's no measurable performance difference, I agree with
> >>> Stanislav
> >>> > that Optional would be better than Integer.
> >>> >
> >>> > 3. We should include the javadoc for the newly introduced method that
> >>> > specifies it and its parameters. In particular, it would good to
> >>> specify if
> >>> > it gets called when an explicit partition id has been provided.
> >>>
> >>> Agreed.
> >>>
> >>> best,
> >>> Colin
> >>>
> >>> >
> >>> > Ismael
> >>> >
> >>> > On Mon, Jun 24, 2019, 2:04 PM Justine Olshan 
> >>> wrote:
> >>> >
> >>> > > Hello,
> >>> > > This is the discussion thread for KIP-480: Sticky Partitioner.
> >>> > >
> >>> > >
> >>> > >
> >>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-480%3A+Sticky+Partitioner
> >>> > >
> >>> > > Thank you,
> >>> > > Justine Olshan
> >>> > >
> >>> >
> >>>
> >>
>

Re: PR review

[M. Manna]

Hello Colin,

I appreciate the time for reviewing this PR. I have now incorporated your
(and Matthias's) comments and the PR has been resubmitted. Please let me
know if there is more that we need to change.

Thanks again,

On Mon, 8 Jul 2019 at 23:50, Colin McCabe  wrote:

> Hi M. Manna,
>
> I left a review.  Take a look.
>
> Sorry for the delays.
>
> best,
> Colin
>
>
> On Mon, Jul 8, 2019, at 14:38, M. Manna wrote:
> > Hello,
> >
> > A few requests have been sent already. Could this please be reviewed ?
> Our
> > business implementation is holding due to this change.
> >
> >
> >
> > On Thu, 4 Jul 2019 at 13:33, M. Manna  wrote:
> >
> > > https://github.com/apache/kafka/pull/6771
> > >
> > > Could this be reviewed please ?
> > >
> > > On Wed, 3 Jul 2019 at 11:35, M. Manna  wrote:
> > >
> > >> https://github.com/apache/kafka/pull/6771
> > >>
> > >> Bouncing both users and dev to get some activity going. We are waiting
> > >> for a while to get this KIP pr merged.
> > >>
> > >> Could someone please review?
> > >>
> > >> Thanks,
> > >>
> > >> On Sun, 30 Jun 2019 at 08:59, M. Manna  wrote:
> > >>
> > >>> https://github.com/apache/kafka/pull/6771
> > >>>
> > >>> Hello,
> > >>>
> > >>> Could the above PR can be reviewed? This has been waiting for a long
> > >>> time.
> > >>>
> > >>> Just to mention, the package name should have "internal". Round-robin
> > >>> partitioning should have been supported without/without a key from
> the
> > >>> beginning. It provides user a guaranteed round-robin partitioning
> without
> > >>> having to regard for key values (e.g. null/not null). From our
> business
> > >>> side, this is a Kafka internal logic. Hence, the placement inside
> > >>> "internal" package.
> > >>>
> > >>> Thanks,
> > >>>
> > >>
> >
>

Re: PR review

[M. Manna]

Hello,

A few requests have been sent already. Could this please be reviewed ? Our
business implementation is holding due to this change.



On Thu, 4 Jul 2019 at 13:33, M. Manna  wrote:

> https://github.com/apache/kafka/pull/6771
>
> Could this be reviewed please ?
>
> On Wed, 3 Jul 2019 at 11:35, M. Manna  wrote:
>
>> https://github.com/apache/kafka/pull/6771
>>
>> Bouncing both users and dev to get some activity going. We are waiting
>> for a while to get this KIP pr merged.
>>
>> Could someone please review?
>>
>> Thanks,
>>
>> On Sun, 30 Jun 2019 at 08:59, M. Manna  wrote:
>>
>>> https://github.com/apache/kafka/pull/6771
>>>
>>> Hello,
>>>
>>> Could the above PR can be reviewed? This has been waiting for a long
>>> time.
>>>
>>> Just to mention, the package name should have "internal". Round-robin
>>> partitioning should have been supported without/without a key from the
>>> beginning. It provides user a guaranteed round-robin partitioning without
>>> having to regard for key values (e.g. null/not null). From our business
>>> side, this is a Kafka internal logic. Hence, the placement inside
>>> "internal" package.
>>>
>>> Thanks,
>>>
>>

Re: PR review

[M. Manna]

https://github.com/apache/kafka/pull/6771

Could this be reviewed please ?

On Wed, 3 Jul 2019 at 11:35, M. Manna  wrote:

> https://github.com/apache/kafka/pull/6771
>
> Bouncing both users and dev to get some activity going. We are waiting for
> a while to get this KIP pr merged.
>
> Could someone please review?
>
> Thanks,
>
> On Sun, 30 Jun 2019 at 08:59, M. Manna  wrote:
>
>> https://github.com/apache/kafka/pull/6771
>>
>> Hello,
>>
>> Could the above PR can be reviewed? This has been waiting for a long time.
>>
>> Just to mention, the package name should have "internal". Round-robin
>> partitioning should have been supported without/without a key from the
>> beginning. It provides user a guaranteed round-robin partitioning without
>> having to regard for key values (e.g. null/not null). From our business
>> side, this is a Kafka internal logic. Hence, the placement inside
>> "internal" package.
>>
>> Thanks,
>>
>

Re: PR review

[M. Manna]

https://github.com/apache/kafka/pull/6771

Bouncing both users and dev to get some activity going. We are waiting for
a while to get this KIP pr merged.

Could someone please review?

Thanks,

On Sun, 30 Jun 2019 at 08:59, M. Manna  wrote:

> https://github.com/apache/kafka/pull/6771
>
> Hello,
>
> Could the above PR can be reviewed? This has been waiting for a long time.
>
> Just to mention, the package name should have "internal". Round-robin
> partitioning should have been supported without/without a key from the
> beginning. It provides user a guaranteed round-robin partitioning without
> having to regard for key values (e.g. null/not null). From our business
> side, this is a Kafka internal logic. Hence, the placement inside
> "internal" package.
>
> Thanks,
>

PR review

[M. Manna]

https://github.com/apache/kafka/pull/6771

Hello,

Could the above PR can be reviewed? This has been waiting for a long time.

Just to mention, the package name should have "internal". Round-robin
partitioning should have been supported without/without a key from the
beginning. It provides user a guaranteed round-robin partitioning without
having to regard for key values (e.g. null/not null). From our business
side, this is a Kafka internal logic. Hence, the placement inside
"internal" package.

Thanks,

Re: [jira] [Created] (KAFKA-8604) kafka log dir was marked as offline because of deleting segments of __consumer_offsets failed

[M. Manna]

This is a known issue for Windows. See KAFKA-6188.

Thanks,

On Wed, 26 Jun 2019 at 14:46, songyingshuan (JIRA)  wrote:

> songyingshuan created KAFKA-8604:
> 
>
>  Summary: kafka log dir was marked as offline because of
> deleting segments of __consumer_offsets failed
>  Key: KAFKA-8604
>  URL: https://issues.apache.org/jira/browse/KAFKA-8604
>  Project: Kafka
>   Issue Type: Bug
>   Components: log cleaner
> Affects Versions: 1.0.1
> Reporter: songyingshuan
>
>
> We encountered a problem in our product env without any foresight. When
> kafka broker trying to clean __consumer_offsets-38 (and only happents to
> this partition), the log shows
> it failed, and marking the whole disk/log dir offline, and this leads to a
> negative impact on some normal partitions (because of the ISR list of those
> partitions decrease).
> we had to restart the broker server to reuse the disk/dir which was marked
> as offline. BUT!! this problem occurs periodically with the same reason so
> we have to restart broker periodically.
>
> we read some source code of kafka-1.0.1, but cannot make sure why this
> happends. And The cluster status had been good until this problem suddenly
> attacked us.
>
> the error log is something like this :
>
>
> {code:java}
> 2019-06-25 00:11:26,241 INFO kafka.log.TimeIndex: Deleting index
> /data6/kafka/data/__consumer_offsets-38/012855596978.timeindex.deleted
> 2019-06-25 00:11:26,258 ERROR kafka.server.LogDirFailureChannel: Error
> while deleting segments for __consumer_offsets-38 in dir /data6/kafka/data
> java.io.IOException: Delete of log .log.deleted failed.
> at kafka.log.LogSegment.delete(LogSegment.scala:496)
> at
> kafka.log.Log$$anonfun$kafka$log$Log$$deleteSeg$1$1.apply$mcV$sp(Log.scala:1596)
> at
> kafka.log.Log$$anonfun$kafka$log$Log$$deleteSeg$1$1.apply(Log.scala:1596)
> at
> kafka.log.Log$$anonfun$kafka$log$Log$$deleteSeg$1$1.apply(Log.scala:1596)
> at kafka.log.Log.maybeHandleIOException(Log.scala:1669)
> at kafka.log.Log.kafka$log$Log$$deleteSeg$1(Log.scala:1595)
> at
> kafka.log.Log$$anonfun$kafka$log$Log$$asyncDeleteSegment$1.apply$mcV$sp(Log.scala:1599)
> at
> kafka.utils.KafkaScheduler$$anonfun$1.apply$mcV$sp(KafkaScheduler.scala:110)
> at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:61)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2019-06-25 00:11:26,265 ERROR kafka.utils.KafkaScheduler: Uncaught
> exception in scheduled task 'delete-file'
> org.apache.kafka.common.errors.KafkaStorageException: Error while deleting
> segments for __consumer_offsets-38 in dir /data6/kafka/data
> Caused by: java.io.IOException: Delete of log
> .log.deleted failed.
> at kafka.log.LogSegment.delete(LogSegment.scala:496)
> at
> kafka.log.Log$$anonfun$kafka$log$Log$$deleteSeg$1$1.apply$mcV$sp(Log.scala:1596)
> at
> kafka.log.Log$$anonfun$kafka$log$Log$$deleteSeg$1$1.apply(Log.scala:1596)
> at
> kafka.log.Log$$anonfun$kafka$log$Log$$deleteSeg$1$1.apply(Log.scala:1596)
> at kafka.log.Log.maybeHandleIOException(Log.scala:1669)
> at kafka.log.Log.kafka$log$Log$$deleteSeg$1(Log.scala:1595)
> at
> kafka.log.Log$$anonfun$kafka$log$Log$$asyncDeleteSegment$1.apply$mcV$sp(Log.scala:1599)
> at
> kafka.utils.KafkaScheduler$$anonfun$1.apply$mcV$sp(KafkaScheduler.scala:110)
> at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:61)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2019-06-25 00:11:26,268 INFO kafka.server.ReplicaManager: [ReplicaManager
> broker=1001] Stopping serving replicas in dir /data6/kafka/data
> {code}
>  and we also find that the cleaning of __consumer_offsets-38 is so
> frequently that almost 85% of output log is related. something like this :
>
>
> {code:java}
> 2019-06-25 20:29:03,222 INFO kafka.log.OffsetIndex: Deleting i

Re: PR review

[M. Manna]

I have updated it with my comments - @mjsax could you kindly comment
I stopped watching the issue since it sends all updates to all issues to my
inbox :)

Thanks,

On Sat, 22 Jun 2019 at 10:48, M. Manna  wrote:

> Thanks Matthias. Moved to dev DL now.
>
> I saw your comment on the PR regarding imports. I’ll fix those and
> resubmit.
>
> Thanks,
>
> On Sat, 22 Jun 2019 at 01:22, Matthias J. Sax 
> wrote:
>
>> I would recommend to move this discussion to the dev list.
>>
>> -Matthias
>>
>> On 6/20/19 7:42 PM, M. Manna wrote:
>> > It's done. Sorry for the confusion.
>> >
>> > The KIP table however, showed that it's been accepted. But yes it's
>> better
>> > to keep all places consistent.
>> >
>> > Thanks,
>> >
>> > On Fri, 21 Jun 2019 at 03:27, Jeff Widman  wrote:
>> >
>> >> The KIP linked to from the JIRA shows the KIP as still under
>> discussion...
>> >> if it's been voted/approved, then can you please update the wiki page?
>> >>
>> >> On Thu, Jun 20, 2019 at 6:45 PM M. Manna  wrote:
>> >>
>> >>> Hello,
>> >>>
>> >>> We’ve been waiting for this PR for a while.
>> >>>
>> >>> https://github.com/apache/kafka/pull/6771
>> >>>
>> >>> Could this Be reviewed for new release ? This is important for our
>> >> project.
>> >>>
>> >>> Thanks,
>> >>>
>> >>
>> >>
>> >> --
>> >>
>> >> *Jeff Widman*
>> >> jeffwidman.com <http://www.jeffwidman.com/> | 740-WIDMAN-J (943-6265)
>> >> <><
>> >>
>> >
>>
>>

Re: PR review

[M. Manna]

Thanks Matthias. Moved to dev DL now.

I saw your comment on the PR regarding imports. I’ll fix those and resubmit.

Thanks,

On Sat, 22 Jun 2019 at 01:22, Matthias J. Sax  wrote:

> I would recommend to move this discussion to the dev list.
>
> -Matthias
>
> On 6/20/19 7:42 PM, M. Manna wrote:
> > It's done. Sorry for the confusion.
> >
> > The KIP table however, showed that it's been accepted. But yes it's
> better
> > to keep all places consistent.
> >
> > Thanks,
> >
> > On Fri, 21 Jun 2019 at 03:27, Jeff Widman  wrote:
> >
> >> The KIP linked to from the JIRA shows the KIP as still under
> discussion...
> >> if it's been voted/approved, then can you please update the wiki page?
> >>
> >> On Thu, Jun 20, 2019 at 6:45 PM M. Manna  wrote:
> >>
> >>> Hello,
> >>>
> >>> We’ve been waiting for this PR for a while.
> >>>
> >>> https://github.com/apache/kafka/pull/6771
> >>>
> >>> Could this Be reviewed for new release ? This is important for our
> >> project.
> >>>
> >>> Thanks,
> >>>
> >>
> >>
> >> --
> >>
> >> *Jeff Widman*
> >> jeffwidman.com <http://www.jeffwidman.com/> | 740-WIDMAN-J (943-6265)
> >> <><
> >>
> >
>
>

Re: Preliminary blog post for the Apache Kafka 2.3.0 release

[M. Manna]

It’s asking for credentials...?

On Tue, 18 Jun 2019 at 15:10, Colin McCabe  wrote:

> Hi all,
>
> I've written up a preliminary blog post about the upcoming Apache Kafka
> 2.3.0 release.  Take a look and let me know what you think.
>
>
> https://blogs.apache.org/roller-ui/authoring/preview/kafka/?previewEntry=what-s-new-in-apache
>
> cheers,
> Colin
>

Re: message.max.bytes

[M. Manna]

Can you see anything from server logs whether this has been purged ? As far
as I can remember, you are not supposed to just make it smaller without
future issues.

Let us here know what you can find from logs. Also, you should think about
configuring request and server level max size to avoid such issues.

Thanks,

Thanks,

On Sun, 16 Jun 2019 at 11:39, 王双双  wrote:

> Hello everyone,
> I have recently encountered a problem with the parameter
> 'message.max.bytes'. Here are the details: There are 1000 messages that I
> have not consumed, each message is 7KB, and a log under
>  the topic is 106MB. I altered the parameter 'message.max.bytes' from 4MB
> to 1MB, then I restarted the kafka server. I checked the message in each
> partition, unfortunately, the message that have not been consumed
> disappeared.
> Note: my kafka  server version is 0.10.0.1, I created a topic with 3
> replicas and 6 partitions.
> Does anyone know where the disappeared message gone? Can I recover this?
> I appreciate it if anyone replies to me soon.
>
>
> Best regards.
> Wang

Re: Newbie on lookout for work

[M. Manna]

Please send a separate email, asking for JIRA contributor permissions.

As always, you’re welcome to read guidelines mentioned in confluence WIKI
for apache Kafka and Kafka official site to understand the practices.

Thanks,

On Thu, 6 Jun 2019 at 11:57, Dulvin Witharane  wrote:

> Hi I’ve found a ticket and wanting to know how to get it assigned to me to
> start working on. Could anyone let me know how to.
>
> On Thu, Jun 6, 2019 at 4:43 AM Gwen Shapira  wrote:
>
> > Hi,
> >
> > You can look at our contribution guidelines:
> > https://kafka.apache.org/contributing
> >
> > We have a number of open "newbie" tickets in JIRA, perhaps one of them
> > will be interesting to you:
> >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20KAFKA%20AND%20status%20%3D%20Open%20AND%20labels%20in%20(Newbie%2C%20newbie)%20ORDER%20BY%20created%20DESC
> >
> > On Wed, Jun 5, 2019 at 9:36 AM Dulvin Witharane 
> wrote:
> > >
> > > Hi All,
> > >
> > > I'm a newbie for FOSS and I'd like to start contributing to Apache
> Kafka.
> > > Any pointers for beginning would be appreciated.
> > >
> > > --
> > > *Witharane, D.R.H.*
> > >
> > > R&D Engineer
> > > Synopsys Inc,
> > > Colombo 08
> > > Mobile : *+94 7 <%2B94%2071%201127241>7 6746781*
> > > Skype  : dulvin.rivindu
> > > Facebook  |
> LinkedIn
> > > 
> >
> >
> >
> > --
> > Gwen Shapira
> > Product Manager | Confluent
> > 650.450.2760 | @gwenshap
> > Follow us: Twitter | blog
> >
> --
> Witharane, DRH
> R & D Engineer
> Synopsys Lanka (Pvt) Ltd.
> Borella, Sri Lanka
> 0776746781
>
> Sent from my iPhone
>

Re: [VOTE] KIP-369 Alternative Partitioner to Support "Always Round-Robin" Selection

[M. Manna]

I have now submitted the PR for review. Thanks Matthias for pointing out
that KAFKA- was raised to address the same.

https://github.com/apache/kafka/pull/6771

if someone reviews after Jenkins build is complete, I would appreciate it.

Thanks,

On Fri, 17 May 2019 at 22:18, M. Manna  wrote:

> Apologies for the delay. As per the original thread, there have been 3
> binding votes.
>
> I will be closing this and update the confluence page with the results.
> Also, I will be submitting the PR soon.
>
> Regards,
>
> On Fri, 5 Apr 2019 at 00:18, M. Manna  wrote:
>
>> Thanks Harsha.
>>
>> As per your comments, I have counted 3 binding votes so far.
>>
>> Thanks everyone for your comments and support. I’ll update the kip next
>> morning and do  the needful.
>>
>> Regards,
>>
>> On Thu, 4 Apr 2019 at 22:10, Harsha  wrote:
>>
>>> Looks like the KIP is passed with 3 binding votes.  From Matthias, Bill
>>> Bejeck and myself you got 3 binding votes.
>>> You can do the full tally of the votes and send out a close of vote
>>> thread.
>>>
>>> Thanks,
>>> Harsha
>>>
>>> On Thu, Apr 4, 2019, at 12:24 PM, M. Manna wrote:
>>> > Hello,
>>> >
>>> > Trying to revive this thread again. Would anyone be interested in
>>> having
>>> > this KiP through
>>> >
>>> >
>>> > Thanks,
>>> >
>>> > On Fri, 25 Jan 2019 at 16:44, M. Manna  wrote:
>>> >
>>> > > Hello,
>>> > >
>>> > > I am trying to revive this thread. I only got 1 binding vote so far.
>>> > >
>>> > > Please feel free to revisit and comment here.
>>> > >
>>> > > Thanks,
>>> > >
>>> > > On Thu, 25 Oct 2018 at 00:15, M. Manna  wrote:
>>> > >
>>> > >> Hey IJ,
>>> > >>
>>> > >> Thanks for your interest in the KIP.
>>> > >>
>>> > >> My point was simply that the round-robin should happen even if the
>>> key is
>>> > >> not null. As for the importance of key in our case, we treat the
>>> key as
>>> > >> metadata. Each key is composed of certain info which are parsed by
>>> our
>>> > >> consumer thread. We will then determine whether it's an actionable
>>> message
>>> > >> (e.g. process it), or a loopback(ignore it). You could argue, "Why
>>> not
>>> > >> append this metadata with the record and parse it there?". But that
>>> means
>>> > >> the following:
>>> > >>
>>> > >> 1) I'm always passing null key to achieve this - I would like to
>>> pass
>>> > >> Null/Not-Null/Other key i.e. flexibility
>>> > >> 2) Suppose the message size is 99 KB and and max message bytes
>>> allowed is
>>> > >> 100K. Now prefixing metadata with message results into the actual
>>> message
>>> > >> being 101K. This will fail at producer level and cause a retry/log
>>> this in
>>> > >> our DB for future pickup.
>>> > >>
>>> > >> To avoid all these, we are simply proposing this new partitioner
>>> class.
>>> > >> but all Kafka new releases will still have DefaultPartitioner as
>>> default,
>>> > >> unless they change the prop file to use our new class.
>>> > >>
>>> > >> Regards,
>>> > >>
>>> > >> On Sun, 21 Oct 2018 at 04:05, Ismael Juma 
>>> wrote:
>>> > >>
>>> > >>> Thanks for the KIP. Can you please elaborate on the need for the
>>> key in
>>> > >>> this case? The KIP simply states that the key is needed for
>>> metadata, but
>>> > >>> doesn't give any more details.
>>> > >>>
>>> > >>> Ismael
>>> > >>>
>>> > >>> On Tue, Sep 4, 2018 at 3:39 AM M. Manna 
>>> wrote:
>>> > >>>
>>> > >>> > Hello,
>>> > >>> >
>>> > >>> > I have made necessary changes as per the original discussion
>>> thread,
>>> > >>> and
>>> > >>> > would like to put it for votes.
>>> > >>> >
>>> > >>> > Thank you very much for your suggestion and guidance so far.
>>> > >>> >
>>> > >>> > Regards,
>>> > >>> >
>>> > >>>
>>> > >>
>>> >
>>>
>>

[jira] [Created] (KAFKA-8394) Cannot Start a build with New Gradle Version

[M. Manna (JIRA)]

M. Manna created KAFKA-8394:
---

 Summary: Cannot Start a build with New Gradle Version
 Key: KAFKA-8394
 URL: https://issues.apache.org/jira/browse/KAFKA-8394
 Project: Kafka
  Issue Type: Bug
  Components: build
Affects Versions: 2.2.0
Reporter: M. Manna


When I downloaded gradle 5.4.1 and ran `gradle wrapper` - the build failed 
because the scoverage plugin dependency encountered some build errors. The 
following is the output

 

org.gradle.api.GradleScriptException: A problem occurred evaluating root
 project 'kafka'.
         at
 
org.gradle.groovy.scripts.internal.DefaultScriptRunnerFactory$ScriptRunnerImpl.run(DefaultScriptRunnerFactory.java:92)
         at
 
org.gradle.configuration.DefaultScriptPluginFactory$ScriptPluginImpl$2.run(DefaultScriptPluginFactory.java:221)
         at
 
org.gradle.configuration.ProjectScriptTarget.addConfiguration(ProjectScriptTarget.java:77)
         at
 
org.gradle.configuration.DefaultScriptPluginFactory$ScriptPluginImpl.apply(DefaultScriptPluginFactory.java:226)
         at
 
org.gradle.configuration.BuildOperationScriptPlugin$1$1.run(BuildOperationScriptPlugin.java:69)
         at
 
org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:402)
         at
 
org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:394)
         at
 
org.gradle.internal.operations.DefaultBuildOperationExecutor$1.execute(DefaultBuildOperationExecutor.java:165)
         at
 
org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:250)
         at
 
org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:158)
         at
 
org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:92)
         at
 
org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
         at
 
org.gradle.configuration.BuildOperationScriptPlugin$1.execute(BuildOperationScriptPlugin.java:66)
         at
 
org.gradle.configuration.BuildOperationScriptPlugin$1.execute(BuildOperationScriptPlugin.java:63)
         at
 
org.gradle.configuration.internal.DefaultUserCodeApplicationContext.apply(DefaultUserCodeApplicationContext.java:48)
         at
 
org.gradle.configuration.BuildOperationScriptPlugin.apply(BuildOperationScriptPlugin.java:63)
         at
 
org.gradle.configuration.project.BuildScriptProcessor$1.run(BuildScriptProcessor.java:44)
         at org.gradle.internal.Factories$1.create(Factories.java:25)
         at
 
org.gradle.api.internal.project.DefaultProjectStateRegistry$ProjectStateImpl.withMutableState(DefaultProjectStateRegistry.java:200)
         at
 
org.gradle.api.internal.project.DefaultProjectStateRegistry$ProjectStateImpl.withMutableState(DefaultProjectStateRegistry.java:186)
         at
 
org.gradle.configuration.project.BuildScriptProcessor.execute(BuildScriptProcessor.java:41)
         at
 
org.gradle.configuration.project.BuildScriptProcessor.execute(BuildScriptProcessor.java:26)
         at
 
org.gradle.configuration.project.ConfigureActionsProjectEvaluator.evaluate(ConfigureActionsProjectEvaluator.java:34)
         at
 
org.gradle.configuration.project.LifecycleProjectEvaluator$EvaluateProject$1.run(LifecycleProjectEvaluator.java:106)
         at org.gradle.internal.Factories$1.create(Factories.java:25)
         at
 
[org.gradle.internal.work|http://org.gradle.internal.work/].DefaultWorkerLeaseService.withLocks(DefaultWorkerLeaseService.java:183)
         at
 
[org.gradle.internal.work|http://org.gradle.internal.work/].StopShieldingWorkerLeaseService.withLocks(StopShieldingWorkerLeaseService.java:40)
         at
 
org.gradle.api.internal.project.DefaultProjectStateRegistry$ProjectStateImpl.withProjectLock(DefaultProjectStateRegistry.java:226)
         at
 
org.gradle.api.internal.project.DefaultProjectStateRegistry$ProjectStateImpl.withMutableState(DefaultProjectStateRegistry.java:220)
         at
 
org.gradle.api.internal.project.DefaultProjectStateRegistry$ProjectStateImpl.withMutableState(DefaultProjectStateRegistry.java:186)
         at
 
org.gradle.configuration.project.LifecycleProjectEvaluator$EvaluateProject.run(LifecycleProjectEvaluator.java:95)
         at
 
org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:402)
         at
 
org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:394)
         at
 
org.gradle.internal.operations.DefaultBuildOperationExecutor$1.execute(DefaultBuildOperationExecutor.java:165)
 

Re: [VOTE] KIP-369 Alternative Partitioner to Support "Always Round-Robin" Selection

[M. Manna]

Apologies for the delay. As per the original thread, there have been 3
binding votes.

I will be closing this and update the confluence page with the results.
Also, I will be submitting the PR soon.

Regards,

On Fri, 5 Apr 2019 at 00:18, M. Manna  wrote:

> Thanks Harsha.
>
> As per your comments, I have counted 3 binding votes so far.
>
> Thanks everyone for your comments and support. I’ll update the kip next
> morning and do  the needful.
>
> Regards,
>
> On Thu, 4 Apr 2019 at 22:10, Harsha  wrote:
>
>> Looks like the KIP is passed with 3 binding votes.  From Matthias, Bill
>> Bejeck and myself you got 3 binding votes.
>> You can do the full tally of the votes and send out a close of vote
>> thread.
>>
>> Thanks,
>> Harsha
>>
>> On Thu, Apr 4, 2019, at 12:24 PM, M. Manna wrote:
>> > Hello,
>> >
>> > Trying to revive this thread again. Would anyone be interested in having
>> > this KiP through
>> >
>> >
>> > Thanks,
>> >
>> > On Fri, 25 Jan 2019 at 16:44, M. Manna  wrote:
>> >
>> > > Hello,
>> > >
>> > > I am trying to revive this thread. I only got 1 binding vote so far.
>> > >
>> > > Please feel free to revisit and comment here.
>> > >
>> > > Thanks,
>> > >
>> > > On Thu, 25 Oct 2018 at 00:15, M. Manna  wrote:
>> > >
>> > >> Hey IJ,
>> > >>
>> > >> Thanks for your interest in the KIP.
>> > >>
>> > >> My point was simply that the round-robin should happen even if the
>> key is
>> > >> not null. As for the importance of key in our case, we treat the key
>> as
>> > >> metadata. Each key is composed of certain info which are parsed by
>> our
>> > >> consumer thread. We will then determine whether it's an actionable
>> message
>> > >> (e.g. process it), or a loopback(ignore it). You could argue, "Why
>> not
>> > >> append this metadata with the record and parse it there?". But that
>> means
>> > >> the following:
>> > >>
>> > >> 1) I'm always passing null key to achieve this - I would like to pass
>> > >> Null/Not-Null/Other key i.e. flexibility
>> > >> 2) Suppose the message size is 99 KB and and max message bytes
>> allowed is
>> > >> 100K. Now prefixing metadata with message results into the actual
>> message
>> > >> being 101K. This will fail at producer level and cause a retry/log
>> this in
>> > >> our DB for future pickup.
>> > >>
>> > >> To avoid all these, we are simply proposing this new partitioner
>> class.
>> > >> but all Kafka new releases will still have DefaultPartitioner as
>> default,
>> > >> unless they change the prop file to use our new class.
>> > >>
>> > >> Regards,
>> > >>
>> > >> On Sun, 21 Oct 2018 at 04:05, Ismael Juma  wrote:
>> > >>
>> > >>> Thanks for the KIP. Can you please elaborate on the need for the
>> key in
>> > >>> this case? The KIP simply states that the key is needed for
>> metadata, but
>> > >>> doesn't give any more details.
>> > >>>
>> > >>> Ismael
>> > >>>
>> > >>> On Tue, Sep 4, 2018 at 3:39 AM M. Manna  wrote:
>> > >>>
>> > >>> > Hello,
>> > >>> >
>> > >>> > I have made necessary changes as per the original discussion
>> thread,
>> > >>> and
>> > >>> > would like to put it for votes.
>> > >>> >
>> > >>> > Thank you very much for your suggestion and guidance so far.
>> > >>> >
>> > >>> > Regards,
>> > >>> >
>> > >>>
>> > >>
>> >
>>
>

Re: [VOTE] KIP-369 Alternative Partitioner to Support "Always Round-Robin" Selection

[M. Manna]

Thanks Harsha.

As per your comments, I have counted 3 binding votes so far.

Thanks everyone for your comments and support. I’ll update the kip next
morning and do  the needful.

Regards,

On Thu, 4 Apr 2019 at 22:10, Harsha  wrote:

> Looks like the KIP is passed with 3 binding votes.  From Matthias, Bill
> Bejeck and myself you got 3 binding votes.
> You can do the full tally of the votes and send out a close of vote thread.
>
> Thanks,
> Harsha
>
> On Thu, Apr 4, 2019, at 12:24 PM, M. Manna wrote:
> > Hello,
> >
> > Trying to revive this thread again. Would anyone be interested in having
> > this KiP through
> >
> >
> > Thanks,
> >
> > On Fri, 25 Jan 2019 at 16:44, M. Manna  wrote:
> >
> > > Hello,
> > >
> > > I am trying to revive this thread. I only got 1 binding vote so far.
> > >
> > > Please feel free to revisit and comment here.
> > >
> > > Thanks,
> > >
> > > On Thu, 25 Oct 2018 at 00:15, M. Manna  wrote:
> > >
> > >> Hey IJ,
> > >>
> > >> Thanks for your interest in the KIP.
> > >>
> > >> My point was simply that the round-robin should happen even if the
> key is
> > >> not null. As for the importance of key in our case, we treat the key
> as
> > >> metadata. Each key is composed of certain info which are parsed by our
> > >> consumer thread. We will then determine whether it's an actionable
> message
> > >> (e.g. process it), or a loopback(ignore it). You could argue, "Why not
> > >> append this metadata with the record and parse it there?". But that
> means
> > >> the following:
> > >>
> > >> 1) I'm always passing null key to achieve this - I would like to pass
> > >> Null/Not-Null/Other key i.e. flexibility
> > >> 2) Suppose the message size is 99 KB and and max message bytes
> allowed is
> > >> 100K. Now prefixing metadata with message results into the actual
> message
> > >> being 101K. This will fail at producer level and cause a retry/log
> this in
> > >> our DB for future pickup.
> > >>
> > >> To avoid all these, we are simply proposing this new partitioner
> class.
> > >> but all Kafka new releases will still have DefaultPartitioner as
> default,
> > >> unless they change the prop file to use our new class.
> > >>
> > >> Regards,
> > >>
> > >> On Sun, 21 Oct 2018 at 04:05, Ismael Juma  wrote:
> > >>
> > >>> Thanks for the KIP. Can you please elaborate on the need for the key
> in
> > >>> this case? The KIP simply states that the key is needed for
> metadata, but
> > >>> doesn't give any more details.
> > >>>
> > >>> Ismael
> > >>>
> > >>> On Tue, Sep 4, 2018 at 3:39 AM M. Manna  wrote:
> > >>>
> > >>> > Hello,
> > >>> >
> > >>> > I have made necessary changes as per the original discussion
> thread,
> > >>> and
> > >>> > would like to put it for votes.
> > >>> >
> > >>> > Thank you very much for your suggestion and guidance so far.
> > >>> >
> > >>> > Regards,
> > >>> >
> > >>>
> > >>
> >
>

Re: [VOTE] KIP-369 Alternative Partitioner to Support "Always Round-Robin" Selection

[M. Manna]

Hello,

Trying to revive this thread again. Would anyone be interested in having
this KiP through


Thanks,

On Fri, 25 Jan 2019 at 16:44, M. Manna  wrote:

> Hello,
>
> I am trying to revive this thread. I only got 1 binding vote so far.
>
> Please feel free to revisit and comment here.
>
> Thanks,
>
> On Thu, 25 Oct 2018 at 00:15, M. Manna  wrote:
>
>> Hey IJ,
>>
>> Thanks for your interest in the KIP.
>>
>> My point was simply that the round-robin should happen even if the key is
>> not null. As for the importance of key in our case, we treat the key as
>> metadata. Each key is composed of certain info which are parsed by our
>> consumer thread. We will then determine whether it's an actionable message
>> (e.g. process it), or a loopback(ignore it). You could argue, "Why not
>> append this metadata with the record and parse it there?". But that means
>> the following:
>>
>> 1) I'm always passing null key to achieve this - I would like to pass
>> Null/Not-Null/Other key i.e. flexibility
>> 2) Suppose the message size is 99 KB and and max message bytes allowed is
>> 100K. Now prefixing metadata with message results into the actual message
>> being 101K. This will fail at producer level and cause a retry/log this in
>> our DB for future pickup.
>>
>> To avoid all these, we are simply proposing this new partitioner class.
>> but all Kafka new releases will still have DefaultPartitioner as default,
>> unless they change the prop file to use our new class.
>>
>> Regards,
>>
>> On Sun, 21 Oct 2018 at 04:05, Ismael Juma  wrote:
>>
>>> Thanks for the KIP. Can you please elaborate on the need for the key in
>>> this case? The KIP simply states that the key is needed for metadata, but
>>> doesn't give any more details.
>>>
>>> Ismael
>>>
>>> On Tue, Sep 4, 2018 at 3:39 AM M. Manna  wrote:
>>>
>>> > Hello,
>>> >
>>> > I have made necessary changes as per the original discussion thread,
>>> and
>>> > would like to put it for votes.
>>> >
>>> > Thank you very much for your suggestion and guidance so far.
>>> >
>>> > Regards,
>>> >
>>>
>>

Re: Need help in setting up security in Kafka systems

[M. Manna]

Hi,

Have you checked the section on security here? It's got a comprehensive
guide

https://kafka.apache.org/documentation/#security_sasl

To answer your questions briefly

- Broker to Broker should be plainTEXT (or SSL if inter-broker security is
enabled then broker2brorker works as a client coms)
- zookeeper to broker should be plaintext
- client to broker should be plaintext

But then, I would recommend you read the guidlines above. We setup our
security nicely with the above guidelines, however they were not using SASL
but only SSL.

Thanks,

On Thu, 4 Apr 2019 at 15:53, Suman B N  wrote:

> Team,
>
> Can anyone help me share the configs to be set to achieve the below
> security in Kafka systems?
>
>- Broker-Broker should be PLAINTEXT(No Authentication and Authorization
>between brokers)
>- Zookeeper-Broker should be PLAINTEXT(No Authentication and
>Authorization between brokers and zookeeper)
>- Client-Broker should be SASL_PLAINTEXT(Using JAAS).
>
> Any help to finalize broker configs and client configs will be very
> helpful.
>
> I am still trying out some configs. I will update the configs with
> respective issues observed very soon.
>
> Thanks,
> Suman
> --
> *Suman*
> *OlaCabs*
>

Kafka Broker Config (logs.dir) on Kubernetes

[M. Manna]

 Hello,

The question might trigger people to reply with "Confluent" - but it's not
related to confluent as the kubernetes offering is not for publi/community
edition. So, discussing Helm charts and intro to Confluent isn't our
objective here.

What I am trying to understand is how does the log files (kafka message
logs, consumer offsets) are managed in Kubernetes (e.g. persistent volume,
statefulsets, etc). I have a 3 node cluster running over 3 physical Linux
VMs, and would like to move this setup to Kubernetes.

The only part where we are strugging is with the following:

1) how does logs.dir configuration work per Pod?
2) Assuming I have 3 PODs (3 brokers), and one of the Pod Goes down - how
do I manage the message log and offset files ? If a POD goes does, that
means it will delete everything in the logs.dir location, won't it?
3) I am assuming broker.id will be supplied using some form of configMap,
but if there is anything worth knowing here, please do share.

We have a reliable service on bare metal cloud, so we don't want to disrupt
it unless we are sure about the changes.

Does anyone have any experience with this? If so, it would be great if you
can share any gists or configs.

Much appreciated.

Regards,

Re: [jira] [Created] (KAFKA-8149) ERROR Disk error while writing to recovery point file

[M. Manna]

This is probably a duplicate of Kafka-6188 and Kafka-1194

Please double check and verify.

Thanks,

On Fri, 22 Mar 2019 at 17:36, wade wu (JIRA)  wrote:

> wade wu created KAFKA-8149:
> --
>
>  Summary: ERROR Disk error while writing to recovery point file
>  Key: KAFKA-8149
>  URL: https://issues.apache.org/jira/browse/KAFKA-8149
>  Project: Kafka
>   Issue Type: Bug
>   Components: core
> Affects Versions: 1.1.1
>  Environment: Windows
> Reporter: wade wu
>
>
> [2019-03-17 02:55:14,458] ERROR Disk error while writing to recovery point
> file in directory I:\data\Kafka\kafka-datalogs
> (kafka.server.LogDirFailureChannel)
> java.nio.file.AccessDeniedException:
> H:\data\Kafka\kafka-datalogs\AzPubSubCompactTestNew1-0\01170892.snapshot
>  at
> sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
>  at
> sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
>  at
> sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
>  at
> sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)
>  at
> sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
>  at java.nio.file.Files.deleteIfExists(Files.java:1165)
>  at
> kafka.log.ProducerStateManager$$anonfun$kafka$log$ProducerStateManager$$deleteSnapshotFiles$2.apply(ProducerStateManager.scala:458)
>  at
> kafka.log.ProducerStateManager$$anonfun$kafka$log$ProducerStateManager$$deleteSnapshotFiles$2.apply(ProducerStateManager.scala:457)
>  at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
>  at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
>  at
> kafka.log.ProducerStateManager$.kafka$log$ProducerStateManager$$deleteSnapshotFiles(ProducerStateManager.scala:457)
>  at
> kafka.log.ProducerStateManager$.deleteSnapshotsBefore(ProducerStateManager.scala:454)
>  at
> kafka.log.ProducerStateManager.deleteSnapshotsBefore(ProducerStateManager.scala:763)
>  at
> kafka.log.Log.deleteSnapshotsAfterRecoveryPointCheckpoint(Log.scala:1461)
>  at
> kafka.log.LogManager$$anonfun$kafka$log$LogManager$$checkpointLogRecoveryOffsetsInDir$1$$anonfun$apply$29$$anonfun$apply$31.apply(LogManager.scala:577)
>  at
> kafka.log.LogManager$$anonfun$kafka$log$LogManager$$checkpointLogRecoveryOffsetsInDir$1$$anonfun$apply$29$$anonfun$apply$31.apply(LogManager.scala:577)
>  at scala.collection.immutable.List.foreach(List.scala:392)
>  at
> kafka.log.LogManager$$anonfun$kafka$log$LogManager$$checkpointLogRecoveryOffsetsInDir$1$$anonfun$apply$29.apply(LogManager.scala:577)
>  at
> kafka.log.LogManager$$anonfun$kafka$log$LogManager$$checkpointLogRecoveryOffsetsInDir$1$$anonfun$apply$29.apply(LogManager.scala:573)
>  at scala.Option.foreach(Option.scala:257)
>  at
> kafka.log.LogManager$$anonfun$kafka$log$LogManager$$checkpointLogRecoveryOffsetsInDir$1.apply(LogManager.scala:573)
>  at
> kafka.log.LogManager$$anonfun$kafka$log$LogManager$$checkpointLogRecoveryOffsetsInDir$1.apply(LogManager.scala:572)
>  at scala.Option.foreach(Option.scala:257)
>  at
> kafka.log.LogManager.kafka$log$LogManager$$checkpointLogRecoveryOffsetsInDir(LogManager.scala:572)
>  at
> kafka.log.LogManager$$anonfun$checkpointLogRecoveryOffsets$1.apply(LogManager.scala:556)
>  at
> kafka.log.LogManager$$anonfun$checkpointLogRecoveryOffsets$1.apply(LogManager.scala:556)
>  at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>  at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
>  at kafka.log.LogManager.checkpointLogRecoveryOffsets(LogManager.scala:556)
>  at kafka.log.LogManager.truncateTo(LogManager.scala:520)
>  at
> kafka.cluster.Partition$$anonfun$truncateTo$1.apply$mcV$sp(Partition.scala:665)
>  at
> kafka.cluster.Partition$$anonfun$truncateTo$1.apply(Partition.scala:665)
>  at
> kafka.cluster.Partition$$anonfun$truncateTo$1.apply(Partition.scala:665)
>  at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:250)
>  at kafka.utils.CoreUtils$.inReadLock(CoreUtils.scala:256)
>  at kafka.cluster.Partition.truncateTo(Partition.scala:664)
>  at
> kafka.server.ReplicaFetcherThread$$anonfun$maybeTruncate$1.apply(ReplicaFetcherThread.scala:320)
>  at
> kafka.server.ReplicaFetcherThread$$anonfun$maybeTruncate$1.apply(ReplicaFetcherThread.scala:301)
>  at scala.collection.immutable.Map$Map2.foreach(Map.scala:137)
>  at
> kafka.server.ReplicaFetcherThread.maybeTruncate(ReplicaFetcherThread.scala:301)
>  at
> kafka.server.AbstractFetcherThread$$anonfun$maybeTruncate$1.apply$mcV$sp(AbstractFetcherThread.scala:133)
>  at
> kafka.server.AbstractFetcherThread$$anonfun$maybeTruncate$1.apply(AbstractFetcherThread.scala:130)
>  at
> kafka.server.AbstractFetcherThread$$anonfun$maybeTruncate$1.apply(AbstractFetcherThread.scala:130)
>  at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:250)

Question on performance data for Kafka vs NATS

[M. Manna]

HI All,

https://nats.io/about/

this shows a general comparison of sender/receiver throughputs for NATS and
other messaging system including our favourite Kafka.

It appears that Kafka, despite taking the 2nd place, has a very low
throughput. My question is, where does Kafka win over NATS? is it the
unique partitioning and delivery semantics? Or, is it something else.

>From what I can see, NATS has traditional pub/sub and queuing. But it
doesn't look like there is any proper retention system built for this.

Has anyone come across this already?

Thanks,

Re: Kafka issue

[M. Manna]

3 broker with replication factor 2 ? Isn't that already problematic?

Have you considered extending the partition as 3 (since you have 3
brokers), and give a try? Can you afford extension to 1 more broker?

On Wed, 20 Mar 2019 at 16:02, Barguje, Bharat, Vodafone Group <
bharat.barg...@vodafone.com> wrote:

> Hi Team,
>
>
>
> I am using Apache Kafka stateful set, I have setup like 3 Zookeeper, 3
> Kafka broker, Replication factor is 2.
>
>
>
> I have 2 microservices with 2 kafka consumers, Kafka topic has 2
> partition, I could see 1 consumer is dead as per the command to display
> consumer group, But it still processing the record.
>
>
>
> So What is ahpepning is both the consumer is consuming from Partition 0,
> Can you please hekp me on this? I can setup a call if you need.
>
>
>
> Thanks & Regards
>
> [image: Vodafone] 
>
> *Bharat Barguje*
>
> Manager
> Vodafone Shared Services India
> Email: bharat.barg...@vodafone.com
>
>
>
>
>
>
> C2 General
>

Kafka Topic Volume and (possibly ACL) question

[M. Manna]

Hello,

We have a requirement where, based on business requirementes, we need to
publish data only for a specific set of clients. For example, an invoice
update shouldn't go to all clients, only the specific client. But a company
remittance info should be published to all clients. Also, in some cases, a
specific client changes some contract info which is published in a P2P
fashion. We have about 8k clients.

What is the ideal way to control this flow?

1) specific topic per client
2) Some form of ACL?

For option 1, we are not 100% sure if Kafka can handle 8k topics (or, the
resource issues for that matter). Has anyone solved a similar business
problem? If so, would you mind sharing your solution?

Btw, we are not using stream platform, it's simply pub-sub. Because we
don't need real-time aggregation of various items. For us, it's key that
the synchronisation occurs, and has "exactly-once" semantics.

Thanks,

Re: [jira] [Created] (KAFKA-7913) Kafka broker halts and messes up the whole cluster

[M. Manna]

Are you saying it’s jdk 11 issue ? Does it work with jdk8

On Sat, 9 Feb 2019 at 10:31, Andrej Urvantsev (JIRA) 
wrote:

> Andrej Urvantsev created KAFKA-7913:
> ---
>
>  Summary: Kafka broker halts and messes up the whole cluster
>  Key: KAFKA-7913
>  URL: https://issues.apache.org/jira/browse/KAFKA-7913
>  Project: Kafka
>   Issue Type: Bug
> Affects Versions: 2.1.0
>  Environment: kafka_2.12-2.1.0,
> openjdk version "11.0.1" 2018-10-16 LTS
> OpenJDK Runtime Environment 18.9 (build 11.0.1+13-LTS),
> CentOS Linux release 7.3.1611 (Core),
> linux 3.10.0-514.26.2.el7.x86_64
> Reporter: Andrej Urvantsev
>
>
> We upgraded cluster recently and running kafka 2.1.0 on java 11.
>
> For a time being everything went ok, but then random brokers started to
> halt from time to time.
>
> When it happens the broker still looks alive to other brokers, but it
> stops to receive network traffic. Other brokers then throw IOException:
> {noformat}
> java.io.IOException: Connection to 36155 was disconnected before the
> response was read
> at
> org.apache.kafka.clients.NetworkClientUtils.sendAndReceive(NetworkClientUtils.java:97)
> at
> kafka.server.ReplicaFetcherBlockingSend.sendRequest(ReplicaFetcherBlockingSend.scala:97)
> at
> kafka.server.ReplicaFetcherThread.fetchFromLeader(ReplicaFetcherThread.scala:190)
> at
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:241)
> at
> kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3(AbstractFetcherThread.scala:130)
> at
> kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3$adapted(AbstractFetcherThread.scala:129)
> at scala.Option.foreach(Option.scala:257)
> at
> kafka.server.AbstractFetcherThread.maybeFetch(AbstractFetcherThread.scala:129)
> at
> kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:111)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
> {noformat}
> On the problematic broker all logging stops. No errors, no exceptions -
> nothing.
>
> This also "breaks" all cluster - since clients and other brokers "think"
> that broker is still alive,
>
> they are trying to connect to it and it seems that leader election leaves
> problematic brokers as a leader.
>
>
>
> I would be glad to provide any further details if somebody could give an
> advice what to investigate when it happens next time.
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v7.6.3#76005)
>

[jira] [Resolved] (KAFKA-7881) Inter.broker.procol.version is incorrect for Rolling Upgrade

[M. Manna (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-7881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

M. Manna resolved KAFKA-7881.
-
Resolution: Cannot Reproduce
  Assignee: M. Manna

Duplicate jars were present - a delete and copy of new jars resolved the issue. 
closing this.

> Inter.broker.procol.version is incorrect for Rolling Upgrade
> 
>
> Key: KAFKA-7881
> URL: https://issues.apache.org/jira/browse/KAFKA-7881
> Project: Kafka
>  Issue Type: Bug
>  Components: admin
>Affects Versions: 2.1.0
>Reporter: M. Manna
>Assignee: M. Manna
>Priority: Critical
>   Original Estimate: 2m
>  Remaining Estimate: 2m
>
> We are getting the following error when upgrading from 1.1.0 to 2.1.0. We 
> have not changed the log message format version.
>  
> [https://kafka.apache.org/21/documentation.html]
>  
> Jan 29 06:06:14 one-drive-loc-01 systemd[1]: Started Zookeeper unit for this 
> machine.
>  Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: [2019-01-29 
> 06:06:15,272] INFO Registered kafka:type=kafka.Log4jController MBean 
> (kafka.utils.Log4jControl
>  Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: [2019-01-29 
> 06:06:15,599] ERROR Exiting Kafka due to fatal exception (kafka.Kafka$)
>  Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: 
> java.lang.IllegalArgumentException: Version `2.1` is not a valid version
>  Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: at 
> kafka.api.ApiVersion$$anonfun$apply$1.apply(ApiVersion.scala:88)
>  Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: at 
> kafka.api.ApiVersion$$anonfun$apply$1.apply(ApiVersion.scala:88)
>  Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: at 
> scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
>  Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: at 
> scala.collection.AbstractMap.getOrElse(Map.scala:59)
>  Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: at 
> kafka.api.ApiVersion$.apply(ApiVersion.scala:88)
>  Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: at 
> kafka.server.KafkaConfig.(KafkaConfig.scala:1127)
>  Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: at 
> kafka.server.KafkaConfig.(KafkaConfig.scala:989)
>  Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: at 
> kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:969)
>  Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: at 
> kafka.server.KafkaServerStartable$.fromProps(KafkaServerStartable.scala:28)
>  Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: at 
> kafka.Kafka$.main(Kafka.scala:82)
>  Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: at 
> kafka.Kafka.main(Kafka.scala)
>  
> we have also tried to make it to 2.1 but no change. If the version map is 
> being keyed using shortVersion, shouldn't it match? This is the first time we 
> are upgrading (from 1.1.0) and we have never had to change log message format.
> Please advise.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-7881) Inter.broker.procol.version is incorrect for Rolling Upgrade

[M. Manna (JIRA)]

M. Manna created KAFKA-7881:
---

 Summary: Inter.broker.procol.version is incorrect for Rolling 
Upgrade
 Key: KAFKA-7881
 URL: https://issues.apache.org/jira/browse/KAFKA-7881
 Project: Kafka
  Issue Type: Bug
  Components: admin
Affects Versions: 2.1.0
Reporter: M. Manna


We are getting the following error when upgrading from 1.1.0 to 2.1.0. We have 
not changed the log message format version.

 

Jan 29 06:06:14 one-drive-loc-01 systemd[1]: Started Zookeeper unit for this 
machine.
Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: [2019-01-29 
06:06:15,272] INFO Registered kafka:type=kafka.Log4jController MBean 
(kafka.utils.Log4jControl
Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: [2019-01-29 
06:06:15,599] ERROR Exiting Kafka due to fatal exception (kafka.Kafka$)
Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: 
java.lang.IllegalArgumentException: Version `2.1` is not a valid version
Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: at 
kafka.api.ApiVersion$$anonfun$apply$1.apply(ApiVersion.scala:88)
Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: at 
kafka.api.ApiVersion$$anonfun$apply$1.apply(ApiVersion.scala:88)
Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: at 
scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: at 
scala.collection.AbstractMap.getOrElse(Map.scala:59)
Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: at 
kafka.api.ApiVersion$.apply(ApiVersion.scala:88)
Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: at 
kafka.server.KafkaConfig.(KafkaConfig.scala:1127)
Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: at 
kafka.server.KafkaConfig.(KafkaConfig.scala:989)
Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: at 
kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:969)
Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: at 
kafka.server.KafkaServerStartable$.fromProps(KafkaServerStartable.scala:28)
Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: at 
kafka.Kafka$.main(Kafka.scala:82)
Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: at 
kafka.Kafka.main(Kafka.scala)

 

we have also tried to make it to 2.1 but no change. If the version map is being 
keyed using shortVersion, shouldn't it match? This is the first time we are 
upgrading (from 1.1.0) and we have never had to change log message format. 

Please advise.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Rolling Upgrade failed after bumping version to 2.1 from 1.1

[M. Manna]

By investigating journalctl logs - I see that it says version 2.1 is not a
valid version? So what is a valid version, does anyone know?

Thanks,

On Tue, 29 Jan 2019 at 11:37, M. Manna  wrote:

> Hello,
>
> I have recently upgraded to 2.12-2.1.0 from 2.11-1.1.0. After bumping
> inter.broker.protocol.version to 2.1 (and doing 2 bleed-and-restart from
> 1.1), my services are all failing to start. Slightly clueless at this stage
> regarding what to do other than reverting.
>
> I followed the steps in main site, and didn't use log message version
> change (since I am doing the uprade from 1.1.0 to 2.1).
>
> Could anyone please help ?
>
> Thanks,
>

Rolling Upgrade failed after bumping version to 2.1 from 1.1

[M. Manna]

Hello,

I have recently upgraded to 2.12-2.1.0 from 2.11-1.1.0. After bumping
inter.broker.protocol.version to 2.1 (and doing 2 bleed-and-restart from
1.1), my services are all failing to start. Slightly clueless at this stage
regarding what to do other than reverting.

I followed the steps in main site, and didn't use log message version
change (since I am doing the uprade from 1.1.0 to 2.1).

Could anyone please help ?

Thanks,

Kafka Version Detection for Rolling Upgrade

[M. Manna]

Hello,

I am struggling a little bit to do the rolling upgrade. My confusion is
mainly where the version numbers should be. I am verifying my steps using
main site guidelines. Any help would be appreciated

https://kafka.apache.org/documentation/#ecosystem

I am upgrading from *2.11-1.1.0* to *2.12-2.1.0. *I have 3 brokers.Could
someone confirm that my versions are correct.

Current version: 1.1
New version: 2.1 (obtained from kafka-version.sh --version).

Also, I don't need to override the log message format since it's 1.1.0 to
2.1.0, is it still correct?

Thanks,

Re: [VOTE] KIP-369 Alternative Partitioner to Support "Always Round-Robin" Selection

[M. Manna]

Hello,

I am trying to revive this thread. I only got 1 binding vote so far.

Please feel free to revisit and comment here.

Thanks,

On Thu, 25 Oct 2018 at 00:15, M. Manna  wrote:

> Hey IJ,
>
> Thanks for your interest in the KIP.
>
> My point was simply that the round-robin should happen even if the key is
> not null. As for the importance of key in our case, we treat the key as
> metadata. Each key is composed of certain info which are parsed by our
> consumer thread. We will then determine whether it's an actionable message
> (e.g. process it), or a loopback(ignore it). You could argue, "Why not
> append this metadata with the record and parse it there?". But that means
> the following:
>
> 1) I'm always passing null key to achieve this - I would like to pass
> Null/Not-Null/Other key i.e. flexibility
> 2) Suppose the message size is 99 KB and and max message bytes allowed is
> 100K. Now prefixing metadata with message results into the actual message
> being 101K. This will fail at producer level and cause a retry/log this in
> our DB for future pickup.
>
> To avoid all these, we are simply proposing this new partitioner class.
> but all Kafka new releases will still have DefaultPartitioner as default,
> unless they change the prop file to use our new class.
>
> Regards,
>
> On Sun, 21 Oct 2018 at 04:05, Ismael Juma  wrote:
>
>> Thanks for the KIP. Can you please elaborate on the need for the key in
>> this case? The KIP simply states that the key is needed for metadata, but
>> doesn't give any more details.
>>
>> Ismael
>>
>> On Tue, Sep 4, 2018 at 3:39 AM M. Manna  wrote:
>>
>> > Hello,
>> >
>> > I have made necessary changes as per the original discussion thread, and
>> > would like to put it for votes.
>> >
>> > Thank you very much for your suggestion and guidance so far.
>> >
>> > Regards,
>> >
>>
>

Re: Kafka official docker image

[M. Manna]

There should be confluent enterprise image-it should work without control
centre etc. and basic open source items should stay the same.

Could you kindly check ?

On Tue, 15 Jan 2019 at 23:06, Олег Иванов  wrote:

> Hi,
>
> Could you please create an official docker image of kafka? There are a lot
> custom images in the dockerhub, but company's security policy allows only
> official images.
>
> Thanks!
>

Re: [VOTE] KIP-379: Multiple Consumer Group Management

[M. Manna]

Hi,

Thanks for this KIP. Could you kindly clarify why CSV format is useful, but
not anything else?

CSV format is ancient and the only reason it keep existing is various
application is because legacy applications aren't moving away from using
them. Would you would consider JSON or YAML?

Also, if you think about the kafka-reassign-partitions - it's also using
JSON, not CSV. That is my only point. However, if majority feels that it's
not
an issue I believe it's a team decision after all :).

Thanks,


On Mon, 14 Jan 2019 at 15:06, Gwen Shapira  wrote:

> +1. Thanks, that will be very helpful.
>
> On Mon, Jan 14, 2019, 4:43 AM Alex D 
> > Hello guys,
> >
> > We need your VOTES for the KIP-379: Multiple Consumer Group Management.
> >
> > KIP-379:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-379%3A+Multiple+Consumer+Group+Management
> >
> > PR: https://github.com/apache/kafka/pull/5726/
> >
> > Let's start the voting session.
> >
> > Thank you,
> >
> > Alex Dunayevsky
> >
>

Re: Windows Testing

[M. Manna]

There’s no specific test or setup for Windows. There are issues with file
cleanup on windows, so your best bet is to run it on containers or Linux.

Thanks,

On Tue, 18 Dec 2018 at 17:06, Jacob Braaten  wrote:

> Hello,
>
> Quick question about your testing setup. I can't find where I read this
> before and it may be incorrect so I wanted to check with you.
>
> Do you guys do any stress or integration testing in windows environments?
> I'm fairly certain I read somewhere that you only do testing in linux
> environments, but I wanted to be sure.
>
> Thanks,
> Jake
>

Re: [VOTE] KIP-369 Alternative Partitioner to Support "Always Round-Robin" Selection

[M. Manna]

Hey IJ,

Thanks for your interest in the KIP.

My point was simply that the round-robin should happen even if the key is
not null. As for the importance of key in our case, we treat the key as
metadata. Each key is composed of certain info which are parsed by our
consumer thread. We will then determine whether it's an actionable message
(e.g. process it), or a loopback(ignore it). You could argue, "Why not
append this metadata with the record and parse it there?". But that means
the following:

1) I'm always passing null key to achieve this - I would like to pass
Null/Not-Null/Other key i.e. flexibility
2) Suppose the message size is 99 KB and and max message bytes allowed is
100K. Now prefixing metadata with message results into the actual message
being 101K. This will fail at producer level and cause a retry/log this in
our DB for future pickup.

To avoid all these, we are simply proposing this new partitioner class. but
all Kafka new releases will still have DefaultPartitioner as default,
unless they change the prop file to use our new class.

Regards,

On Sun, 21 Oct 2018 at 04:05, Ismael Juma  wrote:

> Thanks for the KIP. Can you please elaborate on the need for the key in
> this case? The KIP simply states that the key is needed for metadata, but
> doesn't give any more details.
>
> Ismael
>
> On Tue, Sep 4, 2018 at 3:39 AM M. Manna  wrote:
>
> > Hello,
> >
> > I have made necessary changes as per the original discussion thread, and
> > would like to put it for votes.
> >
> > Thank you very much for your suggestion and guidance so far.
> >
> > Regards,
> >
>

Re: Java cluster vs Scala cluster

[M. Manna]

Not sure I got your question ?

Kafka is written mainly in Scala with additional admin functionality and
Client API available in Java. There are some subtle differences but they
are not specific to Kafka?

What is the context of your question?

On Fri, 19 Oct 2018 at 21:32,  wrote:

> Hi,
> what is difference between class cluster written in java and class cluster
> written in scala.
>
> Thanks
>

Re: [VOTE] KIP-369 Alternative Partitioner to Support "Always Round-Robin" Selection

[M. Manna]

Since this has gone quiet, could I prequest 1 more vote here - if anyone
thinks it's worth doing?

Thanks,



On Wed, 3 Oct 2018 at 16:14, M. Manna  wrote:

> Thanks for reminding me about the "Binding" vote Bill. I remember some
> people with non-binding vote, so jumped the gun a bit too early.
> We will wait for 2 more as you stated.
>
> Regards,
>
> On Wed, 3 Oct 2018 at 16:07, Bill Bejeck  wrote:
>
>> +1 from me.
>>
>> As for closing the vote, it needs to be open for a minimum of 72 and
>> requires three binding +1 votes.  Additionally, there needs more +1
>> binding
>> votes than -1 votes.  The description for the lazy majority vote process
>> is
>> described here
>> https://cwiki.apache.org/confluence/display/KAFKA/Bylaws#Bylaws-Approvals
>> .
>>
>> I haven't been tracking the vote results, but from what I can see in the
>> voting thread, this KIP still requires two more +1 binding votes.
>>
>> HTH,
>> BIll
>>
>> On Wed, Oct 3, 2018 at 8:58 AM M. Manna  wrote:
>>
>> > Since this has been open for a while, I am assuming that it's good to
>> go?
>> >
>> > if so, I will update the KIP page - and start coding this. I would
>> prefer
>> > re-using existing tests written for DefaultPartitioner, so that we don't
>> > need to write new tests.
>> >
>> > Regards,
>> >
>> > On Sun, 30 Sep 2018 at 19:34, Matthias J. Sax 
>> > wrote:
>> >
>> > > +1 (binding)
>> > >
>> > > @Abhimanyu: can you please update the table in
>> > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
>> > > and add a link to the KIP. Thanks.
>> > >
>> > > -Matthias
>> > >
>> > > On 9/4/18 9:56 PM, Abhimanyu Nagrath wrote:
>> > > > +1
>> > > >
>> > > > On Wed, Sep 5, 2018 at 2:39 AM Magesh Nandakumar <
>> mage...@confluent.io
>> > >
>> > > > wrote:
>> > > >
>> > > >> +1 ( non-binding)
>> > > >>
>> > > >> On Tue, Sep 4, 2018 at 3:39 AM M. Manna 
>> wrote:
>> > > >>
>> > > >>> Hello,
>> > > >>>
>> > > >>> I have made necessary changes as per the original discussion
>> thread,
>> > > and
>> > > >>> would like to put it for votes.
>> > > >>>
>> > > >>> Thank you very much for your suggestion and guidance so far.
>> > > >>>
>> > > >>> Regards,
>> > > >>>
>> > > >>
>> > > >
>> > >
>> > >
>> >
>>
>

Re: [VOTE] KIP-369 Alternative Partitioner to Support "Always Round-Robin" Selection

[M. Manna]

Thanks for reminding me about the "Binding" vote Bill. I remember some
people with non-binding vote, so jumped the gun a bit too early.
We will wait for 2 more as you stated.

Regards,

On Wed, 3 Oct 2018 at 16:07, Bill Bejeck  wrote:

> +1 from me.
>
> As for closing the vote, it needs to be open for a minimum of 72 and
> requires three binding +1 votes.  Additionally, there needs more +1 binding
> votes than -1 votes.  The description for the lazy majority vote process is
> described here
> https://cwiki.apache.org/confluence/display/KAFKA/Bylaws#Bylaws-Approvals.
>
> I haven't been tracking the vote results, but from what I can see in the
> voting thread, this KIP still requires two more +1 binding votes.
>
> HTH,
> BIll
>
> On Wed, Oct 3, 2018 at 8:58 AM M. Manna  wrote:
>
> > Since this has been open for a while, I am assuming that it's good to go?
> >
> > if so, I will update the KIP page - and start coding this. I would prefer
> > re-using existing tests written for DefaultPartitioner, so that we don't
> > need to write new tests.
> >
> > Regards,
> >
> > On Sun, 30 Sep 2018 at 19:34, Matthias J. Sax 
> > wrote:
> >
> > > +1 (binding)
> > >
> > > @Abhimanyu: can you please update the table in
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > > and add a link to the KIP. Thanks.
> > >
> > > -Matthias
> > >
> > > On 9/4/18 9:56 PM, Abhimanyu Nagrath wrote:
> > > > +1
> > > >
> > > > On Wed, Sep 5, 2018 at 2:39 AM Magesh Nandakumar <
> mage...@confluent.io
> > >
> > > > wrote:
> > > >
> > > >> +1 ( non-binding)
> > > >>
> > > >> On Tue, Sep 4, 2018 at 3:39 AM M. Manna  wrote:
> > > >>
> > > >>> Hello,
> > > >>>
> > > >>> I have made necessary changes as per the original discussion
> thread,
> > > and
> > > >>> would like to put it for votes.
> > > >>>
> > > >>> Thank you very much for your suggestion and guidance so far.
> > > >>>
> > > >>> Regards,
> > > >>>
> > > >>
> > > >
> > >
> > >
> >
>

Re: [VOTE] KIP-369 Alternative Partitioner to Support "Always Round-Robin" Selection

[M. Manna]

Since this has been open for a while, I am assuming that it's good to go?

if so, I will update the KIP page - and start coding this. I would prefer
re-using existing tests written for DefaultPartitioner, so that we don't
need to write new tests.

Regards,

On Sun, 30 Sep 2018 at 19:34, Matthias J. Sax  wrote:

> +1 (binding)
>
> @Abhimanyu: can you please update the table in
>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> and add a link to the KIP. Thanks.
>
> -Matthias
>
> On 9/4/18 9:56 PM, Abhimanyu Nagrath wrote:
> > +1
> >
> > On Wed, Sep 5, 2018 at 2:39 AM Magesh Nandakumar 
> > wrote:
> >
> >> +1 ( non-binding)
> >>
> >> On Tue, Sep 4, 2018 at 3:39 AM M. Manna  wrote:
> >>
> >>> Hello,
> >>>
> >>> I have made necessary changes as per the original discussion thread,
> and
> >>> would like to put it for votes.
> >>>
> >>> Thank you very much for your suggestion and guidance so far.
> >>>
> >>> Regards,
> >>>
> >>
> >
>
>

Re: Help me to get Started !

[M. Manna]

This is it.

https://cwiki.apache.org/confluence/display/KAFKA/Index

Keep browsing through all pages.

On Sat, 22 Sep 2018 at 11:56, $uMe$h <1cool.1...@gmail.com> wrote:

> I unable to find dev wiki link.
> Can anyone point me to the dev wiki link ?
>
> On Sun 23 Sep, 2018, 12:02 AM $uMe$h, <1cool.1...@gmail.com> wrote:
>
> > Thanks a lot !
> >
> > I will start doing the same as suggested !
> >
> > On Sat 22 Sep, 2018, 11:58 PM M. Manna,  wrote:
> >
> >> You are on the same boat as I was a year ago.
> >>
> >> Before you do anything, try to visit wiki of apache Kafka. Also, try to
> >> visit JIRA and see a list of all minor bugs being raised. These are not
> >> necessarily blockers, but useful to understand why something should
> >> change.
> >> From there you should be able to reproduce the issue, and follow the
> >> workflow to learn about different components e.g. client, admin, core,
> >> etc.
> >>
> >>
> >> An old fashioned, but useful way to also start is to debug the startup
> of
> >> Kafka broker, and see how different worker threads are being scheduled.
> >>
> >>
> >>
> >> On Sat, 22 Sep 2018 at 11:24, $uMe$h <1cool.1...@gmail.com> wrote:
> >>
> >> > Hi all,
> >> >
> >> > Would like to start contributing to Kafka project.
> >> > I am new to Scala and have good knowledge in java.
> >> >
> >> > Please help me to get started.
> >> >
> >> > Thanks and Regards,
> >> > Sumesh
> >> >
> >>
> >
>

Re: Help me to get Started !

[M. Manna]

You are on the same boat as I was a year ago.

Before you do anything, try to visit wiki of apache Kafka. Also, try to
visit JIRA and see a list of all minor bugs being raised. These are not
necessarily blockers, but useful to understand why something should change.
>From there you should be able to reproduce the issue, and follow the
workflow to learn about different components e.g. client, admin, core, etc.


An old fashioned, but useful way to also start is to debug the startup of
Kafka broker, and see how different worker threads are being scheduled.



On Sat, 22 Sep 2018 at 11:24, $uMe$h <1cool.1...@gmail.com> wrote:

> Hi all,
>
> Would like to start contributing to Kafka project.
> I am new to Scala and have good knowledge in java.
>
> Please help me to get started.
>
> Thanks and Regards,
> Sumesh
>

[VOTE] KIP-369 Alternative Partitioner to Support "Always Round-Robin" Selection

[M. Manna]

Hello,

I have made necessary changes as per the original discussion thread, and
would like to put it for votes.

Thank you very much for your suggestion and guidance so far.

Regards,

Could not find or load main class kafka.Kafka (Eclipse Startup)

[M. Manna]

Hello,

I completed set up according to cwiki instruction -
https://cwiki.apache.org/confluence/display/KAFKA/Developer+Setup

I have also built all the sources and jars using `gradlew jarAll`

After that, I created a Scala application configuration in Eclipse to
debug. I provided the sources to log4j config file and server.properties
file location as arguments. After that, I also included the projects
"Clients", "Tools", and "Core" in the classpath. But the broker couldn't be
started in debug mode. I always get the following error.

Error: Could not find or load main class kafka.Kafka

Could someone guide me to fixing this issue? I was able to run this in the
past, but for some reason it has stopped working and I am struggling to get
this back up. I hope I've followed the correct process above.

Regards,

Re: KAFKA-1194

[M. Manna]

Hi Stephan,

https://jira.apache.org/jira/browse/KAFKA-7278

This might (according to Dong) be a better fix for the issue. Could you
kindly apply to patch to your kafka (assuming that your version is
reasonably close to the latest), and see if that helps?

Regards,

On Mon, 3 Sep 2018 at 02:27, Ismael Juma  wrote:

> Hi Stephane,
>
> Does https://github.com/apache/kafka/pull/4947/files fix it for you?
>
> Ismael
>
> On Sun, Sep 2, 2018 at 11:25 AM Stephane Maarek <
> steph...@simplemachines.com.au> wrote:
>
> > Hi,
> >
> > I've seen Dong has done some work on
> > https://issues.apache.org/jira/browse/KAFKA-7278 which is said from the
> > comments that it could have possibly fixed
> > https://issues.apache.org/jira/browse/KAFKA-1194.
> >
> > I tested and it is unfortunately not the case...
> > I have posted in KAFKA-1194 a way to reproduce the issue in a
> deterministic
> > way:
> > ===
> > C:\kafka_2.11-2.1.0-SNAPSHOT>bin\windows\kafka-server-start.bat
> > config\server.properties
> >
> > C:\kafka_2.11-2.1.0-SNAPSHOT>kafka-topics.bat --zookeeper 127.0.0.1:2181
> > --topic second_topic --create --partitions 3 --replication-factor 1
> >
> > C:\kafka_2.11-2.1.0-SNAPSHOT>kafka-console-producer.bat --broker-list
> > 127.0.0.1:9092 --topic second_topic
> > >hello
> > >world
> > >hello
> > >Terminate batch job (Y/N)? Y
> >
> > C:\kafka_2.11-2.1.0-SNAPSHOT>kafka-topics.bat --zookeeper 127.0.0.1:2181
> > --topic second_topic --delete
> > 
> >
> >
> > I usually wouldn't push for any issues for windows to be fixed, but this
> > one is triggered from basic CLI usage and triggers a full broker failure
> > that can't be recovered.
> > It actually completely destroys the faith of Kafka newcomers learning
> using
> > Windows (which are 50% of the users I have in my course).
> >
> > I'm happy to help out in any way I can to test any patch.
> >
> > Thanks
> > Stephane
> >
>

Re: [DISCUSS] KIP-369 Alternative Partitioner to Support "Always Round-Robin" Selection

[M. Manna]

Thanks for all your comments and taking it easy on me for my first KIP :)

 I am trying to check if it's okay for us to start a vote on this? As per
some recent comment I'll change the name to RoundRobinPartitioner.

I'll need to put some effort in writing Scala tests etc. since I'm a novice
with Scala.

Please let me know your thoughts, and I'll update the status accordingly
(and start working on the JIRA once it's approved).

Regards,

On Fri, 31 Aug 2018, 10:22 M. Manna,  wrote:

> Yes I’m more than happy to change it to a more appropriate name.
>
> The issue with RoundRobinPatitoner is that the DefaultPartitioner already
> has a Round-Robin associated to it. But if community doesn’t mind the name,
> I don’t either.
>
> Thanks for reading the KIP btw.
>
> Regards,
>
> On Fri, 31 Aug 2018 at 05:47, Magesh Nandakumar 
> wrote:
>
>> +1 for this. The only small suggestion would be to possibly call this
>> RondRobinPartitioner which makes the intent obvious.
>>
>> On Thu, Aug 30, 2018 at 5:31 PM Stephen Powis 
>> wrote:
>>
>> > Neat, this would be super helpful! I submitted this ages ago:
>> > https://issues.apache.org/jira/browse/KAFKA-
>> >
>> > On Fri, Aug 31, 2018 at 5:04 AM, Satish Duggana <
>> satish.dugg...@gmail.com>
>> > wrote:
>> >
>> > > +including both dev and user mailing lists.
>> > >
>> > > Hi,
>> > > Thanks for the KIP.
>> > >
>> > > "* For us, the message keys represent some metadata which we use to
>> > either
>> > > ignore messages (if a loop-back to the sender), or log some
>> > information.*"
>> > >
>> > > Above statement was mentioned in the KIP about how key value is used.
>> I
>> > > guess the topic is not configured to be compacted and you do not want
>> to
>> > > have partitioning based on that key. IMHO, it qualifies more as a
>> header
>> > > than a key. What do you think about building records with a specific
>> > header
>> > > and consumers to execute the logic whether to process or ignore the
>> > > messages based on that header value.
>> > >
>> > > Thanks,
>> > > Satish.
>> > >
>> > >
>> > > On Fri, Aug 31, 2018 at 1:32 AM, Satish Duggana <
>> > satish.dugg...@gmail.com>
>> > > wrote:
>> > >
>> > > > Hi,
>> > > > Thanks for the KIP.
>> > > >
>> > > > "* For us, the message keys represent some metadata which we use to
>> > > > either ignore messages (if a loop-back to the sender), or log some
>> > > > information.*"
>> > > >
>> > > > Above statement was mentioned in the KIP about how key value is
>> used. I
>> > > > guess the topic is not configured to be compacted and you do not
>> want
>> > to
>> > > > have partitioning based on that key. IMHO, it qualifies more as a
>> > header
>> > > > than a key. What do you think about building records with a specific
>> > > header
>> > > > and consumers to execute the logic whether to process or ignore the
>> > > > messages based on that header value.
>> > > >
>> > > > Thanks,
>> > > > Satish.
>> > > >
>> > > >
>> > > > On Fri, Aug 31, 2018 at 12:02 AM, M. Manna 
>> wrote:
>> > > >
>> > > >> Hi Harsha,
>> > > >>
>> > > >> thanks for reading the KIP.
>> > > >>
>> > > >> The intent is to use the DefaultPartitioner logic for round-robin
>> > > >> selection
>> > > >> of partition regardless of key type.
>> > > >>
>> > > >> Implementing Partitioner interface isn’t the issue here, you would
>> > have
>> > > to
>> > > >> do that anyway if  you are implementing your own. But we also want
>> > this
>> > > to
>> > > >> be part of formal codebase.
>> > > >>
>> > > >> Regards,
>> > > >>
>> > > >> On Thu, 30 Aug 2018 at 16:58, Harsha  wrote:
>> > > >>
>> > > >> > Hi,
>> > > >> >   Thanks for the KIP. I am trying to understand the intent of
>> > the
>> > > >> > KIP.  Is the use case you specified can't be achieved by
>> > implementing
>> > > >> the
>> > > >> > Partitioner interface here?
>> > > >> > https://github.com/apache/kafka/blob/trunk/clients/src/main/
>> > > >> java/org/apache/kafka/clients/producer/Partitioner.java#L28
>> > > >> > .
>> > > >> > Use your custom partitioner to be configured in your producer
>> > clients.
>> > > >> >
>> > > >> > Thanks,
>> > > >> > Harsha
>> > > >> >
>> > > >> > On Thu, Aug 30, 2018, at 1:45 AM, M. Manna wrote:
>> > > >> > > Hello,
>> > > >> > >
>> > > >> > > I opened a very simple KIP and there exists a JIRA for it.
>> > > >> > >
>> > > >> > > I would be grateful if any comments are available for action.
>> > > >> > >
>> > > >> > > Regards,
>> > > >> >
>> > > >>
>> > > >
>> > > >
>> > >
>> >
>>
>

Re: [jira] [Created] (KAFKA-7370) Enhance FileConfigProvider to read a directory

[M. Manna]

Do you want to raise a KIP and explain the motivation behind it ?

Please ignore if you’re doing that already.

Thanks ,

On Sat, 1 Sep 2018 at 00:15, Robert Yokota (JIRA)  wrote:

> Robert Yokota created KAFKA-7370:
> 
>
>  Summary: Enhance FileConfigProvider to read a directory
>  Key: KAFKA-7370
>  URL: https://issues.apache.org/jira/browse/KAFKA-7370
>  Project: Kafka
>   Issue Type: Improvement
>   Components: config
> Affects Versions: 2.0.0
> Reporter: Robert Yokota
> Assignee: Robert Yokota
>
>
> Currently FileConfigProvider can read a Properties file as a set of
> key-value pairs.  This enhancement is to augment FileConfigProvider so that
> it can also read a directory, where the file names are the keys and the
> corresponding file contents are the values.
>
> This will allow for easier integration with secret management systems
> where each secret is often an individual file, such as in Docker and
> Kubernetes.
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v7.6.3#76005)
>

Re: [DISCUSS] KIP-369 Alternative Partitioner to Support "Always Round-Robin" Selection

[M. Manna]

Yes I’m more than happy to change it to a more appropriate name.

The issue with RoundRobinPatitoner is that the DefaultPartitioner already
has a Round-Robin associated to it. But if community doesn’t mind the name,
I don’t either.

Thanks for reading the KIP btw.

Regards,

On Fri, 31 Aug 2018 at 05:47, Magesh Nandakumar 
wrote:

> +1 for this. The only small suggestion would be to possibly call this
> RondRobinPartitioner which makes the intent obvious.
>
> On Thu, Aug 30, 2018 at 5:31 PM Stephen Powis 
> wrote:
>
> > Neat, this would be super helpful! I submitted this ages ago:
> > https://issues.apache.org/jira/browse/KAFKA-
> >
> > On Fri, Aug 31, 2018 at 5:04 AM, Satish Duggana <
> satish.dugg...@gmail.com>
> > wrote:
> >
> > > +including both dev and user mailing lists.
> > >
> > > Hi,
> > > Thanks for the KIP.
> > >
> > > "* For us, the message keys represent some metadata which we use to
> > either
> > > ignore messages (if a loop-back to the sender), or log some
> > information.*"
> > >
> > > Above statement was mentioned in the KIP about how key value is used. I
> > > guess the topic is not configured to be compacted and you do not want
> to
> > > have partitioning based on that key. IMHO, it qualifies more as a
> header
> > > than a key. What do you think about building records with a specific
> > header
> > > and consumers to execute the logic whether to process or ignore the
> > > messages based on that header value.
> > >
> > > Thanks,
> > > Satish.
> > >
> > >
> > > On Fri, Aug 31, 2018 at 1:32 AM, Satish Duggana <
> > satish.dugg...@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > > Thanks for the KIP.
> > > >
> > > > "* For us, the message keys represent some metadata which we use to
> > > > either ignore messages (if a loop-back to the sender), or log some
> > > > information.*"
> > > >
> > > > Above statement was mentioned in the KIP about how key value is
> used. I
> > > > guess the topic is not configured to be compacted and you do not want
> > to
> > > > have partitioning based on that key. IMHO, it qualifies more as a
> > header
> > > > than a key. What do you think about building records with a specific
> > > header
> > > > and consumers to execute the logic whether to process or ignore the
> > > > messages based on that header value.
> > > >
> > > > Thanks,
> > > > Satish.
> > > >
> > > >
> > > > On Fri, Aug 31, 2018 at 12:02 AM, M. Manna 
> wrote:
> > > >
> > > >> Hi Harsha,
> > > >>
> > > >> thanks for reading the KIP.
> > > >>
> > > >> The intent is to use the DefaultPartitioner logic for round-robin
> > > >> selection
> > > >> of partition regardless of key type.
> > > >>
> > > >> Implementing Partitioner interface isn’t the issue here, you would
> > have
> > > to
> > > >> do that anyway if  you are implementing your own. But we also want
> > this
> > > to
> > > >> be part of formal codebase.
> > > >>
> > > >> Regards,
> > > >>
> > > >> On Thu, 30 Aug 2018 at 16:58, Harsha  wrote:
> > > >>
> > > >> > Hi,
> > > >> >   Thanks for the KIP. I am trying to understand the intent of
> > the
> > > >> > KIP.  Is the use case you specified can't be achieved by
> > implementing
> > > >> the
> > > >> > Partitioner interface here?
> > > >> > https://github.com/apache/kafka/blob/trunk/clients/src/main/
> > > >> java/org/apache/kafka/clients/producer/Partitioner.java#L28
> > > >> > .
> > > >> > Use your custom partitioner to be configured in your producer
> > clients.
> > > >> >
> > > >> > Thanks,
> > > >> > Harsha
> > > >> >
> > > >> > On Thu, Aug 30, 2018, at 1:45 AM, M. Manna wrote:
> > > >> > > Hello,
> > > >> > >
> > > >> > > I opened a very simple KIP and there exists a JIRA for it.
> > > >> > >
> > > >> > > I would be grateful if any comments are available for action.
> > > >> > >
> > > >> > > Regards,
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-369 Alternative Partitioner to Support "Always Round-Robin" Selection

[M. Manna]

Hey Bill,

Thanks for reading the KIP, much appreciated.

The reason we want it to be a separate Partitioner is because:

a) We don’t want to specify partition anywhere.

b) we want to be able to reuse what’s been done for NULL key in
DefaultPartitioner.

Using the constructor means we need to assign partition manually in the
code. I’m not sure if it I managed to clarify our need.

Also, this means no change in any existing code. It’s a new class in Kafka
trunk which doesn’t disrupt any exising operation.

Thanks,

On Thu, 30 Aug 2018 at 18:12, Bill Bejeck  wrote:

> Hi,
>
> NOTE: I sent this earlier, but that message just went to the dev list.  I'm
> including both users and dev now.
>
> Thanks for the KIP.
>
> Have you considered using the overloaded ProducerRecord constructor where
> you can specify the partition?   I mention this as an option as I
> encountered the same issue on a previous project and that is how we handled
> round-robin distribution with non-null keys.
>
> Would that suit your needs?
>
> Thanks,
> Bill
>
> On Thu, Aug 30, 2018 at 11:58 AM Harsha  wrote:
>
> > Hi,
> >   Thanks for the KIP. I am trying to understand the intent of the
> > KIP.  Is the use case you specified can't be achieved by implementing the
> > Partitioner interface here?
> >
> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/Partitioner.java#L28
> > .
> > Use your custom partitioner to be configured in your producer clients.
> >
> > Thanks,
> > Harsha
> >
> > On Thu, Aug 30, 2018, at 1:45 AM, M. Manna wrote:
> > > Hello,
> > >
> > > I opened a very simple KIP and there exists a JIRA for it.
> > >
> > > I would be grateful if any comments are available for action.
> > >
> > > Regards,
> >
>

Re: [DISCUSS] KIP-369: Alternative Partitioner to Support "Always Round-Robin" Selection

[M. Manna]

I just added you as a "Watcher". I can see the page via this link -
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=89070828

1) visited cwiki.apache.org
2) Opened "Apache Kafka" from Space Directory
3) Browsed to Kafka Improvement Proposals (or Design Proposals).
4) From the side bar menu - > clicked on 363 more child pages.
5) Found the KIP.

I used the KIP template described in the "Process" section to create this.
If something is not correct, please advise and I'll amend it.

Thanks,

On Thu, 30 Aug 2018 at 11:32, Eno Thereska  wrote:

> Hi there,
>
> I can't see this KIP on the KIP wiki, could you double check?
>
> Thanks
> Eno
>
> On Thu, Aug 30, 2018 at 9:56 AM, M. Manna  wrote:
>
> >  Hello,
> >
> > I opened a very simple KIP and there exists a JIRA for it.
> >
> > I have already sent it to users' list - but the mail thread doesn't seem
> to
> > get updated.
> >
> > I would be grateful if any comments are available for action.
> >
> > Regards,
> >
>

[DISCUSS] KIP-369: Alternative Partitioner to Support "Always Round-Robin" Selection

[M. Manna]

 Hello,

I opened a very simple KIP and there exists a JIRA for it.

I have already sent it to users' list - but the mail thread doesn't seem to
get updated.

I would be grateful if any comments are available for action.

Regards,

[jira] [Created] (KAFKA-7358) Extended Partitioner to Support "Always Round Robin" Selection

[M. Manna (JIRA)]

M. Manna created KAFKA-7358:
---

 Summary: Extended Partitioner to Support "Always Round Robin" 
Selection
 Key: KAFKA-7358
 URL: https://issues.apache.org/jira/browse/KAFKA-7358
 Project: Kafka
  Issue Type: Wish
  Components: clients
Reporter: M. Manna
Assignee: M. Manna


In my organisation, we have been using kafka as the basic publish-subscribe 
messaging system provider. Our goal is the send event-based messages reliably 
and securely, and perform data synchronisation based on the messages. For us, 
the message keys represent some metadata which we use to either ignore messages 
(if a loopback) or log some information. We have the following use case for 
messaging:

1) A Database transaction event takes place

2) The event is captured and messaged across 10 data centres all around the 
world.

3) A group of consumers (for each data centre with a unique consumer-group ID) 
are will process messages from their respective partitions. 1 consumer per 
partition.

Under the circumstances, we only need a guarantee that same message won't be 
sent to multiple partitions. Using DefaultPartitioner, we can achieve this only 
with NULL keys. But since we need keys for metadata, we cannot maintain 
"Round-robin" selection of partitions because a key hash will determine which 
partition to choose. We need both non-null key with round-robin partition 
selection for KafkaProducer.

We believe this solution is achievable and stable, because multiple consumer 
will not be consuming from the same partition as per Kafka's fundamental 
architecture. Hence, the ask for an extended partitioner.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Kafka Producer Partition Key Selection

[M. Manna]

Why can't we override the DefaultPartitioner, and simply override
paritition()  method, such that it will redistribute to all partitions in
round robin fashion.

Round-Robin partitioner and StickyAssignor (consumer) should work nicely
for any publish subscribe system.

On Wed, 29 Aug 2018 at 09:39, SenthilKumar K  wrote:

> Thanks Gaurav.  Did you notice side effect mentioned in this page :
>
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyisdatanotevenlydistributedamongpartitionswhenapartitioningkeyisnotspecified
> ?
>
>
> --Senthil
>
> On Wed, Aug 29, 2018 at 2:02 PM Gaurav Bajaj 
> wrote:
>
> > Hello Senthil,
> >
> > In our case we use NULL as message Key to achieve even distribution in
> > producer.
> > With that we were able to achieve very even distribution with that.
> > Our Kafka client version is 0.10.1.0 and Kafka broker version is 1.1
> >
> >
> > Thanks,
> > Gaurav
> >
> > On Wed, Aug 29, 2018 at 9:15 AM, SenthilKumar K 
> > wrote:
> >
> >> Hello Experts, We want to distribute data across partitions in Kafka
> >> Cluster.
> >>  Option 1 : Use Null Partition Key which can distribute data across
> >> paritions.
> >>  Option 2 :  Choose Key ( Random UUID ? ) which can help to distribute
> >> data
> >> 70-80%.
> >>
> >> I have seen below side effect on Confluence Page about sending null Keys
> >> to
> >> Producer. Is this still valid on newer version of Kafka Producer Lib ?
> >> Why is data not evenly distributed among partitions when a partitioning
> >> key
> >> is not specified?
> >>
> >> In Kafka producer, a partition key can be specified to indicate the
> >> destination partition of the message. By default, a hashing-based
> >> partitioner is used to determine the partition id given the key, and
> >> people
> >> can use customized partitioners also.
> >>
> >> To reduce # of open sockets, in 0.8.0 (
> >> https://issues.apache.org/jira/browse/KAFKA-1017), when the
> partitioning
> >> key is not specified or null, a producer will pick a random partition
> and
> >> stick to it for some time (default is 10 mins) before switching to
> another
> >> one. So, if there are fewer producers than partitions, at a given point
> of
> >> time, some partitions may not receive any data. To alleviate this
> problem,
> >> one can either reduce the metadata refresh interval or specify a message
> >> key and a customized random partitioner. For more detail see this thread
> >>
> >>
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201310.mbox/%3CCAFbh0Q0aVh%2Bvqxfy7H-%2BMnRFBt6BnyoZk1LWBoMspwSmTqUKMg%40mail.gmail.com%3E
> >>
> >> Pls advise on Choosing Partition Key which should not have side effects.
> >>
> >> --Senthil
> >>
> >
> >
>

Re: Can't start kafka

[M. Manna]

You know people are still trying to find your error message, right :) ?

On 14 August 2018 at 15:19, Mohamed Salah Baccouche  wrote:

> hello , i was trying to connect kafka and cassandra when i finally
> succeeded kafka stopped and i got this error message that i sent you with
> the email , and from then kafka wouldn't start , please help me
>
> Thank you
>
>

Re: Running Kafka locally - no logs

[M. Manna]

I have managed to load the broker from within Eclipse Scala IDE. I only
needed to put server.properties and -Dlog4j.configuration in program
arguments area.

How did you set up classpath? You need to set up classpath such that it can
find all the jars necessary. And you shouldn’t have to remove any jar. For
simplicity, could you r move everything and keep the jars which are only
found under your KAFKA_HOME/libs folder? And try again.

Regards,



On Wed, 23 May 2018 at 07:11, Lerh Chuan Low  wrote:

> Hi Kafka devs,
>
> I've been trying to debug Kafka locally. I've been following the guides
> available on the website/confluence, which are:
>
> - Installing gradle (already have Java and Scala)
> - git clone kafka
> - cd git repository
> - gradle
> - ./gradlew idea
>
> I then open IntellIJ IDEA and open the kafka repository. It works! The
> only problem is that it refuses to log anything.
>
>
> ​
> I have verified that Log4j JAR is in the classpath
>
>
>
> In this case I've purposely removed log4j12:1.7.21 because some
> Stackoverflow answers suggested using 1 and only 1 logging framework JAR.
> It still outputs the same error message.
>
> I also tried to specify the configurationFile explicitly using
> -Dlog4j.configurationFile=config/log4j.properties (even though Log4J should
> default to an Appender otherwise), it didn't work still (which is to be
> expected I guess since SLF4J can't even find the Log4J implementation).
>
> Not sure if anyone has any ideas or ran into this issue when running Kafka
> from IntelliJ locally? I've been stuck on it for quite some time. If I
> debug and check LoggerFactory#findPossibleStaticLoggerBinderPathSet(), it's
> not able to find the SLF4J implementation. Is there anything else I need to
> setup, or does Kafka log somewhere else?
>
> Thanks!!
>
> Lerh
> ​
>

Re: Making sure all of you know about Kafka Summit

[M. Manna]

I guess It's kinda late since I am already in transit for work.

Is there any plan to do something in Europe e.g. London or some other place?

On 18 Aug 2017 4:41 pm, "Gwen Shapira"  wrote:

> Hi,
>
> I figured everyone in this list kinda cares about Kafka, so just making
> sure you all know.
>
> Kafka Summit SF happens in about a week:
> https://kafka-summit.org/events/kafka-summit-sf/
>
> August 28 in San Francisco. It is not too late to register.
>
> The talks are pretty great (and very relevant to everyone here) - from new
> Kafka features to how different companies run Kafka in different ways.
>
> Even better, most of the Apache Kafka committers will attend. So if you
> ever wanted to discuss Kafka internals with the people who are writing this
> thing and reviewing every line of code, this is a good opportunity. Also a
> good time to discuss your favorite KIPs in person and argue about that
> JIRA.
>
> There will also be good food, parties and fun swag.
>
> As a member of the mailing list and a fellow procrastinator you can use the
> code kafpcf17 for 30% off your conference pass.
>
> Looking forward to see you there and to toast for Apache Kafka 1.0 :)
>
> Gwen
>

Re: Usage of SSL_ENDPOINT_IDENTIFICATION_ALGORITHM_CONFIG

[M. Manna]

Thanks got it :)

On 10 Aug 2017 7:02 am, "Manikumar"  wrote:

> check here:
> https://github.com/apache/kafka/blob/trunk/clients/src/
> main/java/org/apache/kafka/common/security/ssl/SslFactory.java#L84
>
> On Thu, Aug 10, 2017 at 3:49 AM, M. Manna  wrote:
>
> > Hello,
> >
> > I have been trying to find the usage of this property within Kafka
> source.
> > All I am trying to understand is how this is looked up for hostname
> > verification.
> >
> > I used notepad++ to search for this and only link was in
> KafkaConfig.scala
> > file - after that I cannot see this scala property being used anywhere
> > directly.
> >
> > Could anyone please tell me the names of some source files which is using
> > this for hostname verification? It will be highly appreciated.
> >
> > Kindest Regards,
> >
>

Usage of SSL_ENDPOINT_IDENTIFICATION_ALGORITHM_CONFIG

[M. Manna]

Hello,

I have been trying to find the usage of this property within Kafka source.
All I am trying to understand is how this is looked up for hostname
verification.

I used notepad++ to search for this and only link was in KafkaConfig.scala
file - after that I cannot see this scala property being used anywhere
directly.

Could anyone please tell me the names of some source files which is using
this for hostname verification? It will be highly appreciated.

Kindest Regards,

Re: Kafka issues during Developement

[M. Manna]

This is a know issue. You cannot delete shared (or opened files) on
Windows, but UNIX/Linux is okay.

Please read Kafka online documentation on WIndows usage. A relevant jira
case is KAFKA-1194.



On 17 July 2017 at 11:31, Dharma Raj  wrote:

> Hi,
>
> I am using kafka as the messaging queue for my application which will run
> on the windows platform. I am facing issues during the log file deletion
> with following erros.
> Also i found some disclaimer is available in the kafka documentation to use
> kafka in windows platform
>
>
>
> Caused by: java.nio.file.FileSystemException:
> \tmp\kafka-logs\*TopicName*-2\7262.log
> -> \tmp\kafka-logs\TopicName -2\7262.log.deleted: The
> process cannot access the file because it is being used by another process.
>
>
>
> Uncaught exception in scheduled task 'kafka-log-retention'
> (kafka.utils.KafkaScheduler)
>
> kafka.common.KafkaStorageException: Failed to change the log file suffix
> from  to .deleted for log segment 7262
>
>
>
> Is it safe to go with kafka in windows platform or not?
>
> Can you please help me on this, my development work got halted.
>
> Thanks
> Dharmaraj.G
>

Re: Delete Kafka Topic After Enable Ranger Plugin

[M. Manna]

Hi,

You got any log for this? Also, which platform are you running this on?

KR,

On Fri, 14 Jul 2017 at 11:26 pm, amazon  wrote:

> Hi:
>
>  Recently I develop some applications about kafka under ranger. But
> when I set enable ranger kafka plugin I can not delete kafka topic
> completely even though set 'delete.topic.enable=true'. And I find when
> enable ranger kafka plugin it must be authrized. How can I delete kafka
> topic completely under ranger. Thank you.
>
> --
> 格律诗
> 祝您工作顺利
>
>
>

Re: Clarification on KafkaConsumer manual partition assignment

[M. Manna]

"A consumer doesn't need to be part of a consumer group"
- where do you see this documented please ?

On 13 Jul 2017 6:36 pm, "Paolo Patierno"  wrote:

Assigning partitions manually has no relation with consumer groups. I mean
... a consumer doesn't need to be part of a consumer group (so specifying
group.id) for having a partition assigned manually.

From: venkata sastry akella 
Sent: Thursday, July 13, 2017 7:09:33 PM
To: dev@kafka.apache.org
Subject: Clarification on KafkaConsumer manual partition assignment

KafkaConsumer API doc has the following statement.

"Note that it isn't possible to mix manual partition assignment (i.e. using
assign
)
with dynamic partition assignment through topic subscription (i.e. using
subscribe

)."

Question:  Does this statement applies to only one consumer group  or
multiple consumer groups ?
Meaning, can one consumer group has manual assignment and other consumer
group has automatic assignment  ?   OR if atleast one consumer group has
manual assignment, then automatic assignment doesnt work for any other
consumer group also ?

Thanks for clarifying this.

[jira] [Created] (KAFKA-5583) Provide a "OS independent" file rename and delete mechanism

[M. Manna (JIRA)]

M. Manna created KAFKA-5583:
---

 Summary: Provide a "OS independent" file rename and delete 
mechanism
 Key: KAFKA-5583
 URL: https://issues.apache.org/jira/browse/KAFKA-5583
 Project: Kafka
  Issue Type: Improvement
  Components: clients, core
Affects Versions: 0.10.2.1
 Environment: Windows
Reporter: M. Manna


This is related to 
[KAFKA-1194|https://issues.apache.org/jira/browse/KAFKA-1194] and specific to 
Windows. I thought it would be better to address this as a KIP item since it's 
OS platform specific.

Quite a lot of unit tests run on Windows platform fails after gradle build 
because of the same. Could I please request you to put some thought whether 
considering a platform-independent way of handling file renaming and deletion? 






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)