[DISCUSS] KAFKA-4169: KafkaProducer max.request.size message wrong when using compression

2023-08-24 Thread Pere Urbón Bayes
Hi,
  my name is Pere, Software Engineer turned Consultant for the last few
years, :-/ , but keep doing Kafka quite a lot of my day-to-day.

I know not a sexy issue, but last few days, I worked with customers who
again hit  https://issues.apache.org/jira/browse/KAFKA-4169 when using
producing with compression, getting a few uncomfortable
RecordTooLargeExceptions.

While I see and agree with the community, this issue is not that important,
it keeps annoying starters, so I thought to try giving it a try to do
something here.

I updated the issue with some comments of mine, but in a nutshell, unless
I'm missing something, it certainly could be. I think the max.request.size
config and the related check are wrong and problematic, Jorge already
mentioned that in the ticket. In my opinion, it mixes requests (batches)
with single messages, and it does not take into account any compression. I
can understand when people get confused, and so do I from time to time.

Before starting a KIP, and writing a ton I wanted to drop here a message
and see how you feel about possible solutions I have thought. To my
understanding, all might need a KIP, but please let me know if I got it
wrong.

* As Jorge proposed in the ticket, use the CompressionRateEstimator to
complement the existing check and be more accurate for single messages than
it is right now. All with having in mind that it will be working with an
estimator. This will require, in my mind to add a new configuration for a
single message size, isn't it?

* Remove completely the fail-fast check earlier mentioned and is currently
in place. Introduce a new check at the tryAppend methods within the
RecordAccumulator. The check-in this case will be using, as in option 1,
the CompressionRateEstimator. In this case, I see the current variable as
being more accurate, as we will be doing request level (ProducerBatch)
level checks.

What do you think?

Looking forward to help out,


-- 
Pere Urbon-Bayes
Software Architect
https://twitter.com/purbon
https://www.linkedin.com/in/purbon/


Re: [ANNOUNCE] New committer: Stanislav Kozlovski

2023-01-18 Thread Pere Urbón Bayes
Congrats Stan,
  very well deserved.

-- Pere

On Wed, Jan 18, 2023 at 7:51 AM Yash Mayya  wrote:

> Congratulations, Stan!
>
> On Tue, Jan 17, 2023 at 9:20 PM Jun Rao  wrote:
>
> > Hi, Everyone,
> >
> > The PMC of Apache Kafka is pleased to announce a new Kafka committer
> > Stanislav Kozlovski.
> >
> > Stan has been contributing to Apache Kafka since June 2018. He made
> various
> > contributions including the following KIPs.
> >
> > KIP-455: Create an Administrative API for Replica Reassignment
> > KIP-412: Extend Admin API to support dynamic application log levels
> >
> > Congratulations, Stan!
> >
> > Thanks,
> >
> > Jun (on behalf of the Apache Kafka PMC)
> >
>


-- 
Pere Urbon-Bayes
Software Architect
https://twitter.com/purbon
https://www.linkedin.com/in/purbon/


[DISCUSS] KIP-703: Add a metrics for reporting idle connections closed

2021-01-08 Thread Pere Urbón Bayes
Hi,
  I would love to kick start a discussion for KIP-703, the proposal is
available
https://cwiki.apache.org/confluence/display/KAFKA/KIP-703%3A+Add+a+metrics+for+reporting+idle+connections+closed

It is a minor change but certainly helps to improve the ops day2day life.

What do you think? Looking forward to hearing your thoughts and feedback.

-- 
Pere Urbon-Bayes
Software Architect
https://twitter.com/purbon
https://www.linkedin.com/in/purbon/


Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication

2019-09-12 Thread Pere Urbón Bayes
True, the number is duplicated.

  do you know how can we get the problem fix? I was not aware, I missed,
sorry the need to add the KIP to the table.

-- Pere

Missatge de Jorge Esteban Quilcate Otoya  del dia
dt., 3 de set. 2019 a les 13:43:

> Hi Pere,
>
> Have you add your KIP to the list here
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals?
>
> I found the KIP number assigned to another.
>
>
>
> On Mon, Sep 2, 2019 at 2:23 PM Pere Urbón Bayes 
> wrote:
>
>> Thanks for your time Harsha,
>>anyone else with comments? looking forward to hearing from you.
>>
>> Stupid question: when do you move from discussion to vote?
>>
>> Missatge de Harsha Chintalapani  del dia dv., 30 d’ag.
>> 2019 a les 21:59:
>>
>> > Thanks Pere. KIP looks good to me.
>> > -Harsha
>> >
>> >
>> > On Fri, Aug 30, 2019 at 10:05 AM, Pere Urbón Bayes <
>> pere.ur...@gmail.com>
>> > wrote:
>> >
>> >> Not really,
>> >>   my idea is to keep the JAAS parameter, so people don't see major
>> >> changes. But if you pass a properties file, then this takes precedence
>> over
>> >> the other, with the idea that you can do sasl as well with the
>> properties
>> >> files.
>> >>
>> >> Makes sense?
>> >>
>> >> -- Pere
>> >>
>> >> Missatge de Harsha Chintalapani  del dia dv., 30
>> d’ag.
>> >> 2019 a les 19:00:
>> >>
>> >>> Hi Pere,
>> >>>   Thanks for the KIP. Enabling SSL for zookeeper for Kafka
>> makes
>> >>> sense.
>> >>> "The changes are planned to be introduced in a compatible way, by
>> >>> keeping the current JAAS variable precedence."
>> >>> Can you elaborate a bit here. If the user configures a JAAS file with
>> >>> Client section it will take precedence over zookeeper SSL configs?
>> >>>
>> >>> Thanks,
>> >>> Harsha
>> >>>
>> >>>
>> >>>
>> >>> On Fri, Aug 30, 2019 at 7:50 AM, Pere Urbón Bayes <
>> pere.ur...@gmail.com>
>> >>> wrote:
>> >>>
>> >>>> Hi,
>> >>>> quick question, I saw in another mail that 2.4 release is planned for
>> >>>> September. I think it would be really awesome to have this for this
>> >>>> release, do you think we can make it?
>> >>>>
>> >>>> -- Pere
>> >>>>
>> >>>> Missatge de Pere Urbón Bayes  del dia dj., 29
>> >>>> d’ag. 2019 a les 20:10:
>> >>>>
>> >>>> Hi,
>> >>>> this is my first KIP for a change in Apache Kafka, so I'm really need
>> >>>> to the process. Looking forward to hearing from you and learn the
>> best
>> >>>> ropes here.
>> >>>>
>> >>>> I would like to propose this KIP-515 to enable the ZookeeperClients
>> to
>> >>>> take full advantage of the TLS communication in the new Zookeeper
>> 3.5.5.
>> >>>> Specially interesting it the Zookeeper Security Migration, that
>> without
>> >>>> this change will not work with TLS, disabling users to use ACLs when
>> the
>> >>>> Zookeeper cluster use TLS.
>> >>>>
>> >>>> link:
>> >>>>
>> >>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication
>> >>>>
>> >>>> Looking forward to hearing from you on this,
>> >>>>
>> >>>> /cheers
>> >>>>
>> >>>> --
>> >>>> Pere Urbon-Bayes
>> >>>> Software Architect
>> >>>> http://www.purbon.com
>> >>>> https://twitter.com/purbon
>> >>>> https://www.linkedin.com/in/purbon/
>> >>>>
>> >>>> --
>> >>>> Pere Urbon-Bayes
>> >>>> Software Architect
>> >>>> http://www.purbon.com
>> >>>> https://twitter.com/purbon
>> >>>> https://www.linkedin.com/in/purbon/
>> >>>>
>> >>>
>> >>>
>> >>
>> >> --
>> >> Pere Urbon-Bayes
>> >> Software Architect
>> >> http://www.purbon.com
>> >> https://twitter.com/purbon
>> >> https://www.linkedin.com/in/purbon/
>> >>
>> >
>> >
>>
>> --
>> Pere Urbon-Bayes
>> Software Architect
>> http://www.purbon.com
>> https://twitter.com/purbon
>> https://www.linkedin.com/in/purbon/
>>
>

-- 
Pere Urbon-Bayes
Software Architect
http://www.purbon.com
https://twitter.com/purbon
https://www.linkedin.com/in/purbon/


Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication

2019-09-12 Thread Pere Urbón Bayes
Hi Ismael,
   fair point and totally reasonable point. I added the CLI part to bring
compatibility, but I can agree with not doing it right now per
compatibility reasons. However, I do have a question for you: You say most
of the CLI tools, is there anyone in your mind that you recall that
would break if the cluster uses a TLS activated Zookeeper cluster? Only one
I can probably think right now is the Zookeeper Security Migration tool,
right?

If I clean the others, would you agree to move the KIP forward?

-- Pere

Missatge de Ismael Juma  del dia dt., 3 de set. 2019 a
les 14:53:

> Hi Pere,
>
> Thanks for the KIP. With regards to the CLI tools, most of them support
> direct access to ZK for compatibility reasons and we encourage usage of the
> Kafka protocol instead. I am not sure we should be extending them as
> described in the KIP. What are your thoughts on that?
>
> Ismael
>
> On Thu, Aug 29, 2019 at 11:11 AM Pere Urbón Bayes 
> wrote:
>
>> Hi,
>>  this is my first KIP for a change in Apache Kafka, so I'm really need to
>> the process. Looking forward to hearing from you and learn the best ropes
>> here.
>>
>> I would like to propose this KIP-515 to enable the ZookeeperClients to
>> take
>> full advantage of the TLS communication in the new Zookeeper 3.5.5.
>> Specially interesting it the Zookeeper Security Migration, that without
>> this change will not work with TLS, disabling users to use ACLs when the
>> Zookeeper cluster use TLS.
>>
>> link:
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication
>>
>> Looking forward to hearing from you on this,
>>
>> /cheers
>>
>> --
>> Pere Urbon-Bayes
>> Software Architect
>> http://www.purbon.com
>> https://twitter.com/purbon
>> https://www.linkedin.com/in/purbon/
>>
>

-- 
Pere Urbon-Bayes
Software Architect
http://www.purbon.com
https://twitter.com/purbon
https://www.linkedin.com/in/purbon/


Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication

2019-09-02 Thread Pere Urbón Bayes
Thanks for your time Harsha,
   anyone else with comments? looking forward to hearing from you.

Stupid question: when do you move from discussion to vote?

Missatge de Harsha Chintalapani  del dia dv., 30 d’ag.
2019 a les 21:59:

> Thanks Pere. KIP looks good to me.
> -Harsha
>
>
> On Fri, Aug 30, 2019 at 10:05 AM, Pere Urbón Bayes 
> wrote:
>
>> Not really,
>>   my idea is to keep the JAAS parameter, so people don't see major
>> changes. But if you pass a properties file, then this takes precedence over
>> the other, with the idea that you can do sasl as well with the properties
>> files.
>>
>> Makes sense?
>>
>> -- Pere
>>
>> Missatge de Harsha Chintalapani  del dia dv., 30 d’ag.
>> 2019 a les 19:00:
>>
>>> Hi Pere,
>>>   Thanks for the KIP. Enabling SSL for zookeeper for Kafka makes
>>> sense.
>>> "The changes are planned to be introduced in a compatible way, by
>>> keeping the current JAAS variable precedence."
>>> Can you elaborate a bit here. If the user configures a JAAS file with
>>> Client section it will take precedence over zookeeper SSL configs?
>>>
>>> Thanks,
>>> Harsha
>>>
>>>
>>>
>>> On Fri, Aug 30, 2019 at 7:50 AM, Pere Urbón Bayes 
>>> wrote:
>>>
>>>> Hi,
>>>> quick question, I saw in another mail that 2.4 release is planned for
>>>> September. I think it would be really awesome to have this for this
>>>> release, do you think we can make it?
>>>>
>>>> -- Pere
>>>>
>>>> Missatge de Pere Urbón Bayes  del dia dj., 29
>>>> d’ag. 2019 a les 20:10:
>>>>
>>>> Hi,
>>>> this is my first KIP for a change in Apache Kafka, so I'm really need
>>>> to the process. Looking forward to hearing from you and learn the best
>>>> ropes here.
>>>>
>>>> I would like to propose this KIP-515 to enable the ZookeeperClients to
>>>> take full advantage of the TLS communication in the new Zookeeper 3.5.5.
>>>> Specially interesting it the Zookeeper Security Migration, that without
>>>> this change will not work with TLS, disabling users to use ACLs when the
>>>> Zookeeper cluster use TLS.
>>>>
>>>> link:
>>>>
>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication
>>>>
>>>> Looking forward to hearing from you on this,
>>>>
>>>> /cheers
>>>>
>>>> --
>>>> Pere Urbon-Bayes
>>>> Software Architect
>>>> http://www.purbon.com
>>>> https://twitter.com/purbon
>>>> https://www.linkedin.com/in/purbon/
>>>>
>>>> --
>>>> Pere Urbon-Bayes
>>>> Software Architect
>>>> http://www.purbon.com
>>>> https://twitter.com/purbon
>>>> https://www.linkedin.com/in/purbon/
>>>>
>>>
>>>
>>
>> --
>> Pere Urbon-Bayes
>> Software Architect
>> http://www.purbon.com
>> https://twitter.com/purbon
>> https://www.linkedin.com/in/purbon/
>>
>
>

-- 
Pere Urbon-Bayes
Software Architect
http://www.purbon.com
https://twitter.com/purbon
https://www.linkedin.com/in/purbon/


Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication

2019-08-30 Thread Pere Urbón Bayes
Not really,
  my idea is to keep the JAAS parameter, so people don't see major changes.
But if you pass a properties file, then this takes precedence over the
other, with the idea that you can do sasl as well with the properties files.

Makes sense?

-- Pere

Missatge de Harsha Chintalapani  del dia dv., 30 d’ag.
2019 a les 19:00:

> Hi Pere,
>   Thanks for the KIP. Enabling SSL for zookeeper for Kafka makes
> sense.
> "The changes are planned to be introduced in a compatible way, by keeping
> the current JAAS variable precedence."
> Can you elaborate a bit here. If the user configures a JAAS file with
> Client section it will take precedence over zookeeper SSL configs?
>
> Thanks,
> Harsha
>
>
>
> On Fri, Aug 30, 2019 at 7:50 AM, Pere Urbón Bayes 
> wrote:
>
>> Hi,
>> quick question, I saw in another mail that 2.4 release is planned for
>> September. I think it would be really awesome to have this for this
>> release, do you think we can make it?
>>
>> -- Pere
>>
>> Missatge de Pere Urbón Bayes  del dia dj., 29
>> d’ag. 2019 a les 20:10:
>>
>> Hi,
>> this is my first KIP for a change in Apache Kafka, so I'm really need to
>> the process. Looking forward to hearing from you and learn the best ropes
>> here.
>>
>> I would like to propose this KIP-515 to enable the ZookeeperClients to
>> take full advantage of the TLS communication in the new Zookeeper 3.5.5.
>> Specially interesting it the Zookeeper Security Migration, that without
>> this change will not work with TLS, disabling users to use ACLs when the
>> Zookeeper cluster use TLS.
>>
>> link:
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication
>>
>> Looking forward to hearing from you on this,
>>
>> /cheers
>>
>> --
>> Pere Urbon-Bayes
>> Software Architect
>> http://www.purbon.com
>> https://twitter.com/purbon
>> https://www.linkedin.com/in/purbon/
>>
>> --
>> Pere Urbon-Bayes
>> Software Architect
>> http://www.purbon.com
>> https://twitter.com/purbon
>> https://www.linkedin.com/in/purbon/
>>
>
>

-- 
Pere Urbon-Bayes
Software Architect
http://www.purbon.com
https://twitter.com/purbon
https://www.linkedin.com/in/purbon/


Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication

2019-08-30 Thread Pere Urbón Bayes
It helps as a beginner! thanks for sharing!

any tip for getting reviewers to take a look? :pray:, :pray: ;-)

-- Pere

Missatge de Colin McCabe  del dia dv., 30 d’ag. 2019 a
les 18:57:

> Hi Pere,
>
> It will depend on how quickly the KIP can be voted in, and then
> implemented.
>
> Manikumar Reddy is going to be the release manager for Apache Kafka
> 2.4.0.  Right now, the KIP freeze is scheduled for Sep 25, 2019.  A KIP
> must be accepted by this date in order to be part of 2.4.0.  (Being
> accepted doesn't mean it will be part of 2.4.0, though-- it also has to be
> implemented.)  Hope this helps.
>
> cheers,
> Colin
>
>
> On Fri, Aug 30, 2019, at 07:50, Pere Urbón Bayes wrote:
> > Hi,
> >  quick question, I saw in another mail that 2.4 release is planned for
> > September. I think it would be really awesome to have this for this
> > release, do you think we can make it?
> >
> > -- Pere
> >
> > Missatge de Pere Urbón Bayes  del dia dj., 29
> d’ag.
> > 2019 a les 20:10:
> >
> > > Hi,
> > >  this is my first KIP for a change in Apache Kafka, so I'm really need
> to
> > > the process. Looking forward to hearing from you and learn the best
> ropes
> > > here.
> > >
> > > I would like to propose this KIP-515 to enable the ZookeeperClients to
> > > take full advantage of the TLS communication in the new Zookeeper
> 3.5.5.
> > > Specially interesting it the Zookeeper Security Migration, that without
> > > this change will not work with TLS, disabling users to use ACLs when
> the
> > > Zookeeper cluster use TLS.
> > >
> > > link:
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication
> > >
> > > Looking forward to hearing from you on this,
> > >
> > > /cheers
> > >
> > > --
> > > Pere Urbon-Bayes
> > > Software Architect
> > > http://www.purbon.com
> > > https://twitter.com/purbon
> > > https://www.linkedin.com/in/purbon/
> > >
> >
> >
> > --
> > Pere Urbon-Bayes
> > Software Architect
> > http://www.purbon.com
> > https://twitter.com/purbon
> > https://www.linkedin.com/in/purbon/
> >
>


-- 
Pere Urbon-Bayes
Software Architect
http://www.purbon.com
https://twitter.com/purbon
https://www.linkedin.com/in/purbon/


Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication

2019-08-30 Thread Pere Urbón Bayes
Hi,
 quick question, I saw in another mail that 2.4 release is planned for
September. I think it would be really awesome to have this for this
release, do you think we can make it?

-- Pere

Missatge de Pere Urbón Bayes  del dia dj., 29 d’ag.
2019 a les 20:10:

> Hi,
>  this is my first KIP for a change in Apache Kafka, so I'm really need to
> the process. Looking forward to hearing from you and learn the best ropes
> here.
>
> I would like to propose this KIP-515 to enable the ZookeeperClients to
> take full advantage of the TLS communication in the new Zookeeper 3.5.5.
> Specially interesting it the Zookeeper Security Migration, that without
> this change will not work with TLS, disabling users to use ACLs when the
> Zookeeper cluster use TLS.
>
> link:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication
>
> Looking forward to hearing from you on this,
>
> /cheers
>
> --
> Pere Urbon-Bayes
> Software Architect
> http://www.purbon.com
> https://twitter.com/purbon
> https://www.linkedin.com/in/purbon/
>


-- 
Pere Urbon-Bayes
Software Architect
http://www.purbon.com
https://twitter.com/purbon
https://www.linkedin.com/in/purbon/


[DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication

2019-08-29 Thread Pere Urbón Bayes
Hi,
 this is my first KIP for a change in Apache Kafka, so I'm really need to
the process. Looking forward to hearing from you and learn the best ropes
here.

I would like to propose this KIP-515 to enable the ZookeeperClients to take
full advantage of the TLS communication in the new Zookeeper 3.5.5.
Specially interesting it the Zookeeper Security Migration, that without
this change will not work with TLS, disabling users to use ACLs when the
Zookeeper cluster use TLS.

link:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication

Looking forward to hearing from you on this,

/cheers

-- 
Pere Urbon-Bayes
Software Architect
http://www.purbon.com
https://twitter.com/purbon
https://www.linkedin.com/in/purbon/


Re: Permissions to create a KIP for KAFKA-8843

2019-08-29 Thread Pere Urbón Bayes
Working on it,
  looking forward to do my best with my first KIP,

I will send details for review as soon as I have them.

-- Pere

Missatge de Colin McCabe  del dia dj., 29 d’ag. 2019 a
les 19:06:

> On Thu, Aug 29, 2019, at 09:27, Pere Urbón Bayes wrote:
> > Thanks,
> >   yes I know the KIP-500, how realistic is to have it landing for 2.4? if
> > not really, end of the day we will need something like KAFKA-8843 for
> this
> > version.
> >
> > Looking forward to help out.
> >
>
> Hi Pere,
>
> Thanks for contributing.  KIP-500 will not be implemented in 2.4.
>
> KAFKA-8843 seems like a good improvement.  Since it involves changing
> command-line arguments, I think a KIP will be needed.  However, it should
> be a relatively short one if the change is straightforward.
>
> best,
> Colin
>
>
> > -- Pere
> >
> > Missatge de Guozhang Wang  del dia dj., 29 d’ag.
> 2019 a
> > les 17:35:
> >
> > > Hello Pere,
> > >
> > > Thanks for you interest in contributing to Kafka, I've added you to the
> > > contributors list and you should be able to create wiki pages now.
> > >
> > > BTW there's an on-going KIP-500 which aims at removing ZK dependency of
> > > Kafka recently; depending when it would be voted and be adopted, I
> think
> > > it's still worth doing KAFKA-8843.
> > >
> > > Guozhang
> > >
> > >
> > > On Thu, Aug 29, 2019 at 3:56 AM Pere Urbón Bayes  >
> > > wrote:
> > >
> > >> Hi,
> > >>   I would like to create, and start the process to fix KAFKA-8843, so
> as I
> > >> understand it I should first create a KIP.
> > >>
> > >> I registered to the wiki:
> > >>   email: pere.ur...@gmail.com
> > >>   user: pere.urbon
> > >>
> > >> would you be so nice to provide permissions to create the KIP?
> > >>
> > >> thanks a lot,
> > >>
> > >> --
> > >> Pere Urbon-Bayes
> > >> Software Architect
> > >> http://www.purbon.com
> > >> https://twitter.com/purbon
> > >> https://www.linkedin.com/in/purbon/
> > >>
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
> >
> > --
> > Pere Urbon-Bayes
> > Software Architect
> > http://www.purbon.com
> > https://twitter.com/purbon
> > https://www.linkedin.com/in/purbon/
> >
>


-- 
Pere Urbon-Bayes
Software Architect
http://www.purbon.com
https://twitter.com/purbon
https://www.linkedin.com/in/purbon/


Re: Permissions to create a KIP for KAFKA-8843

2019-08-29 Thread Pere Urbón Bayes
Thanks,
  yes I know the KIP-500, how realistic is to have it landing for 2.4? if
not really, end of the day we will need something like KAFKA-8843 for this
version.

Looking forward to help out.

-- Pere

Missatge de Guozhang Wang  del dia dj., 29 d’ag. 2019 a
les 17:35:

> Hello Pere,
>
> Thanks for you interest in contributing to Kafka, I've added you to the
> contributors list and you should be able to create wiki pages now.
>
> BTW there's an on-going KIP-500 which aims at removing ZK dependency of
> Kafka recently; depending when it would be voted and be adopted, I think
> it's still worth doing KAFKA-8843.
>
> Guozhang
>
>
> On Thu, Aug 29, 2019 at 3:56 AM Pere Urbón Bayes 
> wrote:
>
>> Hi,
>>   I would like to create, and start the process to fix KAFKA-8843, so as I
>> understand it I should first create a KIP.
>>
>> I registered to the wiki:
>>   email: pere.ur...@gmail.com
>>   user: pere.urbon
>>
>> would you be so nice to provide permissions to create the KIP?
>>
>> thanks a lot,
>>
>> --
>> Pere Urbon-Bayes
>> Software Architect
>> http://www.purbon.com
>> https://twitter.com/purbon
>> https://www.linkedin.com/in/purbon/
>>
>
>
> --
> -- Guozhang
>


-- 
Pere Urbon-Bayes
Software Architect
http://www.purbon.com
https://twitter.com/purbon
https://www.linkedin.com/in/purbon/


Permissions to create a KIP for KAFKA-8843

2019-08-29 Thread Pere Urbón Bayes
Hi,
  I would like to create, and start the process to fix KAFKA-8843, so as I
understand it I should first create a KIP.

I registered to the wiki:
  email: pere.ur...@gmail.com
  user: pere.urbon

would you be so nice to provide permissions to create the KIP?

thanks a lot,

-- 
Pere Urbon-Bayes
Software Architect
http://www.purbon.com
https://twitter.com/purbon
https://www.linkedin.com/in/purbon/


[Discuss] Question on KIP-298: Error Handling in Kafka Connect

2019-01-01 Thread Pere Urbón Bayes
Hi,
 a quick question on the KIP-298 Dead letter queue, as I read from the KIP
is only available for the Sink connectors.

While I know the challenges of defining a dead-letter queue for the
incoming messages, I wanted ask/discuss what is the sense in here for this,
do you completely discard the option?

I sort of see it useful for messages that where pulled from the source, but
somehow could not be ingested in Kafka, might be because of serialisation
for example.

What do you think?

-- 
Pere Urbon-Bayes
Software Architect
http://www.purbon.com
https://twitter.com/purbon
https://www.linkedin.com/in/purbon/


Re: Kafka tests on a remote cluster

2019-01-01 Thread Pere Urbón Bayes
Hi,
 if I understand your question properly you are aiming to validate failure
scenarios. I usually see this done for learning purposes, basically to
answer how Kafka would react to such and such situation? or to validate the
current setup / configuration an Organisation have.

For the first one, or as well the second one but not directly on the
cluster, colleagues of mine actually wrote
https://github.com/Dabz/kafka-boom-boom, this uses docker to simulate the
scenarios. I've actually seen this approach used often.

This days I would only run a chaos testing if supported by a tooling like
the Netflix Chaos Monkey (https://github.com/Netflix/chaosmonkey) or the
other tooling available for the cloud providers, Kubernetes or Cloud
Foundry. A nice list is available from
https://github.com/dastergon/awesome-chaos-engineering#notable-tools

Cheers

-- Pere




Missatge de Parviz deyhim  del dia dj., 27 de des. 2018 a
les 21:57:

> +dev@kafka.apache.org
>
> On Wed, Dec 26, 2018 at 8:53 PM Parviz deyhim  wrote:
>
> > Thanks fair points. Probably best if I simplify the question: How does
> > Kafka community run tests besides using mocked local Kafka components?
> > Surely there are tests to confirm different failure scenarios such as
> > losing a broker in a real clustered environment (multi node cluster with
> > Ip, port, hostnsmes and etc). The answer would be a good starting point
> for
> > me.
> >
> > On Wed, Dec 26, 2018 at 6:11 PM Stephen Powis 
> > wrote:
> >
> >> Without looking into how the integration tests work my best guess is
> >> within
> >> the context they were written to run in, it doesn't make sense to run
> them
> >> against a remote cluster.  The "internal" cluster is running the same
> >> code,
> >> so why require having to coordinate with an external dependency?
> >>
> >> For the use case you gave, and I'm not sure if tests exist that cover
> this
> >> behavior or not -- Running the brokers locally in the context of the
> tests
> >> mean that those tests have control over the brokers (IE shut them off,
> >> restart them, etc.. programmatically) and validate behavior.  To
> >> coordinate
> >> these operations on a remote broker would be significantly more
> difficult.
> >>
> >> Not sure this helps...but perhaps you're either asking the wrong
> questions
> >> or trying to go about solving your problem using the wrong set of tools?
> >> My gut feeling says if you want to do a full scale multi-server load /
> HA
> >> test, Kafka's test suite is not the best place to start.
> >>
> >> Stephen
> >>
> >>
> >>
> >> On Thu, Dec 27, 2018 at 10:53 AM Parviz deyhim 
> wrote:
> >>
> >> > Hi,
> >> >
> >> > I'm looking to see who has done this before and get some guidance. On
> >> > frequent basis I like to run basic tests on a remote Kafka cluster
> while
> >> > some random chaos/faults are being performed. In other words I like to
> >> run
> >> > chaos engineering tasks (network outage, disk outage, etc) and see how
> >> > Kafka behaves. For example:
> >> >
> >> > 1) bring some random Broker node down
> >> > 2) send 2000 messages
> >> > 3) consumes messages
> >> > 4) confirm there's no data loss
> >> >
> >> > My questions: I'm pretty sure most of the scenarios I'm looking to
> test
> >> > have been covered under Kafka's integration,unit and other existing
> >> tests.
> >> > What I cannot figure out is how to run those tests on a remote cluster
> >> vs.
> >> > a local one which the tests seems to run on. For example I like to run
> >> the
> >> > following command but the tests to be executed on a remote cluster of
> my
> >> > choice:
> >> >
> >> > ./gradlew cleanTest integrationTest
> >> >
> >> > Any guidance/help would be appreciated.
> >> >
> >> > Thanks
> >> >
> >>
> >
>


-- 
Pere Urbon-Bayes
Software Architect
http://www.purbon.com
https://twitter.com/purbon
https://www.linkedin.com/in/purbon/