Requesting Permission

[Brian Byrne]

Hello,

I'm requesting permission to the Kafka Wiki, specifically to create a KIP.
Wiki ID is 'bbyrne'.

Thanks,
Brian

Build failed in Jenkins: kafka-trunk-jdk8 #3905

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: add unsigned varint support (#7338)

[jason] MINOR: Cleanup scala warnings (#7335)

--
[...truncated 2.63 MB...]

org.apache.kafka.connect.transforms.ExtractFieldTest > testNullSchemaless 
STARTED

org.apache.kafka.connect.transforms.ExtractFieldTest > testNullSchemaless PASSED

org.apache.kafka.connect.transforms.util.NonEmptyListValidatorTest > 
testNullList STARTED

org.apache.kafka.connect.transforms.util.NonEmptyListValidatorTest > 
testNullList PASSED

org.apache.kafka.connect.transforms.util.NonEmptyListValidatorTest > 
testEmptyList STARTED

org.apache.kafka.connect.transforms.util.NonEmptyListValidatorTest > 
testEmptyList PASSED

org.apache.kafka.connect.transforms.util.NonEmptyListValidatorTest > 
testValidList STARTED

org.apache.kafka.connect.transforms.util.NonEmptyListValidatorTest > 
testValidList PASSED

org.apache.kafka.connect.transforms.ReplaceFieldTest > withSchema STARTED

org.apache.kafka.connect.transforms.ReplaceFieldTest > withSchema PASSED

org.apache.kafka.connect.transforms.ReplaceFieldTest > schemaless STARTED

org.apache.kafka.connect.transforms.ReplaceFieldTest > schemaless PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testSchemalessFieldConversion STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testSchemalessFieldConversion PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testConfigInvalidTargetType STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testConfigInvalidTargetType PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testSchemalessStringToTimestamp STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testSchemalessStringToTimestamp PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > testKey STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > testKey PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testSchemalessDateToTimestamp STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testSchemalessDateToTimestamp PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testSchemalessTimestampToString STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testSchemalessTimestampToString PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testSchemalessTimeToTimestamp STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testSchemalessTimeToTimestamp PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaTimestampToDate STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaTimestampToDate PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaTimestampToTime STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaTimestampToTime PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaTimestampToUnix STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaTimestampToUnix PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaNullValueToTimestamp STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaNullValueToTimestamp PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaNullValueToDate STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaNullValueToDate PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaNullValueToTime STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaNullValueToTime PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaNullValueToUnix STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaNullValueToUnix PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaUnixToTimestamp STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaUnixToTimestamp PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaNullValueToString STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaNullValueToString PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testSchemalessIdentity STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testSchemalessIdentity PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaNullFieldToTimestamp STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaNullFieldToTimestamp PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaFieldConversion STARTED

Re: [DISCUSS] KIP-495: Dynamically Adjust Log Levels in Connect

[Randall Hauch]

Thanks for the updates, Arjun. If possible, it'd be great to have the KIP
clarify a few things:

1) IIUC, the loggers returned by the GET methods are only those loggers
that have been activated/used/set in the JVM. If this is the case, this
should be specified.

2) It's possible to set a log level on an ancestor of other loggers, so we
should also specify whether or not ancestors are included in the GET
responses. Doing so would be helpful, but might not be feasible since two
different descendants might have different log levels.

Otherwise this looks good!

Best regards,

Randall

On Mon, Sep 16, 2019 at 4:15 AM Arjun Satish  wrote:

> Good catch, Randall. Yes, we should be able to set the level of any logger
> given its name. If this is an ancestor, then the levels of all child
> classes are updated. I updated the KIP to be more explicit about what
> loggers we can set, and how they affect child classes, if any.
>
> Let me know what you think.
>
> Best,
>
> On Thu, Sep 12, 2019 at 5:02 PM Randall Hauch  wrote:
>
> > Thanks for the KIP, Arjun. It's going to be really nice to be able to set
> > the log levels dynamically, especially through the REST API.
> >
> > However, I think it's not clear what behavior the KIP is actually
> proposing
> > with respect to which loggers are exposed (just those that are used, or
> > common ancestors) and the behavior when I change the log level on a
> > specific logger (is just that logger affected, or are descendants
> affected,
> > too).
> >
> > For example, in a Log4J configuration file we can set the logger for
> > packages (e.g., `org.apache.kafka`, `org.apache.kafka.connect`, etc.) or
> > classes (e.g., `org.apache.kafka.connect.runtime.WorkerSinkTask`).
> Really,
> > those are just conventions, because if the code created a logger for the
> > "kafka.connect.worker" context then we could set that, too. So by
> > convention, the loggers map to Kafka classes and packages.
> >
> > But it's unclear what behavior the KIP is proposing. Are the intermediate
> > loggers such as all packages exposed as loggers? If so, if I set the
> logger
> > on `org.apache.kafka.connect.runtime`, will this set the log level for
> all
> > loggers below this?
> >
> > My concern is that if the behavior is (a) only concrete classes, and/or
> (b)
> > setting a log level for a specific logger sets only that logger, then
> this
> > deviates from what our users are familiar with when setting the log
> levels
> > in the Log4J configuration files, and would be a difficult user
> experience
> > if I have to set 30+ loggers rather than 1 or 2.
> >
> > On Thu, Sep 12, 2019 at 1:04 PM Jason Gustafson 
> > wrote:
> >
> > > Thanks Arjun. +1
> > >
> > > On Thu, Sep 12, 2019 at 9:58 AM Gwen Shapira 
> wrote:
> > >
> > > > The new REST API for logger management looks great to me.
> > > >
> > > >
> > > > On Thu, Sep 12, 2019 at 8:36 AM Arjun Satish  >
> > > > wrote:
> > > > >
> > > > > Bumping this thread.
> > > > >
> > > > > If there are no further comments, please add your votes here:
> > > > > https://www.mail-archive.com/dev@kafka.apache.org/msg100313.html
> > > > >
> > > > > Thanks in advance,
> > > > > Arjun
> > > > >
> > > > > On Fri, Sep 6, 2019 at 4:22 PM Arjun Satish <
> arjun.sat...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > Thanks a lot, Jason! Answers inline. I'll also modify the kip to
> > make
> > > > > > these clear.
> > > > > >
> > > > > > On Fri, Sep 6, 2019 at 4:01 PM Jason Gustafson <
> ja...@confluent.io
> > >
> > > > wrote:
> > > > > >
> > > > > >> Hi Arjun,
> > > > > >>
> > > > > >> The updated KIP looks good. Just a couple questions:
> > > > > >>
> > > > > >> 1. Is the /admin endpoint on the normal listener by default? If
> > not,
> > > > is
> > > > > >> there a way to have it use the same listener?
> > > > > >>
> > > > > >
> > > > > > Uses the normal listener by default.
> > > > > >
> > > > > >
> > > > > >> 2. Changes to logging configuration are not intended to be
> > > > persistent, is
> > > > > >> that right? Also, I assume changes only apply to the worker that
> > > > received
> > > > > >> the request?
> > > > > >>
> > > > > >
> > > > > > Changes will not be persistent and only apply to the worker that
> > > > received
> > > > > > the request.
> > > > > >
> > > > > >
> > > > > >> Thanks,
> > > > > >> Jason
> > > > > >>
> > > > > >> On Fri, Aug 30, 2019 at 1:25 AM Arjun Satish <
> > > arjun.sat...@gmail.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >> > OK. I didn't realize the plan was to deprecate and remove the
> > JMX
> > > > > >> endpoint.
> > > > > >> > KIP-412 says brokers will continue to expose the JMX API. JMX
> > was
> > > > > >> selected
> > > > > >> > so all components could follow the brokers. In light of this,
> I
> > > > think we
> > > > > >> > should simply aim for semantic equivalency across the
> different
> > > API
> > > > for
> > > > > >> > this functionality.
> > > > > >> >
> > > > > >> > REST is convenient for Connect. We can modify the KIP to have
> 

Re: [DISCUSS] KIP-481: SerDe Improvements for Connect Decimal type in JSON

[Randall Hauch]

Thanks for all the changes, Almog. The current KIP looks good to me.

Randall

On Fri, Aug 30, 2019 at 11:28 AM Almog Gavra  wrote:

> Thanks again Randall! Here are the changes I made:
>
> - Defaults. The KIP had mentioned that the default would be BASE64 in the
> "Public Interfaces" section. I have also added your suggestion in "Proposed
> Changes".
> - I've added a bullet point to change the internal converter to use
> decimal.format=NUMERIC in Proposed Changes. If I see that this is not
> possible or cumbersome during implementation I will amend the KIP to keep
> this change minimally scoped.
> - Adopted your third suggestion
> - Adopted your migration plan proposal
> - Added rejected alternative for creating a new converter
>
> > calling `NumericNode.decimalValue()` will always return a
> java.math.BigDecimal even
> > if the underlying Number was not a BigDecimal
>
> I'm not sure I understand your comment here. NumericNode#decimalValue will
> always return a BigDecimal, regardless of what the underlying JSON data is
> - that hasn't changed with this KIP. The decimalValue() will only be called
> when converting to a DECIMAL logical type.
>
> What has changed, is that the NumericNode will store a BigDecimal instead
> of a double whenever the JSON value is a number with a decimal point in it
> (this should answer your fourth point, about deserializing a BigDecimal for
> all floating points).
>
> > Shouldn't the `convertToConnect(Schema, JsonNode) instead use the type of
> Number value as
> > parsed and returned by the JsonDeserializer to determine the proper
> schema
> > type
>
> There is no way to infer the "proper schema type" in the JsonDeserializer.
> If my JSON value is {"foo": 1.234} I have no idea whether "foo" is a
> decimal or a double - that's the reason we need a configuration value in
> first place. This means that in order to avoid precision loss, we must
> deserialize any floating point number first as a BigDecimal.
>
> > and then get the proper value type using that schema type's converter?
>
> That's exactly the proposal. I think this will be clear in the code.
>
> Almog
>
> On Fri, Aug 30, 2019 at 8:00 AM Randall Hauch  wrote:
>
> > Thanks for the updates, Almog. This looks really good, but I have a few
> > more comments (most wording, though one potentially thorny issue):
> >
> > First, the KIP should define the default value for the `decimal.format`
> > property. IIUC, it will be BASE64, and to gain the new behavior users
> will
> > have to explicitly set this to NUMERIC. In fact, I'd recommend changing
> > this bullet item from:
> >
> >- JsonConverter will accept the decimal.format configuration to
> >determine the serialization format. If the value is BASE64, the
> behavior
> >remains unchanged (i.e. it serializes it as a base64 text). If the
> > value is
> >NUMERIC, the JSON node will be a number representing that decimal
> (e.g.
> >10.2345 instead of "D3J5").
> >
> > to something like:
> >
> >- Define a new `decimal.format` configuration property on
> JsonConverter
> >to specify the serialization format for Connect DECIMAL logical type
> >values, with two allowed literal values for the configuration
> property:
> >- `BASE64` specifies the existing behavior of serializing DECIMAL
> >   logical types as base64 encoded binary data (e.g., "D3J5" in the
> > example
> >   above), and will be the default; and
> >   - `NUMERIC` will serialize Connect DECIMAL logical type values in
> >   JSON as a number representing that decimal
> >
> > Second, since the default will be the existing BASE64 representation,
> what
> > do you think about changing the JsonConverter instances used by the
> Connect
> > worker for internal topics to enable `decimal.format=NUMERIC`? I don't
> > think we actually use decimals in the internal messages, but if we do at
> > some point then the converters will store them with the improved natural
> > representation.
> >
> > Third, the following bullet item could be more clear:
> >
> >- JsonConverter will automatically handle deserialization of either
> >serialization format given a Decimal logical type schema, i.e. it will
> >accept both a deserialized BinaryNode and NumericNode. If the value
> is a
> >BinaryNode, it will construct a java BigDecimal from the binaryValue()
> >(which is a btye[]). If the value is a NumericNode, it will simply
> pass
> >through the decimalValue() deserialized by the JsonDeserializer.
> >
> > such as maybe:
> >
> >- The JsonConverter deserialization method currently expects only a
> >BinaryNode, but will be changed to also handle NumericNode by calling
> >NumericNode.decimalValue().
> >
> > This brings up an interesting potential issue: if `schemas.enable=false`,
> > then there will be no schema in the record, and calling
> > `NumericNode.decimalValue()` will always return a java.math.BigDecimal
> even
> > if the underlying Number was not a BigDecimal. 

Re: [VOTE] KIP-481: SerDe Improvements for Connect Decimal type in JSON

[Randall Hauch]

Thanks for the nice improvement, Almog!

+1 (binding)

Randall

On Thu, Aug 15, 2019 at 11:59 AM Konstantine Karantasis <
konstant...@confluent.io> wrote:

> Thanks Almog!
> Nicely designed and concise KIP.
>
> +1 non-binding
>
> Konstantine
>
> On Tue, Aug 6, 2019 at 11:44 PM Almog Gavra  wrote:
>
> > Hello Everyone,
> >
> > After discussions on
> >
> >
> https://lists.apache.org/thread.html/fa665a6dc59f73ca294a00bcbef2eaa3ad00cc69626e91c516fa4fca@%3Cdev.kafka.apache.org%3E
> > I've opened this KIP up for formal voting.
> >
> > KIP:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-481%3A+SerDe+Improvements+for+Connect+Decimal+type+in+JSON
> >
> > Please update the DISCUSS thread with any concerns/comments.
> >
> > Cheers!
> > Almog
> >
>

Re: [DISCUSS] KIP-521: Enable redirection of Connect's log4j messages to a file by default

[Randall Hauch]

Thanks for tackling this, Konstantine.

The KIP looks great. My only question is about whether to enable the recent
MDC variable in the file log format, but for backward compatibility reasons
keep it as-is for the console. I suspect using the same format in the log
files and the console would be preferred, and to have users change it in
one place. WDYT?

Randall

On Wed, Sep 11, 2019 at 7:06 PM Konstantine Karantasis <
konstant...@confluent.io> wrote:

> Thanks Gwen!
> Indeed, it's a common setup and it's been missing for some time. I agree,
> it'll be nice to have this in place by default.
> I'm guessing previous attempts missed that such a change needs a KIP.
>
> Cheers,
> Konstantine
>
>
>
> On Wed, Sep 11, 2019 at 2:16 PM Gwen Shapira  wrote:
>
> > Great idea. It will greatly improve the ops experience. Can't believe
> > we didn't do it before.
> >
> > On Wed, Sep 11, 2019 at 2:07 PM Konstantine Karantasis
> >  wrote:
> > >
> > > *** Missed the [DISCUSS] tag in the previous email. Reposting here,
> > please
> > > reply in this thread instead ***
> > >
> > > Hi all.
> > >
> > > While we are in the midst of some very interesting KIP discussions, I'd
> > > like to bring a brief and useful KIP on the table as well.
> > >
> > > It's about enabling redirection of log4j logging to a file for Kafka
> > > Connect by default, in a way similar to how this is done for Kafka
> > brokers
> > > today.
> > >
> > > You might find it short and straightforward but, still, needs to be
> > > discussed as a KIP since it's an externally visible (yet compatible)
> > change
> > > in how Connect logs its status during runtime.
> > >
> > > Here's a link to the KIP:
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-521%3A+Enable+redirection+of+Connect%27s+log4j+messages+to+a+file+by+default
> > >
> > > Looking forward to your comments!
> > > Konstantine
> >
>

Re: [DISCUSS] KIP-507: Securing Internal Connect REST Endpoints

[Randall Hauch]

On Mon, Sep 16, 2019 at 3:06 PM Chris Egerton  wrote:

> Hi Randall,
>
> The new default value for the key size configuration will be "null". I've
> clarified this in the KIP. This will still preserve backwards compatibility
> and should not be an issue.
>

Thanks for clarifying this in the KIP.


>
> I understand your point about key generation vs MAC signing algorithms;
> like I said, I'll need to do more research.
>
> I respectfully disagree that a single algorithm list would be easier to
> understand as opposed to a list of accepted algorithms and a signature
> algorithm. Placing special priority on the first element in that list is
> fairly implicit and leaves room for confusion where users might have the
> same algorithms in their lists for that config but in different orders for
> different workers. The group coordination protocol selection behavior isn't
> really a great example since users don't configure that directly
> themselves.
>
> RE the proposed set of new configs: like I stated in my previous response,
> I object to the use of "cluster." as a configuration prefix for any worker
> configs: "most configurations deal with the worker and, transitively, the
> cluster; there doesn't seem to be good enough cause for including that
> prefix for these configurations." I also think this discussion should
> probably continue a little more before we start proposing new configuration
> names, given that it's still undecided which configurations we're going to
> expose.
>
> > I don't think we have any data about how how often a follower will be
> fully caught up, and it's possible that a worker's consumer fails to keep
> the worker caught up quickly enough with the configs topic and the new
> session key. So can we really say that a follower making a request with an
> expired key will be rare?
> It would depend on the key rotation interval, but I can't imagine the
> likelihood of this occurring with an interval as low as even 10 minutes
> would be that high. The config topic is low-volume; the consumer for that
> topic isn't going to be flooded with writes and it seems fine to expect
> fairly low latency for the consumer of this topic.
>

My concern is that we're really *assuming* it's not a problem. All I'm
asking for is more clarity on what happens when a worker doesn't know about
the new session key when it makes a request to this REST resource? The KIP
makes it clear that It will be retried and that the existing error message
will be replaced with a debug message, at least for a time being. Perhaps
the KIP paragraph that talks about this can be reworded to make this more
clear, something akin to:

"The leader will only accept requests signed with the most current key.
However, Connect follower workers may routinely experience small delays
when reading the new key. Rather than always logging such task
configuration failure and retry attempts as errors (the current behavior),
Connect's distributed herder will be modified slightly to handle such HTTP
403 responses for this task configuration request by quietly retrying them
with the latest key for up to 1 minute. If failures persist for more than 1
minute, they will be logged as errors."


> > Can we not retry for longer than 1 second if the request fails with HTTP
> 403? I'm concerned what the UX will be if/when this happens, and that the
> user sees a very obtuse and seemingly unrelated error message they won't
> know how to fix.
> To be clear, the KIP doesn't propose any changes to the infinite retry
> logic that's present in the worker today. All that the KIP proposes is that
> an existing error-level message be replaced with a debug-level message if
> it's suspected that a request has failed due to an out-of-date key. With
> that out of the way, sure, we can bump the grace period before beginning to
> emit error-level log messages. I think going as high as minute might be
> acceptable; we should try to stay fairly low, however, in case the request
> failures are due to some other reason that should be surfaced as soon as
> possible and with some urgency.
>

Ack. See above .

>
> > The text in the "Failure to relay task configurations to leader due to
> incorrect configuration" section is similarly ambiguous. How will a user
> know that this is occurring, and is this really an unlikely situation
> (e.g., "This scenario is unlikely to occur with any normal usage of
> Connect.")? Seems like users unintentionally misconfiguring some of their
> Connect workers is quite common.
> A user will know that this is occurring based on error-level log messages
> emitted by the worker about a failure to relay task ("Failed to reconfigure
> connector's tasks, retrying after backoff:"), plus a stack trace. Yes, this
> situation is unlikely; most worker files will never contain the
> newly-proposed configurations for this KIP since the default values should
> suffice for most cases. If anyone tries to adjust these values, we will
> have documentation available on 

Re: [DISCUSS] Streams-Broker compatibility regression in 2.2.1 release

[Ismael Juma]

+1 to the proposal. Let's also highlight this in the release notes for
2.2.2 and 2.3.0 please.

Ismael

On Wed, Sep 11, 2019 at 10:23 AM Matthias J. Sax 
wrote:

> Hi,
>
> recently a user reported an issue upgrading a Kafka Streams application
> from 2.2.0 to 2.2.1 (cf
> https://mail-archives.apache.org/mod_mbox/kafka-users/201908.mbox/
> )
>
> After some investigation, we identified
> https://issues.apache.org/jira/browse/KAFKA-7895 to be the root cause of
> the problem.
>
> The fix for KAFKA-7895 is using message headers and thus requires broker
> version 0.11.0 (or newer) and message format 0.11 (or newer). Hence,
> while a Kafka Streams application version 2.2.0 is compatible to older
> brokers (0.10.1 and 0.10.2) and only requires message format 0.10, the
> backward compatibility was broken accidentally in 2.2.1.
>
> The fix is also contained in 2.3.0 release and cherry-picked to 2.1
> branch (2.1.2 is not released yet, and thus 2.1 users are not affected
> as this point).
>
> Note: only users that use `suppress()` operator in their program are
> affected.
>
> We should not break streams-broker backward compatibility in bug-fix
> releases at all and avoid for minor releases. However, it seems
> necessary to have the fix in 2.3.0 though -- otherwise, `suppress()` is
> effectively useless and it does not seem to be a good idea to fix the
> bug only in the next major release. Hence, trading-off some backward
> compatibility in a minor release seems to be acceptable for this case,
> considering that 0.11.0 was release 2 years ago.
>
> For 2.2.1, it is more challenging to decide how to move forward, because
> we should not have broken streams-broker compatibility but 2.2.1 is
> already released and we can only react after the fact.
>
>   From my point of view, the best way is to keep the fix and update the
> release notes and documentation accordingly. The main reason for my
> suggestions is that we would expect a majority of users to be on 0.11.0
> brokers already and the fix will work for them -- reverting the fix in
> 2.2.2 seems to be worse for all those users on newer broker versions. We
> also know that `suppress()` is a highly demanded feature and a lot of
> people waited for a fix.
>
>   The expected minority of users that are on 0.10.1 / 0.10.2 brokers, or
> newer brokers but still on message format 0.10 would either need to stay
> on Kafka Streams 2.2.0 or upgrade their brokers/message format
> accordingly. However, upgrading brokers/message format is de-facto
> already required for 2.2.1 and thus keeping the fix would not make the
> situation worse.
>
> For 2.1, I would suggest to revert the fix to make sure we don't break
> streams-broker compatibility for 2.1.x users. If those users need the
> fix for `suppress()` they need to upgrade to 2.2.1/2.3.0 or newer and
> make sure their brokers are on 0.11.0 with message format 0.11, too.
>
>
> TL;DR; the proposal is:
>
> (1) revert the fix for KAFKA-7895 in 2.1 branch
> (2) keep the fix for KAFKA-7895 in 2.2.1 and 2.3.0
>
> Impact:
>
>  - Kafka Streams 2.1.x and 2.2.0 applications are backward compatible
> back to 0.10.1 brokers, requiring message format 0.10
>  - Kafka Streams 2.2.2 / 2.3.0 application (or newer) are backward
> compatible back to 0.11.0 brokers, requiring message format 0.11
>
>
> Please let us know what you think about this proposal.
>
>
> -Matthias
>
>
>
>

Re: [DISCUSS] KIP-524: Allow users to choose config source when describing configs

[Jason Gustafson]

Thanks Gwen and Rajini. Since this seems to be a non-controversial change,
I'll plan to go ahead with a vote later this week.

-Jason

On Mon, Sep 16, 2019 at 2:41 AM Rajini Sivaram 
wrote:

> Thanks for the KIP Jason! Good idea.
>
> On Sun, Sep 15, 2019 at 5:41 AM Gwen Shapira  wrote:
>
> > Ah, great idea.
> >
> > On Fri, Sep 13, 2019 at 8:47 AM Jason Gustafson 
> > wrote:
> > >
> > > Hi All,
> > >
> > > I have a minor KIP to improve the config tool:
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-524%3A+Allow+users+to+choose+config+source+when+describing+configs
> > .
> > > Let me know what you think.
> > >
> > > -Jason
> >
>

Re: [VOTE] KIP-373: Allow users to create delegation tokens for other users

[Ryanne Dolan]

+1 non-binding

Ryanne

On Mon, Sep 16, 2019, 5:11 PM Harsha Ch  wrote:

> +1 (binding). Thanks for the KIP Viktor
>
> Thanks,
>
> Harsha
>
> On Mon, Sep 16, 2019 at 3:02 AM, Viktor Somogyi-Vass <
> viktorsomo...@gmail.com > wrote:
>
> >
> >
> >
> > Hi All,
> >
> >
> >
> > I'd like to bump this again in order to get some more binding votes
> and/or
> > feedback in the hope we can push this in for 2.4.
> >
> >
> >
> > Thank you Manikumar, Gabor and Ryanne so far for the votes! (the last two
> > were on the discussion thread after starting the vote but I think it
> still
> > counts :) )
> >
> >
> >
> > Thanks,
> > Viktor
> >
> >
> >
> > On Wed, Aug 21, 2019 at 1:44 PM Manikumar < manikumar. reddy@ gmail.
> com (
> > manikumar.re...@gmail.com ) > wrote:
> >
> >
> >>
> >>
> >> Hi,
> >>
> >>
> >>
> >> +1 (binding).
> >>
> >>
> >>
> >> Thanks for the updated KIP. LGTM.
> >>
> >>
> >>
> >> Thanks,
> >> Manikumar
> >>
> >>
> >>
> >> On Tue, Aug 6, 2019 at 3:14 PM Viktor Somogyi-Vass < viktorsomogyi@
> gmail.
> >> com ( viktorsomo...@gmail.com ) >
> >> wrote:
> >>
> >>
> >>>
> >>>
> >>> Hi All,
> >>>
> >>>
> >>>
> >>> Bumping this, I'd be happy to get some additional feedback and/or
> votes.
> >>>
> >>>
> >>>
> >>> Thanks,
> >>> Viktor
> >>>
> >>>
> >>>
> >>> On Wed, Jul 31, 2019 at 11:04 AM Viktor Somogyi-Vass < viktorsomogyi@
> gmail.
> >>> com ( viktorsomo...@gmail.com ) > wrote:
> >>>
> >>>
> 
> 
>  Hi All,
> 
> 
> 
>  I'd like to start a vote on this KIP.
> 
> 
> >>>
> >>>
> >>
> >>
> >>
> >> https:/ / cwiki. apache. org/ confluence/ display/ KAFKA/
> KIP-373%3A+Allow+users+to+create+delegation+tokens+for+other+users
> >> (
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-373%3A+Allow+users+to+create+delegation+tokens+for+other+users
> >> )
> >>
> >>
> >>>
> 
> 
>  To summarize it: the proposed feature would allow users (usually
>  superusers) to create delegation tokens for other users. This is
> 
> 
> >>>
> >>>
> >>>
> >>> especially
> >>>
> >>>
> 
> 
>  helpful in Spark where the delegation token created this way can be
>  distributed to workers.
> 
> 
> 
>  I'd be happy to receive any votes or additional feedback.
> 
> 
> 
>  Viktor
> 
> 
> >>>
> >>>
> >>
> >>
> >
> >
> >

Re: [ DISCUSS ] KIP-512:Adding headers to RecordMetaData

[Renuka M]

Hi All,

The motivation behind this KIP is have info about record in
ProducerInterceptor.onAcknowledgement(..), so that we can link record to
its metadata which is missing now. Initially we proposed to add headers to
RecordMetadata, so that we can link record to its metadata.
After thinking little bit more about it, we thought adding a record itself
to ProducerInterceptor.onAcknowledgement(..) makes sense since we can have
all data to link record to its metadata in Interceptors.

*public interface ProducerInterceptor extends Configurable *{






*@Deprecated   public void onAcknowledgement(RecordMetadata metadata,
Exception exception);   public default void
onAcknowledgement(RecordMetadata metadata, Exception exception,
ProducerRecord record){ onAcknowledgement(metadata, exception);
  }*
}

we will not be adding new method to Callback which has same method
signature, since in callback the record information is  already available
to link.


Please let us know your thoughts on this? If its makes sense, we will drop
original plan of adding headers to metadata, instead we can add an
overloaded onAcknowledgement(...) method to ProducerInterceptor with record
itself.
By providing above default implementation, we will have backward
compatibility, at the same time we can have deprecation plan for existing
method.


Thanks
Renuka M


On Thu, Sep 12, 2019 at 12:22 PM Renuka M  wrote:

> Bumping this in the hope I can get more feedback and/or votes.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-512%3AAdding+headers+to+RecordMetaData
>
> Thanks,
> Renuka M
>
> On Mon, Sep 9, 2019 at 2:41 PM Renuka M  wrote:
>
>>
>> Hi All,
>>
>> Could you please take a look and let us know what you think on KIP-512 --
>> adding record headers to RecordMetaData.
>> Since headers also provides metadata about the record, adding these
>> to RecordMetaData, will allow to link record to its acknowledgement in
>> Interceptors.
>>
>> The message tracing solutions like inca -
>> https://medium.com/@NetflixTechBlog/inca-message-tracing-and-loss-detection-for-streaming-data-netflix-de4836fc38c9
>> using producer callback to link records to its acknowledgement.
>> If we have headers info as part of RecordMetaData, we can capture the
>> record tracking in ProducerInterceptors itself.
>>
>> This is what we are trying to solve.
>>
>> Thanks
>> Renuka M
>>
>>
>>
>>
>>
>> On Thu, Aug 29, 2019 at 10:49 AM Renuka M  wrote:
>>
>>> Hi Colin,
>>>
>>> yes we agree but RecordMetadata in Interceptors and callbacks  will not
>>> have headers which gives context on for which record this MetaData belongs
>>> to. To fill that Gap, we are proposing these changes.
>>>
>>>
>>> Thanks
>>> Renuka M
>>>
>>> On Thu, Aug 29, 2019 at 10:20 AM Colin McCabe 
>>> wrote:
>>>
 As Gwen commented earlier, the client already has the record that it
 sent, including all the headers.

 >
 > Future future = producer.send(myRecord, null);
 > future.get();
 > System.out.println("I sent myRecord with headers " +
 myRecord.headers());
 >

 best,
 Colin


 On Tue, Aug 27, 2019, at 17:06, Renuka M wrote:
 > Hi  Gwen/Team
 >
 > Can you please review the KIP. Hope we have clarified the question
 you have
 > regarding proposal.
 >
 > Thanks
 > Renuka M
 >
 > On Mon, Aug 26, 2019 at 3:35 PM Renuka M 
 wrote:
 >
 > > Hi Eric,
 > >
 > > We thought about that but we didn't find the strong  enough reason
 for
 > > having record itself in Acknowledgement.
 > > Headers are supposed to carry metadata and that is the reason
 headers are
 > > added to producer/consumer records.
 > > Also we feel having headers information in record metadata is good
 enough
 > > to bridge the gap and link the record to its metadata.
 > > Its simple change since we are not adding any new method signatures.
 > > Adding new method signatures requires adoption and deprecation of
 old ones
 > > to reduce duplication.
 > > If we get enough votes on adding new method signature, we are open
 to add
 > > it.
 > >
 > > Thanks
 > > Renuka M
 > >
 > > On Mon, Aug 26, 2019 at 10:54 AM Eric Azama 
 wrote:
 > >
 > >> Have you considered adding a new onAcknowledgement method to the
 > >> ProducerInterceptor with the signature
 onAcknowledgement(RecordMetadata
 > >> metadata, Exception exception, ProducerRecord record)? I would also
 > >> consider adding this to Producer Callbacks as well, since linking a
 > >> Callback to a specific record currently requires creating a new
 Callback
 > >> for every ProducerRecord sent.
 > >>
 > >> This seems like a more robust strategy compared to using Headers.
 Headers
 > >> don't necessarily contain anything that connects them to the
 original
 > >> ProducerRecord, and forcibly including information in the Headers

Build failed in Jenkins: kafka-trunk-jdk11 #814

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: add unsigned varint support (#7338)

[jason] MINOR: Cleanup scala warnings (#7335)

--
[...truncated 2.63 MB...]
org.apache.kafka.connect.converters.FloatConverterTest > testBytesNullToNumber 
STARTED

org.apache.kafka.connect.converters.FloatConverterTest > testBytesNullToNumber 
PASSED

org.apache.kafka.connect.converters.FloatConverterTest > 
testSerializingIncorrectType STARTED

org.apache.kafka.connect.converters.FloatConverterTest > 
testSerializingIncorrectType PASSED

org.apache.kafka.connect.converters.FloatConverterTest > 
testDeserializingHeaderWithTooManyBytes STARTED

org.apache.kafka.connect.converters.FloatConverterTest > 
testDeserializingHeaderWithTooManyBytes PASSED

org.apache.kafka.connect.converters.FloatConverterTest > testNullToBytes STARTED

org.apache.kafka.connect.converters.FloatConverterTest > testNullToBytes PASSED

org.apache.kafka.connect.converters.FloatConverterTest > 
testSerializingIncorrectHeader STARTED

org.apache.kafka.connect.converters.FloatConverterTest > 
testSerializingIncorrectHeader PASSED

org.apache.kafka.connect.converters.FloatConverterTest > 
testDeserializingDataWithTooManyBytes STARTED

org.apache.kafka.connect.converters.FloatConverterTest > 
testDeserializingDataWithTooManyBytes PASSED

org.apache.kafka.connect.converters.FloatConverterTest > 
testConvertingSamplesToAndFromBytes STARTED

org.apache.kafka.connect.converters.FloatConverterTest > 
testConvertingSamplesToAndFromBytes PASSED

org.apache.kafka.connect.converters.ShortConverterTest > testBytesNullToNumber 
STARTED

org.apache.kafka.connect.converters.ShortConverterTest > testBytesNullToNumber 
PASSED

org.apache.kafka.connect.converters.ShortConverterTest > 
testSerializingIncorrectType STARTED

org.apache.kafka.connect.converters.ShortConverterTest > 
testSerializingIncorrectType PASSED

org.apache.kafka.connect.converters.ShortConverterTest > 
testDeserializingHeaderWithTooManyBytes STARTED

org.apache.kafka.connect.converters.ShortConverterTest > 
testDeserializingHeaderWithTooManyBytes PASSED

org.apache.kafka.connect.converters.ShortConverterTest > testNullToBytes STARTED

org.apache.kafka.connect.converters.ShortConverterTest > testNullToBytes PASSED

org.apache.kafka.connect.converters.ShortConverterTest > 
testSerializingIncorrectHeader STARTED

org.apache.kafka.connect.converters.ShortConverterTest > 
testSerializingIncorrectHeader PASSED

org.apache.kafka.connect.converters.ShortConverterTest > 
testDeserializingDataWithTooManyBytes STARTED

org.apache.kafka.connect.converters.ShortConverterTest > 
testDeserializingDataWithTooManyBytes PASSED

org.apache.kafka.connect.converters.ShortConverterTest > 
testConvertingSamplesToAndFromBytes STARTED

org.apache.kafka.connect.converters.ShortConverterTest > 
testConvertingSamplesToAndFromBytes PASSED

org.apache.kafka.connect.converters.DoubleConverterTest > testBytesNullToNumber 
STARTED

org.apache.kafka.connect.converters.DoubleConverterTest > testBytesNullToNumber 
PASSED

org.apache.kafka.connect.converters.DoubleConverterTest > 
testSerializingIncorrectType STARTED

org.apache.kafka.connect.converters.DoubleConverterTest > 
testSerializingIncorrectType PASSED

org.apache.kafka.connect.converters.DoubleConverterTest > 
testDeserializingHeaderWithTooManyBytes STARTED

org.apache.kafka.connect.converters.DoubleConverterTest > 
testDeserializingHeaderWithTooManyBytes PASSED

org.apache.kafka.connect.converters.DoubleConverterTest > testNullToBytes 
STARTED

org.apache.kafka.connect.converters.DoubleConverterTest > testNullToBytes PASSED

org.apache.kafka.connect.converters.DoubleConverterTest > 
testSerializingIncorrectHeader STARTED

org.apache.kafka.connect.converters.DoubleConverterTest > 
testSerializingIncorrectHeader PASSED

org.apache.kafka.connect.converters.DoubleConverterTest > 
testDeserializingDataWithTooManyBytes STARTED

org.apache.kafka.connect.converters.DoubleConverterTest > 
testDeserializingDataWithTooManyBytes PASSED

org.apache.kafka.connect.converters.DoubleConverterTest > 
testConvertingSamplesToAndFromBytes STARTED

org.apache.kafka.connect.converters.DoubleConverterTest > 
testConvertingSamplesToAndFromBytes PASSED

org.apache.kafka.connect.converters.IntegerConverterTest > 
testBytesNullToNumber STARTED

org.apache.kafka.connect.converters.IntegerConverterTest > 
testBytesNullToNumber PASSED

org.apache.kafka.connect.converters.IntegerConverterTest > 
testSerializingIncorrectType STARTED

org.apache.kafka.connect.converters.IntegerConverterTest > 
testSerializingIncorrectType PASSED

org.apache.kafka.connect.converters.IntegerConverterTest > 
testDeserializingHeaderWithTooManyBytes STARTED

org.apache.kafka.connect.converters.IntegerConverterTest > 
testDeserializingHeaderWithTooManyBytes PASSED

org.apache.kafka.connect.converters.IntegerConverterTest > 

Re: [DISCUSS] KIP-511: Collect and Expose Client's Name and Version in the Brokers

[Jose Armando Garcia Sancio]

On Mon, Sep 9, 2019 at 4:00 PM Colin McCabe  wrote:

>
> One solution to consider is adding the clientVersion and clientType to the
> request header as optional (tagged) fields.  This would let us skip the
> extra round trip.  I don't think it's that much more messy than having a
> separate request type to set the client version and type.  In both cases,
> you have to handle connections that set the version later than others, or
> don't set the version at all (for compatibility).  So the version/type has
> to be mutable and added after the TCP connection itself is established.


Hey David,

Did we consider Colin's suggestion of adding this information to every
request using tagged field? If so can we add a section to the KIP
documenting the decision?

The HTTP protocol solves a similar problem by using a special header called
User-Agent. In that field clients encode much more information than just
client version and type. For example Mozilla uses this to include platform
version and engine version. E.g.

User-Agent: Mozilla/ () 
() 

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/User-Agent

Thanks!

-Jose

KAFKA-8584: Support of ByteBuffer for bytes field implemented[Convert Kafka RPCs to use automatically generated code]

[Nikolay Izhikov]

Hello, Kafka team.

I implemented KAFKA-8584 [1].
PR - [2]
Please, do the review.

[1] https://issues.apache.org/jira/browse/KAFKA-8584
[2] https://github.com/apache/kafka/pull/7342


signature.asc
Description: This is a digitally signed message part

[jira] [Created] (KAFKA-8913) Document topic based configs & ISR settings for Streams apps

[Vinoth Chandar (Jira)]

Vinoth Chandar created KAFKA-8913:
-

 Summary: Document topic based configs & ISR settings for Streams 
apps
 Key: KAFKA-8913
 URL: https://issues.apache.org/jira/browse/KAFKA-8913
 Project: Kafka
  Issue Type: Improvement
  Components: documentation, streams
Reporter: Vinoth Chandar
Assignee: Vinoth Chandar


Noticed that it was not clear how to configure the internal topics . 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Re: [VOTE] KIP-373: Allow users to create delegation tokens for other users

[Harsha Ch]

+1 (binding). Thanks for the KIP Viktor

Thanks,

Harsha

On Mon, Sep 16, 2019 at 3:02 AM, Viktor Somogyi-Vass < viktorsomo...@gmail.com 
> wrote:

> 
> 
> 
> Hi All,
> 
> 
> 
> I'd like to bump this again in order to get some more binding votes and/or
> feedback in the hope we can push this in for 2.4.
> 
> 
> 
> Thank you Manikumar, Gabor and Ryanne so far for the votes! (the last two
> were on the discussion thread after starting the vote but I think it still
> counts :) )
> 
> 
> 
> Thanks,
> Viktor
> 
> 
> 
> On Wed, Aug 21, 2019 at 1:44 PM Manikumar < manikumar. reddy@ gmail. com (
> manikumar.re...@gmail.com ) > wrote:
> 
> 
>> 
>> 
>> Hi,
>> 
>> 
>> 
>> +1 (binding).
>> 
>> 
>> 
>> Thanks for the updated KIP. LGTM.
>> 
>> 
>> 
>> Thanks,
>> Manikumar
>> 
>> 
>> 
>> On Tue, Aug 6, 2019 at 3:14 PM Viktor Somogyi-Vass < viktorsomogyi@ gmail.
>> com ( viktorsomo...@gmail.com ) >
>> wrote:
>> 
>> 
>>> 
>>> 
>>> Hi All,
>>> 
>>> 
>>> 
>>> Bumping this, I'd be happy to get some additional feedback and/or votes.
>>> 
>>> 
>>> 
>>> Thanks,
>>> Viktor
>>> 
>>> 
>>> 
>>> On Wed, Jul 31, 2019 at 11:04 AM Viktor Somogyi-Vass < viktorsomogyi@ gmail.
>>> com ( viktorsomo...@gmail.com ) > wrote:
>>> 
>>> 
 
 
 Hi All,
 
 
 
 I'd like to start a vote on this KIP.
 
 
>>> 
>>> 
>> 
>> 
>> 
>> https:/ / cwiki. apache. org/ confluence/ display/ KAFKA/ 
>> KIP-373%3A+Allow+users+to+create+delegation+tokens+for+other+users
>> (
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-373%3A+Allow+users+to+create+delegation+tokens+for+other+users
>> )
>> 
>> 
>>> 
 
 
 To summarize it: the proposed feature would allow users (usually
 superusers) to create delegation tokens for other users. This is
 
 
>>> 
>>> 
>>> 
>>> especially
>>> 
>>> 
 
 
 helpful in Spark where the delegation token created this way can be
 distributed to workers.
 
 
 
 I'd be happy to receive any votes or additional feedback.
 
 
 
 Viktor
 
 
>>> 
>>> 
>> 
>> 
> 
> 
>

Re: [VOTE] KIP-309: Add toUpperCase support to sasl.kerberos.principal.to.local rule

[Harsha Chintalapani]

+1 (binding).
Thanks,
Harsha


On Mon, Sep 16, 2019 at 4:21 AM, Manikumar 
wrote:

> Hi all,
>
> I would like to start vote on this trivial KIP-309: https://cwiki.apache.
> org/confluence/display/KAFKA/KIP
> -309%3A+Add+toUpperCase+support+to+sasl.kerberos.principal.to.local+rule
>
> Thanks,
>

[jira] [Created] (KAFKA-8912) TimeoutException cause is insufficiently articulated for the KafkProducer

[Ramkishan (Jira)]

Ramkishan created KAFKA-8912:


 Summary: TimeoutException cause is insufficiently articulated for 
the KafkProducer
 Key: KAFKA-8912
 URL: https://issues.apache.org/jira/browse/KAFKA-8912
 Project: Kafka
  Issue Type: Bug
  Components: documentation, producer 
Affects Versions: 2.2.1, 0.10.0.1
Reporter: Ramkishan


The javadocs of the org.apache.kafka.clients.producer.KafkaProducer class has 
partially articulated description for the TimeoutException, wherever applicable.

The document reads - 
"{{[TimeoutException|https://kafka.apache.org/23/javadoc/org/apache/kafka/common/errors/TimeoutException.html]}}
 - If the time taken for fetching metadata or allocating memory for the record 
has surpassed {{max.block.ms}}."

While we are aware that this exception can also be thrown if the RTM occurs 
when the message is on the accumulator and the batch expires. The current 
description is misleading the developers who try to optimize the producer 
config and if they face this issue

[https://kafka.apache.org/23/javadoc/org/apache/kafka/common/errors/TimeoutException.html]



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Re: [DISCUSS] KIP-507: Securing Internal Connect REST Endpoints

[Chris Egerton]

Hi Randall,

The new default value for the key size configuration will be "null". I've
clarified this in the KIP. This will still preserve backwards compatibility
and should not be an issue.

I understand your point about key generation vs MAC signing algorithms;
like I said, I'll need to do more research.

I respectfully disagree that a single algorithm list would be easier to
understand as opposed to a list of accepted algorithms and a signature
algorithm. Placing special priority on the first element in that list is
fairly implicit and leaves room for confusion where users might have the
same algorithms in their lists for that config but in different orders for
different workers. The group coordination protocol selection behavior isn't
really a great example since users don't configure that directly
themselves.

RE the proposed set of new configs: like I stated in my previous response,
I object to the use of "cluster." as a configuration prefix for any worker
configs: "most configurations deal with the worker and, transitively, the
cluster; there doesn't seem to be good enough cause for including that
prefix for these configurations." I also think this discussion should
probably continue a little more before we start proposing new configuration
names, given that it's still undecided which configurations we're going to
expose.

> I don't think we have any data about how how often a follower will be
fully caught up, and it's possible that a worker's consumer fails to keep
the worker caught up quickly enough with the configs topic and the new
session key. So can we really say that a follower making a request with an
expired key will be rare?
It would depend on the key rotation interval, but I can't imagine the
likelihood of this occurring with an interval as low as even 10 minutes
would be that high. The config topic is low-volume; the consumer for that
topic isn't going to be flooded with writes and it seems fine to expect
fairly low latency for the consumer of this topic.

> Can we not retry for longer than 1 second if the request fails with HTTP
403? I'm concerned what the UX will be if/when this happens, and that the
user sees a very obtuse and seemingly unrelated error message they won't
know how to fix.
To be clear, the KIP doesn't propose any changes to the infinite retry
logic that's present in the worker today. All that the KIP proposes is that
an existing error-level message be replaced with a debug-level message if
it's suspected that a request has failed due to an out-of-date key. With
that out of the way, sure, we can bump the grace period before beginning to
emit error-level log messages. I think going as high as minute might be
acceptable; we should try to stay fairly low, however, in case the request
failures are due to some other reason that should be surfaced as soon as
possible and with some urgency.

> The text in the "Failure to relay task configurations to leader due to
incorrect configuration" section is similarly ambiguous. How will a user
know that this is occurring, and is this really an unlikely situation
(e.g., "This scenario is unlikely to occur with any normal usage of
Connect.")? Seems like users unintentionally misconfiguring some of their
Connect workers is quite common.
A user will know that this is occurring based on error-level log messages
emitted by the worker about a failure to relay task ("Failed to reconfigure
connector's tasks, retrying after backoff:"), plus a stack trace. Yes, this
situation is unlikely; most worker files will never contain the
newly-proposed configurations for this KIP since the default values should
suffice for most cases. If anyone tries to adjust these values, we will
have documentation available on what their behavior is and what to do to
ensure your cluster doesn't break while changing them (including the
rolling upgrade procedure outlined in the KIP). These configurations make
it no likelier that a user will accidentally break their Connect cluster
than the group ID, key/value converter, REST extension, and broker
authentication configs (to name just a few), and since they will be left
out of any sample worker configurations included in Kafka, the odds of
someone accidentally messing with them are low enough that it doesn't seem
worth investing a lot of effort into making it harder for someone to shoot
themselves in the foot.
I'll update the KIP to include possible indicators that the cluster has
been misconfigured, but I don't think this scenario deserves a ton of
priority.

> Maybe provide a bit more description of what these error messages will be.
I believe this is more of an implementation detail and should be left to PR
review. KIPs should only be expected to be so fine-grained; proposing
actual log messages doesn't seem necessary for the overall
adoption/rejection of the mechanisms proposed here and the iteration cycle
on GitHub is significantly faster and more efficient than on a mailing list.

> Do we need a JMX metric that shows 

Re: Delivery Status Notification (Failure)

[David Arthur]

And here's a passing build for the 2.3 branch
https://builds.apache.org/view/All/job/kafka-2.3-jdk8/108/

On Mon, Sep 16, 2019 at 3:46 PM David Arthur  wrote:

> And here's a passing build for the 2.3 branch
> https://builds.apache.org/view/All/job/kafka-2.3-jdk8/108/
>
> On Fri, Sep 13, 2019 at 6:53 PM Mail Delivery Subsystem <
> mailer-dae...@googlemail.com> wrote:
>
>> Hello davidart...@apache.org,
>>
>> We're writing to let you know that the group you tried to contact
>> (kafka-clients) may not exist, or you may not have permission to post
>> messages to the group. A few more details on why you weren't able to post:
>>
>>  * You might have spelled or formatted the group name incorrectly.
>>  * The owner of the group may have removed this group.
>>  * You may need to join the group before receiving permission to post.
>>  * This group may not be open to posting.
>>
>> If you have questions related to this or any other Google Group, visit
>> the Help Center at https://groups.google.com/support/.
>>
>> Thanks,
>>
>> Google Groups
>>
>>
>>
>> - Original message -
>>
>> X-Google-Smtp-Source:
>> APXvYqzR4ecTqF5eQ+zbyuBxevrqEwPh8iwuX3JqXoKJrMBJp7djgdedjT2zyrbtVIrUeG6BwVA8
>> X-Received: by 2002:a2e:a408:: with SMTP id
>> p8mr31061788ljn.54.1568415187213;
>> Fri, 13 Sep 2019 15:53:07 -0700 (PDT)
>> ARC-Seal: i=1; a=rsa-sha256; t=1568415187; cv=none;
>> d=google.com; s=arc-20160816;
>>
>> b=lFaSoS3I6a2CXozRGM3EmhfndkH0TurGXBP9+hWIfDIcoNjnr3ARGwMKY7AWCDZPs3
>>
>>  ov7Q0bS1Q6p0sYNteXCQL/sV6/mgc2V/xyDSGG5o1KVIgZFfK9ufnwcMk4aO+WrXpDAW
>>
>>  j7LdU4dASdd+Xx7XStZv4q6MwXscMm4jQo0i8rUUDntcP4att8pHOMOLi1xPviWm16Fj
>>
>>  8hRHBhP3q3cVwJ5tEsDNgXBNpI6VsZ9QpMbqGyc5utoVc8SN2ga+8mE4hdBZER/dCA3N
>>
>>  z4ZShmQUeC1Ke8AkoSbnQ2xCSjHC9/WIjP2OFCglMGCTpnxKKBW7XS6WdC73tSKwCgqM
>>  gNdA==
>> ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com;
>> s=arc-20160816;
>> h=to:subject:message-id:date:from:mime-version;
>> bh=2IB75WkaHSQnbnrcwcxo9nzKnjVzTOZ3fxahUUU2E4A=;
>>
>> b=ItkjikNLKn9+gEytT805Fz6dm3386ciF2CFBtwmRwv/oR77fsGxREbIrats1BIvp3W
>>
>>  RE91FZbTRo3i9p4EbHpKpjpm1kLetiUrbaXVw2Ti85c7v2D+BoLEwpMAsVvRCQcnEG/K
>>
>>  oLLZP4I39alEFzH3RzUqXVbmdmBx5G/UGXEVvo6rtOEsvZm7r3Cg5/QZIee3jTNQL0Tv
>>
>>  1iVk3O1OUqtiEuaxg7e/x48fzwpMSg1Xo1xmXLRCfmVVGPsvc1pAsoMBwYHrCp5Fz6pS
>>
>>  p6pEtPZDKfZJ4xgGveJuawT4OyMkhcZVREot9KoEOzRA6zi/o2iPq93urcTQqskF13ze
>>  /+yQ==
>> ARC-Authentication-Results: i=1; gmr-mx.google.com;
>>spf=pass (google.com: domain of davidart...@apache.org designates
>> 207.244.88.153 as permitted sender) smtp.mailfrom=davidart...@apache.org
>> Return-Path: 
>> Received: from mail.apache.org (hermes.apache.org. [207.244.88.153])
>> by gmr-mx.google.com with SMTP id
>> o30si1535368lfi.0.2019.09.13.15.53.06
>> for ;
>> Fri, 13 Sep 2019 15:53:07 -0700 (PDT)
>> Received-SPF: pass (google.com: domain of davidart...@apache.org
>> designates 207.244.88.153 as permitted sender) client-ip=207.244.88.153;
>> Authentication-Results: gmr-mx.google.com;
>>spf=pass (google.com: domain of davidart...@apache.org designates
>> 207.244.88.153 as permitted sender) smtp.mailfrom=davidart...@apache.org
>> Received: (qmail 16798 invoked by uid 99); 13 Sep 2019 22:53:05 -
>> Received: from Unknown (HELO mailrelay1-lw-us.apache.org) (10.10.3.159)
>> by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Sep 2019 22:53:05
>> +
>> Received: from mail-lj1-f179.google.com (mail-lj1-f179.google.com
>> [209.85.208.179])
>> by mailrelay1-lw-us.apache.org (ASF Mail Server at
>> mailrelay1-lw-us.apache.org) with ESMTPSA id 51D8C5A46
>> for ; Fri, 13 Sep 2019 22:53:05
>> + (UTC)
>> Received: by mail-lj1-f179.google.com with SMTP id m13so621468ljj.11
>> for ; Fri, 13 Sep 2019 15:53:05
>> -0700 (PDT)
>> X-Gm-Message-State:
>> APjAAAXWPIv9Dwy38bntGR/3Ohm5LevO97RH2xWTmubiYBHn99xVzzPX
>> BiCE0sUZAWUyGlzIzWDF8YoZOrAzpwrn7B3O8AA=
>> X-Received: by 2002:a2e:9c87:: with SMTP id
>> x7mr18958540lji.207.1568415184417;
>>  Fri, 13 Sep 2019 15:53:04 -0700 (PDT)
>> MIME-Version: 1.0
>> From: David Arthur 
>> Date: Fri, 13 Sep 2019 18:52:53 -0400
>> X-Gmail-Original-Message-ID: <
>> ca+0ze6rcdwmmc0e+usuekcttyr7r2ecck5tti_28eosfcve...@mail.gmail.com>
>> Message-ID: <
>> ca+0ze6rcdwmmc0e+usuekcttyr7r2ecck5tti_28eosfcve...@mail.gmail.com>
>> Subject: [VOTE] 2.3.1 RC0
>> To: dev@kafka.apache.org, users ,
>> kafka-clie...@googlegroups.com
>> Content-Type: multipart/alternative;
>> boundary="ed791e0592771d44"
>>
>> --ed791e0592771d44
>> Content-Type: text/plain; charset="UTF-8"
>>
>> Hello Kafka users, developers and client-developers,
>>
>>
>> This is the first candidate for release of Apache Kafka 2.3.1 which
>> includes many bug fixes for Apache Kafka 2.3.
>>
>>
>> Release notes for the 2.3.1 release:
>>
>> https://home.apache.org/~davidarthur/kafka-2.3.1-rc0/RELEASE_NOTES.html
>>
>>
>> *** Please 

Re: Request to be added as a contributor

[Shyam P]

Thanks Matthias J S

On Tue, Sep 17, 2019 at 12:24 AM Matthias J. Sax 
wrote:

> Shyam,
>
> also check out https://kafka.apache.org/contributing
>
>
> -Matthias
>
> On 9/16/19 11:51 AM, Jun Rao wrote:
> > Hi, Nigel,
> >
> > Thanks for your interest. Just added you to the contributor list.
> >
> > Hi, Shyam,
> >
> > Thanks for your interest. What's you jira id?
> >
> > To get started, you can probably look for recent jiras labelled with the
> > newbie tag.
> >
> > Jun
> >
> > On Mon, Sep 16, 2019 at 11:38 AM Shyam P 
> wrote:
> >
> >> Me too , want to contribute?
> >> But not sure where to start ...
> >>
> >>
> >>
> >> On Mon, Sep 16, 2019 at 11:05 PM Nigel Liang 
> wrote:
> >>
> >>> Hi there,
> >>>
> >>> I'd like to contribute to Kafka, and would like permissions for the
> >>> following account:
> >>> ASF JIRA: nigel.liang
> >>>
> >>> Thanks!
> >>> -Nigel
> >>>
> >>
> >
>
>

Re: [DISCUSS] KIP-507: Securing Internal Connect REST Endpoints

[Randall Hauch]

Thanks, Chris. I still have a number of questions and requests for
clarification.

If we don't provide a default value for the key size, then the following
statement in the "Backwards compatibility" subsection is no longer true:
"All of the proposed configurations here have default values, making them
backwards compatible." IIUC, a user will not be able to upgrade an existing
Connect cluster without editing their configurations, and this pretty much
means it is not backwards compatible, which is a non-starter.

Regarding the new configurations and our earlier discussion about whether
all are required, I'm not convinced that we need separate configs for
signing and key generation algorithms. If this is a common need, then the
KIP should justify that with an explanation. But as it stands now, exposing
multiple algorithm properties now means the UX is more complicated and we
can't make things simpler in the future. I also think that a single
algorithm list where the first is used for signing would be easier to
understand (fewer is better) and would be more in line with the way
rebalance protocols are chosen the broker (see
https://www.youtube.com/watch?v=MmLezWRI3Ys for a decent explanation). If
at some point in the future we do want that extra flexibility, then we can
expose additional properties.

I also asked earlier about renaming the config properties so they can be
used in other places within the cluster other than the task configs
request, which is something I think we'll want to do. If we minimize the
number of configs, then how about `cluster.session.algorithms`,
`cluster.session.key.size` and `cluster.session.key.ttl.ms`?

The "Proposed Changes" section now mentions:

The leader will only accept requests signed with the most current key. This
> should not cause any major problems; if a follower attempts to make a
> request with an expired key (which should be quite rare and only occur if
> the request is made by a follower that is not fully caught up to the end of
> the config topic), the initial request will fail, but will be subsequently
> retried after a backoff period. This backoff period should leave sufficient
> room for the rebalance to complete.
>

I don't think we have any data about how how often a follower will be fully
caught up, and it's possible that a worker's consumer fails to keep the
worker caught up quickly enough with the configs topic and the new session
key. So can we really say that a follower making a request with an expired
key will be rare?

If the first four requests fail with HTTP 403, it will be assumed that this
> is due to an out-of-date session key; a debug-level message about the
> subsequent retry will be logged in place of the current error-level log
> message of "Failed to reconfigure connector's tasks, retrying after
> backoff: " followed by a stack trace. Since the backoff period is 250
> milliseconds, this should give at least one second of leeway for an
> outdated key to be updated. If longer than that is required, the usual
> error-level log messages will begin to be generated by the worker.
>

Can we not retry for longer than 1 second if the request fails with HTTP
403? I'm concerned what the UX will be if/when this happens, and that the
user sees a very obtuse and seemingly unrelated error message they won't
know how to fix.

The text in the "Failure to relay task configurations to leader due to
incorrect configuration" section is similarly ambiguous. How will a user
know that this is occurring, and is this really an unlikely situation
(e.g., "This scenario is unlikely to occur with any normal usage of
Connect.")? Seems like users unintentionally misconfiguring some of their
Connect workers is quite common.

Additionally, it will be vital to design appropriate error messages for
> this scenario so that users can (hopefully) dig themselves out of that hole
> on their own.


Maybe provide a bit more description of what these error messages will be.

Do we need a JMX metric that shows the protocol that each worker is
configured to use, and whether the workers are using session keys? This
would be a great way to monitor the cluster's use of this feature.

Best regards,

Randall


On Wed, Sep 11, 2019 at 7:00 PM Chris Egerton  wrote:

> Hi all,
>
> I've updated KIP-507 to reflect the changes inspired by Randall's recent
> feedback. In addition, after some further research, I've decided to remove
> the proposed default value for the internal.request.key.size and instead,
> should no value be provided, rely on the default key size for the given key
> algorithm. More detail can be found in the KIP if anyone's interested.
>
> Cheers,
>
> Chris
>
> On Tue, Sep 10, 2019 at 3:18 PM Chris Egerton  wrote:
>
> > Hi Randall,
> >
> > Thanks so much for your comprehensive feedback! To avoid creating an
> > extremely long response I'll address your comments/questions by
> referencing
> > the identifiers you've provided for them as opposed to copying them and
> > responding 

Re: Request to be added as a contributor

[Matthias J. Sax]

Done.

On 9/16/19 1:30 AM, Csenge Virág Maruzsi wrote:
> Hi Team,
> 
> I'd like to contribute to Kafka, and would like permissions for that.
> Jira username: csenge
> Thanks!
> 
> Regards,
> Csenge
> 



signature.asc
Description: OpenPGP digital signature

Re: Request to be added as a contributor

[Matthias J. Sax]

Shyam,

also check out https://kafka.apache.org/contributing


-Matthias

On 9/16/19 11:51 AM, Jun Rao wrote:
> Hi, Nigel,
> 
> Thanks for your interest. Just added you to the contributor list.
> 
> Hi, Shyam,
> 
> Thanks for your interest. What's you jira id?
> 
> To get started, you can probably look for recent jiras labelled with the
> newbie tag.
> 
> Jun
> 
> On Mon, Sep 16, 2019 at 11:38 AM Shyam P  wrote:
> 
>> Me too , want to contribute?
>> But not sure where to start ...
>>
>>
>>
>> On Mon, Sep 16, 2019 at 11:05 PM Nigel Liang  wrote:
>>
>>> Hi there,
>>>
>>> I'd like to contribute to Kafka, and would like permissions for the
>>> following account:
>>> ASF JIRA: nigel.liang
>>>
>>> Thanks!
>>> -Nigel
>>>
>>
> 



signature.asc
Description: OpenPGP digital signature

Re: Request to be added as a contributor

[Jun Rao]

Hi, Nigel,

Thanks for your interest. Just added you to the contributor list.

Hi, Shyam,

Thanks for your interest. What's you jira id?

To get started, you can probably look for recent jiras labelled with the
newbie tag.

Jun

On Mon, Sep 16, 2019 at 11:38 AM Shyam P  wrote:

> Me too , want to contribute?
> But not sure where to start ...
>
>
>
> On Mon, Sep 16, 2019 at 11:05 PM Nigel Liang  wrote:
>
> > Hi there,
> >
> > I'd like to contribute to Kafka, and would like permissions for the
> > following account:
> > ASF JIRA: nigel.liang
> >
> > Thanks!
> > -Nigel
> >
>

Re: Request to be added as a contributor

[Shyam P]

Me too , want to contribute?
But not sure where to start ...



On Mon, Sep 16, 2019 at 11:05 PM Nigel Liang  wrote:

> Hi there,
>
> I'd like to contribute to Kafka, and would like permissions for the
> following account:
> ASF JIRA: nigel.liang
>
> Thanks!
> -Nigel
>

Jenkins build is back to normal : kafka-2.3-jdk8 #108

[Apache Jenkins Server]

See 

Request to be added as a contributor

[Nigel Liang]

Hi there,

I'd like to contribute to Kafka, and would like permissions for the
following account:
ASF JIRA: nigel.liang

Thanks!
-Nigel

Re: [DISCUSS] KIP-517: Add consumer metric indicating time between poll calls

[Harsha Chintalapani]

Thanks. +1 LGTM.


On Mon, Sep 16, 2019 at 9:19 AM, Kevin Lu  wrote:

> Hi Harsha,
>
> Thanks for the feedback. I have added *last-poll-seconds-ago* to the KIP
> (being consistent with *last-heartbeat-seconds-ago*).
>
> Regards,
> Kevin
>
> On Sat, Sep 14, 2019 at 9:44 AM Harsha Chintalapani 
> wrote:
>
> Thanks Kevin for the KIP. Overall LGTM.
> On you second point, I think the metric will be really useful to indicate
> the perf bottlenecks on user code vs kakfa consumer/broker.
>
> Thanks,
> Harsha
>
> On Fri, Sep 13, 2019 at 2:41 PM, Kevin Lu  wrote:
>
> Hi Radai & Jason,
>
> Thanks for the support and suggestion.
>
> 1. I think ratio is a good additional metric since the current proposed
> metrics are only absolute times which may not be useful in all scenarios.
>
> I have added this to the KIP:
> * - poll-idle-ratio*: The fraction of time the consumer spent waiting for
> the user to process records from poll.
>
> Thoughts on the metric name/description?
>
> 2. Would it be useful to include a metric measuring the time since poll
> was last called? Similar to *heartbeat-last-seconds-ago*, it would be
> *poll-last-ms-ago.
> *This could be useful if (1) the user has a very high *max.poll.interval.
> ms
> * configured and typically spends a long
> time processing, or (2) comparing this metric with others such as
> *heartbeat-last-seconds-ago* or something else for gathering data in root
> cause analyses (or identifying potential consumer bugs related to poll).
>
> Regards,
> Kevin
>
> On Fri, Sep 13, 2019 at 10:39 AM Jason Gustafson 
> wrote:
>
> Hi Kevin,
>
> This looks reasonable to me. I'd also +1 Radai's suggestion if you're
> willing. Something like an idle ratio for the consumer would be helpful.
>
> Thanks,
> Jason
>
> On Fri, Sep 13, 2019 at 10:08 AM radai 
> wrote:
>
> while youre at it another metric that we have found to be useful is %
>
> time
>
> spent in user code vs time spent in poll() (so time between poll calls /
> time inside poll calls) - the higher the % value the more indicative of
> user code being the cause of performance bottlenecks.
>
> On Fri, Sep 13, 2019 at 9:14 AM Kevin Lu  wrote:
>
> Hi All,
>
> Happy Friday! Bumping this. Any thoughts?
>
> Thanks.
>
> Regards,
> Kevin
>
> On Thu, Sep 5, 2019 at 9:35 AM Kevin Lu  wrote:
>
> Hi All,
>
> I'd like to propose a new consumer metric that measures the time
>
> between
>
> calls to poll() for use in issues related to hitting
>
> max.poll.interval.ms
>
> due to long processing time.
>
> https://cwiki.apache.org/confluence/display/KAFKA/
> KIP-517%3A+Add+consumer+metric+indicating+time+between+poll+calls
>
> Please give it a read and let me know what you think.
>
> Thanks!
>
> Regards,
> Kevin
>
>

Re: [DISCUSS] KIP-517: Add consumer metric indicating time between poll calls

[Kevin Lu]

Hi Harsha,

Thanks for the feedback. I have added *last-poll-seconds-ago* to the KIP
(being consistent with *last-heartbeat-seconds-ago*).

Regards,
Kevin

On Sat, Sep 14, 2019 at 9:44 AM Harsha Chintalapani  wrote:

> Thanks Kevin for the KIP. Overall LGTM.
> On you second point, I think the metric will be really useful to indicate
> the perf bottlenecks on user code vs kakfa consumer/broker.
>
> Thanks,
> Harsha
>
>
> On Fri, Sep 13, 2019 at 2:41 PM, Kevin Lu  wrote:
>
> > Hi Radai & Jason,
> >
> > Thanks for the support and suggestion.
> >
> > 1. I think ratio is a good additional metric since the current proposed
> > metrics are only absolute times which may not be useful in all scenarios.
> >
> > I have added this to the KIP:
> > * - poll-idle-ratio*: The fraction of time the consumer spent waiting for
> > the user to process records from poll.
> >
> > Thoughts on the metric name/description?
> >
> > 2. Would it be useful to include a metric measuring the time since poll
> > was last called? Similar to *heartbeat-last-seconds-ago*, it would be
> > *poll-last-ms-ago.
> > *This could be useful if (1) the user has a very high *max.poll.interval.
> > ms
> > * configured and typically spends a long
> > time processing, or (2) comparing this metric with others such as
> > *heartbeat-last-seconds-ago* or something else for gathering data in root
> > cause analyses (or identifying potential consumer bugs related to poll).
> >
> > Regards,
> > Kevin
> >
> > On Fri, Sep 13, 2019 at 10:39 AM Jason Gustafson 
> > wrote:
> >
> > Hi Kevin,
> >
> > This looks reasonable to me. I'd also +1 Radai's suggestion if you're
> > willing. Something like an idle ratio for the consumer would be helpful.
> >
> > Thanks,
> > Jason
> >
> > On Fri, Sep 13, 2019 at 10:08 AM radai 
> > wrote:
> >
> > while youre at it another metric that we have found to be useful is %
> time
> > spent in user code vs time spent in poll() (so time between poll calls /
> > time inside poll calls) - the higher the % value the more indicative of
> > user code being the cause of performance bottlenecks.
> >
> > On Fri, Sep 13, 2019 at 9:14 AM Kevin Lu  wrote:
> >
> > Hi All,
> >
> > Happy Friday! Bumping this. Any thoughts?
> >
> > Thanks.
> >
> > Regards,
> > Kevin
> >
> > On Thu, Sep 5, 2019 at 9:35 AM Kevin Lu  wrote:
> >
> > Hi All,
> >
> > I'd like to propose a new consumer metric that measures the time
> >
> > between
> >
> > calls to poll() for use in issues related to hitting
> >
> > max.poll.interval.ms
> >
> > due to long processing time.
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/
> > KIP-517%3A+Add+consumer+metric+indicating+time+between+poll+calls
> >
> > Please give it a read and let me know what you think.
> >
> > Thanks!
> >
> > Regards,
> > Kevin
> >
> >
>

Re: [DISCUSS] KIP-491: Preferred Leader Deprioritized List (Temporary Blacklist)

[Satish Duggana]

Hi George,
Thanks for explaining the usecase for topic level preferred leader
blacklist. As I mentioned earlier, I am fine with broker level config
for now.

~Satish.


On Sat, Sep 7, 2019 at 12:29 AM George Li
 wrote:
>
>  Hi,
>
> Just want to ping and bubble up the discussion of KIP-491.
>
> On a large scale of Kafka clusters with thousands of brokers in many 
> clusters.  Frequent hardware failures are common, although the reassignments 
> to change the preferred leaders is a workaround, it incurs unnecessary 
> additional work than the proposed preferred leader blacklist in KIP-491, and 
> hard to scale.
>
> I am wondering whether others using Kafka in a big scale running into same 
> problem.
>
>
> Satish,
>
> Regarding your previous question about whether there is use-case for 
> TopicLevel preferred leader "blacklist",  I thought about one use-case:  to 
> improve rebalance/reassignment, the large partition will usually cause 
> performance/stability issues, planning to change the say the New Replica will 
> start with Leader's latest offset(this way the replica is almost instantly in 
> the ISR and reassignment completed), and put this partition's NewReplica into 
> Preferred Leader "Blacklist" at the Topic Level config for that partition. 
> After sometime(retention time), this new replica has caught up and ready to 
> serve traffic, update/remove the TopicConfig for this partition's preferred 
> leader blacklist.
>
> I will update the KIP-491 later for this use case of Topic Level config for 
> Preferred Leader Blacklist.
>
>
> Thanks,
> George
>
> On Wednesday, August 7, 2019, 07:43:55 PM PDT, George Li 
>  wrote:
>
>   Hi Colin,
>
> > In your example, I think we're comparing apples and oranges.  You started 
> > by outlining a scenario where "an empty broker... comes up... [without] any 
> > > leadership[s]."  But then you criticize using reassignment to switch the 
> > order of preferred replicas because it "would not actually switch the 
> > leader > automatically."  If the empty broker doesn't have any leaderships, 
> > there is nothing to be switched, right?
>
> Let me explained in details of this particular use case example for comparing 
> apples to apples.
>
> Let's say a healthy broker hosting 3000 partitions, and of which 1000 are the 
> preferred leaders (leader count is 1000). There is a hardware failure 
> (disk/memory, etc.), and kafka process crashed. We swap this host with 
> another host but keep the same broker.id, when this new broker coming up, it 
> has no historical data, and we manage to have the current last offsets of all 
> partitions set in the replication-offset-checkpoint (if we don't set them, it 
> could cause crazy ReplicaFetcher pulling of historical data from other 
> brokers and cause cluster high latency and other instabilities), so when 
> Kafka is brought up, it is quickly catching up as followers in the ISR.  
> Note, we have auto.leader.rebalance.enable  disabled, so it's not serving any 
> traffic as leaders (leader count = 0), even there are 1000 partitions that 
> this broker is the Preferred Leader.
>
> We need to make this broker not serving traffic for a few hours or days 
> depending on the SLA of the topic retention requirement until after it's 
> having enough historical data.
>
>
> * The traditional way using the reassignments to move this broker in that 
> 1000 partitions where it's the preferred leader to the end of  assignment, 
> this is O(N) operation. and from my experience, we can't submit all 1000 at 
> the same time, otherwise cause higher latencies even the reassignment in this 
> case can complete almost instantly.  After  a few hours/days whatever, this 
> broker is ready to serve traffic,  we have to run reassignments again to 
> restore that 1000 partitions preferred leaders for this broker: O(N) 
> operation.  then run preferred leader election O(N) again.  So total 3 x O(N) 
> operations.  The point is since the new empty broker is expected to be the 
> same as the old one in terms of hosting partition/leaders, it would seem 
> unnecessary to do reassignments (ordering of replica) during the broker 
> catching up time.
>
>
>
> * The new feature Preferred Leader "Blacklist":  just need to put a dynamic 
> config to indicate that this broker should be considered leader (preferred 
> leader election or broker failover or unclean leader election) to the lowest 
> priority. NO need to run any reassignments. After a few hours/days, when this 
> broker is ready, remove the dynamic config, and run preferred leader election 
> and this broker will serve traffic for that 1000 original partitions it was 
> the preferred leader. So total  1 x O(N) operation.
>
>
> If auto.leader.rebalance.enable  is enabled,  the Preferred Leader 
> "Blacklist" can be put it before Kafka is started to prevent this broker 
> serving traffic.  In the traditional way of running reassignments, once the 
> broker is up, with auto.leader.rebalance.enable  , if 

RE: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

[Pellerin, Clement]

>> Pluggable instances have a default constructor and implement Configurable.
>> Why wouldn't we just do that for SslEngineFactory?
Because the constructor that you are replacing has more than just a map.
You need to pass those extra arguments somewhere.
I once proposed to fake extra arguments as synthetic configs and I was 
rightfully shut down.
I agree though that the map should be passed in configure()

>> We can rename SslEngineFactory.reconfigurableConfigs() if it helps
You don't want to rename this method because it should contain only the 
reconfigurable configs, not all configs.

>> Perhaps we need to focus on the aspects of your use case that are not
>> handled with the proposed approach.
My use case is documented in KIP-383. In more detail, I want to pass an object 
in a config and the custom implementation can call this object to get the 
SslEngine from elsewhere in my application. There should be no 
keystore/truststore validation what so ever and there should not be any SSL 
Configs except the implementation class name and that object.

The only known objection why SslFactory cannot be the extension point is 
because it contains a lot of validation code that would likely be by-passed by 
custom implementations. I don't agree with this objection because the keystore 
validation must be moved to DefaultSslEngineFactory and the rest can be moved 
to the SslChannelBuilder. This way SslFactory would fit the existing extension 
pattern in Kafka. An extension point that extends Reconfigurable is a lot 
cleaner.

If you wonder why I changed my mind so strongly, it's because I realized there 
is less and less validation left in SslFactory and I expect SslChannelBuilder 
to be the only caller so it gives an anchor where to attach the remaining 
validation code.

Can we discuss the possibility of moving the validation code to 
SslChannelBuilder? I did not look at the code and I don't know the challenges 
we would face.

-Original Message-
From: Rajini Sivaram [mailto:rajinisiva...@gmail.com] 
Sent: Monday, September 16, 2019 8:59 AM
To: dev
Subject: Re: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

I am not sure what the confusion regarding Configurable interface is.
Pluggable instances have a default constructor and implement Configurable.
Why wouldn't we just do that for SslEngineFactory?

We can rename SslEngineFactory.reconfigurableConfigs() if it helps, but
basically it is the factory that decides what configs it is interested in.
It doesn't really matter that we don't know what the reconfigurable configs
are until the SslFactory is configured.

`sslContext()` method was restored to support some connectors that were
using it, but we don't need to add that to the interface since the plan is
to remove that anyway. We do need `keystore()` and `truststore()` methods
for validation.

It is always the responsibility of plugins to validate the configs passed
to them. Custom plugins must validate configs and fail configure() if
configs are invalid. We would expect the same for SslEngineFactory.

Perhaps we need to focus on the aspects of your use case that are not
handled with the proposed approach.

On Mon, Sep 16, 2019 at 1:26 PM Pellerin, Clement 
wrote:

> There are pending issues we need to address.
>
> We want to be able to call config.getConfiguredInstance(key, Class) to
> instantiate the extension point. This requires a default constructor. The
> former constructor arguments must now be passed in a separate init()
> method. This has the advantage of moving the constructor signature from the
> comment prose to the compiled language. I took inspiration from
> MetricsReporter for the init() method.
>
> I question the object oriented design that requires the
> reconfigurableConfigs() method but declares the interface to be
> non-reconfigurable with just the Configurable interface.
>
> My use case removes all built-in SSL configs (except the interface class
> name of course). SslFactory should not hardcode any SSL configs in the
> reconfigurableConfigs. It should delegate to the interface instance for all
> reconfigurableConfigs. In particular, it cannot assume there are keystores
> and truststores to validate. These checks should be moved to
> DefaultSslEngineFactory. We can then consider moving the SslEngine
> validation from SslFactory to SslChannelBuilder. What would be left in
> SslFactory that forces us to keep it instead of making it the
> Reconfigurable extension point itself?
>
> I believe we don't need the sslContext() method since it is only used by a
> junit.
> If we support my use case, there is a good chance we don't need the
> keystore() and truststore() method.
>
> I am still not comfortable with the fact that reconfigurableConfigs() are
> not known until the SslEngineFactory implementation is created and that
> happens only after configure() is called. Notice this goes away if
> SslFactory is the extension point, which would explain why this might not
> 

Re: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

[Rajini Sivaram]

I am not sure what the confusion regarding Configurable interface is.
Pluggable instances have a default constructor and implement Configurable.
Why wouldn't we just do that for SslEngineFactory?

We can rename SslEngineFactory.reconfigurableConfigs() if it helps, but
basically it is the factory that decides what configs it is interested in.
It doesn't really matter that we don't know what the reconfigurable configs
are until the SslFactory is configured.

`sslContext()` method was restored to support some connectors that were
using it, but we don't need to add that to the interface since the plan is
to remove that anyway. We do need `keystore()` and `truststore()` methods
for validation.

It is always the responsibility of plugins to validate the configs passed
to them. Custom plugins must validate configs and fail configure() if
configs are invalid. We would expect the same for SslEngineFactory.

Perhaps we need to focus on the aspects of your use case that are not
handled with the proposed approach.

On Mon, Sep 16, 2019 at 1:26 PM Pellerin, Clement 
wrote:

> There are pending issues we need to address.
>
> We want to be able to call config.getConfiguredInstance(key, Class) to
> instantiate the extension point. This requires a default constructor. The
> former constructor arguments must now be passed in a separate init()
> method. This has the advantage of moving the constructor signature from the
> comment prose to the compiled language. I took inspiration from
> MetricsReporter for the init() method.
>
> I question the object oriented design that requires the
> reconfigurableConfigs() method but declares the interface to be
> non-reconfigurable with just the Configurable interface.
>
> My use case removes all built-in SSL configs (except the interface class
> name of course). SslFactory should not hardcode any SSL configs in the
> reconfigurableConfigs. It should delegate to the interface instance for all
> reconfigurableConfigs. In particular, it cannot assume there are keystores
> and truststores to validate. These checks should be moved to
> DefaultSslEngineFactory. We can then consider moving the SslEngine
> validation from SslFactory to SslChannelBuilder. What would be left in
> SslFactory that forces us to keep it instead of making it the
> Reconfigurable extension point itself?
>
> I believe we don't need the sslContext() method since it is only used by a
> junit.
> If we support my use case, there is a good chance we don't need the
> keystore() and truststore() method.
>
> I am still not comfortable with the fact that reconfigurableConfigs() are
> not known until the SslEngineFactory implementation is created and that
> happens only after configure() is called. Notice this goes away if
> SslFactory is the extension point, which would explain why this might not
> have been an issue with the other extension points exposing reconfigurable
> custom configs.
>
> We must document if the configs sent to the extension point implementation
> have been validated or not. I am pretty sure they are not since
> config.getConfiguredInstance() passes the originals() and there is no
> configurableConfigs() method (there is only the subset in
> reconfigurableConfigs()). Non-validated configs might be of the wrong type,
> be out of range, or missing since the default value is not applied. This is
> a burden to the extension point developer and Kafka should provide
> utilities for this.
>
> Can you confirm where the special case for the reconfiguration of
> keystore/truststore is implemented? I am still trying to determine if it is
> possible to trigger a reconfiguration when none of the known configs have
> changed.
>
>
> -Original Message-
> From: Rajini Sivaram [mailto:rajinisiva...@gmail.com]
> Sent: Monday, September 16, 2019 5:29 AM
> To: dev
> Subject: Re: [DISCUSS] KIP-519: Make SSL context/engine configuration
> extensible
>
> Hi Maulin,
>
> Thanks for writing that up. I think we are getting close.
>
> As an example interface (which can be adjusted as necessary, but just
> including something here to refer to), I think we would have an interface
>  along these lines:
>
> public interface SslEngineFactory extends Configurable, Closeable {
>
> Set reconfigurableConfigs();
> boolean shouldBeRebuilt(Map nextConfigs);
>
> SSLEngine createSslEngine(Mode mode, String peerHost, int
> peerPort, String endpointIdentification);
> SSLContext sslContext();
> KeyStore keystore();
> KeyStore truststore();
> }
>
> A) *Creation:*
>
>- We typically tend to include configs in constructor for non-pluggable
>classes, but use Configurable for pluggable classes. So Option 2 works
>better
>
> B) *Returning reconfigurable configs: *
>
>- Custom SslEngineFactory implementations will only return what they
>care about in their implementation of reconfigurableConfigs().
>- SslChannelBuilder will delegate to SslFactory as you mentioned.
>- 

[jira] [Created] (KAFKA-8911) Implicit TimeWindowedSerde creates Serde with null inner serializer

[Jira]

Michał created KAFKA-8911:
-

 Summary: Implicit TimeWindowedSerde creates Serde with null inner 
serializer
 Key: KAFKA-8911
 URL: https://issues.apache.org/jira/browse/KAFKA-8911
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 2.3.0
Reporter: Michał


{{Serdes.scala}} contains an implicit def timeWindowedSerde as below:

 
{code:java}
implicit def timeWindowedSerde[T]: WindowedSerdes.TimeWindowedSerde[T] = new 
WindowedSerdes.TimeWindowedSerde[T]()
{code}

It creates a new {{TimeWindowedSerde}} without inner serializer, which is a 
bug. Even in {{WindowedSerdes.java}} it says that empty constructor is for 
reflection.

{code:java}
// Default constructor needed for reflection object creation
public TimeWindowedSerde() {
super(new TimeWindowedSerializer<>(), new TimeWindowedDeserializer<>());
}

public TimeWindowedSerde(final Serde inner) {
 super(new TimeWindowedSerializer<>(inner.serializer()), new 
TimeWindowedDeserializer<>(inner.deserializer()));
}
{code}

All above failes for me when I try to implicitly access the right Serde:
{code:java}
private val twSerde = implicitly[TimeWindowedSerde[String]]
{code}

and I have to create the object properly on my own
{code}
  private val twSerde = new 
WindowedSerdes.TimeWindowedSerde[String](implicitly[Serde[String]])
{code}

it could be fixed with a proper call in {{Serdes.scala}}
{code}
  implicit def timeWindowedSerde[T](implicit tSerde: Serde[T]): 
WindowedSerdes.TimeWindowedSerde[T] =
new WindowedSerdes.TimeWindowedSerde[T](tSerde)
{code}

But maybe also the scope of the default constructor for {{TimeWindowedSerde}} 
should be changed?

BR, Michał








--
This message was sent by Atlassian Jira
(v8.3.2#803003)

RE: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

[Pellerin, Clement]

There are pending issues we need to address.

We want to be able to call config.getConfiguredInstance(key, Class) to 
instantiate the extension point. This requires a default constructor. The 
former constructor arguments must now be passed in a separate init() method. 
This has the advantage of moving the constructor signature from the comment 
prose to the compiled language. I took inspiration from MetricsReporter for the 
init() method.

I question the object oriented design that requires the reconfigurableConfigs() 
method but declares the interface to be non-reconfigurable with just the 
Configurable interface.

My use case removes all built-in SSL configs (except the interface class name 
of course). SslFactory should not hardcode any SSL configs in the 
reconfigurableConfigs. It should delegate to the interface instance for all 
reconfigurableConfigs. In particular, it cannot assume there are keystores and 
truststores to validate. These checks should be moved to 
DefaultSslEngineFactory. We can then consider moving the SslEngine validation 
from SslFactory to SslChannelBuilder. What would be left in SslFactory that 
forces us to keep it instead of making it the Reconfigurable extension point 
itself?

I believe we don't need the sslContext() method since it is only used by a 
junit.
If we support my use case, there is a good chance we don't need the keystore() 
and truststore() method.

I am still not comfortable with the fact that reconfigurableConfigs() are not 
known until the SslEngineFactory implementation is created and that happens 
only after configure() is called. Notice this goes away if SslFactory is the 
extension point, which would explain why this might not have been an issue with 
the other extension points exposing reconfigurable custom configs.

We must document if the configs sent to the extension point implementation have 
been validated or not. I am pretty sure they are not since 
config.getConfiguredInstance() passes the originals() and there is no 
configurableConfigs() method (there is only the subset in 
reconfigurableConfigs()). Non-validated configs might be of the wrong type, be 
out of range, or missing since the default value is not applied. This is a 
burden to the extension point developer and Kafka should provide utilities for 
this.

Can you confirm where the special case for the reconfiguration of 
keystore/truststore is implemented? I am still trying to determine if it is 
possible to trigger a reconfiguration when none of the known configs have 
changed.


-Original Message-
From: Rajini Sivaram [mailto:rajinisiva...@gmail.com] 
Sent: Monday, September 16, 2019 5:29 AM
To: dev
Subject: Re: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

Hi Maulin,

Thanks for writing that up. I think we are getting close.

As an example interface (which can be adjusted as necessary, but just
including something here to refer to), I think we would have an interface
 along these lines:

public interface SslEngineFactory extends Configurable, Closeable {

Set reconfigurableConfigs();
boolean shouldBeRebuilt(Map nextConfigs);

SSLEngine createSslEngine(Mode mode, String peerHost, int
peerPort, String endpointIdentification);
SSLContext sslContext();
KeyStore keystore();
KeyStore truststore();
}

A) *Creation:*

   - We typically tend to include configs in constructor for non-pluggable
   classes, but use Configurable for pluggable classes. So Option 2 works
   better

B) *Returning reconfigurable configs: *

   - Custom SslEngineFactory implementations will only return what they
   care about in their implementation of reconfigurableConfigs().
   - SslChannelBuilder will delegate to SslFactory as you mentioned.
   - SslFactory.reconfigurableConfigs() will return
   SslConfigs.RECONFIGURABLE_CONFIGS plus SslEngineFactory.
reconfigurableConfigs(). So one day if we make endpoint validation
   reconfigurable, it would all just work. We can easily find a different way
   of continuing to reconfigure SslFactory without config changes if we
   needed to since it is not a pluggable class.

C) *Triggering reconfiguration:*

   - We continue to use AdminClient for reconfiguration in brokers with
   validation. That goes through SslEngineFactory.shouldBeRebuilt() and
   creates a new SslEngineFactory instance if needed.

D) *Handling push notification kind of needs*

   - For brokers, we want SslFactory to manage reconfiguration since
   configs may change. Also, AdminClient requests in brokers give a clear
   auditable path for config updates where update failures are returned to the
   caller. C) enables this.
   - For client-side, custom SslEngineFactory implementations could
   reconfigure themselves, we don't really need SslFactory to be involved
   at all.



On Sat, Sep 14, 2019 at 8:17 AM Maulin Vasavada 
wrote:

> Hi Clement/Rajini
>
> Based on what I get from my own understanding and your latest inputs, I'll
> write down what 

Re: [VOTE] KIP-511: Collect and Expose Client's Name and Version in the Brokers

[Mickael Maison]

+1 (non binding)
Thanks for the KIP!

On Mon, Sep 16, 2019 at 12:07 PM David Jacot  wrote:
>
> Hi all,
>
> I would like to start a vote on KIP-511:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-511%3A+Collect+and+Expose+Client%27s+Name+and+Version+in+the+Brokers
>
> Best,
> David

[VOTE] KIP-309: Add toUpperCase support to sasl.kerberos.principal.to.local rule

[Manikumar]

Hi all,

I would like to start vote on this trivial KIP-309:
https://cwiki.apache.org/confluence/display/KAFKA/KIP
-309%3A+Add+toUpperCase+support+to+sasl.kerberos.principal.to.local+rule

Thanks,

Re: [DISCUSS] Apache Kafka 2.4.0 release

[Manikumar]

Hi All,

Just a reminder that any new/pending KIP must pass vote by next Wednesday
(Sep 25, 2019) to be included
in Apache Kafka 2.4.0 release.

Also keep in mind that deadline for feature freeze is Oct 2, 2019.
In order to be included in the release, major features/KIPs need to be
merged and minor features need to be
have PR ready.  Any feature/KIP not in this state will be automatically
moved to the next release after Oct 2.

There are total 84 open

JIRAs. Please update your assigned JIRAs, if you know they cannot make it
to 2.4.0.
There are also quite a few JIRAs related to flaky tests. We really
appreciate any help on fixing these failing tests.

Thanks,
Manikumar

On Mon, Sep 16, 2019 at 4:08 PM Manikumar  wrote:

> Hi Mickael,
>
> Yes, we can include. Added KIP-396 to the wiki page for tracking.
>
>
> Thanks,
> Manikumar
>
> On Mon, Sep 16, 2019 at 3:36 PM Mickael Maison 
> wrote:
>
>> Hi Manikumar,
>>
>> Can we also include KIP-396?
>> (
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=97551484
>> )
>> It has been accepted and the PR is ready for review:
>> https://github.com/apache/kafka/pull/7296
>>
>> Thanks
>>
>> On Mon, Sep 16, 2019 at 10:56 AM Manikumar 
>> wrote:
>> >
>> > Hi Viktor,
>> >
>> > Yes, we can include KIP-434.
>> >
>> > Thanks,
>> >
>> > On Mon, Sep 16, 2019 at 3:09 PM Viktor Somogyi-Vass <
>> viktorsomo...@gmail.com>
>> > wrote:
>> >
>> > > Hi Manikumar,
>> > >
>> > > Can we please also include KIP-434?
>> > >
>> > >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-434%3A+Add+Replica+Fetcher+and+Log+Cleaner+Count+Metrics
>> > > It has been accepted and there is already a pull request under review.
>> > >
>> > > Thanks,
>> > > Viktor
>> > >
>> > > On Fri, Sep 6, 2019 at 9:59 AM Manikumar 
>> > > wrote:
>> > >
>> > > > Hi David,
>> > > >
>> > > > Yes, we can include KIP-511.  KIP must be accepted by KIP Freeze
>> date
>> > > (Sep
>> > > > 25, 2019 )
>> > > >
>> > > > Thanks,
>> > > >
>> > > >
>> > > > On Fri, Sep 6, 2019 at 12:53 PM David Jacot 
>> wrote:
>> > > >
>> > > > > Hi Manikumar,
>> > > > >
>> > > > > Could we add KIP-511 to the plan? I think it will make it.
>> > > > >
>> > > > > Thanks,
>> > > > > David
>> > > > >
>> > > > > On Tue, Aug 27, 2019 at 5:32 PM Manikumar <
>> manikumar.re...@gmail.com>
>> > > > > wrote:
>> > > > >
>> > > > > > Hi all,
>> > > > > >
>> > > > > > I put together a draft release plan with Oct 2019 as the release
>> > > month
>> > > > > and
>> > > > > > a list of KIPs that have already been voted:
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=125307901
>> > > > > >
>> > > > > > Here are the dates:
>> > > > > >
>> > > > > > 1) KIP Freeze:  Sep 25, 2019 (A KIP must be accepted by this
>> date in
>> > > > > order
>> > > > > > to be considered for this release)
>> > > > > >
>> > > > > > 2) Feature Freeze:  Oct 2, 2019 (Major features merged &
>> working on
>> > > > > > stabilization, minor features have PR,
>> > > > > >  release branch cut; anything not in this state will be
>> automatically
>> > > > > moved
>> > > > > > to the next release in JIRA.
>> > > > > >
>> > > > > > 3) Code Freeze:  Oct 16, 2019
>> > > > > >
>> > > > > > 4) Release Date: Oct 30, 2019 (tentative)
>> > > > > >
>> > > > > > Please plan accordingly for the features you want push into
>> Apache
>> > > > Kafka
>> > > > > > 2.4.0 release.
>> > > > > >
>> > > > > > Regards,
>> > > > > > Manikumar
>> > > > > >
>> > > > > > On Mon, Aug 12, 2019 at 9:08 PM Ismael Juma 
>> > > wrote:
>> > > > > >
>> > > > > > > Thanks for volunteering Manikumar. +1
>> > > > > > >
>> > > > > > > Ismael
>> > > > > > >
>> > > > > > > On Mon, Aug 12, 2019 at 7:54 AM Manikumar <
>> > > manikumar.re...@gmail.com
>> > > > >
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > Hi all,
>> > > > > > > >
>> > > > > > > > I would like to volunteer to be the release manager for our
>> next
>> > > > > > > time-based
>> > > > > > > > feature release (v2.4.0).
>> > > > > > > >
>> > > > > > > > If that sounds good, I'll post the release plan over the
>> next few
>> > > > > days.
>> > > > > > > >
>> > > > > > > > Thanks,
>> > > > > > > > Manikumar
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>>
>

[VOTE] KIP-511: Collect and Expose Client's Name and Version in the Brokers

[David Jacot]

Hi all,

I would like to start a vote on KIP-511:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-511%3A+Collect+and+Expose+Client%27s+Name+and+Version+in+the+Brokers

Best,
David

Re: [DISCUSS] KIP-511: Collect and Expose Client's Name and Version in the Brokers

[David Jacot]

Hi all,

I have updated the KIP to reflect the offline discussion that we have had.
It seems that we have finally
reached a consensus on the approach. Therefore, I will start a vote.

Best,
David


On Wed, Sep 11, 2019 at 3:53 PM David Jacot  wrote:

> Hi all,
>
> I have discussed with Magnus about the various options to get his view
> from a librdkafka perspective
> and he has suggested a good alternative.
>
> It seems we could continue with the idea to use the
> ApiVersionsRequest/Response but we a different
> failing back strategy. When a broker get an unknown ApiVersionsRequest, it
> could continue to fail back
> to version 0 as today but instead of sending back the UNSUPPORTED_VERSION
> error alone in the
> response, it could also provide at minimum the supported version of the
> ApiVersionsRequest. This way,
> a recent client could leverage that information when the error is received
> and send the correct version
> to the broker instead of failing all the way back to version 0.
>
> This way, we can evolve the ApiVersionsRequest while keeping forward
> compatibility of the Request
> Header. It doesn't add any extra round trip.
>
> Knowing this, I think that putting the information in the
> ApiVersionsRequest remains the best option.
>
> What do you think?
>
> Best,
> David
>
> On Tue, Sep 10, 2019 at 1:00 AM Colin McCabe  wrote:
>
>> Hi all,
>>
>> I agree that freezing the request header is not very appealing.  We might
>> want to add something there later.
>>
>> Having a separate request type is very clean, but as David mentioned, it
>> does add an extra request/response cycle to the time required to get a
>> connection into a usable state.
>>
>> One solution to consider is adding the clientVersion and clientType to
>> the request header as optional (tagged) fields.  This would let us skip the
>> extra round trip.  I don't think it's that much more messy than having a
>> separate request type to set the client version and type.  In both cases,
>> you have to handle connections that set the version later than others, or
>> don't set the version at all (for compatibility).  So the version/type has
>> to be mutable and added after the TCP connection itself is established.
>>
>> best,
>> Colin
>>
>>
>> On Mon, Sep 9, 2019, at 06:11, David Jacot wrote:
>> > Hi Gwen,
>> >
>> > The reasoning behind having the information before the Sasl
>> authentication
>> > was to have the information for troubleshooting purposes. For instance,
>> when
>> > there are errors in the handshake, it would be great to know if it comes
>> > from
>> > a specific buggy clients. I think we could live without though but was
>> nice
>> > to
>> > have.
>> >
>> > Yeah. I agree with you. It seems that evolution of the RequestHeader
>> (RH)
>> > and the
>> > ApiVersionsRequest (AVR) is conjunction is more complicated than I
>> > anticipated.
>> > The evolution if the AVR alone works well but it would make the
>> evolution
>> > of the
>> > RH hard in the future. Please, check my other email to see my thoughts.
>> >
>> > Best,
>> > David
>> >
>> > On Mon, Sep 9, 2019 at 3:18 AM Gwen Shapira  wrote:
>> >
>> > > Hey,
>> > >
>> > > Since modifying ApiVersionsRequest seems to be quite involved, do we
>> > > want to re-examine the rejected option of adding another
>> > > request/response pair? It will add another roundtrip, but Kafka
>> > > already expects client connections to be long-lived, so the overhead
>> > > is probably negligible.
>> > >
>> > > In addition, in the rejected alternatives it says: "It also would
>> > > require to be done before the authentication (TLS AuthN aside) and
>> > > thus requiring specific treatment, similarly to the
>> > > ApiVersionsRequest." - which I don't quite understand. Why do we think
>> > > this has to happen before authenticating?
>> > >
>> > > It sounds like addition another protocol will allow us to keep the
>> > > special assumptions around ApiVersionsRequest and will remove the
>> > > dependency on KIP-482. Which will make KIP-511 much simpler than the
>> > > alternative we are currently discussing.
>> > >
>> > > Gwen
>> > >
>> > > On Fri, Sep 6, 2019 at 3:16 PM Colin McCabe 
>> wrote:
>> > > >
>> > > > Hi David,
>> > > >
>> > > > Yeah, this is kind of difficult.
>> > > >
>> > > > From the high level, we either need forward compatibility (old
>> brokers
>> > > able to read new ApiVersionsRequests) or some kind of
>> challenge/response
>> > > system.  I think you're on the right track to be thinking about
>> forward
>> > > compatibility...  a challenge/response system would have a lot of
>> overhead
>> > > in cases where the client opens a lot of new connections.
>> > > >
>> > > > I agree that we can have the new versions "add stuff at the end" to
>> > > maintain forward compatibility.  Tagged fields will also help here, by
>> > > avoiding the need for version bumps.  We also have to think about the
>> > > impact this will have on the request header.  It seems like we might
>> need
>> > > to freeze 

Re: [DISCUSS] Apache Kafka 2.4.0 release

[Manikumar]

Hi Mickael,

Yes, we can include. Added KIP-396 to the wiki page for tracking.


Thanks,
Manikumar

On Mon, Sep 16, 2019 at 3:36 PM Mickael Maison 
wrote:

> Hi Manikumar,
>
> Can we also include KIP-396?
> (https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=97551484
> )
> It has been accepted and the PR is ready for review:
> https://github.com/apache/kafka/pull/7296
>
> Thanks
>
> On Mon, Sep 16, 2019 at 10:56 AM Manikumar 
> wrote:
> >
> > Hi Viktor,
> >
> > Yes, we can include KIP-434.
> >
> > Thanks,
> >
> > On Mon, Sep 16, 2019 at 3:09 PM Viktor Somogyi-Vass <
> viktorsomo...@gmail.com>
> > wrote:
> >
> > > Hi Manikumar,
> > >
> > > Can we please also include KIP-434?
> > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-434%3A+Add+Replica+Fetcher+and+Log+Cleaner+Count+Metrics
> > > It has been accepted and there is already a pull request under review.
> > >
> > > Thanks,
> > > Viktor
> > >
> > > On Fri, Sep 6, 2019 at 9:59 AM Manikumar 
> > > wrote:
> > >
> > > > Hi David,
> > > >
> > > > Yes, we can include KIP-511.  KIP must be accepted by KIP Freeze date
> > > (Sep
> > > > 25, 2019 )
> > > >
> > > > Thanks,
> > > >
> > > >
> > > > On Fri, Sep 6, 2019 at 12:53 PM David Jacot 
> wrote:
> > > >
> > > > > Hi Manikumar,
> > > > >
> > > > > Could we add KIP-511 to the plan? I think it will make it.
> > > > >
> > > > > Thanks,
> > > > > David
> > > > >
> > > > > On Tue, Aug 27, 2019 at 5:32 PM Manikumar <
> manikumar.re...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I put together a draft release plan with Oct 2019 as the release
> > > month
> > > > > and
> > > > > > a list of KIPs that have already been voted:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=125307901
> > > > > >
> > > > > > Here are the dates:
> > > > > >
> > > > > > 1) KIP Freeze:  Sep 25, 2019 (A KIP must be accepted by this
> date in
> > > > > order
> > > > > > to be considered for this release)
> > > > > >
> > > > > > 2) Feature Freeze:  Oct 2, 2019 (Major features merged & working
> on
> > > > > > stabilization, minor features have PR,
> > > > > >  release branch cut; anything not in this state will be
> automatically
> > > > > moved
> > > > > > to the next release in JIRA.
> > > > > >
> > > > > > 3) Code Freeze:  Oct 16, 2019
> > > > > >
> > > > > > 4) Release Date: Oct 30, 2019 (tentative)
> > > > > >
> > > > > > Please plan accordingly for the features you want push into
> Apache
> > > > Kafka
> > > > > > 2.4.0 release.
> > > > > >
> > > > > > Regards,
> > > > > > Manikumar
> > > > > >
> > > > > > On Mon, Aug 12, 2019 at 9:08 PM Ismael Juma 
> > > wrote:
> > > > > >
> > > > > > > Thanks for volunteering Manikumar. +1
> > > > > > >
> > > > > > > Ismael
> > > > > > >
> > > > > > > On Mon, Aug 12, 2019 at 7:54 AM Manikumar <
> > > manikumar.re...@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > I would like to volunteer to be the release manager for our
> next
> > > > > > > time-based
> > > > > > > > feature release (v2.4.0).
> > > > > > > >
> > > > > > > > If that sounds good, I'll post the release plan over the
> next few
> > > > > days.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Manikumar
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>

Re: [DISCUSS] Apache Kafka 2.4.0 release

[Mickael Maison]

Hi Manikumar,

Can we also include KIP-396?
(https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=97551484)
It has been accepted and the PR is ready for review:
https://github.com/apache/kafka/pull/7296

Thanks

On Mon, Sep 16, 2019 at 10:56 AM Manikumar  wrote:
>
> Hi Viktor,
>
> Yes, we can include KIP-434.
>
> Thanks,
>
> On Mon, Sep 16, 2019 at 3:09 PM Viktor Somogyi-Vass 
> wrote:
>
> > Hi Manikumar,
> >
> > Can we please also include KIP-434?
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-434%3A+Add+Replica+Fetcher+and+Log+Cleaner+Count+Metrics
> > It has been accepted and there is already a pull request under review.
> >
> > Thanks,
> > Viktor
> >
> > On Fri, Sep 6, 2019 at 9:59 AM Manikumar 
> > wrote:
> >
> > > Hi David,
> > >
> > > Yes, we can include KIP-511.  KIP must be accepted by KIP Freeze date
> > (Sep
> > > 25, 2019 )
> > >
> > > Thanks,
> > >
> > >
> > > On Fri, Sep 6, 2019 at 12:53 PM David Jacot  wrote:
> > >
> > > > Hi Manikumar,
> > > >
> > > > Could we add KIP-511 to the plan? I think it will make it.
> > > >
> > > > Thanks,
> > > > David
> > > >
> > > > On Tue, Aug 27, 2019 at 5:32 PM Manikumar 
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I put together a draft release plan with Oct 2019 as the release
> > month
> > > > and
> > > > > a list of KIPs that have already been voted:
> > > > >
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=125307901
> > > > >
> > > > > Here are the dates:
> > > > >
> > > > > 1) KIP Freeze:  Sep 25, 2019 (A KIP must be accepted by this date in
> > > > order
> > > > > to be considered for this release)
> > > > >
> > > > > 2) Feature Freeze:  Oct 2, 2019 (Major features merged & working on
> > > > > stabilization, minor features have PR,
> > > > >  release branch cut; anything not in this state will be automatically
> > > > moved
> > > > > to the next release in JIRA.
> > > > >
> > > > > 3) Code Freeze:  Oct 16, 2019
> > > > >
> > > > > 4) Release Date: Oct 30, 2019 (tentative)
> > > > >
> > > > > Please plan accordingly for the features you want push into Apache
> > > Kafka
> > > > > 2.4.0 release.
> > > > >
> > > > > Regards,
> > > > > Manikumar
> > > > >
> > > > > On Mon, Aug 12, 2019 at 9:08 PM Ismael Juma 
> > wrote:
> > > > >
> > > > > > Thanks for volunteering Manikumar. +1
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Mon, Aug 12, 2019 at 7:54 AM Manikumar <
> > manikumar.re...@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I would like to volunteer to be the release manager for our next
> > > > > > time-based
> > > > > > > feature release (v2.4.0).
> > > > > > >
> > > > > > > If that sounds good, I'll post the release plan over the next few
> > > > days.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Manikumar
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >

Re: [VOTE] KIP-373: Allow users to create delegation tokens for other users

[Viktor Somogyi-Vass]

Hi All,

I'd like to bump this again in order to get some more binding votes and/or
feedback in the hope we can push this in for 2.4.

Thank you Manikumar, Gabor and Ryanne so far for the votes! (the last two
were on the discussion thread after starting the vote but I think it still
counts :) )

Thanks,
Viktor

On Wed, Aug 21, 2019 at 1:44 PM Manikumar  wrote:

> Hi,
>
> +1 (binding).
>
> Thanks for the updated KIP. LGTM.
>
> Thanks,
> Manikumar
>
>
>
> On Tue, Aug 6, 2019 at 3:14 PM Viktor Somogyi-Vass <
> viktorsomo...@gmail.com>
> wrote:
>
> > Hi All,
> >
> > Bumping this, I'd be happy to get some additional feedback and/or votes.
> >
> > Thanks,
> > Viktor
> >
> > On Wed, Jul 31, 2019 at 11:04 AM Viktor Somogyi-Vass <
> > viktorsomo...@gmail.com> wrote:
> >
> > > Hi All,
> > >
> > > I'd like to start a vote on this KIP.
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-373%3A+Allow+users+to+create+delegation+tokens+for+other+users
> > >
> > > To summarize it: the proposed feature would allow users (usually
> > > superusers) to create delegation tokens for other users. This is
> > especially
> > > helpful in Spark where the delegation token created this way can be
> > > distributed to workers.
> > >
> > > I'd be happy to receive any votes or additional feedback.
> > >
> > > Viktor
> > >
> >
>

Re: [DISCUSS] Apache Kafka 2.4.0 release

[Manikumar]

Hi Viktor,

Yes, we can include KIP-434.

Thanks,

On Mon, Sep 16, 2019 at 3:09 PM Viktor Somogyi-Vass 
wrote:

> Hi Manikumar,
>
> Can we please also include KIP-434?
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-434%3A+Add+Replica+Fetcher+and+Log+Cleaner+Count+Metrics
> It has been accepted and there is already a pull request under review.
>
> Thanks,
> Viktor
>
> On Fri, Sep 6, 2019 at 9:59 AM Manikumar 
> wrote:
>
> > Hi David,
> >
> > Yes, we can include KIP-511.  KIP must be accepted by KIP Freeze date
> (Sep
> > 25, 2019 )
> >
> > Thanks,
> >
> >
> > On Fri, Sep 6, 2019 at 12:53 PM David Jacot  wrote:
> >
> > > Hi Manikumar,
> > >
> > > Could we add KIP-511 to the plan? I think it will make it.
> > >
> > > Thanks,
> > > David
> > >
> > > On Tue, Aug 27, 2019 at 5:32 PM Manikumar 
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I put together a draft release plan with Oct 2019 as the release
> month
> > > and
> > > > a list of KIPs that have already been voted:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=125307901
> > > >
> > > > Here are the dates:
> > > >
> > > > 1) KIP Freeze:  Sep 25, 2019 (A KIP must be accepted by this date in
> > > order
> > > > to be considered for this release)
> > > >
> > > > 2) Feature Freeze:  Oct 2, 2019 (Major features merged & working on
> > > > stabilization, minor features have PR,
> > > >  release branch cut; anything not in this state will be automatically
> > > moved
> > > > to the next release in JIRA.
> > > >
> > > > 3) Code Freeze:  Oct 16, 2019
> > > >
> > > > 4) Release Date: Oct 30, 2019 (tentative)
> > > >
> > > > Please plan accordingly for the features you want push into Apache
> > Kafka
> > > > 2.4.0 release.
> > > >
> > > > Regards,
> > > > Manikumar
> > > >
> > > > On Mon, Aug 12, 2019 at 9:08 PM Ismael Juma 
> wrote:
> > > >
> > > > > Thanks for volunteering Manikumar. +1
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Mon, Aug 12, 2019 at 7:54 AM Manikumar <
> manikumar.re...@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I would like to volunteer to be the release manager for our next
> > > > > time-based
> > > > > > feature release (v2.4.0).
> > > > > >
> > > > > > If that sounds good, I'll post the release plan over the next few
> > > days.
> > > > > >
> > > > > > Thanks,
> > > > > > Manikumar
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-524: Allow users to choose config source when describing configs

[Rajini Sivaram]

Thanks for the KIP Jason! Good idea.

On Sun, Sep 15, 2019 at 5:41 AM Gwen Shapira  wrote:

> Ah, great idea.
>
> On Fri, Sep 13, 2019 at 8:47 AM Jason Gustafson 
> wrote:
> >
> > Hi All,
> >
> > I have a minor KIP to improve the config tool:
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-524%3A+Allow+users+to+choose+config+source+when+describing+configs
> .
> > Let me know what you think.
> >
> > -Jason
>

Re: [DISCUSS] Apache Kafka 2.4.0 release

[Viktor Somogyi-Vass]

Hi Manikumar,

Can we please also include KIP-434?
https://cwiki.apache.org/confluence/display/KAFKA/KIP-434%3A+Add+Replica+Fetcher+and+Log+Cleaner+Count+Metrics
It has been accepted and there is already a pull request under review.

Thanks,
Viktor

On Fri, Sep 6, 2019 at 9:59 AM Manikumar  wrote:

> Hi David,
>
> Yes, we can include KIP-511.  KIP must be accepted by KIP Freeze date (Sep
> 25, 2019 )
>
> Thanks,
>
>
> On Fri, Sep 6, 2019 at 12:53 PM David Jacot  wrote:
>
> > Hi Manikumar,
> >
> > Could we add KIP-511 to the plan? I think it will make it.
> >
> > Thanks,
> > David
> >
> > On Tue, Aug 27, 2019 at 5:32 PM Manikumar 
> > wrote:
> >
> > > Hi all,
> > >
> > > I put together a draft release plan with Oct 2019 as the release month
> > and
> > > a list of KIPs that have already been voted:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=125307901
> > >
> > > Here are the dates:
> > >
> > > 1) KIP Freeze:  Sep 25, 2019 (A KIP must be accepted by this date in
> > order
> > > to be considered for this release)
> > >
> > > 2) Feature Freeze:  Oct 2, 2019 (Major features merged & working on
> > > stabilization, minor features have PR,
> > >  release branch cut; anything not in this state will be automatically
> > moved
> > > to the next release in JIRA.
> > >
> > > 3) Code Freeze:  Oct 16, 2019
> > >
> > > 4) Release Date: Oct 30, 2019 (tentative)
> > >
> > > Please plan accordingly for the features you want push into Apache
> Kafka
> > > 2.4.0 release.
> > >
> > > Regards,
> > > Manikumar
> > >
> > > On Mon, Aug 12, 2019 at 9:08 PM Ismael Juma  wrote:
> > >
> > > > Thanks for volunteering Manikumar. +1
> > > >
> > > > Ismael
> > > >
> > > > On Mon, Aug 12, 2019 at 7:54 AM Manikumar  >
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I would like to volunteer to be the release manager for our next
> > > > time-based
> > > > > feature release (v2.4.0).
> > > > >
> > > > > If that sounds good, I'll post the release plan over the next few
> > days.
> > > > >
> > > > > Thanks,
> > > > > Manikumar
> > > > >
> > > >
> > >
> >
>

[DISCUSS] KIP-522: Update BrokerApiVersionsCommand to use AdminClient

[Mickael Maison]

Hi all,

I've created a KIP to add listApiVersions support to the AdminClient.
This will allow us to update the BrokerApiVersionsCommand tool and
more importantly allow users to detect API support and build flexible
client applications:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-522%3A+Update+BrokerApiVersionsCommand+to+use+AdminClient

As always, feedback and comments are welcome!
Thanks

Re: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

[Rajini Sivaram]

Hi Maulin,

Thanks for writing that up. I think we are getting close.

As an example interface (which can be adjusted as necessary, but just
including something here to refer to), I think we would have an interface
 along these lines:

public interface SslEngineFactory extends Configurable, Closeable {

Set reconfigurableConfigs();
boolean shouldBeRebuilt(Map nextConfigs);

SSLEngine createSslEngine(Mode mode, String peerHost, int
peerPort, String endpointIdentification);
SSLContext sslContext();
KeyStore keystore();
KeyStore truststore();
}

A) *Creation:*

   - We typically tend to include configs in constructor for non-pluggable
   classes, but use Configurable for pluggable classes. So Option 2 works
   better

B) *Returning reconfigurable configs: *

   - Custom SslEngineFactory implementations will only return what they
   care about in their implementation of reconfigurableConfigs().
   - SslChannelBuilder will delegate to SslFactory as you mentioned.
   - SslFactory.reconfigurableConfigs() will return
   SslConfigs.RECONFIGURABLE_CONFIGS plus SslEngineFactory.
reconfigurableConfigs(). So one day if we make endpoint validation
   reconfigurable, it would all just work. We can easily find a different way
   of continuing to reconfigure SslFactory without config changes if we
   needed to since it is not a pluggable class.

C) *Triggering reconfiguration:*

   - We continue to use AdminClient for reconfiguration in brokers with
   validation. That goes through SslEngineFactory.shouldBeRebuilt() and
   creates a new SslEngineFactory instance if needed.

D) *Handling push notification kind of needs*

   - For brokers, we want SslFactory to manage reconfiguration since
   configs may change. Also, AdminClient requests in brokers give a clear
   auditable path for config updates where update failures are returned to the
   caller. C) enables this.
   - For client-side, custom SslEngineFactory implementations could
   reconfigure themselves, we don't really need SslFactory to be involved
   at all.



On Sat, Sep 14, 2019 at 8:17 AM Maulin Vasavada 
wrote:

> Hi Clement/Rajini
>
> Based on what I get from my own understanding and your latest inputs, I'll
> write down what my proposal is here and let us see if we can discuss on
> that,
>
> A) Creation
>
> 1. SslEngineFactory interface will not extend Configurable. Instead it will
> require the implementation to have constructor with config map
> 2. Currently, the caller is co-ordinating the call sequences between
> configure() and reconfigurableConfigs() in existing code base for
> SslChannelBuilder, SslFactory, ChannelBuilders and SocketServer. Hence I am
> not sure if we should get stuck arguing having a constructor vs having
> configure() to initialize the object. Personally I don't have a preference.
>
> B) Returning reconfigurable configs
>
> 1. SslEngineFactory interface will have reconfigurableConfigs() method
> which will return Set with values for custom (or with existing)
> config keys that it cares about.
> Here I agree that all existing SSL configurations may not at all be
> applicable to the pluggable engine. I don't see a point in keep referring
> to existing SSL Configs. Of course, the default implementation of the
> SslEngineFactory will return current SSL Configs since it relies on them to
> initialize keys/certs.
>
> 2. SslFacotry's reconfigurableConfigs() will delegate the call to the
> SslEngineFactory's reconfigurableConfigs(). The similar delegation will
> need to be done by SslChannelBuilder/SaslChannelBuilder to SslFactory for
> the same method (currently that are returning SslConfigs.
> RECONFIGURABLE_CONFIGS)
>
> C) Triggering reconfiguration
>
> 1. We can use AdminClient to trigger AlterConfig for any configs needed by
> the SslEngineFactory's implementation. For the default implementation it
> will follow the SSL Configs it uses today. For the custom implementation
> everything will funnel via 'custom config' as it is dealt with in below
> method,
>
>
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/DynamicBrokerConfig.scala#L507
>
> Btw, I don't see any check like if the reconfiguration needs to be
> triggered at the below line for Listeners,
>
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/DynamicBrokerConfig.scala#L515
>
> We do seem to have a check on Line#518 for other reconfigurables.
>
> D) Handling push notification kind of needs
>
> As noted by both of you, the reconfiguration will come via AdminClient
> probably from the SslEngineFactory's implementation itself when it knows
> the keys/certs changed and I need to ask Kafka to re-create myself.
> However, that approach ONLY works for Brokers. With keys/cert rotations we
> should address Client side challenge also. As you would have noticed in
> KIP-486 motivation the deployment challenge on Broker and Client side both
> makes it challenging to manage key/cert rotations etc. Hence I 

Re: [DISCUSS] KIP-495: Dynamically Adjust Log Levels in Connect

[Arjun Satish]

Good catch, Randall. Yes, we should be able to set the level of any logger
given its name. If this is an ancestor, then the levels of all child
classes are updated. I updated the KIP to be more explicit about what
loggers we can set, and how they affect child classes, if any.

Let me know what you think.

Best,

On Thu, Sep 12, 2019 at 5:02 PM Randall Hauch  wrote:

> Thanks for the KIP, Arjun. It's going to be really nice to be able to set
> the log levels dynamically, especially through the REST API.
>
> However, I think it's not clear what behavior the KIP is actually proposing
> with respect to which loggers are exposed (just those that are used, or
> common ancestors) and the behavior when I change the log level on a
> specific logger (is just that logger affected, or are descendants affected,
> too).
>
> For example, in a Log4J configuration file we can set the logger for
> packages (e.g., `org.apache.kafka`, `org.apache.kafka.connect`, etc.) or
> classes (e.g., `org.apache.kafka.connect.runtime.WorkerSinkTask`). Really,
> those are just conventions, because if the code created a logger for the
> "kafka.connect.worker" context then we could set that, too. So by
> convention, the loggers map to Kafka classes and packages.
>
> But it's unclear what behavior the KIP is proposing. Are the intermediate
> loggers such as all packages exposed as loggers? If so, if I set the logger
> on `org.apache.kafka.connect.runtime`, will this set the log level for all
> loggers below this?
>
> My concern is that if the behavior is (a) only concrete classes, and/or (b)
> setting a log level for a specific logger sets only that logger, then this
> deviates from what our users are familiar with when setting the log levels
> in the Log4J configuration files, and would be a difficult user experience
> if I have to set 30+ loggers rather than 1 or 2.
>
> On Thu, Sep 12, 2019 at 1:04 PM Jason Gustafson 
> wrote:
>
> > Thanks Arjun. +1
> >
> > On Thu, Sep 12, 2019 at 9:58 AM Gwen Shapira  wrote:
> >
> > > The new REST API for logger management looks great to me.
> > >
> > >
> > > On Thu, Sep 12, 2019 at 8:36 AM Arjun Satish 
> > > wrote:
> > > >
> > > > Bumping this thread.
> > > >
> > > > If there are no further comments, please add your votes here:
> > > > https://www.mail-archive.com/dev@kafka.apache.org/msg100313.html
> > > >
> > > > Thanks in advance,
> > > > Arjun
> > > >
> > > > On Fri, Sep 6, 2019 at 4:22 PM Arjun Satish 
> > > wrote:
> > > >
> > > > > Thanks a lot, Jason! Answers inline. I'll also modify the kip to
> make
> > > > > these clear.
> > > > >
> > > > > On Fri, Sep 6, 2019 at 4:01 PM Jason Gustafson  >
> > > wrote:
> > > > >
> > > > >> Hi Arjun,
> > > > >>
> > > > >> The updated KIP looks good. Just a couple questions:
> > > > >>
> > > > >> 1. Is the /admin endpoint on the normal listener by default? If
> not,
> > > is
> > > > >> there a way to have it use the same listener?
> > > > >>
> > > > >
> > > > > Uses the normal listener by default.
> > > > >
> > > > >
> > > > >> 2. Changes to logging configuration are not intended to be
> > > persistent, is
> > > > >> that right? Also, I assume changes only apply to the worker that
> > > received
> > > > >> the request?
> > > > >>
> > > > >
> > > > > Changes will not be persistent and only apply to the worker that
> > > received
> > > > > the request.
> > > > >
> > > > >
> > > > >> Thanks,
> > > > >> Jason
> > > > >>
> > > > >> On Fri, Aug 30, 2019 at 1:25 AM Arjun Satish <
> > arjun.sat...@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >> > OK. I didn't realize the plan was to deprecate and remove the
> JMX
> > > > >> endpoint.
> > > > >> > KIP-412 says brokers will continue to expose the JMX API. JMX
> was
> > > > >> selected
> > > > >> > so all components could follow the brokers. In light of this, I
> > > think we
> > > > >> > should simply aim for semantic equivalency across the different
> > API
> > > for
> > > > >> > this functionality.
> > > > >> >
> > > > >> > REST is convenient for Connect. We can modify the KIP to have a
> > > /admin
> > > > >> > endpoint, and /admin/loggers specifically for getting/setting
> the
> > > log
> > > > >> > levels of different loggers. The /admin/loggers will only impact
> > > loggers
> > > > >> > running in the specific worker we target requests to, and upon
> > > > >> restarting
> > > > >> > the worker, these loggers will reset back to their original
> level.
> > > > >> >
> > > > >> > Since configuring the rest server already has multiple config
> > keys,
> > > I am
> > > > >> > inclined to bundle this /admin endpoint on to the same listener,
> > and
> > > > >> > provide a single new config key that enables or disables the
> > entire
> > > > >> /admin
> > > > >> > endpoint. This keeps the initial approach simple and doesn't
> > require
> > > > >> users
> > > > >> > to configure/discover a new endpoint.
> > > > >> >
> > > > >> > If this works with you all, I can update the KIP. Please let me
> > know
> > > > >> what
> > > 

[jira] [Created] (KAFKA-8910) Incorrect javadoc at org.apache.kafka.clients.producer.Callback

[Sergey Ushakov (Jira)]

Sergey Ushakov created KAFKA-8910:
-

 Summary: Incorrect javadoc at 
org.apache.kafka.clients.producer.Callback
 Key: KAFKA-8910
 URL: https://issues.apache.org/jira/browse/KAFKA-8910
 Project: Kafka
  Issue Type: Bug
Reporter: Sergey Ushakov


h1. Problem

Javadoc for org.apache.kafka.clients.producer.Callback says:
{noformat}
* @param metadata The metadata for the record that was sent (i.e. the partition 
and offset). Null if an error
*occurred. {noformat}
In fact, metadata is never null after KAFKA-6180. See 
org.apache.kafka.clients.producer.KafkaProducer.InterceptorCallback#onCompletion
 for details.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Request to be added as a contributor

[Csenge Virág Maruzsi]

Hi Team,

I'd like to contribute to Kafka, and would like permissions for that.
Jira username: csenge
Thanks!

Regards,
Csenge