Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

On Sat, Jan 6, 2018 at 5:57 PM, Colin McCabe wrote: > On Thu, Jan 4, 2018, at 10:37, Jun Rao wrote: >> >> 4. The process to mark partition as dirty requires updating every fetch >> session having the partition. This may add some overhead. An alternative >> approach is to check the difference btw c

Build failed in Jenkins: kafka-1.0-jdk7 #120

See Changes: [jason] KAFKA-6422 Mirror maker will throw null pointer exception when the -- [...truncated 206.47 KB...] kafka.api.SslEndToEndAuthorizationTest > testNoConsum

Build failed in Jenkins: kafka-trunk-jdk8 #2318

See Changes: [jason] KAFKA-6096: Add multi-threaded tests for group coordinator, txn manager -- [...truncated 407.45 KB...] kafka.utils.CoreUtilsTest > testReadInt STARTE

Re: [VOTE] KIP-219 - Improve Quota Communication

Thanks for the comments, Jun. 1. Good point. 2. Also makes sense. Usually the connection.max.idle.ms is high enough so the throttling is impacted. I have updated the KIP to reflect the changes. Thanks, Jiangjie (Becket) Qin On Mon, Jan 8, 2018 at 6:30 PM, Jun Rao wrote: > Hi, Jiangjie, > >

Build failed in Jenkins: kafka-trunk-jdk8 #2317

See Changes: [jason] KAFKA-6422 Mirror maker will throw null pointer exception when the -- [...truncated 3.38 MB...] kafka.utils.CoreUtilsTest > testReadInt STARTED kafk

Re: [VOTE] KIP-219 - Improve Quota Communication

Hi, Jiangjie, Sorry for the late response. The proposal sounds good overall. A couple of minor comments below. 1. For throttling a fetch request, we could potentially just send an empty response. We can return a throttle time calculated from a full response, but only mute the channel on the serve

Build failed in Jenkins: kafka-trunk-jdk7 #3084

See Changes: [jason] KAFKA-6096: Add multi-threaded tests for group coordinator, txn manager -- [...truncated 400.63 KB...] kafka.tools.ConsoleProducerTest > testParseKey

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

Hi, Dong, Thanks for the updated KIP. A few more comments. 60. Perhaps having a partition epoch is more flexible since in the future, we may support deleting a partition as well. 61. It seems that the leader epoch returned in the position() call should the the leader epoch returned in the fetch

Re: [VOTE] KIP-174 Deprecate and remove internal converter configs in WorkerConfig

+1 binding On Mon, Jan 8, 2018 at 4:59 PM Ewen Cheslack-Postava wrote: > +1 binding. Thanks for the KIP! > > -Ewen > > On Mon, Jan 8, 2018 at 8:34 AM, Ted Yu wrote: > > > +1 > > > > On Mon, Jan 8, 2018 at 4:27 AM, UMESH CHAUDHARY > > wrote: > > > > > Hello All, > > > Since there are no outstan

Re: [VOTE] KIP-174 Deprecate and remove internal converter configs in WorkerConfig

+1 binding. Thanks for the KIP! -Ewen On Mon, Jan 8, 2018 at 8:34 AM, Ted Yu wrote: > +1 > > On Mon, Jan 8, 2018 at 4:27 AM, UMESH CHAUDHARY > wrote: > > > Hello All, > > Since there are no outstanding comments on this, so I'd like to start a > > vote. > > > > Please find the KIP here > >

Jenkins build is back to normal : kafka-trunk-jdk7 #3083

See

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

Hey Jason, Certainly. This sounds good. I have updated the KIP to clarity that the global epoch will be incremented by 1 each time a topic is deleted. Thanks, Dong On Mon, Jan 8, 2018 at 4:09 PM, Jason Gustafson wrote: > Hi Dong, > > > I think your approach will allow user to distinguish betwe

Re: [DISCUSS] KIP-208: Add SSL support to Kafka Connect REST interface

On Mon, Jan 8, 2018 at 11:39 AM, Randall Hauch wrote: > Nice feedback, Ewen. Thanks! > > On Thu, Jan 4, 2018 at 5:11 PM, Ewen Cheslack-Postava > wrote: > > > Hey Jakub, > > > > Sorry for not getting to this sooner. Overall the proposal looks good to > > me, I just had a couple of questions. > >

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

Hi Dong, I think your approach will allow user to distinguish between the metadata > before and after the topic deletion. I also agree that this can be > potentially be useful to user. I am just not very sure whether we already > have a good use-case to make the additional complexity worthwhile.

Jenkins build is back to normal : kafka-trunk-jdk8 #2316

See

[jira] [Created] (KAFKA-6433) Connect distributed workers should fail if their config is "incompatible" with leader's

Randall Hauch created KAFKA-6433: Summary: Connect distributed workers should fail if their config is "incompatible" with leader's Key: KAFKA-6433 URL: https://issues.apache.org/jira/browse/KAFKA-6433

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

Hey Ismael, I guess we actually need user to see this field so that user can store this value in the external store together with the offset. We just prefer the value to be opaque to discourage most users from interpreting this value. One more advantage of using such an opaque field is to be able

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

Hey Jason, Thanks for the comments. On Mon, Jan 8, 2018 at 2:27 PM, Jason Gustafson wrote: > > > > I am not sure I understand the benefit of incrementing this epoch after > > topic deletion. At a high level, client can not differentiate between > topic > > deletion and topic creation when the g

Re: [VOTE] KIP-226 - Dynamic Broker Configuration

Hi, Rajini, Thanks for the explanation. Then your suggestion sounds good to me. Jun On Mon, Jan 8, 2018 at 1:32 PM, Rajini Sivaram wrote: > Hi Jun, > > No, password.encoder.secret cannot be updated dynamically at the moment. > Dynamic configs are stored in ZooKeeper and since ZK is not secure

Build failed in Jenkins: kafka-trunk-jdk7 #3082

See Changes: [wangguoz] MINOR: fix image 404s in streams doc -- [...truncated 189.35 KB...] kafka.admin.ReassignPartitionsCommandTest > shouldFindMovingReplicasMultipleT

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

On Fri, Jan 5, 2018 at 7:15 PM, Jason Gustafson wrote: > > class OffsetAndMetadata { > long offset; > byte[] offsetMetadata; > String metadata; > } > Admittedly, the naming is a bit annoying, but we can probably come up with > something better. Internally the byte array would have a versio

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

> > I am not sure I understand the benefit of incrementing this epoch after > topic deletion. At a high level, client can not differentiate between topic > deletion and topic creation when the global epoch is incremented. Can you > provide more specific use-case? Say you send two metadata request

[jira] [Resolved] (KAFKA-6428) Fail builds on findbugs warnings

[ https://issues.apache.org/jira/browse/KAFKA-6428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava resolved KAFKA-6428. -- Resolution: Invalid Seems it was already setup to report & fail, the reporting o

Re: [VOTE] KIP-226 - Dynamic Broker Configuration

Hi Jun, No, password.encoder.secret cannot be updated dynamically at the moment. Dynamic configs are stored in ZooKeeper and since ZK is not secure, all password configs in ZK are encrypted using password.encoder.secret. We cannot make password.encoder.secret dynamic since it would need another s

Re: [VOTE] KIP-243: Make ProducerConfig and ConsumerConfig constructors public

We cannot simply change to without breaking a lot of code and making it a nightmare to fix it up... :( I will leave the VOTE open for some more time if people still want to comment. Otherwise, we would have 3 bindings vote now. -Matthias On 1/8/18 10:58 AM, Ewen Cheslack-Postava wrote: > +1 (b

Build failed in Jenkins: kafka-trunk-jdk8 #2315

See -- [...truncated 3.87 MB...] org.apache.kafka.connect.runtime.TransformationConfigTest > testEmbeddedConfigRegexRouter PASSED org.apache.kafka.connect.runtime.TransformationConfigTe

Re: [DISCUSS] KIP-212: Enforce set of legal characters for connector names

So are we ready to start a vote on this KIP? On Sat, Jan 6, 2018 at 6:00 PM, Ewen Cheslack-Postava wrote: > re: whitespace characters, I'm fine with the restriction since I don't see > it becoming an issue in practice. I just don't see any reason to restrict > it so it seems like we're going out

Re: [DISCUSS] KIP-208: Add SSL support to Kafka Connect REST interface

Nice feedback, Ewen. Thanks! On Thu, Jan 4, 2018 at 5:11 PM, Ewen Cheslack-Postava wrote: > Hey Jakub, > > Sorry for not getting to this sooner. Overall the proposal looks good to > me, I just had a couple of questions. > > 1. For the configs/overrides, does this happen on a per-setting basis or

[jira] [Resolved] (KAFKA-6387) Worker's producer and consumer configs should inherit from worker configs

[ https://issues.apache.org/jira/browse/KAFKA-6387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Randall Hauch resolved KAFKA-6387. -- Resolution: Won't Fix Because of the inability to maintain backward compatibility of the behavio

Re: [VOTE] KIP-243: Make ProducerConfig and ConsumerConfig constructors public

+1 (binding), though I still think the Map should be instead of . I also think its better to just expose the defaults as constants on the class. Apparently there was discussion of this before and the concern is that people write code that rely on the default configs and we might break their code

Re: [VOTE] KIP-226 - Dynamic Broker Configuration

Hi, Rajini, Could password.encoder.secret be updated dynamically? If so, each broker will still have access to the old secret when password.encoder.secret is updated. Perhaps that's a simpler way to handle changing secret than introducing an extra config. Thanks, Jun On Fri, Jan 5, 2018 at 3:09

Build failed in Jenkins: kafka-trunk-jdk8 #2314

See Changes: [romain_hardouin] Catch JsonMappingException subclass [damian.guy] MINOR: Add documentation for KAFKA-6086 (ProductionExceptionHandler) -- [...truncated 267

Re: [VOTE] KIP-174 Deprecate and remove internal converter configs in WorkerConfig

+1 On Mon, Jan 8, 2018 at 4:27 AM, UMESH CHAUDHARY wrote: > Hello All, > Since there are no outstanding comments on this, so I'd like to start a > vote. > > Please find the KIP here > 174+-+Deprecate+and+remove+internal+converter+configs+

Build failed in Jenkins: kafka-trunk-jdk7 #3081

See Changes: [ismael] MINOR: Catch JsonMappingException subclass (#3821) -- [...truncated 401.14 KB...] kafka.tools.ConsoleProducerTest > testParseKeyProp STARTED kafka.t

[VOTE] KIP-174 Deprecate and remove internal converter configs in WorkerConfig

Hello All, Since there are no outstanding comments on this, so I'd like to start a vote. Please find the KIP here and the related JIRA here

[jira] [Resolved] (KAFKA-4315) Kafka Connect documentation problems

[ https://issues.apache.org/jira/browse/KAFKA-4315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seweryn Habdank-Wojewodzki resolved KAFKA-4315. --- Resolution: Done I do not care anymore about this matter. > Kafka Con

[jira] [Resolved] (KAFKA-4908) consumer.properties logging warnings

[ https://issues.apache.org/jira/browse/KAFKA-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seweryn Habdank-Wojewodzki resolved KAFKA-4908. --- Resolution: Done Not an issue for me anymore. > consumer.properties l