Re: [DISCUSS] KIP-43: Kafka SASL enhancements

[Rajini Sivaram]

Harsha,

You are right, we don't expect to override callback handler or login for
Digest-MD5.

Pluggable CallbackHandler and Login modules enable custom SASL mechanisms
to be implemented without modifying Kafka. For instance, it would enable
KIP-44 (
https://cwiki.apache.org/confluence/display/KAFKA/KIP-44+-+Allow+Kafka+to+have+a+customized+security+protocol)
to be implemented without making the whole security protocol pluggable. Tao
Xiao has already confirmed earlier in this discussion thread that the
proposed callback handler and login interfaces are suitable for their
custom authentication.



On Sun, Mar 13, 2016 at 6:59 PM, Harsha  wrote:

> Agree with Gwen here. I feel like these additional pluggable Login
> Modules are making this KIP complex. Since the main goal of the KIP is
> to enable additional mechanism , can we limit the scope to that and If
> we feel necessary for pluggable Login and callback handler classes we
> can address in another JIRA.
>
> Adding digest-md5 ,password callbacks can be done to existing
> callbackhandler without  expose it as pluggable class. It would be
> useful to have broker support multiple mechanisms.  I haven't seen
> anyone using more than this in hadoop . It might be different for Kafka
> but I personally haven't seen anyone asking for this yet.
>
> Thanks,
> Harsha
>
>
> On Thu, Mar 10, 2016, at 01:44 AM, Rajini Sivaram wrote:
> > Gwen,
> >
> > Just to be clear, the alternative would be:
> >
> > *jaas.conf:*
> >
> > GssapiKafkaServer {
> >
> > com.ibm.security.auth.module.Krb5LoginModule required
> > credsType=both
> > useKeytab="file:/kafka/key.tab"
> > principal="kafka/localh...@example.com ";
> >
> > };
> >
> > SmartcardKafkaServer {
> >
> >   example.SmartcardLoginModule required
> >
> >   cardNumber=123;
> >
> > };
> >
> >
> > *KafkaConfig*
> >
> >
> >
> >- login.context.map={"GSSAPI="GssapiKafkaServer",
> >   "SMARTCARD"=SmartcardKafkaServer}
> >   - login.class.map={"GSSAPI=GssapiLogin.class,
> >   "SMARTCARD"=SmartcardLogin.class}
> >   - callback.handler.class.map={"GSSAPI"=GssapiCallbackHandler.class,
> >   "SMARTCARD"=SmartcardCallbackHandler.class}
> >
> > *Client Config *
> > Same as the server, but with only one entry allowed in each map and
> > jaas.conf
> >
> >
> >
> > This is a different model from the Java standard for supporting multiple
> > logins. As a developer, I am inclined to stick with approaches that are
> > widely in use like JSSE. But this alternative can be made to work if the
> > Kafka community feels it is more appropriate for Kafka. If you know of
> > other systems which use this approach, that would be helpful.
> >
> >
> >
> > On Thu, Mar 10, 2016 at 2:07 AM, Gwen Shapira  wrote:
> >
> > > What I'm hearing is that:
> > >
> > > 1. In order to support authentication mechanisms that were not written
> > > specifically with Kafka in mind, someone will need to write the
> > > integration between the mechanism and Kafka. This may include Login
> > > and CallbackHandler classes. This can be the mechanism vendor, the
> > > user or a 3rd party vendor.
> > > 2. If someone wrote the code to support a mechanism in Kafka, and a
> > > user will want to use more than one mechanism, they will still need to
> > > write a wrapper.
> > > 3. In reality, #2 will not be necessary ("edge-case") because Kafka
> > > will actually already provide the callback needed (and presumably also
> > > the code to load the LoginModule provided by Example.com)?
> > >
> > > Tradeoff #1 sounds reasonable.
> > > #2 and #3 do not sound reasonable considering one of the goals of the
> > > patch is to support multiple mechanisms. I don't think we should force
> > > our users to write code just to avoid writing it ourselves.
> > > Configuring security is complex enough as is.
> > > Furthermore, if we believe that "Smartcard is likely to use standard
> > > NameCallback and PasswordCallback already implemented in Kafka" - why
> > > do we even provide configuration for Login and CallbackHandler
> > > classes? Either we support multiple mechanisms written by different
> > > vendors, or we don't.
> > >
> > > Gwen
> > >
> > >
> > > On Wed, Mar 9, 2016 at 12:32 AM, Rajini Sivaram
> > >  wrote:
> > > > I am not saying that the developer at Example Inc. would develop a
> Login
> > > > implementation that combines Smartcard and Kerberos because Retailer
> uses
> > > > both. I am saying that Example Inc develops the LoginModule (similar
> to
> > > JVM
> > > > security providers developing Kerberos modules). But there is no
> standard
> > > > interface for Login to allow ticket refresh. So, it is very unlikely
> that
> > > > Example Inc would develop a Login implementation that works with an
> > > Apache
> > > > Kafka defined interface ( Kafka developers wrote this code for
> Kerberos).
> > > > For a custom integration, the user (i.e. Retailer) would be expected
> to
> > > > develop this code if required.
> > > >
> > > > You could imagine t

[jira] [Commented] (KAFKA-2165) ReplicaFetcherThread: data loss on unknown exception

[jiang tao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15192931#comment-15192931
 ] 

jiang tao commented on KAFKA-2165:
--

@Jun Rao  Recently,our production cluster have suddenly high network 
flow,increased 50M/s. after some days analyse, I find is the replication flow.

the cause is the follower replica offset is larger than master,and the follower 
replica offset  have been reset of the most begin,then the follower send 
another fetch request with the new offset,cause the follower out of ISR. and 
because the partition is also 100G,the follower make almost 1 hour to catch 
up,during the replication, the broker network load is very high,almost 
saturated.

> ReplicaFetcherThread: data loss on unknown exception
> 
>
> Key: KAFKA-2165
> URL: https://issues.apache.org/jira/browse/KAFKA-2165
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.1
>Reporter: Alexey Ozeritskiy
> Attachments: KAFKA-2165.patch
>
>
> Sometimes in our cluster some replica gets out of the isr. Then broker 
> redownloads the partition from the beginning. We got the following messages 
> in logs:
> {code}
> # The leader:
> [2015-03-25 11:11:07,796] ERROR [Replica Manager on Broker 21]: Error when 
> processing fetch request for partition [topic,11] offset 54369274 from 
> follower with correlation id 2634499. Possible cause: Request for offset 
> 54369274 but we only have log segments in the range 49322124 to 54369273. 
> (kafka.server.ReplicaManager)
> {code}
> {code}
> # The follower:
> [2015-03-25 11:11:08,816] WARN [ReplicaFetcherThread-0-21], Replica 31 for 
> partition [topic,11] reset its fetch offset from 49322124 to current leader 
> 21's start offset 49322124 (kafka.server.ReplicaFetcherThread)
> [2015-03-25 11:11:08,816] ERROR [ReplicaFetcherThread-0-21], Current offset 
> 54369274 for partition [topic,11] out of range; reset offset to 49322124 
> (kafka.server.ReplicaFetcherThread)
> {code}
> This occures because we update fetchOffset 
> [here|https://github.com/apache/kafka/blob/0.8.2/core/src/main/scala/kafka/server/AbstractFetcherThread.scala#L124]
>  and then try to process message. 
> If any exception except OffsetOutOfRangeCode occures we get unsynchronized 
> fetchOffset and replica.logEndOffset.
> On next fetch iteration we can get 
> fetchOffset>replica.logEndOffset==leaderEndOffset and OffsetOutOfRangeCode.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (KAFKA-2165) ReplicaFetcherThread: data loss on unknown exception

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma reopened KAFKA-2165:


> ReplicaFetcherThread: data loss on unknown exception
> 
>
> Key: KAFKA-2165
> URL: https://issues.apache.org/jira/browse/KAFKA-2165
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.1
>Reporter: Alexey Ozeritskiy
> Attachments: KAFKA-2165.patch
>
>
> Sometimes in our cluster some replica gets out of the isr. Then broker 
> redownloads the partition from the beginning. We got the following messages 
> in logs:
> {code}
> # The leader:
> [2015-03-25 11:11:07,796] ERROR [Replica Manager on Broker 21]: Error when 
> processing fetch request for partition [topic,11] offset 54369274 from 
> follower with correlation id 2634499. Possible cause: Request for offset 
> 54369274 but we only have log segments in the range 49322124 to 54369273. 
> (kafka.server.ReplicaManager)
> {code}
> {code}
> # The follower:
> [2015-03-25 11:11:08,816] WARN [ReplicaFetcherThread-0-21], Replica 31 for 
> partition [topic,11] reset its fetch offset from 49322124 to current leader 
> 21's start offset 49322124 (kafka.server.ReplicaFetcherThread)
> [2015-03-25 11:11:08,816] ERROR [ReplicaFetcherThread-0-21], Current offset 
> 54369274 for partition [topic,11] out of range; reset offset to 49322124 
> (kafka.server.ReplicaFetcherThread)
> {code}
> This occures because we update fetchOffset 
> [here|https://github.com/apache/kafka/blob/0.8.2/core/src/main/scala/kafka/server/AbstractFetcherThread.scala#L124]
>  and then try to process message. 
> If any exception except OffsetOutOfRangeCode occures we get unsynchronized 
> fetchOffset and replica.logEndOffset.
> On next fetch iteration we can get 
> fetchOffset>replica.logEndOffset==leaderEndOffset and OffsetOutOfRangeCode.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-2165) ReplicaFetcherThread: data loss on unknown exception

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-2165.

Resolution: Duplicate

> ReplicaFetcherThread: data loss on unknown exception
> 
>
> Key: KAFKA-2165
> URL: https://issues.apache.org/jira/browse/KAFKA-2165
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.1
>Reporter: Alexey Ozeritskiy
> Attachments: KAFKA-2165.patch
>
>
> Sometimes in our cluster some replica gets out of the isr. Then broker 
> redownloads the partition from the beginning. We got the following messages 
> in logs:
> {code}
> # The leader:
> [2015-03-25 11:11:07,796] ERROR [Replica Manager on Broker 21]: Error when 
> processing fetch request for partition [topic,11] offset 54369274 from 
> follower with correlation id 2634499. Possible cause: Request for offset 
> 54369274 but we only have log segments in the range 49322124 to 54369273. 
> (kafka.server.ReplicaManager)
> {code}
> {code}
> # The follower:
> [2015-03-25 11:11:08,816] WARN [ReplicaFetcherThread-0-21], Replica 31 for 
> partition [topic,11] reset its fetch offset from 49322124 to current leader 
> 21's start offset 49322124 (kafka.server.ReplicaFetcherThread)
> [2015-03-25 11:11:08,816] ERROR [ReplicaFetcherThread-0-21], Current offset 
> 54369274 for partition [topic,11] out of range; reset offset to 49322124 
> (kafka.server.ReplicaFetcherThread)
> {code}
> This occures because we update fetchOffset 
> [here|https://github.com/apache/kafka/blob/0.8.2/core/src/main/scala/kafka/server/AbstractFetcherThread.scala#L124]
>  and then try to process message. 
> If any exception except OffsetOutOfRangeCode occures we get unsynchronized 
> fetchOffset and replica.logEndOffset.
> On next fetch iteration we can get 
> fetchOffset>replica.logEndOffset==leaderEndOffset and OffsetOutOfRangeCode.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2165) ReplicaFetcherThread: data loss on unknown exception

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15192948#comment-15192948
 ] 

Ismael Juma commented on KAFKA-2165:


I changed the resolution to duplicate of KAFKA-2143, which was fixed in 
0.9.0.1. If you are seeing this with 0.9.0.1, then it's probably best to file a 
new issue with as much detail as possible.

> ReplicaFetcherThread: data loss on unknown exception
> 
>
> Key: KAFKA-2165
> URL: https://issues.apache.org/jira/browse/KAFKA-2165
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.1
>Reporter: Alexey Ozeritskiy
> Attachments: KAFKA-2165.patch
>
>
> Sometimes in our cluster some replica gets out of the isr. Then broker 
> redownloads the partition from the beginning. We got the following messages 
> in logs:
> {code}
> # The leader:
> [2015-03-25 11:11:07,796] ERROR [Replica Manager on Broker 21]: Error when 
> processing fetch request for partition [topic,11] offset 54369274 from 
> follower with correlation id 2634499. Possible cause: Request for offset 
> 54369274 but we only have log segments in the range 49322124 to 54369273. 
> (kafka.server.ReplicaManager)
> {code}
> {code}
> # The follower:
> [2015-03-25 11:11:08,816] WARN [ReplicaFetcherThread-0-21], Replica 31 for 
> partition [topic,11] reset its fetch offset from 49322124 to current leader 
> 21's start offset 49322124 (kafka.server.ReplicaFetcherThread)
> [2015-03-25 11:11:08,816] ERROR [ReplicaFetcherThread-0-21], Current offset 
> 54369274 for partition [topic,11] out of range; reset offset to 49322124 
> (kafka.server.ReplicaFetcherThread)
> {code}
> This occures because we update fetchOffset 
> [here|https://github.com/apache/kafka/blob/0.8.2/core/src/main/scala/kafka/server/AbstractFetcherThread.scala#L124]
>  and then try to process message. 
> If any exception except OffsetOutOfRangeCode occures we get unsynchronized 
> fetchOffset and replica.logEndOffset.
> On next fetch iteration we can get 
> fetchOffset>replica.logEndOffset==leaderEndOffset and OffsetOutOfRangeCode.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-2551) Unclean leader election docs outdated

[Manikumar Reddy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar Reddy resolved KAFKA-2551.

Resolution: Fixed

> Unclean leader election docs outdated
> -
>
> Key: KAFKA-2551
> URL: https://issues.apache.org/jira/browse/KAFKA-2551
> Project: Kafka
>  Issue Type: Bug
>  Components: website
>Affects Versions: 0.8.2.2
>Reporter: Stevo Slavic
>Assignee: Manikumar Reddy
>Priority: Trivial
>  Labels: documentation, newbie
> Fix For: 0.10.0.0
>
>
> Current unclean leader election docs state:
> {quote}
> In the future, we would like to make this configurable to better support use 
> cases where downtime is preferable to inconsistency.
> {quote}
> Since 0.8.2.0, unclean leader election strategy (whether to allow it or not) 
> is already configurable via {{unclean.leader.election.enable}} broker config 
> property.
> That sentence is in both 
> https://svn.apache.org/repos/asf/kafka/site/083/design.html and 
> https://svn.apache.org/repos/asf/kafka/site/082/design.html near the end of 
> "Unclean leader election: What if they all die?" section. Next section, 
> "Availability and Durability Guarantees", mentions ability to disable unclean 
> leader election, so likely just this one reference needs to be updated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Work started] (KAFKA-3188) Add system test for KIP-31 and KIP-32 - Compatibility Test

[Eno Thereska (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-3188 started by Eno Thereska.
---
> Add system test for KIP-31 and KIP-32 - Compatibility Test
> --
>
> Key: KAFKA-3188
> URL: https://issues.apache.org/jira/browse/KAFKA-3188
> Project: Kafka
>  Issue Type: Sub-task
>Affects Versions: 0.9.0.0
>Reporter: Jiangjie Qin
>Assignee: Eno Thereska
>Priority: Blocker
> Fix For: 0.10.0.0
>
>
> The integration test should test the compatibility between 0.10.0 broker with 
> clients on older versions. The clients version should include 0.9.0 and 0.8.x.
> We already cover 0.10 brokers with old producers/consumers in upgrade tests. 
> So, the main thing to test is a mix of 0.9 and 0.10 producers and consumers. 
> E.g., test1: 0.9 producer/0.10 consumer and then test2: 0.10 producer/0.9 
> consumer. And then, each of them: compression/no compression (like in upgrade 
> test). And we could probably add another dimension : topic configured with 
> CreateTime (default) and LogAppendTime. So, total 2x2x2 combinations (but 
> maybe can reduce that — eg. do LogAppendTime with compression only).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-493) High CPU usage on inactive server

[Cosmin Marginean (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15193245#comment-15193245
 ] 

Cosmin Marginean commented on KAFKA-493:


Thanks Andrew. Will do this in the next few weeks.

> High CPU usage on inactive server
> -
>
> Key: KAFKA-493
> URL: https://issues.apache.org/jira/browse/KAFKA-493
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.0
>Reporter: Jay Kreps
> Attachments: Kafka-2014-11-10.snapshot.zip, Kafka-sampling1.zip, 
> Kafka-sampling2.zip, Kafka-sampling3.zip, Kafka-trace1.zip, Kafka-trace2.zip, 
> Kafka-trace3.zip, backtraces.txt, stacktrace.txt
>
>
> > I've been playing with the 0.8 branch of Kafka and noticed that idle CPU 
> > usage is fairly high (13% of a 
> > core). Is that to be expected? I did look at the stack, but didn't see 
> > anything obvious. A background 
> > task?
> > I wanted to mention how I am getting into this state. I've set up two 
> > machines with the latest 0.8 
> > code base and am using a replication factor of 2. On starting the brokers 
> > there is no idle CPU activity. 
> > Then I run a test that essential does 10k publish operations followed by 
> > immediate consume operations 
> > (I was measuring latency). Once this has run the kafka nodes seem to 
> > consistently be consuming CPU 
> > essentially forever.
> hprof results:
> THREAD START (obj=53ae, id = 24, name="RMI TCP Accept-0", 
> group="system")
> THREAD START (obj=53ae, id = 25, name="RMI TCP Accept-", 
> group="system")
> THREAD START (obj=53ae, id = 26, name="RMI TCP Accept-0", 
> group="system")
> THREAD START (obj=53ae, id = 21, name="main", group="main")
> THREAD START (obj=53ae, id = 27, name="Thread-2", group="main")
> THREAD START (obj=53ae, id = 28, name="Thread-3", group="main")
> THREAD START (obj=53ae, id = 29, name="kafka-processor-9092-0", 
> group="main")
> THREAD START (obj=53ae, id = 200010, name="kafka-processor-9092-1", 
> group="main")
> THREAD START (obj=53ae, id = 200011, name="kafka-acceptor", group="main")
> THREAD START (obj=574b, id = 200012, 
> name="ZkClient-EventThread-20-localhost:2181", group="main")
> THREAD START (obj=576e, id = 200014, name="main-SendThread()", 
> group="main")
> THREAD START (obj=576d, id = 200013, name="main-EventThread", 
> group="main")
> THREAD START (obj=53ae, id = 200015, name="metrics-meter-tick-thread-1", 
> group="main")
> THREAD START (obj=53ae, id = 200016, name="metrics-meter-tick-thread-2", 
> group="main")
> THREAD START (obj=53ae, id = 200017, name="request-expiration-task", 
> group="main")
> THREAD START (obj=53ae, id = 200018, name="request-expiration-task", 
> group="main")
> THREAD START (obj=53ae, id = 200019, name="kafka-request-handler-0", 
> group="main")
> THREAD START (obj=53ae, id = 200020, name="kafka-request-handler-1", 
> group="main")
> THREAD START (obj=53ae, id = 200021, name="Thread-6", group="main")
> THREAD START (obj=53ae, id = 200022, name="Thread-7", group="main")
> THREAD START (obj=5899, id = 200023, name="ReplicaFetcherThread-0-2 on 
> broker 1, ", group="main")
> THREAD START (obj=5899, id = 200024, name="ReplicaFetcherThread-0-3 on 
> broker 1, ", group="main")
> THREAD START (obj=5899, id = 200025, name="ReplicaFetcherThread-0-0 on 
> broker 1, ", group="main")
> THREAD START (obj=5899, id = 200026, name="ReplicaFetcherThread-0-1 on 
> broker 1, ", group="main")
> THREAD START (obj=53ae, id = 200028, name="SIGINT handler", 
> group="system")
> THREAD START (obj=53ae, id = 200029, name="Thread-5", group="main")
> THREAD START (obj=574b, id = 200030, name="Thread-1", group="main")
> THREAD START (obj=574b, id = 200031, name="Thread-0", group="main")
> THREAD END (id = 200031)
> THREAD END (id = 200029)
> THREAD END (id = 200020)
> THREAD END (id = 200019)
> THREAD END (id = 28)
> THREAD END (id = 200021)
> THREAD END (id = 27)
> THREAD END (id = 200022)
> THREAD END (id = 200018)
> THREAD END (id = 200017)
> THREAD END (id = 200012)
> THREAD END (id = 200013)
> THREAD END (id = 200014)
> THREAD END (id = 200025)
> THREAD END (id = 200023)
> THREAD END (id = 200026)
> THREAD END (id = 200024)
> THREAD END (id = 200011)
> THREAD END (id = 29)
> THREAD END (id = 200010)
> THREAD END (id = 200030)
> THREAD END (id = 200028)
> TRACE 301281:
> sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java:Unknown 
> line)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> sun.nio.ch.SelectorImpl.select(Selecto

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ismael Juma]

Hi Ashish,

A few comments below.

On Fri, Mar 11, 2016 at 9:59 PM, Ashish Singh  wrote:

> Sounds like we are mostly in agreement. Following are the key points.
>
>1. Every time a protocol version changes, for any request/response,
>broker version, ApiVersion, will be bumped up.
>

Any thoughts on how we will enforce this?


>2. Protocol documentation will be versioned with broker version. Every
>time there is a broker version change, protocol documentation version
> needs
>to be updated and linked to main documentation page.
>3. Deprecation of protocol version will be done via marking the version
>as deprecated on the protocol documentation.
>

I think this is fine for the first version. We may consider doing more in
the future (logging a warning perhaps).


>4. On getting unknown protocol version, broker will send an empty
>response, instead of simply closing client connection.
>

I am not sure about this one. It's an unusual pattern and feels like a hack.

   5. Metadata response will be enhanced to also contain broker version,
>VersionInt and VersionString. VersionString will contain internal
>version information.
>

Even though Magnus suggested that it's OK for clients to parse
`VersionString`, I personally would rather avoid that. Do we really need 3
separate versions or could we get away with 2? I think it would be good to
elaborate on this a bit and explain how each of these versions would be
used (both from the broker and clients perspective).


>6. Metadata request with single null topic and size of -1 can be used to
>fetch metadata response with only broker version information and no
>topic/broker info.

   7. On receiving a metadata request with single null topic with size of
>-1, broker will respond with only broker version.
>

As Magnus says, the broker information should be returned. This would also
help us reduce unnecessary data transfer during NetworkClient's metadata
updates (KAFKA-3358). At the moment, we get information for all topics in
situations where we actually want no topics.

Also, I think it's a bit odd to say a `single null topic with size -1`. Do
we mean an array of topics with size -1 and no elements? That would imply
introducing a NULLABLE_ARRAY type (we currently have NULLABLE_STRING and
NULLABLE_BYTES).

Ismael

[GitHub] kafka pull request: MINOR: Review change to block.on.buffer.full c...

[granthenke]

GitHub user granthenke opened a pull request:

https://github.com/apache/kafka/pull/1058

MINOR: Review change to block.on.buffer.full config



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/granthenke/kafka block-on-buffer-full

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1058.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1058


commit 48690cb12311cf138128a30d81a35edb4315fdaf
Author: Grant Henke 
Date:   2016-03-14T14:03:52Z

MINOR: Review change to block.on.buffer.full config




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3388) Producer should only timeout a batch in the accumulator when metadata is missing.

[Mayuresh Gharat (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15193448#comment-15193448
 ] 

Mayuresh Gharat commented on KAFKA-3388:


If the batch is retried and re-enqueued, the last append time gets updated and 
that is used to time out the batch. So it should not expire immediately.

> Producer should only timeout a batch in the accumulator when metadata is 
> missing.
> -
>
> Key: KAFKA-3388
> URL: https://issues.apache.org/jira/browse/KAFKA-3388
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.1
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Blocker
> Fix For: 0.10.0.0
>
>
> In KIP-19 we are reusing the request.timeout.ms to timeout the batches in the 
> accumulator. We were intended to avoid the case that the batches sitting in 
> the accumulator forever when topic metadata is missing.
> Currently we are not checking if metadata is available or not when we timeout 
> the batches in the accumulator (although the comments says we will check the 
> metadata). This causes problem that once the previous batch hit a request 
> timeout and got retried, all the subsequent batches will fail with timeout 
> exception. We should only timeout the batches in the accumulator when the 
> metadata of the partition is missing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-3390) ReplicaManager may infinitely try-fail to shrink ISR set of deleted partition

[Mayuresh Gharat (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mayuresh Gharat reassigned KAFKA-3390:
--

Assignee: Mayuresh Gharat

> ReplicaManager may infinitely try-fail to shrink ISR set of deleted partition
> -
>
> Key: KAFKA-3390
> URL: https://issues.apache.org/jira/browse/KAFKA-3390
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.1
>Reporter: Stevo Slavic
>Assignee: Mayuresh Gharat
>
> For a topic whose deletion has been requested, Kafka replica manager may end 
> up infinitely trying and failing to shrink ISR.
> Here is fragment from server.log where this recurring and never ending 
> condition has been noticed:
> {noformat}
> [2016-03-04 09:42:13,894] INFO Partition [foo,0] on broker 1: Shrinking ISR 
> for partition [foo,0] from 1,3,2 to 1 (kafka.cluster.Partition)
> [2016-03-04 09:42:13,897] WARN Conditional update of path 
> /brokers/topics/foo/partitions/0/state with data 
> {"controller_epoch":53,"leader":1,"version":1,"leader_epoch":34,"isr":[1]} 
> and expected version 68 failed due to 
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = 
> NoNode for /brokers/topics/foo/partitions/0/state (kafka.utils.ZkUtils)
> [2016-03-04 09:42:13,898] INFO Partition [foo,0] on broker 1: Cached 
> zkVersion [68] not equal to that in zookeeper, skip updating ISR 
> (kafka.cluster.Partition)
> [2016-03-04 09:42:23,894] INFO Partition [foo,0] on broker 1: Shrinking ISR 
> for partition [foo,0] from 1,3,2 to 1 (kafka.cluster.Partition)
> [2016-03-04 09:42:23,897] WARN Conditional update of path 
> /brokers/topics/foo/partitions/0/state with data 
> {"controller_epoch":53,"leader":1,"version":1,"leader_epoch":34,"isr":[1]} 
> and expected version 68 failed due to 
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = 
> NoNode for /brokers/topics/foo/partitions/0/state (kafka.utils.ZkUtils)
> [2016-03-04 09:42:23,897] INFO Partition [foo,0] on broker 1: Cached 
> zkVersion [68] not equal to that in zookeeper, skip updating ISR 
> (kafka.cluster.Partition)
> [2016-03-04 09:42:33,894] INFO Partition [foo,0] on broker 1: Shrinking ISR 
> for partition [foo,0] from 1,3,2 to 1 (kafka.cluster.Partition)
> [2016-03-04 09:42:33,897] WARN Conditional update of path 
> /brokers/topics/foo/partitions/0/state with data 
> {"controller_epoch":53,"leader":1,"version":1,"leader_epoch":34,"isr":[1]} 
> and expected version 68 failed due to 
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = 
> NoNode for /brokers/topics/foo/partitions/0/state (kafka.utils.ZkUtils)
> [2016-03-04 09:42:33,897] INFO Partition [foo,0] on broker 1: Cached 
> zkVersion [68] not equal to that in zookeeper, skip updating ISR 
> (kafka.cluster.Partition)
> ...
> {noformat}
> Before topic deletion was requested, this was state in ZK of its sole 
> partition:
> {noformat}
> Zxid: 0x181045
> Cxid: 0xc92
> Client id:0x3532dd88fd2
> Time: Mon Feb 29 16:46:23 CET 2016
> Operation:setData
> Path: /brokers/topics/foo/partitions/0/state
> Data: 
> {"controller_epoch":53,"leader":1,"version":1,"leader_epoch":34,"isr":[1,3,2]}
> Version:  68
> {noformat}
> Topic (sole partition) had no data ever published to it. I guess at some 
> point after topic deletion has been requested, partition state first got 
> updated and this was updated state:
> {noformat}
> Zxid: 0x18b0be
> Cxid: 0x141e4
> Client id:0x3532dd88fd2
> Time: Fri Mar 04 9:41:52 CET 2016
> Operation:setData
> Path: /brokers/topics/foo/partitions/0/state
> Data: 
> {"controller_epoch":54,"leader":1,"version":1,"leader_epoch":35,"isr":[1,3]}
> Version:  69
> {noformat}
> For whatever reason replica manager (some cache it uses, I guess 
> ReplicaManager.allPartitions) never sees this update, nor does it see that 
> the partition state, partition, partitions node and finally topic node got 
> deleted:
> {noformat}
> Zxid: 0x18b0bf
> Cxid: 0x40fb
> Client id:0x3532dd88fd2000a
> Time: Fri Mar 04 9:41:52 CET 2016
> Operation:delete
> Path: /brokers/topics/foo/partitions/0/state
> ---
> Zxid: 0x18b0c0
> Cxid: 0x40fe
> Client id:0x3532dd88fd2000a
> Time: Fri Mar 04 9:41:52 CET 2016
> Operation:delete
> Path: /brokers/topics/foo/partitions/0
> ---
> Zxid: 0x18b0c1
> Cxid: 0x4100
> Client id:0x3532dd88fd2000a
> Time: Fri Mar 04 9:41:52 CET 2016
> Operation:delete
> Path: /brokers/topics/foo/partitions
> ---
> Zxid: 0x18b0c2
> Cxid: 0x4102
> Client id:0x3532dd88fd2000a
> Time: Fri Mar 04 9:41:52 CET 2

[jira] [Updated] (KAFKA-3248) AdminClient Blocks Forever in send Method

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3248:
---
Status: Patch Available  (was: Open)

> AdminClient Blocks Forever in send Method
> -
>
> Key: KAFKA-3248
> URL: https://issues.apache.org/jira/browse/KAFKA-3248
> Project: Kafka
>  Issue Type: Bug
>  Components: admin
>Affects Versions: 0.9.0.0
>Reporter: John Tylwalk
>Assignee: Warren Green
>Priority: Critical
> Fix For: 0.10.0.0
>
>
> AdminClient will block forever when performing operations involving the 
> {{send()}} method, due to usage of 
> {{ConsumerNetworkClient.poll(RequestFuture)}} - which blocks indefinitely.
> Suggested fix is to use {{ConsumerNetworkClient.poll(RequestFuture, long 
> timeout)}} in {{AdminClient.send()}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Gwen Shapira]

On Mon, Mar 14, 2016 at 7:05 AM, Ismael Juma  wrote:
> Hi Ashish,
>
> A few comments below.
>
> On Fri, Mar 11, 2016 at 9:59 PM, Ashish Singh  wrote:
>
>> Sounds like we are mostly in agreement. Following are the key points.
>>
>>1. Every time a protocol version changes, for any request/response,
>>broker version, ApiVersion, will be bumped up.
>>
>
> Any thoughts on how we will enforce this?

Code reviews? :)

We are already doing it in ApiVersion (and have been since
0.8.2.0-SNAPSHOT). Enforcing is awesome, but not necessarily part of
this KIP.

>
>
>>2. Protocol documentation will be versioned with broker version. Every
>>time there is a broker version change, protocol documentation version
>> needs
>>to be updated and linked to main documentation page.
>>3. Deprecation of protocol version will be done via marking the version
>>as deprecated on the protocol documentation.
>>
>
> I think this is fine for the first version. We may consider doing more in
> the future (logging a warning perhaps).
>
>
>>4. On getting unknown protocol version, broker will send an empty
>>response, instead of simply closing client connection.
>>
>
> I am not sure about this one. It's an unusual pattern and feels like a hack.

We discussed this and failed to come up with a better solution that
doesn't break compatibility.
Improvements can be added in the future - nothing can be worse than
current state (where the broker silently closes the connection)

>
>5. Metadata response will be enhanced to also contain broker version,
>>VersionInt and VersionString. VersionString will contain internal
>>version information.
>>
>
> Even though Magnus suggested that it's OK for clients to parse
> `VersionString`, I personally would rather avoid that. Do we really need 3
> separate versions or could we get away with 2? I think it would be good to
> elaborate on this a bit and explain how each of these versions would be
> used (both from the broker and clients perspective).

Agree! I'm also confused.

>
>>6. Metadata request with single null topic and size of -1 can be used to
>>fetch metadata response with only broker version information and no
>>topic/broker info.
>
>7. On receiving a metadata request with single null topic with size of
>>-1, broker will respond with only broker version.
>>
>
> As Magnus says, the broker information should be returned. This would also
> help us reduce unnecessary data transfer during NetworkClient's metadata
> updates (KAFKA-3358). At the moment, we get information for all topics in
> situations where we actually want no topics.
>
> Also, I think it's a bit odd to say a `single null topic with size -1`. Do
> we mean an array of topics with size -1 and no elements? That would imply
> introducing a NULLABLE_ARRAY type (we currently have NULLABLE_STRING and
> NULLABLE_BYTES).
>
> Ismael

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ismael Juma]

Hi Gwen,

On Mon, Mar 14, 2016 at 4:37 PM, Gwen Shapira  wrote:

> On Mon, Mar 14, 2016 at 7:05 AM, Ismael Juma  wrote:>
> >>1. Every time a protocol version changes, for any request/response,
> >>broker version, ApiVersion, will be bumped up.
> >>
> >
> > Any thoughts on how we will enforce this?
>
> Code reviews? :)
>
> We are already doing it in ApiVersion (and have been since
> 0.8.2.0-SNAPSHOT). Enforcing is awesome, but not necessarily part of
> this KIP.
>

What we have been doing since 0.8.2.0-SNAPSHOT is to create a new
`ApiVersion` for each new release. What is being proposed here is to create
a new `ApiVersion` every time a protocol version changes for any
request/response. This is much easier to miss. Admittedly, this approach
was introduced as part of KIP-31/32, but if we are going to expose this
version to clients, I think it is good to think about ways to ensure
correctness. It may be that we decide that it's out of scope or that we can
do it later, but I don't think we should just dismiss it without even
thinking about it.

>>4. On getting unknown protocol version, broker will send an empty
> >>response, instead of simply closing client connection.
> >>
> >
> > I am not sure about this one. It's an unusual pattern and feels like a
> hack.
>
> We discussed this and failed to come up with a better solution that
> doesn't break compatibility.
> Improvements can be added in the future - nothing can be worse than
> current state (where the broker silently closes the connection)
>

My understanding is that this doesn't help clients that support KIP-35
since they will know the broker version. And for older clients, they will
fail with a parsing exception, which is a bit better, but not much better.
So, is it really worth doing? In the KIP call we had about this months ago,
there was no consensus on this one from what I remember.

Ismael

[jira] [Created] (KAFKA-3393) Update site docs and javadoc based on max.block.ms changes

[Grant Henke (JIRA)]

Grant Henke created KAFKA-3393:
--

 Summary: Update site docs and javadoc based on max.block.ms changes
 Key: KAFKA-3393
 URL: https://issues.apache.org/jira/browse/KAFKA-3393
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.9.0.0
Reporter: Grant Henke
Assignee: Mayuresh Gharat


KAFKA-2120 deprecated block.on.buffer.full in favor of max.block.ms. This 
change alters the behavior of the KafkaProducer. Users may not be expecting 
that change when upgrading from the 0.8.x clients. We should:
- Update the KafkaProducer javadoc
- Update the ProducerConfig docs and the generated site docs
- Add an entry to the 0.9 upgrade notes (if appropriate) 

Related discussion can be seen here: https://github.com/apache/kafka/pull/1058



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3188: Compatibility test for old and new...

[enothereska]

GitHub user enothereska opened a pull request:

https://github.com/apache/kafka/pull/1059

KAFKA-3188: Compatibility test for old and new clients with 0.10 broker

@apovzner @becketqin please have a look if you can. Thanks.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/enothereska/kafka kafka-3188-compatibility

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1059.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1059


commit 18ce7127757f96fd256e67f6974a62881c3f6736
Author: Eno Thereska 
Date:   2016-03-14T17:23:49Z

Compatibility test for old and new clients with 0.10 broker




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3188) Add system test for KIP-31 and KIP-32 - Compatibility Test

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15193689#comment-15193689
 ] 

ASF GitHub Bot commented on KAFKA-3188:
---

GitHub user enothereska opened a pull request:

https://github.com/apache/kafka/pull/1059

KAFKA-3188: Compatibility test for old and new clients with 0.10 broker

@apovzner @becketqin please have a look if you can. Thanks.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/enothereska/kafka kafka-3188-compatibility

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1059.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1059


commit 18ce7127757f96fd256e67f6974a62881c3f6736
Author: Eno Thereska 
Date:   2016-03-14T17:23:49Z

Compatibility test for old and new clients with 0.10 broker




> Add system test for KIP-31 and KIP-32 - Compatibility Test
> --
>
> Key: KAFKA-3188
> URL: https://issues.apache.org/jira/browse/KAFKA-3188
> Project: Kafka
>  Issue Type: Sub-task
>Affects Versions: 0.9.0.0
>Reporter: Jiangjie Qin
>Assignee: Eno Thereska
>Priority: Blocker
> Fix For: 0.10.0.0
>
>
> The integration test should test the compatibility between 0.10.0 broker with 
> clients on older versions. The clients version should include 0.9.0 and 0.8.x.
> We already cover 0.10 brokers with old producers/consumers in upgrade tests. 
> So, the main thing to test is a mix of 0.9 and 0.10 producers and consumers. 
> E.g., test1: 0.9 producer/0.10 consumer and then test2: 0.10 producer/0.9 
> consumer. And then, each of them: compression/no compression (like in upgrade 
> test). And we could probably add another dimension : topic configured with 
> CreateTime (default) and LogAppendTime. So, total 2x2x2 combinations (but 
> maybe can reduce that — eg. do LogAppendTime with compression only).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Work started] (KAFKA-3202) Add system test for KIP-31 and KIP-32 - Change message format version on the fly

[Eno Thereska (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-3202 started by Eno Thereska.
---
> Add system test for KIP-31 and KIP-32 - Change message format version on the 
> fly
> 
>
> Key: KAFKA-3202
> URL: https://issues.apache.org/jira/browse/KAFKA-3202
> Project: Kafka
>  Issue Type: Sub-task
>  Components: system tests
>Reporter: Jiangjie Qin
>Assignee: Eno Thereska
>Priority: Blocker
> Fix For: 0.10.0.0
>
>
> The system test should cover the case that message format changes are made 
> when clients are producing/consuming. The message format change should not 
> cause client side issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[Reminder] 0.10.0 release candidate coming up in a week!

[Gwen Shapira]

Hey Team,

Just a reminder, we are planning to roll out the first release candidate
for 0.10.0 in exactly one week.

According to our release plan (
https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+0.10.0), we
have 13 open blockers (up from 12 on Thursday). Here's the list:

KeySummaryAssigneeReviewerStatus
KAFKA-2370


KAFKA-2365  Add
pause/unpause connector support

Jason Gustafson
 Gwen
Shapira
 OPEN
*Actions*

KAFKA-2832


support exclude.internal.topics in new consumer

Vahid Hashemian
 PATCH
AVAILABLE
*Actions*

KAFKA-3188


KAFKA-2511  Add system
test for KIP-31 and KIP-32 - Compatibility Test

Eno Thereska
 Ewen
Cheslack-Postava
 IN
PROGRESS
*Actions*

KAFKA-3190


KafkaProducer should not invoke callback in send()

Jiangjie Qin

Guozhang
Wang 
PATCH
AVAILABLE
*Actions*

KAFKA-3202


KAFKA-2511  Add system
test for KIP-31 and KIP-32 - Change message format version on the fly

Eno Thereska
 Ewen
Cheslack-Postava
 IN
PROGRESS
*Actions*

KAFKA-3303


Pass partial record metadata to Interceptor onAcknowledgement in case of
errors 
Anna Povzner
 Jun
Rao  PATCH
AVAILABLE
*Actions*

KAFKA-3315


Add Connect API to expose connector configuration info

Liquan Pei
 Ewen
Cheslack-Postava
 OPEN
*Actions*

KAFKA-3316


Add Connect REST API to list available connector classes

Ewen Cheslack-Postava
 Gwen
Shapira
 OPEN
*Actions*

[jira] [Commented] (KAFKA-3367) Delete topic dont delete the complete log from kafka

[Akshath Patkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15193801#comment-15193801
 ] 

Akshath Patkar commented on KAFKA-3367:
---

Thanks [~mgharat] will close this issue for now.

> Delete topic dont delete the complete log from kafka
> 
>
> Key: KAFKA-3367
> URL: https://issues.apache.org/jira/browse/KAFKA-3367
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Akshath Patkar
>
> Delete topic Just marks the topic as deleted. But data still remain in logs.
> How can we delete the topic completely with out doing manual delete of logs 
> from kafka and zookeeper



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ashish Singh]

Hello Gwen/ Ismael,

On Mon, Mar 14, 2016 at 9:55 AM, Ismael Juma  wrote:

> Hi Gwen,
>
> On Mon, Mar 14, 2016 at 4:37 PM, Gwen Shapira  wrote:
>
> > On Mon, Mar 14, 2016 at 7:05 AM, Ismael Juma  wrote:>
> > >>1. Every time a protocol version changes, for any request/response,
> > >>broker version, ApiVersion, will be bumped up.
> > >>
> > >
> > > Any thoughts on how we will enforce this?
> >
> > Code reviews? :)
> >
> > We are already doing it in ApiVersion (and have been since
> > 0.8.2.0-SNAPSHOT). Enforcing is awesome, but not necessarily part of
> > this KIP.
> >
>
> What we have been doing since 0.8.2.0-SNAPSHOT is to create a new
> `ApiVersion` for each new release. What is being proposed here is to create
> a new `ApiVersion` every time a protocol version changes for any
> request/response. This is much easier to miss. Admittedly, this approach
> was introduced as part of KIP-31/32, but if we are going to expose this
> version to clients, I think it is good to think about ways to ensure
> correctness. It may be that we decide that it's out of scope or that we can
> do it later, but I don't think we should just dismiss it without even
> thinking about it.
>
I think ApiVersion aims to identify protocol version changes, more than
release change,
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/api/ApiVersion.scala#L30.
I do agree that having an automated check to ensure this happens will be
really useful.

>
> >>4. On getting unknown protocol version, broker will send an empty
> > >>response, instead of simply closing client connection.
> > >>
> > >
> > > I am not sure about this one. It's an unusual pattern and feels like a
> > hack.
> >
> > We discussed this and failed to come up with a better solution that
> > doesn't break compatibility.
> > Improvements can be added in the future - nothing can be worse than
> > current state (where the broker silently closes the connection)
> >
>
> My understanding is that this doesn't help clients that support KIP-35
> since they will know the broker version. And for older clients, they will
> fail with a parsing exception, which is a bit better, but not much better.
> So, is it really worth doing? In the KIP call we had about this months ago,
> there was no consensus on this one from what I remember.
>
That is a good point! However, what about a client that wants to support
broker versions that do not provide broker version in metadata and broker
versions that provides version info in metadata. I think having this does
not cost us anything, but enables such clients to be smart.

>
> Ismael
>



-- 

Regards,
Ashish

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ashish Singh]

On Mon, Mar 14, 2016 at 9:37 AM, Gwen Shapira  wrote:

> On Mon, Mar 14, 2016 at 7:05 AM, Ismael Juma  wrote:
> > Hi Ashish,
> >
> > A few comments below.
> >
> > On Fri, Mar 11, 2016 at 9:59 PM, Ashish Singh 
> wrote:
> >
> >> Sounds like we are mostly in agreement. Following are the key points.
> >>
> >>1. Every time a protocol version changes, for any request/response,
> >>broker version, ApiVersion, will be bumped up.
> >>
> >
> > Any thoughts on how we will enforce this?
>
> Code reviews? :)
>
> We are already doing it in ApiVersion (and have been since
> 0.8.2.0-SNAPSHOT). Enforcing is awesome, but not necessarily part of
> this KIP.
>
> >
> >
> >>2. Protocol documentation will be versioned with broker version.
> Every
> >>time there is a broker version change, protocol documentation version
> >> needs
> >>to be updated and linked to main documentation page.
> >>3. Deprecation of protocol version will be done via marking the
> version
> >>as deprecated on the protocol documentation.
> >>
> >
> > I think this is fine for the first version. We may consider doing more in
> > the future (logging a warning perhaps).
> >
> >
> >>4. On getting unknown protocol version, broker will send an empty
> >>response, instead of simply closing client connection.
> >>
> >
> > I am not sure about this one. It's an unusual pattern and feels like a
> hack.
>
> We discussed this and failed to come up with a better solution that
> doesn't break compatibility.
> Improvements can be added in the future - nothing can be worse than
> current state (where the broker silently closes the connection)
>
> >
> >5. Metadata response will be enhanced to also contain broker version,
> >>VersionInt and VersionString. VersionString will contain internal
> >>version information.
> >>
> >
> > Even though Magnus suggested that it's OK for clients to parse
> > `VersionString`, I personally would rather avoid that. Do we really need
> 3
> > separate versions or could we get away with 2? I think it would be good
> to
> > elaborate on this a bit and explain how each of these versions would be
> > used (both from the broker and clients perspective).
>
> Agree! I'm also confused.
>
I am working on updating KIP and hopefully that will be less confusing.
What I meant was metadata response will have broker-version, which will be
made up of VersionInt and VersionString. For example, (4, "0.10.0-IV0"),
this will be based on respective ApiVersions,
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/api/ApiVersion.scala#L100
.

>
> >
> >>6. Metadata request with single null topic and size of -1 can be
> used to
> >>fetch metadata response with only broker version information and no
> >>topic/broker info.
> >
> >7. On receiving a metadata request with single null topic with size of
> >>-1, broker will respond with only broker version.
> >>
> >
> > As Magnus says, the broker information should be returned. This would
> also
> > help us reduce unnecessary data transfer during NetworkClient's metadata
> > updates (KAFKA-3358). At the moment, we get information for all topics in
> > situations where we actually want no topics.
> >
> > Also, I think it's a bit odd to say a `single null topic with size -1`.
> Do
> > we mean an array of topics with size -1 and no elements? That would imply
> > introducing a NULLABLE_ARRAY type (we currently have NULLABLE_STRING and
> > NULLABLE_BYTES).
> >
> > Ismael
>



-- 

Regards,
Ashish

[jira] [Commented] (KAFKA-2464) Client-side assignment and group generalization

[Jason Gustafson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15193852#comment-15193852
 ] 

Jason Gustafson commented on KAFKA-2464:


[~becket_qin] Looks like I missed this renaming. I doubt there are any 
compatibility concerns if we fix it, so I'll go ahead and submit a followup 
patch.

> Client-side assignment and group generalization
> ---
>
> Key: KAFKA-2464
> URL: https://issues.apache.org/jira/browse/KAFKA-2464
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Blocker
> Fix For: 0.9.0.0
>
>
> Add support for client-side assignment and generalization of join group 
> protocol as documented here: 
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Client-side+Assignment+Proposal.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Dana Powers]

There seems to be a lot of tension between support for release-deploys vs.
trunk-deploys. Is there a better way to handle this?

In my experience the vast majority of third-party client code is written
managing compatibility on the first case (release-deploys). I would prefer
a simple approach that is optimized for third-party clients that rarely
connect to trunk-deploys. As Magnus mentioned, the approach we take in
kafka-python is to attempt to identify the broker version -- 0.9, 0.8.2,
0.8.1, 0.8.0 -- and gate "blocks" of features based on that information.

Would it be reasonable to put the onus on the user to manage connecting to
trunk-deploys? If the broker always returns 'trunk' and the client is
configured to manually override the "broker version" via configuration,
would that work for people running trunk-deploys? For example, I might run
a trunk-deploy broker and configure my client to assume broker version
'0.10-dev' and write some client code to support that.

To be honest, I do not plan to release any code publicly (i.e., to pypi)
that is intended to support trunk-deploys. That really sounds like a
maintenance nightmare. I would expect anyone running a server pulled from
trunk to also run clients that aren't officially released / are in active
development.

-Dana


On Mon, Mar 14, 2016 at 11:19 AM, Ashish Singh  wrote:

> On Mon, Mar 14, 2016 at 9:37 AM, Gwen Shapira  wrote:
>
> > On Mon, Mar 14, 2016 at 7:05 AM, Ismael Juma  wrote:
> > > Hi Ashish,
> > >
> > > A few comments below.
> > >
> > > On Fri, Mar 11, 2016 at 9:59 PM, Ashish Singh 
> > wrote:
> > >
> > >> Sounds like we are mostly in agreement. Following are the key points.
> > >>
> > >>1. Every time a protocol version changes, for any request/response,
> > >>broker version, ApiVersion, will be bumped up.
> > >>
> > >
> > > Any thoughts on how we will enforce this?
> >
> > Code reviews? :)
> >
> > We are already doing it in ApiVersion (and have been since
> > 0.8.2.0-SNAPSHOT). Enforcing is awesome, but not necessarily part of
> > this KIP.
> >
> > >
> > >
> > >>2. Protocol documentation will be versioned with broker version.
> > Every
> > >>time there is a broker version change, protocol documentation
> version
> > >> needs
> > >>to be updated and linked to main documentation page.
> > >>3. Deprecation of protocol version will be done via marking the
> > version
> > >>as deprecated on the protocol documentation.
> > >>
> > >
> > > I think this is fine for the first version. We may consider doing more
> in
> > > the future (logging a warning perhaps).
> > >
> > >
> > >>4. On getting unknown protocol version, broker will send an empty
> > >>response, instead of simply closing client connection.
> > >>
> > >
> > > I am not sure about this one. It's an unusual pattern and feels like a
> > hack.
> >
> > We discussed this and failed to come up with a better solution that
> > doesn't break compatibility.
> > Improvements can be added in the future - nothing can be worse than
> > current state (where the broker silently closes the connection)
> >
> > >
> > >5. Metadata response will be enhanced to also contain broker
> version,
> > >>VersionInt and VersionString. VersionString will contain internal
> > >>version information.
> > >>
> > >
> > > Even though Magnus suggested that it's OK for clients to parse
> > > `VersionString`, I personally would rather avoid that. Do we really
> need
> > 3
> > > separate versions or could we get away with 2? I think it would be good
> > to
> > > elaborate on this a bit and explain how each of these versions would be
> > > used (both from the broker and clients perspective).
> >
> > Agree! I'm also confused.
> >
> I am working on updating KIP and hopefully that will be less confusing.
> What I meant was metadata response will have broker-version, which will be
> made up of VersionInt and VersionString. For example, (4, "0.10.0-IV0"),
> this will be based on respective ApiVersions,
>
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/api/ApiVersion.scala#L100
> .
>
> >
> > >
> > >>6. Metadata request with single null topic and size of -1 can be
> > used to
> > >>fetch metadata response with only broker version information and no
> > >>topic/broker info.
> > >
> > >7. On receiving a metadata request with single null topic with size
> of
> > >>-1, broker will respond with only broker version.
> > >>
> > >
> > > As Magnus says, the broker information should be returned. This would
> > also
> > > help us reduce unnecessary data transfer during NetworkClient's
> metadata
> > > updates (KAFKA-3358). At the moment, we get information for all topics
> in
> > > situations where we actually want no topics.
> > >
> > > Also, I think it's a bit odd to say a `single null topic with size -1`.
> > Do
> > > we mean an array of topics with size -1 and no elements? That would
> imply
> > > introducing a NULLABLE_ARRAY type (we currently have

[GitHub] kafka pull request: KAFKA-3393 : Updated the docs to reflect the d...

[MayureshGharat]

GitHub user MayureshGharat opened a pull request:

https://github.com/apache/kafka/pull/1060

KAFKA-3393 : Updated the docs to reflect the deprecation of 
block.on.buffer.full and usage of max.block.ms



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/MayureshGharat/kafka KAFKA-3393

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1060.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1060


commit 76ab834fff1f49d07bea621626df090937c3a276
Author: MayureshGharat 
Date:   2016-03-14T18:46:06Z

Updated the docs to reflect the deprecation of block.on.buffer.full and 
usage of max.block.ms




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3393) Update site docs and javadoc based on max.block.ms changes

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15193877#comment-15193877
 ] 

ASF GitHub Bot commented on KAFKA-3393:
---

GitHub user MayureshGharat opened a pull request:

https://github.com/apache/kafka/pull/1060

KAFKA-3393 : Updated the docs to reflect the deprecation of 
block.on.buffer.full and usage of max.block.ms



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/MayureshGharat/kafka KAFKA-3393

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1060.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1060


commit 76ab834fff1f49d07bea621626df090937c3a276
Author: MayureshGharat 
Date:   2016-03-14T18:46:06Z

Updated the docs to reflect the deprecation of block.on.buffer.full and 
usage of max.block.ms




> Update site docs and javadoc based on max.block.ms changes
> --
>
> Key: KAFKA-3393
> URL: https://issues.apache.org/jira/browse/KAFKA-3393
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.0
>Reporter: Grant Henke
>Assignee: Mayuresh Gharat
>
> KAFKA-2120 deprecated block.on.buffer.full in favor of max.block.ms. This 
> change alters the behavior of the KafkaProducer. Users may not be expecting 
> that change when upgrading from the 0.8.x clients. We should:
> - Update the KafkaProducer javadoc
> - Update the ProducerConfig docs and the generated site docs
> - Add an entry to the 0.9 upgrade notes (if appropriate) 
> Related discussion can be seen here: https://github.com/apache/kafka/pull/1058



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: TRIVIAL: remove TODO in ConsumerNetworkClient ...

[christian-posta]

GitHub user christian-posta opened a pull request:

https://github.com/apache/kafka/pull/1061

TRIVIAL: remove TODO in ConsumerNetworkClient after KAFKA-2120



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/christian-posta/kafka 
ceposta-trivial-remove-todo-after-KAFKA-2120

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1061.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1061


commit e5baf7cae7028c3840195328111fd45b4a403616
Author: Christian Posta 
Date:   2016-03-14T19:03:46Z

TRIVIAL: remove TODO in ConsumerNetworkClient after KAFKA-2120




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2120) Add a request timeout to NetworkClient

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15193917#comment-15193917
 ] 

ASF GitHub Bot commented on KAFKA-2120:
---

GitHub user christian-posta opened a pull request:

https://github.com/apache/kafka/pull/1061

TRIVIAL: remove TODO in ConsumerNetworkClient after KAFKA-2120



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/christian-posta/kafka 
ceposta-trivial-remove-todo-after-KAFKA-2120

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1061.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1061


commit e5baf7cae7028c3840195328111fd45b4a403616
Author: Christian Posta 
Date:   2016-03-14T19:03:46Z

TRIVIAL: remove TODO in ConsumerNetworkClient after KAFKA-2120




> Add a request timeout to NetworkClient
> --
>
> Key: KAFKA-2120
> URL: https://issues.apache.org/jira/browse/KAFKA-2120
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Jiangjie Qin
>Assignee: Mayuresh Gharat
>Priority: Blocker
> Fix For: 0.9.0.0
>
> Attachments: KAFKA-2120.patch, KAFKA-2120_2015-07-27_15:31:19.patch, 
> KAFKA-2120_2015-07-29_15:57:02.patch, KAFKA-2120_2015-08-10_19:55:18.patch, 
> KAFKA-2120_2015-08-12_10:59:09.patch, KAFKA-2120_2015-09-03_15:12:02.patch, 
> KAFKA-2120_2015-09-04_17:49:01.patch, KAFKA-2120_2015-09-09_16:45:44.patch, 
> KAFKA-2120_2015-09-09_18:56:18.patch, KAFKA-2120_2015-09-10_21:38:55.patch, 
> KAFKA-2120_2015-09-11_14:54:15.patch, KAFKA-2120_2015-09-15_18:57:20.patch, 
> KAFKA-2120_2015-09-18_19:27:48.patch, KAFKA-2120_2015-09-28_16:13:02.patch
>
>
> Currently NetworkClient does not have a timeout setting for requests. So if 
> no response is received for a request due to reasons such as broker is down, 
> the request will never be completed.
> Request timeout will also be used as implicit timeout for some methods such 
> as KafkaProducer.flush() and kafkaProducer.close().
> KIP-19 is created for this public interface change.
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-19+-+Add+a+request+timeout+to+NetworkClient



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-3394) Broker fails to parse Null Metadata in OffsetCommit requests

[Magnus Edenhill (JIRA)]

Magnus Edenhill created KAFKA-3394:
--

 Summary: Broker fails to parse Null Metadata in OffsetCommit 
requests
 Key: KAFKA-3394
 URL: https://issues.apache.org/jira/browse/KAFKA-3394
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.10.0.0
Reporter: Magnus Edenhill


librdkafka sends a Null Metadata string (size -1) in its OffsetCommitRequests 
when there is no metadata, this unfortunately leads to an exception on the 
broker that expects a non-null string.

{noformat}
[2016-03-11 11:11:57,623] ERROR Closing socket for 
10.191.0.33:9092-10.191.0.33:56503 because of error (kafka.network.Processor)
kafka.network.InvalidRequestException: Error getting request for apiKey: 8 and 
apiVersion: 1
at 
kafka.network.RequestChannel$Request.liftedTree2$1(RequestChannel.scala:91)
at kafka.network.RequestChannel$Request.(RequestChannel.scala:88)
at kafka.network.Processor$$anonfun$run$11.apply(SocketServer.scala:426)
at kafka.network.Processor$$anonfun$run$11.apply(SocketServer.scala:421)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at kafka.network.Processor.run(SocketServer.scala:421)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.kafka.common.protocol.types.SchemaException: Error 
reading field 'topics': Error reading field 'partitions': Error reading field 
'metadata': java.lang.NegativeArraySizeException
at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:73)
at 
org.apache.kafka.common.requests.OffsetCommitRequest.parse(OffsetCommitRequest.java:260)
at 
org.apache.kafka.common.requests.AbstractRequest.getRequest(AbstractRequest.java:50)
at 
kafka.network.RequestChannel$Request.liftedTree2$1(RequestChannel.scala:88)
... 9 more
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3394) Broker fails to parse Null Metadata in OffsetCommit requests

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3394:
---
Fix Version/s: 0.10.0.0

> Broker fails to parse Null Metadata in OffsetCommit requests
> 
>
> Key: KAFKA-3394
> URL: https://issues.apache.org/jira/browse/KAFKA-3394
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.0.0
>Reporter: Magnus Edenhill
> Fix For: 0.10.0.0
>
>
> librdkafka sends a Null Metadata string (size -1) in its OffsetCommitRequests 
> when there is no metadata, this unfortunately leads to an exception on the 
> broker that expects a non-null string.
> {noformat}
> [2016-03-11 11:11:57,623] ERROR Closing socket for 
> 10.191.0.33:9092-10.191.0.33:56503 because of error (kafka.network.Processor)
> kafka.network.InvalidRequestException: Error getting request for apiKey: 8 
> and apiVersion: 1
> at 
> kafka.network.RequestChannel$Request.liftedTree2$1(RequestChannel.scala:91)
> at kafka.network.RequestChannel$Request.(RequestChannel.scala:88)
> at kafka.network.Processor$$anonfun$run$11.apply(SocketServer.scala:426)
> at kafka.network.Processor$$anonfun$run$11.apply(SocketServer.scala:421)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at kafka.network.Processor.run(SocketServer.scala:421)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.kafka.common.protocol.types.SchemaException: Error 
> reading field 'topics': Error reading field 'partitions': Error reading field 
> 'metadata': java.lang.NegativeArraySizeException
> at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:73)
> at 
> org.apache.kafka.common.requests.OffsetCommitRequest.parse(OffsetCommitRequest.java:260)
> at 
> org.apache.kafka.common.requests.AbstractRequest.getRequest(AbstractRequest.java:50)
> at 
> kafka.network.RequestChannel$Request.liftedTree2$1(RequestChannel.scala:88)
> ... 9 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3394) Broker fails to parse Null Metadata in OffsetCommit requests

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3394:
---
Priority: Blocker  (was: Major)

> Broker fails to parse Null Metadata in OffsetCommit requests
> 
>
> Key: KAFKA-3394
> URL: https://issues.apache.org/jira/browse/KAFKA-3394
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.0.0
>Reporter: Magnus Edenhill
>Priority: Blocker
> Fix For: 0.10.0.0
>
>
> librdkafka sends a Null Metadata string (size -1) in its OffsetCommitRequests 
> when there is no metadata, this unfortunately leads to an exception on the 
> broker that expects a non-null string.
> {noformat}
> [2016-03-11 11:11:57,623] ERROR Closing socket for 
> 10.191.0.33:9092-10.191.0.33:56503 because of error (kafka.network.Processor)
> kafka.network.InvalidRequestException: Error getting request for apiKey: 8 
> and apiVersion: 1
> at 
> kafka.network.RequestChannel$Request.liftedTree2$1(RequestChannel.scala:91)
> at kafka.network.RequestChannel$Request.(RequestChannel.scala:88)
> at kafka.network.Processor$$anonfun$run$11.apply(SocketServer.scala:426)
> at kafka.network.Processor$$anonfun$run$11.apply(SocketServer.scala:421)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at kafka.network.Processor.run(SocketServer.scala:421)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.kafka.common.protocol.types.SchemaException: Error 
> reading field 'topics': Error reading field 'partitions': Error reading field 
> 'metadata': java.lang.NegativeArraySizeException
> at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:73)
> at 
> org.apache.kafka.common.requests.OffsetCommitRequest.parse(OffsetCommitRequest.java:260)
> at 
> org.apache.kafka.common.requests.AbstractRequest.getRequest(AbstractRequest.java:50)
> at 
> kafka.network.RequestChannel$Request.liftedTree2$1(RequestChannel.scala:88)
> ... 9 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3394) Broker fails to parse Null Metadata in OffsetCommit requests

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15194032#comment-15194032
 ] 

Ismael Juma commented on KAFKA-3394:


This is a regression caused by changing `KafkaApis` to use the Java protocol 
classes. Marking as a blocker since it breaks existing clients.

> Broker fails to parse Null Metadata in OffsetCommit requests
> 
>
> Key: KAFKA-3394
> URL: https://issues.apache.org/jira/browse/KAFKA-3394
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.0.0
>Reporter: Magnus Edenhill
>Priority: Blocker
> Fix For: 0.10.0.0
>
>
> librdkafka sends a Null Metadata string (size -1) in its OffsetCommitRequests 
> when there is no metadata, this unfortunately leads to an exception on the 
> broker that expects a non-null string.
> {noformat}
> [2016-03-11 11:11:57,623] ERROR Closing socket for 
> 10.191.0.33:9092-10.191.0.33:56503 because of error (kafka.network.Processor)
> kafka.network.InvalidRequestException: Error getting request for apiKey: 8 
> and apiVersion: 1
> at 
> kafka.network.RequestChannel$Request.liftedTree2$1(RequestChannel.scala:91)
> at kafka.network.RequestChannel$Request.(RequestChannel.scala:88)
> at kafka.network.Processor$$anonfun$run$11.apply(SocketServer.scala:426)
> at kafka.network.Processor$$anonfun$run$11.apply(SocketServer.scala:421)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at kafka.network.Processor.run(SocketServer.scala:421)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.kafka.common.protocol.types.SchemaException: Error 
> reading field 'topics': Error reading field 'partitions': Error reading field 
> 'metadata': java.lang.NegativeArraySizeException
> at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:73)
> at 
> org.apache.kafka.common.requests.OffsetCommitRequest.parse(OffsetCommitRequest.java:260)
> at 
> org.apache.kafka.common.requests.AbstractRequest.getRequest(AbstractRequest.java:50)
> at 
> kafka.network.RequestChannel$Request.liftedTree2$1(RequestChannel.scala:88)
> ... 9 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-3394) Broker fails to parse Null Metadata in OffsetCommit requests

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson reassigned KAFKA-3394:
--

Assignee: Jason Gustafson

> Broker fails to parse Null Metadata in OffsetCommit requests
> 
>
> Key: KAFKA-3394
> URL: https://issues.apache.org/jira/browse/KAFKA-3394
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.0.0
>Reporter: Magnus Edenhill
>Assignee: Jason Gustafson
>Priority: Blocker
> Fix For: 0.10.0.0
>
>
> librdkafka sends a Null Metadata string (size -1) in its OffsetCommitRequests 
> when there is no metadata, this unfortunately leads to an exception on the 
> broker that expects a non-null string.
> {noformat}
> [2016-03-11 11:11:57,623] ERROR Closing socket for 
> 10.191.0.33:9092-10.191.0.33:56503 because of error (kafka.network.Processor)
> kafka.network.InvalidRequestException: Error getting request for apiKey: 8 
> and apiVersion: 1
> at 
> kafka.network.RequestChannel$Request.liftedTree2$1(RequestChannel.scala:91)
> at kafka.network.RequestChannel$Request.(RequestChannel.scala:88)
> at kafka.network.Processor$$anonfun$run$11.apply(SocketServer.scala:426)
> at kafka.network.Processor$$anonfun$run$11.apply(SocketServer.scala:421)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at kafka.network.Processor.run(SocketServer.scala:421)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.kafka.common.protocol.types.SchemaException: Error 
> reading field 'topics': Error reading field 'partitions': Error reading field 
> 'metadata': java.lang.NegativeArraySizeException
> at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:73)
> at 
> org.apache.kafka.common.requests.OffsetCommitRequest.parse(OffsetCommitRequest.java:260)
> at 
> org.apache.kafka.common.requests.AbstractRequest.getRequest(AbstractRequest.java:50)
> at 
> kafka.network.RequestChannel$Request.liftedTree2$1(RequestChannel.scala:88)
> ... 9 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: MINOR: Review change to block.on.buffer.full c...

[granthenke]

Github user granthenke closed the pull request at:

https://github.com/apache/kafka/pull/1058


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-3338) Add print / writeAsTex / etc functions to the DSL

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-3338:
-
Priority: Major  (was: Blocker)

> Add print / writeAsTex / etc functions to the DSL
> -
>
> Key: KAFKA-3338
> URL: https://issues.apache.org/jira/browse/KAFKA-3338
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Guozhang Wang
> Fix For: 0.10.0.0
>
>
> We want to provide some REPL-like pattern for users for better debuggability; 
> this would be like a finer grained trace-level logging. Such example APIs can 
> be found in Flink (search for class DataSet):
> https://ci.apache.org/projects/flink/flink-docs-master/api/java/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3338) Add print / writeAsTex / etc functions to the DSL

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15194044#comment-15194044
 ] 

Guozhang Wang commented on KAFKA-3338:
--

[~gwenshap] Down grading to major as this may not end up in 0.10.0.0 release.

> Add print / writeAsTex / etc functions to the DSL
> -
>
> Key: KAFKA-3338
> URL: https://issues.apache.org/jira/browse/KAFKA-3338
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Guozhang Wang
> Fix For: 0.10.0.0
>
>
> We want to provide some REPL-like pattern for users for better debuggability; 
> this would be like a finer grained trace-level logging. Such example APIs can 
> be found in Flink (search for class DataSet):
> https://ci.apache.org/projects/flink/flink-docs-master/api/java/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3337) Extract selector as a separate groupBy operator for KTable aggregations

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-3337:
-
Priority: Major  (was: Blocker)

> Extract selector as a separate groupBy operator for KTable aggregations
> ---
>
> Key: KAFKA-3337
> URL: https://issues.apache.org/jira/browse/KAFKA-3337
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Guozhang Wang
> Fix For: 0.10.0.0
>
>
> Currently KTable aggregation takes a selector and an aggregator, which makes 
> the function a little bit "heavy". It is better to extract the selector in a 
> separate groupBy function such that
> {code}
> table.groupBy(selector).aggregate(aggregator);
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3337) Extract selector as a separate groupBy operator for KTable aggregations

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15194046#comment-15194046
 ] 

Guozhang Wang commented on KAFKA-3337:
--

[~gwenshap] Down grading to major as it may not be included in 0.10.0.0.

> Extract selector as a separate groupBy operator for KTable aggregations
> ---
>
> Key: KAFKA-3337
> URL: https://issues.apache.org/jira/browse/KAFKA-3337
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Guozhang Wang
> Fix For: 0.10.0.0
>
>
> Currently KTable aggregation takes a selector and an aggregator, which makes 
> the function a little bit "heavy". It is better to extract the selector in a 
> separate groupBy function such that
> {code}
> table.groupBy(selector).aggregate(aggregator);
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-3395) prefix job id to internal topic names

[Yasuhiro Matsuda (JIRA)]

Yasuhiro Matsuda created KAFKA-3395:
---

 Summary: prefix job id to internal topic names
 Key: KAFKA-3395
 URL: https://issues.apache.org/jira/browse/KAFKA-3395
 Project: Kafka
  Issue Type: Sub-task
  Components: kafka streams
Affects Versions: 0.9.0.1
Reporter: Yasuhiro Matsuda
 Fix For: 0.10.0.0


Names of internal repartition topics are not prefixed by a job id right now.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3395: prefix job id to internal topic na...

[ymatsuda]

GitHub user ymatsuda opened a pull request:

https://github.com/apache/kafka/pull/1062

KAFKA-3395: prefix job id to internal topic names

@guozhangwang 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ymatsuda/kafka k3395

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1062.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1062


commit 6bd08b21adaa8373349918cd0e249fa4222c8e43
Author: Yasuhiro Matsuda 
Date:   2016-03-14T20:15:05Z

KAFKA-3395: prefix job id to internal topic names




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3395) prefix job id to internal topic names

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15194051#comment-15194051
 ] 

ASF GitHub Bot commented on KAFKA-3395:
---

GitHub user ymatsuda opened a pull request:

https://github.com/apache/kafka/pull/1062

KAFKA-3395: prefix job id to internal topic names

@guozhangwang 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ymatsuda/kafka k3395

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1062.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1062


commit 6bd08b21adaa8373349918cd0e249fa4222c8e43
Author: Yasuhiro Matsuda 
Date:   2016-03-14T20:15:05Z

KAFKA-3395: prefix job id to internal topic names




> prefix job id to internal topic names
> -
>
> Key: KAFKA-3395
> URL: https://issues.apache.org/jira/browse/KAFKA-3395
> Project: Kafka
>  Issue Type: Sub-task
>  Components: kafka streams
>Affects Versions: 0.9.0.1
>Reporter: Yasuhiro Matsuda
> Fix For: 0.10.0.0
>
>
> Names of internal repartition topics are not prefixed by a job id right now.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2464) Client-side assignment and group generalization

[Jiangjie Qin (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15194053#comment-15194053
 ] 

Jiangjie Qin commented on KAFKA-2464:
-

Thanks, sounds good. I don't think there will be compatibility concerns here 
either.

> Client-side assignment and group generalization
> ---
>
> Key: KAFKA-2464
> URL: https://issues.apache.org/jira/browse/KAFKA-2464
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Blocker
> Fix For: 0.9.0.0
>
>
> Add support for client-side assignment and generalization of join group 
> protocol as documented here: 
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Client-side+Assignment+Proposal.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: KStreams Proposal

[Guozhang Wang]

Thanks Bill for the detailed description of your use case.

I think part of the issue is that today KStream does not allow non-keyed
aggregation, and hence users need to call a `map` function if they want to
do aggregations on other fields. We did this by design for KStream since if
users write the processor of your own for this case, they also need to
manually extract the aggregation key while updating the state store, which
is exactly what `map` would do, and hence making it explicit would help
guiding the programming. This `map` function does not generate any overhead
in terms of storage, all it does is extract the key for the use of the next
processor.

As for your proposed change, I think we need to be a bit careful about the
return types here if we want to add a return KStream<>, since its key value
type would be different from input KStream but rather KStream
for your case. But process() function itself does not have any method to
enforce typing on the returned value.

Guozhang



On Sun, Mar 13, 2016 at 3:38 PM, Bill Bejeck  wrote:

> Hi Guozhang,
>
> Possibly, but the use case I'm working with  is having a of collector
> object , for aggregate statistics for example, that would output results
> intermittently (via punctuate).
>
> The issue for me is that 'transform(..)'  returns a key-value pair for each
> message, possibly of a different type.
>
> I've achieved something similar in the KStream api using the form of
>  map(...).aggregateByKey().to(...)  but using that approach I need to
> map each message to an intermediate form and do the periodic aggregations
> of "stats" objects.
>
> What I'd really like is a way to attach a sink to a processor.
>
> With that in mind, instead of introducing a "proccessTo" method, another
> option could to change the return type of "process" from void to
> KStream.
>
> Then the use case becomes 'process(..).to(...)', similar to
> 'transform(..).to(..).
>
> I've made those changes locally and everything compiles fine and running my
> simple drive program achieves the desired results.
>
> I know I could be splitting hairs here,  but in my opinion, it would be
> nice to have.
>
> Thanks for your time!
>
> Bill
>
>
> On Sun, Mar 13, 2016 at 4:28 PM, Guozhang Wang  wrote:
>
> > Hello Bill,
> >
> > We added transform() together with process() to support any
> user-customized
> > stateful processor that can still concatenate to another KStream.
> >
> > So for your case, would `transform(...).to(topic)` provide the same
> > functionality as "processTo(topic, ...)"?
> >
> > Guozhang
> >
> >
> > On Sat, Mar 12, 2016 at 12:20 PM, Bill Bejeck  wrote:
> >
> > > Hi All,
> > >
> > > While working with KStream/KStreamImp I discovered that there does not
> > seem
> > > to be any way to connect the results of the KStream.process method
> with a
> > > sink node.
> > >
> > > I'd like to propose an addition to the API a "processTo" method.
> > >
> > > I've looked at and used the "transform", "reduceByKey" and
> > "aggregateByKey"
> > >  methods, but "processTo" would work like a more general purpose
> > collector
> > > terminating the KStream and allow for writing out results to an
> arbitrary
> > > topic (regardless of key type).
> > >
> > >
> > >  I've done a quick prototype and some  initial testing locally on my
> > fork.
> > > If you think this could be useful I can add unit tests and create a PR.
> > > I've included the proposed code changes and the test driver code below
> > >
> > >
> > > KStream.java additions
> > >
> > > void processTo(String topic,  ProcessorSupplier processorSupplier,
> > > String... stateStoreNames);
> > >
> > > void processTo(String topic, ProcessorSupplier processorSupplier,
> > >  Serializer keySerializer, Serializer valSerializer, String...
> > > stateStoreNames);
> > >
> > >
> > > KStreamImpl.java additions
> > >
> > >  @Override
> > > public void processTo(String topic, ProcessorSupplier
> > > processorSupplier,  String... stateStoreNames) {
> > > processTo(topic, processorSupplier, null, null,
> stateStoreNames);
> > > }
> > >
> > > @SuppressWarnings("unchecked")
> > > @Override
> > > public void processTo(String topic,ProcessorSupplier
> > > processorSupplier,  Serializer keySerializer, Serializer
> > > valSerializer, String... stateStoreNames) {
> > > String processorName = topology.newName(PROCESSOR_NAME);
> > > String sinkName = topology.newName(SINK_NAME);
> > > StreamPartitioner streamPartitioner = null;
> > >
> > > if (keySerializer != null && keySerializer instanceof
> > > WindowedSerializer) {
> > > WindowedSerializer windowedSerializer =
> > > (WindowedSerializer) keySerializer;
> > > streamPartitioner = (StreamPartitioner) new
> > > WindowedStreamPartitioner(windowedSerializer);
> > > }
> > >
> > > topology.addProcessor(processorName, processorSupplier,
> > this.name
> > > );
> > > topology.addSink(sinkName,topic, keySerialize

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ismael Juma]

Hi Ashish,

Comments inline.

On Mon, Mar 14, 2016 at 6:15 PM, Ashish Singh  wrote:

> I think ApiVersion aims to identify protocol version changes, more than
> release change,
>
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/api/ApiVersion.scala#L30
> .
>

That is something that has changed as part of KIP-31/32 (as I mentioned),
it wasn't like that in 0.9.0.x for example:

https://github.com/apache/kafka/blob/0.9.0/core/src/main/scala/kafka/api/ApiVersion.scala#L21


> I do agree that having an automated check to ensure this happens will be
> really useful.
>

Great.

> My understanding is that this doesn't help clients that support KIP-35
> > since they will know the broker version. And for older clients, they will
> > fail with a parsing exception, which is a bit better, but not much
> better.
> > So, is it really worth doing? In the KIP call we had about this months
> ago,
> > there was no consensus on this one from what I remember.
> >
> That is a good point! However, what about a client that wants to support
> broker versions that do not provide broker version in metadata and broker
> versions that provides version info in metadata. I think having this does
> not cost us anything, but enables such clients to be smart.
>

I don't see how it helps. If the client is communicating with a broker that
does not support KIP-35, that broker will simply close the connection. If
the broker supports KIP-35, then it will provide the broker version. I
don't envisage a scenario where a broker does not support KIP-35, but
implements the new behaviour of sending an empty response. Do you?

Ismael

Kafka KIP meeting Mar 15 at 11:00am PST

[Jun Rao]

Hi, Everyone,

We will have a Kafka KIP meeting tomorrow at 11:00am PST. If you plan to
attend but haven't received an invite, please let me know. The following is
the agenda.

Agenda:

KIP-35 - Retrieving protocol version
KIP-45 - Standardize all client sequence interaction on j.u.Collection
KIP-33 - Add a time based log index to Kafka (discuss if should be in
0.10.0)
KIP-43 - Kafka SASL enhancements (discuss if should be in 0.10.0)

Thanks,

Jun

[jira] [Commented] (KAFKA-3394) Broker fails to parse Null Metadata in OffsetCommit requests

[Grant Henke (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15194089#comment-15194089
 ] 

Grant Henke commented on KAFKA-3394:


This could happen for basically any string in the protocol. The java protocol 
implementation assumes that string was not nullable, while the old Scala 
messages allowed all strings to be nullable. This came up in KAFKA-3088 where 
we added NULLABLE_STRING and a default for the client id. 

If this is considered a regression, we should consider holistically how we want 
to handle the remaining strings that are not nullable. 

> Broker fails to parse Null Metadata in OffsetCommit requests
> 
>
> Key: KAFKA-3394
> URL: https://issues.apache.org/jira/browse/KAFKA-3394
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.0.0
>Reporter: Magnus Edenhill
>Assignee: Jason Gustafson
>Priority: Blocker
> Fix For: 0.10.0.0
>
>
> librdkafka sends a Null Metadata string (size -1) in its OffsetCommitRequests 
> when there is no metadata, this unfortunately leads to an exception on the 
> broker that expects a non-null string.
> {noformat}
> [2016-03-11 11:11:57,623] ERROR Closing socket for 
> 10.191.0.33:9092-10.191.0.33:56503 because of error (kafka.network.Processor)
> kafka.network.InvalidRequestException: Error getting request for apiKey: 8 
> and apiVersion: 1
> at 
> kafka.network.RequestChannel$Request.liftedTree2$1(RequestChannel.scala:91)
> at kafka.network.RequestChannel$Request.(RequestChannel.scala:88)
> at kafka.network.Processor$$anonfun$run$11.apply(SocketServer.scala:426)
> at kafka.network.Processor$$anonfun$run$11.apply(SocketServer.scala:421)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at kafka.network.Processor.run(SocketServer.scala:421)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.kafka.common.protocol.types.SchemaException: Error 
> reading field 'topics': Error reading field 'partitions': Error reading field 
> 'metadata': java.lang.NegativeArraySizeException
> at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:73)
> at 
> org.apache.kafka.common.requests.OffsetCommitRequest.parse(OffsetCommitRequest.java:260)
> at 
> org.apache.kafka.common.requests.AbstractRequest.getRequest(AbstractRequest.java:50)
> at 
> kafka.network.RequestChannel$Request.liftedTree2$1(RequestChannel.scala:88)
> ... 9 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Gwen Shapira]

>
>
> I don't see how it helps. If the client is communicating with a broker that
> does not support KIP-35, that broker will simply close the connection. If
> the broker supports KIP-35, then it will provide the broker version. I
> don't envisage a scenario where a broker does not support KIP-35, but
> implements the new behaviour of sending an empty response. Do you?
>
> Are you sure about that? Per KIP-35, the broker supplies the version in
response to Metadata request, not in response to anything else.
If the client sends producer request version 42 (accidentally or due to
premature upgrade) to KIP-35-compactible broker - we want to see an empty
packet and not a connection close.
Sending a broker version was deemed impractical IIRC.


> Ismael
>

[jira] [Commented] (KAFKA-3394) Broker fails to parse Null Metadata in OffsetCommit requests

[Jason Gustafson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15194116#comment-15194116
 ] 

Jason Gustafson commented on KAFKA-3394:


[~granthenke] I considered this when I first created NULLABLE_BYTES. 
Technically the Kafka protocol as documented has no such thing: all byte arrays 
and strings are nullable. I thought about changing BYTES to allow null values, 
but I was reluctant to suddenly have to deal with null values popping up where 
they hadn't been possible before. But maybe that's actually the safer way to 
ensure compatibility?

> Broker fails to parse Null Metadata in OffsetCommit requests
> 
>
> Key: KAFKA-3394
> URL: https://issues.apache.org/jira/browse/KAFKA-3394
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.0.0
>Reporter: Magnus Edenhill
>Assignee: Jason Gustafson
>Priority: Blocker
> Fix For: 0.10.0.0
>
>
> librdkafka sends a Null Metadata string (size -1) in its OffsetCommitRequests 
> when there is no metadata, this unfortunately leads to an exception on the 
> broker that expects a non-null string.
> {noformat}
> [2016-03-11 11:11:57,623] ERROR Closing socket for 
> 10.191.0.33:9092-10.191.0.33:56503 because of error (kafka.network.Processor)
> kafka.network.InvalidRequestException: Error getting request for apiKey: 8 
> and apiVersion: 1
> at 
> kafka.network.RequestChannel$Request.liftedTree2$1(RequestChannel.scala:91)
> at kafka.network.RequestChannel$Request.(RequestChannel.scala:88)
> at kafka.network.Processor$$anonfun$run$11.apply(SocketServer.scala:426)
> at kafka.network.Processor$$anonfun$run$11.apply(SocketServer.scala:421)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at kafka.network.Processor.run(SocketServer.scala:421)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.kafka.common.protocol.types.SchemaException: Error 
> reading field 'topics': Error reading field 'partitions': Error reading field 
> 'metadata': java.lang.NegativeArraySizeException
> at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:73)
> at 
> org.apache.kafka.common.requests.OffsetCommitRequest.parse(OffsetCommitRequest.java:260)
> at 
> org.apache.kafka.common.requests.AbstractRequest.getRequest(AbstractRequest.java:50)
> at 
> kafka.network.RequestChannel$Request.liftedTree2$1(RequestChannel.scala:88)
> ... 9 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3394) Broker fails to parse Null Metadata in OffsetCommit requests

[Grant Henke (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15194134#comment-15194134
 ] 

Grant Henke commented on KAFKA-3394:


[~hachikuji] I agree that allowing nulls everywhere is pretty risky. I wasn't 
making a comment on which approach we should take. I was just commenting that 
we should evaluate/understand the problem holistically given that its a 
widespread issue.

If a (third party) client runs into an issue with any string in the protocol 
are we going to make that string nullable? If the answer is yes, should we wait 
until they run into the issue and handle it on a case by cases basis?

> Broker fails to parse Null Metadata in OffsetCommit requests
> 
>
> Key: KAFKA-3394
> URL: https://issues.apache.org/jira/browse/KAFKA-3394
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.0.0
>Reporter: Magnus Edenhill
>Assignee: Jason Gustafson
>Priority: Blocker
> Fix For: 0.10.0.0
>
>
> librdkafka sends a Null Metadata string (size -1) in its OffsetCommitRequests 
> when there is no metadata, this unfortunately leads to an exception on the 
> broker that expects a non-null string.
> {noformat}
> [2016-03-11 11:11:57,623] ERROR Closing socket for 
> 10.191.0.33:9092-10.191.0.33:56503 because of error (kafka.network.Processor)
> kafka.network.InvalidRequestException: Error getting request for apiKey: 8 
> and apiVersion: 1
> at 
> kafka.network.RequestChannel$Request.liftedTree2$1(RequestChannel.scala:91)
> at kafka.network.RequestChannel$Request.(RequestChannel.scala:88)
> at kafka.network.Processor$$anonfun$run$11.apply(SocketServer.scala:426)
> at kafka.network.Processor$$anonfun$run$11.apply(SocketServer.scala:421)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at kafka.network.Processor.run(SocketServer.scala:421)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.kafka.common.protocol.types.SchemaException: Error 
> reading field 'topics': Error reading field 'partitions': Error reading field 
> 'metadata': java.lang.NegativeArraySizeException
> at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:73)
> at 
> org.apache.kafka.common.requests.OffsetCommitRequest.parse(OffsetCommitRequest.java:260)
> at 
> org.apache.kafka.common.requests.AbstractRequest.getRequest(AbstractRequest.java:50)
> at 
> kafka.network.RequestChannel$Request.liftedTree2$1(RequestChannel.scala:88)
> ... 9 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ismael Juma]

On Mon, Mar 14, 2016 at 8:45 PM, Gwen Shapira  wrote:
>
> > I don't see how it helps. If the client is communicating with a broker
> that
> > does not support KIP-35, that broker will simply close the connection. If
> > the broker supports KIP-35, then it will provide the broker version. I
> > don't envisage a scenario where a broker does not support KIP-35, but
> > implements the new behaviour of sending an empty response. Do you?
> >
> > Are you sure about that? Per KIP-35, the broker supplies the version in
> response to Metadata request, not in response to anything else.
> If the client sends producer request version 42 (accidentally or due to
> premature upgrade) to KIP-35-compactible broker - we want to see an empty
> packet and not a connection close.
> Sending a broker version was deemed impractical IIRC.
>

OK, so this is a different case than the one Ashish described ("client that
wants to support broker versions that do not provide broker version in
metadata and broker versions that provides version info in metadata"). So,
you are suggesting that if a client is communicating with a broker that
implements KIP-35 and it receives an empty response, it will assume that
the broker doesn't support the request version and it won't try to parse
the response? I think it would be good to explain this kind of thing in
detail in the KIP.

Ismael

[jira] [Commented] (KAFKA-3394) Broker fails to parse Null Metadata in OffsetCommit requests

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15194152#comment-15194152
 ] 

Ismael Juma commented on KAFKA-3394:


Given how the Scala request classes work, it seems to me that we have to make 
all Strings nullable in the Java request classes that are handling requests 
that were previously handled by Scala request classes. Is there any alternative 
that won't break clients? It is a bit too late to change this after we actually 
release a particular version.

> Broker fails to parse Null Metadata in OffsetCommit requests
> 
>
> Key: KAFKA-3394
> URL: https://issues.apache.org/jira/browse/KAFKA-3394
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.0.0
>Reporter: Magnus Edenhill
>Assignee: Jason Gustafson
>Priority: Blocker
> Fix For: 0.10.0.0
>
>
> librdkafka sends a Null Metadata string (size -1) in its OffsetCommitRequests 
> when there is no metadata, this unfortunately leads to an exception on the 
> broker that expects a non-null string.
> {noformat}
> [2016-03-11 11:11:57,623] ERROR Closing socket for 
> 10.191.0.33:9092-10.191.0.33:56503 because of error (kafka.network.Processor)
> kafka.network.InvalidRequestException: Error getting request for apiKey: 8 
> and apiVersion: 1
> at 
> kafka.network.RequestChannel$Request.liftedTree2$1(RequestChannel.scala:91)
> at kafka.network.RequestChannel$Request.(RequestChannel.scala:88)
> at kafka.network.Processor$$anonfun$run$11.apply(SocketServer.scala:426)
> at kafka.network.Processor$$anonfun$run$11.apply(SocketServer.scala:421)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at kafka.network.Processor.run(SocketServer.scala:421)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.kafka.common.protocol.types.SchemaException: Error 
> reading field 'topics': Error reading field 'partitions': Error reading field 
> 'metadata': java.lang.NegativeArraySizeException
> at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:73)
> at 
> org.apache.kafka.common.requests.OffsetCommitRequest.parse(OffsetCommitRequest.java:260)
> at 
> org.apache.kafka.common.requests.AbstractRequest.getRequest(AbstractRequest.java:50)
> at 
> kafka.network.RequestChannel$Request.liftedTree2$1(RequestChannel.scala:88)
> ... 9 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3394) Broker fails to parse Null Metadata in OffsetCommit requests

[Jason Gustafson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15194157#comment-15194157
 ] 

Jason Gustafson commented on KAFKA-3394:


Yeah, I guess what I was thinking is whether we should keep the current 
protocol definitions and change the implementation to make sure that we support 
what is documented (and deal with the consequence of having nulls all over the 
place). Or if we want to have non-nullable strings, maybe we should update the 
protocol documentation.

As for how we're going to detect other such cases, I don't have any great 
ideas. I asked [~edenhill] if there were other cases where he depends on null 
values and the only other case he thought of was message keys and values, which 
I'm sure we already handle.

> Broker fails to parse Null Metadata in OffsetCommit requests
> 
>
> Key: KAFKA-3394
> URL: https://issues.apache.org/jira/browse/KAFKA-3394
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.0.0
>Reporter: Magnus Edenhill
>Assignee: Jason Gustafson
>Priority: Blocker
> Fix For: 0.10.0.0
>
>
> librdkafka sends a Null Metadata string (size -1) in its OffsetCommitRequests 
> when there is no metadata, this unfortunately leads to an exception on the 
> broker that expects a non-null string.
> {noformat}
> [2016-03-11 11:11:57,623] ERROR Closing socket for 
> 10.191.0.33:9092-10.191.0.33:56503 because of error (kafka.network.Processor)
> kafka.network.InvalidRequestException: Error getting request for apiKey: 8 
> and apiVersion: 1
> at 
> kafka.network.RequestChannel$Request.liftedTree2$1(RequestChannel.scala:91)
> at kafka.network.RequestChannel$Request.(RequestChannel.scala:88)
> at kafka.network.Processor$$anonfun$run$11.apply(SocketServer.scala:426)
> at kafka.network.Processor$$anonfun$run$11.apply(SocketServer.scala:421)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at kafka.network.Processor.run(SocketServer.scala:421)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.kafka.common.protocol.types.SchemaException: Error 
> reading field 'topics': Error reading field 'partitions': Error reading field 
> 'metadata': java.lang.NegativeArraySizeException
> at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:73)
> at 
> org.apache.kafka.common.requests.OffsetCommitRequest.parse(OffsetCommitRequest.java:260)
> at 
> org.apache.kafka.common.requests.AbstractRequest.getRequest(AbstractRequest.java:50)
> at 
> kafka.network.RequestChannel$Request.liftedTree2$1(RequestChannel.scala:88)
> ... 9 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-3396) Unauthorized topics are returned to the user

[Grant Henke (JIRA)]

Grant Henke created KAFKA-3396:
--

 Summary: Unauthorized topics are returned to the user
 Key: KAFKA-3396
 URL: https://issues.apache.org/jira/browse/KAFKA-3396
 Project: Kafka
  Issue Type: Bug
Reporter: Grant Henke


Kafka's clients and protocol exposes unauthorized topics to the end user. This 
is often considered a security hole. To some, the topic name is considered 
sensitive information. Those that do not consider the name sensitive, still 
consider it more information that allows a user to try and circumvent security. 
 Instead, if a user does not have access to the topic, the servers should act 
as if the topic does not exist. 

To solve this some of the changes could include:
  - The broker should not return a TOPIC_AUTHORIZATION(29) error for 
requests (metadata, produce, fetch, etc) that include a topic that the user 
does not have DESCRIBE access to.
  - A user should not receive a TopicAuthorizationException when they do 
not have DESCRIBE access to a topic or the cluster.
 - The client should not maintain and expose a list of unauthorized topics 
in org.apache.kafka.common.Cluster. 

Other changes may be required that are not listed here. Further analysis is 
needed. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-3336) Unify ser/de pair classes into one serde class

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang reassigned KAFKA-3336:


Assignee: Guozhang Wang

> Unify ser/de pair classes into one serde class
> --
>
> Key: KAFKA-3336
> URL: https://issues.apache.org/jira/browse/KAFKA-3336
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
>Priority: Blocker
> Fix For: 0.10.0.0
>
>
> Right now users must provide two separate classes for serializers and 
> deserializers, respectively.  This means the current API functions have at 
> least 2 * numberOfTypes parameters.
> *Example (current, bad): "foo(..., longSerializer, longDeserializer)".*
> Because the serde aspect of the API is already one of the biggest UX issues, 
> we should unify the serde functionality into a single serde class, i.e. one 
> class that provides both serialization and deserialization functionality.  
> This will reduce the number of required serde parameters in the API by 50%.
> *Example (suggested, better): "foo(..., longSerializerDeserializer)"*. 
> * Note: This parameter name is horrible and only used to highlight the 
> difference to the "current" example above.
> We also want to 1) add a pairing function for each operator that does not 
> require serialization, and 2) provide default serdes for common data types.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3336) Unify ser/de pair classes into one serde class

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-3336:
-
Description: 
Right now users must provide two separate classes for serializers and 
deserializers, respectively.  This means the current API functions have at 
least 2 * numberOfTypes parameters.

*Example (current, bad): "foo(..., longSerializer, longDeserializer)".*

Because the serde aspect of the API is already one of the biggest UX issues, we 
should unify the serde functionality into a single serde class, i.e. one class 
that provides both serialization and deserialization functionality.  This will 
reduce the number of required serde parameters in the API by 50%.

*Example (suggested, better): "foo(..., longSerializerDeserializer)"*. 
* Note: This parameter name is horrible and only used to highlight the 
difference to the "current" example above.

We also want to add a pairing function for each operator that does not require 
serialization.

  was:
Right now users must provide two separate classes for serializers and 
deserializers, respectively.  This means the current API functions have at 
least 2 * numberOfTypes parameters.

*Example (current, bad): "foo(..., longSerializer, longDeserializer)".*

Because the serde aspect of the API is already one of the biggest UX issues, we 
should unify the serde functionality into a single serde class, i.e. one class 
that provides both serialization and deserialization functionality.  This will 
reduce the number of required serde parameters in the API by 50%.

*Example (suggested, better): "foo(..., longSerializerDeserializer)"*. 
* Note: This parameter name is horrible and only used to highlight the 
difference to the "current" example above.

We also want to 1) add a pairing function for each operator that does not 
require serialization, and 2) provide default serdes for common data types.


> Unify ser/de pair classes into one serde class
> --
>
> Key: KAFKA-3336
> URL: https://issues.apache.org/jira/browse/KAFKA-3336
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
>Priority: Blocker
> Fix For: 0.10.0.0
>
>
> Right now users must provide two separate classes for serializers and 
> deserializers, respectively.  This means the current API functions have at 
> least 2 * numberOfTypes parameters.
> *Example (current, bad): "foo(..., longSerializer, longDeserializer)".*
> Because the serde aspect of the API is already one of the biggest UX issues, 
> we should unify the serde functionality into a single serde class, i.e. one 
> class that provides both serialization and deserialization functionality.  
> This will reduce the number of required serde parameters in the API by 50%.
> *Example (suggested, better): "foo(..., longSerializerDeserializer)"*. 
> * Note: This parameter name is horrible and only used to highlight the 
> difference to the "current" example above.
> We also want to add a pairing function for each operator that does not 
> require serialization.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3336) Unify ser/de pair classes into one serde class

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-3336:
-
Description: 
Right now users must provide two separate classes for serializers and 
deserializers, respectively.  This means the current API functions have at 
least 2 * numberOfTypes parameters.

*Example (current, bad): "foo(..., longSerializer, longDeserializer)".*

Because the serde aspect of the API is already one of the biggest UX issues, we 
should unify the serde functionality into a single serde class, i.e. one class 
that provides both serialization and deserialization functionality.  This will 
reduce the number of required serde parameters in the API by 50%.

*Example (suggested, better): "foo(..., longSerializerDeserializer)"*. 
* Note: This parameter name is horrible and only used to highlight the 
difference to the "current" example above.

We also want to 1) add a pairing function for each operator that does not 
require serialization and 2) add a default serde in the configs to make these 
not required configs.

  was:
Right now users must provide two separate classes for serializers and 
deserializers, respectively.  This means the current API functions have at 
least 2 * numberOfTypes parameters.

*Example (current, bad): "foo(..., longSerializer, longDeserializer)".*

Because the serde aspect of the API is already one of the biggest UX issues, we 
should unify the serde functionality into a single serde class, i.e. one class 
that provides both serialization and deserialization functionality.  This will 
reduce the number of required serde parameters in the API by 50%.

*Example (suggested, better): "foo(..., longSerializerDeserializer)"*. 
* Note: This parameter name is horrible and only used to highlight the 
difference to the "current" example above.

We also want to add a pairing function for each operator that does not require 
serialization.


> Unify ser/de pair classes into one serde class
> --
>
> Key: KAFKA-3336
> URL: https://issues.apache.org/jira/browse/KAFKA-3336
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
>Priority: Blocker
> Fix For: 0.10.0.0
>
>
> Right now users must provide two separate classes for serializers and 
> deserializers, respectively.  This means the current API functions have at 
> least 2 * numberOfTypes parameters.
> *Example (current, bad): "foo(..., longSerializer, longDeserializer)".*
> Because the serde aspect of the API is already one of the biggest UX issues, 
> we should unify the serde functionality into a single serde class, i.e. one 
> class that provides both serialization and deserialization functionality.  
> This will reduce the number of required serde parameters in the API by 50%.
> *Example (suggested, better): "foo(..., longSerializerDeserializer)"*. 
> * Note: This parameter name is horrible and only used to highlight the 
> difference to the "current" example above.
> We also want to 1) add a pairing function for each operator that does not 
> require serialization and 2) add a default serde in the configs to make these 
> not required configs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3395: prefix job id to internal topic na...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1062


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-3395) prefix job id to internal topic names

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-3395:
-
Assignee: Yasuhiro Matsuda

> prefix job id to internal topic names
> -
>
> Key: KAFKA-3395
> URL: https://issues.apache.org/jira/browse/KAFKA-3395
> Project: Kafka
>  Issue Type: Sub-task
>  Components: kafka streams
>Affects Versions: 0.9.0.1
>Reporter: Yasuhiro Matsuda
>Assignee: Yasuhiro Matsuda
> Fix For: 0.10.0.0
>
>
> Names of internal repartition topics are not prefixed by a job id right now.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-3395) prefix job id to internal topic names

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-3395.
--
Resolution: Fixed

Issue resolved by pull request 1062
[https://github.com/apache/kafka/pull/1062]

> prefix job id to internal topic names
> -
>
> Key: KAFKA-3395
> URL: https://issues.apache.org/jira/browse/KAFKA-3395
> Project: Kafka
>  Issue Type: Sub-task
>  Components: kafka streams
>Affects Versions: 0.9.0.1
>Reporter: Yasuhiro Matsuda
> Fix For: 0.10.0.0
>
>
> Names of internal repartition topics are not prefixed by a job id right now.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3395) prefix job id to internal topic names

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15194224#comment-15194224
 ] 

ASF GitHub Bot commented on KAFKA-3395:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1062


> prefix job id to internal topic names
> -
>
> Key: KAFKA-3395
> URL: https://issues.apache.org/jira/browse/KAFKA-3395
> Project: Kafka
>  Issue Type: Sub-task
>  Components: kafka streams
>Affects Versions: 0.9.0.1
>Reporter: Yasuhiro Matsuda
> Fix For: 0.10.0.0
>
>
> Names of internal repartition topics are not prefixed by a job id right now.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: [MINOR] cleanup

[Ishiihara]

GitHub user Ishiihara opened a pull request:

https://github.com/apache/kafka/pull/1063

[MINOR] cleanup

@guozhangwang Very minor cleanup. 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Ishiihara/kafka minor-cleanup

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1063.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1063


commit 2131b2a9069de38906a1c25f1deec9b5b7e8867b
Author: Liquan Pei 
Date:   2016-03-14T21:48:55Z

minor cleanup




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Gwen Shapira]

Agree. Regardless of which one of us understood the KIP correctly, this
kind of thing should be crystal clear in the KIP.

On Mon, Mar 14, 2016 at 2:12 PM, Ismael Juma  wrote:

> On Mon, Mar 14, 2016 at 8:45 PM, Gwen Shapira  wrote:
> >
> > > I don't see how it helps. If the client is communicating with a broker
> > that
> > > does not support KIP-35, that broker will simply close the connection.
> If
> > > the broker supports KIP-35, then it will provide the broker version. I
> > > don't envisage a scenario where a broker does not support KIP-35, but
> > > implements the new behaviour of sending an empty response. Do you?
> > >
> > > Are you sure about that? Per KIP-35, the broker supplies the version in
> > response to Metadata request, not in response to anything else.
> > If the client sends producer request version 42 (accidentally or due to
> > premature upgrade) to KIP-35-compactible broker - we want to see an empty
> > packet and not a connection close.
> > Sending a broker version was deemed impractical IIRC.
> >
>
> OK, so this is a different case than the one Ashish described ("client that
> wants to support broker versions that do not provide broker version in
> metadata and broker versions that provides version info in metadata"). So,
> you are suggesting that if a client is communicating with a broker that
> implements KIP-35 and it receives an empty response, it will assume that
> the broker doesn't support the request version and it won't try to parse
> the response? I think it would be good to explain this kind of thing in
> detail in the KIP.
>
> Ismael
>

KIP-4 Wiki Update

[Grant Henke]

I have been updating the KIP-4 wiki page based on the last KIP call and
wanted to get some review and discussion around the server side
implementation for admin requests. Both the "ideal" functionality and the
"intermediated" functionality. The updates are still in progress, but this
section is the most critical and will likely have the most discussion. This
topic has had a few shifts in perspective and various discussions on
synchronous vs asynchronous server support. The wiki contains my current
perspective on the challenges and approach.

If you have any thoughts or feedback on the "Server-side Admin Request
handlers" section here
.
Lets discuss them in this thread.

For reference the last KIP discussion can be viewed here:
https://youtu.be/rFW0-zJqg5I?t=12m30s

Thank you,
Grant
-- 
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke

[jira] [Commented] (KAFKA-3394) Broker fails to parse Null Metadata in OffsetCommit requests

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15194269#comment-15194269
 ] 

ASF GitHub Bot commented on KAFKA-3394:
---

GitHub user hachikuji opened a pull request:

https://github.com/apache/kafka/pull/1064

KAFKA-3394: allow null offset metadata in commit API



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hachikuji/kafka KAFKA-3394

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1064.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1064






> Broker fails to parse Null Metadata in OffsetCommit requests
> 
>
> Key: KAFKA-3394
> URL: https://issues.apache.org/jira/browse/KAFKA-3394
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.0.0
>Reporter: Magnus Edenhill
>Assignee: Jason Gustafson
>Priority: Blocker
> Fix For: 0.10.0.0
>
>
> librdkafka sends a Null Metadata string (size -1) in its OffsetCommitRequests 
> when there is no metadata, this unfortunately leads to an exception on the 
> broker that expects a non-null string.
> {noformat}
> [2016-03-11 11:11:57,623] ERROR Closing socket for 
> 10.191.0.33:9092-10.191.0.33:56503 because of error (kafka.network.Processor)
> kafka.network.InvalidRequestException: Error getting request for apiKey: 8 
> and apiVersion: 1
> at 
> kafka.network.RequestChannel$Request.liftedTree2$1(RequestChannel.scala:91)
> at kafka.network.RequestChannel$Request.(RequestChannel.scala:88)
> at kafka.network.Processor$$anonfun$run$11.apply(SocketServer.scala:426)
> at kafka.network.Processor$$anonfun$run$11.apply(SocketServer.scala:421)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at kafka.network.Processor.run(SocketServer.scala:421)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.kafka.common.protocol.types.SchemaException: Error 
> reading field 'topics': Error reading field 'partitions': Error reading field 
> 'metadata': java.lang.NegativeArraySizeException
> at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:73)
> at 
> org.apache.kafka.common.requests.OffsetCommitRequest.parse(OffsetCommitRequest.java:260)
> at 
> org.apache.kafka.common.requests.AbstractRequest.getRequest(AbstractRequest.java:50)
> at 
> kafka.network.RequestChannel$Request.liftedTree2$1(RequestChannel.scala:88)
> ... 9 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: [MINOR]: Remove unused method, redundant in in...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1063


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: KAFKA-3394: allow null offset metadata in comm...

[hachikuji]

GitHub user hachikuji opened a pull request:

https://github.com/apache/kafka/pull/1064

KAFKA-3394: allow null offset metadata in commit API



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hachikuji/kafka KAFKA-3394

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1064.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1064






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: MINOR: kstream/ktable counting method with def...

[ymatsuda]

GitHub user ymatsuda opened a pull request:

https://github.com/apache/kafka/pull/1065

MINOR: kstream/ktable counting method with default long serdes

@guozhangwang @miguno 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ymatsuda/kafka count_serdes

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1065.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1065


commit f9350c6068010901e538052d8c039eaeccdf229a
Author: Yasuhiro Matsuda 
Date:   2016-03-14T22:24:12Z

MINOR: kstream/ktable counting method with default long serdes




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ashish Singh]

On Mon, Mar 14, 2016 at 2:12 PM, Ismael Juma  wrote:

> On Mon, Mar 14, 2016 at 8:45 PM, Gwen Shapira  wrote:
> >
> > > I don't see how it helps. If the client is communicating with a broker
> > that
> > > does not support KIP-35, that broker will simply close the connection.
> If
> > > the broker supports KIP-35, then it will provide the broker version. I
> > > don't envisage a scenario where a broker does not support KIP-35, but
> > > implements the new behaviour of sending an empty response. Do you?
> > >
> > > Are you sure about that? Per KIP-35, the broker supplies the version in
> > response to Metadata request, not in response to anything else.
> > If the client sends producer request version 42 (accidentally or due to
> > premature upgrade) to KIP-35-compactible broker - we want to see an empty
> > packet and not a connection close.
> > Sending a broker version was deemed impractical IIRC.
> >
>
> OK, so this is a different case than the one Ashish described ("client that
> wants to support broker versions that do not provide broker version in
> metadata and broker versions that provides version info in metadata"). So,
> you are suggesting that if a client is communicating with a broker that
> implements KIP-35 and it receives an empty response, it will assume that
> the broker doesn't support the request version and it won't try to parse
> the response? I think it would be good to explain this kind of thing in
> detail in the KIP.
>
Actually even in this case and the case I mentioned, closing connection
should be fine. Lets think about possible reasons that could lead to this
issue.

1. Client has incorrect mapping of supported protocols for a broker version.
2. Client misread broker version from metadata response.
3. Client constructed unsupported protocol version by mistake.

In all the above cases irrespective of what broker does, client will keep
sending wrong request version.

At this point, I think sending an empty packet instead of closing
connection is a nice to have and not mandatory requirement. Like in the
above case, a client can catch parsing error and be sure that there is
something wrong in the protocol version it is sending. However, a generic
connection close does not really provide any information on probable cause.

What do you guys suggest?

>
> Ismael
>



-- 

Regards,
Ashish

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ashish Singh]

On Mon, Mar 14, 2016 at 7:05 AM, Ismael Juma  wrote:

> Hi Ashish,
>
> A few comments below.
>
> On Fri, Mar 11, 2016 at 9:59 PM, Ashish Singh  wrote:
>
> > Sounds like we are mostly in agreement. Following are the key points.
> >
> >1. Every time a protocol version changes, for any request/response,
> >broker version, ApiVersion, will be bumped up.
> >
>
> Any thoughts on how we will enforce this?
>
>
> >2. Protocol documentation will be versioned with broker version. Every
> >time there is a broker version change, protocol documentation version
> > needs
> >to be updated and linked to main documentation page.
> >3. Deprecation of protocol version will be done via marking the
> version
> >as deprecated on the protocol documentation.
> >
>
> I think this is fine for the first version. We may consider doing more in
> the future (logging a warning perhaps).
>
>
> >4. On getting unknown protocol version, broker will send an empty
> >response, instead of simply closing client connection.
> >
>
> I am not sure about this one. It's an unusual pattern and feels like a
> hack.
>
>5. Metadata response will be enhanced to also contain broker version,
> >VersionInt and VersionString. VersionString will contain internal
> >version information.
> >
>
> Even though Magnus suggested that it's OK for clients to parse
> `VersionString`, I personally would rather avoid that. Do we really need 3
> separate versions or could we get away with 2? I think it would be good to
> elaborate on this a bit and explain how each of these versions would be
> used (both from the broker and clients perspective).
>
>
> >6. Metadata request with single null topic and size of -1 can be used
> to
> >fetch metadata response with only broker version information and no
> >topic/broker info.
>
>7. On receiving a metadata request with single null topic with size of
> >-1, broker will respond with only broker version.
> >
>
> As Magnus says, the broker information should be returned. This would also
> help us reduce unnecessary data transfer during NetworkClient's metadata
> updates (KAFKA-3358). At the moment, we get information for all topics in
> situations where we actually want no topics.
>
> Also, I think it's a bit odd to say a `single null topic with size -1`. Do
> we mean an array of topics with size -1 and no elements? That would imply
> introducing a NULLABLE_ARRAY type (we currently have NULLABLE_STRING and
> NULLABLE_BYTES).
>
Missed this point. I was thinking of an array with one null string,
ArrayOf[NULLABLE_STRING]. However, we can add NullableArrayOf and use
NullableArrayOf[STRING] as well. What do you think is better here?

>
> Ismael
>



-- 

Regards,
Ashish

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ismael Juma]

On Mon, Mar 14, 2016 at 10:35 PM, Ashish Singh  wrote:

> > Also, I think it's a bit odd to say a `single null topic with size -1`.
> Do
> > we mean an array of topics with size -1 and no elements? That would imply
> > introducing a NULLABLE_ARRAY type (we currently have NULLABLE_STRING and
> > NULLABLE_BYTES).
> >
> Missed this point. I was thinking of an array with one null string,
> ArrayOf[NULLABLE_STRING]. However, we can add NullableArrayOf and use
> NullableArrayOf[STRING] as well. What do you think is better here?
>

I prefer the latter.

Ismael

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ashish Singh]

On Mon, Mar 14, 2016 at 3:37 PM, Ismael Juma  wrote:

> On Mon, Mar 14, 2016 at 10:35 PM, Ashish Singh 
> wrote:
>
> > > Also, I think it's a bit odd to say a `single null topic with size -1`.
> > Do
> > > we mean an array of topics with size -1 and no elements? That would
> imply
> > > introducing a NULLABLE_ARRAY type (we currently have NULLABLE_STRING
> and
> > > NULLABLE_BYTES).
> > >
> > Missed this point. I was thinking of an array with one null string,
> > ArrayOf[NULLABLE_STRING]. However, we can add NullableArrayOf and use
> > NullableArrayOf[STRING] as well. What do you think is better here?
> >
>
> I prefer the latter.
>
> So be it. Updating KIP. Thanks!

> Ismael
>



-- 

Regards,
Ashish

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ashish Singh]

I have updated the KIP based on the discussions so far. Will initiate a
VOTE thread soon. There are some minor details that are still under
discussion, but nothing major to stop us from voting.

On Mon, Mar 14, 2016 at 3:39 PM, Ashish Singh  wrote:

>
>
> On Mon, Mar 14, 2016 at 3:37 PM, Ismael Juma  wrote:
>
>> On Mon, Mar 14, 2016 at 10:35 PM, Ashish Singh 
>> wrote:
>>
>> > > Also, I think it's a bit odd to say a `single null topic with size
>> -1`.
>> > Do
>> > > we mean an array of topics with size -1 and no elements? That would
>> imply
>> > > introducing a NULLABLE_ARRAY type (we currently have NULLABLE_STRING
>> and
>> > > NULLABLE_BYTES).
>> > >
>> > Missed this point. I was thinking of an array with one null string,
>> > ArrayOf[NULLABLE_STRING]. However, we can add NullableArrayOf and use
>> > NullableArrayOf[STRING] as well. What do you think is better here?
>> >
>>
>> I prefer the latter.
>>
>> So be it. Updating KIP. Thanks!
>
>> Ismael
>>
>
>
>
> --
>
> Regards,
> Ashish
>



-- 

Regards,
Ashish

[VOTE] KIP-35: Retrieving protocol version

[Ashish Singh]

Hey Guys,

I would like to start voting process for *KIP-35: Retrieving protocol
version*. The KIP is available here
.
Here

is a brief summary of the KIP.

The vote will run for 72 hours.
​
-- 

Regards,
Ashish

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Jason Gustafson]

Perhaps clients should always send the oldest version of the metadata
request which supports KIP-35 when initially connecting to the cluster.
Depending on the versions in the response, it can upgrade to a more recent
version. Then maybe we don't need the empty response hack?

One thing that's not clear to me is whether the ultimate goal of this KIP
is to have our clients support multiple broker versions. It would be a
little weird to have this feature if our own clients don't use it.

-Jason

On Mon, Mar 14, 2016 at 3:34 PM, Ashish Singh  wrote:

> On Mon, Mar 14, 2016 at 2:12 PM, Ismael Juma  wrote:
>
> > On Mon, Mar 14, 2016 at 8:45 PM, Gwen Shapira  wrote:
> > >
> > > > I don't see how it helps. If the client is communicating with a
> broker
> > > that
> > > > does not support KIP-35, that broker will simply close the
> connection.
> > If
> > > > the broker supports KIP-35, then it will provide the broker version.
> I
> > > > don't envisage a scenario where a broker does not support KIP-35, but
> > > > implements the new behaviour of sending an empty response. Do you?
> > > >
> > > > Are you sure about that? Per KIP-35, the broker supplies the version
> in
> > > response to Metadata request, not in response to anything else.
> > > If the client sends producer request version 42 (accidentally or due to
> > > premature upgrade) to KIP-35-compactible broker - we want to see an
> empty
> > > packet and not a connection close.
> > > Sending a broker version was deemed impractical IIRC.
> > >
> >
> > OK, so this is a different case than the one Ashish described ("client
> that
> > wants to support broker versions that do not provide broker version in
> > metadata and broker versions that provides version info in metadata").
> So,
> > you are suggesting that if a client is communicating with a broker that
> > implements KIP-35 and it receives an empty response, it will assume that
> > the broker doesn't support the request version and it won't try to parse
> > the response? I think it would be good to explain this kind of thing in
> > detail in the KIP.
> >
> Actually even in this case and the case I mentioned, closing connection
> should be fine. Lets think about possible reasons that could lead to this
> issue.
>
> 1. Client has incorrect mapping of supported protocols for a broker
> version.
> 2. Client misread broker version from metadata response.
> 3. Client constructed unsupported protocol version by mistake.
>
> In all the above cases irrespective of what broker does, client will keep
> sending wrong request version.
>
> At this point, I think sending an empty packet instead of closing
> connection is a nice to have and not mandatory requirement. Like in the
> above case, a client can catch parsing error and be sure that there is
> something wrong in the protocol version it is sending. However, a generic
> connection close does not really provide any information on probable cause.
>
> What do you guys suggest?
>
> >
> > Ismael
> >
>
>
>
> --
>
> Regards,
> Ashish
>

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ashish Singh]

Hi Jason,

On Mon, Mar 14, 2016 at 4:04 PM, Jason Gustafson  wrote:

> Perhaps clients should always send the oldest version of the metadata
> request which supports KIP-35 when initially connecting to the cluster.
> Depending on the versions in the response, it can upgrade to a more recent
> version. Then maybe we don't need the empty response hack?
>
Are you suggesting this as a solution for the problem where a KIP-35 aware
client sends a higher version of metadata req, say v2, to a KIP-35 aware
broker that only supports up to v1.

We had a bit of discussion on such scenarios, and it seemed to be a chicken
and egg problem that is hard to avoid. Your suggestion definitely makes
sense, however it falls under the purview of clients.

>
> One thing that's not clear to me is whether the ultimate goal of this KIP
> is to have our clients support multiple broker versions. It would be a
> little weird to have this feature if our own clients don't use it.
>
KIP-35 only aims on adding support for getting the version info from a
broker. This definitely can be utilized by our clients. However, that can
follow KIP-35 changes. Does this sound reasonable to you?

>
> -Jason
>
> On Mon, Mar 14, 2016 at 3:34 PM, Ashish Singh  wrote:
>
> > On Mon, Mar 14, 2016 at 2:12 PM, Ismael Juma  wrote:
> >
> > > On Mon, Mar 14, 2016 at 8:45 PM, Gwen Shapira 
> wrote:
> > > >
> > > > > I don't see how it helps. If the client is communicating with a
> > broker
> > > > that
> > > > > does not support KIP-35, that broker will simply close the
> > connection.
> > > If
> > > > > the broker supports KIP-35, then it will provide the broker
> version.
> > I
> > > > > don't envisage a scenario where a broker does not support KIP-35,
> but
> > > > > implements the new behaviour of sending an empty response. Do you?
> > > > >
> > > > > Are you sure about that? Per KIP-35, the broker supplies the
> version
> > in
> > > > response to Metadata request, not in response to anything else.
> > > > If the client sends producer request version 42 (accidentally or due
> to
> > > > premature upgrade) to KIP-35-compactible broker - we want to see an
> > empty
> > > > packet and not a connection close.
> > > > Sending a broker version was deemed impractical IIRC.
> > > >
> > >
> > > OK, so this is a different case than the one Ashish described ("client
> > that
> > > wants to support broker versions that do not provide broker version in
> > > metadata and broker versions that provides version info in metadata").
> > So,
> > > you are suggesting that if a client is communicating with a broker that
> > > implements KIP-35 and it receives an empty response, it will assume
> that
> > > the broker doesn't support the request version and it won't try to
> parse
> > > the response? I think it would be good to explain this kind of thing in
> > > detail in the KIP.
> > >
> > Actually even in this case and the case I mentioned, closing connection
> > should be fine. Lets think about possible reasons that could lead to this
> > issue.
> >
> > 1. Client has incorrect mapping of supported protocols for a broker
> > version.
> > 2. Client misread broker version from metadata response.
> > 3. Client constructed unsupported protocol version by mistake.
> >
> > In all the above cases irrespective of what broker does, client will keep
> > sending wrong request version.
> >
> > At this point, I think sending an empty packet instead of closing
> > connection is a nice to have and not mandatory requirement. Like in the
> > above case, a client can catch parsing error and be sure that there is
> > something wrong in the protocol version it is sending. However, a generic
> > connection close does not really provide any information on probable
> cause.
> >
> > What do you guys suggest?
> >
> > >
> > > Ismael
> > >
> >
> >
> >
> > --
> >
> > Regards,
> > Ashish
> >
>



-- 

Regards,
Ashish

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Jason Gustafson]

>
> Are you suggesting this as a solution for the problem where a KIP-35 aware
> client sends a higher version of metadata req, say v2, to a KIP-35 aware
> broker that only supports up to v1.


Yes, that's right. In that case, the client first sends v1, finds out that
the broker supports v2, and then sends v2 (if it has any reason to do so).

We had a bit of discussion on such scenarios, and it seemed to be a chicken
> and egg problem that is hard to avoid. Your suggestion definitely makes
> sense, however it falls under the purview of clients.


That basically means clients should figure it out for themselves? Might be
nice to have a better answer.

KIP-35 only aims on adding support for getting the version info from a
> broker. This definitely can be utilized by our clients. However, that can
> follow KIP-35 changes. Does this sound reasonable to you?


It may be OK, but I'm a little concerned about offering a feature that we
don't support ourselves. Sometimes it's not until implementation that we
find out whether it really works as expected. And if we're eventually
planning to support it, I feel we should think through some of the cases a
bit more. For example, the upgrade and downgrade cases that Becket
mentioned earlier. It doesn't feel too great to support this feature unless
we can offer guidance on how to use it.

-Jason



On Mon, Mar 14, 2016 at 4:20 PM, Ashish Singh  wrote:

> Hi Jason,
>
> On Mon, Mar 14, 2016 at 4:04 PM, Jason Gustafson 
> wrote:
>
> > Perhaps clients should always send the oldest version of the metadata
> > request which supports KIP-35 when initially connecting to the cluster.
> > Depending on the versions in the response, it can upgrade to a more
> recent
> > version. Then maybe we don't need the empty response hack?
> >
> Are you suggesting this as a solution for the problem where a KIP-35 aware
> client sends a higher version of metadata req, say v2, to a KIP-35 aware
> broker that only supports up to v1.
>
> We had a bit of discussion on such scenarios, and it seemed to be a chicken
> and egg problem that is hard to avoid. Your suggestion definitely makes
> sense, however it falls under the purview of clients.
>
> >
> > One thing that's not clear to me is whether the ultimate goal of this KIP
> > is to have our clients support multiple broker versions. It would be a
> > little weird to have this feature if our own clients don't use it.
> >
> KIP-35 only aims on adding support for getting the version info from a
> broker. This definitely can be utilized by our clients. However, that can
> follow KIP-35 changes. Does this sound reasonable to you?
>
> >
> > -Jason
> >
> > On Mon, Mar 14, 2016 at 3:34 PM, Ashish Singh 
> wrote:
> >
> > > On Mon, Mar 14, 2016 at 2:12 PM, Ismael Juma 
> wrote:
> > >
> > > > On Mon, Mar 14, 2016 at 8:45 PM, Gwen Shapira 
> > wrote:
> > > > >
> > > > > > I don't see how it helps. If the client is communicating with a
> > > broker
> > > > > that
> > > > > > does not support KIP-35, that broker will simply close the
> > > connection.
> > > > If
> > > > > > the broker supports KIP-35, then it will provide the broker
> > version.
> > > I
> > > > > > don't envisage a scenario where a broker does not support KIP-35,
> > but
> > > > > > implements the new behaviour of sending an empty response. Do
> you?
> > > > > >
> > > > > > Are you sure about that? Per KIP-35, the broker supplies the
> > version
> > > in
> > > > > response to Metadata request, not in response to anything else.
> > > > > If the client sends producer request version 42 (accidentally or
> due
> > to
> > > > > premature upgrade) to KIP-35-compactible broker - we want to see an
> > > empty
> > > > > packet and not a connection close.
> > > > > Sending a broker version was deemed impractical IIRC.
> > > > >
> > > >
> > > > OK, so this is a different case than the one Ashish described
> ("client
> > > that
> > > > wants to support broker versions that do not provide broker version
> in
> > > > metadata and broker versions that provides version info in
> metadata").
> > > So,
> > > > you are suggesting that if a client is communicating with a broker
> that
> > > > implements KIP-35 and it receives an empty response, it will assume
> > that
> > > > the broker doesn't support the request version and it won't try to
> > parse
> > > > the response? I think it would be good to explain this kind of thing
> in
> > > > detail in the KIP.
> > > >
> > > Actually even in this case and the case I mentioned, closing connection
> > > should be fine. Lets think about possible reasons that could lead to
> this
> > > issue.
> > >
> > > 1. Client has incorrect mapping of supported protocols for a broker
> > > version.
> > > 2. Client misread broker version from metadata response.
> > > 3. Client constructed unsupported protocol version by mistake.
> > >
> > > In all the above cases irrespective of what broker does, client will
> keep
> > > sending wrong request version.
> > >
> > > At this point, I think se

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Ashish Singh]

On Mon, Mar 14, 2016 at 4:51 PM, Jason Gustafson  wrote:

> >
> > Are you suggesting this as a solution for the problem where a KIP-35
> aware
> > client sends a higher version of metadata req, say v2, to a KIP-35 aware
> > broker that only supports up to v1.
>
>
> Yes, that's right. In that case, the client first sends v1, finds out that
> the broker supports v2, and then sends v2 (if it has any reason to do so).
>
> We had a bit of discussion on such scenarios, and it seemed to be a chicken
> > and egg problem that is hard to avoid. Your suggestion definitely makes
> > sense, however it falls under the purview of clients.
>
>
> That basically means clients should figure it out for themselves? Might be
> nice to have a better answer.
>
You mean to provide guidance for clients? I think that would makes sense.
However, I do not see how server can alleviate this issue.

>
> KIP-35 only aims on adding support for getting the version info from a
> > broker. This definitely can be utilized by our clients. However, that can
> > follow KIP-35 changes. Does this sound reasonable to you?
>
>
> It may be OK, but I'm a little concerned about offering a feature that we
> don't support ourselves. Sometimes it's not until implementation that we
> find out whether it really works as expected. And if we're eventually
> planning to support it, I feel we should think through some of the cases a
> bit more. For example, the upgrade and downgrade cases that Becket
> mentioned earlier. It doesn't feel too great to support this feature unless
> we can offer guidance on how to use it.
>
> Totally agreed with implementation is required to make sure the feature is
working as expected. Magnus and I had some offline chat and he is willing
to test this out with librdkafka.

Making our client utilize KIP-35 changes will be super awesome. My only
concern is that if we decide to make those changes as part of this KIP, we
will have to move KIP-35 out of 0.10 release scope.

If testing is the only concern, would librdkafka validation be good enough?

> -Jason
>
>
>
> On Mon, Mar 14, 2016 at 4:20 PM, Ashish Singh  wrote:
>
> > Hi Jason,
> >
> > On Mon, Mar 14, 2016 at 4:04 PM, Jason Gustafson 
> > wrote:
> >
> > > Perhaps clients should always send the oldest version of the metadata
> > > request which supports KIP-35 when initially connecting to the cluster.
> > > Depending on the versions in the response, it can upgrade to a more
> > recent
> > > version. Then maybe we don't need the empty response hack?
> > >
> > Are you suggesting this as a solution for the problem where a KIP-35
> aware
> > client sends a higher version of metadata req, say v2, to a KIP-35 aware
> > broker that only supports up to v1.
> >
> > We had a bit of discussion on such scenarios, and it seemed to be a
> chicken
> > and egg problem that is hard to avoid. Your suggestion definitely makes
> > sense, however it falls under the purview of clients.
> >
> > >
> > > One thing that's not clear to me is whether the ultimate goal of this
> KIP
> > > is to have our clients support multiple broker versions. It would be a
> > > little weird to have this feature if our own clients don't use it.
> > >
> > KIP-35 only aims on adding support for getting the version info from a
> > broker. This definitely can be utilized by our clients. However, that can
> > follow KIP-35 changes. Does this sound reasonable to you?
> >
> > >
> > > -Jason
> > >
> > > On Mon, Mar 14, 2016 at 3:34 PM, Ashish Singh 
> > wrote:
> > >
> > > > On Mon, Mar 14, 2016 at 2:12 PM, Ismael Juma 
> > wrote:
> > > >
> > > > > On Mon, Mar 14, 2016 at 8:45 PM, Gwen Shapira 
> > > wrote:
> > > > > >
> > > > > > > I don't see how it helps. If the client is communicating with a
> > > > broker
> > > > > > that
> > > > > > > does not support KIP-35, that broker will simply close the
> > > > connection.
> > > > > If
> > > > > > > the broker supports KIP-35, then it will provide the broker
> > > version.
> > > > I
> > > > > > > don't envisage a scenario where a broker does not support
> KIP-35,
> > > but
> > > > > > > implements the new behaviour of sending an empty response. Do
> > you?
> > > > > > >
> > > > > > > Are you sure about that? Per KIP-35, the broker supplies the
> > > version
> > > > in
> > > > > > response to Metadata request, not in response to anything else.
> > > > > > If the client sends producer request version 42 (accidentally or
> > due
> > > to
> > > > > > premature upgrade) to KIP-35-compactible broker - we want to see
> an
> > > > empty
> > > > > > packet and not a connection close.
> > > > > > Sending a broker version was deemed impractical IIRC.
> > > > > >
> > > > >
> > > > > OK, so this is a different case than the one Ashish described
> > ("client
> > > > that
> > > > > wants to support broker versions that do not provide broker version
> > in
> > > > > metadata and broker versions that provides version info in
> > metadata").
> > > > So,
> > > > > you are suggesting that if a client is comm

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Jay Kreps]

Couple of missing things:

This KIP doesn't have a proposal on versioning it just gives different
options, it'd be good to get a concrete proposal in the KIP. Here is my
understanding of what we are proposing (can someone sanity check and if
correct, update the kip):

   1. We will augment the existing api_version field in the header with a
   protocol_version that will begin at some initial value and increment by 1
   every time we make a changes to any of the api_versions (question:
   including internal apis?).
   2. The protocol_version will be added to the metadata request
   3. We will also add a string that this proposal is calling VersionString
   which will describe the build of kafka in some way. The clients should not
   under any circumstances do anything with this string other than print it
   out to the user.

One thing I'm not sure about: I think currently metadata sits in the client
for 10 mins by default. Say a client bootstraps and then a server is
downgraded to an earlier version, won't the client's metadata version
indicate that that client handles a version it doesn't actually handle any
more? We need to document how clients will handle this.

Here are some comments on other details:

   1. As a minor thing I think we should avoid naming the fields VersionId
   and VersionString which sort of implies they are both used for versioning.
   I think we should call them something like ProtocolVersion and
   BuildDescription, with BuildDescription being totally unspecified other
   than that it is some kind of human readable string describing a particular
   Kafka build. We really don't want a client attempting to use this string in
   any way as that would always be the wrong thing to do in the versioning
   scheme we are proposing, you should always use the protocol version.
   2. Does making the topics field in the metadata request nullable
   actually make sense? We have a history of wanting to add magical values
   rather than fields. Currently topics=[a] means give me information about
   topic a, topics=[] means give me information about all topics, and we are
   proposing topics=null would mean don't give me topics. I don't have a
   strong opinion.
   3. I prefer Jason's proposal on using a conservative metadata version
   versus the empty response hack. However I think that may actually
   exacerbate the downgrade scenario I described.
   4. I agree with Jason that we should really look at the details of the
   implementation so we know it works--implementing server support without
   actually trying it is kind of risky.

As a meta comment: I'd really like to encourage us to think of the protocol
as a document that includes the following things:

   - The binary format, error codes, etc
   - The request/response interaction
   - The semantics of each request in different cases
   - Instructions on how to use this to implement a client

This document is versioned with the protocol number and is the source of
truth for the protocol.

Part of any protocol change needs to be an update to the instructions on
how to use that part of the protocol. We should be opinionated. If there
are two options there should be a reason, and then we need to document both
and say exactly when to use each.

I think we also need to get a "how to" document on protocol changes just so
people know what they need to do to add a new protocol feature.

-Jay

On Mon, Mar 14, 2016 at 4:51 PM, Jason Gustafson  wrote:

> >
> > Are you suggesting this as a solution for the problem where a KIP-35
> aware
> > client sends a higher version of metadata req, say v2, to a KIP-35 aware
> > broker that only supports up to v1.
>
>
> Yes, that's right. In that case, the client first sends v1, finds out that
> the broker supports v2, and then sends v2 (if it has any reason to do so).
>
> We had a bit of discussion on such scenarios, and it seemed to be a chicken
> > and egg problem that is hard to avoid. Your suggestion definitely makes
> > sense, however it falls under the purview of clients.
>
>
> That basically means clients should figure it out for themselves? Might be
> nice to have a better answer.
>
> KIP-35 only aims on adding support for getting the version info from a
> > broker. This definitely can be utilized by our clients. However, that can
> > follow KIP-35 changes. Does this sound reasonable to you?
>
>
> It may be OK, but I'm a little concerned about offering a feature that we
> don't support ourselves. Sometimes it's not until implementation that we
> find out whether it really works as expected. And if we're eventually
> planning to support it, I feel we should think through some of the cases a
> bit more. For example, the upgrade and downgrade cases that Becket
> mentioned earlier. It doesn't feel too great to support this feature unless
> we can offer guidance on how to use it.
>
> -Jason
>
>
>
> On Mon, Mar 14, 2016 at 4:20 PM, Ashish Singh  wrote:
>
> > Hi Jason,
> >
> > On Mon, Mar 14, 2016 at 4:04 PM, J

Jenkins build is back to normal : kafka-trunk-jdk7 #1114

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Gwen Shapira]

On Mon, Mar 14, 2016 at 5:31 PM, Jay Kreps  wrote:

> Couple of missing things:
>
> This KIP doesn't have a proposal on versioning it just gives different
> options, it'd be good to get a concrete proposal in the KIP. Here is my
> understanding of what we are proposing (can someone sanity check and if
> correct, update the kip):
>
>1. We will augment the existing api_version field in the header with a
>protocol_version that will begin at some initial value and increment by
> 1
>every time we make a changes to any of the api_versions (question:
>including internal apis?).
>

Jay, this part was not in the KIP and was never discussed.
Are you proposing adding this? Or is it just an assumption you made?



>2. The protocol_version will be added to the metadata request
>3. We will also add a string that this proposal is calling VersionString
>which will describe the build of kafka in some way. The clients should
> not
>under any circumstances do anything with this string other than print it
>out to the user.
>
> One thing I'm not sure about: I think currently metadata sits in the client
> for 10 mins by default. Say a client bootstraps and then a server is
> downgraded to an earlier version, won't the client's metadata version
> indicate that that client handles a version it doesn't actually handle any
> more? We need to document how clients will handle this.
>
> Here are some comments on other details:
>
>1. As a minor thing I think we should avoid naming the fields VersionId
>and VersionString which sort of implies they are both used for
> versioning.
>I think we should call them something like ProtocolVersion and
>BuildDescription, with BuildDescription being totally unspecified other
>than that it is some kind of human readable string describing a
> particular
>Kafka build. We really don't want a client attempting to use this
> string in
>any way as that would always be the wrong thing to do in the versioning
>scheme we are proposing, you should always use the protocol version.
>2. Does making the topics field in the metadata request nullable
>actually make sense? We have a history of wanting to add magical values
>rather than fields. Currently topics=[a] means give me information about
>topic a, topics=[] means give me information about all topics, and we
> are
>proposing topics=null would mean don't give me topics. I don't have a
>strong opinion.
>3. I prefer Jason's proposal on using a conservative metadata version
>versus the empty response hack. However I think that may actually
>exacerbate the downgrade scenario I described.
>4. I agree with Jason that we should really look at the details of the
>implementation so we know it works--implementing server support without
>actually trying it is kind of risky.
>
> As a meta comment: I'd really like to encourage us to think of the protocol
> as a document that includes the following things:
>
>- The binary format, error codes, etc
>- The request/response interaction
>- The semantics of each request in different cases
>- Instructions on how to use this to implement a client
>
> This document is versioned with the protocol number and is the source of
> truth for the protocol.
>
> Part of any protocol change needs to be an update to the instructions on
> how to use that part of the protocol. We should be opinionated. If there
> are two options there should be a reason, and then we need to document both
> and say exactly when to use each.
>
> I think we also need to get a "how to" document on protocol changes just so
> people know what they need to do to add a new protocol feature.
>
> -Jay
>
> On Mon, Mar 14, 2016 at 4:51 PM, Jason Gustafson 
> wrote:
>
> > >
> > > Are you suggesting this as a solution for the problem where a KIP-35
> > aware
> > > client sends a higher version of metadata req, say v2, to a KIP-35
> aware
> > > broker that only supports up to v1.
> >
> >
> > Yes, that's right. In that case, the client first sends v1, finds out
> that
> > the broker supports v2, and then sends v2 (if it has any reason to do
> so).
> >
> > We had a bit of discussion on such scenarios, and it seemed to be a
> chicken
> > > and egg problem that is hard to avoid. Your suggestion definitely makes
> > > sense, however it falls under the purview of clients.
> >
> >
> > That basically means clients should figure it out for themselves? Might
> be
> > nice to have a better answer.
> >
> > KIP-35 only aims on adding support for getting the version info from a
> > > broker. This definitely can be utilized by our clients. However, that
> can
> > > follow KIP-35 changes. Does this sound reasonable to you?
> >
> >
> > It may be OK, but I'm a little concerned about offering a feature that we
> > don't support ourselves. Sometimes it's not until implementation that we
> > find out whether it really works as expected. And if we're eventually
> > plan

Build failed in Jenkins: kafka-trunk-jdk8 #442

[Apache Jenkins Server]

See 

Changes:

[wangguoz] KAFKA-3395: prefix job id to internal topic names

[wangguoz] MINOR: Remove unused method, redundant in interface definition and 
add

--
[...truncated 3615 lines...]

org.apache.kafka.clients.NetworkClientTest > testClose PASSED

org.apache.kafka.clients.NetworkClientTest > testLeastLoadedNode PASSED

org.apache.kafka.clients.NetworkClientTest > testConnectionDelayConnected PASSED

org.apache.kafka.clients.NetworkClientTest > testRequestTimeout PASSED

org.apache.kafka.clients.NetworkClientTest > 
testSimpleRequestResponseWithStaticNodes PASSED

org.apache.kafka.clients.NetworkClientTest > testConnectionDelay PASSED

org.apache.kafka.clients.NetworkClientTest > testSendToUnreadyNode PASSED

org.apache.kafka.clients.NetworkClientTest > testConnectionDelayDisconnected 
PASSED

org.apache.kafka.clients.MetadataTest > testListenerCanUnregister PASSED

org.apache.kafka.clients.MetadataTest > testFailedUpdate PASSED

org.apache.kafka.clients.MetadataTest > testMetadataUpdateWaitTime PASSED

org.apache.kafka.clients.MetadataTest > testUpdateWithNeedMetadataForAllTopics 
PASSED

org.apache.kafka.clients.MetadataTest > testMetadata PASSED

org.apache.kafka.clients.MetadataTest > testListenerGetsNotifiedOfUpdate PASSED

org.apache.kafka.common.SerializeCompatibilityTopicPartitionTest > 
testSerializationRoundtrip PASSED

org.apache.kafka.common.SerializeCompatibilityTopicPartitionTest > 
testTopiPartitionSerializationCompatibility PASSED

org.apache.kafka.common.serialization.SerializationTest > testStringSerializer 
PASSED

org.apache.kafka.common.serialization.SerializationTest > testIntegerSerializer 
PASSED

org.apache.kafka.common.serialization.SerializationTest > 
testByteBufferSerializer PASSED

org.apache.kafka.common.config.AbstractConfigTest > testOriginalsWithPrefix 
PASSED

org.apache.kafka.common.config.AbstractConfigTest > testConfiguredInstances 
PASSED

org.apache.kafka.common.config.ConfigDefTest > testBasicTypes PASSED

org.apache.kafka.common.config.ConfigDefTest > testNullDefault PASSED

org.apache.kafka.common.config.ConfigDefTest > testInvalidDefaultRange PASSED

org.apache.kafka.common.config.ConfigDefTest > testValidators PASSED

org.apache.kafka.common.config.ConfigDefTest > testInvalidDefaultString PASSED

org.apache.kafka.common.config.ConfigDefTest > testSslPasswords PASSED

org.apache.kafka.common.config.ConfigDefTest > testInvalidDefault PASSED

org.apache.kafka.common.config.ConfigDefTest > testMissingRequired PASSED

org.apache.kafka.common.config.ConfigDefTest > testNullDefaultWithValidator 
PASSED

org.apache.kafka.common.config.ConfigDefTest > testDefinedTwice PASSED

org.apache.kafka.common.config.ConfigDefTest > testBadInputs PASSED

org.apache.kafka.common.protocol.ErrorsTest > testForExceptionDefault PASSED

org.apache.kafka.common.protocol.ErrorsTest > testUniqueExceptions PASSED

org.apache.kafka.common.protocol.ErrorsTest > testForExceptionInheritance PASSED

org.apache.kafka.common.protocol.ErrorsTest > testNoneException PASSED

org.apache.kafka.common.protocol.ErrorsTest > testUniqueErrorCodes PASSED

org.apache.kafka.common.protocol.ErrorsTest > testExceptionsAreNotGeneric PASSED

org.apache.kafka.common.protocol.ApiKeysTest > testForIdWithInvalidIdLow PASSED

org.apache.kafka.common.protocol.ApiKeysTest > testForIdWithInvalidIdHigh PASSED

org.apache.kafka.common.protocol.ProtoUtilsTest > schemaVersionOutOfRange PASSED

org.apache.kafka.common.protocol.types.ProtocolSerializationTest > testArray 
PASSED

org.apache.kafka.common.protocol.types.ProtocolSerializationTest > testNulls 
PASSED

org.apache.kafka.common.protocol.types.ProtocolSerializationTest > 
testNullableDefault PASSED

org.apache.kafka.common.protocol.types.ProtocolSerializationTest > testDefault 
PASSED

org.apache.kafka.common.protocol.types.ProtocolSerializationTest > testSimple 
PASSED

org.apache.kafka.common.requests.RequestResponseTest > testSerialization PASSED

org.apache.kafka.common.requests.RequestResponseTest > fetchResponseVersionTest 
PASSED

org.apache.kafka.common.requests.RequestResponseTest > 
produceResponseVersionTest PASSED

org.apache.kafka.common.requests.RequestResponseTest > 
testControlledShutdownResponse PASSED

org.apache.kafka.common.requests.RequestResponseTest > 
testRequestHeaderWithNullClientId PASSED

org.apache.kafka.common.security.auth.KafkaPrincipalTest > 
testPrincipalNameCanContainSeparator PASSED

org.apache.kafka.common.security.auth.KafkaPrincipalTest > 
testEqualsAndHashCode PASSED

org.apache.kafka.common.security.kerberos.KerberosNameTest > testParse PASSED

org.apache.kafka.common.security.ssl.SslFactoryTest > testClientMode PASSED

org.apache.kafka.common.security.ssl.SslFactoryTest > 
testSslFactoryConfiguration PASSED

org.apache.kafka.common.metrics.MetricsTest > testSimpleStats PASSED

org.apache.kafka.common.metrics.MetricsTest 

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Jay Kreps]

Yeah I think that is the point--we have a proposal for a new protocol
versioning scheme and a vote on it but it doesn't actually describe
how versioning will work yet! I gave my vague impression based on this
thread, but I want to make sure that is correct and get it written
down before we adopt it.

-Jay

On Mon, Mar 14, 2016 at 5:58 PM, Gwen Shapira  wrote:
> On Mon, Mar 14, 2016 at 5:31 PM, Jay Kreps  wrote:
>
>> Couple of missing things:
>>
>> This KIP doesn't have a proposal on versioning it just gives different
>> options, it'd be good to get a concrete proposal in the KIP. Here is my
>> understanding of what we are proposing (can someone sanity check and if
>> correct, update the kip):
>>
>>1. We will augment the existing api_version field in the header with a
>>protocol_version that will begin at some initial value and increment by
>> 1
>>every time we make a changes to any of the api_versions (question:
>>including internal apis?).
>>
>
> Jay, this part was not in the KIP and was never discussed.
> Are you proposing adding this? Or is it just an assumption you made?
>
>
>
>>2. The protocol_version will be added to the metadata request
>>3. We will also add a string that this proposal is calling VersionString
>>which will describe the build of kafka in some way. The clients should
>> not
>>under any circumstances do anything with this string other than print it
>>out to the user.
>>
>> One thing I'm not sure about: I think currently metadata sits in the client
>> for 10 mins by default. Say a client bootstraps and then a server is
>> downgraded to an earlier version, won't the client's metadata version
>> indicate that that client handles a version it doesn't actually handle any
>> more? We need to document how clients will handle this.
>>
>> Here are some comments on other details:
>>
>>1. As a minor thing I think we should avoid naming the fields VersionId
>>and VersionString which sort of implies they are both used for
>> versioning.
>>I think we should call them something like ProtocolVersion and
>>BuildDescription, with BuildDescription being totally unspecified other
>>than that it is some kind of human readable string describing a
>> particular
>>Kafka build. We really don't want a client attempting to use this
>> string in
>>any way as that would always be the wrong thing to do in the versioning
>>scheme we are proposing, you should always use the protocol version.
>>2. Does making the topics field in the metadata request nullable
>>actually make sense? We have a history of wanting to add magical values
>>rather than fields. Currently topics=[a] means give me information about
>>topic a, topics=[] means give me information about all topics, and we
>> are
>>proposing topics=null would mean don't give me topics. I don't have a
>>strong opinion.
>>3. I prefer Jason's proposal on using a conservative metadata version
>>versus the empty response hack. However I think that may actually
>>exacerbate the downgrade scenario I described.
>>4. I agree with Jason that we should really look at the details of the
>>implementation so we know it works--implementing server support without
>>actually trying it is kind of risky.
>>
>> As a meta comment: I'd really like to encourage us to think of the protocol
>> as a document that includes the following things:
>>
>>- The binary format, error codes, etc
>>- The request/response interaction
>>- The semantics of each request in different cases
>>- Instructions on how to use this to implement a client
>>
>> This document is versioned with the protocol number and is the source of
>> truth for the protocol.
>>
>> Part of any protocol change needs to be an update to the instructions on
>> how to use that part of the protocol. We should be opinionated. If there
>> are two options there should be a reason, and then we need to document both
>> and say exactly when to use each.
>>
>> I think we also need to get a "how to" document on protocol changes just so
>> people know what they need to do to add a new protocol feature.
>>
>> -Jay
>>
>> On Mon, Mar 14, 2016 at 4:51 PM, Jason Gustafson 
>> wrote:
>>
>> > >
>> > > Are you suggesting this as a solution for the problem where a KIP-35
>> > aware
>> > > client sends a higher version of metadata req, say v2, to a KIP-35
>> aware
>> > > broker that only supports up to v1.
>> >
>> >
>> > Yes, that's right. In that case, the client first sends v1, finds out
>> that
>> > the broker supports v2, and then sends v2 (if it has any reason to do
>> so).
>> >
>> > We had a bit of discussion on such scenarios, and it seemed to be a
>> chicken
>> > > and egg problem that is hard to avoid. Your suggestion definitely makes
>> > > sense, however it falls under the purview of clients.
>> >
>> >
>> > That basically means clients should figure it out for themselves? Might
>> be
>> > nice to have a better answer

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Dana Powers]

Is a linear protocol int consistent with the current release model? It
seems like that would break down w/ the multiple release branches that are
all simultaneously maintained? Or is it implicit that no patch release can
ever bump the protocol int? Or maybe the protocol int gets some extra
"wiggle" on minor / major releases to create unallocated version ints that
could be used on future patch releases / backports?

I think the protocol version int does make sense for folks deploying from
trunk.

-Dana

On Mon, Mar 14, 2016 at 6:13 PM, Jay Kreps  wrote:

> Yeah I think that is the point--we have a proposal for a new protocol
> versioning scheme and a vote on it but it doesn't actually describe
> how versioning will work yet! I gave my vague impression based on this
> thread, but I want to make sure that is correct and get it written
> down before we adopt it.
>
> -Jay
>
> On Mon, Mar 14, 2016 at 5:58 PM, Gwen Shapira  wrote:
> > On Mon, Mar 14, 2016 at 5:31 PM, Jay Kreps  wrote:
> >
> >> Couple of missing things:
> >>
> >> This KIP doesn't have a proposal on versioning it just gives different
> >> options, it'd be good to get a concrete proposal in the KIP. Here is my
> >> understanding of what we are proposing (can someone sanity check and if
> >> correct, update the kip):
> >>
> >>1. We will augment the existing api_version field in the header with
> a
> >>protocol_version that will begin at some initial value and increment
> by
> >> 1
> >>every time we make a changes to any of the api_versions (question:
> >>including internal apis?).
> >>
> >
> > Jay, this part was not in the KIP and was never discussed.
> > Are you proposing adding this? Or is it just an assumption you made?
> >
> >
> >
> >>2. The protocol_version will be added to the metadata request
> >>3. We will also add a string that this proposal is calling
> VersionString
> >>which will describe the build of kafka in some way. The clients
> should
> >> not
> >>under any circumstances do anything with this string other than
> print it
> >>out to the user.
> >>
> >> One thing I'm not sure about: I think currently metadata sits in the
> client
> >> for 10 mins by default. Say a client bootstraps and then a server is
> >> downgraded to an earlier version, won't the client's metadata version
> >> indicate that that client handles a version it doesn't actually handle
> any
> >> more? We need to document how clients will handle this.
> >>
> >> Here are some comments on other details:
> >>
> >>1. As a minor thing I think we should avoid naming the fields
> VersionId
> >>and VersionString which sort of implies they are both used for
> >> versioning.
> >>I think we should call them something like ProtocolVersion and
> >>BuildDescription, with BuildDescription being totally unspecified
> other
> >>than that it is some kind of human readable string describing a
> >> particular
> >>Kafka build. We really don't want a client attempting to use this
> >> string in
> >>any way as that would always be the wrong thing to do in the
> versioning
> >>scheme we are proposing, you should always use the protocol version.
> >>2. Does making the topics field in the metadata request nullable
> >>actually make sense? We have a history of wanting to add magical
> values
> >>rather than fields. Currently topics=[a] means give me information
> about
> >>topic a, topics=[] means give me information about all topics, and we
> >> are
> >>proposing topics=null would mean don't give me topics. I don't have a
> >>strong opinion.
> >>3. I prefer Jason's proposal on using a conservative metadata version
> >>versus the empty response hack. However I think that may actually
> >>exacerbate the downgrade scenario I described.
> >>4. I agree with Jason that we should really look at the details of
> the
> >>implementation so we know it works--implementing server support
> without
> >>actually trying it is kind of risky.
> >>
> >> As a meta comment: I'd really like to encourage us to think of the
> protocol
> >> as a document that includes the following things:
> >>
> >>- The binary format, error codes, etc
> >>- The request/response interaction
> >>- The semantics of each request in different cases
> >>- Instructions on how to use this to implement a client
> >>
> >> This document is versioned with the protocol number and is the source of
> >> truth for the protocol.
> >>
> >> Part of any protocol change needs to be an update to the instructions on
> >> how to use that part of the protocol. We should be opinionated. If there
> >> are two options there should be a reason, and then we need to document
> both
> >> and say exactly when to use each.
> >>
> >> I think we also need to get a "how to" document on protocol changes
> just so
> >> people know what they need to do to add a new protocol feature.
> >>
> >> -Jay
> >>
> >> On Mon, Mar 14, 2016 at 4:51 PM, Jason

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Gwen Shapira]

Thats a great point, Dana. Thanks for bringing this up.

Ewen raised the same concern and suggested something called "feature
detection" where the broker would advertise support for specific features
that clients may need and by that avoid a linear model (it also allows for
advertising features such as compression codecs that are not part of the
protocol at all). I hope he'll jump in to explain :)

Do you have other suggestions?

Gwen



On Mon, Mar 14, 2016 at 6:22 PM, Dana Powers  wrote:

> Is a linear protocol int consistent with the current release model? It
> seems like that would break down w/ the multiple release branches that are
> all simultaneously maintained? Or is it implicit that no patch release can
> ever bump the protocol int? Or maybe the protocol int gets some extra
> "wiggle" on minor / major releases to create unallocated version ints that
> could be used on future patch releases / backports?
>
> I think the protocol version int does make sense for folks deploying from
> trunk.
>
> -Dana
>
> On Mon, Mar 14, 2016 at 6:13 PM, Jay Kreps  wrote:
>
> > Yeah I think that is the point--we have a proposal for a new protocol
> > versioning scheme and a vote on it but it doesn't actually describe
> > how versioning will work yet! I gave my vague impression based on this
> > thread, but I want to make sure that is correct and get it written
> > down before we adopt it.
> >
> > -Jay
> >
> > On Mon, Mar 14, 2016 at 5:58 PM, Gwen Shapira  wrote:
> > > On Mon, Mar 14, 2016 at 5:31 PM, Jay Kreps  wrote:
> > >
> > >> Couple of missing things:
> > >>
> > >> This KIP doesn't have a proposal on versioning it just gives different
> > >> options, it'd be good to get a concrete proposal in the KIP. Here is
> my
> > >> understanding of what we are proposing (can someone sanity check and
> if
> > >> correct, update the kip):
> > >>
> > >>1. We will augment the existing api_version field in the header
> with
> > a
> > >>protocol_version that will begin at some initial value and
> increment
> > by
> > >> 1
> > >>every time we make a changes to any of the api_versions (question:
> > >>including internal apis?).
> > >>
> > >
> > > Jay, this part was not in the KIP and was never discussed.
> > > Are you proposing adding this? Or is it just an assumption you made?
> > >
> > >
> > >
> > >>2. The protocol_version will be added to the metadata request
> > >>3. We will also add a string that this proposal is calling
> > VersionString
> > >>which will describe the build of kafka in some way. The clients
> > should
> > >> not
> > >>under any circumstances do anything with this string other than
> > print it
> > >>out to the user.
> > >>
> > >> One thing I'm not sure about: I think currently metadata sits in the
> > client
> > >> for 10 mins by default. Say a client bootstraps and then a server is
> > >> downgraded to an earlier version, won't the client's metadata version
> > >> indicate that that client handles a version it doesn't actually handle
> > any
> > >> more? We need to document how clients will handle this.
> > >>
> > >> Here are some comments on other details:
> > >>
> > >>1. As a minor thing I think we should avoid naming the fields
> > VersionId
> > >>and VersionString which sort of implies they are both used for
> > >> versioning.
> > >>I think we should call them something like ProtocolVersion and
> > >>BuildDescription, with BuildDescription being totally unspecified
> > other
> > >>than that it is some kind of human readable string describing a
> > >> particular
> > >>Kafka build. We really don't want a client attempting to use this
> > >> string in
> > >>any way as that would always be the wrong thing to do in the
> > versioning
> > >>scheme we are proposing, you should always use the protocol
> version.
> > >>2. Does making the topics field in the metadata request nullable
> > >>actually make sense? We have a history of wanting to add magical
> > values
> > >>rather than fields. Currently topics=[a] means give me information
> > about
> > >>topic a, topics=[] means give me information about all topics, and
> we
> > >> are
> > >>proposing topics=null would mean don't give me topics. I don't
> have a
> > >>strong opinion.
> > >>3. I prefer Jason's proposal on using a conservative metadata
> version
> > >>versus the empty response hack. However I think that may actually
> > >>exacerbate the downgrade scenario I described.
> > >>4. I agree with Jason that we should really look at the details of
> > the
> > >>implementation so we know it works--implementing server support
> > without
> > >>actually trying it is kind of risky.
> > >>
> > >> As a meta comment: I'd really like to encourage us to think of the
> > protocol
> > >> as a document that includes the following things:
> > >>
> > >>- The binary format, error codes, etc
> > >>- The request/response interaction
> > >>- The semant

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Jay Kreps]

Hey Dana,

I am actually thinking about it differently. Basically I think you are
imagining a world in which the Kafka code is the source of truth, and
the Kafka developers make random changes that inflict pain on you at
will. The protocol documentation is basically just some semi-accurate
description of what the code does. It sounds like this isn't too far
from the actual world. :-) In that world I agree that the best we
could do would be to assign some id to versions (the md5 of the Kafka
jar, maybe) and put in various checks around that in clients to try to
keep things working.

But imagine a different approach where we try to really treat the
protocol as a document and treat that as the source of truth. We try
to make this document cover what is and isn't specified and make it
cover enough to support client implementations and a given Kafka
version covers some range of protocols explicitly. There is a version
of this document for each protocol version. The code implements the
protocol rather than vice versa.

So in other words protocol changes are totally ordered and separate
from code development (we might develop them together but the protocol
assignment would come when you checked in the new protocol version
which would happen with your code).

This was really the intention with the protocol originally (though we
were doing it on a per-api basis), but I think that understanding was
not shared by the full team and we have not done a great job of
important things like documentation or explaining how this are
supposed to work so we fall back on the "the protocol is whatever the
code does" thing.

Does that make sense? In that sense think one of the more important
things we could get out of this would not be more versioning features
so much as clear docs and processes around protocol versioning.

-Jay

On Mon, Mar 14, 2016 at 6:22 PM, Dana Powers  wrote:
> Is a linear protocol int consistent with the current release model? It
> seems like that would break down w/ the multiple release branches that are
> all simultaneously maintained? Or is it implicit that no patch release can
> ever bump the protocol int? Or maybe the protocol int gets some extra
> "wiggle" on minor / major releases to create unallocated version ints that
> could be used on future patch releases / backports?
>
> I think the protocol version int does make sense for folks deploying from
> trunk.
>
> -Dana
>
> On Mon, Mar 14, 2016 at 6:13 PM, Jay Kreps  wrote:
>
>> Yeah I think that is the point--we have a proposal for a new protocol
>> versioning scheme and a vote on it but it doesn't actually describe
>> how versioning will work yet! I gave my vague impression based on this
>> thread, but I want to make sure that is correct and get it written
>> down before we adopt it.
>>
>> -Jay
>>
>> On Mon, Mar 14, 2016 at 5:58 PM, Gwen Shapira  wrote:
>> > On Mon, Mar 14, 2016 at 5:31 PM, Jay Kreps  wrote:
>> >
>> >> Couple of missing things:
>> >>
>> >> This KIP doesn't have a proposal on versioning it just gives different
>> >> options, it'd be good to get a concrete proposal in the KIP. Here is my
>> >> understanding of what we are proposing (can someone sanity check and if
>> >> correct, update the kip):
>> >>
>> >>1. We will augment the existing api_version field in the header with
>> a
>> >>protocol_version that will begin at some initial value and increment
>> by
>> >> 1
>> >>every time we make a changes to any of the api_versions (question:
>> >>including internal apis?).
>> >>
>> >
>> > Jay, this part was not in the KIP and was never discussed.
>> > Are you proposing adding this? Or is it just an assumption you made?
>> >
>> >
>> >
>> >>2. The protocol_version will be added to the metadata request
>> >>3. We will also add a string that this proposal is calling
>> VersionString
>> >>which will describe the build of kafka in some way. The clients
>> should
>> >> not
>> >>under any circumstances do anything with this string other than
>> print it
>> >>out to the user.
>> >>
>> >> One thing I'm not sure about: I think currently metadata sits in the
>> client
>> >> for 10 mins by default. Say a client bootstraps and then a server is
>> >> downgraded to an earlier version, won't the client's metadata version
>> >> indicate that that client handles a version it doesn't actually handle
>> any
>> >> more? We need to document how clients will handle this.
>> >>
>> >> Here are some comments on other details:
>> >>
>> >>1. As a minor thing I think we should avoid naming the fields
>> VersionId
>> >>and VersionString which sort of implies they are both used for
>> >> versioning.
>> >>I think we should call them something like ProtocolVersion and
>> >>BuildDescription, with BuildDescription being totally unspecified
>> other
>> >>than that it is some kind of human readable string describing a
>> >> particular
>> >>Kafka build. We really don't want a client attempting to use this
>> >> string in

Re: [DISCUSS] KIP-35 - Retrieve protocol version

[Gwen Shapira]

Jay,

Ewen had a good example:

1. 0.10.0 (protocol v4)  is released with current KIP-35
2. On trunk, modify produce requests and bump to v5
3. On trunk, we modify metadata requests and bump to v6
4. Now we decide that the metadata change fixes a super critical issue and
want to backport the change. What's the protocol version of the next
release of 0.10.0 - which supports v6 protocol only partially?

Gwen

On Mon, Mar 14, 2016 at 6:38 PM, Jay Kreps  wrote:

> Hey Dana,
>
> I am actually thinking about it differently. Basically I think you are
> imagining a world in which the Kafka code is the source of truth, and
> the Kafka developers make random changes that inflict pain on you at
> will. The protocol documentation is basically just some semi-accurate
> description of what the code does. It sounds like this isn't too far
> from the actual world. :-) In that world I agree that the best we
> could do would be to assign some id to versions (the md5 of the Kafka
> jar, maybe) and put in various checks around that in clients to try to
> keep things working.
>
> But imagine a different approach where we try to really treat the
> protocol as a document and treat that as the source of truth. We try
> to make this document cover what is and isn't specified and make it
> cover enough to support client implementations and a given Kafka
> version covers some range of protocols explicitly. There is a version
> of this document for each protocol version. The code implements the
> protocol rather than vice versa.
>
> So in other words protocol changes are totally ordered and separate
> from code development (we might develop them together but the protocol
> assignment would come when you checked in the new protocol version
> which would happen with your code).
>
> This was really the intention with the protocol originally (though we
> were doing it on a per-api basis), but I think that understanding was
> not shared by the full team and we have not done a great job of
> important things like documentation or explaining how this are
> supposed to work so we fall back on the "the protocol is whatever the
> code does" thing.
>
> Does that make sense? In that sense think one of the more important
> things we could get out of this would not be more versioning features
> so much as clear docs and processes around protocol versioning.
>
> -Jay
>
> On Mon, Mar 14, 2016 at 6:22 PM, Dana Powers 
> wrote:
> > Is a linear protocol int consistent with the current release model? It
> > seems like that would break down w/ the multiple release branches that
> are
> > all simultaneously maintained? Or is it implicit that no patch release
> can
> > ever bump the protocol int? Or maybe the protocol int gets some extra
> > "wiggle" on minor / major releases to create unallocated version ints
> that
> > could be used on future patch releases / backports?
> >
> > I think the protocol version int does make sense for folks deploying from
> > trunk.
> >
> > -Dana
> >
> > On Mon, Mar 14, 2016 at 6:13 PM, Jay Kreps  wrote:
> >
> >> Yeah I think that is the point--we have a proposal for a new protocol
> >> versioning scheme and a vote on it but it doesn't actually describe
> >> how versioning will work yet! I gave my vague impression based on this
> >> thread, but I want to make sure that is correct and get it written
> >> down before we adopt it.
> >>
> >> -Jay
> >>
> >> On Mon, Mar 14, 2016 at 5:58 PM, Gwen Shapira 
> wrote:
> >> > On Mon, Mar 14, 2016 at 5:31 PM, Jay Kreps  wrote:
> >> >
> >> >> Couple of missing things:
> >> >>
> >> >> This KIP doesn't have a proposal on versioning it just gives
> different
> >> >> options, it'd be good to get a concrete proposal in the KIP. Here is
> my
> >> >> understanding of what we are proposing (can someone sanity check and
> if
> >> >> correct, update the kip):
> >> >>
> >> >>1. We will augment the existing api_version field in the header
> with
> >> a
> >> >>protocol_version that will begin at some initial value and
> increment
> >> by
> >> >> 1
> >> >>every time we make a changes to any of the api_versions (question:
> >> >>including internal apis?).
> >> >>
> >> >
> >> > Jay, this part was not in the KIP and was never discussed.
> >> > Are you proposing adding this? Or is it just an assumption you made?
> >> >
> >> >
> >> >
> >> >>2. The protocol_version will be added to the metadata request
> >> >>3. We will also add a string that this proposal is calling
> >> VersionString
> >> >>which will describe the build of kafka in some way. The clients
> >> should
> >> >> not
> >> >>under any circumstances do anything with this string other than
> >> print it
> >> >>out to the user.
> >> >>
> >> >> One thing I'm not sure about: I think currently metadata sits in the
> >> client
> >> >> for 10 mins by default. Say a client bootstraps and then a server is
> >> >> downgraded to an earlier version, won't the client's metadata version
> >> >> indicate that that client handles

[GitHub] kafka pull request: KAFKA-3336: Unify Serializer and Deserializer ...

[guozhangwang]

GitHub user guozhangwang opened a pull request:

https://github.com/apache/kafka/pull/1066

KAFKA-3336: Unify Serializer and Deserializer into Serialization



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/guozhangwang/kafka K3336

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1066.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1066


commit ed8245a35397ed1a1a28141a18e5182acaaeceda
Author: Guozhang Wang 
Date:   2016-03-14T22:08:55Z

reuse Serde for unification

commit 6264c4eadf37f8cfe16909f1ee328c33eb00c30f
Author: Guozhang Wang 
Date:   2016-03-15T01:46:50Z

K3336 v1




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3336) Unify ser/de pair classes into one serde class

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15194545#comment-15194545
 ] 

ASF GitHub Bot commented on KAFKA-3336:
---

GitHub user guozhangwang opened a pull request:

https://github.com/apache/kafka/pull/1066

KAFKA-3336: Unify Serializer and Deserializer into Serialization



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/guozhangwang/kafka K3336

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1066.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1066


commit ed8245a35397ed1a1a28141a18e5182acaaeceda
Author: Guozhang Wang 
Date:   2016-03-14T22:08:55Z

reuse Serde for unification

commit 6264c4eadf37f8cfe16909f1ee328c33eb00c30f
Author: Guozhang Wang 
Date:   2016-03-15T01:46:50Z

K3336 v1




> Unify ser/de pair classes into one serde class
> --
>
> Key: KAFKA-3336
> URL: https://issues.apache.org/jira/browse/KAFKA-3336
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
>Priority: Blocker
> Fix For: 0.10.0.0
>
>
> Right now users must provide two separate classes for serializers and 
> deserializers, respectively.  This means the current API functions have at 
> least 2 * numberOfTypes parameters.
> *Example (current, bad): "foo(..., longSerializer, longDeserializer)".*
> Because the serde aspect of the API is already one of the biggest UX issues, 
> we should unify the serde functionality into a single serde class, i.e. one 
> class that provides both serialization and deserialization functionality.  
> This will reduce the number of required serde parameters in the API by 50%.
> *Example (suggested, better): "foo(..., longSerializerDeserializer)"*. 
> * Note: This parameter name is horrible and only used to highlight the 
> difference to the "current" example above.
> We also want to 1) add a pairing function for each operator that does not 
> require serialization and 2) add a default serde in the configs to make these 
> not required configs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2982; Mark the old Scala producer and re...

[ijuma]

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/1067

KAFKA-2982; Mark the old Scala producer and related classes as deprecated



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka kafka-2982-deprecate-old-producers

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1067.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1067


commit 335d58e035bc4980ff6fba66ec51a1fe47b4fa8b
Author: Ismael Juma 
Date:   2016-03-10T11:56:47Z

Fix and suppress number of unchecked warnings

Ignored Kafka Streams on this iteration.

commit 965684a599ad438bf4668309c7ef7219e9836030
Author: Ismael Juma 
Date:   2016-03-10T11:57:18Z

Add `@SafeVarargs` annotation to fix warning

commit 0730839f1d671448f62349877ac80fd7ec139243
Author: Ismael Juma 
Date:   2016-03-10T11:58:02Z

Suppress unfixable deprecation warnings

commit 8d0e3fb5727791b7173e4800e41fd6870e829cd4
Author: Ismael Juma 
Date:   2016-03-10T11:58:25Z

Replace deprecated by non-deprecated usage

commit 9d1347b3f62e21f69cf0f0879b6ac22872e63c21
Author: Ismael Juma 
Date:   2016-03-10T11:58:57Z

Avoid reflective calls via structural types in Scala

commit 1eb054a6a6cd4b3f5f6303dbe193f27deffe11b4
Author: Ismael Juma 
Date:   2016-03-10T11:59:57Z

Tweak compiler settings for scalac and javac

commit ae25e88045f606bf51d38e067fa6ca3dec15118d
Author: Ismael Juma 
Date:   2016-03-11T22:08:03Z

Use `sourceCompatibility` instead of hardcoding `1.7`

commit 5629d3891993ea96acf82d4f3784bf199df1d28f
Author: Ismael Juma 
Date:   2016-03-15T01:49:05Z

Deprecate old producers




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2982) Mark the old Scala producer and related classes as deprecated

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15194549#comment-15194549
 ] 

ASF GitHub Bot commented on KAFKA-2982:
---

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/1067

KAFKA-2982; Mark the old Scala producer and related classes as deprecated



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka kafka-2982-deprecate-old-producers

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1067.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1067


commit 335d58e035bc4980ff6fba66ec51a1fe47b4fa8b
Author: Ismael Juma 
Date:   2016-03-10T11:56:47Z

Fix and suppress number of unchecked warnings

Ignored Kafka Streams on this iteration.

commit 965684a599ad438bf4668309c7ef7219e9836030
Author: Ismael Juma 
Date:   2016-03-10T11:57:18Z

Add `@SafeVarargs` annotation to fix warning

commit 0730839f1d671448f62349877ac80fd7ec139243
Author: Ismael Juma 
Date:   2016-03-10T11:58:02Z

Suppress unfixable deprecation warnings

commit 8d0e3fb5727791b7173e4800e41fd6870e829cd4
Author: Ismael Juma 
Date:   2016-03-10T11:58:25Z

Replace deprecated by non-deprecated usage

commit 9d1347b3f62e21f69cf0f0879b6ac22872e63c21
Author: Ismael Juma 
Date:   2016-03-10T11:58:57Z

Avoid reflective calls via structural types in Scala

commit 1eb054a6a6cd4b3f5f6303dbe193f27deffe11b4
Author: Ismael Juma 
Date:   2016-03-10T11:59:57Z

Tweak compiler settings for scalac and javac

commit ae25e88045f606bf51d38e067fa6ca3dec15118d
Author: Ismael Juma 
Date:   2016-03-11T22:08:03Z

Use `sourceCompatibility` instead of hardcoding `1.7`

commit 5629d3891993ea96acf82d4f3784bf199df1d28f
Author: Ismael Juma 
Date:   2016-03-15T01:49:05Z

Deprecate old producers




> Mark the old Scala producer and related classes as deprecated
> -
>
> Key: KAFKA-2982
> URL: https://issues.apache.org/jira/browse/KAFKA-2982
> Project: Kafka
>  Issue Type: Sub-task
>Affects Versions: 0.9.0.0
>Reporter: Grant Henke
>Assignee: Ismael Juma
>Priority: Critical
> Fix For: 0.10.0.0
>
>
> Now that the new producer and consumer are released the old Scala producer 
> and consumer clients should be deprecated to encourage use of the new clients 
> and facilitate the removal of the old clients.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2982) Mark the old Scala producer and related classes as deprecated

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-2982:
---
Status: Patch Available  (was: In Progress)

> Mark the old Scala producer and related classes as deprecated
> -
>
> Key: KAFKA-2982
> URL: https://issues.apache.org/jira/browse/KAFKA-2982
> Project: Kafka
>  Issue Type: Sub-task
>Affects Versions: 0.9.0.0
>Reporter: Grant Henke
>Assignee: Ismael Juma
>Priority: Critical
> Fix For: 0.10.0.0
>
>
> Now that the new producer and consumer are released the old Scala producer 
> and consumer clients should be deprecated to encourage use of the new clients 
> and facilitate the removal of the old clients.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-3397) add missing 'REQUIRED' for tools.GetOffsetShell

[Xin Wang (JIRA)]

Xin Wang created KAFKA-3397:
---

 Summary: add missing 'REQUIRED' for tools.GetOffsetShell
 Key: KAFKA-3397
 URL: https://issues.apache.org/jira/browse/KAFKA-3397
 Project: Kafka
  Issue Type: Bug
  Components: tools
Reporter: Xin Wang
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3373 add 'log' prefix to configurations ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1049


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-3373) Add `log` prefix to KIP-31/32 configs

[Gwen Shapira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira updated KAFKA-3373:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

Issue resolved by pull request 1049
[https://github.com/apache/kafka/pull/1049]

> Add `log` prefix to KIP-31/32 configs
> -
>
> Key: KAFKA-3373
> URL: https://issues.apache.org/jira/browse/KAFKA-3373
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ismael Juma
>Assignee: Jiangjie Qin
>Priority: Blocker
> Fix For: 0.10.0.0
>
>
> [~jjkoshy] suggested that we should prefix the configs introduced as part of 
> KIP-31/32 to include a `log` prefix:
> message.format.version
> message.timestamp.type
> message.timestamp.difference.max.ms
> If we do it, we must update the KIP.
> Marking it as blocker because we should decide either way before 0.10.0.0.
> Discussion here:
> https://github.com/apache/kafka/pull/907#issuecomment-193950768



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3375; Suppress deprecated warnings where...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1042


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-3375) Suppress and fix compiler warnings where reasonable and tweak compiler settings

[Gwen Shapira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira updated KAFKA-3375:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

Issue resolved by pull request 1042
[https://github.com/apache/kafka/pull/1042]

> Suppress and fix compiler warnings where reasonable and tweak compiler 
> settings
> ---
>
> Key: KAFKA-3375
> URL: https://issues.apache.org/jira/browse/KAFKA-3375
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Ismael Juma
>Assignee: Ismael Juma
> Fix For: 0.10.0.0
>
>
> This will make it easier to do KAFKA-2982.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3375) Suppress and fix compiler warnings where reasonable and tweak compiler settings

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15194580#comment-15194580
 ] 

ASF GitHub Bot commented on KAFKA-3375:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1042


> Suppress and fix compiler warnings where reasonable and tweak compiler 
> settings
> ---
>
> Key: KAFKA-3375
> URL: https://issues.apache.org/jira/browse/KAFKA-3375
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Ismael Juma
>Assignee: Ismael Juma
> Fix For: 0.10.0.0
>
>
> This will make it easier to do KAFKA-2982.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3373) Add `log` prefix to KIP-31/32 configs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15194577#comment-15194577
 ] 

ASF GitHub Bot commented on KAFKA-3373:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1049


> Add `log` prefix to KIP-31/32 configs
> -
>
> Key: KAFKA-3373
> URL: https://issues.apache.org/jira/browse/KAFKA-3373
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ismael Juma
>Assignee: Jiangjie Qin
>Priority: Blocker
> Fix For: 0.10.0.0
>
>
> [~jjkoshy] suggested that we should prefix the configs introduced as part of 
> KIP-31/32 to include a `log` prefix:
> message.format.version
> message.timestamp.type
> message.timestamp.difference.max.ms
> If we do it, we must update the KIP.
> Marking it as blocker because we should decide either way before 0.10.0.0.
> Discussion here:
> https://github.com/apache/kafka/pull/907#issuecomment-193950768



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3013

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/695


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---