Re: [DISCUSSION] KIP-2: Refactor Brokers to Allow Multiple Endpoints
Todd, Could you elaborate on the benefit for having a separate endpoint for intra-cluster communication? Is it mainly for giving intra-cluster requests a high priority? At this moment, having a separate endpoint just means that the socket connection for the intra-cluster communication is handled by a separate acceptor thread. The processing of the requests from the network and the handling of the requests are still shared by a single thread pool. So, if any of the thread pool is exhausted, the intra-cluster requests will still be delayed. We can potentially change this model, but this requires more work. An alternative is to just rely on quotas. Intra-cluster requests will be exempt from any kind of throttling. Gwen, I agree that defaulting wire.protocol.version to the current version is probably better. It just means that we need to document the migration path for previous versions. Thanks, Jun On Wed, Feb 11, 2015 at 6:33 PM, Todd Palino tpal...@gmail.com wrote: Thanks, Gwen. This looks good to me as far as the wire protocol versioning goes. I agree with you on defaulting to the new wire protocol version for new installs. I think it will also need to be very clear (to general installer of Kafka, and not just developers) in documentation when the wire protocol version changes moving forwards, and what the risk/benefit of changing to the new version is. Since a rolling upgrade of the intra-cluster protocol is supported, will a rolling downgrade work as well? Should a flaw (bug, security, or otherwise) be discovered after upgrade, is it possible to change the wire.protocol.version back to 0.8.2 and do a rolling bounce? On the host/port/protocol specification, specifically the ZK config format, is it possible to have an un-advertised endpoint? I would see this as potentially useful if you wanted to have an endpoint that you are reserving for intra-cluster communication, and you would prefer to not have it advertised at all. Perhaps it is blocked by a firewall rule or other authentication method. This could also allow you to duplicate a security protocol type but segregate it on a different port or interface (if it is unadvertised, there is no ambiguity to the clients as to which endpoint should be selected). I believe I asked about that previously, and I didn't track what the final outcome was or even if it was discussed further. -Todd On Wed, Feb 11, 2015 at 4:38 PM, Gwen Shapira gshap...@cloudera.com wrote: Added Jun's notes to the KIP (Thanks for explaining so clearly, Jun. I was clearly struggling with this...) and removed the reference to use.new.wire.protocol. On Wed, Feb 11, 2015 at 4:19 PM, Joel Koshy jjkosh...@gmail.com wrote: The description that Jun gave for (2) was the detail I was looking for - Gwen can you update the KIP with that for completeness/clarity? I'm +1 as well overall. However, I think it would be good if we also get an ack from someone who is more experienced on the operations side (say, Todd) to review especially the upgrade plan. On Wed, Feb 11, 2015 at 09:40:50AM -0800, Jun Rao wrote: +1 for proposed changes in 1 and 2. 1. The impact is that if someone uses SimpleConsumer and references Broker explicitly, the application needs code change to compile with 0.8.3. Since SimpleConsumer is not widely used, breaking the API in SimpleConsumer but maintaining overall code cleanness seems to be a better tradeoff. 2. For clarification, the issue is the following. In 0.8.3, we will be evolving the wire protocol of UpdateMedataRequest (to send info about endpoints for different security protocols). Since this is used in intra-cluster communication, we need to do the upgrade in two steps. The idea is that in 0.8.3, we will default wire.protocol.version to 0.8.2. When upgrading to 0.8.3, in step 1, we do a rolling upgrade to 0.8.3. After step 1, all brokers will be capable for processing the new protocol in 0.8.3, but without actually using it. In step 2, we configure wire.protocol.version to 0.8.3 in each broker and do another rolling restart. After step 2, all brokers will start using the new protocol in 0.8.3. Let's say that in the next release 0.9, we are changing the intra-cluster wire protocol again. We will do the similar thing: defaulting wire.protocol.version to 0.8.3 in 0.9 so that people can upgrade from 0.8.3 to 0.9 in two steps. For people who want to upgrade from 0.8.2 to 0.9 directly, they will have to configure wire.protocol.version to 0.8.2 first and then do the two-step upgrade to 0.9. Gwen, In KIP2, there is still a reference to use.new.protocol. This needs to be removed. Also, would it be better to use intra.cluster.wire.protocol.version since this only applies to the wire protocol among brokers? Others, The patch in KAFKA-1809
Re: [DISCUSSION] KIP-2: Refactor Brokers to Allow Multiple Endpoints
Hi, 1. We have another issue with reserving a port for inter-broker communication (at least by not advertising the endpoint). Currently brokers are pretty much normal consumers when replicating. Not exactly, but close. Which mean they also learn about other brokers from the registration in ZK. If a broker fails to advertise an endpoint, no one will know about it - neither clients nor other brokers. If Todd has a good use-case for reserving ports for inter-broker messages (firewalls?), we can support it as a follow-up JIRA. 2. I updated the upgrade guide in the KIP and created KAFKA-1949 to follow up in the docs. Gwen On Thu, Feb 12, 2015 at 11:38 AM, Jun Rao j...@confluent.io wrote: Todd, Could you elaborate on the benefit for having a separate endpoint for intra-cluster communication? Is it mainly for giving intra-cluster requests a high priority? At this moment, having a separate endpoint just means that the socket connection for the intra-cluster communication is handled by a separate acceptor thread. The processing of the requests from the network and the handling of the requests are still shared by a single thread pool. So, if any of the thread pool is exhausted, the intra-cluster requests will still be delayed. We can potentially change this model, but this requires more work. An alternative is to just rely on quotas. Intra-cluster requests will be exempt from any kind of throttling. Gwen, I agree that defaulting wire.protocol.version to the current version is probably better. It just means that we need to document the migration path for previous versions. Thanks, Jun On Wed, Feb 11, 2015 at 6:33 PM, Todd Palino tpal...@gmail.com wrote: Thanks, Gwen. This looks good to me as far as the wire protocol versioning goes. I agree with you on defaulting to the new wire protocol version for new installs. I think it will also need to be very clear (to general installer of Kafka, and not just developers) in documentation when the wire protocol version changes moving forwards, and what the risk/benefit of changing to the new version is. Since a rolling upgrade of the intra-cluster protocol is supported, will a rolling downgrade work as well? Should a flaw (bug, security, or otherwise) be discovered after upgrade, is it possible to change the wire.protocol.version back to 0.8.2 and do a rolling bounce? On the host/port/protocol specification, specifically the ZK config format, is it possible to have an un-advertised endpoint? I would see this as potentially useful if you wanted to have an endpoint that you are reserving for intra-cluster communication, and you would prefer to not have it advertised at all. Perhaps it is blocked by a firewall rule or other authentication method. This could also allow you to duplicate a security protocol type but segregate it on a different port or interface (if it is unadvertised, there is no ambiguity to the clients as to which endpoint should be selected). I believe I asked about that previously, and I didn't track what the final outcome was or even if it was discussed further. -Todd On Wed, Feb 11, 2015 at 4:38 PM, Gwen Shapira gshap...@cloudera.com wrote: Added Jun's notes to the KIP (Thanks for explaining so clearly, Jun. I was clearly struggling with this...) and removed the reference to use.new.wire.protocol. On Wed, Feb 11, 2015 at 4:19 PM, Joel Koshy jjkosh...@gmail.com wrote: The description that Jun gave for (2) was the detail I was looking for - Gwen can you update the KIP with that for completeness/clarity? I'm +1 as well overall. However, I think it would be good if we also get an ack from someone who is more experienced on the operations side (say, Todd) to review especially the upgrade plan. On Wed, Feb 11, 2015 at 09:40:50AM -0800, Jun Rao wrote: +1 for proposed changes in 1 and 2. 1. The impact is that if someone uses SimpleConsumer and references Broker explicitly, the application needs code change to compile with 0.8.3. Since SimpleConsumer is not widely used, breaking the API in SimpleConsumer but maintaining overall code cleanness seems to be a better tradeoff. 2. For clarification, the issue is the following. In 0.8.3, we will be evolving the wire protocol of UpdateMedataRequest (to send info about endpoints for different security protocols). Since this is used in intra-cluster communication, we need to do the upgrade in two steps. The idea is that in 0.8.3, we will default wire.protocol.version to 0.8.2. When upgrading to 0.8.3, in step 1, we do a rolling upgrade to 0.8.3. After step 1, all brokers will be capable for processing the new protocol in 0.8.3, but without actually using it. In step 2, we configure wire.protocol.version to 0.8.3 in each broker and do another rolling
Re: [DISCUSSION] KIP-2: Refactor Brokers to Allow Multiple Endpoints
The idea is more about isolating the intra-cluster traffic from the normal clients as much as possible. There's a couple situations we've seen where this would be useful that I can think of immediately: 1) Normal operation - just having the intra-cluster traffic on a separate network interface would allow it to not get overwhelmed by something like a bootstrapping client who is saturating the network interface. We see this fairly often, where the replication falls behind because of heavy traffic from one application. We can always adjust the network threads, but segregating the traffic is the first step. 2) Isolation in case of an error - We have had situations, more than once, where we are needing to rebuild a cluster after a catastrophic problem and the clients are causing that process to take too long, or are causing additional failures. This has mostly come into play with file descriptor limits in the past, but it's certainly not the only situation. Constantly reconnecting clients continue to cause the brokers to fall over while we are trying to recover a down cluster. The only solution was to firewall off all the clients temporarily. This is a great deal more complicated if the brokers and the clients are all operating over the same port. Now, that said, quotas can be a partial solution to this. I don't want to jump the gun on that discussion (because it's going to come up separately and in more detail), but it is possible to structure quotas in a way that will allow the intra-cluster replication to continue to function in the case of high load. That would partially address case 1, but it does nothing for case 2. Additionally, I think it is also desirable to segregate the traffic even with quotas, so that regardless of the client load, the cluster itself is able to be healthy. -Todd On Thu, Feb 12, 2015 at 11:38 AM, Jun Rao j...@confluent.io wrote: Todd, Could you elaborate on the benefit for having a separate endpoint for intra-cluster communication? Is it mainly for giving intra-cluster requests a high priority? At this moment, having a separate endpoint just means that the socket connection for the intra-cluster communication is handled by a separate acceptor thread. The processing of the requests from the network and the handling of the requests are still shared by a single thread pool. So, if any of the thread pool is exhausted, the intra-cluster requests will still be delayed. We can potentially change this model, but this requires more work. An alternative is to just rely on quotas. Intra-cluster requests will be exempt from any kind of throttling. Gwen, I agree that defaulting wire.protocol.version to the current version is probably better. It just means that we need to document the migration path for previous versions. Thanks, Jun On Wed, Feb 11, 2015 at 6:33 PM, Todd Palino tpal...@gmail.com wrote: Thanks, Gwen. This looks good to me as far as the wire protocol versioning goes. I agree with you on defaulting to the new wire protocol version for new installs. I think it will also need to be very clear (to general installer of Kafka, and not just developers) in documentation when the wire protocol version changes moving forwards, and what the risk/benefit of changing to the new version is. Since a rolling upgrade of the intra-cluster protocol is supported, will a rolling downgrade work as well? Should a flaw (bug, security, or otherwise) be discovered after upgrade, is it possible to change the wire.protocol.version back to 0.8.2 and do a rolling bounce? On the host/port/protocol specification, specifically the ZK config format, is it possible to have an un-advertised endpoint? I would see this as potentially useful if you wanted to have an endpoint that you are reserving for intra-cluster communication, and you would prefer to not have it advertised at all. Perhaps it is blocked by a firewall rule or other authentication method. This could also allow you to duplicate a security protocol type but segregate it on a different port or interface (if it is unadvertised, there is no ambiguity to the clients as to which endpoint should be selected). I believe I asked about that previously, and I didn't track what the final outcome was or even if it was discussed further. -Todd On Wed, Feb 11, 2015 at 4:38 PM, Gwen Shapira gshap...@cloudera.com wrote: Added Jun's notes to the KIP (Thanks for explaining so clearly, Jun. I was clearly struggling with this...) and removed the reference to use.new.wire.protocol. On Wed, Feb 11, 2015 at 4:19 PM, Joel Koshy jjkosh...@gmail.com wrote: The description that Jun gave for (2) was the detail I was looking for - Gwen can you update the KIP with that for completeness/clarity? I'm +1 as well overall. However, I think it would be good if we also get an ack from someone who is more experienced on the operations side
Re: [DISCUSSION] KIP-2: Refactor Brokers to Allow Multiple Endpoints
For rolling downgrading of the intra-cluster protocol, we mostly just need to reverse the steps. If none of the brokers have set wire.protocol.version to the newer version, downgrading can be done by just downgrading the jar, followed by a rolling restart. Otherwise, downgrading needs to be done in two steps: (1) Reconfigure wire.protocol.version to an old version and do a rolling restart of the brokers. After this step, all brokers will be using the old protocol for intra-cluster communication. (2) Downgrade the jar and do another round of rolling restart. Gwen, Could you document that in the WIP-2 wiki too? Thanks, Jun On Wed, Feb 11, 2015 at 6:33 PM, Todd Palino tpal...@gmail.com wrote: Thanks, Gwen. This looks good to me as far as the wire protocol versioning goes. I agree with you on defaulting to the new wire protocol version for new installs. I think it will also need to be very clear (to general installer of Kafka, and not just developers) in documentation when the wire protocol version changes moving forwards, and what the risk/benefit of changing to the new version is. Since a rolling upgrade of the intra-cluster protocol is supported, will a rolling downgrade work as well? Should a flaw (bug, security, or otherwise) be discovered after upgrade, is it possible to change the wire.protocol.version back to 0.8.2 and do a rolling bounce? On the host/port/protocol specification, specifically the ZK config format, is it possible to have an un-advertised endpoint? I would see this as potentially useful if you wanted to have an endpoint that you are reserving for intra-cluster communication, and you would prefer to not have it advertised at all. Perhaps it is blocked by a firewall rule or other authentication method. This could also allow you to duplicate a security protocol type but segregate it on a different port or interface (if it is unadvertised, there is no ambiguity to the clients as to which endpoint should be selected). I believe I asked about that previously, and I didn't track what the final outcome was or even if it was discussed further. -Todd On Wed, Feb 11, 2015 at 4:38 PM, Gwen Shapira gshap...@cloudera.com wrote: Added Jun's notes to the KIP (Thanks for explaining so clearly, Jun. I was clearly struggling with this...) and removed the reference to use.new.wire.protocol. On Wed, Feb 11, 2015 at 4:19 PM, Joel Koshy jjkosh...@gmail.com wrote: The description that Jun gave for (2) was the detail I was looking for - Gwen can you update the KIP with that for completeness/clarity? I'm +1 as well overall. However, I think it would be good if we also get an ack from someone who is more experienced on the operations side (say, Todd) to review especially the upgrade plan. On Wed, Feb 11, 2015 at 09:40:50AM -0800, Jun Rao wrote: +1 for proposed changes in 1 and 2. 1. The impact is that if someone uses SimpleConsumer and references Broker explicitly, the application needs code change to compile with 0.8.3. Since SimpleConsumer is not widely used, breaking the API in SimpleConsumer but maintaining overall code cleanness seems to be a better tradeoff. 2. For clarification, the issue is the following. In 0.8.3, we will be evolving the wire protocol of UpdateMedataRequest (to send info about endpoints for different security protocols). Since this is used in intra-cluster communication, we need to do the upgrade in two steps. The idea is that in 0.8.3, we will default wire.protocol.version to 0.8.2. When upgrading to 0.8.3, in step 1, we do a rolling upgrade to 0.8.3. After step 1, all brokers will be capable for processing the new protocol in 0.8.3, but without actually using it. In step 2, we configure wire.protocol.version to 0.8.3 in each broker and do another rolling restart. After step 2, all brokers will start using the new protocol in 0.8.3. Let's say that in the next release 0.9, we are changing the intra-cluster wire protocol again. We will do the similar thing: defaulting wire.protocol.version to 0.8.3 in 0.9 so that people can upgrade from 0.8.3 to 0.9 in two steps. For people who want to upgrade from 0.8.2 to 0.9 directly, they will have to configure wire.protocol.version to 0.8.2 first and then do the two-step upgrade to 0.9. Gwen, In KIP2, there is still a reference to use.new.protocol. This needs to be removed. Also, would it be better to use intra.cluster.wire.protocol.version since this only applies to the wire protocol among brokers? Others, The patch in KAFKA-1809 is almost ready. It would be good to wrap up the discussion on KIP2 soon. So, if you haven't looked at this KIP, please take a look and send your comments. Thanks, Jun On Mon, Jan 26, 2015 at 8:02 PM, Gwen Shapira gshap...@cloudera.com
Re: [DISCUSSION] KIP-2: Refactor Brokers to Allow Multiple Endpoints
I REALLY like the idea of supporting separate network for inter-broker communication (and probably Zookeeper too). I think its actually a pretty typical configuration in clusters, so I'm surprised we didn't think of it before :) Servers arrive with multiple cards specifically for admin nic vs. clients nic vs storage nic. That said, I'd like to handle it in a separate patch. First because KAFKA-1809 is big enough already, and second because this really deserve its own requirement-gathering and design. Does that make sense? Gwen On Thu, Feb 12, 2015 at 12:34 PM, Todd Palino tpal...@gmail.com wrote: The idea is more about isolating the intra-cluster traffic from the normal clients as much as possible. There's a couple situations we've seen where this would be useful that I can think of immediately: 1) Normal operation - just having the intra-cluster traffic on a separate network interface would allow it to not get overwhelmed by something like a bootstrapping client who is saturating the network interface. We see this fairly often, where the replication falls behind because of heavy traffic from one application. We can always adjust the network threads, but segregating the traffic is the first step. 2) Isolation in case of an error - We have had situations, more than once, where we are needing to rebuild a cluster after a catastrophic problem and the clients are causing that process to take too long, or are causing additional failures. This has mostly come into play with file descriptor limits in the past, but it's certainly not the only situation. Constantly reconnecting clients continue to cause the brokers to fall over while we are trying to recover a down cluster. The only solution was to firewall off all the clients temporarily. This is a great deal more complicated if the brokers and the clients are all operating over the same port. Now, that said, quotas can be a partial solution to this. I don't want to jump the gun on that discussion (because it's going to come up separately and in more detail), but it is possible to structure quotas in a way that will allow the intra-cluster replication to continue to function in the case of high load. That would partially address case 1, but it does nothing for case 2. Additionally, I think it is also desirable to segregate the traffic even with quotas, so that regardless of the client load, the cluster itself is able to be healthy. -Todd On Thu, Feb 12, 2015 at 11:38 AM, Jun Rao j...@confluent.io wrote: Todd, Could you elaborate on the benefit for having a separate endpoint for intra-cluster communication? Is it mainly for giving intra-cluster requests a high priority? At this moment, having a separate endpoint just means that the socket connection for the intra-cluster communication is handled by a separate acceptor thread. The processing of the requests from the network and the handling of the requests are still shared by a single thread pool. So, if any of the thread pool is exhausted, the intra-cluster requests will still be delayed. We can potentially change this model, but this requires more work. An alternative is to just rely on quotas. Intra-cluster requests will be exempt from any kind of throttling. Gwen, I agree that defaulting wire.protocol.version to the current version is probably better. It just means that we need to document the migration path for previous versions. Thanks, Jun On Wed, Feb 11, 2015 at 6:33 PM, Todd Palino tpal...@gmail.com wrote: Thanks, Gwen. This looks good to me as far as the wire protocol versioning goes. I agree with you on defaulting to the new wire protocol version for new installs. I think it will also need to be very clear (to general installer of Kafka, and not just developers) in documentation when the wire protocol version changes moving forwards, and what the risk/benefit of changing to the new version is. Since a rolling upgrade of the intra-cluster protocol is supported, will a rolling downgrade work as well? Should a flaw (bug, security, or otherwise) be discovered after upgrade, is it possible to change the wire.protocol.version back to 0.8.2 and do a rolling bounce? On the host/port/protocol specification, specifically the ZK config format, is it possible to have an un-advertised endpoint? I would see this as potentially useful if you wanted to have an endpoint that you are reserving for intra-cluster communication, and you would prefer to not have it advertised at all. Perhaps it is blocked by a firewall rule or other authentication method. This could also allow you to duplicate a security protocol type but segregate it on a different port or interface (if it is unadvertised, there is no ambiguity to the clients as to which endpoint should be selected). I believe I asked about that previously, and I didn't track what the
Re: [DISCUSSION] KIP-2: Refactor Brokers to Allow Multiple Endpoints
+1 on separating the end points for different purposes. On 2/12/15, 5:47 PM, Gwen Shapira gshap...@cloudera.com wrote: I REALLY like the idea of supporting separate network for inter-broker communication (and probably Zookeeper too). I think its actually a pretty typical configuration in clusters, so I'm surprised we didn't think of it before :) Servers arrive with multiple cards specifically for admin nic vs. clients nic vs storage nic. That said, I'd like to handle it in a separate patch. First because KAFKA-1809 is big enough already, and second because this really deserve its own requirement-gathering and design. Does that make sense? Gwen On Thu, Feb 12, 2015 at 12:34 PM, Todd Palino tpal...@gmail.com wrote: The idea is more about isolating the intra-cluster traffic from the normal clients as much as possible. There's a couple situations we've seen where this would be useful that I can think of immediately: 1) Normal operation - just having the intra-cluster traffic on a separate network interface would allow it to not get overwhelmed by something like a bootstrapping client who is saturating the network interface. We see this fairly often, where the replication falls behind because of heavy traffic from one application. We can always adjust the network threads, but segregating the traffic is the first step. 2) Isolation in case of an error - We have had situations, more than once, where we are needing to rebuild a cluster after a catastrophic problem and the clients are causing that process to take too long, or are causing additional failures. This has mostly come into play with file descriptor limits in the past, but it's certainly not the only situation. Constantly reconnecting clients continue to cause the brokers to fall over while we are trying to recover a down cluster. The only solution was to firewall off all the clients temporarily. This is a great deal more complicated if the brokers and the clients are all operating over the same port. Now, that said, quotas can be a partial solution to this. I don't want to jump the gun on that discussion (because it's going to come up separately and in more detail), but it is possible to structure quotas in a way that will allow the intra-cluster replication to continue to function in the case of high load. That would partially address case 1, but it does nothing for case 2. Additionally, I think it is also desirable to segregate the traffic even with quotas, so that regardless of the client load, the cluster itself is able to be healthy. -Todd On Thu, Feb 12, 2015 at 11:38 AM, Jun Rao j...@confluent.io wrote: Todd, Could you elaborate on the benefit for having a separate endpoint for intra-cluster communication? Is it mainly for giving intra-cluster requests a high priority? At this moment, having a separate endpoint just means that the socket connection for the intra-cluster communication is handled by a separate acceptor thread. The processing of the requests from the network and the handling of the requests are still shared by a single thread pool. So, if any of the thread pool is exhausted, the intra-cluster requests will still be delayed. We can potentially change this model, but this requires more work. An alternative is to just rely on quotas. Intra-cluster requests will be exempt from any kind of throttling. Gwen, I agree that defaulting wire.protocol.version to the current version is probably better. It just means that we need to document the migration path for previous versions. Thanks, Jun On Wed, Feb 11, 2015 at 6:33 PM, Todd Palino tpal...@gmail.com wrote: Thanks, Gwen. This looks good to me as far as the wire protocol versioning goes. I agree with you on defaulting to the new wire protocol version for new installs. I think it will also need to be very clear (to general installer of Kafka, and not just developers) in documentation when the wire protocol version changes moving forwards, and what the risk/benefit of changing to the new version is. Since a rolling upgrade of the intra-cluster protocol is supported, will a rolling downgrade work as well? Should a flaw (bug, security, or otherwise) be discovered after upgrade, is it possible to change the wire.protocol.version back to 0.8.2 and do a rolling bounce? On the host/port/protocol specification, specifically the ZK config format, is it possible to have an un-advertised endpoint? I would see this as potentially useful if you wanted to have an endpoint that you are reserving for intra-cluster communication, and you would prefer to not have it advertised at all. Perhaps it is blocked by a firewall rule or other authentication method. This could also allow you to duplicate a security protocol type but segregate it on a different port or interface (if it is unadvertised, there is no ambiguity to the clients
Re: [DISCUSSION] KIP-2: Refactor Brokers to Allow Multiple Endpoints
+1 on investigating it further as a separate feature that will improve ops significantly (especially since an expert on the operations side has described use cases from actual experience). On Thu, Feb 12, 2015 at 05:47:50PM -0800, Gwen Shapira wrote: I REALLY like the idea of supporting separate network for inter-broker communication (and probably Zookeeper too). I think its actually a pretty typical configuration in clusters, so I'm surprised we didn't think of it before :) Servers arrive with multiple cards specifically for admin nic vs. clients nic vs storage nic. That said, I'd like to handle it in a separate patch. First because KAFKA-1809 is big enough already, and second because this really deserve its own requirement-gathering and design. Does that make sense? Gwen On Thu, Feb 12, 2015 at 12:34 PM, Todd Palino tpal...@gmail.com wrote: The idea is more about isolating the intra-cluster traffic from the normal clients as much as possible. There's a couple situations we've seen where this would be useful that I can think of immediately: 1) Normal operation - just having the intra-cluster traffic on a separate network interface would allow it to not get overwhelmed by something like a bootstrapping client who is saturating the network interface. We see this fairly often, where the replication falls behind because of heavy traffic from one application. We can always adjust the network threads, but segregating the traffic is the first step. 2) Isolation in case of an error - We have had situations, more than once, where we are needing to rebuild a cluster after a catastrophic problem and the clients are causing that process to take too long, or are causing additional failures. This has mostly come into play with file descriptor limits in the past, but it's certainly not the only situation. Constantly reconnecting clients continue to cause the brokers to fall over while we are trying to recover a down cluster. The only solution was to firewall off all the clients temporarily. This is a great deal more complicated if the brokers and the clients are all operating over the same port. Now, that said, quotas can be a partial solution to this. I don't want to jump the gun on that discussion (because it's going to come up separately and in more detail), but it is possible to structure quotas in a way that will allow the intra-cluster replication to continue to function in the case of high load. That would partially address case 1, but it does nothing for case 2. Additionally, I think it is also desirable to segregate the traffic even with quotas, so that regardless of the client load, the cluster itself is able to be healthy. -Todd On Thu, Feb 12, 2015 at 11:38 AM, Jun Rao j...@confluent.io wrote: Todd, Could you elaborate on the benefit for having a separate endpoint for intra-cluster communication? Is it mainly for giving intra-cluster requests a high priority? At this moment, having a separate endpoint just means that the socket connection for the intra-cluster communication is handled by a separate acceptor thread. The processing of the requests from the network and the handling of the requests are still shared by a single thread pool. So, if any of the thread pool is exhausted, the intra-cluster requests will still be delayed. We can potentially change this model, but this requires more work. An alternative is to just rely on quotas. Intra-cluster requests will be exempt from any kind of throttling. Gwen, I agree that defaulting wire.protocol.version to the current version is probably better. It just means that we need to document the migration path for previous versions. Thanks, Jun On Wed, Feb 11, 2015 at 6:33 PM, Todd Palino tpal...@gmail.com wrote: Thanks, Gwen. This looks good to me as far as the wire protocol versioning goes. I agree with you on defaulting to the new wire protocol version for new installs. I think it will also need to be very clear (to general installer of Kafka, and not just developers) in documentation when the wire protocol version changes moving forwards, and what the risk/benefit of changing to the new version is. Since a rolling upgrade of the intra-cluster protocol is supported, will a rolling downgrade work as well? Should a flaw (bug, security, or otherwise) be discovered after upgrade, is it possible to change the wire.protocol.version back to 0.8.2 and do a rolling bounce? On the host/port/protocol specification, specifically the ZK config format, is it possible to have an un-advertised endpoint? I would see this as potentially useful if you wanted to have an endpoint that you are reserving for intra-cluster communication, and you would prefer to not have it advertised at all. Perhaps it is
Re: [DISCUSSION] KIP-2: Refactor Brokers to Allow Multiple Endpoints
I'll be happy to give the initial design a go, but will probably only get to it after Strata. So either wait a bit (there are enough KIPs to review ;) or someone else can get started. Gwen On Thu, Feb 12, 2015 at 6:55 PM, Joel Koshy jjkosh...@gmail.com wrote: +1 on investigating it further as a separate feature that will improve ops significantly (especially since an expert on the operations side has described use cases from actual experience). On Thu, Feb 12, 2015 at 05:47:50PM -0800, Gwen Shapira wrote: I REALLY like the idea of supporting separate network for inter-broker communication (and probably Zookeeper too). I think its actually a pretty typical configuration in clusters, so I'm surprised we didn't think of it before :) Servers arrive with multiple cards specifically for admin nic vs. clients nic vs storage nic. That said, I'd like to handle it in a separate patch. First because KAFKA-1809 is big enough already, and second because this really deserve its own requirement-gathering and design. Does that make sense? Gwen On Thu, Feb 12, 2015 at 12:34 PM, Todd Palino tpal...@gmail.com wrote: The idea is more about isolating the intra-cluster traffic from the normal clients as much as possible. There's a couple situations we've seen where this would be useful that I can think of immediately: 1) Normal operation - just having the intra-cluster traffic on a separate network interface would allow it to not get overwhelmed by something like a bootstrapping client who is saturating the network interface. We see this fairly often, where the replication falls behind because of heavy traffic from one application. We can always adjust the network threads, but segregating the traffic is the first step. 2) Isolation in case of an error - We have had situations, more than once, where we are needing to rebuild a cluster after a catastrophic problem and the clients are causing that process to take too long, or are causing additional failures. This has mostly come into play with file descriptor limits in the past, but it's certainly not the only situation. Constantly reconnecting clients continue to cause the brokers to fall over while we are trying to recover a down cluster. The only solution was to firewall off all the clients temporarily. This is a great deal more complicated if the brokers and the clients are all operating over the same port. Now, that said, quotas can be a partial solution to this. I don't want to jump the gun on that discussion (because it's going to come up separately and in more detail), but it is possible to structure quotas in a way that will allow the intra-cluster replication to continue to function in the case of high load. That would partially address case 1, but it does nothing for case 2. Additionally, I think it is also desirable to segregate the traffic even with quotas, so that regardless of the client load, the cluster itself is able to be healthy. -Todd On Thu, Feb 12, 2015 at 11:38 AM, Jun Rao j...@confluent.io wrote: Todd, Could you elaborate on the benefit for having a separate endpoint for intra-cluster communication? Is it mainly for giving intra-cluster requests a high priority? At this moment, having a separate endpoint just means that the socket connection for the intra-cluster communication is handled by a separate acceptor thread. The processing of the requests from the network and the handling of the requests are still shared by a single thread pool. So, if any of the thread pool is exhausted, the intra-cluster requests will still be delayed. We can potentially change this model, but this requires more work. An alternative is to just rely on quotas. Intra-cluster requests will be exempt from any kind of throttling. Gwen, I agree that defaulting wire.protocol.version to the current version is probably better. It just means that we need to document the migration path for previous versions. Thanks, Jun On Wed, Feb 11, 2015 at 6:33 PM, Todd Palino tpal...@gmail.com wrote: Thanks, Gwen. This looks good to me as far as the wire protocol versioning goes. I agree with you on defaulting to the new wire protocol version for new installs. I think it will also need to be very clear (to general installer of Kafka, and not just developers) in documentation when the wire protocol version changes moving forwards, and what the risk/benefit of changing to the new version is. Since a rolling upgrade of the intra-cluster protocol is supported, will a rolling downgrade work as well? Should a flaw (bug, security, or otherwise) be discovered after upgrade, is it possible to change the wire.protocol.version
Re: [DISCUSSION] KIP-2: Refactor Brokers to Allow Multiple Endpoints
+1 for proposed changes in 1 and 2. 1. The impact is that if someone uses SimpleConsumer and references Broker explicitly, the application needs code change to compile with 0.8.3. Since SimpleConsumer is not widely used, breaking the API in SimpleConsumer but maintaining overall code cleanness seems to be a better tradeoff. 2. For clarification, the issue is the following. In 0.8.3, we will be evolving the wire protocol of UpdateMedataRequest (to send info about endpoints for different security protocols). Since this is used in intra-cluster communication, we need to do the upgrade in two steps. The idea is that in 0.8.3, we will default wire.protocol.version to 0.8.2. When upgrading to 0.8.3, in step 1, we do a rolling upgrade to 0.8.3. After step 1, all brokers will be capable for processing the new protocol in 0.8.3, but without actually using it. In step 2, we configure wire.protocol.version to 0.8.3 in each broker and do another rolling restart. After step 2, all brokers will start using the new protocol in 0.8.3. Let's say that in the next release 0.9, we are changing the intra-cluster wire protocol again. We will do the similar thing: defaulting wire.protocol.version to 0.8.3 in 0.9 so that people can upgrade from 0.8.3 to 0.9 in two steps. For people who want to upgrade from 0.8.2 to 0.9 directly, they will have to configure wire.protocol.version to 0.8.2 first and then do the two-step upgrade to 0.9. Gwen, In KIP2, there is still a reference to use.new.protocol. This needs to be removed. Also, would it be better to use intra.cluster.wire.protocol.version since this only applies to the wire protocol among brokers? Others, The patch in KAFKA-1809 is almost ready. It would be good to wrap up the discussion on KIP2 soon. So, if you haven't looked at this KIP, please take a look and send your comments. Thanks, Jun On Mon, Jan 26, 2015 at 8:02 PM, Gwen Shapira gshap...@cloudera.com wrote: Hi Kafka Devs, While reviewing the patch for KAFKA-1809, we came across two questions that we are interested in hearing the community out on. 1. This patch changes the Broker class and adds a new class BrokerEndPoint that behaves like the previous broker. While technically kafka.cluster.Broker is not part of the public API, it is returned by javaapi, used with the SimpleConsumer. Getting replicas from PartitionMetadata will now return BrokerEndPoint instead of Broker. All method calls remain the same, but since we return a new type, we break the API. Note that this breakage does not prevent upgrades - existing SimpleConsumers will continue working (because we are wire-compatible). The only thing that won't work is building SimpleConsumers with dependency on Kafka versions higher than 0.8.2. Arguably, we don't want anyone to do it anyway :) So: Do we state that the highest release on which SimpleConsumers can depend is 0.8.2? Or shall we keep Broker as is and create an UberBroker which will contain multiple brokers as its endpoints? 2. The KIP suggests use.new.wire.protocol configuration to decide which protocols the brokers will use to talk to each other. The problem is that after the next upgrade, the wire protocol is no longer new, so we'll have to reset it to false for the following upgrade, then change to true again... and upgrading more than a single version will be impossible. Bad idea :) As an alternative, we can have a property for each version and set one of them to true. Or (simple, I think) have wire.protocol.version property and accept version numbers (0.8.2, 0.8.3, 0.9) as values. Please share your thoughts :) Gwen
Re: [DISCUSSION] KIP-2: Refactor Brokers to Allow Multiple Endpoints
The description that Jun gave for (2) was the detail I was looking for - Gwen can you update the KIP with that for completeness/clarity? I'm +1 as well overall. However, I think it would be good if we also get an ack from someone who is more experienced on the operations side (say, Todd) to review especially the upgrade plan. On Wed, Feb 11, 2015 at 09:40:50AM -0800, Jun Rao wrote: +1 for proposed changes in 1 and 2. 1. The impact is that if someone uses SimpleConsumer and references Broker explicitly, the application needs code change to compile with 0.8.3. Since SimpleConsumer is not widely used, breaking the API in SimpleConsumer but maintaining overall code cleanness seems to be a better tradeoff. 2. For clarification, the issue is the following. In 0.8.3, we will be evolving the wire protocol of UpdateMedataRequest (to send info about endpoints for different security protocols). Since this is used in intra-cluster communication, we need to do the upgrade in two steps. The idea is that in 0.8.3, we will default wire.protocol.version to 0.8.2. When upgrading to 0.8.3, in step 1, we do a rolling upgrade to 0.8.3. After step 1, all brokers will be capable for processing the new protocol in 0.8.3, but without actually using it. In step 2, we configure wire.protocol.version to 0.8.3 in each broker and do another rolling restart. After step 2, all brokers will start using the new protocol in 0.8.3. Let's say that in the next release 0.9, we are changing the intra-cluster wire protocol again. We will do the similar thing: defaulting wire.protocol.version to 0.8.3 in 0.9 so that people can upgrade from 0.8.3 to 0.9 in two steps. For people who want to upgrade from 0.8.2 to 0.9 directly, they will have to configure wire.protocol.version to 0.8.2 first and then do the two-step upgrade to 0.9. Gwen, In KIP2, there is still a reference to use.new.protocol. This needs to be removed. Also, would it be better to use intra.cluster.wire.protocol.version since this only applies to the wire protocol among brokers? Others, The patch in KAFKA-1809 is almost ready. It would be good to wrap up the discussion on KIP2 soon. So, if you haven't looked at this KIP, please take a look and send your comments. Thanks, Jun On Mon, Jan 26, 2015 at 8:02 PM, Gwen Shapira gshap...@cloudera.com wrote: Hi Kafka Devs, While reviewing the patch for KAFKA-1809, we came across two questions that we are interested in hearing the community out on. 1. This patch changes the Broker class and adds a new class BrokerEndPoint that behaves like the previous broker. While technically kafka.cluster.Broker is not part of the public API, it is returned by javaapi, used with the SimpleConsumer. Getting replicas from PartitionMetadata will now return BrokerEndPoint instead of Broker. All method calls remain the same, but since we return a new type, we break the API. Note that this breakage does not prevent upgrades - existing SimpleConsumers will continue working (because we are wire-compatible). The only thing that won't work is building SimpleConsumers with dependency on Kafka versions higher than 0.8.2. Arguably, we don't want anyone to do it anyway :) So: Do we state that the highest release on which SimpleConsumers can depend is 0.8.2? Or shall we keep Broker as is and create an UberBroker which will contain multiple brokers as its endpoints? 2. The KIP suggests use.new.wire.protocol configuration to decide which protocols the brokers will use to talk to each other. The problem is that after the next upgrade, the wire protocol is no longer new, so we'll have to reset it to false for the following upgrade, then change to true again... and upgrading more than a single version will be impossible. Bad idea :) As an alternative, we can have a property for each version and set one of them to true. Or (simple, I think) have wire.protocol.version property and accept version numbers (0.8.2, 0.8.3, 0.9) as values. Please share your thoughts :) Gwen
Re: [DISCUSSION] KIP-2: Refactor Brokers to Allow Multiple Endpoints
Added Jun's notes to the KIP (Thanks for explaining so clearly, Jun. I was clearly struggling with this...) and removed the reference to use.new.wire.protocol. On Wed, Feb 11, 2015 at 4:19 PM, Joel Koshy jjkosh...@gmail.com wrote: The description that Jun gave for (2) was the detail I was looking for - Gwen can you update the KIP with that for completeness/clarity? I'm +1 as well overall. However, I think it would be good if we also get an ack from someone who is more experienced on the operations side (say, Todd) to review especially the upgrade plan. On Wed, Feb 11, 2015 at 09:40:50AM -0800, Jun Rao wrote: +1 for proposed changes in 1 and 2. 1. The impact is that if someone uses SimpleConsumer and references Broker explicitly, the application needs code change to compile with 0.8.3. Since SimpleConsumer is not widely used, breaking the API in SimpleConsumer but maintaining overall code cleanness seems to be a better tradeoff. 2. For clarification, the issue is the following. In 0.8.3, we will be evolving the wire protocol of UpdateMedataRequest (to send info about endpoints for different security protocols). Since this is used in intra-cluster communication, we need to do the upgrade in two steps. The idea is that in 0.8.3, we will default wire.protocol.version to 0.8.2. When upgrading to 0.8.3, in step 1, we do a rolling upgrade to 0.8.3. After step 1, all brokers will be capable for processing the new protocol in 0.8.3, but without actually using it. In step 2, we configure wire.protocol.version to 0.8.3 in each broker and do another rolling restart. After step 2, all brokers will start using the new protocol in 0.8.3. Let's say that in the next release 0.9, we are changing the intra-cluster wire protocol again. We will do the similar thing: defaulting wire.protocol.version to 0.8.3 in 0.9 so that people can upgrade from 0.8.3 to 0.9 in two steps. For people who want to upgrade from 0.8.2 to 0.9 directly, they will have to configure wire.protocol.version to 0.8.2 first and then do the two-step upgrade to 0.9. Gwen, In KIP2, there is still a reference to use.new.protocol. This needs to be removed. Also, would it be better to use intra.cluster.wire.protocol.version since this only applies to the wire protocol among brokers? Others, The patch in KAFKA-1809 is almost ready. It would be good to wrap up the discussion on KIP2 soon. So, if you haven't looked at this KIP, please take a look and send your comments. Thanks, Jun On Mon, Jan 26, 2015 at 8:02 PM, Gwen Shapira gshap...@cloudera.com wrote: Hi Kafka Devs, While reviewing the patch for KAFKA-1809, we came across two questions that we are interested in hearing the community out on. 1. This patch changes the Broker class and adds a new class BrokerEndPoint that behaves like the previous broker. While technically kafka.cluster.Broker is not part of the public API, it is returned by javaapi, used with the SimpleConsumer. Getting replicas from PartitionMetadata will now return BrokerEndPoint instead of Broker. All method calls remain the same, but since we return a new type, we break the API. Note that this breakage does not prevent upgrades - existing SimpleConsumers will continue working (because we are wire-compatible). The only thing that won't work is building SimpleConsumers with dependency on Kafka versions higher than 0.8.2. Arguably, we don't want anyone to do it anyway :) So: Do we state that the highest release on which SimpleConsumers can depend is 0.8.2? Or shall we keep Broker as is and create an UberBroker which will contain multiple brokers as its endpoints? 2. The KIP suggests use.new.wire.protocol configuration to decide which protocols the brokers will use to talk to each other. The problem is that after the next upgrade, the wire protocol is no longer new, so we'll have to reset it to false for the following upgrade, then change to true again... and upgrading more than a single version will be impossible. Bad idea :) As an alternative, we can have a property for each version and set one of them to true. Or (simple, I think) have wire.protocol.version property and accept version numbers (0.8.2, 0.8.3, 0.9) as values. Please share your thoughts :) Gwen
Re: [DISCUSSION] KIP-2: Refactor Brokers to Allow Multiple Endpoints
Jun, I'm not sure we should default wire.protocol.version to the previous version. This will make fresh installs a bit weird :) I think we should default to the new version and assume that when I'm upgrading a broker, I'm re-using an existing configuration file. This way, if I'm upgrading 0.8.3.0 to 0.9.0.0, the configuration file already says wire.protocol.version=0.8.3.0 and I need to bump it post upgrade. Fresh install will include 0.9.0.0, so I won't need to bump anything. The only exception is with 0.8.2.0, where I'll need to add wire.protocol.version=0.8.2.0 before upgrading to 0.8.3.0. Does that make sense? Regarding the naming, I agree that this parameter only controls the protocol between brokers (clients control the version of the protocol when they are involved, on a per-message basis). However, inter.broker.wire.protocol.version makes it sound like there may be other types of wire.protocol.version in the future, and I'm pretty sure we want a single parameter for controlling protocol versions from broker side. Not a big deal for me either way. On Wed, Feb 11, 2015 at 9:40 AM, Jun Rao j...@confluent.io wrote: +1 for proposed changes in 1 and 2. 1. The impact is that if someone uses SimpleConsumer and references Broker explicitly, the application needs code change to compile with 0.8.3. Since SimpleConsumer is not widely used, breaking the API in SimpleConsumer but maintaining overall code cleanness seems to be a better tradeoff. 2. For clarification, the issue is the following. In 0.8.3, we will be evolving the wire protocol of UpdateMedataRequest (to send info about endpoints for different security protocols). Since this is used in intra-cluster communication, we need to do the upgrade in two steps. The idea is that in 0.8.3, we will default wire.protocol.version to 0.8.2. When upgrading to 0.8.3, in step 1, we do a rolling upgrade to 0.8.3. After step 1, all brokers will be capable for processing the new protocol in 0.8.3, but without actually using it. In step 2, we configure wire.protocol.version to 0.8.3 in each broker and do another rolling restart. After step 2, all brokers will start using the new protocol in 0.8.3. Let's say that in the next release 0.9, we are changing the intra-cluster wire protocol again. We will do the similar thing: defaulting wire.protocol.version to 0.8.3 in 0.9 so that people can upgrade from 0.8.3 to 0.9 in two steps. For people who want to upgrade from 0.8.2 to 0.9 directly, they will have to configure wire.protocol.version to 0.8.2 first and then do the two-step upgrade to 0.9. Gwen, In KIP2, there is still a reference to use.new.protocol. This needs to be removed. Also, would it be better to use intra.cluster.wire.protocol.version since this only applies to the wire protocol among brokers? Others, The patch in KAFKA-1809 is almost ready. It would be good to wrap up the discussion on KIP2 soon. So, if you haven't looked at this KIP, please take a look and send your comments. Thanks, Jun On Mon, Jan 26, 2015 at 8:02 PM, Gwen Shapira gshap...@cloudera.com wrote: Hi Kafka Devs, While reviewing the patch for KAFKA-1809, we came across two questions that we are interested in hearing the community out on. 1. This patch changes the Broker class and adds a new class BrokerEndPoint that behaves like the previous broker. While technically kafka.cluster.Broker is not part of the public API, it is returned by javaapi, used with the SimpleConsumer. Getting replicas from PartitionMetadata will now return BrokerEndPoint instead of Broker. All method calls remain the same, but since we return a new type, we break the API. Note that this breakage does not prevent upgrades - existing SimpleConsumers will continue working (because we are wire-compatible). The only thing that won't work is building SimpleConsumers with dependency on Kafka versions higher than 0.8.2. Arguably, we don't want anyone to do it anyway :) So: Do we state that the highest release on which SimpleConsumers can depend is 0.8.2? Or shall we keep Broker as is and create an UberBroker which will contain multiple brokers as its endpoints? 2. The KIP suggests use.new.wire.protocol configuration to decide which protocols the brokers will use to talk to each other. The problem is that after the next upgrade, the wire protocol is no longer new, so we'll have to reset it to false for the following upgrade, then change to true again... and upgrading more than a single version will be impossible. Bad idea :) As an alternative, we can have a property for each version and set one of them to true. Or (simple, I think) have wire.protocol.version property and accept version numbers (0.8.2, 0.8.3, 0.9) as values. Please share your thoughts :) Gwen
Re: [DISCUSSION] KIP-2: Refactor Brokers to Allow Multiple Endpoints
Thanks, Gwen. This looks good to me as far as the wire protocol versioning goes. I agree with you on defaulting to the new wire protocol version for new installs. I think it will also need to be very clear (to general installer of Kafka, and not just developers) in documentation when the wire protocol version changes moving forwards, and what the risk/benefit of changing to the new version is. Since a rolling upgrade of the intra-cluster protocol is supported, will a rolling downgrade work as well? Should a flaw (bug, security, or otherwise) be discovered after upgrade, is it possible to change the wire.protocol.version back to 0.8.2 and do a rolling bounce? On the host/port/protocol specification, specifically the ZK config format, is it possible to have an un-advertised endpoint? I would see this as potentially useful if you wanted to have an endpoint that you are reserving for intra-cluster communication, and you would prefer to not have it advertised at all. Perhaps it is blocked by a firewall rule or other authentication method. This could also allow you to duplicate a security protocol type but segregate it on a different port or interface (if it is unadvertised, there is no ambiguity to the clients as to which endpoint should be selected). I believe I asked about that previously, and I didn't track what the final outcome was or even if it was discussed further. -Todd On Wed, Feb 11, 2015 at 4:38 PM, Gwen Shapira gshap...@cloudera.com wrote: Added Jun's notes to the KIP (Thanks for explaining so clearly, Jun. I was clearly struggling with this...) and removed the reference to use.new.wire.protocol. On Wed, Feb 11, 2015 at 4:19 PM, Joel Koshy jjkosh...@gmail.com wrote: The description that Jun gave for (2) was the detail I was looking for - Gwen can you update the KIP with that for completeness/clarity? I'm +1 as well overall. However, I think it would be good if we also get an ack from someone who is more experienced on the operations side (say, Todd) to review especially the upgrade plan. On Wed, Feb 11, 2015 at 09:40:50AM -0800, Jun Rao wrote: +1 for proposed changes in 1 and 2. 1. The impact is that if someone uses SimpleConsumer and references Broker explicitly, the application needs code change to compile with 0.8.3. Since SimpleConsumer is not widely used, breaking the API in SimpleConsumer but maintaining overall code cleanness seems to be a better tradeoff. 2. For clarification, the issue is the following. In 0.8.3, we will be evolving the wire protocol of UpdateMedataRequest (to send info about endpoints for different security protocols). Since this is used in intra-cluster communication, we need to do the upgrade in two steps. The idea is that in 0.8.3, we will default wire.protocol.version to 0.8.2. When upgrading to 0.8.3, in step 1, we do a rolling upgrade to 0.8.3. After step 1, all brokers will be capable for processing the new protocol in 0.8.3, but without actually using it. In step 2, we configure wire.protocol.version to 0.8.3 in each broker and do another rolling restart. After step 2, all brokers will start using the new protocol in 0.8.3. Let's say that in the next release 0.9, we are changing the intra-cluster wire protocol again. We will do the similar thing: defaulting wire.protocol.version to 0.8.3 in 0.9 so that people can upgrade from 0.8.3 to 0.9 in two steps. For people who want to upgrade from 0.8.2 to 0.9 directly, they will have to configure wire.protocol.version to 0.8.2 first and then do the two-step upgrade to 0.9. Gwen, In KIP2, there is still a reference to use.new.protocol. This needs to be removed. Also, would it be better to use intra.cluster.wire.protocol.version since this only applies to the wire protocol among brokers? Others, The patch in KAFKA-1809 is almost ready. It would be good to wrap up the discussion on KIP2 soon. So, if you haven't looked at this KIP, please take a look and send your comments. Thanks, Jun On Mon, Jan 26, 2015 at 8:02 PM, Gwen Shapira gshap...@cloudera.com wrote: Hi Kafka Devs, While reviewing the patch for KAFKA-1809, we came across two questions that we are interested in hearing the community out on. 1. This patch changes the Broker class and adds a new class BrokerEndPoint that behaves like the previous broker. While technically kafka.cluster.Broker is not part of the public API, it is returned by javaapi, used with the SimpleConsumer. Getting replicas from PartitionMetadata will now return BrokerEndPoint instead of Broker. All method calls remain the same, but since we return a new type, we break the API. Note that this breakage does not prevent upgrades - existing SimpleConsumers will continue working (because we are wire-compatible). The only thing that won't work is
Re: [DISCUSSION] KIP-2: Refactor Brokers to Allow Multiple Endpoints
On Mon, Feb 9, 2015 at 5:47 PM, Joel Koshy jjkosh...@gmail.com wrote: For (1) - +1 especially since the existing clients will keep working. For (2) - I'm less clear on the proposal. Can you incorporate it into the KIP and/or linked wiki? Added detail on wire.protocol.version to the KIP (under upgrade plan). Let me know if still unclear. Also, on the KIP itself, can you clarify what the TRACE protocol is? The RB has a brief comment (plan is to add instrumentation in the future) but I'm not sure what that means. Added details in KIP (I already started looking at an earlier version of your RB couple of days ago, but did not finish. I'll look at your latest RB.) Thank you! Thanks, Joel On Wed, Jan 28, 2015 at 10:28:39AM -0800, Gwen Shapira wrote: Bumping :) If there are no objections, I'd like to go with the following: 1. Do not support javaapi (SimpleConsumer) with dependency on versions higher than 0.8.2. Existing clients will keep working. 2. The configuration parameter for upgrades will be inter.broker.protocol.version={0.8.2.0, 0.8.3.0, 0.9.0.0...} Gwen On Mon, Jan 26, 2015 at 8:02 PM, Gwen Shapira gshap...@cloudera.com wrote: Hi Kafka Devs, While reviewing the patch for KAFKA-1809, we came across two questions that we are interested in hearing the community out on. 1. This patch changes the Broker class and adds a new class BrokerEndPoint that behaves like the previous broker. While technically kafka.cluster.Broker is not part of the public API, it is returned by javaapi, used with the SimpleConsumer. Getting replicas from PartitionMetadata will now return BrokerEndPoint instead of Broker. All method calls remain the same, but since we return a new type, we break the API. Note that this breakage does not prevent upgrades - existing SimpleConsumers will continue working (because we are wire-compatible). The only thing that won't work is building SimpleConsumers with dependency on Kafka versions higher than 0.8.2. Arguably, we don't want anyone to do it anyway :) So: Do we state that the highest release on which SimpleConsumers can depend is 0.8.2? Or shall we keep Broker as is and create an UberBroker which will contain multiple brokers as its endpoints? 2. The KIP suggests use.new.wire.protocol configuration to decide which protocols the brokers will use to talk to each other. The problem is that after the next upgrade, the wire protocol is no longer new, so we'll have to reset it to false for the following upgrade, then change to true again... and upgrading more than a single version will be impossible. Bad idea :) As an alternative, we can have a property for each version and set one of them to true. Or (simple, I think) have wire.protocol.version property and accept version numbers (0.8.2, 0.8.3, 0.9) as values. Please share your thoughts :) Gwen
Re: [DISCUSSION] KIP-2: Refactor Brokers to Allow Multiple Endpoints
For (1) - +1 especially since the existing clients will keep working. For (2) - I'm less clear on the proposal. Can you incorporate it into the KIP and/or linked wiki? Also, on the KIP itself, can you clarify what the TRACE protocol is? The RB has a brief comment (plan is to add instrumentation in the future) but I'm not sure what that means. (I already started looking at an earlier version of your RB couple of days ago, but did not finish. I'll look at your latest RB.) Thanks, Joel On Wed, Jan 28, 2015 at 10:28:39AM -0800, Gwen Shapira wrote: Bumping :) If there are no objections, I'd like to go with the following: 1. Do not support javaapi (SimpleConsumer) with dependency on versions higher than 0.8.2. Existing clients will keep working. 2. The configuration parameter for upgrades will be inter.broker.protocol.version={0.8.2.0, 0.8.3.0, 0.9.0.0...} Gwen On Mon, Jan 26, 2015 at 8:02 PM, Gwen Shapira gshap...@cloudera.com wrote: Hi Kafka Devs, While reviewing the patch for KAFKA-1809, we came across two questions that we are interested in hearing the community out on. 1. This patch changes the Broker class and adds a new class BrokerEndPoint that behaves like the previous broker. While technically kafka.cluster.Broker is not part of the public API, it is returned by javaapi, used with the SimpleConsumer. Getting replicas from PartitionMetadata will now return BrokerEndPoint instead of Broker. All method calls remain the same, but since we return a new type, we break the API. Note that this breakage does not prevent upgrades - existing SimpleConsumers will continue working (because we are wire-compatible). The only thing that won't work is building SimpleConsumers with dependency on Kafka versions higher than 0.8.2. Arguably, we don't want anyone to do it anyway :) So: Do we state that the highest release on which SimpleConsumers can depend is 0.8.2? Or shall we keep Broker as is and create an UberBroker which will contain multiple brokers as its endpoints? 2. The KIP suggests use.new.wire.protocol configuration to decide which protocols the brokers will use to talk to each other. The problem is that after the next upgrade, the wire protocol is no longer new, so we'll have to reset it to false for the following upgrade, then change to true again... and upgrading more than a single version will be impossible. Bad idea :) As an alternative, we can have a property for each version and set one of them to true. Or (simple, I think) have wire.protocol.version property and accept version numbers (0.8.2, 0.8.3, 0.9) as values. Please share your thoughts :) Gwen
Re: [DISCUSSION] KIP-2: Refactor Brokers to Allow Multiple Endpoints
Bumping :) If there are no objections, I'd like to go with the following: 1. Do not support javaapi (SimpleConsumer) with dependency on versions higher than 0.8.2. Existing clients will keep working. 2. The configuration parameter for upgrades will be inter.broker.protocol.version={0.8.2.0, 0.8.3.0, 0.9.0.0...} Gwen On Mon, Jan 26, 2015 at 8:02 PM, Gwen Shapira gshap...@cloudera.com wrote: Hi Kafka Devs, While reviewing the patch for KAFKA-1809, we came across two questions that we are interested in hearing the community out on. 1. This patch changes the Broker class and adds a new class BrokerEndPoint that behaves like the previous broker. While technically kafka.cluster.Broker is not part of the public API, it is returned by javaapi, used with the SimpleConsumer. Getting replicas from PartitionMetadata will now return BrokerEndPoint instead of Broker. All method calls remain the same, but since we return a new type, we break the API. Note that this breakage does not prevent upgrades - existing SimpleConsumers will continue working (because we are wire-compatible). The only thing that won't work is building SimpleConsumers with dependency on Kafka versions higher than 0.8.2. Arguably, we don't want anyone to do it anyway :) So: Do we state that the highest release on which SimpleConsumers can depend is 0.8.2? Or shall we keep Broker as is and create an UberBroker which will contain multiple brokers as its endpoints? 2. The KIP suggests use.new.wire.protocol configuration to decide which protocols the brokers will use to talk to each other. The problem is that after the next upgrade, the wire protocol is no longer new, so we'll have to reset it to false for the following upgrade, then change to true again... and upgrading more than a single version will be impossible. Bad idea :) As an alternative, we can have a property for each version and set one of them to true. Or (simple, I think) have wire.protocol.version property and accept version numbers (0.8.2, 0.8.3, 0.9) as values. Please share your thoughts :) Gwen
[DISCUSSION] KIP-2: Refactor Brokers to Allow Multiple Endpoints
Hi Kafka Devs, While reviewing the patch for KAFKA-1809, we came across two questions that we are interested in hearing the community out on. 1. This patch changes the Broker class and adds a new class BrokerEndPoint that behaves like the previous broker. While technically kafka.cluster.Broker is not part of the public API, it is returned by javaapi, used with the SimpleConsumer. Getting replicas from PartitionMetadata will now return BrokerEndPoint instead of Broker. All method calls remain the same, but since we return a new type, we break the API. Note that this breakage does not prevent upgrades - existing SimpleConsumers will continue working (because we are wire-compatible). The only thing that won't work is building SimpleConsumers with dependency on Kafka versions higher than 0.8.2. Arguably, we don't want anyone to do it anyway :) So: Do we state that the highest release on which SimpleConsumers can depend is 0.8.2? Or shall we keep Broker as is and create an UberBroker which will contain multiple brokers as its endpoints? 2. The KIP suggests use.new.wire.protocol configuration to decide which protocols the brokers will use to talk to each other. The problem is that after the next upgrade, the wire protocol is no longer new, so we'll have to reset it to false for the following upgrade, then change to true again... and upgrading more than a single version will be impossible. Bad idea :) As an alternative, we can have a property for each version and set one of them to true. Or (simple, I think) have wire.protocol.version property and accept version numbers (0.8.2, 0.8.3, 0.9) as values. Please share your thoughts :) Gwen