Hi, Boyang, Thanks for the reply.
For the metric, we just need to define a metric of meter type and of name NumRequestsForwardingToControllerPerSec. Meter exposes a few standard JMX attributes including an accumulated total and rates ( https://metrics.dropwizard.io/2.2.0/apidocs/com/yammer/metrics/core/Meter.html ). Jun On Wed, Jun 10, 2020 at 10:38 AM Boyang Chen <reluctanthero...@gmail.com> wrote: > Thanks Jun for the suggestions! I have addressed suggestion and simplified > the metrics part. > > On Tue, Jun 9, 2020 at 5:46 PM Jun Rao <j...@confluent.io> wrote: > > > Hi, Boyang, > > > > Thanks for the KIP. Just a few comments on the metrics. > > > > 1. It would be useful to document the full JMX metric names (package, > type, > > etc) of the new metrics. Also, for rates on the server side, we > > typically use Yammer Meter. > > > > Sounds good. > > 2. For num-messages-redirected-rate, would num-requests-redirected-rate be > > better? > > > > Actually for the scope of this KIP, we are no longer needing to have a > separate tracking > of forwarded request rate, because the Envelope RPC is dropped. > > > > 3. num-client-forwarding-to-controller-rate: Is that based on clientId, > > client IP, client request version or sth else? How do you plan to > implement > > that since it seems to require tracking the current unique client set > > somehow. An alternative approach is to maintain a > > num-requests-redirected-rate metric with a client tag. > > > The clientId tag approach makes sense, will add to the KIP. > > Jun > > > > > > > > On Mon, Jun 8, 2020 at 9:36 AM Boyang Chen <reluctanthero...@gmail.com> > > wrote: > > > > > Hey there, > > > > > > If no more question is raised, I will go ahead and start the vote > > shortly. > > > > > > On Thu, Jun 4, 2020 at 12:39 PM Boyang Chen < > reluctanthero...@gmail.com> > > > wrote: > > > > > > > Hey there, > > > > > > > > bumping this thread for any further KIP-590 discussion, since it's > been > > > > quiet for a while. > > > > > > > > Boyang > > > > > > > > On Thu, May 21, 2020 at 10:31 AM Boyang Chen < > > reluctanthero...@gmail.com > > > > > > > > wrote: > > > > > > > >> Thanks David, I agree the wording here is not clear, and the fellow > > > >> broker should just send a new CreateTopicRequest in this case. > > > >> > > > >> In the meantime, we had some offline discussion about the Envelope > API > > > as > > > >> well. Although it provides certain privileges like data embedding > and > > > >> principal embedding, it creates a security hole by letting a > malicious > > > user > > > >> impersonate any forwarding broker, thus pretending to be any admin > > user. > > > >> Passing the principal around also increases the vulnerability, > > compared > > > >> with other standard ways such as passing a verified token, but it is > > > >> unfortunately not fully supported with Kafka security. > > > >> > > > >> So for the security concerns, we are abandoning the Envelope > approach > > > and > > > >> fallback to just forward the raw admin requests. The authentication > > will > > > >> happen on the receiving broker side instead of on the controller, so > > > that > > > >> we are able to stripped off the principal fields and only include > the > > > >> principal in header as optional field for audit logging purpose. > > > >> Furthermore, we shall propose adding a separate endpoint for > > > >> broker-controller communication which should be recommended to > enable > > > >> secure connections so that a malicious client could not pretend to > be > > a > > > >> broker and perform impersonation. > > > >> > > > >> Let me know your thoughts. > > > >> > > > >> Best, > > > >> Boyang > > > >> > > > >> On Tue, May 19, 2020 at 12:17 AM David Jacot <dja...@confluent.io> > > > wrote: > > > >> > > > >>> Hi Boyang, > > > >>> > > > >>> I've got another question regarding the auto topic creation case. > The > > > KIP > > > >>> says: "Currently the target broker shall just utilize its own ZK > > client > > > >>> to > > > >>> create > > > >>> internal topics, which is disallowed in the bridge release. For > above > > > >>> scenarios, > > > >>> non-controller broker shall just forward a CreateTopicRequest to > the > > > >>> controller > > > >>> instead and let controller take care of the rest, while waiting for > > the > > > >>> response > > > >>> in the meantime." There will be no request to forward in this case, > > > >>> right? > > > >>> Instead, > > > >>> a CreateTopicsRequest is created and sent to the controller node. > > > >>> > > > >>> When the CreateTopicsRequest is sent as a side effect of the > > > >>> MetadataRequest, > > > >>> it would be good to know the principal and the clientId in the > > > controller > > > >>> (quota, > > > >>> audit, etc.). Do you plan to use the Envelope API for this case as > > well > > > >>> or > > > >>> to call > > > >>> the regular API directly? Another was to phrase it would be: Shall > > the > > > >>> internal > > > >>> CreateTopicsRequest be sent with the original metadata (principal, > > > >>> clientId, etc.) > > > >>> of the MetadataRequest or as an admin request? > > > >>> > > > >>> Best, > > > >>> David > > > >>> > > > >>> On Fri, May 8, 2020 at 2:04 AM Guozhang Wang <wangg...@gmail.com> > > > wrote: > > > >>> > > > >>> > Just to adds a bit more FYI here related to the last question > from > > > >>> David: > > > >>> > in KIP-595 while implementing the new requests we are also > adding a > > > >>> > "KafkaNetworkChannel" which is used for brokers to send vote / > > fetch > > > >>> > records, so besides the discussion on listeners I think > > > implementation > > > >>> wise > > > >>> > we can also consider consolidating a lot of those into the same > > > >>> call-trace > > > >>> > as well -- of course this is not related to public APIs so maybe > > just > > > >>> needs > > > >>> > to be coordinated among developments: > > > >>> > > > > >>> > 1. Broker -> Controller: ISR Change, Topic Creation, Admin > Redirect > > > >>> > (KIP-497). > > > >>> > 2. Controller -> Broker: LeaderAndISR / MetadataUpdate; though > > these > > > >>> are > > > >>> > likely going to be deprecated post KIP-500. > > > >>> > 3. Txn Coordinator -> Broker: TxnMarker > > > >>> > > > > >>> > > > > >>> > Guozhang > > > >>> > > > > >>> > On Wed, May 6, 2020 at 8:58 PM Boyang Chen < > > > reluctanthero...@gmail.com > > > >>> > > > > >>> > wrote: > > > >>> > > > > >>> > > Hey David, > > > >>> > > > > > >>> > > thanks for the feedbacks! > > > >>> > > > > > >>> > > On Wed, May 6, 2020 at 2:06 AM David Jacot < > dja...@confluent.io> > > > >>> wrote: > > > >>> > > > > > >>> > > > Hi Boyang, > > > >>> > > > > > > >>> > > > While re-reading the KIP, I've got few small > > questions/comments: > > > >>> > > > > > > >>> > > > 1. When auto topic creation is enabled, brokers will send a > > > >>> > > > CreateTopicRequest > > > >>> > > > to the controller instead of writing to ZK directly. It means > > > that > > > >>> > > > creation of these > > > >>> > > > topics are subject to be rejected with an error if a > > > >>> CreateTopicPolicy > > > >>> > is > > > >>> > > > used. Today, > > > >>> > > > it bypasses the policy entirely. I suppose that clusters > > allowing > > > >>> auto > > > >>> > > > topic creation > > > >>> > > > don't have a policy in place so it is not a big deal. I > suggest > > > to > > > >>> call > > > >>> > > > out explicitly the > > > >>> > > > limitation in the KIP though. > > > >>> > > > > > > >>> > > > That's a good idea, will add to the KIP. > > > >>> > > > > > >>> > > > > > >>> > > > 2. In the same vein as my first point. How do you plan to > > handle > > > >>> errors > > > >>> > > > when internal > > > >>> > > > topics are created by a broker? Do you plan to retry > retryable > > > >>> errors > > > >>> > > > indefinitely? > > > >>> > > > > > > >>> > > > I checked a bit on the admin client handling of the create > > topic > > > >>> RPC. > > > >>> > It > > > >>> > > seems that > > > >>> > > the only retriable exceptions at the moment are NOT_CONTROLLER > > and > > > >>> > > REQUEST_TIMEOUT. > > > >>> > > So I guess we just need to retry on these exceptions? > > > >>> > > > > > >>> > > > > > >>> > > > 3. Could you clarify which listener will be used for the > > internal > > > >>> > > requests? > > > >>> > > > Do you plan > > > >>> > > > to use the control plane listener or perhaps the inter-broker > > > >>> listener? > > > >>> > > > > > > >>> > > > As we discussed in the KIP, currently the internal design for > > > >>> > > broker->controller channel has not been > > > >>> > > done yet, and I feel it makes sense to consolidate redirect RPC > > and > > > >>> > > internal topic creation RPC to this future channel, > > > >>> > > which are details to be filled in the near future, right now > some > > > >>> > > controller refactoring effort is still WIP. > > > >>> > > > > > >>> > > > > > >>> > > > Thanks, > > > >>> > > > David > > > >>> > > > > > > >>> > > > On Mon, May 4, 2020 at 9:37 AM Sönke Liebau > > > >>> > > > <soenke.lie...@opencore.com.invalid> wrote: > > > >>> > > > > > > >>> > > > > Ah, I see, thanks for the clarification! > > > >>> > > > > > > > >>> > > > > Shouldn't be an issue I think. My understanding of KIPs was > > > >>> always > > > >>> > that > > > >>> > > > > they are mostly intended as a place to discuss and agree > > > changes > > > >>> up > > > >>> > > > front, > > > >>> > > > > whereas tracking the actual releases that things go into > > should > > > >>> be > > > >>> > > > handled > > > >>> > > > > in Jira. > > > >>> > > > > So maybe we just create new jiras for any subsequent work > and > > > >>> either > > > >>> > > link > > > >>> > > > > those or make them subtasks (even though this jira is > > already a > > > >>> > subtask > > > >>> > > > > itself), that should allow us to properly track all > releases > > > that > > > >>> > work > > > >>> > > > goes > > > >>> > > > > into. > > > >>> > > > > > > > >>> > > > > Thanks for your work on this!! > > > >>> > > > > > > > >>> > > > > Best, > > > >>> > > > > Sönke > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > On Sat, 2 May 2020 at 00:31, Boyang Chen < > > > >>> reluctanthero...@gmail.com > > > >>> > > > > > >>> > > > > wrote: > > > >>> > > > > > > > >>> > > > > > Sure thing Sonke, what I suggest is that usual KIPs get > > > >>> accepted to > > > >>> > > go > > > >>> > > > > into > > > >>> > > > > > next release. It could span for a couple of releases > > because > > > of > > > >>> > > > > engineering > > > >>> > > > > > time, but no change has to be shipped in specific future > > > >>> releases, > > > >>> > > like > > > >>> > > > > the > > > >>> > > > > > backward incompatible change for KafkaPrincipal. But I > > guess > > > >>> it's > > > >>> > not > > > >>> > > > > > really a blocker, as long as we stated clearly in the KIP > > how > > > >>> we > > > >>> > are > > > >>> > > > > going > > > >>> > > > > > to roll things out, and let it partially finish in 2.6. > > > >>> > > > > > > > > >>> > > > > > Boyang > > > >>> > > > > > > > > >>> > > > > > On Fri, May 1, 2020 at 2:32 PM Sönke Liebau > > > >>> > > > > > <soenke.lie...@opencore.com.invalid> wrote: > > > >>> > > > > > > > > >>> > > > > > > Hi Boyang, > > > >>> > > > > > > > > > >>> > > > > > > thanks for the update, sounds reasonable to me. Making > > it a > > > >>> > > breaking > > > >>> > > > > > change > > > >>> > > > > > > is definitely the safer route to go. > > > >>> > > > > > > > > > >>> > > > > > > Just one quick question regarding your mail, I didn't > > fully > > > >>> > > > understand > > > >>> > > > > > what > > > >>> > > > > > > you mean by "I think this is the first time we need to > > > >>> introduce > > > >>> > a > > > >>> > > > KIP > > > >>> > > > > > > without having it > > > >>> > > > > > > fully accepted in next release." - could you perhaps > > > explain > > > >>> > that > > > >>> > > > some > > > >>> > > > > > > more very briefly? > > > >>> > > > > > > > > > >>> > > > > > > Best regards, > > > >>> > > > > > > Sönke > > > >>> > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > On Fri, 1 May 2020 at 23:03, Boyang Chen < > > > >>> > > reluctanthero...@gmail.com > > > >>> > > > > > > > >>> > > > > > > wrote: > > > >>> > > > > > > > > > >>> > > > > > > > Hey Tom, > > > >>> > > > > > > > > > > >>> > > > > > > > thanks for the suggestion. As long as we could > > correctly > > > >>> > > serialize > > > >>> > > > > the > > > >>> > > > > > > > principal and embed in the Envelope, I think we could > > > still > > > >>> > > > leverage > > > >>> > > > > > the > > > >>> > > > > > > > controller to do the client request authentication. > > > >>> Although > > > >>> > this > > > >>> > > > > pays > > > >>> > > > > > an > > > >>> > > > > > > > extra round trip if the authorization is doomed to > fail > > > on > > > >>> the > > > >>> > > > > receiver > > > >>> > > > > > > > side, having a centralized processing unit is more > > > >>> favorable > > > >>> > such > > > >>> > > > as > > > >>> > > > > > > > ensuring the audit log is consistent instead of > > > scattering > > > >>> > > between > > > >>> > > > > > > > forwarder and receiver. > > > >>> > > > > > > > > > > >>> > > > > > > > Boyang > > > >>> > > > > > > > > > > >>> > > > > > > > On Wed, Apr 29, 2020 at 9:50 AM Tom Bentley < > > > >>> > tbent...@redhat.com > > > >>> > > > > > > >>> > > > > > wrote: > > > >>> > > > > > > > > > > >>> > > > > > > > > Hi Boyang, > > > >>> > > > > > > > > > > > >>> > > > > > > > > Thanks for the update. In the EnvelopeRequest > > handling > > > >>> > section > > > >>> > > of > > > >>> > > > > the > > > >>> > > > > > > KIP > > > >>> > > > > > > > > it might be worth saying explicitly that > > authorization > > > >>> of the > > > >>> > > > > request > > > >>> > > > > > > > will > > > >>> > > > > > > > > happen as normal. Otherwise what you're proposing > > makes > > > >>> sense > > > >>> > > to > > > >>> > > > > me. > > > >>> > > > > > > > > > > > >>> > > > > > > > > Thanks again, > > > >>> > > > > > > > > > > > >>> > > > > > > > > Tom > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > On Wed, Apr 29, 2020 at 5:27 PM Boyang Chen < > > > >>> > > > > > > reluctanthero...@gmail.com> > > > >>> > > > > > > > > wrote: > > > >>> > > > > > > > > > > > >>> > > > > > > > > > Thanks for the proposed idea Sonke. I reviewed it > > and > > > >>> had > > > >>> > > some > > > >>> > > > > > > offline > > > >>> > > > > > > > > > discussion with Colin, Rajini and Mathew. > > > >>> > > > > > > > > > > > > >>> > > > > > > > > > We do need to add serializability to the > > > >>> PrincipalBuilder > > > >>> > > > > > interface, > > > >>> > > > > > > > but > > > >>> > > > > > > > > we > > > >>> > > > > > > > > > should not make any default implementation which > > > could > > > >>> go > > > >>> > > wrong > > > >>> > > > > and > > > >>> > > > > > > > messy > > > >>> > > > > > > > > > up with the security in a production environment > if > > > the > > > >>> > user > > > >>> > > > > > neglects > > > >>> > > > > > > > it. > > > >>> > > > > > > > > > Instead we need to make it required and backward > > > >>> > > incompatible. > > > >>> > > > > So I > > > >>> > > > > > > > > > integrated your proposed methods and expand the > > > >>> Envelope > > > >>> > RPC > > > >>> > > > > with a > > > >>> > > > > > > > > couple > > > >>> > > > > > > > > > of more fields for audit log purpose as well. > > > >>> > > > > > > > > > > > > >>> > > > > > > > > > Since the KafkaPrincipal builder serializability > > is a > > > >>> > binary > > > >>> > > > > > > > incompatible > > > >>> > > > > > > > > > change, I propose (also stated in the KIP) the > > > >>> following > > > >>> > > > > > > implementation > > > >>> > > > > > > > > > plan: > > > >>> > > > > > > > > > > > > >>> > > > > > > > > > 1. For next *2.x* release: > > > >>> > > > > > > > > > 1. Get new admin client forwarding changes > > > >>> > > > > > > > > > 2. Get the Envelope RPC implementation > > > >>> > > > > > > > > > 3. Get the forwarding path working and > > validate > > > >>> the > > > >>> > > > > function > > > >>> > > > > > > with > > > >>> > > > > > > > > > fake principals in testing environment, > > without > > > >>> > actual > > > >>> > > > > > > triggering > > > >>> > > > > > > > > in > > > >>> > > > > > > > > > the > > > >>> > > > > > > > > > production system > > > >>> > > > > > > > > > 2. For next *3.0 *release: > > > >>> > > > > > > > > > 1. Introduce serializability to > > > PrincipalBuilder > > > >>> > > > > > > > > > 2. Turn on forwarding path in production > and > > > >>> perform > > > >>> > > > > > end-to-end > > > >>> > > > > > > > > > testing > > > >>> > > > > > > > > > > > > >>> > > > > > > > > > > > > >>> > > > > > > > > > I think this is the first time we need to > > introduce a > > > >>> KIP > > > >>> > > > without > > > >>> > > > > > > > having > > > >>> > > > > > > > > it > > > >>> > > > > > > > > > fully accepted in next release. Let me know if > this > > > >>> sounds > > > >>> > > > > > > reasonable. > > > >>> > > > > > > > > > > > > >>> > > > > > > > > > On Fri, Apr 24, 2020 at 1:00 AM Sönke Liebau > > > >>> > > > > > > > > > <soenke.lie...@opencore.com.invalid> wrote: > > > >>> > > > > > > > > > > > > >>> > > > > > > > > > > After thinking on this a little bit, maybe this > > > >>> would be > > > >>> > an > > > >>> > > > > > option: > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > > add default methods serialize and deserialize > to > > > the > > > >>> > > > > > > > > > KafkaPrincipalBuilder > > > >>> > > > > > > > > > > interface, these could be very short: > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > > default String serialize(KafkaPrincipal > > principal) > > > { > > > >>> > > > > > > > > > > return principal.toString(); > > > >>> > > > > > > > > > > } > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > > default KafkaPrincipal deserialize(String > > > >>> > principalString) > > > >>> > > { > > > >>> > > > > > > > > > > return > > > >>> > > > SecurityUtils.parseKafkaPrincipal(principalString); > > > >>> > > > > > > > > > > } > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > > This would mean that all existing > implementations > > > of > > > >>> that > > > >>> > > > > > interface > > > >>> > > > > > > > > > > are unaffected, as this code is pretty much > what > > is > > > >>> > > currently > > > >>> > > > > > being > > > >>> > > > > > > > > > > used when their principals need to be > serialized. > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > > But it offers people using custom principals > the > > > >>> chance > > > >>> > to > > > >>> > > > > > override > > > >>> > > > > > > > > > > these methods and ensure that all information > > gets > > > >>> > > serialized > > > >>> > > > > for > > > >>> > > > > > > > > > > delegation tokens or request forwarding. > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > > Wherever we need to de/serialize principals > (for > > > >>> example > > > >>> > in > > > >>> > > > the > > > >>> > > > > > > > > > > DelegationTokenManager [1]) we obtain an > instance > > > of > > > >>> the > > > >>> > > > > > configured > > > >>> > > > > > > > > > > PrincipalBuilder class and use that to do the > > > actual > > > >>> > work. > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > > What do you think? > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > > Best regards, > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > > Sönke > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > > [1] > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > > > > https://github.com/apache/kafka/blob/33d06082117d971cdcddd4f01392006b543f3c01/core/src/main/scala/kafka/server/DelegationTokenManager.scala#L122 > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > > On Thu, 23 Apr 2020 at 17:42, Boyang Chen < > > > >>> > > > > > > > reluctanthero...@gmail.com> > > > >>> > > > > > > > > > > wrote: > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > Thanks all, > > > >>> > > > > > > > > > > > > > > >>> > > > > > > > > > > > IIUC, the necessity of doing the audit log on > > the > > > >>> > > > controller > > > >>> > > > > > side > > > >>> > > > > > > > is > > > >>> > > > > > > > > > > > because we need to make sure the authorized > > > >>> resource > > > >>> > > > > > > modifications > > > >>> > > > > > > > > > > > eventually arrive on the target broker side, > > but > > > is > > > >>> > that > > > >>> > > > > really > > > >>> > > > > > > > > > > necessary? > > > >>> > > > > > > > > > > > > > > >>> > > > > > > > > > > > I'm thinking the possibility of doing the > audit > > > >>> log on > > > >>> > > the > > > >>> > > > > > > > forwarding > > > >>> > > > > > > > > > > > broker side, which could simplify the > > discussion > > > of > > > >>> > > > principal > > > >>> > > > > > > > > > > serialization > > > >>> > > > > > > > > > > > here. The other option I could think of is to > > > >>> serialize > > > >>> > > the > > > >>> > > > > > > entire > > > >>> > > > > > > > > > audit > > > >>> > > > > > > > > > > > log message if we were supposed to approve, > and > > > >>> pass it > > > >>> > > as > > > >>> > > > > part > > > >>> > > > > > > of > > > >>> > > > > > > > > the > > > >>> > > > > > > > > > > > Envelope. > > > >>> > > > > > > > > > > > > > > >>> > > > > > > > > > > > Let me know if you think either of these > > > approaches > > > >>> > would > > > >>> > > > > work. > > > >>> > > > > > > > > > > > > > > >>> > > > > > > > > > > > On Thu, Apr 23, 2020 at 7:01 AM Sönke Liebau > > > >>> > > > > > > > > > > > <soenke.lie...@opencore.com.invalid> wrote: > > > >>> > > > > > > > > > > > > > > >>> > > > > > > > > > > > > I agree that this would be useful to have > and > > > >>> > shouldn't > > > >>> > > > > > create > > > >>> > > > > > > > > issues > > > >>> > > > > > > > > > > in > > > >>> > > > > > > > > > > > > 99% of all cases. But it would be a > breaking > > > >>> change > > > >>> > to > > > >>> > > a > > > >>> > > > > > public > > > >>> > > > > > > > > API. > > > >>> > > > > > > > > > > > > I had a quick look at the two large > projects > > > that > > > >>> > come > > > >>> > > to > > > >>> > > > > > mind > > > >>> > > > > > > > > which > > > >>> > > > > > > > > > > > might > > > >>> > > > > > > > > > > > > be affected: Ranger and Sentry - both seem > to > > > >>> operate > > > >>> > > > > > directly > > > >>> > > > > > > > with > > > >>> > > > > > > > > > > > > KafkaPrincipal instead of subclassing it. > But > > > >>> anybody > > > >>> > > who > > > >>> > > > > > > > > > > > > extended KafkaPrincipal would probably need > > to > > > >>> update > > > >>> > > > their > > > >>> > > > > > > > code.. > > > >>> > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > Writing this sparked the thought that this > > > issue > > > >>> > should > > > >>> > > > > also > > > >>> > > > > > > > > concern > > > >>> > > > > > > > > > > > > delegation tokens, as Principals need to be > > > >>> > stored/sent > > > >>> > > > > > around > > > >>> > > > > > > > for > > > >>> > > > > > > > > > > those > > > >>> > > > > > > > > > > > > too. > > > >>> > > > > > > > > > > > > Had a brief look at the code and for > > Delegation > > > >>> > Tokens > > > >>> > > we > > > >>> > > > > > seem > > > >>> > > > > > > to > > > >>> > > > > > > > > use > > > >>> > > > > > > > > > > > > SecurityUtils#parseKafkaPrincipal[1] which > > > would > > > >>> > ignore > > > >>> > > > any > > > >>> > > > > > > > > > additional > > > >>> > > > > > > > > > > > > information from custom Principals. > > > >>> > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > We'll probably want to at least add a note > on > > > >>> that to > > > >>> > > the > > > >>> > > > > > docs > > > >>> > > > > > > - > > > >>> > > > > > > > > > unless > > > >>> > > > > > > > > > > > it > > > >>> > > > > > > > > > > > > is there already, I've only looked for > about > > 30 > > > >>> > > seconds.. > > > >>> > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > Best regards, > > > >>> > > > > > > > > > > > > Sönke > > > >>> > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > [1] > > > >>> > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > > > > https://github.com/apache/kafka/blob/e9fcfe4fb7b9ae2f537ce355fe2ab51a58034c64/clients/src/main/java/org/apache/kafka/common/utils/SecurityUtils.java#L52 > > > >>> > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > On Thu, 23 Apr 2020 at 14:35, Colin McCabe > < > > > >>> > > > > > cmcc...@apache.org > > > >>> > > > > > > > > > > >>> > > > > > > > > > wrote: > > > >>> > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > Hmm... Maybe we need to add some way to > > > >>> serialize > > > >>> > and > > > >>> > > > > > > > deserialize > > > >>> > > > > > > > > > > > > > KafkaPrincipal subclasses to/from string. > > We > > > >>> could > > > >>> > > > add a > > > >>> > > > > > > > method > > > >>> > > > > > > > > to > > > >>> > > > > > > > > > > > > > KafkaPrincipalBuilder#deserialize and a > > > method > > > >>> > > > > > > > > > > > KafkaPrincipal#serialize, > > > >>> > > > > > > > > > > > > I > > > >>> > > > > > > > > > > > > > suppose. > > > >>> > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > best, > > > >>> > > > > > > > > > > > > > Colin > > > >>> > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > On Thu, Apr 23, 2020, at 02:14, Tom > Bentley > > > >>> wrote: > > > >>> > > > > > > > > > > > > > > Hi folks, > > > >>> > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > Colin wrote: > > > >>> > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > The final broker knows it can trust > the > > > >>> > principal > > > >>> > > > > name > > > >>> > > > > > in > > > >>> > > > > > > > the > > > >>> > > > > > > > > > > > > envelope > > > >>> > > > > > > > > > > > > > > > (since EnvelopeRequest requires > > > >>> CLUSTERACTION > > > >>> > on > > > >>> > > > > > > CLUSTER). > > > >>> > > > > > > > > So > > > >>> > > > > > > > > > it > > > >>> > > > > > > > > > > > can > > > >>> > > > > > > > > > > > > > use > > > >>> > > > > > > > > > > > > > > > that principal name for authorization > > > >>> (given > > > >>> > who > > > >>> > > > you > > > >>> > > > > > are, > > > >>> > > > > > > > > what > > > >>> > > > > > > > > > > can > > > >>> > > > > > > > > > > > > you > > > >>> > > > > > > > > > > > > > > > do?) The forwarded principal name > will > > > >>> also be > > > >>> > > > used > > > >>> > > > > > for > > > >>> > > > > > > > > > logging. > > > >>> > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > My understanding (and I'm happy to be > > > >>> corrected) > > > >>> > is > > > >>> > > > > that > > > >>> > > > > > a > > > >>> > > > > > > > > custom > > > >>> > > > > > > > > > > > > > > authoriser might rely on the > > KafkaPrincipal > > > >>> > > instance > > > >>> > > > > > being > > > >>> > > > > > > a > > > >>> > > > > > > > > > > subclass > > > >>> > > > > > > > > > > > > of > > > >>> > > > > > > > > > > > > > > KafkaPrincipal (e.g. the subclass has > > extra > > > >>> > fields > > > >>> > > > with > > > >>> > > > > > the > > > >>> > > > > > > > > > > > principal's > > > >>> > > > > > > > > > > > > > > "roles"). So you can't construct a > > > >>> KafkaPrinicpal > > > >>> > > on > > > >>> > > > > the > > > >>> > > > > > > > > > controller > > > >>> > > > > > > > > > > > > which > > > >>> > > > > > > > > > > > > > > would be guaranteed to work for > arbitrary > > > >>> > > > authorizers. > > > >>> > > > > > You > > > >>> > > > > > > > have > > > >>> > > > > > > > > > to > > > >>> > > > > > > > > > > > > > perform > > > >>> > > > > > > > > > > > > > > authorization on the first broker > > > (rejecting > > > >>> some > > > >>> > > of > > > >>> > > > > the > > > >>> > > > > > > > > batched > > > >>> > > > > > > > > > > > > > requests), > > > >>> > > > > > > > > > > > > > > forward the authorized ones to the > > > >>> controller, > > > >>> > > which > > > >>> > > > > then > > > >>> > > > > > > has > > > >>> > > > > > > > > to > > > >>> > > > > > > > > > > > trust > > > >>> > > > > > > > > > > > > > that > > > >>> > > > > > > > > > > > > > > the authorization has been done and > make > > > the > > > >>> ZK > > > >>> > > > writes > > > >>> > > > > > > > without > > > >>> > > > > > > > > > > > > > > authorization. Because the controller > > > cannot > > > >>> > invoke > > > >>> > > > the > > > >>> > > > > > > > > > authorizer > > > >>> > > > > > > > > > > > that > > > >>> > > > > > > > > > > > > > > means that the authorizer audit logging > > (on > > > >>> the > > > >>> > > > > > controller) > > > >>> > > > > > > > > would > > > >>> > > > > > > > > > > not > > > >>> > > > > > > > > > > > > > > include these operations. But they > would > > be > > > >>> in > > > >>> > the > > > >>> > > > > audit > > > >>> > > > > > > > > logging > > > >>> > > > > > > > > > > from > > > >>> > > > > > > > > > > > > the > > > >>> > > > > > > > > > > > > > > original broker, so the information > would > > > >>> not be > > > >>> > > > lost. > > > >>> > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > There's also a problem with using the > > > >>> constructed > > > >>> > > > > > principal > > > >>> > > > > > > > for > > > >>> > > > > > > > > > > other > > > >>> > > > > > > > > > > > > > > logging (i.e. non authorizer logging) > on > > > the > > > >>> > > > > controller: > > > >>> > > > > > > > > There's > > > >>> > > > > > > > > > > > > nothing > > > >>> > > > > > > > > > > > > > > stopping a custom KafkaPrincipal > subclass > > > >>> from > > > >>> > > > > overriding > > > >>> > > > > > > > > > > toString() > > > >>> > > > > > > > > > > > to > > > >>> > > > > > > > > > > > > > > return something different from > > > >>> > "${type}:${name}". > > > >>> > > If > > > >>> > > > > > > you're > > > >>> > > > > > > > > > > > building a > > > >>> > > > > > > > > > > > > > > "fake" KafkaPrincipal on the controller > > > from > > > >>> the > > > >>> > > type > > > >>> > > > > and > > > >>> > > > > > > > name > > > >>> > > > > > > > > > then > > > >>> > > > > > > > > > > > its > > > >>> > > > > > > > > > > > > > > toString() would be "wrong". A solution > > to > > > >>> this > > > >>> > > would > > > >>> > > > > be > > > >>> > > > > > to > > > >>> > > > > > > > > also > > > >>> > > > > > > > > > > > > > serialize > > > >>> > > > > > > > > > > > > > > the toString() into the envelope and > have > > > >>> some > > > >>> > > > > > > > > > > ProxiedKafkaPrincipal > > > >>> > > > > > > > > > > > > > class > > > >>> > > > > > > > > > > > > > > which you instantiate on the controller > > > >>> which has > > > >>> > > > > > > overridden > > > >>> > > > > > > > > > > toString > > > >>> > > > > > > > > > > > > to > > > >>> > > > > > > > > > > > > > > return the right value. Obviously you > > could > > > >>> > > optimize > > > >>> > > > > this > > > >>> > > > > > > > using > > > >>> > > > > > > > > > an > > > >>> > > > > > > > > > > > > > optional > > > >>> > > > > > > > > > > > > > > field so you only serialize the > > toString() > > > >>> if it > > > >>> > > > > differed > > > >>> > > > > > > > from > > > >>> > > > > > > > > > the > > > >>> > > > > > > > > > > > > value > > > >>> > > > > > > > > > > > > > > you'd get from > KafkaPrincipal.toString(). > > > >>> Maybe > > > >>> > > > > non-audit > > > >>> > > > > > > > > logging > > > >>> > > > > > > > > > > > using > > > >>> > > > > > > > > > > > > > the > > > >>> > > > > > > > > > > > > > > wrong string value of a principal is > > > >>> sufficiently > > > >>> > > > minor > > > >>> > > > > > > that > > > >>> > > > > > > > > this > > > >>> > > > > > > > > > > > isn't > > > >>> > > > > > > > > > > > > > > even worth trying to solve. > > > >>> > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > Kind regards, > > > >>> > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > Tom > > > >>> > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > On Wed, Apr 22, 2020 at 10:59 PM Sönke > > > Liebau > > > >>> > > > > > > > > > > > > > > <soenke.lie...@opencore.com.invalid> > > > wrote: > > > >>> > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > Hi Colin, > > > >>> > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > thanks for your summary! Just one > > > question > > > >>> - > > > >>> > and > > > >>> > > I > > > >>> > > > > may > > > >>> > > > > > be > > > >>> > > > > > > > > > missing > > > >>> > > > > > > > > > > > an > > > >>> > > > > > > > > > > > > > > > obvious point here.. > > > >>> > > > > > > > > > > > > > > > You write: > > > >>> > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > "The initial broker should do > > > >>> authentication > > > >>> > (who > > > >>> > > > are > > > >>> > > > > > > you?) > > > >>> > > > > > > > > and > > > >>> > > > > > > > > > > > come > > > >>> > > > > > > > > > > > > up > > > >>> > > > > > > > > > > > > > > > with a principal name. Then it > creates > > > an > > > >>> > > envelope > > > >>> > > > > > > > request, > > > >>> > > > > > > > > > > which > > > >>> > > > > > > > > > > > > will > > > >>> > > > > > > > > > > > > > > > contain that principal name, and > sends > > it > > > >>> along > > > >>> > > > with > > > >>> > > > > > the > > > >>> > > > > > > > > > > unmodified > > > >>> > > > > > > > > > > > > > > > original request to the final broker. > > > >>> [... ] > > > >>> > > The > > > >>> > > > > > final > > > >>> > > > > > > > > broker > > > >>> > > > > > > > > > > > knows > > > >>> > > > > > > > > > > > > > it > > > >>> > > > > > > > > > > > > > > > can trust the principal name in the > > > >>> envelope > > > >>> > > (since > > > >>> > > > > > > > > > > EnvelopeRequest > > > >>> > > > > > > > > > > > > > > > requires CLUSTERACTION on CLUSTER). > So > > > it > > > >>> can > > > >>> > > use > > > >>> > > > > that > > > >>> > > > > > > > > > principal > > > >>> > > > > > > > > > > > > name > > > >>> > > > > > > > > > > > > > for > > > >>> > > > > > > > > > > > > > > > authorization (given who you are, > what > > > can > > > >>> you > > > >>> > > > do?) " > > > >>> > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > My understanding is, that you don't > > want > > > to > > > >>> > > > serialize > > > >>> > > > > > the > > > >>> > > > > > > > > > > Principal > > > >>> > > > > > > > > > > > > > (due to > > > >>> > > > > > > > > > > > > > > > the discussed issues with custom > > > >>> principals) > > > >>> > but > > > >>> > > > > reduce > > > >>> > > > > > > the > > > >>> > > > > > > > > > > > principal > > > >>> > > > > > > > > > > > > > down > > > >>> > > > > > > > > > > > > > > > to a string representation that would > > be > > > >>> used > > > >>> > for > > > >>> > > > > > logging > > > >>> > > > > > > > and > > > >>> > > > > > > > > > > > > > > > authorization? > > > >>> > > > > > > > > > > > > > > > If that understanding is correct > then I > > > >>> don't > > > >>> > > think > > > >>> > > > > we > > > >>> > > > > > > > could > > > >>> > > > > > > > > > use > > > >>> > > > > > > > > > > > the > > > >>> > > > > > > > > > > > > > > > regular Authorizer on the target > > broker, > > > >>> > because > > > >>> > > > that > > > >>> > > > > > > would > > > >>> > > > > > > > > > need > > > >>> > > > > > > > > > > > the > > > >>> > > > > > > > > > > > > > actual > > > >>> > > > > > > > > > > > > > > > principal object to work on. > > > >>> > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > Also, a thought that just occurred to > > me, > > > >>> we > > > >>> > > might > > > >>> > > > > > > actually > > > >>> > > > > > > > > > need > > > >>> > > > > > > > > > > to > > > >>> > > > > > > > > > > > > log > > > >>> > > > > > > > > > > > > > > > different principal strings for the > > case > > > of > > > >>> > > queries > > > >>> > > > > > like > > > >>> > > > > > > > > > > > AlterConfigs > > > >>> > > > > > > > > > > > > > > > (mentioned by Rajini) which may > contain > > > >>> > multiple > > > >>> > > > > > > resources. > > > >>> > > > > > > > > > Take > > > >>> > > > > > > > > > > an > > > >>> > > > > > > > > > > > > > LDAP > > > >>> > > > > > > > > > > > > > > > authorizer that grants access based > on > > > >>> group > > > >>> > > > > > membership - > > > >>> > > > > > > > the > > > >>> > > > > > > > > > > same > > > >>> > > > > > > > > > > > > > > > alterconfig request may contain > > resources > > > >>> that > > > >>> > > are > > > >>> > > > > > > > authorized > > > >>> > > > > > > > > > > based > > > >>> > > > > > > > > > > > > on > > > >>> > > > > > > > > > > > > > > > group1 as well as resources > authorized > > > >>> based on > > > >>> > > > > > > membership > > > >>> > > > > > > > in > > > >>> > > > > > > > > > > > group 2 > > > >>> > > > > > > > > > > > > > .. > > > >>> > > > > > > > > > > > > > > > And in all cases we'd need to log the > > > >>> specific > > > >>> > > > > reason I > > > >>> > > > > > > > > think.. > > > >>> > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > Basically I think that we might have > a > > > hard > > > >>> > time > > > >>> > > > > > properly > > > >>> > > > > > > > > > > > authorizing > > > >>> > > > > > > > > > > > > > and > > > >>> > > > > > > > > > > > > > > > logging without being able to forward > > the > > > >>> > entire > > > >>> > > > > > > > principal.. > > > >>> > > > > > > > > > but > > > >>> > > > > > > > > > > > > > again, I > > > >>> > > > > > > > > > > > > > > > might be heading down an entirely > wrong > > > >>> path > > > >>> > here > > > >>> > > > :) > > > >>> > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > Best regards, > > > >>> > > > > > > > > > > > > > > > Sönke > > > >>> > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > On Wed, 22 Apr 2020 at 23:13, > Guozhang > > > >>> Wang < > > > >>> > > > > > > > > > wangg...@gmail.com> > > > >>> > > > > > > > > > > > > > wrote: > > > >>> > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > Colin, Boyang: thanks for the > > updates, > > > I > > > >>> > agree > > > >>> > > > that > > > >>> > > > > > an > > > >>> > > > > > > > > > > > > > EnvelopeRequest > > > >>> > > > > > > > > > > > > > > > > would be a less vulnerable approach > > > than > > > >>> > > optional > > > >>> > > > > > > fields, > > > >>> > > > > > > > > and > > > >>> > > > > > > > > > > I'm > > > >>> > > > > > > > > > > > > > just > > > >>> > > > > > > > > > > > > > > > > wondering if we would keep the > > > >>> > EnvelopeRequest > > > >>> > > > for > > > >>> > > > > a > > > >>> > > > > > > long > > > >>> > > > > > > > > > > time. I > > > >>> > > > > > > > > > > > > was > > > >>> > > > > > > > > > > > > > > > > thinking that, potentially if we > > > require > > > >>> > > clients > > > >>> > > > to > > > >>> > > > > > be > > > >>> > > > > > > on > > > >>> > > > > > > > > > newer > > > >>> > > > > > > > > > > > > > version > > > >>> > > > > > > > > > > > > > > > > when talking to a 3.0+ (assuming > 3.0 > > is > > > >>> the > > > >>> > > > bridge > > > >>> > > > > > > > release) > > > >>> > > > > > > > > > > > > brokers, > > > >>> > > > > > > > > > > > > > then > > > >>> > > > > > > > > > > > > > > > > we do not need to keep this code > for > > > too > > > >>> > long, > > > >>> > > > but > > > >>> > > > > I > > > >>> > > > > > > > think > > > >>> > > > > > > > > > that > > > >>> > > > > > > > > > > > > > would be > > > >>> > > > > > > > > > > > > > > > a > > > >>> > > > > > > > > > > > > > > > > very hasty compatibility breaking > so > > > >>> maybe we > > > >>> > > > > indeed > > > >>> > > > > > > need > > > >>> > > > > > > > > to > > > >>> > > > > > > > > > > keep > > > >>> > > > > > > > > > > > > > this > > > >>> > > > > > > > > > > > > > > > > forwarding mechanism many years. > > > >>> > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > Regarding future use cases, I think > > the > > > >>> > example > > > >>> > > > > that > > > >>> > > > > > > > Boyang > > > >>> > > > > > > > > > > > > > mentioned may > > > >>> > > > > > > > > > > > > > > > > not be very practical honestly, > > because > > > >>> when > > > >>> > > > > there's > > > >>> > > > > > a > > > >>> > > > > > > > > > > > connectivity > > > >>> > > > > > > > > > > > > > > > issue, > > > >>> > > > > > > > > > > > > > > > > it is either a network partition > > > between > > > >>> > > > > "controller, > > > >>> > > > > > > A | > > > >>> > > > > > > > > B", > > > >>> > > > > > > > > > > or > > > >>> > > > > > > > > > > > > > > > > "controller | A, B". In other > words, > > if > > > >>> the > > > >>> > > > > > controller > > > >>> > > > > > > > can > > > >>> > > > > > > > > > talk > > > >>> > > > > > > > > > > > to > > > >>> > > > > > > > > > > > > A, > > > >>> > > > > > > > > > > > > > > > then > > > >>> > > > > > > > > > > > > > > > > very likely A would not be able to > > talk > > > >>> to B > > > >>> > > > > > either... > > > >>> > > > > > > > > > anyways, > > > >>> > > > > > > > > > > > > > since the > > > >>> > > > > > > > > > > > > > > > > forwarding would be there for a > > > >>> sufficiently > > > >>> > > long > > > >>> > > > > > > time, I > > > >>> > > > > > > > > > think > > > >>> > > > > > > > > > > > > > keeping > > > >>> > > > > > > > > > > > > > > > the > > > >>> > > > > > > > > > > > > > > > > additional envelope makes sense. > > > >>> > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > Guozhang > > > >>> > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > On Wed, Apr 22, 2020 at 1:47 PM > > Boyang > > > >>> Chen < > > > >>> > > > > > > > > > > > > > reluctanthero...@gmail.com> > > > >>> > > > > > > > > > > > > > > > > wrote: > > > >>> > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > Thanks Colin for the summary! And > > > >>> Guozhang, > > > >>> > > > > > regarding > > > >>> > > > > > > > the > > > >>> > > > > > > > > > > > future > > > >>> > > > > > > > > > > > > > use > > > >>> > > > > > > > > > > > > > > > > cases, > > > >>> > > > > > > > > > > > > > > > > > consider a scenario where there > are > > > >>> > temporary > > > >>> > > > > > > > > connectivity > > > >>> > > > > > > > > > > > issue > > > >>> > > > > > > > > > > > > > > > between > > > >>> > > > > > > > > > > > > > > > > > controller to a fellow broker A, > > the > > > >>> > > controller > > > >>> > > > > > could > > > >>> > > > > > > > > then > > > >>> > > > > > > > > > > > > leverage > > > >>> > > > > > > > > > > > > > > > > another > > > >>> > > > > > > > > > > > > > > > > > healthy broker B to do a > forwarding > > > >>> request > > > >>> > > to > > > >>> > > > A > > > >>> > > > > in > > > >>> > > > > > > > order > > > >>> > > > > > > > > > to > > > >>> > > > > > > > > > > > > > maintain a > > > >>> > > > > > > > > > > > > > > > > > communication. > > > >>> > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > Even for KIP-590 scope, the > > > forwarding > > > >>> > > > mechanism > > > >>> > > > > > > shall > > > >>> > > > > > > > > not > > > >>> > > > > > > > > > be > > > >>> > > > > > > > > > > > > > transit > > > >>> > > > > > > > > > > > > > > > as > > > >>> > > > > > > > > > > > > > > > > we > > > >>> > > > > > > > > > > > > > > > > > do need to support older version > of > > > >>> admin > > > >>> > > > clients > > > >>> > > > > > > for a > > > >>> > > > > > > > > > > couple > > > >>> > > > > > > > > > > > of > > > >>> > > > > > > > > > > > > > years > > > >>> > > > > > > > > > > > > > > > > > IIUC. > > > >>> > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > Boyang > > > >>> > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > On Wed, Apr 22, 2020 at 1:29 PM > > Colin > > > >>> > McCabe > > > >>> > > < > > > >>> > > > > > > > > > > > cmcc...@apache.org > > > >>> > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > wrote: > > > >>> > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > Hi all, > > > >>> > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > I guess the way I see this > > working > > > is > > > >>> > that > > > >>> > > > the > > > >>> > > > > > > > request > > > >>> > > > > > > > > > gets > > > >>> > > > > > > > > > > > > sent > > > >>> > > > > > > > > > > > > > from > > > >>> > > > > > > > > > > > > > > > > the > > > >>> > > > > > > > > > > > > > > > > > > client, to the initial broker, > > and > > > >>> then > > > >>> > > > > forwarded > > > >>> > > > > > > to > > > >>> > > > > > > > > the > > > >>> > > > > > > > > > > > final > > > >>> > > > > > > > > > > > > > > > broker. > > > >>> > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > The initial broker should do > > > >>> > authentication > > > >>> > > > > (who > > > >>> > > > > > > are > > > >>> > > > > > > > > > you?) > > > >>> > > > > > > > > > > > and > > > >>> > > > > > > > > > > > > > come > > > >>> > > > > > > > > > > > > > > > up > > > >>> > > > > > > > > > > > > > > > > > > with a principal name. Then it > > > >>> creates > > > >>> > an > > > >>> > > > > > envelope > > > >>> > > > > > > > > > > request, > > > >>> > > > > > > > > > > > > > which > > > >>> > > > > > > > > > > > > > > > will > > > >>> > > > > > > > > > > > > > > > > > > contain that principal name, > and > > > >>> sends it > > > >>> > > > along > > > >>> > > > > > > with > > > >>> > > > > > > > > the > > > >>> > > > > > > > > > > > > > unmodified > > > >>> > > > > > > > > > > > > > > > > > > original request to the final > > > >>> broker. (I > > > >>> > > > agree > > > >>> > > > > > > with > > > >>> > > > > > > > > Tom > > > >>> > > > > > > > > > > and > > > >>> > > > > > > > > > > > > > Rajini > > > >>> > > > > > > > > > > > > > > > > that > > > >>> > > > > > > > > > > > > > > > > > we > > > >>> > > > > > > > > > > > > > > > > > > can't forward the information > > > needed > > > >>> to > > > >>> > do > > > >>> > > > > > > > > authentication > > > >>> > > > > > > > > > > on > > > >>> > > > > > > > > > > > > the > > > >>> > > > > > > > > > > > > > > > final > > > >>> > > > > > > > > > > > > > > > > > > broker, but I also think we > don't > > > >>> need > > > >>> > to, > > > >>> > > > > since > > > >>> > > > > > we > > > >>> > > > > > > > can > > > >>> > > > > > > > > > do > > > >>> > > > > > > > > > > it > > > >>> > > > > > > > > > > > > on > > > >>> > > > > > > > > > > > > > the > > > >>> > > > > > > > > > > > > > > > > > > initial broker.) > > > >>> > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > The final broker knows it can > > trust > > > >>> the > > > >>> > > > > principal > > > >>> > > > > > > > name > > > >>> > > > > > > > > in > > > >>> > > > > > > > > > > the > > > >>> > > > > > > > > > > > > > > > envelope > > > >>> > > > > > > > > > > > > > > > > > > (since EnvelopeRequest requires > > > >>> > > CLUSTERACTION > > > >>> > > > > on > > > >>> > > > > > > > > > CLUSTER). > > > >>> > > > > > > > > > > > So > > > >>> > > > > > > > > > > > > > it can > > > >>> > > > > > > > > > > > > > > > > use > > > >>> > > > > > > > > > > > > > > > > > > that principal name for > > > authorization > > > >>> > > (given > > > >>> > > > > who > > > >>> > > > > > > you > > > >>> > > > > > > > > are, > > > >>> > > > > > > > > > > > what > > > >>> > > > > > > > > > > > > > can > > > >>> > > > > > > > > > > > > > > > you > > > >>> > > > > > > > > > > > > > > > > > > do?) The forwarded principal > > name > > > >>> will > > > >>> > > also > > > >>> > > > be > > > >>> > > > > > > used > > > >>> > > > > > > > > for > > > >>> > > > > > > > > > > > > logging. > > > >>> > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > One question is why we need an > > > >>> > > > EnvelopeRequest. > > > >>> > > > > > > > Well, > > > >>> > > > > > > > > if > > > >>> > > > > > > > > > > we > > > >>> > > > > > > > > > > > > > don't > > > >>> > > > > > > > > > > > > > > > have > > > >>> > > > > > > > > > > > > > > > > > an > > > >>> > > > > > > > > > > > > > > > > > > EnvelopeRequest, we need > > somewhere > > > >>> else > > > >>> > to > > > >>> > > > put > > > >>> > > > > > the > > > >>> > > > > > > > > > > forwarded > > > >>> > > > > > > > > > > > > > > > principal > > > >>> > > > > > > > > > > > > > > > > > > name. I don't think overriding > > an > > > >>> > existing > > > >>> > > > > field > > > >>> > > > > > > > (like > > > >>> > > > > > > > > > > > > > clientId) is > > > >>> > > > > > > > > > > > > > > > a > > > >>> > > > > > > > > > > > > > > > > > good > > > >>> > > > > > > > > > > > > > > > > > > option for this. It's messy, > and > > > >>> loses > > > >>> > > > > > > information. > > > >>> > > > > > > > > It > > > >>> > > > > > > > > > > also > > > >>> > > > > > > > > > > > > > raises > > > >>> > > > > > > > > > > > > > > > > the > > > >>> > > > > > > > > > > > > > > > > > > question of how the final > broker > > > >>> knows > > > >>> > that > > > >>> > > > the > > > >>> > > > > > > > > clientId > > > >>> > > > > > > > > > in > > > >>> > > > > > > > > > > > the > > > >>> > > > > > > > > > > > > > > > > received > > > >>> > > > > > > > > > > > > > > > > > > message is not "really" a > > clientId, > > > >>> but > > > >>> > is > > > >>> > > a > > > >>> > > > > > > > principal > > > >>> > > > > > > > > > > name. > > > >>> > > > > > > > > > > > > > Without > > > >>> > > > > > > > > > > > > > > > > an > > > >>> > > > > > > > > > > > > > > > > > > envelope, there's no way to > > clearly > > > >>> mark > > > >>> > a > > > >>> > > > > > request > > > >>> > > > > > > as > > > >>> > > > > > > > > > > > > forwarded, > > > >>> > > > > > > > > > > > > > so > > > >>> > > > > > > > > > > > > > > > > > there's > > > >>> > > > > > > > > > > > > > > > > > > no reason for the final broker > to > > > >>> treat > > > >>> > > this > > > >>> > > > > > > > > differently > > > >>> > > > > > > > > > > > than a > > > >>> > > > > > > > > > > > > > > > regular > > > >>> > > > > > > > > > > > > > > > > > > clientId (or whatever). > > > >>> > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > We talked about using optional > > > >>> fields to > > > >>> > > > > contain > > > >>> > > > > > > the > > > >>> > > > > > > > > > > > forwarded > > > >>> > > > > > > > > > > > > > > > > principal > > > >>> > > > > > > > > > > > > > > > > > > name, but this is also messy > and > > > >>> > > potentially > > > >>> > > > > > > > dangerous. > > > >>> > > > > > > > > > > > Older > > > >>> > > > > > > > > > > > > > > > brokers > > > >>> > > > > > > > > > > > > > > > > > will > > > >>> > > > > > > > > > > > > > > > > > > simply ignore the optional > > fields, > > > >>> which > > > >>> > > > could > > > >>> > > > > > > result > > > >>> > > > > > > > > in > > > >>> > > > > > > > > > > them > > > >>> > > > > > > > > > > > > > > > executing > > > >>> > > > > > > > > > > > > > > > > > > operations as the wrong > > principal. > > > >>> Of > > > >>> > > > course, > > > >>> > > > > > this > > > >>> > > > > > > > > would > > > >>> > > > > > > > > > > > > > require a > > > >>> > > > > > > > > > > > > > > > > > > misconfiguration in order to > > > happen, > > > >>> but > > > >>> > it > > > >>> > > > > still > > > >>> > > > > > > > seems > > > >>> > > > > > > > > > > > better > > > >>> > > > > > > > > > > > > > to set > > > >>> > > > > > > > > > > > > > > > > up > > > >>> > > > > > > > > > > > > > > > > > > the system so that this > > > >>> misconfiguration > > > >>> > is > > > >>> > > > > > > detected, > > > >>> > > > > > > > > > > rather > > > >>> > > > > > > > > > > > > than > > > >>> > > > > > > > > > > > > > > > > > silently > > > >>> > > > > > > > > > > > > > > > > > > ignored. > > > >>> > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > It's true that the need for > > > >>> forwarding is > > > >>> > > > > > > "temporary" > > > >>> > > > > > > > > in > > > >>> > > > > > > > > > > some > > > >>> > > > > > > > > > > > > > sense, > > > >>> > > > > > > > > > > > > > > > > > since > > > >>> > > > > > > > > > > > > > > > > > > we only need it for older > > clients. > > > >>> > > However, > > > >>> > > > we > > > >>> > > > > > > will > > > >>> > > > > > > > > want > > > >>> > > > > > > > > > > to > > > >>> > > > > > > > > > > > > > support > > > >>> > > > > > > > > > > > > > > > > > these > > > >>> > > > > > > > > > > > > > > > > > > older clients for many years to > > > come. > > > >>> > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > I agree that the usefulness of > > > >>> > > > EnvelopeRequest > > > >>> > > > > is > > > >>> > > > > > > > > limited > > > >>> > > > > > > > > > > by > > > >>> > > > > > > > > > > > it > > > >>> > > > > > > > > > > > > > > > being a > > > >>> > > > > > > > > > > > > > > > > > > superuser-only request at the > > > moment. > > > >>> > > > Perhaps > > > >>> > > > > > > there > > > >>> > > > > > > > > are > > > >>> > > > > > > > > > > some > > > >>> > > > > > > > > > > > > > changes > > > >>> > > > > > > > > > > > > > > > > to > > > >>> > > > > > > > > > > > > > > > > > > how custom principals work that > > > would > > > >>> > allow > > > >>> > > > us > > > >>> > > > > to > > > >>> > > > > > > get > > > >>> > > > > > > > > > > around > > > >>> > > > > > > > > > > > > > this in > > > >>> > > > > > > > > > > > > > > > > the > > > >>> > > > > > > > > > > > > > > > > > > future. We should think about > > that > > > >>> so > > > >>> > that > > > >>> > > > we > > > >>> > > > > > have > > > >>> > > > > > > > > this > > > >>> > > > > > > > > > > > > > > > functionality > > > >>> > > > > > > > > > > > > > > > > in > > > >>> > > > > > > > > > > > > > > > > > > the future if it's needed. > > > >>> > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > best, > > > >>> > > > > > > > > > > > > > > > > > > Colin > > > >>> > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > On Wed, Apr 22, 2020, at 11:21, > > > >>> Guozhang > > > >>> > > Wang > > > >>> > > > > > > wrote: > > > >>> > > > > > > > > > > > > > > > > > > > Hello Gwen, > > > >>> > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > The purpose here is for > > > maintaining > > > >>> > > > > > compatibility > > > >>> > > > > > > > old > > > >>> > > > > > > > > > > > > clients, > > > >>> > > > > > > > > > > > > > who > > > >>> > > > > > > > > > > > > > > > do > > > >>> > > > > > > > > > > > > > > > > > not > > > >>> > > > > > > > > > > > > > > > > > > > have functionality to do > > > re-routing > > > >>> > admin > > > >>> > > > > > > requests > > > >>> > > > > > > > > > > > > themselves. > > > >>> > > > > > > > > > > > > > New > > > >>> > > > > > > > > > > > > > > > > > > clients > > > >>> > > > > > > > > > > > > > > > > > > > can of course do this > > themselves > > > by > > > >>> > > > detecting > > > >>> > > > > > > who's > > > >>> > > > > > > > > the > > > >>> > > > > > > > > > > > > > controller. > > > >>> > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > Hello Colin / Boyang, > > > >>> > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > Regarding the usage of the > > > >>> envelope, > > > >>> > I'm > > > >>> > > > > > curious > > > >>> > > > > > > > what > > > >>> > > > > > > > > > are > > > >>> > > > > > > > > > > > the > > > >>> > > > > > > > > > > > > > > > > potential > > > >>> > > > > > > > > > > > > > > > > > > > future use cases that would > > > require > > > >>> > > request > > > >>> > > > > > > > > forwarding > > > >>> > > > > > > > > > > and > > > >>> > > > > > > > > > > > > > hence > > > >>> > > > > > > > > > > > > > > > > > envelope > > > >>> > > > > > > > > > > > > > > > > > > > would be useful? Originally I > > > >>> thought > > > >>> > > that > > > >>> > > > > the > > > >>> > > > > > > > > > forwarding > > > >>> > > > > > > > > > > > > > mechanism > > > >>> > > > > > > > > > > > > > > > > is > > > >>> > > > > > > > > > > > > > > > > > > only > > > >>> > > > > > > > > > > > > > > > > > > > temporary as we need it for > the > > > >>> bridge > > > >>> > > > > release, > > > >>> > > > > > > and > > > >>> > > > > > > > > > > moving > > > >>> > > > > > > > > > > > > > forward > > > >>> > > > > > > > > > > > > > > > we > > > >>> > > > > > > > > > > > > > > > > > > will > > > >>> > > > > > > > > > > > > > > > > > > > get rid of this to simplify > the > > > >>> code > > > >>> > > base. > > > >>> > > > If > > > >>> > > > > > we > > > >>> > > > > > > do > > > >>> > > > > > > > > > have > > > >>> > > > > > > > > > > > > valid > > > >>> > > > > > > > > > > > > > use > > > >>> > > > > > > > > > > > > > > > > > cases > > > >>> > > > > > > > > > > > > > > > > > > in > > > >>> > > > > > > > > > > > > > > > > > > > the future which makes us > > believe > > > >>> that > > > >>> > > > > request > > > >>> > > > > > > > > > forwarding > > > >>> > > > > > > > > > > > > would > > > >>> > > > > > > > > > > > > > > > > > actually > > > >>> > > > > > > > > > > > > > > > > > > be > > > >>> > > > > > > > > > > > > > > > > > > > a permanent feature retained > on > > > the > > > >>> > > broker > > > >>> > > > > > side, > > > >>> > > > > > > > I'm > > > >>> > > > > > > > > on > > > >>> > > > > > > > > > > > board > > > >>> > > > > > > > > > > > > > with > > > >>> > > > > > > > > > > > > > > > > > adding > > > >>> > > > > > > > > > > > > > > > > > > > the envelope request > protocol. > > > >>> > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > Guozhang > > > >>> > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > On Wed, Apr 22, 2020 at 8:22 > AM > > > >>> Gwen > > > >>> > > > Shapira > > > >>> > > > > < > > > >>> > > > > > > > > > > > > > g...@confluent.io> > > > >>> > > > > > > > > > > > > > > > > > wrote: > > > >>> > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > Hey Boyang, > > > >>> > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > Sorry if this was already > > > >>> discussed, > > > >>> > > but > > > >>> > > > I > > > >>> > > > > > > didn't > > > >>> > > > > > > > > see > > > >>> > > > > > > > > > > > this > > > >>> > > > > > > > > > > > > as > > > >>> > > > > > > > > > > > > > > > > > rejected > > > >>> > > > > > > > > > > > > > > > > > > > > alternative: > > > >>> > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > Until now, we always did > > client > > > >>> side > > > >>> > > > > routing > > > >>> > > > > > - > > > >>> > > > > > > > the > > > >>> > > > > > > > > > > client > > > >>> > > > > > > > > > > > > > itself > > > >>> > > > > > > > > > > > > > > > > > found > > > >>> > > > > > > > > > > > > > > > > > > the > > > >>> > > > > > > > > > > > > > > > > > > > > controller via metadata and > > > >>> directed > > > >>> > > > > requests > > > >>> > > > > > > > > > > > accordingly. > > > >>> > > > > > > > > > > > > > > > Brokers > > > >>> > > > > > > > > > > > > > > > > > that > > > >>> > > > > > > > > > > > > > > > > > > > > were not the controller, > > > rejected > > > >>> > those > > > >>> > > > > > > requests. > > > >>> > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > Why did we decide to move > to > > > >>> broker > > > >>> > > side > > > >>> > > > > > > routing? > > > >>> > > > > > > > > Was > > > >>> > > > > > > > > > > the > > > >>> > > > > > > > > > > > > > > > > client-side > > > >>> > > > > > > > > > > > > > > > > > > > > option discussed and > rejected > > > >>> > somewhere > > > >>> > > > > and I > > > >>> > > > > > > > > missed > > > >>> > > > > > > > > > > it? > > > >>> > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > Gwen > > > >>> > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > On Fri, Apr 3, 2020, 4:45 > PM > > > >>> Boyang > > > >>> > > Chen > > > >>> > > > < > > > >>> > > > > > > > > > > > > > > > > reluctanthero...@gmail.com > > > >>> > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > wrote: > > > >>> > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > Hey all, > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > I would like to start off > > the > > > >>> > > > discussion > > > >>> > > > > > for > > > >>> > > > > > > > > > > KIP-590, a > > > >>> > > > > > > > > > > > > > > > follow-up > > > >>> > > > > > > > > > > > > > > > > > > > > > initiative after KIP-500: > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-590%3A+Redirect+Zookeeper+Mutation+Protocols+to+The+Controller > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > This KIP proposes to > > migrate > > > >>> > existing > > > >>> > > > > > > Zookeeper > > > >>> > > > > > > > > > > > mutation > > > >>> > > > > > > > > > > > > > paths, > > > >>> > > > > > > > > > > > > > > > > > > including > > > >>> > > > > > > > > > > > > > > > > > > > > > configuration, security > and > > > >>> quota > > > >>> > > > > changes, > > > >>> > > > > > to > > > >>> > > > > > > > > > > > > > controller-only > > > >>> > > > > > > > > > > > > > > > by > > > >>> > > > > > > > > > > > > > > > > > > always > > > >>> > > > > > > > > > > > > > > > > > > > > > routing these alterations > > to > > > >>> the > > > >>> > > > > > controller. > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > Let me know your > thoughts! > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > Best, > > > >>> > > > > > > > > > > > > > > > > > > > > > Boyang > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > -- > > > >>> > > > > > > > > > > > > > > > > > > > -- Guozhang > > > >>> > > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > -- > > > >>> > > > > > > > > > > > > > > > > -- Guozhang > > > >>> > > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > -- > > > >>> > > > > > > > > > > > > > > > Sönke Liebau > > > >>> > > > > > > > > > > > > > > > Partner > > > >>> > > > > > > > > > > > > > > > Tel. +49 179 7940878 > > > >>> > > > > > > > > > > > > > > > OpenCore GmbH & Co. KG - > > > >>> Thomas-Mann-Straße 8 - > > > >>> > > > 22880 > > > >>> > > > > > > > Wedel - > > > >>> > > > > > > > > > > > Germany > > > >>> > > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > -- > > > >>> > > > > > > > > > > > > Sönke Liebau > > > >>> > > > > > > > > > > > > Partner > > > >>> > > > > > > > > > > > > Tel. +49 179 7940878 > > > >>> > > > > > > > > > > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße > > 8 - > > > >>> 22880 > > > >>> > > > > Wedel - > > > >>> > > > > > > > > Germany > > > >>> > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > > -- > > > >>> > > > > > > > > > > Sönke Liebau > > > >>> > > > > > > > > > > Partner > > > >>> > > > > > > > > > > Tel. +49 179 7940878 > > > >>> > > > > > > > > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - > > > 22880 > > > >>> > > Wedel - > > > >>> > > > > > > Germany > > > >>> > > > > > > > > > > > > > >>> > > > > > > > > > > > > >>> > > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > -- > > > >>> > > > > > > Sönke Liebau > > > >>> > > > > > > Partner > > > >>> > > > > > > Tel. +49 179 7940878 > > > >>> > > > > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 > > > Wedel - > > > >>> > > Germany > > > >>> > > > > > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > -- > > > >>> > > > > Sönke Liebau > > > >>> > > > > Partner > > > >>> > > > > Tel. +49 179 7940878 > > > >>> > > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 > Wedel - > > > >>> Germany > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > > >>> > -- > > > >>> > -- Guozhang > > > >>> > > > > >>> > > > >> > > > > > >
