Thanks Jun for the suggestions! I have addressed suggestion and simplified
the metrics part.
On Tue, Jun 9, 2020 at 5:46 PM Jun Rao <j...@confluent.io> wrote:
> Hi, Boyang,
>
> Thanks for the KIP. Just a few comments on the metrics.
>
> 1. It would be useful to document the full JMX metric names (package, type,
> etc) of the new metrics. Also, for rates on the server side, we
> typically use Yammer Meter.
>
> Sounds good.
2. For num-messages-redirected-rate, would num-requests-redirected-rate be
> better?
>
> Actually for the scope of this KIP, we are no longer needing to have a
separate tracking
of forwarded request rate, because the Envelope RPC is dropped.
> 3. num-client-forwarding-to-controller-rate: Is that based on clientId,
> client IP, client request version or sth else? How do you plan to implement
> that since it seems to require tracking the current unique client set
> somehow. An alternative approach is to maintain a
> num-requests-redirected-rate metric with a client tag.
>
The clientId tag approach makes sense, will add to the KIP.
Jun
>
>
>
> On Mon, Jun 8, 2020 at 9:36 AM Boyang Chen <reluctanthero...@gmail.com>
> wrote:
>
> > Hey there,
> >
> > If no more question is raised, I will go ahead and start the vote
> shortly.
> >
> > On Thu, Jun 4, 2020 at 12:39 PM Boyang Chen <reluctanthero...@gmail.com>
> > wrote:
> >
> > > Hey there,
> > >
> > > bumping this thread for any further KIP-590 discussion, since it's been
> > > quiet for a while.
> > >
> > > Boyang
> > >
> > > On Thu, May 21, 2020 at 10:31 AM Boyang Chen <
> reluctanthero...@gmail.com
> > >
> > > wrote:
> > >
> > >> Thanks David, I agree the wording here is not clear, and the fellow
> > >> broker should just send a new CreateTopicRequest in this case.
> > >>
> > >> In the meantime, we had some offline discussion about the Envelope API
> > as
> > >> well. Although it provides certain privileges like data embedding and
> > >> principal embedding, it creates a security hole by letting a malicious
> > user
> > >> impersonate any forwarding broker, thus pretending to be any admin
> user.
> > >> Passing the principal around also increases the vulnerability,
> compared
> > >> with other standard ways such as passing a verified token, but it is
> > >> unfortunately not fully supported with Kafka security.
> > >>
> > >> So for the security concerns, we are abandoning the Envelope approach
> > and
> > >> fallback to just forward the raw admin requests. The authentication
> will
> > >> happen on the receiving broker side instead of on the controller, so
> > that
> > >> we are able to stripped off the principal fields and only include the
> > >> principal in header as optional field for audit logging purpose.
> > >> Furthermore, we shall propose adding a separate endpoint for
> > >> broker-controller communication which should be recommended to enable
> > >> secure connections so that a malicious client could not pretend to be
> a
> > >> broker and perform impersonation.
> > >>
> > >> Let me know your thoughts.
> > >>
> > >> Best,
> > >> Boyang
> > >>
> > >> On Tue, May 19, 2020 at 12:17 AM David Jacot <dja...@confluent.io>
> > wrote:
> > >>
> > >>> Hi Boyang,
> > >>>
> > >>> I've got another question regarding the auto topic creation case. The
> > KIP
> > >>> says: "Currently the target broker shall just utilize its own ZK
> client
> > >>> to
> > >>> create
> > >>> internal topics, which is disallowed in the bridge release. For above
> > >>> scenarios,
> > >>> non-controller broker shall just forward a CreateTopicRequest to the
> > >>> controller
> > >>> instead and let controller take care of the rest, while waiting for
> the
> > >>> response
> > >>> in the meantime." There will be no request to forward in this case,
> > >>> right?
> > >>> Instead,
> > >>> a CreateTopicsRequest is created and sent to the controller node.
> > >>>
> > >>> When the CreateTopicsRequest is sent as a side effect of the
> > >>> MetadataRequest,
> > >>> it would be good to know the principal and the clientId in the
> > controller
> > >>> (quota,
> > >>> audit, etc.). Do you plan to use the Envelope API for this case as
> well
> > >>> or
> > >>> to call
> > >>> the regular API directly? Another was to phrase it would be: Shall
> the
> > >>> internal
> > >>> CreateTopicsRequest be sent with the original metadata (principal,
> > >>> clientId, etc.)
> > >>> of the MetadataRequest or as an admin request?
> > >>>
> > >>> Best,
> > >>> David
> > >>>
> > >>> On Fri, May 8, 2020 at 2:04 AM Guozhang Wang <wangg...@gmail.com>
> > wrote:
> > >>>
> > >>> > Just to adds a bit more FYI here related to the last question from
> > >>> David:
> > >>> > in KIP-595 while implementing the new requests we are also adding a
> > >>> > "KafkaNetworkChannel" which is used for brokers to send vote /
> fetch
> > >>> > records, so besides the discussion on listeners I think
> > implementation
> > >>> wise
> > >>> > we can also consider consolidating a lot of those into the same
> > >>> call-trace
> > >>> > as well -- of course this is not related to public APIs so maybe
> just
> > >>> needs
> > >>> > to be coordinated among developments:
> > >>> >
> > >>> > 1. Broker -> Controller: ISR Change, Topic Creation, Admin Redirect
> > >>> > (KIP-497).
> > >>> > 2. Controller -> Broker: LeaderAndISR / MetadataUpdate; though
> these
> > >>> are
> > >>> > likely going to be deprecated post KIP-500.
> > >>> > 3. Txn Coordinator -> Broker: TxnMarker
> > >>> >
> > >>> >
> > >>> > Guozhang
> > >>> >
> > >>> > On Wed, May 6, 2020 at 8:58 PM Boyang Chen <
> > reluctanthero...@gmail.com
> > >>> >
> > >>> > wrote:
> > >>> >
> > >>> > > Hey David,
> > >>> > >
> > >>> > > thanks for the feedbacks!
> > >>> > >
> > >>> > > On Wed, May 6, 2020 at 2:06 AM David Jacot <dja...@confluent.io>
> > >>> wrote:
> > >>> > >
> > >>> > > > Hi Boyang,
> > >>> > > >
> > >>> > > > While re-reading the KIP, I've got few small
> questions/comments:
> > >>> > > >
> > >>> > > > 1. When auto topic creation is enabled, brokers will send a
> > >>> > > > CreateTopicRequest
> > >>> > > > to the controller instead of writing to ZK directly. It means
> > that
> > >>> > > > creation of these
> > >>> > > > topics are subject to be rejected with an error if a
> > >>> CreateTopicPolicy
> > >>> > is
> > >>> > > > used. Today,
> > >>> > > > it bypasses the policy entirely. I suppose that clusters
> allowing
> > >>> auto
> > >>> > > > topic creation
> > >>> > > > don't have a policy in place so it is not a big deal. I suggest
> > to
> > >>> call
> > >>> > > > out explicitly the
> > >>> > > > limitation in the KIP though.
> > >>> > > >
> > >>> > > > That's a good idea, will add to the KIP.
> > >>> > >
> > >>> > >
> > >>> > > > 2. In the same vein as my first point. How do you plan to
> handle
> > >>> errors
> > >>> > > > when internal
> > >>> > > > topics are created by a broker? Do you plan to retry retryable
> > >>> errors
> > >>> > > > indefinitely?
> > >>> > > >
> > >>> > > > I checked a bit on the admin client handling of the create
> topic
> > >>> RPC.
> > >>> > It
> > >>> > > seems that
> > >>> > > the only retriable exceptions at the moment are NOT_CONTROLLER
> and
> > >>> > > REQUEST_TIMEOUT.
> > >>> > > So I guess we just need to retry on these exceptions?
> > >>> > >
> > >>> > >
> > >>> > > > 3. Could you clarify which listener will be used for the
> internal
> > >>> > > requests?
> > >>> > > > Do you plan
> > >>> > > > to use the control plane listener or perhaps the inter-broker
> > >>> listener?
> > >>> > > >
> > >>> > > > As we discussed in the KIP, currently the internal design for
> > >>> > > broker->controller channel has not been
> > >>> > > done yet, and I feel it makes sense to consolidate redirect RPC
> and
> > >>> > > internal topic creation RPC to this future channel,
> > >>> > > which are details to be filled in the near future, right now some
> > >>> > > controller refactoring effort is still WIP.
> > >>> > >
> > >>> > >
> > >>> > > > Thanks,
> > >>> > > > David
> > >>> > > >
> > >>> > > > On Mon, May 4, 2020 at 9:37 AM Sönke Liebau
> > >>> > > > <soenke.lie...@opencore.com.invalid> wrote:
> > >>> > > >
> > >>> > > > > Ah, I see, thanks for the clarification!
> > >>> > > > >
> > >>> > > > > Shouldn't be an issue I think. My understanding of KIPs was
> > >>> always
> > >>> > that
> > >>> > > > > they are mostly intended as a place to discuss and agree
> > changes
> > >>> up
> > >>> > > > front,
> > >>> > > > > whereas tracking the actual releases that things go into
> should
> > >>> be
> > >>> > > > handled
> > >>> > > > > in Jira.
> > >>> > > > > So maybe we just create new jiras for any subsequent work and
> > >>> either
> > >>> > > link
> > >>> > > > > those or make them subtasks (even though this jira is
> already a
> > >>> > subtask
> > >>> > > > > itself), that should allow us to properly track all releases
> > that
> > >>> > work
> > >>> > > > goes
> > >>> > > > > into.
> > >>> > > > >
> > >>> > > > > Thanks for your work on this!!
> > >>> > > > >
> > >>> > > > > Best,
> > >>> > > > > Sönke
> > >>> > > > >
> > >>> > > > >
> > >>> > > > > On Sat, 2 May 2020 at 00:31, Boyang Chen <
> > >>> reluctanthero...@gmail.com
> > >>> > >
> > >>> > > > > wrote:
> > >>> > > > >
> > >>> > > > > > Sure thing Sonke, what I suggest is that usual KIPs get
> > >>> accepted to
> > >>> > > go
> > >>> > > > > into
> > >>> > > > > > next release. It could span for a couple of releases
> because
> > of
> > >>> > > > > engineering
> > >>> > > > > > time, but no change has to be shipped in specific future
> > >>> releases,
> > >>> > > like
> > >>> > > > > the
> > >>> > > > > > backward incompatible change for KafkaPrincipal. But I
> guess
> > >>> it's
> > >>> > not
> > >>> > > > > > really a blocker, as long as we stated clearly in the KIP
> how
> > >>> we
> > >>> > are
> > >>> > > > > going
> > >>> > > > > > to roll things out, and let it partially finish in 2.6.
> > >>> > > > > >
> > >>> > > > > > Boyang
> > >>> > > > > >
> > >>> > > > > > On Fri, May 1, 2020 at 2:32 PM Sönke Liebau
> > >>> > > > > > <soenke.lie...@opencore.com.invalid> wrote:
> > >>> > > > > >
> > >>> > > > > > > Hi Boyang,
> > >>> > > > > > >
> > >>> > > > > > > thanks for the update, sounds reasonable to me. Making
> it a
> > >>> > > breaking
> > >>> > > > > > change
> > >>> > > > > > > is definitely the safer route to go.
> > >>> > > > > > >
> > >>> > > > > > > Just one quick question regarding your mail, I didn't
> fully
> > >>> > > > understand
> > >>> > > > > > what
> > >>> > > > > > > you mean by "I think this is the first time we need to
> > >>> introduce
> > >>> > a
> > >>> > > > KIP
> > >>> > > > > > > without having it
> > >>> > > > > > > fully accepted in next release." - could you perhaps
> > explain
> > >>> > that
> > >>> > > > some
> > >>> > > > > > > more very briefly?
> > >>> > > > > > >
> > >>> > > > > > > Best regards,
> > >>> > > > > > > Sönke
> > >>> > > > > > >
> > >>> > > > > > >
> > >>> > > > > > >
> > >>> > > > > > > On Fri, 1 May 2020 at 23:03, Boyang Chen <
> > >>> > > reluctanthero...@gmail.com
> > >>> > > > >
> > >>> > > > > > > wrote:
> > >>> > > > > > >
> > >>> > > > > > > > Hey Tom,
> > >>> > > > > > > >
> > >>> > > > > > > > thanks for the suggestion. As long as we could
> correctly
> > >>> > > serialize
> > >>> > > > > the
> > >>> > > > > > > > principal and embed in the Envelope, I think we could
> > still
> > >>> > > > leverage
> > >>> > > > > > the
> > >>> > > > > > > > controller to do the client request authentication.
> > >>> Although
> > >>> > this
> > >>> > > > > pays
> > >>> > > > > > an
> > >>> > > > > > > > extra round trip if the authorization is doomed to fail
> > on
> > >>> the
> > >>> > > > > receiver
> > >>> > > > > > > > side, having a centralized processing unit is more
> > >>> favorable
> > >>> > such
> > >>> > > > as
> > >>> > > > > > > > ensuring the audit log is consistent instead of
> > scattering
> > >>> > > between
> > >>> > > > > > > > forwarder and receiver.
> > >>> > > > > > > >
> > >>> > > > > > > > Boyang
> > >>> > > > > > > >
> > >>> > > > > > > > On Wed, Apr 29, 2020 at 9:50 AM Tom Bentley <
> > >>> > tbent...@redhat.com
> > >>> > > >
> > >>> > > > > > wrote:
> > >>> > > > > > > >
> > >>> > > > > > > > > Hi Boyang,
> > >>> > > > > > > > >
> > >>> > > > > > > > > Thanks for the update. In the EnvelopeRequest
> handling
> > >>> > section
> > >>> > > of
> > >>> > > > > the
> > >>> > > > > > > KIP
> > >>> > > > > > > > > it might be worth saying explicitly that
> authorization
> > >>> of the
> > >>> > > > > request
> > >>> > > > > > > > will
> > >>> > > > > > > > > happen as normal. Otherwise what you're proposing
> makes
> > >>> sense
> > >>> > > to
> > >>> > > > > me.
> > >>> > > > > > > > >
> > >>> > > > > > > > > Thanks again,
> > >>> > > > > > > > >
> > >>> > > > > > > > > Tom
> > >>> > > > > > > > >
> > >>> > > > > > > > >
> > >>> > > > > > > > >
> > >>> > > > > > > > > On Wed, Apr 29, 2020 at 5:27 PM Boyang Chen <
> > >>> > > > > > > reluctanthero...@gmail.com>
> > >>> > > > > > > > > wrote:
> > >>> > > > > > > > >
> > >>> > > > > > > > > > Thanks for the proposed idea Sonke. I reviewed it
> and
> > >>> had
> > >>> > > some
> > >>> > > > > > > offline
> > >>> > > > > > > > > > discussion with Colin, Rajini and Mathew.
> > >>> > > > > > > > > >
> > >>> > > > > > > > > > We do need to add serializability to the
> > >>> PrincipalBuilder
> > >>> > > > > > interface,
> > >>> > > > > > > > but
> > >>> > > > > > > > > we
> > >>> > > > > > > > > > should not make any default implementation which
> > could
> > >>> go
> > >>> > > wrong
> > >>> > > > > and
> > >>> > > > > > > > messy
> > >>> > > > > > > > > > up with the security in a production environment if
> > the
> > >>> > user
> > >>> > > > > > neglects
> > >>> > > > > > > > it.
> > >>> > > > > > > > > > Instead we need to make it required and backward
> > >>> > > incompatible.
> > >>> > > > > So I
> > >>> > > > > > > > > > integrated your proposed methods and expand the
> > >>> Envelope
> > >>> > RPC
> > >>> > > > > with a
> > >>> > > > > > > > > couple
> > >>> > > > > > > > > > of more fields for audit log purpose as well.
> > >>> > > > > > > > > >
> > >>> > > > > > > > > > Since the KafkaPrincipal builder serializability
> is a
> > >>> > binary
> > >>> > > > > > > > incompatible
> > >>> > > > > > > > > > change, I propose (also stated in the KIP) the
> > >>> following
> > >>> > > > > > > implementation
> > >>> > > > > > > > > > plan:
> > >>> > > > > > > > > >
> > >>> > > > > > > > > > 1. For next *2.x* release:
> > >>> > > > > > > > > > 1. Get new admin client forwarding changes
> > >>> > > > > > > > > > 2. Get the Envelope RPC implementation
> > >>> > > > > > > > > > 3. Get the forwarding path working and
> validate
> > >>> the
> > >>> > > > > function
> > >>> > > > > > > with
> > >>> > > > > > > > > > fake principals in testing environment,
> without
> > >>> > actual
> > >>> > > > > > > triggering
> > >>> > > > > > > > > in
> > >>> > > > > > > > > > the
> > >>> > > > > > > > > > production system
> > >>> > > > > > > > > > 2. For next *3.0 *release:
> > >>> > > > > > > > > > 1. Introduce serializability to
> > PrincipalBuilder
> > >>> > > > > > > > > > 2. Turn on forwarding path in production and
> > >>> perform
> > >>> > > > > > end-to-end
> > >>> > > > > > > > > > testing
> > >>> > > > > > > > > >
> > >>> > > > > > > > > >
> > >>> > > > > > > > > > I think this is the first time we need to
> introduce a
> > >>> KIP
> > >>> > > > without
> > >>> > > > > > > > having
> > >>> > > > > > > > > it
> > >>> > > > > > > > > > fully accepted in next release. Let me know if this
> > >>> sounds
> > >>> > > > > > > reasonable.
> > >>> > > > > > > > > >
> > >>> > > > > > > > > > On Fri, Apr 24, 2020 at 1:00 AM Sönke Liebau
> > >>> > > > > > > > > > <soenke.lie...@opencore.com.invalid> wrote:
> > >>> > > > > > > > > >
> > >>> > > > > > > > > > > After thinking on this a little bit, maybe this
> > >>> would be
> > >>> > an
> > >>> > > > > > option:
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > > > add default methods serialize and deserialize to
> > the
> > >>> > > > > > > > > > KafkaPrincipalBuilder
> > >>> > > > > > > > > > > interface, these could be very short:
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > > > default String serialize(KafkaPrincipal
> principal)
> > {
> > >>> > > > > > > > > > > return principal.toString();
> > >>> > > > > > > > > > > }
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > > > default KafkaPrincipal deserialize(String
> > >>> > principalString)
> > >>> > > {
> > >>> > > > > > > > > > > return
> > >>> > > > SecurityUtils.parseKafkaPrincipal(principalString);
> > >>> > > > > > > > > > > }
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > > > This would mean that all existing implementations
> > of
> > >>> that
> > >>> > > > > > interface
> > >>> > > > > > > > > > > are unaffected, as this code is pretty much what
> is
> > >>> > > currently
> > >>> > > > > > being
> > >>> > > > > > > > > > > used when their principals need to be serialized.
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > > > But it offers people using custom principals the
> > >>> chance
> > >>> > to
> > >>> > > > > > override
> > >>> > > > > > > > > > > these methods and ensure that all information
> gets
> > >>> > > serialized
> > >>> > > > > for
> > >>> > > > > > > > > > > delegation tokens or request forwarding.
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > > > Wherever we need to de/serialize principals (for
> > >>> example
> > >>> > in
> > >>> > > > the
> > >>> > > > > > > > > > > DelegationTokenManager [1]) we obtain an instance
> > of
> > >>> the
> > >>> > > > > > configured
> > >>> > > > > > > > > > > PrincipalBuilder class and use that to do the
> > actual
> > >>> > work.
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > > > What do you think?
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > > > Best regards,
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > > > Sönke
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > > > [1]
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > >
> > >>> > > > > > > > >
> > >>> > > > > > > >
> > >>> > > > > > >
> > >>> > > > > >
> > >>> > > > >
> > >>> > > >
> > >>> > >
> > >>> >
> > >>>
> >
> https://github.com/apache/kafka/blob/33d06082117d971cdcddd4f01392006b543f3c01/core/src/main/scala/kafka/server/DelegationTokenManager.scala#L122
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > > > On Thu, 23 Apr 2020 at 17:42, Boyang Chen <
> > >>> > > > > > > > reluctanthero...@gmail.com>
> > >>> > > > > > > > > > > wrote:
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > > > > Thanks all,
> > >>> > > > > > > > > > > >
> > >>> > > > > > > > > > > > IIUC, the necessity of doing the audit log on
> the
> > >>> > > > controller
> > >>> > > > > > side
> > >>> > > > > > > > is
> > >>> > > > > > > > > > > > because we need to make sure the authorized
> > >>> resource
> > >>> > > > > > > modifications
> > >>> > > > > > > > > > > > eventually arrive on the target broker side,
> but
> > is
> > >>> > that
> > >>> > > > > really
> > >>> > > > > > > > > > > necessary?
> > >>> > > > > > > > > > > >
> > >>> > > > > > > > > > > > I'm thinking the possibility of doing the audit
> > >>> log on
> > >>> > > the
> > >>> > > > > > > > forwarding
> > >>> > > > > > > > > > > > broker side, which could simplify the
> discussion
> > of
> > >>> > > > principal
> > >>> > > > > > > > > > > serialization
> > >>> > > > > > > > > > > > here. The other option I could think of is to
> > >>> serialize
> > >>> > > the
> > >>> > > > > > > entire
> > >>> > > > > > > > > > audit
> > >>> > > > > > > > > > > > log message if we were supposed to approve, and
> > >>> pass it
> > >>> > > as
> > >>> > > > > part
> > >>> > > > > > > of
> > >>> > > > > > > > > the
> > >>> > > > > > > > > > > > Envelope.
> > >>> > > > > > > > > > > >
> > >>> > > > > > > > > > > > Let me know if you think either of these
> > approaches
> > >>> > would
> > >>> > > > > work.
> > >>> > > > > > > > > > > >
> > >>> > > > > > > > > > > > On Thu, Apr 23, 2020 at 7:01 AM Sönke Liebau
> > >>> > > > > > > > > > > > <soenke.lie...@opencore.com.invalid> wrote:
> > >>> > > > > > > > > > > >
> > >>> > > > > > > > > > > > > I agree that this would be useful to have and
> > >>> > shouldn't
> > >>> > > > > > create
> > >>> > > > > > > > > issues
> > >>> > > > > > > > > > > in
> > >>> > > > > > > > > > > > > 99% of all cases. But it would be a breaking
> > >>> change
> > >>> > to
> > >>> > > a
> > >>> > > > > > public
> > >>> > > > > > > > > API.
> > >>> > > > > > > > > > > > > I had a quick look at the two large projects
> > that
> > >>> > come
> > >>> > > to
> > >>> > > > > > mind
> > >>> > > > > > > > > which
> > >>> > > > > > > > > > > > might
> > >>> > > > > > > > > > > > > be affected: Ranger and Sentry - both seem to
> > >>> operate
> > >>> > > > > > directly
> > >>> > > > > > > > with
> > >>> > > > > > > > > > > > > KafkaPrincipal instead of subclassing it. But
> > >>> anybody
> > >>> > > who
> > >>> > > > > > > > > > > > > extended KafkaPrincipal would probably need
> to
> > >>> update
> > >>> > > > their
> > >>> > > > > > > > code..
> > >>> > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > Writing this sparked the thought that this
> > issue
> > >>> > should
> > >>> > > > > also
> > >>> > > > > > > > > concern
> > >>> > > > > > > > > > > > > delegation tokens, as Principals need to be
> > >>> > stored/sent
> > >>> > > > > > around
> > >>> > > > > > > > for
> > >>> > > > > > > > > > > those
> > >>> > > > > > > > > > > > > too.
> > >>> > > > > > > > > > > > > Had a brief look at the code and for
> Delegation
> > >>> > Tokens
> > >>> > > we
> > >>> > > > > > seem
> > >>> > > > > > > to
> > >>> > > > > > > > > use
> > >>> > > > > > > > > > > > > SecurityUtils#parseKafkaPrincipal[1] which
> > would
> > >>> > ignore
> > >>> > > > any
> > >>> > > > > > > > > > additional
> > >>> > > > > > > > > > > > > information from custom Principals.
> > >>> > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > We'll probably want to at least add a note on
> > >>> that to
> > >>> > > the
> > >>> > > > > > docs
> > >>> > > > > > > -
> > >>> > > > > > > > > > unless
> > >>> > > > > > > > > > > > it
> > >>> > > > > > > > > > > > > is there already, I've only looked for about
> 30
> > >>> > > seconds..
> > >>> > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > Best regards,
> > >>> > > > > > > > > > > > > Sönke
> > >>> > > > > > > > > > > > >
> > >>> > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > [1]
> > >>> > > > > > > > > > > > >
> > >>> > > > > > > > > > > > >
> > >>> > > > > > > > > > > >
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > >
> > >>> > > > > > > > >
> > >>> > > > > > > >
> > >>> > > > > > >
> > >>> > > > > >
> > >>> > > > >
> > >>> > > >
> > >>> > >
> > >>> >
> > >>>
> >
> https://github.com/apache/kafka/blob/e9fcfe4fb7b9ae2f537ce355fe2ab51a58034c64/clients/src/main/java/org/apache/kafka/common/utils/SecurityUtils.java#L52
> > >>> > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > On Thu, 23 Apr 2020 at 14:35, Colin McCabe <
> > >>> > > > > > cmcc...@apache.org
> > >>> > > > > > > >
> > >>> > > > > > > > > > wrote:
> > >>> > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > Hmm... Maybe we need to add some way to
> > >>> serialize
> > >>> > and
> > >>> > > > > > > > deserialize
> > >>> > > > > > > > > > > > > > KafkaPrincipal subclasses to/from string.
> We
> > >>> could
> > >>> > > > add a
> > >>> > > > > > > > method
> > >>> > > > > > > > > to
> > >>> > > > > > > > > > > > > > KafkaPrincipalBuilder#deserialize and a
> > method
> > >>> > > > > > > > > > > > KafkaPrincipal#serialize,
> > >>> > > > > > > > > > > > > I
> > >>> > > > > > > > > > > > > > suppose.
> > >>> > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > best,
> > >>> > > > > > > > > > > > > > Colin
> > >>> > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > On Thu, Apr 23, 2020, at 02:14, Tom Bentley
> > >>> wrote:
> > >>> > > > > > > > > > > > > > > Hi folks,
> > >>> > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > Colin wrote:
> > >>> > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > The final broker knows it can trust the
> > >>> > principal
> > >>> > > > > name
> > >>> > > > > > in
> > >>> > > > > > > > the
> > >>> > > > > > > > > > > > > envelope
> > >>> > > > > > > > > > > > > > > > (since EnvelopeRequest requires
> > >>> CLUSTERACTION
> > >>> > on
> > >>> > > > > > > CLUSTER).
> > >>> > > > > > > > > So
> > >>> > > > > > > > > > it
> > >>> > > > > > > > > > > > can
> > >>> > > > > > > > > > > > > > use
> > >>> > > > > > > > > > > > > > > > that principal name for authorization
> > >>> (given
> > >>> > who
> > >>> > > > you
> > >>> > > > > > are,
> > >>> > > > > > > > > what
> > >>> > > > > > > > > > > can
> > >>> > > > > > > > > > > > > you
> > >>> > > > > > > > > > > > > > > > do?) The forwarded principal name will
> > >>> also be
> > >>> > > > used
> > >>> > > > > > for
> > >>> > > > > > > > > > logging.
> > >>> > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > My understanding (and I'm happy to be
> > >>> corrected)
> > >>> > is
> > >>> > > > > that
> > >>> > > > > > a
> > >>> > > > > > > > > custom
> > >>> > > > > > > > > > > > > > > authoriser might rely on the
> KafkaPrincipal
> > >>> > > instance
> > >>> > > > > > being
> > >>> > > > > > > a
> > >>> > > > > > > > > > > subclass
> > >>> > > > > > > > > > > > > of
> > >>> > > > > > > > > > > > > > > KafkaPrincipal (e.g. the subclass has
> extra
> > >>> > fields
> > >>> > > > with
> > >>> > > > > > the
> > >>> > > > > > > > > > > > principal's
> > >>> > > > > > > > > > > > > > > "roles"). So you can't construct a
> > >>> KafkaPrinicpal
> > >>> > > on
> > >>> > > > > the
> > >>> > > > > > > > > > controller
> > >>> > > > > > > > > > > > > which
> > >>> > > > > > > > > > > > > > > would be guaranteed to work for arbitrary
> > >>> > > > authorizers.
> > >>> > > > > > You
> > >>> > > > > > > > have
> > >>> > > > > > > > > > to
> > >>> > > > > > > > > > > > > > perform
> > >>> > > > > > > > > > > > > > > authorization on the first broker
> > (rejecting
> > >>> some
> > >>> > > of
> > >>> > > > > the
> > >>> > > > > > > > > batched
> > >>> > > > > > > > > > > > > > requests),
> > >>> > > > > > > > > > > > > > > forward the authorized ones to the
> > >>> controller,
> > >>> > > which
> > >>> > > > > then
> > >>> > > > > > > has
> > >>> > > > > > > > > to
> > >>> > > > > > > > > > > > trust
> > >>> > > > > > > > > > > > > > that
> > >>> > > > > > > > > > > > > > > the authorization has been done and make
> > the
> > >>> ZK
> > >>> > > > writes
> > >>> > > > > > > > without
> > >>> > > > > > > > > > > > > > > authorization. Because the controller
> > cannot
> > >>> > invoke
> > >>> > > > the
> > >>> > > > > > > > > > authorizer
> > >>> > > > > > > > > > > > that
> > >>> > > > > > > > > > > > > > > means that the authorizer audit logging
> (on
> > >>> the
> > >>> > > > > > controller)
> > >>> > > > > > > > > would
> > >>> > > > > > > > > > > not
> > >>> > > > > > > > > > > > > > > include these operations. But they would
> be
> > >>> in
> > >>> > the
> > >>> > > > > audit
> > >>> > > > > > > > > logging
> > >>> > > > > > > > > > > from
> > >>> > > > > > > > > > > > > the
> > >>> > > > > > > > > > > > > > > original broker, so the information would
> > >>> not be
> > >>> > > > lost.
> > >>> > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > There's also a problem with using the
> > >>> constructed
> > >>> > > > > > principal
> > >>> > > > > > > > for
> > >>> > > > > > > > > > > other
> > >>> > > > > > > > > > > > > > > logging (i.e. non authorizer logging) on
> > the
> > >>> > > > > controller:
> > >>> > > > > > > > > There's
> > >>> > > > > > > > > > > > > nothing
> > >>> > > > > > > > > > > > > > > stopping a custom KafkaPrincipal subclass
> > >>> from
> > >>> > > > > overriding
> > >>> > > > > > > > > > > toString()
> > >>> > > > > > > > > > > > to
> > >>> > > > > > > > > > > > > > > return something different from
> > >>> > "${type}:${name}".
> > >>> > > If
> > >>> > > > > > > you're
> > >>> > > > > > > > > > > > building a
> > >>> > > > > > > > > > > > > > > "fake" KafkaPrincipal on the controller
> > from
> > >>> the
> > >>> > > type
> > >>> > > > > and
> > >>> > > > > > > > name
> > >>> > > > > > > > > > then
> > >>> > > > > > > > > > > > its
> > >>> > > > > > > > > > > > > > > toString() would be "wrong". A solution
> to
> > >>> this
> > >>> > > would
> > >>> > > > > be
> > >>> > > > > > to
> > >>> > > > > > > > > also
> > >>> > > > > > > > > > > > > > serialize
> > >>> > > > > > > > > > > > > > > the toString() into the envelope and have
> > >>> some
> > >>> > > > > > > > > > > ProxiedKafkaPrincipal
> > >>> > > > > > > > > > > > > > class
> > >>> > > > > > > > > > > > > > > which you instantiate on the controller
> > >>> which has
> > >>> > > > > > > overridden
> > >>> > > > > > > > > > > toString
> > >>> > > > > > > > > > > > > to
> > >>> > > > > > > > > > > > > > > return the right value. Obviously you
> could
> > >>> > > optimize
> > >>> > > > > this
> > >>> > > > > > > > using
> > >>> > > > > > > > > > an
> > >>> > > > > > > > > > > > > > optional
> > >>> > > > > > > > > > > > > > > field so you only serialize the
> toString()
> > >>> if it
> > >>> > > > > differed
> > >>> > > > > > > > from
> > >>> > > > > > > > > > the
> > >>> > > > > > > > > > > > > value
> > >>> > > > > > > > > > > > > > > you'd get from KafkaPrincipal.toString().
> > >>> Maybe
> > >>> > > > > non-audit
> > >>> > > > > > > > > logging
> > >>> > > > > > > > > > > > using
> > >>> > > > > > > > > > > > > > the
> > >>> > > > > > > > > > > > > > > wrong string value of a principal is
> > >>> sufficiently
> > >>> > > > minor
> > >>> > > > > > > that
> > >>> > > > > > > > > this
> > >>> > > > > > > > > > > > isn't
> > >>> > > > > > > > > > > > > > > even worth trying to solve.
> > >>> > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > Kind regards,
> > >>> > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > Tom
> > >>> > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > On Wed, Apr 22, 2020 at 10:59 PM Sönke
> > Liebau
> > >>> > > > > > > > > > > > > > > <soenke.lie...@opencore.com.invalid>
> > wrote:
> > >>> > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > Hi Colin,
> > >>> > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > thanks for your summary! Just one
> > question
> > >>> -
> > >>> > and
> > >>> > > I
> > >>> > > > > may
> > >>> > > > > > be
> > >>> > > > > > > > > > missing
> > >>> > > > > > > > > > > > an
> > >>> > > > > > > > > > > > > > > > obvious point here..
> > >>> > > > > > > > > > > > > > > > You write:
> > >>> > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > "The initial broker should do
> > >>> authentication
> > >>> > (who
> > >>> > > > are
> > >>> > > > > > > you?)
> > >>> > > > > > > > > and
> > >>> > > > > > > > > > > > come
> > >>> > > > > > > > > > > > > up
> > >>> > > > > > > > > > > > > > > > with a principal name. Then it creates
> > an
> > >>> > > envelope
> > >>> > > > > > > > request,
> > >>> > > > > > > > > > > which
> > >>> > > > > > > > > > > > > will
> > >>> > > > > > > > > > > > > > > > contain that principal name, and sends
> it
> > >>> along
> > >>> > > > with
> > >>> > > > > > the
> > >>> > > > > > > > > > > unmodified
> > >>> > > > > > > > > > > > > > > > original request to the final broker.
> > >>> [... ]
> > >>> > > The
> > >>> > > > > > final
> > >>> > > > > > > > > broker
> > >>> > > > > > > > > > > > knows
> > >>> > > > > > > > > > > > > > it
> > >>> > > > > > > > > > > > > > > > can trust the principal name in the
> > >>> envelope
> > >>> > > (since
> > >>> > > > > > > > > > > EnvelopeRequest
> > >>> > > > > > > > > > > > > > > > requires CLUSTERACTION on CLUSTER). So
> > it
> > >>> can
> > >>> > > use
> > >>> > > > > that
> > >>> > > > > > > > > > principal
> > >>> > > > > > > > > > > > > name
> > >>> > > > > > > > > > > > > > for
> > >>> > > > > > > > > > > > > > > > authorization (given who you are, what
> > can
> > >>> you
> > >>> > > > do?) "
> > >>> > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > My understanding is, that you don't
> want
> > to
> > >>> > > > serialize
> > >>> > > > > > the
> > >>> > > > > > > > > > > Principal
> > >>> > > > > > > > > > > > > > (due to
> > >>> > > > > > > > > > > > > > > > the discussed issues with custom
> > >>> principals)
> > >>> > but
> > >>> > > > > reduce
> > >>> > > > > > > the
> > >>> > > > > > > > > > > > principal
> > >>> > > > > > > > > > > > > > down
> > >>> > > > > > > > > > > > > > > > to a string representation that would
> be
> > >>> used
> > >>> > for
> > >>> > > > > > logging
> > >>> > > > > > > > and
> > >>> > > > > > > > > > > > > > > > authorization?
> > >>> > > > > > > > > > > > > > > > If that understanding is correct then I
> > >>> don't
> > >>> > > think
> > >>> > > > > we
> > >>> > > > > > > > could
> > >>> > > > > > > > > > use
> > >>> > > > > > > > > > > > the
> > >>> > > > > > > > > > > > > > > > regular Authorizer on the target
> broker,
> > >>> > because
> > >>> > > > that
> > >>> > > > > > > would
> > >>> > > > > > > > > > need
> > >>> > > > > > > > > > > > the
> > >>> > > > > > > > > > > > > > actual
> > >>> > > > > > > > > > > > > > > > principal object to work on.
> > >>> > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > Also, a thought that just occurred to
> me,
> > >>> we
> > >>> > > might
> > >>> > > > > > > actually
> > >>> > > > > > > > > > need
> > >>> > > > > > > > > > > to
> > >>> > > > > > > > > > > > > log
> > >>> > > > > > > > > > > > > > > > different principal strings for the
> case
> > of
> > >>> > > queries
> > >>> > > > > > like
> > >>> > > > > > > > > > > > AlterConfigs
> > >>> > > > > > > > > > > > > > > > (mentioned by Rajini) which may contain
> > >>> > multiple
> > >>> > > > > > > resources.
> > >>> > > > > > > > > > Take
> > >>> > > > > > > > > > > an
> > >>> > > > > > > > > > > > > > LDAP
> > >>> > > > > > > > > > > > > > > > authorizer that grants access based on
> > >>> group
> > >>> > > > > > membership -
> > >>> > > > > > > > the
> > >>> > > > > > > > > > > same
> > >>> > > > > > > > > > > > > > > > alterconfig request may contain
> resources
> > >>> that
> > >>> > > are
> > >>> > > > > > > > authorized
> > >>> > > > > > > > > > > based
> > >>> > > > > > > > > > > > > on
> > >>> > > > > > > > > > > > > > > > group1 as well as resources authorized
> > >>> based on
> > >>> > > > > > > membership
> > >>> > > > > > > > in
> > >>> > > > > > > > > > > > group 2
> > >>> > > > > > > > > > > > > > ..
> > >>> > > > > > > > > > > > > > > > And in all cases we'd need to log the
> > >>> specific
> > >>> > > > > reason I
> > >>> > > > > > > > > think..
> > >>> > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > Basically I think that we might have a
> > hard
> > >>> > time
> > >>> > > > > > properly
> > >>> > > > > > > > > > > > authorizing
> > >>> > > > > > > > > > > > > > and
> > >>> > > > > > > > > > > > > > > > logging without being able to forward
> the
> > >>> > entire
> > >>> > > > > > > > principal..
> > >>> > > > > > > > > > but
> > >>> > > > > > > > > > > > > > again, I
> > >>> > > > > > > > > > > > > > > > might be heading down an entirely wrong
> > >>> path
> > >>> > here
> > >>> > > > :)
> > >>> > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > Best regards,
> > >>> > > > > > > > > > > > > > > > Sönke
> > >>> > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > On Wed, 22 Apr 2020 at 23:13, Guozhang
> > >>> Wang <
> > >>> > > > > > > > > > wangg...@gmail.com>
> > >>> > > > > > > > > > > > > > wrote:
> > >>> > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > Colin, Boyang: thanks for the
> updates,
> > I
> > >>> > agree
> > >>> > > > that
> > >>> > > > > > an
> > >>> > > > > > > > > > > > > > EnvelopeRequest
> > >>> > > > > > > > > > > > > > > > > would be a less vulnerable approach
> > than
> > >>> > > optional
> > >>> > > > > > > fields,
> > >>> > > > > > > > > and
> > >>> > > > > > > > > > > I'm
> > >>> > > > > > > > > > > > > > just
> > >>> > > > > > > > > > > > > > > > > wondering if we would keep the
> > >>> > EnvelopeRequest
> > >>> > > > for
> > >>> > > > > a
> > >>> > > > > > > long
> > >>> > > > > > > > > > > time. I
> > >>> > > > > > > > > > > > > was
> > >>> > > > > > > > > > > > > > > > > thinking that, potentially if we
> > require
> > >>> > > clients
> > >>> > > > to
> > >>> > > > > > be
> > >>> > > > > > > on
> > >>> > > > > > > > > > newer
> > >>> > > > > > > > > > > > > > version
> > >>> > > > > > > > > > > > > > > > > when talking to a 3.0+ (assuming 3.0
> is
> > >>> the
> > >>> > > > bridge
> > >>> > > > > > > > release)
> > >>> > > > > > > > > > > > > brokers,
> > >>> > > > > > > > > > > > > > then
> > >>> > > > > > > > > > > > > > > > > we do not need to keep this code for
> > too
> > >>> > long,
> > >>> > > > but
> > >>> > > > > I
> > >>> > > > > > > > think
> > >>> > > > > > > > > > that
> > >>> > > > > > > > > > > > > > would be
> > >>> > > > > > > > > > > > > > > > a
> > >>> > > > > > > > > > > > > > > > > very hasty compatibility breaking so
> > >>> maybe we
> > >>> > > > > indeed
> > >>> > > > > > > need
> > >>> > > > > > > > > to
> > >>> > > > > > > > > > > keep
> > >>> > > > > > > > > > > > > > this
> > >>> > > > > > > > > > > > > > > > > forwarding mechanism many years.
> > >>> > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > Regarding future use cases, I think
> the
> > >>> > example
> > >>> > > > > that
> > >>> > > > > > > > Boyang
> > >>> > > > > > > > > > > > > > mentioned may
> > >>> > > > > > > > > > > > > > > > > not be very practical honestly,
> because
> > >>> when
> > >>> > > > > there's
> > >>> > > > > > a
> > >>> > > > > > > > > > > > connectivity
> > >>> > > > > > > > > > > > > > > > issue,
> > >>> > > > > > > > > > > > > > > > > it is either a network partition
> > between
> > >>> > > > > "controller,
> > >>> > > > > > > A |
> > >>> > > > > > > > > B",
> > >>> > > > > > > > > > > or
> > >>> > > > > > > > > > > > > > > > > "controller | A, B". In other words,
> if
> > >>> the
> > >>> > > > > > controller
> > >>> > > > > > > > can
> > >>> > > > > > > > > > talk
> > >>> > > > > > > > > > > > to
> > >>> > > > > > > > > > > > > A,
> > >>> > > > > > > > > > > > > > > > then
> > >>> > > > > > > > > > > > > > > > > very likely A would not be able to
> talk
> > >>> to B
> > >>> > > > > > either...
> > >>> > > > > > > > > > anyways,
> > >>> > > > > > > > > > > > > > since the
> > >>> > > > > > > > > > > > > > > > > forwarding would be there for a
> > >>> sufficiently
> > >>> > > long
> > >>> > > > > > > time, I
> > >>> > > > > > > > > > think
> > >>> > > > > > > > > > > > > > keeping
> > >>> > > > > > > > > > > > > > > > the
> > >>> > > > > > > > > > > > > > > > > additional envelope makes sense.
> > >>> > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > Guozhang
> > >>> > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > On Wed, Apr 22, 2020 at 1:47 PM
> Boyang
> > >>> Chen <
> > >>> > > > > > > > > > > > > > reluctanthero...@gmail.com>
> > >>> > > > > > > > > > > > > > > > > wrote:
> > >>> > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > > Thanks Colin for the summary! And
> > >>> Guozhang,
> > >>> > > > > > regarding
> > >>> > > > > > > > the
> > >>> > > > > > > > > > > > future
> > >>> > > > > > > > > > > > > > use
> > >>> > > > > > > > > > > > > > > > > cases,
> > >>> > > > > > > > > > > > > > > > > > consider a scenario where there are
> > >>> > temporary
> > >>> > > > > > > > > connectivity
> > >>> > > > > > > > > > > > issue
> > >>> > > > > > > > > > > > > > > > between
> > >>> > > > > > > > > > > > > > > > > > controller to a fellow broker A,
> the
> > >>> > > controller
> > >>> > > > > > could
> > >>> > > > > > > > > then
> > >>> > > > > > > > > > > > > leverage
> > >>> > > > > > > > > > > > > > > > > another
> > >>> > > > > > > > > > > > > > > > > > healthy broker B to do a forwarding
> > >>> request
> > >>> > > to
> > >>> > > > A
> > >>> > > > > in
> > >>> > > > > > > > order
> > >>> > > > > > > > > > to
> > >>> > > > > > > > > > > > > > maintain a
> > >>> > > > > > > > > > > > > > > > > > communication.
> > >>> > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > > Even for KIP-590 scope, the
> > forwarding
> > >>> > > > mechanism
> > >>> > > > > > > shall
> > >>> > > > > > > > > not
> > >>> > > > > > > > > > be
> > >>> > > > > > > > > > > > > > transit
> > >>> > > > > > > > > > > > > > > > as
> > >>> > > > > > > > > > > > > > > > > we
> > >>> > > > > > > > > > > > > > > > > > do need to support older version of
> > >>> admin
> > >>> > > > clients
> > >>> > > > > > > for a
> > >>> > > > > > > > > > > couple
> > >>> > > > > > > > > > > > of
> > >>> > > > > > > > > > > > > > years
> > >>> > > > > > > > > > > > > > > > > > IIUC.
> > >>> > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > > Boyang
> > >>> > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > > On Wed, Apr 22, 2020 at 1:29 PM
> Colin
> > >>> > McCabe
> > >>> > > <
> > >>> > > > > > > > > > > > cmcc...@apache.org
> > >>> > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > wrote:
> > >>> > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > > > Hi all,
> > >>> > > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > > > I guess the way I see this
> working
> > is
> > >>> > that
> > >>> > > > the
> > >>> > > > > > > > request
> > >>> > > > > > > > > > gets
> > >>> > > > > > > > > > > > > sent
> > >>> > > > > > > > > > > > > > from
> > >>> > > > > > > > > > > > > > > > > the
> > >>> > > > > > > > > > > > > > > > > > > client, to the initial broker,
> and
> > >>> then
> > >>> > > > > forwarded
> > >>> > > > > > > to
> > >>> > > > > > > > > the
> > >>> > > > > > > > > > > > final
> > >>> > > > > > > > > > > > > > > > broker.
> > >>> > > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > > > The initial broker should do
> > >>> > authentication
> > >>> > > > > (who
> > >>> > > > > > > are
> > >>> > > > > > > > > > you?)
> > >>> > > > > > > > > > > > and
> > >>> > > > > > > > > > > > > > come
> > >>> > > > > > > > > > > > > > > > up
> > >>> > > > > > > > > > > > > > > > > > > with a principal name. Then it
> > >>> creates
> > >>> > an
> > >>> > > > > > envelope
> > >>> > > > > > > > > > > request,
> > >>> > > > > > > > > > > > > > which
> > >>> > > > > > > > > > > > > > > > will
> > >>> > > > > > > > > > > > > > > > > > > contain that principal name, and
> > >>> sends it
> > >>> > > > along
> > >>> > > > > > > with
> > >>> > > > > > > > > the
> > >>> > > > > > > > > > > > > > unmodified
> > >>> > > > > > > > > > > > > > > > > > > original request to the final
> > >>> broker. (I
> > >>> > > > agree
> > >>> > > > > > > with
> > >>> > > > > > > > > Tom
> > >>> > > > > > > > > > > and
> > >>> > > > > > > > > > > > > > Rajini
> > >>> > > > > > > > > > > > > > > > > that
> > >>> > > > > > > > > > > > > > > > > > we
> > >>> > > > > > > > > > > > > > > > > > > can't forward the information
> > needed
> > >>> to
> > >>> > do
> > >>> > > > > > > > > authentication
> > >>> > > > > > > > > > > on
> > >>> > > > > > > > > > > > > the
> > >>> > > > > > > > > > > > > > > > final
> > >>> > > > > > > > > > > > > > > > > > > broker, but I also think we don't
> > >>> need
> > >>> > to,
> > >>> > > > > since
> > >>> > > > > > we
> > >>> > > > > > > > can
> > >>> > > > > > > > > > do
> > >>> > > > > > > > > > > it
> > >>> > > > > > > > > > > > > on
> > >>> > > > > > > > > > > > > > the
> > >>> > > > > > > > > > > > > > > > > > > initial broker.)
> > >>> > > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > > > The final broker knows it can
> trust
> > >>> the
> > >>> > > > > principal
> > >>> > > > > > > > name
> > >>> > > > > > > > > in
> > >>> > > > > > > > > > > the
> > >>> > > > > > > > > > > > > > > > envelope
> > >>> > > > > > > > > > > > > > > > > > > (since EnvelopeRequest requires
> > >>> > > CLUSTERACTION
> > >>> > > > > on
> > >>> > > > > > > > > > CLUSTER).
> > >>> > > > > > > > > > > > So
> > >>> > > > > > > > > > > > > > it can
> > >>> > > > > > > > > > > > > > > > > use
> > >>> > > > > > > > > > > > > > > > > > > that principal name for
> > authorization
> > >>> > > (given
> > >>> > > > > who
> > >>> > > > > > > you
> > >>> > > > > > > > > are,
> > >>> > > > > > > > > > > > what
> > >>> > > > > > > > > > > > > > can
> > >>> > > > > > > > > > > > > > > > you
> > >>> > > > > > > > > > > > > > > > > > > do?) The forwarded principal
> name
> > >>> will
> > >>> > > also
> > >>> > > > be
> > >>> > > > > > > used
> > >>> > > > > > > > > for
> > >>> > > > > > > > > > > > > logging.
> > >>> > > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > > > One question is why we need an
> > >>> > > > EnvelopeRequest.
> > >>> > > > > > > > Well,
> > >>> > > > > > > > > if
> > >>> > > > > > > > > > > we
> > >>> > > > > > > > > > > > > > don't
> > >>> > > > > > > > > > > > > > > > have
> > >>> > > > > > > > > > > > > > > > > > an
> > >>> > > > > > > > > > > > > > > > > > > EnvelopeRequest, we need
> somewhere
> > >>> else
> > >>> > to
> > >>> > > > put
> > >>> > > > > > the
> > >>> > > > > > > > > > > forwarded
> > >>> > > > > > > > > > > > > > > > principal
> > >>> > > > > > > > > > > > > > > > > > > name. I don't think overriding
> an
> > >>> > existing
> > >>> > > > > field
> > >>> > > > > > > > (like
> > >>> > > > > > > > > > > > > > clientId) is
> > >>> > > > > > > > > > > > > > > > a
> > >>> > > > > > > > > > > > > > > > > > good
> > >>> > > > > > > > > > > > > > > > > > > option for this. It's messy, and
> > >>> loses
> > >>> > > > > > > information.
> > >>> > > > > > > > > It
> > >>> > > > > > > > > > > also
> > >>> > > > > > > > > > > > > > raises
> > >>> > > > > > > > > > > > > > > > > the
> > >>> > > > > > > > > > > > > > > > > > > question of how the final broker
> > >>> knows
> > >>> > that
> > >>> > > > the
> > >>> > > > > > > > > clientId
> > >>> > > > > > > > > > in
> > >>> > > > > > > > > > > > the
> > >>> > > > > > > > > > > > > > > > > received
> > >>> > > > > > > > > > > > > > > > > > > message is not "really" a
> clientId,
> > >>> but
> > >>> > is
> > >>> > > a
> > >>> > > > > > > > principal
> > >>> > > > > > > > > > > name.
> > >>> > > > > > > > > > > > > > Without
> > >>> > > > > > > > > > > > > > > > > an
> > >>> > > > > > > > > > > > > > > > > > > envelope, there's no way to
> clearly
> > >>> mark
> > >>> > a
> > >>> > > > > > request
> > >>> > > > > > > as
> > >>> > > > > > > > > > > > > forwarded,
> > >>> > > > > > > > > > > > > > so
> > >>> > > > > > > > > > > > > > > > > > there's
> > >>> > > > > > > > > > > > > > > > > > > no reason for the final broker to
> > >>> treat
> > >>> > > this
> > >>> > > > > > > > > differently
> > >>> > > > > > > > > > > > than a
> > >>> > > > > > > > > > > > > > > > regular
> > >>> > > > > > > > > > > > > > > > > > > clientId (or whatever).
> > >>> > > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > > > We talked about using optional
> > >>> fields to
> > >>> > > > > contain
> > >>> > > > > > > the
> > >>> > > > > > > > > > > > forwarded
> > >>> > > > > > > > > > > > > > > > > principal
> > >>> > > > > > > > > > > > > > > > > > > name, but this is also messy and
> > >>> > > potentially
> > >>> > > > > > > > dangerous.
> > >>> > > > > > > > > > > > Older
> > >>> > > > > > > > > > > > > > > > brokers
> > >>> > > > > > > > > > > > > > > > > > will
> > >>> > > > > > > > > > > > > > > > > > > simply ignore the optional
> fields,
> > >>> which
> > >>> > > > could
> > >>> > > > > > > result
> > >>> > > > > > > > > in
> > >>> > > > > > > > > > > them
> > >>> > > > > > > > > > > > > > > > executing
> > >>> > > > > > > > > > > > > > > > > > > operations as the wrong
> principal.
> > >>> Of
> > >>> > > > course,
> > >>> > > > > > this
> > >>> > > > > > > > > would
> > >>> > > > > > > > > > > > > > require a
> > >>> > > > > > > > > > > > > > > > > > > misconfiguration in order to
> > happen,
> > >>> but
> > >>> > it
> > >>> > > > > still
> > >>> > > > > > > > seems
> > >>> > > > > > > > > > > > better
> > >>> > > > > > > > > > > > > > to set
> > >>> > > > > > > > > > > > > > > > > up
> > >>> > > > > > > > > > > > > > > > > > > the system so that this
> > >>> misconfiguration
> > >>> > is
> > >>> > > > > > > detected,
> > >>> > > > > > > > > > > rather
> > >>> > > > > > > > > > > > > than
> > >>> > > > > > > > > > > > > > > > > > silently
> > >>> > > > > > > > > > > > > > > > > > > ignored.
> > >>> > > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > > > It's true that the need for
> > >>> forwarding is
> > >>> > > > > > > "temporary"
> > >>> > > > > > > > > in
> > >>> > > > > > > > > > > some
> > >>> > > > > > > > > > > > > > sense,
> > >>> > > > > > > > > > > > > > > > > > since
> > >>> > > > > > > > > > > > > > > > > > > we only need it for older
> clients.
> > >>> > > However,
> > >>> > > > we
> > >>> > > > > > > will
> > >>> > > > > > > > > want
> > >>> > > > > > > > > > > to
> > >>> > > > > > > > > > > > > > support
> > >>> > > > > > > > > > > > > > > > > > these
> > >>> > > > > > > > > > > > > > > > > > > older clients for many years to
> > come.
> > >>> > > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > > > I agree that the usefulness of
> > >>> > > > EnvelopeRequest
> > >>> > > > > is
> > >>> > > > > > > > > limited
> > >>> > > > > > > > > > > by
> > >>> > > > > > > > > > > > it
> > >>> > > > > > > > > > > > > > > > being a
> > >>> > > > > > > > > > > > > > > > > > > superuser-only request at the
> > moment.
> > >>> > > > Perhaps
> > >>> > > > > > > there
> > >>> > > > > > > > > are
> > >>> > > > > > > > > > > some
> > >>> > > > > > > > > > > > > > changes
> > >>> > > > > > > > > > > > > > > > > to
> > >>> > > > > > > > > > > > > > > > > > > how custom principals work that
> > would
> > >>> > allow
> > >>> > > > us
> > >>> > > > > to
> > >>> > > > > > > get
> > >>> > > > > > > > > > > around
> > >>> > > > > > > > > > > > > > this in
> > >>> > > > > > > > > > > > > > > > > the
> > >>> > > > > > > > > > > > > > > > > > > future. We should think about
> that
> > >>> so
> > >>> > that
> > >>> > > > we
> > >>> > > > > > have
> > >>> > > > > > > > > this
> > >>> > > > > > > > > > > > > > > > functionality
> > >>> > > > > > > > > > > > > > > > > in
> > >>> > > > > > > > > > > > > > > > > > > the future if it's needed.
> > >>> > > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > > > best,
> > >>> > > > > > > > > > > > > > > > > > > Colin
> > >>> > > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > > > On Wed, Apr 22, 2020, at 11:21,
> > >>> Guozhang
> > >>> > > Wang
> > >>> > > > > > > wrote:
> > >>> > > > > > > > > > > > > > > > > > > > Hello Gwen,
> > >>> > > > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > > > > The purpose here is for
> > maintaining
> > >>> > > > > > compatibility
> > >>> > > > > > > > old
> > >>> > > > > > > > > > > > > clients,
> > >>> > > > > > > > > > > > > > who
> > >>> > > > > > > > > > > > > > > > do
> > >>> > > > > > > > > > > > > > > > > > not
> > >>> > > > > > > > > > > > > > > > > > > > have functionality to do
> > re-routing
> > >>> > admin
> > >>> > > > > > > requests
> > >>> > > > > > > > > > > > > themselves.
> > >>> > > > > > > > > > > > > > New
> > >>> > > > > > > > > > > > > > > > > > > clients
> > >>> > > > > > > > > > > > > > > > > > > > can of course do this
> themselves
> > by
> > >>> > > > detecting
> > >>> > > > > > > who's
> > >>> > > > > > > > > the
> > >>> > > > > > > > > > > > > > controller.
> > >>> > > > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > > > > Hello Colin / Boyang,
> > >>> > > > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > > > > Regarding the usage of the
> > >>> envelope,
> > >>> > I'm
> > >>> > > > > > curious
> > >>> > > > > > > > what
> > >>> > > > > > > > > > are
> > >>> > > > > > > > > > > > the
> > >>> > > > > > > > > > > > > > > > > potential
> > >>> > > > > > > > > > > > > > > > > > > > future use cases that would
> > require
> > >>> > > request
> > >>> > > > > > > > > forwarding
> > >>> > > > > > > > > > > and
> > >>> > > > > > > > > > > > > > hence
> > >>> > > > > > > > > > > > > > > > > > envelope
> > >>> > > > > > > > > > > > > > > > > > > > would be useful? Originally I
> > >>> thought
> > >>> > > that
> > >>> > > > > the
> > >>> > > > > > > > > > forwarding
> > >>> > > > > > > > > > > > > > mechanism
> > >>> > > > > > > > > > > > > > > > > is
> > >>> > > > > > > > > > > > > > > > > > > only
> > >>> > > > > > > > > > > > > > > > > > > > temporary as we need it for the
> > >>> bridge
> > >>> > > > > release,
> > >>> > > > > > > and
> > >>> > > > > > > > > > > moving
> > >>> > > > > > > > > > > > > > forward
> > >>> > > > > > > > > > > > > > > > we
> > >>> > > > > > > > > > > > > > > > > > > will
> > >>> > > > > > > > > > > > > > > > > > > > get rid of this to simplify the
> > >>> code
> > >>> > > base.
> > >>> > > > If
> > >>> > > > > > we
> > >>> > > > > > > do
> > >>> > > > > > > > > > have
> > >>> > > > > > > > > > > > > valid
> > >>> > > > > > > > > > > > > > use
> > >>> > > > > > > > > > > > > > > > > > cases
> > >>> > > > > > > > > > > > > > > > > > > in
> > >>> > > > > > > > > > > > > > > > > > > > the future which makes us
> believe
> > >>> that
> > >>> > > > > request
> > >>> > > > > > > > > > forwarding
> > >>> > > > > > > > > > > > > would
> > >>> > > > > > > > > > > > > > > > > > actually
> > >>> > > > > > > > > > > > > > > > > > > be
> > >>> > > > > > > > > > > > > > > > > > > > a permanent feature retained on
> > the
> > >>> > > broker
> > >>> > > > > > side,
> > >>> > > > > > > > I'm
> > >>> > > > > > > > > on
> > >>> > > > > > > > > > > > board
> > >>> > > > > > > > > > > > > > with
> > >>> > > > > > > > > > > > > > > > > > adding
> > >>> > > > > > > > > > > > > > > > > > > > the envelope request protocol.
> > >>> > > > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > > > > Guozhang
> > >>> > > > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > > > > On Wed, Apr 22, 2020 at 8:22 AM
> > >>> Gwen
> > >>> > > > Shapira
> > >>> > > > > <
> > >>> > > > > > > > > > > > > > g...@confluent.io>
> > >>> > > > > > > > > > > > > > > > > > wrote:
> > >>> > > > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > > > > > Hey Boyang,
> > >>> > > > > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > > > > > Sorry if this was already
> > >>> discussed,
> > >>> > > but
> > >>> > > > I
> > >>> > > > > > > didn't
> > >>> > > > > > > > > see
> > >>> > > > > > > > > > > > this
> > >>> > > > > > > > > > > > > as
> > >>> > > > > > > > > > > > > > > > > > rejected
> > >>> > > > > > > > > > > > > > > > > > > > > alternative:
> > >>> > > > > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > > > > > Until now, we always did
> client
> > >>> side
> > >>> > > > > routing
> > >>> > > > > > -
> > >>> > > > > > > > the
> > >>> > > > > > > > > > > client
> > >>> > > > > > > > > > > > > > itself
> > >>> > > > > > > > > > > > > > > > > > found
> > >>> > > > > > > > > > > > > > > > > > > the
> > >>> > > > > > > > > > > > > > > > > > > > > controller via metadata and
> > >>> directed
> > >>> > > > > requests
> > >>> > > > > > > > > > > > accordingly.
> > >>> > > > > > > > > > > > > > > > Brokers
> > >>> > > > > > > > > > > > > > > > > > that
> > >>> > > > > > > > > > > > > > > > > > > > > were not the controller,
> > rejected
> > >>> > those
> > >>> > > > > > > requests.
> > >>> > > > > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > > > > > Why did we decide to move to
> > >>> broker
> > >>> > > side
> > >>> > > > > > > routing?
> > >>> > > > > > > > > Was
> > >>> > > > > > > > > > > the
> > >>> > > > > > > > > > > > > > > > > client-side
> > >>> > > > > > > > > > > > > > > > > > > > > option discussed and rejected
> > >>> > somewhere
> > >>> > > > > and I
> > >>> > > > > > > > > missed
> > >>> > > > > > > > > > > it?
> > >>> > > > > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > > > > > Gwen
> > >>> > > > > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > > > > > On Fri, Apr 3, 2020, 4:45 PM
> > >>> Boyang
> > >>> > > Chen
> > >>> > > > <
> > >>> > > > > > > > > > > > > > > > > reluctanthero...@gmail.com
> > >>> > > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > > > > > wrote:
> > >>> > > > > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > > > > > > Hey all,
> > >>> > > > > > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > > > > > > I would like to start off
> the
> > >>> > > > discussion
> > >>> > > > > > for
> > >>> > > > > > > > > > > KIP-590, a
> > >>> > > > > > > > > > > > > > > > follow-up
> > >>> > > > > > > > > > > > > > > > > > > > > > initiative after KIP-500:
> > >>> > > > > > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > >
> > >>> > > > > > > > > > > >
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > >
> > >>> > > > > > > > >
> > >>> > > > > > > >
> > >>> > > > > > >
> > >>> > > > > >
> > >>> > > > >
> > >>> > > >
> > >>> > >
> > >>> >
> > >>>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-590%3A+Redirect+Zookeeper+Mutation+Protocols+to+The+Controller
> > >>> > > > > > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > > > > > > This KIP proposes to
> migrate
> > >>> > existing
> > >>> > > > > > > Zookeeper
> > >>> > > > > > > > > > > > mutation
> > >>> > > > > > > > > > > > > > paths,
> > >>> > > > > > > > > > > > > > > > > > > including
> > >>> > > > > > > > > > > > > > > > > > > > > > configuration, security and
> > >>> quota
> > >>> > > > > changes,
> > >>> > > > > > to
> > >>> > > > > > > > > > > > > > controller-only
> > >>> > > > > > > > > > > > > > > > by
> > >>> > > > > > > > > > > > > > > > > > > always
> > >>> > > > > > > > > > > > > > > > > > > > > > routing these alterations
> to
> > >>> the
> > >>> > > > > > controller.
> > >>> > > > > > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > > > > > > Let me know your thoughts!
> > >>> > > > > > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > > > > > > Best,
> > >>> > > > > > > > > > > > > > > > > > > > > > Boyang
> > >>> > > > > > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > > > > --
> > >>> > > > > > > > > > > > > > > > > > > > -- Guozhang
> > >>> > > > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > > --
> > >>> > > > > > > > > > > > > > > > > -- Guozhang
> > >>> > > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > > --
> > >>> > > > > > > > > > > > > > > > Sönke Liebau
> > >>> > > > > > > > > > > > > > > > Partner
> > >>> > > > > > > > > > > > > > > > Tel. +49 179 7940878
> > >>> > > > > > > > > > > > > > > > OpenCore GmbH & Co. KG -
> > >>> Thomas-Mann-Straße 8 -
> > >>> > > > 22880
> > >>> > > > > > > > Wedel -
> > >>> > > > > > > > > > > > Germany
> > >>> > > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > >
> > >>> > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > --
> > >>> > > > > > > > > > > > > Sönke Liebau
> > >>> > > > > > > > > > > > > Partner
> > >>> > > > > > > > > > > > > Tel. +49 179 7940878
> > >>> > > > > > > > > > > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße
> 8 -
> > >>> 22880
> > >>> > > > > Wedel -
> > >>> > > > > > > > > Germany
> > >>> > > > > > > > > > > > >
> > >>> > > > > > > > > > > >
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > > > --
> > >>> > > > > > > > > > > Sönke Liebau
> > >>> > > > > > > > > > > Partner
> > >>> > > > > > > > > > > Tel. +49 179 7940878
> > >>> > > > > > > > > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 -
> > 22880
> > >>> > > Wedel -
> > >>> > > > > > > Germany
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > >
> > >>> > > > > > > > >
> > >>> > > > > > > >
> > >>> > > > > > >
> > >>> > > > > > >
> > >>> > > > > > > --
> > >>> > > > > > > Sönke Liebau
> > >>> > > > > > > Partner
> > >>> > > > > > > Tel. +49 179 7940878
> > >>> > > > > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880
> > Wedel -
> > >>> > > Germany
> > >>> > > > > > >
> > >>> > > > > >
> > >>> > > > >
> > >>> > > > >
> > >>> > > > > --
> > >>> > > > > Sönke Liebau
> > >>> > > > > Partner
> > >>> > > > > Tel. +49 179 7940878
> > >>> > > > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel -
> > >>> Germany
> > >>> > > > >
> > >>> > > >
> > >>> > >
> > >>> >
> > >>> >
> > >>> > --
> > >>> > -- Guozhang
> > >>> >
> > >>>
> > >>
> >
>
