Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #778

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-13751) On the broker side, OAUTHBEARER is not compatible with other SASL mechanisms

[RivenSun (Jira)]

RivenSun created KAFKA-13751:


 Summary: On the broker side, OAUTHBEARER is not compatible with 
other SASL mechanisms
 Key: KAFKA-13751
 URL: https://issues.apache.org/jira/browse/KAFKA-13751
 Project: Kafka
  Issue Type: Bug
  Components: security
Affects Versions: 3.0.1
Reporter: RivenSun


h1. Phenomenon:

 SASL/OAUTHBEARER, whether implemented by default or customized by the user, is 
not compatible with other SASL mechanisms.
h3. 
case1:

kafka_server_jaas_oauth.conf
{code:java}
KafkaServer {
  org.apache.kafka.common.security.plain.PlainLoginModule required
  username="admin"
  password="admin"
  user_admin="admin"
  user_alice="alice"; 

   org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required;

   org.apache.kafka.common.security.scram.ScramLoginModule required
  username="admin"
  password="admin_scram";
}; {code}
 server.properties
{code:java}
advertised.listeners=SASL_PLAINTEXT://publicIp:8779,SASL_SSL://publicIp:8889,OAUTH://publicIp:8669
 
listener.security.protocol.map=INTERNAL_SSL:SASL_SSL,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL,OAUTH:SASL_SSL
sasl.enabled.mechanisms=PLAIN,SCRAM-SHA-256,SCRAM-SHA-512,OAUTHBEARER{code}
Error when starting kafka:
server.log
{code:java}
[2022-03-16 13:18:42,658] ERROR [KafkaServer id=1] Fatal error during 
KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.apache.kafka.common.KafkaException: java.lang.IllegalArgumentException: 
Must supply exactly 1 non-null JAAS mechanism configuration (size was 3)
        at 
org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:172)
        at kafka.network.Processor.(SocketServer.scala:724)
        at kafka.network.SocketServer.newProcessor(SocketServer.scala:367)
        at 
kafka.network.SocketServer.$anonfun$addDataPlaneProcessors$1(SocketServer.scala:252)
        at 
kafka.network.SocketServer.addDataPlaneProcessors(SocketServer.scala:251)
        at 
kafka.network.SocketServer.$anonfun$createDataPlaneAcceptorsAndProcessors$1(SocketServer.scala:214)
        at 
kafka.network.SocketServer.$anonfun$createDataPlaneAcceptorsAndProcessors$1$adapted(SocketServer.scala:211)
        at 
scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
        at 
scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
        at 
kafka.network.SocketServer.createDataPlaneAcceptorsAndProcessors(SocketServer.scala:211)
        at kafka.network.SocketServer.startup(SocketServer.scala:122)
        at kafka.server.KafkaServer.startup(KafkaServer.scala:266)
        at 
kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44)
        at kafka.Kafka$.main(Kafka.scala:82)
        at kafka.Kafka.main(Kafka.scala)
Caused by: java.lang.IllegalArgumentException: Must supply exactly 1 non-null 
JAAS mechanism configuration (size was 3)
        at 
org.apache.kafka.common.security.oauthbearer.internals.unsecured.OAuthBearerUnsecuredValidatorCallbackHandler.configure(OAuthBearerUnsecuredValidatorCallbackHandler.java:117)
        at 
org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:139)
        ... 17 more
[2022-03-16 13:18:42,662] INFO [KafkaServer id=1] shutting down 
(kafka.server.KafkaServer)
[2022-03-16 13:18:42,664] INFO [SocketServer brokerId=1] Stopping socket server 
request processors (kafka.network.SocketServer) {code}
The default implementation class of oauthbearer's 
`sasl.server.callback.handler.class` is 
OAuthBearerUnsecuredValidatorCallbackHandler. 
In the OAuthBearerUnsecuredValidatorCallbackHandler#configure(...) method, the 
jaasConfigEntries parameter is verified.
What I want to say is that {*}the verification logic here is completely 
reasonable{*}, but the jaasConfigEntries passed in from the upper layer should 
not contain the AppConfigurationEntry of other loginModules. There are several 
other codes for the check of the same keyword *"Must supply exactly 1 non-null 
JAAS mechanism configuration".*

Rootcause elaborates later.


By the way, at present, KafkaServer allows {*}the same LoginModule to be 
configured multiple times in kafkaJaasConfigFile{*}, which will also lead to 
the phenomenon of case1.

kafka_server_jaas_oauth.conf eg：
{code:java}
KafkaServer {
   org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required;

  org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required;
 
}; {code}
 




h3. case2:
On the basis of case1, modify the default implementation of oauthbearer's 
`sasl.server.callback.handler.class`



 server.properties add new configuration
{code:java}

[jira] [Resolved] (KAFKA-13740) kafka-stream-client-shutdown

[Prashanth Joseph Babu (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prashanth Joseph Babu resolved KAFKA-13740.
---
Resolution: Fixed

> kafka-stream-client-shutdown
> 
>
> Key: KAFKA-13740
> URL: https://issues.apache.org/jira/browse/KAFKA-13740
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 3.1.0
>Reporter: Prashanth Joseph Babu
>Priority: Major
>
> I have an apache kafka streams application . I notice that it sometimes 
> shutsdown when a rebalancing occurs with no real reason for the shutdown . It 
> doesn't even throw an exception.
> Here are some logs on the same 
> {code:java}
> [2022-03-08 17:13:37,024] INFO [Consumer 
> clientId=svc-stream-collector-StreamThread-1-consumer, 
> groupId=svc-stream-collector] Adding newly assigned partitions:  
> (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
> [2022-03-08 17:13:37,024] ERROR stream-thread 
> [svc-stream-collector-StreamThread-1] A Kafka Streams client in this Kafka 
> Streams application is requesting to shutdown the application 
> (org.apache.kafka.streams.processor.internals.StreamThread)
> [2022-03-08 17:13:37,030] INFO stream-client [svc-stream-collector] State 
> transition from REBALANCING to PENDING_ERROR 
> (org.apache.kafka.streams.KafkaStreams)
> old state:REBALANCING new state:PENDING_ERROR
> [2022-03-08 17:13:37,031] INFO [Consumer 
> clientId=svc-stream-collector-StreamThread-1-consumer, 
> groupId=svc-stream-collector] (Re-)joining group 
> (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
> [2022-03-08 17:13:37,032] INFO stream-thread 
> [svc-stream-collector-StreamThread-1] Informed to shut down 
> (org.apache.kafka.streams.processor.internals.StreamThread)
> [2022-03-08 17:13:37,032] INFO stream-thread 
> [svc-stream-collector-StreamThread-1] State transition from 
> PARTITIONS_REVOKED to PENDING_SHUTDOWN 
> (org.apache.kafka.streams.processor.internals.StreamThread)
> [2022-03-08 17:13:37,067] INFO stream-thread 
> [svc-stream-collector-StreamThread-1] Thread state is already 
> PENDING_SHUTDOWN, skipping the run once call after poll request 
> (org.apache.kafka.streams.processor.internals.StreamThread)
> [2022-03-08 17:13:37,067] WARN stream-thread 
> [svc-stream-collector-StreamThread-1] Detected that shutdown was requested. 
> All clients in this app will now begin to shutdown 
> (org.apache.kafka.streams.processor.internals.StreamThread)
> {code}
> I'm suspecting its because there are no `newly assigned partitions in the log 
> below`
> {code:java}
> [2022-03-08 17:13:37,024] INFO [Consumer 
> clientId=svc-stream-collector-StreamThread-1-consumer, 
> groupId=svc-stream-collector] Adding newly assigned partitions:  
> (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
> {code}
> However I'm not exactly sure why this error occurs . Any help would be 
> appreciated.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (KAFKA-13750) Client Compatability KafkaTest uses invalid idempotency configs

[Justine Olshan (Jira)]

Justine Olshan created KAFKA-13750:
--

 Summary: Client Compatability KafkaTest uses invalid idempotency 
configs
 Key: KAFKA-13750
 URL: https://issues.apache.org/jira/browse/KAFKA-13750
 Project: Kafka
  Issue Type: Bug
Reporter: Justine Olshan


With the switch to idempotency as a default, some of our tests broke including 

ClientCompatibilityFeaturesTest.run_compatibility_test for versions prior to 
0.11 where EOS was enabled. We need to configure the producer correctly for 
these earlier versions.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: Need permission to be contributor on Apache Kafka project

[Bill Bejeck]

Done.

Thanks for your interest in Apache Kafka.

-Bill

On Wed, Mar 16, 2022 at 6:03 PM Akhilesh Chaganti 
wrote:

> My username is akhileshchg.
>
> Thanks
> Akhilesh
>
>
> On Wed, Mar 16, 2022 at 2:46 PM Bill Bejeck  wrote:
>
> > Hi Akhilesh,
> >
> > Can you provide your Jira username? We can get you setup after that.
> >
> > Thanks,
> > Bill
> >
> > On Wed, Mar 16, 2022 at 4:38 PM Akhilesh Chaganti <
> akhilesh@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > I raised a bug (https://issues.apache.org/jira/browse/KAFKA-13749) for
> > > Apache Kafka. I want to submit a PR for this by becoming a contributor.
> > Can
> > > anyone help with getting the approval to assign the bug to me?
> > >
> > >
> > > Thanks
> > > Akhilesh
> > >
> >
>

Re: Need permission to be contributor on Apache Kafka project

[Akhilesh Chaganti]

My username is akhileshchg.

Thanks
Akhilesh


On Wed, Mar 16, 2022 at 2:46 PM Bill Bejeck  wrote:

> Hi Akhilesh,
>
> Can you provide your Jira username? We can get you setup after that.
>
> Thanks,
> Bill
>
> On Wed, Mar 16, 2022 at 4:38 PM Akhilesh Chaganti 
> wrote:
>
> > Hi,
> >
> > I raised a bug (https://issues.apache.org/jira/browse/KAFKA-13749) for
> > Apache Kafka. I want to submit a PR for this by becoming a contributor.
> Can
> > anyone help with getting the approval to assign the bug to me?
> >
> >
> > Thanks
> > Akhilesh
> >
>

Re: Need permission to be contributor on Apache Kafka project

[Bill Bejeck]

Hi Akhilesh,

Can you provide your Jira username? We can get you setup after that.

Thanks,
Bill

On Wed, Mar 16, 2022 at 4:38 PM Akhilesh Chaganti 
wrote:

> Hi,
>
> I raised a bug (https://issues.apache.org/jira/browse/KAFKA-13749) for
> Apache Kafka. I want to submit a PR for this by becoming a contributor. Can
> anyone help with getting the approval to assign the bug to me?
>
>
> Thanks
> Akhilesh
>

Need permission to be contributor on Apache Kafka project

[Akhilesh Chaganti]

Hi,

I raised a bug (https://issues.apache.org/jira/browse/KAFKA-13749) for
Apache Kafka. I want to submit a PR for this by becoming a contributor. Can
anyone help with getting the approval to assign the bug to me?


Thanks
Akhilesh

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #777

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-13749) TopicConfigs and ErrorCode are not set in createTopics response in KRaft

[Akhilesh Chaganti (Jira)]

Akhilesh Chaganti created KAFKA-13749:
-

 Summary: TopicConfigs and ErrorCode are not set in createTopics 
response in KRaft
 Key: KAFKA-13749
 URL: https://issues.apache.org/jira/browse/KAFKA-13749
 Project: Kafka
  Issue Type: Bug
  Components: kraft
Reporter: Akhilesh Chaganti


Once the createTopics request is process in KRaft, the `CreatableTopicResult` 
is not set with the appropriate topic configs and error and this breaks KIP-525



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [DISCUSS] KIP-825: introduce a new API to control when aggregated results are produced

[Matthias J. Sax]

`allowedLateness` may not be a good name. What I have in mind is to use
this to control how frequently we try to emit final results. Maybe it's
more flexible to be used as config in properties as we don't need to
recompile DSL to change it.


I see; making it a config seems better. Frankly, I am not even sure if 
we need a config at all or if we can just hard code it? For the 
stream-stream join left/outer join fix, there is only an internal config 
but no public config either.


Option 1: Your proposal is?

  stream
.groupByKey()
.windowBy(TimeWindow.ofSizeNoGrace(...))
.configure(EmitConfig.emitFinal()
.count()

Does not change my argument that it seems to be misplace from an API 
flow POV.


Option 1 seems to be the least desirable to me.

For option 2 and 3, and not sure which one I like better. Might be good 
if other could chime in, too. I think I slightly prefer option 2 over 
option 3.



-Matthias

On 3/15/22 5:33 PM, Hao Li wrote:

Thanks for the feedback Matthias.

`allowedLateness` may not be a good name. What I have in mind is to use
this to control how frequently we try to emit final results. Maybe it's
more flexible to be used as config in properties as we don't need to
recompile DSL to change it.

For option 1, I intend to use `emitFinal` to configure how
`TimeWindowedKStream` should be outputted to `KTable` after aggregation.
But `emitFinal` is not an action to the `TimeWindowedKStream` interface.
Maybe adding `configure(EmitConfig config)` makes more sense?

For option 2, config can be created using `WindowConfig.emitFinal()` or
`EmitConfig.emitFinal`

For option 3, it will be something like `TimeWindows(..., EmitConfig
emitConfig)`.

For putting `EmitConfig` in aggregation operator, I think it doesn't
control how we do aggregation but how we output to `KTable`. That's why I
feel option 1 makes more sense as it applies to `TimeWindowedKStream`. But
I'm also OK with option 2.

Hao




On Tue, Mar 15, 2022 at 4:48 PM Matthias J. Sax  wrote:


Thanks for the KIP.

A general comment: it seem that we won't need any new `allowedLateness`
parameter because the grace-period is defined on the window itself already?

(On the other hand, if I think about it once more, maybe the
`grace-period` is actually not a property of the window but a property
of the aggregation operator? _thinking_)

  From an API flow point of view, option 1 might not be desirable IMHO:

stream
  .groupByKey()
  .windowBy(TimeWindow.ofSizeNoGrace(...))
  .emitFinal()
  .count()

The call to `emitFinal(0` seems not to be on the right place for this case?


Option 2 might work (I think we need to discuss a few details of the API
though):

stream
  .groupByKey()
  .windowBy(
TimeWindow.ofSizeNoGrace(...),
EmitConfig.emitFinal() -- just made this up; it's not in the KIP
  )
  .count()

I made up the `WindowConfig.emitFinal()` call -- from the KIP it's
unclear what API you have in mind? `EmitFinalConfig` has not public
constructor not any builder method.


For option 3, I am not sure what you really have in mind. Can you given
a concrete example (similar to above) how users would write their code?



Did you consider to actually pass in the `EmitConfig` into the
aggregation operator? In the end, it seems not to be property of the
window definition or windowing step, but a property of the actual operator:

stream
  .groupByKey()
  .windowBy(
TimeWindow.ofSizeNoGrace(...)
  )
  .count(EmitConfig.emitFinal())

The API surface area that need to be updated might be larger for this
case though...


-Matthias



On 3/14/22 9:21 PM, Hao Li wrote:

Thanks Guozhang!

1. I agree `EmitConfig` is better than `WindowConfig` and option 2

modifies

less places. What do you think of option 1 which doesn't change the

current

`windowedBy` api but configures `EmitConfig` separately. The benefit of
option 1 is if we need to configure something else later, we don't need

to

pile them on `windowedBy` but can add separate APIs.
2. I added it to `Stores` mainly to conform to


https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/kstream/internals/TimeWindowedKStreamImpl.java#L227-L231
.

But We can also create an internal API to do that without modifying
`Stores`.

Hao

On Mon, Mar 14, 2022 at 7:52 PM Guozhang Wang 

wrote:



Hello Hao,

Thanks for the proposal, I have some preference among the options here

so I

will copy them here:

I'm now thinking if it's better to not add this new config on each of

the

Window interfaces, but instead add that at the KGroupedStream#windowedBy
function. Also instead of adding just a boolean flag, maybe we can add a
Configured class like Grouped, Suppressed, etc, e.g. let's call it a
Emitted which for now would just have a single construct as
Emitted.atWindowClose whose semantics is the same as emitFinal == true.

I

think the benefits are:

1) you do not need to modify 

[jira] [Created] (KAFKA-13748) Do not include file stream connectors in Connect's CLASSPATH and plugin.path by default

[Konstantine Karantasis (Jira)]

Konstantine Karantasis created KAFKA-13748:
--

 Summary: Do not include file stream connectors in Connect's 
CLASSPATH and plugin.path by default
 Key: KAFKA-13748
 URL: https://issues.apache.org/jira/browse/KAFKA-13748
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Reporter: Konstantine Karantasis
Assignee: Konstantine Karantasis
 Fix For: 3.2.0, 3.1.1, 3.0.2


File stream connectors have been included with Kafka Connect distributions from 
the very beginning. These simple connectors were included to show case 
connector implementation but were never meant to be used in production and have 
been only available for the straightforward demonstration of Connect's 
capabilities through our quick start guides. 
 
 Given that these connectors are not production ready and yet they offer access 
to the local filesystem, with this ticket I propose to remove them from our 
deployments by default by excluding these connectors from the {{CLASSPATH}} or 
the default {{{}plugin.path{}}}. 
 
 The impact will be minimal. Quick start guides will require a single 
additional step of editing the {{plugin.path}} to include the single package 
that includes these connectors. Production deployments will remain unaffected 
because these are not production grade connectors. 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #776

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #775

[Apache Jenkins Server]

See 

[GitHub] [kafka-site] mimaison commented on pull request #400: MINOR: Updates for 3.0.1

[GitBox]

mimaison commented on pull request #400:
URL: https://github.com/apache/kafka-site/pull/400#issuecomment-1068997010


   @dajac I opened https://github.com/apache/kafka/pull/11906
   I believe it's the only change that can be applied to `kafka/trunk`
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org