Fips Compliant

[brajesh . shishir]

Hi,

I was wondering if Apache Kafka is Fips Compliant or not. I see that 
Confluent Kafka is FIPS 
Complaint(https://docs.confluent.io/platform/current/security/compliance.html).


What about Apache Kafka, is there any documentation like above for 
apache kafka also?


Regards,

Brajesh

Re: [VOTE] 2.6.1 RC3

[Ismael Juma]

Looks like you have your votes Mickael. :)

Ismael

On Fri, Dec 11, 2020 at 7:23 AM Mickael Maison  wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the fourth candidate for release of Apache Kafka 2.6.1.
>
> Since RC2, the following JIRAs have been fixed: KAFKA-10811, KAFKA-10802
>
> Release notes for the 2.6.1 release:
> https://home.apache.org/~mimaison/kafka-2.6.1-rc3/RELEASE_NOTES.html
>
> *** Please download, test and vote by Friday, December 18, 12 PM ET ***
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~mimaison/kafka-2.6.1-rc3/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> https://home.apache.org/~mimaison/kafka-2.6.1-rc3/javadoc/
>
> * Tag to be voted upon (off 2.6 branch) is the 2.6.1 tag:
> https://github.com/apache/kafka/releases/tag/2.6.1-rc3
>
> * Documentation:
> https://kafka.apache.org/26/documentation.html
>
> * Protocol:
> https://kafka.apache.org/26/protocol.html
>
> * Successful Jenkins builds for the 2.6 branch:
> Unit/integration tests:
> https://ci-builds.apache.org/job/Kafka/job/kafka-2.6-jdk8/62/
>
> /**
>
> Thanks,
> Mickael
>

Re: [VOTE] 2.7.0 RC6

[Gwen Shapira]

+1 (binding)

Thank you for the release, Bill!
I validated signatures, built from source package and ran the perf
producer/consumer to validate.

On Wed, Dec 16, 2020 at 6:53 AM Bill Bejeck  wrote:
>
> Hello Kafka users, developers and client-developers,
>
> This is the seventh candidate for release of Apache Kafka 2.7.0.
>
> * Configurable TCP connection timeout and improve the initial metadata fetch
> * Enforce broker-wide and per-listener connection creation rate (KIP-612,
> part 1)
> * Throttle Create Topic, Create Partition and Delete Topic Operations
> * Add TRACE-level end-to-end latency metrics to Streams
> * Add Broker-side SCRAM Config API
> * Support PEM format for SSL certificates and private key
> * Add RocksDB Memory Consumption to RocksDB Metrics
> * Add Sliding-Window support for Aggregations
>
> This release also includes a few other features, 53 improvements, and 91
> bug fixes.
>
> *** Please download, test and vote by Monday, December 21, 12 PM ET
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~bbejeck/kafka-2.7.0-rc6/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> https://home.apache.org/~bbejeck/kafka-2.7.0-rc6/javadoc/
>
> * Tag to be voted upon (off 2.7 branch) is the 2.7.0 tag:
> https://github.com/apache/kafka/releases/tag/2.7.0-rc6
>
> * Documentation:
> https://kafka.apache.org/27/documentation.html
>
> * Protocol:
> https://kafka.apache.org/27/protocol.html
>
> * Successful Jenkins builds for the 2.7 branch:
> Unit/integration tests:
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-2.7-jdk8/detail/kafka-2.7-jdk8/81/
>
> Thanks,
> Bill



-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: Punctuate NPE

[Blake Miller]

Navneeth,

You may need this JVM option:

 -XX:-OmitStackTraceInFastThrow

as some stack frame information can be optimized away by default by some
JVMs leaving us blind to the real issue.

HTH

On Wed, Dec 16, 2020 at 4:15 AM John Roesler  wrote:

> Hi Navneeth,
>
> I'm sorry for the trouble.
>
> Which version of Streams are you using? Also, this doesn't
> look like the full stacktrace, since we can't see the NPE
> itself. Can you share the whole thing?
>
> Thanks,
> -John
>
>
> On Tue, 2020-12-15 at 00:30 -0800, Navneeth Krishnan wrote:
> > Hi All,
> >
> > I have a scheduled function that runs every 10 seconds and in some cases
> I
> > see this NPE. Not sure how to debug this issue. Any pointers would really
> > help. Thanks
> >
> > context.schedule(this.scheduleInterval, PunctuationType.STREAM_TIME,
> > this::flush);
> >
> >
> > 2020-12-15 07:40:14.214
> > [userapp-c2db617d-ed40-4c9a-a3b3-e9942c19d28a-StreamThread-4] ERROR
> > Pipeline -
> > org.apache.kafka.streams.errors.StreamsException: stream-thread
> > [userapp-c2db617d-ed40-4c9a-a3b3-e9942c19d28a-StreamThread-4] task
> [12_16]
> > Exception caught while punctuating processor 'Window_processor'
> > at
> >
> org.apache.kafka.streams.processor.internals.StreamTask.punctuate(StreamTask.java:760)
> > ~[kafka-streams-2.6.0.jar!/:?]
> > at
> >
> org.apache.kafka.streams.processor.internals.PunctuationQueue.mayPunctuate(PunctuationQueue.java:54)
> > ~[kafka-streams-2.6.0.jar!/:?]
> > at
> >
> org.apache.kafka.streams.processor.internals.StreamTask.maybePunctuateStreamTime(StreamTask.java:941)
> > ~[kafka-streams-2.6.0.jar!/:?]
> > at
> >
> org.apache.kafka.streams.processor.internals.TaskManager.punctuate(TaskManager.java:1066)
> > ~[kafka-streams-2.6.0.jar!/:?]
> > at
> >
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:707)
> > ~[kafka-streams-2.6.0.jar!/:?]
> > at
> >
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:551)
> > ~[kafka-streams-2.6.0.jar!/:?]
> > at
> >
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:510)
> > ~[kafka-streams-2.6.0.jar!/:?]
> >
> > Thanks,
> > Navneeth
>
>
>

Re: [VOTE] 2.7.0 RC6

[Sophie Blee-Goldman]

Thanks for driving this release! I built from the tag and ran the tests,
and verified the signatures.

+1 (binding)

Sophie

On Thu, Dec 17, 2020 at 3:53 PM Jakub Scholz  wrote:

> +1 (non-binding) ... I used the binaries (Scala 2.12) and the staged Maven
> artifacts - all seems to work fine. Thanks.
>
> Jakub
>
> On Wed, Dec 16, 2020 at 3:53 PM Bill Bejeck  wrote:
>
> > Hello Kafka users, developers and client-developers,
> >
> > This is the seventh candidate for release of Apache Kafka 2.7.0.
> >
> > * Configurable TCP connection timeout and improve the initial metadata
> > fetch
> > * Enforce broker-wide and per-listener connection creation rate (KIP-612,
> > part 1)
> > * Throttle Create Topic, Create Partition and Delete Topic Operations
> > * Add TRACE-level end-to-end latency metrics to Streams
> > * Add Broker-side SCRAM Config API
> > * Support PEM format for SSL certificates and private key
> > * Add RocksDB Memory Consumption to RocksDB Metrics
> > * Add Sliding-Window support for Aggregations
> >
> > This release also includes a few other features, 53 improvements, and 91
> > bug fixes.
> >
> > *** Please download, test and vote by Monday, December 21, 12 PM ET
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://kafka.apache.org/KEYS
> >
> > * Release artifacts to be voted upon (source and binary):
> > https://home.apache.org/~bbejeck/kafka-2.7.0-rc6/
> >
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >
> > * Javadoc:
> > https://home.apache.org/~bbejeck/kafka-2.7.0-rc6/javadoc/
> >
> > * Tag to be voted upon (off 2.7 branch) is the 2.7.0 tag:
> > https://github.com/apache/kafka/releases/tag/2.7.0-rc6
> >
> > * Documentation:
> > https://kafka.apache.org/27/documentation.html
> >
> > * Protocol:
> > https://kafka.apache.org/27/protocol.html
> >
> > * Successful Jenkins builds for the 2.7 branch:
> > Unit/integration tests:
> >
> >
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-2.7-jdk8/detail/kafka-2.7-jdk8/81/
> >
> > Thanks,
> > Bill
> >
>

Re: When using MM2, how should the consumer be switched from source to target?

[Aki Yoshida]

I'm answering to my own mail and would like to hear if this assumption
is correct.
it looks like I need 2.7.0 to have the automatic translation.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-545%3A+support+automated+consumer+offset+sync+across+clusters+in+MM+2.0
for earlier versions, there is no way to use the console client but
use java API RemoteClusterUtils.translateOffsets() to get the offset
and re-consuming from the new topic.
Is this correct?

El jue, 17 dic 2020 a las 21:18, Aki Yoshida () escribió:
>
> Hi,
> I have a question regarding how to migrate a consumer when the
> subscribing topic has been migrated to the target Kafka broker.
> Suppose a consumer is consuming from topic1 at the source Kafka
> broker. When MM2 mirrors this topic using options
> source.cluster.alias="", replication.policy.separator="", topic named
> topic1 shows up at the target Kafka broker.
>
> When the consumer instance simply switches the bootstrap server to
> start consuming from this topic1 at the target broker, the consumer
> seems to start subscribing from the latest offset. I thought the
> consumer offsets were also migrated to the target and the consumer
> could simply start subscribing from the mirrored topic from the
> continued offset. Did I miss some configuration parameters or does the
> consumer need to perform some actions to be able to consume the
> records seamlessly?
>
> I appreciate for your help. Thanks.
>
> regards, aki

Re: Guidance to downgrade the stateful Kafka stream application?

[Sophie Blee-Goldman]

Hey Ming,

There should not be any issues in downgrading from 2.5 to 2.2, and if
you stayed on the eager protocol then you can do the downgrade in a
single rolling bounce.

It sounds like your main concern here is with the RocksDB version bump,
and whether there would be any problems reading/opening a newer version
of rocksdb using an older version? AFAICT this should not be the case,
especially if you haven't made any config changes or tried to leverage new
features that are only present in the newer version of rocksdb. We also
only bumped the rocksdb version by a small amount. There was no major
version change or any compatibility-breakage that we are aware of.

If you're paranoid, or if you do happen to run into an issue after
downgrading,
you can always just wipe out the local state stores. Streams will rebuild
all
local rocksdb instances from scratch from the changelog topic.

Best,
Sophie

On Thu, Dec 17, 2020 at 11:59 AM Ming Liu  wrote:

> Hi Team,
> I can't find any documentation or guidance on the expectation of the
> downgrade of stateful Kafka stream application (which has different rocksdb
> versions embedded).
> For example, if we upgrade from 2.2 to 2.5 (with binary upgrade only
> and using the same eager protocol) and somehow found some problem and
> downgrade to 2.2. What is the expectation?
>
> Thanks!
> Ming
>

Re: [VOTE] 2.7.0 RC6

[Jakub Scholz]

+1 (non-binding) ... I used the binaries (Scala 2.12) and the staged Maven
artifacts - all seems to work fine. Thanks.

Jakub

On Wed, Dec 16, 2020 at 3:53 PM Bill Bejeck  wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the seventh candidate for release of Apache Kafka 2.7.0.
>
> * Configurable TCP connection timeout and improve the initial metadata
> fetch
> * Enforce broker-wide and per-listener connection creation rate (KIP-612,
> part 1)
> * Throttle Create Topic, Create Partition and Delete Topic Operations
> * Add TRACE-level end-to-end latency metrics to Streams
> * Add Broker-side SCRAM Config API
> * Support PEM format for SSL certificates and private key
> * Add RocksDB Memory Consumption to RocksDB Metrics
> * Add Sliding-Window support for Aggregations
>
> This release also includes a few other features, 53 improvements, and 91
> bug fixes.
>
> *** Please download, test and vote by Monday, December 21, 12 PM ET
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~bbejeck/kafka-2.7.0-rc6/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> https://home.apache.org/~bbejeck/kafka-2.7.0-rc6/javadoc/
>
> * Tag to be voted upon (off 2.7 branch) is the 2.7.0 tag:
> https://github.com/apache/kafka/releases/tag/2.7.0-rc6
>
> * Documentation:
> https://kafka.apache.org/27/documentation.html
>
> * Protocol:
> https://kafka.apache.org/27/protocol.html
>
> * Successful Jenkins builds for the 2.7 branch:
> Unit/integration tests:
>
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-2.7-jdk8/detail/kafka-2.7-jdk8/81/
>
> Thanks,
> Bill
>

When using MM2, how should the consumer be switched from source to target?

[Aki Yoshida]

Hi,
I have a question regarding how to migrate a consumer when the
subscribing topic has been migrated to the target Kafka broker.
Suppose a consumer is consuming from topic1 at the source Kafka
broker. When MM2 mirrors this topic using options
source.cluster.alias="", replication.policy.separator="", topic named
topic1 shows up at the target Kafka broker.

When the consumer instance simply switches the bootstrap server to
start consuming from this topic1 at the target broker, the consumer
seems to start subscribing from the latest offset. I thought the
consumer offsets were also migrated to the target and the consumer
could simply start subscribing from the mirrored topic from the
continued offset. Did I miss some configuration parameters or does the
consumer need to perform some actions to be able to consume the
records seamlessly?

I appreciate for your help. Thanks.

regards, aki

RE: Apache Kafka Powered By : Proposition La Redoute

[CRASKE Antoine]

Hi Anthony,

Great thanks for the update, glad to be present within the community!

We'll also update our tech blog to this link,

Thanks for your help, and stay well,

Antoine

From: Anthony Middleton 
Sent: 16 December 2020 22:14
To: CRASKE Antoine 
Cc: users@kafka.apache.org; Ale Murray 
Subject: Re: Apache Kafka Powered By : Proposition La Redoute

Hey Antonie,

I hope your week is going well. The interactive team has completed the webpage 
update. Your company's logo is now displayed on the Powered By Kafka webpage - 
http://kafka.apache.org/powered-by.

Please let me know if you need anything else.


Best,
Anthony Middleton

[https://lh6.googleusercontent.com/h7-K-Q5QFoTBxh2KENFQywQTZKJYBnSOr-ZiJRhm9DoqwAf7aMX3gqrFM2IqlkvYXdAgkjOKlRe-UGYoTzw5UzCvnkwbihzDHteSLNDw1cMLp4_t2grxVFgMHQ--737WBtzNQSv3]



Anthony Middleton

Community Associate



Follow us:  
Website
 * Developer 
Resources
 * 
Blog
 * 
Slack
 * 
Twitter

[https://lh5.googleusercontent.com/C_FWa6Nkj9f7cZp6RELj6cc6KPHpqQRf10gFUFTiITpt12B8AlAZ0FCA1kehFCrJA2DL8Zw0L2egHuzFlg-grCJFhmr4MvdZmPhiV3kkkzEgrv5OgFuJOdqcYWks4fslPnrRp7n1]


On Wed, Dec 16, 2020 at 5:04 AM CRASKE Antoine 
mailto:acra...@redoute.pt>> wrote:
Hello John,

Thanks for your feedback and no problem,

I just did the PR with the text and the logo file 
https://github.com/apache/kafka-site/pull/315

Re: kafka config problem

[Xia Qu]

Did you ever restart your brokers to pick up the config changes?

On Wed, Dec 16, 2020 at 9:52 AM 路人  wrote:

> Dear Sir/Madam
>  I am using kafka,but one problem in here. I changed
> "auto.create.topics.enable=false", but kafka producer can build topic. I
> would to konw the true methods to producer can not build topic.
>   I hope to receive your favors at early date.

Guidance to downgrade the stateful Kafka stream application?

[Ming Liu]

Hi Team,
I can't find any documentation or guidance on the expectation of the
downgrade of stateful Kafka stream application (which has different rocksdb
versions embedded).
For example, if we upgrade from 2.2 to 2.5 (with binary upgrade only
and using the same eager protocol) and somehow found some problem and
downgrade to 2.2. What is the expectation?

Thanks!
Ming