[jira] [Commented] (KAFKA-7190) Under low traffic conditions purging repartition topics cause WARN statements about UNKNOWN_PRODUCER_ID
[ https://issues.apache.org/jira/browse/KAFKA-7190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564694#comment-16564694 ] lambdaliu commented on KAFKA-7190: -- Hi [~hachikuji], What do you think about the solution of the problem? Looking forward to your opinion. Thanks! > Under low traffic conditions purging repartition topics cause WARN statements > about UNKNOWN_PRODUCER_ID > - > > Key: KAFKA-7190 > URL: https://issues.apache.org/jira/browse/KAFKA-7190 > Project: Kafka > Issue Type: Improvement > Components: core, streams >Affects Versions: 1.1.0, 1.1.1 >Reporter: Bill Bejeck >Assignee: lambdaliu >Priority: Major > > When a streams application has little traffic, then it is possible that > consumer purging would delete > even the last message sent by a producer (i.e., all the messages sent by > this producer have been consumed and committed), and as a result, the broker > would delete that producer's ID. The next time when this producer tries to > send, it will get this UNKNOWN_PRODUCER_ID error code, but in this case, > this error is retriable: the producer would just get a new producer id and > retries, and then this time it will succeed. > > Possible fixes could be on the broker side, i.e., delaying the deletion of > the produderIDs for a more extended period or on the streams side developing > a more conservative approach to deleting offsets from repartition topics > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7192) State-store can desynchronise with changelog
[ https://issues.apache.org/jira/browse/KAFKA-7192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-7192: --- Fix Version/s: 2.0.1 > State-store can desynchronise with changelog > > > Key: KAFKA-7192 > URL: https://issues.apache.org/jira/browse/KAFKA-7192 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 1.1.1 >Reporter: Jon Bates >Assignee: Guozhang Wang >Priority: Critical > Labels: bugs > Fix For: 2.0.1, 2.1.0 > > > n.b. this bug has been verified with exactly-once processing enabled > Consider the following scenario: > * A record, N is read into a Kafka topology > * the state store is updated > * the topology crashes > h3. *Expected behaviour:* > # Node is restarted > # Offset was never updated, so record N is reprocessed > # State-store is reset to position N-1 > # Record is reprocessed > h3. *Actual Behaviour* > # Node is restarted > # Record N is reprocessed (good) > # The state store has the state from the previous processing > I'd consider this a corruption of the state-store, hence the critical > Priority, although High may be more appropriate. > I wrote a proof-of-concept here, which demonstrates the problem on Linux: > [https://github.com/spadger/kafka-streams-sad-state-store] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7131) Update release script to generate announcement email text
[
https://issues.apache.org/jira/browse/KAFKA-7131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564587#comment-16564587
]
Matthias J. Sax commented on KAFKA-7131:
No this part. The ticker refers to this:
{noformat}
- Include a paragraph in the announcement email like: "According to git
shortlog people contributed to this release:
" where:
- number_of_contributors is determined via `git shortlog -sn --no-merges
.. | wc -l` (eg `git shortlog -sn
--no-merges 0.8.2.2..0.9.0.0 | wc -l`)
- contributors is determined via: `git shortlog -sn --no-merges
.. | cut -f2 | tr '\n' ',' | sed -e
's/,/, /g'` (eg `git shortlog -sn --no-merges 0.8.2.2..0.9.0.0 | cut -f2 | sort
--ignore-case | tr '\n' ',' | sed -e 's/,/, /g'`)
- Template (maybe update Scala versions accordingly):{noformat}
Ie, the script should generate the template will filled information about
number of contributors and list of contributors. This allows the release
manager to c the email text and send out the email without running the
command manually and c the information into the template.
> Update release script to generate announcement email text
> -
>
> Key: KAFKA-7131
> URL: https://issues.apache.org/jira/browse/KAFKA-7131
> Project: Kafka
> Issue Type: Improvement
>Reporter: Matthias J. Sax
>Assignee: bibin sebastian
>Priority: Minor
> Labels: newbie
>
> When a release is finalized, we send out an email to announce the release.
> Atm, we have a template in the wiki
> ([https://cwiki.apache.org/confluence/display/KAFKA/Release+Process|https://cwiki.apache.org/confluence/display/KAFKA/Release+Process]).
> However, the template needs some manual changes to fill in the release
> number, number of contributors, etc.
> Some parts could be automated – the corresponding commands are document in
> the wiki already.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7224) KIP-328: Add spill-to-disk for Suppression
[ https://issues.apache.org/jira/browse/KAFKA-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-7224: --- Component/s: streams > KIP-328: Add spill-to-disk for Suppression > -- > > Key: KAFKA-7224 > URL: https://issues.apache.org/jira/browse/KAFKA-7224 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: John Roesler >Assignee: John Roesler >Priority: Major > > As described in > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-328%3A+Ability+to+suppress+updates+for+KTables] > Following on KAFKA-7223, implement the spill-to-disk buffering strategy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7223) KIP-328: Add in-memory Suppression
[ https://issues.apache.org/jira/browse/KAFKA-7223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-7223: --- Component/s: streams > KIP-328: Add in-memory Suppression > -- > > Key: KAFKA-7223 > URL: https://issues.apache.org/jira/browse/KAFKA-7223 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: John Roesler >Assignee: John Roesler >Priority: Major > > As described in > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-328%3A+Ability+to+suppress+updates+for+KTables.] > > This ticket is to implement Suppress, but only for in-memory buffers. > (depends on KAFKA-7222) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KAFKA-6690) Priorities for Source Topics
[ https://issues.apache.org/jira/browse/KAFKA-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564413#comment-16564413 ] Nick Afshartous edited comment on KAFKA-6690 at 7/31/18 10:00 PM: -- [~balaprassanna] Please review before I take this to the dev mailing list for discussion https://cwiki.apache.org/confluence/display/KAFKA/KIP-349%3A+Priorities+for+Source+Topics was (Author: nafshartous): [~balaprassanna] Please review before I take this to the dev mailing list for discussion https://cwiki.apache.org/confluence/display/KAFKA/KIP-349%3A+Priorities+for+Source+Topics#KIP-349:PrioritiesforSourceTopics-Motivation > Priorities for Source Topics > > > Key: KAFKA-6690 > URL: https://issues.apache.org/jira/browse/KAFKA-6690 > Project: Kafka > Issue Type: New Feature > Components: streams >Reporter: Bala Prassanna I >Assignee: Nick Afshartous >Priority: Major > > We often encounter use cases where we need to prioritise source topics. If a > consumer is listening more than one topic, say, HighPriorityTopic and > LowPriorityTopic, it should consume events from LowPriorityTopic only when > all the events from HighPriorityTopic are consumed. This is needed in Kafka > Streams processor topologies as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-6690) Priorities for Source Topics
[ https://issues.apache.org/jira/browse/KAFKA-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564413#comment-16564413 ] Nick Afshartous commented on KAFKA-6690: [~balaprassanna] Please review before I take this to the dev mailing list for discussion https://cwiki.apache.org/confluence/display/KAFKA/KIP-349%3A+Priorities+for+Source+Topics#KIP-349:PrioritiesforSourceTopics-Motivation > Priorities for Source Topics > > > Key: KAFKA-6690 > URL: https://issues.apache.org/jira/browse/KAFKA-6690 > Project: Kafka > Issue Type: New Feature > Components: streams >Reporter: Bala Prassanna I >Assignee: Nick Afshartous >Priority: Major > > We often encounter use cases where we need to prioritise source topics. If a > consumer is listening more than one topic, say, HighPriorityTopic and > LowPriorityTopic, it should consume events from LowPriorityTopic only when > all the events from HighPriorityTopic are consumed. This is needed in Kafka > Streams processor topologies as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-5407) Mirrormaker dont start after upgrade
[
https://issues.apache.org/jira/browse/KAFKA-5407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564177#comment-16564177
]
Manikumar commented on KAFKA-5407:
--
[~fvegaucr] I am not sure what is happening. It is very difficult without
broker error logs. If you are still getting below broker exception, then try
increasing message.max.bytes to large value ( say 10mb/20mb) and set
offsets.topic.compression.codec=1 on all brokers.
{code:java}
[2017-09-14 15:16:56,688] ERROR [Group Metadata Manager on Broker 43]:
Appending metadata message for group MirrorMaker_hkg1 generation 14 failed due
to org.apache.kafka.common.errors.RecordTooLargeException, returning UNKNOWN
error code to the client (kafka.coordinator.GroupMetadataManager
{code}
Suggested configs:
{code:java}
message.max.bytes=10485760
offsets.topic.compression.codec=1
{code}
> Mirrormaker dont start after upgrade
>
>
> Key: KAFKA-5407
> URL: https://issues.apache.org/jira/browse/KAFKA-5407
> Project: Kafka
> Issue Type: Bug
> Components: consumer
>Affects Versions: 0.10.2.1
> Environment: Operating system
> CentOS 6.8
> HW
> Board Mfg : HP
> Board Product : ProLiant DL380p Gen8
> CPU's x2
> Product Manufacturer : Intel
> Product Name : Intel(R) Xeon(R) CPU E5-2660 v2 @ 2.20GHz
> Memory Type : DDR3 SDRAM
> SDRAM Capacity: 2048 MB
> Total Memory: : 64GB
> Hardrives size and layout:
> 9 drives using jbod
> drive size 3.6TB each
>Reporter: Fernando Vega
>Priority: Critical
> Attachments: broker.hkg1.new, debug.hkg1.new,
> mirrormaker-repl-sjc2-to-hkg1.log.8
>
>
> Currently Im upgrading the cluster from 0.8.2-beta to 0.10.2.1
> So I followed the rolling procedure:
> Here the config files:
> Consumer
> {noformat}
> #
> # Cluster: repl
> # Topic list(goes into command line):
> REPL-ams1-global,REPL-atl1-global,REPL-sjc2-global,REPL-ams1-global-PN_HXIDMAP_.*,REPL-atl1-global-PN_HXIDMAP_.*,REPL-sjc2-global-PN_HXIDMAP_.*,REPL-ams1-global-PN_HXCONTEXTUALV2_.*,REPL-atl1-global-PN_HXCONTEXTUALV2_.*,REPL-sjc2-global-PN_HXCONTEXTUALV2_.*
> bootstrap.servers=app001:9092,app002:9092,app003:9092,app004:9092
> group.id=hkg1_cluster
> auto.commit.interval.ms=6
> partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor
> {noformat}
> Producer
> {noformat}
> hkg1
> # # Producer
> # # hkg1
> bootstrap.servers=app001:9092,app002:9092,app003:9092,app004:9092
> compression.type=gzip
> acks=0
> {noformat}
> Broker
> {noformat}
> auto.leader.rebalance.enable=true
> delete.topic.enable=true
> socket.receive.buffer.bytes=1048576
> socket.send.buffer.bytes=1048576
> default.replication.factor=2
> auto.create.topics.enable=true
> num.partitions=1
> num.network.threads=8
> num.io.threads=40
> log.retention.hours=1
> log.roll.hours=1
> num.replica.fetchers=8
> zookeeper.connection.timeout.ms=3
> zookeeper.session.timeout.ms=3
> inter.broker.protocol.version=0.10.2
> log.message.format.version=0.8.2
> {noformat}
> I tried also using stock configuraiton with no luck.
> The error that I get is this:
> {noformat}
> 2017-06-07 12:24:45,476] INFO ConsumerConfig values:
> auto.commit.interval.ms = 6
> auto.offset.reset = latest
> bootstrap.servers = [app454.sjc2.mytest.com:9092,
> app455.sjc2.mytest.com:9092, app456.sjc2.mytest.com:9092,
> app457.sjc2.mytest.com:9092, app458.sjc2.mytest.com:9092,
> app459.sjc2.mytest.com:9092]
> check.crcs = true
> client.id = MirrorMaker_hkg1-1
> connections.max.idle.ms = 54
> enable.auto.commit = false
> exclude.internal.topics = true
> fetch.max.bytes = 52428800
> fetch.max.wait.ms = 500
> fetch.min.bytes = 1
> group.id = MirrorMaker_hkg1
> heartbeat.interval.ms = 3000
> interceptor.classes = null
> key.deserializer = class
> org.apache.kafka.common.serialization.ByteArrayDeserializer
> max.partition.fetch.bytes = 1048576
> max.poll.interval.ms = 30
> max.poll.records = 500
> metadata.max.age.ms = 30
> metric.reporters = []
> metrics.num.samples = 2
> metrics.recording.level = INFO
> metrics.sample.window.ms = 3
> partition.assignment.strategy =
> [org.apache.kafka.clients.consumer.RoundRobinAssignor]
> receive.buffer.bytes = 65536
> reconnect.backoff.ms = 50
> request.timeout.ms = 305000
> retry.backoff.ms = 100
> sasl.jaas.config = null
> sasl.kerberos.kinit.cmd = /usr/bin/kinit
> sasl.kerberos.min.time.before.relogin = 6
> sasl.kerberos.service.name = null
> sasl.kerberos.ticket.renew.jitter = 0.05
> sasl.kerberos.ticket.renew.window.factor = 0.8
> sasl.mechanism = GSSAPI
> security.protocol
[jira] [Commented] (KAFKA-6605) Flatten SMT does not properly handle fields that are null
[
https://issues.apache.org/jira/browse/KAFKA-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564164#comment-16564164
]
Quinton Parker commented on KAFKA-6605:
---
Excuse me for a second
With recent support for nested data types does that present a suitable
workaround for this issue?
[https://www.confluent.io/blog/introducing-confluent-platform-5-0/]
Just curious
> Flatten SMT does not properly handle fields that are null
> -
>
> Key: KAFKA-6605
> URL: https://issues.apache.org/jira/browse/KAFKA-6605
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 1.0.0
>Reporter: Randall Hauch
>Priority: Major
>
> When a message has a null field, the `Flatten` SMT does not properly handle
> this and throws an NPE. Consider this message from Debezium:
> {code}
> {
> "before": null,
> "after": {
> "dbserver1.mydb.team.Value": {
> "id": 1,
> "name": "kafka",
> "email": "ka...@apache.org",
> "last_modified": 1519939449000
> }
> },
> "source": {
> "version": {
> "string": "0.7.3"
> },
> "name": "dbserver1",
> "server_id": 0,
> "ts_sec": 0,
> "gtid": null,
> "file": "mysql-bin.03",
> "pos": 154,
> "row": 0,
> "snapshot": {
> "boolean": true
> },
> "thread": null,
> "db": {
> "string": "mydb"
> },
> "table": {
> "string": "team"
> }
> },
> "op": "c",
> "ts_ms": {
> "long": 1519939520285
> }
> }
> {code}
> Note how `before` is null; this event represents a row was INSERTED and thus
> there is no `before` state of the row. This results in an NPE:
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.kafka.connect.transforms.Flatten.buildWithSchema(Flatten.java:219)
> at
> org.apache.kafka.connect.transforms.Flatten.buildWithSchema(Flatten.java:234)
> at
> org.apache.kafka.connect.transforms.Flatten.applyWithSchema(Flatten.java:151)
> at org.apache.kafka.connect.transforms.Flatten.apply(Flatten.java:75)
> at
> org.apache.kafka.connect.runtime.TransformationChain.apply(TransformationChain.java:38)
> at
> org.apache.kafka.connect.runtime.WorkerSourceTask.sendRecords(WorkerSourceTask.java:211)
> at
> org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:187)
> at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:170)
> at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:214)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> Here's the connector configuration that was used:
> {code}
> {
> "name": "debezium-connector-flatten",
> "config": {
> "connector.class": "io.debezium.connector.mysql.MySqlConnector",
> "tasks.max": "1",
> "database.hostname": "mysql",
> "database.port": "3306",
> "database.user": "debezium",
> "database.password": "dbz",
> "database.server.id": "223345",
> "database.server.name": "dbserver-flatten",
> "database.whitelist": "mydb",
> "database.history.kafka.bootstrap.servers":
> "kafka-1:9092,kafka-2:9092,kafka-3:9092",
> "database.history.kafka.topic": "schema-flatten.mydb",
> "include.schema.changes": "true",
> "transforms": "flatten",
> "transforms.flatten.type":
> "org.apache.kafka.connect.transforms.Flatten$Value",
> "transforms.flatten.delimiter": "_"
> }
> }
> {code}
> Note that the above configuration sets the delimiter to `_`. The default
> delimiter is `.`, which is not a valid character within an Avro field, and
> doing this results in the following exception:
> {noformat}
> org.apache.avro.SchemaParseException: Illegal character in: source.version
> at org.apache.avro.Schema.validateName(Schema.java:1151)
> at org.apache.avro.Schema.access$200(Schema.java:81)
> at org.apache.avro.Schema$Field.(Schema.java:403)
> at
> org.apache.avro.SchemaBuilder$FieldBuilder.completeField(SchemaBuilder.java:2124)
> at
> org.apache.avro.SchemaBuilder$FieldBuilder.completeField(SchemaBuilder.java:2116)
> at
> org.apache.avro.SchemaBuilder$FieldBuilder.access$5300(SchemaBuilder.java:2034)
> at
> org.apache.avro.SchemaBuilder$GenericDefault.withDefault(SchemaBuilder.java:2423)
> at
> io.confluent.connect.avro.AvroData.addAvroRecordField(AvroData.java:898)
> at
>
[jira] [Assigned] (KAFKA-7223) KIP-328: Add in-memory Suppression
[ https://issues.apache.org/jira/browse/KAFKA-7223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Roesler reassigned KAFKA-7223: --- Assignee: John Roesler > KIP-328: Add in-memory Suppression > -- > > Key: KAFKA-7223 > URL: https://issues.apache.org/jira/browse/KAFKA-7223 > Project: Kafka > Issue Type: Improvement >Reporter: John Roesler >Assignee: John Roesler >Priority: Major > > As described in > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-328%3A+Ability+to+suppress+updates+for+KTables.] > > This ticket is to implement Suppress, but only for in-memory buffers. > (depends on KAFKA-7222) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7224) KIP-328: Add spill-to-disk for Suppression
John Roesler created KAFKA-7224: --- Summary: KIP-328: Add spill-to-disk for Suppression Key: KAFKA-7224 URL: https://issues.apache.org/jira/browse/KAFKA-7224 Project: Kafka Issue Type: Improvement Reporter: John Roesler Assignee: John Roesler As described in [https://cwiki.apache.org/confluence/display/KAFKA/KIP-328%3A+Ability+to+suppress+updates+for+KTables] Following on KAFKA-7223, implement the spill-to-disk buffering strategy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7223) KIP-328: Add in-memory Suppression
John Roesler created KAFKA-7223: --- Summary: KIP-328: Add in-memory Suppression Key: KAFKA-7223 URL: https://issues.apache.org/jira/browse/KAFKA-7223 Project: Kafka Issue Type: Improvement Reporter: John Roesler As described in [https://cwiki.apache.org/confluence/display/KAFKA/KIP-328%3A+Ability+to+suppress+updates+for+KTables.] This ticket is to implement Suppress, but only for in-memory buffers. (depends on KAFKA-7222) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7222) KIP-328: Add Window Grace Period (and deprecate Window Retention)
John Roesler created KAFKA-7222: --- Summary: KIP-328: Add Window Grace Period (and deprecate Window Retention) Key: KAFKA-7222 URL: https://issues.apache.org/jira/browse/KAFKA-7222 Project: Kafka Issue Type: Improvement Reporter: John Roesler Assignee: John Roesler As described in [https://cwiki.apache.org/confluence/display/KAFKA/KIP-328%3A+Ability+to+suppress+updates+for+KTables] This ticket only covers the grace period portion of the work. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7027) Overloaded StreamsBuilder Build Method to Accept java.util.Properties
[
https://issues.apache.org/jira/browse/KAFKA-7027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563922#comment-16563922
]
ASF GitHub Bot commented on KAFKA-7027:
---
guozhangwang closed pull request #5437: KAFKA-7027: Add overloaded build method
to StreamsBuilder
URL: https://github.com/apache/kafka/pull/5437
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/streams/src/main/java/org/apache/kafka/streams/StreamsBuilder.java
b/streams/src/main/java/org/apache/kafka/streams/StreamsBuilder.java
index ae6d44c449e..ba74f61973b 100644
--- a/streams/src/main/java/org/apache/kafka/streams/StreamsBuilder.java
+++ b/streams/src/main/java/org/apache/kafka/streams/StreamsBuilder.java
@@ -41,6 +41,7 @@
import java.util.Collection;
import java.util.Collections;
import java.util.Objects;
+import java.util.Properties;
import java.util.regex.Pattern;
/**
@@ -513,10 +514,23 @@ public synchronized StreamsBuilder addGlobalStore(final
StoreBuilder storeBuilde
/**
* Returns the {@link Topology} that represents the specified processing
logic.
+ * Note that using this method means no optimizations are performed.
*
* @return the {@link Topology} that represents the specified processing
logic
*/
public synchronized Topology build() {
+return build(null);
+}
+
+/**
+ * Returns the {@link Topology} that represents the specified processing
logic and accepts
+ * a {@link Properties} instance used to indicate whether to optimize
topology or not.
+ *
+ * @param props the {@link Properties} used for building possibly
optimized topology
+ * @return the {@link Topology} that represents the specified processing
logic
+ */
+public synchronized Topology build(final Properties props) {
+// the props instance will be used once optimization framework merged
return topology;
}
}
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Overloaded StreamsBuilder Build Method to Accept java.util.Properties
> -
>
> Key: KAFKA-7027
> URL: https://issues.apache.org/jira/browse/KAFKA-7027
> Project: Kafka
> Issue Type: New Feature
> Components: streams
>Reporter: Bill Bejeck
>Assignee: Bill Bejeck
>Priority: Major
> Labels: kip
> Fix For: 2.1.0
>
>
> Add overloaded method to {{StreamsBuilder}} accepting a
> {{java.utils.Properties}} instance.
>
> KIP can be found here
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-312%3A+Add+Overloaded+StreamsBuilder+Build+Method+to+Accept+java.util.Properties
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7027) Overloaded StreamsBuilder Build Method to Accept java.util.Properties
[
https://issues.apache.org/jira/browse/KAFKA-7027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563821#comment-16563821
]
ASF GitHub Bot commented on KAFKA-7027:
---
bbejeck opened a new pull request #5437: KAFKA-7027: Add overloaded build
method to StreamsBuilder
URL: https://github.com/apache/kafka/pull/5437
Implementation of
[KIP-312](https://cwiki.apache.org/confluence/display/KAFKA/KIP-312%3A+Add+Overloaded+StreamsBuilder+Build+Method+to+Accept+java.util.Properties)
required for enabling the use of the optimization framework.
This PR is required to allow for a 4th PR following on from
https://github.com/apache/kafka/pull/5201 to enable optimizations.
The existing streams tests were used for testing.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Overloaded StreamsBuilder Build Method to Accept java.util.Properties
> -
>
> Key: KAFKA-7027
> URL: https://issues.apache.org/jira/browse/KAFKA-7027
> Project: Kafka
> Issue Type: New Feature
> Components: streams
>Reporter: Bill Bejeck
>Assignee: Bill Bejeck
>Priority: Major
> Labels: kip
> Fix For: 2.1.0
>
>
> Add overloaded method to {{StreamsBuilder}} accepting a
> {{java.utils.Properties}} instance.
>
> KIP can be found here
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-312%3A+Add+Overloaded+StreamsBuilder+Build+Method+to+Accept+java.util.Properties
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KAFKA-7221) Add Build method to StreamsBuilder accepting Properties
[ https://issues.apache.org/jira/browse/KAFKA-7221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-7221. Resolution: Fixed Duplicated by KAFKA-7027 > Add Build method to StreamsBuilder accepting Properties > > > Key: KAFKA-7221 > URL: https://issues.apache.org/jira/browse/KAFKA-7221 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 2.1.0 >Reporter: Bill Bejeck >Assignee: Bill Bejeck >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7221) Add Build method to StreamsBuilder accepting Properties
Bill Bejeck created KAFKA-7221: -- Summary: Add Build method to StreamsBuilder accepting Properties Key: KAFKA-7221 URL: https://issues.apache.org/jira/browse/KAFKA-7221 Project: Kafka Issue Type: Improvement Components: streams Affects Versions: 2.1.0 Reporter: Bill Bejeck Assignee: Bill Bejeck -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7219) Add topic/partition level metrics.
[ https://issues.apache.org/jira/browse/KAFKA-7219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563562#comment-16563562 ] Satish Duggana commented on KAFKA-7219: --- I will work on having KIP with the details and update this issue with the location when it is ready. > Add topic/partition level metrics. > -- > > Key: KAFKA-7219 > URL: https://issues.apache.org/jira/browse/KAFKA-7219 > Project: Kafka > Issue Type: Improvement > Components: metrics >Reporter: Satish Duggana >Assignee: Satish Duggana >Priority: Major > Labels: needs-kip > > Currently, Kafka generates different metrics for topics on a broker. > - MessagesInPerSec > - BytesInPerSec > - BytesOutPerSec > - BytesRejectedPerSec > - ReplicationBytesInPerSec > - ReplicationBytesOutPerSec > - FailedProduceRequestsPerSec > - FailedFetchRequestsPerSec > - TotalProduceRequestsPerSec > - TotalFetchRequestsPerSec > - FetchMessageConversionsPerSec > - ProduceMessageConversionsPerSec > Add metrics for individual partitions instead of having only at topic level. > Some of these partition level metrics are useful for monitoring applications > to monitor individual topic/partitions. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7220) Kafka could use RollingFileAppender instead of DailyRollingFileAppender
[ https://issues.apache.org/jira/browse/KAFKA-7220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Swathi Mocharla updated KAFKA-7220: --- Description: We have huge broker logs that are getting generated that currently only roll over to create a new log each day. An automatic mechanism of deletion of old logs will be helpful, which is already provided by the RollingFileAppender. DailyRollingFileAppender doesn't support MaxBackupIndex, and only RollingFileAppender supports it. was: We have huge logs that are getting generated that currently only roll over to create a new log each day. An automatic mechanism of deletion of old logs will be helpful, which is already provided by the RollingFileAppender. DailyRollingFileAppender doesn't support MaxBackupIndex, and only RollingFileAppender supports it. > Kafka could use RollingFileAppender instead of DailyRollingFileAppender > --- > > Key: KAFKA-7220 > URL: https://issues.apache.org/jira/browse/KAFKA-7220 > Project: Kafka > Issue Type: Improvement > Components: core >Affects Versions: 1.1.0 >Reporter: Swathi Mocharla >Priority: Minor > > We have huge broker logs that are getting generated that currently only roll > over to create a new log each day. An automatic mechanism of deletion of old > logs will be helpful, which is already provided by the RollingFileAppender. > DailyRollingFileAppender doesn't support MaxBackupIndex, and only > RollingFileAppender supports it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7220) Kafka could use RollingFileAppender instead of DailyRollingFileAppender
Swathi Mocharla created KAFKA-7220: -- Summary: Kafka could use RollingFileAppender instead of DailyRollingFileAppender Key: KAFKA-7220 URL: https://issues.apache.org/jira/browse/KAFKA-7220 Project: Kafka Issue Type: Improvement Components: core Affects Versions: 1.1.0 Reporter: Swathi Mocharla We have huge logs that are getting generated that currently only roll over to create a new log each day. An automatic mechanism of deletion of old logs will be helpful, which is already provided by the RollingFileAppender. DailyRollingFileAppender doesn't support MaxBackupIndex, and only RollingFileAppender supports it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KAFKA-7165) Error while creating ephemeral at /brokers/ids/BROKER_ID
[ https://issues.apache.org/jira/browse/KAFKA-7165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563458#comment-16563458 ] Manikumar edited comment on KAFKA-7165 at 7/31/18 10:46 AM: [~pachilo] your solution may work, but we should be careful not to remove the ephemeral nodes created by another broker. if someone starts a broker with same brokerId, then the registration should fail. Another option is to maintain previous zk session id and do a check [here|https://github.com/apache/kafka/blob/90e0bbec94dd85e1c5b1af0b6426df0a02e5da3f/core/src/main/scala/kafka/zk/KafkaZkClient.scala#L1512]. If the owner matches with previous sessionID, we can delete and recreate the node. [~cthunes] Since you have analyzed the ZOOKEEPER-2985, any thoughts on handling this on Kafka side. also can you share the code to reproduce this this issue? was (Author: omkreddy): [~pachilo] your solution may work, but we should be careful not to remove the ephemeral nodes created by another broker. if someone starts a broker with same brokerId, then the registration should fail. Another option is to maintain previous zk session id and do a check [here|https://github.com/apache/kafka/blob/90e0bbec94dd85e1c5b1af0b6426df0a02e5da3f/core/src/main/scala/kafka/zk/KafkaZkClient.scala#L1512 If the owner matches with previous sessionID, we can delete and recreate the node. [~cthunes] Since you have analyzed the ZOOKEEPER-2985, any thoughts on handling this on Kafka side. also can you share the code to reproduce this this issue? > Error while creating ephemeral at /brokers/ids/BROKER_ID > > > Key: KAFKA-7165 > URL: https://issues.apache.org/jira/browse/KAFKA-7165 > Project: Kafka > Issue Type: Bug > Components: core, zkclient >Affects Versions: 1.1.0 >Reporter: Jonathan Santilli >Priority: Major > > Kafka version: 1.1.0 > Zookeeper version: 3.4.12 > 4 Kafka Brokers > 4 Zookeeper servers > > In one of the 4 brokers of the cluster, we detect the following error: > [2018-07-14 04:38:23,784] INFO Unable to read additional data from server > sessionid 0x3000c2420cb458d, likely server has closed socket, closing socket > connection and attempting reconnect (org.apache.zookeeper.ClientCnxn) > [2018-07-14 04:38:24,509] INFO Opening socket connection to server > *ZOOKEEPER_SERVER_1:PORT*. Will not attempt to authenticate using SASL > (unknown error) (org.apache.zookeeper.ClientCnxn) > [2018-07-14 04:38:24,510] INFO Socket connection established to > *ZOOKEEPER_SERVER_1:PORT*, initiating session > (org.apache.zookeeper.ClientCnxn) > [2018-07-14 04:38:24,513] INFO Unable to read additional data from server > sessionid 0x3000c2420cb458d, likely server has closed socket, closing socket > connection and attempting reconnect (org.apache.zookeeper.ClientCnxn) > [2018-07-14 04:38:25,287] INFO Opening socket connection to server > *ZOOKEEPER_SERVER_2:PORT*. Will not attempt to authenticate using SASL > (unknown error) (org.apache.zookeeper.ClientCnxn) > [2018-07-14 04:38:25,287] INFO Socket connection established to > *ZOOKEEPER_SERVER_2:PORT*, initiating session > (org.apache.zookeeper.ClientCnxn) > [2018-07-14 04:38:25,954] INFO [Partition TOPIC_NAME-PARTITION-# broker=1|#* > broker=1] Shrinking ISR from 1,3,4,2 to 1,4,2 (kafka.cluster.Partition) > [2018-07-14 04:38:26,444] WARN Unable to reconnect to ZooKeeper service, > session 0x3000c2420cb458d has expired (org.apache.zookeeper.ClientCnxn) > [2018-07-14 04:38:26,444] INFO Unable to reconnect to ZooKeeper service, > session 0x3000c2420cb458d has expired, closing socket connection > (org.apache.zookeeper.ClientCnxn) > [2018-07-14 04:38:26,445] INFO EventThread shut down for session: > 0x3000c2420cb458d (org.apache.zookeeper.ClientCnxn) > [2018-07-14 04:38:26,446] INFO [ZooKeeperClient] Session expired. > (kafka.zookeeper.ZooKeeperClient) > [2018-07-14 04:38:26,459] INFO [ZooKeeperClient] Initializing a new session > to > *ZOOKEEPER_SERVER_1:PORT*,*ZOOKEEPER_SERVER_2:PORT*,*ZOOKEEPER_SERVER_3:PORT*,*ZOOKEEPER_SERVER_4:PORT*. > (kafka.zookeeper.ZooKeeperClient) > [2018-07-14 04:38:26,459] INFO Initiating client connection, > connectString=*ZOOKEEPER_SERVER_1:PORT*,*ZOOKEEPER_SERVER_2:PORT*,*ZOOKEEPER_SERVER_3:PORT*,*ZOOKEEPER_SERVER_4:PORT* > sessionTimeout=6000 > watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@44821a96 > (org.apache.zookeeper.ZooKeeper) > [2018-07-14 04:38:26,465] INFO Opening socket connection to server > *ZOOKEEPER_SERVER_1:PORT*. Will not attempt to authenticate using SASL > (unknown error) (org.apache.zookeeper.ClientCnxn) > [2018-07-14 04:38:26,477] INFO Socket connection established to > *ZOOKEEPER_SERVER_1:PORT*, initiating session >
[jira] [Commented] (KAFKA-7165) Error while creating ephemeral at /brokers/ids/BROKER_ID
[ https://issues.apache.org/jira/browse/KAFKA-7165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563458#comment-16563458 ] Manikumar commented on KAFKA-7165: -- [~pachilo] your solution may work, but we should be careful not to remove the ephemeral nodes created by another broker. if someone starts a broker with same brokerId, then the registration should fail. Another option is to maintain previous zk session id and do a check [here|https://github.com/apache/kafka/blob/90e0bbec94dd85e1c5b1af0b6426df0a02e5da3f/core/src/main/scala/kafka/zk/KafkaZkClient.scala#L1512 If the owner matches with previous sessionID, we can delete and recreate the node. [~cthunes] Since you have analyzed the ZOOKEEPER-2985, any thoughts on handling this on Kafka side. also can you share the code to reproduce this this issue? > Error while creating ephemeral at /brokers/ids/BROKER_ID > > > Key: KAFKA-7165 > URL: https://issues.apache.org/jira/browse/KAFKA-7165 > Project: Kafka > Issue Type: Bug > Components: core, zkclient >Affects Versions: 1.1.0 >Reporter: Jonathan Santilli >Priority: Major > > Kafka version: 1.1.0 > Zookeeper version: 3.4.12 > 4 Kafka Brokers > 4 Zookeeper servers > > In one of the 4 brokers of the cluster, we detect the following error: > [2018-07-14 04:38:23,784] INFO Unable to read additional data from server > sessionid 0x3000c2420cb458d, likely server has closed socket, closing socket > connection and attempting reconnect (org.apache.zookeeper.ClientCnxn) > [2018-07-14 04:38:24,509] INFO Opening socket connection to server > *ZOOKEEPER_SERVER_1:PORT*. Will not attempt to authenticate using SASL > (unknown error) (org.apache.zookeeper.ClientCnxn) > [2018-07-14 04:38:24,510] INFO Socket connection established to > *ZOOKEEPER_SERVER_1:PORT*, initiating session > (org.apache.zookeeper.ClientCnxn) > [2018-07-14 04:38:24,513] INFO Unable to read additional data from server > sessionid 0x3000c2420cb458d, likely server has closed socket, closing socket > connection and attempting reconnect (org.apache.zookeeper.ClientCnxn) > [2018-07-14 04:38:25,287] INFO Opening socket connection to server > *ZOOKEEPER_SERVER_2:PORT*. Will not attempt to authenticate using SASL > (unknown error) (org.apache.zookeeper.ClientCnxn) > [2018-07-14 04:38:25,287] INFO Socket connection established to > *ZOOKEEPER_SERVER_2:PORT*, initiating session > (org.apache.zookeeper.ClientCnxn) > [2018-07-14 04:38:25,954] INFO [Partition TOPIC_NAME-PARTITION-# broker=1|#* > broker=1] Shrinking ISR from 1,3,4,2 to 1,4,2 (kafka.cluster.Partition) > [2018-07-14 04:38:26,444] WARN Unable to reconnect to ZooKeeper service, > session 0x3000c2420cb458d has expired (org.apache.zookeeper.ClientCnxn) > [2018-07-14 04:38:26,444] INFO Unable to reconnect to ZooKeeper service, > session 0x3000c2420cb458d has expired, closing socket connection > (org.apache.zookeeper.ClientCnxn) > [2018-07-14 04:38:26,445] INFO EventThread shut down for session: > 0x3000c2420cb458d (org.apache.zookeeper.ClientCnxn) > [2018-07-14 04:38:26,446] INFO [ZooKeeperClient] Session expired. > (kafka.zookeeper.ZooKeeperClient) > [2018-07-14 04:38:26,459] INFO [ZooKeeperClient] Initializing a new session > to > *ZOOKEEPER_SERVER_1:PORT*,*ZOOKEEPER_SERVER_2:PORT*,*ZOOKEEPER_SERVER_3:PORT*,*ZOOKEEPER_SERVER_4:PORT*. > (kafka.zookeeper.ZooKeeperClient) > [2018-07-14 04:38:26,459] INFO Initiating client connection, > connectString=*ZOOKEEPER_SERVER_1:PORT*,*ZOOKEEPER_SERVER_2:PORT*,*ZOOKEEPER_SERVER_3:PORT*,*ZOOKEEPER_SERVER_4:PORT* > sessionTimeout=6000 > watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@44821a96 > (org.apache.zookeeper.ZooKeeper) > [2018-07-14 04:38:26,465] INFO Opening socket connection to server > *ZOOKEEPER_SERVER_1:PORT*. Will not attempt to authenticate using SASL > (unknown error) (org.apache.zookeeper.ClientCnxn) > [2018-07-14 04:38:26,477] INFO Socket connection established to > *ZOOKEEPER_SERVER_1:PORT*, initiating session > (org.apache.zookeeper.ClientCnxn) > [2018-07-14 04:38:26,484] INFO Session establishment complete on server > *ZOOKEEPER_SERVER_1:PORT*, sessionid = 0x4005b59eb6a, negotiated timeout > = 6000 (org.apache.zookeeper.ClientCnxn) > [2018-07-14 04:38:26,496] *INFO Creating /brokers/ids/1* (is it secure? > false) (kafka.zk.KafkaZkClient) > [2018-07-14 04:38:26,500] INFO Processing notification(s) to /config/changes > (kafka.common.ZkNodeChangeNotificationListener) > *[2018-07-14 04:38:26,547] ERROR Error while creating ephemeral at > /brokers/ids/1, node already exists and owner '216186131422332301' does not > match current session '288330817911521280' > (kafka.zk.KafkaZkClient$CheckedEphemeral)* > [2018-07-14 04:38:26,547] *INFO Result of znode
[jira] [Commented] (KAFKA-7219) Add topic/partition level metrics.
[ https://issues.apache.org/jira/browse/KAFKA-7219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563199#comment-16563199 ] Ismael Juma commented on KAFKA-7219: Thanks for the JIRA. Note that adding metrics requires a KIP (https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals). The main drawback is the additional memory usage of all these new metrics. > Add topic/partition level metrics. > -- > > Key: KAFKA-7219 > URL: https://issues.apache.org/jira/browse/KAFKA-7219 > Project: Kafka > Issue Type: Improvement > Components: metrics >Reporter: Satish Duggana >Assignee: Satish Duggana >Priority: Major > Labels: needs-kip > > Currently, Kafka generates different metrics for topics on a broker. > - MessagesInPerSec > - BytesInPerSec > - BytesOutPerSec > - BytesRejectedPerSec > - ReplicationBytesInPerSec > - ReplicationBytesOutPerSec > - FailedProduceRequestsPerSec > - FailedFetchRequestsPerSec > - TotalProduceRequestsPerSec > - TotalFetchRequestsPerSec > - FetchMessageConversionsPerSec > - ProduceMessageConversionsPerSec > Add metrics for individual partitions instead of having only at topic level. > Some of these partition level metrics are useful for monitoring applications > to monitor individual topic/partitions. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7219) Add topic/partition level metrics.
[ https://issues.apache.org/jira/browse/KAFKA-7219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-7219: --- Labels: needs-kip (was: ) > Add topic/partition level metrics. > -- > > Key: KAFKA-7219 > URL: https://issues.apache.org/jira/browse/KAFKA-7219 > Project: Kafka > Issue Type: Improvement > Components: metrics >Reporter: Satish Duggana >Assignee: Satish Duggana >Priority: Major > Labels: needs-kip > > Currently, Kafka generates different metrics for topics on a broker. > - MessagesInPerSec > - BytesInPerSec > - BytesOutPerSec > - BytesRejectedPerSec > - ReplicationBytesInPerSec > - ReplicationBytesOutPerSec > - FailedProduceRequestsPerSec > - FailedFetchRequestsPerSec > - TotalProduceRequestsPerSec > - TotalFetchRequestsPerSec > - FetchMessageConversionsPerSec > - ProduceMessageConversionsPerSec > Add metrics for individual partitions instead of having only at topic level. > Some of these partition level metrics are useful for monitoring applications > to monitor individual topic/partitions. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KAFKA-7218) Log thread names & ids when KafkaConsumer throws ConcurrentModificationException
[ https://issues.apache.org/jira/browse/KAFKA-7218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563194#comment-16563194 ] Kevin Lu edited comment on KAFKA-7218 at 7/31/18 6:42 AM: -- Hm... Does this require a KIP? This is a minor change, but it does affect KafkaConsumer exception log message... was (Author: lu.kevin): Hm... Does this require a KIP? > Log thread names & ids when KafkaConsumer throws > ConcurrentModificationException > > > Key: KAFKA-7218 > URL: https://issues.apache.org/jira/browse/KAFKA-7218 > Project: Kafka > Issue Type: Improvement > Components: consumer, log >Reporter: Kevin Lu >Assignee: Kevin Lu >Priority: Minor > > KafkaConsumer does not support multi-threaded usage and any access by a > thread that does not have the lock will cause ConcurrentModificationException > to be thrown. > For some users leveraging frameworks on top of Kafka that abstract the actual > KafkaConsumer class/calls, it can be hard to identify user and/or framework > bugs when this exception is thrown. > It would be incredibly helpful to log both the thread name and the thread ID > of the thread that has acquired the lock, and the current thread that is > attempting to obtain the lock in the exception message. > > KafkaConsumer currently only tracks the id of the thread that has acquired > the lock. Additionally, we should also keep track of the thread name. > An example of the exception message: "KafkaConsumer is not safe for > multi-threaded access: acquiredThreadName=acquiredThread, acquiredThreadId=1, > currentThreadName=rejectedThread, currentThreadId=2" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7218) Log thread names & ids when KafkaConsumer throws ConcurrentModificationException
[ https://issues.apache.org/jira/browse/KAFKA-7218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563194#comment-16563194 ] Kevin Lu commented on KAFKA-7218: - Hm... Does this require a KIP? > Log thread names & ids when KafkaConsumer throws > ConcurrentModificationException > > > Key: KAFKA-7218 > URL: https://issues.apache.org/jira/browse/KAFKA-7218 > Project: Kafka > Issue Type: Improvement > Components: consumer, log >Reporter: Kevin Lu >Assignee: Kevin Lu >Priority: Minor > > KafkaConsumer does not support multi-threaded usage and any access by a > thread that does not have the lock will cause ConcurrentModificationException > to be thrown. > For some users leveraging frameworks on top of Kafka that abstract the actual > KafkaConsumer class/calls, it can be hard to identify user and/or framework > bugs when this exception is thrown. > It would be incredibly helpful to log both the thread name and the thread ID > of the thread that has acquired the lock, and the current thread that is > attempting to obtain the lock in the exception message. > > KafkaConsumer currently only tracks the id of the thread that has acquired > the lock. Additionally, we should also keep track of the thread name. > An example of the exception message: "KafkaConsumer is not safe for > multi-threaded access: acquiredThreadName=acquiredThread, acquiredThreadId=1, > currentThreadName=rejectedThread, currentThreadId=2" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7219) Add topic/partition level metrics.
Satish Duggana created KAFKA-7219: - Summary: Add topic/partition level metrics. Key: KAFKA-7219 URL: https://issues.apache.org/jira/browse/KAFKA-7219 Project: Kafka Issue Type: Improvement Components: metrics Reporter: Satish Duggana Assignee: Satish Duggana Currently, Kafka generates different metrics for topics on a broker. - MessagesInPerSec - BytesInPerSec - BytesOutPerSec - BytesRejectedPerSec - ReplicationBytesInPerSec - ReplicationBytesOutPerSec - FailedProduceRequestsPerSec - FailedFetchRequestsPerSec - TotalProduceRequestsPerSec - TotalFetchRequestsPerSec - FetchMessageConversionsPerSec - ProduceMessageConversionsPerSec Add metrics for individual partitions instead of having only at topic level. Some of these partition level metrics are useful for monitoring applications to monitor individual topic/partitions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7218) Log thread names & ids when KafkaConsumer throws ConcurrentModificationException
Kevin Lu created KAFKA-7218: --- Summary: Log thread names & ids when KafkaConsumer throws ConcurrentModificationException Key: KAFKA-7218 URL: https://issues.apache.org/jira/browse/KAFKA-7218 Project: Kafka Issue Type: Improvement Components: consumer, log Reporter: Kevin Lu Assignee: Kevin Lu KafkaConsumer does not support multi-threaded usage and any access by a thread that does not have the lock will cause ConcurrentModificationException to be thrown. For some users leveraging frameworks on top of Kafka that abstract the actual KafkaConsumer class/calls, it can be hard to identify user and/or framework bugs when this exception is thrown. It would be incredibly helpful to log both the thread name and the thread ID of the thread that has acquired the lock, and the current thread that is attempting to obtain the lock in the exception message. KafkaConsumer currently only tracks the id of the thread that has acquired the lock. Additionally, we should also keep track of the thread name. An example of the exception message: "KafkaConsumer is not safe for multi-threaded access: acquiredThreadName=acquiredThread, acquiredThreadId=1, currentThreadName=rejectedThread, currentThreadId=2" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
