[jira] [Updated] (KAFKA-16900) kafka-producer-perf-test reports error when using transaction.
[ https://issues.apache.org/jira/browse/KAFKA-16900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen He updated KAFKA-16900: Labels: perf-test (was: ) > kafka-producer-perf-test reports error when using transaction. > -- > > Key: KAFKA-16900 > URL: https://issues.apache.org/jira/browse/KAFKA-16900 > Project: Kafka > Issue Type: Bug > Components: producer >Affects Versions: 2.9 >Reporter: Chen He >Priority: Minor > Labels: perf-test > > [https://lists.apache.org/thread/dmrbx8kzv2w5t1v0xjvyjbp5y23omlq8] > encounter the same issue as mentioned above. > Did not found the 2.13 version in affects versions so mark it as the most > latest it provided. 2.9. Please feel free to change if possible. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16900) kafka-producer-perf-test reports error when using transaction.
Chen He created KAFKA-16900: --- Summary: kafka-producer-perf-test reports error when using transaction. Key: KAFKA-16900 URL: https://issues.apache.org/jira/browse/KAFKA-16900 Project: Kafka Issue Type: Bug Components: producer Affects Versions: 2.9 Reporter: Chen He [https://lists.apache.org/thread/dmrbx8kzv2w5t1v0xjvyjbp5y23omlq8] encounter the same issue as mentioned above. Did not found the 2.13 version in affects versions so mark it as the most latest it provided. 2.9. Please feel free to change if possible. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-15983) Kafka-acls should return authorization already done if repeating work is issued
[
https://issues.apache.org/jira/browse/KAFKA-15983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen He updated KAFKA-15983:
Summary: Kafka-acls should return authorization already done if repeating
work is issued (was: Kafka-acls should return authorization already done if
repeat work is issued)
> Kafka-acls should return authorization already done if repeating work is
> issued
> ---
>
> Key: KAFKA-15983
> URL: https://issues.apache.org/jira/browse/KAFKA-15983
> Project: Kafka
> Issue Type: Improvement
> Components: security
>Affects Versions: 3.6.0
>Reporter: Chen He
>Priority: Minor
> Labels: newbie
>
> kafka-acls.sh cmd will always issue normal operation for a cmd if customer
> already authorized a user. It should reports something like "user {} already
> authorized with {} resources" instead of do it again and again.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-15983) Kafka-acls should return authorization already done if repeat work is issued
[
https://issues.apache.org/jira/browse/KAFKA-15983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen He updated KAFKA-15983:
Labels: newbie (was: )
> Kafka-acls should return authorization already done if repeat work is issued
>
>
> Key: KAFKA-15983
> URL: https://issues.apache.org/jira/browse/KAFKA-15983
> Project: Kafka
> Issue Type: Improvement
> Components: security
>Affects Versions: 3.6.0
>Reporter: Chen He
>Priority: Minor
> Labels: newbie
>
> kafka-acls.sh cmd will always issue normal operation for a cmd if customer
> already authorized a user. It should reports something like "user {} already
> authorized with {} resources" instead of do it again and again.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-15983) Kafka-acls should return authorization already done if repeat work is issued
Chen He created KAFKA-15983:
---
Summary: Kafka-acls should return authorization already done if
repeat work is issued
Key: KAFKA-15983
URL: https://issues.apache.org/jira/browse/KAFKA-15983
Project: Kafka
Issue Type: Improvement
Components: security
Affects Versions: 3.6.0
Reporter: Chen He
kafka-acls.sh cmd will always issue normal operation for a cmd if customer
already authorized a user. It should reports something like "user {} already
authorized with {} resources" instead of do it again and again.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-7381) Parameterize connector rebalancing behavior
Chen He created KAFKA-7381: -- Summary: Parameterize connector rebalancing behavior Key: KAFKA-7381 URL: https://issues.apache.org/jira/browse/KAFKA-7381 Project: Kafka Issue Type: Improvement Components: KafkaConnect Affects Versions: 1.0.0 Reporter: Chen He Assignee: Chen He I have a question about connector rebalancing issue. Why don't we make it option, I mean have a parameter that turn on/off it instead of having it as a must? We can have a parameter like: "connector.rebalancing.enable" parameter and make it as "true" by default. It allows users to turn it off if they want. There are some cases that connector rebalancing is not needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-6602) Support Kafka to save credentials in Java Key Store on Zookeeper node
Chen He created KAFKA-6602: -- Summary: Support Kafka to save credentials in Java Key Store on Zookeeper node Key: KAFKA-6602 URL: https://issues.apache.org/jira/browse/KAFKA-6602 Project: Kafka Issue Type: New Feature Components: security Reporter: Chen He Kafka connect needs to talk to multifarious distributed systems. However, each system has its own authentication mechanism. How we manage these credentials become a common problem. Here are my thoughts: # We may need to save it in java key store; # We may need to put this key store in a distributed system (topic or zookeeper); # Key store password may be configured in Kafka configuration; I have implement the feature that allows store java key store in zookeeper node. If Kafka community likes this idea, I am happy to contribute. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KAFKA-6602) Support Kafka to save credentials in Java Key Store on Zookeeper node
[ https://issues.apache.org/jira/browse/KAFKA-6602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen He reassigned KAFKA-6602: -- Assignee: Chen He > Support Kafka to save credentials in Java Key Store on Zookeeper node > - > > Key: KAFKA-6602 > URL: https://issues.apache.org/jira/browse/KAFKA-6602 > Project: Kafka > Issue Type: New Feature > Components: security >Reporter: Chen He >Assignee: Chen He >Priority: Major > > Kafka connect needs to talk to multifarious distributed systems. However, > each system has its own authentication mechanism. How we manage these > credentials become a common problem. > Here are my thoughts: > # We may need to save it in java key store; > # We may need to put this key store in a distributed system (topic or > zookeeper); > # Key store password may be configured in Kafka configuration; > I have implement the feature that allows store java key store in zookeeper > node. If Kafka community likes this idea, I am happy to contribute. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-6353) Connector status shows FAILED but actually task is in RUNNING status
[
https://issues.apache.org/jira/browse/KAFKA-6353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295377#comment-16295377
]
Chen He commented on KAFKA-6353:
Thank you, Mr. [~rhauch]. Just sent the email.
Also some updates on this issue. The problem is when user first submit request
to create a connector, they announce the number of tasks. It means there are 2
levels of status: Connector level and task level. We need first clarify the
relationship between these 2 levels.
1. What is the definition of the connector level "RUNNING": all tasks are
running, some tasks are running, or not related to task status and just reflect
the ability to create new tasks through connecor;
2. How to deal with the case, if all tasks are "FAILED", should we still mark
connector as "RUNNING" ?
IMHO
a) Task status only means there is a task still running, even all tasks failed,
we can still mark connector as running if it can create new task.
b) Connector Status should also reflect current tasks existence. It means if
there is one task running, but connector lose the ability to create new task,
we should mark it a different status (like "SUSPENDED") other than "FAILED", in
this way, UI or other monitoring system knows there is problem in connector but
there still some tasks well functioning. It can help them to make right
decision.
c) The reason why this issue happens is that tasks are still running correctly,
however, connector failed to restart (timeout reach or any other problem
happens in the connector level.) We may need to find an answer about why we set
timeout.
> Connector status shows FAILED but actually task is in RUNNING status
>
>
> Key: KAFKA-6353
> URL: https://issues.apache.org/jira/browse/KAFKA-6353
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 0.10.2.1
>Reporter: Chen He
>
> {"name":"test","connector":{"state":"FAILED","trace":"ERROR
> MESSAGE","worker_id":"localhost:8083"},"tasks":[{"state":"RUNNING","id":0,"worker_id":"localhost:8083"}]}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (KAFKA-6353) Connector status shows FAILED but actually task is in RUNNING status
Chen He created KAFKA-6353:
--
Summary: Connector status shows FAILED but actually task is in
RUNNING status
Key: KAFKA-6353
URL: https://issues.apache.org/jira/browse/KAFKA-6353
Project: Kafka
Issue Type: Bug
Components: KafkaConnect
Affects Versions: 0.10.2.1
Reporter: Chen He
{"name":"test","connector":{"state":"FAILED","trace":"ERROR
MESSAGE","worker_id":"localhost:8083"},"tasks":[{"state":"RUNNING","id":0,"worker_id":"localhost:8083"}]}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-6353) Connector status shows FAILED but actually task is in RUNNING status
[
https://issues.apache.org/jira/browse/KAFKA-6353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16288329#comment-16288329
]
Chen He commented on KAFKA-6353:
Can anyone make me contributor then I can work on this. Thanks!
> Connector status shows FAILED but actually task is in RUNNING status
>
>
> Key: KAFKA-6353
> URL: https://issues.apache.org/jira/browse/KAFKA-6353
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 0.10.2.1
>Reporter: Chen He
>
> {"name":"test","connector":{"state":"FAILED","trace":"ERROR
> MESSAGE","worker_id":"localhost:8083"},"tasks":[{"state":"RUNNING","id":0,"worker_id":"localhost:8083"}]}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-3554) Generate actual data with specific compression ratio and add multi-thread support in the ProducerPerformance tool.
[ https://issues.apache.org/jira/browse/KAFKA-3554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16245266#comment-16245266 ] Chen He commented on KAFKA-3554: This is a great feature, [~becket_qin]. What can I do for you to make it checked in to Kafka. I am happy to contribute my time and efforts. > Generate actual data with specific compression ratio and add multi-thread > support in the ProducerPerformance tool. > -- > > Key: KAFKA-3554 > URL: https://issues.apache.org/jira/browse/KAFKA-3554 > Project: Kafka > Issue Type: Improvement >Affects Versions: 0.9.0.1 >Reporter: Jiangjie Qin >Assignee: Jiangjie Qin > Fix For: 1.1.0 > > > Currently the ProducerPerformance always generate the payload with same > bytes. This does not quite well to test the compressed data because the > payload is extremely compressible no matter how big the payload is. > We can make some changes to make it more useful for compressed messages. > Currently I am generating the payload containing integer from a given range. > By adjusting the range of the integers, we can get different compression > ratios. > API wise, we can either let user to specify the integer range or the expected > compression ratio (we will do some probing to get the corresponding range for > the users) > Besides that, in many cases, it is useful to have multiple producer threads > when the producer threads themselves are bottleneck. Admittedly people can > run multiple ProducerPerformance to achieve similar result, but it is still > different from the real case when people actually use the producer. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5935) Kafka connect should provide configurable CONNECTOR_EXCLUDES
[ https://issues.apache.org/jira/browse/KAFKA-5935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen He updated KAFKA-5935: --- Labels: connector (was: ) > Kafka connect should provide configurable CONNECTOR_EXCLUDES > > > Key: KAFKA-5935 > URL: https://issues.apache.org/jira/browse/KAFKA-5935 > Project: Kafka > Issue Type: Improvement >Reporter: Chen He >Priority: Minor > Labels: connector > > In o.a.kafka.connect.runtime.rest.resources.ConnectorPluginsResource, there > is a CONNECTOR_EXCLUDES list which is in charge of filtering connectors > classes that will be shown through the REST_API. > hostname:8083/connector-plugins/ > It will be great if we can have a similar config in kafka-connect.properties > that exclude given connectors. Then, we can select which connector should be > exposed through the REST_API instead of directly post all founded classname > there. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5935) Kafka connect should provide configurable CONNECTOR_EXCLUDES
[ https://issues.apache.org/jira/browse/KAFKA-5935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen He updated KAFKA-5935: --- Description: In o.a.kafka.connect.runtime.rest.resources.ConnectorPluginsResource, there is a CONNECTOR_EXCLUDES list which is in charge of filtering connectors classes that will be shown through the REST_API. hostname:8083/connector-plugins/ It will be great if we can have a similar config in kafka-connect.properties that exclude given connectors. Then, we can select which connector should be exposed through the REST_API instead of directly post all founded classname there. was: In o.a.kafka.connect.runtime.rest.resources.ConnectorPluginsResource, there is a CONNECTOR_EXCLUDES list which is in charge of filtering connectors classes that will be shown through the REST_API. It will be great if we can have a similar config in kafka-connect.properties that exclude given connectors. Then, we can select which connector should be exposed through the REST_API instead of directly post all founded classname there. > Kafka connect should provide configurable CONNECTOR_EXCLUDES > > > Key: KAFKA-5935 > URL: https://issues.apache.org/jira/browse/KAFKA-5935 > Project: Kafka > Issue Type: Improvement >Reporter: Chen He >Priority: Minor > > In o.a.kafka.connect.runtime.rest.resources.ConnectorPluginsResource, there > is a CONNECTOR_EXCLUDES list which is in charge of filtering connectors > classes that will be shown through the REST_API. > hostname:8083/connector-plugins/ > It will be great if we can have a similar config in kafka-connect.properties > that exclude given connectors. Then, we can select which connector should be > exposed through the REST_API instead of directly post all founded classname > there. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-5935) Kafka connect should provide configurable CONNECTOR_EXCLUDES
Chen He created KAFKA-5935: -- Summary: Kafka connect should provide configurable CONNECTOR_EXCLUDES Key: KAFKA-5935 URL: https://issues.apache.org/jira/browse/KAFKA-5935 Project: Kafka Issue Type: Improvement Reporter: Chen He Priority: Minor In o.a.kafka.connect.runtime.rest.resources.ConnectorPluginsResource, there is a CONNECTOR_EXCLUDES list which is in charge of filtering connectors classes that will be shown through the REST_API. It will be great if we can have a similar config in kafka-connect.properties that exclude given connectors. Then, we can select which connector should be exposed through the REST_API instead of directly post all founded classname there. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-2499) kafka-producer-perf-test should use something more realistic than empty byte arrays
[ https://issues.apache.org/jira/browse/KAFKA-2499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16115040#comment-16115040 ] Chen He commented on KAFKA-2499: Does Kafka-3554 resolve this problem? User can control the compression ratio. > kafka-producer-perf-test should use something more realistic than empty byte > arrays > --- > > Key: KAFKA-2499 > URL: https://issues.apache.org/jira/browse/KAFKA-2499 > Project: Kafka > Issue Type: Bug >Reporter: Ben Stopford >Assignee: Edward Ribeiro > Labels: newbie > > ProducerPerformance.scala (There are two of these, one used by the shell > script and one used by the system tests. Both exhibit this problem) > creates messags from empty byte arrays. > This is likely to provide unrealistically fast compression and hence > unrealistically fast results. > Suggest randomised bytes or more realistic sample messages are used. > Thanks to Prabhjot Bharaj for reporting this. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-3408) consumer rebalance fail
[ https://issues.apache.org/jira/browse/KAFKA-3408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16115038#comment-16115038 ] Chen He commented on KAFKA-3408: Is this jira still in the latest Kafka? If so, may I work on it if no one is currently working on it? > consumer rebalance fail > --- > > Key: KAFKA-3408 > URL: https://issues.apache.org/jira/browse/KAFKA-3408 > Project: Kafka > Issue Type: Bug > Components: clients >Affects Versions: 0.9.0.1 > Environment: centos linux >Reporter: zhongkai liu > Labels: newbie > > I use "/bin/kafka-console-consumer" command to start two consumers of group > "page_group",then the first conumer console report rebalance failure like > this: > ERROR [page_view_group1_slave2-1458095694092-80c33086], error during > syncedRebalance (kafka.consumer.ZookeeperConsumerConnector) > kafka.common.ConsumerRebalanceFailedException: > page_view_group1_slave2-1458095694092-80c33086 can't rebalance after 10 > retries > at > kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:660) > at > kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anon$1.run(ZookeeperConsumerConnector.scala:579) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-5705) Kafka Server start failed and reports "unsafe memory access operation"
Chen He created KAFKA-5705: -- Summary: Kafka Server start failed and reports "unsafe memory access operation" Key: KAFKA-5705 URL: https://issues.apache.org/jira/browse/KAFKA-5705 Project: Kafka Issue Type: Bug Components: log Affects Versions: 0.10.2.0 Reporter: Chen He [2017-08-02 15:50:23,361] FATAL Fatal error during KafkaServerStartable startup. Prepare to shutdown (kafka.server.KafkaServerStartable) java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code at kafka.log.TimeIndex$$anonfun$maybeAppend$1.apply$mcV$sp(TimeIndex.scala:128) at kafka.log.TimeIndex$$anonfun$maybeAppend$1.apply(TimeIndex.scala:107) at kafka.log.TimeIndex$$anonfun$maybeAppend$1.apply(TimeIndex.scala:107) at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:213) at kafka.log.TimeIndex.maybeAppend(TimeIndex.scala:107) at kafka.log.LogSegment.recover(LogSegment.scala:252) at kafka.log.Log$$anonfun$loadSegments$4.apply(Log.scala:231) at kafka.log.Log$$anonfun$loadSegments$4.apply(Log.scala:188) at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186) at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732) at kafka.log.Log.loadSegments(Log.scala:188) at kafka.log.Log.(Log.scala:116) at kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$10$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:157) at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:57) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-3554) Generate actual data with specific compression ratio and add multi-thread support in the ProducerPerformance tool.
[ https://issues.apache.org/jira/browse/KAFKA-3554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063452#comment-16063452 ] Chen He commented on KAFKA-3554: Thank you for the quick reply [~becket_qin]. This work is really valuable. It provides us a tool that can exploit kafka system's capacity. For example, we can get lowest latency by only use 1 thread, at the same time, by increasing thread, we can find what is the maximum throughput for a kafka cluster. Only one question, I did applied this patch to latest kafka and comparing results with old ProducerPerformance.java file. I found out, if we set ack=all with snappy compression, with 100M record(100B each), it does not work as well as old PproducerPerformance.java file. > Generate actual data with specific compression ratio and add multi-thread > support in the ProducerPerformance tool. > -- > > Key: KAFKA-3554 > URL: https://issues.apache.org/jira/browse/KAFKA-3554 > Project: Kafka > Issue Type: Improvement >Affects Versions: 0.9.0.1 >Reporter: Jiangjie Qin >Assignee: Jiangjie Qin > Fix For: 0.11.1.0 > > > Currently the ProducerPerformance always generate the payload with same > bytes. This does not quite well to test the compressed data because the > payload is extremely compressible no matter how big the payload is. > We can make some changes to make it more useful for compressed messages. > Currently I am generating the payload containing integer from a given range. > By adjusting the range of the integers, we can get different compression > ratios. > API wise, we can either let user to specify the integer range or the expected > compression ratio (we will do some probing to get the corresponding range for > the users) > Besides that, in many cases, it is useful to have multiple producer threads > when the producer threads themselves are bottleneck. Admittedly people can > run multiple ProducerPerformance to achieve similar result, but it is still > different from the real case when people actually use the producer. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-5518) General Kafka connector performanc workload
Chen He created KAFKA-5518: -- Summary: General Kafka connector performanc workload Key: KAFKA-5518 URL: https://issues.apache.org/jira/browse/KAFKA-5518 Project: Kafka Issue Type: Bug Components: KafkaConnect Affects Versions: 0.10.2.1 Reporter: Chen He Sorry, first time to create Kafka JIRA. Just curious whether there is a general purpose performance workload for Kafka connector (hdfs, s3, etc). Then, we can setup an standard and evaluate the performance for further connectors such as swift, etc. Please feel free to comment or mark as dup if there already is one jira tracking this. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
