Re: test of producer's delay and consumer's delay

To Christian Posta , I have taken into account the interpretation of time. my producer and consumer are deployed on the same machine, the machine’s configuration is very good, so it will not

[jira] [Updated] (KAFKA-3632) ConsumerLag metrics persist after partition migration

[ https://issues.apache.org/jira/browse/KAFKA-3632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-3632: --- Fix Version/s: 0.9.0.2 > ConsumerLag metrics persist after partition migration >

[DISCUSS] KIP-64 -Allow underlying distributed filesystem to take over replication depending on configuration

Hi, Let's use this thread to discuss the above mentioned KIP. Here is the motivation for it: "Distributed data stores can be vastly improved by integrating with Kafka. Some of these improvements are: 1. They can participate easily in the whole Kafka ecosystem 2. Data ingesting speeds can

[jira] [Work started] (KAFKA-3866) KerberosLogin refresh time bug and other improvements

[ https://issues.apache.org/jira/browse/KAFKA-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-3866 started by Ismael Juma. -- > KerberosLogin refresh time bug and other improvements >

[jira] [Commented] (KAFKA-3866) KerberosLogin refresh time bug and other improvements

[ https://issues.apache.org/jira/browse/KAFKA-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338276#comment-15338276 ] ASF GitHub Bot commented on KAFKA-3866: --- GitHub user ijuma opened a pull request:

[GitHub] kafka pull request #1522: KAFKA-3866; KerberosLogin refresh time bug and oth...

GitHub user ijuma opened a pull request: https://github.com/apache/kafka/pull/1522 KAFKA-3866; KerberosLogin refresh time bug and other improvements * Handle correctly the case where the refresh is supposed to happen now * Replace `now + minTimeBeforeLogin` with `lastLogin +

[jira] [Created] (KAFKA-3867) KIP-64 -Allow underlying distributed filesystem to take over replication depending on configuration

Arvind Kandhare created KAFKA-3867: -- Summary: KIP-64 -Allow underlying distributed filesystem to take over replication depending on configuration Key: KAFKA-3867 URL:

[jira] [Created] (KAFKA-3866) KerberosLogin refresh time bug and other improvements

Ismael Juma created KAFKA-3866: -- Summary: KerberosLogin refresh time bug and other improvements Key: KAFKA-3866 URL: https://issues.apache.org/jira/browse/KAFKA-3866 Project: Kafka Issue Type:

Build failed in Jenkins: kafka-trunk-jdk7 #1374

See Changes: [wangguoz] MINOR: Check null keys in KTableSource -- [...truncated 10582 lines...] org.apache.kafka.clients.consumer.KafkaConsumerTest > testPause PASSED

Build failed in Jenkins: kafka-trunk-jdk7 #1373

See Changes: [wangguoz] HOTFIX: Check hasNext in KStreamWindowReduce -- [...truncated 10526 lines...] org.apache.kafka.clients.consumer.internals.SubscriptionStateTest > partitionAssignment

Build failed in Jenkins: kafka-trunk-jdk8 #706

See Changes: [wangguoz] HOTFIX: Check hasNext in KStreamWindowReduce [wangguoz] MINOR: Check null keys in KTableSource -- [...truncated 3331 lines...]

Re: Initiating a new KIP

Hi Arvind. What is your wiki id? Ismael On Sat, Jun 18, 2016 at 7:53 PM, Arvind Kandhare wrote: > Hi, > I would like to initiate a new KIP. It looks like I do not have access to > the wiki. Can I get the access please? > > Thanks and regards, > Arvind >

Re: FYI - Apache ZooKeeper Backup, a Treatise

Hi Flavio, Comments below. On Sat, Jun 18, 2016 at 4:50 PM, Flavio Junqueira wrote: > For acls, you can simply re-run the acl command to re-introduce them, > unless you assume that no record of acls is maintained once they are > introduced. If that's the case, then another way

[GitHub] kafka pull request #1521: MINOR: Check null keys in KTableSource

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/1521 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Re: [VOTE] KIP-62: Allow consumer to send heartbeats from a background thread

If we do that, shouldn't `max.poll.records` remain with the current default of `Integer.MAX_VALUE`? On Sat, Jun 18, 2016 at 5:18 PM, Jay Kreps wrote: > +1 > > One small but important thing I think we should consider changing: I think > we should consider setting the default

[GitHub] kafka pull request #1520: HOTFIX: Check hasNext in KStreamWindowReduce

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/1520 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Created] (KAFKA-3865) Transient failures in org.apache.kafka.connect.runtime.WorkerSourceTaskTest.testSlowTaskStart

Guozhang Wang created KAFKA-3865: Summary: Transient failures in org.apache.kafka.connect.runtime.WorkerSourceTaskTest.testSlowTaskStart Key: KAFKA-3865 URL: https://issues.apache.org/jira/browse/KAFKA-3865

[jira] [Reopened] (KAFKA-3103) Transient Failure in kafka.integration.PlaintextTopicMetadataTest.testIsrAfterBrokerShutDownAndJoinsBack

[ https://issues.apache.org/jira/browse/KAFKA-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang reopened KAFKA-3103: -- Found another case of this issue, re-opening it:

[jira] [Commented] (KAFKA-3246) Transient Failure in kafka.producer.SyncProducerTest.testProduceCorrectlyReceivesResponse

[ https://issues.apache.org/jira/browse/KAFKA-3246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15338091#comment-15338091 ] Guozhang Wang commented on KAFKA-3246: -- One more observation:

Re: Operator order

Hi Matthias, Thank you for your answer. In my use-case, depending on statistics of every operator, some tuples can be escaped for specific operators, so that we can get approximate but faster result. I think this is somehow similar to TelegraphCQ in dynamism of operators. In my case, the goal is

Initiating a new KIP

Hi, I would like to initiate a new KIP. It looks like I do not have access to the wiki. Can I get the access please? Thanks and regards, Arvind

Re: Operator order

Hi Jeyhun, there is no support by the library itself. But you could build a custom solution by building the DAG with all required edges (ie, additional edges from A->E, and B->sink etc.). For this, each output message from A would be duplicate and send to B and E. Therefore, A should "tag" each

Re: test of producer's delay and consumer's delay

So this is a crappy way to test. I think either you should test based on what the broker sees or you should test based on what the consumer or producer sees. Testing across machines with differnt interpretation of time isn't a good method. On Saturday, June 18, 2016, Kafka

Operator order

Hi community, Is there a way in Kafka Streams to change the order of operators in runtime? For example, I have operators Source->A->B->C->D->E->Sink and I want to forward some tuples from A to E, from B to Sink and etc. As far as I know, the stream execution graph is computed in compile time

Re: FYI - Apache ZooKeeper Backup, a Treatise

Why are you seeking to undo it? On 06/17/2016 09:34 PM, Andrew Purtell wrote: > HBase stores replication peering configuration in ZK. We're working on > undoing that, but for now that information exists nowhere else. > > > On Thu, Jun 16, 2016 at 2:47 PM, Ismael Juma wrote: >

Re: [VOTE] KIP-62: Allow consumer to send heartbeats from a background thread

+1 One small but important thing I think we should consider changing: I think we should consider setting the default for max.poll.interval to infinite. Previously our definition of alive was "polls within session timeout". Now our definition of alive is "pings from b/g thread w/in session

Re: FYI - Apache ZooKeeper Backup, a Treatise

For acls, you can simply re-run the acl command to re-introduce them, unless you assume that no record of acls is maintained once they are introduced. If that's the case, then another way is to simply read periodically the zk state and keep that information somewhere else to be extra safe. This

Re: test of producer's delay and consumer's delay

I send every message with timestamp, and when I receive a message，I do a subtraction between current timestamp and message’s timestamp. then I get the consumer’s delay. > 在 2016年6月18日，上午11:28，Kafka 写道： > >

[jira] [Created] (KAFKA-3864) Kafka Connect Struct.get returning defaultValue from Struct not the field schema

Andrew Stevenson created KAFKA-3864: --- Summary: Kafka Connect Struct.get returning defaultValue from Struct not the field schema Key: KAFKA-3864 URL: https://issues.apache.org/jira/browse/KAFKA-3864

[jira] [Commented] (KAFKA-1789) Issue with Async producer

[ https://issues.apache.org/jira/browse/KAFKA-1789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15337883#comment-15337883 ] Ismael Juma commented on KAFKA-1789: To add to Gwen's comment, the Scala producers have been

[jira] [Resolved] (KAFKA-1789) Issue with Async producer

[ https://issues.apache.org/jira/browse/KAFKA-1789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira resolved KAFKA-1789. - Resolution: Unresolved Unlikely that this will get resolved. > Issue with Async producer >

Re: test of producer's delay and consumer's delay

How did you measure consumer delay? On Sat, Jun 18, 2016 at 6:28 AM, Kafka wrote: > hello，I have done a series of tests on kafka 0.9.0，and one of the results > confused me. > > test enviroment: > kafka cluster: 3 brokers，8core cpu ／ 8g mem ／1g netcard > client：4core