[jira] [Resolved] (KAFKA-3918) Broker faills to start after ungraceful shutdown due to non-monotonically incrementing offsets in logs

[ https://issues.apache.org/jira/browse/KAFKA-3918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andy Coates resolved KAFKA-3918. Resolution: Duplicate > Broker faills to start after ungraceful shutdown due to non-monotonically >

[jira] [Updated] (KAFKA-3854) Subsequent regex subscription calls fail

[ https://issues.apache.org/jira/browse/KAFKA-3854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vahid Hashemian updated KAFKA-3854: --- Status: Patch Available (was: In Progress) > Subsequent regex subscription calls fail > -

[GitHub] kafka pull request #1555: MINOR: Fix ambiguous log message in RecordCollecto...

GitHub user kawamuray reopened a pull request: https://github.com/apache/kafka/pull/1555 MINOR: Fix ambiguous log message in RecordCollector When producing fails in Kafka Streams, it gives an error like below: ``` Error sending record: null ``` by this line:

[GitHub] kafka pull request #1555: MINOR: Fix ambiguous log message in RecordCollecto...

Github user kawamuray closed the pull request at: https://github.com/apache/kafka/pull/1555 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is e

[jira] [Commented] (KAFKA-3794) Add Stream / Table prefix in print functions

[ https://issues.apache.org/jira/browse/KAFKA-3794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15356376#comment-15356376 ] Guozhang Wang commented on KAFKA-3794: -- Sounds good to me. Generally speaking, the a

[jira] [Commented] (KAFKA-3794) Add Stream / Table prefix in print functions

[ https://issues.apache.org/jira/browse/KAFKA-3794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15356319#comment-15356319 ] Bill Bejeck commented on KAFKA-3794: [~guozhang] I was thinking of adding the parent

Build failed in Jenkins: kafka-trunk-jdk7 #1394

See Changes: [wangguoz] MINOR: Follow-up from KAFKA-3842 to fix tempDir [wangguoz] KAFKA-3890: Streams use same task assignment on cluster rolling restart -- [...truncated 3369 lines...] kafk

Jenkins build is back to normal : kafka-trunk-jdk8 #724

See

[jira] [Commented] (KAFKA-3628) Native Schema Registry in Kafka

[ https://issues.apache.org/jira/browse/KAFKA-3628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15356233#comment-15356233 ] Sriharsha Chintalapani commented on KAFKA-3628: --- draft version is here http

[GitHub] kafka pull request #1575: hotfix: check join window boundaries

GitHub user mjsax opened a pull request: https://github.com/apache/kafka/pull/1575 hotfix: check join window boundaries @guozhangwang You can merge this pull request into a Git repository by running: $ git pull https://github.com/mjsax/kafka hotfix2 Alternatively you can revi

[jira] [Commented] (KAFKA-3920) Add Schema source connector to Kafka Connect

[ https://issues.apache.org/jira/browse/KAFKA-3920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15356224#comment-15356224 ] ASF GitHub Bot commented on KAFKA-3920: --- GitHub user Ishiihara opened a pull request

[GitHub] kafka pull request #1574: KAFKA-3920: Add Schema source connector to Kafka C...

GitHub user Ishiihara opened a pull request: https://github.com/apache/kafka/pull/1574 KAFKA-3920: Add Schema source connector to Kafka Connect You can merge this pull request into a Git repository by running: $ git pull https://github.com/Ishiihara/kafka schema-source Altern

[GitHub] kafka pull request #1573: hotfix: check join window boundaries

GitHub user mjsax opened a pull request: https://github.com/apache/kafka/pull/1573 hotfix: check join window boundaries @guozhangwang Might need to rebase after https://github.com/apache/kafka/commit/b669b2786c69d2dfa719033eb7c02ac444f115f4 got dropped... You can merge this pull r

[jira] [Created] (KAFKA-3920) Add Schema source connector to Kafka Connect

Liquan Pei created KAFKA-3920: - Summary: Add Schema source connector to Kafka Connect Key: KAFKA-3920 URL: https://issues.apache.org/jira/browse/KAFKA-3920 Project: Kafka Issue Type: Improvement

Build failed in Jenkins: kafka-trunk-jdk7 #1393

See Changes: [ismael] KAFKA-3761; Remove BrokerState "RunningAsController" -- [...truncated 1632 lines...] kafka.api.PlaintextConsumerTest > testCommitSpecifiedOffsets STARTED kafka.api.Plainte

[jira] [Commented] (KAFKA-3854) Subsequent regex subscription calls fail

[ https://issues.apache.org/jira/browse/KAFKA-3854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15355981#comment-15355981 ] ASF GitHub Bot commented on KAFKA-3854: --- GitHub user vahidhashemian opened a pull re

[GitHub] kafka pull request #1572: KAFKA-3854: Fix issues with new consumer's subsequ...

GitHub user vahidhashemian opened a pull request: https://github.com/apache/kafka/pull/1572 KAFKA-3854: Fix issues with new consumer's subsequent regex (pattern) subscriptions This patch fixes two issues: 1. Subsequent regex subscriptions fail with the new consumer. 2. Subse

[jira] [Commented] (KAFKA-3522) Consider adding version information into rocksDB storage format

[ https://issues.apache.org/jira/browse/KAFKA-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15355948#comment-15355948 ] Guozhang Wang commented on KAFKA-3522: -- @migno @imandhan [~jkreps] Here is my proposa

Re: KTable.filter usage, memory consumption and materialized view semantics

good. On Wed, Jun 29, 2016 at 6:44 PM, Guozhang Wang wrote: > Yes, they are related in the sense that if we always materialize a source > KTable, then we can completely replace the `sendOldValues` as it will > always be true. But since 3911 is a rather big change, I'd prefer to > complete this t

Re: KTable.filter usage, memory consumption and materialized view semantics

Yes, they are related in the sense that if we always materialize a source KTable, then we can completely replace the `sendOldValues` as it will always be true. But since 3911 is a rather big change, I'd prefer to complete this ticket first, and refactor it when we decided to work on 3911 later. Fe

[jira] [Resolved] (KAFKA-3890) Kafka Streams: task assignment is not maintained on cluster restart or rolling restart

[ https://issues.apache.org/jira/browse/KAFKA-3890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-3890. -- Resolution: Fixed Fix Version/s: 0.10.0.1 Issue resolved by pull request 1543 [https://gi

[jira] [Commented] (KAFKA-3842) Add Helper Functions Into TestUtils

[ https://issues.apache.org/jira/browse/KAFKA-3842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15355909#comment-15355909 ] Guozhang Wang commented on KAFKA-3842: -- https://github.com/apache/kafka/pull/1554 has

[jira] [Commented] (KAFKA-3890) Kafka Streams: task assignment is not maintained on cluster restart or rolling restart

[ https://issues.apache.org/jira/browse/KAFKA-3890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15355908#comment-15355908 ] ASF GitHub Bot commented on KAFKA-3890: --- Github user asfgit closed the pull request

[GitHub] kafka pull request #1543: KAFKA-3890 Kafka Streams: task assignment is not m...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/1543 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[jira] [Commented] (KAFKA-3842) Add Helper Functions Into TestUtils

[ https://issues.apache.org/jira/browse/KAFKA-3842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15355904#comment-15355904 ] Guozhang Wang commented on KAFKA-3842: -- https://github.com/apache/kafka/pull/1554 has

[jira] [Commented] (KAFKA-3842) Add Helper Functions Into TestUtils

[ https://issues.apache.org/jira/browse/KAFKA-3842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15355903#comment-15355903 ] ASF GitHub Bot commented on KAFKA-3842: --- Github user asfgit closed the pull request

[GitHub] kafka pull request #1554: MINOR: Follow-up from KAFKA-3842 with suggested fi...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/1554 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

Re: Use Kafka 0.9.0.1 or 0.10.0.0 with Zookeeper 3.5

Hi Luciano, I can't remember seeing a discussion in this community about 3.5. I suspect your real question is how to set it up to use TLS/SSL because everything else should remain the same. You should be able to run a Kafka cluster as is against a 3.5 ensemble, but you won't have a secure conne

Build failed in Jenkins: kafka-trunk-jdk8 #723

See Changes: [ismael] KAFKA-3761; Remove BrokerState "RunningAsController" -- [...truncated 4726 lines...] kafka.utils.ZkUtilsTest > testAbortedConditionalDeletePath STARTED kafka.utils.ZkUtils

[jira] [Updated] (KAFKA-3854) Subsequent regex subscription calls fail

[ https://issues.apache.org/jira/browse/KAFKA-3854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vahid Hashemian updated KAFKA-3854: --- Description: There are a couple of issues with regex subscription in the new consumer: h6. Su

[jira] [Updated] (KAFKA-3854) Subsequent regex subscription calls fail

[ https://issues.apache.org/jira/browse/KAFKA-3854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vahid Hashemian updated KAFKA-3854: --- Description: There are a couple of issues with regex subscription in the new consumer: h6. Su

Use Kafka 0.9.0.1 or 0.10.0.0 with Zookeeper 3.5

Hi I would like some advice about what are the changes, at high level, required to use kafka 0.9.0.1 or 0.10.0.0 with Zookeeper 3.5.x (3.5.1-alpha for example) using TLS/SSL. How big are the changes required in Kafka in order to be able to do this? Regards Luciano

[GitHub] kafka pull request #1437: KAFKA-3761: Remove BrokerState "RunningAsControlle...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/1437 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] kafka pull request #1571: Fix few documentation errors in streams quickstart

GitHub user glikson opened a pull request: https://github.com/apache/kafka/pull/1571 Fix few documentation errors in streams quickstart Plus a minor enhancement You can merge this pull request into a Git repository by running: $ git pull https://github.com/glikson/kafka patch-1

Re: [DISCUSS] Client Side Auto Topic Creation

Hi Roger and Constantine, Thanks for the feedback. I agree that configuration to maintain guarantees is commonly spread across enterprise teams, making it difficult to get right. That said its also hard to solve for every company structure too. I think there is room for an open discussion about w

Re: [Kafka Streams] Source Nodes

Thanks Matthias! Got it working with through, still use 0.10.0.0 version. > On Jun 29, 2016, at 22:42, Matthias J. Sax wrote: > > Hi, > > for joins, data of both inputs must be co-located, ie, partitioned on > the same key and have the same number of partitions: > > See "Note" box at: > http:

Re: [DISCUSS] Client Side Auto Topic Creation

We currently run with auto topic creation enabled, largely to ensure that our topics all get created with the cluster defaults. My understanding is that this is the only to ensure this, since the defaults are not accessible to clients. We run a cluster per deployment, with the defaults are set

Re: [DISCUSS] Client Side Auto Topic Creation

Roger, I concur with everything you said. Couple more use cases to prove the point: 1. Some topics should always have 1 and only one partition. 2. CDC application based on Kafka Connect. Those type of application absolutely must know how to create properly configured topics: compacted,

Re: [Kafka Streams] Source Nodes

Hi, for joins, data of both inputs must be co-located, ie, partitioned on the same key and have the same number of partitions: See "Note" box at: http://docs.confluent.io/3.0.0/streams/developer-guide.html?highlight=join#joining-streams From an older email thread about the same issue: >>> If yo

[jira] [Commented] (KAFKA-3822) Kafka Consumer close() hangs indefinitely if Kafka Broker shutdown while connected

[ https://issues.apache.org/jira/browse/KAFKA-3822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15355584#comment-15355584 ] Jason Gustafson commented on KAFKA-3822: [~singhashish] There are two reasons why

Re: [DISCUSS] Client Side Auto Topic Creation

My comments go a bit beyond just topic creation but I'd like to see Kafka make it easier for application developers to specify their requirements declaratively in a single place. Today, for example, if your application requires strong guarantees against data loss, you must set a mix of topic-level

[jira] [Updated] (KAFKA-3900) High CPU util on broker

[ https://issues.apache.org/jira/browse/KAFKA-3900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Konyaev updated KAFKA-3900: -- Component/s: replication network > High CPU util on broker > --

[jira] [Updated] (KAFKA-3900) High CPU util on broker

[ https://issues.apache.org/jira/browse/KAFKA-3900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Konyaev updated KAFKA-3900: -- Affects Version/s: 0.10.0.0 > High CPU util on broker > --- > >

[jira] [Updated] (KAFKA-3854) Subsequent regex subscription calls fail

[ https://issues.apache.org/jira/browse/KAFKA-3854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vahid Hashemian updated KAFKA-3854: --- Summary: Subsequent regex subscription calls fail (was: Consecutive regex subscription calls

[jira] [Updated] (KAFKA-3854) Consecutive regex subscription calls fail

[ https://issues.apache.org/jira/browse/KAFKA-3854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vahid Hashemian updated KAFKA-3854: --- Description: There are a couple of issues with regex subscription in the new consumer: h6. Su

[jira] [Updated] (KAFKA-3854) Consecutive regex subscription calls fail

[ https://issues.apache.org/jira/browse/KAFKA-3854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vahid Hashemian updated KAFKA-3854: --- Description: There are a couple of issues with regex subscription in the new consumer: # When

[Kafka Streams] Source Nodes

Hey, I am trying to understand kafka-streams and doing a simple prototype for joining KStream with KTable, but stuck at the error: Invalid topology building: KSTREAM-MAP-01 and KSTREAM-AGGREGATE-04 are not joinable I tracked down the issue is thrown where stream or table have s

[jira] [Updated] (KAFKA-3854) Consecutive regex subscription calls fail

[ https://issues.apache.org/jira/browse/KAFKA-3854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vahid Hashemian updated KAFKA-3854: --- Description: # When consecutive calls are made to new consumer's [regex subscription|https://

Re: [DISCUSS] Client Side Auto Topic Creation

Thanks for the discussion, below are some thoughts and responses. One of the problems that we currently have with > the clients is that we retry silently on unknown topics under the > expectation that they will eventually be created (automatically or not). > This makes it difficult to detect misco

[jira] [Work started] (KAFKA-3914) Global discovery of state stores

[ https://issues.apache.org/jira/browse/KAFKA-3914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-3914 started by Damian Guy. - > Global discovery of state stores > > >

Kafka:3663 : Help with Jenkin Check Failures in Pull Request on Github

Hi, I submitted a pull request on Github - https://github.com/apache/kafka/pull/1539 Apparently the pull request shows a Jenkins failure - but without any details. Can anyone help me identify the issue (if any) or point in the right direction? Thanks,Jayesh

[GitHub] kafka pull request #1570: KAFKA-3914: Global discovery of state stores

GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/1570 KAFKA-3914: Global discovery of state stores @guozhangwang @enothereska please take a look. A few things that need to be clarified 1. I've added `StreamsConfig.USER_ENDPOINT_CONFIG`, but shoul

[jira] [Commented] (KAFKA-3914) Global discovery of state stores

[ https://issues.apache.org/jira/browse/KAFKA-3914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15355415#comment-15355415 ] ASF GitHub Bot commented on KAFKA-3914: --- GitHub user dguy opened a pull request:

[jira] [Commented] (KAFKA-3919) Broker faills to start after ungraceful shutdown due to non-monotonically incrementing offsets in logs

[ https://issues.apache.org/jira/browse/KAFKA-3919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15355393#comment-15355393 ] Jun Rao commented on KAFKA-3919: [~BigAndy], thanks for reporting and investigating this.

Re: [DISCUSS] Client Side Auto Topic Creation

For what it's worth the use case for auto-creation isn't using a dynamic set of topics, but rather letting apps flow through different dev/staging/prod/integration_testing/unit_testing environments without having the app configure appropriate replication/partitioning stuff in each environment and h

[jira] [Commented] (KAFKA-3919) Broker faills to start after ungraceful shutdown due to non-monotonically incrementing offsets in logs

[ https://issues.apache.org/jira/browse/KAFKA-3919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15355316#comment-15355316 ] Ismael Juma commented on KAFKA-3919: Thanks for the report. Did you mean to file both

[jira] [Commented] (KAFKA-3915) LogCleaner IO buffers do not account for potential size difference due to message format change

[ https://issues.apache.org/jira/browse/KAFKA-3915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15355150#comment-15355150 ] Tommy Becker commented on KAFKA-3915: - Actually, I don't think this is true. The buffe

[jira] [Updated] (KAFKA-3919) Broker faills to start after ungraceful shutdown due to non-monotonically incrementing offsets in logs

[ https://issues.apache.org/jira/browse/KAFKA-3919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andy Coates updated KAFKA-3919: --- Description: Hi All, I encountered an issue with Kafka following a power outage that saw a proportio

[jira] [Created] (KAFKA-3918) Broker faills to start after ungraceful shutdown due to non-monotonically incrementing offsets in logs

Andy Coates created KAFKA-3918: -- Summary: Broker faills to start after ungraceful shutdown due to non-monotonically incrementing offsets in logs Key: KAFKA-3918 URL: https://issues.apache.org/jira/browse/KAFKA-3918

[jira] [Created] (KAFKA-3919) Broker faills to start after ungraceful shutdown due to non-monotonically incrementing offsets in logs

Andy Coates created KAFKA-3919: -- Summary: Broker faills to start after ungraceful shutdown due to non-monotonically incrementing offsets in logs Key: KAFKA-3919 URL: https://issues.apache.org/jira/browse/KAFKA-3919

[jira] [Commented] (KAFKA-3915) LogCleaner IO buffers do not account for potential size difference due to message format change

[ https://issues.apache.org/jira/browse/KAFKA-3915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15355135#comment-15355135 ] Tommy Becker commented on KAFKA-3915: - Should a change to the default maxMessageSize b

[GitHub] kafka pull request #1569: Kafka 3836: KStreamReduce and KTableReduce should ...

GitHub user jeyhunkarimov opened a pull request: https://github.com/apache/kafka/pull/1569 Kafka 3836: KStreamReduce and KTableReduce should not pass nulls to Deserializers You can merge this pull request into a Git repository by running: $ git pull https://github.com/jeyhunk

[jira] [Commented] (KAFKA-1614) Partition log directory name and segments information exposed via JMX

[ https://issues.apache.org/jira/browse/KAFKA-1614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15354883#comment-15354883 ] Sergey Vergun commented on KAFKA-1614: -- This is patch doesn't work with Kafka 0.10.0.

Re: [DISCUSS] Client Side Auto Topic Creation

Thanks Jason, I'm 100% with you on this subject, The current auto creation of topic by the broker appear to be the only reason an unknown topic error is retriable which leads to bugs (like https://issues.apache.org/jira/browse/KAFKA-3727 ) where the consumer hangs forever (or until woken up) and