Re: Vote for KIP-393 (Fix time windowed serde to deserialize changelog topic)

2018-12-06 Thread Liquan Pei
> > I wanted to start a vote on approval of KIP-393 > > < > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-393%3A+Time+windowed+serde+to+properly+deserialize+changelog+input+topic > > > > > to > > fix the current time windowed serde for properly deserializing changelog > > input topics. Let me know what you guys think. > > > > Thanks, > > Shawn > > > > > -- > -- Guozhang > -- Liquan Pei Software Engineer, Confluent Inc

Re: [DISCUSSION] KIP-300: Add Windowed KTable API in StreamsBuilder

2018-05-21 Thread Liquan Pei
derstood that the community is busy working on 2.0 release, but this > KIP is really important for our internal use case. So if any of you got > time, please focus on clarifying the use case and reaching the agreement of > API. Really appreciate your time! > > > Best, > > Boyang > > > -- Liquan Pei Software Engineer, Confluent Inc

Re: [VOTE] KIP-429: Kafka Consumer Incremental Rebalance Protocol

2019-05-22 Thread Liquan Pei
col#KIP-429:KafkaConsumerIncrementalRebalanceProtocol-RebalanceCallbackErrorHandling > > And the on-going PRs available for review: > > Part I: https://github.com/apache/kafka/pull/6528 > Part II: https://github.com/apache/kafka/pull/6778 > > > Thanks > -- Guozhang > -- Liquan Pei Software Engineer, Confluent Inc

Re: [VOTE] KIP-51 - List Connectors REST API

2016-03-22 Thread Liquan Pei
gt; feel there's something to still be discussed. > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-51+-+List+Connectors+REST+API > > > > I'll obviously kick things off with a +1. > > > > -Ewen > > > -- Liquan Pei Department of Physics University of Massachusetts Amherst

Re: kafka-connector sink task flush() interval

2016-04-09 Thread Liquan Pei
1551 > Email: victory_...@163.com > ResearchLab Homepage: http://pasa-bigdata.nju.edu.cn -- Liquan Pei Software Engineer, Confluent Inc

Re: Problem with kafka doFlush method

2016-04-13 Thread Liquan Pei
201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecu

Re: [VOTE] KIP-33 - Add a time based log index

2016-04-17 Thread Liquan Pei
gt;> > > > has > >> > > > >> an entry for the largest timestamp. Is that only for restarting > >> > after > >> > > a > >> > > > >> hard failure? > >> > > > >> > >> > > > >> 11. On broker startup, if a log segment misses the time index, > do > >> we > >> > > > >> always > >> > > > >> rebuild it? This can happen when the broker is upgraded. > >> > > > >> > >> > > > >> 12. Related to Guozhang's question #1. It seems it's simpler to > >> add > >> > > time > >> > > > >> index entries independent of the offset index since at index > entry > >> > may > >> > > > not > >> > > > >> be added to the offset and the time index at the same time. > Also, > >> > this > >> > > > >> allows time index to be rebuilt independently if needed. > >> > > > >> > >> > > > >> Thanks, > >> > > > >> > >> > > > >> Jun > >> > > > >> > >> > > > >> > >> > > > >> On Wed, Apr 6, 2016 at 5:44 PM, Becket Qin < > becket@gmail.com> > >> > > > wrote: > >> > > > >> > >> > > > >> > Hi all, > >> > > > >> > > >> > > > >> > I updated KIP-33 based on the initial implementation. Per > >> > discussion > >> > > > on > >> > > > >> > yesterday's KIP hangout, I would like to initiate the new > vote > >> > > thread > >> > > > >> for > >> > > > >> > KIP-33. > >> > > > >> > > >> > > > >> > The KIP wiki: > >> > > > >> > > >> > > > >> > > >> > > > >> > >> > > > > >> > > > >> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-33+-+Add+a+time+based+log+index > >> > > > >> > > >> > > > >> > Here is a brief summary of the KIP: > >> > > > >> > 1. We propose to add a time index for each log segment. > >> > > > >> > 2. The time indices are going to be used of log retention, > log > >> > > rolling > >> > > > >> and > >> > > > >> > message search by timestamp. > >> > > > >> > > >> > > > >> > There was an old voting thread which has some discussions on > >> this > >> > > KIP. > >> > > > >> The > >> > > > >> > mail thread link is following: > >> > > > >> > > >> > > > >> > > >> > > > >> > >> > > > > >> > > > >> > > >> > http://mail-archives.apache.org/mod_mbox/kafka-dev/201602.mbox/%3ccabtagwgoebukyapfpchmycjk2tepq3ngtuwnhtr2tjvsnc8...@mail.gmail.com%3E > >> > > > >> > > >> > > > >> > I have the following WIP patch for reference. It needs a few > >> more > >> > > unit > >> > > > >> > tests and documentation. Other than that it should run fine. > >> > > > >> > > >> > > > >> > > >> > > > >> > >> > > > > >> > > > >> > > >> > https://github.com/becketqin/kafka/commit/712357a3fbf1423e05f9eed7d2fed5b6fe6c37b7 > >> > > > >> > > >> > > > >> > Thanks, > >> > > > >> > > >> > > > >> > Jiangjie (Becket) Qin > >> > > > >> > > >> > > > >> > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > > > > > > > -- > > -- Guozhang > -- Liquan Pei Software Engineer, Confluent Inc

Re: KTable.count(...)

2016-04-17 Thread Liquan Pei
ly get a record where the key is A and the value is 5, i.e, the > number of times the key A was seen in the input stream. However, this is > not the case. What i receive on the count topic is: > A:1 > A:2 > A:1 > A:2 > A:1 > A:2 > A:1 > A:2 > A:1 > > Is

[DISCUSS] KIP-56 Allow cross origin HTTP requests on all HTTP methods

2016-04-18 Thread Liquan Pei
Hi I would like to start a quick discussion on KIP-56 https://cwiki.apache.org/confluence/display/KAFKA/KIP-56%3A+Allow+cross+origin+HTTP+requests+on+all+HTTP+methods This proposal is to allow cross origin HTTP requests on all HTTP methods. Thanks, Liquan -- Liquan Pei Software Engineer

[VOTE] KIP-56 Allow cross origin HTTP requests on all HTTP methods

2016-04-21 Thread Liquan Pei
Hi I would like to start vote on KIP-56. https://cwiki.apache.org/confluence/display/KAFKA/KIP-56%3A+Allow+cross+origin+HTTP+requests+on+all+HTTP+methods Thanks, -- Liquan Pei Software Engineer, Confluent Inc

Re: Kafka missing from ASF Jira?

2016-04-25 Thread Liquan Pei
; Jira, Kafka is not visible in the list of projects in the first field > of the ticket. I see the hundred+ other Apache projects, but no Kafka > :( > -- Liquan Pei Software Engineer, Confluent Inc

Re: [VOTE] 0.10.0.0 RC4

2016-05-10 Thread Liquan Pei
tml > > * Protocol: > http://kafka.apache.org/0100/protocol.html > > /** > > Thanks, > > Gwen > -- Liquan Pei Software Engineer, Confluent Inc

Re: [ANNOUNCE] New Kafka Committer Ewen Cheslack-Postava

2015-12-08 Thread Liquan Pei
d > numerous patches to Kafka. His most significant contribution is Kafka > Connect which was released few days ago as part of 0.9. > > Please join me on welcoming and congratulating Ewen. > > Ewen, we look forward to your continued contributions to the Kafka > community! > &

Re: [VOTE] KIP-41: Consumer Max Records

2016-01-14 Thread Liquan Pei
gineer > Tink AB > > Email: jens.ran...@tink.se > Phone: +46 708 84 18 32 > Web: www.tink.se > > Facebook <https://www.facebook.com/#!/tink.se> Linkedin > < > http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_photo&trkInfo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary > > > Twitter <https://twitter.com/tink> > -- Liquan Pei Department of Physics University of Massachusetts Amherst

Re: [DISCUSS] scalability limits in the coordinator

2016-05-23 Thread Liquan Pei
>>>>> 60 consumers 1598621 bytes > > >>>>>>> 70 consumers 1837359 bytes > > >>>>>>> 80 consumers 2066934 bytes > > >>>>>>> 90 consumers 2310970 bytes > > >>>>>>> 100 consumers 2542735 bytes > > >>>>>>> > > >>>>>>> Note that the growth itself is pretty gradual. Plotting the > > >> points > > >>>>> makes > > >>>>>> it > > >>>>>>> look roughly linear w.r.t the number of consumers: > > >>>>>>> > > >>>>>>> > > >>>>>> > > >>>>> > > >>>> > > >>> > > >> > > > https://www.wolframalpha.com/input/?i=(1,+54739),+(5,+261524),+(10,+459804),+(20,+702499),+(30,+930525),+(40,+1115657),+(50,+1363112),+(60,+1598621),+(70,+1837359),+(80,+2066934),+(90,+2310970),+(100,+2542735) > > >>>>>>> > > >>>>>>> Also note that these numbers aren't averages or medians or > > >> anything > > >>>>> like > > >>>>>>> that. It's just the byte size from a given run. I did run them a > > >>> few > > >>>>>> times > > >>>>>>> and saw similar results. > > >>>>>>> > > >>>>>>> Impact: > > >>>>>>> Even after adding gzip to the __consumer_offsets topic with my > > >>>> pending > > >>>>>>> KAFKA-3718 patch, the AwaitingSync phase of the group fails with > > >>>>>>> RecordTooLargeException. This means the combined size of each > > >>>> member's > > >>>>>>> subscriptions and assignments exceeded the > > >>>> KafkaConfig.messageMaxBytes > > >>>>> of > > >>>>>>> 112 bytes. The group ends up dying. > > >>>>>>> > > >>>>>>> Options: > > >>>>>>> 1. Config change: reduce the number of consumers in the group. > > >> This > > >>>>> isn't > > >>>>>>> always a realistic answer in more strenuous use cases like > > >>>> MirrorMaker > > >>>>>>> clusters or for auditing. > > >>>>>>> 2. Config change: split the group into smaller groups which > > >>> together > > >>>>> will > > >>>>>>> get full coverage of the topics. This gives each group member a > > >>>> smaller > > >>>>>>> subscription.(ex: g1 has topics starting with a-m while g2 has > > >>> topics > > >>>>>>> starting ith n-z). This would be operationally painful to manage. > > >>>>>>> 3. Config change: split the topics among members of the group. > > >>> Again > > >>>>> this > > >>>>>>> gives each group member a smaller subscription. This would also > > >> be > > >>>>>>> operationally painful to manage. > > >>>>>>> 4. Config change: bump up KafkaConfig.messageMaxBytes (a > > >>> topic-level > > >>>>>>> config) and KafkaConfig.replicaFetchMaxBytes (a broker-level > > >>> config). > > >>>>>>> Applying messageMaxBytes to just the __consumer_offsets topic > > >> seems > > >>>>>>> relatively harmless, but bumping up the broker-level > > >>>>> replicaFetchMaxBytes > > >>>>>>> would probably need more attention. > > >>>>>>> 5. Config change: try different compression codecs. Based on 2 > > >>>> minutes > > >>>>> of > > >>>>>>> googling, it seems like lz4 and snappy are faster than gzip but > > >>> have > > >>>>>> worse > > >>>>>>> compression, so this probably won't help. > > >>>>>>> 6. Implementation change: support sending the regex over the wire > > >>>>> instead > > >>>>>>> of the fully expanded topic subscriptions. I think people said in > > >>> the > > >>>>>> past > > >>>>>>> that different languages have subtle differences in regex, so > > >> this > > >>>>>> doesn't > > >>>>>>> play nicely with cross-language groups. > > >>>>>>> 7. Implementation change: maybe we can reverse the mapping? > > >> Instead > > >>>> of > > >>>>>>> mapping from member to subscriptions, we can map a subscription > > >> to > > >>> a > > >>>>> list > > >>>>>>> of members. > > >>>>>>> 8. Implementation change: maybe we can try to break apart the > > >>>>>> subscription > > >>>>>>> and assignments from the same SyncGroupRequest into multiple > > >>> records? > > >>>>>> They > > >>>>>>> can still go to the same message set and get appended together. > > >>> This > > >>>>> way > > >>>>>>> the limit become the segment size, which shouldn't be a problem. > > >>> This > > >>>>> can > > >>>>>>> be tricky to get right because we're currently keying these > > >>> messages > > >>>> on > > >>>>>> the > > >>>>>>> group, so I think records from the same rebalance might > > >>> accidentally > > >>>>>>> compact one another, but my understanding of compaction isn't > > >> that > > >>>>> great. > > >>>>>>> > > >>>>>>> Todo: > > >>>>>>> It would be interesting to rerun the tests with no compression > > >> just > > >>>> to > > >>>>>> see > > >>>>>>> how much gzip is helping but it's getting late. Maybe tomorrow? > > >>>>>>> > > >>>>>>> - Onur > > >>>>>>> > > >>>>>> > > >>>>> > > >>>> > > >>>> > > >>>> > > >>>> -- > > >>>> -- Guozhang > > >>>> > > >>> > > >> > > >> > > >> > > >> -- > > >> -- Guozhang > > >> > > > > > -- Liquan Pei Software Engineer, Confluent Inc

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

2016-05-24 Thread Liquan Pei
It seems that the links to images in the KIP are broken. Liquan On Tue, May 24, 2016 at 9:33 AM, parth brahmbhatt < brahmbhatt.pa...@gmail.com> wrote: > 110. What does getDelegationTokenAs mean? > In the current proposal we only allow a user to get delegation token for > the identity that it aut

Re: Kafka Connect connector monitoring tasks via offset storage reader

2016-05-24 Thread Liquan Pei
> could use the OffsetStorageReader to read previously-recorded offsets to > more intelligently configure its tasks. This seems very straightforward, > backward compatible, and non-intrusive. > > Is there any interest in this? If so, I can create an issue and work on a > pull req

Re: kafka consumer group is rebalancing

2016-05-31 Thread Liquan Pei
s > as 3 and heartbeat.interval.ms as 1 to consumer and polling > happens > for sure with in 3. Can anyone help me out. Please let me know if any > information is needed. > I am using 3 node kafka cluster. > Thanks, > Sunny > -- Liquan Pei Software Engineer, Confluent Inc

Re: [ANNOUNCE] New committer: Boyang Chen

2020-06-24 Thread Liquan Pei
ode contributions, Boyang > > has > > > also helped reviewing even more PRs and KIPs than his own. > > > > > > Thanks for all the contributions Boyang! And look forward to more > > > collaborations with you on Apache Kafka. > > > > > > > > > -- Guozhang, on behalf of the Apache Kafka PMC > > > > > > -- Liquan Pei Software Engineer, Confluent Inc

[jira] [Created] (KAFKA-7023) Kafka Streams RocksDB bulk loading config may not be honored with customized RocksDBConfigSetter

2018-06-07 Thread Liquan Pei (JIRA)
Liquan Pei created KAFKA-7023: - Summary: Kafka Streams RocksDB bulk loading config may not be honored with customized RocksDBConfigSetter Key: KAFKA-7023 URL: https://issues.apache.org/jira/browse/KAFKA-7023

[jira] [Created] (KAFKA-7103) Use bulkloading for RocksDBSegmentedBytesStore during init

2018-06-26 Thread Liquan Pei (JIRA)
Liquan Pei created KAFKA-7103: - Summary: Use bulkloading for RocksDBSegmentedBytesStore during init Key: KAFKA-7103 URL: https://issues.apache.org/jira/browse/KAFKA-7103 Project: Kafka Issue

[jira] [Created] (KAFKA-7105) Refactor RocksDBSegmentsBatchingRestoreCallback and RocksDBBatchingRestoreCallback into a single class

2018-06-26 Thread Liquan Pei (JIRA)
Liquan Pei created KAFKA-7105: - Summary: Refactor RocksDBSegmentsBatchingRestoreCallback and RocksDBBatchingRestoreCallback into a single class Key: KAFKA-7105 URL: https://issues.apache.org/jira/browse/KAFKA-7105

[jira] [Updated] (KAFKA-3316) Add Connect REST API to list available connector classes

2016-03-19 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated KAFKA-3316: -- Affects Version/s: 0.10.0.0 Status: Patch Available (was: In Progress) > Add Conn

[jira] [Work started] (KAFKA-3316) Add Connect REST API to list available connector classes

2016-03-19 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-3316 started by Liquan Pei. - > Add Connect REST API to list available connector clas

[jira] [Work started] (KAFKA-3462) Allow SinkTasks to disable consumer offset commit

2016-03-24 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-3462 started by Liquan Pei. - > Allow SinkTasks to disable consumer offset com

[jira] [Created] (KAFKA-3462) Allow SinkTasks to disable consumer offset commit

2016-03-24 Thread Liquan Pei (JIRA)
Liquan Pei created KAFKA-3462: - Summary: Allow SinkTasks to disable consumer offset commit Key: KAFKA-3462 URL: https://issues.apache.org/jira/browse/KAFKA-3462 Project: Kafka Issue Type: Bug

[jira] [Updated] (KAFKA-3462) Allow SinkTasks to disable consumer offset commit

2016-03-24 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated KAFKA-3462: -- Description: SinkTasks should be able to disable consumer offset commit if they manage offsets in the

[jira] [Updated] (KAFKA-3462) Allow SinkTasks to disable consumer offset commit

2016-03-24 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated KAFKA-3462: -- Status: Patch Available (was: In Progress) > Allow SinkTasks to disable consumer offset com

[jira] [Updated] (KAFKA-3500) KafkaOffsetBackingStore set method needs to handle null

2016-04-04 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated KAFKA-3500: -- Environment: (was: In some cases, the key or value for the offset map can be null. However, it

[jira] [Created] (KAFKA-3500) KafkaOffsetBackingStore set method needs to handle null

2016-04-04 Thread Liquan Pei (JIRA)
Liquan Pei created KAFKA-3500: - Summary: KafkaOffsetBackingStore set method needs to handle null Key: KAFKA-3500 URL: https://issues.apache.org/jira/browse/KAFKA-3500 Project: Kafka Issue Type

[jira] [Created] (KAFKA-3520) System tests of config validate and list connectors REST APIs

2016-04-06 Thread Liquan Pei (JIRA)
Liquan Pei created KAFKA-3520: - Summary: System tests of config validate and list connectors REST APIs Key: KAFKA-3520 URL: https://issues.apache.org/jira/browse/KAFKA-3520 Project: Kafka Issue

[jira] [Work started] (KAFKA-3520) System tests of config validate and list connectors REST APIs

2016-04-06 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-3520 started by Liquan Pei. - > System tests of config validate and list connectors REST A

[jira] [Updated] (KAFKA-3520) System tests of config validate and list connectors REST APIs

2016-04-06 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated KAFKA-3520: -- Status: Patch Available (was: In Progress) > System tests of config validate and list connectors R

[jira] [Updated] (KAFKA-3526) REST APIs return object representation instead of string for config values, default values and recommended values

2016-04-07 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated KAFKA-3526: -- Description: In the response of PUT /connector-plugins/{name}/config/validate, The value.value

[jira] [Created] (KAFKA-3526) REST APIs return object representation instead of string for config values, default values and recommended values

2016-04-07 Thread Liquan Pei (JIRA)
Liquan Pei created KAFKA-3526: - Summary: REST APIs return object representation instead of string for config values, default values and recommended values Key: KAFKA-3526 URL: https://issues.apache.org/jira/browse

[jira] [Updated] (KAFKA-3526) REST APIs return object representation instead of string for config values, default values and recommended values

2016-04-07 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated KAFKA-3526: -- Description: In the response of {code} PUT /connector-plugins/{name}/config/validate {code} The

[jira] [Updated] (KAFKA-3526) REST APIs return object representation instead of string for config values, default values and recommended values

2016-04-07 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated KAFKA-3526: -- Description: In the response of {code} PUT /connector-plugins/{name}/config/validate {code} The

[jira] [Updated] (KAFKA-3526) REST APIs return object representation instead of string for config values, default values and recommended values

2016-04-07 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated KAFKA-3526: -- Status: Patch Available (was: In Progress) > REST APIs return object representation instead of str

[jira] [Work started] (KAFKA-3526) REST APIs return object representation instead of string for config values, default values and recommended values

2016-04-07 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-3526 started by Liquan Pei. - > REST APIs return object representation instead of string for config val

[jira] [Assigned] (KAFKA-3530) Making the broker-list option consistent across all tools

2016-04-08 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei reassigned KAFKA-3530: - Assignee: Liquan Pei > Making the broker-list option consistent across all to

[jira] [Commented] (KAFKA-3527) Consumer commitAsync should not expose internal exceptions

2016-04-09 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15233722#comment-15233722 ] Liquan Pei commented on KAFKA-3527: --- [~jasong35] Do you mind if I take

[jira] [Assigned] (KAFKA-3552) New Consumer: java.lang.OutOfMemoryError: Direct buffer memory

2016-04-12 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei reassigned KAFKA-3552: - Assignee: Liquan Pei (was: Neha Narkhede) > New Consumer: java.lang.OutOfMemoryError: Dir

[jira] [Commented] (KAFKA-3552) New Consumer: java.lang.OutOfMemoryError: Direct buffer memory

2016-04-12 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15238372#comment-15238372 ] Liquan Pei commented on KAFKA-3552: --- Hi Kanak, Can you share with us the cons

[jira] [Assigned] (KAFKA-3421) Update docs with new connector features

2016-04-14 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei reassigned KAFKA-3421: - Assignee: Liquan Pei (was: Ewen Cheslack-Postava) > Update docs with new connector featu

[jira] [Work started] (KAFKA-3421) Update docs with new connector features

2016-04-15 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-3421 started by Liquan Pei. - > Update docs with new connector featu

[jira] [Assigned] (KAFKA-3571) Traits for utilities like ConsumerGroupCommand

2016-04-16 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei reassigned KAFKA-3571: - Assignee: Liquan Pei > Traits for utilities like ConsumerGroupComm

[jira] [Updated] (KAFKA-3421) Update docs with new connector features

2016-04-17 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated KAFKA-3421: -- Status: Patch Available (was: In Progress) > Update docs with new connector featu

[jira] [Commented] (KAFKA-3573) JSON (de)serialization for kafka-client

2016-04-17 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15244768#comment-15244768 ] Liquan Pei commented on KAFKA-3573: --- Thanks for working on this. Currently, there

[jira] [Created] (KAFKA-3578) Allow cross origin HTTP requests on all HTTP methods

2016-04-18 Thread Liquan Pei (JIRA)
Liquan Pei created KAFKA-3578: - Summary: Allow cross origin HTTP requests on all HTTP methods Key: KAFKA-3578 URL: https://issues.apache.org/jira/browse/KAFKA-3578 Project: Kafka Issue Type

[jira] [Assigned] (KAFKA-3578) Allow cross origin HTTP requests on all HTTP methods

2016-04-18 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei reassigned KAFKA-3578: - Assignee: Liquan Pei (was: Ewen Cheslack-Postava) > Allow cross origin HTTP requests on

[jira] [Created] (KAFKA-3583) Docs on pause/resume/restart APIs.

2016-04-18 Thread Liquan Pei (JIRA)
Liquan Pei created KAFKA-3583: - Summary: Docs on pause/resume/restart APIs. Key: KAFKA-3583 URL: https://issues.apache.org/jira/browse/KAFKA-3583 Project: Kafka Issue Type: Improvement

[jira] [Assigned] (KAFKA-3582) remove references to Copcyat from connect property files

2016-04-18 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei reassigned KAFKA-3582: - Assignee: Liquan Pei > remove references to Copcyat from connect property fi

[jira] [Work started] (KAFKA-3582) remove references to Copcyat from connect property files

2016-04-18 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-3582 started by Liquan Pei. - > remove references to Copcyat from connect property fi

[jira] [Updated] (KAFKA-3582) remove references to Copcyat from connect property files

2016-04-18 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated KAFKA-3582: -- Status: Patch Available (was: In Progress) > remove references to Copcyat from connect property fi

[jira] [Assigned] (KAFKA-3459) Returning zero task configurations from a connector does not properly clean up existing tasks

2016-04-20 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei reassigned KAFKA-3459: - Assignee: Liquan Pei (was: Ewen Cheslack-Postava) > Returning zero task configurations fro

[jira] [Assigned] (KAFKA-3035) Transient: kafka.api.PlaintextConsumerTest > testAutoOffsetReset FAILED

2016-04-20 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei reassigned KAFKA-3035: - Assignee: Liquan Pei > Transient: kafka.api.PlaintextConsumerTest > testAutoOffsetReset

[jira] [Work started] (KAFKA-3459) Returning zero task configurations from a connector does not properly clean up existing tasks

2016-04-20 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-3459 started by Liquan Pei. - > Returning zero task configurations from a connector does not properly clean &

[jira] [Updated] (KAFKA-3459) Returning zero task configurations from a connector does not properly clean up existing tasks

2016-04-20 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated KAFKA-3459: -- Status: Patch Available (was: In Progress) > Returning zero task configurations from a connector d

[jira] [Created] (KAFKA-3606) Traverse CLASSPATH during herder start to list connectors

2016-04-21 Thread Liquan Pei (JIRA)
Liquan Pei created KAFKA-3606: - Summary: Traverse CLASSPATH during herder start to list connectors Key: KAFKA-3606 URL: https://issues.apache.org/jira/browse/KAFKA-3606 Project: Kafka Issue Type

[jira] [Resolved] (KAFKA-3605) Connector REST endpoint allows incorrectly overriding the connector name

2016-04-21 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei resolved KAFKA-3605. --- Resolution: Fixed > Connector REST endpoint allows incorrectly overriding the connector n

[jira] [Reopened] (KAFKA-3605) Connector REST endpoint allows incorrectly overriding the connector name

2016-04-21 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei reopened KAFKA-3605: --- > Connector REST endpoint allows incorrectly overriding the connector n

[jira] [Assigned] (KAFKA-3457) KafkaConsumer.committed(...) hangs forever if port number is wrong

2016-04-21 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei reassigned KAFKA-3457: - Assignee: Liquan Pei > KafkaConsumer.committed(...) hangs forever if port number is wr

[jira] [Assigned] (KAFKA-3527) Consumer commitAsync should not expose internal exceptions

2016-04-21 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei reassigned KAFKA-3527: - Assignee: Liquan Pei (was: Jason Gustafson) > Consumer commitAsync should not expose inter

[jira] [Resolved] (KAFKA-2479) Add CopycatExceptions to indicate transient and permanent errors in a connector/task

2016-04-21 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-2479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei resolved KAFKA-2479. --- Resolution: Fixed > Add CopycatExceptions to indicate transient and permanent errors i

[jira] [Assigned] (KAFKA-3556) Improve group coordinator metrics

2016-04-21 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei reassigned KAFKA-3556: - Assignee: Liquan Pei > Improve group coordinator metr

[jira] [Created] (KAFKA-3611) Remove WARNs when using reflections

2016-04-22 Thread Liquan Pei (JIRA)
Liquan Pei created KAFKA-3611: - Summary: Remove WARNs when using reflections Key: KAFKA-3611 URL: https://issues.apache.org/jira/browse/KAFKA-3611 Project: Kafka Issue Type: Improvement

[jira] [Work stopped] (KAFKA-3611) Remove WARNs when using reflections

2016-04-22 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-3611 stopped by Liquan Pei. - > Remove WARNs when using reflecti

[jira] [Updated] (KAFKA-3611) Remove WARNs when using reflections

2016-04-22 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated KAFKA-3611: -- Status: Patch Available (was: In Progress) > Remove WARNs when using reflecti

[jira] [Work started] (KAFKA-3611) Remove WARNs when using reflections

2016-04-22 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-3611 started by Liquan Pei. - > Remove WARNs when using reflecti

[jira] [Work started] (KAFKA-3611) Remove WARNs when using reflections

2016-04-22 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-3611 started by Liquan Pei. - > Remove WARNs when using reflecti

[jira] [Created] (KAFKA-3615) Exclude test jars in CLASSPATH of kafka-run-class.sh

2016-04-23 Thread Liquan Pei (JIRA)
Liquan Pei created KAFKA-3615: - Summary: Exclude test jars in CLASSPATH of kafka-run-class.sh Key: KAFKA-3615 URL: https://issues.apache.org/jira/browse/KAFKA-3615 Project: Kafka Issue Type

[jira] [Commented] (KAFKA-3615) Exclude test jars in CLASSPATH of kafka-run-class.sh

2016-04-24 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255889#comment-15255889 ] Liquan Pei commented on KAFKA-3615: --- [~enothereska] Kafka Connect provides a REST

[jira] [Assigned] (KAFKA-3626) Transient failure in testGetAllTopicMetadata

2016-04-26 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei reassigned KAFKA-3626: - Assignee: Liquan Pei > Transient failure in testGetAllTopicMetad

[jira] [Assigned] (KAFKA-3627) New consumer doesn't run delayed tasks while under load

2016-04-26 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei reassigned KAFKA-3627: - Assignee: Liquan Pei (was: Neha Narkhede) > New consumer doesn't run delayed tasks whi

[jira] [Assigned] (KAFKA-3637) Add method that checks if streams are initialised

2016-04-29 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei reassigned KAFKA-3637: - Assignee: Liquan Pei > Add method that checks if streams are initiali

[jira] [Assigned] (KAFKA-3213) [CONNECT] It looks like we are not backing off properly when reconfiguring tasks

2016-04-29 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei reassigned KAFKA-3213: - Assignee: Liquan Pei (was: Ewen Cheslack-Postava) > [CONNECT] It looks like we are not back

[jira] [Closed] (KAFKA-3582) remove references to Copcyat from connect property files

2016-04-29 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei closed KAFKA-3582. - > remove references to Copcyat from connect property fi

[jira] [Closed] (KAFKA-3615) Exclude test jars in CLASSPATH of kafka-run-class.sh

2016-04-29 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei closed KAFKA-3615. - > Exclude test jars in CLASSPATH of kafka-run-class

[jira] [Closed] (KAFKA-3606) Traverse CLASSPATH during herder start to list connectors

2016-04-29 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei closed KAFKA-3606. - > Traverse CLASSPATH during herder start to list connect

[jira] [Closed] (KAFKA-3578) Allow cross origin HTTP requests on all HTTP methods

2016-04-29 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei closed KAFKA-3578. - > Allow cross origin HTTP requests on all HTTP meth

[jira] [Closed] (KAFKA-3611) Remove WARNs when using reflections

2016-04-29 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei closed KAFKA-3611. - > Remove WARNs when using reflections > > >

[jira] [Created] (KAFKA-3654) ConnectorConfig defs short circuit implementation config checks

2016-05-03 Thread Liquan Pei (JIRA)
Liquan Pei created KAFKA-3654: - Summary: ConnectorConfig defs short circuit implementation config checks Key: KAFKA-3654 URL: https://issues.apache.org/jira/browse/KAFKA-3654 Project: Kafka

[jira] [Updated] (KAFKA-3654) ConnectorConfig short circuit implementation config validation

2016-05-03 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated KAFKA-3654: -- Summary: ConnectorConfig short circuit implementation config validation (was: ConnectorConfig defs

[jira] [Updated] (KAFKA-3654) ConnectorConfig short circuit implementation config validation

2016-05-03 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated KAFKA-3654: -- Description: Right now if you call the validate endpoint with a config that has an invalid value for

[jira] [Updated] (KAFKA-3654) Config validation should validate both common and connector specific configurations

2016-05-03 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated KAFKA-3654: -- Summary: Config validation should validate both common and connector specific configurations (was

[jira] [Assigned] (KAFKA-3656) Avoid stressing system more when already under stress

2016-05-04 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei reassigned KAFKA-3656: - Assignee: Liquan Pei > Avoid stressing system more when already under str

[jira] [Assigned] (KAFKA-3649) Add capability to query broker process for configuration properties

2016-05-04 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei reassigned KAFKA-3649: - Assignee: Liquan Pei > Add capability to query broker process for configuration propert

[jira] [Updated] (KAFKA-3656) Avoid stressing system more when already under stress

2016-05-04 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated KAFKA-3656: -- Status: Patch Available (was: In Progress) > Avoid stressing system more when already under str

[jira] [Work started] (KAFKA-3656) Avoid stressing system more when already under stress

2016-05-04 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-3656 started by Liquan Pei. - > Avoid stressing system more when already under str

[jira] [Commented] (KAFKA-3587) LogCleaner fails due to incorrect offset map computation on a replica

2016-05-05 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15273305#comment-15273305 ] Liquan Pei commented on KAFKA-3587: --- Hi [~ecomar] I have a WIP PR, mind to take a

[jira] [Commented] (KAFKA-3587) LogCleaner fails due to incorrect offset map computation on a replica

2016-05-05 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15273584#comment-15273584 ] Liquan Pei commented on KAFKA-3587: --- [~ecomar] [~alekar] Moving the pessimistic c

[jira] [Assigned] (KAFKA-3487) Support per-connector/per-task classloaders in Connect

2016-05-05 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei reassigned KAFKA-3487: - Assignee: Liquan Pei (was: Ewen Cheslack-Postava) > Support per-connector/per-task classload

[jira] [Created] (KAFKA-3671) topics should not be in common ConnectorConfig definitions

2016-05-06 Thread Liquan Pei (JIRA)
Liquan Pei created KAFKA-3671: - Summary: topics should not be in common ConnectorConfig definitions Key: KAFKA-3671 URL: https://issues.apache.org/jira/browse/KAFKA-3671 Project: Kafka Issue

[jira] [Updated] (KAFKA-3671) Topics should not be in common ConnectorConfig definitions

2016-05-06 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated KAFKA-3671: -- Summary: Topics should not be in common ConnectorConfig definitions (was: topics should not be in

[jira] [Work started] (KAFKA-3671) Topics should not be in common ConnectorConfig definitions

2016-05-06 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-3671 started by Liquan Pei. - > Topics should not be in common ConnectorConfig definiti

[jira] [Created] (KAFKA-3673) Connect tests dont handle concurrent config changes

2016-05-06 Thread Liquan Pei (JIRA)
Liquan Pei created KAFKA-3673: - Summary: Connect tests dont handle concurrent config changes Key: KAFKA-3673 URL: https://issues.apache.org/jira/browse/KAFKA-3673 Project: Kafka Issue Type: Bug

[jira] [Updated] (KAFKA-3673) Connect tests dont handle concurrent config changes

2016-05-06 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated KAFKA-3673: -- Description: [INFO - 2016-04-28 18:42:45,346 - runner - log - lineno:221]: SerialTestRunner

[jira] [Created] (KAFKA-3684) Not all SinkConnectorConfig configs are returned

2016-05-09 Thread Liquan Pei (JIRA)
Liquan Pei created KAFKA-3684: - Summary: Not all SinkConnectorConfig configs are returned Key: KAFKA-3684 URL: https://issues.apache.org/jira/browse/KAFKA-3684 Project: Kafka Issue Type: Bug

[jira] [Updated] (KAFKA-3684) Not all SinkConnectorConfig configs are returned

2016-05-09 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated KAFKA-3684: -- Status: Patch Available (was: In Progress) > Not all SinkConnectorConfig configs are retur

[jira] [Work started] (KAFKA-3684) Not all SinkConnectorConfig configs are returned

2016-05-09 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-3684 started by Liquan Pei. - > Not all SinkConnectorConfig configs are retur

  1   2   >