[jira] [Commented] (KAFKA-4397) Refactor Connect backing stores for thread-safety

[ https://issues.apache.org/jira/browse/KAFKA-4397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15656420#comment-15656420 ] ASF GitHub Bot commented on KAFKA-4397: --- GitHub user kkonstantine opened a pull requ

[GitHub] kafka pull request #2123: KAFKA-4397: Refactor Connect backing stores for th...

GitHub user kkonstantine opened a pull request: https://github.com/apache/kafka/pull/2123 KAFKA-4397: Refactor Connect backing stores for thread safety You can merge this pull request into a Git repository by running: $ git pull https://github.com/kkonstantine/kafka KAFKA-439

[jira] [Created] (KAFKA-4397) Refactor Connect backing stores for thread-safety

Konstantine Karantasis created KAFKA-4397: - Summary: Refactor Connect backing stores for thread-safety Key: KAFKA-4397 URL: https://issues.apache.org/jira/browse/KAFKA-4397 Project: Kafka

[GitHub] kafka pull request #2122: failing to close this iterator causes leaks in roc...

GitHub user norwood opened a pull request: https://github.com/apache/kafka/pull/2122 failing to close this iterator causes leaks in rocksdb @guozhangwang @dguy You can merge this pull request into a Git repository by running: $ git pull https://github.com/norwood/kafka close-c

Re: ReplicaFetcherThread stopped after ReplicaFetcherThread received a corrupted message

Jun, The shallow iterator actually does not check the CRC, right? The CRC is only checked when the log.append() is called. That is why the exception was thrown from the processPartitionData(). Thanks, Jiangjie (Becket) Qin On Thu, Nov 10, 2016 at 4:27 PM, Guozhang Wang wrote: > Thanks Jiangj

Build failed in Jenkins: kafka-trunk-jdk7 #1683

See Changes: [wangguoz] MINOR: improve exception message for incompatible Serdes to actual -- [...truncated 14337 lines...] org.apache.kafka.streams.processor.internals.StreamsMetadataStateTest

Build failed in Jenkins: kafka-trunk-jdk8 #1034

See Changes: [wangguoz] MINOR: improve exception message for incompatible Serdes to actual -- [...truncated 3877 lines...] kafka.integration.SaslPlaintextTopicMetadataTest > testIsrAfterBroker

[GitHub] kafka pull request #2118: MINOR: improve exception message for incompatible ...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/2118 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

Re: ReplicaFetcherThread stopped after ReplicaFetcherThread received a corrupted message

Thanks Jiangjie. Makes sense. Currently we cannot simply distinguish between "corrupted messages over the wire, and hence we should retry" and "corrupted messages on leader's log files, and hence we should probably just fail fast and notify admin". So https://issues.apache.org/jira/browse/KAFKA-43

Re: ReplicaFetcherThread stopped after ReplicaFetcherThread received a corrupted message

Hmm, was the corruption detected inside processPartitionData()? Typically, the corruption should be detected at https://github.com/apache/kafka/blob/0.9.0.1/core/src/main/scala/kafka/server/AbstractFetcherThread.scala#L130 when we do a shallow iteration of the message set. If the corruption is dete

Re: [VOTE] KIP-72 - Allow putting a bound on memory consumed by Incoming requests

Hi, Radai, 1. Yes, I am concerned about the trickiness of having to deal with wreak refs. I think it's simpler to just have the simple version instrumented with enough debug/trace logging and do enough stress testing. Since we still have queued.max.requests, one can always fall back to that if a m

[jira] [Commented] (KAFKA-4392) Failed to lock the state directory due to an unexpected exception

[ https://issues.apache.org/jira/browse/KAFKA-4392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15655259#comment-15655259 ] Ara Ebrahimi commented on KAFKA-4392: - 8 streaming threads. > Failed to lock the stat

[jira] [Commented] (KAFKA-4392) Failed to lock the state directory due to an unexpected exception

[ https://issues.apache.org/jira/browse/KAFKA-4392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15655234#comment-15655234 ] Guozhang Wang commented on KAFKA-4392: -- [~arae] Here you go: https://github.com/apach

[GitHub] kafka pull request #2121: KAFKA-4392: race condition in directory cleanup

GitHub user guozhangwang opened a pull request: https://github.com/apache/kafka/pull/2121 KAFKA-4392: race condition in directory cleanup You can merge this pull request into a Git repository by running: $ git pull https://github.com/guozhangwang/kafka K4392-race-dir-cleanup

[jira] [Commented] (KAFKA-4392) Failed to lock the state directory due to an unexpected exception

[ https://issues.apache.org/jira/browse/KAFKA-4392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15655231#comment-15655231 ] ASF GitHub Bot commented on KAFKA-4392: --- GitHub user guozhangwang opened a pull requ

[jira] [Comment Edited] (KAFKA-4392) Failed to lock the state directory due to an unexpected exception

[ https://issues.apache.org/jira/browse/KAFKA-4392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15655211#comment-15655211 ] Guozhang Wang edited comment on KAFKA-4392 at 11/10/16 9:34 PM:

[jira] [Commented] (KAFKA-4392) Failed to lock the state directory due to an unexpected exception

[ https://issues.apache.org/jira/browse/KAFKA-4392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15655217#comment-15655217 ] Ara Ebrahimi commented on KAFKA-4392: - Centos 6.8. I'll be testing on Centos 7 next we

[jira] [Commented] (KAFKA-4392) Failed to lock the state directory due to an unexpected exception

[ https://issues.apache.org/jira/browse/KAFKA-4392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15655211#comment-15655211 ] Guozhang Wang commented on KAFKA-4392: -- BTW which unix system are you running on? >

[jira] [Commented] (KAFKA-4392) Failed to lock the state directory due to an unexpected exception

[ https://issues.apache.org/jira/browse/KAFKA-4392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15655209#comment-15655209 ] Guozhang Wang commented on KAFKA-4392: -- Bottom line: "FileChannel.open(WRITE,APPEND)

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

I am not sure if we are bumping magicByte value as the KIP does not mention it (If I didn't miss anything). @Nacho : I did not understand what you meant by : Conversion from one to the other may be possible but I would rather not have to do it. You will have to do the conversion for the scenario

[jira] [Commented] (KAFKA-4392) Failed to lock the state directory due to an unexpected exception

[ https://issues.apache.org/jira/browse/KAFKA-4392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15655029#comment-15655029 ] Ara Ebrahimi commented on KAFKA-4392: - Hi, Note that this is not happening on a box w

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

Nacho, In Kafka protocol, a negative length is null, a zero length means empty byte array. I am still confused about the two stages. It seems that the broker only needs to understand three versions of messages. 1. MagicValue=0 - no timestamp, absolute offset, no tombstone flag 2. MagicValue=1 - w

[jira] [Commented] (KAFKA-4392) Failed to lock the state directory due to an unexpected exception

[ https://issues.apache.org/jira/browse/KAFKA-4392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15654932#comment-15654932 ] Guozhang Wang commented on KAFKA-4392: -- Thanks [~arae], from the stack trace I think

[jira] [Commented] (KAFKA-4394) Processor API does not receive timestamp and/or ConsumerRecord

[ https://issues.apache.org/jira/browse/KAFKA-4394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15654916#comment-15654916 ] Mark Shelton commented on KAFKA-4394: - Great thank you for the pointer. This is suffic

[jira] [Resolved] (KAFKA-4394) Processor API does not receive timestamp and/or ConsumerRecord

[ https://issues.apache.org/jira/browse/KAFKA-4394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Shelton resolved KAFKA-4394. - Resolution: Not A Problem > Processor API does not receive timestamp and/or ConsumerRecord >

Re: ReplicaFetcherThread stopped after ReplicaFetcherThread received a corrupted message

Hi Guozhang, The reason the fetcher thread died was because it expects the FetchRequest to be always fetching from the current log end offset. We could loose this restriction. I agree with the current design that he replica fetcher thread should not die so the other partitions will not lag behind

Re: any plans to switch to java 8?

http://markmail.org/message/gnrn5ccql7a2pmc5 We can bump that up to revisit the discussion. That thread didn't have any closure, but has a lot of background information. On Thu, Nov 10, 2016 at 10:37 AM, Sean McCauliff wrote: > Wait for JDK 9 which is supposed to be 4-5 months from now? > > Sean

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

A quick differentiation I would like to make is null is not the same as size 0. Many times these are made equal, but they're not. When serializing data, we have to make a choice in null values and many times these are translated to zero length blobs. This is not really the same thing. >From this

Re: any plans to switch to java 8?

Wait for JDK 9 which is supposed to be 4-5 months from now? Sean On Thu, Nov 10, 2016 at 10:23 AM, radai wrote: > with java 7 being EOL'ed for more than a year and a half now (apr 2015, see > http://www.oracle.com/technetwork/java/eol-135779.html) i was wondering if > there's an official plan/ti

any plans to switch to java 8?

with java 7 being EOL'ed for more than a year and a half now (apr 2015, see http://www.oracle.com/technetwork/java/eol-135779.html) i was wondering if there's an official plan/timetable for transitioning the kafka codebase over to java 8?

Jenkins build is back to normal : kafka-trunk-jdk8 #1033

See

[jira] [Commented] (KAFKA-4384) ReplicaFetcherThread stopped after ReplicaFetcherThread received a corrupted message

[ https://issues.apache.org/jira/browse/KAFKA-4384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15654659#comment-15654659 ] Jiangjie Qin commented on KAFKA-4384: - [~junhe] Thanks for the patch. Kafka actually u

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

+1 Michael. Thanks, Mayuresh On Thu, Nov 10, 2016 at 4:03 AM, Michael Pearce wrote: > I have finally got around to updating the KIP from some of the bits so far > discussed in this discussion thread. > Please re-read. > > I have added the extra uses cases over and above those already detaile

Jenkins build is back to normal : kafka-trunk-jdk7 #1682

See

[jira] [Commented] (KAFKA-1894) Avoid long or infinite blocking in the consumer

[ https://issues.apache.org/jira/browse/KAFKA-1894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15654191#comment-15654191 ] Stig Rohde Døssing commented on KAFKA-1894: --- Suggestion for a fix for this issue

[jira] [Comment Edited] (KAFKA-1894) Avoid long or infinite blocking in the consumer

[ https://issues.apache.org/jira/browse/KAFKA-1894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15654191#comment-15654191 ] Stig Rohde Døssing edited comment on KAFKA-1894 at 11/10/16 2:31 PM: ---

[jira] [Commented] (KAFKA-4322) StateRestoreCallback begin and end indication

[ https://issues.apache.org/jira/browse/KAFKA-4322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15654104#comment-15654104 ] ASF GitHub Bot commented on KAFKA-4322: --- GitHub user markcshelton reopened a pull re

[jira] [Commented] (KAFKA-4322) StateRestoreCallback begin and end indication

[ https://issues.apache.org/jira/browse/KAFKA-4322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15654103#comment-15654103 ] ASF GitHub Bot commented on KAFKA-4322: --- Github user markcshelton closed the pull re

[GitHub] kafka pull request #2105: KAFKA-4322 StateRestoreCallback begin and end indi...

Github user markcshelton closed the pull request at: https://github.com/apache/kafka/pull/2105 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature i

[GitHub] kafka pull request #2105: KAFKA-4322 StateRestoreCallback begin and end indi...

GitHub user markcshelton reopened a pull request: https://github.com/apache/kafka/pull/2105 KAFKA-4322 StateRestoreCallback begin and end indication This adds a begin and end callback to StateRestoreCallback. The contribution is my original work and I license the work to Apa

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

I have finally got around to updating the KIP from some of the bits so far discussed in this discussion thread. Please re-read. I have added the extra uses cases over and above those already detailed during kip-82’s discussion, mentioned by James (thank you) I have add’d some clarity over what o

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

Agree on this point by Raidai, im happy having a two stage roll out a suggested by yourself Mayuresh, so for a period we have both as obviously a transition period will be required. But I think the end goal is that we should just be reliant on a tombstone marker eventually. (as in remove method