[jira] [Commented] (KAFKA-1374) LogCleaner (compaction) does not support compressed topics
[ https://issues.apache.org/jira/browse/KAFKA-1374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15057052#comment-15057052 ] Manikumar Reddy commented on KAFKA-1374: [~granthenke] Yes, I will update the docs. > LogCleaner (compaction) does not support compressed topics > -- > > Key: KAFKA-1374 > URL: https://issues.apache.org/jira/browse/KAFKA-1374 > Project: Kafka > Issue Type: Bug >Reporter: Joel Koshy >Assignee: Manikumar Reddy > Labels: newbie++ > Fix For: 0.9.0.0 > > Attachments: KAFKA-1374.patch, KAFKA-1374.patch, > KAFKA-1374_2014-08-09_16:18:55.patch, KAFKA-1374_2014-08-12_22:23:06.patch, > KAFKA-1374_2014-09-23_21:47:12.patch, KAFKA-1374_2014-10-03_18:49:16.patch, > KAFKA-1374_2014-10-03_19:17:17.patch, KAFKA-1374_2015-01-18_00:19:21.patch, > KAFKA-1374_2015-05-18_22:55:48.patch, KAFKA-1374_2015-05-19_17:20:44.patch > > > This is a known issue, but opening a ticket to track. > If you try to compact a topic that has compressed messages you will run into > various exceptions - typically because during iteration we advance the > position based on the decompressed size of the message. I have a bunch of > stack traces, but it should be straightforward to reproduce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1374) LogCleaner (compaction) does not support compressed topics
[ https://issues.apache.org/jira/browse/KAFKA-1374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15034056#comment-15034056 ] Grant Henke commented on KAFKA-1374: With this being resolved, should we update this section of the docs? http://kafka.apache.org/documentation.html#design_compactionlimitations > LogCleaner (compaction) does not support compressed topics > -- > > Key: KAFKA-1374 > URL: https://issues.apache.org/jira/browse/KAFKA-1374 > Project: Kafka > Issue Type: Bug >Reporter: Joel Koshy >Assignee: Manikumar Reddy > Labels: newbie++ > Fix For: 0.9.0.0 > > Attachments: KAFKA-1374.patch, KAFKA-1374.patch, > KAFKA-1374_2014-08-09_16:18:55.patch, KAFKA-1374_2014-08-12_22:23:06.patch, > KAFKA-1374_2014-09-23_21:47:12.patch, KAFKA-1374_2014-10-03_18:49:16.patch, > KAFKA-1374_2014-10-03_19:17:17.patch, KAFKA-1374_2015-01-18_00:19:21.patch, > KAFKA-1374_2015-05-18_22:55:48.patch, KAFKA-1374_2015-05-19_17:20:44.patch > > > This is a known issue, but opening a ticket to track. > If you try to compact a topic that has compressed messages you will run into > various exceptions - typically because during iteration we advance the > position based on the decompressed size of the message. I have a bunch of > stack traces, but it should be straightforward to reproduce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1374) LogCleaner (compaction) does not support compressed topics
[ https://issues.apache.org/jira/browse/KAFKA-1374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14552840#comment-14552840 ] Joel Koshy commented on KAFKA-1374: --- I just wanted the commit to record Manikumar as author. Unfortunately our git repo does not seem to let me force push an amended commit so nm. > LogCleaner (compaction) does not support compressed topics > -- > > Key: KAFKA-1374 > URL: https://issues.apache.org/jira/browse/KAFKA-1374 > Project: Kafka > Issue Type: Bug >Reporter: Joel Koshy >Assignee: Manikumar Reddy > Labels: newbie++ > Fix For: 0.8.3 > > Attachments: KAFKA-1374.patch, KAFKA-1374.patch, > KAFKA-1374_2014-08-09_16:18:55.patch, KAFKA-1374_2014-08-12_22:23:06.patch, > KAFKA-1374_2014-09-23_21:47:12.patch, KAFKA-1374_2014-10-03_18:49:16.patch, > KAFKA-1374_2014-10-03_19:17:17.patch, KAFKA-1374_2015-01-18_00:19:21.patch, > KAFKA-1374_2015-05-18_22:55:48.patch, KAFKA-1374_2015-05-19_17:20:44.patch > > > This is a known issue, but opening a ticket to track. > If you try to compact a topic that has compressed messages you will run into > various exceptions - typically because during iteration we advance the > position based on the decompressed size of the message. I have a bunch of > stack traces, but it should be straightforward to reproduce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1374) LogCleaner (compaction) does not support compressed topics
[ https://issues.apache.org/jira/browse/KAFKA-1374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14552712#comment-14552712 ] Guozhang Wang commented on KAFKA-1374: -- Committed to trunk. I think we need some integration tests before feeling confident about this issue being resolved, so leave it open just for now. > LogCleaner (compaction) does not support compressed topics > -- > > Key: KAFKA-1374 > URL: https://issues.apache.org/jira/browse/KAFKA-1374 > Project: Kafka > Issue Type: Bug >Reporter: Joel Koshy >Assignee: Manikumar Reddy > Labels: newbie++ > Fix For: 0.8.3 > > Attachments: KAFKA-1374.patch, KAFKA-1374.patch, > KAFKA-1374_2014-08-09_16:18:55.patch, KAFKA-1374_2014-08-12_22:23:06.patch, > KAFKA-1374_2014-09-23_21:47:12.patch, KAFKA-1374_2014-10-03_18:49:16.patch, > KAFKA-1374_2014-10-03_19:17:17.patch, KAFKA-1374_2015-01-18_00:19:21.patch, > KAFKA-1374_2015-05-18_22:55:48.patch, KAFKA-1374_2015-05-19_17:20:44.patch > > > This is a known issue, but opening a ticket to track. > If you try to compact a topic that has compressed messages you will run into > various exceptions - typically because during iteration we advance the > position based on the decompressed size of the message. I have a bunch of > stack traces, but it should be straightforward to reproduce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1374) LogCleaner (compaction) does not support compressed topics
[ https://issues.apache.org/jira/browse/KAFKA-1374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14551535#comment-14551535 ] Joel Koshy commented on KAFKA-1374: --- Updated reviewboard https://reviews.apache.org/r/34397/diff/ against branch origin/trunk > LogCleaner (compaction) does not support compressed topics > -- > > Key: KAFKA-1374 > URL: https://issues.apache.org/jira/browse/KAFKA-1374 > Project: Kafka > Issue Type: Bug >Reporter: Joel Koshy >Assignee: Manikumar Reddy > Labels: newbie++ > Fix For: 0.8.3 > > Attachments: KAFKA-1374.patch, KAFKA-1374.patch, > KAFKA-1374_2014-08-09_16:18:55.patch, KAFKA-1374_2014-08-12_22:23:06.patch, > KAFKA-1374_2014-09-23_21:47:12.patch, KAFKA-1374_2014-10-03_18:49:16.patch, > KAFKA-1374_2014-10-03_19:17:17.patch, KAFKA-1374_2015-01-18_00:19:21.patch, > KAFKA-1374_2015-05-18_22:55:48.patch, KAFKA-1374_2015-05-19_17:20:44.patch > > > This is a known issue, but opening a ticket to track. > If you try to compact a topic that has compressed messages you will run into > various exceptions - typically because during iteration we advance the > position based on the decompressed size of the message. I have a bunch of > stack traces, but it should be straightforward to reproduce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1374) LogCleaner (compaction) does not support compressed topics
[ https://issues.apache.org/jira/browse/KAFKA-1374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14549998#comment-14549998 ] Joel Koshy commented on KAFKA-1374: --- Created reviewboard https://reviews.apache.org/r/34397/diff/ against branch origin/trunk > LogCleaner (compaction) does not support compressed topics > -- > > Key: KAFKA-1374 > URL: https://issues.apache.org/jira/browse/KAFKA-1374 > Project: Kafka > Issue Type: Bug >Reporter: Joel Koshy >Assignee: Manikumar Reddy > Labels: newbie++ > Fix For: 0.8.3 > > Attachments: KAFKA-1374.patch, KAFKA-1374.patch, > KAFKA-1374_2014-08-09_16:18:55.patch, KAFKA-1374_2014-08-12_22:23:06.patch, > KAFKA-1374_2014-09-23_21:47:12.patch, KAFKA-1374_2014-10-03_18:49:16.patch, > KAFKA-1374_2014-10-03_19:17:17.patch, KAFKA-1374_2015-01-18_00:19:21.patch, > KAFKA-1374_2015-05-18_22:55:48.patch > > > This is a known issue, but opening a ticket to track. > If you try to compact a topic that has compressed messages you will run into > various exceptions - typically because during iteration we advance the > position based on the decompressed size of the message. I have a bunch of > stack traces, but it should be straightforward to reproduce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1374) LogCleaner (compaction) does not support compressed topics
[ https://issues.apache.org/jira/browse/KAFKA-1374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14548354#comment-14548354 ] Manikumar Reddy commented on KAFKA-1374: Updated reviewboard https://reviews.apache.org/r/24214/diff/ against branch origin/trunk > LogCleaner (compaction) does not support compressed topics > -- > > Key: KAFKA-1374 > URL: https://issues.apache.org/jira/browse/KAFKA-1374 > Project: Kafka > Issue Type: Bug >Reporter: Joel Koshy >Assignee: Manikumar Reddy > Labels: newbie++ > Fix For: 0.8.3 > > Attachments: KAFKA-1374.patch, KAFKA-1374_2014-08-09_16:18:55.patch, > KAFKA-1374_2014-08-12_22:23:06.patch, KAFKA-1374_2014-09-23_21:47:12.patch, > KAFKA-1374_2014-10-03_18:49:16.patch, KAFKA-1374_2014-10-03_19:17:17.patch, > KAFKA-1374_2015-01-18_00:19:21.patch, KAFKA-1374_2015-05-18_22:55:48.patch > > > This is a known issue, but opening a ticket to track. > If you try to compact a topic that has compressed messages you will run into > various exceptions - typically because during iteration we advance the > position based on the decompressed size of the message. I have a bunch of > stack traces, but it should be straightforward to reproduce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1374) LogCleaner (compaction) does not support compressed topics
[ https://issues.apache.org/jira/browse/KAFKA-1374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14539856#comment-14539856 ] Joel Koshy commented on KAFKA-1374: --- Sorry I dropped this. I just reviewed the patch. I think it looks good, but needs a rebase. Let me know if you are swamped though and we can help with it. > LogCleaner (compaction) does not support compressed topics > -- > > Key: KAFKA-1374 > URL: https://issues.apache.org/jira/browse/KAFKA-1374 > Project: Kafka > Issue Type: Bug >Reporter: Joel Koshy >Assignee: Manikumar Reddy > Labels: newbie++ > Fix For: 0.8.3 > > Attachments: KAFKA-1374.patch, KAFKA-1374_2014-08-09_16:18:55.patch, > KAFKA-1374_2014-08-12_22:23:06.patch, KAFKA-1374_2014-09-23_21:47:12.patch, > KAFKA-1374_2014-10-03_18:49:16.patch, KAFKA-1374_2014-10-03_19:17:17.patch, > KAFKA-1374_2015-01-18_00:19:21.patch > > > This is a known issue, but opening a ticket to track. > If you try to compact a topic that has compressed messages you will run into > various exceptions - typically because during iteration we advance the > position based on the decompressed size of the message. I have a bunch of > stack traces, but it should be straightforward to reproduce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1374) LogCleaner (compaction) does not support compressed topics
[ https://issues.apache.org/jira/browse/KAFKA-1374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14316735#comment-14316735 ] Jay Kreps commented on KAFKA-1374: -- Great! > LogCleaner (compaction) does not support compressed topics > -- > > Key: KAFKA-1374 > URL: https://issues.apache.org/jira/browse/KAFKA-1374 > Project: Kafka > Issue Type: Bug >Reporter: Joel Koshy >Assignee: Manikumar Reddy > Labels: newbie++ > Fix For: 0.8.3 > > Attachments: KAFKA-1374.patch, KAFKA-1374_2014-08-09_16:18:55.patch, > KAFKA-1374_2014-08-12_22:23:06.patch, KAFKA-1374_2014-09-23_21:47:12.patch, > KAFKA-1374_2014-10-03_18:49:16.patch, KAFKA-1374_2014-10-03_19:17:17.patch, > KAFKA-1374_2015-01-18_00:19:21.patch > > > This is a known issue, but opening a ticket to track. > If you try to compact a topic that has compressed messages you will run into > various exceptions - typically because during iteration we advance the > position based on the decompressed size of the message. I have a bunch of > stack traces, but it should be straightforward to reproduce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1374) LogCleaner (compaction) does not support compressed topics
[ https://issues.apache.org/jira/browse/KAFKA-1374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14316714#comment-14316714 ] Joel Koshy commented on KAFKA-1374: --- I can review this next week. However, as far as checking in is concerned I would strongly prefer to get KAFKA-1755 done first (for which I have a patch almost ready). The reason for that is that this patch is a significant change to the log cleaner and I would rather get some defensive code in first since the log cleaner health is critical for offset management as well as for Samza use-cases. > LogCleaner (compaction) does not support compressed topics > -- > > Key: KAFKA-1374 > URL: https://issues.apache.org/jira/browse/KAFKA-1374 > Project: Kafka > Issue Type: Bug >Reporter: Joel Koshy >Assignee: Manikumar Reddy > Labels: newbie++ > Fix For: 0.8.3 > > Attachments: KAFKA-1374.patch, KAFKA-1374_2014-08-09_16:18:55.patch, > KAFKA-1374_2014-08-12_22:23:06.patch, KAFKA-1374_2014-09-23_21:47:12.patch, > KAFKA-1374_2014-10-03_18:49:16.patch, KAFKA-1374_2014-10-03_19:17:17.patch, > KAFKA-1374_2015-01-18_00:19:21.patch > > > This is a known issue, but opening a ticket to track. > If you try to compact a topic that has compressed messages you will run into > various exceptions - typically because during iteration we advance the > position based on the decompressed size of the message. I have a bunch of > stack traces, but it should be straightforward to reproduce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1374) LogCleaner (compaction) does not support compressed topics
[
https://issues.apache.org/jira/browse/KAFKA-1374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14316691#comment-14316691
]
Manikumar Reddy commented on KAFKA-1374:
yes, we are using TestLogCleaning tool to test the changes.
TestLogCleaning stress test output for compressed messages
{code}
Producing 10 messages...
Logging produce requests to
/tmp/kafka-log-cleaner-produced-6014466306002699464.txt
Sleeping for 120 seconds...
Consuming messages...
Logging consumed messages to
/tmp/kafka-log-cleaner-consumed-177538909590644701.txt
10 rows of data produced, 13165 rows of data consumed (86.8% reduction).
De-duplicating and validating output files...
Validated 9005 values, 0 mismatches.
Producing 100 messages...
Logging produce requests to
/tmp/kafka-log-cleaner-produced-3298578695475992991.txt
Sleeping for 120 seconds...
Consuming messages...
Logging consumed messages to
/tmp/kafka-log-cleaner-consumed-7192293977610206930.txt
100 rows of data produced, 119926 rows of data consumed (88.0% reduction).
De-duplicating and validating output files...
Validated 89947 values, 0 mismatches.
Producing 1000 messages...
Logging produce requests to
/tmp/kafka-log-cleaner-produced-3336255463347572934.txt
Sleeping for 120 seconds...
Consuming messages...
Logging consumed messages to
/tmp/kafka-log-cleaner-consumed-9149188270705707725.txt
1000 rows of data produced, 1645281 rows of data consumed (83.5% reduction).
De-duplicating and validating output files...
Validated 899853 values, 0 mismatches.
{code}
> LogCleaner (compaction) does not support compressed topics
> --
>
> Key: KAFKA-1374
> URL: https://issues.apache.org/jira/browse/KAFKA-1374
> Project: Kafka
> Issue Type: Bug
>Reporter: Joel Koshy
>Assignee: Manikumar Reddy
> Labels: newbie++
> Fix For: 0.8.3
>
> Attachments: KAFKA-1374.patch, KAFKA-1374_2014-08-09_16:18:55.patch,
> KAFKA-1374_2014-08-12_22:23:06.patch, KAFKA-1374_2014-09-23_21:47:12.patch,
> KAFKA-1374_2014-10-03_18:49:16.patch, KAFKA-1374_2014-10-03_19:17:17.patch,
> KAFKA-1374_2015-01-18_00:19:21.patch
>
>
> This is a known issue, but opening a ticket to track.
> If you try to compact a topic that has compressed messages you will run into
> various exceptions - typically because during iteration we advance the
> position based on the decompressed size of the message. I have a bunch of
> stack traces, but it should be straightforward to reproduce.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1374) LogCleaner (compaction) does not support compressed topics
[ https://issues.apache.org/jira/browse/KAFKA-1374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14316526#comment-14316526 ] Jay Kreps commented on KAFKA-1374: -- Hey guys, the test kafka.tools.TestLogCleaning is a very aggressive test that runs against a kafka cluster configured for log compaction. It produces a bunch of messages and compacts them continuously and then does an out of band comparison of the two. It would be good to ensure that stills works on really large cleaner runs with deletes with this patch. > LogCleaner (compaction) does not support compressed topics > -- > > Key: KAFKA-1374 > URL: https://issues.apache.org/jira/browse/KAFKA-1374 > Project: Kafka > Issue Type: Bug >Reporter: Joel Koshy >Assignee: Manikumar Reddy > Labels: newbie++ > Fix For: 0.8.3 > > Attachments: KAFKA-1374.patch, KAFKA-1374_2014-08-09_16:18:55.patch, > KAFKA-1374_2014-08-12_22:23:06.patch, KAFKA-1374_2014-09-23_21:47:12.patch, > KAFKA-1374_2014-10-03_18:49:16.patch, KAFKA-1374_2014-10-03_19:17:17.patch, > KAFKA-1374_2015-01-18_00:19:21.patch > > > This is a known issue, but opening a ticket to track. > If you try to compact a topic that has compressed messages you will run into > various exceptions - typically because during iteration we advance the > position based on the decompressed size of the message. I have a bunch of > stack traces, but it should be straightforward to reproduce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1374) LogCleaner (compaction) does not support compressed topics
[ https://issues.apache.org/jira/browse/KAFKA-1374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14281480#comment-14281480 ] Manikumar Reddy commented on KAFKA-1374: Updated reviewboard https://reviews.apache.org/r/24214/diff/ against branch origin/trunk > LogCleaner (compaction) does not support compressed topics > -- > > Key: KAFKA-1374 > URL: https://issues.apache.org/jira/browse/KAFKA-1374 > Project: Kafka > Issue Type: Bug >Reporter: Joel Koshy >Assignee: Manikumar Reddy > Labels: newbie++ > Fix For: 0.8.3 > > Attachments: KAFKA-1374.patch, KAFKA-1374_2014-08-09_16:18:55.patch, > KAFKA-1374_2014-08-12_22:23:06.patch, KAFKA-1374_2014-09-23_21:47:12.patch, > KAFKA-1374_2014-10-03_18:49:16.patch, KAFKA-1374_2014-10-03_19:17:17.patch, > KAFKA-1374_2015-01-18_00:19:21.patch > > > This is a known issue, but opening a ticket to track. > If you try to compact a topic that has compressed messages you will run into > various exceptions - typically because during iteration we advance the > position based on the decompressed size of the message. I have a bunch of > stack traces, but it should be straightforward to reproduce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1374) LogCleaner (compaction) does not support compressed topics
[ https://issues.apache.org/jira/browse/KAFKA-1374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14157987#comment-14157987 ] Manikumar Reddy commented on KAFKA-1374: Updated reviewboard https://reviews.apache.org/r/24214/diff/ against branch origin/trunk > LogCleaner (compaction) does not support compressed topics > -- > > Key: KAFKA-1374 > URL: https://issues.apache.org/jira/browse/KAFKA-1374 > Project: Kafka > Issue Type: Bug >Reporter: Joel Koshy >Assignee: Manikumar Reddy > Labels: newbie++ > Fix For: 0.8.2 > > Attachments: KAFKA-1374.patch, KAFKA-1374_2014-08-09_16:18:55.patch, > KAFKA-1374_2014-08-12_22:23:06.patch, KAFKA-1374_2014-09-23_21:47:12.patch, > KAFKA-1374_2014-10-03_18:49:16.patch, KAFKA-1374_2014-10-03_19:17:17.patch > > > This is a known issue, but opening a ticket to track. > If you try to compact a topic that has compressed messages you will run into > various exceptions - typically because during iteration we advance the > position based on the decompressed size of the message. I have a bunch of > stack traces, but it should be straightforward to reproduce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1374) LogCleaner (compaction) does not support compressed topics
[ https://issues.apache.org/jira/browse/KAFKA-1374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14157960#comment-14157960 ] Manikumar Reddy commented on KAFKA-1374: Updated reviewboard https://reviews.apache.org/r/24214/diff/ against branch origin/trunk > LogCleaner (compaction) does not support compressed topics > -- > > Key: KAFKA-1374 > URL: https://issues.apache.org/jira/browse/KAFKA-1374 > Project: Kafka > Issue Type: Bug >Reporter: Joel Koshy >Assignee: Manikumar Reddy > Labels: newbie++ > Fix For: 0.8.2 > > Attachments: KAFKA-1374.patch, KAFKA-1374_2014-08-09_16:18:55.patch, > KAFKA-1374_2014-08-12_22:23:06.patch, KAFKA-1374_2014-09-23_21:47:12.patch, > KAFKA-1374_2014-10-03_18:49:16.patch > > > This is a known issue, but opening a ticket to track. > If you try to compact a topic that has compressed messages you will run into > various exceptions - typically because during iteration we advance the > position based on the decompressed size of the message. I have a bunch of > stack traces, but it should be straightforward to reproduce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1374) LogCleaner (compaction) does not support compressed topics
[ https://issues.apache.org/jira/browse/KAFKA-1374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14144971#comment-14144971 ] Manikumar Reddy commented on KAFKA-1374: Updated reviewboard https://reviews.apache.org/r/24214/diff/ against branch origin/trunk > LogCleaner (compaction) does not support compressed topics > -- > > Key: KAFKA-1374 > URL: https://issues.apache.org/jira/browse/KAFKA-1374 > Project: Kafka > Issue Type: Bug >Reporter: Joel Koshy >Assignee: Manikumar Reddy > Labels: newbie++ > Fix For: 0.8.2 > > Attachments: KAFKA-1374.patch, KAFKA-1374_2014-08-09_16:18:55.patch, > KAFKA-1374_2014-08-12_22:23:06.patch, KAFKA-1374_2014-09-23_21:47:12.patch > > > This is a known issue, but opening a ticket to track. > If you try to compact a topic that has compressed messages you will run into > various exceptions - typically because during iteration we advance the > position based on the decompressed size of the message. I have a bunch of > stack traces, but it should be straightforward to reproduce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1374) LogCleaner (compaction) does not support compressed topics
[ https://issues.apache.org/jira/browse/KAFKA-1374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14133498#comment-14133498 ] Neha Narkhede commented on KAFKA-1374: -- Bump. This is marked for 0.8.2. Feel free to reassign for review. > LogCleaner (compaction) does not support compressed topics > -- > > Key: KAFKA-1374 > URL: https://issues.apache.org/jira/browse/KAFKA-1374 > Project: Kafka > Issue Type: Bug >Reporter: Joel Koshy >Assignee: Manikumar Reddy > Labels: newbie++ > Fix For: 0.8.2 > > Attachments: KAFKA-1374.patch, KAFKA-1374_2014-08-09_16:18:55.patch, > KAFKA-1374_2014-08-12_22:23:06.patch > > > This is a known issue, but opening a ticket to track. > If you try to compact a topic that has compressed messages you will run into > various exceptions - typically because during iteration we advance the > position based on the decompressed size of the message. I have a bunch of > stack traces, but it should be straightforward to reproduce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1374) LogCleaner (compaction) does not support compressed topics
[ https://issues.apache.org/jira/browse/KAFKA-1374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14094307#comment-14094307 ] Manikumar Reddy commented on KAFKA-1374: Updated reviewboard https://reviews.apache.org/r/24214/diff/ against branch origin/trunk > LogCleaner (compaction) does not support compressed topics > -- > > Key: KAFKA-1374 > URL: https://issues.apache.org/jira/browse/KAFKA-1374 > Project: Kafka > Issue Type: Bug >Reporter: Joel Koshy >Assignee: Joel Koshy > Labels: newbie++ > Fix For: 0.8.2 > > Attachments: KAFKA-1374.patch, KAFKA-1374_2014-08-09_16:18:55.patch, > KAFKA-1374_2014-08-12_22:23:06.patch > > > This is a known issue, but opening a ticket to track. > If you try to compact a topic that has compressed messages you will run into > various exceptions - typically because during iteration we advance the > position based on the decompressed size of the message. I have a bunch of > stack traces, but it should be straightforward to reproduce. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1374) LogCleaner (compaction) does not support compressed topics
[ https://issues.apache.org/jira/browse/KAFKA-1374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14091719#comment-14091719 ] Manikumar Reddy commented on KAFKA-1374: Updated reviewboard https://reviews.apache.org/r/24214/diff/ against branch origin/trunk > LogCleaner (compaction) does not support compressed topics > -- > > Key: KAFKA-1374 > URL: https://issues.apache.org/jira/browse/KAFKA-1374 > Project: Kafka > Issue Type: Bug >Reporter: Joel Koshy >Assignee: Joel Koshy > Labels: newbie++ > Fix For: 0.8.2 > > Attachments: KAFKA-1374.patch, KAFKA-1374_2014-08-09_16:05:08.patch, > KAFKA-1374_2014-08-09_16:18:55.patch > > > This is a known issue, but opening a ticket to track. > If you try to compact a topic that has compressed messages you will run into > various exceptions - typically because during iteration we advance the > position based on the decompressed size of the message. I have a bunch of > stack traces, but it should be straightforward to reproduce. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1374) LogCleaner (compaction) does not support compressed topics
[ https://issues.apache.org/jira/browse/KAFKA-1374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14091717#comment-14091717 ] Manikumar Reddy commented on KAFKA-1374: Updated reviewboard https://reviews.apache.org/r/24214/diff/ against branch origin/trunk > LogCleaner (compaction) does not support compressed topics > -- > > Key: KAFKA-1374 > URL: https://issues.apache.org/jira/browse/KAFKA-1374 > Project: Kafka > Issue Type: Bug >Reporter: Joel Koshy >Assignee: Joel Koshy > Labels: newbie++ > Fix For: 0.8.2 > > Attachments: KAFKA-1374.patch, KAFKA-1374_2014-08-09_16:05:08.patch > > > This is a known issue, but opening a ticket to track. > If you try to compact a topic that has compressed messages you will run into > various exceptions - typically because during iteration we advance the > position based on the decompressed size of the message. I have a bunch of > stack traces, but it should be straightforward to reproduce. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1374) LogCleaner (compaction) does not support compressed topics
[ https://issues.apache.org/jira/browse/KAFKA-1374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14087675#comment-14087675 ] Neha Narkhede commented on KAFKA-1374: -- Assigning to [~jjkoshy] for review > LogCleaner (compaction) does not support compressed topics > -- > > Key: KAFKA-1374 > URL: https://issues.apache.org/jira/browse/KAFKA-1374 > Project: Kafka > Issue Type: Bug >Reporter: Joel Koshy >Assignee: Joel Koshy > Labels: newbie++ > Fix For: 0.8.2 > > Attachments: KAFKA-1374.patch > > > This is a known issue, but opening a ticket to track. > If you try to compact a topic that has compressed messages you will run into > various exceptions - typically because during iteration we advance the > position based on the decompressed size of the message. I have a bunch of > stack traces, but it should be straightforward to reproduce. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1374) LogCleaner (compaction) does not support compressed topics
[ https://issues.apache.org/jira/browse/KAFKA-1374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14084047#comment-14084047 ] Manikumar Reddy commented on KAFKA-1374: In this patch, LogCleaner decompress the compressed messages and writes back the retained messages in compressed form. I tested the following scenarios 1. Topic with non-compressed messages 2. Topic with compressed messages 3. Topic with both compressed and non-compressed messages. > LogCleaner (compaction) does not support compressed topics > -- > > Key: KAFKA-1374 > URL: https://issues.apache.org/jira/browse/KAFKA-1374 > Project: Kafka > Issue Type: Bug >Reporter: Joel Koshy >Assignee: Manikumar Reddy > Labels: newbie++ > Fix For: 0.8.2 > > Attachments: KAFKA-1374.patch > > > This is a known issue, but opening a ticket to track. > If you try to compact a topic that has compressed messages you will run into > various exceptions - typically because during iteration we advance the > position based on the decompressed size of the message. I have a bunch of > stack traces, but it should be straightforward to reproduce. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1374) LogCleaner (compaction) does not support compressed topics
[ https://issues.apache.org/jira/browse/KAFKA-1374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14084044#comment-14084044 ] Manikumar Reddy commented on KAFKA-1374: Created reviewboard https://reviews.apache.org/r/24214/diff/ against branch origin/trunk > LogCleaner (compaction) does not support compressed topics > -- > > Key: KAFKA-1374 > URL: https://issues.apache.org/jira/browse/KAFKA-1374 > Project: Kafka > Issue Type: Bug >Reporter: Joel Koshy > Labels: newbie++ > Fix For: 0.8.2 > > Attachments: KAFKA-1374.patch > > > This is a known issue, but opening a ticket to track. > If you try to compact a topic that has compressed messages you will run into > various exceptions - typically because during iteration we advance the > position based on the decompressed size of the message. I have a bunch of > stack traces, but it should be straightforward to reproduce. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1374) LogCleaner (compaction) does not support compressed topics
[
https://issues.apache.org/jira/browse/KAFKA-1374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14078858#comment-14078858
]
Manikumar Reddy commented on KAFKA-1374:
I am trying to look in to the issue and WIP Patch.
we are using the following code to traverse segment offsets .
{code}
var currOffset = segment.baseOffset
while (currOffset < segment.index.lastOffset) {
currOffset = entry.nextOffset
}
{code}
As per my observation, segment.index.lastOffset is not giving the last offset
of a given segment.
I have a segment with startingOffset=0 and lastOffset=7140. I am getting
segment.index.lastOffset=7118.
This is creating some issue in the code.
Any idea on why segment.index.lastOffset is not returning proper lastOffset.?
> LogCleaner (compaction) does not support compressed topics
> --
>
> Key: KAFKA-1374
> URL: https://issues.apache.org/jira/browse/KAFKA-1374
> Project: Kafka
> Issue Type: Bug
>Reporter: Joel Koshy
> Labels: newbie++
> Fix For: 0.8.2
>
>
> This is a known issue, but opening a ticket to track.
> If you try to compact a topic that has compressed messages you will run into
> various exceptions - typically because during iteration we advance the
> position based on the decompressed size of the message. I have a bunch of
> stack traces, but it should be straightforward to reproduce.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (KAFKA-1374) LogCleaner (compaction) does not support compressed topics
[ https://issues.apache.org/jira/browse/KAFKA-1374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14037669#comment-14037669 ] Joel Koshy commented on KAFKA-1374: --- I had started on this a while ago, but did not finish. Here's a WIP patch that doesn't quite work yet: https://gist.github.com/jjkoshy/4657a44e52e3f88be1c1 Another nuance with compression and compaction is what compression-codec do we use when writing out the compacted data? We could adopt a broker-side compression config that can be overridden on a per-topic basis. This would not only enable a consistent compression codec for each topic, but it will also make the above decision more straightforward - i.e., write out compacted messages in the configured compression codec for that topic (or broker-default if the topic does not have any override). Will file a separate jira for the above. > LogCleaner (compaction) does not support compressed topics > -- > > Key: KAFKA-1374 > URL: https://issues.apache.org/jira/browse/KAFKA-1374 > Project: Kafka > Issue Type: Bug >Reporter: Joel Koshy > Fix For: 0.8.2 > > > This is a known issue, but opening a ticket to track. > If you try to compact a topic that has compressed messages you will run into > various exceptions - typically because during iteration we advance the > position based on the decompressed size of the message. I have a bunch of > stack traces, but it should be straightforward to reproduce. -- This message was sent by Atlassian JIRA (v6.2#6252)
