[ https://issues.apache.org/jira/browse/KAFKA-6150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16286934#comment-16286934 ]
ASF GitHub Bot commented on KAFKA-6150: --------------------------------------- GitHub user guozhangwang opened a pull request: https://github.com/apache/kafka/pull/4315 KAFKA-6150: KIP-204 part III; Change repartition topic segment size and ms 1. Create default internal topic configs in StreamsConfig, especially for repartition topics change the segment size and time to smaller value. 2. Consolidate the default internal topic settings to InternalTopicManager and simplify InternalTopicConfig correspondingly. 3. Add an integration test for purging data. 4. MINOR: change TopologyBuilderException to IllegalStateException in StreamPartitionAssignor (part of https://issues.apache.org/jira/browse/KAFKA-5660). Here are a few public facing APIs that get added: 1. AbstractConfig#originalsWithPrefix(String prefix, boolean strip). 2. KafkaStreams constructor with Time object for convienent mocking in tests. Will update KIP-204 accordingly if people re-votes these changes. ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) You can merge this pull request into a Git repository by running: $ git pull https://github.com/guozhangwang/kafka K6150-segment-size Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/4315.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4315 ---- commit 7f394817cc0a7a5606ee2cc1bae0d3e271c307b9 Author: Matthias J. Sax <matth...@confluent.io> Date: 2017-11-21T02:17:57Z KAFKA-4857: Replace StreamsKafkaClient with AdminClient in Kafka Streams commit 49e832c297b93c432cdbffddca5635dbdecfc565 Author: Matthias J. Sax <matth...@confluent.io> Date: 2017-12-04T19:36:29Z Github comments - fixed parameter passing for `retries` - added couple of tests commit ad7ccde56910a0b52825470fdd6a54339358445d Author: Matthias J. Sax <matth...@confluent.io> Date: 2017-12-04T21:01:46Z rebased commit bfd3937666ecc82e7584083fd1f17d0bacba8962 Author: Matthias J. Sax <matth...@confluent.io> Date: 2017-12-05T05:38:36Z fixed system test error message commit 81b17ba0b60e9d11cd6a23d4f91786d499869393 Author: Guozhang Wang <wangg...@gmail.com> Date: 2017-12-07T02:53:33Z Merge branch 'kafka-4857-admit-client' of https://github.com/mjsax/kafka into K6150-segment-size commit 3c286b94e2d698ab41c90668b2b9ada8212ab2d8 Author: Guozhang Wang <wangg...@gmail.com> Date: 2017-12-08T00:14:04Z add internal topic types commit 6c223b89c0b81769930c86390de7a8f745e71780 Author: Guozhang Wang <wangg...@gmail.com> Date: 2017-12-08T02:30:12Z use default props for internal topics commit 177d11a400f1574f7c1fbac1e233493e3d43b944 Author: Guozhang Wang <wangg...@gmail.com> Date: 2017-12-11T17:36:11Z Merge branch 'trunk' of https://git-wip-us.apache.org/repos/asf/kafka into K6150-segment-size commit ef57e80633bd39335bc184bed59d55179d18b69b Author: Guozhang Wang <wangg...@gmail.com> Date: 2017-12-12T01:20:55Z add integration test for purging repartition topic ---- > Make Repartition Topics Transient > --------------------------------- > > Key: KAFKA-6150 > URL: https://issues.apache.org/jira/browse/KAFKA-6150 > Project: Kafka > Issue Type: Improvement > Components: streams > Reporter: Guozhang Wang > Assignee: Guozhang Wang > Labels: operability > Fix For: 1.1.0 > > > Unlike changelog topics, the repartition topics could just be short-lived. > Today users have different ways to configure them with short retention such > as enforce a short retention period or use AppendTime for repartition topics. > All these would be cumbersome and Streams should just do this for the users. > One way to do it is use the “purgeData” admin API (KIP-107) such that after > the offset of the input topics are committed, if the input topics are > actually repartition topics, we would purge the data immediately. -- This message was sent by Atlassian JIRA (v6.4.14#64029)