[GitHub] carbondata issue #1812: [CARBONDATA-2033]Support user specified segments in ...
Github user manishgupta88 commented on the issue: https://github.com/apache/carbondata/pull/1812 LGTM ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]Support user specified segments in ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1812 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4626/ ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]Support user specified segments in ...
Github user manishgupta88 commented on the issue: https://github.com/apache/carbondata/pull/1812 retest sdv please ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]Support user specified segments in ...
Github user Xaprice commented on the issue: https://github.com/apache/carbondata/pull/1812 I've raised a sub-task for custom compaction for child tables/datamaps: https://issues.apache.org/jira/browse/CARBONDATA-2412 ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]Support user specified segments in ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1812 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4296/ ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]Support user specified segments in ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1812 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5460/ ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]Support user specified segments in ...
Github user manishgupta88 commented on the issue: https://github.com/apache/carbondata/pull/1812 LGTM...can be merged once build is passed Please raise a sub-jira task under the same jira to track the Custom compaction implementation for child tables/datamaps and add the jira link link here as we need to implement custom compaction for child tables/datamaps also. ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]Support user specified segments in ...
Github user Xaprice commented on the issue: https://github.com/apache/carbondata/pull/1812 @manishgupta88, I've submitted some changes, have a look please. ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]Support user specified segments in ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1812 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4527/ ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]Support user specified segments in ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1812 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5389/ ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]Support user specified segments in ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1812 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4524/ ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]Support user specified segments in ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1812 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4222/ ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]Support user specified segments in ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1812 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5387/ ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]Support user specified segments in ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1812 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4220/ ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]Support user specified segments in ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1812 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5362/ ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]Support user specified segments in ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1812 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4188/ ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]Support user specified segments in ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1812 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4500/ ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]Support user specified segments in ...
Github user Xaprice commented on the issue: https://github.com/apache/carbondata/pull/1812 retest this please ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]Support user specified segments in ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1812 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/4185/ ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]Support user specified segments in ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1812 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4465/ ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]Support user specified segments in ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1812 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3972/ ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]Support user specified segments in ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1812 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5182/ ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]Support user specified segments in ...
Github user bill1208 commented on the issue: https://github.com/apache/carbondata/pull/1812 I agree with @gvramana ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]Support user specified segments in ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1812 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4293/ ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]Support user specified segments in ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1812 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3049/ ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]Support user specified segments in ...
Github user manishgupta88 commented on the issue: https://github.com/apache/carbondata/pull/1812 I agree with @gvramana 1. We should not use Major/Minor compaction type as they have a specific meaning and both are controlled by the system for taking decisions whether segment is valid to be compacted or not. 2. We should not use carbon.input.segments.default.seg_compact to set the segments to be compacted. 3. We should introduce a new compaction type in the DDL 'CUSTOM' as suggested above because it is something like force compaction for the given segments as it will not check for size and frequency of segments. We can work on using the below syntax for custom compaction. **ALTER TABLE [db_name.]table_name COMPACT 'CUSTOM' WHERE SEGMENT.ID IN (0,5,8)** Once a table is compacted using Custom compaction, then minor compaction does not hold good for the custom compacted segment. Custom compacted segment should only participate during major compaction if it satisfies the major compaction size property. ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]Support user specified segments in ...
Github user xuchuanyin commented on the issue: https://github.com/apache/carbondata/pull/1812 @gvramana I think âmajorâ and âminorâ is enough to describe compaction, there is no need to add another on. And 'custom' is somewhat ambiguous. As it is described in readme, ``` In Major compaction, multiple segments can be merged into one large segment. User will specify the compaction size until which segments can be merged. ``` The previous (default without condition) major compaction is size based, carbondata choose the segments by size. And for the newly major compaction (with condition), we specify the segments and let carbondata merge them into one large segment. They are no different. So we don't need an another compaction type. ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]Support user specified segments in ...
Github user xuchuanyin commented on the issue: https://github.com/apache/carbondata/pull/1812 @Xaprice @chenliang613 @ravipesala @gvramana I think the syntax of segment compaction should be similar with that of other management on segment. Currently in carbondata, we delete segment using syntax: ``` DELETE FROM TABLE CarbonDatabase.CarbonTable WHERE SEGMENT.ID IN (0,5,8) ``` And ``` DELETE FROM TABLE CarbonDatabase.CarbonTable WHERE SEGMENT.STARTTIME BEFORE '2017-06-01 12:05:06' ``` So, we can imitate the above syntax and get the followings: ``` ALTER TABLE [db_name.]table_name COMPACT 'MINOR/MAJOR' WHERE SEGMENT.ID IN (0,5,8) ``` And ``` ALTER TABLE [db_name.]table_name COMPACT 'MINOR/MAJOR' WHERE SEGMENT.STARTTIME BEFORE '2017-06-01 12:05:06' AND SEGMENT.STARTTIME AFTER '2017-05-01 12:05:06' ``` We can support compact segment by specifying IDs and dates. ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]Support user specified segments in ...
Github user gvramana commented on the issue: https://github.com/apache/carbondata/pull/1812 @Xaprice Currently Minor and Major compaction has fixed meaning, minor is based on frequency of segments and Major is based on size. So better to not to change the current meaning. Also CARBON_INPUT_SEGMENTS will impact only read query but will not impact any other DDL/DML. So you can add a new compaction type CUSTOM and pass the required segments in the same command, so that it will not create any confusion. so command can be ALTER TABLE tablename compact 'CUSTOM' '1, 2, 3, 4' It is also required to mention in documentation that it will not respect other features like preserve_segments, size etc. Also invalid segments in list are ignored. Also CUSTOM compacted segments will not participate in minor compaction triggered later. ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]Support user specified segments in ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1812 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4085/ ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]Support user specified segments in ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1812 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2840/ ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]Support user specified segments in ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1812 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2786/ ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]Support user specified segments in ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1812 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4030/ ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]Support user specified segments in ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1812 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2781/ ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]Support user specified segments in ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1812 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4026/ ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]Support user specified segments in ...
Github user Xaprice commented on the issue: https://github.com/apache/carbondata/pull/1812 retest this please ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]support user specified segments in ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1812 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2778/ ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]support user specified segments in ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1812 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4022/ ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]support user specified segments in ...
Github user Xaprice commented on the issue: https://github.com/apache/carbondata/pull/1812 @ravipesala Compacting adjacent segments is certainly the best practice in most cases. But is it not flexible enough to take it as a mandatory rule? ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]support user specified segments in ...
Github user Xaprice commented on the issue: https://github.com/apache/carbondata/pull/1812 @chenliang613 For question 1: I thought minor compaction are mainly used in auto-merging scenario. But after reconsidering this feature, maybe it's better to support both major and minor compaction. I will add support of minor compaction soon. For question 2: I will follow your advice and modify the syntax to keep consistent syntax as "query with specified segments". ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]support user specified segments in ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1812 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2923/ ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]support user specified segments in ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1812 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1690/ ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]support user specified segments in ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1812 @Xaprice I think we should have validation for the order of segments to be merged. For suppose we have segments of 1 to 8, and the user gives the compaction on 1, 5, 8 then this should not be valid as it will impact the order of data it is inserted. ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]support user specified segments in ...
Github user chenliang613 commented on the issue: https://github.com/apache/carbondata/pull/1812 Two questions: 1. Why only consider major compaction with specified segments, no need to consider minor compaction? 2. Whether can keep consistent syntax as "query with specified segments", or not ? a. First set segment id : "SET carbon.input.segments.dbname.tablename=1,3" b.Do compaction : ALTER TABLE tablename compact 'MAJOR' ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]support user specified segments in ...
Github user chenliang613 commented on the issue: https://github.com/apache/carbondata/pull/1812 please change the title to : [CARBONDATA-2033] Support user specified segments in major compaction ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]support user specified segments in ...
Github user chenliang613 commented on the issue: https://github.com/apache/carbondata/pull/1812 retest this please ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]support user specified segments in ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1812 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2923/ ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]support user specified segments in ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1812 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1593/ ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]support user specified segments in ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1812 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2828/ ---
[GitHub] carbondata issue #1812: [CARBONDATA-2033]support user specified segments in ...
Github user Xaprice commented on the issue: https://github.com/apache/carbondata/pull/1812 Hi @chenliang613 , can you please take a look? ---
