[DISCUSSION] Forceful minor Compaction
Hi all, I was looking into compaction and had a query regarding the same. If you have auto compaction turned on and the threshold level is 4,3 Now try loading data 7 times which will make 7 segments. Now as the auto compaction was on therefore 4 segments will be merged to 0.1. The segments visible will be 0.1, 4, 5, 6. When i try to run the compaction command nothing happens because the threshold level is not reached for compaction to happen. What if i want to merge the 3 segments i.e (4, 5, 6) and make a level-1 compacted segment. The proposed solution would be to add a new option to the compaction command which contains what level of compaction the user would like to do Example:- alter table carbon_table compact 'minor' level '1'. This would forcefully combine the segments(4, 5, 6) in to a level 1 compacted segment called 4.1 giving me 2 level-1 compacted segment (0.1 and 4.1) Similar operation can be done with level-1 compacted segments.
Re: [DISCUSSION] Forceful minor Compaction
Hi Liang, Currently the user cannot choose which segments should be merged or not. We should give a DDL which supports both forceful minor and major compaction separately. Another point would be if the user can specify the segments which he needs merged. Obviously the segments that can be given should meet some criteria. for example:-- no major compacted or 2nd Level compacted segments can be provided during forced minor compaction. -- Regards *Kunal Kapoor* Software Consultant *Knoldus Software LLP <http://www.knoldus.com/home.knol>* Twitter <http://twitter/kunal642> | LinkedIn <https://in.linkedin.com/in/kunal642> | FB <https://www.facebook.com/kunalkapoor642> Mobile: +91-9711013430 On Wed, Apr 19, 2017 at 2:33 PM, Liang Chen wrote: > Hi Kunal > > Thank you for taking the good topic for discussion. > First , let us think about : why users want to do forceful minor > compaction, > which cases? > Current "MAJOR compaction" whether can cover "forceful MINOR compaction" > scenarios ? > > As we know, compaction is mainly for optimizing index effectiveness by > merging for multiple times data loading(segments), so current system > provides two options for users to choose :"soft compaction(minor) and > strong > compaction(major)"。 > > So i could not find the typical cases to add a new strong > compaction(forceful minor)。 > > Regards > Liang > > > Kunal Kapoor wrote > > Hi all, > > I was looking into compaction and had a query regarding the same. > > If you have auto compaction turned on and the threshold level is 4,3 > > Now try loading data 7 times which will make 7 segments. Now as the auto > > compaction was on therefore 4 segments will be merged to 0.1. > > The segments visible will be 0.1, 4, 5, 6. > > > > When i try to run the compaction command nothing happens because the > > threshold level is not reached for compaction to happen. > > What if i want to merge the 3 segments i.e (4, 5, 6) and make a level-1 > > compacted segment. > > > > The proposed solution would be to add a new option to the compaction > > command which contains what level of compaction the user would like to do > > > > Example:- alter table carbon_table compact 'minor' level '1'. > > This would forcefully combine the segments(4, 5, 6) in to a level 1 > > compacted segment called 4.1 giving me 2 level-1 compacted segment (0.1 > > and > > 4.1) > > Similar operation can be done with level-1 compacted segments. > > > > > > -- > View this message in context: http://apache-carbondata- > mailing-list-archive.1130556.n5.nabble.com/DISCUSSION- > Forceful-minor-Compaction-tp10924p11319.html > Sent from the Apache CarbonData Mailing List archive mailing list archive > at Nabble.com. >
[jira] [Created] (CARBONDATA-346) Update unit test for core module
Kunal Kapoor created CARBONDATA-346: --- Summary: Update unit test for core module Key: CARBONDATA-346 URL: https://issues.apache.org/jira/browse/CARBONDATA-346 Project: CarbonData Issue Type: Test Reporter: Kunal Kapoor Priority: Trivial Improve code coverage for util package. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CARBONDATA-773) During parallel load multiple instances of DictionaryServer are being created.
Kunal Kapoor created CARBONDATA-773: --- Summary: During parallel load multiple instances of DictionaryServer are being created. Key: CARBONDATA-773 URL: https://issues.apache.org/jira/browse/CARBONDATA-773 Project: CarbonData Issue Type: Improvement Reporter: Kunal Kapoor Priority: Minor During parallel load multiple instances of DictionaryServer are being created. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (CARBONDATA-839) Table lock file is not getting deleted after table rename is successful
Kunal Kapoor created CARBONDATA-839: --- Summary: Table lock file is not getting deleted after table rename is successful Key: CARBONDATA-839 URL: https://issues.apache.org/jira/browse/CARBONDATA-839 Project: CarbonData Issue Type: Bug Reporter: Kunal Kapoor Assignee: Kunal Kapoor Priority: Minor Table lock file is not getting deleted after table rename is successful -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (CARBONDATA-846) Add support to revert changes to alter table commands if there is a failure while executing the changes on hive.
Kunal Kapoor created CARBONDATA-846: --- Summary: Add support to revert changes to alter table commands if there is a failure while executing the changes on hive. Key: CARBONDATA-846 URL: https://issues.apache.org/jira/browse/CARBONDATA-846 Project: CarbonData Issue Type: Improvement Reporter: Kunal Kapoor Assignee: Kunal Kapoor -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (CARBONDATA-926) Higher MAXCOLUMNS value in load DML options is leading to out of memory error
Kunal Kapoor created CARBONDATA-926: --- Summary: Higher MAXCOLUMNS value in load DML options is leading to out of memory error Key: CARBONDATA-926 URL: https://issues.apache.org/jira/browse/CARBONDATA-926 Project: CarbonData Issue Type: Improvement Reporter: Kunal Kapoor Assignee: Kunal Kapoor -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (CARBONDATA-957) Table not found exception in rename table after lock acquire failure
Kunal Kapoor created CARBONDATA-957: --- Summary: Table not found exception in rename table after lock acquire failure Key: CARBONDATA-957 URL: https://issues.apache.org/jira/browse/CARBONDATA-957 Project: CarbonData Issue Type: Bug Reporter: Kunal Kapoor Assignee: Kunal Kapoor Priority: Minor During rename table if an exception is thrown during acquiring locks then table not exists exception was thrown while reverting the changes. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
