[GitHub] carbondata issue #1364: [CARBONDATA-1483]open auto merge when loading and in...
Github user chenliang613 commented on the issue: https://github.com/apache/carbondata/pull/1364 @xubo245 please correct your pr title with the correct format , like : [CARBONDATA-1483] Open auto merge when loading and inserting into ---
[jira] [Resolved] (CARBONDATA-1425) Inappropriate Exception displays while creating a new partition with incorrect partition type
[ https://issues.apache.org/jira/browse/CARBONDATA-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen resolved CARBONDATA-1425. Resolution: Fixed Fix Version/s: 1.2.0 > Inappropriate Exception displays while creating a new partition with > incorrect partition type > - > > Key: CARBONDATA-1425 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1425 > Project: CarbonData > Issue Type: Improvement > Components: data-query >Affects Versions: 1.2.0 > Environment: spark 2.1 >Reporter: Vandana Yadav >Assignee: Jarck >Priority: Minor > Fix For: 1.2.0 > > Attachments: 2000_UniqData.csv > > Time Spent: 6h > Remaining Estimate: 0h > > Inappropriate Exception displays while creating a new partition with > incorrect partition type > Steps to Reproduce: > 1)Create table: > CREATE TABLE uniqdata_part1 (CUST_NAME String,ACTIVE_EMUI_VERSION string,DOB > Timestamp,DOJ timestamp, BIGINT_COLUMN1 bigint,BIGINT_COLUMN2 > bigint,DECIMAL_COLUMN1 decimal(30,10), DECIMAL_COLUMN2 > decimal(36,10),Double_COLUMN1 double, Double_COLUMN2 double,INTEGER_COLUMN1 > int) PARTITIONED BY (CUST_ID int) STORED BY 'org.apache.carbondata.format' > TBLPROPERTIES > ('PARTITION_TYPE'='RANGE','RANGE_INFO'='9090,9500,9800',"TABLE_BLOCKSIZE"= > "256 MB") > 2)Load data: > LOAD DATA INPATH 'hdfs://localhost:54310/uniqdata/2000_UniqData.csv' into > table uniqdata_part1 OPTIONS('DELIMITER'=',' , > 'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1') > 3) Execute Query: > ALTER TABLE uniqdata_part1 ADD PARTITION ('abc') > 4) Result on beeline > Error: java.lang.RuntimeException: Range info must be in ascending order, > please check again! (state=,code=0) > 5) Expected Result: it should display a proper exception like invalid > partition definition as we are providing a string value at the place of the > integer value. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] carbondata pull request #1336: [CARBONDATA-1425] Inappropriate Exception dis...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/1336 ---
[GitHub] carbondata issue #1364: [CARBONDATA-1483]open auto merge when loading and in...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1364 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/840/ ---
[GitHub] carbondata issue #1322: [CARBONDATA-1450] Support timestamp more than 68 yea...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1322 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/210/ ---
[GitHub] carbondata issue #1364: [CARBONDATA-1483]open auto merge when loading and in...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1364 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/839/ ---
[GitHub] carbondata issue #1322: [CARBONDATA-1450] Support timestamp more than 68 yea...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1322 Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/86/ ---
[GitHub] carbondata issue #1322: [CARBONDATA-1450] Support timestamp more than 68 yea...
Github user dhatchayani commented on the issue: https://github.com/apache/carbondata/pull/1322 Retest this please ---
[GitHub] carbondata issue #1364: [CARBONDATA-1483]open auto merge when loading and in...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1364 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/208/ ---
[GitHub] carbondata issue #1364: [CARBONDATA-1483]open auto merge when loading and in...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1364 Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/85/ ---
[jira] [Updated] (CARBONDATA-1483) open auto merge
[ https://issues.apache.org/jira/browse/CARBONDATA-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xubo245 updated CARBONDATA-1483: Description: auto compaction when loading and inserting into * set DEFAULT_ENABLE_AUTO_LOAD_MERGE as true * add test case and verify the change is ok was: auto compaction when loading data * set DEFAULT_ENABLE_AUTO_LOAD_MERGE as true * add test case and verify the change is ok Summary: open auto merge (was: open auto merge when loading data) > open auto merge > --- > > Key: CARBONDATA-1483 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1483 > Project: CarbonData > Issue Type: Improvement >Reporter: xubo245 >Assignee: xubo245 >Priority: Minor > Original Estimate: 120h > Time Spent: 0.5h > Remaining Estimate: 119.5h > > auto compaction when loading and inserting into > * set DEFAULT_ENABLE_AUTO_LOAD_MERGE as true > * add test case and verify the change is ok -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] carbondata issue #1364: [CARBONDATA-1483]open auto merge when loading data
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1364 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/207/ ---
[GitHub] carbondata issue #1364: [CARBONDATA-1483]open auto merge when loading data
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1364 Build Failed with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/84/ ---
[jira] [Updated] (CARBONDATA-1483) open auto merge when loading data
[ https://issues.apache.org/jira/browse/CARBONDATA-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xubo245 updated CARBONDATA-1483: Description: auto compaction when loading data * set DEFAULT_ENABLE_AUTO_LOAD_MERGE as true * add test case and verify the change is ok was: auto compaction when loading data set DEFAULT_ENABLE_AUTO_LOAD_MERGE as true add test case and verify the change is ok > open auto merge when loading data > - > > Key: CARBONDATA-1483 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1483 > Project: CarbonData > Issue Type: Improvement >Reporter: xubo245 >Assignee: xubo245 >Priority: Minor > Original Estimate: 120h > Remaining Estimate: 120h > > auto compaction when loading data > * set DEFAULT_ENABLE_AUTO_LOAD_MERGE as true > * add test case and verify the change is ok -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] carbondata pull request #1364: [CARBONDATA-1483]open auto merge when loading...
GitHub user xubo245 opened a pull request: https://github.com/apache/carbondata/pull/1364 [CARBONDATA-1483]open auto merge when loading data auto compaction when loading data * set DEFAULT_ENABLE_AUTO_LOAD_MERGE as true * add test case and verify the change is ok You can merge this pull request into a Git repository by running: $ git pull https://github.com/xubo245/carbondata compaction Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/1364.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1364 commit 1b2be64cebe55baf5661262a008bc1bbbcb16b67 Author: xubo245 <601450...@qq.com> Date: 2017-09-18T03:41:03Z open auto merge when loading data ---
[jira] [Created] (CARBONDATA-1483) open auto merge when loading data
xubo245 created CARBONDATA-1483: --- Summary: open auto merge when loading data Key: CARBONDATA-1483 URL: https://issues.apache.org/jira/browse/CARBONDATA-1483 Project: CarbonData Issue Type: Improvement Reporter: xubo245 Assignee: xubo245 Priority: Minor auto compaction when loading data set DEFAULT_ENABLE_AUTO_LOAD_MERGE as true add test case and verify the change is ok -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] carbondata issue #1317: [CARBONDATA-1316] drop partition
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1317 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/838/ ---
[GitHub] carbondata issue #1361: [CARBONDATA-1481]Compaction support global sort
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/1361 Have SDV accident error? I haven't changed the SDV code after SDV Build fail, but now it shows Success. ---
[GitHub] carbondata issue #1317: [CARBONDATA-1316] drop partition
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1317 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/206/ ---
[GitHub] carbondata issue #1361: [CARBONDATA-1481]Compaction support global sort
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1361 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/837/ ---
[GitHub] carbondata issue #1317: [CARBONDATA-1316] drop partition
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1317 Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/83/ ---
[GitHub] carbondata pull request #1259: [Review][CARBONDATA-1381] Add test cases for ...
Github user lionelcao commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1259#discussion_r139330679 --- Diff: core/src/main/java/org/apache/carbondata/core/metadata/schema/PartitionInfo.java --- @@ -17,17 +17,21 @@ package org.apache.carbondata.core.metadata.schema; +import java.io.DataInput; +import java.io.DataOutput; +import java.io.IOException; import java.io.Serializable; import java.util.ArrayList; import java.util.List; import org.apache.carbondata.core.metadata.schema.partition.PartitionType; +import org.apache.carbondata.core.metadata.schema.table.Writable; import org.apache.carbondata.core.metadata.schema.table.column.ColumnSchema; /** * Partition information of carbon partition table */ -public class PartitionInfo implements Serializable { +public class PartitionInfo implements Serializable, Writable { --- End diff -- This modification will fix a bug : PartitionInfo is null in new CarbonTable in getSplits function in CarbonTableInputFormat. Could you split it into a independent PR? Looks inappropriate with the description of current PR. ---
[GitHub] carbondata issue #1361: [CARBONDATA-1481]Compaction support global sort
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1361 Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/82/ ---
[GitHub] carbondata issue #1361: [CARBONDATA-1481]Compaction support global sort
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1361 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/205/ ---
[GitHub] carbondata issue #1361: [CARBONDATA-1481]Compaction support global sort
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1361 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/836/ ---
[GitHub] carbondata issue #1361: [CARBONDATA-1481]Compaction support global sort
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1361 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/204/ ---
[GitHub] carbondata issue #1361: [CARBONDATA-1481]Compaction support global sort
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1361 Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/81/ ---
[GitHub] carbondata issue #1317: [CARBONDATA-1316] drop partition
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1317 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/835/ ---
[GitHub] carbondata pull request #1291: [CARBONDATA-1343] Hive can't query data when ...
Github user anubhav100 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1291#discussion_r139315604 --- Diff: hadoop/src/main/java/org/apache/carbondata/hadoop/CarbonInputFormat.java --- @@ -133,7 +133,7 @@ public static void setTableInfo(Configuration configuration, TableInfo tableInfo /** * Get TableInfo object from `configuration` */ - private TableInfo getTableInfo(Configuration configuration) throws IOException { + public TableInfo getTableInfo(Configuration configuration) throws IOException { --- End diff -- @cenyuhai change in this file is not required now because MapredCarbonInputFormat now extends CarbonTableInputFormat ---
[GitHub] carbondata issue #1317: [CARBONDATA-1316] drop partition
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1317 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/834/ ---
[GitHub] carbondata issue #1317: [CARBONDATA-1316] drop partition
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1317 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/203/ ---
[GitHub] carbondata issue #1317: [CARBONDATA-1316] drop partition
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1317 Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/80/ ---
[GitHub] carbondata issue #1317: [CARBONDATA-1316] drop partition
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1317 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/202/ ---
[GitHub] carbondata issue #1317: [CARBONDATA-1316] drop partition
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1317 Build Failed with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/79/ ---
[GitHub] carbondata issue #1336: [CARBONDATA-1425] Inappropriate Exception displays w...
Github user lionelcao commented on the issue: https://github.com/apache/carbondata/pull/1336 LGTM ---
[GitHub] carbondata issue #1317: [CARBONDATA-1316] drop partition
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1317 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/833/ ---
[GitHub] carbondata issue #1363: [CARBONDATA-1378] support creating table in hive
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1363 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/832/ ---
[GitHub] carbondata issue #1317: [CARBONDATA-1316] drop partition
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1317 Build Failed with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/78/ ---
[GitHub] carbondata issue #1317: [CARBONDATA-1316] drop partition
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1317 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/201/ ---
[GitHub] carbondata issue #1317: [CARBONDATA-1316] drop partition
Github user chenliang613 commented on the issue: https://github.com/apache/carbondata/pull/1317 retest this please ---
[GitHub] carbondata issue #1363: [CARBONDATA-1378] support creating table in hive
Github user cenyuhai commented on the issue: https://github.com/apache/carbondata/pull/1363 retest please ---
[GitHub] carbondata issue #1363: [CARBONDATA-1378] support creating table in hive
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1363 Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/77/ ---
[GitHub] carbondata issue #1363: [CARBONDATA-1378] support creating table in hive
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1363 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/200/ ---
[jira] [Assigned] (CARBONDATA-1378) Support create carbon table in Hive
[ https://issues.apache.org/jira/browse/CARBONDATA-1378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] cen yuhai reassigned CARBONDATA-1378: - Assignee: cen yuhai > Support create carbon table in Hive > --- > > Key: CARBONDATA-1378 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1378 > Project: CarbonData > Issue Type: Sub-task > Components: hive-integration >Reporter: cen yuhai >Assignee: cen yuhai > Time Spent: 40m > Remaining Estimate: 0h > > Support create carbon table in Hive -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] carbondata issue #1363: [CARBONDATA-1378] support creating table in hive
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1363 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/831/ ---
[GitHub] carbondata issue #1363: [CARBONDATA-1378] support creating table in hive
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1363 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/199/ ---
[GitHub] carbondata issue #1363: [CARBONDATA-1378] support creating table in hive
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1363 Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/76/ ---
[GitHub] carbondata pull request #1363: [CARBONDATA-1378] support creating table in h...
GitHub user cenyuhai opened a pull request: https://github.com/apache/carbondata/pull/1363 [CARBONDATA-1378] support creating table in hive support creating table in hive You can merge this pull request into a Git repository by running: $ git pull https://github.com/cenyuhai/incubator-carbondata CARBONDATA-1378 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/1363.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1363 commit d4196f823edc4a732a1d0081d2e71f8b87041d41 Author: CenYuhai Date: 2017-09-17T10:28:30Z support creating table in hive ---
[GitHub] carbondata issue #1361: [CARBONDATA-1481]Compaction support global sort
Github user jackylk commented on the issue: https://github.com/apache/carbondata/pull/1361 please rebase to master ---
[GitHub] carbondata pull request #1361: [CARBONDATA-1481]Compaction support global so...
Github user jackylk commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1361#discussion_r139307748 --- Diff: integration/spark-common-cluster-test/src/test/scala/org/apache/carbondata/cluster/sdv/generated/BatchSortLoad3TestCase.scala --- @@ -112,33 +112,15 @@ class BatchSortLoad3TestCase extends QueryTest with BeforeAndAfterAll { sql(s"""drop table if exists t3""").collect } - - //Batch_sort_Loading_001-01-01-01_001-TC_056 - test("Batch_sort_Loading_001-01-01-01_001-TC_056", Include) { --- End diff -- why removing it? ---
[jira] [Commented] (CARBONDATA-1444) CarbonData unsupport Boolean data type,
[ https://issues.apache.org/jira/browse/CARBONDATA-1444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16169264#comment-16169264 ] Jacky Li commented on CARBONDATA-1444: -- Can you explain what kind of encoding will be added to support boolean data type? > CarbonData unsupport Boolean data type, > --- > > Key: CARBONDATA-1444 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1444 > Project: CarbonData > Issue Type: Improvement > Components: sql >Affects Versions: 1.2.0 >Reporter: xubo245 >Assignee: xubo245 >Priority: Minor > Fix For: 1.2.0 > > Original Estimate: 240h > Time Spent: 2h > Remaining Estimate: 238h > > Spark/Hive table support Boolean data type, the internal table also should > support Boolean data type. >Boolean data type Range: TRUE or FALSE. Do not use quotation marks around > the TRUE and FALSE literal values. You can write the literal values in > uppercase, lowercase, or mixed case. The values queried from a table are > always returned in lowercase, true or false. > Boolean data type column can be added to dimension and use single-byte to > store in carbon file. > CarbonData should support boolean data type in following aspects: > 1. create table >2.insert into table values and select >3.load data >4.filter -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] carbondata pull request #1362: [CARBONDATA-1444]Support boolean
Github user jackylk commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1362#discussion_r139307667 --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/bool/BooleanEncoderMeta.java --- @@ -0,0 +1,40 @@ +package org.apache.carbondata.core.datastore.page.encoding.bool; + +import java.io.DataInput; +import java.io.DataOutput; +import java.io.IOException; + +import org.apache.carbondata.core.datastore.TableSpec; +import org.apache.carbondata.core.datastore.page.encoding.ColumnPageEncoderMeta; +import org.apache.carbondata.core.datastore.page.statistics.SimpleStatsResult; +import org.apache.carbondata.core.metadata.datatype.DataType; +import org.apache.carbondata.core.metadata.schema.table.Writable; + +public class BooleanEncoderMeta extends ColumnPageEncoderMeta implements Writable { + private String compressorName; + + public BooleanEncoderMeta() { + } + + public BooleanEncoderMeta(TableSpec.ColumnSpec columnSpec, DataType storeDataType, +SimpleStatsResult stats, String compressorName) { +super(columnSpec,storeDataType,stats,compressorName); +this.compressorName = compressorName; --- End diff -- compressorName is already stored in ColumnPageEncoderMeta, this BooleanEncoderMeta is not required. ---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444]Support boolean
Github user jackylk commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1362#discussion_r139307650 --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/bool/BooleanConvert.java --- @@ -0,0 +1,61 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.carbondata.core.datastore.page.encoding.bool; + +/** + * convert tools for boolean data type + */ +public class BooleanConvert { --- End diff -- I think this is not required, it is simple conversion only. ---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444]Support boolean
Github user jackylk commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1362#discussion_r139307631 --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/EncodingFactory.java --- @@ -92,6 +90,10 @@ public ColumnPageDecoder createDecoder(List encodings, List
[GitHub] carbondata pull request #1362: [CARBONDATA-1444]Support boolean
Github user jackylk commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1362#discussion_r139307603 --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/EncodingFactory.java --- @@ -92,6 +90,10 @@ public ColumnPageDecoder createDecoder(List encodings, List
[GitHub] carbondata pull request #1362: [CARBONDATA-1444]Support boolean
Github user jackylk commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1362#discussion_r139307286 --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/ColumnPage.java --- @@ -271,6 +267,18 @@ private static ColumnPage newIntPage(TableSpec.ColumnSpec columnSpec, int[] intD return columnPage; } + private static ColumnPage newBooleanPage(TableSpec.ColumnSpec columnSpec, byte[] booleanData) { +ColumnPage columnPage = createPage(columnSpec, BOOLEAN, booleanData.length); --- End diff -- To make it simple, I think it is better to store boolean as byte in ColumnPage, so you can create a ColumnPage of Byte type. When come to encoding, you still can do proper encoding of this column page. ---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444]Support boolean
Github user jackylk commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1362#discussion_r139307244 --- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/ColumnPage.java --- @@ -187,6 +179,7 @@ public static ColumnPage newPage(TableSpec.ColumnSpec columnSpec, DataType dataT case BYTE: case SHORT: case SHORT_INT: +case BOOLEAN: --- End diff -- Move it before BYTE. Please modify the same in all switch case ---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444]Support boolean
Github user jackylk commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1362#discussion_r139307234 --- Diff: core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java --- @@ -23,6 +23,10 @@ public final class CarbonCommonConstants { /** + * Boolean size in bytes + */ + public static final int BOOLEAN_SIZE_IN_BYTE = 1; --- End diff -- This is not required, it should be added in DataType enum ---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444]Support boolean
Github user jackylk commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1362#discussion_r139307225 --- Diff: core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java --- @@ -707,6 +711,10 @@ */ public static final char BIG_INT_MEASURE = 'd'; /** + * BOOLEAN_VALUE_MEASURE + */ + public static final char BOOLEAN_MEASURE = 'a'; --- End diff -- This is not required. Others are for backward compatible only. ---
[GitHub] carbondata pull request #1336: [CARBONDATA-1425] Inappropriate Exception dis...
Github user sraghunandan commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1336#discussion_r139307125 --- Diff: integration/spark2/src/test/scala/org/apache/carbondata/spark/testsuite/partition/TestAlterPartitionTable.scala --- @@ -466,6 +467,111 @@ class TestAlterPartitionTable extends QueryTest with BeforeAndAfterAll { checkAnswer(result_after5, result_origin5) } + test("test exception when alter partition and the values" --- End diff -- That is also wrong.you can rework both and split into multiple test cases in another pr ---
[GitHub] carbondata issue #1362: [CARBONDATA-1444]Support boolean
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1362 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/198/ ---
[GitHub] carbondata issue #1362: [CARBONDATA-1444]Support boolean
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1362 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/830/ ---
[GitHub] carbondata issue #1362: [CARBONDATA-1444]Support boolean
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1362 Build Failed with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/75/ ---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444]Support boolean
GitHub user xubo245 opened a pull request: https://github.com/apache/carbondata/pull/1362 [CARBONDATA-1444]Support boolean CarbonData should support boolean data type in following aspects: 1. create table 2.insert into table values and select 3.load data 4.filter You can merge this pull request into a Git repository by running: $ git pull https://github.com/xubo245/carbondata supportBoolean Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/1362.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1362 commit 581288d82c53247e4f68e37d638bfe7cc7fb7ded Author: xubo245 <601450...@qq.com> Date: 2017-09-15T15:42:44Z support Boolean data type commit e7d279a41f6e58b6640f828d9fcb1c9962097cb8 Author: xubo245 <601450...@qq.com> Date: 2017-09-17T07:57:40Z fix findbugs commit 6e8f5da599038649b22a1866425941f706650135 Author: xubo245 <601450...@qq.com> Date: 2017-09-17T10:01:06Z add test case ---
[jira] [Resolved] (CARBONDATA-1438) Unify the sort column and sort scope in create table command
[ https://issues.apache.org/jira/browse/CARBONDATA-1438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacky Li resolved CARBONDATA-1438. -- Resolution: Fixed Fix Version/s: 1.2.0 > Unify the sort column and sort scope in create table command > > > Key: CARBONDATA-1438 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1438 > Project: CarbonData > Issue Type: Improvement >Reporter: chenerlu > Fix For: 1.2.0 > > Time Spent: 14h 40m > Remaining Estimate: 0h > > 1 Requirement > Currently, Users can specify sort column in table properties when create > table. And when load data, users can also specify sort scope in load options. > In order to improve the ease of use for users, it will be better to specify > the sort related parameters all in create table command. > Once sort scope is specified in create table command, it will be used in load > data even users have specified in load options. > 2 Detailed design > 2.1 Task-01 > Requirement: Create table can support specify sort scope > Implement: Take use of table properties (Map), will specify > sort scope in table properties by key/value pair, then existing interface > will be called to write this key/value pair into metastore. > Will support Global Sort,Local Sort and No Sort,it can be specified in sql > command: > CREATE TABLE tableWithGlobalSort ( > shortField SHORT, > intField INT, > bigintField LONG, > doubleField DOUBLE, > stringField STRING, > timestampField TIMESTAMP, > decimalField DECIMAL(18,2), > dateField DATE, > charField CHAR(5) > ) > STORED BY 'carbondata' > TBLPROPERTIES('SORT_COLUMNS'='stringField', 'SORT_SCOPE'='GLOBAL_SORT') > > Tips:If the sort scope is global Sort, users should specify > GLOBAL_SORT_PARTITIONS. If users do not specify it, it will use the number of > map task. GLOBAL_SORT_PARTITIONS should be Integer type, the range is > [1,Integer.MaxValue],it is only used when the sort scope is global sort. > Global Sort Use orderby operator in spark, data is ordered in segment level. > Local SortNode ordered, carbondata file is ordered if it is written by > one task. > No Sort No sort > Tips:key and value is case-insensitive. > 2.2 Task-02 > Requirement: > Load data in will support local sort, no sort, global sort > Ignore the sort scope specified in load data and use the parameter which > specified in create table. > Currently, user can specify the sort scope and global sort partitions in load > options, After modification, it will ignore the sort scope which specified in > load options and will get sort scope from table properties. > Current logic: sort scope is from load options > NumberPrerequisiteSort scope > 1 isSortTable is true && Sort Scope is Global SortGlobal > Sort(first check) > 2 isSortTable is falseNo Sort > 3 isSortTable is true Local Sort > Tips: isSortTable is true means this table contains sort column or it > contains dimensions (except complex type), like string type. > For example: > Create table xxx1 (col1 string col2 int) stored by ‘carbondata’ --- sort table > Create table xx1 (col1 int, col2 int) stored by ‘carbondata’ --- not sort > table > Create table xx (col1 int, col2 string) stored by ‘carbondata’ tblproperties > (‘sort_column’=’col1’) –- sort table > New logic:sort scope is from create table > NumberPrerequisiteCode branch > 1 isSortTable = true && Sort Scope is Global Sort Global Sort(first check) > 2 isSortTable= false || Sort Scope is No Sort No Sort > 3 isSortTable is true && Sort Scope is Local Sort Local Sort > 4 isSortTable is true,without specify Sort Scope Local Sort, (Keep > current logic) > 3 Acceptance standard > NumberAcceptance standard > 1 Use can specify sort scope(global, local, no sort) when create carbon > table in sql type > 2 Load data will ignore the sort scope specified in load options and will > use the parameter which specify in create table command. If user still > specify the sort scope in load options, will give warning and inform user > that he will use the sort scope which specified in create table. > 4 Feature restrictions > NA > 5 Dependencies > NA > 6 Technical risk > NA -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] carbondata pull request #1321: [CARBONDATA-1438] Unify the sort column and s...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/1321 ---
[GitHub] carbondata issue #1321: [CARBONDATA-1438] Unify the sort column and sort sco...
Github user jackylk commented on the issue: https://github.com/apache/carbondata/pull/1321 LGTM ---
[GitHub] carbondata issue #1321: [CARBONDATA-1438] Unify the sort column and sort sco...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1321 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/829/ ---
[GitHub] carbondata issue #1321: [CARBONDATA-1438] Unify the sort column and sort sco...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1321 Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/74/ ---
[GitHub] carbondata issue #1321: [CARBONDATA-1438] Unify the sort column and sort sco...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1321 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/197/ ---
[GitHub] carbondata issue #1321: [CARBONDATA-1438] Unify the sort column and sort sco...
Github user chenerlu commented on the issue: https://github.com/apache/carbondata/pull/1321 retest this please ---
[GitHub] carbondata issue #1321: [CARBONDATA-1438] Unify the sort column and sort sco...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1321 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/828/ ---
[GitHub] carbondata issue #1321: [CARBONDATA-1438] Unify the sort column and sort sco...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1321 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/196/ ---
[GitHub] carbondata issue #1321: [CARBONDATA-1438] Unify the sort column and sort sco...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1321 Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/73/ ---