[GitHub] carbondata issue #1443: [CARBONDATA-1524][CARBONDATA-1525] Added support for...
Github user kunal642 commented on the issue: https://github.com/apache/carbondata/pull/1443 retest this please ---
[GitHub] carbondata issue #1439: [CARBONDATA-1628] Re-factory LoadTableCommand to reu...
Github user mohammadshahidkhan commented on the issue: https://github.com/apache/carbondata/pull/1439 +1 LGTM ---
[GitHub] carbondata issue #1439: [CARBONDATA-1628] Re-factory LoadTableCommand to reu...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1439 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1344/ ---
[GitHub] carbondata issue #1439: [CARBONDATA-1628] Re-factory LoadTableCommand to reu...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1439 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1343/ ---
[GitHub] carbondata issue #1439: [CARBONDATA-1628] Re-factory LoadTableCommand to reu...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1439 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/708/ ---
[GitHub] carbondata issue #1443: [CARBONDATA-1524][CARBONDATA-1525] Added support for...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1443 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1342/ ---
[GitHub] carbondata issue #1443: [CARBONDATA-1524][CARBONDATA-1525] Added support for...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1443 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/706/ ---
[GitHub] carbondata pull request #1443: [CARBONDATA-1524[CARBONDATA-1525] Added suppo...
GitHub user kunal642 opened a pull request: https://github.com/apache/carbondata/pull/1443 [CARBONDATA-1524[CARBONDATA-1525] Added support for aggregate table drop Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [X] Make sure the PR title is formatted like: `[CARBONDATA-] Description of pull request` - [X] Make sure to add PR description including - the root cause/problem statement - What is the implemented solution - [X] Any interfaces changed? Metastore interface is changed to support child table drop - [X] Any backward compatibility impacted? No - [X] Document update required? No - [X] Testing done Test cases are added for child and parent table drop scenarios. - [X] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/kunal642/carbondata drop_agg_support Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/1443.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1443 commit 4400f65d3ad588230a84ca67bb96c6675d60ef02 Author: kunal642 Date: 2017-10-18T14:39:04Z added support for aggregate table drop ---
[GitHub] carbondata issue #1429: [WIP] Add StructType and ArrayType class
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1429 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1341/ ---
[GitHub] carbondata issue #1442: [WIP] Add examples for Carbon usage when integrating...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1442 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1340/ ---
[GitHub] carbondata pull request #1428: [WIP] Drop agg support
Github user kunal642 closed the pull request at: https://github.com/apache/carbondata/pull/1428 ---
[GitHub] carbondata issue #1429: [WIP] Add child DataType to complex data type
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1429 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/705/ ---
[GitHub] carbondata issue #1417: [CARBONDATA-1594] Add precision and scale to Decimal...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1417 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1339/ ---
[GitHub] carbondata issue #1442: [WIP] Add examples for Carbon usage when integrating...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1442 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/704/ ---
[GitHub] carbondata issue #1417: [CARBONDATA-1594] Add precision and scale to Decimal...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1417 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/703/ ---
[GitHub] carbondata pull request #1442: [WIP] Add examples
GitHub user jackylk opened a pull request: https://github.com/apache/carbondata/pull/1442 [WIP] Add examples It is good to have more examples for user reference. This PR adds back examples from spark-example module in earlier spark 1 integration You can merge this pull request into a Git repository by running: $ git pull https://github.com/jackylk/incubator-carbondata example Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/1442.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1442 ---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] Support Database Location Configur...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1418 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1338/ ---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] Support Database Location Configur...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1418 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/702/ ---
[GitHub] carbondata pull request #1435: [CARBONDATA-1626]add data size and index size...
Github user mohammadshahidkhan commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1435#discussion_r147471059 --- Diff: core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java --- @@ -1376,6 +1376,32 @@ public static final String BITSET_PIPE_LINE_DEFAULT = "true"; + /** + * The total size of carbon data + */ + public static final String CARBON_TOTAL_DATA_SIZE = "datasize"; + + /** + * The total size of carbon index + */ + public static final String CARBON_TOTAL_INDEX_SIZE = "indexsize"; + + /** + * ENABLE_CALCULATE_DATA_INDEX_SIZE + */ + @CarbonProperty public static final String ENABLE_CALCULATE_SIZE = "carbon.enable.calculate.size"; + + /** + * DEFAULT_ENABLE_CALCULATE_DATA_INDEX_SIZE + */ + @CarbonProperty public static final String DEFAULT_ENABLE_CALCULATE_SIZE = "true"; --- End diff -- for constant variable/keys CarbonProperty anotation not required ---
[GitHub] carbondata issue #1418: [CARBONDATA-1573] Support Database Location Configur...
Github user mohammadshahidkhan commented on the issue: https://github.com/apache/carbondata/pull/1418 retest this please ---
[jira] [Resolved] (CARBONDATA-1517) 1. Support CTAS in carbon and support creating aggregation tables using CTAS.And update aggregation table information to main table schema.
[ https://issues.apache.org/jira/browse/CARBONDATA-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindra Pesala resolved CARBONDATA-1517. - Resolution: Fixed Assignee: kumar vishal Fix Version/s: 1.3.0 > 1. Support CTAS in carbon and support creating aggregation tables using > CTAS.And update aggregation table information to main table schema. > --- > > Key: CARBONDATA-1517 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1517 > Project: CarbonData > Issue Type: Sub-task >Reporter: Ravindra Pesala >Assignee: kumar vishal > Fix For: 1.3.0 > > Time Spent: 3h 10m > Remaining Estimate: 0h > > User should add aggregation tables with following syntax > {code} > CREATE TABLE agg_sales > STORED BY 'carbondata' > TBLPROPERTIES ('parent_table'='sales') > AS SELECT user_id, sum(quantity), avg(price) FROM sales GROUP BY user_id > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] carbondata issue #1433: [CARBONDATA-1517]- Pre Aggregate Create Table Suppor...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1433 LGTM ---
[GitHub] carbondata pull request #1439: [CARBONDATA-1628] Re-factory LoadTableCommand...
Github user kumarvishal09 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1439#discussion_r147454328 --- Diff: integration/spark-common/src/main/scala/org/apache/carbondata/spark/util/CommonUtil.scala --- @@ -831,4 +835,207 @@ object CommonUtil { LOGGER.error(s) } } + + def getFinalOptions(carbonProperty: CarbonProperties, --- End diff -- Please change the method name ---
[GitHub] carbondata issue #1437: [CARBONDATA-1618] Fix issue of not support table com...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1437 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1337/ ---
[GitHub] carbondata issue #1417: [CARBONDATA-1594] Add precision and scale to Decimal...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1417 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1336/ ---
[jira] [Updated] (CARBONDATA-1651) Unsupported Spark2 BooleanType
[ https://issues.apache.org/jira/browse/CARBONDATA-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Timrov updated CARBONDATA-1651: - Description: Unable to save Dataset if it contains field with BooleanType class CarbonDataFrameWriter method convertToCarbonType doesn't support it > Unsupported Spark2 BooleanType > -- > > Key: CARBONDATA-1651 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1651 > Project: CarbonData > Issue Type: Bug > Components: spark-integration >Affects Versions: 1.3.0 >Reporter: Roman Timrov > > Unable to save Dataset if it contains field with BooleanType > class CarbonDataFrameWriter > method convertToCarbonType doesn't support it -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] carbondata issue #1437: [CARBONDATA-1618] Fix issue of not support table com...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1437 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/701/ ---
[jira] [Created] (CARBONDATA-1651) Unsupported Spark2 BooleanType
Roman Timrov created CARBONDATA-1651: Summary: Unsupported Spark2 BooleanType Key: CARBONDATA-1651 URL: https://issues.apache.org/jira/browse/CARBONDATA-1651 Project: CarbonData Issue Type: Bug Components: spark-integration Affects Versions: 1.3.0 Reporter: Roman Timrov -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] carbondata pull request #1437: [CARBONDATA-1618] Fix issue of not support ta...
Github user chenerlu commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1437#discussion_r147437929 --- Diff: integration/spark-common/src/main/scala/org/apache/spark/sql/catalyst/CarbonDDLSqlParser.scala --- @@ -247,7 +247,8 @@ abstract class CarbonDDLSqlParser extends AbstractCarbonSparkSQLParser { , tableName: String, fields: Seq[Field], partitionCols: Seq[PartitionerField], tableProperties: mutable.Map[String, String], - bucketFields: Option[BucketFields], isAlterFlow: Boolean = false): TableModel = { + bucketFields: Option[BucketFields], isAlterFlow: Boolean = false, + comment: Option[String] = None): TableModel = { --- End diff -- Have renamed. ---
[GitHub] carbondata issue #1429: [WIP] Add child DataType to complex data type
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1429 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1335/ ---
[GitHub] carbondata pull request #1437: [CARBONDATA-1618] Fix issue of not support ta...
Github user chenerlu commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1437#discussion_r147436112 --- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/createTable/TestCreateTableWithTableComment.scala --- @@ -0,0 +1,74 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.carbondata.spark.testsuite.createTable + +import org.apache.spark.sql.test.util.QueryTest +import org.scalatest.BeforeAndAfterAll + +/** + * test functionality for create table with table comment + */ +class TestCreateTableWithTableComment extends QueryTest with BeforeAndAfterAll { + + override def beforeAll { +sql("use default") +sql("drop table if exists withTableComment") +sql("drop table if exists withoutTableComment") + } + + test("test create table with table comment") { +sql( + s""" + | create table withTableComment( + | id int, + | name string + | ) + | comment "This table has table comment" + | STORED BY 'carbondata' + """.stripMargin +) + +val result = sql("describe formatted withTableComment") + +checkExistence(result, true, "Comment:") +checkExistence(result, true, "This table has table comment") + } + + test("test create table without table comment") { +sql( + s""" + | create table withoutTableComment( + | id int, + | name string + | ) + | STORED BY 'carbondata' + """.stripMargin +) + --- End diff -- This PR not contains this functions. ---
[GitHub] carbondata pull request #1437: [CARBONDATA-1618] Fix issue of not support ta...
Github user chenerlu commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1437#discussion_r147435775 --- Diff: integration/spark-common/src/main/scala/org/apache/spark/sql/catalyst/CarbonDDLSqlParser.scala --- @@ -247,7 +247,8 @@ abstract class CarbonDDLSqlParser extends AbstractCarbonSparkSQLParser { , tableName: String, fields: Seq[Field], partitionCols: Seq[PartitionerField], tableProperties: mutable.Map[String, String], - bucketFields: Option[BucketFields], isAlterFlow: Boolean = false): TableModel = { + bucketFields: Option[BucketFields], isAlterFlow: Boolean = false, + comment: Option[String] = None): TableModel = { --- End diff -- Carbon already support column comment. ---
[GitHub] carbondata issue #1417: [CARBONDATA-1594] Add precision and scale to Decimal...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1417 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/700/ ---
[GitHub] carbondata issue #1429: [WIP] Add child DataType to complex data type
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1429 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1334/ ---
[GitHub] carbondata issue #1417: [CARBONDATA-1594] Add precision and scale to Decimal...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1417 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1333/ ---
[GitHub] carbondata issue #1439: [CARBONDATA-1628] Re-factory LoadTableCommand to reu...
Github user sraghunandan commented on the issue: https://github.com/apache/carbondata/pull/1439 @mohammadshahidkhan @kumarvishal09 please help to review this PR ---
[GitHub] carbondata issue #1433: [CARBONDATA-1517]- Pre Aggregate Create Table Suppor...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1433 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1332/ ---
[GitHub] carbondata issue #1439: [CARBONDATA-1628] Re-factory LoadTableCommand to reu...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1439 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1331/ ---
[GitHub] carbondata issue #1429: [WIP] Add child DataType to complex data type
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1429 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/699/ ---
[GitHub] carbondata issue #1417: [CARBONDATA-1594] Add precision and scale to Decimal...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1417 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/698/ ---
[GitHub] carbondata issue #1433: [CARBONDATA-1517]- Pre Aggregate Create Table Suppor...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1433 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/697/ ---
[GitHub] carbondata issue #1441: [CARBONDATA-1489] Added testcases Insert and Delete ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1441 Can one of the admins verify this patch? ---
[GitHub] carbondata issue #1441: [CARBONDATA-1489] Added testcases Insert and Delete ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1441 Can one of the admins verify this patch? ---
[GitHub] carbondata pull request #1441: [CARBONDATA-1489] Added testcases Insert and ...
GitHub user ashwini-krishnakumar opened a pull request: https://github.com/apache/carbondata/pull/1441 [CARBONDATA-1489] Added testcases Insert and Delete testcases for Struct and Array data type Added testcases Insert and Delete testcases for Struct and Array data type Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like: `[CARBONDATA-] Description of pull request` - [ ] Make sure to add PR description including - the root cause/problem statement - What is the implemented solution - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/ashwini-krishnakumar/carbondata CARBONDATA-1489 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/1441.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1441 ---
[GitHub] carbondata issue #1439: [CARBONDATA-1628] Re-factory LoadTableCommand to reu...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1439 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1330/ ---
[GitHub] carbondata issue #1440: [WIP][CARBONDATA-1581][CARBONDATA-1582] Implement St...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1440 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1329/ ---
[GitHub] carbondata pull request #1430: [CARBONDATA-1444] Update documentation about ...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/1430 ---
[GitHub] carbondata issue #1430: [CARBONDATA-1444] Update documentation about Boolean...
Github user chenliang613 commented on the issue: https://github.com/apache/carbondata/pull/1430 LGTM ---
[GitHub] carbondata issue #1426: DOCUMENTATION for SORT_SCOPE
Github user chenliang613 commented on the issue: https://github.com/apache/carbondata/pull/1426 @sgururajshetty please add scenarios info for SORT_SCOPE ---
[jira] [Commented] (CARBONDATA-1624) If SORT_SCOPE is non-GLOBAL_SORT with Spark, set 'carbon.number.of.cores.while.loading' dynamically as per the available executor cores
[ https://issues.apache.org/jira/browse/CARBONDATA-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16222176#comment-16222176 ] Ravindra Pesala commented on CARBONDATA-1624: - Ok, I got your point. You mean just get the configured spark.executor.cores from spark while initializing session and set to carbon properties. I was thinking little more about it, get the currently available cores from spark for every load and block those cores to control the parallel loads. Any way we can do your way first later we can extend to get cores for each load. > If SORT_SCOPE is non-GLOBAL_SORT with Spark, set > 'carbon.number.of.cores.while.loading' dynamically as per the available > executor cores > > > Key: CARBONDATA-1624 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1624 > Project: CarbonData > Issue Type: Improvement > Components: data-load, spark-integration >Affects Versions: 1.3.0 >Reporter: Zhichao Zhang >Assignee: Zhichao Zhang >Priority: Minor > > If we are using carbondata + spark to load data, we can set > carbon.number.of.cores.while.loading to the number of executor cores. > For example, when set the number of executor cores to 6, it shows that there > are at > least 6 cores per node for loading data, so we can set > carbon.number.of.cores.while.loading to 6 automatically. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] carbondata issue #1439: [CARBONDATA-1628] Re-factory LoadTableCommand to reu...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1439 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1328/ ---
[GitHub] carbondata pull request #1437: [CARBONDATA-1618] Fix issue of not support ta...
Github user sounakr commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1437#discussion_r147354669 --- Diff: integration/spark-common/src/main/scala/org/apache/spark/sql/catalyst/CarbonDDLSqlParser.scala --- @@ -247,7 +247,8 @@ abstract class CarbonDDLSqlParser extends AbstractCarbonSparkSQLParser { , tableName: String, fields: Seq[Field], partitionCols: Seq[PartitionerField], tableProperties: mutable.Map[String, String], - bucketFields: Option[BucketFields], isAlterFlow: Boolean = false): TableModel = { + bucketFields: Option[BucketFields], isAlterFlow: Boolean = false, + comment: Option[String] = None): TableModel = { --- End diff -- Better to rename it to tableComment in order to avoid confusion with Column Comment which will come in future. ---
[GitHub] carbondata pull request #1437: [CARBONDATA-1618] Fix issue of not support ta...
Github user sounakr commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1437#discussion_r147350873 --- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/createTable/TestCreateTableWithTableComment.scala --- @@ -0,0 +1,74 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.carbondata.spark.testsuite.createTable + +import org.apache.spark.sql.test.util.QueryTest +import org.scalatest.BeforeAndAfterAll + +/** + * test functionality for create table with table comment + */ +class TestCreateTableWithTableComment extends QueryTest with BeforeAndAfterAll { + + override def beforeAll { +sql("use default") +sql("drop table if exists withTableComment") +sql("drop table if exists withoutTableComment") + } + + test("test create table with table comment") { +sql( + s""" + | create table withTableComment( + | id int, + | name string + | ) + | comment "This table has table comment" + | STORED BY 'carbondata' + """.stripMargin +) + +val result = sql("describe formatted withTableComment") + +checkExistence(result, true, "Comment:") +checkExistence(result, true, "This table has table comment") + } + + test("test create table without table comment") { +sql( + s""" + | create table withoutTableComment( + | id int, + | name string + | ) + | STORED BY 'carbondata' + """.stripMargin +) + --- End diff -- Can existing table will be able to add comment through Alter table command? ---
[GitHub] carbondata pull request #1437: [CARBONDATA-1618] Fix issue of not support ta...
Github user sounakr commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1437#discussion_r147354409 --- Diff: integration/spark-common/src/main/scala/org/apache/spark/sql/catalyst/CarbonDDLSqlParser.scala --- @@ -287,7 +288,8 @@ abstract class CarbonDDLSqlParser extends AbstractCarbonSparkSQLParser { groupCols, Some(colProps), bucketFields: Option[BucketFields], - partitionInfo) + partitionInfo, + comment) --- End diff -- Please check if the all the callers for TableModel case class is covered the "comment" parameter or have handled it. For e.g. in def createTableInfoFromParam. ---
[GitHub] carbondata pull request #1434: [CARBONDATA-1593]Add partition to table cause...
Github user sounakr commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1434#discussion_r147375453 --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/partition/AlterTableSplitCarbonPartitionCommand.scala --- @@ -109,7 +109,7 @@ case class AlterTableSplitCarbonPartitionCommand( CarbonUtil.writeThriftTableToSchemaFile(schemaFilePath, thriftTable) // update the schema modified time carbonMetaStore.updateAndTouchSchemasUpdatedTime(storePath) -sparkSession.catalog.refreshTable(tableName) +sparkSession.catalog.refreshTable(s"$dbName.$tableName") --- End diff -- Better to call SessionCatalog.refreshTable rather than Catalog refreshTable as it will honour the DBName. sparkSession.sessionState.catalog.refreshTable() and pass TableIdentifier. ---
[jira] [Closed] (CARBONDATA-1630) load data into hive table fail
[ https://issues.apache.org/jira/browse/CARBONDATA-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xujie closed CARBONDATA-1630. - Resolution: Fixed > load data into hive table fail > -- > > Key: CARBONDATA-1630 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1630 > Project: CarbonData > Issue Type: Bug > Components: hive-integration >Affects Versions: 1.2.0 > Environment: hive.version:1.1.0-cdh5.10.0 > hadoop:version:2.6.0-cdh5.10.0 >Reporter: xujie >Priority: Critical > > import org.apache.spark.sql.SparkSession > import org.apache.spark.sql.CarbonSession._ > val rootPath = "hdfs://namenodeb:8020/app/carbondata" > val storeLocation = s"$rootPath/store" > val warehouse = s"$rootPath/warehouse" > val metastoredb = s"$rootPath/metastore_db" > val carbon = > SparkSession.builder().enableHiveSupport().config("spark.sql.warehouse.dir", > warehouse).config(org.apache.carbondata.core.constants.CarbonCommonConstants.STORE_LOCATION, > storeLocation).getOrCreateCarbonSession(storeLocation, metastoredb) > import org.apache.spark.sql.types._ > import org.apache.spark.sql.Row > val rdd = sc.textFile("/data/home/hadoop/test.txt"); > val schemaString = "id name city" > val fields = schemaString.split(" ").map(fieldName => StructField(fieldName, > StringType, nullable = true)) > val schema = StructType(fields) > val rowRDD = rdd.map(_.split(",")).map(attributes => > Row(attributes(0),attributes(1),attributes(2))) > val peopleDF = spark.createDataFrame(rowRDD, schema) > peopleDF.createOrReplaceTempView("tmp_table") > spark.sql("insert into target_table SELECT * FROM tmp_table") > java.lang.RuntimeException: Failed to add entry in table status for > default.target_table > at scala.sys.package$.error(package.scala:27) > at > org.apache.carbondata.spark.util.CommonUtil$.readAndUpdateLoadProgressInTableMeta(CommonUtil.scala:533) > at > org.apache.spark.sql.execution.command.LoadTable.processData(carbonTableSchema.scala:928) > at > org.apache.spark.sql.execution.command.LoadTable.run(carbonTableSchema.scala:754) > at > org.apache.spark.sql.execution.command.LoadTableByInsert.processData(carbonTableSchema.scala:651) > at > org.apache.spark.sql.execution.command.LoadTableByInsert.run(carbonTableSchema.scala:637) > at > org.apache.spark.sql.CarbonDatasourceHadoopRelation.insert(CarbonDatasourceHadoopRelation.scala:98) > at > org.apache.spark.sql.execution.datasources.InsertIntoDataSourceCommand.run(InsertIntoDataSourceCommand.scala:43) > at > org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58) > at > org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56) > at > org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74) > at > org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114) > at > org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114) > at > org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:135) > at > org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) > at > org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:132) > at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:113) > at > org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:92) > at > org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:92) > at org.apache.spark.sql.Dataset.(Dataset.scala:185) > at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:64) > at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:592) > ... 52 elided -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] carbondata issue #1433: [CARBONDATA-1517]- Pre Aggregate Create Table Suppor...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1433 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1327/ ---
[jira] [Commented] (CARBONDATA-1650) load data into hive table fail
[ https://issues.apache.org/jira/browse/CARBONDATA-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16222115#comment-16222115 ] xujie commented on CARBONDATA-1650: --- scala> spark.sql("desc formatted target_table").show +++---+ |col_name| data_type|comment| +++---+ | id| int| null| |name| string| null| |city| string| null| ||| | |# Detailed Table ...|| | | Database:| default| | | Owner:| hadoop| | |Create Time:|Wed Oct 25 14:20:...| | | Last Access Time:|Thu Jan 01 08:00:...| | | Location:|hdfs://namenodeb:...| | | Table Type:| MANAGED| | | Table Parameters:|| | | rawDataSize| -1| | |numFiles| 0| | | transient_lastD...| 1509091869| | | last_modified_time| 1509088050| | |last_modified_by| hadoop| | | totalSize| 0| | | COLUMN_STATS_AC...| false| | | numRows| -1| | > load data into hive table fail > -- > > Key: CARBONDATA-1650 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1650 > Project: CarbonData > Issue Type: Bug > Components: hive-integration >Affects Versions: 1.2.0 > Environment: hive.version:1.1.0-cdh5.10.0 > hadoop:version:2.6.0-cdh5.10.0 >Reporter: xujie >Priority: Critical > > import org.apache.spark.sql.SparkSession > import org.apache.spark.sql.CarbonSession._ > val rootPath = "hdfs://namenodeb:8020/app/carbondata" > val storeLocation = s"$rootPath/store" > val warehouse = s"$rootPath/warehouse" > val metastoredb = s"$rootPath/metastore_db" > val carbon = > SparkSession.builder().enableHiveSupport().config("spark.sql.warehouse.dir", > warehouse).config(org.apache.carbondata.core.constants.CarbonCommonConstants.STORE_LOCATION, > storeLocation).getOrCreateCarbonSession(storeLocation, metastoredb) > import org.apache.spark.sql.types._ > import org.apache.spark.sql.Row > val rdd = sc.textFile("/data/home/hadoop/test.txt"); > val schemaString = "id name city" > val fields = schemaString.split(" ").map(fieldName => StructField(fieldName, > StringType, nullable = true)) > val schema = StructType(fields) > val rowRDD = rdd.map(_.split(",")).map(attributes => > Row(attributes(0),attributes(1),attributes(2))) > val peopleDF = spark.createDataFrame(rowRDD, schema) > peopleDF.createOrReplaceTempView("tmp_table") > spark.sql("insert into target_table SELECT * FROM tmp_table") > java.lang.RuntimeException: Failed to add entry in table status for > default.target_table > at scala.sys.package$.error(package.scala:27) > at > org.apache.carbondata.spark.util.CommonUtil$.readAndUpdateLoadProgressInTableMeta(CommonUtil.scala:533) > at > org.apache.spark.sql.execution.command.LoadTable.processData(carbonTableSchema.scala:928) > at > org.apache.spark.sql.execution.command.LoadTable.run(carbonTableSchema.scala:754) > at > org.apache.spark.sql.execution.command.LoadTableByInsert.processData(carbonTableSchema.scala:651) > at > org.apache.spark.sql.execution.command.LoadTableByInsert.run(carbonTableSchema.scala:637) > at > org.apache.spark.sql.CarbonDatasourceHadoopRelation.insert(CarbonDatasourceHadoopRelation.scala:98) > at > org.apache.spark.sql.execution.datasources.InsertIntoDataSourceCommand.run(InsertIntoDataSourceCommand.scala:43) > at > org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58) > at > org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56) > at > org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74) > at > org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114) > at > org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114) > at > org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:135) > at > org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) > at > org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:132) > at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:113) > at > org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:92) > at > org.apache.spark.sql.execution.QueryExecution.toRdd
[jira] [Created] (CARBONDATA-1650) load data into hive table fail
xujie created CARBONDATA-1650: - Summary: load data into hive table fail Key: CARBONDATA-1650 URL: https://issues.apache.org/jira/browse/CARBONDATA-1650 Project: CarbonData Issue Type: Bug Components: hive-integration Affects Versions: 1.2.0 Environment: hive.version:1.1.0-cdh5.10.0 hadoop:version:2.6.0-cdh5.10.0 Reporter: xujie Priority: Critical import org.apache.spark.sql.SparkSession import org.apache.spark.sql.CarbonSession._ val rootPath = "hdfs://namenodeb:8020/app/carbondata" val storeLocation = s"$rootPath/store" val warehouse = s"$rootPath/warehouse" val metastoredb = s"$rootPath/metastore_db" val carbon = SparkSession.builder().enableHiveSupport().config("spark.sql.warehouse.dir", warehouse).config(org.apache.carbondata.core.constants.CarbonCommonConstants.STORE_LOCATION, storeLocation).getOrCreateCarbonSession(storeLocation, metastoredb) import org.apache.spark.sql.types._ import org.apache.spark.sql.Row val rdd = sc.textFile("/data/home/hadoop/test.txt"); val schemaString = "id name city" val fields = schemaString.split(" ").map(fieldName => StructField(fieldName, StringType, nullable = true)) val schema = StructType(fields) val rowRDD = rdd.map(_.split(",")).map(attributes => Row(attributes(0),attributes(1),attributes(2))) val peopleDF = spark.createDataFrame(rowRDD, schema) peopleDF.createOrReplaceTempView("tmp_table") spark.sql("insert into target_table SELECT * FROM tmp_table") java.lang.RuntimeException: Failed to add entry in table status for default.target_table at scala.sys.package$.error(package.scala:27) at org.apache.carbondata.spark.util.CommonUtil$.readAndUpdateLoadProgressInTableMeta(CommonUtil.scala:533) at org.apache.spark.sql.execution.command.LoadTable.processData(carbonTableSchema.scala:928) at org.apache.spark.sql.execution.command.LoadTable.run(carbonTableSchema.scala:754) at org.apache.spark.sql.execution.command.LoadTableByInsert.processData(carbonTableSchema.scala:651) at org.apache.spark.sql.execution.command.LoadTableByInsert.run(carbonTableSchema.scala:637) at org.apache.spark.sql.CarbonDatasourceHadoopRelation.insert(CarbonDatasourceHadoopRelation.scala:98) at org.apache.spark.sql.execution.datasources.InsertIntoDataSourceCommand.run(InsertIntoDataSourceCommand.scala:43) at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58) at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56) at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74) at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114) at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114) at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:135) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:132) at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:113) at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:92) at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:92) at org.apache.spark.sql.Dataset.(Dataset.scala:185) at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:64) at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:592) ... 52 elided -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] carbondata issue #1433: [CARBONDATA-1517]- Pre Aggregate Create Table Suppor...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1433 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1326/ ---
[GitHub] carbondata issue #1438: [WIP][CARBONDATA-1649]insert overwrite fix during jo...
Github user akashrn5 commented on the issue: https://github.com/apache/carbondata/pull/1438 @ravipesala @gvramana @QiangCai @manishgupta88 please review this PR ---
[GitHub] carbondata issue #1439: [CARBONDATA-1628] Re-factory LoadTableCommand to reu...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1439 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/696/ ---
[jira] [Created] (CARBONDATA-1649) delete segment and update fail status if insertoverwrite job is cancelled
Akash R Nilugal created CARBONDATA-1649: --- Summary: delete segment and update fail status if insertoverwrite job is cancelled Key: CARBONDATA-1649 URL: https://issues.apache.org/jira/browse/CARBONDATA-1649 Project: CarbonData Issue Type: Bug Reporter: Akash R Nilugal Assignee: Akash R Nilugal Priority: Minor when insert overwrite job is in progress and if that job is cancelled, then table status should be changed and segment should be deleted during the interruption -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1648) Change alter table drop partition to support two level partitions
Cao, Lionel created CARBONDATA-1648: --- Summary: Change alter table drop partition to support two level partitions Key: CARBONDATA-1648 URL: https://issues.apache.org/jira/browse/CARBONDATA-1648 Project: CarbonData Issue Type: Sub-task Reporter: Cao, Lionel -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1647) Change show partition to support two level partitions
Cao, Lionel created CARBONDATA-1647: --- Summary: Change show partition to support two level partitions Key: CARBONDATA-1647 URL: https://issues.apache.org/jira/browse/CARBONDATA-1647 Project: CarbonData Issue Type: Sub-task Reporter: Cao, Lionel -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] carbondata issue #1436: [WIP][CARBONDATA-1617] Merging carbonindex files wit...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1436 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1325/ ---
[GitHub] carbondata issue #1439: [CARBONDATA-1628] Re-factory LoadTableCommand to reu...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1439 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/695/ ---
[GitHub] carbondata issue #1440: [WIP][CARBONDATA-1581][CARBONDATA-1582] Implement St...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1440 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/694/ ---
[jira] [Created] (CARBONDATA-1646) Concurrent performance testing of partition tables
Cao, Lionel created CARBONDATA-1646: --- Summary: Concurrent performance testing of partition tables Key: CARBONDATA-1646 URL: https://issues.apache.org/jira/browse/CARBONDATA-1646 Project: CarbonData Issue Type: Sub-task Reporter: Cao, Lionel -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1645) Change alter table add/split partition to support two level partitions
Cao, Lionel created CARBONDATA-1645: --- Summary: Change alter table add/split partition to support two level partitions Key: CARBONDATA-1645 URL: https://issues.apache.org/jira/browse/CARBONDATA-1645 Project: CarbonData Issue Type: Sub-task Reporter: Cao, Lionel -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1644) Change query process to support two level partitions
Cao, Lionel created CARBONDATA-1644: --- Summary: Change query process to support two level partitions Key: CARBONDATA-1644 URL: https://issues.apache.org/jira/browse/CARBONDATA-1644 Project: CarbonData Issue Type: Sub-task Reporter: Cao, Lionel -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1643) Change load process to support two level partitions
Cao, Lionel created CARBONDATA-1643: --- Summary: Change load process to support two level partitions Key: CARBONDATA-1643 URL: https://issues.apache.org/jira/browse/CARBONDATA-1643 Project: CarbonData Issue Type: Sub-task Reporter: Cao, Lionel -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1642) Implement Hash-Hash partitioner
Cao, Lionel created CARBONDATA-1642: --- Summary: Implement Hash-Hash partitioner Key: CARBONDATA-1642 URL: https://issues.apache.org/jira/browse/CARBONDATA-1642 Project: CarbonData Issue Type: Sub-task Reporter: Cao, Lionel -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1641) Implement Hash-List partitioner
Cao, Lionel created CARBONDATA-1641: --- Summary: Implement Hash-List partitioner Key: CARBONDATA-1641 URL: https://issues.apache.org/jira/browse/CARBONDATA-1641 Project: CarbonData Issue Type: Sub-task Reporter: Cao, Lionel -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1640) Implement Hash-Range partitioner
Cao, Lionel created CARBONDATA-1640: --- Summary: Implement Hash-Range partitioner Key: CARBONDATA-1640 URL: https://issues.apache.org/jira/browse/CARBONDATA-1640 Project: CarbonData Issue Type: Sub-task Reporter: Cao, Lionel -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1639) Implement List-Hash partitioner
Cao, Lionel created CARBONDATA-1639: --- Summary: Implement List-Hash partitioner Key: CARBONDATA-1639 URL: https://issues.apache.org/jira/browse/CARBONDATA-1639 Project: CarbonData Issue Type: Sub-task Reporter: Cao, Lionel -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1638) Implement List-List partitioner
Cao, Lionel created CARBONDATA-1638: --- Summary: Implement List-List partitioner Key: CARBONDATA-1638 URL: https://issues.apache.org/jira/browse/CARBONDATA-1638 Project: CarbonData Issue Type: Sub-task Reporter: Cao, Lionel -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1637) Implement List-Range partitioner
Cao, Lionel created CARBONDATA-1637: --- Summary: Implement List-Range partitioner Key: CARBONDATA-1637 URL: https://issues.apache.org/jira/browse/CARBONDATA-1637 Project: CarbonData Issue Type: Sub-task Reporter: Cao, Lionel -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1636) Implement Range-Hash partitioner
Cao, Lionel created CARBONDATA-1636: --- Summary: Implement Range-Hash partitioner Key: CARBONDATA-1636 URL: https://issues.apache.org/jira/browse/CARBONDATA-1636 Project: CarbonData Issue Type: Sub-task Reporter: Cao, Lionel -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1635) Implement Range-List partitioner
Cao, Lionel created CARBONDATA-1635: --- Summary: Implement Range-List partitioner Key: CARBONDATA-1635 URL: https://issues.apache.org/jira/browse/CARBONDATA-1635 Project: CarbonData Issue Type: Sub-task Reporter: Cao, Lionel -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1634) Implement Range-Range partitioner
Cao, Lionel created CARBONDATA-1634: --- Summary: Implement Range-Range partitioner Key: CARBONDATA-1634 URL: https://issues.apache.org/jira/browse/CARBONDATA-1634 Project: CarbonData Issue Type: Sub-task Reporter: Cao, Lionel -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1633) Change parser to support two level partitions
Cao, Lionel created CARBONDATA-1633: --- Summary: Change parser to support two level partitions Key: CARBONDATA-1633 URL: https://issues.apache.org/jira/browse/CARBONDATA-1633 Project: CarbonData Issue Type: Sub-task Reporter: Cao, Lionel -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (CARBONDATA-1631) Implement Range Interval Partition
[ https://issues.apache.org/jira/browse/CARBONDATA-1631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cao, Lionel reassigned CARBONDATA-1631: --- Assignee: (was: Cao, Lionel) > Implement Range Interval Partition > -- > > Key: CARBONDATA-1631 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1631 > Project: CarbonData > Issue Type: Sub-task > Components: core, spark-integration, sql >Reporter: Cao, Lionel > Fix For: 1.3.0 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] carbondata pull request #1440: [WIP][CARBONDATA-1581][CARBONDATA-1582] Imple...
GitHub user QiangCai opened a pull request: https://github.com/apache/carbondata/pull/1440 [WIP][CARBONDATA-1581][CARBONDATA-1582] Implement StreamSinkProvider and stream file writer 1. Change hadoop.version to 2.7.2 as default Require using truncate operation of the filesystem. 2. CarbonSource extend StreamSinkProvider Provide stream sink to support streaming ingest 3. Implement CarbonStreamOutputFormat and CarbonStreamRecordWriter CarbonStreamRecordWriter write input data to CarbonData stream file. 4. Avoid Small file issue Append new blocklet to old file to avoid small file issue 5. Support fault tolerant Stream segment has a CarbonIndex file, this index file record the information of the CarbonData files. We can recover data to last successful commit. You can merge this pull request into a Git repository by running: $ git pull https://github.com/QiangCai/carbondata streaming Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/1440.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1440 commit 6c94c9311ea1b260e75bf576eec75aea17ce8984 Author: QiangCai Date: 2017-10-18T03:13:00Z support streaming ingest ---
[jira] [Created] (CARBONDATA-1632) Change PartitionInfo and related model to support two level partitions
Cao, Lionel created CARBONDATA-1632: --- Summary: Change PartitionInfo and related model to support two level partitions Key: CARBONDATA-1632 URL: https://issues.apache.org/jira/browse/CARBONDATA-1632 Project: CarbonData Issue Type: Sub-task Reporter: Cao, Lionel -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1631) Implement Range Interval Partition
Cao, Lionel created CARBONDATA-1631: --- Summary: Implement Range Interval Partition Key: CARBONDATA-1631 URL: https://issues.apache.org/jira/browse/CARBONDATA-1631 Project: CarbonData Issue Type: Sub-task Reporter: Cao, Lionel Assignee: Cao, Lionel -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] carbondata issue #1437: [CARBONDATA-1618] Fix issue of not support table com...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1437 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1324/ ---
[jira] [Created] (CARBONDATA-1630) load data into hive table fail
xujie created CARBONDATA-1630: - Summary: load data into hive table fail Key: CARBONDATA-1630 URL: https://issues.apache.org/jira/browse/CARBONDATA-1630 Project: CarbonData Issue Type: Bug Components: hive-integration Affects Versions: 1.2.0 Environment: hive.version:1.1.0-cdh5.10.0 hadoop:version:2.6.0-cdh5.10.0 Reporter: xujie Priority: Critical import org.apache.spark.sql.SparkSession import org.apache.spark.sql.CarbonSession._ val rootPath = "hdfs://namenodeb:8020/app/carbondata" val storeLocation = s"$rootPath/store" val warehouse = s"$rootPath/warehouse" val metastoredb = s"$rootPath/metastore_db" val carbon = SparkSession.builder().enableHiveSupport().config("spark.sql.warehouse.dir", warehouse).config(org.apache.carbondata.core.constants.CarbonCommonConstants.STORE_LOCATION, storeLocation).getOrCreateCarbonSession(storeLocation, metastoredb) import org.apache.spark.sql.types._ import org.apache.spark.sql.Row val rdd = sc.textFile("/data/home/hadoop/test.txt"); val schemaString = "id name city" val fields = schemaString.split(" ").map(fieldName => StructField(fieldName, StringType, nullable = true)) val schema = StructType(fields) val rowRDD = rdd.map(_.split(",")).map(attributes => Row(attributes(0),attributes(1),attributes(2))) val peopleDF = spark.createDataFrame(rowRDD, schema) peopleDF.createOrReplaceTempView("tmp_table") spark.sql("insert into target_table SELECT * FROM tmp_table") java.lang.RuntimeException: Failed to add entry in table status for default.target_table at scala.sys.package$.error(package.scala:27) at org.apache.carbondata.spark.util.CommonUtil$.readAndUpdateLoadProgressInTableMeta(CommonUtil.scala:533) at org.apache.spark.sql.execution.command.LoadTable.processData(carbonTableSchema.scala:928) at org.apache.spark.sql.execution.command.LoadTable.run(carbonTableSchema.scala:754) at org.apache.spark.sql.execution.command.LoadTableByInsert.processData(carbonTableSchema.scala:651) at org.apache.spark.sql.execution.command.LoadTableByInsert.run(carbonTableSchema.scala:637) at org.apache.spark.sql.CarbonDatasourceHadoopRelation.insert(CarbonDatasourceHadoopRelation.scala:98) at org.apache.spark.sql.execution.datasources.InsertIntoDataSourceCommand.run(InsertIntoDataSourceCommand.scala:43) at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58) at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56) at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74) at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114) at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114) at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:135) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:132) at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:113) at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:92) at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:92) at org.apache.spark.sql.Dataset.(Dataset.scala:185) at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:64) at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:592) ... 52 elided -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1629) Partition Function Enhancement
Cao, Lionel created CARBONDATA-1629: --- Summary: Partition Function Enhancement Key: CARBONDATA-1629 URL: https://issues.apache.org/jira/browse/CARBONDATA-1629 Project: CarbonData Issue Type: New Feature Components: core, spark-integration, sql Affects Versions: 1.3.0 Reporter: Cao, Lionel Assignee: Cao, Lionel Fix For: 1.3.0 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] carbondata issue #1439: [CARBONDATA-1628] Re-factory LoadTableCommand to reu...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1439 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/693/ ---
[GitHub] carbondata pull request #1439: [CARBONDATA-1628] Re-factory LoadTableCommand...
GitHub user QiangCai opened a pull request: https://github.com/apache/carbondata/pull/1439 [CARBONDATA-1628] Re-factory LoadTableCommand to reuse code for streaming ingest in the future Re-factory LoadTableCommand to reuse code for streaming ingest in the future You can merge this pull request into a Git repository by running: $ git pull https://github.com/QiangCai/carbondata refactory_loadtable Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/1439.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1439 commit 53e4ad765cc530446980626a638dd6f90da5a692 Author: QiangCai Date: 2017-10-27T08:06:06Z refactory_loadtable ---
[jira] [Created] (CARBONDATA-1628) Re-factory LoadTableCommand to reuse code for streaming ingest in the future
QiangCai created CARBONDATA-1628: Summary: Re-factory LoadTableCommand to reuse code for streaming ingest in the future Key: CARBONDATA-1628 URL: https://issues.apache.org/jira/browse/CARBONDATA-1628 Project: CarbonData Issue Type: Improvement Components: spark-integration Reporter: QiangCai Priority: Minor Re-factory LoadTableCommand to reuse code for streaming ingest in the future -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] carbondata issue #1438: [WIP]insert overwrite fix
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1438 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1323/ ---
[GitHub] carbondata issue #1433: [CARBONDATA-1517]- Pre Aggregate Create Table Suppor...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1433 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/692/ ---
[GitHub] carbondata issue #1436: [WIP][CARBONDATA-1617] Merging carbonindex files wit...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/1436 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/691/ ---
[jira] [Commented] (CARBONDATA-1624) If SORT_SCOPE is non-GLOBAL_SORT with Spark, set 'carbon.number.of.cores.while.loading' dynamically as per the available executor cores
[ https://issues.apache.org/jira/browse/CARBONDATA-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16221856#comment-16221856 ] Zhichao Zhang commented on CARBONDATA-1624: Hi [~ravi.pesala], why does it impact other loads? I will implement this as following steps: 1. Add a parameter called 'carbon.number.of.cores.in.executor' to CarbonCommonConstants; 2. In 'CarbonSession.getOrCreateCarbonSession', get the core number of executor from SparkConf (using sparkContext.getConf.get("spark.executor.cores")) , and then set this number to CarbonProperties, the key is 'carbon.number.of.cores.in.executor'; 3. Change CarbonProperties.getNumberOfCores(), use the value of 'carbon.number.of.cores.in.executor' as the default value of 'carbon.number.of.cores.while.loading'; if the value of 'carbon.number.of.cores.in.executor' is null or empty, use 'NUM_CORES_DEFAULT_VAL' as the default value. Are the above steps OK? Do you mean that it can't change the below code snippet to 'CarbonProperties.getNumberOfCores()'? > If SORT_SCOPE is non-GLOBAL_SORT with Spark, set > 'carbon.number.of.cores.while.loading' dynamically as per the available > executor cores > > > Key: CARBONDATA-1624 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1624 > Project: CarbonData > Issue Type: Improvement > Components: data-load, spark-integration >Affects Versions: 1.3.0 >Reporter: Zhichao Zhang >Assignee: Zhichao Zhang >Priority: Minor > > If we are using carbondata + spark to load data, we can set > carbon.number.of.cores.while.loading to the number of executor cores. > For example, when set the number of executor cores to 6, it shows that there > are at > least 6 cores per node for loading data, so we can set > carbon.number.of.cores.while.loading to 6 automatically. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] carbondata issue #1418: [CARBONDATA-1573] Support Database Location Configur...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/1418 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1322/ ---