[GitHub] carbondata issue #3012: [CARBONDATA-3127]Fix the HiveExample & TestCarbonSer...
Github user jackylk commented on the issue: https://github.com/apache/carbondata/pull/3012 LGTM ---
[GitHub] carbondata issue #2998: [CARBONDATA-3184]Fix DataLoad Failure with 'using ca...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2998 Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10151/ ---
[GitHub] carbondata issue #3014: [WIP] Added load level SORT_SCOPE
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/3014 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1897/ ---
[GitHub] carbondata pull request #3014: [WIP] Added load level SORT_SCOPE
GitHub user NamanRastogi opened a pull request: https://github.com/apache/carbondata/pull/3014 [WIP] Added load level SORT_SCOPE ### Load level SORT_SCOPE ```sql LOAD DATA INPATH 'path/to/data.csv' INTO TABLE my_table OPTIONS ( 'sort_scope'='no_sort' ) ``` ### Priority of SORT_SCOPE 1. Load Level (if provided) 2. Table level (if provided) 3. Default = - [x] Any interfaces changed? ---> No - [x] Any backward compatibility impacted? ---> No - [x] Document update required? ---> Yes - [ ] Testing done - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/NamanRastogi/carbondata load_sort_scope Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/3014.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3014 commit f2bb306acfb3af095b6306be4793acaf8b8380f2 Author: namanrastogi Date: 2018-12-21T07:33:30Z Added load level SORT_SCOPE ---
[GitHub] carbondata issue #2897: [CARBONDATA-3080] Supporting local dictionary enable...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2897 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2104/ ---
[GitHub] carbondata issue #2998: [CARBONDATA-3184]Fix DataLoad Failure with 'using ca...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2998 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1896/ ---
[GitHub] carbondata issue #3012: [CARBONDATA-3127]Fix the HiveExample & TestCarbonSer...
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/3012 LGTM ---
[GitHub] carbondata issue #3012: [CARBONDATA-3127]Fix the HiveExample & TestCarbonSer...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/3012 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2102/ ---
[GitHub] carbondata issue #3012: [CARBONDATA-3127]Fix the HiveExample & TestCarbonSer...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/3012 Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10148/ ---
[GitHub] carbondata issue #2897: [CARBONDATA-3080] Supporting local dictionary enable...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2897 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1895/ ---
[GitHub] carbondata issue #2998: [CARBONDATA-3184]Fix DataLoad Failure with 'using ca...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2998 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1894/ ---
[GitHub] carbondata issue #3012: [CARBONDATA-3127]Fix the HiveExample & TestCarbonSer...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/3012 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1893/ ---
[GitHub] carbondata pull request #2998: [CARBONDATA-3184]Fix DataLoad Failure with 'u...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2998#discussion_r243489099
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/CarbonSource.scala ---
@@ -331,7 +333,17 @@ object CarbonSource {
properties,
query)
// updating params
- val updatedFormat = storageFormat.copy(properties = map)
+ var updatedFormat: CatalogStorageFormat = null
+ // check if catalog table location is empty or if differs from
tablepath, then change the
+ // value of locationUri to tablepath
+ if (SparkUtil.isSparkVersionXandAbove("2.2") &&
tableDesc.storage.locationUri.isEmpty) {
+updatedFormat =
CarbonToSparkAdapater.getUpdatedStorageFormat(storageFormat, map, tablePath)
+ } else if (SparkUtil.isSparkVersionXandAbove("2.2") &&
+
!tableDesc.storage.locationUri.toString.equalsIgnoreCase(tablePath)) {
--- End diff --
Please check this equals check , because `locationUri` may comes with file
protocol also like `file:/` or `hdfs:/` , in case of local file the `file:/`
may not be present so this equals check fails.
---
[GitHub] carbondata pull request #3008: Carbonfile
Github user BeyondYourself closed the pull request at: https://github.com/apache/carbondata/pull/3008 ---
[GitHub] carbondata pull request #3009: Carbonfile
Github user BeyondYourself closed the pull request at: https://github.com/apache/carbondata/pull/3009 ---
[GitHub] carbondata issue #3012: [CARBONDATA-3127]Fix the HiveExample & TestCarbonSer...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/3012 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2101/ ---
[GitHub] carbondata issue #3012: [CARBONDATA-3127]Fix the HiveExample & TestCarbonSer...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/3012 Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10147/ ---
[GitHub] carbondata issue #3013: [WIP]Test
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/3013 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2100/ ---
[GitHub] carbondata issue #3013: [WIP]Test
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/3013 Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10146/ ---
[GitHub] carbondata issue #3004: [CARBONDATA-3188] Create carbon table as hive unders...
Github user SteNicholas commented on the issue: https://github.com/apache/carbondata/pull/3004 @ravipesala This pull request doesn't work in #3012 .HiveExample run exception through your changes.But this update does work in beeline.Therefore will you check the cause of HiveExample exception with meï¼ ---
[GitHub] carbondata issue #3012: [CARBONDATA-3127]Fix the HiveExample & TestCarbonSer...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/3012 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1892/ ---
[GitHub] carbondata pull request #3012: [CARBONDATA-3127]Fix the HiveExample & TestCa...
Github user SteNicholas commented on a diff in the pull request: https://github.com/apache/carbondata/pull/3012#discussion_r243478610 --- Diff: examples/spark2/src/test/scala/org/apache/carbondata/examplesCI/RunExamples.scala --- @@ -19,12 +19,12 @@ package org.apache.carbondata.examplesCI import org.apache.spark.sql.test.util.QueryTest import org.scalatest.BeforeAndAfterAll - import org.apache.carbondata.examples._ import org.apache.carbondata.core.constants.CarbonCommonConstants import org.apache.carbondata.core.util.CarbonProperties import org.apache.carbondata.examples.sdk.CarbonReaderExample import org.apache.carbondata.examples.sql.JavaCarbonSessionExample +import org.apache.spark.sql.test.TestQueryExecutor --- End diff -- @xubo245 Yeah, I move to another group.TestCarbonSerDe is also updated as same as the way of RunExamples.scala. ---
[jira] [Created] (CARBONDATA-3190) There are many code style errors in example module when I open the code style check
xubo245 created CARBONDATA-3190:
---
Summary: There are many code style errors in example module when I
open the code style check
Key: CARBONDATA-3190
URL: https://issues.apache.org/jira/browse/CARBONDATA-3190
Project: CarbonData
Issue Type: Bug
Affects Versions: 1.5.1
Reporter: xubo245
mvn -DskipTests -Pspark-2.2 -Pbuild-with-format clean install
PR test:
https://github.com/apache/carbondata/pull/3013
{code:java}
[INFO] --- maven-checkstyle-plugin:2.17:check (default) @
carbondata-examples-spark2 ---
[INFO] Starting audit...
/Users/xubo/Desktop/xubo/git/carbondata1/examples/spark2/src/main/java/org/apache/carbondata/examples/sdk/SDKS3Example.java:40:
error: 'method def modifier' have incorrect indentation level 4, expected
level should be 2.
/Users/xubo/Desktop/xubo/git/carbondata1/examples/spark2/src/main/java/org/apache/carbondata/examples/sdk/SDKS3Example.java:41:
error: 'member def type' have incorrect indentation level 8, expected level
should be 4.
/Users/xubo/Desktop/xubo/git/carbondata1/examples/spark2/src/main/java/org/apache/carbondata/examples/sdk/SDKS3Example.java:41:
error: 'method def' child have incorrect indentation level 8, expected level
should be 4.
/Users/xubo/Desktop/xubo/git/carbondata1/examples/spark2/src/main/java/org/apache/carbondata/examples/sdk/SDKS3Example.java:42:
error: 'if' have incorrect indentation level 8, expected level should be 4.
/Users/xubo/Desktop/xubo/git/carbondata1/examples/spark2/src/main/java/org/apache/carbondata/examples/sdk/SDKS3Example.java:43:
error: 'if' child have incorrect indentation level 12, expected level should
be 6.
/Users/xubo/Desktop/xubo/git/carbondata1/examples/spark2/src/main/java/org/apache/carbondata/examples/sdk/SDKS3Example.java:45:
error: 'if' child have incorrect indentation level 12, expected level should
be 6.
/Users/xubo/Desktop/xubo/git/carbondata1/examples/spark2/src/main/java/org/apache/carbondata/examples/sdk/SDKS3Example.java:46:
error: 'if rcurly' have incorrect indentation level 8, expected level should
be 4.
/Users/xubo/Desktop/xubo/git/carbondata1/examples/spark2/src/main/java/org/apache/carbondata/examples/sdk/SDKS3Example.java:48:
error: 'member def type' have incorrect indentation level 8, expected level
should be 4.
/Users/xubo/Desktop/xubo/git/carbondata1/examples/spark2/src/main/java/org/apache/carbondata/examples/sdk/SDKS3Example.java:48:
error: 'method def' child have incorrect indentation level 8, expected level
should be 4.
/Users/xubo/Desktop/xubo/git/carbondata1/examples/spark2/src/main/java/org/apache/carbondata/examples/sdk/SDKS3Example.java:51:
error: 'method def' child have incorrect indentation level 8, expected level
should be 4.
/Users/xubo/Desktop/xubo/git/carbondata1/examples/spark2/src/main/java/org/apache/carbondata/examples/sdk/SDKS3Example.java:52:
error: Line is longer than 100 characters (found 106).
/Users/xubo/Desktop/xubo/git/carbondata1/examples/spark2/src/main/java/org/apache/carbondata/examples/sdk/SDKS3Example.java:54:
error: 'member def type' have incorrect indentation level 8, expected level
should be 4.
/Users/xubo/Desktop/xubo/git/carbondata1/examples/spark2/src/main/java/org/apache/carbondata/examples/sdk/SDKS3Example.java:54:
error: 'method def' child have incorrect indentation level 8, expected level
should be 4.
/Users/xubo/Desktop/xubo/git/carbondata1/examples/spark2/src/main/java/org/apache/carbondata/examples/sdk/SDKS3Example.java:55:
error: 'if' have incorrect indentation level 8, expected level should be 4.
/Users/xubo/Desktop/xubo/git/carbondata1/examples/spark2/src/main/java/org/apache/carbondata/examples/sdk/SDKS3Example.java:56:
error: 'if' child have incorrect indentation level 12, expected level should
be 6.
/Users/xubo/Desktop/xubo/git/carbondata1/examples/spark2/src/main/java/org/apache/carbondata/examples/sdk/SDKS3Example.java:56:17:
error: '=' is not preceded with whitespace.
/Users/xubo/Desktop/xubo/git/carbondata1/examples/spark2/src/main/java/org/apache/carbondata/examples/sdk/SDKS3Example.java:56:18:
error: '=' is not followed by whitespace.
/Users/xubo/Desktop/xubo/git/carbondata1/examples/spark2/src/main/java/org/apache/carbondata/examples/sdk/SDKS3Example.java:57:
error: 'if rcurly' have incorrect indentation level 8, expected level should
be 4.
/Users/xubo/Desktop/xubo/git/carbondata1/examples/spark2/src/main/java/org/apache/carbondata/examples/sdk/SDKS3Example.java:59:
error: 'member def type' have incorrect indentation level 8, expected level
should be 4.
/Users/xubo/Desktop/xubo/git/carbondata1/examples/spark2/src/main/java/org/apache/carbondata/examples/sdk/SDKS3Example.java:59:
error: 'method def' child have incorrect indentation level 8, expected level
should be 4.
/Users/xubo/Desktop/xubo/git/carbondata1/examples/spark2/src/main/java/org/apache/carbondata/examples/sdk/SDKS3Example.java:60:
error
[GitHub] carbondata pull request #3012: [CARBONDATA-3127]Fix the HiveExample & TestCa...
Github user SteNicholas commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3012#discussion_r243478424
--- Diff:
examples/spark2/src/main/scala/org/apache/carbondata/examples/HiveExample.scala
---
@@ -30,44 +33,38 @@ object HiveExample {
private val driverName: String = "org.apache.hive.jdbc.HiveDriver"
def main(args: Array[String]) {
-val rootPath = new File(this.getClass.getResource("/").getPath
-+ "../../../..").getCanonicalPath
-val store = s"$rootPath/integration/hive/target/store"
-val warehouse = s"$rootPath/integration/hive/target/warehouse"
-val metaStore_Db =
s"$rootPath/integration/hive/target/carbon_metaStore_db"
-val logger =
LogServiceFactory.getLogService(this.getClass.getCanonicalName)
-var resultId = ""
-var resultName = ""
-var resultSalary = ""
-
+val carbonSession = ExampleUtils.createCarbonSession("HiveExample")
+exampleBody(carbonSession, CarbonProperties.getStorePath
+ + CarbonCommonConstants.FILE_SEPARATOR
+ + CarbonCommonConstants.DATABASE_DEFAULT_NAME)
+carbonSession.close()
-import org.apache.spark.sql.CarbonSession._
+System.exit(0)
+ }
-val carbonSession = SparkSession
- .builder()
- .master("local")
- .appName("HiveExample")
- .config("carbonSession.sql.warehouse.dir",
warehouse).enableHiveSupport()
- .getOrCreateCarbonSession(
-store, metaStore_Db)
+ def exampleBody(sparkSession: SparkSession, store: String): Unit = {
--- End diff --
@xubo245 I update parameter name in exampleBody method according to your
suggestion.
---
[GitHub] carbondata pull request #3012: [CARBONDATA-3127]Fix the HiveExample & TestCa...
Github user SteNicholas commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3012#discussion_r243478363
--- Diff: examples/spark2/pom.xml ---
@@ -78,7 +83,12 @@
org.apache.httpcomponents
httpclient
- ${httpclient.version}
+ 4.3.4
+
+
+ org.apache.httpcomponents
+ httpcore
+ 4.3-alpha1
--- End diff --
@xubo245 I have already use the unify version of http core library.
---
[GitHub] carbondata pull request #3012: [CARBONDATA-3127]Fix the HiveExample & TestCa...
Github user SteNicholas commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3012#discussion_r243478043
--- Diff: examples/spark2/pom.xml ---
@@ -78,7 +83,12 @@
org.apache.httpcomponents
httpclient
- ${httpclient.version}
+ 4.3.4
--- End diff --
@xubo245 I have already use the unify version of http client library.
---
[GitHub] carbondata issue #3013: [WIP]Test
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/3013 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1891/ ---
[GitHub] carbondata issue #3013: [WIP]Test
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/3013 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2099/ ---
[GitHub] carbondata issue #2161: [CARBONDATA-2218] AlluxioCarbonFile while trying to ...
Github user kevinjmh commented on the issue: https://github.com/apache/carbondata/pull/2161 > Since user normally uses Alluxio as a read cache, I think we can firstly verify carbon on alluxio for the query scenario. As I am still not very sure what is the correct way to implement rename for Alluxio, in the meantime, we can merge this PR first. So please rebase it. @chandrasaripaka @jackylk My first try wants to use alluxio for query only as you said. But hive stores table location and carbon blocks `alter location` command, I have to create table after carbon runs on alluxio. Please mind this problem. ---
[GitHub] carbondata pull request #3012: [CARBONDATA-3127]Fix the HiveExample & TestCa...
Github user xubo245 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/3012#discussion_r243470594 --- Diff: examples/spark2/src/test/scala/org/apache/carbondata/examplesCI/RunExamples.scala --- @@ -19,12 +19,12 @@ package org.apache.carbondata.examplesCI import org.apache.spark.sql.test.util.QueryTest import org.scalatest.BeforeAndAfterAll - import org.apache.carbondata.examples._ import org.apache.carbondata.core.constants.CarbonCommonConstants import org.apache.carbondata.core.util.CarbonProperties import org.apache.carbondata.examples.sdk.CarbonReaderExample import org.apache.carbondata.examples.sql.JavaCarbonSessionExample +import org.apache.spark.sql.test.TestQueryExecutor --- End diff -- please move o another group ---
[GitHub] carbondata pull request #3012: [CARBONDATA-3127]Fix the HiveExample & TestCa...
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3012#discussion_r243470427
--- Diff:
examples/spark2/src/main/scala/org/apache/carbondata/examples/HiveExample.scala
---
@@ -30,44 +33,38 @@ object HiveExample {
private val driverName: String = "org.apache.hive.jdbc.HiveDriver"
def main(args: Array[String]) {
-val rootPath = new File(this.getClass.getResource("/").getPath
-+ "../../../..").getCanonicalPath
-val store = s"$rootPath/integration/hive/target/store"
-val warehouse = s"$rootPath/integration/hive/target/warehouse"
-val metaStore_Db =
s"$rootPath/integration/hive/target/carbon_metaStore_db"
-val logger =
LogServiceFactory.getLogService(this.getClass.getCanonicalName)
-var resultId = ""
-var resultName = ""
-var resultSalary = ""
-
+val carbonSession = ExampleUtils.createCarbonSession("HiveExample")
+exampleBody(carbonSession, CarbonProperties.getStorePath
+ + CarbonCommonConstants.FILE_SEPARATOR
+ + CarbonCommonConstants.DATABASE_DEFAULT_NAME)
+carbonSession.close()
-import org.apache.spark.sql.CarbonSession._
+System.exit(0)
+ }
-val carbonSession = SparkSession
- .builder()
- .master("local")
- .appName("HiveExample")
- .config("carbonSession.sql.warehouse.dir",
warehouse).enableHiveSupport()
- .getOrCreateCarbonSession(
-store, metaStore_Db)
+ def exampleBody(sparkSession: SparkSession, store: String): Unit = {
--- End diff --
carbonSession is better
---
[GitHub] carbondata pull request #3012: [CARBONDATA-3127]Fix the HiveExample & TestCa...
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3012#discussion_r243470291
--- Diff: examples/spark2/pom.xml ---
@@ -78,7 +83,12 @@
org.apache.httpcomponents
httpclient
- ${httpclient.version}
+ 4.3.4
--- End diff --
Please use the unify version
---
[GitHub] carbondata pull request #3012: [CARBONDATA-3127]Fix the HiveExample & TestCa...
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/3012#discussion_r243470296
--- Diff: examples/spark2/pom.xml ---
@@ -78,7 +83,12 @@
org.apache.httpcomponents
httpclient
- ${httpclient.version}
+ 4.3.4
+
+
+ org.apache.httpcomponents
+ httpcore
+ 4.3-alpha1
--- End diff --
Please use the unify version
---
[GitHub] carbondata issue #3013: Test
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/3013 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1890/ ---
[GitHub] carbondata pull request #3013: Test
GitHub user xubo245 reopened a pull request: https://github.com/apache/carbondata/pull/3013 Test Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/xubo245/carbondata Test Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/3013.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3013 commit b41eff1c3d129dc3d8776f3eeeb31eb23ebd1f9b Author: xubo245 Date: 2018-11-14T09:03:17Z [CARBONDATA-3097] Support folder path in getVersionDetails and support getVersionDetails in CSDK fix ci error test remove commit f487130468f2ea8444b077d22728e872e0c62239 Author: xubo245 Date: 2018-12-19T03:11:06Z support get Verson from S3 ---
[GitHub] carbondata pull request #3013: Test
Github user xubo245 closed the pull request at: https://github.com/apache/carbondata/pull/3013 ---
[GitHub] carbondata pull request #3013: Test
GitHub user xubo245 reopened a pull request: https://github.com/apache/carbondata/pull/3013 Test Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/xubo245/carbondata Test Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/3013.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3013 commit b41eff1c3d129dc3d8776f3eeeb31eb23ebd1f9b Author: xubo245 Date: 2018-11-14T09:03:17Z [CARBONDATA-3097] Support folder path in getVersionDetails and support getVersionDetails in CSDK fix ci error test remove commit f487130468f2ea8444b077d22728e872e0c62239 Author: xubo245 Date: 2018-12-19T03:11:06Z support get Verson from S3 ---
[GitHub] carbondata pull request #3013: Test
Github user xubo245 closed the pull request at: https://github.com/apache/carbondata/pull/3013 ---
[GitHub] carbondata pull request #3013: Test
GitHub user xubo245 opened a pull request: https://github.com/apache/carbondata/pull/3013 Test Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/xubo245/carbondata Test Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/3013.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3013 commit b41eff1c3d129dc3d8776f3eeeb31eb23ebd1f9b Author: xubo245 Date: 2018-11-14T09:03:17Z [CARBONDATA-3097] Support folder path in getVersionDetails and support getVersionDetails in CSDK fix ci error test remove commit f487130468f2ea8444b077d22728e872e0c62239 Author: xubo245 Date: 2018-12-19T03:11:06Z support get Verson from S3 ---
[GitHub] carbondata issue #3010: [CARBONDATA-3189] Fix PreAggregate Datamap Issue
Github user qiuchenjian commented on the issue: https://github.com/apache/carbondata/pull/3010 Can you show up the error message when load and query failing ---
[GitHub] carbondata issue #3012: [CARBONDATA-3127]Fix the HiveExample & TestCarbonSer...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/3012 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2098/ ---
[GitHub] carbondata issue #3012: [CARBONDATA-3127]Fix the HiveExample & TestCarbonSer...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/3012 Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10143/ ---
[GitHub] carbondata issue #3012: [CARBONDATA-3127]Fix the HiveExample & TestCarbonSer...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/3012 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1889/ ---
[GitHub] carbondata pull request #3012: [CARBONDATA-3127]Fix the HiveExample & TestCa...
GitHub user SteNicholas opened a pull request: https://github.com/apache/carbondata/pull/3012 [CARBONDATA-3127]Fix the HiveExample & TestCarbonSerde exception [CARBONDATA-3127]Hive module test case has been commented off,can' t run. This pull request fix TestCarbonSerde test case exception with maven compile.Method called deserializeAndSerializeLazySimple of TestCarbonSerde class calls the interface called serializeStartKey which is not exsit in CarbonHiveSerDe class.Just modify replace serializeStartKey with serialize and remove comments of TestCarbonSerde. Previous hive-integration HiveExample exists obvious bug that there is no data when select HIVE_CARBON_EXAMPLE table.I have already fixed the bug above in this pull request.Location was configured wrong after creating hive table.And block lack of detail info and footer offset when do querying hive table operation.Therefore I set detail info of block with footer offset through input split. - [x] Any interfaces changed? No - [x] Any backward compatibility impacted? No - [x] Document update required? No - [x] Testing done Modify TestCarbonSerde test - [x] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. No You can merge this pull request into a Git repository by running: $ git pull https://github.com/SteNicholas/carbondata CARBONDATA-3127 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/3012.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3012 commit f403455748aa98a224559dac7e8953725d98b41c Author: Nicholas Jiang Date: 2018-12-20T20:14:09Z Hive Integration ---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the HiveExample & TestCarbonSer...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2969 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2097/ ---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the HiveExample & TestCarbonSer...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2969 Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10142/ ---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the HiveExample & TestCarbonSer...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2969 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1888/ ---
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the HiveExample & TestCa...
Github user SteNicholas closed the pull request at: https://github.com/apache/carbondata/pull/2969 ---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the HiveExample & TestCarbonSer...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2969 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1887/ ---
[GitHub] carbondata pull request #2991: [CARBONDATA-3043] Add build script and add te...
Github user KanakaKumar commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2991#discussion_r243361889
--- Diff: store/CSDK/test/main.cpp ---
@@ -665,6 +699,208 @@ bool readFromS3(JNIEnv *env, char *path, char
*argv[]) {
printResult(env, reader);
}
+TEST(CSDKTest,tryCatchException) {
+bool gotExp=tryCatchException(env);
+EXPECT_TRUE(gotExp);
+}
+
+
+TEST(CSDKTest,tryCarbonRowException) {
+char *smallFilePath = "../../../../resources/carbondata";
+try {
+bool result = tryCarbonRowException(env, smallFilePath);;
+EXPECT_TRUE(result) << "Expected Exception as No Index File" <<
result;
+} catch (runtime_error e) {
+EXPECT_TRUE(true);
+}
+}
+
+TEST(CSDKTest,testCarbonProperties) {
+try {
+bool result = testCarbonProperties(env);
+EXPECT_TRUE(result) << "Carbon set properties not working ";
+} catch (runtime_error e) {
+EXPECT_TRUE(false) << " Exception is not expected while setting
carbon properties ";
+}
+}
+
+TEST(CSDKTest,testWriteData) {
+try {
+bool result =testWriteData(env, "./data", my_argc, my_argv);
+result = result && testWriteData(env, "./data", my_argc, my_argv);
+EXPECT_TRUE(result) << "Either Data Loading Or Carbon Reader is
failed";
+} catch (runtime_error e) {
+EXPECT_TRUE(false) << " Exception is not expected while data
loading";
+}
+}
+
+TEST(CSDKTest,readFromLocalWithoutProjection) {
+try {
+char *smallFilePath = "./data_withoutpro";
+bool result =testWriteData(env, smallFilePath, my_argc, my_argv);
+if(result){
+bool proj_result = readFromLocalWithoutProjection(env,
smallFilePath);
+EXPECT_TRUE(proj_result) << "Without Projection is failed";
+} else {
+EXPECT_TRUE(result) << "Either Data Loading Or Carbon Reader
is failed";
+}
+} catch (runtime_error e) {
+EXPECT_TRUE(false) << " Exception is not expected ,During without
projection selection";
+}
+}
+
+TEST(CSDKTest,readFromLocalWithProjection) {
+try {
+char *smallFilePath = "./data_pro";
+bool result =testWriteData(env, smallFilePath, my_argc, my_argv);
+if(result){
+bool proj_result = readFromLocalWithProjection(env,
smallFilePath);
+EXPECT_TRUE(proj_result) << "With Projection is failed";
+} else {
+EXPECT_TRUE(result) << "Either Data Loading Or Carbon Reader
is failed";
+}
+} catch (runtime_error e) {
+EXPECT_TRUE(false) << " Exception is not expected ,During With
projection selection";
+}
+}
+
+
+TEST(CSDKTest,readSchemaWithoutValidation) {
+try {
+char *path = "./data_readPath";
+bool result =testWriteData(env, path, my_argc, my_argv);
+if(result){
+bool schema_result = readSchema(env, path, false);
+EXPECT_TRUE(schema_result) << "Not Able to read readSchema
from given path";
+} else {
+EXPECT_TRUE(result) << "Either Data Loading Or Carbon Reader
is failed";
+}
+} catch (runtime_error e) {
+EXPECT_TRUE(false) << " Exception is not expected ,During Read
Schema";
+}
+}
+
+
+TEST(CSDKTest,readSchemaWithValidation) {
+try {
+char *path = "./data_readPathWithValidation";
+bool result =testWriteData(env, path, my_argc, my_argv);
+if(result){
+bool schema_result = readSchema(env, path, true);
+EXPECT_TRUE(schema_result) << "Not Able to read readSchema Or
validate from given path";
+} else {
+EXPECT_TRUE(result) << "Either Data Loading Or Carbon Reader
is failed";
+}
+} catch (runtime_error e) {
+EXPECT_TRUE(false) << " Exception is not expected ,During Read
Schema";
+}
+}
+
+TEST(CSDKTest,testReadNextRowWthtVector) {
+try {
+int printNum = 32000;
+char *path = "./data_forVector";
+bool result =testWriteData(env, path, my_argc, my_argv);
+if(result){
+bool readresultWithVector= testReadNextRow(env, path,
printNum, my_argv, 0, true);
+EXPECT_TRUE(readresultWithVector) << "Vector reading is
failed";
+} else {
+EXPECT_TRUE(result) << "Either Data Loading Or Carbon Reader
is failed";
+}
+} catch (runtime_error e) {
+EXPECT_TRUE(false) << " Exception is not expected ,During Vector
read";
+}
+}
+
+
+
[GitHub] carbondata pull request #2991: [CARBONDATA-3043] Add build script and add te...
Github user KanakaKumar commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2991#discussion_r243360358
--- Diff: store/CSDK/test/main.cpp ---
@@ -665,6 +699,208 @@ bool readFromS3(JNIEnv *env, char *path, char
*argv[]) {
printResult(env, reader);
}
+TEST(CSDKTest,tryCatchException) {
+bool gotExp=tryCatchException(env);
+EXPECT_TRUE(gotExp);
+}
+
+
+TEST(CSDKTest,tryCarbonRowException) {
+char *smallFilePath = "../../../../resources/carbondata";
+try {
+bool result = tryCarbonRowException(env, smallFilePath);;
+EXPECT_TRUE(result) << "Expected Exception as No Index File" <<
result;
+} catch (runtime_error e) {
+EXPECT_TRUE(true);
+}
+}
+
+TEST(CSDKTest,testCarbonProperties) {
+try {
+bool result = testCarbonProperties(env);
+EXPECT_TRUE(result) << "Carbon set properties not working ";
+} catch (runtime_error e) {
+EXPECT_TRUE(false) << " Exception is not expected while setting
carbon properties ";
+}
+}
+
+TEST(CSDKTest,testWriteData) {
+try {
+bool result =testWriteData(env, "./data", my_argc, my_argv);
+result = result && testWriteData(env, "./data", my_argc, my_argv);
+EXPECT_TRUE(result) << "Either Data Loading Or Carbon Reader is
failed";
+} catch (runtime_error e) {
+EXPECT_TRUE(false) << " Exception is not expected while data
loading";
+}
+}
+
+TEST(CSDKTest,readFromLocalWithoutProjection) {
+try {
+char *smallFilePath = "./data_withoutpro";
+bool result =testWriteData(env, smallFilePath, my_argc, my_argv);
+if(result){
+bool proj_result = readFromLocalWithoutProjection(env,
smallFilePath);
+EXPECT_TRUE(proj_result) << "Without Projection is failed";
+} else {
+EXPECT_TRUE(result) << "Either Data Loading Or Carbon Reader
is failed";
+}
+} catch (runtime_error e) {
+EXPECT_TRUE(false) << " Exception is not expected ,During without
projection selection";
+}
+}
+
+TEST(CSDKTest,readFromLocalWithProjection) {
+try {
+char *smallFilePath = "./data_pro";
+bool result =testWriteData(env, smallFilePath, my_argc, my_argv);
+if(result){
+bool proj_result = readFromLocalWithProjection(env,
smallFilePath);
+EXPECT_TRUE(proj_result) << "With Projection is failed";
+} else {
+EXPECT_TRUE(result) << "Either Data Loading Or Carbon Reader
is failed";
+}
+} catch (runtime_error e) {
+EXPECT_TRUE(false) << " Exception is not expected ,During With
projection selection";
+}
+}
+
+
+TEST(CSDKTest,readSchemaWithoutValidation) {
+try {
+char *path = "./data_readPath";
+bool result =testWriteData(env, path, my_argc, my_argv);
+if(result){
+bool schema_result = readSchema(env, path, false);
+EXPECT_TRUE(schema_result) << "Not Able to read readSchema
from given path";
+} else {
+EXPECT_TRUE(result) << "Either Data Loading Or Carbon Reader
is failed";
+}
+} catch (runtime_error e) {
+EXPECT_TRUE(false) << " Exception is not expected ,During Read
Schema";
+}
+}
+
+
+TEST(CSDKTest,readSchemaWithValidation) {
+try {
+char *path = "./data_readPathWithValidation";
+bool result =testWriteData(env, path, my_argc, my_argv);
+if(result){
+bool schema_result = readSchema(env, path, true);
+EXPECT_TRUE(schema_result) << "Not Able to read readSchema Or
validate from given path";
+} else {
+EXPECT_TRUE(result) << "Either Data Loading Or Carbon Reader
is failed";
+}
+} catch (runtime_error e) {
+EXPECT_TRUE(false) << " Exception is not expected ,During Read
Schema";
+}
+}
+
+TEST(CSDKTest,testReadNextRowWthtVector) {
+try {
+int printNum = 32000;
+char *path = "./data_forVector";
+bool result =testWriteData(env, path, my_argc, my_argv);
+if(result){
+bool readresultWithVector= testReadNextRow(env, path,
printNum, my_argv, 0, true);
+EXPECT_TRUE(readresultWithVector) << "Vector reading is
failed";
+} else {
+EXPECT_TRUE(result) << "Either Data Loading Or Carbon Reader
is failed";
+}
+} catch (runtime_error e) {
+EXPECT_TRUE(false) << " Exception is not expected ,During Vector
read";
+}
+}
+
+
+
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the HiveExample & TestCarbonSer...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2969 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1886/ ---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the HiveExample & TestCarbonSer...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2969 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2094/ ---
[jira] [Resolved] (CARBONDATA-3179) DataLoad Failure in Map Data Type
[ https://issues.apache.org/jira/browse/CARBONDATA-3179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kumar vishal resolved CARBONDATA-3179. -- Resolution: Fixed > DataLoad Failure in Map Data Type > - > > Key: CARBONDATA-3179 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3179 > Project: CarbonData > Issue Type: Bug >Reporter: MANISH NALLA >Assignee: MANISH NALLA >Priority: Major > Time Spent: 2h 40m > Remaining Estimate: 0h > > Data Load failing for insert into table select * from table containing Map > datatype -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2993: [CARBONDATA-3179] Map Data Load Failure and S...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2993 ---
[GitHub] carbondata issue #2993: [CARBONDATA-3179] Map Data Load Failure and Struct P...
Github user kumarvishal09 commented on the issue: https://github.com/apache/carbondata/pull/2993 LGTM ---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the HiveExample & TestCarbonSer...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2969 Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10140/ ---
[jira] [Resolved] (CARBONDATA-3149) Support alter table column rename
[ https://issues.apache.org/jira/browse/CARBONDATA-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacky Li resolved CARBONDATA-3149. -- Resolution: Fixed Fix Version/s: 1.5.2 > Support alter table column rename > - > > Key: CARBONDATA-3149 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3149 > Project: CarbonData > Issue Type: New Feature >Reporter: Akash R Nilugal >Assignee: Akash R Nilugal >Priority: Major > Fix For: 1.5.2 > > Attachments: Alter table column rename.odt > > Time Spent: 15h 10m > Remaining Estimate: 0h > > Please find the mailing list link for the same > [http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/Discussion-Alter-table-column-rename-feature-tt69814.html|http://example.com] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2990: [CARBONDATA-3149]Support alter table column r...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2990 ---
[GitHub] carbondata issue #2990: [CARBONDATA-3149]Support alter table column rename
Github user akashrn5 commented on the issue: https://github.com/apache/carbondata/pull/2990 @jackylk thank you ---
[GitHub] carbondata issue #2990: [CARBONDATA-3149]Support alter table column rename
Github user jackylk commented on the issue: https://github.com/apache/carbondata/pull/2990 LGTM. Thanks for working on this ---
[GitHub] carbondata issue #3011: [WIP] multiple issue fixes in presto-carbon integrat...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/3011 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2093/ ---
[GitHub] carbondata issue #3011: [WIP] multiple issue fixes in presto-carbon integrat...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/3011 Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10139/ ---
[GitHub] carbondata issue #2990: [CARBONDATA-3149]Support alter table column rename
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2990 Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10138/ ---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the HiveExample & TestCarbonSer...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2969 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1885/ ---
[GitHub] carbondata issue #2990: [CARBONDATA-3149]Support alter table column rename
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2990 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2092/ ---
[GitHub] carbondata issue #2161: [CARBONDATA-2218] AlluxioCarbonFile while trying to ...
Github user jackylk commented on the issue: https://github.com/apache/carbondata/pull/2161 Since user normally uses Alluxio as a read cache, I think we can firstly verify carbon on alluxio for the query scenario. As I am still not very sure what is the correct way to implement rename for Alluxio, in the meantime, we can merge this PR first. So please rebase it. @chandrasaripaka ---
[GitHub] carbondata issue #2998: [CARBONDATA-3184]Fix DataLoad Failure with 'using ca...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2998 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2091/ ---
[GitHub] carbondata issue #3011: [WIP] multiple issue fixes in presto-carbon integrat...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/3011 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1884/ ---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the HiveExample & TestCarbonSer...
Github user jackylk commented on the issue: https://github.com/apache/carbondata/pull/2969 LGTM, can you squash your commits and rebase to master. I could not squash it ---
[GitHub] carbondata issue #2998: [CARBONDATA-3184]Fix DataLoad Failure with 'using ca...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2998 Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10137/ ---
[GitHub] carbondata pull request #3011: [WIP] multiple issue fixes in presto-carbon i...
GitHub user ajantha-bhat opened a pull request: https://github.com/apache/carbondata/pull/3011 [WIP] multiple issue fixes in presto-carbon integration. [WIP] issue1: NPE in HDFS, when carbon table is already cached in presto issue2: Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ajantha-bhat/carbondata presto Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/3011.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3011 commit eeb03ca64f5af120929bf06135ba4c2037406ea2 Author: ajantha-bhat Date: 2018-12-20T14:26:01Z nonTransactional table NPE, when table present in cache ---
[GitHub] carbondata issue #2990: [CARBONDATA-3149]Support alter table column rename
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2990 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1883/ ---
[GitHub] carbondata issue #2990: [CARBONDATA-3149]Support alter table column rename
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2990 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2090/ ---
[GitHub] carbondata issue #2990: [CARBONDATA-3149]Support alter table column rename
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2990 Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10135/ ---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the HiveExample & TestCarbonSer...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2969 Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10133/ ---
[GitHub] carbondata issue #2998: [CARBONDATA-3184]Fix DataLoad Failure with 'using ca...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2998 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1882/ ---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the HiveExample & TestCarbonSer...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2969 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2088/ ---
[jira] [Issue Comment Deleted] (CARBONDATA-3173) Add hive-guide and other guides to the root of the file ReadMe
[ https://issues.apache.org/jira/browse/CARBONDATA-3173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] beyond updated CARBONDATA-3173: --- Comment: was deleted (was: add the hive /presto /flink /alluxio documents index to the root of file ReadMe.md ，it helps us to find the relevant integration documents) > Add hive-guide and other guides to the root of the file ReadMe > -- > > Key: CARBONDATA-3173 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3173 > Project: CarbonData > Issue Type: Improvement > Components: docs >Affects Versions: NONE >Reporter: beyond >Assignee: beyond >Priority: Minor > Labels: documentation > Fix For: NONE > > Time Spent: 50m > Remaining Estimate: 0h > > In order to facilitate everyone's reading, we unified the integrated document > index into the ReadMe file in the root directory. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (CARBONDATA-3173) Add hive-guide and other guides to the root of the file ReadMe
[ https://issues.apache.org/jira/browse/CARBONDATA-3173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] beyond updated CARBONDATA-3173: --- Summary: Add hive-guide and other guides to the root of the file ReadMe (was: add hive-guide and other guides to the root of the file ReadMe) > Add hive-guide and other guides to the root of the file ReadMe > -- > > Key: CARBONDATA-3173 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3173 > Project: CarbonData > Issue Type: Improvement > Components: docs >Affects Versions: NONE >Reporter: beyond >Assignee: beyond >Priority: Minor > Labels: documentation > Fix For: NONE > > Time Spent: 50m > Remaining Estimate: 0h > > In order to facilitate everyone's reading, we unified the integrated document > index into the ReadMe file in the root directory. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2990: [CARBONDATA-3149]Support alter table column rename
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2990 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1880/ ---
[GitHub] carbondata issue #2990: [CARBONDATA-3149]Support alter table column rename
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2990 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1879/ ---
[GitHub] carbondata issue #2990: [CARBONDATA-3149]Support alter table column rename
Github user akashrn5 commented on the issue: https://github.com/apache/carbondata/pull/2990 @jackylk handled comments, please review ---
[jira] [Updated] (CARBONDATA-3189) Load and Select query was failing on table with preaggregate datamap.
[ https://issues.apache.org/jira/browse/CARBONDATA-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shubhdeep Singh updated CARBONDATA-3189: Description: Load and Select query was failing on table with preaggregate datamap. > Load and Select query was failing on table with preaggregate datamap. > - > > Key: CARBONDATA-3189 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3189 > Project: CarbonData > Issue Type: Bug >Reporter: Shubhdeep Singh >Priority: Minor > > Load and Select query was failing on table with preaggregate datamap. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CARBONDATA-3189) Load and Select query was failing on table with preaggregate datamap.
Shubhdeep Singh created CARBONDATA-3189: --- Summary: Load and Select query was failing on table with preaggregate datamap. Key: CARBONDATA-3189 URL: https://issues.apache.org/jira/browse/CARBONDATA-3189 Project: CarbonData Issue Type: Bug Reporter: Shubhdeep Singh -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #3010: Fix PreAggregate Datamap Issue
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/3010 Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10132/ ---
[GitHub] carbondata pull request #2990: [CARBONDATA-3149]Support alter table column r...
Github user akashrn5 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2990#discussion_r243261978
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/schema/CarbonAlterTableColRenameDataTypeChangeCommand.scala
---
@@ -0,0 +1,324 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.schema
+
+import scala.collection.JavaConverters._
+import scala.collection.mutable
+
+import org.apache.spark.sql.{CarbonEnv, Row, SparkSession}
+import
org.apache.spark.sql.execution.command.{AlterTableDataTypeChangeModel,
DataTypeInfo,
+ MetadataCommand}
+import org.apache.spark.sql.hive.CarbonSessionCatalog
+import org.apache.spark.util.AlterTableUtil
+
+import
org.apache.carbondata.common.exceptions.sql.MalformedCarbonCommandException
+import org.apache.carbondata.common.logging.LogServiceFactory
+import org.apache.carbondata.core.features.TableOperation
+import org.apache.carbondata.core.locks.{ICarbonLock, LockUsage}
+import
org.apache.carbondata.core.metadata.converter.ThriftWrapperSchemaConverterImpl
+import org.apache.carbondata.core.metadata.datatype.DecimalType
+import org.apache.carbondata.core.metadata.schema.table.CarbonTable
+import org.apache.carbondata.core.metadata.schema.table.column.CarbonColumn
+import
org.apache.carbondata.events.{AlterTableColRenameAndDataTypeChangePostEvent,
+ AlterTableColRenameAndDataTypeChangePreEvent, OperationContext,
OperationListenerBus}
+import org.apache.carbondata.format.{ColumnSchema, SchemaEvolutionEntry,
TableInfo}
+import org.apache.carbondata.spark.util.DataTypeConverterUtil
+
+abstract class CarbonAlterTableColumnRenameCommand(oldColumnName: String,
newColumnName: String)
+ extends MetadataCommand {
+
+ protected def validColumnsForRenaming(carbonColumns:
mutable.Buffer[CarbonColumn],
+ oldCarbonColumn: CarbonColumn,
+ carbonTable: CarbonTable): Unit = {
+// check whether new column name is already an existing column name
+if
(carbonColumns.exists(_.getColName.equalsIgnoreCase(newColumnName))) {
+ throw new MalformedCarbonCommandException(s"Column Rename Operation
failed. New " +
+s"column name
$newColumnName already exists" +
+s" in table ${
carbonTable.getTableName }")
+}
+
+// if the column rename is for complex column, block the operation
+if (oldCarbonColumn.isComplex) {
+ throw new MalformedCarbonCommandException(s"Column Rename Operation
failed. Rename " +
+s"column is unsupported
for complex datatype " +
+s"column ${
oldCarbonColumn.getColName }")
+}
+
+// if column rename operation is on partition column, then fail the
rename operation
+if (null != carbonTable.getPartitionInfo) {
+ val partitionColumns =
carbonTable.getPartitionInfo.getColumnSchemaList
+ partitionColumns.asScala.foreach {
+col =>
+ if (col.getColumnName.equalsIgnoreCase(oldColumnName)) {
+throw new MalformedCarbonCommandException(
+ s"Column Rename Operation failed. Renaming " +
+ s"the partition column $newColumnName is not " +
+ s"allowed")
+ }
+ }
+}
+
+ }
+}
+
+private[sql] case class CarbonAlterTableColRenameDataTypeChangeCommand(
+alterTableColRenameAndDataTypeChangeModel:
AlterTableDataTypeChangeModel,
+childTableColumnRename: Boolean = false)
+ extends
CarbonAlterTableColumnRenameCommand(alterTableColRenameAndDataTypeChangeModel.columnName,
+alterTableColRenameAndDataTypeChangeModel.newColumnName) {
+
+ override def processMetadata(sparkSession: SparkSession): Seq[Row] = {
+val LOGGER =
LogServ
[GitHub] carbondata issue #3010: Fix PreAggregate Datamap Issue
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/3010 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2087/ ---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2969 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1878/ ---
[GitHub] carbondata pull request #2990: [CARBONDATA-3149]Support alter table column r...
Github user akashrn5 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2990#discussion_r243260747
--- Diff:
integration/spark-common/src/main/scala/org/apache/spark/sql/catalyst/CarbonDDLSqlParser.scala
---
@@ -1511,7 +1514,16 @@ abstract class CarbonDDLSqlParser extends
AbstractCarbonSparkSQLParser {
}
DataTypeInfo("decimal", precision, scale)
case _ =>
-throw new MalformedCarbonCommandException("Data type provided is
invalid.")
+if (isColumnRename) {
+ dataType match {
--- End diff --
ok, refactored, if rename operation then just return the datatypeInfo
object, else throw invalid datatype exception as before
---
[GitHub] carbondata issue #2998: [CARBONDATA-3184]Fix DataLoad Failure with 'using ca...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2998 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2085/ ---
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exce...
Github user SteNicholas commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2969#discussion_r243259414
--- Diff:
integration/hive/src/main/scala/org/apache/carbondata/hiveexample/HiveExample.scala
---
@@ -22,52 +22,49 @@ import java.sql.{DriverManager, ResultSet, Statement}
import org.apache.spark.sql.SparkSession
--- End diff --
@jackylk I have already moved this example to example folder and rename
module named carbondata-example-spark2.
---
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exce...
Github user SteNicholas commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2969#discussion_r243259131
--- Diff:
integration/hive/src/main/scala/org/apache/carbondata/hiveexample/HiveExample.scala
---
@@ -85,25 +82,35 @@ object HiveExample {
logger.info(s"HIVE CLI IS STARTED ON PORT $port
==")
-statement.execute("CREATE TABLE IF NOT EXISTS " + "HIVE_CARBON_EXAMPLE
" +
- " (ID int, NAME string,SALARY double)")
-statement
- .execute(
-"ALTER TABLE HIVE_CARBON_EXAMPLE SET FILEFORMAT INPUTFORMAT
\"org.apache.carbondata." +
-"hive.MapredCarbonInputFormat\"OUTPUTFORMAT
\"org.apache.carbondata.hive." +
-"MapredCarbonOutputFormat\"SERDE \"org.apache.carbondata.hive." +
-"CarbonHiveSerDe\" ")
+statement.execute(
+ s"""
+ | CREATE TABLE IF NOT EXISTS HIVE_CARBON_EXAMPLE
+ | (ID int, NAME string,SALARY double)
+ | ROW FORMAT SERDE 'org.apache.carbondata.hive.CarbonHiveSerDe'
+ | WITH SERDEPROPERTIES
('mapreduce.input.carboninputformat.databaseName'='default',
+ |
'mapreduce.input.carboninputformat.tableName'='HIVE_CARBON_EXAMPLE')
+ """.stripMargin)
+
+statement.execute(
+ s"""
+ | ALTER TABLE HIVE_CARBON_EXAMPLE
+ | SET FILEFORMAT
+ | INPUTFORMAT
\"org.apache.carbondata.hive.MapredCarbonInputFormat\"
--- End diff --
@jackylk Yeah, I really agree with you.But this pull request #3004 doesn't
work in HiveExample.
---
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exce...
Github user SteNicholas commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2969#discussion_r243259047
--- Diff:
integration/hive/src/main/scala/org/apache/carbondata/hiveexample/HiveExample.scala
---
@@ -85,25 +82,35 @@ object HiveExample {
logger.info(s"HIVE CLI IS STARTED ON PORT $port
==")
-statement.execute("CREATE TABLE IF NOT EXISTS " + "HIVE_CARBON_EXAMPLE
" +
- " (ID int, NAME string,SALARY double)")
-statement
- .execute(
-"ALTER TABLE HIVE_CARBON_EXAMPLE SET FILEFORMAT INPUTFORMAT
\"org.apache.carbondata." +
-"hive.MapredCarbonInputFormat\"OUTPUTFORMAT
\"org.apache.carbondata.hive." +
-"MapredCarbonOutputFormat\"SERDE \"org.apache.carbondata.hive." +
-"CarbonHiveSerDe\" ")
+statement.execute(
+ s"""
+ | CREATE TABLE IF NOT EXISTS HIVE_CARBON_EXAMPLE
+ | (ID int, NAME string,SALARY double)
+ | ROW FORMAT SERDE 'org.apache.carbondata.hive.CarbonHiveSerDe'
+ | WITH SERDEPROPERTIES
('mapreduce.input.carboninputformat.databaseName'='default',
+ |
'mapreduce.input.carboninputformat.tableName'='HIVE_CARBON_EXAMPLE')
+ """.stripMargin)
+
+statement.execute(
+ s"""
+ | ALTER TABLE HIVE_CARBON_EXAMPLE
+ | SET FILEFORMAT
+ | INPUTFORMAT
\"org.apache.carbondata.hive.MapredCarbonInputFormat\"
--- End diff --
Yeah, I really agree with you.But this pull request #3004 doesn't work in
HiveExample.
---
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exce...
Github user SteNicholas commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2969#discussion_r243258830
--- Diff:
integration/hive/src/main/scala/org/apache/carbondata/hiveexample/HiveExample.scala
---
@@ -49,17 +49,19 @@ object HiveExample {
sparkSession.sql("""DROP TABLE IF EXISTS
HIVE_CARBON_EXAMPLE""".stripMargin)
-sparkSession
- .sql(
-"CREATE TABLE HIVE_CARBON_EXAMPLE (ID int,NAME string,SALARY
double) " +
- "STORED BY 'CARBONDATA' ")
+sparkSession.sql(
+ s"""
+ | CREATE TABLE HIVE_CARBON_EXAMPLE
+ | (ID int,NAME string,SALARY double)
+ | STORED BY 'CARBONDATA'
--- End diff --
@jackylk This suggestion doesn't work
@xubo245 I have already lower CARBONDATA
---
[GitHub] carbondata pull request #3007: [CARBONDATA-3173] add hive-guide and other gu...
Github user BeyondYourself commented on a diff in the pull request: https://github.com/apache/carbondata/pull/3007#discussion_r243258655 --- Diff: README.md --- @@ -67,6 +67,11 @@ CarbonData is built using Apache Maven, to [build CarbonData](https://github.com * [Carbon as Spark's Datasource](https://github.com/apache/carbondata/blob/master/docs/carbon-as-spark-datasource-guide.md) * [FAQs](https://github.com/apache/carbondata/blob/master/docs/faq.md) +## Integrations +* [Hive](https://github.com/apache/carbondata/blob/master/docs/hive-guide.md) +* [Presto](https://github.com/apache/carbondata/tree/master/integration/presto) +* [Flink]( ) --- End diff -- ok ,I get it ---
[GitHub] carbondata pull request #2997: [CARBONDATA-3161]Pipe dilimiter is not workin...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2997 ---
[GitHub] carbondata pull request #3007: [CARBONDATA-3173] add hive-guide and other gu...
Github user BeyondYourself closed the pull request at: https://github.com/apache/carbondata/pull/3007 ---
[GitHub] carbondata issue #2993: [CARBONDATA-3179] Map Data Load Failure and Struct P...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2993 Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10131/ ---
[GitHub] carbondata issue #2997: [CARBONDATA-3161]Pipe dilimiter is not working for s...
Github user QiangCai commented on the issue: https://github.com/apache/carbondata/pull/2997 LGTM ---
