[GitHub] incubator-carbondata pull request #417: fix conf issue for scanrdd
Github user scwf commented on a diff in the pull request: https://github.com/apache/incubator-carbondata/pull/417#discussion_r91848570 --- Diff: integration/spark/src/main/scala/org/apache/spark/sql/CarbonEnv.scala --- @@ -21,22 +21,24 @@ import org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend import org.apache.spark.sql.hive.CarbonMetastore import org.apache.carbondata.hadoop.readsupport.impl.RawDataReadSupport -import org.apache.carbondata.spark.rdd.SparkCommonEnv +import org.apache.carbondata.spark.rdd.SparkReadSupport case class CarbonEnv(carbonMetastore: CarbonMetastore) object CarbonEnv { @volatile private var carbonEnv: CarbonEnv = _ + // set readsupport class global so that the executor can get it. + SparkReadSupport.readSupportClass = classOf[RawDataReadSupport] --- End diff -- No, this is the key point, init is called in driver, see the comment, place here is to let executor get the read support class --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #417: fix conf issue for scanrdd
Github user scwf commented on a diff in the pull request: https://github.com/apache/incubator-carbondata/pull/417#discussion_r91848571 --- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/CarbonEnv.scala --- @@ -37,6 +36,9 @@ object CarbonEnv { @volatile private var carbonEnv: CarbonEnv = _ + // set readsupport class global so that the executor can get it. + SparkReadSupport.readSupportClass = classOf[SparkRowReadSupportImpl] --- End diff -- also the same --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #417: fix conf issue for scanrdd
Github user scwf commented on the issue: https://github.com/apache/incubator-carbondata/pull/417 /cc @QiangCai @jackylk plz review this --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #417: fix conf issue for scanrdd
GitHub user scwf opened a pull request: https://github.com/apache/incubator-carbondata/pull/417 fix conf issue for scanrdd we should sync the conf in executor side You can merge this pull request into a Git repository by running: $ git pull https://github.com/KirinKing/incubator-carbondata fix-scan Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/417.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #417 commit 1c54cd4ef942be0f5e2eef40c34db98a2233a0dc Author: wangfei Date: 2016-12-10T23:24:03Z fix conf issue for scanrdd --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #414: [CARBONDATA-517] Use carbon property to get...
Github user scwf commented on the issue: https://github.com/apache/incubator-carbondata/pull/414 @jackylk plz review this, i think this ready to merge --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #415: Fix integrate with spark 2.1
Github user scwf commented on the issue: https://github.com/apache/incubator-carbondata/pull/415 /cc @QiangCai can you try this fix --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #415: Fix integrate with spark 2.1
GitHub user scwf opened a pull request: https://github.com/apache/incubator-carbondata/pull/415 Fix integrate with spark 2.1 You can merge this pull request into a Git repository by running: $ git pull https://github.com/KirinKing/incubator-carbondata fix-2.1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/415.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #415 commit 74e9b98d2596d5c52fa8ef43aeddeaf6ca689c01 Author: wangfei Date: 2016-12-09T09:28:59Z use carbon property to get the store path commit 906ab88ed4ebe178bf1a52e1bb26f3dc12d4f07c Author: wangfei Date: 2016-12-09T09:51:32Z remove nouse imports commit 14cfe617f46e84a27f83da2fe2edd1e73298b7d1 Author: wangfei Date: 2016-12-09T13:14:04Z fix bugs with spark 2.1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #414: Use carbon property to get the store...
GitHub user scwf opened a pull request: https://github.com/apache/incubator-carbondata/pull/414 Use carbon property to get the store path Spark conf will ignore the config which does not start with "spark.", so let's use carbon property to get the carbon store path /cc @QiangCai plz review this You can merge this pull request into a Git repository by running: $ git pull https://github.com/KirinKing/incubator-carbondata fix-storepath-config Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/414.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #414 commit 74e9b98d2596d5c52fa8ef43aeddeaf6ca689c01 Author: wangfei Date: 2016-12-09T09:28:59Z use carbon property to get the store path --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #408: [CARBONDATA-512] Make timestamp formatter s...
Github user scwf commented on the issue: https://github.com/apache/incubator-carbondata/pull/408 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #243: [CARBONDATA-323] load data not suppo...
Github user scwf closed the pull request at: https://github.com/apache/incubator-carbondata/pull/243 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #243: [CARBONDATA-323] load data not support loca...
Github user scwf commented on the issue: https://github.com/apache/incubator-carbondata/pull/243 i will close this, and in spark2 integration we do not support load local for carbon, in spark1 i think there is no problem --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #393: [SPARK2] exclude dependency
Github user scwf commented on the issue: https://github.com/apache/incubator-carbondata/pull/393 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #389: [CARBONDATA-491] spark2 integration: Do not...
Github user scwf commented on the issue: https://github.com/apache/incubator-carbondata/pull/389 @jackylk i fix the test suite, plz review again --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #390: [CARBONDATA-492]fix a bug of profile spark-...
Github user scwf commented on the issue: https://github.com/apache/incubator-carbondata/pull/390 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #390: [CARBONDATA-492]fix a bug of profile spark-...
Github user scwf commented on the issue: https://github.com/apache/incubator-carbondata/pull/390 can you explain the root cause here, so it's easy to review --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #389: Do not use runnable command in spark...
GitHub user scwf opened a pull request: https://github.com/apache/incubator-carbondata/pull/389 Do not use runnable command in spark2 Runnable command is a inner interface in spark, it may changed from versions, we'd better not use it. You can merge this pull request into a Git repository by running: $ git pull https://github.com/KirinKing/incubator-carbondata runcommand Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/389.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #389 commit 59d8cd5fd90c9cb952a487a9ca4592c3ed5823ee Author: wangfei Date: 2016-12-03T12:40:39Z do not reply on runnable command --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #386: Fix spark2 decimal issue
Github user scwf commented on a diff in the pull request: https://github.com/apache/incubator-carbondata/pull/386#discussion_r90750992 --- Diff: core/src/main/java/org/apache/carbondata/core/datastorage/store/impl/FileFactory.java --- @@ -397,6 +397,38 @@ public static boolean createNewFile(String filePath, FileType fileType) throws I } } + public static boolean deleteFile(String filePath, FileType fileType) throws IOException { --- End diff -- added for carbon metasore to clean store path --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #386: Fix spark2 decimal issue
GitHub user scwf opened a pull request: https://github.com/apache/incubator-carbondata/pull/386 Fix spark2 decimal issue also added a test suite for decimal You can merge this pull request into a Git repository by running: $ git pull https://github.com/KirinKing/incubator-carbondata fix-decimal Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/386.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #386 commit 4da98d3c54021d8744809df0e779fac2dbc34a5a Author: wangfei Date: 2016-12-03T04:44:07Z fix spark2 decimal commit f351c8d6e586019d13a087dddaddb759127ab948 Author: wangfei Date: 2016-12-03T04:52:30Z code clean --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #385: Fix spark2 decimal
Github user scwf closed the pull request at: https://github.com/apache/incubator-carbondata/pull/385 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #385: Fix spark2 decimal
GitHub user scwf opened a pull request: https://github.com/apache/incubator-carbondata/pull/385 Fix spark2 decimal also added test suite for decimal You can merge this pull request into a Git repository by running: $ git pull https://github.com/KirinKing/incubator-carbondata SJS Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/385.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #385 commit cf11617c2e8c4480873356ce2e2333a4bc8a180f Author: wangfei Date: 2016-12-03T00:34:53Z add sjs testsuite commit c1587b87d8de2bc9a95da1bd35f2dc59aa6376a6 Author: wangfei Date: 2016-12-03T01:26:01Z decimal => double and date => string commit 76c8f66486dabf51ae5cf184846d991e6affce6d Author: wangfei Date: 2016-12-03T01:59:13Z use original datatype to create table commit 7b27e416fa8575a7f0abb0e01689f9bf1b7a7da5 Author: wangfei Date: 2016-12-03T01:59:50Z date => string commit 4b40a64051b2dfa13e17f53e88915e92d8b04182 Author: wangfei Date: 2016-12-03T02:25:42Z added clean store commit 34537e8ff9681c84102e006c1132709b0912ea60 Author: wangfei Date: 2016-12-03T02:26:28Z fix 4.2.7 cartession join commit e3280cf6647ee958f35b8ce623d17cac5195bcd2 Author: wangfei Date: 2016-12-03T02:29:45Z fix 4.2.4 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #372: [CARBONDATA-473] integrate with spark 2 sta...
Github user scwf commented on the issue: https://github.com/apache/incubator-carbondata/pull/372 @jacklk added test case from empty table --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #372: [CARBONDATA-473] integrate with spark 2 sta...
Github user scwf commented on the issue: https://github.com/apache/incubator-carbondata/pull/372 the result has some issues for the carbon example: ``` +--++---+---+---+ |shortField|intField|bigintField|doubleField|stringField| +--++---+---+---+ | 1| 11| null| 44.4| flink| | 3| 14| null| 43.4| hive| | 2| 10| null| 43.4| impala| | 4| 16| null| 42.4| impala| | 1| 10| null| 48.4| spark| | 5| 17| null| 43.4| spark| | 1| 10| null| 43.4| spark| | 1| 10| null| 47.4| spark| | 1| 10| null| 43.4| spark| | 1| 10| null| 43.4| spark| +--++---+---+---+ ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #372: [CARBONDATA-473] integrate with spar...
Github user scwf commented on a diff in the pull request: https://github.com/apache/incubator-carbondata/pull/372#discussion_r90207315 --- Diff: integration/spark-common/pom.xml --- @@ -84,6 +79,11 @@ junit + com.databricks + spark-csv_${scala.binary.version} + 1.2.0 --- End diff -- we can extract this as a property --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #372: [CARBONDATA-473] integrate with spar...
GitHub user scwf opened a pull request: https://github.com/apache/incubator-carbondata/pull/372 [CARBONDATA-473] integrate with spark 2 stable datasource api 1. read path: based on spark2 datasource api 2. load path: give a tool to do data load You can merge this pull request into a Git repository by running: $ git pull https://github.com/KirinKing/incubator-carbondata spark2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/372.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #372 commit a78efaea951194b1e9a3273b817d5e1641ed0f2c Author: jackylk Date: 2016-11-30T08:28:34Z add spark2 module commit 8d536ced3c1c147bb734c769a28cebbf33f2f035 Author: jackylk Date: 2016-11-30T08:49:03Z rebase commit 308fe24500a81373dac7d87d5f2a563a35ff547d Author: jackylk Date: 2016-11-30T08:49:57Z rebase commit d3416dc128c0d4c37d9f8191796e596d9ca7a77b Author: jackylk Date: 2016-11-29T12:12:22Z rebase commit 120914467ee93a6613911fcf25171fa952eccd83 Author: jackylk Date: 2016-11-30T08:50:32Z rebase commit d4cee19fbd54aa008662e36479b0ee0dda0ef384 Author: QiangCai Date: 2016-11-29T13:25:58Z carbon api for spark2 commit bdfa20346d2ce432f925c02c47a9d2dd7dd5d807 Author: QiangCai Date: 2016-11-29T16:38:53Z fix late decoder commit 9a96466209a5918b2fc4f55a728055bc0093 Author: jackylk Date: 2016-11-30T09:16:16Z fix commit 68f61c195a90b027e9d0dbc5a87915ca68d90c00 Author: Fei Wang Date: 2016-11-30T09:31:56Z Merge pull request #26 from jackylk/kingmaster Kingmaster --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #365: [CARBONDATA-463] Extract code to spark-comm...
Github user scwf commented on the issue: https://github.com/apache/incubator-carbondata/pull/365 i think we should merge this into our spark2 branch, otherwise it will leads to large conflicts --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #352: [CARBONDATA-448] Solve compilation e...
Github user scwf commented on a diff in the pull request: https://github.com/apache/incubator-carbondata/pull/352#discussion_r89570118 --- Diff: pom.xml --- @@ -291,27 +286,36 @@ 1.5.2 +2.10 +2.10.4 integration/spark +examples/spark --- End diff -- should also include spark common module --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #351: [CARBONDATA-447] Use Carbon log service ins...
Github user scwf commented on the issue: https://github.com/apache/incubator-carbondata/pull/351 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---