[GitHub] incubator-carbondata issue #704: [CARBONDATA-826] Create carbondata-connecto...
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/704 Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1372/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #635: [CARBONDATA-782]support SORT_COLUMNS
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/635 Build Failed with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1373/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (CARBONDATA-830) Incorrect schedule for NewCarbonDataLoadRDD
Weizhong created CARBONDATA-830: --- Summary: Incorrect schedule for NewCarbonDataLoadRDD Key: CARBONDATA-830 URL: https://issues.apache.org/jira/browse/CARBONDATA-830 Project: CarbonData Issue Type: Bug Components: spark-integration Affects Versions: 1.0.0-incubating Environment: Spark 2.1 + Carbon 1.0.0 Reporter: Weizhong Assignee: Weizhong Priority: Minor Currently NewCarbonDataLoadRDD's getPreferredLocations will return all locs rather than 1, then on Spark may pick the same node for two tasks, so one node is getting over loaded with the task and one has no task to do, and impacting the performance despite of any failure. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] incubator-carbondata pull request #704: [CARBONDATA-826] Create carbondata-c...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/704 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Resolved] (CARBONDATA-826) Create carbondata-connector for query carbon data in presto
[ https://issues.apache.org/jira/browse/CARBONDATA-826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen resolved CARBONDATA-826. --- Resolution: Fixed Fix Version/s: 1.2.0 > Create carbondata-connector for query carbon data in presto > --- > > Key: CARBONDATA-826 > URL: https://issues.apache.org/jira/browse/CARBONDATA-826 > Project: CarbonData > Issue Type: Sub-task > Components: presto-integration >Reporter: Liang Chen >Assignee: Liang Chen >Priority: Minor > Fix For: 1.2.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > 1.In CarbonData project, generate carbondata-connector of presto > 2.Copy carbondata-connector to presto/plugin/ > 3.Run query in presto to read carbon data. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] incubator-carbondata pull request #708: [CARBONDATA-830] Incorrect schedule ...
GitHub user Sephiroth-Lin opened a pull request: https://github.com/apache/incubator-carbondata/pull/708 [CARBONDATA-830] Incorrect schedule for NewCarbonDataLoadRDD Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like: `[CARBONDATA-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [ ] Replace `` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - What manual testing you have done? - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/Sephiroth-Lin/incubator-carbondata carbon-830 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/708.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #708 commit e4d51e5a9367d54da485b6a2f522295119e89602 Author: l00251599 Date: 2017-03-29T09:07:50Z Only return 1 preferred loc to confirm each node run 1 task --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #708: [CARBONDATA-830] Incorrect schedule for New...
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/708 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #708: [CARBONDATA-830] Incorrect schedule for New...
Github user Sephiroth-Lin commented on the issue: https://github.com/apache/incubator-carbondata/pull/708 @QiangCai Please help review this --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (CARBONDATA-831) can't run PerfTest example
sehriff created CARBONDATA-831: -- Summary: can't run PerfTest example Key: CARBONDATA-831 URL: https://issues.apache.org/jira/browse/CARBONDATA-831 Project: CarbonData Issue Type: Bug Reporter: sehriff can not run PerfTest in incubator-carbondata-apache-carbondata-1.0.0-incubating-rc2\examples\spark\src\main\scala\org\apache\carbondata\examples\ errlog: Exception in thread "main" java.lang.NoSuchMethodError: scala.reflect.api.JavaUniverse.runtimeMirror(Ljava/lang/ClassLoader;)Lscala/reflect/api/JavaUniverse$JavaMirror; at org.apache.spark.util.ScalaCompilerUtil$.compiledCode(ScalaCompilerUtil.scala:29) at org.apache.spark.sql.CodeGenerateFactory.(CodeGenerateFactory.scala:32) at org.apache.spark.sql.CodeGenerateFactory$.init(CodeGenerateFactory.scala:67) at org.apache.spark.sql.CarbonContext.(CarbonContext.scala:58) at org.apache.carbondata.examples.util.ExampleUtils$.createCarbonContext(ExampleUtils.scala:44) at org.apache.carbondata.examples.PerfTest$.main(PerfTest.scala:271) at org.apache.carbondata.examples.PerfTest.main(PerfTest.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] incubator-carbondata issue #699: [CARBONDATA-821] Removed kettle related cod...
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/699 Build Failed with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1374/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #635: [CARBONDATA-782]support SORT_COLUMNS
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/635 Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1375/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #672: [CARBONDATA-815] add hive integration for c...
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/672 Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1376/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #699: [CARBONDATA-821] Removed kettle related cod...
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/699 Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1377/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #697: [CARBONDATA-708] Fixed Between and L...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/697 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Resolved] (CARBONDATA-708) Between operator does not work properly in carbondata.
[
https://issues.apache.org/jira/browse/CARBONDATA-708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liang Chen resolved CARBONDATA-708.
---
Resolution: Fixed
Assignee: Bhavya Aggarwal
Fix Version/s: 1.1.0-incubating
> Between operator does not work properly in carbondata.
> --
>
> Key: CARBONDATA-708
> URL: https://issues.apache.org/jira/browse/CARBONDATA-708
> Project: CarbonData
> Issue Type: Bug
> Components: sql
>Affects Versions: 1.1.0-incubating
> Environment: Spark 2.1
>Reporter: SWATI RAO
>Assignee: Bhavya Aggarwal
>Priority: Minor
> Fix For: 1.1.0-incubating
>
> Attachments: 100_hive_test.csv
>
> Time Spent: 4h
> Remaining Estimate: 0h
>
> Incorrect result displays.
> Note:Issue exist when you take Latest_HOUR as a String but if you take
> Latest_HOUR as a int then it's working fine.
> Steps to reproduce:
> 1:Create table using following Command
> " create table Carbon_automation (imei string,deviceInformationId int,MAC
> string,deviceColor string,device_backColor string,modelId string,marketName
> string,AMSize string,ROMSize string,CUPAudit string,CPIClocked string,series
> string,productionDate timestamp,bomCode string,internalModels string,
> deliveryTime string, channelsId string, channelsName string , deliveryAreaId
> string, deliveryCountry string, deliveryProvince string, deliveryCity
> string,deliveryDistrict string, deliveryStreet string, oxSingleNumber string,
> ActiveCheckTime string, ActiveAreaId string, ActiveCountry string,
> ActiveProvince string, Activecity string, ActiveDistrict string, ActiveStreet
> string, ActiveOperatorId string, Active_releaseId string, Active_EMUIVersion
> string, Active_operaSysVersion string, Active_BacVerNumber string,
> Active_BacFlashVer string, Active_webUIVersion string,
> Active_webUITypeCarrVer string,Active_webTypeDataVerNumber string,
> Active_operatorsVersion string, Active_phonePADPartitionedVersions string,
> Latest_YEAR int, Latest_MONTH int, Latest_DAY int, Latest_HOUR string,
> Latest_areaId string, Latest_country string, Latest_province string,
> Latest_city string, Latest_district string, Latest_street string,
> Latest_releaseId string, Latest_EMUIVersion string, Latest_operaSysVersion
> string, Latest_BacVerNumber string, Latest_BacFlashVer string,
> Latest_webUIVersion string, Latest_webUITypeCarrVer string,
> Latest_webTypeDataVerNumber string, Latest_operatorsVersion string,
> Latest_phonePADPartitionedVersions string, Latest_operatorId string,
> gamePointDescription string,gamePointId double,contractNumber
> double,imei_count int) STORED BY 'org.apache.carbondata.format' TBLPROPERTIES
> ('DICTIONARY_INCLUDE'='deviceInformationId,Latest_YEAR,Latest_MONTH,Latest_DAY')"
> 2:Load Data with following command
> " LOAD DATA INPATH 'HDFS_URL/BabuStore/Data/HiveData' INTO TABLE
> Carbon_automation
> OPTIONS('DELIMITER'=',','QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='imei,deviceInformationId,MAC,deviceColor,device_backColor,modelId,marketName,AMSize,ROMSize,CUPAudit,CPIClocked,series,productionDate,bomCode,internalModels,deliveryTime,channelsId,channelsName,deliveryAreaId,deliveryCountry,deliveryProvince,deliveryCity,deliveryDistrict,deliveryStreet,oxSingleNumber,contractNumber,ActiveCheckTime,ActiveAreaId,ActiveCountry,ActiveProvince,Activecity,ActiveDistrict,ActiveStreet,ActiveOperatorId,Active_releaseId,Active_EMUIVersion,Active_operaSysVersion,Active_BacVerNumber,Active_BacFlashVer,Active_webUIVersion,Active_webUITypeCarrVer,Active_webTypeDataVerNumber,Active_operatorsVersion,Active_phonePADPartitionedVersions,Latest_YEAR,Latest_MONTH,Latest_DAY,Latest_HOUR,Latest_areaId,Latest_country,Latest_province,Latest_city,Latest_district,Latest_street,Latest_releaseId,Latest_EMUIVersion,Latest_operaSysVersion,Latest_BacVerNumber,Latest_BacFlashVer,Latest_webUIVersion,Latest_webUITypeCarrVer,Latest_webTypeDataVerNumber,Latest_operatorsVersion,Latest_phonePADPartitionedVersions,Latest_operatorId,gamePointId,gamePointDescription,imei_count')"
> 3:Run the Query
> select Latest_DAY,Latest_HOUR,count(distinct AMSize) as
> AMSize_number,sum(gamePointId+contractNumber) as total from Carbon_automation
> where Latest_HOUR between 12 and 15 group by Latest_DAY,Latest_HOUR order by
> total desc
> 4:No Result display:
> +-+--+++--+
> | Latest_DAY | Latest_HOUR | AMSize_number | total |
> +-+--+++--+
> +-+--+++--+
> No rows selected (2.133 seconds).
> 5:CSV Attached "100_hive_test.csv"
> Expected Result:Correct Result should be display.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (CARBONDATA-832) Data loading is failing with duplicate header column in csv file
kumar vishal created CARBONDATA-832: --- Summary: Data loading is failing with duplicate header column in csv file Key: CARBONDATA-832 URL: https://issues.apache.org/jira/browse/CARBONDATA-832 Project: CarbonData Issue Type: Bug Reporter: kumar vishal Assignee: kumar vishal Problem : data mismatch issue when csv column having duplicate column header. Solution: row parser impl logic of getting indexes is having issue -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] incubator-carbondata issue #635: [CARBONDATA-782]support SORT_COLUMNS
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/635 Build Failed with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1378/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #699: [CARBONDATA-821] Removed kettle related cod...
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/699 Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1379/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #709: [WIP] Improvements in query
GitHub user ravipesala opened a pull request: https://github.com/apache/incubator-carbondata/pull/709 [WIP] Improvements in query Following are the list of improvements done in this part of PR. 1. Removed multiple creation of array and copy of it in Dimension and measure chunk readers. 2. Simplified logic of finding offsets of nodictionary keys in the class SafeVariableLengthDimensionDataChunkStore. 3. Avoided byte array creation and copy for nodictionary columns in case of vectorized reader. Instead directly sending the length and offset to vector. 4. Removed unnecessary decoder plan additions to oprtimized plan. It can optimize the codegen flow. 5. Updated CompareTest to take table blocksize and kept as 32 Mb in order to make use of small sorting when doing take ordered in spark. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ravipesala/incubator-carbondata minor-perf-improv Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/709.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #709 commit eaa964425ffa784d905b12045e6e719c55eb1164 Author: ravipesala Date: 2017-03-05T15:02:35Z Removed unnecessary array copy and bitset checking commit 62914d866063a2606f6396b9912cf4466cbacef9 Author: ravipesala Date: 2017-03-28T15:24:26Z OPtimized code commit 45a4dcab42842f61d7cf28c5834bb4810c77bcbc Author: ravipesala Date: 2017-03-29T13:57:11Z Added table_blocksize option. commit 57d135937843ce89eb3805cddad6034cf9db3aaf Author: ravipesala Date: 2017-03-29T18:49:36Z Removed unnecessary plan from optimized plan. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #709: [WIP] Improvements in query
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/709 Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1380/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (CARBONDATA-759) How to check a table's properties?
[ https://issues.apache.org/jira/browse/CARBONDATA-759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15948172#comment-15948172 ] sehriff commented on CARBONDATA-759: thank you. > How to check a table's properties? > -- > > Key: CARBONDATA-759 > URL: https://issues.apache.org/jira/browse/CARBONDATA-759 > Project: CarbonData > Issue Type: Improvement >Reporter: sehriff >Priority: Minor > > Is there a way to check what colums of a table is treated as dimensions or > measures? > How to check default table > properties(COLUMN_GROUPS,DICTIONARY_EXCLUDE,DICTIONARY_INCLUDE,NO_INVERTED_INDEX) > and what of this tables's mdk looks like? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] incubator-carbondata pull request #708: [CARBONDATA-830] Incorrect schedule ...
Github user watermen commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/708#discussion_r108831010
--- Diff:
integration/spark-common/src/main/scala/org/apache/carbondata/spark/rdd/NewCarbonDataLoadRDD.scala
---
@@ -292,12 +292,7 @@ class NewCarbonDataLoadRDD[K, V](
}
}
- val sortedList = blockMap.entrySet().asScala.toSeq.sortWith {
(nodeCount1, nodeCount2) =>
-nodeCount1.getValue > nodeCount2.getValue
- }
-
- val sortedNodesList = sortedList.map(nodeCount =>
nodeCount.getKey).take(2)
- firstOptionLocation ++ sortedNodesList
+ firstOptionLocation
--- End diff --
Add some notes to explain why we do this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #699: [CARBONDATA-821] Removed kettle related cod...
Github user jackylk commented on the issue: https://github.com/apache/incubator-carbondata/pull/699 LGTM. I will merge this now. Thanks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #699: [CARBONDATA-821] Removed kettle rela...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/699 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #705: [CARBONDATA-827] Fix query stats format err...
Github user jackylk commented on the issue: https://github.com/apache/incubator-carbondata/pull/705 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #682: [CARBONDATA-832] Fixed data loading issue w...
Github user jackylk commented on the issue: https://github.com/apache/incubator-carbondata/pull/682 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #682: [CARBONDATA-832] Fixed data loading ...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/682 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Resolved] (CARBONDATA-832) Data loading is failing with duplicate header column in csv file
[ https://issues.apache.org/jira/browse/CARBONDATA-832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacky Li resolved CARBONDATA-832. - Resolution: Fixed Fix Version/s: 1.1.0-incubating > Data loading is failing with duplicate header column in csv file > > > Key: CARBONDATA-832 > URL: https://issues.apache.org/jira/browse/CARBONDATA-832 > Project: CarbonData > Issue Type: Bug >Reporter: kumar vishal >Assignee: kumar vishal > Fix For: 1.1.0-incubating > > Time Spent: 20m > Remaining Estimate: 0h > > Problem : data mismatch issue when csv column having duplicate column header. > Solution: row parser impl logic of getting indexes is having issue -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (CARBONDATA-821) Remove Kettle related code and flow from carbon.
[ https://issues.apache.org/jira/browse/CARBONDATA-821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacky Li resolved CARBONDATA-821. - Resolution: Fixed Fix Version/s: 1.1.0-incubating > Remove Kettle related code and flow from carbon. > > > Key: CARBONDATA-821 > URL: https://issues.apache.org/jira/browse/CARBONDATA-821 > Project: CarbonData > Issue Type: Bug >Reporter: Ravindra Pesala >Assignee: Ravindra Pesala > Fix For: 1.1.0-incubating > > Time Spent: 2h 20m > Remaining Estimate: 0h > > Remove Kettle related code and flow from carbon. It becomes difficult to > developers to handle all bugs and features in both the flows. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] incubator-carbondata issue #708: [CARBONDATA-830] Incorrect schedule for New...
Github user jackylk commented on the issue: https://github.com/apache/incubator-carbondata/pull/708 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #708: [CARBONDATA-830] Incorrect schedule ...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/708 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Resolved] (CARBONDATA-830) Incorrect schedule for NewCarbonDataLoadRDD
[ https://issues.apache.org/jira/browse/CARBONDATA-830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacky Li resolved CARBONDATA-830. - Resolution: Fixed Fix Version/s: 1.1.0-incubating > Incorrect schedule for NewCarbonDataLoadRDD > --- > > Key: CARBONDATA-830 > URL: https://issues.apache.org/jira/browse/CARBONDATA-830 > Project: CarbonData > Issue Type: Bug > Components: spark-integration >Affects Versions: 1.0.0-incubating > Environment: Spark 2.1 + Carbon 1.0.0 >Reporter: Weizhong >Assignee: Weizhong >Priority: Minor > Fix For: 1.1.0-incubating > > Time Spent: 1h > Remaining Estimate: 0h > > Currently NewCarbonDataLoadRDD's getPreferredLocations will return all locs > rather than 1, then on Spark may pick the same node for two tasks, so one > node is getting over loaded with the task and one has no task to do, and > impacting the performance despite of any failure. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (CARBONDATA-833) load data from dataframe,generater data row may be error when delimiterLevel1 or delimiterLevel2 is special character
tianli created CARBONDATA-833:
-
Summary: load data from dataframe,generater data row may be error
when delimiterLevel1 or delimiterLevel2 is special character
Key: CARBONDATA-833
URL: https://issues.apache.org/jira/browse/CARBONDATA-833
Project: CarbonData
Issue Type: Bug
Components: spark-integration
Affects Versions: 1.0.0-incubating, 1.1.0-incubating
Reporter: tianli
Assignee: tianli
Fix For: 1.1.0-incubating, 1.0.0-incubating
load data from dataframe,generater data row may be error by delimiterLevel1 or
delimiterLevel2 is special character
because delimiterLevel1 and delimiterLevel2 when carbonLoadModel is create by
CarbonUtil.delimiterConverter(), CarbonScalaUtil.getString direct use
carbonLoadModel.getComplexDelimiterLevel1 and
carbonLoadModel.getComplexDelimiterLevel2
val delimiter = if (level == 1) {
delimiterLevel1
} else {
delimiterLevel2
}
val builder = new StringBuilder()
s.foreach { x =>
builder.append(getString(x, serializationNullFormat,
delimiterLevel1,
delimiterLevel2, timeStampFormat, dateFormat, level +
1)).append(delimiter)
}
make primitive data added a more char \ when datatype is complex
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[GitHub] incubator-carbondata pull request #710: [CARBONDATA-833]load data from dataf...
GitHub user tianlileer opened a pull request:
https://github.com/apache/incubator-carbondata/pull/710
[CARBONDATA-833]load data from dataframe,generater data row may be error
when delimiterâ¦
load data from dataframe,generater data row may be error by delimiterLevel1
or delimiterLevel2 is special character
because delimiterLevel1 and delimiterLevel2 when carbonLoadModel is create
by CarbonUtil.delimiterConverter(), CarbonScalaUtil.getString direct use
carbonLoadModel.getComplexDelimiterLevel1 and
carbonLoadModel.getComplexDelimiterLevel2
val delimiter = if (level == 1)
{ delimiterLevel1 }
else
{ delimiterLevel2 }
val builder = new StringBuilder()
s.foreach
{ x => builder.append(getString(x, serializationNullFormat,
delimiterLevel1, delimiterLevel2, timeStampFormat, dateFormat, level +
1)).append(delimiter) }
make primitive data added a more char \ when datatype is complex
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tianlileer/incubator-carbondata dev
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-carbondata/pull/710.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #710
commit da638fb8fb9ee74c0846d16563b524decabfbf13
Author: tianli
Date: 2017-03-30T06:01:41Z
load data from dataframe,generater data row may be error by delimiterLevel1
or delimiterLevel2 is special character
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #705: [CARBONDATA-827] Fix query stats format err...
Github user chenliang613 commented on the issue: https://github.com/apache/incubator-carbondata/pull/705 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #705: [CARBONDATA-827] Fix query stats for...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/705 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
