[jira] [Created] (CARBONDATA-2318) Remove invalid table name(.ds_store) of presto integration
Liang Chen created CARBONDATA-2318: -- Summary: Remove invalid table name(.ds_store) of presto integration Key: CARBONDATA-2318 URL: https://issues.apache.org/jira/browse/CARBONDATA-2318 Project: CarbonData Issue Type: Improvement Components: presto-integration Reporter: Liang Chen For presto integration , will get the invalid table name via "show tables from default" As below. presto:default> show tables from default; Table .ds_store carbon_table carbontable partition_bigtable partition_table (5 rows) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CARBONDATA-2282) presto carbon does not support reading specific partition on which query is fired mapreduce.input.carboninputformat.partitions.to.prune property is null
[ https://issues.apache.org/jira/browse/CARBONDATA-2282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16429203#comment-16429203 ] Liang Chen commented on CARBONDATA-2282: ok > presto carbon does not support reading specific partition on which query is > fired mapreduce.input.carboninputformat.partitions.to.prune property is null > > > Key: CARBONDATA-2282 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2282 > Project: CarbonData > Issue Type: Bug > Components: core, presto-integration >Affects Versions: 1.3.0 >Reporter: zhangwei >Assignee: anubhav tarar >Priority: Major > Fix For: 1.3.0 > > Attachments: partitonToPrune.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2127: [CARBONDATA-2301][SDK] CarbonStore interface ...
Github user ravipesala commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2127#discussion_r179905093 --- Diff: store/sdk/src/main/java/org/apache/carbondata/store/LocalCarbonStore.java --- @@ -0,0 +1,111 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.carbondata.store; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.List; + +import org.apache.carbondata.common.annotations.InterfaceAudience; +import org.apache.carbondata.core.datastore.row.CarbonRow; +import org.apache.carbondata.core.metadata.schema.table.CarbonTable; +import org.apache.carbondata.core.scan.expression.Expression; +import org.apache.carbondata.hadoop.CarbonProjection; +import org.apache.carbondata.hadoop.api.CarbonTableInputFormat; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.mapreduce.InputSplit; +import org.apache.hadoop.mapreduce.Job; +import org.apache.hadoop.mapreduce.JobID; +import org.apache.hadoop.mapreduce.RecordReader; +import org.apache.hadoop.mapreduce.TaskAttemptID; +import org.apache.hadoop.mapreduce.task.JobContextImpl; +import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl; + +/** + * A CarbonStore implementation that works locally, without other compute framework dependency. + * It can be used to read data in local disk. + * + * Note that this class is experimental, it is not intended to be used in production. + */ +@InterfaceAudience.Internal +class LocalCarbonStore extends MetaCachedCarbonStore { + + @Override + public CarbonRow[] scan(String path, String[] projectColumns) throws IOException { +return scan(path, projectColumns, null); + } + + @Override + public CarbonRow[] scan(String path, String[] projectColumns, Expression filter) + throws IOException { +CarbonTable table = getTable(path); +if (table.isStreamingTable() || table.isHivePartitionTable()) { + throw new UnsupportedOperationException("streaming and partition table is not supported"); +} +// TODO: use InputFormat to prune data and read data + +final CarbonTableInputFormat format = new CarbonTableInputFormat(); +final Job job = new Job(new Configuration()); +format.setTableInfo(job.getConfiguration(), table.getTableInfo()); +format.setTablePath(job.getConfiguration(), table.getTablePath()); +format.setTableName(job.getConfiguration(), table.getTableName()); +format.setDatabaseName(job.getConfiguration(), table.getDatabaseName()); +format.setCarbonReadSupport(job.getConfiguration(), CarbonRowReadSupport.class); +if (filter != null) { + format.setFilterPredicates(job.getConfiguration(), filter); +} +if (projectColumns != null) { --- End diff -- what is the behavior if the user does not provide projections? ---
[GitHub] carbondata pull request #2127: [CARBONDATA-2301][SDK] CarbonStore interface ...
Github user ravipesala commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2127#discussion_r179905076 --- Diff: store/sdk/src/main/java/org/apache/carbondata/store/CarbonStore.java --- @@ -0,0 +1,67 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.carbondata.store; + +import java.io.Closeable; +import java.io.IOException; + +import org.apache.carbondata.common.annotations.InterfaceAudience; +import org.apache.carbondata.common.annotations.InterfaceStability; +import org.apache.carbondata.core.datastore.row.CarbonRow; +import org.apache.carbondata.core.scan.expression.Expression; + +/** + * User can use {@link CarbonStore} to query data + */ +@InterfaceAudience.User +@InterfaceStability.Unstable +public interface CarbonStore extends Closeable { + + /** + * Scan query on the data in the table path + * @param path table path + * @param projectColumns column names to read + * @return rows + * @throws IOException if unable to read files in table path + */ + CarbonRow[] scan( --- End diff -- I think it is better to return iterator rather than array, because in case of returning large rows it is not memory efficient to return all rows to driver. Internal implementation can have batch read from each executor and gives to drver. ---
[jira] [Comment Edited] (CARBONDATA-2282) presto carbon does not support reading specific partition on which query is fired mapreduce.input.carboninputformat.partitions.to.prune property is null
[ https://issues.apache.org/jira/browse/CARBONDATA-2282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16428473#comment-16428473 ] anubhav tarar edited comment on CARBONDATA-2282 at 4/6/18 3:55 PM: --- [~chenliang613] title of this issue was not completely correct by zhangwei i corrected it now as it can confuse and it is duplicated with my Jira https://issues.apache.org/jira/browse/CARBONDATA-2267 on which my pr is linked and has the correct ttile was (Author: anubhavtarar): [~chenliang613] title of this issue was not completely correct by zhangwei i corrected it now as it can confuse and it is duplicated with my Jira https://issues.apache.org/jira/browse/CARBONDATA-2267 on which my pr is linked > presto carbon does not support reading specific partition on which query is > fired mapreduce.input.carboninputformat.partitions.to.prune property is null > > > Key: CARBONDATA-2282 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2282 > Project: CarbonData > Issue Type: Bug > Components: core, presto-integration >Affects Versions: 1.3.0 >Reporter: zhangwei >Assignee: anubhav tarar >Priority: Major > Fix For: 1.3.0 > > Attachments: partitonToPrune.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CARBONDATA-2282) presto carbon does not support reading specific partition on which query is fired mapreduce.input.carboninputformat.partitions.to.prune property is null
[ https://issues.apache.org/jira/browse/CARBONDATA-2282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16428479#comment-16428479 ] anubhav tarar commented on CARBONDATA-2282: --- pr is already raised for this issue[ #2139|https://github.com/apache/carbondata/pull/2139] > presto carbon does not support reading specific partition on which query is > fired mapreduce.input.carboninputformat.partitions.to.prune property is null > > > Key: CARBONDATA-2282 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2282 > Project: CarbonData > Issue Type: Bug > Components: core, presto-integration >Affects Versions: 1.3.0 >Reporter: zhangwei >Assignee: anubhav tarar >Priority: Major > Fix For: 1.3.0 > > Attachments: partitonToPrune.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (CARBONDATA-2282) presto carbon does not support reading specific partition on which query is fired mapreduce.input.carboninputformat.partitions.to.prune property is null
[ https://issues.apache.org/jira/browse/CARBONDATA-2282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16428463#comment-16428463 ] anubhav tarar edited comment on CARBONDATA-2282 at 4/6/18 3:42 PM: --- [~chenliang613] In this Issue we are handling the scenario of querying data to standard hive partitioned table in carbondata. although you are getting the correct data from partition table but in current master branch we were iterating over all the partitions to get the data not the one which is required,which will be slow you can check value of property mapreduce.input.carboninputformat.partitions.to.prune in carboninputformat class after executing query on a particular partition it is null so when getPartitionsToPrune method is called it is returning null value instead of required partition specs but now in this pr we have identified the required partitions, so the query will be performed only on the required partition. Firstly we are identifying the partition columns through domain constraints in presto and after that we are adding the Partition Spec to the configuration object to query partition was (Author: anubhavtarar): [~chenliang613] In this PR we are handling the scenario of querying data to standard hive partitioned table in carbondata. although you are getting the correct data from partition table but in current master branch we were iterating over all the partitions to get the data not the one which is required,which will be slow you can check value of property mapreduce.input.carboninputformat.partitions.to.prune in carboninputformat class after executing query on a particular partition it is null so when getPartitionsToPrune method is called it is returning null value instead of required partition specs but now in this pr we have identified the required partitions, so the query will be performed only on the required partition. Firstly we are identifying the partition columns through domain constraints in presto and after that we are adding the Partition Spec to the configuration object to query partition > presto carbon does not support reading specific partition on which query is > fired mapreduce.input.carboninputformat.partitions.to.prune property is null > > > Key: CARBONDATA-2282 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2282 > Project: CarbonData > Issue Type: Bug > Components: core, presto-integration >Affects Versions: 1.3.0 >Reporter: zhangwei >Assignee: anubhav tarar >Priority: Major > Fix For: 1.3.0 > > Attachments: partitonToPrune.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CARBONDATA-2282) presto carbon does not support reading specific partition on which query is fired mapreduce.input.carboninputformat.partitions.to.prune property is null
[ https://issues.apache.org/jira/browse/CARBONDATA-2282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16428473#comment-16428473 ] anubhav tarar commented on CARBONDATA-2282: --- [~chenliang613] title of this issue was not completely correct by zhangwei i corrected it now as it can confuse and it is duplicated with my Jira https://issues.apache.org/jira/browse/CARBONDATA-2267 on which my pr is linked > presto carbon does not support reading specific partition on which query is > fired mapreduce.input.carboninputformat.partitions.to.prune property is null > > > Key: CARBONDATA-2282 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2282 > Project: CarbonData > Issue Type: Bug > Components: core, presto-integration >Affects Versions: 1.3.0 >Reporter: zhangwei >Assignee: anubhav tarar >Priority: Major > Fix For: 1.3.0 > > Attachments: partitonToPrune.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (CARBONDATA-2282) presto carbon does not support reading specific partition on which query is fired mapreduce.input.carboninputformat.partitions.to.prune property is null
[ https://issues.apache.org/jira/browse/CARBONDATA-2282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] anubhav tarar updated CARBONDATA-2282: -- Summary: presto carbon does not support reading specific partition on which query is fired mapreduce.input.carboninputformat.partitions.to.prune property is null (was: presto carbon does not support reading specific partition on which query is fired ) > presto carbon does not support reading specific partition on which query is > fired mapreduce.input.carboninputformat.partitions.to.prune property is null > > > Key: CARBONDATA-2282 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2282 > Project: CarbonData > Issue Type: Bug > Components: core, presto-integration >Affects Versions: 1.3.0 >Reporter: zhangwei >Assignee: anubhav tarar >Priority: Major > Fix For: 1.3.0 > > Attachments: partitonToPrune.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (CARBONDATA-2282) presto carbon does not support reading specific partition on which query is fired
[ https://issues.apache.org/jira/browse/CARBONDATA-2282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] anubhav tarar updated CARBONDATA-2282: -- Summary: presto carbon does not support reading specific partition on which query is fired (was: presto carbon not support query partition table) > presto carbon does not support reading specific partition on which query is > fired > -- > > Key: CARBONDATA-2282 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2282 > Project: CarbonData > Issue Type: Bug > Components: core, presto-integration >Affects Versions: 1.3.0 >Reporter: zhangwei >Assignee: anubhav tarar >Priority: Major > Fix For: 1.3.0 > > Attachments: partitonToPrune.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CARBONDATA-2282) presto carbon not support query partition table
[ https://issues.apache.org/jira/browse/CARBONDATA-2282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16428463#comment-16428463 ] anubhav tarar commented on CARBONDATA-2282: --- [~chenliang613] In this PR we are handling the scenario of querying data to standard hive partitioned table in carbondata. although you are getting the correct data from partition table but in current master branch we were iterating over all the partitions to get the data not the one which is required,which will be slow you can check value of property mapreduce.input.carboninputformat.partitions.to.prune in carboninputformat class after executing query on a particular partition it is null so when getPartitionsToPrune method is called it is returning null value instead of required partition specs but now in this pr we have identified the required partitions, so the query will be performed only on the required partition. Firstly we are identifying the partition columns through domain constraints in presto and after that we are adding the Partition Spec to the configuration object to query partition > presto carbon not support query partition table > --- > > Key: CARBONDATA-2282 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2282 > Project: CarbonData > Issue Type: Bug > Components: core, presto-integration >Affects Versions: 1.3.0 >Reporter: zhangwei >Assignee: anubhav tarar >Priority: Major > Fix For: 1.3.0 > > Attachments: partitonToPrune.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2143: [CARBONDATA-2317] oncurrent datamap with same name a...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2143 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4322/ ---
[GitHub] carbondata issue #2143: [CARBONDATA-2317] oncurrent datamap with same name a...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2143 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3631/ ---
[GitHub] carbondata issue #2143: [CARBONDATA-2317] oncurrent datamap with same name a...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2143 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4853/ ---
[GitHub] carbondata issue #2142: [CARBONDATA-2316] [WIP] Executor task is failed but ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2142 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4321/ ---
[GitHub] carbondata pull request #2143: [CARBONDATA-2317] oncurrent datamap with same...
GitHub user rahulforallp opened a pull request: https://github.com/apache/carbondata/pull/2143 [CARBONDATA-2317] oncurrent datamap with same name and schema creation throws exception â¦n throws exception Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rahulforallp/incubator-carbondata concur_preagg Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2143.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2143 commit c82a046849562dc61b761f3aaefff1653f585a0f Author: rahulforallp Date: 2018-04-06T09:47:54Z [CARBONDATA-2317] oncurrent datamap with same name and schema creation throws exception ---
[jira] [Created] (CARBONDATA-2317) concurrent datamap with same name and schema creation throws exception
Rahul Kumar created CARBONDATA-2317: --- Summary: concurrent datamap with same name and schema creation throws exception Key: CARBONDATA-2317 URL: https://issues.apache.org/jira/browse/CARBONDATA-2317 Project: CarbonData Issue Type: Improvement Reporter: Rahul Kumar Assignee: Rahul Kumar *Steps to reproduce :* # From Beeline user creates a table. # From 4 concurrent terminals user tries to create datamaps. *Query used to reproduce:* # CREATE TABLE uniqdata(CUST_ID int,CUST_NAME String,ACTIVE_EMUI_VERSION string, DOB timestamp, DOJ timestamp, BIGINT_COLUMN1 bigint,BIGINT_COLUMN2 bigint,DECIMAL_COLUMN1 decimal(30,10), DECIMAL_COLUMN2 decimal(36,10),Double_COLUMN1 double, Double_COLUMN2 double,INTEGER_COLUMN1 int) STORED BY 'org.apache.carbondata.format' TBLPROPERTIES('DICTIONARY_INCLUDE'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1'); # create datamap uniqdata_agg on table uniqdata using 'preaggregate' as select cust_name, avg(cust_id) from uniqdata group by cust_id, cust_name; # create datamap uniqdata_agg_sum on table uniqdata using 'preaggregate' as select cust_name, sum(cust_id) from uniqdata group by cust_id, cust_name; # create datamap uniqdata_agg_count on table uniqdata using 'preaggregate' as select cust_name, count(cust_id) from uniqdata group by cust_id, cust_name; # create datamap uniqdata_agg_min on table uniqdata using 'preaggregate' as select cust_name, min(cust_id) from uniqdata group by cust_id, cust_name; # create datamap uniqdata_agg_max on table uniqdata using 'preaggregate' as select cust_name, max(cust_id) from uniqdata group by cust_id, cust_name; --->*The datamaps are tried to be created from 4 concurrent terminals.* *2 of the datamaps fails with below error* {quote}0: jdbc:hive2://ha-cluster/default> create datamap uniqdata_agg_min on table uniqdata using 'preaggregate' as select cust_name, min(cust_id) from uniqdata group by cust_id, cust_name;Error: org.apache.carbondata.spark.exception.ProcessMetaDataException: operation failed for default.uniqdata_uniqdata_agg_min: Create table'uniqdata_uniqdata_agg_min' in database 'default' failed, File does not exist: /user/hive/warehouse/carbon.store/default/uniqdata_uniqdata_agg_min/Metadata/schema.write (inode 20219) Holder DFSClient_NONMAPREDUCE_1307577692_216 does not have any open files.at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2686)at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.completeFileInternal(FSDirWriteFileOp.java:625)at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.completeFile(FSDirWriteFileOp.java:605)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:2731)at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:883)at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:561)at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:447)at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:847)at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:790)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:422)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2486) (state=,code=0){quote} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2142: [CARBONDATA-2316] [WIP] Executor task is failed but ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2142 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3630/ ---
[GitHub] carbondata issue #2142: [CARBONDATA-2316] [WIP] Executor task is failed but ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2142 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4852/ ---
[GitHub] carbondata issue #2141: [CARBONDATA-2313] Added comment for SDK writer API m...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2141 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4851/ ---
[GitHub] carbondata issue #2141: [CARBONDATA-2313] Added comment for SDK writer API m...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2141 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4320/ ---
[GitHub] carbondata issue #2141: [CARBONDATA-2313] Added comment for SDK writer API m...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2141 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3629/ ---
[GitHub] carbondata pull request #2142: [CARBONDATA-2316] [WIP] Executor task is fail...
GitHub user rahulforallp opened a pull request: https://github.com/apache/carbondata/pull/2142 [CARBONDATA-2316] [WIP] Executor task is failed but UI shows success issue is fixed ⦠- [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? No - [ ] Document update required? No - [ ] Testing done - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rahulforallp/incubator-carbondata compact_task_faail_issue Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2142.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2142 commit 1ac4d350c4b405b0ff65cb04909b02e3a43d7737 Author: rahulforallp Date: 2018-04-06T08:46:06Z [CARBONDATA-2316] Executor task is failed but UI shows success issue fixed ---
[GitHub] carbondata issue #2139: [CARBONDATA-2267] [Presto] Support Reading CarbonDat...
Github user geetikagupta16 commented on the issue: https://github.com/apache/carbondata/pull/2139 In this PR we are handling the scenario of querying data to standard hive partitioned table in carbondata. Earlier we were iterating over all the partitions to get the data but now as we have identified the required partitions, so the query will be performed only on the required partition. Firstly we are identifying the partition columns through domain constraints in presto and after that we are adding the Partition Spec to the configuration object to query partition. ---
[GitHub] carbondata pull request #2141: Added comment for SDK writer API methods
GitHub user ajantha-bhat opened a pull request: https://github.com/apache/carbondata/pull/2141 Added comment for SDK writer API methods Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? NO - [ ] Any backward compatibility impacted? NO - [ ] Document update required? NO - [ ] Testing done. NA You can merge this pull request into a Git repository by running: $ git pull https://github.com/ajantha-bhat/carbondata unmanaged_table Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2141.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2141 commit 49970829d56b91f2a5ca6e46f3957b248cf9ccde Author: ajantha-bhat Date: 2018-04-06T08:40:38Z Added comment for SDK writer API methods ---
[jira] [Created] (CARBONDATA-2316) Even though one of the Compaction task failed at executor. All the executor task is showing success in UI and Job fails from driver.
Rahul Kumar created CARBONDATA-2316: --- Summary: Even though one of the Compaction task failed at executor. All the executor task is showing success in UI and Job fails from driver. Key: CARBONDATA-2316 URL: https://issues.apache.org/jira/browse/CARBONDATA-2316 Project: CarbonData Issue Type: Improvement Reporter: Rahul Kumar Assignee: Rahul Kumar *At UI:* *!http://dts.huawei.com/net/dts/fckeditor/download.ashx?Path=9wrYLnPvC4cVkbqrwtlHFbeC2%2fnVXKrZXwY7%2bglgnCQfywSBPcodImAcj29S40n0M8Rx%2bHUEMVQ%3d!* *At Executor :* !http://dts.huawei.com/net/dts/fckeditor/download.ashx?Path=9wrYLnPvC4cVkbqrwtlHFbeC2%2fnVXKrZXwY7%2bglgnCQfywSBPcodInHgG%2fMfoX2avMU6b0kZ9Jo%3d! *At driver* !http://dts.huawei.com/net/dts/fckeditor/download.ashx?Path=9wrYLnPvC4cVkbqrwtlHFbeC2%2fnVXKrZXwY7%2bglgnCQfywSBPcodIruNtISaQlvSdxWgQJLSrNA%3d! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (CARBONDATA-2315) DataLoad is showing success and failure message in log,when no data is loaded into table during LOAD
[ https://issues.apache.org/jira/browse/CARBONDATA-2315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manish Gupta resolved CARBONDATA-2315. -- Resolution: Fixed Fix Version/s: 1.3.2 1.4.0 > DataLoad is showing success and failure message in log,when no data is loaded > into table during LOAD > > > Key: CARBONDATA-2315 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2315 > Project: CarbonData > Issue Type: Improvement >Reporter: Indhumathi Muthumurugesh >Assignee: Indhumathi Muthumurugesh >Priority: Minor > Fix For: 1.4.0, 1.3.2 > > Time Spent: 1.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2140: [CARBONDATA-2315] success and failure log fix...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2140 ---
[GitHub] carbondata issue #2140: [CARBONDATA-2315] success and failure log fixed when...
Github user manishgupta88 commented on the issue: https://github.com/apache/carbondata/pull/2140 LGTM ---
[jira] [Commented] (CARBONDATA-2282) presto carbon not support query partition table
[ https://issues.apache.org/jira/browse/CARBONDATA-2282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16428065#comment-16428065 ] Liang Chen commented on CARBONDATA-2282: Just i tried: 1.create partition table as below in sparkshell scala> carbon.sql("create table partition_table(id Int,vin String, phonenumber Long, country String, area String, salary Int) PARTITIONED BY (logdate Date) STORED BY 'carbondata' TBLPROPERTIES('SORT_COLUMNS'='id,vin')") 2.Can query successfully in presto : presto:default> select * from partition_table; id | vin | phonenumber | country | area | salary | logdate +--+-+-+--++ 6 | A42258434831 | 125371346 | China | Asia | 10005 | 2016-12-13 7 | A42158475831 | 125371347 | UK | OutSpace | 10006 | 2016-12-13 8 | A41158494830 | 225371348 | China | Asia | 10007 | 2016-12-13 12 | A42151477823 | 425371312 | China | Asia | 10011 | 2016-12-13 14 | A42258434835 | 525371314 | China | Asia | 10013 | 2016-12-13 20 | A42151477824 | 225371320 | China | Asia | 10019 | 2016-12-13 22 | A42258434837 | 25371322 | China | Asia | 10021 | 2016-12-13 24 | A41158494839 | 625371324 | China | Asia | 10023 | 2016-12-13 15 | A42158475836 | 625371315 | UK | OutSpace | 10014 | 2014-05-15 16 | A41158494838 | 525371316 | China | Asia | 10015 | 2014-05-15 18 | A42158473832 | 325371318 | China | Asia | 10017 | 2014-05-15 19 | A42152474834 | 225371319 | US | America | 10018 | 2014-05-15 21 | A42158474137 | 325371321 | Japan | Asia | 10020 | 2014-05-15 23 | A42158475838 | 425371323 | UK | OutSpace | 10022 | 2014-05-15 25 | A41158494840 | 626381324 | Good | OutSpace | 10024 | 2014-05-15 26 | A41158494843 | 625378824 | NotGood | OutSpace | 10025 | 2014-05-15 1 | A42158424831 | 125371341 | China | Asia | 1 | 2016-02-12 2 | A42158473831 | 125371342 | China | Asia | 10001 | 2016-02-12 3 | A42152474832 | 125371343 | US | America | 10002 | 2016-02-12 4 | A42151477823 | 125371344 | China | OutSpace | 10003 | 2016-02-12 5 | A42158474135 | 125371345 | Japan | OutSpace | 10004 | 2016-02-12 9 | A42158424831 | 225371349 | China | OutSpace | 10008 | 2016-02-12 10 | A42158473830 | 225371310 | China | Asia | 10009 | 2016-02-12 11 | A42152474830 | 325371311 | US | America | 10010 | 2016-02-12 13 | A42158474133 | 325371313 | Japan | Asia | 10012 | 2016-02-12 17 | A42158424833 | 425371317 | China | Asia | 10016 | 2016-02-12 (26 rows) > presto carbon not support query partition table > --- > > Key: CARBONDATA-2282 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2282 > Project: CarbonData > Issue Type: Bug > Components: core, presto-integration >Affects Versions: 1.3.0 >Reporter: zhangwei >Assignee: anubhav tarar >Priority: Major > Fix For: 1.3.0 > > Attachments: partitonToPrune.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)