[GitHub] incubator-carbondata issue #551: [CARBONDATA-603] Fixed Date Filter issue
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/551 Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/709/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #551: [CARBONDATA-603] Fixed Date Filter issue
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/551 Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/708/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Resolved] (CARBONDATA-562) Carbon Context initialization is failed with spark 1.6.3
[ https://issues.apache.org/jira/browse/CARBONDATA-562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindra Pesala resolved CARBONDATA-562. Resolution: Fixed Fix Version/s: 1.0.0-incubating > Carbon Context initialization is failed with spark 1.6.3 > - > > Key: CARBONDATA-562 > URL: https://issues.apache.org/jira/browse/CARBONDATA-562 > Project: CarbonData > Issue Type: Bug > Components: core >Affects Versions: 0.2.0-incubating >Reporter: Babulal >Assignee: Naresh P R > Fix For: 1.0.0-incubating > > Time Spent: 40m > Remaining Estimate: 0h > > Carbon is compiled with below command > mvn clean -P spark-1.6,provided -Dspark.version=1.6.3 -DskipTests package > start carbon context in yarn client mode > bin/spark-shell --master yarn-client --executor-memory 2G --executor-cores > 2 --driver-memory 2G --num-executors 3 > __ > / __/__ ___ _/ /__ > _\ \/ _ \/ _ `/ __/ '_/ >/___/ .__/\_,_/_/ /_/\_\ version 1.6.3 > /_/ > Using Scala version 2.10.5 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_111) > Type in expressions to have them evaluated. > scala> import org.apache.spark.sql.CarbonContext; > import org.apache.spark.sql.CarbonContext > scala> val cc=new CarbonContext(sc,"hdfs://hacluser/opt/CarbonStore"); > e of dependencies) > scala.tools.reflect.ToolBoxError: reflective compilation has failed: > not enough arguments for constructor Optimizer: (conf: > org.apache.spark.sql.catalyst.CatalystConf)org.apache.spark.sql.catalyst.optimizer.Optimizer. > Unspecified value parameter conf. > at > scala.tools.reflect.ToolBoxFactory$ToolBoxImpl$ToolBoxGlobal.throwIfErrors(ToolBoxFactory.scala:314) > at > org.apache.spark.sql.CodeGenerateFactory.(CodeGenerateFactory.scala:32) > at > org.apache.spark.sql.CodeGenerateFactory$.init(CodeGenerateFactory.scala:67) > at org.apache.spark.sql.CarbonContext.(CarbonContext.scala:58) > at org.apache.spark.sql.CarbonContext.(CarbonContext.scala:52) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-carbondata pull request #558: [CARBONDATA-562] Supporting Spark 1....
Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/558 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #560: [WIP] Do not merge added log for the...
GitHub user mohammadshahidkhan reopened a pull request: https://github.com/apache/incubator-carbondata/pull/560 [WIP] Do not merge added log for the random failing test case Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like: `[CARBONDATA-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [ ] Replace `` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - What manual testing you have done? - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/mohammadshahidkhan/incubator-carbondata ut_fail_analysis Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/560.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #560 commit 70863ba19306f69c9a19e90b6eb2cc89a5c93e7a Author: mohammadshahidkhan Date: 2017-01-19T15:53:13Z Fixed random test case failure --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #560: [WIP] Do not merge added log for the...
Github user mohammadshahidkhan closed the pull request at: https://github.com/apache/incubator-carbondata/pull/560 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #560: [WIP] Do not merge added log for the random...
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/560 Build Failed with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/707/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #555: [WIP] All dictionary path issue
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/555 Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/706/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #560: [WIP] Do not merge added log for the random...
Github user ravipesala commented on the issue: https://github.com/apache/incubator-carbondata/pull/560 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (CARBONDATA-598) Not using tableName option in Create table command Shows Strange Behaviour
[
https://issues.apache.org/jira/browse/CARBONDATA-598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15831306#comment-15831306
]
Naresh P R commented on CARBONDATA-598:
---
I have analyzed this issue and found DatasourceAPI CarbonSource createRelation
is getting called from SchemaRelationProvider with options map and schema
information only. CarbonSource will not get table identifier provided in create
table using datasource command.
Hence its always required to pass tableName, dbName in options same as table
identifier while creating table using datasource.
Alternatively, we can use SQL statements to create carbon table, please refer
below link for the same.
https://github.com/apache/incubator-carbondata/blob/master/docs/ddl-operation-on-carbondata.md#create-table
> Not using tableName option in Create table command Shows Strange Behaviour
> ---
>
> Key: CARBONDATA-598
> URL: https://issues.apache.org/jira/browse/CARBONDATA-598
> Project: CarbonData
> Issue Type: Bug
> Components: spark-integration
>Affects Versions: 1.0.0-incubating
> Environment: spark 2.1
>Reporter: anubhav tarar
>Assignee: Naresh P R
>
> if you dont use the tableName Option when creating table with Bucketing it
> shows strange behaviour and do not validate any check
> here are the logs
> spark.sql("""CREATE TABLE t3219(ID Int)USING
> org.apache.spark.sql.CarbonSource
> OPTIONS("bucketnumber"="1","bucketcolumns"="id","tableName"="t3q21000")""");
> here file t3q21000 get created in hdfs
> which should not be created it is quite confusing
> either there should be a check that both table name in create table statement
> and tableName in Option both are same and if it is allowed it should valid
> all the checks
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CARBONDATA-667) after setting carbon property carbon.kettle.home in carbon.properties , while loading data, it is not referring to the carbon.properties file in carbonlib directory
Akash R Nilugal created CARBONDATA-667: -- Summary: after setting carbon property carbon.kettle.home in carbon.properties , while loading data, it is not referring to the carbon.properties file in carbonlib directory Key: CARBONDATA-667 URL: https://issues.apache.org/jira/browse/CARBONDATA-667 Project: CarbonData Issue Type: Bug Components: data-load Reporter: Akash R Nilugal Priority: Minor after setting carbon property carbon.kettle.home in carbon.properties , placed it in carbonlib directory. But while loading it is not referring to that path. and gives error as carboin.kettle.home is not set. when carbon.properties file is placed inside /conf/ directory , it works fine. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CARBONDATA-665) Comparision Failure occurs when we execute the same query in hive and Carbondata
[
https://issues.apache.org/jira/browse/CARBONDATA-665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
SWATI RAO updated CARBONDATA-665:
-
Attachment: Test_Data1.csv
Description:
Orderby is not working , so records are not coming in sequence as well there is
data difference and some values being stored as null
Data itself is stored incorrectly and is different from Hive
Spark version :1.6.2
Create 1 query : create table Test_Boundary (c1_int int,c2_Bigint
Bigint,c3_Decimal Decimal(38,30),c4_double double,c5_string string,c6_Timestamp
Timestamp,c7_Datatype_Desc string) STORED BY 'org.apache.carbondata.format'
Load 1 Query : LOAD DATA INPATH 'HDFS_URL/BabuStore/Data/Test_Data1.csv' INTO
table Test_Boundary
OPTIONS('DELIMITER'=',','QUOTECHAR'='','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='')
Create 2 query : create table Test_Boundary1 (c1_int int,c2_Bigint
Bigint,c3_Decimal Decimal(38,30),c4_double double,c5_string string,c6_Timestamp
Timestamp,c7_Datatype_Desc string) STORED BY 'org.apache.carbondata.format'
Load 2 query: LOAD DATA INPATH 'HDFS_URL/BabuStore/Data/Test_Data1.csv' INTO
table Test_Boundary1
OPTIONS('DELIMITER'=',','QUOTECHAR'='','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='')
Select Query :
select
c1_int,c2_Bigint,c3_Decimal,c4_double,c5_string,c6_Timestamp,c7_Datatype_Desc
from Test_Boundary where c2_bigint=c2_bigint
was:
Create 1 query : create table Test_Boundary (c1_int int,c2_Bigint
Bigint,c3_Decimal Decimal(38,30),c4_double double,c5_string string,c6_Timestamp
Timestamp,c7_Datatype_Desc string) STORED BY 'org.apache.carbondata.format'
Load 1 Query : LOAD DATA INPATH 'HDFS_URL/BabuStore/Data/Test_Data1.csv' INTO
table Test_Boundary
OPTIONS('DELIMITER'=',','QUOTECHAR'='','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='')
Create 2 query : create table Test_Boundary1 (c1_int int,c2_Bigint
Bigint,c3_Decimal Decimal(38,30),c4_double double,c5_string string,c6_Timestamp
Timestamp,c7_Datatype_Desc string) STORED BY 'org.apache.carbondata.format'
Load 2 query: LOAD DATA INPATH 'HDFS_URL/BabuStore/Data/Test_Data1.csv' INTO
table Test_Boundary1
OPTIONS('DELIMITER'=',','QUOTECHAR'='','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='')
Select Query :
select
c1_int,c2_Bigint,c3_Decimal,c4_double,c5_string,c6_Timestamp,c7_Datatype_Desc
from Test_Boundary where c2_bigint=c2_bigint
> Comparision Failure occurs when we execute the same query in hive and
> Carbondata
>
>
> Key: CARBONDATA-665
> URL: https://issues.apache.org/jira/browse/CARBONDATA-665
> Project: CarbonData
> Issue Type: Bug
>Reporter: SWATI RAO
> Attachments: Test_Data1.csv
>
>
> Orderby is not working , so records are not coming in sequence as well there
> is data difference and some values being stored as null
> Data itself is stored incorrectly and is different from Hive
> Spark version :1.6.2
> Create 1 query : create table Test_Boundary (c1_int int,c2_Bigint
> Bigint,c3_Decimal Decimal(38,30),c4_double double,c5_string
> string,c6_Timestamp Timestamp,c7_Datatype_Desc string) STORED BY
> 'org.apache.carbondata.format'
> Load 1 Query : LOAD DATA INPATH 'HDFS_URL/BabuStore/Data/Test_Data1.csv' INTO
> table Test_Boundary
> OPTIONS('DELIMITER'=',','QUOTECHAR'='','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='')
> Create 2 query : create table Test_Boundary1 (c1_int int,c2_Bigint
> Bigint,c3_Decimal Decimal(38,30),c4_double double,c5_string
> string,c6_Timestamp Timestamp,c7_Datatype_Desc string) STORED BY
> 'org.apache.carbondata.format'
> Load 2 query: LOAD DATA INPATH 'HDFS_URL/BabuStore/Data/Test_Data1.csv' INTO
> table Test_Boundary1
> OPTIONS('DELIMITER'=',','QUOTECHAR'='','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='')
> Select Query :
> select
> c1_int,c2_Bigint,c3_Decimal,c4_double,c5_string,c6_Timestamp,c7_Datatype_Desc
> from Test_Boundary where c2_bigint=c2_bigint
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] incubator-carbondata pull request #557: [WIP] Set carbon properties location...
Github user chenliang613 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/557#discussion_r97020867
--- Diff:
integration/spark2/src/main/scala/org/apache/carbondata/spark/thriftserver/CarbonThriftServer.scala
---
@@ -39,14 +42,15 @@ object CarbonThriftServer {
val sparkHome = System.getenv.get("SPARK_HOME")
if (null != sparkHome) {
- builder.config("carbon.properties.filepath",
-sparkHome + '/' + "conf" + '/' + "carbon.properties")
- System.setProperty("carbon.properties.filepath",
-sparkHome + '/' + "conf" + '/' + "carbon.properties")
+ val file = new File(sparkHome + '/' + "conf" + '/' +
"carbon.properties")
+ if (file.exists()) {
+builder.config("carbon.properties.filepath", file.getCanonicalPath)
+System.setProperty("carbon.properties.filepath",
file.getCanonicalPath)
+ }
}
-CarbonProperties.getInstance().addProperty("carbon.storelocation",
args.head)
+
CarbonProperties.getInstance().addProperty(CarbonCommonConstants.STORE_LOCATION,
args.head)
--- End diff --
CarbonCommonConstants.STORE_LOCATION also get value from
carbon.properties.carbon.storelocation, why need do this change ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #560: [WIP] Do not merge added log for the random...
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/560 Build Failed with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/705/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (CARBONDATA-646) Bad record handling is not correct for Int data type
[ https://issues.apache.org/jira/browse/CARBONDATA-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15831282#comment-15831282 ] Manish Gupta commented on CARBONDATA-646: - This issue is fixed as part of PR-535 (https://github.com/apache/incubator-carbondata/pull/535). This issue can be closed. I have tested and attached the snapshot of data and output. > Bad record handling is not correct for Int data type > > > Key: CARBONDATA-646 > URL: https://issues.apache.org/jira/browse/CARBONDATA-646 > Project: CarbonData > Issue Type: Bug >Affects Versions: 1.0.0-incubating, 0.1.1-incubating > Environment: Spark 1.6 >Reporter: Ramakrishna >Assignee: Manish Gupta >Priority: Minor > Attachments: 646_1.PNG, 646_2.PNG > > > With Bad record handling as default, > If Char value is given for Int data type, that is handled properly(moving > NULL). > If Decimal values is given for Int Data type, it is stripping the decimal, > where it should consider this as bad record and move NULL. > Bad record csv: > TRUE,2.7,423.0,A,2003454300, > 121.5,4.99,2.44,SE3423ee,asfdsffdfg,EtryTRWT,2012-01-12 > 03:14:05.123456729,2012-01-20 > 0: jdbc:hive2://172.168.100.212:23040> select * from t_carbn01 where > qty_total is NULL; > ++---+--++-+--+-+-++-+--++--+--+ > | active_status | item_type_cd | qty_day_avg | qty_total | sell_price > | sell_pricep | discount_price | profit | item_code | item_name | > outlet_name | update_time | create_date | > ++---+--++-+--+-+-++-+--++--+--+ > | TRUE | 2 | 423 | NULL | > 2003454304 | 121.5| 4.99| 2.44| SE3423ee | > asfdsffdfg | EtryTRWT | 2012-01-12 03:14:05.0 | 2012-01-20 | > ++---+--++-+--+-+-++-+--++-- > 0: jdbc:hive2://172.168.100.212:23040> desc t_carbn01; > +-+---+--+--+ > |col_name | data_type | comment | > +-+---+--+--+ > | active_status | string| | > | item_type_cd| bigint| | > | qty_day_avg | bigint| | > | qty_total | bigint| | > | sell_price | bigint| | > | sell_pricep | double| | > | discount_price | double| | > | profit | decimal(3,2) | | > | item_code | string| | > | item_name | string| | > | outlet_name | string| | > | update_time | timestamp | | > | create_date | string| | > +-+---+--+--+ > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-carbondata issue #558: [CARBONDATA-562] Supporting Spark 1.6.3 Ver...
Github user ravipesala commented on the issue: https://github.com/apache/incubator-carbondata/pull/558 LGTM. Verified and built locally, works fine. Thank you --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (CARBONDATA-663) Major compaction is not working properly as per the configuration
[
https://issues.apache.org/jira/browse/CARBONDATA-663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15831280#comment-15831280
]
ravikiran commented on CARBONDATA-663:
--
hi ,
Please find below the working of the Major compaction .
1. major compaction is a size based compaction. assume X is the size
configured . for example X = 10 mb
2. Major compaction will be done as long as the segments are inside this 10 mb
size. here the size considered is not the CSV input file size. size will be
calculated from the segments files i.e carbondata files , index files of a
segment.
3. if a size of 1 segment is above 10 mb then that segment wont be considered
for merging.
In the above description please check if the size is calculated properly.
> Major compaction is not working properly as per the configuration
> -
>
> Key: CARBONDATA-663
> URL: https://issues.apache.org/jira/browse/CARBONDATA-663
> Project: CarbonData
> Issue Type: Bug
> Components: data-query
>Affects Versions: 1.0.0-incubating
> Environment: Spark - 2.1
>Reporter: Anurag Srivastava
> Attachments: logs, sample_str_more1.csv, show_segment.png,
> show_segments_after_compaction.png
>
>
> I have set property *carbon.major.compaction.size= 3* and load data which is
> the size of 5 MB and when I perform compaction it compacted, but initially it
> shouldn't be perform. Here is the queries :
> *create table :* create table test_major_compaction(id Int,name string)stored
> by 'carbondata';
> *Load Data :* Load two segments.
> LOAD DATA inpath 'hdfs://localhost:54310/sample_str_more1.csv' INTO table
> test_major_compaction options('DELIMITER'=',', 'FILEHEADER'='id,
> name','QUOTECHAR'='"');
> *Show segments :* show segments for table test_major_compaction;
> !https://issues.apache.org/jira/secure/attachment/12848287/show_segment.png!
> *Alter Table :* ALTER TABLE test_major_compaction COMPACT 'MAJOR';
> *Show segments :* Again see the segments :
> show segments for table test_major_compaction;
> !https://issues.apache.org/jira/secure/attachment/12848286/show_segments_after_compaction.png!
> I have attached all the data with the it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CARBONDATA-646) Bad record handling is not correct for Int data type
[ https://issues.apache.org/jira/browse/CARBONDATA-646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manish Gupta updated CARBONDATA-646: Attachment: 646_2.PNG 646_1.PNG > Bad record handling is not correct for Int data type > > > Key: CARBONDATA-646 > URL: https://issues.apache.org/jira/browse/CARBONDATA-646 > Project: CarbonData > Issue Type: Bug >Affects Versions: 1.0.0-incubating, 0.1.1-incubating > Environment: Spark 1.6 >Reporter: Ramakrishna >Assignee: Manish Gupta >Priority: Minor > Attachments: 646_1.PNG, 646_2.PNG > > > With Bad record handling as default, > If Char value is given for Int data type, that is handled properly(moving > NULL). > If Decimal values is given for Int Data type, it is stripping the decimal, > where it should consider this as bad record and move NULL. > Bad record csv: > TRUE,2.7,423.0,A,2003454300, > 121.5,4.99,2.44,SE3423ee,asfdsffdfg,EtryTRWT,2012-01-12 > 03:14:05.123456729,2012-01-20 > 0: jdbc:hive2://172.168.100.212:23040> select * from t_carbn01 where > qty_total is NULL; > ++---+--++-+--+-+-++-+--++--+--+ > | active_status | item_type_cd | qty_day_avg | qty_total | sell_price > | sell_pricep | discount_price | profit | item_code | item_name | > outlet_name | update_time | create_date | > ++---+--++-+--+-+-++-+--++--+--+ > | TRUE | 2 | 423 | NULL | > 2003454304 | 121.5| 4.99| 2.44| SE3423ee | > asfdsffdfg | EtryTRWT | 2012-01-12 03:14:05.0 | 2012-01-20 | > ++---+--++-+--+-+-++-+--++-- > 0: jdbc:hive2://172.168.100.212:23040> desc t_carbn01; > +-+---+--+--+ > |col_name | data_type | comment | > +-+---+--+--+ > | active_status | string| | > | item_type_cd| bigint| | > | qty_day_avg | bigint| | > | qty_total | bigint| | > | sell_price | bigint| | > | sell_pricep | double| | > | discount_price | double| | > | profit | decimal(3,2) | | > | item_code | string| | > | item_name | string| | > | outlet_name | string| | > | update_time | timestamp | | > | create_date | string| | > +-+---+--+--+ > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-carbondata issue #555: [WIP] All dictionary path issue
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/555 Build Failed with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/704/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #398: [CARBONDATA-400] Error message for dataload...
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/398 Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/703/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (CARBONDATA-666) Select Query fails with NullPoint exception if using 2 or more logical operators
[
https://issues.apache.org/jira/browse/CARBONDATA-666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ramakrishna updated CARBONDATA-666:
---
Environment: Spark 1.6 (was: Carbon 1.6)
> Select Query fails with NullPoint exception if using 2 or more logical
> operators
>
>
> Key: CARBONDATA-666
> URL: https://issues.apache.org/jira/browse/CARBONDATA-666
> Project: CarbonData
> Issue Type: Bug
>Affects Versions: 1.0.0-incubating
> Environment: Spark 1.6
>Reporter: Ramakrishna
>Priority: Minor
>
> select * from communication.flow_carbon where cus_ac = '6222621350672465397'
> and txn_bk IN ('000', '001','002') OR own_bk IN
> ('124','175','034','231','167','182','206')
> limit 10;
> Error: org.apache.spark.SparkException: Job aborted due to stage failure:
> Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in
> stage 0.0 (TID 3, linux-79): java.lang.RuntimeException:
> java.util.concurrent.ExecutionException: java.lang.NullPointerException
> at
> org.apache.carbondata.scan.result.iterator.DetailQueryResultIterator.next(DetailQueryResultIterator.java:65)
> at
> org.apache.carbondata.scan.result.iterator.DetailQueryResultIterator.next(DetailQueryResultIterator.java:35)
> at
> org.apache.carbondata.scan.result.iterator.ChunkRowIterator.(ChunkRowIterator.java:43)
> at
> org.apache.carbondata.hadoop.CarbonRecordReader.initialize(CarbonRecordReader.java:81)
> at
> org.apache.carbondata.spark.rdd.CarbonScanRDD.compute(CarbonScanRDD.scala:194)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
> at org.apache.spark.scheduler.Task.run(Task.scala:89)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.NullPointerException
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.carbondata.scan.result.iterator.DetailQueryResultIterator.next(DetailQueryResultIterator.java:52)
> ... 24 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.carbondata.scan.result.AbstractScannedResult.getDictionaryKeyIntegerArray(AbstractScannedResult.java:186)
> at
> org.apache.carbondata.scan.result.impl.FilterQueryScannedResult.getDictionaryKeyIntegerArray(FilterQueryScannedResult.java:53)
> at
> org.apache.carbondata.scan.collector.impl.DictionaryBasedResultCollector.collectData(DictionaryBasedResultCollector.java:111)
> at
> org.apache.carbondata.scan.processor.impl.DataBlockIteratorImpl.next(DataBlockIteratorImpl.java:52)
> at
> org.apache.carbondata.scan.processor.impl.DataBlockIteratorImpl.next(DataBlockIteratorImpl.java:33)
> at
> org.apache.carbondata.scan.result.iterator.DetailQueryResultIterator$1.call(DetailQueryResultIterator.java:78)
> at
> org.apache.carbondata.scan.result.iterator.DetailQueryResultIterator$1.call(DetailQueryResultIterator.java:72)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CARBONDATA-666) Select Query fails with NullPoint exception if using 2 or more logical operators
Ramakrishna created CARBONDATA-666:
--
Summary: Select Query fails with NullPoint exception if using 2 or
more logical operators
Key: CARBONDATA-666
URL: https://issues.apache.org/jira/browse/CARBONDATA-666
Project: CarbonData
Issue Type: Bug
Affects Versions: 1.0.0-incubating
Environment: Carbon 1.6
Reporter: Ramakrishna
Priority: Minor
select * from communication.flow_carbon where cus_ac = '6222621350672465397'
and txn_bk IN ('000', '001','002') OR own_bk IN
('124','175','034','231','167','182','206')
limit 10;
Error: org.apache.spark.SparkException: Job aborted due to stage failure: Task
0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0
(TID 3, linux-79): java.lang.RuntimeException:
java.util.concurrent.ExecutionException: java.lang.NullPointerException
at
org.apache.carbondata.scan.result.iterator.DetailQueryResultIterator.next(DetailQueryResultIterator.java:65)
at
org.apache.carbondata.scan.result.iterator.DetailQueryResultIterator.next(DetailQueryResultIterator.java:35)
at
org.apache.carbondata.scan.result.iterator.ChunkRowIterator.(ChunkRowIterator.java:43)
at
org.apache.carbondata.hadoop.CarbonRecordReader.initialize(CarbonRecordReader.java:81)
at
org.apache.carbondata.spark.rdd.CarbonScanRDD.compute(CarbonScanRDD.scala:194)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException:
java.lang.NullPointerException
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
org.apache.carbondata.scan.result.iterator.DetailQueryResultIterator.next(DetailQueryResultIterator.java:52)
... 24 more
Caused by: java.lang.NullPointerException
at
org.apache.carbondata.scan.result.AbstractScannedResult.getDictionaryKeyIntegerArray(AbstractScannedResult.java:186)
at
org.apache.carbondata.scan.result.impl.FilterQueryScannedResult.getDictionaryKeyIntegerArray(FilterQueryScannedResult.java:53)
at
org.apache.carbondata.scan.collector.impl.DictionaryBasedResultCollector.collectData(DictionaryBasedResultCollector.java:111)
at
org.apache.carbondata.scan.processor.impl.DataBlockIteratorImpl.next(DataBlockIteratorImpl.java:52)
at
org.apache.carbondata.scan.processor.impl.DataBlockIteratorImpl.next(DataBlockIteratorImpl.java:33)
at
org.apache.carbondata.scan.result.iterator.DetailQueryResultIterator$1.call(DetailQueryResultIterator.java:78)
at
org.apache.carbondata.scan.result.iterator.DetailQueryResultIterator$1.call(DetailQueryResultIterator.java:72)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CARBONDATA-596) unable to crete filter resolver tree exception when using equals expression with carbon input format api
[
https://issues.apache.org/jira/browse/CARBONDATA-596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15831263#comment-15831263
]
anubhav tarar commented on CARBONDATA-596:
--
i am not able to reproduce this bug i am closing it
> unable to crete filter resolver tree exception when using equals expression
> with carbon input format api
>
>
> Key: CARBONDATA-596
> URL: https://issues.apache.org/jira/browse/CARBONDATA-596
> Project: CarbonData
> Issue Type: Bug
> Components: hadoop-integration
>Affects Versions: 1.0.0-incubating
> Environment: Spark 2.1 Spark 1.6
>Reporter: anubhav tarar
>Assignee: Manish Gupta
>Priority: Minor
>
> unable to crete filter resolver tree exception when using equals expression
> Expression expression = new EqualToExpression(new ColumnExpression("c1",
> DataType.STRING),
> new LiteralExpression("a", DataType.STRING));
> CarbonInputFormat.setFilterPredicates(job.getConfiguration(),
> expression);
> List splits = carbonInputFormat.getSplits(job);
> it throws java.io.IOException: Error while resolving filter expression
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (CARBONDATA-596) unable to crete filter resolver tree exception when using equals expression with carbon input format api
[
https://issues.apache.org/jira/browse/CARBONDATA-596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
anubhav tarar closed CARBONDATA-596.
Resolution: Fixed
this bug is not reproduced
> unable to crete filter resolver tree exception when using equals expression
> with carbon input format api
>
>
> Key: CARBONDATA-596
> URL: https://issues.apache.org/jira/browse/CARBONDATA-596
> Project: CarbonData
> Issue Type: Bug
> Components: hadoop-integration
>Affects Versions: 1.0.0-incubating
> Environment: Spark 2.1 Spark 1.6
>Reporter: anubhav tarar
>Assignee: Manish Gupta
>Priority: Minor
>
> unable to crete filter resolver tree exception when using equals expression
> Expression expression = new EqualToExpression(new ColumnExpression("c1",
> DataType.STRING),
> new LiteralExpression("a", DataType.STRING));
> CarbonInputFormat.setFilterPredicates(job.getConfiguration(),
> expression);
> List splits = carbonInputFormat.getSplits(job);
> it throws java.io.IOException: Error while resolving filter expression
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (CARBONDATA-600) Should reuse unit test case for integration module
[ https://issues.apache.org/jira/browse/CARBONDATA-600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen closed CARBONDATA-600. - Resolution: Won't Fix > Should reuse unit test case for integration module > -- > > Key: CARBONDATA-600 > URL: https://issues.apache.org/jira/browse/CARBONDATA-600 > Project: CarbonData > Issue Type: Bug > Components: spark-integration >Affects Versions: 1.0.0-incubating >Reporter: QiangCai >Assignee: QiangCai >Priority: Minor > Fix For: 1.0.0-incubating > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (CARBONDATA-287) Save the sorted temp files to multi local dirs to improve dataloading perfomance
[ https://issues.apache.org/jira/browse/CARBONDATA-287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen resolved CARBONDATA-287. --- Resolution: Fixed > Save the sorted temp files to multi local dirs to improve dataloading > perfomance > - > > Key: CARBONDATA-287 > URL: https://issues.apache.org/jira/browse/CARBONDATA-287 > Project: CarbonData > Issue Type: Improvement > Components: data-load >Affects Versions: 0.2.0-incubating >Reporter: QiangCai >Assignee: QiangCai >Priority: Minor > Fix For: 1.0.0-incubating > > Time Spent: 20m > Remaining Estimate: 0h > > Now for each dataloading, we use only a different local dir to save the > sorted temp files. I think it is neccessary to use multi local dirs for each > dataloading to improve dataloading performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CARBONDATA-303) 8. Add CarbonTableOutpuFormat to write data to carbon.
[ https://issues.apache.org/jira/browse/CARBONDATA-303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen updated CARBONDATA-303: -- Fix Version/s: (was: 1.0.0-incubating) 1.1.0-incubating > 8. Add CarbonTableOutpuFormat to write data to carbon. > -- > > Key: CARBONDATA-303 > URL: https://issues.apache.org/jira/browse/CARBONDATA-303 > Project: CarbonData > Issue Type: Sub-task >Reporter: Ravindra Pesala > Fix For: 1.1.0-incubating > > > Add CarbonTableOutpuFormat to write data to carbon. It should use > DataProcessorStep interface to load the data. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CARBONDATA-45) Support MAP type
[ https://issues.apache.org/jira/browse/CARBONDATA-45?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen updated CARBONDATA-45: - Fix Version/s: (was: 1.0.0-incubating) NONE > Support MAP type > > > Key: CARBONDATA-45 > URL: https://issues.apache.org/jira/browse/CARBONDATA-45 > Project: CarbonData > Issue Type: New Feature >Reporter: cen yuhai >Assignee: Vimal Das Kammath > Fix For: NONE > > > We have many tables which use map type, and general file format orc and > parquet support map type. So can carbondata support map type? > As sql "select map['id'] from table", orc will read all keys in map type. Can > we just read key 'id' ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CARBONDATA-631) Select,Delete and Insert Query Failing for table created in 0.2 with data loaded in 1.0
[
https://issues.apache.org/jira/browse/CARBONDATA-631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liang Chen updated CARBONDATA-631:
--
Fix Version/s: (was: 0.1.0-incubating)
NONE
> Select,Delete and Insert Query Failing for table created in 0.2 with data
> loaded in 1.0
> ---
>
> Key: CARBONDATA-631
> URL: https://issues.apache.org/jira/browse/CARBONDATA-631
> Project: CarbonData
> Issue Type: Bug
> Environment: Spark 1.6
>Reporter: Pallavi Singh
>Assignee: kumar vishal
> Fix For: NONE
>
>
> Created table with the 0.2 jar:
> CREATE TABLE uniqdata (CUST_ID int,CUST_NAME String,ACTIVE_EMUI_VERSION
> string, DOB timestamp, DOJ timestamp, BIGINT_COLUMN1 bigint,BIGINT_COLUMN2
> bigint,DECIMAL_COLUMN1 decimal(30,10), DECIMAL_COLUMN2
> decimal(36,10),Double_COLUMN1 double, Double_COLUMN2 double,INTEGER_COLUMN1
> int) STORED BY 'org.apache.carbondata.format' TBLPROPERTIES
> ("TABLE_BLOCKSIZE"= "256 MB");
> then
> LOAD DATA INPATH 'hdfs://localhost:54310/csv/2000_UniqData.csv' into table
> uniqdata OPTIONS('DELIMITER'=',' ,
> 'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1');
> Switched to 1.0 jar
> LOAD DATA INPATH 'hdfs://localhost:54310/csv/2000_UniqData.csv' into table
> uniqdata OPTIONS('DELIMITER'=',' ,
> 'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1');
> After successful load :
> select count(*) from uniqdata;
> I get following error :
> INFO 12-01 18:31:04,057 - Running query 'select count(*) from uniqdata' with
> 81129cf3-fcd4-429d-9adf-d37d35cdf051
> INFO 12-01 18:31:04,058 - pool-27-thread-46 Query [SELECT COUNT(*) FROM
> UNIQDATA]
> INFO 12-01 18:31:04,060 - Parsing command: select count(*) from uniqdata
> INFO 12-01 18:31:04,060 - Parse Completed
> INFO 12-01 18:31:04,061 - Parsing command: select count(*) from uniqdata
> INFO 12-01 18:31:04,061 - Parse Completed
> INFO 12-01 18:31:04,061 - 27: get_table : db=12jan17 tbl=uniqdata
> INFO 12-01 18:31:04,061 - ugi=pallaviip=unknown-ip-addr
> cmd=get_table : db=12jan17 tbl=uniqdata
> INFO 12-01 18:31:04,061 - 27: Opening raw store with implemenation
> class:org.apache.hadoop.hive.metastore.ObjectStore
> INFO 12-01 18:31:04,063 - ObjectStore, initialize called
> INFO 12-01 18:31:04,068 - Reading in results for query
> "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is
> closing
> INFO 12-01 18:31:04,069 - Using direct SQL, underlying DB is DERBY
> INFO 12-01 18:31:04,069 - Initialized ObjectStore
> INFO 12-01 18:31:04,101 - pool-27-thread-46 Starting to optimize plan
> ERROR 12-01 18:31:04,168 - pool-27-thread-46 Cannot convert12-01-2017
> 16:02:28 to Time/Long type valueUnparseable date: "12-01-2017 16:02:28"
> ERROR 12-01 18:31:04,185 - pool-27-thread-46 Cannot convert12-01-2017
> 16:02:08 to Time/Long type valueUnparseable date: "12-01-2017 16:02:08"
> ERROR 12-01 18:31:04,185 - pool-27-thread-46 Cannot convert12-01-2017
> 16:02:08 to Time/Long type valueUnparseable date: "12-01-2017 16:02:08"
> ERROR 12-01 18:31:04,204 - pool-27-thread-46 Cannot convert12-01-2017
> 16:02:08 to Time/Long type valueUnparseable date: "12-01-2017 16:02:08"
> ERROR 12-01 18:31:04,210 - Error executing query, currentState RUNNING,
> org.apache.spark.sql.catalyst.errors.package$TreeNodeException: execute, tree:
> CarbonDictionaryDecoder [CarbonDecoderRelation(Map(dob#280 -> dob#280,
> double_column1#287 -> double_column1#287, decimal_column1#285 ->
> decimal_column1#285, cust_id#282L -> cust_id#282L, integer_column1#289L ->
> integer_column1#289L, decimal_column2#286 -> decimal_column2#286,
> cust_name#278 -> cust_name#278, double_column2#288 -> double_column2#288,
> active_emui_version#279 -> active_emui_version#279, bigint_column1#283L ->
> bigint_column1#283L, bigint_column2#284L -> bigint_column2#284L, doj#281 ->
> doj#281),CarbonDatasourceRelation(`12jan17`.`uniqdata`,None))],
> ExcludeProfile(ArrayBuffer()), CarbonAliasDecoderRelation()
> +- TungstenAggregate(key=[],
> functions=[(count(1),mode=Final,isDistinct=false)], output=[_c0#750L])
>+- TungstenExchange SinglePartition, None
> +- TungstenAggregate(key=[],
> functions=[(count(1),mode=Partial,isDistinct=false)], output=[count#754L])
> +- CarbonScan CarbonRelation 12jan17, uniqdata,
> CarbonMetaData(ArrayBuffer(cust_name, active_emui_version, dob,
> doj),ArrayBuffer(cust_id, bigint_column1, bigint_column2, decimal_
[jira] [Resolved] (CARBONDATA-2) Remove kettle for loading data
[ https://issues.apache.org/jira/browse/CARBONDATA-2?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen resolved CARBONDATA-2. - Resolution: Fixed > Remove kettle for loading data > -- > > Key: CARBONDATA-2 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2 > Project: CarbonData > Issue Type: Improvement > Components: data-load >Reporter: Liang Chen >Assignee: Ravindra Pesala >Priority: Critical > Labels: features > Fix For: 1.0.0-incubating > > Attachments: CarbonDataLoadingdesign.pdf > > Time Spent: 5h 10m > Remaining Estimate: 0h > > Remove kettle for loading data module -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CARBONDATA-307) Support executor side scan using CarbonInputFormat
[ https://issues.apache.org/jira/browse/CARBONDATA-307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen updated CARBONDATA-307: -- Fix Version/s: (was: 1.0.0-incubating) 1.1.0-incubating > Support executor side scan using CarbonInputFormat > -- > > Key: CARBONDATA-307 > URL: https://issues.apache.org/jira/browse/CARBONDATA-307 > Project: CarbonData > Issue Type: Improvement > Components: spark-integration >Affects Versions: 0.1.0-incubating >Reporter: Jacky Li > Fix For: 1.1.0-incubating > > > Currently, there are two read path in carbon-spark module: > 1. CarbonContext => CarbonDatasourceRelation => CarbonScanRDD => QueryExecutor > In this case, CarbonScanRDD uses CarbonInputFormat to get the split, and use > QueryExecutor for scan. > 2. SqlContext => CarbonDatasourceHadoopRelation => CarbonHadoopFSRDD => > CarbonInputFormat(CarbonRecordReader) => QueryExecutor > In this case, CarbonHadoopFSRDD uses CarbonInputFormat to do both get split > and scan > Because of this, there are unnecessary duplicate code, they need to be > unified. > The target approach should be: > sqlContext/carbonContext => CarbonDatasourceHadoopRelation => CarbonScanRDD > => CarbonInputFormat(CarbonRecordReader) => QueryExecutor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CARBONDATA-309) Support two types of ReadSupport in CarbonRecordReader
[ https://issues.apache.org/jira/browse/CARBONDATA-309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen updated CARBONDATA-309: -- Fix Version/s: (was: 1.0.0-incubating) 1.1.0-incubating > Support two types of ReadSupport in CarbonRecordReader > -- > > Key: CARBONDATA-309 > URL: https://issues.apache.org/jira/browse/CARBONDATA-309 > Project: CarbonData > Issue Type: Sub-task > Components: spark-integration >Reporter: Jacky Li > Fix For: 1.1.0-incubating > > > CarbonRecordReader should support late decode based on passed Configuration > A config indicating late decode need to be added in CarbonInputFormat for > this purpose. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CARBONDATA-313) Update CarbonSource to use CarbonDatasourceHadoopRelation
[ https://issues.apache.org/jira/browse/CARBONDATA-313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen updated CARBONDATA-313: -- Fix Version/s: (was: 1.0.0-incubating) 1.1.0-incubating > Update CarbonSource to use CarbonDatasourceHadoopRelation > - > > Key: CARBONDATA-313 > URL: https://issues.apache.org/jira/browse/CARBONDATA-313 > Project: CarbonData > Issue Type: Sub-task > Components: spark-integration >Reporter: Jacky Li > Fix For: 1.1.0-incubating > > > Change CarbonSource to use CarbonDatasourceHadoopRelation only, remove > extension of BaseRelation, extend from HadoopFsRelation only -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CARBONDATA-312) Unify two datasource: CarbonDatasourceHadoopRelation and CarbonDatasourceRelation
[ https://issues.apache.org/jira/browse/CARBONDATA-312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen updated CARBONDATA-312: -- Fix Version/s: (was: 1.0.0-incubating) 1.1.0-incubating > Unify two datasource: CarbonDatasourceHadoopRelation and > CarbonDatasourceRelation > - > > Key: CARBONDATA-312 > URL: https://issues.apache.org/jira/browse/CARBONDATA-312 > Project: CarbonData > Issue Type: Sub-task > Components: spark-integration >Reporter: Jacky Li > Fix For: 1.1.0-incubating > > > Take CarbonDatasourceHadoopRelation as the target datasource definition, > after that, CarbonContext can use standard Datasource strategy > Since CarbonHadoopFSRDD need to be removed, and it is used by > CarbonDatasourceHadoopRelation. So we need to change > CarbonDatasourceHadoopRelation.buildScan function to return CarbonScanRDD > instead CarbonHadoopFSRDD, then CarbonHadoopFSRDD can be removed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CARBONDATA-314) Make CarbonContext to use standard Datasource strategy
[ https://issues.apache.org/jira/browse/CARBONDATA-314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen updated CARBONDATA-314: -- Fix Version/s: (was: 1.0.0-incubating) 1.1.0-incubating > Make CarbonContext to use standard Datasource strategy > -- > > Key: CARBONDATA-314 > URL: https://issues.apache.org/jira/browse/CARBONDATA-314 > Project: CarbonData > Issue Type: Sub-task > Components: spark-integration >Reporter: Jacky Li > Fix For: 1.1.0-incubating > > > Move the dictionary stratey out of CarbonTableScan, make a separate strategy > for it. > Then make CarbonContext use standard datasource strategy for creation of > relation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (CARBONDATA-322) Integration with spark 2.x
[ https://issues.apache.org/jira/browse/CARBONDATA-322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen resolved CARBONDATA-322. --- Resolution: Fixed > Integration with spark 2.x > > > Key: CARBONDATA-322 > URL: https://issues.apache.org/jira/browse/CARBONDATA-322 > Project: CarbonData > Issue Type: New Feature > Components: spark-integration >Affects Versions: 0.2.0-incubating >Reporter: Fei Wang >Assignee: Fei Wang > Fix For: 1.0.0-incubating > > > Since spark 2.0 released. there are many nice features such as more efficient > parser, vectorized execution, adaptive execution. > It is good to integrate with spark 2.x > current integration up to Spark v1.6 is tightly coupled with spark, we would > like to cleanup the interface with following design points in mind: > 1. decoupled with Spark, integration based on Spark's v2 datasource API > 2. Enable vectorized carbon reader > 3. Support saving DataFrame to Carbondata file through Carbondata's output > format. > ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (CARBONDATA-440) Provide Update/Delete functionality support in CarbonData
[ https://issues.apache.org/jira/browse/CARBONDATA-440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen resolved CARBONDATA-440. --- Resolution: Fixed > Provide Update/Delete functionality support in CarbonData > - > > Key: CARBONDATA-440 > URL: https://issues.apache.org/jira/browse/CARBONDATA-440 > Project: CarbonData > Issue Type: New Feature > Components: core, data-query, file-format, spark-integration, sql >Affects Versions: 0.1.1-incubating >Reporter: Aniket Adnaik >Assignee: Aniket Adnaik > Fix For: 1.0.0-incubating > > Attachments: ApacheCarbonData_UpdateDelete_Support_jira440.pdf > > Time Spent: 13h > Remaining Estimate: 0h > > Currently, CarbonData does not support modification of existing rows in the > table. This is a major limitation for many real world desirable use cases in > data warehousing, such as slow changing dimension tables, data correction of > fact tables or data cleanup, etc. Many users want to be able to update and > delete rows from the CarbonData table. > Following are some high level design goals to support this functionality, > 1. Provide a standard SQL interface to perform Update and Delete operations. > 2. Perform Update and Delete operations on CarbonData table without having to > rewrite the entire CarbonData block (file) by making use of differential > files (a.k.a delta files). > 3. After Update or Delete operation, CarbonData readers should skip deleted > records and read updated records seamlessly without having to modify user > applications. > 4. Perform compaction of Update and Delete delta files to maintain adequate > query performance -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CARBONDATA-440) Provide Update/Delete functionality support in CarbonData
[ https://issues.apache.org/jira/browse/CARBONDATA-440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen updated CARBONDATA-440: -- Assignee: Aniket Adnaik > Provide Update/Delete functionality support in CarbonData > - > > Key: CARBONDATA-440 > URL: https://issues.apache.org/jira/browse/CARBONDATA-440 > Project: CarbonData > Issue Type: New Feature > Components: core, data-query, file-format, spark-integration, sql >Affects Versions: 0.1.1-incubating >Reporter: Aniket Adnaik >Assignee: Aniket Adnaik > Fix For: 1.0.0-incubating > > Attachments: ApacheCarbonData_UpdateDelete_Support_jira440.pdf > > Time Spent: 13h > Remaining Estimate: 0h > > Currently, CarbonData does not support modification of existing rows in the > table. This is a major limitation for many real world desirable use cases in > data warehousing, such as slow changing dimension tables, data correction of > fact tables or data cleanup, etc. Many users want to be able to update and > delete rows from the CarbonData table. > Following are some high level design goals to support this functionality, > 1. Provide a standard SQL interface to perform Update and Delete operations. > 2. Perform Update and Delete operations on CarbonData table without having to > rewrite the entire CarbonData block (file) by making use of differential > files (a.k.a delta files). > 3. After Update or Delete operation, CarbonData readers should skip deleted > records and read updated records seamlessly without having to modify user > applications. > 4. Perform compaction of Update and Delete delta files to maintain adequate > query performance -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CARBONDATA-440) Provide Update/Delete functionality support in CarbonData
[ https://issues.apache.org/jira/browse/CARBONDATA-440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen updated CARBONDATA-440: -- Affects Version/s: (was: 0.2.0-incubating) (was: 0.1.0-incubating) > Provide Update/Delete functionality support in CarbonData > - > > Key: CARBONDATA-440 > URL: https://issues.apache.org/jira/browse/CARBONDATA-440 > Project: CarbonData > Issue Type: New Feature > Components: core, data-query, file-format, spark-integration, sql >Affects Versions: 0.1.1-incubating >Reporter: Aniket Adnaik > Fix For: 1.0.0-incubating > > Attachments: ApacheCarbonData_UpdateDelete_Support_jira440.pdf > > Time Spent: 13h > Remaining Estimate: 0h > > Currently, CarbonData does not support modification of existing rows in the > table. This is a major limitation for many real world desirable use cases in > data warehousing, such as slow changing dimension tables, data correction of > fact tables or data cleanup, etc. Many users want to be able to update and > delete rows from the CarbonData table. > Following are some high level design goals to support this functionality, > 1. Provide a standard SQL interface to perform Update and Delete operations. > 2. Perform Update and Delete operations on CarbonData table without having to > rewrite the entire CarbonData block (file) by making use of differential > files (a.k.a delta files). > 3. After Update or Delete operation, CarbonData readers should skip deleted > records and read updated records seamlessly without having to modify user > applications. > 4. Perform compaction of Update and Delete delta files to maintain adequate > query performance -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (CARBONDATA-443) Enable non-sort data loading
[ https://issues.apache.org/jira/browse/CARBONDATA-443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen resolved CARBONDATA-443. --- Resolution: Fixed > Enable non-sort data loading > > > Key: CARBONDATA-443 > URL: https://issues.apache.org/jira/browse/CARBONDATA-443 > Project: CarbonData > Issue Type: Improvement > Components: data-load >Affects Versions: 1.0.0-incubating >Reporter: QiangCai >Assignee: QiangCai > Fix For: 1.0.0-incubating > > Time Spent: 0.5h > Remaining Estimate: 0h > > Improving data ingestion rate for fast ingestion for special use cases with > potential side-effect of compromising query performance -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (CARBONDATA-495) Unify compressor interface
[ https://issues.apache.org/jira/browse/CARBONDATA-495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen resolved CARBONDATA-495. --- Resolution: Fixed > Unify compressor interface > -- > > Key: CARBONDATA-495 > URL: https://issues.apache.org/jira/browse/CARBONDATA-495 > Project: CarbonData > Issue Type: Improvement >Affects Versions: 0.2.0-incubating >Reporter: Jacky Li >Assignee: Jacky Li > Fix For: 1.0.0-incubating > > Time Spent: 40m > Remaining Estimate: 0h > > Use compressor factory to unify the interface and eliminate small objects -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (CARBONDATA-478) Separate SparkRowReadSupportImpl implementation for integrating with Spark1.x vs. Spark 2.x
[ https://issues.apache.org/jira/browse/CARBONDATA-478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen resolved CARBONDATA-478. --- Resolution: Fixed > Separate SparkRowReadSupportImpl implementation for integrating with Spark1.x > vs. Spark 2.x > --- > > Key: CARBONDATA-478 > URL: https://issues.apache.org/jira/browse/CARBONDATA-478 > Project: CarbonData > Issue Type: New Feature > Components: data-query >Affects Versions: 0.2.0-incubating >Reporter: QiangCai >Assignee: QiangCai > Fix For: 1.0.0-incubating > > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (CARBONDATA-539) Return empty row in map reduce application
[ https://issues.apache.org/jira/browse/CARBONDATA-539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen resolved CARBONDATA-539. --- Resolution: Fixed > Return empty row in map reduce application > -- > > Key: CARBONDATA-539 > URL: https://issues.apache.org/jira/browse/CARBONDATA-539 > Project: CarbonData > Issue Type: Bug >Reporter: Jacky Li >Assignee: Jacky Li > Fix For: 1.0.0-incubating > > Time Spent: 1h 50m > Remaining Estimate: 0h > > There is a bug that Carbon will return empty row in map reduce app if > projection columns are not set. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CARBONDATA-565) Clean up code suggested by IDE analyzer
[ https://issues.apache.org/jira/browse/CARBONDATA-565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen updated CARBONDATA-565: -- Fix Version/s: (was: 1.0.0-incubating) 1.1.0 > Clean up code suggested by IDE analyzer > --- > > Key: CARBONDATA-565 > URL: https://issues.apache.org/jira/browse/CARBONDATA-565 > Project: CarbonData > Issue Type: Improvement >Reporter: Jacky Li >Assignee: Jacky Li > Fix For: 1.1.0 > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (CARBONDATA-605) Add Update-delete related documentation
[ https://issues.apache.org/jira/browse/CARBONDATA-605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen resolved CARBONDATA-605. --- Resolution: Fixed > Add Update-delete related documentation > --- > > Key: CARBONDATA-605 > URL: https://issues.apache.org/jira/browse/CARBONDATA-605 > Project: CarbonData > Issue Type: Sub-task > Components: docs >Affects Versions: 1.0.0-incubating >Reporter: Venkata Ramana G >Assignee: Gururaj Shetty >Priority: Minor > Fix For: 1.0.0-incubating > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CARBONDATA-605) Add Update-delete related documentation
[ https://issues.apache.org/jira/browse/CARBONDATA-605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen updated CARBONDATA-605: -- Assignee: Gururaj Shetty > Add Update-delete related documentation > --- > > Key: CARBONDATA-605 > URL: https://issues.apache.org/jira/browse/CARBONDATA-605 > Project: CarbonData > Issue Type: Sub-task > Components: docs >Affects Versions: 1.0.0-incubating >Reporter: Venkata Ramana G >Assignee: Gururaj Shetty >Priority: Minor > Fix For: 1.0.0-incubating > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CARBONDATA-605) Add Update-delete related documentation
[ https://issues.apache.org/jira/browse/CARBONDATA-605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen updated CARBONDATA-605: -- Component/s: (was: core) docs > Add Update-delete related documentation > --- > > Key: CARBONDATA-605 > URL: https://issues.apache.org/jira/browse/CARBONDATA-605 > Project: CarbonData > Issue Type: Sub-task > Components: docs >Affects Versions: 1.0.0-incubating >Reporter: Venkata Ramana G >Priority: Minor > Fix For: 1.0.0-incubating > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-carbondata pull request #554: [CARBONDATA-661] misc cleanup in car...
Github user JihongMA commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/554#discussion_r96988394
--- Diff:
core/src/main/java/org/apache/carbondata/core/keygenerator/columnar/impl/MultiDimKeyVarLengthEquiSplitGenerator.java
---
@@ -143,17 +142,18 @@ private void intialize() {
private int[] convertToArray(List list) {
--- End diff --
this is all depending on if we want to hold a primitive type array not not.
these type of boxing/unboxing for primitive array type is annoying
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #554: [CARBONDATA-661] misc cleanup in car...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/554#discussion_r96984537
--- Diff:
core/src/main/java/org/apache/carbondata/core/keygenerator/columnar/impl/MultiDimKeyVarLengthEquiSplitGenerator.java
---
@@ -143,17 +142,18 @@ private void intialize() {
private int[] convertToArray(List list) {
--- End diff --
After second thought, I think these kind of functions are unnecessary, we
should use
```
List list = new LinkedList<>();
Integer[] array = x.toArray(new Integer[list.size()]);
```
from JDK instead of writing our own code
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #561: [WIP]Reverting big decimal compression as i...
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/561 Build Failed with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/700/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #561: [WIP]Reverting big decimal compressi...
GitHub user ashokblend opened a pull request: https://github.com/apache/incubator-carbondata/pull/561 [WIP]Reverting big decimal compression as it has below issue **Problem** We had added code for big decimal compression for tpc-h activity. It turns out there is data inaccuracy when big decimal scale value is more then 18 then result is not accurate **Solution** Revert the code to old way **TO-DO** Find another way to compress big decimal data type. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ashokblend/incubator-carbondata bigdecimal_compression_revert Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/561.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #561 commit ed630151012e930b3b9756f420a0965e76aab5ea Author: ashok.blend Date: 2017-01-19T23:15:16Z Reverting big decimal compression as it has below issue when big decimal scale value is more then 18 then result is not accurate --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Assigned] (CARBONDATA-660) Bad Records Logs and Raw CSVs should get display under segment id instead of Tasks id
[
https://issues.apache.org/jira/browse/CARBONDATA-660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohammad Shahid Khan reassigned CARBONDATA-660:
---
Assignee: Mohammad Shahid Khan
> Bad Records Logs and Raw CSVs should get display under segment id instead of
> Tasks id
> -
>
> Key: CARBONDATA-660
> URL: https://issues.apache.org/jira/browse/CARBONDATA-660
> Project: CarbonData
> Issue Type: Improvement
> Components: data-load
>Reporter: Priyal Sachdeva
>Assignee: Mohammad Shahid Khan
>Priority: Minor
>
> create table if not exists Badrecords_test (imei string,AMSize int) STORED BY
> 'org.apache.carbondata.format';
> LOAD DATA INPATH 'hdfs://hacluster/CSVs/bad_records.csv' into table
> Badrecords_test OPTIONS('DELIMITER'=',' ,
> 'QUOTECHAR'='"','BAD_RECORDS_LOGGER_ENABLE'='TRUE',
> 'BAD_RECORDS_ACTION'='REDIRECT','FILEHEADER'='imei,AMSize');
> Bad Records Logs and raw csvs are getting display under Task ID
> linux-61:/srv/OSCON/BigData/HACluster/install/hadoop/datanode #
> bin/hadoop fs -ls /tmp/carbon/default/badrecords_test
> drwxr-xr-x - root users 0 2017-01-18 21:08
> /tmp/carbon/default/badrecords_test/0--->Task ID
> 0: jdbc:hive2://172.168.100.205:23040> show segments for table
> Badrecords_test;
> ++--+--+--+--+
> | SegmentSequenceId | Status | Load Start Time |
> Load End Time |
> ++--+--+--+--+
> | 8 | Partial Success | 2017-01-18 21:12:58.018 |
> 2017-01-18 21:12:59.652 |
> | 7 | Partial Success | 2017-01-18 21:08:07.426 |
> 2017-01-18 21:08:11.791 |
> | 6 | Partial Success | 2017-01-18 21:07:07.645 |
> 2017-01-18 21:07:08.747 |
> | 5 | Partial Success | 2017-01-18 19:34:16.163 |
> 2017-01-18 19:34:18.163 |
> | 4 | Partial Success | 2017-01-18 19:34:13.669 |
> 2017-01-18 19:34:15.811 |
> | 3 | Partial Success | 2017-01-18 19:30:18.753 |
> 2017-01-18 19:30:19.644 |
> | 2 | Partial Success | 2017-01-18 19:30:13.508 |
> 2017-01-18 19:30:15.578 |
> | 1 | Partial Success | 2017-01-18 19:18:54.787 |
> 2017-01-18 19:18:54.94 |
> | 0 | Partial Success | 2017-01-18 19:18:53.741 |
> 2017-01-18 19:18:54.614 |
> ++--+--+--+--+
> Bad Records Logs and raw csvs are getting display under Task ID. It would be
> good to have the information of bad records as per the load i.e under segment
> id..
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] incubator-carbondata pull request #560: [WIP] Do not merge added log for the...
GitHub user mohammadshahidkhan reopened a pull request: https://github.com/apache/incubator-carbondata/pull/560 [WIP] Do not merge added log for the random failing test case Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like: `[CARBONDATA-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace `` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - What manual testing you have done? - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/mohammadshahidkhan/incubator-carbondata ut_fail_analysis Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/560.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #560 commit 1caac5c4c54f0121f1cdcd77a9c348d0df72baa1 Author: mohammadshahidkhan Date: 2017-01-19T15:53:13Z For test case fail analysis --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #560: [WIP] Do not merge added log for the...
Github user mohammadshahidkhan closed the pull request at: https://github.com/apache/incubator-carbondata/pull/560 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Resolved] (CARBONDATA-654) Add data update and deletion example
[ https://issues.apache.org/jira/browse/CARBONDATA-654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindra Pesala resolved CARBONDATA-654. Resolution: Fixed > Add data update and deletion example > > > Key: CARBONDATA-654 > URL: https://issues.apache.org/jira/browse/CARBONDATA-654 > Project: CarbonData > Issue Type: Bug > Components: examples >Affects Versions: 1.0.0-incubating >Reporter: Liang Chen >Assignee: Liang Chen >Priority: Minor > Fix For: 1.0.0-incubating > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Add data update and deletion example -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-carbondata pull request #540: [CARBONDATA-654] Add data update and...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/540 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #557: [WIP] Set carbon properties location proper
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/557 Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/695/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #540: [CARBONDATA-654] Add data update and deleti...
Github user ravipesala commented on the issue: https://github.com/apache/incubator-carbondata/pull/540 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #560: [WIP] Do not merge added log for the random...
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/560 Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/694/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (CARBONDATA-665) Comparision Failure occurs when we execute the same query in hive and Carbondata
SWATI RAO created CARBONDATA-665:
Summary: Comparision Failure occurs when we execute the same query
in hive and Carbondata
Key: CARBONDATA-665
URL: https://issues.apache.org/jira/browse/CARBONDATA-665
Project: CarbonData
Issue Type: Bug
Reporter: SWATI RAO
Create 1 query : create table Test_Boundary (c1_int int,c2_Bigint
Bigint,c3_Decimal Decimal(38,30),c4_double double,c5_string string,c6_Timestamp
Timestamp,c7_Datatype_Desc string) STORED BY 'org.apache.carbondata.format'
Load 1 Query : LOAD DATA INPATH 'HDFS_URL/BabuStore/Data/Test_Data1.csv' INTO
table Test_Boundary
OPTIONS('DELIMITER'=',','QUOTECHAR'='','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='')
Create 2 query : create table Test_Boundary1 (c1_int int,c2_Bigint
Bigint,c3_Decimal Decimal(38,30),c4_double double,c5_string string,c6_Timestamp
Timestamp,c7_Datatype_Desc string) STORED BY 'org.apache.carbondata.format'
Load 2 query: LOAD DATA INPATH 'HDFS_URL/BabuStore/Data/Test_Data1.csv' INTO
table Test_Boundary1
OPTIONS('DELIMITER'=',','QUOTECHAR'='','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='')
Select Query :
select
c1_int,c2_Bigint,c3_Decimal,c4_double,c5_string,c6_Timestamp,c7_Datatype_Desc
from Test_Boundary where c2_bigint=c2_bigint
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] incubator-carbondata issue #560: [WIP] Do not merge added log for the random...
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/560 Build Failed with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/693/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #560: [WIP] Do not merge added log for the...
GitHub user mohammadshahidkhan opened a pull request: https://github.com/apache/incubator-carbondata/pull/560 [WIP] Do not merge added log for the random failing test case Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like: `[CARBONDATA-] Description of pull request` - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [ ] Replace `` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - What manual testing you have done? - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/mohammadshahidkhan/incubator-carbondata ut_fail_analysis Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/560.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #560 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #551: [CARBONDATA-603] Fixed Date Filter i...
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/551#discussion_r96887126
--- Diff:
core/src/main/java/org/apache/carbondata/core/cache/dictionary/ColumnDictionaryInfo.java
---
@@ -285,9 +285,17 @@ private int compareFilterKeyWithDictionaryKey(String
dictionaryVal, String membe
.compare((Boolean.parseBoolean(dictionaryVal)),
(Boolean.parseBoolean(memberVal)));
case DATE:
case TIMESTAMP:
- SimpleDateFormat parser = new
SimpleDateFormat(CarbonProperties.getInstance()
- .getProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
- CarbonCommonConstants.CARBON_TIMESTAMP_DEFAULT_FORMAT));
+ String format= null;
+ if(dataType==DataType.DATE) {
--- End diff --
ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #551: [CARBONDATA-603] Fixed Date Filter i...
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/551#discussion_r96887109
--- Diff:
core/src/main/java/org/apache/carbondata/core/keygenerator/directdictionary/DirectDictionaryKeyGeneratorFactory.java
---
@@ -57,9 +57,16 @@ public static DirectDictionaryGenerator
getDirectDictionaryGenerator(DataType da
}
public static DirectDictionaryGenerator
getDirectDictionaryGenerator(DataType dataType) {
-return getDirectDictionaryGenerator(dataType,
-CarbonProperties.getInstance()
-.getProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
-CarbonCommonConstants.CARBON_TIMESTAMP_DEFAULT_FORMAT));
+String format= null;
--- End diff --
ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #551: [CARBONDATA-603] Fixed Date Filter i...
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/551#discussion_r96887094
--- Diff:
core/src/main/java/org/apache/carbondata/core/scan/expression/ExpressionResult.java
---
@@ -175,9 +175,17 @@ public String getString() throws
FilterIllegalMemberException {
switch (this.getDataType()) {
case DATE:
case TIMESTAMP:
- SimpleDateFormat parser = new
SimpleDateFormat(CarbonProperties.getInstance()
- .getProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
- CarbonCommonConstants.CARBON_TIMESTAMP_DEFAULT_FORMAT));
+ String format= null;
+ if(dataType==DataType.DATE) {
--- End diff --
ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #551: [CARBONDATA-603] Fixed Date Filter i...
Github user kumarvishal09 commented on a diff in the pull request: https://github.com/apache/incubator-carbondata/pull/551#discussion_r96887076 --- Diff: core/src/main/java/org/apache/carbondata/core/keygenerator/directdictionary/timestamp/DateDirectDictionaryGenerator.java --- @@ -35,11 +37,16 @@ static final int cutOffDate = Integer.MAX_VALUE >> 1; --- End diff -- ok --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #557: [WIP] Set carbon properties location proper
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/557 Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/692/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #551: [CARBONDATA-603] Fixed Date Filter i...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/551#discussion_r96885350
--- Diff:
core/src/main/java/org/apache/carbondata/core/writer/sortindex/CarbonDictionarySortModel.java
---
@@ -102,9 +102,17 @@ public CarbonDictionarySortModel(int key, DataType
dataType, String memberValue)
return val1.compareTo(val2);
case DATE:
case TIMESTAMP:
-SimpleDateFormat parser = new
SimpleDateFormat(CarbonProperties.getInstance()
-.getProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
-CarbonCommonConstants.CARBON_TIMESTAMP_DEFAULT_FORMAT));
+String format= null;
+if(dataType==DataType.DATE) {
--- End diff --
instead of using if else, please use inside case and use break;
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #551: [CARBONDATA-603] Fixed Date Filter i...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/551#discussion_r96885139
--- Diff:
core/src/main/java/org/apache/carbondata/core/scan/filter/FilterUtil.java ---
@@ -1078,9 +1078,17 @@ public static int
compareFilterKeyBasedOnDataType(String dictionaryVal, String m
.compare((Boolean.parseBoolean(dictionaryVal)),
(Boolean.parseBoolean(memberVal)));
case DATE:
case TIMESTAMP:
- SimpleDateFormat parser = new
SimpleDateFormat(CarbonProperties.getInstance()
- .getProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
- CarbonCommonConstants.CARBON_TIMESTAMP_DEFAULT_FORMAT));
+ String format= null;
+ if(dataType==DataType.DATE) {
--- End diff --
instead of using if else, please use inside case and use break;
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #551: [CARBONDATA-603] Fixed Date Filter i...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/551#discussion_r96885076
--- Diff:
core/src/main/java/org/apache/carbondata/core/scan/expression/ExpressionResult.java
---
@@ -523,9 +531,17 @@ public boolean isNull() {
return val1.compareTo(val2);
case DATE:
case TIMESTAMP:
- SimpleDateFormat parser = new
SimpleDateFormat(CarbonProperties.getInstance()
- .getProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
- CarbonCommonConstants.CARBON_TIMESTAMP_DEFAULT_FORMAT));
+ String format= null;
+ if(dataType==DataType.DATE) {
--- End diff --
instead of using if else, please use inside case and use break;
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #551: [CARBONDATA-603] Fixed Date Filter i...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/551#discussion_r96884908
--- Diff:
core/src/main/java/org/apache/carbondata/core/scan/expression/ExpressionResult.java
---
@@ -175,9 +175,17 @@ public String getString() throws
FilterIllegalMemberException {
switch (this.getDataType()) {
case DATE:
case TIMESTAMP:
- SimpleDateFormat parser = new
SimpleDateFormat(CarbonProperties.getInstance()
- .getProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
- CarbonCommonConstants.CARBON_TIMESTAMP_DEFAULT_FORMAT));
+ String format= null;
+ if(dataType==DataType.DATE) {
--- End diff --
instead of using if else, please use inside case and use break;
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #551: [CARBONDATA-603] Fixed Date Filter i...
Github user ravipesala commented on a diff in the pull request: https://github.com/apache/incubator-carbondata/pull/551#discussion_r96884951 --- Diff: core/src/main/java/org/apache/carbondata/core/keygenerator/directdictionary/timestamp/DateDirectDictionaryGenerator.java --- @@ -35,11 +37,16 @@ static final int cutOffDate = Integer.MAX_VALUE >> 1; --- End diff -- Please remove unused cutOffDate --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #551: [CARBONDATA-603] Fixed Date Filter i...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/551#discussion_r96883596
--- Diff:
core/src/main/java/org/apache/carbondata/core/cache/dictionary/ColumnDictionaryInfo.java
---
@@ -285,9 +285,17 @@ private int compareFilterKeyWithDictionaryKey(String
dictionaryVal, String membe
.compare((Boolean.parseBoolean(dictionaryVal)),
(Boolean.parseBoolean(memberVal)));
case DATE:
case TIMESTAMP:
- SimpleDateFormat parser = new
SimpleDateFormat(CarbonProperties.getInstance()
- .getProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
- CarbonCommonConstants.CARBON_TIMESTAMP_DEFAULT_FORMAT));
+ String format= null;
+ if(dataType==DataType.DATE) {
--- End diff --
why if else is required? please keep in the DATE case.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #551: [CARBONDATA-603] Fixed Date Filter i...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/551#discussion_r96883829
--- Diff:
core/src/main/java/org/apache/carbondata/core/keygenerator/directdictionary/DirectDictionaryKeyGeneratorFactory.java
---
@@ -57,9 +57,16 @@ public static DirectDictionaryGenerator
getDirectDictionaryGenerator(DataType da
}
public static DirectDictionaryGenerator
getDirectDictionaryGenerator(DataType dataType) {
-return getDirectDictionaryGenerator(dataType,
-CarbonProperties.getInstance()
-.getProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT,
-CarbonCommonConstants.CARBON_TIMESTAMP_DEFAULT_FORMAT));
+String format= null;
--- End diff --
Please use switch case
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #553: [CARBONDATA-635]Insert query fails w...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/553 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #553: [CARBONDATA-635]Insert query fails when fun...
Github user jackylk commented on the issue: https://github.com/apache/incubator-carbondata/pull/553 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #332: [CARBONDATA-424] Data Load will fail...
Github user akashrn5 closed the pull request at: https://github.com/apache/incubator-carbondata/pull/332 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #332: [CARBONDATA-424] Data Load will fail for ba...
Github user akashrn5 commented on the issue: https://github.com/apache/incubator-carbondata/pull/332 @ravipesala i have added test case for no_kettle flow. Please review --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #332: [CARBONDATA-424] Data Load will fail...
GitHub user akashrn5 reopened a pull request: https://github.com/apache/incubator-carbondata/pull/332 [CARBONDATA-424] Data Load will fail for badrecord when "bad_records_action" is "FAIL" **Why :** Data loading should fail for any bad record found when bad_records_action is FAIL **solution:** One more action for BAD_RECORDS_ACTION is added as 'FAIL' . When we give action is FAIL during data load, then the data load will fail with a proper error message which tells, for which bad record the data load is failed. You can merge this pull request into a Git repository by running: $ git pull https://github.com/akashrn5/incubator-carbondata br_bad_record_fail_action Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/332.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #332 commit 28e7ae37bf7ef1c29b5d0421f65df4c741120fae Author: Akash R Nilugal Date: 2016-11-18T13:01:38Z test case added for no kettle --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #551: [CARBONDATA-603] Fixed Date Filter issue
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/551 Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/691/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #332: [CARBONDATA-424] Data Load will fail for ba...
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/332 Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/690/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #559: Fix issues for document link
Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/559 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #559: Fix issues for document link
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/559 Build Failed with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/689/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #558: [CARBONDATA-562] Supporting Spark 1.6.3 Ver...
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/558 Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/688/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Resolved] (CARBONDATA-655) Make nokettle dataload flow as default in carbon
[ https://issues.apache.org/jira/browse/CARBONDATA-655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacky Li resolved CARBONDATA-655. - Resolution: Fixed Fix Version/s: 1.0.0-incubating > Make nokettle dataload flow as default in carbon > > > Key: CARBONDATA-655 > URL: https://issues.apache.org/jira/browse/CARBONDATA-655 > Project: CarbonData > Issue Type: Improvement >Reporter: Ravindra Pesala >Assignee: Ravindra Pesala >Priority: Minor > Fix For: 1.0.0-incubating > > Time Spent: 2h 20m > Remaining Estimate: 0h > > Make nokettle dataload flow as default in carbon -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-carbondata pull request #559: Fix issues for document link
GitHub user chenliang613 opened a pull request: https://github.com/apache/incubator-carbondata/pull/559 Fix issues for document link Fix issues for document link You can merge this pull request into a Git repository by running: $ git pull https://github.com/chenliang613/incubator-carbondata fix_doc_issues Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/559.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #559 commit c671c965d9fd81c2db3c1ce50ac26cfd54d31b6b Author: chenliang613 Date: 2017-01-19T13:50:52Z fix docs issues commit cbe535eefdbbc6cb113a1f46aace66e82356c6a5 Author: chenliang613 Date: 2017-01-19T13:56:22Z fix docs issues commit 99125fcbc65961dafc5fefdfe5353ad5dfda21cd Author: chenliang613 Date: 2017-01-19T13:59:19Z fix docs issues commit 4a30b42ae70992a2b76008cdee5b207a57725422 Author: chenliang613 Date: 2017-01-19T14:06:46Z fix docs issues --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #546: [CARBONDATA-655][CARBONDATA-604]Make...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/546 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #546: [CARBONDATA-655][CARBONDATA-604]Make no ket...
Github user chenliang613 commented on the issue: https://github.com/apache/incubator-carbondata/pull/546 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #332: [CARBONDATA-424] Data Load will fail for ba...
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/332 Build Failed with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/687/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #558: [CARBONDATA-562] Supporting Spark 1....
GitHub user nareshpr opened a pull request: https://github.com/apache/incubator-carbondata/pull/558 [CARBONDATA-562] Supporting Spark 1.6.3 Version in CarbonData Implemented support for Spark 1.6.3 in CarbonData You can merge this pull request into a Git repository by running: $ git pull https://github.com/nareshpr/incubator-carbondata Support163Spark Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/558.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #558 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #551: [CARBONDATA-603] Fixed Date Filter issue
Github user CarbonDataQA commented on the issue: https://github.com/apache/incubator-carbondata/pull/551 Build Success with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/686/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #557: [WIP] Set carbon properties location...
GitHub user ravipesala opened a pull request: https://github.com/apache/incubator-carbondata/pull/557 [WIP] Set carbon properties location proper You can merge this pull request into a Git repository by running: $ git pull https://github.com/ravipesala/incubator-carbondata yarn_carbon-properties-issue Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/557.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #557 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #398: [CARBONDATA-400] Error message for d...
Github user akashrn5 closed the pull request at: https://github.com/apache/incubator-carbondata/pull/398 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #398: [CARBONDATA-400] Error message for d...
GitHub user akashrn5 reopened a pull request: https://github.com/apache/incubator-carbondata/pull/398 [CARBONDATA-400] Error message for dataload with a column having more than 10 characters. * Problem: When the number of characters in a column exceeds 10 characters whole string appears in beeline with exception. Analysis: In univocity csv parser settings , the maximum number of characters per column is 10 and when it exceeds that limit, TextparsingException is thrown with the complete string as error in beeline during data load. Fix: Now a proper error message is displayed in beeline and complete error messages and parser settings details will be present in logs. Impact area: Data loading with more than 10 characters in a single column. You can merge this pull request into a Git repository by running: $ git pull https://github.com/akashrn5/incubator-carbondata br_message_correction Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/398.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #398 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #398: [CARBONDATA-400] Error message for d...
Github user akashrn5 closed the pull request at: https://github.com/apache/incubator-carbondata/pull/398 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #398: [CARBONDATA-400] Error message for d...
GitHub user akashrn5 reopened a pull request: https://github.com/apache/incubator-carbondata/pull/398 [CARBONDATA-400] Error message for dataload with a column having more than 10 characters. * Problem: When the number of characters in a column exceeds 10 characters whole string appears in beeline with exception. Analysis: In univocity csv parser settings , the maximum number of characters per column is 10 and when it exceeds that limit, TextparsingException is thrown with the complete string as error in beeline during data load. Fix: Now a proper error message is displayed in beeline and complete error messages and parser settings details will be present in logs. Impact area: Data loading with more than 10 characters in a single column. You can merge this pull request into a Git repository by running: $ git pull https://github.com/akashrn5/incubator-carbondata br_message_correction Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/398.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #398 commit a303d30b07d80aec5e2d3c4526145c96a1e28b9f Author: Akash R Nilugal Date: 2016-12-05T09:46:10Z [CARBONDATA-400] * Problem: When the number of characters in a column exceeds 10 characters whole string appears in beeline with exception. Analysis: In univocity csv parser settings , the maximum number of characters per column is 10 and when it exceeds that limit, TextparsingException is thrown with the complete string as error in beeline during data load. Fix: Now a proper error message is displayed in beeline and complete error messages and parser settings details will be present in logs. Impact area: Data loading with more than 10 characters in a single column. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (CARBONDATA-664) Select queries fail when BAD_RECORDS_ACTION as FORCED is used in load query.
[
https://issues.apache.org/jira/browse/CARBONDATA-664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh Sharma updated CARBONDATA-664:
Summary: Select queries fail when BAD_RECORDS_ACTION as FORCED is used in
load query. (was: Select queries fail when BAD_RECORDS_ACTION is used in load
query.)
> Select queries fail when BAD_RECORDS_ACTION as FORCED is used in load query.
>
>
> Key: CARBONDATA-664
> URL: https://issues.apache.org/jira/browse/CARBONDATA-664
> Project: CarbonData
> Issue Type: Bug
> Components: data-query
>Affects Versions: 1.0.0-incubating
> Environment: Spark 1.6
>Reporter: Harsh Sharma
> Labels: bug
> Attachments: 100_olap_C20.csv, Driver Logs, Executor Logs
>
>
> Below scenario is working on Spark 2.1, but not on Spark 1.6
> create table VMALL_DICTIONARY_INCLUDE (imei string,deviceInformationId
> int,MAC string,deviceColor string,device_backColor string,modelId
> string,marketName string,AMSize string,ROMSize string,CUPAudit
> string,CPIClocked string,series string,productionDate timestamp,bomCode
> string,internalModels string, deliveryTime string, channelsId string,
> channelsName string , deliveryAreaId string, deliveryCountry string,
> deliveryProvince string, deliveryCity string,deliveryDistrict string,
> deliveryStreet string, oxSingleNumber string, ActiveCheckTime string,
> ActiveAreaId string, ActiveCountry string, ActiveProvince string, Activecity
> string, ActiveDistrict string, ActiveStreet string, ActiveOperatorId string,
> Active_releaseId string, Active_EMUIVersion string, Active_operaSysVersion
> string, Active_BacVerNumber string, Active_BacFlashVer string,
> Active_webUIVersion string, Active_webUITypeCarrVer
> string,Active_webTypeDataVerNumber string, Active_operatorsVersion string,
> Active_phonePADPartitionedVersions string, Latest_YEAR int, Latest_MONTH int,
> Latest_DAY Decimal(30,10), Latest_HOUR string, Latest_areaId string,
> Latest_country string, Latest_province string, Latest_city string,
> Latest_district string, Latest_street string, Latest_releaseId string,
> Latest_EMUIVersion string, Latest_operaSysVersion string, Latest_BacVerNumber
> string, Latest_BacFlashVer string, Latest_webUIVersion string,
> Latest_webUITypeCarrVer string, Latest_webTypeDataVerNumber string,
> Latest_operatorsVersion string, Latest_phonePADPartitionedVersions string,
> Latest_operatorId string, gamePointDescription string,gamePointId
> double,contractNumber BigInt) STORED BY 'org.apache.carbondata.format'
> TBLPROPERTIES('DICTIONARY_INCLUDE'='imei,deviceInformationId,productionDate,gamePointId,Latest_DAY,contractNumber');
> LOAD DATA INPATH 'hdfs://hadoop-master:54311/data/100_olap_C20.csv' INTO
> table VMALL_DICTIONARY_INCLUDE
> options('DELIMITER'=',','QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='imei,deviceInformationId,MAC,deviceColor,device_backColor,modelId,marketName,AMSize,ROMSize,CUPAudit,CPIClocked,series,productionDate,bomCode,internalModels,deliveryTime,channelsId,channelsName,deliveryAreaId,deliveryCountry,deliveryProvince,deliveryCity,deliveryDistrict,deliveryStreet,oxSingleNumber,contractNumber,ActiveCheckTime,ActiveAreaId,ActiveCountry,ActiveProvince,Activecity,ActiveDistrict,ActiveStreet,ActiveOperatorId,Active_releaseId,Active_EMUIVersion,Active_operaSysVersion,Active_BacVerNumber,Active_BacFlashVer,Active_webUIVersion,Active_webUITypeCarrVer,Active_webTypeDataVerNumber,Active_operatorsVersion,Active_phonePADPartitionedVersions,Latest_YEAR,Latest_MONTH,Latest_DAY,Latest_HOUR,Latest_areaId,Latest_country,Latest_province,Latest_city,Latest_district,Latest_street,Latest_releaseId,Latest_EMUIVersion,Latest_operaSysVersion,Latest_BacVerNumber,Latest_BacFlashVer,Latest_webUIVersion,Latest_webUITypeCarrVer,Latest_webTypeDataVerNumber,Latest_operatorsVersion,Latest_phonePADPartitionedVersions,Latest_operatorId,gamePointId,gamePointDescription');
> select sum(deviceinformationId) from VMALL_DICTIONARY_INCLUDE where
> deviceColor ='5Device Color' and modelId != '109' or Latest_DAY >
> '1234567890123540.00' and contractNumber == '92233720368547800' or
> Active_operaSysVersion like 'Operating System Version' and gamePointId <=>
> '8.1366141918611E39' and deviceInformationId < '100' and productionDate
> not like '2016-07-01' and imei is null and Latest_HOUR is not null and
> channelsId <= '7' and Latest_releaseId >= '1' and Latest_MONTH between 6 and
> 8 and Latest_YEAR not between 2016 and 2017 and Latest_HOUR RLIKE '12' and
> gamePointDescription REGEXP 'Site' and imei in
> ('1AA1','1AA100','1AA10','1AA1000','1AA1','1AA10','1AA100','1AA11','1AA12','1AA14','','NULL')
> and Active_BacVerNumber not in ('Background version number1','','
[jira] [Updated] (CARBONDATA-664) Select queries fail when BAD_RECORDS_ACTION is used in load query.
[
https://issues.apache.org/jira/browse/CARBONDATA-664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh Sharma updated CARBONDATA-664:
Description:
Below scenario is working on Spark 2.1, but not on Spark 1.6
create table VMALL_DICTIONARY_INCLUDE (imei string,deviceInformationId int,MAC
string,deviceColor string,device_backColor string,modelId string,marketName
string,AMSize string,ROMSize string,CUPAudit string,CPIClocked string,series
string,productionDate timestamp,bomCode string,internalModels string,
deliveryTime string, channelsId string, channelsName string , deliveryAreaId
string, deliveryCountry string, deliveryProvince string, deliveryCity
string,deliveryDistrict string, deliveryStreet string, oxSingleNumber string,
ActiveCheckTime string, ActiveAreaId string, ActiveCountry string,
ActiveProvince string, Activecity string, ActiveDistrict string, ActiveStreet
string, ActiveOperatorId string, Active_releaseId string, Active_EMUIVersion
string, Active_operaSysVersion string, Active_BacVerNumber string,
Active_BacFlashVer string, Active_webUIVersion string, Active_webUITypeCarrVer
string,Active_webTypeDataVerNumber string, Active_operatorsVersion string,
Active_phonePADPartitionedVersions string, Latest_YEAR int, Latest_MONTH int,
Latest_DAY Decimal(30,10), Latest_HOUR string, Latest_areaId string,
Latest_country string, Latest_province string, Latest_city string,
Latest_district string, Latest_street string, Latest_releaseId string,
Latest_EMUIVersion string, Latest_operaSysVersion string, Latest_BacVerNumber
string, Latest_BacFlashVer string, Latest_webUIVersion string,
Latest_webUITypeCarrVer string, Latest_webTypeDataVerNumber string,
Latest_operatorsVersion string, Latest_phonePADPartitionedVersions string,
Latest_operatorId string, gamePointDescription string,gamePointId
double,contractNumber BigInt) STORED BY 'org.apache.carbondata.format'
TBLPROPERTIES('DICTIONARY_INCLUDE'='imei,deviceInformationId,productionDate,gamePointId,Latest_DAY,contractNumber');
LOAD DATA INPATH 'hdfs://hadoop-master:54311/data/100_olap_C20.csv' INTO table
VMALL_DICTIONARY_INCLUDE
options('DELIMITER'=',','QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='imei,deviceInformationId,MAC,deviceColor,device_backColor,modelId,marketName,AMSize,ROMSize,CUPAudit,CPIClocked,series,productionDate,bomCode,internalModels,deliveryTime,channelsId,channelsName,deliveryAreaId,deliveryCountry,deliveryProvince,deliveryCity,deliveryDistrict,deliveryStreet,oxSingleNumber,contractNumber,ActiveCheckTime,ActiveAreaId,ActiveCountry,ActiveProvince,Activecity,ActiveDistrict,ActiveStreet,ActiveOperatorId,Active_releaseId,Active_EMUIVersion,Active_operaSysVersion,Active_BacVerNumber,Active_BacFlashVer,Active_webUIVersion,Active_webUITypeCarrVer,Active_webTypeDataVerNumber,Active_operatorsVersion,Active_phonePADPartitionedVersions,Latest_YEAR,Latest_MONTH,Latest_DAY,Latest_HOUR,Latest_areaId,Latest_country,Latest_province,Latest_city,Latest_district,Latest_street,Latest_releaseId,Latest_EMUIVersion,Latest_operaSysVersion,Latest_BacVerNumber,Latest_BacFlashVer,Latest_webUIVersion,Latest_webUITypeCarrVer,Latest_webTypeDataVerNumber,Latest_operatorsVersion,Latest_phonePADPartitionedVersions,Latest_operatorId,gamePointId,gamePointDescription');
select sum(deviceinformationId) from VMALL_DICTIONARY_INCLUDE where deviceColor
='5Device Color' and modelId != '109' or Latest_DAY >
'1234567890123540.00' and contractNumber == '92233720368547800' or
Active_operaSysVersion like 'Operating System Version' and gamePointId <=>
'8.1366141918611E39' and deviceInformationId < '100' and productionDate not
like '2016-07-01' and imei is null and Latest_HOUR is not null and channelsId
<= '7' and Latest_releaseId >= '1' and Latest_MONTH between 6 and 8 and
Latest_YEAR not between 2016 and 2017 and Latest_HOUR RLIKE '12' and
gamePointDescription REGEXP 'Site' and imei in
('1AA1','1AA100','1AA10','1AA1000','1AA1','1AA10','1AA100','1AA11','1AA12','1AA14','','NULL')
and Active_BacVerNumber not in ('Background version number1','','null');
This scenario results in the following exception,
Error: org.apache.spark.SparkException: Job aborted due to stage failure: Task
0 in stage 48.0 failed 4 times, most recent failure: Lost task 0.3 in stage
48.0 (TID 152, hadoop-master): java.lang.RuntimeException:
java.util.concurrent.ExecutionException: java.lang.NullPointerException
at
org.apache.carbondata.scan.result.iterator.DetailQueryResultIterator.next(DetailQueryResultIterator.java:65)
at
org.apache.carbondata.scan.result.iterator.DetailQueryResultIterator.next(DetailQueryResultIterator.java:35)
at
org.apache.carbondata.scan.result.iterator.ChunkRowIterator.(ChunkRowIterator.java:43)
at
org.apache.carbondata.hadoop.CarbonRecordReader.initialize(CarbonRecordReader.java:81)
at
org.
[jira] [Updated] (CARBONDATA-664) Select queries fail when BAD_RECORDS_ACTION is used in load query.
[
https://issues.apache.org/jira/browse/CARBONDATA-664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh Sharma updated CARBONDATA-664:
Affects Version/s: (was: 1.1.0)
1.0.0-incubating
Summary: Select queries fail when BAD_RECORDS_ACTION is used in
load query. (was: Bug in automation test case in BAD_RECORDS_ACTION select
query in Spark 1.6)
> Select queries fail when BAD_RECORDS_ACTION is used in load query.
> --
>
> Key: CARBONDATA-664
> URL: https://issues.apache.org/jira/browse/CARBONDATA-664
> Project: CarbonData
> Issue Type: Bug
> Components: data-query
>Affects Versions: 1.0.0-incubating
> Environment: Spark 1.6
>Reporter: Harsh Sharma
> Labels: bug
> Attachments: 100_olap_C20.csv, Driver Logs, Executor Logs
>
>
> Below scenario is working on Spark 2.1, but not on Spark 1.6
> create table VMALL_DICTIONARY_INCLUDE (imei string,deviceInformationId
> int,MAC string,deviceColor string,device_backColor string,modelId
> string,marketName string,AMSize string,ROMSize string,CUPAudit
> string,CPIClocked string,series string,productionDate timestamp,bomCode
> string,internalModels string, deliveryTime string, channelsId string,
> channelsName string , deliveryAreaId string, deliveryCountry string,
> deliveryProvince string, deliveryCity string,deliveryDistrict string,
> deliveryStreet string, oxSingleNumber string, ActiveCheckTime string,
> ActiveAreaId string, ActiveCountry string, ActiveProvince string, Activecity
> string, ActiveDistrict string, ActiveStreet string, ActiveOperatorId string,
> Active_releaseId string, Active_EMUIVersion string, Active_operaSysVersion
> string, Active_BacVerNumber string, Active_BacFlashVer string,
> Active_webUIVersion string, Active_webUITypeCarrVer
> string,Active_webTypeDataVerNumber string, Active_operatorsVersion string,
> Active_phonePADPartitionedVersions string, Latest_YEAR int, Latest_MONTH int,
> Latest_DAY Decimal(30,10), Latest_HOUR string, Latest_areaId string,
> Latest_country string, Latest_province string, Latest_city string,
> Latest_district string, Latest_street string, Latest_releaseId string,
> Latest_EMUIVersion string, Latest_operaSysVersion string, Latest_BacVerNumber
> string, Latest_BacFlashVer string, Latest_webUIVersion string,
> Latest_webUITypeCarrVer string, Latest_webTypeDataVerNumber string,
> Latest_operatorsVersion string, Latest_phonePADPartitionedVersions string,
> Latest_operatorId string, gamePointDescription string,gamePointId
> double,contractNumber BigInt) STORED BY 'org.apache.carbondata.format'
> TBLPROPERTIES('DICTIONARY_INCLUDE'='imei,deviceInformationId,productionDate,gamePointId,Latest_DAY,contractNumber');
> LOAD DATA INPATH 'hdfs://hadoop-master:54311/data/100_olap_C20.csv' INTO
> table VMALL_DICTIONARY_INCLUDE
> options('DELIMITER'=',','QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='imei,deviceInformationId,MAC,deviceColor,device_backColor,modelId,marketName,AMSize,ROMSize,CUPAudit,CPIClocked,series,productionDate,bomCode,internalModels,deliveryTime,channelsId,channelsName,deliveryAreaId,deliveryCountry,deliveryProvince,deliveryCity,deliveryDistrict,deliveryStreet,oxSingleNumber,contractNumber,ActiveCheckTime,ActiveAreaId,ActiveCountry,ActiveProvince,Activecity,ActiveDistrict,ActiveStreet,ActiveOperatorId,Active_releaseId,Active_EMUIVersion,Active_operaSysVersion,Active_BacVerNumber,Active_BacFlashVer,Active_webUIVersion,Active_webUITypeCarrVer,Active_webTypeDataVerNumber,Active_operatorsVersion,Active_phonePADPartitionedVersions,Latest_YEAR,Latest_MONTH,Latest_DAY,Latest_HOUR,Latest_areaId,Latest_country,Latest_province,Latest_city,Latest_district,Latest_street,Latest_releaseId,Latest_EMUIVersion,Latest_operaSysVersion,Latest_BacVerNumber,Latest_BacFlashVer,Latest_webUIVersion,Latest_webUITypeCarrVer,Latest_webTypeDataVerNumber,Latest_operatorsVersion,Latest_phonePADPartitionedVersions,Latest_operatorId,gamePointId,gamePointDescription');
> select sum(deviceinformationId) from VMALL_DICTIONARY_INCLUDE where
> deviceColor ='5Device Color' and modelId != '109' or Latest_DAY >
> '1234567890123540.00' and contractNumber == '92233720368547800' or
> Active_operaSysVersion like 'Operating System Version' and gamePointId <=>
> '8.1366141918611E39' and deviceInformationId < '100' and productionDate
> not like '2016-07-01' and imei is null and Latest_HOUR is not null and
> channelsId <= '7' and Latest_releaseId >= '1' and Latest_MONTH between 6 and
> 8 and Latest_YEAR not between 2016 and 2017 and Latest_HOUR RLIKE '12' and
> gamePointDescription REGEXP 'Site' and imei in
> ('1AA1','1AA100','1AA10','1AA1000','1AA1','1AA10','1AA100','1AA11','1AA12','1AA14','','NULL')
[jira] [Created] (CARBONDATA-664) Bug in automation test case in BAD_RECORDS_ACTION select query in Spark 1.6
Harsh Sharma created CARBONDATA-664:
---
Summary: Bug in automation test case in BAD_RECORDS_ACTION select
query in Spark 1.6
Key: CARBONDATA-664
URL: https://issues.apache.org/jira/browse/CARBONDATA-664
Project: CarbonData
Issue Type: Bug
Components: data-query
Affects Versions: 1.1.0
Environment: Spark 1.6
Reporter: Harsh Sharma
Attachments: 100_olap_C20.csv, Driver Logs, Executor Logs
Below scenario is working on Spark 2.1, but not on Spark 1.6
create table VMALL_DICTIONARY_INCLUDE (imei string,deviceInformationId int,MAC
string,deviceColor string,device_backColor string,modelId string,marketName
string,AMSize string,ROMSize string,CUPAudit string,CPIClocked string,series
string,productionDate timestamp,bomCode string,internalModels string,
deliveryTime string, channelsId string, channelsName string , deliveryAreaId
string, deliveryCountry string, deliveryProvince string, deliveryCity
string,deliveryDistrict string, deliveryStreet string, oxSingleNumber string,
ActiveCheckTime string, ActiveAreaId string, ActiveCountry string,
ActiveProvince string, Activecity string, ActiveDistrict string, ActiveStreet
string, ActiveOperatorId string, Active_releaseId string, Active_EMUIVersion
string, Active_operaSysVersion string, Active_BacVerNumber string,
Active_BacFlashVer string, Active_webUIVersion string, Active_webUITypeCarrVer
string,Active_webTypeDataVerNumber string, Active_operatorsVersion string,
Active_phonePADPartitionedVersions string, Latest_YEAR int, Latest_MONTH int,
Latest_DAY Decimal(30,10), Latest_HOUR string, Latest_areaId string,
Latest_country string, Latest_province string, Latest_city string,
Latest_district string, Latest_street string, Latest_releaseId string,
Latest_EMUIVersion string, Latest_operaSysVersion string, Latest_BacVerNumber
string, Latest_BacFlashVer string, Latest_webUIVersion string,
Latest_webUITypeCarrVer string, Latest_webTypeDataVerNumber string,
Latest_operatorsVersion string, Latest_phonePADPartitionedVersions string,
Latest_operatorId string, gamePointDescription string,gamePointId
double,contractNumber BigInt) STORED BY 'org.apache.carbondata.format'
TBLPROPERTIES('DICTIONARY_INCLUDE'='imei,deviceInformationId,productionDate,gamePointId,Latest_DAY,contractNumber');
LOAD DATA INPATH 'hdfs://hadoop-master:54311/data/100_olap_C20.csv' INTO table
VMALL_DICTIONARY_INCLUDE
options('DELIMITER'=',','QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='imei,deviceInformationId,MAC,deviceColor,device_backColor,modelId,marketName,AMSize,ROMSize,CUPAudit,CPIClocked,series,productionDate,bomCode,internalModels,deliveryTime,channelsId,channelsName,deliveryAreaId,deliveryCountry,deliveryProvince,deliveryCity,deliveryDistrict,deliveryStreet,oxSingleNumber,contractNumber,ActiveCheckTime,ActiveAreaId,ActiveCountry,ActiveProvince,Activecity,ActiveDistrict,ActiveStreet,ActiveOperatorId,Active_releaseId,Active_EMUIVersion,Active_operaSysVersion,Active_BacVerNumber,Active_BacFlashVer,Active_webUIVersion,Active_webUITypeCarrVer,Active_webTypeDataVerNumber,Active_operatorsVersion,Active_phonePADPartitionedVersions,Latest_YEAR,Latest_MONTH,Latest_DAY,Latest_HOUR,Latest_areaId,Latest_country,Latest_province,Latest_city,Latest_district,Latest_street,Latest_releaseId,Latest_EMUIVersion,Latest_operaSysVersion,Latest_BacVerNumber,Latest_BacFlashVer,Latest_webUIVersion,Latest_webUITypeCarrVer,Latest_webTypeDataVerNumber,Latest_operatorsVersion,Latest_phonePADPartitionedVersions,Latest_operatorId,gamePointId,gamePointDescription');
select sum(deviceinformationId) from VMALL_DICTIONARY_INCLUDE where deviceColor
='5Device Color' and modelId != '109' or Latest_DAY >
'1234567890123540.00' and contractNumber == '92233720368547800' or
Active_operaSysVersion like 'Operating System Version' and gamePointId <=>
'8.1366141918611E39' and deviceInformationId < '100' and productionDate not
like '2016-07-01' and imei is null and Latest_HOUR is not null and channelsId
<= '7' and Latest_releaseId >= '1' and Latest_MONTH between 6 and 8 and
Latest_YEAR not between 2016 and 2017 and Latest_HOUR RLIKE '12' and
gamePointDescription REGEXP 'Site' and imei in
('1AA1','1AA100','1AA10','1AA1000','1AA1','1AA10','1AA100','1AA11','1AA12','1AA14','','NULL')
and Active_BacVerNumber not in ('Background version number1','','null');
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
