[jira] [Commented] (CARBONDATA-759) How to check a table's properties?
[ https://issues.apache.org/jira/browse/CARBONDATA-759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942277#comment-15942277 ] Babulal commented on CARBONDATA-759: please try "desc formatted " command . In comment column you can find the column type . MEASURE = Measure type column DICTIONARY, KEY COLUMN = dimension with Dictionary KEY COLUMN = dimension with NO Dictionary In ##Column Group Informationyou can find the COLUMN_GROUPS info . MDK info is used internally and only dimensions are used to form it . Regards Babu > How to check a table's properties? > -- > > Key: CARBONDATA-759 > URL: https://issues.apache.org/jira/browse/CARBONDATA-759 > Project: CarbonData > Issue Type: Improvement >Reporter: sehriff >Priority: Minor > > Is there a way to check what colums of a table is treated as dimensions or > measures? > How to check default table > properties(COLUMN_GROUPS,DICTIONARY_EXCLUDE,DICTIONARY_INCLUDE,NO_INVERTED_INDEX) > and what of this tables's mdk looks like? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (CARBONDATA-697) single_pass is not used while doing data load
[
https://issues.apache.org/jira/browse/CARBONDATA-697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Babulal reassigned CARBONDATA-697:
--
Assignee: Babulal
> single_pass is not used while doing data load
> -
>
> Key: CARBONDATA-697
> URL: https://issues.apache.org/jira/browse/CARBONDATA-697
> Project: CarbonData
> Issue Type: Bug
>Affects Versions: 0.1.1-incubating
> Environment: Spark 2.1
>Reporter: Ramakrishna
>Assignee: Babulal
> Fix For: 1.1.0-incubating
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> 1. 1st load with 'USE_KETTLE'='FALSE','SINGLE_PASS'='FALSE'
> 2. 2nd load with 'USE_KETTLE'='FALSE','SINGLE_PASS'='TRUE'
> Expected: GlobalDictionaryUtil should not be triggered in the 2nd time load
> Actual : GlobalDictionaryUtil was triggered
> Job Id (Job Group) ▾
> Description
> Submitted
> Duration
> Stages: Succeeded/Total Tasks (for all stages): Succeeded/Total
> 3 (891a750d-1e08-432f-bcf4-5501332d6d09)
> LOAD DATA inpath 'hdfs://hacluster/commbank/splits/FG2' into table
> flow_carbon
> options('USE_KETTLE'='FALSE','SINGLE_PASS'='TRUE','DELIMITER'=',',
> 'QUOTECHAR'='"','FILEHEADER'='aco_ac,ac_dte,txn_cnt,jrn_par,mfm_jrn_no,cbn_jrn_no,ibs_jrn_no,vch_no,vch_seq,srv_cde,cus_no,bus_cd_no,id_flg,cus_ac,bv_cde,bv_no,txn_dte,txn_time,txn_tlr,txn_bk,txn_br,ety_tlr,ety_bk,ety_br,bus_pss_no,chk_flg,chk_tlr,chk_jrn_no,bus_sys_no,bus_opr_cde,txn_sub_cde,fin_bus_cde,fin_bus_sub_cde,opt_prd_cde,chl,tml_id,sus_no,sus_seq,cho_seq,itm_itm,itm_sub,itm_sss,dc_flg,amt,bal,ccy,spv_flg,vch_vld_dte,pst_bk,pst_br,ec_flg,aco_tlr,opp_ac,opp_ac_nme,opp_bk,gen_flg,his_rec_sum_flg,his_flg,vch_typ,val_dte,opp_ac_flg,cmb_flg,ass_vch_flg,cus_pps_flg,bus_rmk_cde,vch_bus_rmk,tec_rmk_cde,vch_tec_rmk,rsv_ara,own_br,own_bk,gems_last_upd_d,gems_last_upd_d_bat,maps_date,maps_job,dt')
> collect at CarbonDataRDDFactory.scala:651 2017/02/06 15:23:05 41 s
> 1/1
> 3/3
> 2 (891a750d-1e08-432f-bcf4-5501332d6d09)
> LOAD DATA inpath 'hdfs://hacluster/commbank/splits/FG2' into table
> flow_carbon
> options('USE_KETTLE'='FALSE','SINGLE_PASS'='TRUE','DELIMITER'=',',
> 'QUOTECHAR'='"','FILEHEADER'='aco_ac,ac_dte,txn_cnt,jrn_par,mfm_jrn_no,cbn_jrn_no,ibs_jrn_no,vch_no,vch_seq,srv_cde,cus_no,bus_cd_no,id_flg,cus_ac,bv_cde,bv_no,txn_dte,txn_time,txn_tlr,txn_bk,txn_br,ety_tlr,ety_bk,ety_br,bus_pss_no,chk_flg,chk_tlr,chk_jrn_no,bus_sys_no,bus_opr_cde,txn_sub_cde,fin_bus_cde,fin_bus_sub_cde,opt_prd_cde,chl,tml_id,sus_no,sus_seq,cho_seq,itm_itm,itm_sub,itm_sss,dc_flg,amt,bal,ccy,spv_flg,vch_vld_dte,pst_bk,pst_br,ec_flg,aco_tlr,opp_ac,opp_ac_nme,opp_bk,gen_flg,his_rec_sum_flg,his_flg,vch_typ,val_dte,opp_ac_flg,cmb_flg,ass_vch_flg,cus_pps_flg,bus_rmk_cde,vch_bus_rmk,tec_rmk_cde,vch_tec_rmk,rsv_ara,own_br,own_bk,gems_last_upd_d,gems_last_upd_d_bat,maps_date,maps_job,dt')
> collect at GlobalDictionaryUtil.scala:742 2017/02/06 15:22:29 36 s
> 2/2
> 42/42
> 1 (c7706941-8184-40ab-9ceb-712af96e15e3)
> LOAD DATA inpath 'hdfs://hacluster/commbank/splits/FG2' into table
> flow_carbon
> options('USE_KETTLE'='FALSE','SINGLE_PASS'='FALSE','DELIMITER'=',',
> 'QUOTECHAR'='"','FILEHEADER'='aco_ac,ac_dte,txn_cnt,jrn_par,mfm_jrn_no,cbn_jrn_no,ibs_jrn_no,vch_no,vch_seq,srv_cde,cus_no,bus_cd_no,id_flg,cus_ac,bv_cde,bv_no,txn_dte,txn_time,txn_tlr,txn_bk,txn_br,ety_tlr,ety_bk,ety_br,bus_pss_no,chk_flg,chk_tlr,chk_jrn_no,bus_sys_no,bus_opr_cde,txn_sub_cde,fin_bus_cde,fin_bus_sub_cde,opt_prd_cde,chl,tml_id,sus_no,sus_seq,cho_seq,itm_itm,itm_sub,itm_sss,dc_flg,amt,bal,ccy,spv_flg,vch_vld_dte,pst_bk,pst_br,ec_flg,aco_tlr,opp_ac,opp_ac_nme,opp_bk,gen_flg,his_rec_sum_flg,his_flg,vch_typ,val_dte,opp_ac_flg,cmb_flg,ass_vch_flg,cus_pps_flg,bus_rmk_cde,vch_bus_rmk,tec_rmk_cde,vch_tec_rmk,rsv_ara,own_br,own_bk,gems_last_upd_d,gems_last_upd_d_bat,maps_date,maps_job,dt')
> collect at CarbonDataRDDFactory.scala:651 2017/02/06 15:21:42 47 s
> 1/1
> 3/3
> 0 (c7706941-8184-40ab-9ceb-712af96e15e3)
> LOAD DATA inpath 'hdfs://hacluster/commbank/splits/FG2' into table
> flow_carbon
> options('USE_KETTLE'='FALSE','SINGLE_PASS'='FALSE','DELIMITER'=',',
>
[jira] [Commented] (CARBONDATA-691) After Compaction records count are mismatched.
[ https://issues.apache.org/jira/browse/CARBONDATA-691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15859379#comment-15859379 ] Babulal commented on CARBONDATA-691: [~anubhavtarar] This issue happens when we have multiple node cluster ,where many tasks in the compaction . please find the create ,load command from CARBONDATA-696. > After Compaction records count are mismatched. > -- > > Key: CARBONDATA-691 > URL: https://issues.apache.org/jira/browse/CARBONDATA-691 > Project: CarbonData > Issue Type: Bug > Components: data-load, data-query, docs >Affects Versions: 1.0.0-incubating >Reporter: Babulal > Attachments: createLoadcmd.txt, driverlog.txt > > > Spark version - Spark 1.6.2 and spark2.1 > After Compaction data showing is wrong. > create table and load 4 times s( compaction threshold is 4,3) > Load 4 times same data .each load 105 records as attached in file . > --+--+ > | SegmentSequenceId | Status | Load Start Time | Load End > Time | > +++--+--+--+ > | 3 | Compacted | 2017-02-01 14:07:51.922 | 2017-02-01 > 14:07:52.591 | > | 2 | Compacted | 2017-02-01 14:07:33.481 | 2017-02-01 > 14:07:34.443 | > | 1 | Compacted | 2017-02-01 14:07:23.495 | 2017-02-01 > 14:07:24.167 | > | 0.1| Success| 2017-02-01 14:07:52.815 | 2017-02-01 > 14:07:57.201 | > | 0 | Compacted | 2017-02-01 14:07:07.541 | 2017-02-01 > 14:07:11.983 | > +++--+--+--+ > 5 rows selected (0.021 seconds) > 0: jdbc:hive2://8.99.61.4:23040> select count(*) from > Comp_VMALL_DICTIONARY_INCLUDE_7; > +---+--+ > | count(1) | > +---+--+ > | 1680 | > +---+--+ > 1 row selected (4.468 seconds) > 0: jdbc:hive2://8.99.61.4:23040> select count(imei) from > Comp_VMALL_DICTIONARY_INCLUDE_7; > +--+--+ > | count(imei) | > +--+--+ > | 1680 | > +--+--+ > Expected :- total records should be 420 . -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (CARBONDATA-696) NPE when select query run on measure having double data type without fraction.
Babulal created CARBONDATA-696:
--
Summary: NPE when select query run on measure having double data
type without fraction.
Key: CARBONDATA-696
URL: https://issues.apache.org/jira/browse/CARBONDATA-696
Project: CarbonData
Issue Type: Bug
Components: data-query
Affects Versions: 1.0.0-incubating
Reporter: Babulal
Attachments: logs, oscon_10.csv
Create table as below
cc.sql("create table oscon_carbon_old (CUST_PRFRD_FLG String,PROD_BRAND_NAME
String,PROD_COLOR String,CUST_LAST_RVW_DATE String,CUST_COUNTRY
String,CUST_CITY String,PRODUCT_NAME String,CUST_JOB_TITLE String,CUST_STATE
String,CUST_BUY_POTENTIAL String,PRODUCT_MODEL String,ITM_ID String,ITM_NAME
String,PRMTION_ID String,PRMTION_NAME String,SHP_MODE_ID String,SHP_MODE
String,DELIVERY_COUNTRY String,DELIVERY_STATE String,DELIVERY_CITY
String,DELIVERY_DISTRICT String,ACTIVE_EMUI_VERSION String,WH_NAME
String,STR_ORDER_DATE String,OL_ORDER_NO String,OL_ORDER_DATE String,OL_SITE
String,CUST_FIRST_NAME String,CUST_LAST_NAME String,CUST_BIRTH_DY
String,CUST_BIRTH_MM String,CUST_BIRTH_YR String,CUST_BIRTH_COUNTRY
String,CUST_SEX String,CUST_ADDRESS_ID String,CUST_STREET_NO
String,CUST_STREET_NAME String,CUST_AGE String,CUST_SUITE_NO String,CUST_ZIP
String,CUST_COUNTY String,PRODUCT_ID String,PROD_SHELL_COLOR String,DEVICE_NAME
String,PROD_SHORT_DESC String,PROD_LONG_DESC String,PROD_THUMB
String,PROD_IMAGE String,PROD_UPDATE_DATE String,PROD_LIVE String,PROD_LOC
String,PROD_RAM String,PROD_ROM String,PROD_CPU_CLOCK String,PROD_SERIES
String,ITM_REC_START_DATE String,ITM_REC_END_DATE String,ITM_BRAND_ID
String,ITM_BRAND String,ITM_CLASS_ID String,ITM_CLASS String,ITM_CATEGORY_ID
String,ITM_CATEGORY String,ITM_MANUFACT_ID String,ITM_MANUFACT
String,ITM_FORMULATION String,ITM_COLOR String,ITM_CONTAINER
String,ITM_MANAGER_ID String,PRM_START_DATE String,PRM_END_DATE
String,PRM_CHANNEL_DMAIL String,PRM_CHANNEL_EMAIL String,PRM_CHANNEL_CAT
String,PRM_CHANNEL_TV String,PRM_CHANNEL_RADIO String,PRM_CHANNEL_PRESS
String,PRM_CHANNEL_EVENT String,PRM_CHANNEL_DEMO String,PRM_CHANNEL_DETAILS
String,PRM_PURPOSE String,PRM_DSCNT_ACTIVE String,SHP_CODE String,SHP_CARRIER
String,SHP_CONTRACT String,CHECK_DATE String,CHECK_YR String,CHECK_MM
String,CHECK_DY String,CHECK_HOUR String,BOM String,INSIDE_NAME
String,PACKING_DATE String,PACKING_YR String,PACKING_MM String,PACKING_DY
String,PACKING_HOUR String,DELIVERY_PROVINCE String,PACKING_LIST_NO
String,ACTIVE_CHECK_TIME String,ACTIVE_CHECK_YR String,ACTIVE_CHECK_MM
String,ACTIVE_CHECK_DY String,ACTIVE_CHECK_HOUR String,ACTIVE_AREA_ID
String,ACTIVE_COUNTRY String,ACTIVE_PROVINCE String,ACTIVE_CITY
String,ACTIVE_DISTRICT String,ACTIVE_NETWORK String,ACTIVE_FIRMWARE_VER
String,ACTIVE_OS_VERSION String,LATEST_CHECK_TIME String,LATEST_CHECK_YR
String,LATEST_CHECK_MM String,LATEST_CHECK_DY String,LATEST_CHECK_HOUR
String,LATEST_AREAID String,LATEST_COUNTRY String,LATEST_PROVINCE
String,LATEST_CITY String,LATEST_DISTRICT String,LATEST_FIRMWARE_VER
String,LATEST_EMUI_VERSION String,LATEST_OS_VERSION String,LATEST_NETWORK
String,WH_ID String,WH_STREET_NO String,WH_STREET_NAME String,WH_STREET_TYPE
String,WH_SUITE_NO String,WH_CITY String,WH_COUNTY String,WH_STATE
String,WH_ZIP String,WH_COUNTRY String,OL_SITE_DESC String,OL_RET_ORDER_NO
String,OL_RET_DATE String,PROD_MODEL_ID String,CUST_ID String,PROD_UNQ_MDL_ID
String,CUST_NICK_NAME String,CUST_LOGIN String,CUST_EMAIL_ADDR
String,PROD_UNQ_DEVICE_ADDR String,PROD_UQ_UUID String,PROD_BAR_CODE
String,TRACKING_NO String,STR_ORDER_NO String,CUST_DEP_COUNT
double,CUST_VEHICLE_COUNT double,CUST_ADDRESS_CNT double,CUST_CRNT_CDEMO_CNT
double,CUST_CRNT_HDEMO_CNT double,CUST_CRNT_ADDR_DM
double,CUST_FIRST_SHIPTO_CNT double,CUST_FIRST_SALES_CNT double,CUST_GMT_OFFSET
double,CUST_DEMO_CNT double,CUST_INCOME double,PROD_UNLIMITED
double,PROD_OFF_PRICE double,PROD_UNITS double,TOTAL_PRD_COST
double,TOTAL_PRD_DISC double,PROD_WEIGHT double,REG_UNIT_PRICE
double,EXTENDED_AMT double,UNIT_PRICE_DSCNT_PCT double,DSCNT_AMT
double,PROD_STD_CST double,TOTAL_TX_AMT double,FREIGHT_CHRG
double,WAITING_PERIOD double,DELIVERY_PERIOD double,ITM_CRNT_PRICE
double,ITM_UNITS double,ITM_WSLE_CST double,ITM_SIZE double,PRM_CST
double,PRM_RESPONSE_TARGET double,PRM_ITM_DM double,SHP_MODE_CNT
double,WH_GMT_OFFSET double,WH_SQ_FT double,STR_ORD_QTY double,STR_WSLE_CST
double,STR_LIST_PRICE double,STR_SALES_PRICE double,STR_EXT_DSCNT_AMT
double,STR_EXT_SALES_PRICE double,STR_EXT_WSLE_CST double,STR_EXT_LIST_PRICE
double,STR_EXT_TX double,STR_COUPON_AMT double,STR_NET_PAID
double,STR_NET_PAID_INC_TX double,STR_NET_PRFT double,STR_SOLD_YR_CNT
double,STR_SOLD_MM_CNT double,STR_SOLD_ITM_CNT double,STR_TOTAL_CUST_CNT
double,STR_AREA_CNT double,STR_DEMO_CNT double,STR_OFFER_CNT double,STR_PRM_CNT
double,STR_TICKET_CNT
[jira] [Created] (CARBONDATA-691) After Compaction records count are mismatched.
Babulal created CARBONDATA-691: -- Summary: After Compaction records count are mismatched. Key: CARBONDATA-691 URL: https://issues.apache.org/jira/browse/CARBONDATA-691 Project: CarbonData Issue Type: Bug Components: data-load, data-query, docs Affects Versions: 1.0.0-incubating Reporter: Babulal Attachments: createLoadcmd.txt, driverlog.txt Spark version - Spark 1.6.2 and spark2.1 After Compaction data showing is wrong. create table and load 4 times s( compaction threshold is 4,3) Load 4 times same data .each load 105 records as attached in file . --+--+ | SegmentSequenceId | Status | Load Start Time | Load End Time | +++--+--+--+ | 3 | Compacted | 2017-02-01 14:07:51.922 | 2017-02-01 14:07:52.591 | | 2 | Compacted | 2017-02-01 14:07:33.481 | 2017-02-01 14:07:34.443 | | 1 | Compacted | 2017-02-01 14:07:23.495 | 2017-02-01 14:07:24.167 | | 0.1| Success| 2017-02-01 14:07:52.815 | 2017-02-01 14:07:57.201 | | 0 | Compacted | 2017-02-01 14:07:07.541 | 2017-02-01 14:07:11.983 | +++--+--+--+ 5 rows selected (0.021 seconds) 0: jdbc:hive2://8.99.61.4:23040> select count(*) from Comp_VMALL_DICTIONARY_INCLUDE_7; +---+--+ | count(1) | +---+--+ | 1680 | +---+--+ 1 row selected (4.468 seconds) 0: jdbc:hive2://8.99.61.4:23040> select count(imei) from Comp_VMALL_DICTIONARY_INCLUDE_7; +--+--+ | count(imei) | +--+--+ | 1680 | +--+--+ Expected :- total records should be 420 . -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CARBONDATA-623) If we drop table after this condition ---(Firstly we load data in table with single pass true and use kettle false and then in same table load data 2nd time with si
[
https://issues.apache.org/jira/browse/CARBONDATA-623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818650#comment-15818650
]
Babulal commented on CARBONDATA-623:
Hi ,
can you please refer CARBONDATA-595 Drop Table for carbon throws NPE
seems it it a same issue.
Thanks
Babu
> If we drop table after this condition ---(Firstly we load data in table with
> single pass true and use kettle false and then in same table load data 2nd
> time with single pass true and use kettle false ), it is throwing Error:
> java.lang.NullPointerException
> ---
>
> Key: CARBONDATA-623
> URL: https://issues.apache.org/jira/browse/CARBONDATA-623
> Project: CarbonData
> Issue Type: Bug
> Components: data-load
>Affects Versions: 1.0.0-incubating
>Reporter: Payal
>Priority: Minor
> Attachments: 7000_UniqData.csv
>
>
> 1.Firstly we load data in table with single pass true and use kettle false
> data load successfully and we are getting result set properly.
> 2.then in same table load data in table with single pass true and use kettle
> false data load successfully and we are getting result set properly.
> 3.But after that if we drop the table ,its is throwing null pointer exception.
> Queries
> 0: jdbc:hive2://hadoop-master:1> CREATE TABLE uniqdata_INCLUDEDICTIONARY
> (CUST_ID int,CUST_NAME String,ACTIVE_EMUI_VERSION string, DOB timestamp, DOJ
> timestamp, BIGINT_COLUMN1 bigint,BIGINT_COLUMN2 bigint,DECIMAL_COLUMN1
> decimal(30,10), DECIMAL_COLUMN2 decimal(36,10),Double_COLUMN1 double,
> Double_COLUMN2 double,INTEGER_COLUMN1 int) STORED BY
> 'org.apache.carbondata.format'
> TBLPROPERTIES('DICTIONARY_INCLUDE'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1');
> +-+--+
> | Result |
> +-+--+
> +-+--+
> No rows selected (1.13 seconds)
> 0: jdbc:hive2://hadoop-master:1> LOAD DATA INPATH
> 'hdfs://hadoop-master:54311/data/uniqdata/7000_UniqData.csv' into table
> uniqdata_INCLUDEDICTIONARY OPTIONS('DELIMITER'=',' ,
> 'QUOTECHAR'='"','BAD_RECORDS_LOGGER_ENABLE'='TRUE',
> 'BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1','SINGLE_PASS'='false','USE_KETTLE'
> ='false');
> +-+--+
> | Result |
> +-+--+
> +-+--+
> No rows selected (22.814 seconds)
> 0: jdbc:hive2://hadoop-master:1>
> 0: jdbc:hive2://hadoop-master:1> select count (distinct CUST_NAME) from
> uniqdata_INCLUDEDICTIONARY ;
> +---+--+
> | _c0 |
> +---+--+
> | 7002 |
> +---+--+
> 1 row selected (3.055 seconds)
> 0: jdbc:hive2://hadoop-master:1> select count(CUST_NAME) from
> uniqdata_INCLUDEDICTIONARY ;
> +---+--+
> | _c0 |
> +---+--+
> | 7013 |
> +---+--+
> 1 row selected (0.366 seconds)
> 0: jdbc:hive2://hadoop-master:1> LOAD DATA INPATH
> 'hdfs://hadoop-master:54311/data/uniqdata/7000_UniqData.csv' into table
> uniqdata_INCLUDEDICTIONARY OPTIONS('DELIMITER'=',' ,
> 'QUOTECHAR'='"','BAD_RECORDS_LOGGER_ENABLE'='TRUE',
> 'BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1','SINGLE_PASS'='true','USE_KETTLE'
> ='false');
> +-+--+
> | Result |
> +-+--+
> +-+--+
> No rows selected (4.837 seconds)
> 0: jdbc:hive2://hadoop-master:1> select count(CUST_NAME) from
> uniqdata_INCLUDEDICTIONARY ;
> ++--+
> | _c0 |
> ++--+
> | 14026 |
> ++--+
> 1 row selected (0.458 seconds)
> 0: jdbc:hive2://hadoop-master:1> select count (distinct CUST_NAME) from
> uniqdata_INCLUDEDICTIONARY ;
> +---+--+
> | _c0 |
> +---+--+
> | 7002 |
> +---+--+
> 1 row selected (3.173 seconds)
> 0: jdbc:hive2://hadoop-master:1> drop table uniqdata_includedictionary;
> Error: java.lang.NullPointerException (state=,code=0)
> Logs
> WARN 11-01 12:56:52,722 - Lost task 0.0 in stage 61.0 (TID 1740,
> hadoop-slave-2): FetchFailed(BlockManagerId(0, hadoop-slave-3, 45331),
> shuffleId=22, mapId=0, reduceId=0, message=
> org.apache.spark.shuffle.FetchFailedException: Failed to connect to
> hadoop-slave-3:45331
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.throwFetchFailedException(ShuffleBlockFetcherIterator.scala:323)
> at
>
[jira] [Created] (CARBONDATA-595) Drop Table for carbon throws NPE with HDFS lock type.
Babulal created CARBONDATA-595: -- Summary: Drop Table for carbon throws NPE with HDFS lock type. Key: CARBONDATA-595 URL: https://issues.apache.org/jira/browse/CARBONDATA-595 Project: CarbonData Issue Type: Bug Affects Versions: 0.2.0-incubating Reporter: Babulal Priority: Minor Start version :- 1.6.2 Start carbon thrift server set HDFS LOCK Type drop table from beeline 0: jdbc:hive2://hacluster> drop table oscon_new_1; Error: java.lang.NullPointerException (state=,code=0) Error in thrftserver 17/01/04 20:40:08 AUDIT DropTableCommand: [hadoop-master][anonymous][Thread-182]Deleted table [oscon_new_1] under database [default] 17/01/04 20:40:08 ERROR AbstractDFSCarbonFile: pool-25-thread-12 Exception occured:File does not exist: hdfs://hacluster/opt/CarbonStore/default/oscon_new_1/droptable.lock 17/01/04 20:40:08 ERROR SparkExecuteStatementOperation: Error executing query, currentState RUNNING, java.lang.NullPointerException at org.apache.carbondata.core.datastorage.store.filesystem.AbstractDFSCarbonFile.delete(AbstractDFSCarbonFile.java:128) at org.apache.carbondata.lcm.locks.HdfsFileLock.unlock(HdfsFileLock.java:110) at org.apache.spark.sql.execution.command.DropTableCommand.run(carbonTableSchema.scala:613) at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:58) at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:56) Note :- lock file and data are deleted successfully but in beeline it throws ERROR message instead of success. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CARBONDATA-563) Select Queries are not working with spark 1.6.2.
[ https://issues.apache.org/jira/browse/CARBONDATA-563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Babulal updated CARBONDATA-563: --- Attachment: issue_snapshot.jpg > Select Queries are not working with spark 1.6.2. > - > > Key: CARBONDATA-563 > URL: https://issues.apache.org/jira/browse/CARBONDATA-563 > Project: CarbonData > Issue Type: Bug > Components: core, data-query >Affects Versions: 0.2.0-incubating >Reporter: Babulal > Attachments: issue_snapshot.jpg > > > Create carbon table > create table x (a int ,b string) stored by 'carbondata' > Load data to carbon table > run query select count(*) from x; > Java.lang.ClassCastException:[Ljava.lang.Object;can not be cast to > org.apache.sql.catalyst.InternalRow > Log snap shot in attached. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-559) Job failed at last step
[
https://issues.apache.org/jira/browse/CARBONDATA-559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15775260#comment-15775260
]
Babulal commented on CARBONDATA-559:
Hi Lionel
Global dictionary is generated successfully but data loading graph is not
started because it seems that kettle home at executor size is not set properly
as displayed in logs.
NFO 23-12 16:58:47,461 - Executor task launch worker-4
{carbon.graph.rowset.size=10, carbon.enable.quick.filter=false,
carbon.number.of.cores=4, carbon.sort.file.buffer.size=20,
carbon.kettle.home=$/carbonlib/carbonplugins,
Carbon property is referred from
/opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/spark/conf/carbon.properties
path .
Below suggested solution can work.
a.. Correct the Kettle home path in carbon.properties and try again .
b. use without Kettle flow (please refer examples) .
Thanks
Babu
> Job failed at last step
> ---
>
> Key: CARBONDATA-559
> URL: https://issues.apache.org/jira/browse/CARBONDATA-559
> Project: CarbonData
> Issue Type: Bug
> Components: core
>Affects Versions: 0.2.0-incubating
> Environment: carbon version: branch-0.2
> hadoop 2.4.0
> spark 1.6.0
> OS centOS
>Reporter: Cao, Lionel
> Attachments: test001.log.zip
>
>
> Hi team,
> My job alwasy failed at last step:
> it said 'yarn' user don't have write access to target data
> path(storeLocation).
> But I tested twice with 1 rows data, both successed. could you help look
> into the log? Please refer to the attachment.
> Search 'access=WRITE' you can see the exception.
> Search 'Exception' for other exceptions.
> thanks,
> Lionel
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CARBONDATA-406) Empty Folder is created when data load from dataframe
Babulal created CARBONDATA-406:
--
Summary: Empty Folder is created when data load from dataframe
Key: CARBONDATA-406
URL: https://issues.apache.org/jira/browse/CARBONDATA-406
Project: CarbonData
Issue Type: Bug
Components: data-load
Affects Versions: 0.1.0-incubating
Reporter: Babulal
Priority: Trivial
Load the data from dataframe to carbon table with tempCSV=false option .
Load is success but emtyFolder is getting created in HDFS
Cluster size= 3 node .
Type:- Stanalone Spark
Steps
val customSchema = StructType(Array(StructField("imei", StringType, true),
StructField("deviceInformationId", IntegerType, true),StructField("mac",
StringType, true),StructField("productdate", TimestampType , true),
StructField("updatetime", TimestampType, true),StructField("gamePointId",
DoubleType, true),StructField("contractNumber", DoubleType, true) ));
val df = cc.read.format("com.databricks.spark.csv").option("header",
"false").schema(customSchema).load("/opt/data/xyz/100_default_date_11_header.csv");
Start data loading
scala> df.write.format("carbondata").option("tableName","mycarbon2").save();
Check Logs
leges:{}, groupPrivileges:null, rolePrivileges:null))
INFO 10-11 23:52:44,005 - Creating directory if it doesn't exist:
hdfs://10.18.102.236:54310/opt/Carbon/Spark/spark/bin/null/bin/carbonshellstore/hivemetadata/mycarbon4
AUDIT 10-11 23:52:44,037 - [BLR107781][root][Thread-1]Table created with
Database name [default] and Table name [mycarbon4]
INFO 10-11 23:52:44,040 - Successfully able to get the table metadata file lock
In the HDFS this Path is empty
hdfs://10.18.102.236:54310/opt/Carbon/Spark/spark/bin/null/bin/carbonshellstore/hivemetadata/mycarbon4
Actual Store location is :- hdfs://10.18.102.236:54310/opt/Carbon/mystore
Expect :- Empty folder should not be created. . It seems that it is created in
SPARK_HOME/bin .
SPARK_HOME is /opt/Carbon/Spark/spark/bin
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CARBONDATA-399) [Bad Records] Data Load is not FAILED even bad_records_action="FAIL" .
Babulal created CARBONDATA-399:
--
Summary: [Bad Records] Data Load is not FAILED even
bad_records_action="FAIL" .
Key: CARBONDATA-399
URL: https://issues.apache.org/jira/browse/CARBONDATA-399
Project: CarbonData
Issue Type: Bug
Components: data-load
Affects Versions: 0.1.0-incubating
Environment: SUSE 11 SP4
YARN HA
3 Nodes
Reporter: Babulal
Priority: Minor
Data Load is not FAILED when string data are loaded in the int column .
1. Create table defect_5 (imei string ,deviceInformationId int,mac
string,productdate timestamp,updatetime timestamp,gamePointId
double,contractNumber double) stored by 'carbondata'
TBLPROPERTIES('DICTIONARY_INCLUDE'='deviceInformationId') ;
deviceInformationId is int ( it will handled as dimension). Now load the
data
2. 0: jdbc:hive2://ha-cluster/default> LOAD DATA inpath
'hdfs://hacluster/tmp/100_default_date_11_header_2.csv' into table defect_5
options('DELIMITER'=',', 'bad_records_action'='FAIL',
'QUOTECHAR'='"','FILEHEADER'='imei,deviceinformationid,mac,productdate,updatetime,gamepointid,contractnumber');
+-+--+
| Result |
+-+--+
+-+--+
No rows selected (0.969 seconds)
3. Data
imei,deviceinformationid,mac,productdate,updatetime,gamepointid,contractnumber
1AA1,babu,Mikaa1,2015-01-01 11:00:00,2015-01-01 13:00:00,10,260
1AA2,3,Mikaa2,2015-01-02 12:00:00,2015-01-01 14:00:00,278,230
1AA3,1,Mikaa1,2015-01-03 13:00:00,2015-01-01 15:00:00,2556,1
1AA4,10,Mikaa2,2015-01-04 14:00:00,2015-01-01 16:00:00,640,254
1AA5,10,Mikaa,2015-01-05 15:00:00,2015-01-01 17:00:00,980,256
1AA6,10,Mikaa,2015-01-06 16:00:00,2015-01-01 18:00:00,1,2378
1AA7,10,Mikaa,2015-01-07 17:00:00,2015-01-01 19:00:00,96,234
1AA8,9,max,2015-01-08 18:00:00,2015-01-01 20:00:00,89,236
1AA9,10,max,2015-01-09 19:00:00,2015-01-01 21:00:00,198.36,239.2
Expect Outoput:- Data Load should FAIL
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
