[jira] [Created] (CARBONDATA-2265) [DFX]-Load]: Load job fails if 1 folder contains 1000 files
Ajeet Rai created CARBONDATA-2265: - Summary: [DFX]-Load]: Load job fails if 1 folder contains 1000 files Key: CARBONDATA-2265 URL: https://issues.apache.org/jira/browse/CARBONDATA-2265 Project: CarbonData Issue Type: Bug Environment: 3 node ant cluster Reporter: Ajeet Rai Load job fails if 1 folder contains 1000 files. 【Precondition】：Thrift server should be running 【Test step】： 1: Create a carbon table 2: Start a load where 1 folder contains 1000 files 3: Observe that load fails Observe that Out of Memory exception is thrown. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (CARBONDATA-1759) (Carbon1.3.0 - Clean Files) Clean command is not working correctly for segments marked for delete due to insert overwrite job
[
https://issues.apache.org/jira/browse/CARBONDATA-1759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ajeet Rai closed CARBONDATA-1759.
-
Issue is fixed. Clean command is working correctly for segments marked for
delete due to insert overwrite job.
> (Carbon1.3.0 - Clean Files) Clean command is not working correctly for
> segments marked for delete due to insert overwrite job
> --
>
> Key: CARBONDATA-1759
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1759
> Project: CarbonData
> Issue Type: Bug
> Components: data-query
>Affects Versions: 1.3.0
> Environment: 3 Node ant cluster
>Reporter: Ajeet Rai
>Assignee: Akash R Nilugal
>Priority: Major
> Labels: dfx
> Fix For: 1.3.0
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> Carbon1.3.0 Clean command is not working correctly for segments marked for
> delete due to insert overwrite job.
> 1: Create a table
> CREATE TABLE IF NOT EXISTS flow_carbon_new999(txn_dte String,dt String,txn_bk
> String,txn_br String,own_bk String,own_br String,opp_bk String,bus_opr_cde
> String,opt_prd_cde String,cus_no String,cus_ac String,opp_ac_nme
> String,opp_ac String,bv_no String,aco_ac String,ac_dte String,txn_cnt
> int,jrn_par int,mfm_jrn_no String,cbn_jrn_no String,ibs_jrn_no String,vch_no
> String,vch_seq String,srv_cde String,bus_cd_no String,id_flg String,bv_cde
> String,txn_time String,txn_tlr String,ety_tlr String,ety_bk String,ety_br
> String,bus_pss_no String,chk_flg String,chk_tlr String,chk_jrn_no String,
> bus_sys_no String,txn_sub_cde String,fin_bus_cde String,fin_bus_sub_cde
> String,chl String,tml_id String,sus_no String,sus_seq String, cho_seq
> String, itm_itm String,itm_sub String,itm_sss String,dc_flg String,amt
> decimal(15,2),bal decimal(15,2),ccy String,spv_flg String,vch_vld_dte
> String,pst_bk String,pst_br String,ec_flg String,aco_tlr String,gen_flg
> String,his_rec_sum_flg String,his_flg String,vch_typ String,val_dte
> String,opp_ac_flg String,cmb_flg String,ass_vch_flg String,cus_pps_flg
> String,bus_rmk_cde String,vch_bus_rmk String,tec_rmk_cde String,vch_tec_rmk
> String,gems_last_upd_d String,maps_date String,maps_job String)STORED BY
> 'org.apache.carbondata.format'
> TBLPROPERTIES('DICTIONARY_INCLUDE'='txn_cnt,jrn_par,amt,bal','No_Inverted_Index'=
> 'txn_dte,dt,txn_bk,txn_br,own_bk ,own_br ,opp_bk ,bus_opr_cde ,opt_prd_cde
> ,cus_no ,cus_ac ,opp_ac_nme ,opp_ac ,bv_no ,aco_ac ,ac_dte ,txn_cnt
> ,jrn_par ,mfm_jrn_no ,cbn_jrn_no ,ibs_jrn_no ,vch_no ,vch_seq ,srv_cde
> ,bus_cd_no ,id_flg ,bv_cde ,txn_time ,txn_tlr ,ety_tlr ,ety_bk ,ety_br
> ,bus_pss_no ,chk_flg ,chk_tlr ,chk_jrn_no , bus_sys_no ,txn_sub_cde
> ,fin_bus_cde ,fin_bus_sub_cde ,chl ,tml_id ,sus_no ,sus_seq , cho_seq ,
> itm_itm ,itm_sub ,itm_sss ,dc_flg ,amt,bal,ccy ,spv_flg ,vch_vld_dte ,pst_bk
> ,pst_br ,ec_flg ,aco_tlr ,gen_flg ,his_rec_sum_flg ,his_flg ,vch_typ ,val_dte
> ,opp_ac_flg ,cmb_flg ,ass_vch_flg ,cus_pps_flg ,bus_rmk_cde ,vch_bus_rmk
> ,tec_rmk_cde ,vch_tec_rmk ,gems_last_upd_d ,maps_date ,maps_job' );
> 2: start a data load.
> LOAD DATA inpath 'hdfs://hacluster/user/test/20140101_1_1.csv' into
> table flow_carbon_new999 options('DELIMITER'=',',
> 'QUOTECHAR'='"','header'='false');
> 3: run a insert overwrite job
> insert into table flow_carbon_new999 select * from flow_carbon_new666;
> 4: run show segment query:
> show segments for table ajeet.flow_carbon_new999
> 5: Observe that all previous segments are marked for delete
> 6: run clean query
> CLEAN FILES FOR TABLE ajeet.flow_carbon_new999;
> 7: again run show segment query
> 8: Observe that still all previous segments which are marked for delete are
> shown as result.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (CARBONDATA-1761) (Carbon1.3.0 - DELETE SEGMENT BY ID) In Progress Segment is marked for delete if respective id is given in delete segment by id query
[
https://issues.apache.org/jira/browse/CARBONDATA-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ajeet Rai closed CARBONDATA-1761.
-
Issue is fixed. A segment is not deleted if load is still in progress for that
segment id.
> (Carbon1.3.0 - DELETE SEGMENT BY ID) In Progress Segment is marked for delete
> if respective id is given in delete segment by id query
> -
>
> Key: CARBONDATA-1761
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1761
> Project: CarbonData
> Issue Type: Bug
> Components: data-load
>Affects Versions: 1.3.0
> Environment: 3 Node ant cluster
> Description
>Reporter: Ajeet Rai
>Assignee: Akash R Nilugal
>Priority: Major
> Labels: dfx
> Fix For: 1.3.0
>
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> (Carbon1.3.0 - DELETE SEGMENT BY ID) In Progress Segment is marked for delete
> if respective id is given in delete segment by id query.
> 1: Create a table
> CREATE TABLE IF NOT EXISTS flow_carbon_new999(txn_dte String,dt String,txn_bk
> String,txn_br String,own_bk String,own_br String,opp_bk String,bus_opr_cde
> String,opt_prd_cde String,cus_no String,cus_ac String,opp_ac_nme
> String,opp_ac String,bv_no String,aco_ac String,ac_dte String,txn_cnt
> int,jrn_par int,mfm_jrn_no String,cbn_jrn_no String,ibs_jrn_no String,vch_no
> String,vch_seq String,srv_cde String,bus_cd_no String,id_flg String,bv_cde
> String,txn_time String,txn_tlr String,ety_tlr String,ety_bk String,ety_br
> String,bus_pss_no String,chk_flg String,chk_tlr String,chk_jrn_no String,
> bus_sys_no String,txn_sub_cde String,fin_bus_cde String,fin_bus_sub_cde
> String,chl String,tml_id String,sus_no String,sus_seq String, cho_seq String,
> itm_itm String,itm_sub String,itm_sss String,dc_flg String,amt
> decimal(15,2),bal decimal(15,2),ccy String,spv_flg String,vch_vld_dte
> String,pst_bk String,pst_br String,ec_flg String,aco_tlr String,gen_flg
> String,his_rec_sum_flg String,his_flg String,vch_typ String,val_dte
> String,opp_ac_flg String,cmb_flg String,ass_vch_flg String,cus_pps_flg
> String,bus_rmk_cde String,vch_bus_rmk String,tec_rmk_cde String,vch_tec_rmk
> String,gems_last_upd_d String,maps_date String,maps_job String)STORED BY
> 'org.apache.carbondata.format'
> TBLPROPERTIES('DICTIONARY_INCLUDE'='txn_cnt,jrn_par,amt,bal','No_Inverted_Index'=
> 'txn_dte,dt,txn_bk,txn_br,own_bk ,own_br ,opp_bk ,bus_opr_cde ,opt_prd_cde
> ,cus_no ,cus_ac ,opp_ac_nme ,opp_ac ,bv_no ,aco_ac ,ac_dte ,txn_cnt ,jrn_par
> ,mfm_jrn_no ,cbn_jrn_no ,ibs_jrn_no ,vch_no ,vch_seq ,srv_cde ,bus_cd_no
> ,id_flg ,bv_cde ,txn_time ,txn_tlr ,ety_tlr ,ety_bk ,ety_br ,bus_pss_no
> ,chk_flg ,chk_tlr ,chk_jrn_no , bus_sys_no ,txn_sub_cde ,fin_bus_cde
> ,fin_bus_sub_cde ,chl ,tml_id ,sus_no ,sus_seq , cho_seq , itm_itm ,itm_sub
> ,itm_sss ,dc_flg ,amt,bal,ccy ,spv_flg ,vch_vld_dte ,pst_bk ,pst_br ,ec_flg
> ,aco_tlr ,gen_flg ,his_rec_sum_flg ,his_flg ,vch_typ ,val_dte ,opp_ac_flg
> ,cmb_flg ,ass_vch_flg ,cus_pps_flg ,bus_rmk_cde ,vch_bus_rmk ,tec_rmk_cde
> ,vch_tec_rmk ,gems_last_upd_d ,maps_date ,maps_job' );
> 2: start a data load.
> LOAD DATA inpath 'hdfs://hacluster/user/test/20140101_1_1.csv' into
> table flow_carbon_new999 options('DELIMITER'=',',
> 'QUOTECHAR'='"','header'='false');
> 3: run a insert into/overwrite job
> insert into table flow_carbon_new999 select * from flow_carbon_new666;
> 4: show segments for table flow_carbon_new999;
> 5: Observe that load/insert/overwrite job is started with new segment id
> 6: now run a delete segment by id query with this id.
> DELETE FROM TABLE ajeet.flow_carbon_new999 WHERE SEGMENT.ID IN (34)
> 7: again run show segment and see this segment which is still in progress is
> marked for delete.
> 8: Observe that insert/load job is still running and after some time(in next
> job of load/insert/overwrite), this job fails with below error:
> Error: java.lang.RuntimeException: It seems insert overwrite has been issued
> during load (state=,code=0)
> This is not correct behaviour and it should be handled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (CARBONDATA-1789) Carbon1.3.0 Concurrent Load-Drop: user is able to drop table even if insert/load job is running
[ https://issues.apache.org/jira/browse/CARBONDATA-1789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajeet Rai closed CARBONDATA-1789. - Issue is fixed.When user tries to drop a table while load is still in progress, a error message is shown to user: Error: org.apache.carbondata.spark.exception.ConcurrentOperationException: loading is in progress for table db1.t1, drop table operation is not allowed (state=,code=0) > Carbon1.3.0 Concurrent Load-Drop: user is able to drop table even if > insert/load job is running > --- > > Key: CARBONDATA-1789 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1789 > Project: CarbonData > Issue Type: Bug > Components: data-load > Environment: 3 Node ant cluster >Reporter: Ajeet Rai >Assignee: Akash R Nilugal >Priority: Major > Labels: dfx > Fix For: 1.3.0 > > Time Spent: 6h 50m > Remaining Estimate: 0h > > Carbon1.3.0 Concurrent Load-Drop: user is able to drop table even if > insert/load job is running > Steps: > 1: Create a table > 2: Start a insert job > 3: Concurrently drop the table > 4: Observe that drop is success > 5: Observe that insert job is running and after some times job fails > Expected behvaiour: drop job should wait for insert job to complete -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (CARBONDATA-1802) Carbon1.3.0 Alter:Alter query fails if a column is dropped and there is no key column
[ https://issues.apache.org/jira/browse/CARBONDATA-1802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajeet Rai closed CARBONDATA-1802. - Issue is fixed.Alter query is success after dropping the column in a table where there is no key column. > Carbon1.3.0 Alter:Alter query fails if a column is dropped and there is no > key column > -- > > Key: CARBONDATA-1802 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1802 > Project: CarbonData > Issue Type: Bug >Affects Versions: 1.3.0 > Environment: 3 Node ant cluster >Reporter: Ajeet Rai >Assignee: dhatchayani >Priority: Major > Labels: functional > Fix For: 1.3.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Carbon1.3.0 Alter:Alter query fails if a column is dropped and there is no > key column. > Steps: > 1: create table ttt(c int,d int,e int) stored by 'carbondata'; > 2: Alter table ttt drop columns(c); > 3: observe that below error is coming: > Error: java.lang.RuntimeException: Alter table drop column operation failed: > Alter drop operation failed. AtLeast one key column should exist after drop. > Expected: Since user is able to create a table with all numeric columns, Same > should be supported in Alter feature. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (CARBONDATA-1735) Carbon1.3.0 Load: Segment created during load is not marked for delete if beeline session is closed while load is still in progress
[ https://issues.apache.org/jira/browse/CARBONDATA-1735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajeet Rai closed CARBONDATA-1735. - Resolution: Fixed This issue has been verified in latest carbon 1.3 version and it is working fine. Hence closing the defect. > Carbon1.3.0 Load: Segment created during load is not marked for delete if > beeline session is closed while load is still in progress > > > Key: CARBONDATA-1735 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1735 > Project: CarbonData > Issue Type: Bug > Components: data-load >Affects Versions: 1.3.0 > Environment: 3 Node ant cluster >Reporter: Ajeet Rai >Priority: Minor > Labels: DFX > > Load: Segment created during load is not marked for delete if beeline session > is closed while load is still in progress. > Steps: > 1: Create a table with dictionary include > 2: Start a load job > 3: close the beeline session when global dictionary generation job is still > in progress. > 4: Observe that global dictionary generation job is completed but next job is > not triggered. > 5: Also observe that table status file is not updated and status of job is > still in progress. > 6: show segment will show this segment with status as in progress. > Expected behaviour: Either job should be completed or load should fail and > segment should be marked for delete. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (CARBONDATA-1806) Carbon1.3.0 Load with global sort: Load fails If a table is created with sort scope as global sort
[
https://issues.apache.org/jira/browse/CARBONDATA-1806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ajeet Rai closed CARBONDATA-1806.
-
Resolution: Fixed
Fix Version/s: 1.3.0
Issue is closed as fixed and not reproducible in latest release.
> Carbon1.3.0 Load with global sort: Load fails If a table is created with sort
> scope as global sort
> --
>
> Key: CARBONDATA-1806
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1806
> Project: CarbonData
> Issue Type: Bug
>Affects Versions: 1.3.0
> Environment: 3 node cluster
>Reporter: Ajeet Rai
> Labels: dfx
> Fix For: 1.3.0
>
>
> Carbon1.3.0 Load with global sort: Load fails If a table is created with sort
> scope as global sort.
> Steps:
> 1: create table dt1 (c1 string, c2 int) STORED BY
> 'org.apache.carbondata.format' tblproperties('sort_scope'='Global_sort');
> 2: LOAD DATA INPATH 'hdfs://hacluster/user/test/dt1.txt' INTO TABLE dt1
> OPTIONS('DELIMITER'=',', 'QUOTECHAR'= '\"');
> 3: Observe that load fails with below error:
> Error: java.lang.Exception: DataLoad failure (state=,code=0)
> 4: Check log:
> org.apache.carbondata.processing.loading.exception.CarbonDataLoadingException:
> There is an unexpected error:
> org.apache.carbondata.core.datastore.exception.CarbonDataWriterException
> at
> org.apache.carbondata.spark.load.DataLoadProcessorStepOnSpark$.writeFunc(DataLoadProcessorStepOnSpark.scala:198)
> at
> org.apache.carbondata.spark.load.DataLoadProcessBuilderOnSpark$$anonfun$loadDataUsingGlobalSort$1.apply(DataLoadProcessBuilderOnSpark.scala:130)
> at
> org.apache.carbondata.spark.load.DataLoadProcessBuilderOnSpark$$anonfun$loadDataUsingGlobalSort$1.apply(DataLoadProcessBuilderOnSpark.scala:129)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
> at org.apache.spark.scheduler.Task.run(Task.scala:99)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Suppressed: org.apache.spark.util.TaskCompletionListenerException:
> There is an unexpected error:
> org.apache.carbondata.core.datastore.exception.CarbonDataWriterException
> Previous exception in task: There is an unexpected error:
> org.apache.carbondata.core.datastore.exception.CarbonDataWriterException
>
> org.apache.carbondata.spark.load.DataLoadProcessorStepOnSpark$.writeFunc(DataLoadProcessorStepOnSpark.scala:198)
>
> org.apache.carbondata.spark.load.DataLoadProcessBuilderOnSpark$$anonfun$loadDataUsingGlobalSort$1.apply(DataLoadProcessBuilderOnSpark.scala:130)
>
> org.apache.carbondata.spark.load.DataLoadProcessBuilderOnSpark$$anonfun$loadDataUsingGlobalSort$1.apply(DataLoadProcessBuilderOnSpark.scala:129)
> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
> org.apache.spark.scheduler.Task.run(Task.scala:99)
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)
> at
> org.apache.spark.TaskContextImpl.invokeListeners(TaskContextImpl.scala:138)
> at
> org.apache.spark.TaskContextImpl.markTaskFailed(TaskContextImpl.scala:106)
> at org.apache.spark.scheduler.Task.run(Task.scala:104)
> ... 4 more
> Caused by:
> org.apache.carbondata.processing.loading.exception.CarbonDataLoadingException:
> org.apache.carbondata.core.datastore.exception.CarbonDataWriterException
> at
> org.apache.carbondata.processing.loading.steps.DataWriterProcessorStepImpl.processingComplete(DataWriterProcessorStepImpl.java:163)
> at
> org.apache.carbondata.processing.loading.steps.DataWriterProcessorStepImpl.finish(DataWriterProcessorStepImpl.java:149)
> at
> org.apache.carbondata.spark.load.DataLoadProcessorStepOnSpark$.writeFunc(DataLoadProcessorStepOnSpark.scala:189)
> ... 8 more
> Caused by:
> org.apache.carbondata.core.datastore.exception.CarbonDataWriterException:
> org.apache.carbondata.core.datastore.exception.CarbonDataWriterException
> at
> org.apache.carbondata.processing.store.CarbonFactDataHandlerColumnar.processWriteTaskSubmitList(CarbonFactDataHandlerColumnar.java:326)
> at
> org.apache.carbondata.processing.store.CarbonFactDataHandlerColumnar.closeHandler(CarbonFactDataHandlerColumnar.java:350)
> at
[jira] [Commented] (CARBONDATA-1708) Carbon1.3.0 Dictionary creation: By default dictionary is not created for string column
[ https://issues.apache.org/jira/browse/CARBONDATA-1708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16298325#comment-16298325 ] Ajeet Rai commented on CARBONDATA-1708: --- Working as per the latest documentation. Hence closing the defect. > Carbon1.3.0 Dictionary creation: By default dictionary is not created for > string column > --- > > Key: CARBONDATA-1708 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1708 > Project: CarbonData > Issue Type: Bug > Components: other >Affects Versions: 1.3.0 > Environment: 3 Node ant cluster >Reporter: Ajeet Rai >Assignee: Vandana Yadav >Priority: Minor > Labels: Functional > > By default dictionary is not created for string column. > steps: > 1: create a table with one column of string data type: > create table check_dict(id int, name string) > 2: insert into check_dict select 1,'abc' > 3: describe the table to check dictionary column: > desc formatted check_dict; > 4: Observe that name column is not DICTIONARY column. > Issue: This is not as per document. > Link: https://carbondata.apache.org/ddl-operation-on-carbondata.html > Expected : Dictionary encoding is enabled by default for all String columns, > and disabled for non-String columns -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (CARBONDATA-1708) Carbon1.3.0 Dictionary creation: By default dictionary is not created for string column
[ https://issues.apache.org/jira/browse/CARBONDATA-1708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajeet Rai closed CARBONDATA-1708. - Resolution: Invalid Working as per the latest documentation. Hence closing the defect. > Carbon1.3.0 Dictionary creation: By default dictionary is not created for > string column > --- > > Key: CARBONDATA-1708 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1708 > Project: CarbonData > Issue Type: Bug > Components: other >Affects Versions: 1.3.0 > Environment: 3 Node ant cluster >Reporter: Ajeet Rai >Assignee: Vandana Yadav >Priority: Minor > Labels: Functional > > By default dictionary is not created for string column. > steps: > 1: create a table with one column of string data type: > create table check_dict(id int, name string) > 2: insert into check_dict select 1,'abc' > 3: describe the table to check dictionary column: > desc formatted check_dict; > 4: Observe that name column is not DICTIONARY column. > Issue: This is not as per document. > Link: https://carbondata.apache.org/ddl-operation-on-carbondata.html > Expected : Dictionary encoding is enabled by default for all String columns, > and disabled for non-String columns -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (CARBONDATA-1735) Carbon1.3.0 Load: Segment created during load is not marked for delete if beeline session is closed while load is still in progress
[ https://issues.apache.org/jira/browse/CARBONDATA-1735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajeet Rai updated CARBONDATA-1735: -- Description: Load: Segment created during load is not marked for delete if beeline session is closed while load is still in progress. Steps: 1: Create a table with dictionary include 2: Start a load job 3: close the beeline session when global dictionary generation job is still in progress. 4: Observe that global dictionary generation job is completed but next job is not triggered. 5: Also observe that table status file is not updated and status of job is still in progress. 6: show segment will show this segment with status as in progress. Expected behaviour: Either job should be completed or load should fail and segment should be marked for delete. was: Load: Segment created during load is not marked for delete if beeline session is closed while load is still in progress. Steps: 1: Create a table 2: Start a load job 3: close the beeline session when global dictionary generation job is still in progress. 4: Observe that global dictionary generation job is completed but next job is not triggered. 5: Also observe that table status file is not updated and status of job is still in progress. 6: show segment will show this segment with status as in progress. Expected behaviour: Either job should be completed or load should fail and segment should be marked for delete. > Carbon1.3.0 Load: Segment created during load is not marked for delete if > beeline session is closed while load is still in progress > > > Key: CARBONDATA-1735 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1735 > Project: CarbonData > Issue Type: Bug > Components: data-load >Affects Versions: 1.3.0 > Environment: 3 Node ant cluster >Reporter: Ajeet Rai >Priority: Minor > Labels: DFX > > Load: Segment created during load is not marked for delete if beeline session > is closed while load is still in progress. > Steps: > 1: Create a table with dictionary include > 2: Start a load job > 3: close the beeline session when global dictionary generation job is still > in progress. > 4: Observe that global dictionary generation job is completed but next job is > not triggered. > 5: Also observe that table status file is not updated and status of job is > still in progress. > 6: show segment will show this segment with status as in progress. > Expected behaviour: Either job should be completed or load should fail and > segment should be marked for delete. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1806) Carbon1.3.0 Load with global sort: Load fails If a table is created with sort scope as global sort
Ajeet Rai created CARBONDATA-1806:
-
Summary: Carbon1.3.0 Load with global sort: Load fails If a table
is created with sort scope as global sort
Key: CARBONDATA-1806
URL: https://issues.apache.org/jira/browse/CARBONDATA-1806
Project: CarbonData
Issue Type: Bug
Affects Versions: 1.3.0
Environment: 3 node cluster
Reporter: Ajeet Rai
Carbon1.3.0 Load with global sort: Load fails If a table is created with sort
scope as global sort.
Steps:
1: create table dt1 (c1 string, c2 int) STORED BY
'org.apache.carbondata.format' tblproperties('sort_scope'='Global_sort');
2: LOAD DATA INPATH 'hdfs://hacluster/user/test/dt1.txt' INTO TABLE dt1
OPTIONS('DELIMITER'=',', 'QUOTECHAR'= '\"');
3: Observe that lod fails with error:
Error: java.lang.Exception: DataLoad failure (state=,code=0)
4: Check log:
org.apache.carbondata.processing.loading.exception.CarbonDataLoadingException:
There is an unexpected error:
org.apache.carbondata.core.datastore.exception.CarbonDataWriterException
at
org.apache.carbondata.spark.load.DataLoadProcessorStepOnSpark$.writeFunc(DataLoadProcessorStepOnSpark.scala:198)
at
org.apache.carbondata.spark.load.DataLoadProcessBuilderOnSpark$$anonfun$loadDataUsingGlobalSort$1.apply(DataLoadProcessBuilderOnSpark.scala:130)
at
org.apache.carbondata.spark.load.DataLoadProcessBuilderOnSpark$$anonfun$loadDataUsingGlobalSort$1.apply(DataLoadProcessBuilderOnSpark.scala:129)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Suppressed: org.apache.spark.util.TaskCompletionListenerException:
There is an unexpected error:
org.apache.carbondata.core.datastore.exception.CarbonDataWriterException
Previous exception in task: There is an unexpected error:
org.apache.carbondata.core.datastore.exception.CarbonDataWriterException
org.apache.carbondata.spark.load.DataLoadProcessorStepOnSpark$.writeFunc(DataLoadProcessorStepOnSpark.scala:198)
org.apache.carbondata.spark.load.DataLoadProcessBuilderOnSpark$$anonfun$loadDataUsingGlobalSort$1.apply(DataLoadProcessBuilderOnSpark.scala:130)
org.apache.carbondata.spark.load.DataLoadProcessBuilderOnSpark$$anonfun$loadDataUsingGlobalSort$1.apply(DataLoadProcessBuilderOnSpark.scala:129)
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
org.apache.spark.scheduler.Task.run(Task.scala:99)
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
at
org.apache.spark.TaskContextImpl.invokeListeners(TaskContextImpl.scala:138)
at
org.apache.spark.TaskContextImpl.markTaskFailed(TaskContextImpl.scala:106)
at org.apache.spark.scheduler.Task.run(Task.scala:104)
... 4 more
Caused by:
org.apache.carbondata.processing.loading.exception.CarbonDataLoadingException:
org.apache.carbondata.core.datastore.exception.CarbonDataWriterException
at
org.apache.carbondata.processing.loading.steps.DataWriterProcessorStepImpl.processingComplete(DataWriterProcessorStepImpl.java:163)
at
org.apache.carbondata.processing.loading.steps.DataWriterProcessorStepImpl.finish(DataWriterProcessorStepImpl.java:149)
at
org.apache.carbondata.spark.load.DataLoadProcessorStepOnSpark$.writeFunc(DataLoadProcessorStepOnSpark.scala:189)
... 8 more
Caused by:
org.apache.carbondata.core.datastore.exception.CarbonDataWriterException:
org.apache.carbondata.core.datastore.exception.CarbonDataWriterException
at
org.apache.carbondata.processing.store.CarbonFactDataHandlerColumnar.processWriteTaskSubmitList(CarbonFactDataHandlerColumnar.java:326)
at
org.apache.carbondata.processing.store.CarbonFactDataHandlerColumnar.closeHandler(CarbonFactDataHandlerColumnar.java:350)
at
org.apache.carbondata.processing.loading.steps.DataWriterProcessorStepImpl.processingComplete(DataWriterProcessorStepImpl.java:160)
... 10 more
Caused by: java.util.concurrent.ExecutionException:
org.apache.carbondata.core.datastore.exception.CarbonDataWriterException
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
[jira] [Updated] (CARBONDATA-1802) Carbon1.3.0 Alter:Alter query fails if a column is dropped and there is no key column
[ https://issues.apache.org/jira/browse/CARBONDATA-1802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajeet Rai updated CARBONDATA-1802: -- Description: Carbon1.3.0 Alter:Alter query fails if a column is dropped and there is no key column. Steps: 1: create table ttt(c int,d int,e int) stored by 'carbondata'; 2: Alter table ttt drop columns(c); 3: observe that below error is coming: Error: java.lang.RuntimeException: Alter table drop column operation failed: Alter drop operation failed. AtLeast one key column should exist after drop. Expected: Since user is able to create a table with all numeric columns, Same should be supported in Alter feature. was: Carbon1.3.0 Alter:Alter query fails if a column is dropped and there is no key column. Steps: 1: create table ttt(c int,d int,e int) stored by 'carbondata'; 2: Alter table ttt drop columns(c); 3: observe that below error is coming: Error: java.lang.RuntimeException: Alter table drop column operation failed: Alter drop operation failed. AtLeast one key column should exist after drop. > Carbon1.3.0 Alter:Alter query fails if a column is dropped and there is no > key column > -- > > Key: CARBONDATA-1802 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1802 > Project: CarbonData > Issue Type: Bug >Affects Versions: 1.3.0 > Environment: 3 Node ant cluster >Reporter: Ajeet Rai > Labels: functional > > Carbon1.3.0 Alter:Alter query fails if a column is dropped and there is no > key column. > Steps: > 1: create table ttt(c int,d int,e int) stored by 'carbondata'; > 2: Alter table ttt drop columns(c); > 3: observe that below error is coming: > Error: java.lang.RuntimeException: Alter table drop column operation failed: > Alter drop operation failed. AtLeast one key column should exist after drop. > Expected: Since user is able to create a table with all numeric columns, Same > should be supported in Alter feature. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1802) Carbon1.3.0 Alter:Alter query fails if a column is dropped and there is no key column
Ajeet Rai created CARBONDATA-1802: - Summary: Carbon1.3.0 Alter:Alter query fails if a column is dropped and there is no key column Key: CARBONDATA-1802 URL: https://issues.apache.org/jira/browse/CARBONDATA-1802 Project: CarbonData Issue Type: Bug Affects Versions: 1.3.0 Environment: 3 Node ant cluster Reporter: Ajeet Rai Carbon1.3.0 Alter:Alter query fails if a column is dropped and there is no key column. Steps: 1: create table ttt(c int,d int,e int) stored by 'carbondata'; 2: Alter table ttt drop columns(c); 3: observe that below error is coming: Error: java.lang.RuntimeException: Alter table drop column operation failed: Alter drop operation failed. AtLeast one key column should exist after drop. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1791) Carbon1.3.0 Concurrent Load-Alter: user is able to Alter table even if insert/load job is running
Ajeet Rai created CARBONDATA-1791: - Summary: Carbon1.3.0 Concurrent Load-Alter: user is able to Alter table even if insert/load job is running Key: CARBONDATA-1791 URL: https://issues.apache.org/jira/browse/CARBONDATA-1791 Project: CarbonData Issue Type: Bug Components: data-load Environment: 3 Node ant cluster Reporter: Ajeet Rai Fix For: 1.3.0 Carbon1.3.0 Concurrent Load-Alter: user is able to Alter table even if insert/load job is running. Steps: 1: Create a table 2: Start a insert job 3: Concurrently Alter the table(add,drop,rename) 4: Observe that alter is success 5: Observe that insert job is running and after some times job fails if table is renamed otherwise alter is success(for add,drop column) Expected behvaiour: drop job should wait for insert job to complete -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (CARBONDATA-1789) Carbon1.3.0 Concurrent Load-Drop: user is able to drop table even if insert/load job is running
[ https://issues.apache.org/jira/browse/CARBONDATA-1789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajeet Rai updated CARBONDATA-1789: -- Labels: dfx (was: ) > Carbon1.3.0 Concurrent Load-Drop: user is able to drop table even if > insert/load job is running > --- > > Key: CARBONDATA-1789 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1789 > Project: CarbonData > Issue Type: Bug > Components: data-load > Environment: 3 Node ant cluster >Reporter: Ajeet Rai > Labels: dfx > Fix For: 1.3.0 > > > Carbon1.3.0 Concurrent Load-Drop: user is able to drop table even if > insert/load job is running > Steps: > 1: Create a table > 2: Start a insert job > 3: Concurrently drop the table > 4: Observe that drop is success > 5: Observe that insert job is running and after some times job fails > Expected behvaiour: drop job should wait for insert job to complete -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1789) Carbon1.3.0 Concurrent Load-Drop: user is able to drop table even if insert/load job is running
Ajeet Rai created CARBONDATA-1789: - Summary: Carbon1.3.0 Concurrent Load-Drop: user is able to drop table even if insert/load job is running Key: CARBONDATA-1789 URL: https://issues.apache.org/jira/browse/CARBONDATA-1789 Project: CarbonData Issue Type: Bug Components: data-load Environment: 3 Node ant cluster Reporter: Ajeet Rai Fix For: 1.3.0 Carbon1.3.0 Concurrent Load-Drop: user is able to drop table even if insert/load job is running Steps: 1: Create a table 2: Start a insert job 3: Concurrently drop the table 4: Observe that drop is success 5: Observe that insert job is running and after some times job fails Expected behvaiour: drop job should wait for insert job to complete -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1761) (Carbon1.3.0 - DELETE SEGMENT BY ID) In Progress Segment is marked for delete if respective id is given in delete segment by id query
Ajeet Rai created CARBONDATA-1761:
-
Summary: (Carbon1.3.0 - DELETE SEGMENT BY ID) In Progress Segment
is marked for delete if respective id is given in delete segment by id query
Key: CARBONDATA-1761
URL: https://issues.apache.org/jira/browse/CARBONDATA-1761
Project: CarbonData
Issue Type: Bug
Components: data-load
Affects Versions: 1.3.0
Environment: 3 Node ant cluster
Description
Reporter: Ajeet Rai
(Carbon1.3.0 - DELETE SEGMENT BY ID) In Progress Segment is marked for delete
if respective id is given in delete segment by id query.
1: Create a table
CREATE TABLE IF NOT EXISTS flow_carbon_new999(txn_dte String,dt String,txn_bk
String,txn_br String,own_bk String,own_br String,opp_bk String,bus_opr_cde
String,opt_prd_cde String,cus_no String,cus_ac String,opp_ac_nme String,opp_ac
String,bv_no String,aco_ac String,ac_dte String,txn_cnt int,jrn_par
int,mfm_jrn_no String,cbn_jrn_no String,ibs_jrn_no String,vch_no String,vch_seq
String,srv_cde String,bus_cd_no String,id_flg String,bv_cde String,txn_time
String,txn_tlr String,ety_tlr String,ety_bk String,ety_br String,bus_pss_no
String,chk_flg String,chk_tlr String,chk_jrn_no String, bus_sys_no
String,txn_sub_cde String,fin_bus_cde String,fin_bus_sub_cde String,chl
String,tml_id String,sus_no String,sus_seq String, cho_seq String, itm_itm
String,itm_sub String,itm_sss String,dc_flg String,amt decimal(15,2),bal
decimal(15,2),ccy String,spv_flg String,vch_vld_dte String,pst_bk String,pst_br
String,ec_flg String,aco_tlr String,gen_flg String,his_rec_sum_flg
String,his_flg String,vch_typ String,val_dte String,opp_ac_flg String,cmb_flg
String,ass_vch_flg String,cus_pps_flg String,bus_rmk_cde String,vch_bus_rmk
String,tec_rmk_cde String,vch_tec_rmk String,gems_last_upd_d String,maps_date
String,maps_job String)STORED BY 'org.apache.carbondata.format'
TBLPROPERTIES('DICTIONARY_INCLUDE'='txn_cnt,jrn_par,amt,bal','No_Inverted_Index'=
'txn_dte,dt,txn_bk,txn_br,own_bk ,own_br ,opp_bk ,bus_opr_cde ,opt_prd_cde
,cus_no ,cus_ac ,opp_ac_nme ,opp_ac ,bv_no ,aco_ac ,ac_dte ,txn_cnt ,jrn_par
,mfm_jrn_no ,cbn_jrn_no ,ibs_jrn_no ,vch_no ,vch_seq ,srv_cde ,bus_cd_no
,id_flg ,bv_cde ,txn_time ,txn_tlr ,ety_tlr ,ety_bk ,ety_br ,bus_pss_no
,chk_flg ,chk_tlr ,chk_jrn_no , bus_sys_no ,txn_sub_cde ,fin_bus_cde
,fin_bus_sub_cde ,chl ,tml_id ,sus_no ,sus_seq , cho_seq , itm_itm ,itm_sub
,itm_sss ,dc_flg ,amt,bal,ccy ,spv_flg ,vch_vld_dte ,pst_bk ,pst_br ,ec_flg
,aco_tlr ,gen_flg ,his_rec_sum_flg ,his_flg ,vch_typ ,val_dte ,opp_ac_flg
,cmb_flg ,ass_vch_flg ,cus_pps_flg ,bus_rmk_cde ,vch_bus_rmk ,tec_rmk_cde
,vch_tec_rmk ,gems_last_upd_d ,maps_date ,maps_job' );
2: start a data load.
LOAD DATA inpath 'hdfs://hacluster/user/test/20140101_1_1.csv' into
table flow_carbon_new999 options('DELIMITER'=',',
'QUOTECHAR'='"','header'='false');
3: run a insert into/overwrite job
insert into table flow_carbon_new999 select * from flow_carbon_new666;
4: show segments for table flow_carbon_new999;
5: Observe that load/insert/overwrite job is started with new segment id
6: now run a delete segment by id query with this id.
DELETE FROM TABLE ajeet.flow_carbon_new999 WHERE SEGMENT.ID IN (34)
7: again run show segment and see this segment which is still in progress is
marked for delete.
8: Observe that insert/load job is still running and after some time(in next
job of load/insert/overwrite), this job fails with below error:
Error: java.lang.RuntimeException: It seems insert overwrite has been issued
during load (state=,code=0)
This is not correct behaviour and it should be handled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (CARBONDATA-1759) Carbon1.3.0 Clean command is not working correctly for segments marked for delete due to insert overwrite job
Ajeet Rai created CARBONDATA-1759:
-
Summary: Carbon1.3.0 Clean command is not working correctly for
segments marked for delete due to insert overwrite job
Key: CARBONDATA-1759
URL: https://issues.apache.org/jira/browse/CARBONDATA-1759
Project: CarbonData
Issue Type: Bug
Components: data-query
Affects Versions: 1.3.0
Environment: 3 Node ant cluster
Reporter: Ajeet Rai
Carbon1.3.0 Clean command is not working correctly for segments marked for
delete due to insert overwrite job.
1: Create a table
CREATE TABLE IF NOT EXISTS flow_carbon_new999(txn_dte String,dt String,txn_bk
String,txn_br String,own_bk String,own_br String,opp_bk String,bus_opr_cde
String,opt_prd_cde String,cus_no String,cus_ac String,opp_ac_nme String,opp_ac
String,bv_no String,aco_ac String,ac_dte String,txn_cnt int,jrn_par
int,mfm_jrn_no String,cbn_jrn_no String,ibs_jrn_no String,vch_no String,vch_seq
String,srv_cde String,bus_cd_no String,id_flg String,bv_cde String,txn_time
String,txn_tlr String,ety_tlr String,ety_bk String,ety_br String,bus_pss_no
String,chk_flg String,chk_tlr String,chk_jrn_no String, bus_sys_no
String,txn_sub_cde String,fin_bus_cde String,fin_bus_sub_cde String,chl
String,tml_id String,sus_no String,sus_seq String, cho_seq String, itm_itm
String,itm_sub String,itm_sss String,dc_flg String,amt decimal(15,2),bal
decimal(15,2),ccy String,spv_flg String,vch_vld_dte String,pst_bk
String,pst_br String,ec_flg String,aco_tlr String,gen_flg
String,his_rec_sum_flg String,his_flg String,vch_typ String,val_dte
String,opp_ac_flg String,cmb_flg String,ass_vch_flg String,cus_pps_flg
String,bus_rmk_cde String,vch_bus_rmk String,tec_rmk_cde String,vch_tec_rmk
String,gems_last_upd_d String,maps_date String,maps_job String)STORED BY
'org.apache.carbondata.format'
TBLPROPERTIES('DICTIONARY_INCLUDE'='txn_cnt,jrn_par,amt,bal','No_Inverted_Index'=
'txn_dte,dt,txn_bk,txn_br,own_bk ,own_br ,opp_bk ,bus_opr_cde ,opt_prd_cde
,cus_no ,cus_ac ,opp_ac_nme ,opp_ac ,bv_no ,aco_ac ,ac_dte ,txn_cnt ,jrn_par
,mfm_jrn_no ,cbn_jrn_no ,ibs_jrn_no ,vch_no ,vch_seq ,srv_cde ,bus_cd_no
,id_flg ,bv_cde ,txn_time ,txn_tlr ,ety_tlr ,ety_bk ,ety_br ,bus_pss_no
,chk_flg ,chk_tlr ,chk_jrn_no , bus_sys_no ,txn_sub_cde ,fin_bus_cde
,fin_bus_sub_cde ,chl ,tml_id ,sus_no ,sus_seq , cho_seq , itm_itm ,itm_sub
,itm_sss ,dc_flg ,amt,bal,ccy ,spv_flg ,vch_vld_dte ,pst_bk ,pst_br ,ec_flg
,aco_tlr ,gen_flg ,his_rec_sum_flg ,his_flg ,vch_typ ,val_dte ,opp_ac_flg
,cmb_flg ,ass_vch_flg ,cus_pps_flg ,bus_rmk_cde ,vch_bus_rmk ,tec_rmk_cde
,vch_tec_rmk ,gems_last_upd_d ,maps_date ,maps_job' );
2: start a data load.
LOAD DATA inpath 'hdfs://hacluster/user/test/20140101_1_1.csv' into
table flow_carbon_new999 options('DELIMITER'=',',
'QUOTECHAR'='"','header'='false');
3: run a insert overwrite job
insert into table flow_carbon_new999 select * from flow_carbon_new666;
4: run show segment query:
show segments for table ajeet.flow_carbon_new999
5: Observe that all previous segments are marked for delete
6: run clean query
CLEAN FILES FOR TABLE ajeet.flow_carbon_new999;
7: again run show segment query
8: Observe that still all previous segments which are marked for delete are
shown as result.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (CARBONDATA-1755) Carbon1.3.0 Concurrent Insert overwrite-update: User is able to run insert overwrite and update job concurrently.
Ajeet Rai created CARBONDATA-1755: - Summary: Carbon1.3.0 Concurrent Insert overwrite-update: User is able to run insert overwrite and update job concurrently. Key: CARBONDATA-1755 URL: https://issues.apache.org/jira/browse/CARBONDATA-1755 Project: CarbonData Issue Type: Bug Components: data-load Affects Versions: 1.3.0 Environment: 3 Node ant cluster Reporter: Ajeet Rai Priority: Minor Carbon1.3.0 Concurrent Insert overwrite-update: User is able to run insert overwrite and update job concurrently. updated data will be overwritten by insert overwrite job. So there is no meaning of running update job if insert overwrite is in progress. Steps: 1: Create a table 2: Do a data load 3: run insert overwrite job. 4: run a update job while overwrite job is still running. 5: Observe that update job is finished and after that overwrite job is also finished. 6: All previous segments are marked for delete and there is no impact of update job. Update job will use the resources unnecessary. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (CARBONDATA-1754) Carbon1.3.0 Concurrent Insert overwrite-Compaction: Compaction job fails at run time if insert overwrite job is running concurrentlyInsert overwrite
[ https://issues.apache.org/jira/browse/CARBONDATA-1754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajeet Rai updated CARBONDATA-1754: -- Description: Carbon1.3.0 Concurrent Insert overwrite-Compaction: Compaction job fails at run time if insert overwrite job is running concurrently. Steps: 1: Create a table 2: Start three load one by one 3: After load is completed, start insert overwrite and minor compaction concurrently from two different session 4: observe that both jobs are are running 5: Observe that Insert overwrite job is success but after that compaction fails with below exception: | ERROR | [pool-23-thread-49] | Error running hive query: | org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1$$anon$2.run(SparkExecuteStatementOperation.scala:167) org.apache.hive.service.cli.HiveSQLException: java.lang.RuntimeException: Compaction failed. Please check logs for more info. Exception in compaction java.lang.Exception: Compaction failed to update metadata for table ajeet.flow_carbon_new999 7: Ideally compaction job should give error in start with message that insert overwrite in progress. was: Carbon1.3.0 Concurrent Load-Compaction: Compaction job fails at run time if insert overwrite job is running concurrently. Steps: 1: Create a table 2: Start three load one by one 3: After load is completed, start insert overwrite and minor compaction concurrently from two different session 4: observe that both jobs are are running 5: Observe that Insert overwrite job is success but after that compaction fails with below exception: | ERROR | [pool-23-thread-49] | Error running hive query: | org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1$$anon$2.run(SparkExecuteStatementOperation.scala:167) org.apache.hive.service.cli.HiveSQLException: java.lang.RuntimeException: Compaction failed. Please check logs for more info. Exception in compaction java.lang.Exception: Compaction failed to update metadata for table ajeet.flow_carbon_new999 7: Ideally compaction job should give error in start with message that insert overwrite in progress. > Carbon1.3.0 Concurrent Insert overwrite-Compaction: Compaction job fails at > run time if insert overwrite job is running concurrentlyInsert overwrite > > > Key: CARBONDATA-1754 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1754 > Project: CarbonData > Issue Type: Bug > Components: data-load >Affects Versions: 1.3.0 > Environment: 3 Node ant cluster >Reporter: Ajeet Rai > Labels: dfx > > Carbon1.3.0 Concurrent Insert overwrite-Compaction: Compaction job fails at > run time if insert overwrite job is running concurrently. > Steps: > 1: Create a table > 2: Start three load one by one > 3: After load is completed, start insert overwrite and minor compaction > concurrently from two different session > 4: observe that both jobs are are running > 5: Observe that Insert overwrite job is success but after that compaction > fails with below exception: > | ERROR | [pool-23-thread-49] | Error running hive query: | > org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1$$anon$2.run(SparkExecuteStatementOperation.scala:167) > org.apache.hive.service.cli.HiveSQLException: java.lang.RuntimeException: > Compaction failed. Please check logs for more info. Exception in compaction > java.lang.Exception: Compaction failed to update metadata for table > ajeet.flow_carbon_new999 > 7: Ideally compaction job should give error in start with message that insert > overwrite in progress. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (CARBONDATA-1754) Carbon1.3.0 Concurrent Insert overwrite-Compaction: Compaction job fails at run time if insert overwrite job is running concurrentlyInsert overwrite
[ https://issues.apache.org/jira/browse/CARBONDATA-1754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajeet Rai updated CARBONDATA-1754: -- Summary: Carbon1.3.0 Concurrent Insert overwrite-Compaction: Compaction job fails at run time if insert overwrite job is running concurrentlyInsert overwrite (was: Carbon1.3.0 Concurrent Load-Compaction: Compaction job fails at run time if insert overwrite job is running concurrently) > Carbon1.3.0 Concurrent Insert overwrite-Compaction: Compaction job fails at > run time if insert overwrite job is running concurrentlyInsert overwrite > > > Key: CARBONDATA-1754 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1754 > Project: CarbonData > Issue Type: Bug > Components: data-load >Affects Versions: 1.3.0 > Environment: 3 Node ant cluster >Reporter: Ajeet Rai > Labels: dfx > > Carbon1.3.0 Concurrent Load-Compaction: Compaction job fails at run time if > insert overwrite job is running concurrently. > Steps: > 1: Create a table > 2: Start three load one by one > 3: After load is completed, start insert overwrite and minor compaction > concurrently from two different session > 4: observe that both jobs are are running > 5: Observe that Insert overwrite job is success but after that compaction > fails with below exception: > | ERROR | [pool-23-thread-49] | Error running hive query: | > org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1$$anon$2.run(SparkExecuteStatementOperation.scala:167) > org.apache.hive.service.cli.HiveSQLException: java.lang.RuntimeException: > Compaction failed. Please check logs for more info. Exception in compaction > java.lang.Exception: Compaction failed to update metadata for table > ajeet.flow_carbon_new999 > 7: Ideally compaction job should give error in start with message that insert > overwrite in progress. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (CARBONDATA-1754) Carbon1.3.0 Concurrent Load-Compaction: Compaction job fails at run time if insert overwrite job is running concurrently
[ https://issues.apache.org/jira/browse/CARBONDATA-1754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajeet Rai updated CARBONDATA-1754: -- Description: Carbon1.3.0 Concurrent Load-Compaction: Compaction job fails at run time if insert overwrite job is running concurrently. Steps: 1: Create a table 2: Start three load one by one 3: After load is completed, start insert overwrite and minor compaction concurrently from two different session 4: observe that both jobs are are running 5: Observe that Insert overwrite job is success but after that compaction fails with below exception: | ERROR | [pool-23-thread-49] | Error running hive query: | org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1$$anon$2.run(SparkExecuteStatementOperation.scala:167) org.apache.hive.service.cli.HiveSQLException: java.lang.RuntimeException: Compaction failed. Please check logs for more info. Exception in compaction java.lang.Exception: Compaction failed to update metadata for table ajeet.flow_carbon_new999 7: Ideally compaction job should give error in start with message that insert overwrite in progress. was: Carbon1.3.0 Concurrent Load-Compaction: Compaction job fails at run time if insert overwrite job is running concurrently. Steps: 1: Create a table 2: Start three load one by one 3: After load is completed, start insert overwrite and minor compaction concurrently from two different session 4: observe that both jobs are are running 5: Observe that Insert overwrite job is success but after that compaction fails with below exception: | ERROR | [pool-23-thread-49] | Error running hive query: | org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1$$anon$2.run(SparkExecuteStatementOperation.scala:167) org.apache.hive.service.cli.HiveSQLException: java.lang.RuntimeException: Compaction failed. Please check logs for more info. Exception in compaction java.lang.Exception: Compaction failed to update metadata for table ajeet.flow_carbon_new999 7: Ideally compaction job should give error with message that insert overwrite in progress. > Carbon1.3.0 Concurrent Load-Compaction: Compaction job fails at run time if > insert overwrite job is running concurrently > > > Key: CARBONDATA-1754 > URL: https://issues.apache.org/jira/browse/CARBONDATA-1754 > Project: CarbonData > Issue Type: Bug > Components: data-load >Affects Versions: 1.3.0 > Environment: 3 Node ant cluster >Reporter: Ajeet Rai > Labels: dfx > > Carbon1.3.0 Concurrent Load-Compaction: Compaction job fails at run time if > insert overwrite job is running concurrently. > Steps: > 1: Create a table > 2: Start three load one by one > 3: After load is completed, start insert overwrite and minor compaction > concurrently from two different session > 4: observe that both jobs are are running > 5: Observe that Insert overwrite job is success but after that compaction > fails with below exception: > | ERROR | [pool-23-thread-49] | Error running hive query: | > org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1$$anon$2.run(SparkExecuteStatementOperation.scala:167) > org.apache.hive.service.cli.HiveSQLException: java.lang.RuntimeException: > Compaction failed. Please check logs for more info. Exception in compaction > java.lang.Exception: Compaction failed to update metadata for table > ajeet.flow_carbon_new999 > 7: Ideally compaction job should give error in start with message that insert > overwrite in progress. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1754) Carbon1.3.0 Concurrent Load-Compaction: Compaction job fails at run time if insert overwrite job is running concurrently
Ajeet Rai created CARBONDATA-1754: - Summary: Carbon1.3.0 Concurrent Load-Compaction: Compaction job fails at run time if insert overwrite job is running concurrently Key: CARBONDATA-1754 URL: https://issues.apache.org/jira/browse/CARBONDATA-1754 Project: CarbonData Issue Type: Bug Components: data-load Affects Versions: 1.3.0 Environment: 3 Node ant cluster Reporter: Ajeet Rai Carbon1.3.0 Concurrent Load-Compaction: Compaction job fails at run time if insert overwrite job is running concurrently. Steps: 1: Create a table 2: Start three load one by one 3: After load is completed, start insert overwrite and minor compaction concurrently from two different session 4: observe that both jobs are are running 5: Observe that Insert overwrite job is success but after that compaction fails with below exception: | ERROR | [pool-23-thread-49] | Error running hive query: | org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1$$anon$2.run(SparkExecuteStatementOperation.scala:167) org.apache.hive.service.cli.HiveSQLException: java.lang.RuntimeException: Compaction failed. Please check logs for more info. Exception in compaction java.lang.Exception: Compaction failed to update metadata for table ajeet.flow_carbon_new999 7: Ideally compaction job should give error with message that insert overwrite in progress. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1744) Carbon1.3.0 Concurrent Load-Delete:Delete query is not working correctly if load is already in process.
Ajeet Rai created CARBONDATA-1744: - Summary: Carbon1.3.0 Concurrent Load-Delete:Delete query is not working correctly if load is already in process. Key: CARBONDATA-1744 URL: https://issues.apache.org/jira/browse/CARBONDATA-1744 Project: CarbonData Issue Type: Bug Components: sql Affects Versions: 1.3.0 Environment: 3 Node ant cluster Reporter: Ajeet Rai Priority: Minor Concurrent Load-Delete:Delete query is not working correctly if load is already in process. steps: 1:Create a table 2: Start a large data load 3: Execute delete query from another session(delete from table_name) 4: Observe that Delete operation doesn't give any error and completed as success. 5: Execute show segment query and observe that status of current segment is in progress. 6: execute delete query again once load is completed. 7: Observe that delete is success but segments are not marked for delete. Current status is still success which is wrong. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1735) Carbon1.3.0 Load: Segment created during load is not marked for delete if beeline session is closed while load is still in progress
Ajeet Rai created CARBONDATA-1735: - Summary: Carbon1.3.0 Load: Segment created during load is not marked for delete if beeline session is closed while load is still in progress Key: CARBONDATA-1735 URL: https://issues.apache.org/jira/browse/CARBONDATA-1735 Project: CarbonData Issue Type: Bug Components: data-load Affects Versions: 1.3.0 Environment: 3 Node ant cluster Reporter: Ajeet Rai Priority: Minor Load: Segment created during load is not marked for delete if beeline session is closed while load is still in progress. Steps: 1: Create a table 2: Start a load job 3: close the beeline session when global dictionary generation job is still in progress. 4: Observe that global dictionary generation job is completed but next job is not triggered. 5: Also observe that table status file is not updated and status of job is still in progress. 6: show segment will show this segment with status as in progress. Expected behaviour: Either job should be completed or load should fail and segment should be marked for delete. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (CARBONDATA-1708) Carbon1.3.0 Dictionary creation: By default dictionary is not created for string column
Ajeet Rai created CARBONDATA-1708: - Summary: Carbon1.3.0 Dictionary creation: By default dictionary is not created for string column Key: CARBONDATA-1708 URL: https://issues.apache.org/jira/browse/CARBONDATA-1708 Project: CarbonData Issue Type: Bug Components: other Affects Versions: 1.3.0 Environment: 3 Node ant cluster Reporter: Ajeet Rai Priority: Minor By default dictionary is not created for string column. steps: 1: create a table with one column of string data type: create table check_dict(id int, name string) 2: insert into check_dict select 1,'abc' 3: describe the table to check dictionary column: desc formatted check_dict; 4: Observe that name column is not DICTIONARY column. Issue: This is not as per document. Link: https://carbondata.apache.org/ddl-operation-on-carbondata.html Expected : Dictionary encoding is enabled by default for all String columns, and disabled for non-String columns -- This message was sent by Atlassian JIRA (v6.4.14#64029)
