Vandana Yadav created CARBONDATA-2079:
-----------------------------------------

             Summary: Error displays while executing minor compaction on the 
cluster
                 Key: CARBONDATA-2079
                 URL: https://issues.apache.org/jira/browse/CARBONDATA-2079
             Project: CarbonData
          Issue Type: Bug
          Components: data-load
    Affects Versions: 1.3.0
         Environment: spark 2.1
            Reporter: Vandana Yadav
         Attachments: 7000_UniqData.csv

Error displays while executing minor compaction on the cluster:

Steps to Reproduce:

1) Create Table:

CREATE TABLE uniqdata_batchsort_compact (CUST_ID int,CUST_NAME 
String,ACTIVE_EMUI_VERSION string, DOB timestamp, DOJ timestamp, BIGINT_COLUMN1 
bigint,BIGINT_COLUMN2 bigint,DECIMAL_COLUMN1 decimal(30,10), DECIMAL_COLUMN2 
decimal(36,10),Double_COLUMN1 double, Double_COLUMN2 double,INTEGER_COLUMN1 
int) STORED BY 'carbondata' TBLPROPERTIES('SORT_SCOPE'='BATCH_SORT')

2) Load Data :

LOAD DATA INPATH 'HDFS_URL/BabuStore/Data/uniqdata/7000_UniqData.csv' into 
table uniqdata_batchsort_compact OPTIONS('DELIMITER'=',' , 
'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1','batch_sort_size_inmb'='1')

LOAD DATA INPATH 'HDFS_URL/BabuStore/Data/uniqdata/7000_UniqData.csv' into 
table uniqdata_batchsort_compact OPTIONS('DELIMITER'=',' , 
'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1','batch_sort_size_inmb'='1')

LOAD DATA INPATH 'HDFS_URL/BabuStore/Data/uniqdata/7000_UniqData.csv' into 
table uniqdata_batchsort_compact OPTIONS('DELIMITER'=',' , 
'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1','batch_sort_size_inmb'='1')

LOAD DATA INPATH 'HDFS_URL/BabuStore/Data/uniqdata/7000_UniqData.csv' into 
table uniqdata_batchsort_compact OPTIONS('DELIMITER'=',' , 
'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1','batch_sort_size_inmb'='1')

3)Execute Queries:

alter table uniqdata_batchsort_compact compact 'minor'

output:

Compaction failed. Please check logs for more info. Exception in compaction / 
by zero

logs:

[exec] 18/01/24 16:47:33 INFO SelectQuery: Executing Query: alter table 
uniqdata_batchsort_compact compact 'minor'
 [exec] 18/01/24 16:47:33 INFO CarbonSparkSqlParser: Parsing command: alter 
table uniqdata_batchsort_compact compact 'minor'
 [exec] 18/01/24 16:47:33 INFO CarbonLateDecodeRule: main skip CarbonOptimizer
 [exec] 18/01/24 16:47:33 INFO CarbonLateDecodeRule: main Skip CarbonOptimizer
 [exec] 18/01/24 16:47:33 INFO HiveMetaStore: 0: get_table : db=default 
tbl=uniqdata_batchsort_compact
 [exec] 18/01/24 16:47:33 INFO audit: ugi=root ip=unknown-ip-addr cmd=get_table 
: db=default tbl=uniqdata_batchsort_compact 
 [exec] 18/01/24 16:47:33 INFO CatalystSqlParser: Parsing command: array<string>
 [exec] 18/01/24 16:47:33 INFO HiveMetaStore: 0: get_table : db=default 
tbl=uniqdata_batchsort_compact
 [exec] 18/01/24 16:47:33 INFO audit: ugi=root ip=unknown-ip-addr cmd=get_table 
: db=default tbl=uniqdata_batchsort_compact 
 [exec] 18/01/24 16:47:33 INFO CatalystSqlParser: Parsing command: array<string>
 [exec] 18/01/24 16:47:33 AUDIT CarbonAlterTableCompactionCommand: 
[hadoop-master][root][Thread-1]Compaction request received for table 
default.uniqdata_batchsort_compact
 [exec] 18/01/24 16:47:33 INFO HdfsFileLock: main HDFS lock 
path:hdfs://hadoop-master:54311//opt/CarbonStore/default/uniqdata_batchsort_compact/compaction.lock
 [exec] 18/01/24 16:47:34 INFO CarbonAlterTableCompactionCommand: main Acquired 
the compaction lock for table default.uniqdata_batchsort_compact
 [exec] 18/01/24 16:47:34 INFO CarbonTableCompactor: main loads identified for 
merge is 0
 [exec] 18/01/24 16:47:34 INFO CarbonTableCompactor: main loads identified for 
merge is 1
 [exec] 18/01/24 16:47:34 INFO CarbonTableCompactor: main loads identified for 
merge is 2
 [exec] 18/01/24 16:47:34 INFO CarbonTableCompactor: main loads identified for 
merge is 3
 [exec] 18/01/24 16:47:34 INFO CarbonTableCompactor: main 
spark.executor.instances property is set to = 3
 [exec] 18/01/24 16:47:34 INFO TableInfo: main Table block size not specified 
for default_uniqdata_batchsort_compact. Therefore considering the default value 
1024 MB
 [exec] 18/01/24 16:47:34 INFO BlockletDataMap: main Time taken to load 
blocklet datamap from file : 
hdfs://hadoop-master:54311//opt/CarbonStore/default/uniqdata_batchsort_compact/Fact/Part0/Segment_0/0_batchno0-0-1516792651041.carbonindexis
 0
 [exec] 18/01/24 16:47:34 INFO BlockletDataMap: main Time taken to load 
blocklet datamap from file : 
hdfs://hadoop-master:54311//opt/CarbonStore/default/uniqdata_batchsort_compact/Fact/Part0/Segment_1/0_batchno0-0-1516792651780.carbonindexis
 1
 [exec] 18/01/24 16:47:34 INFO BlockletDataMap: main Time taken to load 
blocklet datamap from file : 
hdfs://hadoop-master:54311//opt/CarbonStore/default/uniqdata_batchsort_compact/Fact/Part0/Segment_2/0_batchno0-0-1516792652481.carbonindexis
 0
 [exec] 18/01/24 16:47:34 INFO BlockletDataMap: main Time taken to load 
blocklet datamap from file : 
hdfs://hadoop-master:54311//opt/CarbonStore/default/uniqdata_batchsort_compact/Fact/Part0/Segment_3/0_batchno0-0-1516792653232.carbonindexis
 0
 [exec] 18/01/24 16:47:34 ERROR CarbonTableCompactor: main Exception in 
compaction thread / by zero
 [exec] java.lang.ArithmeticException: / by zero
 [exec] at 
org.apache.carbondata.processing.util.CarbonLoaderUtil.nodeBlockMapping(CarbonLoaderUtil.java:524)
 [exec] at 
org.apache.carbondata.processing.util.CarbonLoaderUtil.nodeBlockMapping(CarbonLoaderUtil.java:453)
 [exec] at 
org.apache.carbondata.spark.rdd.CarbonMergerRDD.getPartitions(CarbonMergerRDD.scala:400)
 [exec] at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)
 [exec] at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250)
 [exec] at scala.Option.getOrElse(Option.scala:121)
 [exec] at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)
 [exec] at org.apache.spark.SparkContext.runJob(SparkContext.scala:1958)
 [exec] at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:935)
 [exec] at 
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
 [exec] at 
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
 [exec] at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
 [exec] at org.apache.spark.rdd.RDD.collect(RDD.scala:934)
 [exec] at 
org.apache.carbondata.spark.rdd.CarbonTableCompactor.triggerCompaction(CarbonTableCompactor.scala:211)
 [exec] at 
org.apache.carbondata.spark.rdd.CarbonTableCompactor.scanSegmentsAndSubmitJob(CarbonTableCompactor.scala:120)
 [exec] at 
org.apache.carbondata.spark.rdd.CarbonTableCompactor.executeCompaction(CarbonTableCompactor.scala:71)
 [exec] at 
org.apache.carbondata.spark.rdd.CarbonDataRDDFactory$$anon$2.run(CarbonDataRDDFactory.scala:182)
 [exec] at 
org.apache.carbondata.spark.rdd.CarbonDataRDDFactory$.startCompactionThreads(CarbonDataRDDFactory.scala:269)
 [exec] at 
org.apache.spark.sql.execution.command.management.CarbonAlterTableCompactionCommand.alterTableForCompaction(CarbonAlterTableCompactionCommand.scala:258)
 [exec] at 
org.apache.spark.sql.execution.command.management.CarbonAlterTableCompactionCommand.processData(CarbonAlterTableCompactionCommand.scala:111)
 [exec] at 
org.apache.spark.sql.execution.command.DataCommand.run(package.scala:71)
 [exec] at 
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58)
 [exec] at 
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56)
 [exec] at 
org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74)
 [exec] at 
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
 [exec] at 
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
 [exec] at 
org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:135)
 [exec] at 
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
 [exec] at 
org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:132)
 [exec] at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:113)
 [exec] at 
org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:87)
 [exec] at 
org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:87)
 [exec] at org.apache.spark.sql.Dataset.<init>(Dataset.scala:185)
 [exec] at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:64)
 [exec] at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:592)
 [exec] at com.huawei.spark.SessionManager.sql(SessionManager.java:42)
 [exec] at 
com.huawei.querymanagement.QueryManagement.sql(QueryManagement.java:62)
 [exec] at com.huawei.querymanagement.SelectQuery.testQuery(SelectQuery.java:70)
 [exec] at sun.reflect.GeneratedMethodAccessor66.invoke(Unknown Source)
 [exec] at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 [exec] at java.lang.reflect.Method.invoke(Method.java:498)
 [exec] at org.junit.internal.runners.TestMethod.invoke(TestMethod.java:59)
 [exec] at 
org.junit.internal.runners.MethodRoadie.runTestMethod(MethodRoadie.java:98)
 [exec] at org.junit.internal.runners.MethodRoadie$2.run(MethodRoadie.java:79)
 [exec] at 
org.junit.internal.runners.MethodRoadie.runBeforesThenTestThenAfters(MethodRoadie.java:87)
 [exec] at org.junit.internal.runners.MethodRoadie.runTest(MethodRoadie.java:77)
 [exec] at org.junit.internal.runners.MethodRoadie.run(MethodRoadie.java:42)
 [exec] at 
org.junit.internal.runners.JUnit4ClassRunner.invokeTestMethod(JUnit4ClassRunner.java:88)
 [exec] at 
org.junit.internal.runners.JUnit4ClassRunner.runMethods(JUnit4ClassRunner.java:51)
 [exec] at 
org.junit.runners.Parameterized$TestClassRunnerForParameters.run(Parameterized.java:98)
 [exec] at 
org.junit.internal.runners.CompositeRunner.runChildren(CompositeRunner.java:33)
 [exec] at org.junit.runners.Parameterized.access$000(Parameterized.java:55)
 [exec] at org.junit.runners.Parameterized$1.run(Parameterized.java:131)
 [exec] at 
org.junit.internal.runners.ClassRoadie.runUnprotected(ClassRoadie.java:27)
 [exec] at 
org.junit.internal.runners.ClassRoadie.runProtected(ClassRoadie.java:37)
 [exec] at org.junit.runners.Parameterized.run(Parameterized.java:129)
 [exec] at 
org.junit.internal.runners.CompositeRunner.runChildren(CompositeRunner.java:33)
 [exec] at 
org.junit.internal.runners.CompositeRunner.run(CompositeRunner.java:28)
 [exec] at org.junit.runner.JUnitCore.run(JUnitCore.java:130)
 [exec] at org.junit.runner.JUnitCore.run(JUnitCore.java:109)
 [exec] at org.junit.runner.JUnitCore.run(JUnitCore.java:100)
 [exec] at org.junit.runner.JUnitCore.runClasses(JUnitCore.java:60)
 [exec] at 
com.huawei.querymanagement.SelectQuerySuite.main(SelectQuerySuite.java:18)
 [exec] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 [exec] at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 [exec] at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 [exec] at java.lang.reflect.Method.invoke(Method.java:498)
 [exec] at 
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738)
 [exec] at 
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
 [exec] at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
 [exec] at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
 [exec] at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
 [exec] 18/01/24 16:47:34 ERROR CarbonDataRDDFactory$: main Exception in 
compaction thread / by zero
 [exec] 18/01/24 16:47:34 INFO HdfsFileLock: main Deleted the lock file 
hdfs://hadoop-master:54311//opt/CarbonStore/default/uniqdata_batchsort_compact/compaction.lock
 [exec] 18/01/24 16:47:34 ERROR CarbonAlterTableCompactionCommand: main 
Exception in start compaction thread. Exception in compaction / by zero
 [exec] 18/01/24 16:47:34 ERROR HdfsFileLock: main Not able to delete the lock 
file because it is not existed in location 
hdfs://hadoop-master:54311//opt/CarbonStore/default/uniqdata_batchsort_compact/compaction.lock
 [exec] 18/01/24 16:47:34 ERROR SelectQuery: An exception has occurred: 
 [exec] org.apache.spark.sql.AnalysisException: Compaction failed. Please check 
logs for more info. Exception in compaction / by zero;
 [exec] at 
org.apache.spark.sql.util.CarbonException$.analysisException(CarbonException.scala:23)
 [exec] at 
org.apache.spark.sql.execution.command.management.CarbonAlterTableCompactionCommand.processData(CarbonAlterTableCompactionCommand.scala:120)
 [exec] at 
org.apache.spark.sql.execution.command.DataCommand.run(package.scala:71)
 [exec] at 
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58)
 [exec] at 
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56)
 [exec] at 
org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74)
 [exec] at 
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
 [exec] at 
org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
 [exec] at 
org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:135)
 [exec] at 
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
 [exec] at 
org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:132)
 [exec] at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:113)
 [exec] at 
org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:87)
 [exec] at 
org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:87)
 [exec] at org.apache.spark.sql.Dataset.<init>(Dataset.scala:185)
 [exec] at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:64)
 [exec] at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:592)
 [exec] at com.huawei.spark.SessionManager.sql(SessionManager.java:42)
 [exec] at 
com.huawei.querymanagement.QueryManagement.sql(QueryManagement.java:62)
 [exec] at com.huawei.querymanagement.SelectQuery.testQuery(SelectQuery.java:70)
 [exec] at sun.reflect.GeneratedMethodAccessor66.invoke(Unknown Source)
 [exec] at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 [exec] at java.lang.reflect.Method.invoke(Method.java:498)
 [exec] at org.junit.internal.runners.TestMethod.invoke(TestMethod.java:59)
 [exec] at 
org.junit.internal.runners.MethodRoadie.runTestMethod(MethodRoadie.java:98)
 [exec] at org.junit.internal.runners.MethodRoadie$2.run(MethodRoadie.java:79)
 [exec] at 
org.junit.internal.runners.MethodRoadie.runBeforesThenTestThenAfters(MethodRoadie.java:87)
 [exec] at org.junit.internal.runners.MethodRoadie.runTest(MethodRoadie.java:77)
 [exec] at org.junit.internal.runners.MethodRoadie.run(MethodRoadie.java:42)
 [exec] at 
org.junit.internal.runners.JUnit4ClassRunner.invokeTestMethod(JUnit4ClassRunner.java:88)
 [exec] at 
org.junit.internal.runners.JUnit4ClassRunner.runMethods(JUnit4ClassRunner.java:51)
 [exec] at 
org.junit.runners.Parameterized$TestClassRunnerForParameters.run(Parameterized.java:98)
 [exec] at 
org.junit.internal.runners.CompositeRunner.runChildren(CompositeRunner.java:33)
 [exec] at org.junit.runners.Parameterized.access$000(Parameterized.java:55)
 [exec] at org.junit.runners.Parameterized$1.run(Parameterized.java:131)
 [exec] at 
org.junit.internal.runners.ClassRoadie.runUnprotected(ClassRoadie.java:27)
 [exec] at 
org.junit.internal.runners.ClassRoadie.runProtected(ClassRoadie.java:37)
 [exec] at org.junit.runners.Parameterized.run(Parameterized.java:129)
 [exec] at 
org.junit.internal.runners.CompositeRunner.runChildren(CompositeRunner.java:33)
 [exec] at 
org.junit.internal.runners.CompositeRunner.run(CompositeRunner.java:28)
 [exec] at org.junit.runner.JUnitCore.run(JUnitCore.java:130)
 [exec] at org.junit.runner.JUnitCore.run(JUnitCore.java:109)
 [exec] at org.junit.runner.JUnitCore.run(JUnitCore.java:100)
 [exec] at org.junit.runner.JUnitCore.runClasses(JUnitCore.java:60)
 [exec] at 
com.huawei.querymanagement.SelectQuerySuite.main(SelectQuerySuite.java:18)
 [exec] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 [exec] at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 [exec] at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 [exec] at java.lang.reflect.Method.invoke(Method.java:498)
 [exec] at 
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738)
 [exec] at 
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
 [exec] at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
 [exec] at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
 [exec] at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to