Per documentation: http://kylin.apache.org/docs/tutorial/cube_migration.html
In the example for "II. Migrate across two Hadoop clusters"
./bin/kylin.sh org.apache.kylin.tool.migration.CubeMigrationCrossClusterCLI
-kylinUriSrc ADMIN:ky...@qa.env:17070 -kylinUriDst
ADMIN:ky...@prod.env:17777 -cube kylin_sales_cube -updateMappingPath
$KYLIN_HOME/updateTableMapping.json -execute true -schemaOnly false
-overwrite true
What is the updateTableMapping.json file? I have not been able to get any
information on what this file should be. Is this a file to be created by us?
can I have a sample?
I am encountering error in trying to copy a cube with cube data to another
hadoop cluster. Here is the log:
Command without -updateMappingPath:
./bin/kylin.sh org.apache.kylin.tool.migration.CubeMigrationCrossClusterCLI
-kylinUriSrc ADMIN:KYLIN@ip-172-31-85-18.ec2.internal:7070 -kylinUriDst
ADMIN:KYLIN@ip-172-31-94-63.ec2.internal:7070 -cube kylin_sales_cube
-execute true -overwrite true -schemaOnly false
/*removed previous log*/
2021-02-26 16:48:30,274 INFO [Cube-kylin_sales_cube-data-migration-pool-0]
migration.CubeMigrationCrossClusterCLI:695 : start to copy hdfs directory
from
hdfs://ip-172-31-85-18.ec2.internal:8020/kylin/kylin_metadata/kylin-8dbe2fbb-7960-c2a5-3a20-ba26958f9691
to
hdfs://ip-172-31-94-63.ec2.internal:8020/kylin/kylin_metadata/kylin-8dbe2fbb-7960-c2a5-3a20-ba26958f9691
2021-02-26 16:48:30,435 INFO [Cube-kylin_sales_cube-data-migration-pool-1]
client.RMProxy:123 : Connecting to ResourceManager at
ip-172-31-85-18.ec2.internal/172.31.85.18:8032
2021-02-26 16:48:30,435 INFO [Cube-kylin_sales_cube-data-migration-pool-0]
client.RMProxy:123 : Connecting to ResourceManager at
ip-172-31-85-18.ec2.internal/172.31.85.18:8032
2021-02-26 16:48:30,659 INFO [Cube-kylin_sales_cube-data-migration-pool-0]
client.RMProxy:123 : Connecting to ResourceManager at
ip-172-31-85-18.ec2.internal/172.31.85.18:8032
2021-02-26 16:48:30,659 INFO [Cube-kylin_sales_cube-data-migration-pool-1]
client.RMProxy:123 : Connecting to ResourceManager at
ip-172-31-85-18.ec2.internal/172.31.85.18:8032
2021-02-26 16:48:30,877 INFO [Cube-kylin_sales_cube-data-migration-pool-0]
tools.SimpleCopyListing:589 : Paths (files+dirs) cnt = 71; dirCnt = 15
2021-02-26 16:48:30,877 INFO [Cube-kylin_sales_cube-data-migration-pool-0]
tools.SimpleCopyListing:384 : Build file listing completed.
2021-02-26 16:48:30,879 INFO [Cube-kylin_sales_cube-data-migration-pool-0]
Configuration.deprecation:1285 : io.sort.mb is deprecated. Instead, use
mapreduce.task.io.sort.mb
2021-02-26 16:48:30,879 INFO [Cube-kylin_sales_cube-data-migration-pool-0]
Configuration.deprecation:1285 : io.sort.factor is deprecated. Instead, use
mapreduce.task.io.sort.factor
2021-02-26 16:48:31,370 INFO [Cube-kylin_sales_cube-data-migration-pool-0]
tools.DistCp:94 : Number of paths in the copy list: 71
2021-02-26 16:48:31,400 INFO [Cube-kylin_sales_cube-data-migration-pool-0]
tools.DistCp:94 : Number of paths in the copy list: 71
2021-02-26 16:48:31,448 INFO [Cube-kylin_sales_cube-data-migration-pool-0]
client.RMProxy:123 : Connecting to ResourceManager at
ip-172-31-85-18.ec2.internal/172.31.85.18:8032
2021-02-26 16:48:31,486 WARN [Cube-kylin_sales_cube-data-migration-pool-0]
mapreduce.JobResourceUploader:64 : Hadoop command-line option parsing not
performed. Implement the Tool interface and execute your application with
ToolRunner to remedy this.
2021-02-26 16:48:32,227 INFO [Cube-kylin_sales_cube-data-migration-pool-1]
tools.SimpleCopyListing:589 : Paths (files+dirs) cnt = 11; dirCnt = 5
2021-02-26 16:48:32,227 INFO [Cube-kylin_sales_cube-data-migration-pool-1]
tools.SimpleCopyListing:384 : Build file listing completed.
2021-02-26 16:48:32,243 INFO [Cube-kylin_sales_cube-data-migration-pool-1]
tools.DistCp:94 : Number of paths in the copy list: 11
2021-02-26 16:48:32,259 INFO [Cube-kylin_sales_cube-data-migration-pool-1]
tools.DistCp:94 : Number of paths in the copy list: 11
2021-02-26 16:48:32,306 INFO [Cube-kylin_sales_cube-data-migration-pool-1]
client.RMProxy:123 : Connecting to ResourceManager at
ip-172-31-85-18.ec2.internal/172.31.85.18:8032
2021-02-26 16:48:32,311 WARN [Cube-kylin_sales_cube-data-migration-pool-1]
mapreduce.JobResourceUploader:64 : Hadoop command-line option parsing not
performed. Implement the Tool interface and execute your application with
ToolRunner to remedy this.
2021-02-26 16:48:32,367 INFO [Cube-kylin_sales_cube-data-migration-pool-1]
mapreduce.JobSubmitter:200 : number of splits:6
2021-02-26 16:48:32,398 INFO [Cube-kylin_sales_cube-data-migration-pool-0]
mapreduce.JobSubmitter:200 : number of splits:26
2021-02-26 16:48:32,490 INFO [Cube-kylin_sales_cube-data-migration-pool-0]
mapreduce.JobSubmitter:289 : Submitting tokens for job:
job_1614247321131_0022
2021-02-26 16:48:32,491 INFO [Cube-kylin_sales_cube-data-migration-pool-1]
mapreduce.JobSubmitter:289 : Submitting tokens for job:
job_1614247321131_0023
2021-02-26 16:48:32,604 INFO [Cube-kylin_sales_cube-data-migration-pool-1]
mapreduce.JobSubmitter:251 : Cleaning up the staging area
/tmp/hadoop-yarn/staging/hadoop/.staging/job_1614247321131_0023
2021-02-26 16:48:32,605 INFO [Cube-kylin_sales_cube-data-migration-pool-0]
mapreduce.JobSubmitter:251 : Cleaning up the staging area
/tmp/hadoop-yarn/staging/hadoop/.staging/job_1614247321131_0022
2021-02-26 16:48:32,609 ERROR [Cube-kylin_sales_cube-data-migration-pool-1]
migration.CubeMigrationCrossClusterCLI:680 : fail to migrate htable
KYLIN_HMATDT9OSR due to {}
java.lang.NullPointerException
at
org.apache.hadoop.mapreduce.v2.util.MRApps.setMRFrameworkClasspath(MRApps.java:210)
at
org.apache.hadoop.mapreduce.v2.util.MRApps.setClasspath(MRApps.java:259)
at
org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:467)
at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:296)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:242)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1338)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1338)
at org.apache.hadoop.tools.DistCp.createAndSubmitJob(DistCp.java:193)
at org.apache.hadoop.tools.DistCp.execute(DistCp.java:155)
at
org.apache.kylin.tool.migration.CubeMigrationCrossClusterCLI.copyHDFSPath(CubeMigrationCrossClusterCLI.java:701)
at
org.apache.kylin.tool.migration.CubeMigrationCrossClusterCLI.copyHFileByDistCp(CubeMigrationCrossClusterCLI.java:690)
at
org.apache.kylin.tool.migration.CubeMigrationCrossClusterCLI.copyHTable(CubeMigrationCrossClusterCLI.java:662)
at
org.apache.kylin.tool.migration.CubeMigrationCrossClusterCLI.copyHTable(CubeMigrationCrossClusterCLI.java:614)
at
org.apache.kylin.tool.migration.CubeMigrationCrossClusterCLI.access$200(CubeMigrationCrossClusterCLI.java:88)
at
org.apache.kylin.tool.migration.CubeMigrationCrossClusterCLI$3.doRun(CubeMigrationCrossClusterCLI.java:505)
at
org.apache.kylin.tool.migration.CubeMigrationCrossClusterCLI$MyRunnable.run(CubeMigrationCrossClusterCLI.java:742)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2021-02-26 16:48:32,610 ERROR [main]
migration.CubeMigrationCrossClusterCLI:431 : java.lang.NullPointerException
2021-02-26 16:48:32,612 ERROR [main]
migration.CubeMigrationCrossClusterCLI:332 : fail to migrate cube
CUBE[name=kylin_sales_cube] due to
java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.lang.NullPointerException
at
org.apache.kylin.tool.migration.CubeMigrationCrossClusterCLI.migrateCube(CubeMigrationCrossClusterCLI.java:432)
at
org.apache.kylin.tool.migration.CubeMigrationCrossClusterCLI.execute(CubeMigrationCrossClusterCLI.java:329)
at
org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
at
org.apache.kylin.tool.migration.CubeMigrationCrossClusterCLI.main(CubeMigrationCrossClusterCLI.java:755)
Caused by: java.util.concurrent.ExecutionException:
java.lang.NullPointerException
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
org.apache.kylin.tool.migration.CubeMigrationCrossClusterCLI.migrateCube(CubeMigrationCrossClusterCLI.java:426)
... 3 more
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.mapreduce.v2.util.MRApps.setMRFrameworkClasspath(MRApps.java:210)
at
org.apache.hadoop.mapreduce.v2.util.MRApps.setClasspath(MRApps.java:259)
at
org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:467)
at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:296)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:242)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1338)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1338)
at org.apache.hadoop.tools.DistCp.createAndSubmitJob(DistCp.java:193)
at org.apache.hadoop.tools.DistCp.execute(DistCp.java:155)
at
org.apache.kylin.tool.migration.CubeMigrationCrossClusterCLI.copyHDFSPath(CubeMigrationCrossClusterCLI.java:701)
at
org.apache.kylin.tool.migration.CubeMigrationCrossClusterCLI.copyHDFSJobInfo(CubeMigrationCrossClusterCLI.java:602)
at
org.apache.kylin.tool.migration.CubeMigrationCrossClusterCLI.access$100(CubeMigrationCrossClusterCLI.java:88)
at
org.apache.kylin.tool.migration.CubeMigrationCrossClusterCLI$2.doRun(CubeMigrationCrossClusterCLI.java:497)
at
org.apache.kylin.tool.migration.CubeMigrationCrossClusterCLI$MyRunnable.run(CubeMigrationCrossClusterCLI.java:742)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2021-02-26 16:48:32,612 INFO [main] migration.DstClusterUtil:363 : update
meta cache for org.apache.kylin.common.restclient.RestClient@3e53c4ad
2021-02-26 16:48:32,617 WARN [main]
migration.CubeMigrationCrossClusterCLI:361 : Failed to migrate cubes
[CUBE[name=kylin_sales_cube]] and need to check the detailed reason and
retry again!!!
2021-02-26 16:48:32,619 INFO [close-hbase-conn] hbase.HBaseConnection:137 :
Closing HBase connections...
2021-02-26 16:48:32,619 INFO [close-hbase-conn]
client.ConnectionManager$HConnectionImplementation:2259 : Closing master
protocol: MasterService
2021-02-26 16:48:32,620 INFO [close-hbase-conn]
client.ConnectionManager$HConnectionImplementation:1776 : Closing zookeeper
sessionid=0x10000029a450025
2021-02-26 16:48:32,622 INFO [close-hbase-conn] zookeeper.ZooKeeper:693 :
Session: 0x10000029a450025 closed
2021-02-26 16:48:32,622 INFO [main-EventThread] zookeeper.ClientCnxn:522 :
EventThread shut down for session: 0x10000029a450025
--
Sent from: http://apache-kylin.74782.x6.nabble.com/
