Hello,
I am running Kylin 3.1.1 on AWS EMR 5.30.1
(Hadoop 2.8.5, Hive 2.3.6, HBase 1.4.13, ZooKeeper 3.4.14).
Hbase is configured to store data on S3, and I use AWS Aurora MySQL
for Kylin metadata.
Whenever I attempt to run a MERGE CUBE job, the job fails at #4 Step
Name: Merge Cuboid Data
Step Parameters:
-conf /usr/local/kylin/conf/kylin_job_conf.xml -cubename xxxxx
-segmentid f6eadc72-e5e4-bdd2-db1d-24ea19fbf9c4 -input
hdfs://xxxxx:8020/kylin/kylin_metadata/kylin-a13671fd-ef51-fa05-89d1-033f0c6e3423/xxxxx/cuboid/*,hdfs://xxxxx:8020/kylin/kylin_metadata/kylin-24ef5daf-38e2-56d3-2b0a-792ffb37a0bf/xxxxx/cuboid/*,hdfs://xxxxx:8020/kylin/kylin_met
adata/kylin-d0a4b4b5-44ba-cdf3-c6d0-231483835b24/xxxxx/cuboid/*
-output
hdfs://xxxxx:8020/kylin/kylin_metadata/kylin-6c86522d-2a6e-2820-1af6-193f7c19b0d0/xxxxx/cuboid/
-jobname Kylin_Merge_Cuboid_xxxxx_Step
Step Logs:
java.io.IOException: No input paths specified in job
at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:245)
at
org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat.listStatus(SequenceFileInputFormat.java:59)
at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:358)
at
org.apache.kylin.engine.mr.common.AbstractHadoopJob.getTotalMapInputMB(AbstractHadoopJob.java:664)
at
org.apache.kylin.storage.hbase.steps.HBaseMROutput2Transition$HBaseMergeMROutputFormat.configureJobOutput(HBaseMROutput2Transition.java:171)
at
org.apache.kylin.engine.mr.steps.MergeCuboidJob.run(MergeCuboidJob.java:88)
at
org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:155)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:179)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:179)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:113)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Prior to this exception I get warnings in Kylin logs like:
common.AbstractHadoopJob:460 : Path not
exist:hdfs://xxxxx:8020/kylin/kylin_metadata/kylin-d0a4b4b5-44ba-cdf3-c6d0-231483835b24/xxxxx/cuboid
Maybe the MERGE CUBE job is ignoring the kylin.metadata.url
configuration value?
For example in #1 Step Name: Merge Cuboid Dictionary I see the
following parameters:
-conf /usr/local/kylin/conf/kylin_job_conf_cube_merge.xml
-cubename xxxxx -segmentid f6eadc72-e5e4-bdd2-db1d-24ea19fbf9c4 -metadataUrl
kylin_metadata@hdfs,path=hdfs://xxxxx:8020/kylin/kylin_metadata/kylin-6c86522d-2a6e-2820-1af6-193f7c19b0d0/xxxxx/metadata
-segmentIds
d4e7c735-48d2-c257-d011-a99573e3f32d,a5aba327-9441-eb8e-b719-c0642c53632b,688df5de-8123-6963-83bd-330d897a25a6
-dictOutputPath
hdfs://xxxxx:8020/kylin/kylin_metadata/kylin-6c86522d-2a6e-2820-1af6-193f7c19b0d0/xxxxx/dict_info
-statOutputPath
hdfs://xxxxx:8020/kylin/kylin_metadata/kylin-6c86522d-2a6e-2820-1af6-193f7c19b0d0/xxxxx/fact_distinct_columns/statistics
-jobname Kylin_Merge_Dictionary_xxxxx_Step
Note that the -metadataUrl parameter refers to an HDFS location even
though my kylin.metadata.url refers to a JDBC connection.
Here are my Kylin configuration values (from web UI, some irrelevant
or sensitive values removed):
kylin.metadata.url=kylin_metadata@jdbc,url=jdbc:mysql://xxxxx:3306/kylin?createDatabaseIfNotExist=true,username=xxxxx,password=xxxxx,driverClassName=org.mariadb.jdbc.Driver
kylin.metadata.sync-retries=3
kylin.env.hdfs-working-dir=hdfs://xxxxx:8020/kylin
kylin.env=QA
kylin.env.zookeeper-base-path=/kylin
kylin.server.mode=all
kylin.server.cluster-servers=xxxxx:7070
kylin.engine.default=2
kylin.storage.default=2
kylin.cube.migration.enabled=false
kylin.source.hive.client=cli
kylin.source.hive.beeline-shell=beeline
kylin.source.hive.enable-sparksql-for-table-ops=false
kylin.source.hive.keep-flat-table=false
kylin.source.hive.database-for-flat-table=default
kylin.source.hive.redistribute-flat-table=false
kylin.source.hive.metadata-type=hcatalog
kylin.storage.url=hbase
kylin.storage.hbase.table-name-prefix=KYLIN_
kylin.storage.hbase.namespace=default
kylin.storage.hbase.compression-codec=snappy
kylin.storage.hbase.region-cut-gb=5
kylin.storage.hbase.hfile-size-gb=2
kylin.storage.hbase.min-region-count=1
kylin.storage.hbase.max-region-count=500
[email protected]<mailto:[email protected]>
kylin.storage.hbase.coprocessor-mem-gb=3
kylin.storage.partition.aggr-spill-enabled=true
kylin.storage.partition.max-scan-bytes=3221225472
kylin.storage.clean-after-delete-operation=false
kylin.job.retry=0
kylin.job.max-concurrent-jobs=10
kylin.job.sampling-percentage=100
kylin.job.scheduler.provider.100=org.apache.kylin.job.impl.curator.CuratorScheduler
kylin.job.scheduler.default=100
kylin.engine.mr.yarn-check-interval-seconds=10
kylin.engine.mr.reduce-input-mb=500
kylin.engine.mr.max-reducer-number=500
kylin.engine.mr.mapper-input-rows=1000000
kylin.engine.mr.build-dict-in-reducer=true
kylin.engine.mr.uhc-reducer-count=3
kylin.engine.mr.build-uhc-dict-in-additional-step=false
kylin.cube.cuboid-scheduler=org.apache.kylin.cube.cuboid.DefaultCuboidScheduler
kylin.cube.segment-advisor=org.apache.kylin.cube.CubeSegmentAdvisor
kylin.cube.algorithm=layer
kylin.cube.algorithm.layer-or-inmem-threshold=7
kylin.cube.algorithm.inmem-auto-optimize=true
kylin.cube.aggrgroup.max-combination=32768
kylin.snapshot.max-mb=300
kylin.storage.hbase.cluster-fs=s3://xxxxx
kylin.metadata.jdbc.dialect=mysql
kylin.metadata.jdbc.json-always-small-cell=true
kylin.server.self-discovery-enabled=true
kylin.server.cluster-name=kylin_metadata
kylin.server.cluster-servers-with-mode=xxxxx:7070:all
Thank you for your assistance,
Gabe
