"java.io.FileNotFoundException: File does not exist: /young/kylin_test/kylin_metadata_test/kylin-678266c0- ba0e-48b4-bdb5-6e578320375a/row_count/000000_0"
Seems be the issue in https://issues.apache.org/jira/browse/KYLIN-2159 It should be fixed in 1.6.0; could you please check whether there is any file in the folder? hadoop fs -ls /young/kylin_test/kylin_metadata_test/kylin-678266c0-ba0e-48b4-bdb5-6e578320375a/row_count/ 2016-12-20 15:23 GMT+08:00 Alberto Ramón <a.ramonporto...@gmail.com>: > other idea: > Can be a problem with permissions?: the user that execute Kylin can't > read data generated by YARN > check if Kylin user can read your folder /young/kylin_test/ > Which Hadoop user are executing Kylin? > > (no more ideas, Good Luck) > > 2016-12-20 7:51 GMT+01:00 雨日听风 <491245...@qq.com>: > >> Thank you! >> We checked the yarn and hard disk. But not found any error. Hard disk >> space and memory and so on is working well. >> Last time its error code was "unknownhost clusterB",now in new server env >> it cant find clusterB(hbase only). but cant find rowCount file. >> === >> the follow command runs ok: >> hdfs dfs -mkdir /young/kylin_test/kylin_metadata_nokia/kylin >> -678c15ba-5375-4f80-831e-1ae0af8ed576/row_count/tmp >> And "ls" cant find file "000000_0" which it said "file does not exist". >> >> ------------------ 原始邮件 ------------------ >> *发件人:* "Alberto Ramón";<a.ramonporto...@gmail.com>; >> *发送时间:* 2016年12月19日(星期一) 晚上9:13 >> *收件人:* "user"<user@kylin.apache.org>; >> *主题:* Re: Error when #2 Step: Redistribute Flat Hive Table - File does >> not exist >> >> i think i had this error last nigth :) >> (go to yarn to find detailed error & find on internet) >> in my case was free space less than 10% of hard disk. Check this please >> >> El 19/12/2016 11:35, "雨日听风" <491245...@qq.com> escribió: >> >>> When I build a cube in kylin1.6, I get error in step #2: Redistribute >>> Flat Hive Table >>> >>> Please help! Thank you very much! >>> >>> env: kylin1.6 is in a independent server, and have 2 other server >>> cluster: clusterA(hive only) and clusterB(hbase only). >>> Error is: >>> >>> 2016-12-19 10:28:00,641 INFO [pool-8-thread-7] >>> execution.AbstractExecutable:36 : Compute row count of flat hive table, >>> cmd: >>> 2016-12-19 10:28:00,642 INFO [pool-8-thread-7] >>> execution.AbstractExecutable:36 : hive -e "USE boco; >>> SET dfs.replication=2; >>> SET hive.exec.compress.output=true; >>> SET hive.auto.convert.join.noconditionaltask=true; >>> SET hive.auto.convert.join.noconditionaltask.size=100000000; >>> SET mapreduce.output.fileoutputformat.compress.type=BLOCK; >>> SET mapreduce.job.split.metainfo.maxsize=-1; >>> SET mapreduce.job.queuename=young; >>> SET tez.queue.name=young; >>> >>> set hive.exec.compress.output=false; >>> >>> set hive.exec.compress.output=false; >>> INSERT OVERWRITE DIRECTORY '/young/kylin_test/kylin_metad >>> ata_test/kylin-678266c0-ba0e-48b4-bdb5-6e578320375a/row_count' SELECT >>> count(*) FROM kylin_intermediate_hbase_in_te >>> stCluster_CUBE_f9468805_eabf_4b54_bf2b_182e4c86214a; >>> >>> " >>> 2016-12-19 10:28:03,277 INFO [pool-8-thread-7] >>> execution.AbstractExecutable:36 : WARNING: Use "yarn jar" to launch >>> YARN applications. >>> 2016-12-19 10:28:04,444 INFO [pool-8-thread-7] >>> execution.AbstractExecutable:36 : >>> 2016-12-19 10:28:04,445 INFO [pool-8-thread-7] >>> execution.AbstractExecutable:36 : Logging initialized using >>> configuration in file:/etc/hive/conf/hive-log4j.properties >>> 2016-12-19 10:28:14,700 INFO [pool-8-thread-7] >>> execution.AbstractExecutable:36 : OK >>> 2016-12-19 10:28:14,703 INFO [pool-8-thread-7] >>> execution.AbstractExecutable:36 : Time taken: 0.935 seconds >>> 2016-12-19 10:28:15,559 INFO [pool-8-thread-7] >>> execution.AbstractExecutable:36 : Query ID = >>> young_20161219102814_a7104fd4-ba83-47fc-ac0b-0c9bef4e1969 >>> 2016-12-19 10:28:15,560 INFO [pool-8-thread-7] >>> execution.AbstractExecutable:36 : Total jobs = 1 >>> 2016-12-19 10:28:15,575 INFO [pool-8-thread-7] >>> execution.AbstractExecutable:36 : Launching Job 1 out of 1 >>> 2016-12-19 10:28:22,842 INFO [pool-8-thread-7] >>> execution.AbstractExecutable:36 : >>> 2016-12-19 10:28:22,842 INFO [pool-8-thread-7] >>> execution.AbstractExecutable:36 : >>> 2016-12-19 10:28:23,104 INFO [pool-8-thread-7] >>> execution.AbstractExecutable:36 : Status: Running (Executing on YARN >>> cluster with App id application_1473415773736_1063281) >>> 2016-12-19 10:28:23,104 INFO [pool-8-thread-7] >>> execution.AbstractExecutable:36 : >>> 2016-12-19 10:28:23,104 INFO [pool-8-thread-7] >>> execution.AbstractExecutable:36 : Map 1: -/- Reducer 2: 0/1 >>> 2016-12-19 10:28:23,307 INFO [pool-8-thread-7] >>> execution.AbstractExecutable:36 : Map 1: 0/2 Reducer 2: 0/1 >>> 2016-12-19 10:28:26,363 INFO [pool-8-thread-7] >>> execution.AbstractExecutable:36 : Map 1: 0/2 Reducer 2: 0/1 >>> 2016-12-19 10:28:26,567 INFO [pool-8-thread-7] >>> execution.AbstractExecutable:36 : Map 1: 0(+1)/2 Reducer 2: 0/1 >>> 2016-12-19 10:28:26,596 INFO [pool-7-thread-1] >>> threadpool.DefaultScheduler:118 : Job Fetcher: 1 should running, 1 >>> actual running, 0 ready, 0 already succeed, 3 error, 1 discarded, 0 others >>> 2016-12-19 10:28:26,769 INFO [pool-8-thread-7] >>> execution.AbstractExecutable:36 : Map 1: 0(+2)/2 Reducer 2: 0/1 >>> 2016-12-19 10:28:29,810 INFO [pool-8-thread-7] >>> execution.AbstractExecutable:36 : Map 1: 0(+2)/2 Reducer 2: 0/1 >>> 2016-12-19 10:28:30,217 INFO [pool-8-thread-7] >>> execution.AbstractExecutable:36 : Map 1: 1(+1)/2 Reducer 2: 0(+1)/1 >>> 2016-12-19 10:28:30,826 INFO [pool-8-thread-7] >>> execution.AbstractExecutable:36 : Map 1: 2/2 Reducer 2: 0(+1)/1 >>> 2016-12-19 10:28:31,232 INFO [pool-8-thread-7] >>> execution.AbstractExecutable:36 : Map 1: 2/2 Reducer 2: 1/1 >>> 2016-12-19 10:28:31,319 INFO [pool-8-thread-7] >>> execution.AbstractExecutable:36 : Moving data to: >>> /young/kylin_test/kylin_metadata_test/kylin-678266c0-ba0e-48 >>> b4-bdb5-6e578320375a/row_count >>> 2016-12-19 10:28:31,406 INFO [pool-8-thread-7] >>> execution.AbstractExecutable:36 : OK >>> 2016-12-19 10:28:31,454 INFO [pool-8-thread-7] >>> execution.AbstractExecutable:36 : Time taken: 16.701 seconds >>> 2016-12-19 10:28:35,074 ERROR [pool-8-thread-7] >>> execution.AbstractExecutable:357 : >>> job:678266c0-ba0e-48b4-bdb5-6e578320375a-01 >>> execute finished with exception >>> java.io.FileNotFoundException: File does not exist: >>> /young/kylin_test/kylin_metadata_test/kylin-678266c0-ba0e-48 >>> b4-bdb5-6e578320375a/row_count/000000_0 >>> at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INo >>> deFile.java:71) >>> at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INo >>> deFile.java:61) >>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlock >>> LocationsInt(FSNamesystem.java:1820) >>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlock >>> Locations(FSNamesystem.java:1791) >>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlock >>> Locations(FSNamesystem.java:1704) >>> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.get >>> BlockLocations(NameNodeRpcServer.java:587) >>> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServ >>> erSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolS >>> erverSideTranslatorPB.java:365) >>> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocol >>> Protos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNam >>> enodeProtocolProtos.java) >>> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcIn >>> voker.call(ProtobufRpcEngine.java:616) >>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) >>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2081) >>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2077) >>> at java.security.AccessController.doPrivileged(Native Method) >>> at javax.security.auth.Subject.doAs(Subject.java:415) >>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro >>> upInformation.java:1657) >>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2075) >>> >>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native >>> Method) >>> at sun.reflect.NativeConstructorAccessorImpl.newInstance(Native >>> ConstructorAccessorImpl.java:57) >>> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(De >>> legatingConstructorAccessorImpl.java:45) >>> at java.lang.reflect.Constructor.newInstance(Constructor.java:526) >>> at org.apache.hadoop.ipc.RemoteException.instantiateException(R >>> emoteException.java:106) >>> at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException( >>> RemoteException.java:73) >>> at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSCl >>> ient.java:1242) >>> at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient. >>> java:1227) >>> at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient. >>> java:1215) >>> at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndG >>> etLastBlockLength(DFSInputStream.java:303) >>> at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStrea >>> m.java:269) >>> at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream. >>> java:261) >>> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1540) >>> at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(Distri >>> butedFileSystem.java:303) >>> at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(Distri >>> butedFileSystem.java:299) >>> at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSyst >>> emLinkResolver.java:81) >>> at org.apache.hadoop.hdfs.DistributedFileSystem.open(Distribute >>> dFileSystem.java:299) >>> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:767) >>> at org.apache.kylin.source.hive.HiveMRInput$RedistributeFlatHiv >>> eTableStep.readRowCountFromFile(HiveMRInput.java:294) >>> at org.apache.kylin.source.hive.HiveMRInput$RedistributeFlatHiv >>> eTableStep.doWork(HiveMRInput.java:337) >>> at org.apache.kylin.job.execution.AbstractExecutable.execute(Ab >>> stractExecutable.java:113) >>> at org.apache.kylin.job.execution.DefaultChainedExecutable.doWo >>> rk(DefaultChainedExecutable.java:57) >>> at org.apache.kylin.job.execution.AbstractExecutable.execute(Ab >>> stractExecutable.java:113) >>> at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRun >>> ner.run(DefaultScheduler.java:136) >>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >>> Executor.java:1145) >>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >>> lExecutor.java:615) >>> at java.lang.Thread.run(Thread.java:745) >>> Caused by: >>> org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): >>> File does not exist: /young/kylin_test/kylin_metada >>> ta_test/kylin-678266c0-ba0e-48b4-bdb5-6e578320375a/row_count/000000_0 >>> at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INo >>> deFile.java:71) >>> at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INo >>> deFile.java:61) >>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlock >>> LocationsInt(FSNamesystem.java:1820) >>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlock >>> Locations(FSNamesystem.java:1791) >>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlock >>> Locations(FSNamesystem.java:1704) >>> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.get >>> BlockLocations(NameNodeRpcServer.java:587) >>> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServ >>> erSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolS >>> erverSideTranslatorPB.java:365) >>> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocol >>> Protos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNam >>> enodeProtocolProtos.java) >>> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcIn >>> voker.call(ProtobufRpcEngine.java:616) >>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) >>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2081) >>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2077) >>> at java.security.AccessController.doPrivileged(Native Method) >>> at javax.security.auth.Subject.doAs(Subject.java:415) >>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro >>> upInformation.java:1657) >>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2075) >>> >>> at org.apache.hadoop.ipc.Client.call(Client.java:1427) >>> at org.apache.hadoop.ipc.Client.call(Client.java:1358) >>> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(Proto >>> bufRpcEngine.java:229) >>> at com.sun.proxy.$Proxy25.getBlockLocations(Unknown Source) >>> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTran >>> slatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:255) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce >>> ssorImpl.java:57) >>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe >>> thodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:606) >>> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMeth >>> od(RetryInvocationHandler.java:252) >>> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(Ret >>> ryInvocationHandler.java:104) >>> at com.sun.proxy.$Proxy26.getBlockLocations(Unknown Source) >>> at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSCl >>> ient.java:1240) >>> ... 20 more >>> 2016-12-19 10:28:35,078 DEBUG [pool-8-thread-7] dao.ExecutableDao:210 : >>> updating job output, id: 678266c0-ba0e-48b4-bdb5-6e578320375a-01 >>> 2016-12-19 10:28:35,082 DEBUG [pool-8-thread-7] >>> hbase.HBaseResourceStore:262 : Update row >>> /execute_output/678266c0-ba0e-48b4-bdb5-6e578320375a-01 >>> from oldTs: 1482114480638, to newTs: 1482114515078, operation result: true >>> 2016-12-19 10:28:35,085 DEBUG [pool-8-thread-7] dao.ExecutableDao:210 : >>> updating job output, id: 678266c0-ba0e-48b4-bdb5-6e578320375a-01 >>> >> > -- Best regards, Shaofeng Shi 史少锋
