Thank you! We checked the yarn and hard disk. But not found any error. Hard disk space and memory and so on is working well. Last time its error code was "unknownhost clusterB",now in new server env it cant find clusterB(hbase only). but cant find rowCount file. === the follow command runs ok: hdfs dfs -mkdir /young/kylin_test/kylin_metadata_nokia/kylin-678c15ba-5375-4f80-831e-1ae0af8ed576/row_count/tmp
And "ls" cant find file "000000_0" which it said "file does not exist". ------------------ ???????? ------------------ ??????: "Alberto Ram??n";<a.ramonporto...@gmail.com>; ????????: 2016??12??19??(??????) ????9:13 ??????: "user"<user@kylin.apache.org>; ????: Re: Error when #2 Step: Redistribute Flat Hive Table - File does not exist i think i had this error last nigth :) (go to yarn to find detailed error & find on internet) in my case was free space less than 10% of hard disk. Check this please El 19/12/2016 11:35, "????????" <491245...@qq.com> escribi??: When I build a cube in kylin1.6, I get error in step #2: Redistribute Flat Hive Table Please help! Thank you very much! env: kylin1.6 is in a independent server, and have 2 other server cluster: clusterA(hive only) and clusterB(hbase only). Error is: 2016-12-19 10:28:00,641 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : Compute row count of flat hive table, cmd: 2016-12-19 10:28:00,642 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : hive -e "USE boco; SET dfs.replication=2; SET hive.exec.compress.output=true; SET hive.auto.convert.join.noconditionaltask=true; SET hive.auto.convert.join.noconditionaltask.size=100000000; SET mapreduce.output.fileoutputformat.compress.type=BLOCK; SET mapreduce.job.split.metainfo.maxsize=-1; SET mapreduce.job.queuename=young; SET tez.queue.name=young; set hive.exec.compress.output=false; set hive.exec.compress.output=false; INSERT OVERWRITE DIRECTORY '/young/kylin_test/kylin_metadata_test/kylin-678266c0-ba0e-48b4-bdb5-6e578320375a/row_count' SELECT count(*) FROM kylin_intermediate_hbase_in_testCluster_CUBE_f9468805_eabf_4b54_bf2b_182e4c86214a; " 2016-12-19 10:28:03,277 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : WARNING: Use "yarn jar" to launch YARN applications. 2016-12-19 10:28:04,444 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : 2016-12-19 10:28:04,445 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : Logging initialized using configuration in file:/etc/hive/conf/hive-log4j.properties 2016-12-19 10:28:14,700 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : OK 2016-12-19 10:28:14,703 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : Time taken: 0.935 seconds 2016-12-19 10:28:15,559 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : Query ID = young_20161219102814_a7104fd4-ba83-47fc-ac0b-0c9bef4e1969 2016-12-19 10:28:15,560 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : Total jobs = 1 2016-12-19 10:28:15,575 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : Launching Job 1 out of 1 2016-12-19 10:28:22,842 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : 2016-12-19 10:28:22,842 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : 2016-12-19 10:28:23,104 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : Status: Running (Executing on YARN cluster with App id application_1473415773736_1063281) 2016-12-19 10:28:23,104 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : 2016-12-19 10:28:23,104 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : Map 1: -/- Reducer 2: 0/1 2016-12-19 10:28:23,307 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : Map 1: 0/2 Reducer 2: 0/1 2016-12-19 10:28:26,363 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : Map 1: 0/2 Reducer 2: 0/1 2016-12-19 10:28:26,567 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : Map 1: 0(+1)/2 Reducer 2: 0/1 2016-12-19 10:28:26,596 INFO [pool-7-thread-1] threadpool.DefaultScheduler:118 : Job Fetcher: 1 should running, 1 actual running, 0 ready, 0 already succeed, 3 error, 1 discarded, 0 others 2016-12-19 10:28:26,769 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : Map 1: 0(+2)/2 Reducer 2: 0/1 2016-12-19 10:28:29,810 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : Map 1: 0(+2)/2 Reducer 2: 0/1 2016-12-19 10:28:30,217 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : Map 1: 1(+1)/2 Reducer 2: 0(+1)/1 2016-12-19 10:28:30,826 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : Map 1: 2/2 Reducer 2: 0(+1)/1 2016-12-19 10:28:31,232 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : Map 1: 2/2 Reducer 2: 1/1 2016-12-19 10:28:31,319 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : Moving data to: /young/kylin_test/kylin_metadata_test/kylin-678266c0-ba0e-48b4-bdb5-6e578320375a/row_count 2016-12-19 10:28:31,406 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : OK 2016-12-19 10:28:31,454 INFO [pool-8-thread-7] execution.AbstractExecutable:36 : Time taken: 16.701 seconds 2016-12-19 10:28:35,074 ERROR [pool-8-thread-7] execution.AbstractExecutable:357 : job:678266c0-ba0e-48b4-bdb5-6e578320375a-01 execute finished with exception java.io.FileNotFoundException: File does not exist: /young/kylin_test/kylin_metadata_test/kylin-678266c0-ba0e-48b4-bdb5-6e578320375a/row_count/000000_0 at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1820) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1791) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1704) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:587) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2081) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2077) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2075) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1242) at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1227) at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1215) at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:303) at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:269) at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:261) at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1540) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:303) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:299) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:299) at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:767) at org.apache.kylin.source.hive.HiveMRInput$RedistributeFlatHiveTableStep.readRowCountFromFile(HiveMRInput.java:294) at org.apache.kylin.source.hive.HiveMRInput$RedistributeFlatHiveTableStep.doWork(HiveMRInput.java:337) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:57) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:136) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /young/kylin_test/kylin_metadata_test/kylin-678266c0-ba0e-48b4-bdb5-6e578320375a/row_count/000000_0 at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1820) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1791) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1704) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:587) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2081) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2077) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2075) at org.apache.hadoop.ipc.Client.call(Client.java:1427) at org.apache.hadoop.ipc.Client.call(Client.java:1358) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy25.getBlockLocations(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:255) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:252) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104) at com.sun.proxy.$Proxy26.getBlockLocations(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1240) ... 20 more 2016-12-19 10:28:35,078 DEBUG [pool-8-thread-7] dao.ExecutableDao:210 : updating job output, id: 678266c0-ba0e-48b4-bdb5-6e578320375a-01 2016-12-19 10:28:35,082 DEBUG [pool-8-thread-7] hbase.HBaseResourceStore:262 : Update row /execute_output/678266c0-ba0e-48b4-bdb5-6e578320375a-01 from oldTs: 1482114480638, to newTs: 1482114515078, operation result: true 2016-12-19 10:28:35,085 DEBUG [pool-8-thread-7] dao.ExecutableDao:210 : updating job output, id: 678266c0-ba0e-48b4-bdb5-6e578320375a-01
