[ https://issues.apache.org/jira/browse/IMPALA-9055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Adriano updated IMPALA-9055: ---------------------------- Description: HDFS Caching with Impala: If we create a pool specifying the maxTtl with the hdfs command: e.g: {{sudo -u hdfs hdfs cacheadmin -addPool foo -owner impala -group hdfs -mode 755 -limit 100000000000 -maxTtl 7d}} when we try to alter a table adding a partition in Impala: e.g: {code:java} alter table foo partition (p1=1) set cached in 'foo' {code} we get a failure with the exception: {code:java} ERROR: ImpalaRuntimeException: Expiration 26687997791:19:48:13.951 exceeds the max relative expiration time of 604800000 ms. at org.apache.hadoop.hdfs.server.namenode.CacheManager.validateExpiryTime(CacheManager.java:378) at org.apache.hadoop.hdfs.server.namenode.CacheManager.addDirective(CacheManager.java:528) at org.apache.hadoop.hdfs.server.namenode.FSNDNCacheOp.addCacheDirective(FSNDNCacheOp.java:45) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.addCacheDirective(FSNamesystem.java:6782) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addCacheDirective(NameNodeRpcServer.java:1883) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addCacheDirective(ClientNamenodeProtocolServerSideTranslatorPB.java:1265) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675) CAUSED BY: InvalidRequestException: Expiration 26687997791:19:48:13.951 exceeds the max relative expiration time of 604800000 ms. at org.apache.hadoop.hdfs.server.namenode.CacheManager.validateExpiryTime(CacheManager.java:378) at org.apache.hadoop.hdfs.server.namenode.CacheManager.addDirective(CacheManager.java:528) at org.apache.hadoop.hdfs.server.namenode.FSNDNCacheOp.addCacheDirective(FSNDNCacheOp.java:45) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.addCacheDirective(FSNamesystem.java:6782) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addCacheDirective(NameNodeRpcServer.java:1883) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addCacheDirective(ClientNamenodeProtocolServerSideTranslatorPB.java:1265) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675) CAUSED BY: RemoteException: Expiration 26687997791:19:48:13.951 exceeds the max relative expiration time of 604800000 ms. at org.apache.hadoop.hdfs.server.namenode.CacheManager.validateExpiryTime(CacheManager.java:378) at org.apache.hadoop.hdfs.server.namenode.CacheManager.addDirective(CacheManager.java:528) at org.apache.hadoop.hdfs.server.namenode.FSNDNCacheOp.addCacheDirective(FSNDNCacheOp.java:45) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.addCacheDirective(FSNamesystem.java:6782) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addCacheDirective(NameNodeRpcServer.java:1883) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addCacheDirective(ClientNamenodeProtocolServerSideTranslatorPB.java:1265) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675) {code} this is always reproducible. The workaround is to do not set the maxTtl in the hdfs command during the pool creation. Here's the repro steps: {code:java} impala-shell -i <ImpalaD hostname> -q "drop table if exists foo;" impala-shell -i <ImpalaD hostname> -q "create table if not exists foo (c1 int, c2 string) partitioned by (p1 int) stored as parquet location '/user/hive/warehouse/foo';" impala-shell -i <ImpalaD hostname> -q "insert into foo partition (p1 = 1 ) values (1 ,'one');" sudo -u hdfs hdfs cacheadmin -removePool foo sudo -u hdfs hdfs cacheadmin -listDirectives -stats sudo -u hdfs hdfs cacheadmin -listPools -stats sudo -u hdfs hdfs cacheadmin -addPool foo -owner impala -group hdfs -mode 755 -limit 100000000000 -maxTtl 7d sudo -u hdfs hdfs cacheadmin -addDirective -path /user/hive/warehouse/foo -pool foo -ttl 7d sudo -u hdfs hdfs cacheadmin -listDirectives -stats sudo -u hdfs hdfs cacheadmin -listPools -stats impala-shell -i <ImpalaD hostname> -q "alter table foo set uncached;" sleep 5; impala-shell -i <ImpalaD hostname> -q "alter table foo partition (p1=1) set cached in 'foo';" {code} I did not found any already open jira for this issue that looks reproducible in many CDH versions, I appreciate if you can take a look on it. was: HDFS Caching with Impala: If we create a pool specifying the maxTtl with the hdfs command: e.g: {{sudo -u hdfs hdfs cacheadmin -addPool case422446 -owner impala -group hdfs -mode 755 -limit 100000000000 -maxTtl 7d}} when we try to alter a table adding a partition in Impala: e.g: {code:java} alter table foo partition (p1=1) set cached in 'foo' {code} we get a failure with the exception: {code:java} ERROR: ImpalaRuntimeException: Expiration 26687997791:19:48:13.951 exceeds the max relative expiration time of 604800000 ms. at org.apache.hadoop.hdfs.server.namenode.CacheManager.validateExpiryTime(CacheManager.java:378) at org.apache.hadoop.hdfs.server.namenode.CacheManager.addDirective(CacheManager.java:528) at org.apache.hadoop.hdfs.server.namenode.FSNDNCacheOp.addCacheDirective(FSNDNCacheOp.java:45) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.addCacheDirective(FSNamesystem.java:6782) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addCacheDirective(NameNodeRpcServer.java:1883) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addCacheDirective(ClientNamenodeProtocolServerSideTranslatorPB.java:1265) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675) CAUSED BY: InvalidRequestException: Expiration 26687997791:19:48:13.951 exceeds the max relative expiration time of 604800000 ms. at org.apache.hadoop.hdfs.server.namenode.CacheManager.validateExpiryTime(CacheManager.java:378) at org.apache.hadoop.hdfs.server.namenode.CacheManager.addDirective(CacheManager.java:528) at org.apache.hadoop.hdfs.server.namenode.FSNDNCacheOp.addCacheDirective(FSNDNCacheOp.java:45) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.addCacheDirective(FSNamesystem.java:6782) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addCacheDirective(NameNodeRpcServer.java:1883) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addCacheDirective(ClientNamenodeProtocolServerSideTranslatorPB.java:1265) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675) CAUSED BY: RemoteException: Expiration 26687997791:19:48:13.951 exceeds the max relative expiration time of 604800000 ms. at org.apache.hadoop.hdfs.server.namenode.CacheManager.validateExpiryTime(CacheManager.java:378) at org.apache.hadoop.hdfs.server.namenode.CacheManager.addDirective(CacheManager.java:528) at org.apache.hadoop.hdfs.server.namenode.FSNDNCacheOp.addCacheDirective(FSNDNCacheOp.java:45) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.addCacheDirective(FSNamesystem.java:6782) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addCacheDirective(NameNodeRpcServer.java:1883) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addCacheDirective(ClientNamenodeProtocolServerSideTranslatorPB.java:1265) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675) {code} this is always reproducible. The workaround is to do not set the maxTtl in the hdfs command during the pool creation. Here's the repro steps: {code:java} impala-shell -i <ImpalaD hostname> -q "drop table if exists foo;" impala-shell -i <ImpalaD hostname> -q "create table if not exists foo (c1 int, c2 string) partitioned by (p1 int) stored as parquet location '/user/hive/warehouse/foo';" impala-shell -i <ImpalaD hostname> -q "insert into foo partition (p1 = 1 ) values (1 ,'one');" sudo -u hdfs hdfs cacheadmin -removePool foo sudo -u hdfs hdfs cacheadmin -listDirectives -stats sudo -u hdfs hdfs cacheadmin -listPools -stats sudo -u hdfs hdfs cacheadmin -addPool foo -owner impala -group hdfs -mode 755 -limit 100000000000 -maxTtl 7d sudo -u hdfs hdfs cacheadmin -addDirective -path /user/hive/warehouse/foo -pool foo -ttl 7d sudo -u hdfs hdfs cacheadmin -listDirectives -stats sudo -u hdfs hdfs cacheadmin -listPools -stats impala-shell -i <ImpalaD hostname> -q "alter table foo set uncached;" sleep 5; impala-shell -i <ImpalaD hostname> -q "alter table foo partition (p1=1) set cached in 'foo';" {code} I did not found any already open jira for this issue that looks reproducible in many CDH versions, I appreciate if you can take a look on it. > HDFS Caching with Impala: Expiration 26687997791:19:48:13.951 exceeds the max > relative expiration time of <maxTtl> > ------------------------------------------------------------------------------------------------------------------ > > Key: IMPALA-9055 > URL: https://issues.apache.org/jira/browse/IMPALA-9055 > Project: IMPALA > Issue Type: Bug > Components: Catalog > Reporter: Adriano > Priority: Major > > HDFS Caching with Impala: > If we create a pool specifying the maxTtl with the hdfs command: > e.g: > {{sudo -u hdfs hdfs cacheadmin -addPool foo -owner impala -group hdfs -mode > 755 -limit 100000000000 -maxTtl 7d}} > when we try to alter a table adding a partition in Impala: > e.g: > {code:java} > alter table foo partition (p1=1) set cached in 'foo' > {code} > we get a failure with the exception: > {code:java} > ERROR: ImpalaRuntimeException: Expiration 26687997791:19:48:13.951 exceeds > the max relative expiration time of 604800000 ms. > at > org.apache.hadoop.hdfs.server.namenode.CacheManager.validateExpiryTime(CacheManager.java:378) > at > org.apache.hadoop.hdfs.server.namenode.CacheManager.addDirective(CacheManager.java:528) > at > org.apache.hadoop.hdfs.server.namenode.FSNDNCacheOp.addCacheDirective(FSNDNCacheOp.java:45) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.addCacheDirective(FSNamesystem.java:6782) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addCacheDirective(NameNodeRpcServer.java:1883) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addCacheDirective(ClientNamenodeProtocolServerSideTranslatorPB.java:1265) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675) > CAUSED BY: InvalidRequestException: Expiration 26687997791:19:48:13.951 > exceeds the max relative expiration time of 604800000 ms. > at > org.apache.hadoop.hdfs.server.namenode.CacheManager.validateExpiryTime(CacheManager.java:378) > at > org.apache.hadoop.hdfs.server.namenode.CacheManager.addDirective(CacheManager.java:528) > at > org.apache.hadoop.hdfs.server.namenode.FSNDNCacheOp.addCacheDirective(FSNDNCacheOp.java:45) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.addCacheDirective(FSNamesystem.java:6782) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addCacheDirective(NameNodeRpcServer.java:1883) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addCacheDirective(ClientNamenodeProtocolServerSideTranslatorPB.java:1265) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675) > CAUSED BY: RemoteException: Expiration 26687997791:19:48:13.951 exceeds the > max relative expiration time of 604800000 ms. > at > org.apache.hadoop.hdfs.server.namenode.CacheManager.validateExpiryTime(CacheManager.java:378) > at > org.apache.hadoop.hdfs.server.namenode.CacheManager.addDirective(CacheManager.java:528) > at > org.apache.hadoop.hdfs.server.namenode.FSNDNCacheOp.addCacheDirective(FSNDNCacheOp.java:45) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.addCacheDirective(FSNamesystem.java:6782) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addCacheDirective(NameNodeRpcServer.java:1883) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addCacheDirective(ClientNamenodeProtocolServerSideTranslatorPB.java:1265) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675) > {code} > this is always reproducible. > The workaround is to do not set the maxTtl in the hdfs command during the > pool creation. > Here's the repro steps: > {code:java} > impala-shell -i <ImpalaD hostname> -q "drop table if exists foo;" > impala-shell -i <ImpalaD hostname> -q "create table if not exists foo (c1 > int, c2 string) partitioned by (p1 int) stored as parquet location > '/user/hive/warehouse/foo';" > impala-shell -i <ImpalaD hostname> -q "insert into foo partition (p1 = 1 ) > values (1 ,'one');" > sudo -u hdfs hdfs cacheadmin -removePool foo > sudo -u hdfs hdfs cacheadmin -listDirectives -stats > sudo -u hdfs hdfs cacheadmin -listPools -stats > sudo -u hdfs hdfs cacheadmin -addPool foo -owner impala -group hdfs -mode 755 > -limit 100000000000 -maxTtl 7d > sudo -u hdfs hdfs cacheadmin -addDirective -path /user/hive/warehouse/foo > -pool foo -ttl 7d > sudo -u hdfs hdfs cacheadmin -listDirectives -stats > sudo -u hdfs hdfs cacheadmin -listPools -stats > impala-shell -i <ImpalaD hostname> -q "alter table foo set uncached;" > sleep 5; > impala-shell -i <ImpalaD hostname> -q "alter table foo partition (p1=1) set > cached in 'foo';" > {code} > I did not found any already open jira for this issue that looks reproducible > in many CDH versions, I appreciate if you can take a look on it. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org