[
https://issues.apache.org/jira/browse/IMPALA-9055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adriano updated IMPALA-9055:
----------------------------
Description:
HDFS Caching with Impala:
If we create a pool specifying the maxTtl with the hdfs command:
e.g:
{{sudo -u hdfs hdfs cacheadmin -addPool foo -owner impala -group hdfs -mode 755
-limit 100000000000 -maxTtl 7d}}
when we try to alter a table adding a partition in Impala:
e.g:
{code:java}
alter table foo partition (p1=1) set cached in 'foo'
{code}
we get a failure with the exception:
{code:java}
ERROR: ImpalaRuntimeException: Expiration 26687997791:19:48:13.951 exceeds the
max relative expiration time of 604800000 ms.
at
org.apache.hadoop.hdfs.server.namenode.CacheManager.validateExpiryTime(CacheManager.java:378)
at
org.apache.hadoop.hdfs.server.namenode.CacheManager.addDirective(CacheManager.java:528)
at
org.apache.hadoop.hdfs.server.namenode.FSNDNCacheOp.addCacheDirective(FSNDNCacheOp.java:45)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.addCacheDirective(FSNamesystem.java:6782)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addCacheDirective(NameNodeRpcServer.java:1883)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addCacheDirective(ClientNamenodeProtocolServerSideTranslatorPB.java:1265)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
CAUSED BY: InvalidRequestException: Expiration 26687997791:19:48:13.951 exceeds
the max relative expiration time of 604800000 ms.
at
org.apache.hadoop.hdfs.server.namenode.CacheManager.validateExpiryTime(CacheManager.java:378)
at
org.apache.hadoop.hdfs.server.namenode.CacheManager.addDirective(CacheManager.java:528)
at
org.apache.hadoop.hdfs.server.namenode.FSNDNCacheOp.addCacheDirective(FSNDNCacheOp.java:45)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.addCacheDirective(FSNamesystem.java:6782)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addCacheDirective(NameNodeRpcServer.java:1883)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addCacheDirective(ClientNamenodeProtocolServerSideTranslatorPB.java:1265)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
CAUSED BY: RemoteException: Expiration 26687997791:19:48:13.951 exceeds the max
relative expiration time of 604800000 ms.
at
org.apache.hadoop.hdfs.server.namenode.CacheManager.validateExpiryTime(CacheManager.java:378)
at
org.apache.hadoop.hdfs.server.namenode.CacheManager.addDirective(CacheManager.java:528)
at
org.apache.hadoop.hdfs.server.namenode.FSNDNCacheOp.addCacheDirective(FSNDNCacheOp.java:45)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.addCacheDirective(FSNamesystem.java:6782)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addCacheDirective(NameNodeRpcServer.java:1883)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addCacheDirective(ClientNamenodeProtocolServerSideTranslatorPB.java:1265)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
{code}
this is always reproducible.
The workaround is to do not set the maxTtl in the hdfs command during the pool
creation.
Here's the repro steps:
{code:java}
impala-shell -i <ImpalaD hostname> -q "drop table if exists foo;"
impala-shell -i <ImpalaD hostname> -q "create table if not exists foo (c1 int,
c2 string) partitioned by (p1 int) stored as parquet location
'/user/hive/warehouse/foo';"
impala-shell -i <ImpalaD hostname> -q "insert into foo partition (p1 = 1 )
values (1 ,'one');"
sudo -u hdfs hdfs cacheadmin -removePool foo
sudo -u hdfs hdfs cacheadmin -listDirectives -stats
sudo -u hdfs hdfs cacheadmin -listPools -stats
sudo -u hdfs hdfs cacheadmin -addPool foo -owner impala -group hdfs -mode 755
-limit 100000000000 -maxTtl 7d
sudo -u hdfs hdfs cacheadmin -addDirective -path /user/hive/warehouse/foo -pool
foo -ttl 7d
sudo -u hdfs hdfs cacheadmin -listDirectives -stats
sudo -u hdfs hdfs cacheadmin -listPools -stats
impala-shell -i <ImpalaD hostname> -q "alter table foo set uncached;"
sleep 5;
impala-shell -i <ImpalaD hostname> -q "alter table foo partition (p1=1) set
cached in 'foo';"
{code}
I did not found any already open jira for this issue that looks reproducible in
many CDH versions, I appreciate if you can take a look on it.
was:
HDFS Caching with Impala:
If we create a pool specifying the maxTtl with the hdfs command:
e.g:
{{sudo -u hdfs hdfs cacheadmin -addPool case422446 -owner impala -group hdfs
-mode 755 -limit 100000000000 -maxTtl 7d}}
when we try to alter a table adding a partition in Impala:
e.g:
{code:java}
alter table foo partition (p1=1) set cached in 'foo'
{code}
we get a failure with the exception:
{code:java}
ERROR: ImpalaRuntimeException: Expiration 26687997791:19:48:13.951 exceeds the
max relative expiration time of 604800000 ms.
at
org.apache.hadoop.hdfs.server.namenode.CacheManager.validateExpiryTime(CacheManager.java:378)
at
org.apache.hadoop.hdfs.server.namenode.CacheManager.addDirective(CacheManager.java:528)
at
org.apache.hadoop.hdfs.server.namenode.FSNDNCacheOp.addCacheDirective(FSNDNCacheOp.java:45)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.addCacheDirective(FSNamesystem.java:6782)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addCacheDirective(NameNodeRpcServer.java:1883)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addCacheDirective(ClientNamenodeProtocolServerSideTranslatorPB.java:1265)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
CAUSED BY: InvalidRequestException: Expiration 26687997791:19:48:13.951 exceeds
the max relative expiration time of 604800000 ms.
at
org.apache.hadoop.hdfs.server.namenode.CacheManager.validateExpiryTime(CacheManager.java:378)
at
org.apache.hadoop.hdfs.server.namenode.CacheManager.addDirective(CacheManager.java:528)
at
org.apache.hadoop.hdfs.server.namenode.FSNDNCacheOp.addCacheDirective(FSNDNCacheOp.java:45)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.addCacheDirective(FSNamesystem.java:6782)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addCacheDirective(NameNodeRpcServer.java:1883)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addCacheDirective(ClientNamenodeProtocolServerSideTranslatorPB.java:1265)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
CAUSED BY: RemoteException: Expiration 26687997791:19:48:13.951 exceeds the max
relative expiration time of 604800000 ms.
at
org.apache.hadoop.hdfs.server.namenode.CacheManager.validateExpiryTime(CacheManager.java:378)
at
org.apache.hadoop.hdfs.server.namenode.CacheManager.addDirective(CacheManager.java:528)
at
org.apache.hadoop.hdfs.server.namenode.FSNDNCacheOp.addCacheDirective(FSNDNCacheOp.java:45)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.addCacheDirective(FSNamesystem.java:6782)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addCacheDirective(NameNodeRpcServer.java:1883)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addCacheDirective(ClientNamenodeProtocolServerSideTranslatorPB.java:1265)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
{code}
this is always reproducible.
The workaround is to do not set the maxTtl in the hdfs command during the pool
creation.
Here's the repro steps:
{code:java}
impala-shell -i <ImpalaD hostname> -q "drop table if exists foo;"
impala-shell -i <ImpalaD hostname> -q "create table if not exists foo (c1 int,
c2 string) partitioned by (p1 int) stored as parquet location
'/user/hive/warehouse/foo';"
impala-shell -i <ImpalaD hostname> -q "insert into foo partition (p1 = 1 )
values (1 ,'one');"
sudo -u hdfs hdfs cacheadmin -removePool foo
sudo -u hdfs hdfs cacheadmin -listDirectives -stats
sudo -u hdfs hdfs cacheadmin -listPools -stats
sudo -u hdfs hdfs cacheadmin -addPool foo -owner impala -group hdfs -mode 755
-limit 100000000000 -maxTtl 7d
sudo -u hdfs hdfs cacheadmin -addDirective -path /user/hive/warehouse/foo -pool
foo -ttl 7d
sudo -u hdfs hdfs cacheadmin -listDirectives -stats
sudo -u hdfs hdfs cacheadmin -listPools -stats
impala-shell -i <ImpalaD hostname> -q "alter table foo set uncached;"
sleep 5;
impala-shell -i <ImpalaD hostname> -q "alter table foo partition (p1=1) set
cached in 'foo';"
{code}
I did not found any already open jira for this issue that looks reproducible in
many CDH versions, I appreciate if you can take a look on it.
> HDFS Caching with Impala: Expiration 26687997791:19:48:13.951 exceeds the max
> relative expiration time of <maxTtl>
> ------------------------------------------------------------------------------------------------------------------
>
> Key: IMPALA-9055
> URL: https://issues.apache.org/jira/browse/IMPALA-9055
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Adriano
> Priority: Major
>
> HDFS Caching with Impala:
> If we create a pool specifying the maxTtl with the hdfs command:
> e.g:
> {{sudo -u hdfs hdfs cacheadmin -addPool foo -owner impala -group hdfs -mode
> 755 -limit 100000000000 -maxTtl 7d}}
> when we try to alter a table adding a partition in Impala:
> e.g:
> {code:java}
> alter table foo partition (p1=1) set cached in 'foo'
> {code}
> we get a failure with the exception:
> {code:java}
> ERROR: ImpalaRuntimeException: Expiration 26687997791:19:48:13.951 exceeds
> the max relative expiration time of 604800000 ms.
> at
> org.apache.hadoop.hdfs.server.namenode.CacheManager.validateExpiryTime(CacheManager.java:378)
> at
> org.apache.hadoop.hdfs.server.namenode.CacheManager.addDirective(CacheManager.java:528)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNDNCacheOp.addCacheDirective(FSNDNCacheOp.java:45)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.addCacheDirective(FSNamesystem.java:6782)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addCacheDirective(NameNodeRpcServer.java:1883)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addCacheDirective(ClientNamenodeProtocolServerSideTranslatorPB.java:1265)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
> CAUSED BY: InvalidRequestException: Expiration 26687997791:19:48:13.951
> exceeds the max relative expiration time of 604800000 ms.
> at
> org.apache.hadoop.hdfs.server.namenode.CacheManager.validateExpiryTime(CacheManager.java:378)
> at
> org.apache.hadoop.hdfs.server.namenode.CacheManager.addDirective(CacheManager.java:528)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNDNCacheOp.addCacheDirective(FSNDNCacheOp.java:45)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.addCacheDirective(FSNamesystem.java:6782)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addCacheDirective(NameNodeRpcServer.java:1883)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addCacheDirective(ClientNamenodeProtocolServerSideTranslatorPB.java:1265)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
> CAUSED BY: RemoteException: Expiration 26687997791:19:48:13.951 exceeds the
> max relative expiration time of 604800000 ms.
> at
> org.apache.hadoop.hdfs.server.namenode.CacheManager.validateExpiryTime(CacheManager.java:378)
> at
> org.apache.hadoop.hdfs.server.namenode.CacheManager.addDirective(CacheManager.java:528)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNDNCacheOp.addCacheDirective(FSNDNCacheOp.java:45)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.addCacheDirective(FSNamesystem.java:6782)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addCacheDirective(NameNodeRpcServer.java:1883)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addCacheDirective(ClientNamenodeProtocolServerSideTranslatorPB.java:1265)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
> {code}
> this is always reproducible.
> The workaround is to do not set the maxTtl in the hdfs command during the
> pool creation.
> Here's the repro steps:
> {code:java}
> impala-shell -i <ImpalaD hostname> -q "drop table if exists foo;"
> impala-shell -i <ImpalaD hostname> -q "create table if not exists foo (c1
> int, c2 string) partitioned by (p1 int) stored as parquet location
> '/user/hive/warehouse/foo';"
> impala-shell -i <ImpalaD hostname> -q "insert into foo partition (p1 = 1 )
> values (1 ,'one');"
> sudo -u hdfs hdfs cacheadmin -removePool foo
> sudo -u hdfs hdfs cacheadmin -listDirectives -stats
> sudo -u hdfs hdfs cacheadmin -listPools -stats
> sudo -u hdfs hdfs cacheadmin -addPool foo -owner impala -group hdfs -mode 755
> -limit 100000000000 -maxTtl 7d
> sudo -u hdfs hdfs cacheadmin -addDirective -path /user/hive/warehouse/foo
> -pool foo -ttl 7d
> sudo -u hdfs hdfs cacheadmin -listDirectives -stats
> sudo -u hdfs hdfs cacheadmin -listPools -stats
> impala-shell -i <ImpalaD hostname> -q "alter table foo set uncached;"
> sleep 5;
> impala-shell -i <ImpalaD hostname> -q "alter table foo partition (p1=1) set
> cached in 'foo';"
> {code}
> I did not found any already open jira for this issue that looks reproducible
> in many CDH versions, I appreciate if you can take a look on it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
