[jira] [Updated] (IMPALA-11421) make idle_session_timeout no overridable in the user session
[ https://issues.apache.org/jira/browse/IMPALA-11421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adriano updated IMPALA-11421: - Description: hi, afaik idle_session_timeout is overridable in the user session, would be a good idea to make it un-overridable as we did with the query MEM_LIMIT (clamping the option). In a big cluster, with thousand of tenants would be nice to have control over this options too. was: hi, afaik idle_session_timeout are overridable in the user session, would be a good idea to make them un-overridable as we did with the query MEM_LIMIT (clamping the option). In a big cluster, with thousand of tenants would be nice to have control over these options too. > make idle_session_timeout no overridable in the user session > - > > Key: IMPALA-11421 > URL: https://issues.apache.org/jira/browse/IMPALA-11421 > Project: IMPALA > Issue Type: New Feature >Reporter: Adriano >Priority: Major > > hi, > afaik idle_session_timeout is overridable in the user session, would be a > good idea to make it un-overridable as we did with the query MEM_LIMIT > (clamping the option). > In a big cluster, with thousand of tenants would be nice to have control over > this options too. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-11421) make idle_session_timeout no overridable in the user session
[ https://issues.apache.org/jira/browse/IMPALA-11421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adriano updated IMPALA-11421: - Description: hi, afaik idle_session_timeout are overridable in the user session, would be a good idea to make them un-overridable as we did with the query MEM_LIMIT (clamping the option). In a big cluster, with thousand of tenants would be nice to have control over these options too. was: hi, afaik idle_[query|session]_timeout are overridable in the user session, would be a good idea to make them un-overridable as we did with the query MEM_LIMIT (clamping the option). In a big cluster, with thousand of tenants would be nice to have control over these options too. > make idle_session_timeout no overridable in the user session > - > > Key: IMPALA-11421 > URL: https://issues.apache.org/jira/browse/IMPALA-11421 > Project: IMPALA > Issue Type: New Feature >Reporter: Adriano >Priority: Major > > hi, > afaik idle_session_timeout are overridable in the user session, would be a > good idea to make them un-overridable as we did with the query MEM_LIMIT > (clamping the option). > In a big cluster, with thousand of tenants would be nice to have control over > these options too. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-11421) make idle_session_timeout no overridable in the user session
[ https://issues.apache.org/jira/browse/IMPALA-11421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adriano updated IMPALA-11421: - Summary: make idle_session_timeout no overridable in the user session (was: make idle_[query|session]_timeout no overridable in the user session) > make idle_session_timeout no overridable in the user session > - > > Key: IMPALA-11421 > URL: https://issues.apache.org/jira/browse/IMPALA-11421 > Project: IMPALA > Issue Type: New Feature >Reporter: Adriano >Priority: Major > > hi, > afaik idle_[query|session]_timeout are overridable in the user session, would > be a good idea to make them un-overridable as we did with the query MEM_LIMIT > (clamping the option). > In a big cluster, with thousand of tenants would be nice to have control over > these options too. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-11421) make idle_[query|session]_timeout no overridable in the user session
Adriano created IMPALA-11421: Summary: make idle_[query|session]_timeout no overridable in the user session Key: IMPALA-11421 URL: https://issues.apache.org/jira/browse/IMPALA-11421 Project: IMPALA Issue Type: New Feature Reporter: Adriano hi, afaik idle_[query|session]_timeout are overridable in the user session, would be a good idea to make them un-overridable as we did with the query MEM_LIMIT (clamping the option). In a big cluster, with thousand of tenants would be nice to have control over these options too. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-10502) delayed 'Invalidated objects in cache' cause 'Table already exists'
Adriano created IMPALA-10502: Summary: delayed 'Invalidated objects in cache' cause 'Table already exists' Key: IMPALA-10502 URL: https://issues.apache.org/jira/browse/IMPALA-10502 Project: IMPALA Issue Type: Bug Components: Catalog, Clients, Frontend Affects Versions: Impala 3.4.0 Reporter: Adriano In fast paced environment where the interval between the step 1 and 2 is # < 100ms (a simplified pipeline looks like): 0- catalog 'on demand' in use and disableHmsSync (enabled or disabled: no difference) 1- open session to coord A -> DROP TABLE X -> close session 2- open session to coord A -> CREATE TABLE X-> close session Results: the step -2- can fail with table already exist. During the internal investigation was discovered that IMPALA-9913 will regress the issue in almost all scenarios. However considering that the investigation are internally ongoing it is nice to have the event tracked also here. Once we are sure that IMPALA-9913 fix these events we can close this as duplicate, in alternative carry on the investigation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-10396) DECIMAL_V2=false: exception during analysis after expr substitution
Adriano created IMPALA-10396:
Summary: DECIMAL_V2=false: exception during analysis after expr
substitution
Key: IMPALA-10396
URL: https://issues.apache.org/jira/browse/IMPALA-10396
Project: IMPALA
Issue Type: Bug
Components: Frontend
Affects Versions: Impala 3.4.0, Impala 3.3.0, Impala 3.2.0, Impala 3.1.0,
Impala 2.12.0, Impala 3.0
Reporter: Adriano
When DECIMAL_V2 is false in some edge cases the precondition check at
[https://github.com/apache/impala/blob/aeeff53e884a67ee7f5980654a1d394c6e3e34ac/fe/src/main/java/org/apache/impala/analysis/TypesUtil.java#L89]
by getArithmeticResultType() when Expr#analyzeImpl() at
[https://github.com/apache/impala/blob/5530b62539e762ddf5825e2b43db2f29d9addae7/fe/src/main/java/org/apache/impala/analysis/Expr.java#L503]
can drop an exception while it try to analyze the Expr tree.
Here the stack:
{code:java}
I1201 09:06:18.680207 31885 jni-util.cc:288] 55470855fd2e6aba:c8d32a28]
java.lang.IllegalStateException: Failed analysis after expr substitution. at
org.apache.impala.analysis.Expr.substitute(Expr.java:1091) at
org.apache.impala.analysis.Analyzer.isTrueWithNullSlots(Analyzer.java:2316) at
org.apache.impala.analysis.TupleIsNullPredicate.requiresNullWrapping(TupleIsNullPredicate.java:175)
at
org.apache.impala.analysis.TupleIsNullPredicate.wrapExpr(TupleIsNullPredicate.java:147)
at
org.apache.impala.analysis.TupleIsNullPredicate.wrapExprs(TupleIsNullPredicate.java:136)
at
org.apache.impala.planner.SingleNodePlanner.createInlineViewPlan(SingleNodePlanner.java:1238)
at
org.apache.impala.planner.SingleNodePlanner.createTableRefNode(SingleNodePlanner.java:2078)
at
org.apache.impala.planner.SingleNodePlanner.createTableRefsPlan(SingleNodePlanner.java:940)
at
org.apache.impala.planner.SingleNodePlanner.createSelectPlan(SingleNodePlanner.java:768)
at
org.apache.impala.planner.SingleNodePlanner.createQueryPlan(SingleNodePlanner.java:276)
at
org.apache.impala.planner.SingleNodePlanner.createInlineViewPlan(SingleNodePlanner.java:1220)
at
org.apache.impala.planner.SingleNodePlanner.createTableRefNode(SingleNodePlanner.java:2078)
at
org.apache.impala.planner.SingleNodePlanner.createTableRefsPlan(SingleNodePlanner.java:940)
at
org.apache.impala.planner.SingleNodePlanner.createSelectPlan(SingleNodePlanner.java:768)
at
org.apache.impala.planner.SingleNodePlanner.createQueryPlan(SingleNodePlanner.java:276)
at
org.apache.impala.planner.SingleNodePlanner.createSingleNodePlan(SingleNodePlanner.java:169)
at org.apache.impala.planner.Planner.createPlanFragments(Planner.java:118) at
org.apache.impala.planner.Planner.createPlans(Planner.java:245) at
org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1504) at
org.apache.impala.service.Frontend.getPlannedExecRequest(Frontend.java:1834) at
org.apache.impala.service.Frontend.doCreateExecRequest(Frontend.java:1692) at
org.apache.impala.service.Frontend.getTExecRequest(Frontend.java:1585) at
org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1555) at
org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:159)
Caused by: java.lang.IllegalStateException at
com.google.common.base.Preconditions.checkState(Preconditions.java:492) at
org.apache.impala.analysis.TypesUtil.getArithmeticResultType(TypesUtil.java:89)
at
org.apache.impala.analysis.ArithmeticExpr.analyzeImpl(ArithmeticExpr.java:202)
at org.apache.impala.analysis.Expr.analyze(Expr.java:503) at
org.apache.impala.analysis.Expr.analyze(Expr.java:497) at
org.apache.impala.analysis.Expr.analyze(Expr.java:497) at
org.apache.impala.analysis.Expr.analyze(Expr.java:497) at
org.apache.impala.analysis.Expr.analyze(Expr.java:497) at
org.apache.impala.analysis.Expr.analyze(Expr.java:497) at
org.apache.impala.analysis.Expr.analyze(Expr.java:497) at
org.apache.impala.analysis.Expr.analyze(Expr.java:497) at
org.apache.impala.analysis.Expr.analyze(Expr.java:497) at
org.apache.impala.analysis.Expr.analyze(Expr.java:497) at
org.apache.impala.analysis.Expr.trySubstitute(Expr.java:1072) at
org.apache.impala.analysis.Expr.substitute(Expr.java:1089)
{code}
This does not happen when DECIMAL_V2 = true.
+WorkAround available:+ SET DECIMAL_V2=true;
Internal investigation tracked with ENGESC-5645
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-9055) HDFS Caching with Impala: Expiration 26687997791:19:48:13.951 exceeds the max relative expiration time of
[
https://issues.apache.org/jira/browse/IMPALA-9055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adriano updated IMPALA-9055:
Description:
HDFS Caching with Impala:
If we create a pool specifying the maxTtl with the hdfs command:
e.g:
{{sudo -u hdfs hdfs cacheadmin -addPool case422446 -owner impala -group hdfs
-mode 755 -limit 1000 -maxTtl 7d}}
when we try to alter a table adding a partition in Impala:
e.g:
{code:java}
alter table foo partition (p1=1) set cached in 'foo'
{code}
we get a failure with the exception:
{code:java}
ERROR: ImpalaRuntimeException: Expiration 26687997791:19:48:13.951 exceeds the
max relative expiration time of 60480 ms.
at
org.apache.hadoop.hdfs.server.namenode.CacheManager.validateExpiryTime(CacheManager.java:378)
at
org.apache.hadoop.hdfs.server.namenode.CacheManager.addDirective(CacheManager.java:528)
at
org.apache.hadoop.hdfs.server.namenode.FSNDNCacheOp.addCacheDirective(FSNDNCacheOp.java:45)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.addCacheDirective(FSNamesystem.java:6782)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addCacheDirective(NameNodeRpcServer.java:1883)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addCacheDirective(ClientNamenodeProtocolServerSideTranslatorPB.java:1265)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
CAUSED BY: InvalidRequestException: Expiration 26687997791:19:48:13.951 exceeds
the max relative expiration time of 60480 ms.
at
org.apache.hadoop.hdfs.server.namenode.CacheManager.validateExpiryTime(CacheManager.java:378)
at
org.apache.hadoop.hdfs.server.namenode.CacheManager.addDirective(CacheManager.java:528)
at
org.apache.hadoop.hdfs.server.namenode.FSNDNCacheOp.addCacheDirective(FSNDNCacheOp.java:45)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.addCacheDirective(FSNamesystem.java:6782)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addCacheDirective(NameNodeRpcServer.java:1883)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addCacheDirective(ClientNamenodeProtocolServerSideTranslatorPB.java:1265)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
CAUSED BY: RemoteException: Expiration 26687997791:19:48:13.951 exceeds the max
relative expiration time of 60480 ms.
at
org.apache.hadoop.hdfs.server.namenode.CacheManager.validateExpiryTime(CacheManager.java:378)
at
org.apache.hadoop.hdfs.server.namenode.CacheManager.addDirective(CacheManager.java:528)
at
org.apache.hadoop.hdfs.server.namenode.FSNDNCacheOp.addCacheDirective(FSNDNCacheOp.java:45)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.addCacheDirective(FSNamesystem.java:6782)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addCacheDirective(NameNodeRpcServer.java:1883)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addCacheDirective(ClientNamenodeProtocolServerSideTranslatorPB.java:1265)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
at
[jira] [Updated] (IMPALA-9055) HDFS Caching with Impala: Expiration 26687997791:19:48:13.951 exceeds the max relative expiration time of
[
https://issues.apache.org/jira/browse/IMPALA-9055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adriano updated IMPALA-9055:
Description:
HDFS Caching with Impala:
If we create a pool specifying the maxTtl with the hdfs command:
e.g:
{{sudo -u hdfs hdfs cacheadmin -addPool foo -owner impala -group hdfs -mode 755
-limit 1000 -maxTtl 7d}}
when we try to alter a table adding a partition in Impala:
e.g:
{code:java}
alter table foo partition (p1=1) set cached in 'foo'
{code}
we get a failure with the exception:
{code:java}
ERROR: ImpalaRuntimeException: Expiration 26687997791:19:48:13.951 exceeds the
max relative expiration time of 60480 ms.
at
org.apache.hadoop.hdfs.server.namenode.CacheManager.validateExpiryTime(CacheManager.java:378)
at
org.apache.hadoop.hdfs.server.namenode.CacheManager.addDirective(CacheManager.java:528)
at
org.apache.hadoop.hdfs.server.namenode.FSNDNCacheOp.addCacheDirective(FSNDNCacheOp.java:45)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.addCacheDirective(FSNamesystem.java:6782)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addCacheDirective(NameNodeRpcServer.java:1883)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addCacheDirective(ClientNamenodeProtocolServerSideTranslatorPB.java:1265)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
CAUSED BY: InvalidRequestException: Expiration 26687997791:19:48:13.951 exceeds
the max relative expiration time of 60480 ms.
at
org.apache.hadoop.hdfs.server.namenode.CacheManager.validateExpiryTime(CacheManager.java:378)
at
org.apache.hadoop.hdfs.server.namenode.CacheManager.addDirective(CacheManager.java:528)
at
org.apache.hadoop.hdfs.server.namenode.FSNDNCacheOp.addCacheDirective(FSNDNCacheOp.java:45)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.addCacheDirective(FSNamesystem.java:6782)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addCacheDirective(NameNodeRpcServer.java:1883)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addCacheDirective(ClientNamenodeProtocolServerSideTranslatorPB.java:1265)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
CAUSED BY: RemoteException: Expiration 26687997791:19:48:13.951 exceeds the max
relative expiration time of 60480 ms.
at
org.apache.hadoop.hdfs.server.namenode.CacheManager.validateExpiryTime(CacheManager.java:378)
at
org.apache.hadoop.hdfs.server.namenode.CacheManager.addDirective(CacheManager.java:528)
at
org.apache.hadoop.hdfs.server.namenode.FSNDNCacheOp.addCacheDirective(FSNDNCacheOp.java:45)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.addCacheDirective(FSNamesystem.java:6782)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addCacheDirective(NameNodeRpcServer.java:1883)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addCacheDirective(ClientNamenodeProtocolServerSideTranslatorPB.java:1265)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
at
[jira] [Created] (IMPALA-9055) HDFS Caching with Impala: Expiration 26687997791:19:48:13.951 exceeds the max relative expiration time of
Adriano created IMPALA-9055:
---
Summary: HDFS Caching with Impala: Expiration
26687997791:19:48:13.951 exceeds the max relative expiration time of
Key: IMPALA-9055
URL: https://issues.apache.org/jira/browse/IMPALA-9055
Project: IMPALA
Issue Type: Bug
Components: Catalog
Reporter: Adriano
HDFS Caching with Impala:
If we create a pool specifying the maxTtl with the hdfs command:
e.g:
{{sudo -u hdfs hdfs cacheadmin -addPool case422446 -owner impala -group hdfs
-mode 755 -limit 1000 -maxTtl 7d}}
when we try to alter a table adding a partition in Impala:
e.g:
{{alter table foo partition (p1=1) set cached in 'foo'
}}
we get a failure with the exception:
ERROR: ImpalaRuntimeException: Expiration 26687997791:19:48:13.951 exceeds the
max relative expiration time of 60480 ms.
at
org.apache.hadoop.hdfs.server.namenode.CacheManager.validateExpiryTime(CacheManager.java:378)
at
org.apache.hadoop.hdfs.server.namenode.CacheManager.addDirective(CacheManager.java:528)
at
org.apache.hadoop.hdfs.server.namenode.FSNDNCacheOp.addCacheDirective(FSNDNCacheOp.java:45)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.addCacheDirective(FSNamesystem.java:6782)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addCacheDirective(NameNodeRpcServer.java:1883)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addCacheDirective(ClientNamenodeProtocolServerSideTranslatorPB.java:1265)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
CAUSED BY: InvalidRequestException: Expiration 26687997791:19:48:13.951 exceeds
the max relative expiration time of 60480 ms.
at
org.apache.hadoop.hdfs.server.namenode.CacheManager.validateExpiryTime(CacheManager.java:378)
at
org.apache.hadoop.hdfs.server.namenode.CacheManager.addDirective(CacheManager.java:528)
at
org.apache.hadoop.hdfs.server.namenode.FSNDNCacheOp.addCacheDirective(FSNDNCacheOp.java:45)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.addCacheDirective(FSNamesystem.java:6782)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addCacheDirective(NameNodeRpcServer.java:1883)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addCacheDirective(ClientNamenodeProtocolServerSideTranslatorPB.java:1265)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
CAUSED BY: RemoteException: Expiration 26687997791:19:48:13.951 exceeds the max
relative expiration time of 60480 ms.
at
org.apache.hadoop.hdfs.server.namenode.CacheManager.validateExpiryTime(CacheManager.java:378)
at
org.apache.hadoop.hdfs.server.namenode.CacheManager.addDirective(CacheManager.java:528)
at
org.apache.hadoop.hdfs.server.namenode.FSNDNCacheOp.addCacheDirective(FSNDNCacheOp.java:45)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.addCacheDirective(FSNamesystem.java:6782)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addCacheDirective(NameNodeRpcServer.java:1883)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addCacheDirective(ClientNamenodeProtocolServerSideTranslatorPB.java:1265)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
[jira] [Commented] (IMPALA-8852) ImpalaD fail to start on a non-datanode with "Invalid short-circuit reads configuration"
[
https://issues.apache.org/jira/browse/IMPALA-8852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16905346#comment-16905346
]
Adriano commented on IMPALA-8852:
-
[~lv] agree, considering we have a solution adding the CM properties.
> ImpalaD fail to start on a non-datanode with "Invalid short-circuit reads
> configuration"
>
>
> Key: IMPALA-8852
> URL: https://issues.apache.org/jira/browse/IMPALA-8852
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.2.0, Impala 3.3.0
>Reporter: Adriano
>Priority: Major
> Labels: ramp-up
>
> On coordinator only nodes ([typically the edge
> nodes|https://www.cloudera.com/documentation/enterprise/5-15-x/topics/impala_dedicated_coordinator.html#concept_omm_gf1_n2b]):
> {code:java}
> --is_coordinator=true
> --is_executor=false
> {code}
> the *dfs.domain.socket.path* (can be nonexistent on the local FS as the
> Datanode role eventually is not installed on the edge node).
> The non existing path prevent the ImpalaD to start with the message:
> {code:java}
> I0809 04:15:53.899714 25364 status.cc:124] Invalid short-circuit reads
> configuration:
> - Impala cannot read or execute the parent directory of
> dfs.domain.socket.path
> @ 0xb35f19
> @ 0x100e2fe
> @ 0x103f274
> @ 0x102836f
> @ 0xa9f573
> @ 0x7f97807e93d4
> @ 0xafb3b8
> E0809 04:15:53.899749 25364 impala-server.cc:278] Invalid short-circuit reads
> configuration:
> - Impala cannot read or execute the parent directory of
> dfs.domain.socket.path
> {code}
> despite a coordinator-only ImpalaD does not do short circuit reads.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Comment Edited] (IMPALA-8852) ImpalaD fail to start on a non-datanode with "Invalid short-circuit reads configuration"
[
https://issues.apache.org/jira/browse/IMPALA-8852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903830#comment-16903830
]
Adriano edited comment on IMPALA-8852 at 8/12/19 4:19 PM:
--
WORKAROUND -1-:
Create the dfs.domain.socket.path manually with the proper hdfs user permission
on the local fs as:
{code:java}
# mkdir /var/run/hdfs-sockets/
# chown hdfs:hadoop /var/run/hdfs-sockets/
# chmod 755 /var/run/hdfs-sockets/
# mkdir /var/run/hdfs-sockets/dn
# chown hdfs:hdfs /var/run/hdfs-sockets/dn
# chmod 1666 /var/run/hdfs-sockets/dn
{code}
*Best Solution*:
1) On the Dedicated Coordinators, where the "-is_executor=false" add the
following property into "Impala Daemon HDFS Advanced Configuration Snippet
(Safety Valve)":
{code:java}
dfs.client.read.shortcircuit
false
Disable shortcircuit on dedicated coordinator
{code}
2) Save the changes and restart the Impala Daemon Coordinator instances.
was (Author: adrenas):
WORKAROUND -1-:
Create the dfs.domain.socket.path manually with the proper hdfs user permission
on the local fs as:
{code:java}
# mkdir /var/run/hdfs-sockets/
# chown hdfs:hadoop /var/run/hdfs-sockets/
# chmod 755 /var/run/hdfs-sockets/
# mkdir /var/run/hdfs-sockets/dn
# chown hdfs:hdfs /var/run/hdfs-sockets/dn
# chmod 1666 /var/run/hdfs-sockets/dn
{code}
WORKAROUND -2-:
1) On the Dedicated Coordinators, where the "-is_executor=false" add the
following property into "Impala Daemon HDFS Advanced Configuration Snippet
(Safety Valve)":
{code:java}
dfs.client.read.shortcircuit
false
Disable shortcircuit on dedicated coordinator
{code}
2) Save the changes and restart the Impala Daemon Coordinator instances.
> ImpalaD fail to start on a non-datanode with "Invalid short-circuit reads
> configuration"
>
>
> Key: IMPALA-8852
> URL: https://issues.apache.org/jira/browse/IMPALA-8852
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.2.0, Impala 3.3.0
>Reporter: Adriano
>Priority: Major
> Labels: ramp-up
>
> On coordinator only nodes ([typically the edge
> nodes|https://www.cloudera.com/documentation/enterprise/5-15-x/topics/impala_dedicated_coordinator.html#concept_omm_gf1_n2b]):
> {code:java}
> --is_coordinator=true
> --is_executor=false
> {code}
> the *dfs.domain.socket.path* (can be nonexistent on the local FS as the
> Datanode role eventually is not installed on the edge node).
> The non existing path prevent the ImpalaD to start with the message:
> {code:java}
> I0809 04:15:53.899714 25364 status.cc:124] Invalid short-circuit reads
> configuration:
> - Impala cannot read or execute the parent directory of
> dfs.domain.socket.path
> @ 0xb35f19
> @ 0x100e2fe
> @ 0x103f274
> @ 0x102836f
> @ 0xa9f573
> @ 0x7f97807e93d4
> @ 0xafb3b8
> E0809 04:15:53.899749 25364 impala-server.cc:278] Invalid short-circuit reads
> configuration:
> - Impala cannot read or execute the parent directory of
> dfs.domain.socket.path
> {code}
> despite a coordinator-only ImpalaD does not do short circuit reads.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8852) ImpalaD fail to start on a non-datanode with "Invalid short-circuit reads configuration"
[
https://issues.apache.org/jira/browse/IMPALA-8852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adriano updated IMPALA-8852:
Summary: ImpalaD fail to start on a non-datanode with "Invalid
short-circuit reads configuration" (was: The dfs.domain.socket.path can be
nonexistent on coordinator only nodes)
> ImpalaD fail to start on a non-datanode with "Invalid short-circuit reads
> configuration"
>
>
> Key: IMPALA-8852
> URL: https://issues.apache.org/jira/browse/IMPALA-8852
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: Adriano
>Priority: Minor
>
> On coordinator only nodes ([typically the edge
> nodes|https://www.cloudera.com/documentation/enterprise/5-15-x/topics/impala_dedicated_coordinator.html#concept_omm_gf1_n2b]):
> {code:java}
> --is_coordinator=true
> --is_executor=false
> {code}
> the *dfs.domain.socket.path* (can be nonexistent on the local FS as the
> Datanode role eventually is not installed on the edge node).
> The non existing path prevent the ImpalaD to start with the message:
> {code:java}
> I0809 04:15:53.899714 25364 status.cc:124] Invalid short-circuit reads
> configuration:
> - Impala cannot read or execute the parent directory of
> dfs.domain.socket.path
> @ 0xb35f19
> @ 0x100e2fe
> @ 0x103f274
> @ 0x102836f
> @ 0xa9f573
> @ 0x7f97807e93d4
> @ 0xafb3b8
> E0809 04:15:53.899749 25364 impala-server.cc:278] Invalid short-circuit reads
> configuration:
> - Impala cannot read or execute the parent directory of
> dfs.domain.socket.path
> {code}
> despite a coordinator-only ImpalaD does not do short circuit reads.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Comment Edited] (IMPALA-8852) The dfs.domain.socket.path can be nonexistent on coordinator only nodes
[
https://issues.apache.org/jira/browse/IMPALA-8852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903830#comment-16903830
]
Adriano edited comment on IMPALA-8852 at 8/12/19 8:33 AM:
--
WORKAROUND -1-:
Create the dfs.domain.socket.path manually with the proper hdfs user permission
on the local fs as:
{code:java}
# mkdir /var/run/hdfs-sockets/
# chown hdfs:hadoop /var/run/hdfs-sockets/
# chmod 755 /var/run/hdfs-sockets/
# mkdir /var/run/hdfs-sockets/dn
# chown hdfs:hdfs /var/run/hdfs-sockets/dn
# chmod 1666 /var/run/hdfs-sockets/dn
{code}
WORKAROUND -2-:
1) On the Dedicated Coordinators, where the "-is_executor=false" add the
following property into "Impala Daemon HDFS Advanced Configuration Snippet
(Safety Valve)":
{code:java}
dfs.client.read.shortcircuit
false
Disable shortcircuit on dedicated coordinator
{code}
2) Save the changes and restart the Impala Daemon Coordinator instances.
was (Author: adrenas):
WORKAROUND -1-:
Create the dfs.domain.socket.path manually with the proper hdfs user permission
on the local fs as:
{code:java}
# mkdir /var/run/hdfs-sockets/
# chown hdfs:hadoop /var/run/hdfs-sockets/
# chmod 755 /var/run/hdfs-sockets/
# mkdir /var/run/hdfs-sockets/dn
# chown hdfs:hdfs /var/run/hdfs-sockets/dn
# chmod 1666 /var/run/hdfs-sockets/dn
{code}
WORKAROUND -2-:
1) On the Dedicated Coordinators, add the following line into "Impala Daemon
Command Line Argument Advanced Configuration Snippet (Safety Valve)":
-is_executor=false
2) Add the following property into "Impala Daemon HDFS Advanced Configuration
Snippet (Safety Valve)":
{code:java}
dfs.client.read.shortcircuit
false
Disable shortcircuit on dedicated coordinator
{code}
3) Save the changes and restart the Impala Daemon Coordinator instances.
> The dfs.domain.socket.path can be nonexistent on coordinator only nodes
> ---
>
> Key: IMPALA-8852
> URL: https://issues.apache.org/jira/browse/IMPALA-8852
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: Adriano
>Priority: Minor
>
> On coordinator only nodes ([typically the edge
> nodes|https://www.cloudera.com/documentation/enterprise/5-15-x/topics/impala_dedicated_coordinator.html#concept_omm_gf1_n2b]):
> {code:java}
> --is_coordinator=true
> --is_executor=false
> {code}
> the *dfs.domain.socket.path* (can be nonexistent on the local FS as the
> Datanode role eventually is not installed on the edge node).
> The non existing path prevent the ImpalaD to start with the message:
> {code:java}
> I0809 04:15:53.899714 25364 status.cc:124] Invalid short-circuit reads
> configuration:
> - Impala cannot read or execute the parent directory of
> dfs.domain.socket.path
> @ 0xb35f19
> @ 0x100e2fe
> @ 0x103f274
> @ 0x102836f
> @ 0xa9f573
> @ 0x7f97807e93d4
> @ 0xafb3b8
> E0809 04:15:53.899749 25364 impala-server.cc:278] Invalid short-circuit reads
> configuration:
> - Impala cannot read or execute the parent directory of
> dfs.domain.socket.path
> {code}
> despite a coordinator-only ImpalaD does not do short circuit reads.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Comment Edited] (IMPALA-8852) The dfs.domain.socket.path can be nonexistent on coordinator only nodes
[
https://issues.apache.org/jira/browse/IMPALA-8852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903830#comment-16903830
]
Adriano edited comment on IMPALA-8852 at 8/12/19 8:31 AM:
--
WORKAROUND -1-:
Create the dfs.domain.socket.path manually with the proper hdfs user permission
on the local fs as:
{code:java}
# mkdir /var/run/hdfs-sockets/
# chown hdfs:hadoop /var/run/hdfs-sockets/
# chmod 755 /var/run/hdfs-sockets/
# mkdir /var/run/hdfs-sockets/dn
# chown hdfs:hdfs /var/run/hdfs-sockets/dn
# chmod 1666 /var/run/hdfs-sockets/dn
{code}
WORKAROUND -2-:
1) On the Dedicated Coordinators, add the following line into "Impala Daemon
Command Line Argument Advanced Configuration Snippet (Safety Valve)":
-is_executor=false
2) Add the following property into "Impala Daemon HDFS Advanced Configuration
Snippet (Safety Valve)":
{code:java}
dfs.client.read.shortcircuit
false
Disable shortcircuit on dedicated coordinator
{code}
3) Save the changes and restart the Impala Daemon Coordinator instances.
was (Author: adrenas):
WORKAROUND:
Create the dfs.domain.socket.path manually with the proper hdfs user permission
on the local fs as:
{code:java}
# mkdir /var/run/hdfs-sockets/
# chown hdfs:hadoop /var/run/hdfs-sockets/
# chmod 755 /var/run/hdfs-sockets/
# mkdir /var/run/hdfs-sockets/dn
# chown hdfs:hdfs /var/run/hdfs-sockets/dn
# chmod 1666 /var/run/hdfs-sockets/dn
{code}
> The dfs.domain.socket.path can be nonexistent on coordinator only nodes
> ---
>
> Key: IMPALA-8852
> URL: https://issues.apache.org/jira/browse/IMPALA-8852
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: Adriano
>Priority: Minor
>
> On coordinator only nodes ([typically the edge
> nodes|https://www.cloudera.com/documentation/enterprise/5-15-x/topics/impala_dedicated_coordinator.html#concept_omm_gf1_n2b]):
> {code:java}
> --is_coordinator=true
> --is_executor=false
> {code}
> the *dfs.domain.socket.path* (can be nonexistent on the local FS as the
> Datanode role eventually is not installed on the edge node).
> The non existing path prevent the ImpalaD to start with the message:
> {code:java}
> I0809 04:15:53.899714 25364 status.cc:124] Invalid short-circuit reads
> configuration:
> - Impala cannot read or execute the parent directory of
> dfs.domain.socket.path
> @ 0xb35f19
> @ 0x100e2fe
> @ 0x103f274
> @ 0x102836f
> @ 0xa9f573
> @ 0x7f97807e93d4
> @ 0xafb3b8
> E0809 04:15:53.899749 25364 impala-server.cc:278] Invalid short-circuit reads
> configuration:
> - Impala cannot read or execute the parent directory of
> dfs.domain.socket.path
> {code}
> despite a coordinator-only ImpalaD does not do short circuit reads.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8852) The dfs.domain.socket.path can be nonexistent on coordinator only nodes
[
https://issues.apache.org/jira/browse/IMPALA-8852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adriano updated IMPALA-8852:
Priority: Minor (was: Major)
> The dfs.domain.socket.path can be nonexistent on coordinator only nodes
> ---
>
> Key: IMPALA-8852
> URL: https://issues.apache.org/jira/browse/IMPALA-8852
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: Adriano
>Priority: Minor
>
> On coordinator only nodes ([typically the edge
> nodes|https://www.cloudera.com/documentation/enterprise/5-15-x/topics/impala_dedicated_coordinator.html#concept_omm_gf1_n2b]):
> {code:java}
> --is_coordinator=true
> --is_executor=false
> {code}
> the *dfs.domain.socket.path* (can be nonexistent on the local FS as the
> Datanode role eventually is not installed on the edge node).
> The non existing path prevent the ImpalaD to start with the message:
> {code:java}
> I0809 04:15:53.899714 25364 status.cc:124] Invalid short-circuit reads
> configuration:
> - Impala cannot read or execute the parent directory of
> dfs.domain.socket.path
> @ 0xb35f19
> @ 0x100e2fe
> @ 0x103f274
> @ 0x102836f
> @ 0xa9f573
> @ 0x7f97807e93d4
> @ 0xafb3b8
> E0809 04:15:53.899749 25364 impala-server.cc:278] Invalid short-circuit reads
> configuration:
> - Impala cannot read or execute the parent directory of
> dfs.domain.socket.path
> {code}
> despite a coordinator-only ImpalaD does not do short circuit reads.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8852) The dfs.domain.socket.path can be nonexistent on coordinator only nodes
[
https://issues.apache.org/jira/browse/IMPALA-8852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adriano updated IMPALA-8852:
Issue Type: Bug (was: Improvement)
> The dfs.domain.socket.path can be nonexistent on coordinator only nodes
> ---
>
> Key: IMPALA-8852
> URL: https://issues.apache.org/jira/browse/IMPALA-8852
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: Adriano
>Priority: Major
>
> On coordinator only nodes ([typically the edge
> nodes|https://www.cloudera.com/documentation/enterprise/5-15-x/topics/impala_dedicated_coordinator.html#concept_omm_gf1_n2b]):
> {code:java}
> --is_coordinator=true
> --is_executor=false
> {code}
> the *dfs.domain.socket.path* (can be nonexistent on the local FS as the
> Datanode role eventually is not installed on the edge node).
> The non existing path prevent the ImpalaD to start with the message:
> {code:java}
> I0809 04:15:53.899714 25364 status.cc:124] Invalid short-circuit reads
> configuration:
> - Impala cannot read or execute the parent directory of
> dfs.domain.socket.path
> @ 0xb35f19
> @ 0x100e2fe
> @ 0x103f274
> @ 0x102836f
> @ 0xa9f573
> @ 0x7f97807e93d4
> @ 0xafb3b8
> E0809 04:15:53.899749 25364 impala-server.cc:278] Invalid short-circuit reads
> configuration:
> - Impala cannot read or execute the parent directory of
> dfs.domain.socket.path
> {code}
> despite a coordinator-only ImpalaD does not do short circuit reads.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Comment Edited] (IMPALA-8852) The dfs.domain.socket.path can be nonexistent on coordinator only nodes
[
https://issues.apache.org/jira/browse/IMPALA-8852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903830#comment-16903830
]
Adriano edited comment on IMPALA-8852 at 8/9/19 1:22 PM:
-
WORKAROUND:
Create the dfs.domain.socket.path manually with the proper hdfs user permission
on the local fs as:
{code:java}
# mkdir /var/run/hdfs-sockets/
# chown hdfs:hadoop /var/run/hdfs-sockets/
# chmod 755 /var/run/hdfs-sockets/
# mkdir /var/run/hdfs-sockets/dn
# chown hdfs:hdfs /var/run/hdfs-sockets/dn
# chmod 1666 /var/run/hdfs-sockets/dn
{code}
was (Author: adrenas):
WORKAROUND:
Create the dfs.domain.socket.path manually with the proper hdfs permission as:
{code:java}
# mkdir /var/run/hdfs-sockets/
# chown hdfs:hadoop /var/run/hdfs-sockets/
# chmod 755 /var/run/hdfs-sockets/
# mkdir /var/run/hdfs-sockets/dn
# chown hdfs:hdfs /var/run/hdfs-sockets/dn
# chmod 1666 /var/run/hdfs-sockets/dn
{code}
> The dfs.domain.socket.path can be nonexistent on coordinator only nodes
> ---
>
> Key: IMPALA-8852
> URL: https://issues.apache.org/jira/browse/IMPALA-8852
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: Adriano
>Priority: Major
>
> On coordinator only nodes ([typically the edge
> nodes|https://www.cloudera.com/documentation/enterprise/5-15-x/topics/impala_dedicated_coordinator.html#concept_omm_gf1_n2b]):
> {code:java}
> --is_coordinator=true
> --is_executor=false
> {code}
> the *dfs.domain.socket.path* (can be nonexistent on the local FS as the
> Datanode role eventually is not installed on the edge node).
> The non existing path prevent the ImpalaD to start with the message:
> {code:java}
> I0809 04:15:53.899714 25364 status.cc:124] Invalid short-circuit reads
> configuration:
> - Impala cannot read or execute the parent directory of
> dfs.domain.socket.path
> @ 0xb35f19
> @ 0x100e2fe
> @ 0x103f274
> @ 0x102836f
> @ 0xa9f573
> @ 0x7f97807e93d4
> @ 0xafb3b8
> E0809 04:15:53.899749 25364 impala-server.cc:278] Invalid short-circuit reads
> configuration:
> - Impala cannot read or execute the parent directory of
> dfs.domain.socket.path
> {code}
> despite a coordinator-only ImpalaD does not do short circuit reads.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8852) The dfs.domain.socket.path can be nonexistent on coordinator only nodes
[
https://issues.apache.org/jira/browse/IMPALA-8852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adriano updated IMPALA-8852:
Issue Type: Improvement (was: New Feature)
> The dfs.domain.socket.path can be nonexistent on coordinator only nodes
> ---
>
> Key: IMPALA-8852
> URL: https://issues.apache.org/jira/browse/IMPALA-8852
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: Adriano
>Priority: Major
>
> On coordinator only nodes ([typically the edge
> nodes|https://www.cloudera.com/documentation/enterprise/5-15-x/topics/impala_dedicated_coordinator.html#concept_omm_gf1_n2b]):
> {code:java}
> --is_coordinator=true
> --is_executor=false
> {code}
> the *dfs.domain.socket.path* (can be nonexistent on the local FS as the
> Datanode role eventually is not installed on the edge node).
> The non existing path prevent the ImpalaD to start with the message:
> {code:java}
> I0809 04:15:53.899714 25364 status.cc:124] Invalid short-circuit reads
> configuration:
> - Impala cannot read or execute the parent directory of
> dfs.domain.socket.path
> @ 0xb35f19
> @ 0x100e2fe
> @ 0x103f274
> @ 0x102836f
> @ 0xa9f573
> @ 0x7f97807e93d4
> @ 0xafb3b8
> E0809 04:15:53.899749 25364 impala-server.cc:278] Invalid short-circuit reads
> configuration:
> - Impala cannot read or execute the parent directory of
> dfs.domain.socket.path
> {code}
> despite a coordinator-only ImpalaD does not do short circuit reads.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8852) The dfs.domain.socket.path can be nonexistent on coordinator only nodes
[
https://issues.apache.org/jira/browse/IMPALA-8852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adriano updated IMPALA-8852:
Description:
On coordinator only nodes ([typically the edge
nodes|https://www.cloudera.com/documentation/enterprise/5-15-x/topics/impala_dedicated_coordinator.html#concept_omm_gf1_n2b]):
{code:java}
--is_coordinator=true
--is_executor=false
{code}
the *dfs.domain.socket.path* (can be nonexistent on the local FS as the
Datanode role eventually is not installed on the edge node).
The non existing path prevent the ImpalaD to start with the message:
{code:java}
I0809 04:15:53.899714 25364 status.cc:124] Invalid short-circuit reads
configuration:
- Impala cannot read or execute the parent directory of dfs.domain.socket.path
@ 0xb35f19
@ 0x100e2fe
@ 0x103f274
@ 0x102836f
@ 0xa9f573
@ 0x7f97807e93d4
@ 0xafb3b8
E0809 04:15:53.899749 25364 impala-server.cc:278] Invalid short-circuit reads
configuration:
- Impala cannot read or execute the parent directory of dfs.domain.socket.path
{code}
despite a coordinator-only ImpalaD does not do short circuit reads.
was:
On coordinator only nodes (typically the edge nodes), the
dfs.domain.socket.path (can be nonexistent on the local FS as the Datanode role
eventually is not installed on the edge node).
The non existing path prevent the ImpalaD to start with the message:
{code:java}
I0809 04:15:53.899714 25364 status.cc:124] Invalid short-circuit reads
configuration:
- Impala cannot read or execute the parent directory of dfs.domain.socket.path
@ 0xb35f19
@ 0x100e2fe
@ 0x103f274
@ 0x102836f
@ 0xa9f573
@ 0x7f97807e93d4
@ 0xafb3b8
E0809 04:15:53.899749 25364 impala-server.cc:278] Invalid short-circuit reads
configuration:
- Impala cannot read or execute the parent directory of dfs.domain.socket.path
{code}
despite a coordinator-only ImpalaD does not do short circuit reads.
> The dfs.domain.socket.path can be nonexistent on coordinator only nodes
> ---
>
> Key: IMPALA-8852
> URL: https://issues.apache.org/jira/browse/IMPALA-8852
> Project: IMPALA
> Issue Type: New Feature
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: Adriano
>Priority: Major
>
> On coordinator only nodes ([typically the edge
> nodes|https://www.cloudera.com/documentation/enterprise/5-15-x/topics/impala_dedicated_coordinator.html#concept_omm_gf1_n2b]):
> {code:java}
> --is_coordinator=true
> --is_executor=false
> {code}
> the *dfs.domain.socket.path* (can be nonexistent on the local FS as the
> Datanode role eventually is not installed on the edge node).
> The non existing path prevent the ImpalaD to start with the message:
> {code:java}
> I0809 04:15:53.899714 25364 status.cc:124] Invalid short-circuit reads
> configuration:
> - Impala cannot read or execute the parent directory of
> dfs.domain.socket.path
> @ 0xb35f19
> @ 0x100e2fe
> @ 0x103f274
> @ 0x102836f
> @ 0xa9f573
> @ 0x7f97807e93d4
> @ 0xafb3b8
> E0809 04:15:53.899749 25364 impala-server.cc:278] Invalid short-circuit reads
> configuration:
> - Impala cannot read or execute the parent directory of
> dfs.domain.socket.path
> {code}
> despite a coordinator-only ImpalaD does not do short circuit reads.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8852) The dfs.domain.socket.path can be nonexistent on coordinator only nodes
[
https://issues.apache.org/jira/browse/IMPALA-8852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903830#comment-16903830
]
Adriano commented on IMPALA-8852:
-
WORKAROUND:
Create the dfs.domain.socket.path manually with the proper hdfs permission as:
{code:java}
# mkdir /var/run/hdfs-sockets/
# chown hdfs:hadoop /var/run/hdfs-sockets/
# chmod 755 /var/run/hdfs-sockets/
# mkdir /var/run/hdfs-sockets/dn
# chown hdfs:hdfs /var/run/hdfs-sockets/dn
# chmod 1666 /var/run/hdfs-sockets/dn
{code}
> The dfs.domain.socket.path can be nonexistent on coordinator only nodes
> ---
>
> Key: IMPALA-8852
> URL: https://issues.apache.org/jira/browse/IMPALA-8852
> Project: IMPALA
> Issue Type: New Feature
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: Adriano
>Priority: Major
>
> On coordinator only nodes (typically the edge nodes), the
> dfs.domain.socket.path (can be nonexistent on the local FS as the Datanode
> role eventually is not installed on the edge node).
> The non existing path prevent the ImpalaD to start with the message:
> {code:java}
> I0809 04:15:53.899714 25364 status.cc:124] Invalid short-circuit reads
> configuration:
> - Impala cannot read or execute the parent directory of
> dfs.domain.socket.path
> @ 0xb35f19
> @ 0x100e2fe
> @ 0x103f274
> @ 0x102836f
> @ 0xa9f573
> @ 0x7f97807e93d4
> @ 0xafb3b8
> E0809 04:15:53.899749 25364 impala-server.cc:278] Invalid short-circuit reads
> configuration:
> - Impala cannot read or execute the parent directory of
> dfs.domain.socket.path
> {code}
> despite a coordinator-only ImpalaD does not do short circuit reads.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8852) The dfs.domain.socket.path can be nonexistent on coordinator only nodes
[
https://issues.apache.org/jira/browse/IMPALA-8852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adriano updated IMPALA-8852:
Description:
On coordinator only nodes (typically the edge nodes), the
dfs.domain.socket.path (can be nonexistent on the local FS as the Datanode role
eventually is not installed on the edge node).
The non existing path prevent the ImpalaD to start with the message:
{code:java}
I0809 04:15:53.899714 25364 status.cc:124] Invalid short-circuit reads
configuration:
- Impala cannot read or execute the parent directory of dfs.domain.socket.path
@ 0xb35f19
@ 0x100e2fe
@ 0x103f274
@ 0x102836f
@ 0xa9f573
@ 0x7f97807e93d4
@ 0xafb3b8
E0809 04:15:53.899749 25364 impala-server.cc:278] Invalid short-circuit reads
configuration:
- Impala cannot read or execute the parent directory of dfs.domain.socket.path
{code}
despite a coordinator-only ImpalaD does not do short circuit reads.
was:
On coordinator only nodes (typically the edge nodes), the
dfs.domain.socket.path (can be nonexistent on the local FS as the Datanode role
eventually is not installed on the edge node).
The non existing path prevent the ImpalaD to start with the message:
{{ I0809 04:15:53.899714 25364 status.cc:124] Invalid short-circuit reads
configuration:}}
- {{Impala cannot read or execute the parent directory of
dfs.domain.socket.path}}
{{@ 0xb35f19}}
{{ @ 0x100e2fe}}
{{ @ 0x103f274}}
{{ @ 0x102836f}}
{{ @ 0xa9f573}}
{{ @ 0x7f97807e93d4}}
{{ @ 0xafb3b8}}
{{ E0809 04:15:53.899749 25364 impala-server.cc:278] Invalid short-circuit
reads configuration:}}
- {{Impala cannot read or execute the parent directory of
dfs.domain.socket.path}}
{{E0809 04:15:53.899786 25364 impala-server.cc:281] Aborting Impala Server
startup due to improper configuration. Impalad exiting.}}
despite a coordinator-only ImpalaD does not do short circuit reads.
> The dfs.domain.socket.path can be nonexistent on coordinator only nodes
> ---
>
> Key: IMPALA-8852
> URL: https://issues.apache.org/jira/browse/IMPALA-8852
> Project: IMPALA
> Issue Type: New Feature
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: Adriano
>Priority: Major
>
> On coordinator only nodes (typically the edge nodes), the
> dfs.domain.socket.path (can be nonexistent on the local FS as the Datanode
> role eventually is not installed on the edge node).
> The non existing path prevent the ImpalaD to start with the message:
> {code:java}
> I0809 04:15:53.899714 25364 status.cc:124] Invalid short-circuit reads
> configuration:
> - Impala cannot read or execute the parent directory of
> dfs.domain.socket.path
> @ 0xb35f19
> @ 0x100e2fe
> @ 0x103f274
> @ 0x102836f
> @ 0xa9f573
> @ 0x7f97807e93d4
> @ 0xafb3b8
> E0809 04:15:53.899749 25364 impala-server.cc:278] Invalid short-circuit reads
> configuration:
> - Impala cannot read or execute the parent directory of
> dfs.domain.socket.path
> {code}
> despite a coordinator-only ImpalaD does not do short circuit reads.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8852) The dfs.domain.socket.path can be nonexistent on coordinator only nodes
[
https://issues.apache.org/jira/browse/IMPALA-8852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adriano updated IMPALA-8852:
Description:
On coordinator only nodes (typically the edge nodes), the
dfs.domain.socket.path (can be nonexistent on the local FS as the Datanode role
eventually is not installed on the edge node).
The non existing path prevent the ImpalaD to start with the message:
{{ I0809 04:15:53.899714 25364 status.cc:124] Invalid short-circuit reads
configuration:}}
- {{Impala cannot read or execute the parent directory of
dfs.domain.socket.path}}
{{@ 0xb35f19}}
{{ @ 0x100e2fe}}
{{ @ 0x103f274}}
{{ @ 0x102836f}}
{{ @ 0xa9f573}}
{{ @ 0x7f97807e93d4}}
{{ @ 0xafb3b8}}
{{ E0809 04:15:53.899749 25364 impala-server.cc:278] Invalid short-circuit
reads configuration:}}
- {{Impala cannot read or execute the parent directory of
dfs.domain.socket.path}}
{{E0809 04:15:53.899786 25364 impala-server.cc:281] Aborting Impala Server
startup due to improper configuration. Impalad exiting.}}
despite a coordinator-only ImpalaD does not do short circuit reads.
was:
On coordinator only nodes (typically the edge nodes), the
dfs.domain.socket.path (can be nonexistent on the local FS as the Datanode role
eventually is not installed on the edge node).
The non existing path prevent the ImpalaD to start with the message:
{{
I0809 04:15:53.899714 25364 status.cc:124] Invalid short-circuit reads
configuration:
- Impala cannot read or execute the parent directory of dfs.domain.socket.path
@ 0xb35f19
@ 0x100e2fe
@ 0x103f274
@ 0x102836f
@ 0xa9f573
@ 0x7f97807e93d4
@ 0xafb3b8
E0809 04:15:53.899749 25364 impala-server.cc:278] Invalid short-circuit reads
configuration:
- Impala cannot read or execute the parent directory of dfs.domain.socket.path
E0809 04:15:53.899786 25364 impala-server.cc:281] Aborting Impala Server
startup due to improper configuration. Impalad exiting.}}
despite a coordinator only ImpalaD does not do short circuit reads.
> The dfs.domain.socket.path can be nonexistent on coordinator only nodes
> ---
>
> Key: IMPALA-8852
> URL: https://issues.apache.org/jira/browse/IMPALA-8852
> Project: IMPALA
> Issue Type: New Feature
> Components: Backend
>Affects Versions: Impala 3.2.0
>Reporter: Adriano
>Priority: Major
>
> On coordinator only nodes (typically the edge nodes), the
> dfs.domain.socket.path (can be nonexistent on the local FS as the Datanode
> role eventually is not installed on the edge node).
> The non existing path prevent the ImpalaD to start with the message:
> {{ I0809 04:15:53.899714 25364 status.cc:124] Invalid short-circuit reads
> configuration:}}
> - {{Impala cannot read or execute the parent directory of
> dfs.domain.socket.path}}
> {{@ 0xb35f19}}
> {{ @ 0x100e2fe}}
> {{ @ 0x103f274}}
> {{ @ 0x102836f}}
> {{ @ 0xa9f573}}
> {{ @ 0x7f97807e93d4}}
> {{ @ 0xafb3b8}}
> {{ E0809 04:15:53.899749 25364 impala-server.cc:278] Invalid short-circuit
> reads configuration:}}
> - {{Impala cannot read or execute the parent directory of
> dfs.domain.socket.path}}
> {{E0809 04:15:53.899786 25364 impala-server.cc:281] Aborting Impala Server
> startup due to improper configuration. Impalad exiting.}}
> despite a coordinator-only ImpalaD does not do short circuit reads.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8852) The dfs.domain.socket.path can be nonexistent on coordinator only nodes
Adriano created IMPALA-8852:
---
Summary: The dfs.domain.socket.path can be nonexistent on
coordinator only nodes
Key: IMPALA-8852
URL: https://issues.apache.org/jira/browse/IMPALA-8852
Project: IMPALA
Issue Type: New Feature
Components: Backend
Affects Versions: Impala 3.2.0
Reporter: Adriano
On coordinator only nodes (typically the edge nodes), the
dfs.domain.socket.path (can be nonexistent on the local FS as the Datanode role
eventually is not installed on the edge node).
The non existing path prevent the ImpalaD to start with the message:
{{
I0809 04:15:53.899714 25364 status.cc:124] Invalid short-circuit reads
configuration:
- Impala cannot read or execute the parent directory of dfs.domain.socket.path
@ 0xb35f19
@ 0x100e2fe
@ 0x103f274
@ 0x102836f
@ 0xa9f573
@ 0x7f97807e93d4
@ 0xafb3b8
E0809 04:15:53.899749 25364 impala-server.cc:278] Invalid short-circuit reads
configuration:
- Impala cannot read or execute the parent directory of dfs.domain.socket.path
E0809 04:15:53.899786 25364 impala-server.cc:281] Aborting Impala Server
startup due to improper configuration. Impalad exiting.}}
despite a coordinator only ImpalaD does not do short circuit reads.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8403) Possible thread leak in impalad
[
https://issues.apache.org/jira/browse/IMPALA-8403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16898711#comment-16898711
]
Adriano commented on IMPALA-8403:
-
Thank [~kwho],
yes, it will be a good improvement after IMPALA-7984 will be done.
> Possible thread leak in impalad
> ---
>
> Key: IMPALA-8403
> URL: https://issues.apache.org/jira/browse/IMPALA-8403
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 2.12.0
>Reporter: Quanlong Huang
>Priority: Major
> Attachments: image-2019-04-10-11-15-11-321.png, reproIMPALA-8403.tgz
>
>
> The metric of thread-manager.running-threads got from
> http://${impalad_host}:25000/metrics?json shows that the number of running
> threads keeps increasing. (See the snapshot) This phenomenon is most
> noticeable in coordinators.
> Maybe a counter bug or threads leak.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-8679) Make the query options set in the dynamic resource pool/admission control un-overridable in the user session
[ https://issues.apache.org/jira/browse/IMPALA-8679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adriano resolved IMPALA-8679. - Resolution: Fixed Fix Version/s: Impala 3.1.0 IMPALA-7349 is applicable here (specifically for the memory settings). With this feature the users cannot override the MEM_LIMIT set in the pool configuration causing potential query failures for memory limit exceeded). [1] https://www.cloudera.com/documentation/enterprise/6/6.1/topics/impala_admission.html#admission_memory > Make the query options set in the dynamic resource pool/admission control > un-overridable in the user session > > > Key: IMPALA-8679 > URL: https://issues.apache.org/jira/browse/IMPALA-8679 > Project: IMPALA > Issue Type: New Feature > Components: Frontend >Reporter: Adriano >Priority: Major > Fix For: Impala 3.1.0 > > > Issue description: Once the admission control is configured (with the > MAX_MEM_ESTIMATE_FOR_ADMISSION, MEM_LIMIT, etc,etc query options), if a user > bypass the default setting setting the query options in the session it can > cause queries failures in the pools configured (e.g decreasing > MAX_MEM_ESTIMATE_FOR_ADMISSION and increasing MEM_LIMIT). > Improvement: It will be great to have a further checkboxes (with those > default values) like "do not allow user to override this value". > The value can be changed eventually by the query optimizer, but we do not > allow users to change MAX_MEM_ESTIMATE_FOR_ADMISSION at all. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8679) Make the query options set in the dynamic resource pool/admission control un-overridable in the user session
Adriano created IMPALA-8679: --- Summary: Make the query options set in the dynamic resource pool/admission control un-overridable in the user session Key: IMPALA-8679 URL: https://issues.apache.org/jira/browse/IMPALA-8679 Project: IMPALA Issue Type: New Feature Components: Frontend Reporter: Adriano Issue description: Once the admission control is configured (with the MAX_MEM_ESTIMATE_FOR_ADMISSION, MEM_LIMIT, etc,etc query options), if a user bypass the default setting setting the query options in the session it can cause queries failures in the pools configured (e.g decreasing MAX_MEM_ESTIMATE_FOR_ADMISSION and increasing MEM_LIMIT). Improvement: It will be great to have a further checkboxes (with those default values) like "do not allow user to override this value". The value can be changed eventually by the query optimizer, but we do not allow users to change MAX_MEM_ESTIMATE_FOR_ADMISSION at all. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8403) Possible thread leak in impalad
[
https://issues.apache.org/jira/browse/IMPALA-8403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832467#comment-16832467
]
Adriano commented on IMPALA-8403:
-
Hi, I would like to add a repro steps [^reproIMPALA-8403.tgz] that i followed
to increase the same of threads (submitting a query with many fragments and
then cancelling it once the fragments were in execution). Maybe is a dup,
however if it is not, I appreciate your help in order to put this jira into the
backlog.
Many thanks,
Adriano
> Possible thread leak in impalad
> ---
>
> Key: IMPALA-8403
> URL: https://issues.apache.org/jira/browse/IMPALA-8403
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 2.12.0
>Reporter: Quanlong Huang
>Priority: Major
> Attachments: image-2019-04-10-11-15-11-321.png, reproIMPALA-8403.tgz
>
>
> The metric of thread-manager.running-threads got from
> http://${impalad_host}:25000/metrics?json shows that the number of running
> threads keeps increasing. (See the snapshot) This phenomenon is most
> noticeable in coordinators.
> Maybe a counter bug or threads leak.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8403) Possible thread leak in impalad
[
https://issues.apache.org/jira/browse/IMPALA-8403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adriano updated IMPALA-8403:
Attachment: reproIMPALA-8403.tgz
> Possible thread leak in impalad
> ---
>
> Key: IMPALA-8403
> URL: https://issues.apache.org/jira/browse/IMPALA-8403
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 2.12.0
>Reporter: Quanlong Huang
>Priority: Major
> Attachments: image-2019-04-10-11-15-11-321.png, reproIMPALA-8403.tgz
>
>
> The metric of thread-manager.running-threads got from
> http://${impalad_host}:25000/metrics?json shows that the number of running
> threads keeps increasing. (See the snapshot) This phenomenon is most
> noticeable in coordinators.
> Maybe a counter bug or threads leak.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8087) Evaluate to adapt the serde property values length as per the latest changes on HMS
[
https://issues.apache.org/jira/browse/IMPALA-8087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16747796#comment-16747796
]
Adriano commented on IMPALA-8087:
-
[~bharathv] thanks to have a look on it.
Just a question regarding:
bq. Existing incremental stats data should be read fine and rewritten with new
chunk size.
bq.
Do you mean that increasing this parameter will increase the size of the new
incremental stats?
Or it continue to be 400 bytes of metadata per column per partition for caching?
> Evaluate to adapt the serde property values length as per the latest changes
> on HMS
>
>
> Key: IMPALA-8087
> URL: https://issues.apache.org/jira/browse/IMPALA-8087
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
>Affects Versions: Impala 2.7.0
>Reporter: Adriano
>Priority: Major
> Labels: ramp-up
>
> Hi,
> with the HIVE-12274 (C5.13) seems that the SERDE_PARAMS.PARAM_VALUE max
> length is increased in all the supported RDBMS to a bigger value while on our
> side we kept to 4k (depending by the RDBMS the datatype used increased the
> length between 32k and 64k).
> I saw that those lines are not changed in the latest releases:
> https://github.com/cloudera/Impala/blob/cdh5-2.7.0_5.10.2/fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java#L50-L51
>
> https://github.com/cloudera/Impala/blob/cdh5-2.7.0_5.10.2/fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java#L145-L155
> Not sure if can be done without breaking anything, however could you evaluate
> if it is possible to safely increase it to a bigger value? If that case could
> you get this Jira in your backlog?
> It will allow as example the creation of avro table with avro.schema.literal
> > 4k as in Hive instead to exit with:
> {code:java}
> AnalysisException: Property value length must be <= 4000:
> {code}
> Thank you,
> Adriano
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8087) Evaluate to adapt the serde property values length as per the latest changes on HMS
Adriano created IMPALA-8087:
---
Summary: Evaluate to adapt the serde property values length as per
the latest changes on HMS
Key: IMPALA-8087
URL: https://issues.apache.org/jira/browse/IMPALA-8087
Project: IMPALA
Issue Type: Improvement
Components: Frontend
Affects Versions: Impala 2.7.0
Reporter: Adriano
Hi,
with the HIVE-12274 (C5.13) seems that the SERDE_PARAMS.PARAM_VALUE max length
is increased in all the supported RDBMS to a bigger value while on our side we
kept to 4k (depending by the RDBMS the datatype used increased the length
between 32k and 64k).
I saw that those lines are not changed in the latest releases:
https://github.com/cloudera/Impala/blob/cdh5-2.7.0_5.10.2/fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java#L50-L51
https://github.com/cloudera/Impala/blob/cdh5-2.7.0_5.10.2/fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java#L145-L155
Not sure if can be done without breaking anything, however could you evaluate
if it is possible to safely increase it to a bigger value? If that case could
you get this Jira in your backlog?
It will allow as example the creation of avro table with avro.schema.literal >
4k as in Hive instead to exit with:
{code:java}
AnalysisException: Property value length must be <= 4000:
{code}
Thank you,
Adriano
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7496) Schedule query taking in account the mem available on the impalad nodes
[ https://issues.apache.org/jira/browse/IMPALA-7496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596341#comment-16596341 ] Adriano commented on IMPALA-7496: - The Impala version used during the issue haven't -IMPALA-1575- The specific situation appeared once many users connected directly from their workstation to the specific impalad (this caused more use of memory, as you guessed). The query options beside the lack of guarantee are very appreciated. Thank to include the jira in the backlog. > Schedule query taking in account the mem available on the impalad nodes > --- > > Key: IMPALA-7496 > URL: https://issues.apache.org/jira/browse/IMPALA-7496 > Project: IMPALA > Issue Type: New Feature > Components: Backend >Reporter: Adriano >Priority: Major > Labels: admission-control, scheduler > > Environment description: cluster scale (50/100/150 nodes and terabyte of ram > available) - Admission Control enabled. > Issue description: > Despite the coordinator chosen (with data and statistics unchanged) a query > will be planned always in the same way based on the metainfo that the > coordinator have. > The query will be scheduled always on the same nodes if the memory > requirements for the admission are satisfied: > https://github.com/cloudera/Impala/blob/cdh5-2.7.0_5.9.1/be/src/scheduling/admission-controller.cc#L307-L333 > Equal queries are planned/scheduled always in the same way (to hit always the > same nodes). > This often lead to queue the queries that are hitting the same nodes are > queued (not admitted) as on those nodes there's no more memory available > within its process limit despite the pool have lot of free memory and the > overall cluster load is low. > When the plan is finished and the query can be evaluated to be admitted often > happen that the admission is denied because one of the node have not enough > memory to run the query operation (and the query is moved in the pool queue) > despite the cluster have 50/100/150 nodes and terabyte of ram available. > Why the scheduler does not take in consideration the memory available on the > nodes involved in the query before to buid the schedule, (maybe preferring a > remote read/operation on a free memory node instead to include in the plan > always the same nodes that will end to be: > 1- overloaded > 2- the query will be not immediately admitted, risking to be timedout in the > pool queue > Since 2.7 the REPLICA_PREFERENCE can possibly help, but it's not good enough > as it does not prevent the scheduler to choose busy nodes (with the same > potential effect: query queued for lack of resource on specific node despite > there are terabytes of free memory). > Feature Request: > It would be good if Impala had an option to execute queries (even with worse > performance) excluding the nodes overloaded and including different nodes in > order to get the query immediately admitted and executed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-7496) Schedule query taking in account the mem available on the impalad nodes
Adriano created IMPALA-7496: --- Summary: Schedule query taking in account the mem available on the impalad nodes Key: IMPALA-7496 URL: https://issues.apache.org/jira/browse/IMPALA-7496 Project: IMPALA Issue Type: New Feature Components: Backend Reporter: Adriano Environment description: cluster scale (50/100/150 nodes and terabyte of ram available) - Admission Control enabled. Issue description: Despite the coordinator chosen (with data and statistics unchanged) a query will be planned always in the same way based on the metainfo that the coordinator have. The query will be scheduled always on the same nodes if the memory requirements for the admission are satisfied: https://github.com/cloudera/Impala/blob/cdh5-2.7.0_5.9.1/be/src/scheduling/admission-controller.cc#L307-L333 Equal queries are planned/scheduled always in the same way (to hit always the same nodes). This often lead to queue the queries that are hitting the same nodes are queued (not admitted) as on those nodes there's no more memory available within its process limit despite the pool have lot of free memory and the overall cluster load is low. When the plan is finished and the query can be evaluated to be admitted often happen that the admission is denied because one of the node have not enough memory to run the query operation (and the query is moved in the pool queue) despite the cluster have 50/100/150 nodes and terabyte of ram available. Why the scheduler does not take in consideration the memory available on the nodes involved in the query before to buid the schedule, (maybe preferring a remote read/operation on a free memory node instead to include in the plan always the same nodes that will end to be: 1- overloaded 2- the query will be not immediately admitted, risking to be timedout in the pool queue Since 2.7 the REPLICA_PREFERENCE can possibly help, but it's not good enough as it does not prevent the scheduler to choose busy nodes (with the same potential effect: query queued for lack of resource on specific node despite there are terabytes of free memory). Feature Request: It would be good if Impala had an option to execute queries (even with worse performance) excluding the nodes overloaded and including different nodes in order to get the query immediately admitted and executed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7031) Detailed failure reason for clients when a query is cancelled from the WebUi
[
https://issues.apache.org/jira/browse/IMPALA-7031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550510#comment-16550510
]
Adriano commented on IMPALA-7031:
-
Hi [~tarmstrong], sorry to answer with delay.
I made some tests on Impala 2.10 (C5.13, but most probably this behaviour is
only in the next versions):
* from the Impala WebUi we can only cancel the query in flight, in this case:
!Screen Shot 2018-07-20 at 10.19.42.png!
* the coordinator will cancel the query as requested:
{code:java}
I0720 01:03:52.288815 28385 status.cc:122] Cancelled from Impala's debug web
interface
@ 0x83e789 impala::Status::Status()
@ 0xabce21 impala::ImpalaHttpHandler::CancelQueryHandler()
@ 0xbdb7da impala::Webserver::RenderUrlWithTemplate()
@ 0xbdcd27 impala::Webserver::BeginRequestCallback()
@ 0xbe9a70 (unknown)
@ 0xbec1ed (unknown)
@ 0xbec87d (unknown)
@ 0x7f301df39aa1 start_thread
@ 0x7f301dc86c4d clone
I0720 01:03:52.288822 28385 impala-server.cc:989] UnregisterQuery():
query_id=944b1add28b19313:eb712db1
I0720 01:03:52.288844 28385 impala-server.cc:1075] Cancel():
query_id=944b1add28b19313:eb712db1
I0720 01:03:52.288862 28385 coordinator.cc:910] Cancel()
query_id=944b1add28b19313:eb712db1
I0720 01:03:52.288868 28385 coordinator-backend-state.cc:367] sending
CancelQueryFInstances rpc for query_id=944b1add28b19313:eb712db1
backend=host-10-17-102-61.coe.cloudera.com:22000
I0720 01:03:52.288873 28385 client-cache.cc:46]
GetClient(host-10-17-102-61.coe.cloudera.com:22000)
I0720 01:03:52.288877 28385 client-cache.cc:56] GetClient(): returning cached
client for host-10-17-102-61.coe.cloudera.com:22000
I0720 01:03:52.288935 28314 rpc-trace.cc:190] RPC call:
ImpalaInternalService.CancelQueryFInstances(from 10.17.102.61:57249)
I0720 01:03:52.288945 28314 impala-internal-service.cc:63]
CancelQueryFInstances(): query_id=944b1add28b19313:eb712db1
{code}
* On the client side the query fail with the error below:
{code:java}
java.sql.SQLException: [Simba][ImpalaJDBCDriver](500051) ERROR processing
query/statement. Error Code: 0, SQL state: TStatus(statusCode:ERROR_STATUS,
sqlState:HY000, errorMessage:Invalid query handle:
944b1add28b19313:eb712db1), Query: SELECT COUNT(*) as `expr_0` FROM
`db`.`table`.
at com.cloudera.hivecommon.api.HS2Client.executeStatementInternal(Unknown
Source)
at com.cloudera.hivecommon.api.HS2Client.executeStatement(Unknown Source)
at
com.cloudera.hivecommon.dataengine.HiveJDBCNativeQueryExecutor.executeQuery(Unknown
Source)
at
com.cloudera.hivecommon.dataengine.HiveJDBCDSIExtQueryExecutor.execute(Unknown
Source)
at com.cloudera.jdbc.common.SStatement.executeNoParams(Unknown Source)
at com.cloudera.jdbc.common.SStatement.executeQuery(Unknown Source)
at JdbcTest.runTest(JdbcTest.java:43)
Caused by: com.cloudera.support.exceptions.GeneralException:
[Simba][ImpalaJDBCDriver](500051) ERROR processing query/statement. Error Code:
0, SQL state: TStatus(statusCode:ERROR_STATUS, sqlState:HY000,
errorMessage:Invalid query handle: 944b1add28b19313:eb712db1), Query:
SELECT COUNT(*) as `expr_0` FROM `db`.`table`.
... 7 more{code}
* Because the coordinator provide this answer if the query is closed/cancelled
{code:java}
I0720 01:03:52.301568 28384 status.cc:55] ReportExecStatus(): Received report
for unknown query ID (probably closed or cancelled):
944b1add28b19313:eb712db1
@ 0x83d66a impala::Status::Status()
@ 0xa90176 impala::ImpalaServer::ReportExecStatus()
@ 0xd4b60e
impala::ImpalaInternalServiceProcessor::process_ReportExecStatus()
@ 0xd47859 impala::ImpalaInternalServiceProcessor::dispatchCall()
@ 0x80e4ac apache::thrift::TDispatchProcessor::process()
@ 0x9d9edf
apache::thrift::server::TAcceptQueueServer::Task::run()
@ 0x9d44f9 impala::ThriftThread::RunRunnable()
@ 0x9d52d2
boost::detail::function::void_function_obj_invoker0<>::invoke()
@ 0xbd3fa2 impala::Thread::SuperviseThread()
@ 0xbd4704 boost::detail::thread_data<>::run()
@ 0xe608ea (unknown)
@ 0x7f301df39aa1 start_thread
@ 0x7f301dc86c4d clone
{code}
>From the client/user prospective the error "Invalid query handle" is not
>indicative and they think that something is broken (ignoring that someone else
>cancelled the query using the Administrative WebUi). If in the future we can
>provide some more info on the reason why the query failed will be good.
PS: I was able to replicate also the "waiting to be closed" state but there we
have no errors on the client side (the effect is that the query will stay in
waiting to be closed on the
[jira] [Updated] (IMPALA-7031) Detailed failure reason for clients when a query is cancelled from the WebUi
[ https://issues.apache.org/jira/browse/IMPALA-7031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adriano updated IMPALA-7031: Attachment: Screen Shot 2018-07-20 at 10.19.42.png > Detailed failure reason for clients when a query is cancelled from the WebUi > > > Key: IMPALA-7031 > URL: https://issues.apache.org/jira/browse/IMPALA-7031 > Project: IMPALA > Issue Type: New Feature >Reporter: Adriano >Priority: Major > Attachments: Screen Shot 2018-07-20 at 10.19.42.png > > > In big clusters with many jdbc/odbc users, in order to save resources are > often implemented scripts that automatically cancel queries (e.g. long > running queries) (the scripts typically are using the Impala Webui). > Typical Scenario: > # A jdbc/odbc client submit a query > # The Coordinator start the query execution > # The query is cancelled from the Coordinator WebUi > # The jdbc/odbc client ask to the Coordinator the query status > (GetOperationStatus) > # The Coordinator answer "unknown query ID" (as the query was cancelled) > # For the client perspective the query failed for "unknown query ID" > Currently, if a running query is cancelled from the impalad WebUI, the client > will just receive an 'unknown query ID' error on the next > fetch/getOperationStatus attempt. It would be good to be able to explicitly > call out this case. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
