[jira] [Created] (IMPALA-12985) Use the new constructor when instantiating RangerAccessRequestImpl
Fang-Yu Rao created IMPALA-12985:
Summary: Use the new constructor when instantiating
RangerAccessRequestImpl
Key: IMPALA-12985
URL: https://issues.apache.org/jira/browse/IMPALA-12985
Project: IMPALA
Issue Type: Task
Components: Frontend
Reporter: Fang-Yu Rao
Assignee: Fang-Yu Rao
After RANGER-2763, we changed the signature of the class
RangerAccessRequestImpl in by adding an additional input argument 'userRoles'
as shown in the following.
{code:java}
public RangerAccessRequestImpl(RangerAccessResource resource, String
accessType, String user, Set userGroups, Set userRoles) {
...
{code}
The new signature is also provided in CDP Ranger. Thus to unblock IMPALA-12921
or to be able to build Apache Impala with Apache Ranger, it may be faster to
switch to the new signature on the Impala side.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (IMPALA-12984) Show inactivity of data exchanges in query profile
Riza Suminto created IMPALA-12984: - Summary: Show inactivity of data exchanges in query profile Key: IMPALA-12984 URL: https://issues.apache.org/jira/browse/IMPALA-12984 Project: IMPALA Issue Type: Improvement Components: Distributed Exec Reporter: Riza Suminto Many-to-many data exchanges can be bottlenecked by hotspot receiver such scenario described in IMPALA-6692 or when data spilling happens in subset of backend. Ideally, this occurrences should be easily figured out in query profile. But triaging this kind of issue often requires correlation analysis of several counters in query profile. There are few ideas on how to improve this identification: # Upon query completion, let coordinator do some profile analysis and print WARNING in query profile pointing at the skew. One group of EXCHANGE senders and receivers can only complete simultaneously since all receivers need to wait for EOS signal from all senders. Let say we take max of TotalNetworkSendTime from all senders and max of DataWaitTime from all receivers, a "mutual wait" time of min(TotalNetworkSendTime,DataWaitTime) can be used as indicator of how long the exchanges are waiting for query operators above them to progress. # Add "Max Inactive" column in ExecSummary table. Existing "Avg Time" and "Max Time" are derived from RuntimeProfileBase::local_time_ns_. If ExecSummary also display maximum value of RuntimeProfileBase::inactive_timer_ of each query operator as "Max Inactive", we can then compare it against "Max Time" and figure out which exchange is mostly idle waiting. The calculation between local_time_ns, children_total_time, and inactive_timer can be seen here at [https://github.com/apache/impala/blob/0721858/be/src/util/runtime-profile.cc#L935-L938] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IMPALA-12983) Push down 'order by' to remote database server
Wenzhe Zhou created IMPALA-12983: Summary: Push down 'order by' to remote database server Key: IMPALA-12983 URL: https://issues.apache.org/jira/browse/IMPALA-12983 Project: IMPALA Issue Type: Sub-task Components: Frontend Reporter: Wenzhe Zhou "order by" cannot be pushed down to JDBC right now, but most of tpcds/tpch queries are using "order by ... limit ...", e.g top n. This results JDBC handler to retrieve all rows of remote table on remote database server, hence bad performance. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IMPALA-12982) COMMIT_TXN event should skipped by EP if there is concurrent DROP DB operation
Sai Hemanth Gantasala created IMPALA-12982:
--
Summary: COMMIT_TXN event should skipped by EP if there is
concurrent DROP DB operation
Key: IMPALA-12982
URL: https://issues.apache.org/jira/browse/IMPALA-12982
Project: IMPALA
Issue Type: Bug
Components: Catalog
Reporter: Sai Hemanth Gantasala
Assignee: Sai Hemanth Gantasala
The error happens in loading a partition while a concurrent DROP DATABASE
CASCADE is running. Impalad logs:
{noformat}
I0217 06:55:58.696295 28500 Frontend.java:2070]
8b4a15df68c9c5dd:f21c5bb2] Analyzing query: DROP DATABASE
`test_read_hive_inserts_8bb30aca` CASCADE db: default{noformat}
catalogd logs:
{noformat}
I0217 06:55:58.696962 28650 JniUtil.java:166]
8b4a15df68c9c5dd:f21c5bb2] execDdl request: DROP_DATABASE
test_read_hive_inserts_8bb30aca issued by jenkins
I0217 06:55:58.751202 5748 MetastoreEventsProcessor.java:909] Received 2
events. Start event id : 51397
I0217 06:55:58.751662 5748 MetastoreEvents.java:293] Total number of events
received: 2 Total number of events filtered out: 0
I0217 06:55:58.751700 5748 MetastoreEvents.java:297] Incremented skipped
metric to 8174
I0217 06:55:58.751801 5748 CatalogServiceCatalog.java:1060] Not a self-event
since the given version is -1 and service id is empty
I0217 06:55:58.751914 5748 HdfsTable.java:2851] Reloading partition metadata
for table: test_read_hive_inserts_8bb30aca.test_read_hive_inserts
(ALTER_PARTITION EVENT FOR TRANSACTIONAL TABLE)
I0217 06:55:58.752924 5749 MetastoreEventsProcessor.java:1023] Latest event in
HMS: id=51399, time=1708181758. Last synced event: id=51397, time=1708181750.
W0217 06:55:58.752954 5749 MetastoreEventsProcessor.java:1026] Lag: 8s. 2
events pending to be processed.
I0217 06:55:58.754891 5748 HdfsTable.java:2943] Setting the latest refresh
event id to 51399 for the reloaded 1 partitions
I0217 06:55:58.756098 5748 HdfsTable.java:2851] Reloading partition metadata
for table: test_read_hive_inserts_8bb30aca.test_read_hive_inserts (Processing
event id: 51399, event type: COMMIT_TXN)
I0217 06:55:58.756839 5748 HdfsTable.java:2943] Setting the latest refresh
event id to 51401 for the reloaded 1 partitions
I0217 06:55:58.756867 5748 HdfsTable.java:2946] for table
test_read_hive_inserts_8bb30aca.test_read_hive_inserts, file metadataOps:
FORCE_LOAD, refreshing file metadata for 1 out of 1 partitions to reload in
reloadPartitions()
E0217 06:55:58.761873 5748 ParallelFileMetadataLoader.java:194] Refreshing
file and block metadata for 1 paths for table
test_read_hive_inserts_8bb30aca.test_read_hive_inserts encountered an error
loading data for path
hdfs://localhost:20500/test-warehouse/managed/test_read_hive_inserts_8bb30aca.db/test_read_hive_inserts/p=1
Java exception follows:
java.util.concurrent.ExecutionException: java.io.FileNotFoundException: File
does not exist:
/test-warehouse/managed/test_read_hive_inserts_8bb30aca.db/test_read_hive_inserts/p=1/delta_010_010_/00_0
...
I0217 06:55:58.762064 5748 CatalogOpExecutor.java:5006] Rolling back the write
id list of table test_read_hive_inserts_8bb30aca.test_read_hive_inserts because
reloading for event 51399 is failed: Refreshing file and block metadata for 1
paths for table test_read_hive_inserts_8bb30aca.test_read_hive_inserts: failed
to load 1 paths. Check the catalog server log for more details.
I0217 06:55:58.762253 5748 MetastoreEventsProcessor.java:1189] Time elapsed in
processing event batch: 10.977ms
E0217 06:55:58.762821 5748 MetastoreEventsProcessor.java:981] Event processing
needs a invalidate command to resolve the state
Java exception follows:
org.apache.impala.catalog.events.MetastoreNotificationNeedsInvalidateException:
EventId: 51399 EventType: COMMIT_TXN Failed to mark committed write ids and
refresh partitions for txn {}. Event processing cannot continue. Issue an
invalidate metadata command to reset event processor.
at
org.apache.impala.compat.MetastoreShim$CommitTxnEvent.process(MetastoreShim.java:870)
at
org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.processIfEnabled(MetastoreEvents.java:672)
at
org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:1164)
at
org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:972)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExec
[jira] [Created] (IMPALA-12981) Support a column list in compute stats that is retrived via a subquery
Manish Maheshwari created IMPALA-12981:
--
Summary: Support a column list in compute stats that is retrived
via a subquery
Key: IMPALA-12981
URL: https://issues.apache.org/jira/browse/IMPALA-12981
Project: IMPALA
Issue Type: Improvement
Components: Backend, Frontend
Reporter: Manish Maheshwari
Support a column list in compute stats that is retrived via a subquery -
Specifically we want to use Impala query history tables where we collect the
columns in a table that are using for joins, aggegrates, filters etc to be
passed into compute stats command -
Suggested Syntax -
{code:java}
compute stats db.tbl (
select distinct join_columns from
from sys.impala_query_log
where contains(tables_queried, "db.tbl")
and query_dttm >current_timestamp()-7
and join_columns rlike 'db.tbl'
) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (IMPALA-12980) Translate CpuAsk into admission control slot to use
Riza Suminto created IMPALA-12980: - Summary: Translate CpuAsk into admission control slot to use Key: IMPALA-12980 URL: https://issues.apache.org/jira/browse/IMPALA-12980 Project: IMPALA Issue Type: Improvement Components: Distributed Exec, Frontend Reporter: Riza Suminto Assignee: Riza Suminto Admission control slot accounting is described in IMPALA-8998. On each backend, number of. It compute 'slots_to_use' for each backend based on the max number of instances of any fragment on that backend. This is simplistic, because multiple fragments with same number of instance count, say 4 non-blocking fragments each with 12 instances, only request the max instance (12) admission slots rather than sum of it (48). When COMPUTE_PROCESSING_COST is enabled, Planner will generate a CpuAsk number that represent the cpu requirement of that query over a particular executor group set. This number is an estimation of what is the largest number of query fragments that can run in-parallel given the blocking operator analysis. Therefore, the fragment trace that sums into that CpuAsk number can be translated into 'slots_to_use' as well, which will be a closer resemblance of maximum parallel execution of fragment instances. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (IMPALA-12640) Remove IcebergDeleteSink
[ https://issues.apache.org/jira/browse/IMPALA-12640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltán Borók-Nagy resolved IMPALA-12640. Fix Version/s: Impala 4.4.0 Resolution: Fixed > Remove IcebergDeleteSink > > > Key: IMPALA-12640 > URL: https://issues.apache.org/jira/browse/IMPALA-12640 > Project: IMPALA > Issue Type: Bug > Components: Backend, Frontend >Reporter: Zoltán Borók-Nagy >Assignee: Zoltán Borók-Nagy >Priority: Major > Labels: impala-iceberg > Fix For: Impala 4.4.0 > > > UPDATE part 3 CR (https://gerrit.cloudera.org/#/c/20760/) introduces a new > sink operator for position delete records: IcebergBufferedDeleteSink. > The new operator can be used in the context of UPDATEs even in the case when > updating a partition column value, or the table has SORT BY properties. > IcebergBufferedDeleteSink doesn't require sorting by delete partitions, file > paths, and positions, as it takes care of it. > The only area where IcebergBufferedDeleteSink lags behind IcebergDeleteSink > is that it cannot spill to disk. But since it stores filepaths and positions > in a compact format it is unlikely that it would ever need to spill to disk > in a real life situation. E.g. even if there are 100M rows need to be deleted > per Impala executor, the amount of memory required is not much larger than > 800 MBs per executor. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (IMPALA-12969) DeserializeThriftMsg may leak JNI resources
[ https://issues.apache.org/jira/browse/IMPALA-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Csaba Ringhofer resolved IMPALA-12969. -- Fix Version/s: Impala 4.4.0 Resolution: Fixed > DeserializeThriftMsg may leak JNI resources > --- > > Key: IMPALA-12969 > URL: https://issues.apache.org/jira/browse/IMPALA-12969 > Project: IMPALA > Issue Type: Bug >Reporter: Csaba Ringhofer >Priority: Major > Fix For: Impala 4.4.0 > > > JNI's GetByteArrayElements should be followed by a ReleaseByteArrayElements > call, but this is not done in case there is an error during deserialization: > [https://github.com/apache/impala/blob/f05eac647647b5e03c3aafc35f785c73d07e2658/be/src/rpc/jni-thrift-util.h#L66] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (IMPALA-12979) Wildcard in CLASSPATH might not work in the RPM package
Quanlong Huang created IMPALA-12979:
---
Summary: Wildcard in CLASSPATH might not work in the RPM package
Key: IMPALA-12979
URL: https://issues.apache.org/jira/browse/IMPALA-12979
Project: IMPALA
Issue Type: Bug
Components: Infrastructure
Reporter: Quanlong Huang
Assignee: Quanlong Huang
I tried deploying the RPM package of Impala-3.4.2 (commit 8e9c5a5) on CentOS
7.9 and found launching catalogd failed by the following error (in
catalogd.INFO):
{noformat}
Wrote minidump to
/var/log/impala-minidumps/catalogd/5e3c8819-0593-4943-555addbc-665470ad.dmp
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x02baf14c, pid=156082, tid=0x7fec0dce59c0
#
# JRE version: Java(TM) SE Runtime Environment (8.0_141-b15) (build
1.8.0_141-b15)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.141-b15 mixed mode linux-amd64
compressed oops)
# Problematic frame:
# C [catalogd+0x27af14c]
llvm::SCEVAddRecExpr::getNumIterationsInRange(llvm::ConstantRange const&,
llvm::ScalarEvolution&) const+0x73c
#
# Core dump written. Default location: /opt/impala/core or core.156082
#
# An error report file with more information is saved as:
# /tmp/hs_err_pid156082.log
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
# {noformat}
There are other logs in catalogd.ERROR
{noformat}
Log file created at: 2024/04/08 04:49:28
Running on machine: ccycloud-1.quanlong.root.comops.site
Log line format: [IWEF]mmdd hh:mm:ss.uu threadid file:line] msg
E0408 04:49:28.979386 158187 logging.cc:146] stderr will be logged to this file.
Wrote minidump to
/var/log/impala-minidumps/catalogd/6c3f550c-be96-4a5b-61171aac-0de15155.dmp
could not find method getRootCauseMessage from class (null) with signature
(Ljava/lang/Throwable;)Ljava/lang/String;
could not find method getStackTrace from class (null) with signature
(Ljava/lang/Throwable;)Ljava/lang/String;
FileSystem: loadFileSystems failed error:
(unable to get root cause for java.lang.NoClassDefFoundError)
(unable to get stack trace for java.lang.NoClassDefFoundError){noformat}
Resolving the minidump shows me the following stacktrace:
{noformat}
(gdb) bt
#0 0x02baf14c in ?? ()
#1 0x02baee24 in getJNIEnv ()
#2 0x02bacb71 in hdfsBuilderConnect ()
#3 0x012e6ae2 in impala::JniUtil::InitLibhdfs() ()
#4 0x012e7897 in impala::JniUtil::Init() ()
#5 0x00be9297 in impala::InitCommonRuntime(int, char**, bool,
impala::TestInfo::Mode) ()
#6 0x00bb604a in CatalogdMain(int, char**) ()
#7 0x00b33f97 in main (){noformat}
It indicates something wrong in initializing the JVM. Here are the env vars:
{noformat}
Environment Variables:
JAVA_HOME=/usr/java/jdk1.8.0_141
CLASSPATH=/opt/impala/conf:/opt/impala/jar/*
PATH=/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/root/bin
LD_LIBRARY_PATH=/opt/impala/lib/:/usr/java/jdk1.8.0_141/jre/lib/amd64/server:/usr/java/jdk1.8.0_141/jre/lib/amd64
SHELL=/bin/bash{noformat}
We use wildcard "*" in the classpath which seems to be the cause. The issue was
resolved after using explicit paths in the classpath. Here are what I changed
in bin/impala-env.sh:
{code:bash}
#export CLASSPATH="/opt/impala/conf:/opt/impala/jar/*"
CLASSPATH=/opt/impala/conf
for jar in /opt/impala/jar/*.jar; do
CLASSPATH="$CLASSPATH:$jar"
done
export CLASSPATH
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (IMPALA-12978) IMPALA-12544 made impala-shell incompatible with old impala servers
Csaba Ringhofer created IMPALA-12978: Summary: IMPALA-12544 made impala-shell incompatible with old impala servers Key: IMPALA-12978 URL: https://issues.apache.org/jira/browse/IMPALA-12978 Project: IMPALA Issue Type: Bug Components: Clients Reporter: Csaba Ringhofer IMPALA-12544 uses "progress.total_fragment_instances > 0:", but total_fragment_instances is None if the server is older and does not know this Thrift member yet (added in IMPALA-12048). [https://github.com/apache/impala/blob/fb3c379f395635f9f6927b40694bc3dd95a2866f/shell/impala_shell.py#L1320] This leads to error messages in interactive shell sessions when progress reporting is enabled. -- This message was sent by Atlassian Jira (v8.20.10#820010)
