[jira] [Closed] (IMPALA-13297) Custom Cluster Test Failure: Error Starting Cluster
[
https://issues.apache.org/jira/browse/IMPALA-13297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Fehr closed IMPALA-13297.
---
Resolution: Cannot Reproduce
> Custom Cluster Test Failure: Error Starting Cluster
> ---
>
> Key: IMPALA-13297
> URL: https://issues.apache.org/jira/browse/IMPALA-13297
> Project: IMPALA
> Issue Type: Bug
>Reporter: Jason Fehr
>Priority: Minor
> Labels: broken-build, flaky-test, test-infra
>
> Custom cluster test TestCatalogdHA.test_catalogd_ha_with_one_catalogd failed
> with "Error starting cluster"
> {noformat}
> E CalledProcessError: Command
> '['/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/bin/start-impala-cluster.py',
> '--state_store_args=--statestore_update_frequency_ms=50
> --statestore_priority_update_frequency_ms=50
> --statestore_heartbeat_frequency_ms=50', '--cluster_size=3',
> '--num_coordinators=3',
> '--log_dir=/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests',
> '--log_level=1', '--state_store_args=--enable_catalogd_ha=true
> --use_subscriber_id_as_catalogd_priority=true
> --catalogd_ha_preemption_wait_period_ms=200 ',
> '--catalogd_args=--enable_catalogd_ha=true ',
> '--impalad_args=--default_query_options=']' returned non-zero exit status 1
> {noformat}
> {noformat}
> MainThread: Error starting cluster
> Traceback (most recent call last):
> File
> "/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/bin/start-impala-cluster.py",
> line 1154, in
> impala_cluster.wait_until_ready(expected_cluster_size,
> expected_num_ready_impalads)
> File
> "/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/common/impala_cluster.py",
> line 228, in wait_until_ready
> early_abort_fn=check_processes_still_running)
> File
> "/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/common/impala_service.py",
> line 360, in wait_for_num_known_live_backends
> early_abort_fn()
> File
> "/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/common/impala_cluster.py",
> line 221, in check_processes_still_running
> assert len(self.impalads) >= expected_num_impalads
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Closed] (IMPALA-13300) Maven Could Not Acquire Lock(s)
[
https://issues.apache.org/jira/browse/IMPALA-13300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Fehr closed IMPALA-13300.
---
Resolution: Fixed
> Maven Could Not Acquire Lock(s)
> ---
>
> Key: IMPALA-13300
> URL: https://issues.apache.org/jira/browse/IMPALA-13300
> Project: IMPALA
> Issue Type: Bug
>Reporter: Jason Fehr
>Assignee: Jason Fehr
>Priority: Critical
> Labels: broken-build, flaky-test, test-infra
>
> {noformat}
> 02:55:36 [INFO] Test Data Generator
> [jar]
> 02:55:36 [INFO] Apache Impala External Data Source Test Library
> [jar]
> 02:55:39 [INFO] Building Test Data Generator 4.5.0-SNAPSHOT
> [2/18]
> 02:56:29 [INFO] Apache Impala Parent POM 4.5.0-SNAPSHOT SUCCESS
> [ 2.224 s]
> 02:56:29 [INFO] Test Data Generator 4.5.0-SNAPSHOT . FAILURE
> [ 50.783 s]
> 02:56:29 [INFO] Apache Impala External Data Source Test Library
> 4.5.0-SNAPSHOT SKIPPED
> 02:56:29 [INFO] BUILD FAILURE
> 02:56:29 [ERROR] Failed to execute goal on project impala-data-generator:
> Could not resolve dependencies for project
> org.apache.impala:impala-data-generator:jar:4.5.0-SNAPSHOT: Failed to collect
> dependencies at org.kitesdk:kite-data-core:jar:1.1.0 ->
> com.fasterxml.jackson.core:jackson-core:jar:2.15.3:
> java.lang.IllegalStateException: Could not acquire lock(s) -> [Help
> 1]{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Closed] (IMPALA-13398) Cannot Set Properties on System Tables
[
https://issues.apache.org/jira/browse/IMPALA-13398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Fehr closed IMPALA-13398.
---
Resolution: Cannot Reproduce
After a full rebuild, error no longer appears and system table properties can
be set.
> Cannot Set Properties on System Tables
> --
>
> Key: IMPALA-13398
> URL: https://issues.apache.org/jira/browse/IMPALA-13398
> Project: IMPALA
> Issue Type: Bug
>Reporter: Jason Fehr
>Priority: Major
>
> When running "alter table set TBLPROPERTIES" DMLs on system tables, the
> response is "IllegalStateException: null".
> Stack trace from catalogd.INFO file:
> {noformat}
> I0920 16:12:15.601699 1292854 JniUtil.java:166] execDdl request: ALTER_TABLE
> sys.impala_query_live issued by impala
> I0920 16:12:15.626621 1292854 CatalogOpExecutor.java:1117] Cancel in-progress
> in-flight event of table sys.impala_query_live. versionNumber=2091
> isRemoved=false inflightEventAdded_=false
> E0920 16:12:15.626955 1292854 JniUtil.java:183] Error in ALTER_TABLE
> sys.impala_query_live issued by impala. Time spent: 25ms
> I0920 16:12:15.627753 1292854 jni-util.cc:321] java.lang.IllegalStateException
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:496)
> at org.apache.impala.catalog.SystemTable.load(SystemTable.java:104)
> at
> org.apache.impala.service.CatalogOpExecutor.loadTableMetadata(CatalogOpExecutor.java:1756)
> at
> org.apache.impala.service.CatalogOpExecutor.loadTableMetadata(CatalogOpExecutor.java:1722)
> at
> org.apache.impala.service.CatalogOpExecutor.alterTable(CatalogOpExecutor.java:1430)
> at
> org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:463)
> at
> org.apache.impala.service.JniCatalog.lambda$execDdl$3(JniCatalog.java:316)
> at
> org.apache.impala.service.JniCatalogOp.lambda$execAndSerialize$1(JniCatalogOp.java:90)
> at org.apache.impala.service.JniCatalogOp.execOp(JniCatalogOp.java:58)
> at
> org.apache.impala.service.JniCatalogOp.execAndSerialize(JniCatalogOp.java:89)
> at
> org.apache.impala.service.JniCatalogOp.execAndSerialize(JniCatalogOp.java:100)
> at
> org.apache.impala.service.JniCatalog.execAndSerialize(JniCatalog.java:245)
> at
> org.apache.impala.service.JniCatalog.execAndSerialize(JniCatalog.java:259)
> at org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:315)
> I0920 16:12:15.627768 1292854 status.cc:129] IllegalStateException: null
> @ 0x302352d impala::Status::Status()
> @ 0x4221b8e impala::JniUtil::GetJniExceptionMsg()
> @ 0x2ed9af4 impala::JniCall::Call<>()
> @ 0x2e74ea0 impala::Catalog::ExecDdl()
> @ 0x2f15f7d impala::CatalogServiceThriftIf::ExecDdl()
> @ 0x2fa793d impala::_upgradeTable()
> @ 0x2fabd52 impala::_tableSchemaManagement()
> @ 0x2fac200
> impala::CatalogServer::InitCatalogWorkloadManagement()
> @ 0x2f2ecd9 CatalogdMain()
> @ 0x2e4992f main
> @ 0x7f9445590083 __libc_start_main
> @ 0x2e4976e _start
> E0920 16:12:15.72 1292854 catalog-server.cc:292] IllegalStateException:
> null
> F0920 16:12:15.720032 1292854 catalogd-main.cc:104] IllegalStateException:
> null{noformat}
> The error is being thrown out of the load function in the SystemTable.java
> class:
> https://github.com/apache/impala/blob/7167f3b4f0b6940f36705c95b8da17941557f721/fe/src/main/java/org/apache/impala/catalog/SystemTable.java#L98
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-13398) Cannot Set Properties on System Tables
Jason Fehr created IMPALA-13398:
---
Summary: Cannot Set Properties on System Tables
Key: IMPALA-13398
URL: https://issues.apache.org/jira/browse/IMPALA-13398
Project: IMPALA
Issue Type: Bug
Reporter: Jason Fehr
When running "alter table set TBLPROPERTIES" DMLs on system tables, the
response is "IllegalStateException: null".
Stack trace from catalogd.INFO file:
{noformat}
I0920 16:12:15.601699 1292854 JniUtil.java:166] execDdl request: ALTER_TABLE
sys.impala_query_live issued by impala
I0920 16:12:15.626621 1292854 CatalogOpExecutor.java:1117] Cancel in-progress
in-flight event of table sys.impala_query_live. versionNumber=2091
isRemoved=false inflightEventAdded_=false
E0920 16:12:15.626955 1292854 JniUtil.java:183] Error in ALTER_TABLE
sys.impala_query_live issued by impala. Time spent: 25ms
I0920 16:12:15.627753 1292854 jni-util.cc:321] java.lang.IllegalStateException
at
com.google.common.base.Preconditions.checkState(Preconditions.java:496)
at org.apache.impala.catalog.SystemTable.load(SystemTable.java:104)
at
org.apache.impala.service.CatalogOpExecutor.loadTableMetadata(CatalogOpExecutor.java:1756)
at
org.apache.impala.service.CatalogOpExecutor.loadTableMetadata(CatalogOpExecutor.java:1722)
at
org.apache.impala.service.CatalogOpExecutor.alterTable(CatalogOpExecutor.java:1430)
at
org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:463)
at
org.apache.impala.service.JniCatalog.lambda$execDdl$3(JniCatalog.java:316)
at
org.apache.impala.service.JniCatalogOp.lambda$execAndSerialize$1(JniCatalogOp.java:90)
at org.apache.impala.service.JniCatalogOp.execOp(JniCatalogOp.java:58)
at
org.apache.impala.service.JniCatalogOp.execAndSerialize(JniCatalogOp.java:89)
at
org.apache.impala.service.JniCatalogOp.execAndSerialize(JniCatalogOp.java:100)
at
org.apache.impala.service.JniCatalog.execAndSerialize(JniCatalog.java:245)
at
org.apache.impala.service.JniCatalog.execAndSerialize(JniCatalog.java:259)
at org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:315)
I0920 16:12:15.627768 1292854 status.cc:129] IllegalStateException: null
@ 0x302352d impala::Status::Status()
@ 0x4221b8e impala::JniUtil::GetJniExceptionMsg()
@ 0x2ed9af4 impala::JniCall::Call<>()
@ 0x2e74ea0 impala::Catalog::ExecDdl()
@ 0x2f15f7d impala::CatalogServiceThriftIf::ExecDdl()
@ 0x2fa793d impala::_upgradeTable()
@ 0x2fabd52 impala::_tableSchemaManagement()
@ 0x2fac200 impala::CatalogServer::InitCatalogWorkloadManagement()
@ 0x2f2ecd9 CatalogdMain()
@ 0x2e4992f main
@ 0x7f9445590083 __libc_start_main
@ 0x2e4976e _start
E0920 16:12:15.72 1292854 catalog-server.cc:292] IllegalStateException: null
F0920 16:12:15.720032 1292854 catalogd-main.cc:104] IllegalStateException:
null{noformat}
The error is being thrown out of the load function in the SystemTable.java
class:
https://github.com/apache/impala/blob/7167f3b4f0b6940f36705c95b8da17941557f721/fe/src/main/java/org/apache/impala/catalog/SystemTable.java#L98
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-13322) Cannot Alter Table sys.impala_query_live
[ https://issues.apache.org/jira/browse/IMPALA-13322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Fehr updated IMPALA-13322: Description: When running "alter table add column if not exists" ddls on sys.impala_query_live, Impala returns "IllegalStateException: null". No stack trace can be found in any log files. Even though that error was returned, the column was still added, it just took about 10 seconds to show up in the table. Once this issue is fixed, the disparate logic to handle upgrades on sys.impala_query_log and sys.impala_query_live can be unified. was: When running "alter table add column if not exists" ddls on sys.impala_query_live, Impala returns "IllegalStateException: null". No stack trace can be found in any log files. Even though that error was returned, the column was still added, it just took about 10 seconds to show up in the table. > Cannot Alter Table sys.impala_query_live > > > Key: IMPALA-13322 > URL: https://issues.apache.org/jira/browse/IMPALA-13322 > Project: IMPALA > Issue Type: Bug >Reporter: Jason Fehr >Priority: Critical > Labels: workload-management > > When running "alter table add column if not exists" ddls on > sys.impala_query_live, Impala returns "IllegalStateException: null". No > stack trace can be found in any log files. > Even though that error was returned, the column was still added, it just took > about 10 seconds to show up in the table. > Once this issue is fixed, the disparate logic to handle upgrades on > sys.impala_query_log and sys.impala_query_live can be unified. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-13322) Cannot Alter Table sys.impala_query_live
Jason Fehr created IMPALA-13322: --- Summary: Cannot Alter Table sys.impala_query_live Key: IMPALA-13322 URL: https://issues.apache.org/jira/browse/IMPALA-13322 Project: IMPALA Issue Type: Bug Reporter: Jason Fehr When running "alter table add column if not exists" ddls on sys.impala_query_live, Impala returns "IllegalStateException: null". No stack trace can be found in any log files. Even though that error was returned, the column was still added, it just took about 10 seconds to show up in the table. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-13298) TestRPCTimeout.test_miss_complete_cb: RPC Failed: Could Not Connect
[
https://issues.apache.org/jira/browse/IMPALA-13298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Fehr reassigned IMPALA-13298:
---
Assignee: Jason Fehr
> TestRPCTimeout.test_miss_complete_cb: RPC Failed: Could Not Connect
> ---
>
> Key: IMPALA-13298
> URL: https://issues.apache.org/jira/browse/IMPALA-13298
> Project: IMPALA
> Issue Type: Bug
>Reporter: Jason Fehr
>Assignee: Jason Fehr
>Priority: Minor
> Labels: broken-build, flaky-test
>
> Custom cluster test assert failure:
> Error:
> {noformat}
> assert 'Query aborted' in 'Query dc4af3c7143ae570:142a80e3
> failed:\nExec() rpc failed: Remote error: Runtime error: Debug Action:
> IMPALA_SERVICE_POOL:FAIL\n\n' + where 'Query
> dc4af3c7143ae570:142a80e3 failed:\nExec() rpc failed: Remote error:
> Runtime error: Debug Action: IMPALA_SERVICE_POOL:FAIL\n\n' =
> str(ImpalaBeeswaxException())
> {noformat}
> Stack Trace:
> {noformat}
> custom_cluster/test_rpc_timeout.py:231: in test_miss_complete_cb
> assert "Query aborted" in str(ex)
> E assert 'Query aborted' in 'Query dc4af3c7143ae570:142a80e3
> failed:\nExec() rpc failed: Remote error: Runtime error: Debug Action:
> IMPALA_SERVICE_POOL:FAIL\n\n'
> E+ where 'Query dc4af3c7143ae570:142a80e3 failed:\nExec() rpc
> failed: Remote error: Runtime error: Debug Action:
> IMPALA_SERVICE_POOL:FAIL\n\n' = str(ImpalaBeeswaxException())
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-13300) Maven Could Not Acquire Lock(s)
Jason Fehr created IMPALA-13300:
---
Summary: Maven Could Not Acquire Lock(s)
Key: IMPALA-13300
URL: https://issues.apache.org/jira/browse/IMPALA-13300
Project: IMPALA
Issue Type: Bug
Reporter: Jason Fehr
Assignee: Jason Fehr
{noformat}
02:55:36 [INFO] Test Data Generator
[jar]
02:55:36 [INFO] Apache Impala External Data Source Test Library
[jar]
02:55:39 [INFO] Building Test Data Generator 4.5.0-SNAPSHOT
[2/18]
02:56:29 [INFO] Apache Impala Parent POM 4.5.0-SNAPSHOT SUCCESS [
2.224 s]
02:56:29 [INFO] Test Data Generator 4.5.0-SNAPSHOT . FAILURE [
50.783 s]
02:56:29 [INFO] Apache Impala External Data Source Test Library 4.5.0-SNAPSHOT
SKIPPED
02:56:29 [INFO] BUILD FAILURE
02:56:29 [ERROR] Failed to execute goal on project impala-data-generator: Could
not resolve dependencies for project
org.apache.impala:impala-data-generator:jar:4.5.0-SNAPSHOT: Failed to collect
dependencies at org.kitesdk:kite-data-core:jar:1.1.0 ->
com.fasterxml.jackson.core:jackson-core:jar:2.15.3:
java.lang.IllegalStateException: Could not acquire lock(s) -> [Help 1]{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-13300) Maven Could Not Acquire Lock(s)
[
https://issues.apache.org/jira/browse/IMPALA-13300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Fehr updated IMPALA-13300:
Labels: broken-build flaky-test test-infra (was: )
> Maven Could Not Acquire Lock(s)
> ---
>
> Key: IMPALA-13300
> URL: https://issues.apache.org/jira/browse/IMPALA-13300
> Project: IMPALA
> Issue Type: Bug
>Reporter: Jason Fehr
>Assignee: Jason Fehr
>Priority: Critical
> Labels: broken-build, flaky-test, test-infra
>
> {noformat}
> 02:55:36 [INFO] Test Data Generator
> [jar]
> 02:55:36 [INFO] Apache Impala External Data Source Test Library
> [jar]
> 02:55:39 [INFO] Building Test Data Generator 4.5.0-SNAPSHOT
> [2/18]
> 02:56:29 [INFO] Apache Impala Parent POM 4.5.0-SNAPSHOT SUCCESS
> [ 2.224 s]
> 02:56:29 [INFO] Test Data Generator 4.5.0-SNAPSHOT . FAILURE
> [ 50.783 s]
> 02:56:29 [INFO] Apache Impala External Data Source Test Library
> 4.5.0-SNAPSHOT SKIPPED
> 02:56:29 [INFO] BUILD FAILURE
> 02:56:29 [ERROR] Failed to execute goal on project impala-data-generator:
> Could not resolve dependencies for project
> org.apache.impala:impala-data-generator:jar:4.5.0-SNAPSHOT: Failed to collect
> dependencies at org.kitesdk:kite-data-core:jar:1.1.0 ->
> com.fasterxml.jackson.core:jackson-core:jar:2.15.3:
> java.lang.IllegalStateException: Could not acquire lock(s) -> [Help
> 1]{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-13298) TestRPCTimeout.test_miss_complete_cb: RPC Failed: Could Not Connect
Jason Fehr created IMPALA-13298:
---
Summary: TestRPCTimeout.test_miss_complete_cb: RPC Failed: Could
Not Connect
Key: IMPALA-13298
URL: https://issues.apache.org/jira/browse/IMPALA-13298
Project: IMPALA
Issue Type: Bug
Reporter: Jason Fehr
Custom cluster test assert failure:
Error:
{noformat}
assert 'Query aborted' in 'Query dc4af3c7143ae570:142a80e3
failed:\nExec() rpc failed: Remote error: Runtime error: Debug Action:
IMPALA_SERVICE_POOL:FAIL\n\n' + where 'Query
dc4af3c7143ae570:142a80e3 failed:\nExec() rpc failed: Remote error:
Runtime error: Debug Action: IMPALA_SERVICE_POOL:FAIL\n\n' =
str(ImpalaBeeswaxException())
{noformat}
Stack Trace:
{noformat}
custom_cluster/test_rpc_timeout.py:231: in test_miss_complete_cb
assert "Query aborted" in str(ex)
E assert 'Query aborted' in 'Query dc4af3c7143ae570:142a80e3
failed:\nExec() rpc failed: Remote error: Runtime error: Debug Action:
IMPALA_SERVICE_POOL:FAIL\n\n'
E+ where 'Query dc4af3c7143ae570:142a80e3 failed:\nExec() rpc
failed: Remote error: Runtime error: Debug Action:
IMPALA_SERVICE_POOL:FAIL\n\n' = str(ImpalaBeeswaxException())
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-13297) Custom Cluster Test Failure: Error Starting Cluster
Jason Fehr created IMPALA-13297:
---
Summary: Custom Cluster Test Failure: Error Starting Cluster
Key: IMPALA-13297
URL: https://issues.apache.org/jira/browse/IMPALA-13297
Project: IMPALA
Issue Type: Bug
Reporter: Jason Fehr
Custom cluster test TestCatalogdHA.test_catalogd_ha_with_one_catalogd failed
with "Error starting cluster"
{noformat}
E CalledProcessError: Command
'['/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/bin/start-impala-cluster.py',
'--state_store_args=--statestore_update_frequency_ms=50
--statestore_priority_update_frequency_ms=50
--statestore_heartbeat_frequency_ms=50', '--cluster_size=3',
'--num_coordinators=3',
'--log_dir=/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/logs/custom_cluster_tests',
'--log_level=1', '--state_store_args=--enable_catalogd_ha=true
--use_subscriber_id_as_catalogd_priority=true
--catalogd_ha_preemption_wait_period_ms=200 ',
'--catalogd_args=--enable_catalogd_ha=true ',
'--impalad_args=--default_query_options=']' returned non-zero exit status 1
{noformat}
{noformat}
MainThread: Error starting cluster
Traceback (most recent call last):
File
"/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/bin/start-impala-cluster.py",
line 1154, in
impala_cluster.wait_until_ready(expected_cluster_size,
expected_num_ready_impalads)
File
"/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/common/impala_cluster.py",
line 228, in wait_until_ready
early_abort_fn=check_processes_still_running)
File
"/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/common/impala_service.py",
line 360, in wait_for_num_known_live_backends
early_abort_fn()
File
"/data/jenkins/workspace/impala-asf-master-core-s3/repos/Impala/tests/common/impala_cluster.py",
line 221, in check_processes_still_running
assert len(self.impalads) >= expected_num_impalads
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-12737) Include List of Referenced Columns in Query Log Table
[ https://issues.apache.org/jira/browse/IMPALA-12737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Fehr updated IMPALA-12737: Priority: Critical (was: Major) > Include List of Referenced Columns in Query Log Table > - > > Key: IMPALA-12737 > URL: https://issues.apache.org/jira/browse/IMPALA-12737 > Project: IMPALA > Issue Type: Improvement >Reporter: Manish Maheshwari >Assignee: Jason Fehr >Priority: Critical > Labels: workload-management > > In the Impala query log table where completed queries are stored, add lists > of columns that were referenced in the query. The purpose behind this > functionality is to know which columns are part of > * Select clause > * Where clause > * Join clause > * Aggegrate clause > * Order by clause > There should be a column for each type of clause, so that decisions can be > made based on specific usage or on the union of those clauses. > With this information, we will feed into compute stats command to collect > stats only on the required columns that are using in joins / filters and > aggegrates and not on all the table columns. > The information can be collected as an array of > [db1.table1.column1,db1.table1.column2] > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-13060) TestQueryLogTableBeeswax.test_query_log_table_query_select_dedicate_coordinator flaky
[
https://issues.apache.org/jira/browse/IMPALA-13060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17867406#comment-17867406
]
Jason Fehr commented on IMPALA-13060:
-
Fixed by
https://github.com/apache/impala/commit/d5b4fdcb016d098ea8c473d97fd1dc514ec214a1
> TestQueryLogTableBeeswax.test_query_log_table_query_select_dedicate_coordinator
> flaky
> -
>
> Key: IMPALA-13060
> URL: https://issues.apache.org/jira/browse/IMPALA-13060
> Project: IMPALA
> Issue Type: Bug
>Reporter: Wenzhe Zhou
>Assignee: Jason Fehr
>Priority: Major
>
> The unit-test failed since the expected message "'Query successfully
> unregistered: query_id=554dcf86d11dbd5f:0ea9f28d'" was not found in
> the log file.
> Stacktrace:
> custom_cluster/test_query_log.py:414: in
> test_query_log_table_query_select_dedicate_coordinator
> client = self.get_client(vector.get_value('protocol'))
> custom_cluster/test_query_log.py:73: in get_client
> self.assert_impalad_log_contains("INFO", finish_re)
> common/impala_test_suite.py:1271: in assert_impalad_log_contains
> "impalad", level, line_regex, expected_count, timeout_s)
> common/impala_test_suite.py:1322: in assert_log_contains
> (expected_count, log_file_path, line_regex, found, line)
> E AssertionError: Expected 1 lines in file
> /data0/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/logs/custom_cluster_tests/impalad.impala-ec2-centos79-m6i-4xlarge-xldisk-04d5.vpc.cloudera.com.jenkins.log.INFO.20240506-052112.12754
> matching regex 'Query successfully unregistered:
> query_id=554dcf86d11dbd5f:0ea9f28d', but found 0 lines. Last line
> was:
> E I0506 05:21:32.810438 14218 TAcceptQueueServer.cpp:355] New connection to
> server beeswax-frontend from client
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Closed] (IMPALA-13060) TestQueryLogTableBeeswax.test_query_log_table_query_select_dedicate_coordinator flaky
[
https://issues.apache.org/jira/browse/IMPALA-13060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Fehr closed IMPALA-13060.
---
Resolution: Fixed
> TestQueryLogTableBeeswax.test_query_log_table_query_select_dedicate_coordinator
> flaky
> -
>
> Key: IMPALA-13060
> URL: https://issues.apache.org/jira/browse/IMPALA-13060
> Project: IMPALA
> Issue Type: Bug
>Reporter: Wenzhe Zhou
>Assignee: Jason Fehr
>Priority: Major
>
> The unit-test failed since the expected message "'Query successfully
> unregistered: query_id=554dcf86d11dbd5f:0ea9f28d'" was not found in
> the log file.
> Stacktrace:
> custom_cluster/test_query_log.py:414: in
> test_query_log_table_query_select_dedicate_coordinator
> client = self.get_client(vector.get_value('protocol'))
> custom_cluster/test_query_log.py:73: in get_client
> self.assert_impalad_log_contains("INFO", finish_re)
> common/impala_test_suite.py:1271: in assert_impalad_log_contains
> "impalad", level, line_regex, expected_count, timeout_s)
> common/impala_test_suite.py:1322: in assert_log_contains
> (expected_count, log_file_path, line_regex, found, line)
> E AssertionError: Expected 1 lines in file
> /data0/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/logs/custom_cluster_tests/impalad.impala-ec2-centos79-m6i-4xlarge-xldisk-04d5.vpc.cloudera.com.jenkins.log.INFO.20240506-052112.12754
> matching regex 'Query successfully unregistered:
> query_id=554dcf86d11dbd5f:0ea9f28d', but found 0 lines. Last line
> was:
> E I0506 05:21:32.810438 14218 TAcceptQueueServer.cpp:355] New connection to
> server beeswax-frontend from client
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-13239) Enable OTel Agents to Scrape Log Files
[ https://issues.apache.org/jira/browse/IMPALA-13239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Fehr updated IMPALA-13239: Description: Impala daemons write logs to local log files. OTel agents have the ability scrap such files. Ensure the Impala logs provided by all the Impala daemons (coordinator, execture, catalog, and statestore) can be scraped by OTel agents by configuring such agents using out-of-the-box configuration methods. The deliverable for this task will be a sample OTel agent configuration for scraping log files from the Impala daemons. was: Metrics are provided by all Impala daemons using a Prometheus endpoint. OTel agents have the ability to scrape such endpoints. Ensure the Impala Prometheus endpoints provided by all the Impala daemons (coordinator, execture, catalog, and statestore) can be scraped by OTel agents by configuring such agents using out-of-the-box configuration methods. The deliverable for this task will be a sample OTel agent configuration for scraping metrics from the Impala daemons. > Enable OTel Agents to Scrape Log Files > -- > > Key: IMPALA-13239 > URL: https://issues.apache.org/jira/browse/IMPALA-13239 > Project: IMPALA > Issue Type: Improvement > Components: Backend, Frontend >Reporter: Jason Fehr >Priority: Critical > Labels: observability > > Impala daemons write logs to local log files. OTel agents have the ability > scrap such files. Ensure the Impala logs provided by all the Impala daemons > (coordinator, execture, catalog, and statestore) can be scraped by OTel > agents by configuring such agents using out-of-the-box configuration methods. > The deliverable for this task will be a sample OTel agent configuration for > scraping log files from the Impala daemons. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-13239) Enable OTel Agents to Scrape Log Files
Jason Fehr created IMPALA-13239: --- Summary: Enable OTel Agents to Scrape Log Files Key: IMPALA-13239 URL: https://issues.apache.org/jira/browse/IMPALA-13239 Project: IMPALA Issue Type: Improvement Components: Backend, Frontend Reporter: Jason Fehr Metrics are provided by all Impala daemons using a Prometheus endpoint. OTel agents have the ability to scrape such endpoints. Ensure the Impala Prometheus endpoints provided by all the Impala daemons (coordinator, execture, catalog, and statestore) can be scraped by OTel agents by configuring such agents using out-of-the-box configuration methods. The deliverable for this task will be a sample OTel agent configuration for scraping metrics from the Impala daemons. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-13238) Enable OTel Agents to Scrape Prometheus Metrics
[ https://issues.apache.org/jira/browse/IMPALA-13238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Fehr updated IMPALA-13238: Description: Metrics are provided by all Impala daemons using a Prometheus endpoint. OTel agents have the ability to scrape such endpoints. Ensure the Impala Prometheus endpoints provided by all the Impala daemons (coordinator, execture, catalog, and statestore) can be scraped by OTel agents by configuring such agents using out-of-the-box configuration methods. (was: Metrics are provided by all Impala daemons using a Prometheus endpoint. OTel agents have the ability to scrape such endpoints. Ensure that the Impala Prometheus endpoints can be scraped by OTel agents by configuring such agents using out-of-the-box configuration methods.) > Enable OTel Agents to Scrape Prometheus Metrics > --- > > Key: IMPALA-13238 > URL: https://issues.apache.org/jira/browse/IMPALA-13238 > Project: IMPALA > Issue Type: Improvement > Components: Backend, Frontend >Reporter: Jason Fehr >Priority: Critical > Labels: observability > > Metrics are provided by all Impala daemons using a Prometheus endpoint. OTel > agents have the ability to scrape such endpoints. Ensure the Impala > Prometheus endpoints provided by all the Impala daemons (coordinator, > execture, catalog, and statestore) can be scraped by OTel agents by > configuring such agents using out-of-the-box configuration methods. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-13238) Enable OTel Agents to Scrape Prometheus Metrics
[ https://issues.apache.org/jira/browse/IMPALA-13238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Fehr updated IMPALA-13238: Description: Metrics are provided by all Impala daemons using a Prometheus endpoint. OTel agents have the ability to scrape such endpoints. Ensure the Impala Prometheus endpoints provided by all the Impala daemons (coordinator, execture, catalog, and statestore) can be scraped by OTel agents by configuring such agents using out-of-the-box configuration methods. The deliverable for this task will be a sample OTel agent configuration for scraping metrics from the Impala daemons. was:Metrics are provided by all Impala daemons using a Prometheus endpoint. OTel agents have the ability to scrape such endpoints. Ensure the Impala Prometheus endpoints provided by all the Impala daemons (coordinator, execture, catalog, and statestore) can be scraped by OTel agents by configuring such agents using out-of-the-box configuration methods. > Enable OTel Agents to Scrape Prometheus Metrics > --- > > Key: IMPALA-13238 > URL: https://issues.apache.org/jira/browse/IMPALA-13238 > Project: IMPALA > Issue Type: Improvement > Components: Backend, Frontend >Reporter: Jason Fehr >Priority: Critical > Labels: observability > > Metrics are provided by all Impala daemons using a Prometheus endpoint. OTel > agents have the ability to scrape such endpoints. Ensure the Impala > Prometheus endpoints provided by all the Impala daemons (coordinator, > execture, catalog, and statestore) can be scraped by OTel agents by > configuring such agents using out-of-the-box configuration methods. > The deliverable for this task will be a sample OTel agent configuration for > scraping metrics from the Impala daemons. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-13238) Enable OTel Agents to Scrape Prometheus Metrics
[ https://issues.apache.org/jira/browse/IMPALA-13238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Fehr updated IMPALA-13238: Description: Metrics are provided by all Impala daemons using a Prometheus endpoint. OTel agents have the ability to scrape such endpoints. Ensure that the Impala Prometheus endpoints can be scraped by OTel agents by configuring such agents using out-of-the-box configuration methods. (was: Throughout the lifecycle of a query, several events happen. Implement the sending of these events to OTel systems using the OTel SDK.) > Enable OTel Agents to Scrape Prometheus Metrics > --- > > Key: IMPALA-13238 > URL: https://issues.apache.org/jira/browse/IMPALA-13238 > Project: IMPALA > Issue Type: Improvement > Components: Backend, Frontend >Reporter: Jason Fehr >Priority: Critical > Labels: observability > > Metrics are provided by all Impala daemons using a Prometheus endpoint. OTel > agents have the ability to scrape such endpoints. Ensure that the Impala > Prometheus endpoints can be scraped by OTel agents by configuring such agents > using out-of-the-box configuration methods. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-13238) Enable OTel Agents to Scrape Prometheus Metrics
Jason Fehr created IMPALA-13238: --- Summary: Enable OTel Agents to Scrape Prometheus Metrics Key: IMPALA-13238 URL: https://issues.apache.org/jira/browse/IMPALA-13238 Project: IMPALA Issue Type: Improvement Components: Backend, Frontend Reporter: Jason Fehr Throughout the lifecycle of a query, several events happen. Implement the sending of these events to OTel systems using the OTel SDK. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-13237) Send Backend/Frontend Query Lifecycle Events to Otel
[ https://issues.apache.org/jira/browse/IMPALA-13237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Fehr updated IMPALA-13237: Description: Throughout the lifecycle of a query, several events happen. Implement the sending of these events to (was: Integrate the Otel SDK for Java into the Impala frontend. This work involves: 1. updating Impala to include the Otel SDK in the build process 2. adding any necessary code to enable its use 3. supporting authentication with OTel systems) > Send Backend/Frontend Query Lifecycle Events to Otel > > > Key: IMPALA-13237 > URL: https://issues.apache.org/jira/browse/IMPALA-13237 > Project: IMPALA > Issue Type: Improvement > Components: Backend, Frontend >Reporter: Jason Fehr >Priority: Critical > Labels: observability > > Throughout the lifecycle of a query, several events happen. Implement the > sending of these events to -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-13237) Send Backend/Frontend Query Lifecycle Events to OTel
[ https://issues.apache.org/jira/browse/IMPALA-13237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Fehr updated IMPALA-13237: Summary: Send Backend/Frontend Query Lifecycle Events to OTel (was: Send Backend/Frontend Query Lifecycle Events to Otel) > Send Backend/Frontend Query Lifecycle Events to OTel > > > Key: IMPALA-13237 > URL: https://issues.apache.org/jira/browse/IMPALA-13237 > Project: IMPALA > Issue Type: Improvement > Components: Backend, Frontend >Reporter: Jason Fehr >Priority: Critical > Labels: observability > > Throughout the lifecycle of a query, several events happen. Implement the > sending of these events to -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-13237) Send Backend/Frontend Query Lifecycle Events to Otel
[ https://issues.apache.org/jira/browse/IMPALA-13237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Fehr updated IMPALA-13237: Component/s: Backend > Send Backend/Frontend Query Lifecycle Events to Otel > > > Key: IMPALA-13237 > URL: https://issues.apache.org/jira/browse/IMPALA-13237 > Project: IMPALA > Issue Type: Improvement > Components: Backend, Frontend >Reporter: Jason Fehr >Priority: Critical > Labels: observability > > Integrate the Otel SDK for Java into the Impala frontend. This work involves: > 1. updating Impala to include the Otel SDK in the build process > 2. adding any necessary code to enable its use > 3. supporting authentication with OTel systems -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-13237) Send Backend/Frontend Query Lifecycle Events to Otel
[ https://issues.apache.org/jira/browse/IMPALA-13237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Fehr updated IMPALA-13237: Summary: Send Backend/Frontend Query Lifecycle Events to Otel (was: Send Backend/Frontend Query Lifecycle Events to Otel via SDK) > Send Backend/Frontend Query Lifecycle Events to Otel > > > Key: IMPALA-13237 > URL: https://issues.apache.org/jira/browse/IMPALA-13237 > Project: IMPALA > Issue Type: Improvement > Components: Frontend >Reporter: Jason Fehr >Priority: Critical > Labels: observability > > Integrate the Otel SDK for Java into the Impala frontend. This work involves: > 1. updating Impala to include the Otel SDK in the build process > 2. adding any necessary code to enable its use > 3. supporting authentication with OTel systems -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-13237) Send Backend/Frontend Query Lifecycle Events to OTel
[ https://issues.apache.org/jira/browse/IMPALA-13237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Fehr updated IMPALA-13237: Description: Throughout the lifecycle of a query, several events happen. Implement the sending of these events to OTel systems using the OTel SDK. (was: Throughout the lifecycle of a query, several events happen. Implement the sending of these events to ) > Send Backend/Frontend Query Lifecycle Events to OTel > > > Key: IMPALA-13237 > URL: https://issues.apache.org/jira/browse/IMPALA-13237 > Project: IMPALA > Issue Type: Improvement > Components: Backend, Frontend >Reporter: Jason Fehr >Priority: Critical > Labels: observability > > Throughout the lifecycle of a query, several events happen. Implement the > sending of these events to OTel systems using the OTel SDK. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-13237) Send Backend/Frontend Query Lifecycle Events to Otel via SDK
Jason Fehr created IMPALA-13237: --- Summary: Send Backend/Frontend Query Lifecycle Events to Otel via SDK Key: IMPALA-13237 URL: https://issues.apache.org/jira/browse/IMPALA-13237 Project: IMPALA Issue Type: Improvement Components: Frontend Reporter: Jason Fehr Integrate the Otel SDK for Java into the Impala frontend. This work involves: 1. updating Impala to include the Otel SDK in the build process 2. adding any necessary code to enable its use 3. supporting authentication with OTel systems -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-13236) Integrate OTel Java SDK into Impala Frontend
[ https://issues.apache.org/jira/browse/IMPALA-13236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Fehr updated IMPALA-13236: Description: Integrate the Otel SDK for Java into the Impala frontend. This work involves: 1. updating Impala to include the Otel SDK in the build process 2. adding any necessary code to enable its use 3. supporting authentication with OTel systems was: Integrate the Otel SDK for C++ into Impala. This work involves: 1. updating Impala to include the Otel SDK in the build process 2. adding any necessary code to enable its use 3. supporting authentication with OTel systems > Integrate OTel Java SDK into Impala Frontend > > > Key: IMPALA-13236 > URL: https://issues.apache.org/jira/browse/IMPALA-13236 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Reporter: Jason Fehr >Priority: Critical > Labels: observability > > Integrate the Otel SDK for Java into the Impala frontend. This work involves: > 1. updating Impala to include the Otel SDK in the build process > 2. adding any necessary code to enable its use > 3. supporting authentication with OTel systems -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-13236) Integrate OTel Java SDK into Impala Frontend
[ https://issues.apache.org/jira/browse/IMPALA-13236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Fehr updated IMPALA-13236: Component/s: Frontend (was: Backend) > Integrate OTel Java SDK into Impala Frontend > > > Key: IMPALA-13236 > URL: https://issues.apache.org/jira/browse/IMPALA-13236 > Project: IMPALA > Issue Type: Improvement > Components: Frontend >Reporter: Jason Fehr >Priority: Critical > Labels: observability > > Integrate the Otel SDK for Java into the Impala frontend. This work involves: > 1. updating Impala to include the Otel SDK in the build process > 2. adding any necessary code to enable its use > 3. supporting authentication with OTel systems -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-13236) Integrate OTel Java SDK into Impala Frontend
Jason Fehr created IMPALA-13236: --- Summary: Integrate OTel Java SDK into Impala Frontend Key: IMPALA-13236 URL: https://issues.apache.org/jira/browse/IMPALA-13236 Project: IMPALA Issue Type: Improvement Components: Backend Reporter: Jason Fehr Integrate the Otel SDK for C++ into Impala. This work involves: 1. updating Impala to include the Otel SDK in the build process 2. adding any necessary code to enable its use 3. supporting authentication with OTel systems -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-13235) Integrate OTel C++ SDK into Impala Backend
Jason Fehr created IMPALA-13235: --- Summary: Integrate OTel C++ SDK into Impala Backend Key: IMPALA-13235 URL: https://issues.apache.org/jira/browse/IMPALA-13235 Project: IMPALA Issue Type: Improvement Components: Backend Reporter: Jason Fehr Integrate the Otel SDK for C++ into Impala. This work involves: 1. updating Impala to include the Otel SDK in the build process 2. adding any necessary code to enable its use 3. supporting authentication with OTel systems -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-13234) OpenTelemetry Integration
Jason Fehr created IMPALA-13234: --- Summary: OpenTelemetry Integration Key: IMPALA-13234 URL: https://issues.apache.org/jira/browse/IMPALA-13234 Project: IMPALA Issue Type: Epic Components: Backend, Frontend, Toolchain Reporter: Jason Fehr Assignee: Jason Fehr Implement functionality for Impala to get metrics, logs, and query lifecycle events into the OpenTelemetry system. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-13228) test_query_log_table_query_select is flaky
[
https://issues.apache.org/jira/browse/IMPALA-13228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17866551#comment-17866551
]
Jason Fehr commented on IMPALA-13228:
-
[~boroknagyz] -- can you please provide a link to the broken build so I can dig
into the log files?
> test_query_log_table_query_select is flaky
> --
>
> Key: IMPALA-13228
> URL: https://issues.apache.org/jira/browse/IMPALA-13228
> Project: IMPALA
> Issue Type: Bug
>Reporter: Zoltán Borók-Nagy
>Assignee: Jason Fehr
>Priority: Major
> Labels: broken-build
>
> h3. Error Message
> {noformat}
> AssertionError: Expected 1 lines in file
> /data0/jenkins/workspace/impala-cdwh-2024.0.18.0-core-ozone/repos/Impala/logs/custom_cluster_tests/impalad.impala-ec2-centos79-m6i-4xlarge-xldisk-17d6.vpc.cloudera.com.jenkins.log.INFO.20240714-141704.28401
> matching regex 'Query successfully unregistered:
> query_id=cf42cded8dac4e9c:5622c462', but found 0 lines. Last line
> was: I0714 14:17:14.487112 29379 impala-server.cc:1564] Query successfully
> unregistered:
> {noformat}
> h3. Stacktrace
> {noformat}
> custom_cluster/test_query_log.py:1033: in test_query_log_table_query_select
> client = self.get_client(vector.get_value('protocol'))
> custom_cluster/test_query_log.py:73: in get_client
> self.assert_impalad_log_contains("INFO", finish_re)
> common/impala_test_suite.py:1337: in assert_impalad_log_contains
> "impalad", level, line_regex, expected_count, timeout_s)
> common/impala_test_suite.py:1388: in assert_log_contains
> (expected_count, log_file_path, line_regex, found, line)
> E AssertionError: Expected 1 lines in file
> /data0/jenkins/workspace/impala-cdwh-2024.0.18.0-core-ozone/repos/Impala/logs/custom_cluster_tests/impalad.impala-ec2-centos79-m6i-4xlarge-xldisk-17d6.vpc.cloudera.com.jenkins.log.INFO.20240714-141704.28401
> matching regex 'Query successfully unregistered:
> query_id=cf42cded8dac4e9c:5622c462', but found 0 lines. Last line
> was:
> E I0714 14:17:14.487112 29379 impala-server.cc:1564] Query successfully
> unregistered:
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-13207) The 'sys' database for system tables blacklisted by default
[
https://issues.apache.org/jira/browse/IMPALA-13207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Fehr updated IMPALA-13207:
Epic Link: IMPALA-12427
> The 'sys' database for system tables blacklisted by default
> ---
>
> Key: IMPALA-13207
> URL: https://issues.apache.org/jira/browse/IMPALA-13207
> Project: IMPALA
> Issue Type: Bug
>Reporter: YifanZhang
>Assignee: Jason Fehr
>Priority: Major
>
> When enabling `enable_workload_mgmt` in a cluster, the impalad failed to
> start with a FATAL error:
> {code:java}
> F0710 17:46:51.211848 456937 workload-management.cc:346] AnalysisException:
> Database does not exist: sys. Impalad exiting. {code}
> There was also a db creation error in the log before this error happened:
> {code:java}
> I0710 17:46:51.169672 456951 client-request-state.cc:1348]
> 744c895a3184553c:0759f540] IllegalStateException: Can't create
> blacklisted database: sys. --blacklisted_dbs may be inconsistent between
> catalogd and coordinators {code}
> We must change the default value of the impalad/catalogd flag
> 'blacklisted_dbs' to enable the query log table. Maybe we should add a
> validator for 'enable_workload_mgmt' to avoid any conflicts.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-13207) The 'sys' database for system tables blacklisted by default
[
https://issues.apache.org/jira/browse/IMPALA-13207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Fehr reassigned IMPALA-13207:
---
Assignee: Jason Fehr
> The 'sys' database for system tables blacklisted by default
> ---
>
> Key: IMPALA-13207
> URL: https://issues.apache.org/jira/browse/IMPALA-13207
> Project: IMPALA
> Issue Type: Bug
>Reporter: YifanZhang
>Assignee: Jason Fehr
>Priority: Major
>
> When enabling `enable_workload_mgmt` in a cluster, the impalad failed to
> start with a FATAL error:
> {code:java}
> F0710 17:46:51.211848 456937 workload-management.cc:346] AnalysisException:
> Database does not exist: sys. Impalad exiting. {code}
> There was also a db creation error in the log before this error happened:
> {code:java}
> I0710 17:46:51.169672 456951 client-request-state.cc:1348]
> 744c895a3184553c:0759f540] IllegalStateException: Can't create
> blacklisted database: sys. --blacklisted_dbs may be inconsistent between
> catalogd and coordinators {code}
> We must change the default value of the impalad/catalogd flag
> 'blacklisted_dbs' to enable the query log table. Maybe we should add a
> validator for 'enable_workload_mgmt' to avoid any conflicts.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-12648) Support killing queries and sessions programatically
[
https://issues.apache.org/jira/browse/IMPALA-12648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17863845#comment-17863845
]
Jason Fehr commented on IMPALA-12648:
-
These termination queries must also be able to run as a simple query not
subject to admission control so that the queries do not get queued waiting with
other queries on admission control.
> Support killing queries and sessions programatically
>
>
> Key: IMPALA-12648
> URL: https://issues.apache.org/jira/browse/IMPALA-12648
> Project: IMPALA
> Issue Type: New Feature
> Components: Frontend
>Affects Versions: Impala 4.3.0
>Reporter: Manish Maheshwari
>Assignee: Jason Fehr
>Priority: Major
>
> Support killing queries and sessions programatically via kill commands to be
> able to manage impala running workloads.
> 1. Killing Queries that are currently running
> {code:java}
> -- Forcibly terminates query with the specified query_id:
> KILL QUERY WHERE query_id='634bf9fcf55278eb:ac0ef053' {code}
> For queries that are the finished and waiting to be closed, this command
> should close them
> 2. Killing sessions that are open
> {code:java}
> -- Forcibly terminates session and closes all queries:
> KILL SESSION WHERE session_id='2644d52c79c4c1e4:e974538f2189ed82' {code}
> this command should terminate the session and kill all active queries and
> close all queries that are finished and are waiting to be closed.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-13128) disk-file-test hangs on ARM + UBSAN test jobs
[
https://issues.apache.org/jira/browse/IMPALA-13128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Fehr reassigned IMPALA-13128:
---
Assignee: Jason Fehr
> disk-file-test hangs on ARM + UBSAN test jobs
> -
>
> Key: IMPALA-13128
> URL: https://issues.apache.org/jira/browse/IMPALA-13128
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 4.5.0
>Reporter: Joe McDonnell
>Assignee: Jason Fehr
>Priority: Critical
> Labels: broken-build, flaky
>
> The UBSAN ARM job (running on Redhat 8) has been hanging then timing out with
> this being the last output:
> {noformat}
> 23:06:47 63/147 Test #63: disk-io-mgr-test . Passed
> 43.42 sec
> 23:07:30 Start 64: disk-file-test
> 23:07:30
> 18:47:00
> 18:47:00 run-all-tests.sh TIMED OUT! {noformat}
> This has happened multiple times, but it looks limited to ARM + UBSAN. The
> jobs take stack traces, but only of the running impalads / HMS.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Closed] (IMPALA-13128) disk-file-test hangs on ARM + UBSAN test jobs
[
https://issues.apache.org/jira/browse/IMPALA-13128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Fehr closed IMPALA-13128.
---
Resolution: Fixed
> disk-file-test hangs on ARM + UBSAN test jobs
> -
>
> Key: IMPALA-13128
> URL: https://issues.apache.org/jira/browse/IMPALA-13128
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 4.5.0
>Reporter: Joe McDonnell
>Assignee: Jason Fehr
>Priority: Critical
> Labels: broken-build, flaky
>
> The UBSAN ARM job (running on Redhat 8) has been hanging then timing out with
> this being the last output:
> {noformat}
> 23:06:47 63/147 Test #63: disk-io-mgr-test . Passed
> 43.42 sec
> 23:07:30 Start 64: disk-file-test
> 23:07:30
> 18:47:00
> 18:47:00 run-all-tests.sh TIMED OUT! {noformat}
> This has happened multiple times, but it looks limited to ARM + UBSAN. The
> jobs take stack traces, but only of the running impalads / HMS.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-12737) Include List of Referenced Columns in Query Log Table
[ https://issues.apache.org/jira/browse/IMPALA-12737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17856589#comment-17856589 ] Jason Fehr commented on IMPALA-12737: - [~mylogi...@gmail.com] -- I'm picking up this Jira again. I see that [~MikaelSmith] had a couple of unanswered questions. Can you please take a look and answer them? > Include List of Referenced Columns in Query Log Table > - > > Key: IMPALA-12737 > URL: https://issues.apache.org/jira/browse/IMPALA-12737 > Project: IMPALA > Issue Type: Improvement >Reporter: Manish Maheshwari >Assignee: Jason Fehr >Priority: Major > Labels: workload-management > > In the Impala query log table where completed queries are stored, add lists > of columns that were referenced in the query. The purpose behind this > functionality is to know which columns are part of > * Select clause > * Where clause > * Join clause > * Aggegrate clause > * Order by clause > There should be a column for each type of clause, so that decisions can be > made based on specific usage or on the union of those clauses. > With this information, we will feed into compute stats command to collect > stats only on the required columns that are using in joins / filters and > aggegrates and not on all the table columns. > The information can be collected as an array of > [db1.table1.column1,db1.table1.column2] > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-13133) Failed LOAD DATA Statements Not Cleaning Up
Jason Fehr created IMPALA-13133: --- Summary: Failed LOAD DATA Statements Not Cleaning Up Key: IMPALA-13133 URL: https://issues.apache.org/jira/browse/IMPALA-13133 Project: IMPALA Issue Type: Bug Components: be Reporter: Jason Fehr When a LOAD DATA statement fails (for example, if there is a mismatch in the number of columns between the parquet files and the destination table), Impala is not cleaning up the table or the temporary hdfs directory. Thus, a "show tables" command still shows the temporary table, and the warehouse hdfs directory still contains a temporary folder. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Comment Edited] (IMPALA-12981) Support a column list in compute stats that is retrieved via a subquery
[
https://issues.apache.org/jira/browse/IMPALA-12981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17851719#comment-17851719
]
Jason Fehr edited comment on IMPALA-12981 at 6/3/24 4:24 PM:
-
Use Case -- customers can run compute stats on the weekends for the most
popular columns on the most popular tables. This use case must be accomplished
using only sql without using a scripting language.
was (Author: JIRAUSER298428):
Use Case -- customers can run compute stats on the weekends for the most
popular columns on the most popular tables.
> Support a column list in compute stats that is retrieved via a subquery
> -
>
> Key: IMPALA-12981
> URL: https://issues.apache.org/jira/browse/IMPALA-12981
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend, Frontend
>Reporter: Manish Maheshwari
>Priority: Major
>
> Support a column list in compute stats that is retrived via a subquery -
> Specifically we want to use Impala query history tables where we collect the
> columns in a table that are using for joins, aggegrates, filters etc to be
> passed into compute stats command -
> Suggested Syntax -
> {code:java}
> compute stats db.tbl (
> select distinct join_columns from
> from sys.impala_query_log
> where contains(tables_queried, "db.tbl")
> and query_dttm >current_timestamp()-7
> and join_columns rlike 'db.tbl'
> ) {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-12981) Support a column list in compute stats that is retrieved via a subquery
[
https://issues.apache.org/jira/browse/IMPALA-12981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17851719#comment-17851719
]
Jason Fehr commented on IMPALA-12981:
-
Use Case -- customers can run compute stats on the weekends for the most
popular columns on the most popular tables.
> Support a column list in compute stats that is retrieved via a subquery
> -
>
> Key: IMPALA-12981
> URL: https://issues.apache.org/jira/browse/IMPALA-12981
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend, Frontend
>Reporter: Manish Maheshwari
>Priority: Major
>
> Support a column list in compute stats that is retrived via a subquery -
> Specifically we want to use Impala query history tables where we collect the
> columns in a table that are using for joins, aggegrates, filters etc to be
> passed into compute stats command -
> Suggested Syntax -
> {code:java}
> compute stats db.tbl (
> select distinct join_columns from
> from sys.impala_query_log
> where contains(tables_queried, "db.tbl")
> and query_dttm >current_timestamp()-7
> and join_columns rlike 'db.tbl'
> ) {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-12648) Support killing queries and sessions programatically
[
https://issues.apache.org/jira/browse/IMPALA-12648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17851714#comment-17851714
]
Jason Fehr commented on IMPALA-12648:
-
security requirement -- kill() function is only allowed for admin users or the
user who owns the query/session.
> Support killing queries and sessions programatically
>
>
> Key: IMPALA-12648
> URL: https://issues.apache.org/jira/browse/IMPALA-12648
> Project: IMPALA
> Issue Type: New Feature
> Components: Frontend
>Affects Versions: Impala 4.3.0
>Reporter: Manish Maheshwari
>Assignee: Jason Fehr
>Priority: Major
>
> Support killing queries and sessions programatically via kill commands to be
> able to manage impala running workloads.
> 1. Killing Queries that are currently running
> {code:java}
> -- Forcibly terminates query with the specified query_id:
> KILL QUERY WHERE query_id='634bf9fcf55278eb:ac0ef053' {code}
> For queries that are the finished and waiting to be closed, this command
> should close them
> 2. Killing sessions that are open
> {code:java}
> -- Forcibly terminates session and closes all queries:
> KILL SESSION WHERE session_id='2644d52c79c4c1e4:e974538f2189ed82' {code}
> this command should terminate the session and kill all active queries and
> close all queries that are finished and are waiting to be closed.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-12737) Include List of Referenced Columns in Query Log Table
[ https://issues.apache.org/jira/browse/IMPALA-12737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Fehr reassigned IMPALA-12737: --- Assignee: Jason Fehr (was: Michael Smith) > Include List of Referenced Columns in Query Log Table > - > > Key: IMPALA-12737 > URL: https://issues.apache.org/jira/browse/IMPALA-12737 > Project: IMPALA > Issue Type: Improvement >Reporter: Manish Maheshwari >Assignee: Jason Fehr >Priority: Major > Labels: workload-management > > In the Impala query log table where completed queries are stored, add lists > of columns that were referenced in the query. The purpose behind this > functionality is to know which columns are part of > * Select clause > * Where clause > * Join clause > * Aggegrate clause > * Order by clause > There should be a column for each type of clause, so that decisions can be > made based on specific usage or on the union of those clauses. > With this information, we will feed into compute stats command to collect > stats only on the required columns that are using in joins / filters and > aggegrates and not on all the table columns. > The information can be collected as an array of > [db1.table1.column1,db1.table1.column2] > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-12652) Limit Length of Completed Queries Insert DML
[ https://issues.apache.org/jira/browse/IMPALA-12652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-12652 started by Jason Fehr. --- > Limit Length of Completed Queries Insert DML > > > Key: IMPALA-12652 > URL: https://issues.apache.org/jira/browse/IMPALA-12652 > Project: IMPALA > Issue Type: Improvement > Components: be >Reporter: Jason Fehr >Assignee: Jason Fehr >Priority: Major > Labels: backend, workload-management > > Implement a coordinator startup flag that limits the max length (number of > characters) of the insert DML statement that inserts records into the > impala_query_log table. > The purpose of this flag is to ensure that workload management does not > generate an insert DML statement that exceeds Impala's max length for a sql > statement (approximately 16 megabytes or 16 million characters). -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-13060) TestQueryLogTableBeeswax.test_query_log_table_query_select_dedicate_coordinator flaky
[
https://issues.apache.org/jira/browse/IMPALA-13060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-13060 started by Jason Fehr.
---
> TestQueryLogTableBeeswax.test_query_log_table_query_select_dedicate_coordinator
> flaky
> -
>
> Key: IMPALA-13060
> URL: https://issues.apache.org/jira/browse/IMPALA-13060
> Project: IMPALA
> Issue Type: Bug
>Reporter: Wenzhe Zhou
>Assignee: Jason Fehr
>Priority: Major
>
> The unit-test failed since the expected message "'Query successfully
> unregistered: query_id=554dcf86d11dbd5f:0ea9f28d'" was not found in
> the log file.
> Stacktrace:
> custom_cluster/test_query_log.py:414: in
> test_query_log_table_query_select_dedicate_coordinator
> client = self.get_client(vector.get_value('protocol'))
> custom_cluster/test_query_log.py:73: in get_client
> self.assert_impalad_log_contains("INFO", finish_re)
> common/impala_test_suite.py:1271: in assert_impalad_log_contains
> "impalad", level, line_regex, expected_count, timeout_s)
> common/impala_test_suite.py:1322: in assert_log_contains
> (expected_count, log_file_path, line_regex, found, line)
> E AssertionError: Expected 1 lines in file
> /data0/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/logs/custom_cluster_tests/impalad.impala-ec2-centos79-m6i-4xlarge-xldisk-04d5.vpc.cloudera.com.jenkins.log.INFO.20240506-052112.12754
> matching regex 'Query successfully unregistered:
> query_id=554dcf86d11dbd5f:0ea9f28d', but found 0 lines. Last line
> was:
> E I0506 05:21:32.810438 14218 TAcceptQueueServer.cpp:355] New connection to
> server beeswax-frontend from client
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-13060) TestQueryLogTableBeeswax.test_query_log_table_query_select_dedicate_coordinator flaky
[
https://issues.apache.org/jira/browse/IMPALA-13060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Fehr updated IMPALA-13060:
Epic Link: IMPALA-12427
> TestQueryLogTableBeeswax.test_query_log_table_query_select_dedicate_coordinator
> flaky
> -
>
> Key: IMPALA-13060
> URL: https://issues.apache.org/jira/browse/IMPALA-13060
> Project: IMPALA
> Issue Type: Bug
>Reporter: Wenzhe Zhou
>Assignee: Jason Fehr
>Priority: Major
>
> The unit-test failed since the expected message "'Query successfully
> unregistered: query_id=554dcf86d11dbd5f:0ea9f28d'" was not found in
> the log file.
> Stacktrace:
> custom_cluster/test_query_log.py:414: in
> test_query_log_table_query_select_dedicate_coordinator
> client = self.get_client(vector.get_value('protocol'))
> custom_cluster/test_query_log.py:73: in get_client
> self.assert_impalad_log_contains("INFO", finish_re)
> common/impala_test_suite.py:1271: in assert_impalad_log_contains
> "impalad", level, line_regex, expected_count, timeout_s)
> common/impala_test_suite.py:1322: in assert_log_contains
> (expected_count, log_file_path, line_regex, found, line)
> E AssertionError: Expected 1 lines in file
> /data0/jenkins/workspace/impala-asf-master-core-s3-data-cache/repos/Impala/logs/custom_cluster_tests/impalad.impala-ec2-centos79-m6i-4xlarge-xldisk-04d5.vpc.cloudera.com.jenkins.log.INFO.20240506-052112.12754
> matching regex 'Query successfully unregistered:
> query_id=554dcf86d11dbd5f:0ea9f28d', but found 0 lines. Last line
> was:
> E I0506 05:21:32.810438 14218 TAcceptQueueServer.cpp:355] New connection to
> server beeswax-frontend from client
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-12768) Set iceberg.engine.hive.lock-enabled To false By Default
[
https://issues.apache.org/jira/browse/IMPALA-12768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Fehr updated IMPALA-12768:
Epic Link: IMPALA-12427
> Set iceberg.engine.hive.lock-enabled To false By Default
> -
>
> Key: IMPALA-12768
> URL: https://issues.apache.org/jira/browse/IMPALA-12768
> Project: IMPALA
> Issue Type: Improvement
> Components: be
>Reporter: Jason Fehr
>Priority: Major
> Labels: backend, iceberg, impala
>
> Iceberg tables are currently prone to leaving a permanent lock in HMS if
> Impala shuts down while the lock is held. This was observed intermittently
> during {{sys.impala_query_log}} testing and noted in
> https://github.com/apache/iceberg/issues/2301. It's a short lock, so it's
> fairly rare, but could happen and the only recourse is to delete rows from
> HIVE_LOCKS in HMS's database.
> https://github.com/apache/iceberg/pull/6570 introduced an alternative update
> mechanism in Iceberg 1.3 that depends on HIVE-26682. [Iceberg
> documentation|https://iceberg.apache.org/docs/latest/configuration/#hadoop-configuration]
> says to set the property "iceberg.engine.hive.lock-enabled
> {noformat}
> Warn: Setting iceberg.engine.hive.lock-enabled=false will cause HiveCatalog
> to commit to tables without using Hive locks. This should only be set to
> false if all following conditions are met:
> * HIVE-26882 is available on the Hive Metastore server
> * All other HiveCatalogs committing to tables that this HiveCatalog commits
> to are also on Iceberg 1.3 or later
> * All other HiveCatalogs committing to tables that this HiveCatalog commits
> to have also disabled Hive locks on commit.
> Failing to ensure these conditions risks corrupting the table.
> {noformat}
> Setting this property to false uses a new Hive atomic operation and avoids
> taking a lock, so it can't get stuck if Impala shuts down at the wrong time.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-4024) Expose Impala metrics as a table
[ https://issues.apache.org/jira/browse/IMPALA-4024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Fehr updated IMPALA-4024: --- Epic Link: IMPALA-12427 > Expose Impala metrics as a table > > > Key: IMPALA-4024 > URL: https://issues.apache.org/jira/browse/IMPALA-4024 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Impala 2.7.0 >Reporter: Tim Armstrong >Priority: Major > Labels: supportability, usability, workload-management > Attachments: 2cb86db8.diff, ce6933d3.diff, d556ca9.diff > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-13051) Speed up test_query_log test runs
[ https://issues.apache.org/jira/browse/IMPALA-13051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Fehr reassigned IMPALA-13051: --- Assignee: Jason Fehr (was: Michael Smith) > Speed up test_query_log test runs > - > > Key: IMPALA-13051 > URL: https://issues.apache.org/jira/browse/IMPALA-13051 > Project: IMPALA > Issue Type: Task >Affects Versions: Impala 4.4.0 >Reporter: Michael Smith >Assignee: Jason Fehr >Priority: Minor > > test_query_log.py takes 11 minutes to run. Most of them use graceful > shutdown, and provide an unnecessary grace period. Optimize test_query_log > test runs, and do some other code cleanup around workload management. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-12951) Investigate RETRY_FAILED_QUERIES with System Tables
[
https://issues.apache.org/jira/browse/IMPALA-12951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17843918#comment-17843918
]
Jason Fehr commented on IMPALA-12951:
-
Is this issue limited to clients when they are reading from the system tables
or does it also cover situations where inserting into the sys.impala_query_log
table fails?
> Investigate RETRY_FAILED_QUERIES with System Tables
> ---
>
> Key: IMPALA-12951
> URL: https://issues.apache.org/jira/browse/IMPALA-12951
> Project: IMPALA
> Issue Type: Task
>Reporter: Michael Smith
>Priority: Major
>
> My theory is that {{RETRY_FAILED_QUERIES}} doesn't work well with querying
> System Tables because coordinators don't get added to a blacklist when they
> fail, so it reschedules them on the same set of nodes.
> Look into whether issues with {{RETRY_FAILED_QUERIES}} when a coordinator
> restarts (without graceful shutdown) are unique to coordinators, and how to
> improve it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Closed] (IMPALA-12440) Dont write profiles for set / Use / Get Tables / Get Schema commands
[ https://issues.apache.org/jira/browse/IMPALA-12440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Fehr closed IMPALA-12440. --- Resolution: Fixed > Dont write profiles for set / Use / Get Tables / Get Schema commands > > > Key: IMPALA-12440 > URL: https://issues.apache.org/jira/browse/IMPALA-12440 > Project: IMPALA > Issue Type: Improvement > Components: Frontend >Reporter: Manish Maheshwari >Assignee: Jason Fehr >Priority: Major > > Dont write profiles for set commands -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Closed] (IMPALA-12624) Create a System Database for Impala
[ https://issues.apache.org/jira/browse/IMPALA-12624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Fehr closed IMPALA-12624. --- Resolution: Fixed This work was implemented by https://gerrit.cloudera.org/c/20770/ thanks to [~MikaelSmith] > Create a System Database for Impala > --- > > Key: IMPALA-12624 > URL: https://issues.apache.org/jira/browse/IMPALA-12624 > Project: IMPALA > Issue Type: Improvement > Components: be >Reporter: Jason Fehr >Assignee: Michael Smith >Priority: Major > Labels: workload-management > > Impala must create a database named "system" for storing Impala related > tables (such as query history). -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Closed] (IMPALA-12913) Refactor Workload Management Custom Cluster Tests
[ https://issues.apache.org/jira/browse/IMPALA-12913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Fehr closed IMPALA-12913. --- Resolution: Fixed Commit has been merged > Refactor Workload Management Custom Cluster Tests > - > > Key: IMPALA-12913 > URL: https://issues.apache.org/jira/browse/IMPALA-12913 > Project: IMPALA > Issue Type: Improvement > Components: be >Reporter: Jason Fehr >Assignee: Jason Fehr >Priority: Minor > Labels: test, workload-management > > The custom cluster test_query_log.py tests can be refactored to leverage the > available tables in the "functional" database. Where custom tables > absolutely must be used, leverage the unique_database and unique_name test > fixtures. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Closed] (IMPALA-12936) test_query_log tests fail during test setup in UBSAN build
[
https://issues.apache.org/jira/browse/IMPALA-12936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Fehr closed IMPALA-12936.
---
Resolution: Fixed
> test_query_log tests fail during test setup in UBSAN build
> --
>
> Key: IMPALA-12936
> URL: https://issues.apache.org/jira/browse/IMPALA-12936
> Project: IMPALA
> Issue Type: Bug
>Reporter: Zoltán Borók-Nagy
>Assignee: Jason Fehr
>Priority: Major
> Labels: broken-build
>
> h2. Error details
> {noformat}
> test setup failure
> {noformat}
> h2. Stack trace
> {noformat}
> common/custom_cluster_test_suite.py:225: in teardown_method
> impalad.wait_for_exit()
> common/impala_cluster.py:471: in wait_for_exit
> while self.__get_pid() is not None:
> common/impala_cluster.py:414: in __get_pid
> assert len(pids) < 2, "Expected single pid but found %s" % ",
> ".join(map(str, pids))
> E AssertionError: Expected single pid but found 21791, 23520
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Closed] (IMPALA-12945) TickerTest.GenericTickerHappyPath failed in exhaustive release build
[
https://issues.apache.org/jira/browse/IMPALA-12945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Fehr closed IMPALA-12945.
---
Resolution: Fixed
This issue should now be fixed with the linked commit.
> TickerTest.GenericTickerHappyPath failed in exhaustive release build
>
>
> Key: IMPALA-12945
> URL: https://issues.apache.org/jira/browse/IMPALA-12945
> Project: IMPALA
> Issue Type: Bug
>Reporter: Zoltán Borók-Nagy
>Assignee: Jason Fehr
>Priority: Major
> Labels: broken-build
>
> h3. Error Message
> {noformat}
> The difference between NsToMs(sw.ElapsedTime()) and static_cast(50) is
> 9, which exceeds 5, where NsToMs(sw.ElapsedTime()) evaluates to 59,
> static_cast(50) evaluates to 50, and 5 evaluates to 5.
> {noformat}
> h3. Stacktrace
> {noformat}
> /data/jenkins/workspace/impala-cdw-master-staging-core-release-arm/repos/Impala/be/src/util/ticker-test.cc:96
> The difference between NsToMs(sw.ElapsedTime()) and static_cast(50) is
> 9, which exceeds 5, where
> NsToMs(sw.ElapsedTime()) evaluates to 59,
> static_cast(50) evaluates to 50, and
> 5 evaluates to 5.
> {noformat}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-12944) test_query_log_table_different_table failed in Ozone build
[
https://issues.apache.org/jira/browse/IMPALA-12944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-12944 started by Jason Fehr.
---
> test_query_log_table_different_table failed in Ozone build
> --
>
> Key: IMPALA-12944
> URL: https://issues.apache.org/jira/browse/IMPALA-12944
> Project: IMPALA
> Issue Type: Bug
>Reporter: Zoltán Borók-Nagy
>Assignee: Jason Fehr
>Priority: Major
> Labels: broken-build
>
> Based on Standard Error, it seems like the issue was that the test couldn't
> connect to the Impala cluster.
>
> h2. Error Message
> {noformat}
> ImpalaBeeswaxException: ImpalaBeeswaxException: INNER EXCEPTION: 'beeswaxd.ttypes.BeeswaxException'> MESSAGE: AnalysisException: Table does
> not exist: sys.completed_queries_table_1711376107
> {noformat}
> h2. Stack trace
> {noformat}
> custom_cluster/test_query_log.py:403: in test_query_log_table_different_table
> client.execute("drop table {0}.{1} purge".format(self.WM_DB,
> self.OTHER_TBL))
> common/impala_connection.py:219: in execute
> fetch_profile_after_close=fetch_profile_after_close)
> beeswax/impala_beeswax.py:191: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:382: in __execute_query
> handle = self.execute_query_async(query_string, user=user)
> beeswax/impala_beeswax.py:376: in execute_query_async
> handle = self.__do_rpc(lambda: self.imp_service.query(query,))
> beeswax/impala_beeswax.py:539: in __do_rpc
> raise ImpalaBeeswaxException(self.__build_error_message(b), b)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> EINNER EXCEPTION:
> EMESSAGE: AnalysisException: Table does not exist:
> sys.completed_queries_table_1711376107
> {noformat}
> Standard Error
> {noformat}
> -- 2024-03-25 09:22:26,560 INFO MainThread: Starting cluster with
> command:
> /data/jenkins/workspace/impala-cdw-master-staging-core-ozone/repos/Impala/bin/start-impala-cluster.py
> '--state_store_args=--statestore_update_frequency_ms=50
> --statestore_priority_update_frequency_ms=50
> --statestore_heartbeat_frequency_ms=50' --cluster_size=3 --num_coordinators=3
> --log_dir=/data/jenkins/workspace/impala-cdw-master-staging-core-ozone/repos/Impala/logs/custom_cluster_tests
> --log_level=1 '--impalad_args=--enable_workload_mgmt
> --query_log_write_interval_s=1 --shutdown_grace_period_s=10
> --shutdown_deadline_s=30 --blacklisted_dbs=information_schema
> --query_log_table_name=completed_queries_table_1711376107 '
> '--state_store_args=None ' '--catalogd_args=--enable_workload_mgmt
> --blacklisted_dbs=information_schema ' --impalad_args=--default_query_options=
> 09:22:27 MainThread: Found 0 impalad/0 statestored/0 catalogd process(es)
> ...
> -- 2024-03-25 09:22:35,972 DEBUGMainThread: Getting
> num_known_live_backends from
> impala-ec2-centos79-m6i-4xlarge-ondemand-1477.vpc.cloudera.com:25002
> -- 2024-03-25 09:22:35,973 INFO MainThread: num_known_live_backends has
> reached value: 3
> SET
> client_identifier=custom_cluster/test_query_log.py::TestQueryLogTableBeeswax::()::test_query_log_table_different_table[protocol:beeswax|exec_option:{'test_replan':1;'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;'disable_codegen':False;'abort_on_error':1;';
> -- connecting to: localhost:21000
> -- 2024-03-25 09:22:35,974 INFO MainThread: Could not connect to ('::1',
> 21000, 0, 0)
> Traceback (most recent call last):
> File
> "/data0/jenkins/workspace/impala-cdw-master-staging-core-ozone/repos/Impala/infra/python/env-gcc10.4.0/lib/python2.7/site-packages/thrift/transport/TSocket.py",
> line 137, in open
> handle.connect(sockaddr)
> File
> "/data/jenkins/workspace/impala-cdw-master-staging-core-ozone/Impala-Toolchain/toolchain-packages-gcc10.4.0/python-2.7.16/lib/python2.7/socket.py",
> line 228, in meth
> return getattr(self._sock,name)(*args)
> error: [Errno 111] Connection refused
> -- connecting to localhost:21050 with impyla
> -- 2024-03-25 09:22:35,974 INFO MainThread: Could not connect to ('::1',
> 21050, 0, 0)
> Traceback (most recent call last):
> File
> "/data0/jenkins/workspace/impala-cdw-master-staging-core-ozone/repos/Impala/infra/python/env-gcc10.4.0/lib/python2.7/site-packages/thrift/transport/TSocket.py",
> line 137, in open
> handle.connect(sockaddr)
> File
> "/data/jenkins/workspace/impala-cdw-master-staging-core-ozone/Impala-Toolchain/toolchain-packages-gcc10.4.0/python-2.7.16/lib/python2.7/socket.py",
> line 228, in meth
> return getattr(self._sock,name)(*args)
> error: [Errno 111] Connection refused
> -- 2024-03-25 09:22:36,001 INFO MainThread: Closing active operation
> -- connecting to localhost:28000 with impyla
> -- 2024-03-25 09:2
[jira] [Comment Edited] (IMPALA-12944) test_query_log_table_different_table failed in Ozone build
[
https://issues.apache.org/jira/browse/IMPALA-12944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17831868#comment-17831868
]
Jason Fehr edited comment on IMPALA-12944 at 3/28/24 3:37 PM:
--
I was able to track down the problem -- this tests ensures the completed
queries table can be renamed which requires the table to be re-created, but the
test is not allowing enough time for the completed queries table to be created.
was (Author: JIRAUSER298428):
I was able to track down the problem -- this tests ensures the completed
queries table can be renamed which requires the table to be re-created, but the
test is not allowing enough time for the completed queries table to be created.
I'm thinking this also explains some of the other test flakiness.
> test_query_log_table_different_table failed in Ozone build
> --
>
> Key: IMPALA-12944
> URL: https://issues.apache.org/jira/browse/IMPALA-12944
> Project: IMPALA
> Issue Type: Bug
>Reporter: Zoltán Borók-Nagy
>Assignee: Jason Fehr
>Priority: Major
> Labels: broken-build
>
> Based on Standard Error, it seems like the issue was that the test couldn't
> connect to the Impala cluster.
>
> h2. Error Message
> {noformat}
> ImpalaBeeswaxException: ImpalaBeeswaxException: INNER EXCEPTION: 'beeswaxd.ttypes.BeeswaxException'> MESSAGE: AnalysisException: Table does
> not exist: sys.completed_queries_table_1711376107
> {noformat}
> h2. Stack trace
> {noformat}
> custom_cluster/test_query_log.py:403: in test_query_log_table_different_table
> client.execute("drop table {0}.{1} purge".format(self.WM_DB,
> self.OTHER_TBL))
> common/impala_connection.py:219: in execute
> fetch_profile_after_close=fetch_profile_after_close)
> beeswax/impala_beeswax.py:191: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:382: in __execute_query
> handle = self.execute_query_async(query_string, user=user)
> beeswax/impala_beeswax.py:376: in execute_query_async
> handle = self.__do_rpc(lambda: self.imp_service.query(query,))
> beeswax/impala_beeswax.py:539: in __do_rpc
> raise ImpalaBeeswaxException(self.__build_error_message(b), b)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> EINNER EXCEPTION:
> EMESSAGE: AnalysisException: Table does not exist:
> sys.completed_queries_table_1711376107
> {noformat}
> Standard Error
> {noformat}
> -- 2024-03-25 09:22:26,560 INFO MainThread: Starting cluster with
> command:
> /data/jenkins/workspace/impala-cdw-master-staging-core-ozone/repos/Impala/bin/start-impala-cluster.py
> '--state_store_args=--statestore_update_frequency_ms=50
> --statestore_priority_update_frequency_ms=50
> --statestore_heartbeat_frequency_ms=50' --cluster_size=3 --num_coordinators=3
> --log_dir=/data/jenkins/workspace/impala-cdw-master-staging-core-ozone/repos/Impala/logs/custom_cluster_tests
> --log_level=1 '--impalad_args=--enable_workload_mgmt
> --query_log_write_interval_s=1 --shutdown_grace_period_s=10
> --shutdown_deadline_s=30 --blacklisted_dbs=information_schema
> --query_log_table_name=completed_queries_table_1711376107 '
> '--state_store_args=None ' '--catalogd_args=--enable_workload_mgmt
> --blacklisted_dbs=information_schema ' --impalad_args=--default_query_options=
> 09:22:27 MainThread: Found 0 impalad/0 statestored/0 catalogd process(es)
> ...
> -- 2024-03-25 09:22:35,972 DEBUGMainThread: Getting
> num_known_live_backends from
> impala-ec2-centos79-m6i-4xlarge-ondemand-1477.vpc.cloudera.com:25002
> -- 2024-03-25 09:22:35,973 INFO MainThread: num_known_live_backends has
> reached value: 3
> SET
> client_identifier=custom_cluster/test_query_log.py::TestQueryLogTableBeeswax::()::test_query_log_table_different_table[protocol:beeswax|exec_option:{'test_replan':1;'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;'disable_codegen':False;'abort_on_error':1;';
> -- connecting to: localhost:21000
> -- 2024-03-25 09:22:35,974 INFO MainThread: Could not connect to ('::1',
> 21000, 0, 0)
> Traceback (most recent call last):
> File
> "/data0/jenkins/workspace/impala-cdw-master-staging-core-ozone/repos/Impala/infra/python/env-gcc10.4.0/lib/python2.7/site-packages/thrift/transport/TSocket.py",
> line 137, in open
> handle.connect(sockaddr)
> File
> "/data/jenkins/workspace/impala-cdw-master-staging-core-ozone/Impala-Toolchain/toolchain-packages-gcc10.4.0/python-2.7.16/lib/python2.7/socket.py",
> line 228, in meth
> return getattr(self._sock,name)(*args)
> error: [Errno 111] Connection refused
> -- connecting to localhost:21050 with impyla
> -- 2024-03-25 09:22:35,974 INFO MainThread: Could not connect to ('::1',
> 21050, 0, 0)
> Traceback (m
[jira] [Commented] (IMPALA-12944) test_query_log_table_different_table failed in Ozone build
[
https://issues.apache.org/jira/browse/IMPALA-12944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17831868#comment-17831868
]
Jason Fehr commented on IMPALA-12944:
-
I was able to track down the problem -- this tests ensures the completed
queries table can be renamed which requires the table to be re-created, but the
test is not allowing enough time for the completed queries table to be created.
I'm thinking this also explains some of the other test flakiness.
> test_query_log_table_different_table failed in Ozone build
> --
>
> Key: IMPALA-12944
> URL: https://issues.apache.org/jira/browse/IMPALA-12944
> Project: IMPALA
> Issue Type: Bug
>Reporter: Zoltán Borók-Nagy
>Assignee: Jason Fehr
>Priority: Major
> Labels: broken-build
>
> Based on Standard Error, it seems like the issue was that the test couldn't
> connect to the Impala cluster.
>
> h2. Error Message
> {noformat}
> ImpalaBeeswaxException: ImpalaBeeswaxException: INNER EXCEPTION: 'beeswaxd.ttypes.BeeswaxException'> MESSAGE: AnalysisException: Table does
> not exist: sys.completed_queries_table_1711376107
> {noformat}
> h2. Stack trace
> {noformat}
> custom_cluster/test_query_log.py:403: in test_query_log_table_different_table
> client.execute("drop table {0}.{1} purge".format(self.WM_DB,
> self.OTHER_TBL))
> common/impala_connection.py:219: in execute
> fetch_profile_after_close=fetch_profile_after_close)
> beeswax/impala_beeswax.py:191: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:382: in __execute_query
> handle = self.execute_query_async(query_string, user=user)
> beeswax/impala_beeswax.py:376: in execute_query_async
> handle = self.__do_rpc(lambda: self.imp_service.query(query,))
> beeswax/impala_beeswax.py:539: in __do_rpc
> raise ImpalaBeeswaxException(self.__build_error_message(b), b)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> EINNER EXCEPTION:
> EMESSAGE: AnalysisException: Table does not exist:
> sys.completed_queries_table_1711376107
> {noformat}
> Standard Error
> {noformat}
> -- 2024-03-25 09:22:26,560 INFO MainThread: Starting cluster with
> command:
> /data/jenkins/workspace/impala-cdw-master-staging-core-ozone/repos/Impala/bin/start-impala-cluster.py
> '--state_store_args=--statestore_update_frequency_ms=50
> --statestore_priority_update_frequency_ms=50
> --statestore_heartbeat_frequency_ms=50' --cluster_size=3 --num_coordinators=3
> --log_dir=/data/jenkins/workspace/impala-cdw-master-staging-core-ozone/repos/Impala/logs/custom_cluster_tests
> --log_level=1 '--impalad_args=--enable_workload_mgmt
> --query_log_write_interval_s=1 --shutdown_grace_period_s=10
> --shutdown_deadline_s=30 --blacklisted_dbs=information_schema
> --query_log_table_name=completed_queries_table_1711376107 '
> '--state_store_args=None ' '--catalogd_args=--enable_workload_mgmt
> --blacklisted_dbs=information_schema ' --impalad_args=--default_query_options=
> 09:22:27 MainThread: Found 0 impalad/0 statestored/0 catalogd process(es)
> ...
> -- 2024-03-25 09:22:35,972 DEBUGMainThread: Getting
> num_known_live_backends from
> impala-ec2-centos79-m6i-4xlarge-ondemand-1477.vpc.cloudera.com:25002
> -- 2024-03-25 09:22:35,973 INFO MainThread: num_known_live_backends has
> reached value: 3
> SET
> client_identifier=custom_cluster/test_query_log.py::TestQueryLogTableBeeswax::()::test_query_log_table_different_table[protocol:beeswax|exec_option:{'test_replan':1;'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;'disable_codegen':False;'abort_on_error':1;';
> -- connecting to: localhost:21000
> -- 2024-03-25 09:22:35,974 INFO MainThread: Could not connect to ('::1',
> 21000, 0, 0)
> Traceback (most recent call last):
> File
> "/data0/jenkins/workspace/impala-cdw-master-staging-core-ozone/repos/Impala/infra/python/env-gcc10.4.0/lib/python2.7/site-packages/thrift/transport/TSocket.py",
> line 137, in open
> handle.connect(sockaddr)
> File
> "/data/jenkins/workspace/impala-cdw-master-staging-core-ozone/Impala-Toolchain/toolchain-packages-gcc10.4.0/python-2.7.16/lib/python2.7/socket.py",
> line 228, in meth
> return getattr(self._sock,name)(*args)
> error: [Errno 111] Connection refused
> -- connecting to localhost:21050 with impyla
> -- 2024-03-25 09:22:35,974 INFO MainThread: Could not connect to ('::1',
> 21050, 0, 0)
> Traceback (most recent call last):
> File
> "/data0/jenkins/workspace/impala-cdw-master-staging-core-ozone/repos/Impala/infra/python/env-gcc10.4.0/lib/python2.7/site-packages/thrift/transport/TSocket.py",
> line 137, in open
> handle.connect(sockaddr)
> File
> "/data/jenkins/workspace/impala-cdw-master-staging-core-
[jira] [Work started] (IMPALA-12945) TickerTest.GenericTickerHappyPath failed in exhaustive release build
[
https://issues.apache.org/jira/browse/IMPALA-12945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-12945 started by Jason Fehr.
---
> TickerTest.GenericTickerHappyPath failed in exhaustive release build
>
>
> Key: IMPALA-12945
> URL: https://issues.apache.org/jira/browse/IMPALA-12945
> Project: IMPALA
> Issue Type: Bug
>Reporter: Zoltán Borók-Nagy
>Assignee: Jason Fehr
>Priority: Major
> Labels: broken-build
>
> h3. Error Message
> {noformat}
> The difference between NsToMs(sw.ElapsedTime()) and static_cast(50) is
> 9, which exceeds 5, where NsToMs(sw.ElapsedTime()) evaluates to 59,
> static_cast(50) evaluates to 50, and 5 evaluates to 5.
> {noformat}
> h3. Stacktrace
> {noformat}
> /data/jenkins/workspace/impala-cdw-master-staging-core-release-arm/repos/Impala/be/src/util/ticker-test.cc:96
> The difference between NsToMs(sw.ElapsedTime()) and static_cast(50) is
> 9, which exceeds 5, where
> NsToMs(sw.ElapsedTime()) evaluates to 59,
> static_cast(50) evaluates to 50, and
> 5 evaluates to 5.
> {noformat}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-12944) test_query_log_table_different_table failed in Ozone build
[
https://issues.apache.org/jira/browse/IMPALA-12944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17831602#comment-17831602
]
Jason Fehr commented on IMPALA-12944:
-
Also, if you could get add a link to a failed build, I would like to look at
some of the other build artifacts.
> test_query_log_table_different_table failed in Ozone build
> --
>
> Key: IMPALA-12944
> URL: https://issues.apache.org/jira/browse/IMPALA-12944
> Project: IMPALA
> Issue Type: Bug
>Reporter: Zoltán Borók-Nagy
>Assignee: Jason Fehr
>Priority: Major
> Labels: broken-build
>
> Based on Standard Error, it seems like the issue was that the test couldn't
> connect to the Impala cluster.
>
> h2. Error Message
> {noformat}
> ImpalaBeeswaxException: ImpalaBeeswaxException: INNER EXCEPTION: 'beeswaxd.ttypes.BeeswaxException'> MESSAGE: AnalysisException: Table does
> not exist: sys.completed_queries_table_1711376107
> {noformat}
> h2. Stack trace
> {noformat}
> custom_cluster/test_query_log.py:403: in test_query_log_table_different_table
> client.execute("drop table {0}.{1} purge".format(self.WM_DB,
> self.OTHER_TBL))
> common/impala_connection.py:219: in execute
> fetch_profile_after_close=fetch_profile_after_close)
> beeswax/impala_beeswax.py:191: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:382: in __execute_query
> handle = self.execute_query_async(query_string, user=user)
> beeswax/impala_beeswax.py:376: in execute_query_async
> handle = self.__do_rpc(lambda: self.imp_service.query(query,))
> beeswax/impala_beeswax.py:539: in __do_rpc
> raise ImpalaBeeswaxException(self.__build_error_message(b), b)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> EINNER EXCEPTION:
> EMESSAGE: AnalysisException: Table does not exist:
> sys.completed_queries_table_1711376107
> {noformat}
> Standard Error
> {noformat}
> -- 2024-03-25 09:22:26,560 INFO MainThread: Starting cluster with
> command:
> /data/jenkins/workspace/impala-cdw-master-staging-core-ozone/repos/Impala/bin/start-impala-cluster.py
> '--state_store_args=--statestore_update_frequency_ms=50
> --statestore_priority_update_frequency_ms=50
> --statestore_heartbeat_frequency_ms=50' --cluster_size=3 --num_coordinators=3
> --log_dir=/data/jenkins/workspace/impala-cdw-master-staging-core-ozone/repos/Impala/logs/custom_cluster_tests
> --log_level=1 '--impalad_args=--enable_workload_mgmt
> --query_log_write_interval_s=1 --shutdown_grace_period_s=10
> --shutdown_deadline_s=30 --blacklisted_dbs=information_schema
> --query_log_table_name=completed_queries_table_1711376107 '
> '--state_store_args=None ' '--catalogd_args=--enable_workload_mgmt
> --blacklisted_dbs=information_schema ' --impalad_args=--default_query_options=
> 09:22:27 MainThread: Found 0 impalad/0 statestored/0 catalogd process(es)
> ...
> -- 2024-03-25 09:22:35,972 DEBUGMainThread: Getting
> num_known_live_backends from
> impala-ec2-centos79-m6i-4xlarge-ondemand-1477.vpc.cloudera.com:25002
> -- 2024-03-25 09:22:35,973 INFO MainThread: num_known_live_backends has
> reached value: 3
> SET
> client_identifier=custom_cluster/test_query_log.py::TestQueryLogTableBeeswax::()::test_query_log_table_different_table[protocol:beeswax|exec_option:{'test_replan':1;'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;'disable_codegen':False;'abort_on_error':1;';
> -- connecting to: localhost:21000
> -- 2024-03-25 09:22:35,974 INFO MainThread: Could not connect to ('::1',
> 21000, 0, 0)
> Traceback (most recent call last):
> File
> "/data0/jenkins/workspace/impala-cdw-master-staging-core-ozone/repos/Impala/infra/python/env-gcc10.4.0/lib/python2.7/site-packages/thrift/transport/TSocket.py",
> line 137, in open
> handle.connect(sockaddr)
> File
> "/data/jenkins/workspace/impala-cdw-master-staging-core-ozone/Impala-Toolchain/toolchain-packages-gcc10.4.0/python-2.7.16/lib/python2.7/socket.py",
> line 228, in meth
> return getattr(self._sock,name)(*args)
> error: [Errno 111] Connection refused
> -- connecting to localhost:21050 with impyla
> -- 2024-03-25 09:22:35,974 INFO MainThread: Could not connect to ('::1',
> 21050, 0, 0)
> Traceback (most recent call last):
> File
> "/data0/jenkins/workspace/impala-cdw-master-staging-core-ozone/repos/Impala/infra/python/env-gcc10.4.0/lib/python2.7/site-packages/thrift/transport/TSocket.py",
> line 137, in open
> handle.connect(sockaddr)
> File
> "/data/jenkins/workspace/impala-cdw-master-staging-core-ozone/Impala-Toolchain/toolchain-packages-gcc10.4.0/python-2.7.16/lib/python2.7/socket.py",
> line 228, in meth
> return getattr(self._sock,name)(*args)
> error: [Errno 111] Connection
[jira] [Commented] (IMPALA-12944) test_query_log_table_different_table failed in Ozone build
[
https://issues.apache.org/jira/browse/IMPALA-12944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17831599#comment-17831599
]
Jason Fehr commented on IMPALA-12944:
-
[~boroknagyz] -- has this error happened more than once? The cluster appears
to have started up correctly and then died very unexpectedly.
> test_query_log_table_different_table failed in Ozone build
> --
>
> Key: IMPALA-12944
> URL: https://issues.apache.org/jira/browse/IMPALA-12944
> Project: IMPALA
> Issue Type: Bug
>Reporter: Zoltán Borók-Nagy
>Assignee: Jason Fehr
>Priority: Major
> Labels: broken-build
>
> Based on Standard Error, it seems like the issue was that the test couldn't
> connect to the Impala cluster.
>
> h2. Error Message
> {noformat}
> ImpalaBeeswaxException: ImpalaBeeswaxException: INNER EXCEPTION: 'beeswaxd.ttypes.BeeswaxException'> MESSAGE: AnalysisException: Table does
> not exist: sys.completed_queries_table_1711376107
> {noformat}
> h2. Stack trace
> {noformat}
> custom_cluster/test_query_log.py:403: in test_query_log_table_different_table
> client.execute("drop table {0}.{1} purge".format(self.WM_DB,
> self.OTHER_TBL))
> common/impala_connection.py:219: in execute
> fetch_profile_after_close=fetch_profile_after_close)
> beeswax/impala_beeswax.py:191: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:382: in __execute_query
> handle = self.execute_query_async(query_string, user=user)
> beeswax/impala_beeswax.py:376: in execute_query_async
> handle = self.__do_rpc(lambda: self.imp_service.query(query,))
> beeswax/impala_beeswax.py:539: in __do_rpc
> raise ImpalaBeeswaxException(self.__build_error_message(b), b)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> EINNER EXCEPTION:
> EMESSAGE: AnalysisException: Table does not exist:
> sys.completed_queries_table_1711376107
> {noformat}
> Standard Error
> {noformat}
> -- 2024-03-25 09:22:26,560 INFO MainThread: Starting cluster with
> command:
> /data/jenkins/workspace/impala-cdw-master-staging-core-ozone/repos/Impala/bin/start-impala-cluster.py
> '--state_store_args=--statestore_update_frequency_ms=50
> --statestore_priority_update_frequency_ms=50
> --statestore_heartbeat_frequency_ms=50' --cluster_size=3 --num_coordinators=3
> --log_dir=/data/jenkins/workspace/impala-cdw-master-staging-core-ozone/repos/Impala/logs/custom_cluster_tests
> --log_level=1 '--impalad_args=--enable_workload_mgmt
> --query_log_write_interval_s=1 --shutdown_grace_period_s=10
> --shutdown_deadline_s=30 --blacklisted_dbs=information_schema
> --query_log_table_name=completed_queries_table_1711376107 '
> '--state_store_args=None ' '--catalogd_args=--enable_workload_mgmt
> --blacklisted_dbs=information_schema ' --impalad_args=--default_query_options=
> 09:22:27 MainThread: Found 0 impalad/0 statestored/0 catalogd process(es)
> ...
> -- 2024-03-25 09:22:35,972 DEBUGMainThread: Getting
> num_known_live_backends from
> impala-ec2-centos79-m6i-4xlarge-ondemand-1477.vpc.cloudera.com:25002
> -- 2024-03-25 09:22:35,973 INFO MainThread: num_known_live_backends has
> reached value: 3
> SET
> client_identifier=custom_cluster/test_query_log.py::TestQueryLogTableBeeswax::()::test_query_log_table_different_table[protocol:beeswax|exec_option:{'test_replan':1;'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;'disable_codegen':False;'abort_on_error':1;';
> -- connecting to: localhost:21000
> -- 2024-03-25 09:22:35,974 INFO MainThread: Could not connect to ('::1',
> 21000, 0, 0)
> Traceback (most recent call last):
> File
> "/data0/jenkins/workspace/impala-cdw-master-staging-core-ozone/repos/Impala/infra/python/env-gcc10.4.0/lib/python2.7/site-packages/thrift/transport/TSocket.py",
> line 137, in open
> handle.connect(sockaddr)
> File
> "/data/jenkins/workspace/impala-cdw-master-staging-core-ozone/Impala-Toolchain/toolchain-packages-gcc10.4.0/python-2.7.16/lib/python2.7/socket.py",
> line 228, in meth
> return getattr(self._sock,name)(*args)
> error: [Errno 111] Connection refused
> -- connecting to localhost:21050 with impyla
> -- 2024-03-25 09:22:35,974 INFO MainThread: Could not connect to ('::1',
> 21050, 0, 0)
> Traceback (most recent call last):
> File
> "/data0/jenkins/workspace/impala-cdw-master-staging-core-ozone/repos/Impala/infra/python/env-gcc10.4.0/lib/python2.7/site-packages/thrift/transport/TSocket.py",
> line 137, in open
> handle.connect(sockaddr)
> File
> "/data/jenkins/workspace/impala-cdw-master-staging-core-ozone/Impala-Toolchain/toolchain-packages-gcc10.4.0/python-2.7.16/lib/python2.7/socket.py",
> line 228, in meth
> return getattr(self._sock,name)(*args)
> er
[jira] [Assigned] (IMPALA-12941) Implement the prettyprint_duration Built-in Function
[
https://issues.apache.org/jira/browse/IMPALA-12941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Fehr reassigned IMPALA-12941:
---
Assignee: Jason Fehr
> Implement the prettyprint_duration Built-in Function
>
>
> Key: IMPALA-12941
> URL: https://issues.apache.org/jira/browse/IMPALA-12941
> Project: IMPALA
> Issue Type: Improvement
> Components: be
>Reporter: Jason Fehr
>Assignee: Jason Fehr
>Priority: Minor
> Labels: workload-management
>
> Implement a new Impala built-in function named prettyprint_duration. This
> function will take in two inputs -- a duration (all integer/floating
> point/decimal types) and a unit (nanoseconds, microseconds, milliseconds, and
> seconds) and output a prettyprinted, human readable string.
> For example:
> {noformat}
> select prettyprint_duration(2147483647, 'nanoseconds,);
> output: 2s147ms
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-12941) Implement the prettyprint_duration Built-in Function
Jason Fehr created IMPALA-12941:
---
Summary: Implement the prettyprint_duration Built-in Function
Key: IMPALA-12941
URL: https://issues.apache.org/jira/browse/IMPALA-12941
Project: IMPALA
Issue Type: Improvement
Components: be
Reporter: Jason Fehr
Implement a new Impala built-in function named prettyprint_duration. This
function will take in two inputs -- a duration (all integer/floating
point/decimal types) and a unit (nanoseconds, microseconds, milliseconds, and
seconds) and output a prettyprinted, human readable string.
For example:
{noformat}
select prettyprint_duration(2147483647, 'nanoseconds,);
output: 2s147ms
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-12558) JSON Web Keys (JWK) Require the Optional alg Property
[ https://issues.apache.org/jira/browse/IMPALA-12558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829673#comment-17829673 ] Jason Fehr commented on IMPALA-12558: - If the "alg" field exists on the JWK, it must be used. If the "alg" field is missing from the JWK, then use the "alg" field from the JWT. Impala only supports JWTs encoded as the payload of a JSON Web Signature (JWS) structure. The "alg" field is required in the headers of JWS structures -- https://datatracker.ietf.org/doc/html/rfc7515#section-4.1.1 Note: if the "alg" field is set to "none", the JWT must be rejected. > JSON Web Keys (JWK) Require the Optional alg Property > - > > Key: IMPALA-12558 > URL: https://issues.apache.org/jira/browse/IMPALA-12558 > Project: IMPALA > Issue Type: Bug > Components: be, Security >Reporter: Jason Fehr >Assignee: gaurav singh >Priority: Critical > Labels: JWT, jwt, security > > According to the [JWK > RFC|https://datatracker.ietf.org/doc/html/rfc7517#section-4.4], the "alg" > property is optional on JWKs. > Update Impala so that the "alg" property is no longer required on the JWKs. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-12932) SPOOL_QUERY_OPTION Default Value Mismatch Between Code and Docs
Jason Fehr created IMPALA-12932: --- Summary: SPOOL_QUERY_OPTION Default Value Mismatch Between Code and Docs Key: IMPALA-12932 URL: https://issues.apache.org/jira/browse/IMPALA-12932 Project: IMPALA Issue Type: Bug Components: be Reporter: Jason Fehr The Impala documentation is written to state that the SPOOL_QUERY_RESULTS query option defaults to false. A few examples: * https://impala.apache.org/docs/build/html/topics/impala_spool_query_results.html * https://impala.apache.org/docs/build/html/topics/impala_query_results_spooling.html#data_sink * https://impala.apache.org/docs/build/html/topics/impala_new_features.html The code sets this query option to true by default -- https://github.com/apache/impala/blob/1e67480ba6cd4a6f01b369eb11cc2b7330554bde/common/thrift/Query.thrift#L411 Either the docs or the code needs to be modified to fix this mismatch. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-12913) Refactor Workload Management Custom Cluster Tests
[ https://issues.apache.org/jira/browse/IMPALA-12913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-12913 started by Jason Fehr. --- > Refactor Workload Management Custom Cluster Tests > - > > Key: IMPALA-12913 > URL: https://issues.apache.org/jira/browse/IMPALA-12913 > Project: IMPALA > Issue Type: Improvement > Components: be >Reporter: Jason Fehr >Assignee: Jason Fehr >Priority: Minor > Labels: test, workload-management > > The custom cluster test_query_log.py tests can be refactored to leverage the > available tables in the "functional" database. Where custom tables > absolutely must be used, leverage the unique_database and unique_name test > fixtures. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-12913) Refactor Workload Management Custom Cluster Tests
Jason Fehr created IMPALA-12913: --- Summary: Refactor Workload Management Custom Cluster Tests Key: IMPALA-12913 URL: https://issues.apache.org/jira/browse/IMPALA-12913 Project: IMPALA Issue Type: Improvement Components: be Reporter: Jason Fehr Assignee: Jason Fehr The custom cluster test_query_log.py tests can be refactored to leverage the available tables in the "functional" database. Where custom tables absolutely must be used, leverage the unique_database and unique_name test fixtures. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-12897) Add Ability to set Memory Limit for Completed Queries Queue
Jason Fehr created IMPALA-12897: --- Summary: Add Ability to set Memory Limit for Completed Queries Queue Key: IMPALA-12897 URL: https://issues.apache.org/jira/browse/IMPALA-12897 Project: IMPALA Issue Type: Improvement Components: be Reporter: Jason Fehr The in-memory queue that holds completed queries until they are written to the completed queries table can grow unbounded in terms of memory usage. Add a flag that sets a limit on the amount of memory this queue can use before it is drained to the completed queries table. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-12859) Most steps fail in internal-server-test during some build types
[
https://issues.apache.org/jira/browse/IMPALA-12859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Fehr reassigned IMPALA-12859:
---
Assignee: Jason Fehr
> Most steps fail in internal-server-test during some build types
> ---
>
> Key: IMPALA-12859
> URL: https://issues.apache.org/jira/browse/IMPALA-12859
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 4.4.0
>Reporter: Laszlo Gaal
>Assignee: Jason Fehr
>Priority: Blocker
> Labels: broken-build
>
> 8 out of 10 test in internal-server-test failed in downstream internal builds
> during the following build types:
> * TSAN
> * JDK17
> * Arm-based RedHat8
> Only {{InvalidQueryOption}} and {{UnclosedComment}} managed to pass, all
> other tests failed with the symptom
> {code}
> Value of: found_state
> Actual: ""
> Expected: expected_state
> Which is: "FINISHED"
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-12859) Most steps fail in internal-server-test during some build types
[
https://issues.apache.org/jira/browse/IMPALA-12859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-12859 started by Jason Fehr.
---
> Most steps fail in internal-server-test during some build types
> ---
>
> Key: IMPALA-12859
> URL: https://issues.apache.org/jira/browse/IMPALA-12859
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 4.4.0
>Reporter: Laszlo Gaal
>Assignee: Jason Fehr
>Priority: Blocker
> Labels: broken-build
>
> 8 out of 10 test in internal-server-test failed in downstream internal builds
> during the following build types:
> * TSAN
> * JDK17
> * Arm-based RedHat8
> Only {{InvalidQueryOption}} and {{UnclosedComment}} managed to pass, all
> other tests failed with the symptom
> {code}
> Value of: found_state
> Actual: ""
> Expected: expected_state
> Which is: "FINISHED"
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-12859) Most steps fail in internal-server-test during some build types
[
https://issues.apache.org/jira/browse/IMPALA-12859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17822297#comment-17822297
]
Jason Fehr commented on IMPALA-12859:
-
I saw this same issue in my other change and fixed it there --
https://gerrit.cloudera.org/c/20770/28/be/src/service/impala-server.h#1186
I will get another change together to fix this issue.
> Most steps fail in internal-server-test during some build types
> ---
>
> Key: IMPALA-12859
> URL: https://issues.apache.org/jira/browse/IMPALA-12859
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 4.4.0
>Reporter: Laszlo Gaal
>Priority: Blocker
> Labels: broken-build
>
> 8 out of 10 test in internal-server-test failed in downstream internal builds
> during the following build types:
> * TSAN
> * JDK17
> * Arm-based RedHat8
> Only {{InvalidQueryOption}} and {{UnclosedComment}} managed to pass, all
> other tests failed with the symptom
> {code}
> Value of: found_state
> Actual: ""
> Expected: expected_state
> Which is: "FINISHED"
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-12559) Support x5u/x5c/x5t Parameter in JSON Web Keys (JWK)
[ https://issues.apache.org/jira/browse/IMPALA-12559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Fehr reassigned IMPALA-12559: --- Assignee: gaurav singh > Support x5u/x5c/x5t Parameter in JSON Web Keys (JWK) > > > Key: IMPALA-12559 > URL: https://issues.apache.org/jira/browse/IMPALA-12559 > Project: IMPALA > Issue Type: Bug > Components: be, Security >Reporter: Jason Fehr >Assignee: gaurav singh >Priority: Critical > Labels: JWT, jwt, security > > The ["x5u"|https://datatracker.ietf.org/doc/html/rfc7517#section-4.6], > ["x5c"|https://datatracker.ietf.org/doc/html/rfc7517#section-4.7], > ["x5t"|https://datatracker.ietf.org/doc/html/rfc7517#section-4.8], and > ["x5t#S256|https://datatracker.ietf.org/doc/html/rfc7517#section-4.9] > parameters in JWKs is not supported by Impala. Implement support for this > parameter using the available methods in the [Thalhammer/jwt-cpp > library|https://github.com/Thalhammer/jwt-cpp/blob/ce1f9df3a9f861d136d6f0c93a6f811c364d1d3d/example/jwks-verify.cpp]. > Note: If the "alg" property is specified and so is "x5u" or "x5c", then the > value of the "alg" property must match the algorithm on the certificate from > the "x5u" or "x5c" property. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-12559) Support x5u/x5c/x5t Parameter in JSON Web Keys (JWK)
[ https://issues.apache.org/jira/browse/IMPALA-12559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Fehr updated IMPALA-12559: Description: The ["x5u"|https://datatracker.ietf.org/doc/html/rfc7517#section-4.6], ["x5c"|https://datatracker.ietf.org/doc/html/rfc7517#section-4.7], ["x5t"|https://datatracker.ietf.org/doc/html/rfc7517#section-4.8], and ["x5t#S256|https://datatracker.ietf.org/doc/html/rfc7517#section-4.9] parameters in JWKs is not supported by Impala. Implement support for this parameter using the available methods in the [Thalhammer/jwt-cpp library|https://github.com/Thalhammer/jwt-cpp/blob/ce1f9df3a9f861d136d6f0c93a6f811c364d1d3d/example/jwks-verify.cpp]. Note: If the "alg" property is specified and so is "x5u" or "x5c", then the value of the "alg" property must match the algorithm on the certificate from the "x5u" or "x5c" property. was:The ["x5c" parameter|https://datatracker.ietf.org/doc/html/rfc7517#section-4.7] in JWKs is not supported by Impala. Implement support for this parameter using the available methods in the [Thalhammer/jwt-cpp library|https://github.com/Thalhammer/jwt-cpp/blob/ce1f9df3a9f861d136d6f0c93a6f811c364d1d3d/example/jwks-verify.cpp]. > Support x5u/x5c/x5t Parameter in JSON Web Keys (JWK) > > > Key: IMPALA-12559 > URL: https://issues.apache.org/jira/browse/IMPALA-12559 > Project: IMPALA > Issue Type: Bug > Components: be, Security >Reporter: Jason Fehr >Priority: Critical > Labels: JWT, jwt, security > > The ["x5u"|https://datatracker.ietf.org/doc/html/rfc7517#section-4.6], > ["x5c"|https://datatracker.ietf.org/doc/html/rfc7517#section-4.7], > ["x5t"|https://datatracker.ietf.org/doc/html/rfc7517#section-4.8], and > ["x5t#S256|https://datatracker.ietf.org/doc/html/rfc7517#section-4.9] > parameters in JWKs is not supported by Impala. Implement support for this > parameter using the available methods in the [Thalhammer/jwt-cpp > library|https://github.com/Thalhammer/jwt-cpp/blob/ce1f9df3a9f861d136d6f0c93a6f811c364d1d3d/example/jwks-verify.cpp]. > Note: If the "alg" property is specified and so is "x5u" or "x5c", then the > value of the "alg" property must match the algorithm on the certificate from > the "x5u" or "x5c" property. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-12559) Support x5u/x5c/x5t Parameter in JSON Web Keys (JWK)
[ https://issues.apache.org/jira/browse/IMPALA-12559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Fehr updated IMPALA-12559: Summary: Support x5u/x5c/x5t Parameter in JSON Web Keys (JWK) (was: Support x5c Parameter in JSON Web Keys (JWK)) > Support x5u/x5c/x5t Parameter in JSON Web Keys (JWK) > > > Key: IMPALA-12559 > URL: https://issues.apache.org/jira/browse/IMPALA-12559 > Project: IMPALA > Issue Type: Bug > Components: be, Security >Reporter: Jason Fehr >Priority: Critical > Labels: JWT, jwt, security > > The ["x5c" > parameter|https://datatracker.ietf.org/doc/html/rfc7517#section-4.7] in JWKs > is not supported by Impala. Implement support for this parameter using the > available methods in the [Thalhammer/jwt-cpp > library|https://github.com/Thalhammer/jwt-cpp/blob/ce1f9df3a9f861d136d6f0c93a6f811c364d1d3d/example/jwks-verify.cpp]. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-12841) Write-Ahead Log for Completed Queries
Jason Fehr created IMPALA-12841: --- Summary: Write-Ahead Log for Completed Queries Key: IMPALA-12841 URL: https://issues.apache.org/jira/browse/IMPALA-12841 Project: IMPALA Issue Type: Improvement Components: be Reporter: Jason Fehr Add the ability to set up a write-ahead log (WAL) for the queued up completed queries that have not yet been inserted into the completed queries table. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-12828) Remove Usage of "this->"
[ https://issues.apache.org/jira/browse/IMPALA-12828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Fehr updated IMPALA-12828: Description: The commit [408c606|https://github.com/apache/impala/commit/408c606dd9b891b3abc7f5ce95c95a92334abb8b] added code that used the pattern "this->" unnecessarily. Remove instances of using "this->" from the code changes in this commit. (was: Remove instances of using "this->" from the internal server code.) > Remove Usage of "this->" > > > Key: IMPALA-12828 > URL: https://issues.apache.org/jira/browse/IMPALA-12828 > Project: IMPALA > Issue Type: Sub-task > Components: Backend >Reporter: Jason Fehr >Assignee: Jason Fehr >Priority: Major > > The commit > [408c606|https://github.com/apache/impala/commit/408c606dd9b891b3abc7f5ce95c95a92334abb8b] > added code that used the pattern "this->" unnecessarily. Remove instances > of using "this->" from the code changes in this commit. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-12828) Remove Usage of "this->"
[ https://issues.apache.org/jira/browse/IMPALA-12828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Fehr updated IMPALA-12828: Parent: IMPALA-12426 Issue Type: Sub-task (was: Task) > Remove Usage of "this->" > > > Key: IMPALA-12828 > URL: https://issues.apache.org/jira/browse/IMPALA-12828 > Project: IMPALA > Issue Type: Sub-task > Components: Backend >Reporter: Jason Fehr >Assignee: Jason Fehr >Priority: Major > > Remove instances of using "this->" from the internal server code. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-12426) SQL Interface to Completed Queries/DDLs/DMLs
[ https://issues.apache.org/jira/browse/IMPALA-12426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Fehr updated IMPALA-12426: Issue Type: New Feature (was: Story) > SQL Interface to Completed Queries/DDLs/DMLs > > > Key: IMPALA-12426 > URL: https://issues.apache.org/jira/browse/IMPALA-12426 > Project: IMPALA > Issue Type: New Feature > Components: Backend, be >Reporter: Jason Fehr >Assignee: Jason Fehr >Priority: Major > Labels: impala, workload-management > > Implement a way of querying (via SQL) information about completed > queries/ddls/dmls. Adds coordinator startup flags for users to specify that > Impala will track completed queries in an internal table. > Impala will create and maintain an internal Iceberg table named > "impala_query_log" in the "system database" that contains all completed > queries. This table is automatically created at startup by each coordinator > if it does not exist. Then, each completed query is queued in memory and > flushed to the query history table either at a set interval (user specified > number of minutes) or when a user specified number of completed queries are > queued in memory. Partition this table by the hour of the query end time. > Data in this table must match the corresponding data in the query profile. > Develop automated testing that asserts this requirement is true. > Don't write use, show, and set queries to this table. > Add the following metrics to the "impala-server" metrics group: > * Number of completed queries queued in memory waiting to be written to the > table. > * Number of completed queries successfully written to the table. > * Number of attempts that failed to write completed queries to the table. > * Number of times completed queries were written at the regularly scheduled > time. > * Number of times completed queries were written before the scheduled time > because the max number of queued records was reached. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-12426) SQL Interface to Completed Queries/DDLs/DMLs
[ https://issues.apache.org/jira/browse/IMPALA-12426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Fehr updated IMPALA-12426: Issue Type: New Feature (was: Improvement) > SQL Interface to Completed Queries/DDLs/DMLs > > > Key: IMPALA-12426 > URL: https://issues.apache.org/jira/browse/IMPALA-12426 > Project: IMPALA > Issue Type: New Feature > Components: Backend, be >Reporter: Jason Fehr >Assignee: Jason Fehr >Priority: Major > Labels: impala, workload-management > > Implement a way of querying (via SQL) information about completed > queries/ddls/dmls. Adds coordinator startup flags for users to specify that > Impala will track completed queries in an internal table. > Impala will create and maintain an internal Iceberg table named > "impala_query_log" in the "system database" that contains all completed > queries. This table is automatically created at startup by each coordinator > if it does not exist. Then, each completed query is queued in memory and > flushed to the query history table either at a set interval (user specified > number of minutes) or when a user specified number of completed queries are > queued in memory. Partition this table by the hour of the query end time. > Data in this table must match the corresponding data in the query profile. > Develop automated testing that asserts this requirement is true. > Don't write use, show, and set queries to this table. > Add the following metrics to the "impala-server" metrics group: > * Number of completed queries queued in memory waiting to be written to the > table. > * Number of completed queries successfully written to the table. > * Number of attempts that failed to write completed queries to the table. > * Number of times completed queries were written at the regularly scheduled > time. > * Number of times completed queries were written before the scheduled time > because the max number of queued records was reached. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-12426) SQL Interface to Completed Queries/DDLs/DMLs
[ https://issues.apache.org/jira/browse/IMPALA-12426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Fehr updated IMPALA-12426: Issue Type: Story (was: New Feature) > SQL Interface to Completed Queries/DDLs/DMLs > > > Key: IMPALA-12426 > URL: https://issues.apache.org/jira/browse/IMPALA-12426 > Project: IMPALA > Issue Type: Story > Components: Backend, be >Reporter: Jason Fehr >Assignee: Jason Fehr >Priority: Major > Labels: impala, workload-management > > Implement a way of querying (via SQL) information about completed > queries/ddls/dmls. Adds coordinator startup flags for users to specify that > Impala will track completed queries in an internal table. > Impala will create and maintain an internal Iceberg table named > "impala_query_log" in the "system database" that contains all completed > queries. This table is automatically created at startup by each coordinator > if it does not exist. Then, each completed query is queued in memory and > flushed to the query history table either at a set interval (user specified > number of minutes) or when a user specified number of completed queries are > queued in memory. Partition this table by the hour of the query end time. > Data in this table must match the corresponding data in the query profile. > Develop automated testing that asserts this requirement is true. > Don't write use, show, and set queries to this table. > Add the following metrics to the "impala-server" metrics group: > * Number of completed queries queued in memory waiting to be written to the > table. > * Number of completed queries successfully written to the table. > * Number of attempts that failed to write completed queries to the table. > * Number of times completed queries were written at the regularly scheduled > time. > * Number of times completed queries were written before the scheduled time > because the max number of queued records was reached. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-12828) Remove Usage of "this->"
Jason Fehr created IMPALA-12828: --- Summary: Remove Usage of "this->" Key: IMPALA-12828 URL: https://issues.apache.org/jira/browse/IMPALA-12828 Project: IMPALA Issue Type: Task Components: Backend Reporter: Jason Fehr Assignee: Jason Fehr Remove instances of using "this->" from the internal server code. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Closed] (IMPALA-12824) Add Built-in Functions to Pretty Print Duration and Bytes
[ https://issues.apache.org/jira/browse/IMPALA-12824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Fehr closed IMPALA-12824. --- Resolution: Fixed > Add Built-in Functions to Pretty Print Duration and Bytes > - > > Key: IMPALA-12824 > URL: https://issues.apache.org/jira/browse/IMPALA-12824 > Project: IMPALA > Issue Type: Improvement > Components: be >Reporter: Jason Fehr >Assignee: Jason Fehr >Priority: Major > Labels: backend, functions > > Implement new built-in Impala string functions to pretty print time duration > from an input of nanoseconds and pretty print a memory value from an input of > bytes. > For example, pretty printing a duration of 2147483648 nanoseconds would > output "2s147ms". Pretty printing a memory value of 32768 would output > "32.00 KB". -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-12824) Add Built-in Functions to Pretty Print Duration and Bytes
[ https://issues.apache.org/jira/browse/IMPALA-12824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Fehr updated IMPALA-12824: Summary: Add Built-in Functions to Pretty Print Duration and Bytes (was: Add Built-in Functions to Pretty Print Duration and Bytes.) > Add Built-in Functions to Pretty Print Duration and Bytes > - > > Key: IMPALA-12824 > URL: https://issues.apache.org/jira/browse/IMPALA-12824 > Project: IMPALA > Issue Type: Improvement > Components: be >Reporter: Jason Fehr >Assignee: Jason Fehr >Priority: Major > Labels: backend, functions > > Implement new built-in Impala string functions to pretty print time duration > from an input of nanoseconds and pretty print a memory value from an input of > bytes. > For example, pretty printing a duration of 2147483648 nanoseconds would > output "2s147ms". Pretty printing a memory value of 32768 would output > "32.00 KB". -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-12824) Add Built-in Functions to Pretty Print Duration and Bytes.
Jason Fehr created IMPALA-12824: --- Summary: Add Built-in Functions to Pretty Print Duration and Bytes. Key: IMPALA-12824 URL: https://issues.apache.org/jira/browse/IMPALA-12824 Project: IMPALA Issue Type: Improvement Components: be Reporter: Jason Fehr Assignee: Jason Fehr Implement new built-in Impala string functions to pretty print time duration from an input of nanoseconds and pretty print a memory value from an input of bytes. For example, pretty printing a duration of 2147483648 nanoseconds would output "2s147ms". Pretty printing a memory value of 32768 would output "32.00 KB". -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-12824) Add Built-in Functions to Pretty Print Duration and Bytes.
[ https://issues.apache.org/jira/browse/IMPALA-12824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-12824 started by Jason Fehr. --- > Add Built-in Functions to Pretty Print Duration and Bytes. > -- > > Key: IMPALA-12824 > URL: https://issues.apache.org/jira/browse/IMPALA-12824 > Project: IMPALA > Issue Type: Improvement > Components: be >Reporter: Jason Fehr >Assignee: Jason Fehr >Priority: Major > Labels: backend, functions > > Implement new built-in Impala string functions to pretty print time duration > from an input of nanoseconds and pretty print a memory value from an input of > bytes. > For example, pretty printing a duration of 2147483648 nanoseconds would > output "2s147ms". Pretty printing a memory value of 32768 would output > "32.00 KB". -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-12768) Set iceberg.engine.hive.lock-enabled To false By Default
Jason Fehr created IMPALA-12768:
---
Summary: Set iceberg.engine.hive.lock-enabled To false By Default
Key: IMPALA-12768
URL: https://issues.apache.org/jira/browse/IMPALA-12768
Project: IMPALA
Issue Type: Improvement
Components: be
Reporter: Jason Fehr
[Iceberg
documentation|https://iceberg.apache.org/docs/latest/configuration/#hadoop-configuration]
says to set the property "iceberg.engine.hive.lock-enabled
{noformat}
Warn: Setting iceberg.engine.hive.lock-enabled=false will cause HiveCatalog to
commit to tables without using Hive locks. This should only be set to false if
all following conditions are met:
* HIVE-26882 is available on the Hive Metastore server
* All other HiveCatalogs committing to tables that this HiveCatalog commits to
are also on Iceberg 1.3 or later
* All other HiveCatalogs committing to tables that this HiveCatalog commits to
have also disabled Hive locks on commit.
Failing to ensure these conditions risks corrupting the table.
{noformat}
Setting this property to false reduces the possibility of stuck table locks
where the lock in HMS does not get released because the Impala process that
took the lock was ungracefully killed before it released the lock.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-12747) internal-server-test crashes with SIGSEGV on ARM in release build
[
https://issues.apache.org/jira/browse/IMPALA-12747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17810136#comment-17810136
]
Jason Fehr commented on IMPALA-12747:
-
Excellent! I was thinking this build failure was happening frequently.
> internal-server-test crashes with SIGSEGV on ARM in release build
> -
>
> Key: IMPALA-12747
> URL: https://issues.apache.org/jira/browse/IMPALA-12747
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 4.4.0
>Reporter: Laszlo Gaal
>Assignee: Michael Smith
>Priority: Blocker
> Labels: arm64, broken-build
>
> internal-server-test crashed in
> InternalServerTest.SimultaneousMultipleQueriesOneSession with minidumps in a
> release-mode build on ARM.
> The minidump was resolved to the below call stack:
> {code}
> Operating system: Linux
> 0.0.0 Linux 4.18.0-477.15.1.el8_8.aarch64 #1 SMP Fri Jun 2
> 08:39:44 EDT 2023 aarch64
> CPU: arm64
> 16 CPUs
> GPU: UNKNOWN
> Crash reason: SIGSEGV /SEGV_MAPERR
> Crash address: 0x8
> Process uptime: not available
> Thread 538 (crashed)
> 0 internal-server-test!impala::TPlanNode::TPlanNode(impala::TPlanNode
> const&) [PlanNodes_types.cpp : 3124 + 0x0]
> x0 = 0x4d600020x1 = 0x0020
> x2 = 0xx3 = 0x
> x4 = 0x4d600900x5 = 0x4d600a00
> x6 = 0xx7 = 0x4d600918
> x8 = 0x4d600060x9 = 0x4d600828
> x10 = 0x04a3ab48 x11 = 0x4d600838
> x12 = 0x4d600868 x13 = 0x4d6007f8
> x14 = 0x4d600818 x15 = 0x4d6007d0
> x16 = 0x4d600848 x17 = 0x4d600878
> x18 = 0x04a3ab98 x19 = 0x4d60
> x20 = 0x x21 = 0x4d600098
> x22 = 0x4d6001e0 x23 = 0x4d600038
> x24 = 0x4d600190 x25 = 0x4d6003d8
> x26 = 0x4d600560 x27 = 0x4d600080
> x28 = 0x4d6008b8fp = 0xfffe4df61990
> lr = 0x4d600798sp = 0xfffe4df61990
> pc = 0x01023dc4
> Found by: given as instruction pointer in context
> 1 internal-server-test!std::vector std::allocator >::operator=(std::vector std::allocator > const&) [stl_construct.h : 109 + 0x8]
> x19 = 0x x20 = 0x4d60
> x21 = 0x4497b038 x22 = 0x4d60
> x23 = 0x45a9bb30 x24 = 0x45a9bb30
> x25 = 0x4497b020 x26 = 0x390da838
> x27 = 0x461d7000 x28 = 0xfffe4df62018
> fp = 0xfffe4df61a50sp = 0xfffe4df61a50
> pc = 0x010242f4
> Found by: call frame info
> 2 internal-server-test!impala::TPlan::operator=(impala::TPlan const&)
> [PlanNodes_types.cpp : 3305 + 0x0]
> x19 = 0x4497b030 x20 = 0x390da800
> x21 = 0x4497b010 x22 = 0x4497b400
> x23 = 0x4497b050 x24 = 0x4497b328
> x25 = 0x4497b020 x26 = 0x4497b030
> x27 = 0x461d7000 x28 = 0xfffe4df62018
> fp = 0xfffe4df61aa0sp = 0xfffe4df61aa0
> pc = 0x010244ac
> Found by: call frame info
> 3
> internal-server-test!impala::TPlanFragment::TPlanFragment(impala::TPlanFragment
> const&) [Planner_types.cpp : 110 + 0x8]
> x19 = 0x4497b000 x20 = 0x390da800
> x21 = 0x4497b010 x22 = 0x4497b400
> x23 = 0x4497b050 x24 = 0x4497b328
> x25 = 0x4497b020 x26 = 0x4497b030
> x27 = 0x461d7000 x28 = 0xfffe4df62018
> fp = 0xfffe4df61ac0sp = 0xfffe4df61ac0
> pc = 0x0102cd40
> Found by: call frame info
> 4
> internal-server-test!impala::ImpalaServer::QueryStateRecord::Init(impala::ClientRequestState
> const&) [stl_construct.h : 109 + 0x8]
> x19 = 0x390da800 x20 = 0x4497b000
> x21 = 0x390daf30 x22 = 0x0730
> x23 = 0x4497b000 x24 = 0xfffe4df61d50
> x25 = 0x x26 = 0x462ed320
> x27 = 0x461d7000 x28 = 0xfffe4df62018
> fp = 0xfffe4df61b10sp = 0xfffe4df61b10
> pc = 0x014d80a4
> Found by: call frame info
> 5
> internal-server-test!impala::ImpalaServer::QueryStateRecord::QueryStateRecord(impala::ClientRequestState
> const&) [impala-server.cc : 2529 + 0x0]
> x19 = 0xfffe4df61d50 x20 = 0xfffe4df61eb8
> x21 = 0xfffe4df61df8 x22 = 0xfffe4df61e38
> x23 = 0xfffe4df61e48 x24 = 0xfffe4df61e98
> x25 = 0xfffe4
[jira] [Commented] (IMPALA-12747) internal-server-test crashes with SIGSEGV on ARM in release build
[
https://issues.apache.org/jira/browse/IMPALA-12747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17810132#comment-17810132
]
Jason Fehr commented on IMPALA-12747:
-
[~MikaelSmith] -- given that this race condition has existed for a long time
and not caused any problems, would it be sufficient to perform synchronization
in the test itself such that the Impala webserver is not called at the exact
same time queries are running?
> internal-server-test crashes with SIGSEGV on ARM in release build
> -
>
> Key: IMPALA-12747
> URL: https://issues.apache.org/jira/browse/IMPALA-12747
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 4.4.0
>Reporter: Laszlo Gaal
>Assignee: Michael Smith
>Priority: Blocker
> Labels: arm64, broken-build
>
> internal-server-test crashed with minidumps in a release-mode build on ARM.
> The minidump was resolved to the below call stack:
> {code}
> Operating system: Linux
> 0.0.0 Linux 4.18.0-477.15.1.el8_8.aarch64 #1 SMP Fri Jun 2
> 08:39:44 EDT 2023 aarch64
> CPU: arm64
> 16 CPUs
> GPU: UNKNOWN
> Crash reason: SIGSEGV /SEGV_MAPERR
> Crash address: 0x8
> Process uptime: not available
> Thread 538 (crashed)
> 0 internal-server-test!impala::TPlanNode::TPlanNode(impala::TPlanNode
> const&) [PlanNodes_types.cpp : 3124 + 0x0]
> x0 = 0x4d600020x1 = 0x0020
> x2 = 0xx3 = 0x
> x4 = 0x4d600900x5 = 0x4d600a00
> x6 = 0xx7 = 0x4d600918
> x8 = 0x4d600060x9 = 0x4d600828
> x10 = 0x04a3ab48 x11 = 0x4d600838
> x12 = 0x4d600868 x13 = 0x4d6007f8
> x14 = 0x4d600818 x15 = 0x4d6007d0
> x16 = 0x4d600848 x17 = 0x4d600878
> x18 = 0x04a3ab98 x19 = 0x4d60
> x20 = 0x x21 = 0x4d600098
> x22 = 0x4d6001e0 x23 = 0x4d600038
> x24 = 0x4d600190 x25 = 0x4d6003d8
> x26 = 0x4d600560 x27 = 0x4d600080
> x28 = 0x4d6008b8fp = 0xfffe4df61990
> lr = 0x4d600798sp = 0xfffe4df61990
> pc = 0x01023dc4
> Found by: given as instruction pointer in context
> 1 internal-server-test!std::vector std::allocator >::operator=(std::vector std::allocator > const&) [stl_construct.h : 109 + 0x8]
> x19 = 0x x20 = 0x4d60
> x21 = 0x4497b038 x22 = 0x4d60
> x23 = 0x45a9bb30 x24 = 0x45a9bb30
> x25 = 0x4497b020 x26 = 0x390da838
> x27 = 0x461d7000 x28 = 0xfffe4df62018
> fp = 0xfffe4df61a50sp = 0xfffe4df61a50
> pc = 0x010242f4
> Found by: call frame info
> 2 internal-server-test!impala::TPlan::operator=(impala::TPlan const&)
> [PlanNodes_types.cpp : 3305 + 0x0]
> x19 = 0x4497b030 x20 = 0x390da800
> x21 = 0x4497b010 x22 = 0x4497b400
> x23 = 0x4497b050 x24 = 0x4497b328
> x25 = 0x4497b020 x26 = 0x4497b030
> x27 = 0x461d7000 x28 = 0xfffe4df62018
> fp = 0xfffe4df61aa0sp = 0xfffe4df61aa0
> pc = 0x010244ac
> Found by: call frame info
> 3
> internal-server-test!impala::TPlanFragment::TPlanFragment(impala::TPlanFragment
> const&) [Planner_types.cpp : 110 + 0x8]
> x19 = 0x4497b000 x20 = 0x390da800
> x21 = 0x4497b010 x22 = 0x4497b400
> x23 = 0x4497b050 x24 = 0x4497b328
> x25 = 0x4497b020 x26 = 0x4497b030
> x27 = 0x461d7000 x28 = 0xfffe4df62018
> fp = 0xfffe4df61ac0sp = 0xfffe4df61ac0
> pc = 0x0102cd40
> Found by: call frame info
> 4
> internal-server-test!impala::ImpalaServer::QueryStateRecord::Init(impala::ClientRequestState
> const&) [stl_construct.h : 109 + 0x8]
> x19 = 0x390da800 x20 = 0x4497b000
> x21 = 0x390daf30 x22 = 0x0730
> x23 = 0x4497b000 x24 = 0xfffe4df61d50
> x25 = 0x x26 = 0x462ed320
> x27 = 0x461d7000 x28 = 0xfffe4df62018
> fp = 0xfffe4df61b10sp = 0xfffe4df61b10
> pc = 0x014d80a4
> Found by: call frame info
> 5
> internal-server-test!impala::ImpalaServer::QueryStateRecord::QueryStateRecord(impala::ClientRequestState
> const&) [impala-server.cc : 2529 + 0x0]
> x19 = 0xfffe4df61d50 x20 = 0xfffe4df61eb8
> x21
[jira] [Updated] (IMPALA-4024) Expose Impala metrics as a table
[ https://issues.apache.org/jira/browse/IMPALA-4024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Fehr updated IMPALA-4024: --- Labels: supportability usability workload-management (was: supportability usability) > Expose Impala metrics as a table > > > Key: IMPALA-4024 > URL: https://issues.apache.org/jira/browse/IMPALA-4024 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Impala 2.7.0 >Reporter: Tim Armstrong >Priority: Major > Labels: supportability, usability, workload-management > Attachments: 2cb86db8.diff, ce6933d3.diff, d556ca9.diff > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-4024) Expose Impala metrics as a table
[ https://issues.apache.org/jira/browse/IMPALA-4024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Fehr updated IMPALA-4024: --- Epic Link: (was: IMPALA-12427) > Expose Impala metrics as a table > > > Key: IMPALA-4024 > URL: https://issues.apache.org/jira/browse/IMPALA-4024 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Affects Versions: Impala 2.7.0 >Reporter: Tim Armstrong >Priority: Major > Labels: supportability, usability > Attachments: 2cb86db8.diff, ce6933d3.diff, d556ca9.diff > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-3984) Impala should provide some interface for showing running config/settings
[ https://issues.apache.org/jira/browse/IMPALA-3984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Fehr updated IMPALA-3984: --- Epic Link: (was: IMPALA-12427) Labels: workload-management (was: ) > Impala should provide some interface for showing running config/settings > > > Key: IMPALA-3984 > URL: https://issues.apache.org/jira/browse/IMPALA-3984 > Project: IMPALA > Issue Type: New Feature > Components: Clients >Affects Versions: Impala 2.7.0 >Reporter: David Knupp >Priority: Minor > Labels: workload-management > > This came about during the course of designing a way to skip Impala > LZO-dependent tests in the case that Impala was not build with LZO-support. > We didn't have a way to query for this, and wound up experimenting with > relying on environment variables and/or the existence of the libimpalalzo.so > file. > A simple way to query Impala itself would have been much simpler and more > reliable. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-12676) UBSAN intermittent int overflow on internal-server-test
[
https://issues.apache.org/jira/browse/IMPALA-12676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-12676 started by Jason Fehr.
---
> UBSAN intermittent int overflow on internal-server-test
> ---
>
> Key: IMPALA-12676
> URL: https://issues.apache.org/jira/browse/IMPALA-12676
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 4.4.0
>Reporter: Michael Smith
>Assignee: Jason Fehr
>Priority: Major
>
> internal-server-test intermittently fails UBSAN runs with
> {code}
> [ RUN ] InternalServerTest.SimultaneousMultipleQueriesOneSession
> /data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/be/src/service/internal-server-test.cc:149:42:
> runtime error: signed integer overflow: 31536000 * 81 cannot be represented
> in type 'int'
> #0 0x1f5a6dc in
> impala::internalservertest::DatabaseTest::DatabaseTest(std::shared_ptr,
> std::__cxx11::basic_string,
> std::allocator >, bool, int)
> /data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/be/src/service/internal-server-test.cc:149:42
> #1 0x1f514f4 in
> impala::internalservertest::InternalServerTest_SimultaneousMultipleQueriesOneSession_Test::TestBody()
>
> /data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/be/src/service/internal-server-test.cc:487:26
> #2 0x8f0161c in void
> testing::internal::HandleExceptionsInMethodIfSupported void>(testing::Test*, void (testing::Test::*)(), char const*)
> (/data0/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/be/build/debug/service/internal-server-test+0x8f0161c)
> #3 0x8efa534 in testing::Test::Run()
> (/data0/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/be/build/debug/service/internal-server-test+0x8efa534)
> #4 0x8efa677 in testing::TestInfo::Run()
> (/data0/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/be/build/debug/service/internal-server-test+0x8efa677)
> #5 0x8efa754 in testing::TestCase::Run()
> (/data0/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/be/build/debug/service/internal-server-test+0x8efa754)
> #6 0x8efac4b in testing::internal::UnitTestImpl::RunAllTests()
> (/data0/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/be/build/debug/service/internal-server-test+0x8efac4b)
> #7 0x8efacf2 in testing::UnitTest::Run()
> (/data0/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/be/build/debug/service/internal-server-test+0x8efacf2)
> #8 0x1f535e5 in main
> /data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/be/src/service/internal-server-test.cc:547:22
> #9 0x7f3802a81554 in __libc_start_main (/lib64/libc.so.6+0x22554)
> #10 0x1f21f26 in _start
> (/data0/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/be/build/debug/service/internal-server-test+0x1f21f26)
> /data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/be/src/service/internal-server-test.cc:172:36:
> runtime error: signed integer overflow: 31536000 * 100 cannot be represented
> in type 'int'
> #0 0x1f5b1fd in
> impala::internalservertest::DatabaseTest::DatabaseTest(std::shared_ptr,
> std::__cxx11::basic_string,
> std::allocator >, bool, int)
> /data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/be/src/service/internal-server-test.cc:172:36
> #1 0x1f514f4 in
> impala::internalservertest::InternalServerTest_SimultaneousMultipleQueriesOneSession_Test::TestBody()
>
> /data/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/be/src/service/internal-server-test.cc:487:26
> #2 0x8f0161c in void
> testing::internal::HandleExceptionsInMethodIfSupported void>(testing::Test*, void (testing::Test::*)(), char const*)
> (/data0/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/be/build/debug/service/internal-server-test+0x8f0161c)
> #3 0x8efa534 in testing::Test::Run()
> (/data0/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/be/build/debug/service/internal-server-test+0x8efa534)
> #4 0x8efa677 in testing::TestInfo::Run()
> (/data0/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/be/build/debug/service/internal-server-test+0x8efa677)
> #5 0x8efa754 in testing::TestCase::Run()
> (/data0/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/be/build/debug/service/internal-server-test+0x8efa754)
> #6 0x8efac4b in testing::internal::UnitTestImpl::RunAllTests()
> (/data0/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/be/build/debug/service/internal-server-test+0x8efac4b)
> #7 0x8efacf2 in testing::UnitTest::Run()
> (/data0/jenkins/workspace/impala-asf-master-core-ubsan/repos/Impala/be/build/debug/service/internal-server-test+0x8efacf2)
> #8 0x1f535e5 in main
> /data/jenkins/workspace/impala-asf-master
[jira] [Updated] (IMPALA-12648) Support killing queries and sessions programatically
[
https://issues.apache.org/jira/browse/IMPALA-12648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Fehr updated IMPALA-12648:
Epic Link: IMPALA-12427
> Support killing queries and sessions programatically
>
>
> Key: IMPALA-12648
> URL: https://issues.apache.org/jira/browse/IMPALA-12648
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
>Affects Versions: Impala 4.3.0
>Reporter: Manish Maheshwari
>Assignee: Michael Smith
>Priority: Major
>
> Support killing queries and sessions programatically via kill commands to be
> able to manage impala running workloads.
> 1. Killing Queries that are currently running
> {code:java}
> -- Forcibly terminates query with the specified query_id:
> KILL QUERY WHERE query_id='634bf9fcf55278eb:ac0ef053' {code}
> For queries that are the finished and waiting to be closed, this command
> should close them
> 2. Killing sessions that are open
> {code:java}
> -- Forcibly terminates session and closes all queries:
> KILL SESSION WHERE session_id='2644d52c79c4c1e4:e974538f2189ed82' {code}
> this command should terminate the session and kill all active queries and
> close all queries that are finished and are waiting to be closed.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-12652) Limit Length of Completed Queries Insert DML
Jason Fehr created IMPALA-12652: --- Summary: Limit Length of Completed Queries Insert DML Key: IMPALA-12652 URL: https://issues.apache.org/jira/browse/IMPALA-12652 Project: IMPALA Issue Type: Improvement Components: be Reporter: Jason Fehr Assignee: Jason Fehr Implement a coordinator startup flag that limits the max length (number of characters) of the insert DML statement that inserts records into the impala_query_log table. The purpose of this flag is to ensure that workload management does not generate an insert DML statement that exceeds Impala's max length for a sql statement (approximately 16 megabytes or 16 million characters). -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-12624) Create a System Database for Impala
[ https://issues.apache.org/jira/browse/IMPALA-12624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Fehr reassigned IMPALA-12624: --- Assignee: Jason Fehr > Create a System Database for Impala > --- > > Key: IMPALA-12624 > URL: https://issues.apache.org/jira/browse/IMPALA-12624 > Project: IMPALA > Issue Type: Improvement > Components: be >Reporter: Jason Fehr >Assignee: Jason Fehr >Priority: Major > Labels: workload-management > > Impala must create a database named "system" for storing Impala related > tables (such as query history). -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-12627) Include Expanded SQL in Query Log Table
Jason Fehr created IMPALA-12627: --- Summary: Include Expanded SQL in Query Log Table Key: IMPALA-12627 URL: https://issues.apache.org/jira/browse/IMPALA-12627 Project: IMPALA Issue Type: Improvement Components: be Reporter: Jason Fehr The current sql stored in the query log table does not have views expanded. Include another sql statement that replaces each referenced view with that view's sql statement. This sql statement must recursively expand the views. If a view references another view, then expand the other view into it's sql statement. Also, add another list of all references tables which will contain all tables referenced in the original query plus all tables referenced in the views. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-12624) Create a System Database for Impala
[ https://issues.apache.org/jira/browse/IMPALA-12624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-12624 started by Jason Fehr. --- > Create a System Database for Impala > --- > > Key: IMPALA-12624 > URL: https://issues.apache.org/jira/browse/IMPALA-12624 > Project: IMPALA > Issue Type: Improvement > Components: be >Reporter: Jason Fehr >Assignee: Jason Fehr >Priority: Major > Labels: workload-management > > Impala must create a database named "system" for storing Impala related > tables (such as query history). -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-12626) Include List of Referenced Tables/Views in Query Log Table
Jason Fehr created IMPALA-12626: --- Summary: Include List of Referenced Tables/Views in Query Log Table Key: IMPALA-12626 URL: https://issues.apache.org/jira/browse/IMPALA-12626 Project: IMPALA Issue Type: Improvement Components: be Reporter: Jason Fehr In the Impala query log table where completed queries are stored, add a list of all tables and views that were referenced in the query. The purpose behind this functionality is for end users to be able to analyze how often each table is used in queries. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-12625) Expire Old Query Log Records
Jason Fehr created IMPALA-12625: --- Summary: Expire Old Query Log Records Key: IMPALA-12625 URL: https://issues.apache.org/jira/browse/IMPALA-12625 Project: IMPALA Issue Type: Improvement Components: be Reporter: Jason Fehr Add the ability for users to specify a number of days after which records should be purged from the query log table (that stores completed queries). Update Impala to automatically purge records older than the set threshold. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-12426) SQL Interface to Completed Queries/DDLs/DMLs
[ https://issues.apache.org/jira/browse/IMPALA-12426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Fehr updated IMPALA-12426: Description: Implement a way of querying (via SQL) information about completed queries/ddls/dmls. Adds coordinator startup flags for users to specify that Impala will track completed queries in an internal table. Impala will create and maintain an internal Iceberg table named "impala_query_log" in the "system database" that contains all completed queries. This table is automatically created at startup by each coordinator if it does not exist. Then, each completed query is queued in memory and flushed to the query history table either at a set interval (user specified number of minutes) or when a user specified number of completed queries are queued in memory. Partition this table by the hour of the query end time. Data in this table must match the corresponding data in the query profile. Develop automated testing that asserts this requirement is true. Don't write use, show, and set queries to this table. Add the following metrics to the "impala-server" metrics group: * Number of completed queries queued in memory waiting to be written to the table. * Number of completed queries successfully written to the table. * Number of attempts that failed to write completed queries to the table. * Number of times completed queries were written at the regularly scheduled time. * Number of times completed queries were written before the scheduled time because the max number of queued records was reached. was: Implement a way of querying (via SQL) information about completed queries/ddls/dmls. Adds coordinator startup flags for users to specify that Impala will track completed queries in an internal table. Impala will create and maintain an internal Iceberg table named "impala_query_log" in the "system database" that contains all completed queries. This table is automatically created at startup by each coordinator if it does not exist. Then, each completed query is queued in memory and flushed to the query history table at a set interval (either minutes or number of records). Data in this table must match the corresponding data in the query profile. Develop automated testing that asserts this requirement is true. Add the following metrics to the "impala-server" metrics group: * Number of completed queries queued in memory waiting to be written to the table. * Number of completed queries successfully written to the table. * Number of attempts that failed to write completed queries to the table. * Number of times completed queries were written at the regularly scheduled time. * Number of times completed queries were written before the scheduled time because the max number of queued records was reached. > SQL Interface to Completed Queries/DDLs/DMLs > > > Key: IMPALA-12426 > URL: https://issues.apache.org/jira/browse/IMPALA-12426 > Project: IMPALA > Issue Type: Improvement > Components: Backend, be >Reporter: Jason Fehr >Assignee: Jason Fehr >Priority: Major > Labels: impala, workload-management > > Implement a way of querying (via SQL) information about completed > queries/ddls/dmls. Adds coordinator startup flags for users to specify that > Impala will track completed queries in an internal table. > Impala will create and maintain an internal Iceberg table named > "impala_query_log" in the "system database" that contains all completed > queries. This table is automatically created at startup by each coordinator > if it does not exist. Then, each completed query is queued in memory and > flushed to the query history table either at a set interval (user specified > number of minutes) or when a user specified number of completed queries are > queued in memory. Partition this table by the hour of the query end time. > Data in this table must match the corresponding data in the query profile. > Develop automated testing that asserts this requirement is true. > Don't write use, show, and set queries to this table. > Add the following metrics to the "impala-server" metrics group: > * Number of completed queries queued in memory waiting to be written to the > table. > * Number of completed queries successfully written to the table. > * Number of attempts that failed to write completed queries to the table. > * Number of times completed queries were written at the regularly scheduled > time. > * Number of times completed queries were written before the scheduled time > because the max number of queued records was reached. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For ad
