[
https://issues.apache.org/jira/browse/IMPALA-12191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18078786#comment-18078786
]
ASF subversion and git services commented on IMPALA-12191:
----------------------------------------------------------
Commit a44f72d8f80677509a98439a5b46fa2cbb933db7 in impala's branch
refs/heads/master from Arnab Karmakar
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=a44f72d8f ]
IMPALA-12191: Add hardware and OS details to runtime profile
This patch adds backend hardware (CPU model, core count) and OS
distribution information to the runtime profile. The information
is displayed in two places:
1. Per-node profiles: Each backend's CPU and OS info appears at
the top of its host profile for easy correlation with performance
2. Aggregated summary: Unique configurations are grouped with counts
Example profile output:
- Per-node profiles (CPU/OS info at top of each host):
Per Node Profiles:
ccycloud.impala-host.root.comops.site:27000:
OS Info: Ubuntu 22.04.5 LTS
CPU Info: Intel(R) Xeon(R) Silver 4215R CPU @ 3.20GHz (32 cores)
- HostCpuIoWaitPercentage (50.000ms): 0, 0, 0, 0
- HostCpuSysPercentage (50.000ms): 3, 4, 4, 4
...
ccycloud.impala-host.root.comops.site:27001:
OS Info: Ubuntu 22.04.5 LTS
CPU Info: Intel(R) Xeon(R) Silver 4215R CPU @ 3.20GHz (32 cores)
- HostCpuIoWaitPercentage (50.000ms): 0, 0, 0, 0
...
- Aggregated summary (for quick overview):
Per Node Peak Memory Usage: host:27000(400 KB) host:27001(352 KB)
host:27002(352 KB)
Per Node Bytes Read: host:27000(161 KB) host:27001(158 KB) host:27002(158 KB)
Backend CPU Info: Intel(R) Xeon(R) Silver 4215R CPU @ 3.20GHz (32 cores) (3)
Backend OS Info: Ubuntu 22.04.5 LTS (3)
- Heterogeneous cluster example:
Backend CPU Info: Intel Xeon E5-2680 v4 (28 cores) (15), Intel Xeon E5-2660
v3 (20 cores) (3)
Backend OS Info: CentOS Linux 7 (15), Ubuntu 20.04.3 LTS (3)
Testing:
- Added Python test test_backend_hardware_info_in_profile() that validates
per-node and aggregated hardware info
- Added C++ unit tests for BackendHardwareInfo and
BackendHardwareInfoAggregator covering single/multiple configs, invalid
data handling, and deterministic ordering
- Manually tested with 3-node minicluster
Change-Id: Ib5a093b6cf9dd7dfa4321b79aeea95d6fc748dd4
Reviewed-on: http://gerrit.cloudera.org:8080/23950
Tested-by: Impala Public Jenkins <[email protected]>
Reviewed-by: Joe McDonnell <[email protected]>
> Add hardware and OS details to runtime profile
> ----------------------------------------------
>
> Key: IMPALA-12191
> URL: https://issues.apache.org/jira/browse/IMPALA-12191
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Reporter: David Rorke
> Assignee: Arnab Karmakar
> Priority: Major
> Labels: ramp-up
>
> The runtime profiles are currently lacking any details about the hardware the
> query ran on (CPU model and core count, cache sizes, etc) OS versions, etc
> which may all be relevant when analyzing performance issues, comparing
> performance metrics across different profiles, etc.
> We should add relevant hardware and OS details to the profile. The
> information currently displayed at the root Impalad web UI page (Hardware
> Info, OS Info) would be a good starting point.
> IMPALA-12118 is also relevant if we want to cover ARM processors.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
