gerashegalov commented on code in PR #8770:
URL: https://github.com/apache/hadoop/pull/8770#discussion_r4188802157
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/timeline/TimelineUtils.java:
##########
@@ -60,6 +72,57 @@ public class TimelineUtils {
YarnJacksonJaxbJsonProvider.configObjectMapper(mapper);
}
+ @Private
+ public static Map<String, Map<String, Object>> getCustomResourceInfo(
+ Resource resource) {
+ Map<String, Map<String, Object>> resources = new HashMap<>();
+ for (ResourceInformation information : resource.getResources()) {
+ String name = information.getName();
+ if (!ResourceInformation.MEMORY_URI.equals(name)
+ && !ResourceInformation.VCORES_URI.equals(name)) {
+ Map<String, Object> allocation = new HashMap<>();
+ allocation.put("value", information.getValue());
+ allocation.put("units", information.getUnits());
+ resources.put(name, allocation);
+ }
+ }
+ return resources;
+ }
+
+ @Private
+ public static Resource getContainerResource(Map<String, Object> entityInfo) {
+ if (entityInfo == null) {
+ return Resource.newInstance(0, 0);
+ }
+ long memory = ((Number) entityInfo.getOrDefault(
+ ContainerMetricsConstants.ALLOCATED_MEMORY_INFO, 0L)).longValue();
+ int vcores = ((Number) entityInfo.getOrDefault(
+ ContainerMetricsConstants.ALLOCATED_VCORE_INFO, 0)).intValue();
+ Resource resource = Resource.newInstance(memory, vcores);
+ Map<?, ?> allocations = (Map<?, ?>) entityInfo.get(
+ ContainerMetricsConstants.ALLOCATED_RESOURCES_INFO);
+ if (allocations != null) {
+ for (Map.Entry<?, ?> entry : allocations.entrySet()) {
+ String name = (String) entry.getKey();
+ if (ResourceInformation.MEMORY_URI.equals(name)
+ || ResourceInformation.VCORES_URI.equals(name)) {
+ continue;
+ }
+ if (!ResourceUtils.getResourceTypes().containsKey(name)) {
+ LOG.warn("Skipping unknown resource type {} in container history",
name);
+ continue;
+ }
+ Map<?, ?> allocation = (Map<?, ?>) entry.getValue();
Review Comment:
Thanks for the concrete cases. Addressed in 48b26fe: malformed custom
entries are skipped independently so later valid allocations survive. The tests
cover missing values, invalid shapes, unknown units, conversion overflow, and
invalid numeric values. The separate pre-existing UnitsConversionUtil issue is
tracked as YARN-12007.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/timeline/TimelineUtils.java:
##########
@@ -60,6 +72,57 @@ public class TimelineUtils {
YarnJacksonJaxbJsonProvider.configObjectMapper(mapper);
}
+ @Private
+ public static Map<String, Map<String, Object>> getCustomResourceInfo(
+ Resource resource) {
+ Map<String, Map<String, Object>> resources = new HashMap<>();
+ for (ResourceInformation information : resource.getResources()) {
+ String name = information.getName();
+ if (!ResourceInformation.MEMORY_URI.equals(name)
+ && !ResourceInformation.VCORES_URI.equals(name)) {
+ Map<String, Object> allocation = new HashMap<>();
+ allocation.put("value", information.getValue());
+ allocation.put("units", information.getUnits());
+ resources.put(name, allocation);
+ }
+ }
+ return resources;
+ }
+
+ @Private
+ public static Resource getContainerResource(Map<String, Object> entityInfo) {
+ if (entityInfo == null) {
+ return Resource.newInstance(0, 0);
+ }
+ long memory = ((Number) entityInfo.getOrDefault(
+ ContainerMetricsConstants.ALLOCATED_MEMORY_INFO, 0L)).longValue();
+ int vcores = ((Number) entityInfo.getOrDefault(
+ ContainerMetricsConstants.ALLOCATED_VCORE_INFO, 0)).intValue();
+ Resource resource = Resource.newInstance(memory, vcores);
+ Map<?, ?> allocations = (Map<?, ?>) entityInfo.get(
+ ContainerMetricsConstants.ALLOCATED_RESOURCES_INFO);
+ if (allocations != null) {
+ for (Map.Entry<?, ?> entry : allocations.entrySet()) {
+ String name = (String) entry.getKey();
+ if (ResourceInformation.MEMORY_URI.equals(name)
+ || ResourceInformation.VCORES_URI.equals(name)) {
+ continue;
+ }
+ if (!ResourceUtils.getResourceTypes().containsKey(name)) {
+ LOG.warn("Skipping unknown resource type {} in container history",
name);
Review Comment:
Addressed in 48b26fe: unknown resource types now log at DEBUG rather than
WARN, and both timeline docs match.
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/timeline/TimelineUtils.java:
##########
@@ -60,6 +72,57 @@ public class TimelineUtils {
YarnJacksonJaxbJsonProvider.configObjectMapper(mapper);
}
+ @Private
+ public static Map<String, Map<String, Object>> getCustomResourceInfo(
Review Comment:
Addressed in 48b26fe: all three publishers omit the field for an empty
custom-resource map. The helper also omits zero-valued custom types, so even a
configured but unused GPU type adds no field. RM v1, RM v2, and NM tests assert
the field is absent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
