Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/535/ No changes - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-10033) TestProportionalCapacityPreemptionPolicy not initializing vcores for effective max resources
Eric Payne created YARN-10033:
-
Summary: TestProportionalCapacityPreemptionPolicy not initializing
vcores for effective max resources
Key: YARN-10033
URL: https://issues.apache.org/jira/browse/YARN-10033
Project: Hadoop YARN
Issue Type: Improvement
Components: capacity scheduler, test
Affects Versions: 3.3.0
Reporter: Eric Payne
TestProportionalCapacityPreemptionPolicy#testPreemptionWithVCoreResource is
preempting more containers than would happen on a real cluster.
This is because the process for mocking CS queues in
{{TestProportionalCapacityPreemptionPolicy}} fails to take into consideration
vcores when mocking effective max resources.
This causes miscalculations for how many vcores to preempt when the DRF is
being used in the test:
{code:title=TempQueuePerPartition#offer}
Resource absMaxCapIdealAssignedDelta = Resources.componentwiseMax(
Resources.subtract(getMax(), idealAssigned),
Resource.newInstance(0, 0));
{code}
In the above code, the preemption policy is offering resources to an
underserved queue. {{getMax()}} will use the effective max resource if it
exists. Since this test is mocking effective max resources, it will return that
value. However, since the mock doesn't include vcores, the test treats memory
as the dominant resource and awards too many preempted containers to the
underserved queue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-10032) Implement regex querying of logs
Adam Antal created YARN-10032: - Summary: Implement regex querying of logs Key: YARN-10032 URL: https://issues.apache.org/jira/browse/YARN-10032 Project: Hadoop YARN Issue Type: Sub-task Components: yarn Affects Versions: 3.2.1 Reporter: Adam Antal Assignee: Adam Antal Fix For: 3.3.0 After YARN-10031, we have query parameters to the log servlet's GET endpoint. To demonstrate the new capabilities of the log servlet and how easy it will be to add a functionality to all log servlets at the same time: let's add the ability to search in the aggregated logs with a given regex. A conceptual use case: User run several MR jobs daily, but some of them fail to localize a particular resource at first. We want to search in the logs of these Yarn applications, and extract some data from them. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-10031) Create a general purpose log request with additional query parameters
Adam Antal created YARN-10031:
-
Summary: Create a general purpose log request with additional
query parameters
Key: YARN-10031
URL: https://issues.apache.org/jira/browse/YARN-10031
Project: Hadoop YARN
Issue Type: Sub-task
Components: yarn
Reporter: Adam Antal
Assignee: Adam Antal
The current endpoints are robust but not very flexible with regards to
filtering options. I suggest to add an endpoint which provides filtering
options.
E.g.:
In ATS we have multiple endpoints:
/containers/{containerid}/logs/{filename}
/containerlogs/{containerid}/{filename}
We could add @QueryParams parameters to the REST endpoints like this:
/containers/{containerid}/logs?fileName=stderr=FAILED=nm45
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-10030) Remove code duplicates from log servlet utility classes
Adam Antal created YARN-10030: - Summary: Remove code duplicates from log servlet utility classes Key: YARN-10030 URL: https://issues.apache.org/jira/browse/YARN-10030 Project: Hadoop YARN Issue Type: Sub-task Components: yarn Reporter: Adam Antal Assignee: Adam Antal Also, build a centralized package for handling the log servlet related functionality (preferably in yarn-common). There are a tons of code duplication in the following classes: - ContainerLogsUtils - LogWebServiceUtils - LogToolUtils - LogAggregationWebUtils - LogCLIHelpers -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-10029) Add option to UIv2 to get container logs from the new JHS API
Adam Antal created YARN-10029: - Summary: Add option to UIv2 to get container logs from the new JHS API Key: YARN-10029 URL: https://issues.apache.org/jira/browse/YARN-10029 Project: Hadoop YARN Issue Type: Sub-task Components: yarn Affects Versions: 3.2.1 Reporter: Adam Antal Assignee: Adam Antal Provided the new API is ready to use (also integrated into JHS in YARN-10028), we can add a new config option to UIv2 that would make the UIv2 to request logs from the JHS API similarly as the ATSv2. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-10028) Integrate the new abstract log servlet to the JobHistory server
Adam Antal created YARN-10028: - Summary: Integrate the new abstract log servlet to the JobHistory server Key: YARN-10028 URL: https://issues.apache.org/jira/browse/YARN-10028 Project: Hadoop YARN Issue Type: Sub-task Components: yarn Reporter: Adam Antal Currently JHS has already incorporates a log servlet, but it in incapable of serving REST calls. We can integrate the new common log servlet to the JHS in order to have a REST interface. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-10027) Add ability for ATS (log servlet) to read logs of running apps
Adam Antal created YARN-10027:
-
Summary: Add ability for ATS (log servlet) to read logs of running
apps
Key: YARN-10027
URL: https://issues.apache.org/jira/browse/YARN-10027
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Adam Antal
Currently neither version of the AHS is able to read logs of running apps
(local logs of NodeManager). YARN log CLI is integrated with NodeManager to
extract local logs as well (see YARN-5224 for reference), the same should be
done for ATS.
Some context:
The local log files are read by the server in
{{NMWebServices#getContainerLogFile}}. This is accessed by the YARN logs CLI
through REST using the /containers/{containerid}/logs/{filename} endpoint in
{{LogsCLI#getResponeFromNMWebService}}.
If YARN-10026 we can pull the common code pieces out of those services, we can
implement this in the common log servlet.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-10026) Pull out common code pieces from ATS v1.5 and v2
Adam Antal created YARN-10026: - Summary: Pull out common code pieces from ATS v1.5 and v2 Key: YARN-10026 URL: https://issues.apache.org/jira/browse/YARN-10026 Project: Hadoop YARN Issue Type: Sub-task Components: ATSv2, yarn Affects Versions: 3.2.1 Reporter: Adam Antal Assignee: Adam Antal ATSv1.5 and ATSv2 has lots of common code that can be pulled to an abstract service / package. The logic is the same, and the code is _almost_ the same. As far as I see, the only ATS specific thing in that AppInfo is constructed from an ApplicationReport, which information is extracted from the TimelineReader client, Later the appInfo object's user and appState fields are used, but I see no other dependency on the timeline part. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-10025) Various improvements in YARN log servlets
Adam Antal created YARN-10025: - Summary: Various improvements in YARN log servlets Key: YARN-10025 URL: https://issues.apache.org/jira/browse/YARN-10025 Project: Hadoop YARN Issue Type: Improvement Components: yarn Affects Versions: 3.2.1 Reporter: Adam Antal Assignee: Adam Antal There are multiple ways how we can enhance the current log servlets in YARN. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
