[jira] [Commented] (YARN-2701) Potential race condition in startLocalizer when using LinuxContainerExecutor
[
https://issues.apache.org/jira/browse/YARN-2701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14180955#comment-14180955
]
Binglin Chang commented on YARN-2701:
-
Hi Xuan Gong, thank for the patch. Path lgtm, just some indent issue in
check_dir definition.
Potential race condition in startLocalizer when using LinuxContainerExecutor
--
Key: YARN-2701
URL: https://issues.apache.org/jira/browse/YARN-2701
Project: Hadoop YARN
Issue Type: Bug
Reporter: Xuan Gong
Assignee: Xuan Gong
Priority: Blocker
Fix For: 2.6.0
Attachments: YARN-2701.1.patch, YARN-2701.2.patch, YARN-2701.3.patch,
YARN-2701.4.patch, YARN-2701.5.patch, YARN-2701.6.patch,
YARN-2701.addendum.1.patch, YARN-2701.addendum.2.patch
When using LinuxContainerExecutor do startLocalizer, we are using native code
container-executor.c.
{code}
if (stat(npath, sb) != 0) {
if (mkdir(npath, perm) != 0) {
{code}
We are using check and create method to create the appDir under /usercache.
But if there are two containers trying to do this at the same time, race
condition may happen.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2161) Fix build on macosx: YARN parts
[ https://issues.apache.org/jira/browse/YARN-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14177969#comment-14177969 ] Binglin Chang commented on YARN-2161: - Hi [~xgong], sorry for break the code. I see in YARN-2701 you already have fix code, but decide to revert the code in the end to be more safe, but this breaks the mac build, how about use #ifdef to use old code when compiling on glibc 2.10(http://linux.die.net/man/2/openat) and use your fixing code otherwise? Fix build on macosx: YARN parts --- Key: YARN-2161 URL: https://issues.apache.org/jira/browse/YARN-2161 Project: Hadoop YARN Issue Type: Bug Reporter: Binglin Chang Assignee: Binglin Chang Fix For: 2.6.0 Attachments: YARN-2161.v1.patch, YARN-2161.v2.patch When compiling on macosx with -Pnative, there are several warning and errors, fix this would help hadoop developers with macosx env. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2161) Fix build on macosx: YARN parts
[ https://issues.apache.org/jira/browse/YARN-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated YARN-2161: Attachment: YARN-2161.v2.patch Thanks for the review Allen. Attach new version of the patch, changes: 1. use cmake CHECK_FUNCTION_EXISTS to check fcloseall exists 2. change user bin to user daemon (which both linux and macosx have) 3. add some fix in YARN-1327 (setpgid and libgen.h) Fix build on macosx: YARN parts --- Key: YARN-2161 URL: https://issues.apache.org/jira/browse/YARN-2161 Project: Hadoop YARN Issue Type: Bug Reporter: Binglin Chang Assignee: Binglin Chang Attachments: YARN-2161.v1.patch, YARN-2161.v2.patch When compiling on macosx with -Pnative, there are several warning and errors, fix this would help hadoop developers with macosx env. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2051) Fix code bug and add more unit tests for PBImpls
[ https://issues.apache.org/jira/browse/YARN-2051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated YARN-2051: Attachment: YARN-2051.v1.patch This path using reflection to test all PBImpl records in hadoop-yarn-common. Basically for each PBImpl record, it tests get/set methods and serialization/deserialization. 95 new test case are added, when running those tests, some classes are found with bug and fixed. Fix code bug and add more unit tests for PBImpls Key: YARN-2051 URL: https://issues.apache.org/jira/browse/YARN-2051 Project: Hadoop YARN Issue Type: Test Reporter: Junping Du Assignee: Binglin Chang Priority: Critical Attachments: YARN-2051.v1.patch From YARN-2016, we can see some bug could exist in PB implementation of protocol. The bad news is most of these PBImpl don't have any unit test to verify the info is not lost or changed after serialization/deserialization. We should add more tests for it. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2051) Fix code bug and add more unit tests for PBImpls
[ https://issues.apache.org/jira/browse/YARN-2051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14061629#comment-14061629 ] Binglin Chang commented on YARN-2051: - The failed test is not related, see YARN-2270 Fix code bug and add more unit tests for PBImpls Key: YARN-2051 URL: https://issues.apache.org/jira/browse/YARN-2051 Project: Hadoop YARN Issue Type: Test Reporter: Junping Du Assignee: Binglin Chang Priority: Critical Attachments: YARN-2051.v1.patch From YARN-2016, we can see some bug could exist in PB implementation of protocol. The bad news is most of these PBImpl don't have any unit test to verify the info is not lost or changed after serialization/deserialization. We should add more tests for it. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2088) Fix code bug in GetApplicationsRequestPBImpl#mergeLocalToBuilder
[ https://issues.apache.org/jira/browse/YARN-2088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14057302#comment-14057302 ] Binglin Chang commented on YARN-2088: - Hi [~jianhe] or [~djp], looks like there are no more comments? Would you help get this committed? Thanks. Fix code bug in GetApplicationsRequestPBImpl#mergeLocalToBuilder Key: YARN-2088 URL: https://issues.apache.org/jira/browse/YARN-2088 Project: Hadoop YARN Issue Type: Bug Reporter: Binglin Chang Assignee: Binglin Chang Attachments: YARN-2088.v1.patch Some fields(set,list) are added to proto builders many times, we need to clear those fields before add, otherwise the result proto contains more contents. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2088) Fix code bug in GetApplicationsRequestPBImpl#mergeLocalToBuilder
[ https://issues.apache.org/jira/browse/YARN-2088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14036859#comment-14036859 ] Binglin Chang commented on YARN-2088: - bq. The new unit is failing without this change is because the equals() method also calls getProto() which adds the fields into proto again, right ? Yes Fix code bug in GetApplicationsRequestPBImpl#mergeLocalToBuilder Key: YARN-2088 URL: https://issues.apache.org/jira/browse/YARN-2088 Project: Hadoop YARN Issue Type: Bug Reporter: Binglin Chang Assignee: Binglin Chang Attachments: YARN-2088.v1.patch Some fields(set,list) are added to proto builders many times, we need to clear those fields before add, otherwise the result proto contains more contents. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2161) Fix build on macosx: YARN parts
[ https://issues.apache.org/jira/browse/YARN-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated YARN-2161: Attachment: YARN-2161.v1.patch Fix build on macosx: YARN parts --- Key: YARN-2161 URL: https://issues.apache.org/jira/browse/YARN-2161 Project: Hadoop YARN Issue Type: Bug Reporter: Binglin Chang Assignee: Binglin Chang Attachments: YARN-2161.v1.patch When compiling on macosx with -Pnative, there are several warning and errors, fix this would help hadoop developers with macosx env. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2161) Fix build on macosx: YARN parts
[ https://issues.apache.org/jira/browse/YARN-2161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14032164#comment-14032164 ] Binglin Chang commented on YARN-2161: - Changes: container-executor.c: 1. make mkdirs more compatible, remove usage of mkdirat/openat 2. use sysconf() to get LOGIN_NAME_MAX 3. macosx doesn't have fcloseall, so close all opened fds on macosx 4. disable cgroup on macosx test-container-executor.c: 1. macosx do not have user bin, skip a check 2. change /etc/passwd(not exists on mac) to /bin/ls Fix build on macosx: YARN parts --- Key: YARN-2161 URL: https://issues.apache.org/jira/browse/YARN-2161 Project: Hadoop YARN Issue Type: Bug Reporter: Binglin Chang Assignee: Binglin Chang Attachments: YARN-2161.v1.patch When compiling on macosx with -Pnative, there are several warning and errors, fix this would help hadoop developers with macosx env. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Moved] (YARN-2160) Fix build native library on mac osx
[ https://issues.apache.org/jira/browse/YARN-2160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang moved HADOOP-9648 to YARN-2160: - Affects Version/s: (was: 2.0.5-alpha) (was: 1.1.2) (was: 1.0.4) (was: 1.2.0) Key: YARN-2160 (was: HADOOP-9648) Project: Hadoop YARN (was: Hadoop Common) Fix build native library on mac osx --- Key: YARN-2160 URL: https://issues.apache.org/jira/browse/YARN-2160 Project: Hadoop YARN Issue Type: Bug Reporter: Kirill A. Korinskiy Assignee: Binglin Chang Attachments: HADOOP-9648-native-osx.1.0.4.patch, HADOOP-9648-native-osx.1.1.2.patch, HADOOP-9648-native-osx.1.2.0.patch, HADOOP-9648-native-osx.2.0.5-alpha-rc1.patch, HADOOP-9648.v2.patch Some patches for fixing build a hadoop native library on os x 10.7/10.8. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (YARN-2161) Fix build on macosx: YARN parts
Binglin Chang created YARN-2161: --- Summary: Fix build on macosx: YARN parts Key: YARN-2161 URL: https://issues.apache.org/jira/browse/YARN-2161 Project: Hadoop YARN Issue Type: Bug Reporter: Binglin Chang Assignee: Binglin Chang When compiling on macosx with -Pnative, there are several warning and errors, fix this would help hadoop developers with macosx env. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2030) Use StateMachine to simplify handleStoreEvent() in RMStateStore
[
https://issues.apache.org/jira/browse/YARN-2030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Binglin Chang updated YARN-2030:
Attachment: YARN-2030.v5.patch
fix compile error
Use StateMachine to simplify handleStoreEvent() in RMStateStore
---
Key: YARN-2030
URL: https://issues.apache.org/jira/browse/YARN-2030
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Junping Du
Assignee: Binglin Chang
Attachments: YARN-2030.v1.patch, YARN-2030.v2.patch,
YARN-2030.v3.patch, YARN-2030.v4.patch, YARN-2030.v5.patch
Now the logic to handle different store events in handleStoreEvent() is as
following:
{code}
if (event.getType().equals(RMStateStoreEventType.STORE_APP)
|| event.getType().equals(RMStateStoreEventType.UPDATE_APP)) {
...
if (event.getType().equals(RMStateStoreEventType.STORE_APP)) {
...
} else {
...
}
...
try {
if (event.getType().equals(RMStateStoreEventType.STORE_APP)) {
...
} else {
...
}
}
...
} else if (event.getType().equals(RMStateStoreEventType.STORE_APP_ATTEMPT)
|| event.getType().equals(RMStateStoreEventType.UPDATE_APP_ATTEMPT)) {
...
if (event.getType().equals(RMStateStoreEventType.STORE_APP_ATTEMPT)) {
...
} else {
...
}
...
if (event.getType().equals(RMStateStoreEventType.STORE_APP_ATTEMPT)) {
...
} else {
...
}
}
...
} else if (event.getType().equals(RMStateStoreEventType.REMOVE_APP)) {
...
} else {
...
}
}
{code}
This is not only confuse people but also led to mistake easily. We may
leverage state machine to simply this even no state transitions.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (YARN-2030) Use StateMachine to simplify handleStoreEvent() in RMStateStore
[
https://issues.apache.org/jira/browse/YARN-2030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Binglin Chang updated YARN-2030:
Attachment: YARN-2030.v4.patch
Thanks for the comments, [~jianhe]. Update the patch to make getProto abstract
method in record class.
Use StateMachine to simplify handleStoreEvent() in RMStateStore
---
Key: YARN-2030
URL: https://issues.apache.org/jira/browse/YARN-2030
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Junping Du
Assignee: Binglin Chang
Attachments: YARN-2030.v1.patch, YARN-2030.v2.patch,
YARN-2030.v3.patch, YARN-2030.v4.patch
Now the logic to handle different store events in handleStoreEvent() is as
following:
{code}
if (event.getType().equals(RMStateStoreEventType.STORE_APP)
|| event.getType().equals(RMStateStoreEventType.UPDATE_APP)) {
...
if (event.getType().equals(RMStateStoreEventType.STORE_APP)) {
...
} else {
...
}
...
try {
if (event.getType().equals(RMStateStoreEventType.STORE_APP)) {
...
} else {
...
}
}
...
} else if (event.getType().equals(RMStateStoreEventType.STORE_APP_ATTEMPT)
|| event.getType().equals(RMStateStoreEventType.UPDATE_APP_ATTEMPT)) {
...
if (event.getType().equals(RMStateStoreEventType.STORE_APP_ATTEMPT)) {
...
} else {
...
}
...
if (event.getType().equals(RMStateStoreEventType.STORE_APP_ATTEMPT)) {
...
} else {
...
}
}
...
} else if (event.getType().equals(RMStateStoreEventType.REMOVE_APP)) {
...
} else {
...
}
}
{code}
This is not only confuse people but also led to mistake easily. We may
leverage state machine to simply this even no state transitions.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (YARN-2030) Use StateMachine to simplify handleStoreEvent() in RMStateStore
[
https://issues.apache.org/jira/browse/YARN-2030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Binglin Chang updated YARN-2030:
Attachment: YARN-2030.v3.patch
Thanks for the comments [~djp] and [~jianhe]. I update the patch to make
ApplicationAttempStateData and ApplicationStateData abstract classes.
bq. Accordingly storeApplicationStateInternal can take in ApplicationStateData
instead of ApplicationStateDataPBImpl as the argument to avoid the type cast.
I try to change updateApplicationAttemptStateInternal paramter type from PBImpl
to abstract records, but looks like some RMStateStore(FileSystemRMStateStore
and ZKRMStateStore) require the parameter to be PBImpl(so they can use toProto
to serialize)
Use StateMachine to simplify handleStoreEvent() in RMStateStore
---
Key: YARN-2030
URL: https://issues.apache.org/jira/browse/YARN-2030
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Junping Du
Assignee: Binglin Chang
Attachments: YARN-2030.v1.patch, YARN-2030.v2.patch,
YARN-2030.v3.patch
Now the logic to handle different store events in handleStoreEvent() is as
following:
{code}
if (event.getType().equals(RMStateStoreEventType.STORE_APP)
|| event.getType().equals(RMStateStoreEventType.UPDATE_APP)) {
...
if (event.getType().equals(RMStateStoreEventType.STORE_APP)) {
...
} else {
...
}
...
try {
if (event.getType().equals(RMStateStoreEventType.STORE_APP)) {
...
} else {
...
}
}
...
} else if (event.getType().equals(RMStateStoreEventType.STORE_APP_ATTEMPT)
|| event.getType().equals(RMStateStoreEventType.UPDATE_APP_ATTEMPT)) {
...
if (event.getType().equals(RMStateStoreEventType.STORE_APP_ATTEMPT)) {
...
} else {
...
}
...
if (event.getType().equals(RMStateStoreEventType.STORE_APP_ATTEMPT)) {
...
} else {
...
}
}
...
} else if (event.getType().equals(RMStateStoreEventType.REMOVE_APP)) {
...
} else {
...
}
}
{code}
This is not only confuse people but also led to mistake easily. We may
leverage state machine to simply this even no state transitions.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2030) Use StateMachine to simplify handleStoreEvent() in RMStateStore
[
https://issues.apache.org/jira/browse/YARN-2030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14016325#comment-14016325
]
Binglin Chang commented on YARN-2030:
-
Hi, [~djp]. I think the patch is ready for review, could you help review the
patch?
Use StateMachine to simplify handleStoreEvent() in RMStateStore
---
Key: YARN-2030
URL: https://issues.apache.org/jira/browse/YARN-2030
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Junping Du
Assignee: Binglin Chang
Attachments: YARN-2030.v1.patch, YARN-2030.v2.patch
Now the logic to handle different store events in handleStoreEvent() is as
following:
{code}
if (event.getType().equals(RMStateStoreEventType.STORE_APP)
|| event.getType().equals(RMStateStoreEventType.UPDATE_APP)) {
...
if (event.getType().equals(RMStateStoreEventType.STORE_APP)) {
...
} else {
...
}
...
try {
if (event.getType().equals(RMStateStoreEventType.STORE_APP)) {
...
} else {
...
}
}
...
} else if (event.getType().equals(RMStateStoreEventType.STORE_APP_ATTEMPT)
|| event.getType().equals(RMStateStoreEventType.UPDATE_APP_ATTEMPT)) {
...
if (event.getType().equals(RMStateStoreEventType.STORE_APP_ATTEMPT)) {
...
} else {
...
}
...
if (event.getType().equals(RMStateStoreEventType.STORE_APP_ATTEMPT)) {
...
} else {
...
}
}
...
} else if (event.getType().equals(RMStateStoreEventType.REMOVE_APP)) {
...
} else {
...
}
}
{code}
This is not only confuse people but also led to mistake easily. We may
leverage state machine to simply this even no state transitions.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (YARN-2103) Inconsistency between viaProto flag and initial value of SerializedExceptionProto.Builder
[
https://issues.apache.org/jira/browse/YARN-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Binglin Chang updated YARN-2103:
Attachment: YARN-2103.v3.patch
Inconsistency between viaProto flag and initial value of
SerializedExceptionProto.Builder
-
Key: YARN-2103
URL: https://issues.apache.org/jira/browse/YARN-2103
Project: Hadoop YARN
Issue Type: Bug
Reporter: Binglin Chang
Assignee: Binglin Chang
Attachments: YARN-2103.v1.patch, YARN-2103.v2.patch,
YARN-2103.v3.patch
Bug 1:
{code}
SerializedExceptionProto proto = SerializedExceptionProto
.getDefaultInstance();
SerializedExceptionProto.Builder builder = null;
boolean viaProto = false;
{code}
Since viaProto is false, we should initiate build rather than proto
Bug 2:
the class does not provide hashcode() and equals() like other PBImpl records,
this class is used in other records, it may affect other records' behavior.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2103) Inconsistency between viaProto flag and initial value of SerializedExceptionProto.Builder
[
https://issues.apache.org/jira/browse/YARN-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14014924#comment-14014924
]
Binglin Chang commented on YARN-2103:
-
Thanks for the view and extra tests, [~ozawa]. I update the patch.
Inconsistency between viaProto flag and initial value of
SerializedExceptionProto.Builder
-
Key: YARN-2103
URL: https://issues.apache.org/jira/browse/YARN-2103
Project: Hadoop YARN
Issue Type: Bug
Reporter: Binglin Chang
Assignee: Binglin Chang
Attachments: YARN-2103.v1.patch, YARN-2103.v2.patch,
YARN-2103.v3.patch
Bug 1:
{code}
SerializedExceptionProto proto = SerializedExceptionProto
.getDefaultInstance();
SerializedExceptionProto.Builder builder = null;
boolean viaProto = false;
{code}
Since viaProto is false, we should initiate build rather than proto
Bug 2:
the class does not provide hashcode() and equals() like other PBImpl records,
this class is used in other records, it may affect other records' behavior.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (YARN-2103) Inconsistency between viaProto flag and initial value of SerializedExceptionProto.Builder
[
https://issues.apache.org/jira/browse/YARN-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Binglin Chang updated YARN-2103:
Attachment: YARN-2103.v1.patch
Thanks for the review [~djp], update patch addressing your comments.
Inconsistency between viaProto flag and initial value of
SerializedExceptionProto.Builder
-
Key: YARN-2103
URL: https://issues.apache.org/jira/browse/YARN-2103
Project: Hadoop YARN
Issue Type: Bug
Reporter: Binglin Chang
Assignee: Binglin Chang
Attachments: YARN-2103.v1.patch, YARN-2103.v2.patch
Bug 1:
{code}
SerializedExceptionProto proto = SerializedExceptionProto
.getDefaultInstance();
SerializedExceptionProto.Builder builder = null;
boolean viaProto = false;
{code}
Since viaProto is false, we should initiate build rather than proto
Bug 2:
the class does not provide hashcode() and equals() like other PBImpl records,
this class is used in other records, it may affect other records' behavior.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (YARN-2103) Inconsistency between viaProto flag and initial value of SerializedExceptionProto.Builder
[
https://issues.apache.org/jira/browse/YARN-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Binglin Chang updated YARN-2103:
Attachment: (was: YARN-2103.v1.patch)
Inconsistency between viaProto flag and initial value of
SerializedExceptionProto.Builder
-
Key: YARN-2103
URL: https://issues.apache.org/jira/browse/YARN-2103
Project: Hadoop YARN
Issue Type: Bug
Reporter: Binglin Chang
Assignee: Binglin Chang
Attachments: YARN-2103.v1.patch, YARN-2103.v2.patch
Bug 1:
{code}
SerializedExceptionProto proto = SerializedExceptionProto
.getDefaultInstance();
SerializedExceptionProto.Builder builder = null;
boolean viaProto = false;
{code}
Since viaProto is false, we should initiate build rather than proto
Bug 2:
the class does not provide hashcode() and equals() like other PBImpl records,
this class is used in other records, it may affect other records' behavior.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (YARN-2103) Inconsistency between viaProto flag and initial value of SerializedExceptionProto.Builder
[
https://issues.apache.org/jira/browse/YARN-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Binglin Chang updated YARN-2103:
Attachment: YARN-2103.v2.patch
updated wrong patch... resubmit
Inconsistency between viaProto flag and initial value of
SerializedExceptionProto.Builder
-
Key: YARN-2103
URL: https://issues.apache.org/jira/browse/YARN-2103
Project: Hadoop YARN
Issue Type: Bug
Reporter: Binglin Chang
Assignee: Binglin Chang
Attachments: YARN-2103.v1.patch, YARN-2103.v2.patch
Bug 1:
{code}
SerializedExceptionProto proto = SerializedExceptionProto
.getDefaultInstance();
SerializedExceptionProto.Builder builder = null;
boolean viaProto = false;
{code}
Since viaProto is false, we should initiate build rather than proto
Bug 2:
the class does not provide hashcode() and equals() like other PBImpl records,
this class is used in other records, it may affect other records' behavior.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2103) Fix code bug in SerializedExceptionPBImpl
[
https://issues.apache.org/jira/browse/YARN-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14009347#comment-14009347
]
Binglin Chang commented on YARN-2103:
-
I plan to add generic test to test all PBImpls in YARN-2051, so separated tests
are not needed.
Fix code bug in SerializedExceptionPBImpl
-
Key: YARN-2103
URL: https://issues.apache.org/jira/browse/YARN-2103
Project: Hadoop YARN
Issue Type: Bug
Reporter: Binglin Chang
Assignee: Binglin Chang
Attachments: YARN-2103.v1.patch
{code}
SerializedExceptionProto proto = SerializedExceptionProto
.getDefaultInstance();
SerializedExceptionProto.Builder builder = null;
boolean viaProto = false;
{code}
Since viaProto is false, we should initiate build rather than proto
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2103) Inconsistency between viaProto flag and initial value of SerializedExceptionProto.Builder
[
https://issues.apache.org/jira/browse/YARN-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14010700#comment-14010700
]
Binglin Chang commented on YARN-2103:
-
Hi [~ozawa], thanks for reviewing the patch and the comments. I use the
original title because the bug isn't just about inconsistent viaProto, but also
lack of equals and hashcode method(which will affect other records who uses
SerializedException), I guess I should point out all bugs in the jira.
about code format, most PBImpl classes use those common code:
{code}
private void maybeInitBuilder() {
if (viaProto || builder == null) {
builder = GetApplicationsRequestProto.newBuilder(proto);
}
viaProto = false;
}
@Override
public int hashCode() {
return getProto().hashCode();
}
@Override
public boolean equals(Object other) {
if (other == null)
return false;
if (other.getClass().isAssignableFrom(this.getClass())) {
return this.getProto().equals(this.getClass().cast(other).getProto());
}
return false;
}
{code}
you can see GetApplicationsRequestPBImpl/GetApplicationsResponsePBImpl, I just
follow those patterns, maybe we can change them all in another JIRA, changing
them may not fit into in this JIRA.
bq. How about adding concrete tests as a first step of generic tests on
YARN-2051.
After generic test are added, those old tests are probably redundant and can be
removed. Guess we can discuss this in the future. I can provide a separate test
currently.
Inconsistency between viaProto flag and initial value of
SerializedExceptionProto.Builder
-
Key: YARN-2103
URL: https://issues.apache.org/jira/browse/YARN-2103
Project: Hadoop YARN
Issue Type: Bug
Reporter: Binglin Chang
Assignee: Binglin Chang
Attachments: YARN-2103.v1.patch
{code}
SerializedExceptionProto proto = SerializedExceptionProto
.getDefaultInstance();
SerializedExceptionProto.Builder builder = null;
boolean viaProto = false;
{code}
Since viaProto is false, we should initiate build rather than proto
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (YARN-2103) Inconsistency between viaProto flag and initial value of SerializedExceptionProto.Builder
[
https://issues.apache.org/jira/browse/YARN-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Binglin Chang updated YARN-2103:
Description:
Bug 1:
{code}
SerializedExceptionProto proto = SerializedExceptionProto
.getDefaultInstance();
SerializedExceptionProto.Builder builder = null;
boolean viaProto = false;
{code}
Since viaProto is false, we should initiate build rather than proto
Bug 2:
the class does not provide hashcode() and equals() like other PBImpl records,
this class is used in other records, it may affect other records' behavior.
was:
{code}
SerializedExceptionProto proto = SerializedExceptionProto
.getDefaultInstance();
SerializedExceptionProto.Builder builder = null;
boolean viaProto = false;
{code}
Since viaProto is false, we should initiate build rather than proto
Inconsistency between viaProto flag and initial value of
SerializedExceptionProto.Builder
-
Key: YARN-2103
URL: https://issues.apache.org/jira/browse/YARN-2103
Project: Hadoop YARN
Issue Type: Bug
Reporter: Binglin Chang
Assignee: Binglin Chang
Attachments: YARN-2103.v1.patch
Bug 1:
{code}
SerializedExceptionProto proto = SerializedExceptionProto
.getDefaultInstance();
SerializedExceptionProto.Builder builder = null;
boolean viaProto = false;
{code}
Since viaProto is false, we should initiate build rather than proto
Bug 2:
the class does not provide hashcode() and equals() like other PBImpl records,
this class is used in other records, it may affect other records' behavior.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1911) NullPointerException in ResourceOption.getOverCommitTimeout()
[
https://issues.apache.org/jira/browse/YARN-1911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008608#comment-14008608
]
Binglin Chang commented on YARN-1911:
-
+1, I also got this bug when doing YARN-2051, hope it will get committed soon.
NullPointerException in ResourceOption.getOverCommitTimeout()
-
Key: YARN-1911
URL: https://issues.apache.org/jira/browse/YARN-1911
Project: Hadoop YARN
Issue Type: Bug
Components: api
Affects Versions: 2.3.0
Reporter: Kenji Kikushima
Assignee: Kenji Kikushima
Attachments: YARN-1911.patch
I'm trying to implement REST API for Dynamic configuration. But
NullPointerException occurred when ResourceOption.getOverCommitTimeout()
called. Here is a sample code with error message.
{code}
@Test
public void testGetOverCommitTimeout() {
Resource capability = Resource.newInstance(4096, 4);
ResourceOption resourceOption = ResourceOption.newInstance(capability,
RMNode.OVER_COMMIT_TIMEOUT_MILLIS_DEFAULT);
Assert.assertEquals(RMNode.OVER_COMMIT_TIMEOUT_MILLIS_DEFAULT,
resourceOption.getOverCommitTimeout());
}
{code}
{code}
java.lang.NullPointerException
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:187)
at
org.apache.hadoop.yarn.api.records.impl.pb.ResourceOptionPBImpl.getOverCommitTimeout(ResourceOptionPBImpl.java:63)
at
org.apache.hadoop.yarn.server.resourcemanager.TestResourceOption.testGetOverCommitTimeout(TestResourceOption.java:33)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at
org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at
org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (YARN-2103) Fix code bug in SerializedExceptionPBImpl
Binglin Chang created YARN-2103:
---
Summary: Fix code bug in SerializedExceptionPBImpl
Key: YARN-2103
URL: https://issues.apache.org/jira/browse/YARN-2103
Project: Hadoop YARN
Issue Type: Bug
Reporter: Binglin Chang
Assignee: Binglin Chang
{code}
SerializedExceptionProto proto = SerializedExceptionProto
.getDefaultInstance();
SerializedExceptionProto.Builder builder = null;
boolean viaProto = false;
{code}
Since viaProto is false, we should initiate build rather than proto
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (YARN-2051) Fix code bug and add more unit tests for PBImpls
[ https://issues.apache.org/jira/browse/YARN-2051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated YARN-2051: Summary: Fix code bug and add more unit tests for PBImpls (was: Add more unit tests for PBImpl that didn't get covered) Fix code bug and add more unit tests for PBImpls Key: YARN-2051 URL: https://issues.apache.org/jira/browse/YARN-2051 Project: Hadoop YARN Issue Type: Test Reporter: Junping Du Assignee: Binglin Chang Priority: Critical From YARN-2016, we can see some bug could exist in PB implementation of protocol. The bad news is most of these PBImpl don't have any unit test to verify the info is not lost or changed after serialization/deserialization. We should add more tests for it. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1871) We should eliminate writing *PBImpl code in YARN
[ https://issues.apache.org/jira/browse/YARN-1871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008650#comment-14008650 ] Binglin Chang commented on YARN-1871: - Good idea on eliminating PBImpl code, some comments: bq. Make record class become a non-abstract class, add simple getters/setters implementation. I like we use simple getter/setter implementation, rather than current complex builder/proto/field mixed cache It's better to remain the old API unchanged, and hide implementation. if we can generate PBImpl toProto/toRecord, why can't we generate simple getters/setters? bq. serialization a record to Proto type using reflection We'd better to generate code, reflection can be used on test code, but I'm afraid using reflection to ser/de-ser in rpc code is not acceptable. bq. There are only some limited patterns in current *PBImpl There are some complex situations: read only type/property, generic types, recursive types and name mismatch in record/proto. When doing YARN-2051, I found some situations hard to automate and need special treatment. We should eliminate writing *PBImpl code in YARN Key: YARN-1871 URL: https://issues.apache.org/jira/browse/YARN-1871 Project: Hadoop YARN Issue Type: Improvement Components: api Affects Versions: 2.4.0 Reporter: Wangda Tan Assignee: Wangda Tan Attachments: YARN-1871.demo.patch Currently, We need write PBImpl classes one by one. After running find . -name *PBImpl*.java | xargs wc -l under hadoop source code directory, we can see, there're more than 25,000 LOC. I think we should improve this, which will be very helpful for YARN developers to make changes for YARN protocols. There're only some limited patterns in current *PBImpl, * Simple types, like string, int32, float. * List? types * Map? types * Enum types Code generation should be enough to generate such PBImpl classes. Some other requirements are, * Leave other related code alone, like service implemention (e.g. ContainerManagerImpl). * (If possible) Forward compatibility, developpers can write their own PBImpl or genereate them. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2051) Fix code bug and add more unit tests for PBImpls
[ https://issues.apache.org/jira/browse/YARN-2051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14008654#comment-14008654 ] Binglin Chang commented on YARN-2051: - Hi [~wangda], thanks for the notice. Fix code bug and add more unit tests for PBImpls Key: YARN-2051 URL: https://issues.apache.org/jira/browse/YARN-2051 Project: Hadoop YARN Issue Type: Test Reporter: Junping Du Assignee: Binglin Chang Priority: Critical From YARN-2016, we can see some bug could exist in PB implementation of protocol. The bad news is most of these PBImpl don't have any unit test to verify the info is not lost or changed after serialization/deserialization. We should add more tests for it. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2103) Fix code bug in SerializedExceptionPBImpl
[
https://issues.apache.org/jira/browse/YARN-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Binglin Chang updated YARN-2103:
Attachment: YARN-2103.v1.patch
Fix code bug in SerializedExceptionPBImpl
-
Key: YARN-2103
URL: https://issues.apache.org/jira/browse/YARN-2103
Project: Hadoop YARN
Issue Type: Bug
Reporter: Binglin Chang
Assignee: Binglin Chang
Attachments: YARN-2103.v1.patch
{code}
SerializedExceptionProto proto = SerializedExceptionProto
.getDefaultInstance();
SerializedExceptionProto.Builder builder = null;
boolean viaProto = false;
{code}
Since viaProto is false, we should initiate build rather than proto
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2088) Fix code bug in GetApplicationsRequestPBImpl#mergeLocalToBuilder
[ https://issues.apache.org/jira/browse/YARN-2088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14005722#comment-14005722 ] Binglin Chang commented on YARN-2088: - Hi Zhiguo, Thanks for the comments, nice catch. Those two lines are used in every record class... so delete them in a single place actually break code conversion, and it's not related to this jira. We may discuss whether to delete them all in other jira. Fix code bug in GetApplicationsRequestPBImpl#mergeLocalToBuilder Key: YARN-2088 URL: https://issues.apache.org/jira/browse/YARN-2088 Project: Hadoop YARN Issue Type: Bug Reporter: Binglin Chang Assignee: Binglin Chang Attachments: YARN-2088.v1.patch Some fields(set,list) are added to proto builders many times, we need to clear those fields before add, otherwise the result proto contains more contents. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2030) Use StateMachine to simplify handleStoreEvent() in RMStateStore
[
https://issues.apache.org/jira/browse/YARN-2030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14005753#comment-14005753
]
Binglin Chang commented on YARN-2030:
-
Hi Jian He,
Thanks for the comments, looks like PBImpl already has ProtoBase as super
class, so we can't change interface to abstract class
{code}
public class ApplicationAttemptStateDataPBImpl
extends ProtoBaseApplicationAttemptStateDataProto
implements ApplicationAttemptStateData {
{code}
Use StateMachine to simplify handleStoreEvent() in RMStateStore
---
Key: YARN-2030
URL: https://issues.apache.org/jira/browse/YARN-2030
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Junping Du
Assignee: Binglin Chang
Attachments: YARN-2030.v1.patch, YARN-2030.v2.patch
Now the logic to handle different store events in handleStoreEvent() is as
following:
{code}
if (event.getType().equals(RMStateStoreEventType.STORE_APP)
|| event.getType().equals(RMStateStoreEventType.UPDATE_APP)) {
...
if (event.getType().equals(RMStateStoreEventType.STORE_APP)) {
...
} else {
...
}
...
try {
if (event.getType().equals(RMStateStoreEventType.STORE_APP)) {
...
} else {
...
}
}
...
} else if (event.getType().equals(RMStateStoreEventType.STORE_APP_ATTEMPT)
|| event.getType().equals(RMStateStoreEventType.UPDATE_APP_ATTEMPT)) {
...
if (event.getType().equals(RMStateStoreEventType.STORE_APP_ATTEMPT)) {
...
} else {
...
}
...
if (event.getType().equals(RMStateStoreEventType.STORE_APP_ATTEMPT)) {
...
} else {
...
}
}
...
} else if (event.getType().equals(RMStateStoreEventType.REMOVE_APP)) {
...
} else {
...
}
}
{code}
This is not only confuse people but also led to mistake easily. We may
leverage state machine to simply this even no state transitions.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (YARN-2088) Fix code bug in GetApplicationsRequestPBImpl#mergeLocalToBuilder
[ https://issues.apache.org/jira/browse/YARN-2088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated YARN-2088: Attachment: YARN-2088.v1.patch Bug and fixes: 1. clear builder before adding Set/Lists 2. remove unnecessary maybeInitBuilder in mergeLocalToBuilder 3. we don't need to construct Iterable manually, just use guava library 4. the property limit is not set properly in mergeLocalToBuilder, this may cause the limit property be reset to Long.MAX... 5. add a test assertion in TestGetApplicationsRequest to verify the bug Run the test on my local laptop, the test failed before the patch, and success after the patch. Fix code bug in GetApplicationsRequestPBImpl#mergeLocalToBuilder Key: YARN-2088 URL: https://issues.apache.org/jira/browse/YARN-2088 Project: Hadoop YARN Issue Type: Bug Reporter: Binglin Chang Assignee: Binglin Chang Attachments: YARN-2088.v1.patch Some fields(set,list) are added to proto builders many times, we need to clear those fields before add, otherwise the result proto contains more contents. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2088) Fix code bug in GetApplicationsRequestPBImpl#mergeLocalToBuilder
[ https://issues.apache.org/jira/browse/YARN-2088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004803#comment-14004803 ] Binglin Chang commented on YARN-2088: - Based on recent bugs related to api records/PBImpl, I have some doubts about the general patterns used in PBImpls(java fields mixed with proto objects, cached states), which causes lots of redundant code and confusion, changes to those code is a mental challenge and can easily generate new bugs... Fix code bug in GetApplicationsRequestPBImpl#mergeLocalToBuilder Key: YARN-2088 URL: https://issues.apache.org/jira/browse/YARN-2088 Project: Hadoop YARN Issue Type: Bug Reporter: Binglin Chang Assignee: Binglin Chang Attachments: YARN-2088.v1.patch Some fields(set,list) are added to proto builders many times, we need to clear those fields before add, otherwise the result proto contains more contents. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2030) Use StateMachine to simplify handleStoreEvent() in RMStateStore
[
https://issues.apache.org/jira/browse/YARN-2030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Binglin Chang updated YARN-2030:
Attachment: YARN-2030.v1.patch
Attach patch.
This is a code refactor, TestFSRMStateStore and TestZKRMStateStore already
cover the code here, so no additional test is added.
Use StateMachine to simplify handleStoreEvent() in RMStateStore
---
Key: YARN-2030
URL: https://issues.apache.org/jira/browse/YARN-2030
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Junping Du
Assignee: Binglin Chang
Attachments: YARN-2030.v1.patch
Now the logic to handle different store events in handleStoreEvent() is as
following:
{code}
if (event.getType().equals(RMStateStoreEventType.STORE_APP)
|| event.getType().equals(RMStateStoreEventType.UPDATE_APP)) {
...
if (event.getType().equals(RMStateStoreEventType.STORE_APP)) {
...
} else {
...
}
...
try {
if (event.getType().equals(RMStateStoreEventType.STORE_APP)) {
...
} else {
...
}
}
...
} else if (event.getType().equals(RMStateStoreEventType.STORE_APP_ATTEMPT)
|| event.getType().equals(RMStateStoreEventType.UPDATE_APP_ATTEMPT)) {
...
if (event.getType().equals(RMStateStoreEventType.STORE_APP_ATTEMPT)) {
...
} else {
...
}
...
if (event.getType().equals(RMStateStoreEventType.STORE_APP_ATTEMPT)) {
...
} else {
...
}
}
...
} else if (event.getType().equals(RMStateStoreEventType.REMOVE_APP)) {
...
} else {
...
}
}
{code}
This is not only confuse people but also led to mistake easily. We may
leverage state machine to simply this even no state transitions.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Assigned] (YARN-2051) Add more unit tests for PBImpl that didn't get covered
[ https://issues.apache.org/jira/browse/YARN-2051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang reassigned YARN-2051: --- Assignee: Binglin Chang Add more unit tests for PBImpl that didn't get covered -- Key: YARN-2051 URL: https://issues.apache.org/jira/browse/YARN-2051 Project: Hadoop YARN Issue Type: Test Reporter: Junping Du Assignee: Binglin Chang Priority: Critical From YARN-2016, we can see some bug could exist in PB implementation of protocol. The bad news is most of these PBImpl don't have any unit test to verify the info is not lost or changed after serialization/deserialization. We should add more tests for it. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2030) Use StateMachine to simplify handleStoreEvent() in RMStateStore
[
https://issues.apache.org/jira/browse/YARN-2030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Binglin Chang updated YARN-2030:
Attachment: YARN-2030.v2.patch
attach v2 patch to fix findbug warnings
Use StateMachine to simplify handleStoreEvent() in RMStateStore
---
Key: YARN-2030
URL: https://issues.apache.org/jira/browse/YARN-2030
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Junping Du
Assignee: Binglin Chang
Attachments: YARN-2030.v1.patch, YARN-2030.v2.patch
Now the logic to handle different store events in handleStoreEvent() is as
following:
{code}
if (event.getType().equals(RMStateStoreEventType.STORE_APP)
|| event.getType().equals(RMStateStoreEventType.UPDATE_APP)) {
...
if (event.getType().equals(RMStateStoreEventType.STORE_APP)) {
...
} else {
...
}
...
try {
if (event.getType().equals(RMStateStoreEventType.STORE_APP)) {
...
} else {
...
}
}
...
} else if (event.getType().equals(RMStateStoreEventType.STORE_APP_ATTEMPT)
|| event.getType().equals(RMStateStoreEventType.UPDATE_APP_ATTEMPT)) {
...
if (event.getType().equals(RMStateStoreEventType.STORE_APP_ATTEMPT)) {
...
} else {
...
}
...
if (event.getType().equals(RMStateStoreEventType.STORE_APP_ATTEMPT)) {
...
} else {
...
}
}
...
} else if (event.getType().equals(RMStateStoreEventType.REMOVE_APP)) {
...
} else {
...
}
}
{code}
This is not only confuse people but also led to mistake easily. We may
leverage state machine to simply this even no state transitions.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2051) Add more unit tests for PBImpl that didn't get covered
[ https://issues.apache.org/jira/browse/YARN-2051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004371#comment-14004371 ] Binglin Chang commented on YARN-2051: - I thought about this, most of the pb serde validation involves the following procedure: 1. set property to record using value(v0) 2. get proto obj from record 3. create new record from proto obj 4. get property from new record value(v1), validate v0 == v1 This can be automated for all set/get pairs, we just need to use reflection to find all get/set pairs of the record class, and test each pair. By doing this, we save lots of testing code. In the future when we add new properties to a record, no need to add/change the testing code:) Note: those record looks like java beans but many of those does not follow strict java bean laws, I try to leverage commons-beanutil but it seems it is not flexible enough, we make a patch soon. Add more unit tests for PBImpl that didn't get covered -- Key: YARN-2051 URL: https://issues.apache.org/jira/browse/YARN-2051 Project: Hadoop YARN Issue Type: Test Reporter: Junping Du Assignee: Binglin Chang Priority: Critical From YARN-2016, we can see some bug could exist in PB implementation of protocol. The bad news is most of these PBImpl don't have any unit test to verify the info is not lost or changed after serialization/deserialization. We should add more tests for it. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (YARN-2030) Use StateMachine to simplify handleStoreEvent() in RMStateStore
[
https://issues.apache.org/jira/browse/YARN-2030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Binglin Chang reassigned YARN-2030:
---
Assignee: Binglin Chang
Use StateMachine to simplify handleStoreEvent() in RMStateStore
---
Key: YARN-2030
URL: https://issues.apache.org/jira/browse/YARN-2030
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Junping Du
Assignee: Binglin Chang
Now the logic to handle different store events in handleStoreEvent() is as
following:
{code}
if (event.getType().equals(RMStateStoreEventType.STORE_APP)
|| event.getType().equals(RMStateStoreEventType.UPDATE_APP)) {
...
if (event.getType().equals(RMStateStoreEventType.STORE_APP)) {
...
} else {
...
}
...
try {
if (event.getType().equals(RMStateStoreEventType.STORE_APP)) {
...
} else {
...
}
}
...
} else if (event.getType().equals(RMStateStoreEventType.STORE_APP_ATTEMPT)
|| event.getType().equals(RMStateStoreEventType.UPDATE_APP_ATTEMPT)) {
...
if (event.getType().equals(RMStateStoreEventType.STORE_APP_ATTEMPT)) {
...
} else {
...
}
...
if (event.getType().equals(RMStateStoreEventType.STORE_APP_ATTEMPT)) {
...
} else {
...
}
}
...
} else if (event.getType().equals(RMStateStoreEventType.REMOVE_APP)) {
...
} else {
...
}
}
{code}
This is not only confuse people but also led to mistake easily. We may
leverage state machine to simply this even no state transitions.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1872) TestDistributedShell occasionally fails in trunk
[ https://issues.apache.org/jira/browse/YARN-1872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13998493#comment-13998493 ] Binglin Chang commented on YARN-1872: - Hi, testDSShell fails with asser failed, don't know whether it is relevant: https://builds.apache.org/job/Hadoop-Yarn-trunk/561/consoleText testDSShell(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell) Time elapsed: 27.557 sec FAILURE! java.lang.AssertionError: expected:1 but was:0 at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.junit.Assert.assertEquals(Assert.java:542) at org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSShell(TestDistributedShell.java:198) Results : Failed tests: TestDistributedShell.testDSShell:198 expected:1 but was:0 Tests run: 8, Failures: 1, Errors: 0, Skipped: 0 TestDistributedShell occasionally fails in trunk Key: YARN-1872 URL: https://issues.apache.org/jira/browse/YARN-1872 Project: Hadoop YARN Issue Type: Bug Reporter: Ted Yu Assignee: Hong Zhiguo Attachments: TestDistributedShell.out, YARN-1872.patch From https://builds.apache.org/job/Hadoop-Yarn-trunk/520/console : TestDistributedShell#testDSShellWithCustomLogPropertyFile failed and TestDistributedShell#testDSShell timed out. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1463) TestContainerManagerSecurity#testContainerManager fails
[
https://issues.apache.org/jira/browse/YARN-1463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13849169#comment-13849169
]
Binglin Chang commented on YARN-1463:
-
Thanks for the input Haohui and Vinod. I will update the patch soon.
TestContainerManagerSecurity#testContainerManager fails
---
Key: YARN-1463
URL: https://issues.apache.org/jira/browse/YARN-1463
Project: Hadoop YARN
Issue Type: Test
Reporter: Ted Yu
Assignee: Binglin Chang
Attachments: YARN-1463.v1.patch, YARN-1463.v2.patch
Here is stack trace:
{code}
testContainerManager[1](org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
Time elapsed: 1.756 sec ERROR!
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException:
ResourceManager failed to start. Final state is STOPPED
at
org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.serviceStart(MiniYARNCluster.java:253)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testContainerManager(TestContainerManagerSecurity.java:110)
{code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (YARN-1463) TestContainerManagerSecurity#testContainerManager fails
[
https://issues.apache.org/jira/browse/YARN-1463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Binglin Chang updated YARN-1463:
Attachment: YARN-1463.v2.patch
Attach new patch adding comments about the affected tests
TestContainerManagerSecurity#testContainerManager fails
---
Key: YARN-1463
URL: https://issues.apache.org/jira/browse/YARN-1463
Project: Hadoop YARN
Issue Type: Test
Reporter: Ted Yu
Assignee: Binglin Chang
Attachments: YARN-1463.v1.patch, YARN-1463.v2.patch
Here is stack trace:
{code}
testContainerManager[1](org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
Time elapsed: 1.756 sec ERROR!
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException:
ResourceManager failed to start. Final state is STOPPED
at
org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.serviceStart(MiniYARNCluster.java:253)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testContainerManager(TestContainerManagerSecurity.java:110)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (YARN-1450) TestUnmanagedAMLauncher#testDSShell fails on trunk
[
https://issues.apache.org/jira/browse/YARN-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Binglin Chang updated YARN-1450:
Attachment: YARN-1450.v5.patch
Thanks for the explanation and review Vinod, attach new patch addressing your
comments.
TestUnmanagedAMLauncher#testDSShell fails on trunk
--
Key: YARN-1450
URL: https://issues.apache.org/jira/browse/YARN-1450
Project: Hadoop YARN
Issue Type: Bug
Components: applications/distributed-shell
Reporter: Akira AJISAKA
Assignee: Binglin Chang
Attachments: YARN-1450.v1.patch, YARN-1450.v2.patch,
YARN-1450.v3.patch, YARN-1450.v4.patch, YARN-1450.v5.patch,
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher-output.txt,
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher.txt
TestUnmanagedAMLauncher fails on trunk. The console output is
{code}
Running
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher
Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 35.937 sec
FAILURE! - in
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher
testDSShell(org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher)
Time elapsed: 14.558 sec ERROR!
java.lang.RuntimeException: Failed to receive final expected state in
ApplicationReport, CurrentState=ACCEPTED,
ExpectedStates=FINISHED,FAILED,KILLED
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.UnmanagedAMLauncher.monitorApplication(UnmanagedAMLauncher.java:447)
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.UnmanagedAMLauncher.run(UnmanagedAMLauncher.java:352)
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher.testDSShell(TestUnmanagedAMLauncher.java:145)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1463) TestContainerManagerSecurity#testContainerManager fails
[
https://issues.apache.org/jira/browse/YARN-1463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13839022#comment-13839022
]
Binglin Chang commented on YARN-1463:
-
There are already 3 tests affected by this, maybe it is better to add some
comment in WebApps rather than each individual tests?
TestContainerManagerSecurity#testContainerManager fails
---
Key: YARN-1463
URL: https://issues.apache.org/jira/browse/YARN-1463
Project: Hadoop YARN
Issue Type: Test
Reporter: Ted Yu
Assignee: Binglin Chang
Attachments: YARN-1463.000.patch, YARN-1463.v1.patch
Here is stack trace:
{code}
testContainerManager[1](org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
Time elapsed: 1.756 sec ERROR!
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException:
ResourceManager failed to start. Final state is STOPPED
at
org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.serviceStart(MiniYARNCluster.java:253)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testContainerManager(TestContainerManagerSecurity.java:110)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (YARN-1450) TestUnmanagedAMLauncher#testDSShell fails on trunk
[
https://issues.apache.org/jira/browse/YARN-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Binglin Chang updated YARN-1450:
Attachment: YARN-1450.v4.patch
Attach new patch, changes:
Write a standalone simple AM to be used in test.
TestUnmanagedAMLauncher#testDSShell fails on trunk
--
Key: YARN-1450
URL: https://issues.apache.org/jira/browse/YARN-1450
Project: Hadoop YARN
Issue Type: Bug
Components: applications/distributed-shell
Reporter: Akira AJISAKA
Assignee: Binglin Chang
Attachments: YARN-1450.v1.patch, YARN-1450.v2.patch,
YARN-1450.v3.patch, YARN-1450.v4.patch,
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher-output.txt,
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher.txt
TestUnmanagedAMLauncher fails on trunk. The console output is
{code}
Running
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher
Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 35.937 sec
FAILURE! - in
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher
testDSShell(org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher)
Time elapsed: 14.558 sec ERROR!
java.lang.RuntimeException: Failed to receive final expected state in
ApplicationReport, CurrentState=ACCEPTED,
ExpectedStates=FINISHED,FAILED,KILLED
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.UnmanagedAMLauncher.monitorApplication(UnmanagedAMLauncher.java:447)
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.UnmanagedAMLauncher.run(UnmanagedAMLauncher.java:352)
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher.testDSShell(TestUnmanagedAMLauncher.java:145)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Assigned] (YARN-1463) TestContainerManagerSecurity#testContainerManager fails
[
https://issues.apache.org/jira/browse/YARN-1463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Binglin Chang reassigned YARN-1463:
---
Assignee: Binglin Chang
TestContainerManagerSecurity#testContainerManager fails
---
Key: YARN-1463
URL: https://issues.apache.org/jira/browse/YARN-1463
Project: Hadoop YARN
Issue Type: Test
Reporter: Ted Yu
Assignee: Binglin Chang
Here is stack trace:
{code}
testContainerManager[1](org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
Time elapsed: 1.756 sec ERROR!
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException:
ResourceManager failed to start. Final state is STOPPED
at
org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.serviceStart(MiniYARNCluster.java:253)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testContainerManager(TestContainerManagerSecurity.java:110)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1463) TestContainerManagerSecurity#testContainerManager fails
[
https://issues.apache.org/jira/browse/YARN-1463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13836713#comment-13836713
]
Binglin Chang commented on YARN-1463:
-
HDFS-5545 introduced this bug: when decide whether to init spnego, original
code logic is broken
TestContainerManagerSecurity#testContainerManager fails
---
Key: YARN-1463
URL: https://issues.apache.org/jira/browse/YARN-1463
Project: Hadoop YARN
Issue Type: Test
Reporter: Ted Yu
Assignee: Binglin Chang
Here is stack trace:
{code}
testContainerManager[1](org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
Time elapsed: 1.756 sec ERROR!
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException:
ResourceManager failed to start. Final state is STOPPED
at
org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.serviceStart(MiniYARNCluster.java:253)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testContainerManager(TestContainerManagerSecurity.java:110)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (YARN-1463) TestContainerManagerSecurity#testContainerManager fails
[
https://issues.apache.org/jira/browse/YARN-1463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Binglin Chang updated YARN-1463:
Attachment: YARN-1463.v1.patch
Attach patch with simple fix, the test can succeed now.
TestContainerManagerSecurity#testContainerManager fails
---
Key: YARN-1463
URL: https://issues.apache.org/jira/browse/YARN-1463
Project: Hadoop YARN
Issue Type: Test
Reporter: Ted Yu
Assignee: Binglin Chang
Attachments: YARN-1463.v1.patch
Here is stack trace:
{code}
testContainerManager[1](org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
Time elapsed: 1.756 sec ERROR!
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException:
ResourceManager failed to start. Final state is STOPPED
at
org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.serviceStart(MiniYARNCluster.java:253)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testContainerManager(TestContainerManagerSecurity.java:110)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1463) TestContainerManagerSecurity#testContainerManager fails
[
https://issues.apache.org/jira/browse/YARN-1463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13837251#comment-13837251
]
Binglin Chang commented on YARN-1463:
-
Hi Haohui,
I originally did the same as your patch did, but it still failed with other
errors on my Macbook pro.
So I add more checks, just as the original code did, and it now passed.
{code}
Running org.apache.hadoop.yarn.server.TestContainerManagerSecurity
Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 26.663 sec
FAILURE! - in org.apache.hadoop.yarn.server.TestContainerManagerSecurity
testContainerManager[1](org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
Time elapsed: 1.735 sec ERROR!
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException:
ResourceManager failed to start. Final state is STOPPED
at
org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.serviceStart(MiniYARNCluster.java:253)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testContainerManager(TestContainerManagerSecurity.java:110)
{code}
TestContainerManagerSecurity#testContainerManager fails
---
Key: YARN-1463
URL: https://issues.apache.org/jira/browse/YARN-1463
Project: Hadoop YARN
Issue Type: Test
Reporter: Ted Yu
Assignee: Binglin Chang
Attachments: YARN-1463.000.patch, YARN-1463.v1.patch
Here is stack trace:
{code}
testContainerManager[1](org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
Time elapsed: 1.756 sec ERROR!
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException:
ResourceManager failed to start. Final state is STOPPED
at
org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.serviceStart(MiniYARNCluster.java:253)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testContainerManager(TestContainerManagerSecurity.java:110)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1463) TestContainerManagerSecurity#testContainerManager fails
[
https://issues.apache.org/jira/browse/YARN-1463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13837257#comment-13837257
]
Binglin Chang commented on YARN-1463:
-
Detail log:
2013-12-03 10:30:44,577 WARN [Thread-321] mortbay.log (Slf4jLog.java:warn(89))
- Failed startup of context
org.mortbay.jetty.webapp.WebAppContext@9ba0281{/,file:/Users/decster/projects/hadoop-trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/classes/webapps/cluster}
javax.servlet.ServletException: javax.servlet.ServletException: Principal not
defined in configuration
at
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.init(KerberosAuthenticationHandler.java:203)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.init(AuthenticationFilter.java:146)
at org.mortbay.jetty.servlet.FilterHolder.doStart(FilterHolder.java:97)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at
org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:713)
at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
at
org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1282)
at
org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:518)
at
org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:499)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at
org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
at
org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at
org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
at org.mortbay.jetty.Server.doStart(Server.java:224)
at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.apache.hadoop.http.HttpServer.start(HttpServer.java:914)
at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:245)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:820)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:471)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:844)
at
org.apache.hadoop.yarn.server.resourcemanager.RMHAProtocolService.transitionToActive(RMHAProtocolService.java:187)
at
org.apache.hadoop.yarn.server.resourcemanager.RMHAProtocolService.serviceStart(RMHAProtocolService.java:101)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:871)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper$3.run(MiniYARNCluster.java:242)
TestContainerManagerSecurity#testContainerManager fails
---
Key: YARN-1463
URL: https://issues.apache.org/jira/browse/YARN-1463
Project: Hadoop YARN
Issue Type: Test
Reporter: Ted Yu
Assignee: Binglin Chang
Attachments: YARN-1463.000.patch, YARN-1463.v1.patch
Here is stack trace:
{code}
testContainerManager[1](org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
Time elapsed: 1.756 sec ERROR!
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException:
ResourceManager failed to start. Final state is STOPPED
at
org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.serviceStart(MiniYARNCluster.java:253)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testContainerManager(TestContainerManagerSecurity.java:110)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1463) TestContainerManagerSecurity#testContainerManager fails
[
https://issues.apache.org/jira/browse/YARN-1463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13837258#comment-13837258
]
Binglin Chang commented on YARN-1463:
-
We can see from the code, HttpServer does not cover null check for keys
{code}
private void initSpnego(Configuration conf, String hostName,
String usernameConfKey, String keytabConfKey) throws IOException {
MapString, String params = new HashMapString, String();
String principalInConf = conf.get(usernameConfKey);
if (principalInConf != null !principalInConf.isEmpty()) {
params.put(kerberos.principal, SecurityUtil.getServerPrincipal(
principalInConf, hostName));
}
String httpKeytab = conf.get(keytabConfKey);
if (httpKeytab != null !httpKeytab.isEmpty()) {
params.put(kerberos.keytab, httpKeytab);
}
params.put(AuthenticationFilter.AUTH_TYPE, kerberos);
defineFilter(webAppContext, SPNEGO_FILTER,
AuthenticationFilter.class.getName(), params, null);
}
{code}
TestContainerManagerSecurity#testContainerManager fails
---
Key: YARN-1463
URL: https://issues.apache.org/jira/browse/YARN-1463
Project: Hadoop YARN
Issue Type: Test
Reporter: Ted Yu
Assignee: Binglin Chang
Attachments: YARN-1463.000.patch, YARN-1463.v1.patch
Here is stack trace:
{code}
testContainerManager[1](org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
Time elapsed: 1.756 sec ERROR!
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException:
ResourceManager failed to start. Final state is STOPPED
at
org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.serviceStart(MiniYARNCluster.java:253)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testContainerManager(TestContainerManagerSecurity.java:110)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1463) TestContainerManagerSecurity#testContainerManager fails
[
https://issues.apache.org/jira/browse/YARN-1463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13837278#comment-13837278
]
Binglin Chang commented on YARN-1463:
-
bq. Webapps enables spnego authentication when security is enabled but no
principals / keytabs are passed in. This configuration is wrong and it should
fail.
I thought the same, but when I looked at the original code:
{code}
if (spnegoPrincipalKey == null
|| conf.get(spnegoPrincipalKey, ).isEmpty()) {
LOG.warn(Principal for spnego filter is not set);
initSpnego = false;
}
if (spnegoKeytabKey == null
|| conf.get(spnegoKeytabKey, ).isEmpty()) {
LOG.warn(Keytab for spnego filter is not set);
initSpnego = false;
}
{code}
The code make a WARN log instead of ERROR, it looks like a intentional
behavior, so I keep the original behavior just for safe, thoughts?
TestContainerManagerSecurity#testContainerManager fails
---
Key: YARN-1463
URL: https://issues.apache.org/jira/browse/YARN-1463
Project: Hadoop YARN
Issue Type: Test
Reporter: Ted Yu
Assignee: Binglin Chang
Attachments: YARN-1463.000.patch, YARN-1463.v1.patch
Here is stack trace:
{code}
testContainerManager[1](org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
Time elapsed: 1.756 sec ERROR!
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException:
ResourceManager failed to start. Final state is STOPPED
at
org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.serviceStart(MiniYARNCluster.java:253)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testContainerManager(TestContainerManagerSecurity.java:110)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1463) TestContainerManagerSecurity#testContainerManager fails
[
https://issues.apache.org/jira/browse/YARN-1463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13837310#comment-13837310
]
Binglin Chang commented on YARN-1463:
-
bq. but the test is broken then
I am sorry. What do you mean? Which test? With my original patch, I didn't see
any test fail?
TestContainerManagerSecurity#testContainerManager fails
---
Key: YARN-1463
URL: https://issues.apache.org/jira/browse/YARN-1463
Project: Hadoop YARN
Issue Type: Test
Reporter: Ted Yu
Assignee: Binglin Chang
Attachments: YARN-1463.000.patch, YARN-1463.v1.patch
Here is stack trace:
{code}
testContainerManager[1](org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
Time elapsed: 1.756 sec ERROR!
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException:
ResourceManager failed to start. Final state is STOPPED
at
org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.serviceStart(MiniYARNCluster.java:253)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testContainerManager(TestContainerManagerSecurity.java:110)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1450) TestUnmanagedAMLauncher#testDSShell fails on trunk
[
https://issues.apache.org/jira/browse/YARN-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13837400#comment-13837400
]
Binglin Chang commented on YARN-1450:
-
Hi Vinod,
Thanks for the explanation, I think I am confused about the propose of
UnmanagedAM, the document of UnmanagedAM says:
bq. The UnmanagedLauncher is a simple client that launches and unmanaged AM. An
unmanagedAM is an AM that is not launched and managed by the RM.
I thought it is used as a tool, new apps can use this to launch AM locally
without create/start YarnClient. If old apps have a workable standalone AM, it
can be launched by this tool too. I did not consider it as a tool to launch old
client based apps, of course it will fail if it is used to launch arbitrarily
old apps. Which leads to the question:
1. What is the usage of UnmanagedAM? Support new special apps which need a
local AM or support all apps? If the later it is far from complete, so either
disable this test or writeuse a simple AM in test.
2. Is distributedshell.ApplicationMaster not supposed to be launched standalone
and must only be called by distributedshell.Client?
TestUnmanagedAMLauncher#testDSShell fails on trunk
--
Key: YARN-1450
URL: https://issues.apache.org/jira/browse/YARN-1450
Project: Hadoop YARN
Issue Type: Bug
Components: applications/distributed-shell
Reporter: Akira AJISAKA
Assignee: Binglin Chang
Attachments: YARN-1450.v1.patch, YARN-1450.v2.patch,
YARN-1450.v3.patch,
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher-output.txt,
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher.txt
TestUnmanagedAMLauncher fails on trunk. The console output is
{code}
Running
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher
Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 35.937 sec
FAILURE! - in
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher
testDSShell(org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher)
Time elapsed: 14.558 sec ERROR!
java.lang.RuntimeException: Failed to receive final expected state in
ApplicationReport, CurrentState=ACCEPTED,
ExpectedStates=FINISHED,FAILED,KILLED
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.UnmanagedAMLauncher.monitorApplication(UnmanagedAMLauncher.java:447)
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.UnmanagedAMLauncher.run(UnmanagedAMLauncher.java:352)
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher.testDSShell(TestUnmanagedAMLauncher.java:145)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1464) TestRMNMSecretKeys#testNMUpdation fails with NullPointerException
[
https://issues.apache.org/jira/browse/YARN-1464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13837416#comment-13837416
]
Binglin Chang commented on YARN-1464:
-
Looks like the exact reason in YARN-1463 caused this. Mark as duplicated
TestRMNMSecretKeys#testNMUpdation fails with NullPointerException
-
Key: YARN-1464
URL: https://issues.apache.org/jira/browse/YARN-1464
Project: Hadoop YARN
Issue Type: Test
Reporter: Ted Yu
Here is the stack trace:
{code}
testNMUpdation(org.apache.hadoop.yarn.server.TestRMNMSecretKeys) Time
elapsed: 2.704 sec ERROR!
java.lang.NullPointerException: null
at java.util.Hashtable.get(Hashtable.java:334)
at java.util.Properties.getProperty(Properties.java:932)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:874)
at org.apache.hadoop.http.HttpServer.initSpnego(HttpServer.java:892)
at org.apache.hadoop.http.HttpServer.access$100(HttpServer.java:101)
at org.apache.hadoop.http.HttpServer$Builder.build(HttpServer.java:323)
at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:232)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:820)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:471)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:844)
at
org.apache.hadoop.yarn.server.resourcemanager.RMHAProtocolService.transitionToActive(RMHAProtocolService.java:187)
at
org.apache.hadoop.yarn.server.resourcemanager.RMHAProtocolService.serviceStart(RMHAProtocolService.java:101)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:871)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.TestRMNMSecretKeys.validateRMNMKeyExchange(TestRMNMSecretKeys.java:69)
at
org.apache.hadoop.yarn.server.TestRMNMSecretKeys.testNMUpdation(TestRMNMSecretKeys.java:49)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Resolved] (YARN-1464) TestRMNMSecretKeys#testNMUpdation fails with NullPointerException
[
https://issues.apache.org/jira/browse/YARN-1464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Binglin Chang resolved YARN-1464.
-
Resolution: Duplicate
TestRMNMSecretKeys#testNMUpdation fails with NullPointerException
-
Key: YARN-1464
URL: https://issues.apache.org/jira/browse/YARN-1464
Project: Hadoop YARN
Issue Type: Test
Reporter: Ted Yu
Here is the stack trace:
{code}
testNMUpdation(org.apache.hadoop.yarn.server.TestRMNMSecretKeys) Time
elapsed: 2.704 sec ERROR!
java.lang.NullPointerException: null
at java.util.Hashtable.get(Hashtable.java:334)
at java.util.Properties.getProperty(Properties.java:932)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:874)
at org.apache.hadoop.http.HttpServer.initSpnego(HttpServer.java:892)
at org.apache.hadoop.http.HttpServer.access$100(HttpServer.java:101)
at org.apache.hadoop.http.HttpServer$Builder.build(HttpServer.java:323)
at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:232)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:820)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:471)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:844)
at
org.apache.hadoop.yarn.server.resourcemanager.RMHAProtocolService.transitionToActive(RMHAProtocolService.java:187)
at
org.apache.hadoop.yarn.server.resourcemanager.RMHAProtocolService.serviceStart(RMHAProtocolService.java:101)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:871)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.TestRMNMSecretKeys.validateRMNMKeyExchange(TestRMNMSecretKeys.java:69)
at
org.apache.hadoop.yarn.server.TestRMNMSecretKeys.testNMUpdation(TestRMNMSecretKeys.java:49)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1463) TestContainerManagerSecurity#testContainerManager fails
[
https://issues.apache.org/jira/browse/YARN-1463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13837420#comment-13837420
]
Binglin Chang commented on YARN-1463:
-
Looks like another test is affected by this: TestRMNMSecretKeys#testNMUpdation.
Those tests all expect RM can start, even if security is enabled but keys are
empty, I don't know whether this is intended and legal, better keep the old
behavior or someone can give more input?
TestContainerManagerSecurity#testContainerManager fails
---
Key: YARN-1463
URL: https://issues.apache.org/jira/browse/YARN-1463
Project: Hadoop YARN
Issue Type: Test
Reporter: Ted Yu
Assignee: Binglin Chang
Attachments: YARN-1463.000.patch, YARN-1463.v1.patch
Here is stack trace:
{code}
testContainerManager[1](org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
Time elapsed: 1.756 sec ERROR!
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException:
ResourceManager failed to start. Final state is STOPPED
at
org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.serviceStart(MiniYARNCluster.java:253)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testContainerManager(TestContainerManagerSecurity.java:110)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Assigned] (YARN-1450) TestUnmanagedAMLauncher#testDSShell fails on trunk
[
https://issues.apache.org/jira/browse/YARN-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Binglin Chang reassigned YARN-1450:
---
Assignee: Binglin Chang
TestUnmanagedAMLauncher#testDSShell fails on trunk
--
Key: YARN-1450
URL: https://issues.apache.org/jira/browse/YARN-1450
Project: Hadoop YARN
Issue Type: Bug
Components: applications
Reporter: Akira AJISAKA
Assignee: Binglin Chang
Attachments:
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher-output.txt,
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher.txt
TestUnmanagedAMLauncher fails on trunk. The console output is
{code}
Running
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher
Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 35.937 sec
FAILURE! - in
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher
testDSShell(org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher)
Time elapsed: 14.558 sec ERROR!
java.lang.RuntimeException: Failed to receive final expected state in
ApplicationReport, CurrentState=ACCEPTED,
ExpectedStates=FINISHED,FAILED,KILLED
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.UnmanagedAMLauncher.monitorApplication(UnmanagedAMLauncher.java:447)
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.UnmanagedAMLauncher.run(UnmanagedAMLauncher.java:352)
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher.testDSShell(TestUnmanagedAMLauncher.java:145)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (YARN-1450) TestUnmanagedAMLauncher#testDSShell fails on trunk
[
https://issues.apache.org/jira/browse/YARN-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Binglin Chang updated YARN-1450:
Attachment: YARN-1450.v1.patch
Bug cause:
YARN-1303 removed parameter --shell_command in DistShell.AppMaster, cause
DistShell.client have parameter --shell_command, unfortunately
TestUnmanagedAMLauncher launch DistShell.AppMaster directly, causing this bug.
Patch changes:
Add parameter --shell_command back to DistShell.AppMaster, this parameter takes
effect only if shell command file does not exists.
TestUnmanagedAMLauncher#testDSShell fails on trunk
--
Key: YARN-1450
URL: https://issues.apache.org/jira/browse/YARN-1450
Project: Hadoop YARN
Issue Type: Bug
Components: applications
Reporter: Akira AJISAKA
Assignee: Binglin Chang
Attachments: YARN-1450.v1.patch,
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher-output.txt,
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher.txt
TestUnmanagedAMLauncher fails on trunk. The console output is
{code}
Running
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher
Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 35.937 sec
FAILURE! - in
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher
testDSShell(org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher)
Time elapsed: 14.558 sec ERROR!
java.lang.RuntimeException: Failed to receive final expected state in
ApplicationReport, CurrentState=ACCEPTED,
ExpectedStates=FINISHED,FAILED,KILLED
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.UnmanagedAMLauncher.monitorApplication(UnmanagedAMLauncher.java:447)
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.UnmanagedAMLauncher.run(UnmanagedAMLauncher.java:352)
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher.testDSShell(TestUnmanagedAMLauncher.java:145)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (YARN-1450) TestUnmanagedAMLauncher#testDSShell fails on trunk
[
https://issues.apache.org/jira/browse/YARN-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Binglin Chang updated YARN-1450:
Component/s: (was: applications)
applications/distributed-shell
TestUnmanagedAMLauncher#testDSShell fails on trunk
--
Key: YARN-1450
URL: https://issues.apache.org/jira/browse/YARN-1450
Project: Hadoop YARN
Issue Type: Bug
Components: applications/distributed-shell
Reporter: Akira AJISAKA
Assignee: Binglin Chang
Attachments: YARN-1450.v1.patch,
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher-output.txt,
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher.txt
TestUnmanagedAMLauncher fails on trunk. The console output is
{code}
Running
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher
Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 35.937 sec
FAILURE! - in
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher
testDSShell(org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher)
Time elapsed: 14.558 sec ERROR!
java.lang.RuntimeException: Failed to receive final expected state in
ApplicationReport, CurrentState=ACCEPTED,
ExpectedStates=FINISHED,FAILED,KILLED
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.UnmanagedAMLauncher.monitorApplication(UnmanagedAMLauncher.java:447)
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.UnmanagedAMLauncher.run(UnmanagedAMLauncher.java:352)
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher.testDSShell(TestUnmanagedAMLauncher.java:145)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1450) TestUnmanagedAMLauncher#testDSShell fails on trunk
[
https://issues.apache.org/jira/browse/YARN-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834382#comment-13834382
]
Binglin Chang commented on YARN-1450:
-
bq. So, if we launch the AM with --shell_command, we will meet the same issue
as before (can not allow multiple commands separating with ; or |). Right ?
Actually there is no issue, the AM is started standalone(no client - RM - NM
- launch AM), we don't need to care about escaping/unescaping issues, allow
multiple commands separating with ;, ;, | or env variable should be fine.
There is nothing wrong about --shell_command, anything you can write to a file
you can pass with cmdline, the issue in YARN-1303 is cuased by client try to
construct a whole cmd without proper escaping support.
TestUnmanagedAMLauncher#testDSShell fails on trunk
--
Key: YARN-1450
URL: https://issues.apache.org/jira/browse/YARN-1450
Project: Hadoop YARN
Issue Type: Bug
Components: applications/distributed-shell
Reporter: Akira AJISAKA
Assignee: Binglin Chang
Attachments: YARN-1450.v1.patch,
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher-output.txt,
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher.txt
TestUnmanagedAMLauncher fails on trunk. The console output is
{code}
Running
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher
Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 35.937 sec
FAILURE! - in
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher
testDSShell(org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher)
Time elapsed: 14.558 sec ERROR!
java.lang.RuntimeException: Failed to receive final expected state in
ApplicationReport, CurrentState=ACCEPTED,
ExpectedStates=FINISHED,FAILED,KILLED
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.UnmanagedAMLauncher.monitorApplication(UnmanagedAMLauncher.java:447)
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.UnmanagedAMLauncher.run(UnmanagedAMLauncher.java:352)
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher.testDSShell(TestUnmanagedAMLauncher.java:145)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (YARN-1450) TestUnmanagedAMLauncher#testDSShell fails on trunk
[
https://issues.apache.org/jira/browse/YARN-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Binglin Chang updated YARN-1450:
Attachment: (was: HADOOP-10130.v2.patch)
TestUnmanagedAMLauncher#testDSShell fails on trunk
--
Key: YARN-1450
URL: https://issues.apache.org/jira/browse/YARN-1450
Project: Hadoop YARN
Issue Type: Bug
Components: applications/distributed-shell
Reporter: Akira AJISAKA
Assignee: Binglin Chang
Attachments: YARN-1450.v1.patch, YARN-1450.v2.patch,
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher-output.txt,
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher.txt
TestUnmanagedAMLauncher fails on trunk. The console output is
{code}
Running
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher
Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 35.937 sec
FAILURE! - in
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher
testDSShell(org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher)
Time elapsed: 14.558 sec ERROR!
java.lang.RuntimeException: Failed to receive final expected state in
ApplicationReport, CurrentState=ACCEPTED,
ExpectedStates=FINISHED,FAILED,KILLED
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.UnmanagedAMLauncher.monitorApplication(UnmanagedAMLauncher.java:447)
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.UnmanagedAMLauncher.run(UnmanagedAMLauncher.java:352)
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher.testDSShell(TestUnmanagedAMLauncher.java:145)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (YARN-1450) TestUnmanagedAMLauncher#testDSShell fails on trunk
[
https://issues.apache.org/jira/browse/YARN-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Binglin Chang updated YARN-1450:
Attachment: HADOOP-10130.v2.patch
Attach patch generated with --no-prefix
TestUnmanagedAMLauncher#testDSShell fails on trunk
--
Key: YARN-1450
URL: https://issues.apache.org/jira/browse/YARN-1450
Project: Hadoop YARN
Issue Type: Bug
Components: applications/distributed-shell
Reporter: Akira AJISAKA
Assignee: Binglin Chang
Attachments: YARN-1450.v1.patch, YARN-1450.v2.patch,
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher-output.txt,
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher.txt
TestUnmanagedAMLauncher fails on trunk. The console output is
{code}
Running
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher
Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 35.937 sec
FAILURE! - in
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher
testDSShell(org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher)
Time elapsed: 14.558 sec ERROR!
java.lang.RuntimeException: Failed to receive final expected state in
ApplicationReport, CurrentState=ACCEPTED,
ExpectedStates=FINISHED,FAILED,KILLED
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.UnmanagedAMLauncher.monitorApplication(UnmanagedAMLauncher.java:447)
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.UnmanagedAMLauncher.run(UnmanagedAMLauncher.java:352)
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher.testDSShell(TestUnmanagedAMLauncher.java:145)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (YARN-1450) TestUnmanagedAMLauncher#testDSShell fails on trunk
[
https://issues.apache.org/jira/browse/YARN-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Binglin Chang updated YARN-1450:
Attachment: YARN-1450.v2.patch
TestUnmanagedAMLauncher#testDSShell fails on trunk
--
Key: YARN-1450
URL: https://issues.apache.org/jira/browse/YARN-1450
Project: Hadoop YARN
Issue Type: Bug
Components: applications/distributed-shell
Reporter: Akira AJISAKA
Assignee: Binglin Chang
Attachments: YARN-1450.v1.patch, YARN-1450.v2.patch,
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher-output.txt,
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher.txt
TestUnmanagedAMLauncher fails on trunk. The console output is
{code}
Running
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher
Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 35.937 sec
FAILURE! - in
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher
testDSShell(org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher)
Time elapsed: 14.558 sec ERROR!
java.lang.RuntimeException: Failed to receive final expected state in
ApplicationReport, CurrentState=ACCEPTED,
ExpectedStates=FINISHED,FAILED,KILLED
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.UnmanagedAMLauncher.monitorApplication(UnmanagedAMLauncher.java:447)
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.UnmanagedAMLauncher.run(UnmanagedAMLauncher.java:352)
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher.testDSShell(TestUnmanagedAMLauncher.java:145)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1450) TestUnmanagedAMLauncher#testDSShell fails on trunk
[
https://issues.apache.org/jira/browse/YARN-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834410#comment-13834410
]
Binglin Chang commented on YARN-1450:
-
bq. JAVA_HOME /bin/java -Xmx512m
org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster
--shell_command pwd;ls --num_containers 5
if you pass this to current distshell client, it will fail too, like your final
example in
YARN-1303[https://issues.apache.org/jira/browse/YARN-1303?focusedCommentId=13828462page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13828462],
you should call:
JAVA_HOME /bin/java -Xmx512m
org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster
--shell_command pwd;ls --num_containers 5
TestUnmanagedAMLauncher#testDSShell fails on trunk
--
Key: YARN-1450
URL: https://issues.apache.org/jira/browse/YARN-1450
Project: Hadoop YARN
Issue Type: Bug
Components: applications/distributed-shell
Reporter: Akira AJISAKA
Assignee: Binglin Chang
Attachments: YARN-1450.v1.patch, YARN-1450.v2.patch,
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher-output.txt,
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher.txt
TestUnmanagedAMLauncher fails on trunk. The console output is
{code}
Running
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher
Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 35.937 sec
FAILURE! - in
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher
testDSShell(org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher)
Time elapsed: 14.558 sec ERROR!
java.lang.RuntimeException: Failed to receive final expected state in
ApplicationReport, CurrentState=ACCEPTED,
ExpectedStates=FINISHED,FAILED,KILLED
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.UnmanagedAMLauncher.monitorApplication(UnmanagedAMLauncher.java:447)
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.UnmanagedAMLauncher.run(UnmanagedAMLauncher.java:352)
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher.testDSShell(TestUnmanagedAMLauncher.java:145)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1450) TestUnmanagedAMLauncher#testDSShell fails on trunk
[
https://issues.apache.org/jira/browse/YARN-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834414#comment-13834414
]
Binglin Chang commented on YARN-1450:
-
bq. Seems like the patch is undoing what YARN-1303 did - shouldn't be doing
that.
This is not the undoing of YARN-1303, I just add another option so some
application need to call distshell.AppMaster directly can use. it is fine that
Distshell.Client can not benefit from it.
TestUnmanagedAMLauncher#testDSShell fails on trunk
--
Key: YARN-1450
URL: https://issues.apache.org/jira/browse/YARN-1450
Project: Hadoop YARN
Issue Type: Bug
Components: applications/distributed-shell
Reporter: Akira AJISAKA
Assignee: Binglin Chang
Attachments: YARN-1450.v1.patch, YARN-1450.v2.patch,
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher-output.txt,
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher.txt
TestUnmanagedAMLauncher fails on trunk. The console output is
{code}
Running
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher
Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 35.937 sec
FAILURE! - in
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher
testDSShell(org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher)
Time elapsed: 14.558 sec ERROR!
java.lang.RuntimeException: Failed to receive final expected state in
ApplicationReport, CurrentState=ACCEPTED,
ExpectedStates=FINISHED,FAILED,KILLED
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.UnmanagedAMLauncher.monitorApplication(UnmanagedAMLauncher.java:447)
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.UnmanagedAMLauncher.run(UnmanagedAMLauncher.java:352)
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher.testDSShell(TestUnmanagedAMLauncher.java:145)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1450) TestUnmanagedAMLauncher#testDSShell fails on trunk
[
https://issues.apache.org/jira/browse/YARN-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834424#comment-13834424
]
Binglin Chang commented on YARN-1450:
-
Let me state the problem more clearly:
Story in YARN-1303:
1. Client accept --shell_command pwd;ls
2. Client can not construct a whole cmdline with proper escape support if
shell_command contains special chars like ;, so it write the pwd;ls to file
instead
3. AM started, read shell_command from file, get pwd;ls
Story in this jira YARN-1450:
1. some app directly launch AM without client, AM directly accept
--shell_command pwd;ls
TestUnmanagedAMLauncher#testDSShell fails on trunk
--
Key: YARN-1450
URL: https://issues.apache.org/jira/browse/YARN-1450
Project: Hadoop YARN
Issue Type: Bug
Components: applications/distributed-shell
Reporter: Akira AJISAKA
Assignee: Binglin Chang
Attachments: YARN-1450.v1.patch, YARN-1450.v2.patch,
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher-output.txt,
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher.txt
TestUnmanagedAMLauncher fails on trunk. The console output is
{code}
Running
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher
Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 35.937 sec
FAILURE! - in
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher
testDSShell(org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher)
Time elapsed: 14.558 sec ERROR!
java.lang.RuntimeException: Failed to receive final expected state in
ApplicationReport, CurrentState=ACCEPTED,
ExpectedStates=FINISHED,FAILED,KILLED
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.UnmanagedAMLauncher.monitorApplication(UnmanagedAMLauncher.java:447)
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.UnmanagedAMLauncher.run(UnmanagedAMLauncher.java:352)
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher.testDSShell(TestUnmanagedAMLauncher.java:145)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (YARN-1450) TestUnmanagedAMLauncher#testDSShell fails on trunk
[
https://issues.apache.org/jira/browse/YARN-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Binglin Chang updated YARN-1450:
Attachment: YARN-1450.v3.patch
The patch didn't apply because my code base was old, update new patch
TestUnmanagedAMLauncher#testDSShell fails on trunk
--
Key: YARN-1450
URL: https://issues.apache.org/jira/browse/YARN-1450
Project: Hadoop YARN
Issue Type: Bug
Components: applications/distributed-shell
Reporter: Akira AJISAKA
Assignee: Binglin Chang
Attachments: YARN-1450.v1.patch, YARN-1450.v2.patch,
YARN-1450.v3.patch,
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher-output.txt,
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher.txt
TestUnmanagedAMLauncher fails on trunk. The console output is
{code}
Running
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher
Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 35.937 sec
FAILURE! - in
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher
testDSShell(org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher)
Time elapsed: 14.558 sec ERROR!
java.lang.RuntimeException: Failed to receive final expected state in
ApplicationReport, CurrentState=ACCEPTED,
ExpectedStates=FINISHED,FAILED,KILLED
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.UnmanagedAMLauncher.monitorApplication(UnmanagedAMLauncher.java:447)
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.UnmanagedAMLauncher.run(UnmanagedAMLauncher.java:352)
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher.testDSShell(TestUnmanagedAMLauncher.java:145)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1450) TestUnmanagedAMLauncher#testDSShell fails on trunk
[
https://issues.apache.org/jira/browse/YARN-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834498#comment-13834498
]
Binglin Chang commented on YARN-1450:
-
The failed test TestUnmanagedAMLauncher#testDSShell can cover this code, so I
think new test case is not needed.
TestUnmanagedAMLauncher#testDSShell fails on trunk
--
Key: YARN-1450
URL: https://issues.apache.org/jira/browse/YARN-1450
Project: Hadoop YARN
Issue Type: Bug
Components: applications/distributed-shell
Reporter: Akira AJISAKA
Assignee: Binglin Chang
Attachments: YARN-1450.v1.patch, YARN-1450.v2.patch,
YARN-1450.v3.patch,
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher-output.txt,
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher.txt
TestUnmanagedAMLauncher fails on trunk. The console output is
{code}
Running
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher
Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 35.937 sec
FAILURE! - in
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher
testDSShell(org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher)
Time elapsed: 14.558 sec ERROR!
java.lang.RuntimeException: Failed to receive final expected state in
ApplicationReport, CurrentState=ACCEPTED,
ExpectedStates=FINISHED,FAILED,KILLED
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.UnmanagedAMLauncher.monitorApplication(UnmanagedAMLauncher.java:447)
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.UnmanagedAMLauncher.run(UnmanagedAMLauncher.java:352)
at
org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher.testDSShell(TestUnmanagedAMLauncher.java:145)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (YARN-1170) yarn proto definitions should specify package as 'hadoop.yarn'
[ https://issues.apache.org/jira/browse/YARN-1170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13764159#comment-13764159 ] Binglin Chang commented on YARN-1170: - This patch only contains proto files change, which does not affect generated java code. I checked manually by generating cpp output from those proto files, and see that the generated cpp files has correct namespaces. yarn proto definitions should specify package as 'hadoop.yarn' -- Key: YARN-1170 URL: https://issues.apache.org/jira/browse/YARN-1170 Project: Hadoop YARN Issue Type: Bug Reporter: Arun C Murthy Priority: Blocker Attachments: YARN-1170.v1.patch yarn proto definitions should specify package as 'hadoop.yarn' similar to protos with 'hadoop.common' 'hadoop.hdfs' in Common HDFS respectively. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-1170) yarn proto definitions should specify package as 'hadoop.yarn'
[ https://issues.apache.org/jira/browse/YARN-1170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated YARN-1170: Attachment: YARN-1170.v1.patch Add namespace to yarn protos also cause some compile error in mapreduce, by the way this patch add hadoop.mapreduce namespace to mapreduce protos. yarn proto definitions should specify package as 'hadoop.yarn' -- Key: YARN-1170 URL: https://issues.apache.org/jira/browse/YARN-1170 Project: Hadoop YARN Issue Type: Bug Reporter: Arun C Murthy Priority: Blocker Attachments: YARN-1170.v1.patch yarn proto definitions should specify package as 'hadoop.yarn' similar to protos with 'hadoop.common' 'hadoop.hdfs' in Common HDFS respectively. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-898) Snapshot support for distcp
Binglin Chang created YARN-898: -- Summary: Snapshot support for distcp Key: YARN-898 URL: https://issues.apache.org/jira/browse/YARN-898 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: Binglin Chang Add snapshot incremental copy ability to distcp, so we can do iterative consistent backup between hadoop clusters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (YARN-898) Snapshot support for distcp
[ https://issues.apache.org/jira/browse/YARN-898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang resolved YARN-898. Resolution: Invalid sorry, should be in HADOOP Snapshot support for distcp --- Key: YARN-898 URL: https://issues.apache.org/jira/browse/YARN-898 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: Binglin Chang Add snapshot incremental copy ability to distcp, so we can do iterative consistent backup between hadoop clusters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-783) RM and NM web server /logs pages link not working
Binglin Chang created YARN-783: -- Summary: RM and NM web server /logs pages link not working Key: YARN-783 URL: https://issues.apache.org/jira/browse/YARN-783 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: Binglin Chang RM and NM web server /logs page link to main default apps page -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-784) YARN does not provide Version info in JMX like hdfs and mapreducev1
Binglin Chang created YARN-784: -- Summary: YARN does not provide Version info in JMX like hdfs and mapreducev1 Key: YARN-784 URL: https://issues.apache.org/jira/browse/YARN-784 Project: Hadoop YARN Issue Type: Improvement Reporter: Binglin Chang Priority: Minor Some third party tool may still need this version info in JMX. Better add this for backward compatibility. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-784) YARN does not provide Version info in JMX like hdfs and mapreducev1
[ https://issues.apache.org/jira/browse/YARN-784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated YARN-784: --- Affects Version/s: 2.1.0-beta YARN does not provide Version info in JMX like hdfs and mapreducev1 --- Key: YARN-784 URL: https://issues.apache.org/jira/browse/YARN-784 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.1.0-beta Reporter: Binglin Chang Priority: Minor Some third party tool may still need this version info in JMX. Better add this for backward compatibility. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-783) RM and NM web server /logs pages link not working
[ https://issues.apache.org/jira/browse/YARN-783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13678112#comment-13678112 ] Binglin Chang commented on YARN-783: The problem seems to only exist in extra jetty context root, like /logs/ and /static/, and none root paths work fine, like /logs/userlogs/ RM and NM web server /logs pages link not working - Key: YARN-783 URL: https://issues.apache.org/jira/browse/YARN-783 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: Binglin Chang RM and NM web server /logs page link to main default apps page -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
