[jira] [Commented] (YARN-9403) GET /apps/{appid}/entities/YARN_APPLICATION accesses application table instead of entity table
[ https://issues.apache.org/jira/browse/YARN-9403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17560680#comment-17560680 ] Ashutosh Gupta commented on YARN-9403: -- Taking it up > GET /apps/{appid}/entities/YARN_APPLICATION accesses application table > instead of entity table > -- > > Key: YARN-9403 > URL: https://issues.apache.org/jira/browse/YARN-9403 > Project: Hadoop YARN > Issue Type: Sub-task > Components: ATSv2 >Affects Versions: 3.2.0 >Reporter: Prabhu Joseph >Assignee: Ashutosh Gupta >Priority: Major > Attachments: YARN-9403-001.patch, YARN-9403-002.patch, > YARN-9403-003.patch, YARN-9403-004.patch > > > {noformat}"GET /apps/{appid}/entities/YARN_APPLICATION"{noformat} accesses > application table instead of entity table. As per the doc, With this API, you > can query generic entities identified by cluster ID, application ID and > per-framework entity type. But it also provides all the apps when entityType > is set to YARN_APPLICATION. It should only access Entity Table through > {{GenericEntityReader}}. > Wrong Output: With YARN_APPLICATION entityType, all applications listed from > application tables. > {code} > [hbase@yarn-ats-3 centos]$ curl -s > "http://yarn-ats-3:8198/ws/v2/timeline/apps/application_1553258815132_0002/entities/YARN_APPLICATION?user.name=hbase&userid=hbase&flowname=word%20count"; > | jq . > [ > { > "metrics": [], > "events": [], > "createdtime": 1553258922721, > "idprefix": 0, > "isrelatedto": {}, > "relatesto": {}, > "info": { > "UID": "ats!application_1553258815132_0002", > "FROM_ID": "ats!hbase!word > count!1553258922721!application_1553258815132_0002" > }, > "configs": {}, > "type": "YARN_APPLICATION", > "id": "application_1553258815132_0002" > }, > { > "metrics": [], > "events": [], > "createdtime": 1553258825918, > "idprefix": 0, > "isrelatedto": {}, > "relatesto": {}, > "info": { > "UID": "ats!application_1553258815132_0001", > "FROM_ID": "ats!hbase!word > count!1553258825918!application_1553258815132_0001" > }, > "configs": {}, > "type": "YARN_APPLICATION", > "id": "application_1553258815132_0001" > } > ] > {code} > Right Output: With correct entity type (MAPREDUCE_JOB) it accesses entity > table for given applicationId and entityType. > {code} > [hbase@yarn-ats-3 centos]$ curl -s > "http://yarn-ats-3:8198/ws/v2/timeline/apps/application_1553258815132_0002/entities/MAPREDUCE_JOB?user.name=hbase&userid=hbase&flowname=word%20count"; > | jq . > [ > { > "metrics": [], > "events": [], > "createdtime": 1553258926667, > "idprefix": 0, > "isrelatedto": {}, > "relatesto": {}, > "info": { > "UID": > "ats!application_1553258815132_0002!MAPREDUCE_JOB!0!job_1553258815132_0002", > "FROM_ID": "ats!hbase!word > count!1553258922721!application_1553258815132_0002!MAPREDUCE_JOB!0!job_1553258815132_0002" > }, > "configs": {}, > "type": "MAPREDUCE_JOB", > "id": "job_1553258815132_0002" > } > ] > {code} > Flow Activity and Flow Run tables can also be accessed using similar way. > {code} > GET /apps/{appid}/entities/YARN_FLOW_ACTIVITY > GET /apps/{appid}/entities/YARN_FLOW_RUN > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9403) GET /apps/{appid}/entities/YARN_APPLICATION accesses application table instead of entity table
[ https://issues.apache.org/jira/browse/YARN-9403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16919363#comment-16919363 ] Hadoop QA commented on YARN-9403: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 26s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} pathlen {color} | {color:red} 0m 0s{color} | {color:red} The patch appears to contain 1 files with names longer than 240 {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 48s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m 16s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 37s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 48s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 5s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 42s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 5s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 32s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 32s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 44s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 3 new + 40 unchanged - 0 fixed = 43 total (was 40) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 57s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 30s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 48s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 13s{color} | {color:green} hadoop-yarn-server-timelineservice in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 24s{color} | {color:green} hadoop-yarn-server-timelineservice-hbase-client in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 11m 37s{color} | {color:green} hadoop-yarn-server-timelineservice-hbase-tests in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 38s{color} | {color:green} The patch does not g
[jira] [Commented] (YARN-9403) GET /apps/{appid}/entities/YARN_APPLICATION accesses application table instead of entity table
[ https://issues.apache.org/jira/browse/YARN-9403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16809557#comment-16809557 ] Prabhu Joseph commented on YARN-9403: - Thanks [~vrushalic] for reviewing this. The rest api {noformat} /apps/{appid}/entities/entityType{noformat} to fetch entities of any particular entityType and an applicationId can be misused to list other apps, flowruns and flow activities. This is not a serious issue since ACLs available but gives negative impression to the user. And also user won't know the internal terms YARN_APPLICATION, YARN_FLOW_RUN, YARN_FLOW_ACTIVITY and i think can use it for custom entities. This Jira fixes the negative scenario where it treats any value user set as entityType to fetch from entities table. {code:java} curl -s "http://yarn-ats-3:8198/ws/v2/timeline/apps/application_1553258815132_0002/entities/YARN_APPLICATION"; curl -s "http://yarn-ats-3:8198/ws/v2/timeline/apps/application_1553258815132_0002/entities/YARN_FLOW_RUN"; curl -s "http://yarn-ats-3:8198/ws/v2/timeline/apps/application_1553258815132_0002/entities/YARN_FLOW_ACTIVITY"; {code} > GET /apps/{appid}/entities/YARN_APPLICATION accesses application table > instead of entity table > -- > > Key: YARN-9403 > URL: https://issues.apache.org/jira/browse/YARN-9403 > Project: Hadoop YARN > Issue Type: Sub-task > Components: ATSv2 >Affects Versions: 3.2.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Attachments: YARN-9403-001.patch, YARN-9403-002.patch, > YARN-9403-003.patch, YARN-9403-004.patch > > > {noformat}"GET /apps/{appid}/entities/YARN_APPLICATION"{noformat} accesses > application table instead of entity table. As per the doc, With this API, you > can query generic entities identified by cluster ID, application ID and > per-framework entity type. But it also provides all the apps when entityType > is set to YARN_APPLICATION. It should only access Entity Table through > {{GenericEntityReader}}. > Wrong Output: With YARN_APPLICATION entityType, all applications listed from > application tables. > {code} > [hbase@yarn-ats-3 centos]$ curl -s > "http://yarn-ats-3:8198/ws/v2/timeline/apps/application_1553258815132_0002/entities/YARN_APPLICATION?user.name=hbase&userid=hbase&flowname=word%20count"; > | jq . > [ > { > "metrics": [], > "events": [], > "createdtime": 1553258922721, > "idprefix": 0, > "isrelatedto": {}, > "relatesto": {}, > "info": { > "UID": "ats!application_1553258815132_0002", > "FROM_ID": "ats!hbase!word > count!1553258922721!application_1553258815132_0002" > }, > "configs": {}, > "type": "YARN_APPLICATION", > "id": "application_1553258815132_0002" > }, > { > "metrics": [], > "events": [], > "createdtime": 1553258825918, > "idprefix": 0, > "isrelatedto": {}, > "relatesto": {}, > "info": { > "UID": "ats!application_1553258815132_0001", > "FROM_ID": "ats!hbase!word > count!1553258825918!application_1553258815132_0001" > }, > "configs": {}, > "type": "YARN_APPLICATION", > "id": "application_1553258815132_0001" > } > ] > {code} > Right Output: With correct entity type (MAPREDUCE_JOB) it accesses entity > table for given applicationId and entityType. > {code} > [hbase@yarn-ats-3 centos]$ curl -s > "http://yarn-ats-3:8198/ws/v2/timeline/apps/application_1553258815132_0002/entities/MAPREDUCE_JOB?user.name=hbase&userid=hbase&flowname=word%20count"; > | jq . > [ > { > "metrics": [], > "events": [], > "createdtime": 1553258926667, > "idprefix": 0, > "isrelatedto": {}, > "relatesto": {}, > "info": { > "UID": > "ats!application_1553258815132_0002!MAPREDUCE_JOB!0!job_1553258815132_0002", > "FROM_ID": "ats!hbase!word > count!1553258922721!application_1553258815132_0002!MAPREDUCE_JOB!0!job_1553258815132_0002" > }, > "configs": {}, > "type": "MAPREDUCE_JOB", > "id": "job_1553258815132_0002" > } > ] > {code} > Flow Activity and Flow Run tables can also be accessed using similar way. > {code} > GET /apps/{appid}/entities/YARN_FLOW_ACTIVITY > GET /apps/{appid}/entities/YARN_FLOW_RUN > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9403) GET /apps/{appid}/entities/YARN_APPLICATION accesses application table instead of entity table
[ https://issues.apache.org/jira/browse/YARN-9403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16809526#comment-16809526 ] Vrushali C commented on YARN-9403: -- I am not sure I understand the issue correctly. For YARN_APPLICATIOn entities, they are being written to the application table, no? If so, why do we need to go to the entities table? If there any information missing in the response that was expected. > GET /apps/{appid}/entities/YARN_APPLICATION accesses application table > instead of entity table > -- > > Key: YARN-9403 > URL: https://issues.apache.org/jira/browse/YARN-9403 > Project: Hadoop YARN > Issue Type: Sub-task > Components: ATSv2 >Affects Versions: 3.2.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Attachments: YARN-9403-001.patch, YARN-9403-002.patch, > YARN-9403-003.patch, YARN-9403-004.patch > > > {noformat}"GET /apps/{appid}/entities/YARN_APPLICATION"{noformat} accesses > application table instead of entity table. As per the doc, With this API, you > can query generic entities identified by cluster ID, application ID and > per-framework entity type. But it also provides all the apps when entityType > is set to YARN_APPLICATION. It should only access Entity Table through > {{GenericEntityReader}}. > Wrong Output: With YARN_APPLICATION entityType, all applications listed from > application tables. > {code} > [hbase@yarn-ats-3 centos]$ curl -s > "http://yarn-ats-3:8198/ws/v2/timeline/apps/application_1553258815132_0002/entities/YARN_APPLICATION?user.name=hbase&userid=hbase&flowname=word%20count"; > | jq . > [ > { > "metrics": [], > "events": [], > "createdtime": 1553258922721, > "idprefix": 0, > "isrelatedto": {}, > "relatesto": {}, > "info": { > "UID": "ats!application_1553258815132_0002", > "FROM_ID": "ats!hbase!word > count!1553258922721!application_1553258815132_0002" > }, > "configs": {}, > "type": "YARN_APPLICATION", > "id": "application_1553258815132_0002" > }, > { > "metrics": [], > "events": [], > "createdtime": 1553258825918, > "idprefix": 0, > "isrelatedto": {}, > "relatesto": {}, > "info": { > "UID": "ats!application_1553258815132_0001", > "FROM_ID": "ats!hbase!word > count!1553258825918!application_1553258815132_0001" > }, > "configs": {}, > "type": "YARN_APPLICATION", > "id": "application_1553258815132_0001" > } > ] > {code} > Right Output: With correct entity type (MAPREDUCE_JOB) it accesses entity > table for given applicationId and entityType. > {code} > [hbase@yarn-ats-3 centos]$ curl -s > "http://yarn-ats-3:8198/ws/v2/timeline/apps/application_1553258815132_0002/entities/MAPREDUCE_JOB?user.name=hbase&userid=hbase&flowname=word%20count"; > | jq . > [ > { > "metrics": [], > "events": [], > "createdtime": 1553258926667, > "idprefix": 0, > "isrelatedto": {}, > "relatesto": {}, > "info": { > "UID": > "ats!application_1553258815132_0002!MAPREDUCE_JOB!0!job_1553258815132_0002", > "FROM_ID": "ats!hbase!word > count!1553258922721!application_1553258815132_0002!MAPREDUCE_JOB!0!job_1553258815132_0002" > }, > "configs": {}, > "type": "MAPREDUCE_JOB", > "id": "job_1553258815132_0002" > } > ] > {code} > Flow Activity and Flow Run tables can also be accessed using similar way. > {code} > GET /apps/{appid}/entities/YARN_FLOW_ACTIVITY > GET /apps/{appid}/entities/YARN_FLOW_RUN > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9403) GET /apps/{appid}/entities/YARN_APPLICATION accesses application table instead of entity table
[ https://issues.apache.org/jira/browse/YARN-9403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16801958#comment-16801958 ] Hadoop QA commented on YARN-9403: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 30s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} pathlen {color} | {color:red} 0m 0s{color} | {color:red} The patch appears to contain 1 files with names longer than 240 {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 36s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 49s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 58s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 6s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 28s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 14m 18s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 8s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 59s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 9s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 41s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 56s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 3 new + 37 unchanged - 0 fixed = 40 total (was 37) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 1s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 43s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 27s{color} | {color:green} hadoop-yarn-server-timelineservice in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 32s{color} | {color:green} hadoop-yarn-server-timelineservice-hbase-client in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 17m 17s{color} | {color:green} hadoop-yarn-server-timelineservice-hbase-tests in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 34s{color} | {color:green} The patch does not g
[jira] [Commented] (YARN-9403) GET /apps/{appid}/entities/YARN_APPLICATION accesses application table instead of entity table
[ https://issues.apache.org/jira/browse/YARN-9403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16801841#comment-16801841 ] Prabhu Joseph commented on YARN-9403: - [~rohithsharma] [~vrushalic] Can you review the patch which fixes below ATSV2 Rest Api from accessing the application table instead of entity table. {code} /apps/{appid}/entities/YARN_APPLICATION {code} > GET /apps/{appid}/entities/YARN_APPLICATION accesses application table > instead of entity table > -- > > Key: YARN-9403 > URL: https://issues.apache.org/jira/browse/YARN-9403 > Project: Hadoop YARN > Issue Type: Sub-task > Components: ATSv2 >Affects Versions: 3.2.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Attachments: YARN-9403-001.patch, YARN-9403-002.patch, > YARN-9403-003.patch, YARN-9403-004.patch > > > {noformat}"GET /apps/{appid}/entities/YARN_APPLICATION"{noformat} accesses > application table instead of entity table. As per the doc, With this API, you > can query generic entities identified by cluster ID, application ID and > per-framework entity type. But it also provides all the apps when entityType > is set to YARN_APPLICATION. It should only access Entity Table through > {{GenericEntityReader}}. > Wrong Output: With YARN_APPLICATION entityType, all applications listed from > application tables. > {code} > [hbase@yarn-ats-3 centos]$ curl -s > "http://yarn-ats-3:8198/ws/v2/timeline/apps/application_1553258815132_0002/entities/YARN_APPLICATION?user.name=hbase&userid=hbase&flowname=word%20count"; > | jq . > [ > { > "metrics": [], > "events": [], > "createdtime": 1553258922721, > "idprefix": 0, > "isrelatedto": {}, > "relatesto": {}, > "info": { > "UID": "ats!application_1553258815132_0002", > "FROM_ID": "ats!hbase!word > count!1553258922721!application_1553258815132_0002" > }, > "configs": {}, > "type": "YARN_APPLICATION", > "id": "application_1553258815132_0002" > }, > { > "metrics": [], > "events": [], > "createdtime": 1553258825918, > "idprefix": 0, > "isrelatedto": {}, > "relatesto": {}, > "info": { > "UID": "ats!application_1553258815132_0001", > "FROM_ID": "ats!hbase!word > count!1553258825918!application_1553258815132_0001" > }, > "configs": {}, > "type": "YARN_APPLICATION", > "id": "application_1553258815132_0001" > } > ] > {code} > Right Output: With correct entity type (MAPREDUCE_JOB) it accesses entity > table for given applicationId and entityType. > {code} > [hbase@yarn-ats-3 centos]$ curl -s > "http://yarn-ats-3:8198/ws/v2/timeline/apps/application_1553258815132_0002/entities/MAPREDUCE_JOB?user.name=hbase&userid=hbase&flowname=word%20count"; > | jq . > [ > { > "metrics": [], > "events": [], > "createdtime": 1553258926667, > "idprefix": 0, > "isrelatedto": {}, > "relatesto": {}, > "info": { > "UID": > "ats!application_1553258815132_0002!MAPREDUCE_JOB!0!job_1553258815132_0002", > "FROM_ID": "ats!hbase!word > count!1553258922721!application_1553258815132_0002!MAPREDUCE_JOB!0!job_1553258815132_0002" > }, > "configs": {}, > "type": "MAPREDUCE_JOB", > "id": "job_1553258815132_0002" > } > ] > {code} > Flow Activity and Flow Run tables can also be accessed using similar way. > {code} > GET /apps/{appid}/entities/YARN_FLOW_ACTIVITY > GET /apps/{appid}/entities/YARN_FLOW_RUN > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9403) GET /apps/{appid}/entities/YARN_APPLICATION accesses application table instead of entity table
[ https://issues.apache.org/jira/browse/YARN-9403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16801830#comment-16801830 ] Hadoop QA commented on YARN-9403: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} pathlen {color} | {color:red} 0m 0s{color} | {color:red} The patch appears to contain 1 files with names longer than 240 {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 26s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 46s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 2s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 25s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 52s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 9s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 59s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 8s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 42s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 58s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 6 new + 37 unchanged - 0 fixed = 43 total (was 37) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 16s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 40s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 12s{color} | {color:green} hadoop-yarn-server-timelineservice in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 26s{color} | {color:green} hadoop-yarn-server-timelineservice-hbase-client in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m 22s{color} | {color:green} hadoop-yarn-server-timelineservice-hbase-tests in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 26s{color} | {color:green} The patch does not g
[jira] [Commented] (YARN-9403) GET /apps/{appid}/entities/YARN_APPLICATION accesses application table instead of entity table
[ https://issues.apache.org/jira/browse/YARN-9403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16801675#comment-16801675 ] Hadoop QA commented on YARN-9403: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 22s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} pathlen {color} | {color:red} 0m 0s{color} | {color:red} The patch appears to contain 1 files with names longer than 240 {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 6 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 26s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 7s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 25s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 13s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 40s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 14m 37s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 21s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 14s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 49s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 58s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 3 new + 50 unchanged - 1 fixed = 53 total (was 51) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 16s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 38s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 20s{color} | {color:green} hadoop-yarn-server-timelineservice in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 28s{color} | {color:green} hadoop-yarn-server-timelineservice-hbase-client in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m 49s{color} | {color:green} hadoop-yarn-server-timelineservice-hbase-tests in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 26s{color} | {color:green} The patch does not g
[jira] [Commented] (YARN-9403) GET /apps/{appid}/entities/YARN_APPLICATION accesses application table instead of entity table
[ https://issues.apache.org/jira/browse/YARN-9403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16801621#comment-16801621 ] Prabhu Joseph commented on YARN-9403: - Without Patch: {code} /apps/{appid}/entities/YARN_APPLICATION {code} throws below error {code} 2019-03-26 17:25:28,750 INFO [2129138446@qtp-1700397528-0] reader.TimelineReaderWebServices (TimelineReaderWebServices.java:handleException(181)) - Processed URL /ws/v2/timeline/apps/application_11_/entities/YARN_APPLICATION but encountered exception (Took 19 ms.) 2019-03-26 17:25:28,751 ERROR [2129138446@qtp-1700397528-0] reader.TimelineReaderWebServices (TimelineReaderWebServices.java:handleException(198)) - Error while processing REST request java.lang.NullPointerException: userId shouldn't be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:208) at org.apache.hadoop.yarn.server.timelineservice.storage.reader.ApplicationEntityReader.validateParams(ApplicationEntityReader.java:350) at org.apache.hadoop.yarn.server.timelineservice.storage.reader.TimelineEntityReader.readEntities(TimelineEntityReader.java:273) at org.apache.hadoop.yarn.server.timelineservice.storage.HBaseTimelineReaderImpl.getEntities(HBaseTimelineReaderImpl.java:150) at org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderManager.getEntities(TimelineReaderManager.java:152) at org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.getEntities(TimelineReaderWebServices.java:622) at org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.getEntities(TimelineReaderWebServices.java:482) {code} With Patch: It accesses Entity table instead of Application Table and returns empty list as there won;t be any entities with YARN_APPLICATION. > GET /apps/{appid}/entities/YARN_APPLICATION accesses application table > instead of entity table > -- > > Key: YARN-9403 > URL: https://issues.apache.org/jira/browse/YARN-9403 > Project: Hadoop YARN > Issue Type: Sub-task > Components: ATSv2 >Affects Versions: 3.2.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Attachments: YARN-9403-001.patch, YARN-9403-002.patch > > > {noformat}"GET /apps/{appid}/entities/YARN_APPLICATION"{noformat} accesses > application table instead of entity table. As per the doc, With this API, you > can query generic entities identified by cluster ID, application ID and > per-framework entity type. But it also provides all the apps when entityType > is set to YARN_APPLICATION. It should only access Entity Table through > {{GenericEntityReader}}. > Wrong Output: With YARN_APPLICATION entityType, all applications listed from > application tables. > {code} > [hbase@yarn-ats-3 centos]$ curl -s > "http://yarn-ats-3:8198/ws/v2/timeline/apps/application_1553258815132_0002/entities/YARN_APPLICATION?user.name=hbase&userid=hbase&flowname=word%20count"; > | jq . > [ > { > "metrics": [], > "events": [], > "createdtime": 1553258922721, > "idprefix": 0, > "isrelatedto": {}, > "relatesto": {}, > "info": { > "UID": "ats!application_1553258815132_0002", > "FROM_ID": "ats!hbase!word > count!1553258922721!application_1553258815132_0002" > }, > "configs": {}, > "type": "YARN_APPLICATION", > "id": "application_1553258815132_0002" > }, > { > "metrics": [], > "events": [], > "createdtime": 1553258825918, > "idprefix": 0, > "isrelatedto": {}, > "relatesto": {}, > "info": { > "UID": "ats!application_1553258815132_0001", > "FROM_ID": "ats!hbase!word > count!1553258825918!application_1553258815132_0001" > }, > "configs": {}, > "type": "YARN_APPLICATION", > "id": "application_1553258815132_0001" > } > ] > {code} > Right Output: With correct entity type (MAPREDUCE_JOB) it accesses entity > table for given applicationId and entityType. > {code} > [hbase@yarn-ats-3 centos]$ curl -s > "http://yarn-ats-3:8198/ws/v2/timeline/apps/application_1553258815132_0002/entities/MAPREDUCE_JOB?user.name=hbase&userid=hbase&flowname=word%20count"; > | jq . > [ > { > "metrics": [], > "events": [], > "createdtime": 1553258926667, > "idprefix": 0, > "isrelatedto": {}, > "relatesto": {}, > "info": { > "UID": > "ats!application_1553258815132_0002!MAPREDUCE_JOB!0!job_1553258815132_0002", > "FROM_ID": "ats!hbase!word > count!1553258922721!application_1553258815132_0002!MAPREDUCE_JOB!0!job_1553258815132_0002" > }, > "configs": {}, > "type": "MAPREDUCE_JOB", > "id": "job_1553258815132_0002" > } > ] > {code} -- Th
[jira] [Commented] (YARN-9403) GET /apps/{appid}/entities/YARN_APPLICATION accesses application table instead of entity table
[ https://issues.apache.org/jira/browse/YARN-9403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16800936#comment-16800936 ] Hadoop QA commented on YARN-9403: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 39s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} pathlen {color} | {color:red} 0m 0s{color} | {color:red} The patch appears to contain 1 files with names longer than 240 {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 8s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m 17s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 45s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 12s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 9s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 15m 8s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 25s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 57s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 26s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 26s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 1m 10s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 5 new + 40 unchanged - 0 fixed = 45 total (was 40) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 58s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 14m 5s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 25s{color} | {color:green} hadoop-yarn-server-timelineservice in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 28s{color} | {color:green} hadoop-yarn-server-timelineservice-hbase-client in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 34s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 73m 56s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f | | JIRA Issue | YARN-9403 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12963641/YARN-9403-001.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 3bded9043cbc 4.4.0-138-generic