[jira] [Commented] (YARN-4985) Refactor the coprocessor code & other definition classes into independent packages
[ https://issues.apache.org/jira/browse/YARN-4985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879335#comment-15879335 ] Sangjin Lee commented on YARN-4985: --- Yes, I agree. > Refactor the coprocessor code & other definition classes into independent > packages > -- > > Key: YARN-4985 > URL: https://issues.apache.org/jira/browse/YARN-4985 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Haibo Chen > Labels: YARN-5355 > Attachments: YARN-4985-YARN-5355.poc.patch, > YARN-4985-YARN-5355.prelim.patch > > > As part of the coprocessor deployment, we have realized that it will be much > cleaner to have the coprocessor code sit in a package which does not depend > on hadoop-yarn-server classes. It only needs hbase and other util classes. > These util classes and tag definition related classes can be refactored into > their own independent "definition" class package so that making changes to > coprocessor code, upgrading hbase, deploying hbase on a different hadoop > version cluster etc all becomes operationally much easier and less error > prone to having different library jars etc. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4985) Refactor the coprocessor code & other definition classes into independent packages
[ https://issues.apache.org/jira/browse/YARN-4985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879219#comment-15879219 ] Haibo Chen commented on YARN-4985: -- Let's leave it as is then. Should we close this as won't fix? > Refactor the coprocessor code & other definition classes into independent > packages > -- > > Key: YARN-4985 > URL: https://issues.apache.org/jira/browse/YARN-4985 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Haibo Chen > Labels: YARN-5355 > Attachments: YARN-4985-YARN-5355.poc.patch, > YARN-4985-YARN-5355.prelim.patch > > > As part of the coprocessor deployment, we have realized that it will be much > cleaner to have the coprocessor code sit in a package which does not depend > on hadoop-yarn-server classes. It only needs hbase and other util classes. > These util classes and tag definition related classes can be refactored into > their own independent "definition" class package so that making changes to > coprocessor code, upgrading hbase, deploying hbase on a different hadoop > version cluster etc all becomes operationally much easier and less error > prone to having different library jars etc. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4985) Refactor the coprocessor code & other definition classes into independent packages
[ https://issues.apache.org/jira/browse/YARN-4985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879210#comment-15879210 ] Sangjin Lee commented on YARN-4985: --- I went over the coprocessor code in some detail, and largely confirmed what you mentioned above. There are a number of classes that the coprocessor code depends on that the hbase "client" code also depends on; e.g. FlowRunColumn. The only code that can probably moved entirely into hbase-server is those HBaseTimelineStorageUtils methods that are exercised by the coprocessor. The only callers are the coprocessor in that case. There might be a way to make maven create a combined jar file for the coprocessor purposes, but I can't think of a straightforward way to do this. It'd probably involve a pretty hacky way to pull that off, and I'm not sure if the benefit justifies the work... > Refactor the coprocessor code & other definition classes into independent > packages > -- > > Key: YARN-4985 > URL: https://issues.apache.org/jira/browse/YARN-4985 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Haibo Chen > Labels: YARN-5355 > Attachments: YARN-4985-YARN-5355.poc.patch, > YARN-4985-YARN-5355.prelim.patch > > > As part of the coprocessor deployment, we have realized that it will be much > cleaner to have the coprocessor code sit in a package which does not depend > on hadoop-yarn-server classes. It only needs hbase and other util classes. > These util classes and tag definition related classes can be refactored into > their own independent "definition" class package so that making changes to > coprocessor code, upgrading hbase, deploying hbase on a different hadoop > version cluster etc all becomes operationally much easier and less error > prone to having different library jars etc. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4985) Refactor the coprocessor code & other definition classes into independent packages
[ https://issues.apache.org/jira/browse/YARN-4985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879030#comment-15879030 ] Haibo Chen commented on YARN-4985: -- bq. What is the implication of not doing this refactoring? Is the original problem of being able to build Hadoop and HBase from source resolved without this, or does it depend on this? The problem of building Hadoop and HBase2.0 has been resolved in YARN-5667. This is more of an effort to make HBase 2.0 & Hadoop 3 integration in cases of HBase 2.0 changes. and improve code organization to explicate, in the form of modules, the issue that we are dealing with two hadoop-common versions. I'd agree with you that if there is too much work, given that the benefits are very limited, we should not do the refactoring. > Refactor the coprocessor code & other definition classes into independent > packages > -- > > Key: YARN-4985 > URL: https://issues.apache.org/jira/browse/YARN-4985 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Haibo Chen > Labels: YARN-5355 > Attachments: YARN-4985-YARN-5355.poc.patch, > YARN-4985-YARN-5355.prelim.patch > > > As part of the coprocessor deployment, we have realized that it will be much > cleaner to have the coprocessor code sit in a package which does not depend > on hadoop-yarn-server classes. It only needs hbase and other util classes. > These util classes and tag definition related classes can be refactored into > their own independent "definition" class package so that making changes to > coprocessor code, upgrading hbase, deploying hbase on a different hadoop > version cluster etc all becomes operationally much easier and less error > prone to having different library jars etc. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4985) Refactor the coprocessor code & other definition classes into independent packages
[ https://issues.apache.org/jira/browse/YARN-4985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15878989#comment-15878989 ] Sangjin Lee commented on YARN-4985: --- Let me look through the code and see what kind of dependencies are emanating from the coprocessor code. I'll come back with some findings. What is the implication of not doing this refactoring? Is the original problem of being able to build Hadoop and HBase from source resolved without this, or does it depend on this? > Refactor the coprocessor code & other definition classes into independent > packages > -- > > Key: YARN-4985 > URL: https://issues.apache.org/jira/browse/YARN-4985 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Haibo Chen > Labels: YARN-5355 > Attachments: YARN-4985-YARN-5355.poc.patch, > YARN-4985-YARN-5355.prelim.patch > > > As part of the coprocessor deployment, we have realized that it will be much > cleaner to have the coprocessor code sit in a package which does not depend > on hadoop-yarn-server classes. It only needs hbase and other util classes. > These util classes and tag definition related classes can be refactored into > their own independent "definition" class package so that making changes to > coprocessor code, upgrading hbase, deploying hbase on a different hadoop > version cluster etc all becomes operationally much easier and less error > prone to having different library jars etc. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4985) Refactor the coprocessor code & other definition classes into independent packages
[ https://issues.apache.org/jira/browse/YARN-4985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15878866#comment-15878866 ] Haibo Chen commented on YARN-4985: -- Sorry for missing your question, [~sjlee0]. One one hand, FlowScanner needs the byte representation of FlowRunColumn FlowRunColumnPrefix and their ValueConverters. Because these getter methods are defined in the superclass/interface, changing them means changing all their sibling classes. On the other hand, FlowScanner also depends on AggregationOperation and AggregationCompactionDimension, which are used extensively by hbase-client side code as well. Based on the two observations, I think having a new common module is probably easier than trying to move coprocessor dependencies into hbase-server module. > Refactor the coprocessor code & other definition classes into independent > packages > -- > > Key: YARN-4985 > URL: https://issues.apache.org/jira/browse/YARN-4985 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Haibo Chen > Labels: YARN-5355 > Attachments: YARN-4985-YARN-5355.poc.patch, > YARN-4985-YARN-5355.prelim.patch > > > As part of the coprocessor deployment, we have realized that it will be much > cleaner to have the coprocessor code sit in a package which does not depend > on hadoop-yarn-server classes. It only needs hbase and other util classes. > These util classes and tag definition related classes can be refactored into > their own independent "definition" class package so that making changes to > coprocessor code, upgrading hbase, deploying hbase on a different hadoop > version cluster etc all becomes operationally much easier and less error > prone to having different library jars etc. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4985) Refactor the coprocessor code & other definition classes into independent packages
[
https://issues.apache.org/jira/browse/YARN-4985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15877352#comment-15877352
]
Hadoop QA commented on YARN-4985:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 13 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
10s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
27s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
43s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m
10s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
20s{color} | {color:green} YARN-5355 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests
hadoop-yarn-project {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
36s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase
in YARN-5355 has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
52s{color} | {color:green} YARN-5355 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
45s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 40s{color} | {color:orange} root: The patch generated 33 new + 13 unchanged
- 2 fixed = 46 total (was 15) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
11s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase
hadoop-yarn-project
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
19s{color} | {color:red} hadoop-yarn-server-timelineservice-hbase-schema in the
patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
20s{color} | {color:red} hadoop-yarn-server-timelineservice-hbase-client in the
patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
26s{color} | {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase
generated 15 new + 0 unchanged - 0 fixed = 15 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
19s{color} | {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-schema
generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 2m
10s{color} | {color:red}
[jira] [Commented] (YARN-4985) Refactor the coprocessor code & other definition classes into independent packages
[ https://issues.apache.org/jira/browse/YARN-4985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15877293#comment-15877293 ] Sangjin Lee commented on YARN-4985: --- Thanks for the update [~haibochen]. bq. With this new module, we do still have the coprocessor installation issue. I am totally speculating here. Is it possible to configure maven so that it will combine hbase-schema and hbase-server into one jar, as a workaround? It might be doable, but it may involve a fairly advance maven trick to pull that off. I haven't thought about what it might take. Going back to the original question, were you able to see if there is any way we can limit the dependency coming from the coprocessor code? If that is possible, that is the best case scenario. > Refactor the coprocessor code & other definition classes into independent > packages > -- > > Key: YARN-4985 > URL: https://issues.apache.org/jira/browse/YARN-4985 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Haibo Chen > Labels: YARN-5355 > Attachments: YARN-4985-YARN-5355.poc.patch, > YARN-4985-YARN-5355.prelim.patch > > > As part of the coprocessor deployment, we have realized that it will be much > cleaner to have the coprocessor code sit in a package which does not depend > on hadoop-yarn-server classes. It only needs hbase and other util classes. > These util classes and tag definition related classes can be refactored into > their own independent "definition" class package so that making changes to > coprocessor code, upgrading hbase, deploying hbase on a different hadoop > version cluster etc all becomes operationally much easier and less error > prone to having different library jars etc. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4985) Refactor the coprocessor code & other definition classes into independent packages
[ https://issues.apache.org/jira/browse/YARN-4985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876620#comment-15876620 ] Haibo Chen commented on YARN-4985: -- Apologies for my delayed response! I did not have answers to your questions, so I took some time to actually try it out. Attached is my POC patch. I managed to extract a common module that both client and server code depend on. The number of dependencies of the new schema module is much smaller, but still include hadoop-yarn-api (for use of ApplicationId) and hadoop-yarn-server-applicationhistoryservice (for use of GenericObjectMapper), as I think ValueConverters belong to schema. With this new module, the undesirable dependency of hbase-server module on hbase-client is no longer necessary. I was not able to, however, redistribute tests in hbase-tests into client and server modules. The reason is that all tests, regardless of whether it is server or client, depend on HBaseTestingUtility which only works with hadoop-common-2.5.1. Therefore, in that sense, I think both tests for hbase-client and tests for hbase-server should still reside in the same module. With this new module, we do still have the coprocessor installation issue. I am totally speculating here. Is it possible to configure maven so that it will combine hbase-schema and hbase-server into one jar, as a workaround? > Refactor the coprocessor code & other definition classes into independent > packages > -- > > Key: YARN-4985 > URL: https://issues.apache.org/jira/browse/YARN-4985 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Haibo Chen > Labels: YARN-5355 > Attachments: YARN-4985-YARN-5355.prelim.patch > > > As part of the coprocessor deployment, we have realized that it will be much > cleaner to have the coprocessor code sit in a package which does not depend > on hadoop-yarn-server classes. It only needs hbase and other util classes. > These util classes and tag definition related classes can be refactored into > their own independent "definition" class package so that making changes to > coprocessor code, upgrading hbase, deploying hbase on a different hadoop > version cluster etc all becomes operationally much easier and less error > prone to having different library jars etc. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4985) Refactor the coprocessor code & other definition classes into independent packages
[ https://issues.apache.org/jira/browse/YARN-4985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860136#comment-15860136 ] Sangjin Lee commented on YARN-4985: --- Thanks for refreshing my memory on this Haibo. Yes, I recall that was the main issue. To tease out client and server correctly, we'd need a separate (common) module both client and server depends on. The only way we could avoid it is if all the fine-grained downstream dependencies can be isolated and moved into the hbase-server module. Do you know if it is feasible? If that is not feasible, maybe this refactoring would be more problematic than beneficial. What would we lose if we didn't do this refactoring? Also, if we didn't do this refactoring and kept thins as is (just timelineservice-hbase), do we still have the dependency issues in terms of installing the coprocessor (we need to follow all transitive dependencies between classes)? If so, that needs to be addressed no matter what. > Refactor the coprocessor code & other definition classes into independent > packages > -- > > Key: YARN-4985 > URL: https://issues.apache.org/jira/browse/YARN-4985 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Haibo Chen > Labels: YARN-5355 > Attachments: YARN-4985-YARN-5355.prelim.patch > > > As part of the coprocessor deployment, we have realized that it will be much > cleaner to have the coprocessor code sit in a package which does not depend > on hadoop-yarn-server classes. It only needs hbase and other util classes. > These util classes and tag definition related classes can be refactored into > their own independent "definition" class package so that making changes to > coprocessor code, upgrading hbase, deploying hbase on a different hadoop > version cluster etc all becomes operationally much easier and less error > prone to having different library jars etc. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4985) Refactor the coprocessor code & other definition classes into independent packages
[ https://issues.apache.org/jira/browse/YARN-4985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15859912#comment-15859912 ] Haibo Chen commented on YARN-4985: -- The dependency is not mainly because all tests are in the hbase-server module as they are now, but because FlowScanner needs to reference FlowRun table schema classes. If we were to extract another module that only includes table/column definitions, we can avoid the dependency from hbase-server on hbase-client. But hbase-client code seems quite coupled to the table schema code due to current code implementation (read & write methods are defined within the Table/Column classes). I will try to break the current hbase-client module in my patch into hbase-schema and hbase-client. This way, we should not have the undesirable dependency any more. That said, I think my previous concern about loading coprocessor from HDFS is still valid, since we will still have hbase-server depend on hbase-schema. > Refactor the coprocessor code & other definition classes into independent > packages > -- > > Key: YARN-4985 > URL: https://issues.apache.org/jira/browse/YARN-4985 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Haibo Chen > Labels: YARN-5355 > Attachments: YARN-4985-YARN-5355.prelim.patch > > > As part of the coprocessor deployment, we have realized that it will be much > cleaner to have the coprocessor code sit in a package which does not depend > on hadoop-yarn-server classes. It only needs hbase and other util classes. > These util classes and tag definition related classes can be refactored into > their own independent "definition" class package so that making changes to > coprocessor code, upgrading hbase, deploying hbase on a different hadoop > version cluster etc all becomes operationally much easier and less error > prone to having different library jars etc. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4985) Refactor the coprocessor code & other definition classes into independent packages
[ https://issues.apache.org/jira/browse/YARN-4985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15859055#comment-15859055 ] Sangjin Lee commented on YARN-4985: --- [~haibochen] Is it possible to create the hbase-server module so that it does *not* depend on the hbase-client module? That's what I meant by watching out for dependencies in the hbase-server code. It would be ideal if the hbase-server code does not have to depend on the hbase-client code. It might mean splitting the tests and putting some in the client and others in the server. > Refactor the coprocessor code & other definition classes into independent > packages > -- > > Key: YARN-4985 > URL: https://issues.apache.org/jira/browse/YARN-4985 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Haibo Chen > Labels: YARN-5355 > Attachments: YARN-4985-YARN-5355.prelim.patch > > > As part of the coprocessor deployment, we have realized that it will be much > cleaner to have the coprocessor code sit in a package which does not depend > on hadoop-yarn-server classes. It only needs hbase and other util classes. > These util classes and tag definition related classes can be refactored into > their own independent "definition" class package so that making changes to > coprocessor code, upgrading hbase, deploying hbase on a different hadoop > version cluster etc all becomes operationally much easier and less error > prone to having different library jars etc. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4985) Refactor the coprocessor code & other definition classes into independent packages
[
https://issues.apache.org/jira/browse/YARN-4985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15858974#comment-15858974
]
Hadoop QA commented on YARN-4985:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 27 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
45s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
19s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
47s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 7m
14s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
53s{color} | {color:green} YARN-5355 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests
hadoop-yarn-project {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
37s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase
in YARN-5355 has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
51s{color} | {color:green} YARN-5355 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
23s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 48s{color} | {color:orange} root: The patch generated 5 new + 10 unchanged -
5 fixed = 15 total (was 15) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 7m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
9s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase
hadoop-yarn-project {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
20s{color} | {color:red} hadoop-yarn-server-timelineservice-hbase-client in the
patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
15s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 61m 2s{color}
| {color:red} hadoop-yarn-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 5m
14s{color} | {color:green} hadoop-yarn-server-timelineservice-hbase in the
patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
25s{color} | {color:green} hadoop-yarn-server-timelineservice-hbase-client in
the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}
[jira] [Commented] (YARN-4985) Refactor the coprocessor code & other definition classes into independent packages
[ https://issues.apache.org/jira/browse/YARN-4985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15858743#comment-15858743 ] Haibo Chen commented on YARN-4985: -- Uploading a patch for preliminary review. The hbase-backend module has been split into two separate modules based on the versions of hadoop-common/hdfs/auth that they need to run under. Coprocessor code and code in hbase-tests are now in the same module, that is, *timelineservice-hbase-server, which depends on *timelineservice-hbase-client that includes all table schema and hbase code executed in YARN trunk. YARN-6094 has enabled dynamic loading of coprocessor from HDFS, now if coprocessor code lives in a hbase-server jar that depends on hbase-client jar, I don't think the dynamic loading from HDFS will work unless HBase somehow knows how to pull in the dependent jars. Thoughts on workaround, [~sjlee0]? In the meantime, I will continue to verify maven dependencies to make sure they are clean. > Refactor the coprocessor code & other definition classes into independent > packages > -- > > Key: YARN-4985 > URL: https://issues.apache.org/jira/browse/YARN-4985 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Haibo Chen > Labels: YARN-5355 > Attachments: YARN-4985-YARN-5355.prelim.patch > > > As part of the coprocessor deployment, we have realized that it will be much > cleaner to have the coprocessor code sit in a package which does not depend > on hadoop-yarn-server classes. It only needs hbase and other util classes. > These util classes and tag definition related classes can be refactored into > their own independent "definition" class package so that making changes to > coprocessor code, upgrading hbase, deploying hbase on a different hadoop > version cluster etc all becomes operationally much easier and less error > prone to having different library jars etc. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4985) Refactor the coprocessor code & other definition classes into independent packages
[
https://issues.apache.org/jira/browse/YARN-4985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15843111#comment-15843111
]
Sangjin Lee commented on YARN-4985:
---
At the end of the day, {{TimelineSchemaCreator}} is a standalone process.
Either {{bin/hbase}} or {{bin/hadoop}} would be able to launch the schema
creator. It doesn't exercise any hbase server code. In that sense, it is
probably closer to the client module than the server module.
> Refactor the coprocessor code & other definition classes into independent
> packages
> --
>
> Key: YARN-4985
> URL: https://issues.apache.org/jira/browse/YARN-4985
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Vrushali C
>Assignee: Haibo Chen
> Labels: YARN-5355
>
> As part of the coprocessor deployment, we have realized that it will be much
> cleaner to have the coprocessor code sit in a package which does not depend
> on hadoop-yarn-server classes. It only needs hbase and other util classes.
> These util classes and tag definition related classes can be refactored into
> their own independent "definition" class package so that making changes to
> coprocessor code, upgrading hbase, deploying hbase on a different hadoop
> version cluster etc all becomes operationally much easier and less error
> prone to having different library jars etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4985) Refactor the coprocessor code & other definition classes into independent packages
[ https://issues.apache.org/jira/browse/YARN-4985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15838583#comment-15838583 ] Haibo Chen commented on YARN-4985: -- [~sjlee0] Can you elaborate a little more on why the schema creator can also be "client"? My understanding is that since we run "bin/hbase org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator", the hadoop dependencies will be provided by the hbase cluster and therefore their versions will be $hbase-compatible-hadoop.version}. > Refactor the coprocessor code & other definition classes into independent > packages > -- > > Key: YARN-4985 > URL: https://issues.apache.org/jira/browse/YARN-4985 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Haibo Chen > Labels: YARN-5355 > > As part of the coprocessor deployment, we have realized that it will be much > cleaner to have the coprocessor code sit in a package which does not depend > on hadoop-yarn-server classes. It only needs hbase and other util classes. > These util classes and tag definition related classes can be refactored into > their own independent "definition" class package so that making changes to > coprocessor code, upgrading hbase, deploying hbase on a different hadoop > version cluster etc all becomes operationally much easier and less error > prone to having different library jars etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4985) Refactor the coprocessor code & other definition classes into independent packages
[ https://issues.apache.org/jira/browse/YARN-4985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830466#comment-15830466 ] Vrushali C commented on YARN-4985: -- thanks [~sjlee0] . [~haibochen] please feel free to reassign the jira to yourself as we dicussed in today's call. > Refactor the coprocessor code & other definition classes into independent > packages > -- > > Key: YARN-4985 > URL: https://issues.apache.org/jira/browse/YARN-4985 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Vrushali C > Labels: YARN-5355 > > As part of the coprocessor deployment, we have realized that it will be much > cleaner to have the coprocessor code sit in a package which does not depend > on hadoop-yarn-server classes. It only needs hbase and other util classes. > These util classes and tag definition related classes can be refactored into > their own independent "definition" class package so that making changes to > coprocessor code, upgrading hbase, deploying hbase on a different hadoop > version cluster etc all becomes operationally much easier and less error > prone to having different library jars etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4985) Refactor the coprocessor code & other definition classes into independent packages
[ https://issues.apache.org/jira/browse/YARN-4985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830428#comment-15830428 ] Sangjin Lee commented on YARN-4985: --- As part of this, it would also be good to see if it is feasible to get rid of the hbase-tests module and relocate the tests into the client and the server modules respectively. Once client and server are separated, the hbase-tests module becomes awkward. Another part about the "server" module: since it should run in the hbase runtime, the pom should prescribe the hbase version and the appropriate hadoop version, rather than using the trunk hadoop version, for its dependencies. In other words, the pom for the server module should strongly resemble the current hbase-tests module. The schema creator is an interesting case. I think a case can be made for it to be consider either "server" or "client". If it becomes too much work to locate it in the server module, I don't think it's a bad idea to keep it in the client module. > Refactor the coprocessor code & other definition classes into independent > packages > -- > > Key: YARN-4985 > URL: https://issues.apache.org/jira/browse/YARN-4985 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Vrushali C > Labels: YARN-5355 > > As part of the coprocessor deployment, we have realized that it will be much > cleaner to have the coprocessor code sit in a package which does not depend > on hadoop-yarn-server classes. It only needs hbase and other util classes. > These util classes and tag definition related classes can be refactored into > their own independent "definition" class package so that making changes to > coprocessor code, upgrading hbase, deploying hbase on a different hadoop > version cluster etc all becomes operationally much easier and less error > prone to having different library jars etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4985) Refactor the coprocessor code & other definition classes into independent packages
[ https://issues.apache.org/jira/browse/YARN-4985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566476#comment-15566476 ] Haibo Chen commented on YARN-4985: -- [~vrushalic] Want to get your take on how we want to extract coprocessor code. IIUC, coprocessor code is mostly self-contained except that it depends on ValueConverter, which is part of the hbase table schema. Both the code run within yarn and the code run within hbase server depend on the schema code. This makes me think that if there need to be three modules for ATSv2-hbase code, hbase-backend-table-schema, hbase-backend-in-yarn and hbase-backend-in-hbase. And the dependency will be like hbase-backend-table-schema +hbase-backend-in-yarn +hbase-backend-in-hbase The downside of this however, is the change will be non-trivial. Another question I had is what about TimelineSchemaCreator. Is this also run in hbase server, thus, should sit along with Coprocessor? Thoughts? > Refactor the coprocessor code & other definition classes into independent > packages > -- > > Key: YARN-4985 > URL: https://issues.apache.org/jira/browse/YARN-4985 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Vrushali C > Labels: YARN-5355 > > As part of the coprocessor deployment, we have realized that it will be much > cleaner to have the coprocessor code sit in a package which does not depend > on hadoop-yarn-server classes. It only needs hbase and other util classes. > These util classes and tag definition related classes can be refactored into > their own independent "definition" class package so that making changes to > coprocessor code, upgrading hbase, deploying hbase on a different hadoop > version cluster etc all becomes operationally much easier and less error > prone to having different library jars etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
