[jira] [Commented] (YARN-3663) Federation State and Policy Store (DBMS implementation)
[ https://issues.apache.org/jira/browse/YARN-3663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116634#comment-17116634 ] Gabor Somogyi commented on YARN-3663: - [~curino] [~giovanni.fumarola] I've the same question just like [~busbey]. Spark upgraded to hadoop 3.2 and mssql-jdbc client side dependency infiltrated. Is this needed on the client side? > Federation State and Policy Store (DBMS implementation) > --- > > Key: YARN-3663 > URL: https://issues.apache.org/jira/browse/YARN-3663 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Affects Versions: YARN-2915 >Reporter: Giovanni Matteo Fumarola >Assignee: Giovanni Matteo Fumarola >Priority: Major > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: YARN-3663-YARN-2915.v1.patch, > YARN-3663-YARN-2915.v2.patch, YARN-3663-YARN-2915.v3.patch, > YARN-3663-YARN-2915.v4.patch > > > This JIRA tracks a SQL-based implementation of the Federation State and > Policy Store, which implements YARN-3662 APIs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3663) Federation State and Policy Store (DBMS implementation)
[ https://issues.apache.org/jira/browse/YARN-3663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16155989#comment-16155989 ] Sean Busbey commented on YARN-3663: --- Sorry for the late comment. The addition of the mssql-jdbc jar here is leaking into the client facing dependencies. Do y'all happen to know off hand if jdbc access is needed client side for this feature to work? > Federation State and Policy Store (DBMS implementation) > --- > > Key: YARN-3663 > URL: https://issues.apache.org/jira/browse/YARN-3663 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Affects Versions: YARN-2915 >Reporter: Giovanni Matteo Fumarola >Assignee: Giovanni Matteo Fumarola > Fix For: 3.0.0-beta1 > > Attachments: YARN-3663-YARN-2915.v1.patch, > YARN-3663-YARN-2915.v2.patch, YARN-3663-YARN-2915.v3.patch, > YARN-3663-YARN-2915.v4.patch > > > This JIRA tracks a SQL-based implementation of the Federation State and > Policy Store, which implements YARN-3662 APIs. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3663) Federation State and Policy Store (DBMS implementation)
[ https://issues.apache.org/jira/browse/YARN-3663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15983770#comment-15983770 ] Giovanni Matteo Fumarola commented on YARN-3663: For tracking the refactoring: YARN-6526 > Federation State and Policy Store (DBMS implementation) > --- > > Key: YARN-3663 > URL: https://issues.apache.org/jira/browse/YARN-3663 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Affects Versions: YARN-2915 >Reporter: Giovanni Matteo Fumarola >Assignee: Giovanni Matteo Fumarola > Fix For: YARN-2915 > > Attachments: YARN-3663-YARN-2915.v1.patch, > YARN-3663-YARN-2915.v2.patch, YARN-3663-YARN-2915.v3.patch, > YARN-3663-YARN-2915.v4.patch > > > This JIRA tracks a SQL-based implementation of the Federation State and > Policy Store, which implements YARN-3662 APIs. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3663) Federation State and Policy Store (DBMS implementation)
[ https://issues.apache.org/jira/browse/YARN-3663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15983736#comment-15983736 ] Carlo Curino commented on YARN-3663: Thanks [~giovanni.fumarola] for the contribution and addressing the review, I committed the patch to branch YARN-2915. > Federation State and Policy Store (DBMS implementation) > --- > > Key: YARN-3663 > URL: https://issues.apache.org/jira/browse/YARN-3663 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Affects Versions: YARN-2915 >Reporter: Giovanni Matteo Fumarola >Assignee: Giovanni Matteo Fumarola > Fix For: YARN-2915 > > Attachments: YARN-3663-YARN-2915.v1.patch, > YARN-3663-YARN-2915.v2.patch, YARN-3663-YARN-2915.v3.patch, > YARN-3663-YARN-2915.v4.patch > > > This JIRA tracks a SQL-based implementation of the Federation State and > Policy Store, which implements YARN-3662 APIs. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3663) Federation State and Policy Store (DBMS implementation)
[ https://issues.apache.org/jira/browse/YARN-3663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15983673#comment-15983673 ] Giovanni Matteo Fumarola commented on YARN-3663: Thanks [~curino]. I just uploaded the new patch (V4). I am going to open a new jira about the optimization and refactoring. > Federation State and Policy Store (DBMS implementation) > --- > > Key: YARN-3663 > URL: https://issues.apache.org/jira/browse/YARN-3663 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Affects Versions: YARN-2915 >Reporter: Giovanni Matteo Fumarola >Assignee: Giovanni Matteo Fumarola > Attachments: YARN-3663-YARN-2915.v1.patch, > YARN-3663-YARN-2915.v2.patch, YARN-3663-YARN-2915.v3.patch, > YARN-3663-YARN-2915.v4.patch > > > This JIRA tracks a SQL-based implementation of the Federation State and > Policy Store, which implements YARN-3662 APIs. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3663) Federation State and Policy Store (DBMS implementation)
[
https://issues.apache.org/jira/browse/YARN-3663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15982263#comment-15982263
]
Carlo Curino commented on YARN-3663:
[~giovanni.fumarola] thanks for addressing my comments. I understand all your
answers, and I am ok to postpone some of the refactorings I asked (we should
think a bit more about especially for {{SQLFederationStateStore}} point 7).
Overall the patch is +1, but please:
# Open all the follow-up JIRAs we mentioned as sub-tasks of YARN-5597
(federation v2)
# Address the first 2 checkstyle ({{UTC_CALENDAR}})... I think it should be
{{private}}, and the naming could be a standard {{utcCalendar}}.
Patch is good to commit once that is done. If you get it done tonight or
tomorrow morning, I will commit it tomorrow late morning, otherwise I asked
[~subru] to take-over reviewing and commit,
as I will be traveling with limited connectivity for a few days.
> Federation State and Policy Store (DBMS implementation)
> ---
>
> Key: YARN-3663
> URL: https://issues.apache.org/jira/browse/YARN-3663
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Affects Versions: YARN-2915
>Reporter: Giovanni Matteo Fumarola
>Assignee: Giovanni Matteo Fumarola
> Attachments: YARN-3663-YARN-2915.v1.patch,
> YARN-3663-YARN-2915.v2.patch, YARN-3663-YARN-2915.v3.patch
>
>
> This JIRA tracks a SQL-based implementation of the Federation State and
> Policy Store, which implements YARN-3662 APIs.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3663) Federation State and Policy Store (DBMS implementation)
[ https://issues.apache.org/jira/browse/YARN-3663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15975805#comment-15975805 ] Giovanni Matteo Fumarola commented on YARN-3663: Same result from the previous execution. The test failed and the asflicense failure are not related to the patch. About checkstyle, we had the same issue with those 2 functions also in YARN-5407. > Federation State and Policy Store (DBMS implementation) > --- > > Key: YARN-3663 > URL: https://issues.apache.org/jira/browse/YARN-3663 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Affects Versions: YARN-2915 >Reporter: Giovanni Matteo Fumarola >Assignee: Giovanni Matteo Fumarola > Attachments: YARN-3663-YARN-2915.v1.patch, > YARN-3663-YARN-2915.v2.patch, YARN-3663-YARN-2915.v3.patch > > > This JIRA tracks a SQL-based implementation of the Federation State and > Policy Store, which implements YARN-3662 APIs. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3663) Federation State and Policy Store (DBMS implementation)
[
https://issues.apache.org/jira/browse/YARN-3663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15975762#comment-15975762
]
Hadoop QA commented on YARN-3663:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 9 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
25s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
6s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
58s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 10m
0s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 3m
1s{color} | {color:green} YARN-2915 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
1s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
33s{color} | {color:green} YARN-2915 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
50s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 9s{color} | {color:orange} root: The patch generated 4 new + 214 unchanged
- 0 fixed = 218 total (was 214) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 10m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
43s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 18m 1s{color}
| {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
40s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}149m 59s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ha.TestZKFailoverController |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:612578f |
| JIRA Issue | YARN-3663 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12864083/YARN-3663-YARN-2915.v3.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml findbugs checkstyle |
| uname | Linux e96643d4403e 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-2915 / 47451e0 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
[jira] [Commented] (YARN-3663) Federation State and Policy Store (DBMS implementation)
[
https://issues.apache.org/jira/browse/YARN-3663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1597#comment-1597
]
Giovanni Matteo Fumarola commented on YARN-3663:
Thanks [~curino] for the accurate review. Let me go in order.
Conf
1. Every process should have at most 1 connection open to the StateStore except
for Router. It has 10. This is for performance and scalability reason of the
Database. FederationInterceptor open one connection for each application
running in the node. More than that can cause a possible DoS to StateStore.
2. Done.
SQLFederationStateStore
1. Done.
2. Done.
3. Done.
4. Done.
5. When I do auto-format from eclipse by using hadoop-format.xml it gives me it
in 2 lines.
6. Done.
7. We can improve/reduce the amount of code later under a different Jira. I am
quite confident in this part and I would avoid to change it.
8. Done.
9. We can improve/reduce the amount of code later under a different Jira. I am
quite confident in this part and I would avoid to change it.
10. That is a safe check, since admin's cluster can setup their own database.
11. Done.
12. Done.
13. Done.
14. Done.
15. Done.
FederationStateStoreUtils
1. Done
HSQLDBFederationStateStore
1. Done
2. I tried to move in a file but I am adding a lot of complexity and I did not
find a way to let it work. I tried to format in SQL standard inside the file
itself but the autoformat messes it up.
3. Done.
4. Fixed it - good catch.
5. In that way I only insert a new application. That code is like if (! app
exists){add app}.
6. Done.
Stored procedure
1. Done.
2. Done.
3. Increased to 6000.
4. My bad, it was a for a performance test - introduced in the patch by mistake.
5. We can DEREGISTER it or put it in a LOST state.
6. We did not run in any issues with those settings. I would keep like this.
7. Let's open a JIRA about it.
8. Done.
9. Increase to 6000.
10. As I said before, let's keep this one since we have one stored procedure
for each function. We can reduce/improve later.
Top level concerns:
1. The current design was not edited over the last year. However in case we
need to evolve the schema we will need to add them.
2. This cannot happen because we have input validators.
3. For that scenario, we have to add only the stored procedure for the specific
database.
4. I edited some tests to cover some scenario we did not check. However, the
current state covers all the possible scenario.
> Federation State and Policy Store (DBMS implementation)
> ---
>
> Key: YARN-3663
> URL: https://issues.apache.org/jira/browse/YARN-3663
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Affects Versions: YARN-2915
>Reporter: Giovanni Matteo Fumarola
>Assignee: Giovanni Matteo Fumarola
> Attachments: YARN-3663-YARN-2915.v1.patch,
> YARN-3663-YARN-2915.v2.patch, YARN-3663-YARN-2915.v3.patch
>
>
> This JIRA tracks a SQL-based implementation of the Federation State and
> Policy Store, which implements YARN-3662 APIs.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3663) Federation State and Policy Store (DBMS implementation)
[
https://issues.apache.org/jira/browse/YARN-3663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968218#comment-15968218
]
Carlo Curino commented on YARN-3663:
Thanks [~giovanni.fumarola] for the updated patch.
Conf:
# You set
{{YarnConfiguration.DEFAULT_FEDERATION_STATESTORE_SQL_MAXCONNECTIONS}} to 1.
Isn't this too small? What values you commonly use? I would put a conservative
but not overly tight value, otherwise users are forced to learn/modify more
params.
# We should omit {{yarn.federation.state-store.sql.max-connections} from
yarn-default.xml like you do for all other params in this patch.
In {{SQLFederationStateStore}}
# For most fields except {{CALL_SP_GET_ALL_POLICY_CONFIGURATIONS}} you simply
use the plural to differentiate the getter of one or all items. Be consistent
(e.g., remove the ALL here, or add it everywhere else)
# Minor: userNameDB --> userName, passwordDB --> password
# When you throw the exceptions (e.g., subcluster registration), it might be
nice in the message to include the sub-clusterID / ip or any other info one can
use to debug.
# Can you comment on why we are using: {{FederationStateStoreErrorCode}}?
They don't seem to be connected to SQL error codes, and they are not used
anywhere else (we normally use named exception, which are easier to
understand/track).
# at line 277-278: formatting
# We should try to remove redundance, e.g., you have lots of things that look
like this:
{code}
try {
FederationMembershipStateStoreInputValidator
.validateGetSubClusterInfoRequest(subClusterRequest);
} catch (FederationStateStoreInvalidInputException e) {
FederationStateStoreUtils.logAndThrowInvalidInputException(LOG,
"Unable to obtain the infomation about the SubCluster. "
+ e.getMessage());
}
{code}
They could be factored out to
{{FederationMembershipStateStoreInputValidator.validate(subClusterRequest)}}
where the type of input param is used to differentiate the method, and the
logAndThrowInvalidInputException is done on that side. Same goes for
{{checkSubClusterInfo}}.
# Similarly to the above we should try to factor out the very repetitive code
to create connection/statements, set params, run, and throw. I don't have a
specific advise on this, but the code is mostly copy and paste, which we should
avoid.
# Move the {{fromStringToSubClusterState}} to the SubclusterState class (and
call it fromString().
# Why {{getSubCluster}} and {{getSubClusters}} use different mechanics for
return values? (registered params vs ResultSet)? Might be worth to be
consistent (probably using ResultSet).
# Line 540: Is this behavior (overwrite existing) consistent with general
YARN? (I think so, but want to check)
# Some of the log are a bit vague {{LOG.debug("Got the information about the
specified application }} say spefically what info where gotten
# if you use {{LOG.debug}} consider to prefix it with a check if we are in
debug mode (save time/objects creations for the String that are then not used).
# You have several {{ if (cstmt.getInt(2) != 1) }} ROWCOUNT checks. This
mix the no tuple where changed to multiple tuple where changed. Distinguishing
the two cases, might help debug (do we have duplicates in DB, or the entry was
not found). (Not mandatory, just somethign to consider)
# {{setPolicyConfiguration}} You are doing {{cstmt.setBytes(3, new
byte[policyConf.getParams().remaining()]);}} which adds an empty byte[] instead
of what is coming in input.
# {{getCurrentVersion}} and {{loadVersion}} throw a NotSupportedException or
something of the sort, a silent return null is easy to confuse people. (I know
the full version will be in V2, let's just have a clear breakage if someone try
to use this methods).
( to be continued ...)
> Federation State and Policy Store (DBMS implementation)
> ---
>
> Key: YARN-3663
> URL: https://issues.apache.org/jira/browse/YARN-3663
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Affects Versions: YARN-2915
>Reporter: Giovanni Matteo Fumarola
>Assignee: Giovanni Matteo Fumarola
> Attachments: YARN-3663-YARN-2915.v1.patch,
> YARN-3663-YARN-2915.v2.patch
>
>
> This JIRA tracks a SQL-based implementation of the Federation State and
> Policy Store, which implements YARN-3662 APIs.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3663) Federation State and Policy Store (DBMS implementation)
[ https://issues.apache.org/jira/browse/YARN-3663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15966954#comment-15966954 ] Giovanni Matteo Fumarola commented on YARN-3663: The test failed and the asflicense failure are not related to the patch. About checkstyle, we had the same issue with those 2 functions also in YARN-5407. > Federation State and Policy Store (DBMS implementation) > --- > > Key: YARN-3663 > URL: https://issues.apache.org/jira/browse/YARN-3663 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Affects Versions: YARN-2915 >Reporter: Giovanni Matteo Fumarola >Assignee: Giovanni Matteo Fumarola > Attachments: YARN-3663-YARN-2915.v1.patch, > YARN-3663-YARN-2915.v2.patch > > > This JIRA tracks a SQL-based implementation of the Federation State and > Policy Store, which implements YARN-3662 APIs. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3663) Federation State and Policy Store (DBMS implementation)
[
https://issues.apache.org/jira/browse/YARN-3663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15966912#comment-15966912
]
Hadoop QA commented on YARN-3663:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m
53s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 22 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
39s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
52s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
0s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 11m
12s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
53s{color} | {color:green} YARN-2915 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
22s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
43s{color} | {color:green} YARN-2915 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m
59s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 1s{color} | {color:orange} root: The patch generated 2 new + 212 unchanged
- 0 fixed = 214 total (was 212) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 9m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
38s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 18m 26s{color}
| {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
40s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}168m 44s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ha.TestZKFailoverController |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:612578f |
| JIRA Issue | YARN-3663 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12863140/YARN-3663-YARN-2915.v2.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml findbugs checkstyle |
| uname | Linux 34e602597828 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-2915 / 5eacf0c |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
[jira] [Commented] (YARN-3663) Federation State and Policy Store (DBMS implementation)
[ https://issues.apache.org/jira/browse/YARN-3663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1599#comment-1599 ] Giovanni Matteo Fumarola commented on YARN-3663: Thanks [~curino] for your feedback. I added SQLServer store procs in the patch, and MSSQL jdbc runtime dependency in hadoop-yarn-server-common as you suggested. Since I am adding a new dependency I updated the license information as well. I fixed the findbugs errors and the failed tests. > Federation State and Policy Store (DBMS implementation) > --- > > Key: YARN-3663 > URL: https://issues.apache.org/jira/browse/YARN-3663 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Affects Versions: YARN-2915 >Reporter: Giovanni Matteo Fumarola >Assignee: Giovanni Matteo Fumarola > Attachments: YARN-3663-YARN-2915.v1.patch, > YARN-3663-YARN-2915.v2.patch > > > This JIRA tracks a SQL-based implementation of the Federation State and > Policy Store, which implements YARN-3662 APIs. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3663) Federation State and Policy Store (DBMS implementation)
[
https://issues.apache.org/jira/browse/YARN-3663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964869#comment-15964869
]
Hadoop QA commented on YARN-3663:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
48s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m
22s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
23s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
34s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
7s{color} | {color:green} YARN-2915 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
52s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} YARN-2915 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m
51s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 5s{color} | {color:orange} root: The patch generated 8 new + 206 unchanged
- 0 fixed = 214 total (was 206) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
7s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
29s{color} | {color:red} hadoop-yarn-server-common in the patch failed. {color}
|
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
21s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 39s{color}
| {color:red} hadoop-yarn-api in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 21s{color}
| {color:red} hadoop-yarn-server-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}103m 23s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs |
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
|
| |
org.apache.hadoop.yarn.server.federation.store.impl.SQLFederationStateStore.getApplicationsHomeSubCluster(GetApplicationsHomeSubClusterRequest)
may fail to clean up java.sql.ResultSet Obligation to clean up resource
[jira] [Commented] (YARN-3663) Federation State and Policy Store (DBMS implementation)
[
https://issues.apache.org/jira/browse/YARN-3663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964832#comment-15964832
]
Carlo Curino commented on YARN-3663:
[~giovanni.fumarola], the current patch doesn't seem to contain the definition
of store procedures except for the HSQLDB tests.
I think we need to include at least the SQLServer version you are currently
running in production and open a JIRA under the Federation V2 umbrella
YARN-5597 for a MySQL or DB-agnostic translation of the store
procedures---which will be less performant but convenient.
I know you were hesitating due to the SQLServer JDBC dependency. I think this
is not an issue anymore since MS open-sourced the JDBC driver for SQLServer:
*https://blogs.msdn.microsoft.com/jdbcteam/2016/11/17/open-source-jdbc-maven/*
Please check whether things compile and run well using that dependency (pom
entry below).
{code}
com.microsoft.sqlserver
mssql-jdbc
6.1.0.jre7
{code}
> Federation State and Policy Store (DBMS implementation)
> ---
>
> Key: YARN-3663
> URL: https://issues.apache.org/jira/browse/YARN-3663
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Giovanni Matteo Fumarola
>Assignee: Giovanni Matteo Fumarola
> Attachments: YARN-3663-YARN-2915.v1.patch
>
>
> This JIRA tracks a SQL-based implementation of the Federation State and
> Policy Store, which implements YARN-3662 APIs.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3663) Federation State and Policy Store (DBMS implementation)
[ https://issues.apache.org/jira/browse/YARN-3663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15963868#comment-15963868 ] Carlo Curino commented on YARN-3663: Now that HADOOP-13545 is committed I kicked yetus to run the usual QA. > Federation State and Policy Store (DBMS implementation) > --- > > Key: YARN-3663 > URL: https://issues.apache.org/jira/browse/YARN-3663 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Giovanni Matteo Fumarola >Assignee: Giovanni Matteo Fumarola > Attachments: YARN-3663-YARN-2915.v1.patch > > > This JIRA tracks a SQL-based implementation of the Federation State and > Policy Store, which implements YARN-3662 APIs. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3663) Federation State and Policy Store (DBMS implementation)
[ https://issues.apache.org/jira/browse/YARN-3663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15959966#comment-15959966 ] Giovanni Matteo Fumarola commented on YARN-3663: [~curino], [~subru] Can you guys take a look of the patch? The new tests will fail since they are a DataSource dependency fixed by [HADOOP-13545|https://issues.apache.org/jira/browse/HADOOP-13545]. > Federation State and Policy Store (DBMS implementation) > --- > > Key: YARN-3663 > URL: https://issues.apache.org/jira/browse/YARN-3663 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Giovanni Matteo Fumarola >Assignee: Giovanni Matteo Fumarola > Attachments: YARN-3663-YARN-2915.v1.patch > > > This JIRA tracks a SQL-based implementation of the Federation State and > Policy Store, which implements YARN-3662 APIs. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
