[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17600603#comment-17600603
]
ASF GitHub Bot commented on YARN-11296:
---
slfan1989 opened a new pull request, #4858:
URL: https://github.com/apache/hadoop/pull/4858
JIRA: YARN-11296. Fix SQLFederationStateStore#Sql script bug.
> Fix SQLFederationStateStore#Sql script bug
> --
>
> Key: YARN-11296
> URL: https://issues.apache.org/jira/browse/YARN-11296
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
>
> *MySQL:*
> FederationStateStoreTables.sql cannot create *membership* table, both in
> MySQL 5.7 and MySQL 8.0.
>
> {code:java}
> CREATE TABLE membership(
>subClusterId varchar(256) NOT NULL,
>amRMServiceAddress varchar(256) NOT NULL,
>clientRMServiceAddress varchar(256) NOT NULL,
>rmAdminServiceAddress varchar(256) NOT NULL,
>rmWebServiceAddress varchar(256) NOT NULL,
>lastHeartBeat datetime NOT NULL,
>state varchar(32) NOT NULL,
>lastStartTime bigint NULL,
>capability varchar(6000),
>CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId), -- missing comma
> UNIQUE(lastStartTime)
> ); {code}
>
> *SQLServer:*
> FederationStateStoreTables.sql
>
> {code:java}
> IF NOT EXISTS ( SELECT * FROM [FederationStateStore].sys.tables
> WHERE name = 'membership'
> AND schema_id = SCHEMA_ID('dbo'))
> BEGIN
> PRINT 'Table membership does not exist, create it...'
> SET ANSI_NULLS ON
> SET QUOTED_IDENTIFIER ON
> SET ANSI_PADDING ON
> CREATE TABLE [dbo].[membership](
> [subClusterId]VARCHAR(256) COLLATE
> Latin1_General_100_BIN2 NOT NULL,
> [amRMServiceAddress] VARCHAR(256) NOT NULL,
> [clientRMServiceAddress] VARCHAR(256) NOT NULL,
> [rmAdminServiceAddress] VARCHAR(256) NOT NULL,
> [rmWebServiceAddress] VARCHAR(256) NOT NULL,
> [lastHeartBeat] DATETIME2 NOT NULL,
> [state] VARCHAR(32) NOT NULL,
> [lastStartTime] BIGINT NOT NULL,
> [capability] VARCHAR(6000) NOT NULL,
> CONSTRAINT [pk_subClusterId] PRIMARY KEY
> (
> [subClusterId]
> ), -- missing comma
> CONSTRAINT [uc_lastStartTime] UNIQUE
> (
> [lastStartTime]
> )
> )
> SET ANSI_PADDING OFF
> PRINT 'Table membership created.'
> END
> ELSE
> PRINT 'Table membership exists, no operation required...'
> GO
> GO {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17600637#comment-17600637
]
ASF GitHub Bot commented on YARN-11296:
---
hadoop-yetus commented on PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#issuecomment-1237763399
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 53s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
_ trunk Compile Tests _ |
| +1 :green_heart: | shadedclient | 33m 52s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | shadedclient | 21m 59s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 42s | | The patch does not
generate ASF License warnings. |
| | | 59m 15s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4858/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4858 |
| Optional Tests | dupname asflicense codespell detsecrets |
| uname | Linux ac4fa9e3546f 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2512f043c415fbfc85d6d3971a84defc04d01532 |
| Max. process+thread count | 534 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn U:
hadoop-yarn-project/hadoop-yarn |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4858/1/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Fix SQLFederationStateStore#Sql script bug
> --
>
> Key: YARN-11296
> URL: https://issues.apache.org/jira/browse/YARN-11296
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
>
> *MySQL:*
> FederationStateStoreTables.sql cannot create *membership* table, both in
> MySQL 5.7 and MySQL 8.0.
>
> {code:java}
> CREATE TABLE membership(
>subClusterId varchar(256) NOT NULL,
>amRMServiceAddress varchar(256) NOT NULL,
>clientRMServiceAddress varchar(256) NOT NULL,
>rmAdminServiceAddress varchar(256) NOT NULL,
>rmWebServiceAddress varchar(256) NOT NULL,
>lastHeartBeat datetime NOT NULL,
>state varchar(32) NOT NULL,
>lastStartTime bigint NULL,
>capability varchar(6000),
>CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId), -- missing comma
> UNIQUE(lastStartTime)
> ); {code}
>
> *SQLServer:*
> FederationStateStoreTables.sql
>
> {code:java}
> IF NOT EXISTS ( SELECT * FROM [FederationStateStore].sys.tables
> WHERE name = 'membership'
> AND schema_id = SCHEMA_ID('dbo'))
> BEGIN
> PRINT 'Table membership does not exist, create it...'
> SET ANSI_NULLS ON
> SET QUOTED_IDENTIFIER ON
> SET ANSI_PADDING ON
> CREATE TABLE [dbo].[membership](
> [subClusterId]VARCHAR(256) COLLATE
> Latin1_General_100_BIN2 NOT NULL,
> [amRMServiceAddress] VARCHAR(256) NOT NULL,
> [clientRMServiceAddress] VARCHAR(256) NOT NULL,
> [rmAdminServiceAddress] VARCHAR(256) NOT NULL,
> [rmWebServiceAddress] VARCHAR(256) NOT NULL,
> [lastHeartBeat] DATETIME2 NOT NULL,
> [state] VARCHAR(32) NOT NULL,
> [lastStartTime] BIGINT NOT NULL,
> [capability] VARCHAR(6000) NOT NULL,
> CONSTRAINT [pk_subClusterId] PRIMARY KEY
> (
> [subClusterId]
> ), -- missing comma
> CONSTRAINT [uc_lastStartTime] UNIQUE
> (
> [lastStartTime]
> )
> )
> SET ANSI_PADDING
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17600700#comment-17600700
]
ASF GitHub Bot commented on YARN-11296:
---
hadoop-yetus commented on PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#issuecomment-1237921393
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
_ trunk Compile Tests _ |
| +1 :green_heart: | shadedclient | 32m 52s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks
issues. |
| +1 :green_heart: | shadedclient | 22m 2s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 42s | | The patch does not
generate ASF License warnings. |
| | | 57m 58s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4858/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4858 |
| Optional Tests | dupname asflicense codespell detsecrets |
| uname | Linux 3b0c251514a6 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / afba705768dd88e1453d451eaeb8e44856156309 |
| Max. process+thread count | 600 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn U:
hadoop-yarn-project/hadoop-yarn |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4858/2/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Fix SQLFederationStateStore#Sql script bug
> --
>
> Key: YARN-11296
> URL: https://issues.apache.org/jira/browse/YARN-11296
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
>
> *MySQL:*
> FederationStateStoreTables.sql cannot create *membership* table, both in
> MySQL 5.7 and MySQL 8.0.
>
> {code:java}
> CREATE TABLE membership(
>subClusterId varchar(256) NOT NULL,
>amRMServiceAddress varchar(256) NOT NULL,
>clientRMServiceAddress varchar(256) NOT NULL,
>rmAdminServiceAddress varchar(256) NOT NULL,
>rmWebServiceAddress varchar(256) NOT NULL,
>lastHeartBeat datetime NOT NULL,
>state varchar(32) NOT NULL,
>lastStartTime bigint NULL,
>capability varchar(6000),
>CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId), -- missing comma
> UNIQUE(lastStartTime)
> ); {code}
>
> *SQLServer:*
> FederationStateStoreTables.sql
>
> {code:java}
> IF NOT EXISTS ( SELECT * FROM [FederationStateStore].sys.tables
> WHERE name = 'membership'
> AND schema_id = SCHEMA_ID('dbo'))
> BEGIN
> PRINT 'Table membership does not exist, create it...'
> SET ANSI_NULLS ON
> SET QUOTED_IDENTIFIER ON
> SET ANSI_PADDING ON
> CREATE TABLE [dbo].[membership](
> [subClusterId]VARCHAR(256) COLLATE
> Latin1_General_100_BIN2 NOT NULL,
> [amRMServiceAddress] VARCHAR(256) NOT NULL,
> [clientRMServiceAddress] VARCHAR(256) NOT NULL,
> [rmAdminServiceAddress] VARCHAR(256) NOT NULL,
> [rmWebServiceAddress] VARCHAR(256) NOT NULL,
> [lastHeartBeat] DATETIME2 NOT NULL,
> [state] VARCHAR(32) NOT NULL,
> [lastStartTime] BIGINT NOT NULL,
> [capability] VARCHAR(6000) NOT NULL,
> CONSTRAINT [pk_subClusterId] PRIMARY KEY
> (
> [subClusterId]
> ), -- missing comma
> CONSTRAINT [uc_lastStartTime] UNIQUE
> (
> [lastStartTime]
> )
> )
> SET ANSI_PADDING
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17601054#comment-17601054
]
ASF GitHub Bot commented on YARN-11296:
---
goiri commented on code in PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#discussion_r964257406
##
hadoop-yarn-project/hadoop-yarn/bin/FederationStateStore/MySQL/FederationStateStoreTables.sql:
##
@@ -36,7 +36,7 @@ CREATE TABLE membership(
state varchar(32) NOT NULL,
lastStartTime bigint NULL,
capability varchar(6000),
- CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId)
+ CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId),
Review Comment:
We have no tests that would catch this?
> Fix SQLFederationStateStore#Sql script bug
> --
>
> Key: YARN-11296
> URL: https://issues.apache.org/jira/browse/YARN-11296
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
>
> *MySQL:*
> FederationStateStoreTables.sql cannot create *membership* table, both in
> MySQL 5.7 and MySQL 8.0.
>
> {code:java}
> CREATE TABLE membership(
>subClusterId varchar(256) NOT NULL,
>amRMServiceAddress varchar(256) NOT NULL,
>clientRMServiceAddress varchar(256) NOT NULL,
>rmAdminServiceAddress varchar(256) NOT NULL,
>rmWebServiceAddress varchar(256) NOT NULL,
>lastHeartBeat datetime NOT NULL,
>state varchar(32) NOT NULL,
>lastStartTime bigint NULL,
>capability varchar(6000),
>CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId), -- missing comma
> UNIQUE(lastStartTime)
> ); {code}
>
> *SQLServer:*
> FederationStateStoreTables.sql
>
> {code:java}
> IF NOT EXISTS ( SELECT * FROM [FederationStateStore].sys.tables
> WHERE name = 'membership'
> AND schema_id = SCHEMA_ID('dbo'))
> BEGIN
> PRINT 'Table membership does not exist, create it...'
> SET ANSI_NULLS ON
> SET QUOTED_IDENTIFIER ON
> SET ANSI_PADDING ON
> CREATE TABLE [dbo].[membership](
> [subClusterId]VARCHAR(256) COLLATE
> Latin1_General_100_BIN2 NOT NULL,
> [amRMServiceAddress] VARCHAR(256) NOT NULL,
> [clientRMServiceAddress] VARCHAR(256) NOT NULL,
> [rmAdminServiceAddress] VARCHAR(256) NOT NULL,
> [rmWebServiceAddress] VARCHAR(256) NOT NULL,
> [lastHeartBeat] DATETIME2 NOT NULL,
> [state] VARCHAR(32) NOT NULL,
> [lastStartTime] BIGINT NOT NULL,
> [capability] VARCHAR(6000) NOT NULL,
> CONSTRAINT [pk_subClusterId] PRIMARY KEY
> (
> [subClusterId]
> ), -- missing comma
> CONSTRAINT [uc_lastStartTime] UNIQUE
> (
> [lastStartTime]
> )
> )
> SET ANSI_PADDING OFF
> PRINT 'Table membership created.'
> END
> ELSE
> PRINT 'Table membership exists, no operation required...'
> GO
> GO {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17601057#comment-17601057
]
ASF GitHub Bot commented on YARN-11296:
---
slfan1989 commented on code in PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#discussion_r964259520
##
hadoop-yarn-project/hadoop-yarn/bin/FederationStateStore/MySQL/FederationStateStoreTables.sql:
##
@@ -36,7 +36,7 @@ CREATE TABLE membership(
state varchar(32) NOT NULL,
lastStartTime bigint NULL,
capability varchar(6000),
- CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId)
+ CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId),
Review Comment:
In the original code, there is no test for this part. This problem is found
when our test database is executed. I will try to submit a new test code to
cover this part.
> Fix SQLFederationStateStore#Sql script bug
> --
>
> Key: YARN-11296
> URL: https://issues.apache.org/jira/browse/YARN-11296
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
>
> *MySQL:*
> FederationStateStoreTables.sql cannot create *membership* table, both in
> MySQL 5.7 and MySQL 8.0.
>
> {code:java}
> CREATE TABLE membership(
>subClusterId varchar(256) NOT NULL,
>amRMServiceAddress varchar(256) NOT NULL,
>clientRMServiceAddress varchar(256) NOT NULL,
>rmAdminServiceAddress varchar(256) NOT NULL,
>rmWebServiceAddress varchar(256) NOT NULL,
>lastHeartBeat datetime NOT NULL,
>state varchar(32) NOT NULL,
>lastStartTime bigint NULL,
>capability varchar(6000),
>CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId), -- missing comma
> UNIQUE(lastStartTime)
> ); {code}
>
> *SQLServer:*
> FederationStateStoreTables.sql
>
> {code:java}
> IF NOT EXISTS ( SELECT * FROM [FederationStateStore].sys.tables
> WHERE name = 'membership'
> AND schema_id = SCHEMA_ID('dbo'))
> BEGIN
> PRINT 'Table membership does not exist, create it...'
> SET ANSI_NULLS ON
> SET QUOTED_IDENTIFIER ON
> SET ANSI_PADDING ON
> CREATE TABLE [dbo].[membership](
> [subClusterId]VARCHAR(256) COLLATE
> Latin1_General_100_BIN2 NOT NULL,
> [amRMServiceAddress] VARCHAR(256) NOT NULL,
> [clientRMServiceAddress] VARCHAR(256) NOT NULL,
> [rmAdminServiceAddress] VARCHAR(256) NOT NULL,
> [rmWebServiceAddress] VARCHAR(256) NOT NULL,
> [lastHeartBeat] DATETIME2 NOT NULL,
> [state] VARCHAR(32) NOT NULL,
> [lastStartTime] BIGINT NOT NULL,
> [capability] VARCHAR(6000) NOT NULL,
> CONSTRAINT [pk_subClusterId] PRIMARY KEY
> (
> [subClusterId]
> ), -- missing comma
> CONSTRAINT [uc_lastStartTime] UNIQUE
> (
> [lastStartTime]
> )
> )
> SET ANSI_PADDING OFF
> PRINT 'Table membership created.'
> END
> ELSE
> PRINT 'Table membership exists, no operation required...'
> GO
> GO {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17602137#comment-17602137
]
ASF GitHub Bot commented on YARN-11296:
---
hadoop-yetus commented on PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#issuecomment-1241529957
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 55s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 36s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 23s | | trunk passed |
| +1 :green_heart: | compile | 10m 45s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 9m 17s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 3s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 10s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 9s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 3m 24s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 14m 48s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 24s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 44s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 5m 31s | | the patch passed |
| +1 :green_heart: | compile | 10m 1s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 10m 1s | | the patch passed |
| +1 :green_heart: | compile | 9m 7s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 9m 7s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 54s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4858/3/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 20 new + 0 unchanged -
1 fixed = 20 total (was 1) |
| +1 :green_heart: | mvnsite | 6m 1s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 49s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 3m 14s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 14m 27s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 40s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 243m 53s | | hadoop-yarn in the patch
passed. |
| +1 :green_heart: | unit | 3m 18s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | asflicense | 1m 12s | | The patch does not
generate ASF License warnings. |
| | | 446m 19s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4858/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4858 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac
javadoc mvninstall mvnsite unit shadedclient xmllint spotbugs checkstyle |
| uname | Linux 2e372e6547db 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c57add83177ffe0e434e2310a41ffd0207679093 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Priv
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17602158#comment-17602158
]
ASF GitHub Bot commented on YARN-11296:
---
hadoop-yetus commented on PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#issuecomment-1241578400
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 47s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 56s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 48s | | trunk passed |
| +1 :green_heart: | compile | 9m 49s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 8m 50s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 10s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 22s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 8s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 3m 33s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 13m 36s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 15s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 5m 25s | | the patch passed |
| +1 :green_heart: | compile | 10m 12s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 10m 12s | | the patch passed |
| +1 :green_heart: | compile | 9m 12s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 9m 12s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 48s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4858/4/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 21 new + 0 unchanged -
1 fixed = 21 total (was 1) |
| +1 :green_heart: | mvnsite | 5m 49s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 59s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 3m 22s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 14m 44s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 17s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 239m 56s | | hadoop-yarn in the patch
passed. |
| +1 :green_heart: | unit | 3m 35s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | asflicense | 1m 20s | | The patch does not
generate ASF License warnings. |
| | | 437m 13s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4858/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4858 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac
javadoc mvninstall mvnsite unit shadedclient xmllint spotbugs checkstyle |
| uname | Linux 827eb3b9e857 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2c1b4a541f37f3c6286643504c60863502dd7fb0 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Priv
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17602211#comment-17602211
]
ASF GitHub Bot commented on YARN-11296:
---
hadoop-yetus commented on PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#issuecomment-1241686561
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 47s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 44s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 27m 52s | | trunk passed |
| +1 :green_heart: | compile | 10m 57s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 9m 38s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 6s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 12s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 1s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 3m 30s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 16m 41s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 30s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 5m 24s | | the patch passed |
| +1 :green_heart: | compile | 9m 57s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| -1 :x: | javac | 9m 57s |
[/results-compile-javac-hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4858/5/artifact/out/results-compile-javac-hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04.txt)
|
hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 generated 2 new + 740
unchanged - 0 fixed = 742 total (was 740) |
| +1 :green_heart: | compile | 9m 6s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| -1 :x: | javac | 9m 6s |
[/results-compile-javac-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4858/5/artifact/out/results-compile-javac-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07.txt)
|
hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 generated 2 new
+ 653 unchanged - 0 fixed = 655 total (was 653) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 53s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4858/5/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 26 new + 0 unchanged -
1 fixed = 26 total (was 1) |
| +1 :green_heart: | mvnsite | 6m 38s | | the patch passed |
| +1 :green_heart: | javadoc | 4m 5s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 3m 15s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 13m 53s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 24s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 238m 25s | | hadoop-yarn in the patch
passed. |
| +1 :green_heart: | unit | 3m 41s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | asflicense | 1m 33s | | The patch does not
generate ASF License warn
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17602262#comment-17602262
]
ASF GitHub Bot commented on YARN-11296:
---
slfan1989 commented on PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#issuecomment-1241825833
The implementation idea of SQL script Check is as follows:
1. From the script directory, synchronize Mysql and SqlServer scripts for
creating tables and creating stored procedures.
2. Use HSQLDB's Mysql and SqlServer compatibility modes respectively to
verify the creation of tables and stored procedures.
> Fix SQLFederationStateStore#Sql script bug
> --
>
> Key: YARN-11296
> URL: https://issues.apache.org/jira/browse/YARN-11296
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
>
> *MySQL:*
> FederationStateStoreTables.sql cannot create *membership* table, both in
> MySQL 5.7 and MySQL 8.0.
>
> {code:java}
> CREATE TABLE membership(
>subClusterId varchar(256) NOT NULL,
>amRMServiceAddress varchar(256) NOT NULL,
>clientRMServiceAddress varchar(256) NOT NULL,
>rmAdminServiceAddress varchar(256) NOT NULL,
>rmWebServiceAddress varchar(256) NOT NULL,
>lastHeartBeat datetime NOT NULL,
>state varchar(32) NOT NULL,
>lastStartTime bigint NULL,
>capability varchar(6000),
>CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId), -- missing comma
> UNIQUE(lastStartTime)
> ); {code}
>
> *SQLServer:*
> FederationStateStoreTables.sql
>
> {code:java}
> IF NOT EXISTS ( SELECT * FROM [FederationStateStore].sys.tables
> WHERE name = 'membership'
> AND schema_id = SCHEMA_ID('dbo'))
> BEGIN
> PRINT 'Table membership does not exist, create it...'
> SET ANSI_NULLS ON
> SET QUOTED_IDENTIFIER ON
> SET ANSI_PADDING ON
> CREATE TABLE [dbo].[membership](
> [subClusterId]VARCHAR(256) COLLATE
> Latin1_General_100_BIN2 NOT NULL,
> [amRMServiceAddress] VARCHAR(256) NOT NULL,
> [clientRMServiceAddress] VARCHAR(256) NOT NULL,
> [rmAdminServiceAddress] VARCHAR(256) NOT NULL,
> [rmWebServiceAddress] VARCHAR(256) NOT NULL,
> [lastHeartBeat] DATETIME2 NOT NULL,
> [state] VARCHAR(32) NOT NULL,
> [lastStartTime] BIGINT NOT NULL,
> [capability] VARCHAR(6000) NOT NULL,
> CONSTRAINT [pk_subClusterId] PRIMARY KEY
> (
> [subClusterId]
> ), -- missing comma
> CONSTRAINT [uc_lastStartTime] UNIQUE
> (
> [lastStartTime]
> )
> )
> SET ANSI_PADDING OFF
> PRINT 'Table membership created.'
> END
> ELSE
> PRINT 'Table membership exists, no operation required...'
> GO
> GO {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17602408#comment-17602408
]
ASF GitHub Bot commented on YARN-11296:
---
slfan1989 commented on PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#issuecomment-1242091483
@goiri Please help to review this pr, thank you very much!
> Fix SQLFederationStateStore#Sql script bug
> --
>
> Key: YARN-11296
> URL: https://issues.apache.org/jira/browse/YARN-11296
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
>
> *MySQL:*
> FederationStateStoreTables.sql cannot create *membership* table, both in
> MySQL 5.7 and MySQL 8.0.
>
> {code:java}
> CREATE TABLE membership(
>subClusterId varchar(256) NOT NULL,
>amRMServiceAddress varchar(256) NOT NULL,
>clientRMServiceAddress varchar(256) NOT NULL,
>rmAdminServiceAddress varchar(256) NOT NULL,
>rmWebServiceAddress varchar(256) NOT NULL,
>lastHeartBeat datetime NOT NULL,
>state varchar(32) NOT NULL,
>lastStartTime bigint NULL,
>capability varchar(6000),
>CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId), -- missing comma
> UNIQUE(lastStartTime)
> ); {code}
>
> *SQLServer:*
> FederationStateStoreTables.sql
>
> {code:java}
> IF NOT EXISTS ( SELECT * FROM [FederationStateStore].sys.tables
> WHERE name = 'membership'
> AND schema_id = SCHEMA_ID('dbo'))
> BEGIN
> PRINT 'Table membership does not exist, create it...'
> SET ANSI_NULLS ON
> SET QUOTED_IDENTIFIER ON
> SET ANSI_PADDING ON
> CREATE TABLE [dbo].[membership](
> [subClusterId]VARCHAR(256) COLLATE
> Latin1_General_100_BIN2 NOT NULL,
> [amRMServiceAddress] VARCHAR(256) NOT NULL,
> [clientRMServiceAddress] VARCHAR(256) NOT NULL,
> [rmAdminServiceAddress] VARCHAR(256) NOT NULL,
> [rmWebServiceAddress] VARCHAR(256) NOT NULL,
> [lastHeartBeat] DATETIME2 NOT NULL,
> [state] VARCHAR(32) NOT NULL,
> [lastStartTime] BIGINT NOT NULL,
> [capability] VARCHAR(6000) NOT NULL,
> CONSTRAINT [pk_subClusterId] PRIMARY KEY
> (
> [subClusterId]
> ), -- missing comma
> CONSTRAINT [uc_lastStartTime] UNIQUE
> (
> [lastStartTime]
> )
> )
> SET ANSI_PADDING OFF
> PRINT 'Table membership created.'
> END
> ELSE
> PRINT 'Table membership exists, no operation required...'
> GO
> GO {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17602483#comment-17602483
]
ASF GitHub Bot commented on YARN-11296:
---
goiri commented on code in PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#discussion_r967322160
##
hadoop-yarn-project/hadoop-yarn/bin/FederationStateStore/SQLServer/FederationStateStoreTables.sql:
##
@@ -140,7 +140,7 @@ IF NOT EXISTS ( SELECT * FROM
[FederationStateStore].sys.tables
CREATE TABLE [dbo].[reservationsHomeSubCluster](
reservationId VARCHAR(128) COLLATE Latin1_General_100_BIN2 NOT
NULL,
homeSubCluster VARCHAR(256) NOT NULL,
-createTime DATETIME2 NOT NULL CONSTRAINT ts_createAppTime
DEFAULT GETUTCDATE(),
+createTime DATETIME2 NOT NULL CONSTRAINT ts_createResTime
DEFAULT GETUTCDATE(),
Review Comment:
This is also used in line 37. Where does this field come from?
##
hadoop-yarn-project/hadoop-yarn/bin/FederationStateStore/MySQL/FederationStateStoreTables.sql:
##
@@ -36,7 +36,7 @@ CREATE TABLE membership(
state varchar(32) NOT NULL,
lastStartTime bigint NULL,
capability varchar(6000),
- CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId)
+ CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId),
Review Comment:
Is this validated by the unit test?
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/sql/TestFDMySQLAccuracy.java:
##
@@ -0,0 +1,73 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.federation.store.sql;
+
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.server.federation.store.FederationStateStore;
+import
org.apache.hadoop.yarn.server.federation.store.impl.MySQLFederationStateStore;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.SQLException;
+import java.util.List;
+
+public class TestFDMySQLAccuracy extends FederationSQLAccuracyTest {
Review Comment:
What is FD?
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/impl/SQLServerFederationStateStore.java:
##
@@ -0,0 +1,114 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.server.federation.store.impl;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.server.federation.store.FederationStateStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.sql.Connection;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * SQLServerFederationStateStore implementation of {@link
FederationStateStore}.
+ */
+public class SQLServerFederationStateStore extends SQLFederationStateStore {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(SQLServerFeder
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17602505#comment-17602505
]
ASF GitHub Bot commented on YARN-11296:
---
hadoop-yetus commented on PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#issuecomment-1242322388
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 5 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 42s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 33s | | trunk passed |
| +1 :green_heart: | compile | 10m 39s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 9m 15s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 3s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 8s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 7s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 3m 25s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 14m 36s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 18s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 5m 28s | | the patch passed |
| +1 :green_heart: | compile | 10m 3s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| -1 :x: | javac | 10m 3s |
[/results-compile-javac-hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4858/6/artifact/out/results-compile-javac-hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04.txt)
|
hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 generated 2 new + 741
unchanged - 0 fixed = 743 total (was 741) |
| +1 :green_heart: | compile | 9m 8s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| -1 :x: | javac | 9m 8s |
[/results-compile-javac-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4858/6/artifact/out/results-compile-javac-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07.txt)
|
hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 generated 2 new
+ 654 unchanged - 0 fixed = 656 total (was 654) |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 51s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4858/6/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 9 new + 0 unchanged -
1 fixed = 9 total (was 1) |
| +1 :green_heart: | mvnsite | 5m 55s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 53s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 3m 14s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 14m 33s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 3s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 242m 52s | | hadoop-yarn in the patch
passed. |
| +1 :green_heart: | unit | 3m 19s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | asflicense | 1m 11s | | The patch does not
generate ASF License warnin
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17602621#comment-17602621
]
ASF GitHub Bot commented on YARN-11296:
---
slfan1989 commented on code in PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#discussion_r967601401
##
hadoop-yarn-project/hadoop-yarn/bin/FederationStateStore/MySQL/FederationStateStoreTables.sql:
##
@@ -36,7 +36,7 @@ CREATE TABLE membership(
state varchar(32) NOT NULL,
lastStartTime bigint NULL,
capability varchar(6000),
- CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId)
+ CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId),
Review Comment:
Thank you very much for your help in reviewing the code. This part of the
modification has been verified. I used the compatibility mode of HSQLDB's Mysql
and SqlServer to verify the table and stored procedure scripts respectively.
> Fix SQLFederationStateStore#Sql script bug
> --
>
> Key: YARN-11296
> URL: https://issues.apache.org/jira/browse/YARN-11296
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
>
> *MySQL:*
> FederationStateStoreTables.sql cannot create *membership* table, both in
> MySQL 5.7 and MySQL 8.0.
>
> {code:java}
> CREATE TABLE membership(
>subClusterId varchar(256) NOT NULL,
>amRMServiceAddress varchar(256) NOT NULL,
>clientRMServiceAddress varchar(256) NOT NULL,
>rmAdminServiceAddress varchar(256) NOT NULL,
>rmWebServiceAddress varchar(256) NOT NULL,
>lastHeartBeat datetime NOT NULL,
>state varchar(32) NOT NULL,
>lastStartTime bigint NULL,
>capability varchar(6000),
>CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId), -- missing comma
> UNIQUE(lastStartTime)
> ); {code}
>
> *SQLServer:*
> FederationStateStoreTables.sql
>
> {code:java}
> IF NOT EXISTS ( SELECT * FROM [FederationStateStore].sys.tables
> WHERE name = 'membership'
> AND schema_id = SCHEMA_ID('dbo'))
> BEGIN
> PRINT 'Table membership does not exist, create it...'
> SET ANSI_NULLS ON
> SET QUOTED_IDENTIFIER ON
> SET ANSI_PADDING ON
> CREATE TABLE [dbo].[membership](
> [subClusterId]VARCHAR(256) COLLATE
> Latin1_General_100_BIN2 NOT NULL,
> [amRMServiceAddress] VARCHAR(256) NOT NULL,
> [clientRMServiceAddress] VARCHAR(256) NOT NULL,
> [rmAdminServiceAddress] VARCHAR(256) NOT NULL,
> [rmWebServiceAddress] VARCHAR(256) NOT NULL,
> [lastHeartBeat] DATETIME2 NOT NULL,
> [state] VARCHAR(32) NOT NULL,
> [lastStartTime] BIGINT NOT NULL,
> [capability] VARCHAR(6000) NOT NULL,
> CONSTRAINT [pk_subClusterId] PRIMARY KEY
> (
> [subClusterId]
> ), -- missing comma
> CONSTRAINT [uc_lastStartTime] UNIQUE
> (
> [lastStartTime]
> )
> )
> SET ANSI_PADDING OFF
> PRINT 'Table membership created.'
> END
> ELSE
> PRINT 'Table membership exists, no operation required...'
> GO
> GO {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17602622#comment-17602622
]
ASF GitHub Bot commented on YARN-11296:
---
slfan1989 commented on code in PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#discussion_r967601774
##
hadoop-yarn-project/hadoop-yarn/bin/FederationStateStore/SQLServer/FederationStateStoreTables.sql:
##
@@ -140,7 +140,7 @@ IF NOT EXISTS ( SELECT * FROM
[FederationStateStore].sys.tables
CREATE TABLE [dbo].[reservationsHomeSubCluster](
reservationId VARCHAR(128) COLLATE Latin1_General_100_BIN2 NOT
NULL,
homeSubCluster VARCHAR(256) NOT NULL,
-createTime DATETIME2 NOT NULL CONSTRAINT ts_createAppTime
DEFAULT GETUTCDATE(),
+createTime DATETIME2 NOT NULL CONSTRAINT ts_createResTime
DEFAULT GETUTCDATE(),
Review Comment:
This is the name of the constraint (newly specified when the table was
created), for reservation I think it might be better to prefix it with res.
> Fix SQLFederationStateStore#Sql script bug
> --
>
> Key: YARN-11296
> URL: https://issues.apache.org/jira/browse/YARN-11296
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
>
> *MySQL:*
> FederationStateStoreTables.sql cannot create *membership* table, both in
> MySQL 5.7 and MySQL 8.0.
>
> {code:java}
> CREATE TABLE membership(
>subClusterId varchar(256) NOT NULL,
>amRMServiceAddress varchar(256) NOT NULL,
>clientRMServiceAddress varchar(256) NOT NULL,
>rmAdminServiceAddress varchar(256) NOT NULL,
>rmWebServiceAddress varchar(256) NOT NULL,
>lastHeartBeat datetime NOT NULL,
>state varchar(32) NOT NULL,
>lastStartTime bigint NULL,
>capability varchar(6000),
>CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId), -- missing comma
> UNIQUE(lastStartTime)
> ); {code}
>
> *SQLServer:*
> FederationStateStoreTables.sql
>
> {code:java}
> IF NOT EXISTS ( SELECT * FROM [FederationStateStore].sys.tables
> WHERE name = 'membership'
> AND schema_id = SCHEMA_ID('dbo'))
> BEGIN
> PRINT 'Table membership does not exist, create it...'
> SET ANSI_NULLS ON
> SET QUOTED_IDENTIFIER ON
> SET ANSI_PADDING ON
> CREATE TABLE [dbo].[membership](
> [subClusterId]VARCHAR(256) COLLATE
> Latin1_General_100_BIN2 NOT NULL,
> [amRMServiceAddress] VARCHAR(256) NOT NULL,
> [clientRMServiceAddress] VARCHAR(256) NOT NULL,
> [rmAdminServiceAddress] VARCHAR(256) NOT NULL,
> [rmWebServiceAddress] VARCHAR(256) NOT NULL,
> [lastHeartBeat] DATETIME2 NOT NULL,
> [state] VARCHAR(32) NOT NULL,
> [lastStartTime] BIGINT NOT NULL,
> [capability] VARCHAR(6000) NOT NULL,
> CONSTRAINT [pk_subClusterId] PRIMARY KEY
> (
> [subClusterId]
> ), -- missing comma
> CONSTRAINT [uc_lastStartTime] UNIQUE
> (
> [lastStartTime]
> )
> )
> SET ANSI_PADDING OFF
> PRINT 'Table membership created.'
> END
> ELSE
> PRINT 'Table membership exists, no operation required...'
> GO
> GO {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17602624#comment-17602624
]
ASF GitHub Bot commented on YARN-11296:
---
slfan1989 commented on code in PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#discussion_r967601774
##
hadoop-yarn-project/hadoop-yarn/bin/FederationStateStore/SQLServer/FederationStateStoreTables.sql:
##
@@ -140,7 +140,7 @@ IF NOT EXISTS ( SELECT * FROM
[FederationStateStore].sys.tables
CREATE TABLE [dbo].[reservationsHomeSubCluster](
reservationId VARCHAR(128) COLLATE Latin1_General_100_BIN2 NOT
NULL,
homeSubCluster VARCHAR(256) NOT NULL,
-createTime DATETIME2 NOT NULL CONSTRAINT ts_createAppTime
DEFAULT GETUTCDATE(),
+createTime DATETIME2 NOT NULL CONSTRAINT ts_createResTime
DEFAULT GETUTCDATE(),
Review Comment:
This is the name of the constraint (newly specified when the table was
created), for reservation I think it might be better to prefix it with res.
```
column_name
[ FILESTREAM ]
[ COLLATE collation_name ]
[ SPARSE ]
[ MASKED WITH ( FUNCTION = 'mask_function' ) ]
[ [ CONSTRAINT constraint_name ] DEFAULT constant_expression ]
[ IDENTITY [ ( seed , increment ) ]
[ NOT FOR REPLICATION ]
[ GENERATED ALWAYS AS { ROW | TRANSACTION_ID | SEQUENCE_NUMBER } { START
| END } [ HIDDEN ] ]
[ NULL | NOT NULL ]
[ ROWGUIDCOL ]
[ ENCRYPTED WITH
( COLUMN_ENCRYPTION_KEY = key_name ,
ENCRYPTION_TYPE = { DETERMINISTIC | RANDOMIZED } ,
ALGORITHM = 'AEAD_AES_256_CBC_HMAC_SHA_256'
) ]
[ [ ,... n ] ]
[ ]
```
We can refer to the following document, which is more detailed
https://docs.microsoft.com/en-us/sql/t-sql/statements/create-table-transact-sql?view=sql-server-ver16
> Fix SQLFederationStateStore#Sql script bug
> --
>
> Key: YARN-11296
> URL: https://issues.apache.org/jira/browse/YARN-11296
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
>
> *MySQL:*
> FederationStateStoreTables.sql cannot create *membership* table, both in
> MySQL 5.7 and MySQL 8.0.
>
> {code:java}
> CREATE TABLE membership(
>subClusterId varchar(256) NOT NULL,
>amRMServiceAddress varchar(256) NOT NULL,
>clientRMServiceAddress varchar(256) NOT NULL,
>rmAdminServiceAddress varchar(256) NOT NULL,
>rmWebServiceAddress varchar(256) NOT NULL,
>lastHeartBeat datetime NOT NULL,
>state varchar(32) NOT NULL,
>lastStartTime bigint NULL,
>capability varchar(6000),
>CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId), -- missing comma
> UNIQUE(lastStartTime)
> ); {code}
>
> *SQLServer:*
> FederationStateStoreTables.sql
>
> {code:java}
> IF NOT EXISTS ( SELECT * FROM [FederationStateStore].sys.tables
> WHERE name = 'membership'
> AND schema_id = SCHEMA_ID('dbo'))
> BEGIN
> PRINT 'Table membership does not exist, create it...'
> SET ANSI_NULLS ON
> SET QUOTED_IDENTIFIER ON
> SET ANSI_PADDING ON
> CREATE TABLE [dbo].[membership](
> [subClusterId]VARCHAR(256) COLLATE
> Latin1_General_100_BIN2 NOT NULL,
> [amRMServiceAddress] VARCHAR(256) NOT NULL,
> [clientRMServiceAddress] VARCHAR(256) NOT NULL,
> [rmAdminServiceAddress] VARCHAR(256) NOT NULL,
> [rmWebServiceAddress] VARCHAR(256) NOT NULL,
> [lastHeartBeat] DATETIME2 NOT NULL,
> [state] VARCHAR(32) NOT NULL,
> [lastStartTime] BIGINT NOT NULL,
> [capability] VARCHAR(6000) NOT NULL,
> CONSTRAINT [pk_subClusterId] PRIMARY KEY
> (
> [subClusterId]
> ), -- missing comma
> CONSTRAINT [uc_lastStartTime] UNIQUE
> (
> [lastStartTime]
> )
> )
> SET ANSI_PADDING OFF
> PRINT 'Table membership created.'
> END
> ELSE
> PRINT 'Table membership exists, no operation required...'
> GO
> GO {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17602626#comment-17602626
]
ASF GitHub Bot commented on YARN-11296:
---
slfan1989 commented on code in PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#discussion_r967602319
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/impl/SQLServerFederationStateStore.java:
##
@@ -0,0 +1,114 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.server.federation.store.impl;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.server.federation.store.FederationStateStore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.sql.Connection;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * SQLServerFederationStateStore implementation of {@link
FederationStateStore}.
+ */
+public class SQLServerFederationStateStore extends SQLFederationStateStore {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(SQLServerFederationStateStore.class);
+
+ private Connection conn;
+
+ private String createTableScriptPath;
+
+ private static String createProcedureScriptPath;
+
+ private List tables = new ArrayList<>();
+
+ private List procedures = new ArrayList<>();
+
+ @Override
+ public void init(Configuration conf) {
+try {
+ super.init(conf);
+
+ // get the sql that creates the table
+ createTableScriptPath = "." + File.separator + "target" + File.separator
+
+ "test-classes" + File.separator +
"SQLServer/FederationStateStoreTables.sql";
+ LOG.info("createTableScriptPath >> {}", createTableScriptPath);
+ String createTableSQL = FileUtils.readFileToString(new
File(createTableScriptPath),
+ StandardCharsets.UTF_8);
+ Pattern p = Pattern.compile("IF NOT EXISTS.*\\n(.*\\n){0,50}.*GO");
+ Matcher m = p.matcher(createTableSQL);
+ while(m!=null && m.find()) {
+tables.add(m.group());
+ }
+
+ // get the sql that creates the stored procedure
+ createProcedureScriptPath = "." + File.separator + "target" +
File.separator +
+ "test-classes" + File.separator +
"SQLServer/FederationStateStoreStoreProcs.sql";
+ String createProcedureSQL = FileUtils.readFileToString(new
File(createProcedureScriptPath),
+ StandardCharsets.UTF_8);
+ String[] results = createProcedureSQL.split("GO");
+ for (String result : results) {
+if (StringUtils.contains(result, "CREATE PROCEDURE")) {
+ procedures.add(result);
+}
+ }
+
+ LOG.info("SqlServer - tables = {}, procedures = {}", tables.size(),
procedures.size());
+
+ conn = super.conn;
+} catch (YarnException | IOException e1) {
+ LOG.error("ERROR: failed to init HSQLDB " + e1.getMessage());
+}
+ }
+
+ public void closeConnection() {
Review Comment:
Thanks for your suggestion, I will refactor this part of the code.
> Fix SQLFederationStateStore#Sql script bug
> --
>
> Key: YARN-11296
> URL: https://issues.apache.org/jira/browse/YARN-11296
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
>
> *MySQL:*
> FederationStateStoreTables.sql cannot create *membership* table, both in
> MySQL 5.7 and MySQL 8.0.
>
> {code:java}
> CREATE TABLE membership(
>subClusterId varchar(256) NOT NULL,
>amRMServiceAddress varchar(256) NOT NULL,
>clientRMServiceA
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17602625#comment-17602625
]
ASF GitHub Bot commented on YARN-11296:
---
slfan1989 commented on code in PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#discussion_r967602276
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/sql/TestFDMySQLAccuracy.java:
##
@@ -0,0 +1,73 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.federation.store.sql;
+
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.server.federation.store.FederationStateStore;
+import
org.apache.hadoop.yarn.server.federation.store.impl.MySQLFederationStateStore;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.SQLException;
+import java.util.List;
+
+public class TestFDMySQLAccuracy extends FederationSQLAccuracyTest {
Review Comment:
FD is short for Federation, it may not be accurate, I will modify the code.
> Fix SQLFederationStateStore#Sql script bug
> --
>
> Key: YARN-11296
> URL: https://issues.apache.org/jira/browse/YARN-11296
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
>
> *MySQL:*
> FederationStateStoreTables.sql cannot create *membership* table, both in
> MySQL 5.7 and MySQL 8.0.
>
> {code:java}
> CREATE TABLE membership(
>subClusterId varchar(256) NOT NULL,
>amRMServiceAddress varchar(256) NOT NULL,
>clientRMServiceAddress varchar(256) NOT NULL,
>rmAdminServiceAddress varchar(256) NOT NULL,
>rmWebServiceAddress varchar(256) NOT NULL,
>lastHeartBeat datetime NOT NULL,
>state varchar(32) NOT NULL,
>lastStartTime bigint NULL,
>capability varchar(6000),
>CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId), -- missing comma
> UNIQUE(lastStartTime)
> ); {code}
>
> *SQLServer:*
> FederationStateStoreTables.sql
>
> {code:java}
> IF NOT EXISTS ( SELECT * FROM [FederationStateStore].sys.tables
> WHERE name = 'membership'
> AND schema_id = SCHEMA_ID('dbo'))
> BEGIN
> PRINT 'Table membership does not exist, create it...'
> SET ANSI_NULLS ON
> SET QUOTED_IDENTIFIER ON
> SET ANSI_PADDING ON
> CREATE TABLE [dbo].[membership](
> [subClusterId]VARCHAR(256) COLLATE
> Latin1_General_100_BIN2 NOT NULL,
> [amRMServiceAddress] VARCHAR(256) NOT NULL,
> [clientRMServiceAddress] VARCHAR(256) NOT NULL,
> [rmAdminServiceAddress] VARCHAR(256) NOT NULL,
> [rmWebServiceAddress] VARCHAR(256) NOT NULL,
> [lastHeartBeat] DATETIME2 NOT NULL,
> [state] VARCHAR(32) NOT NULL,
> [lastStartTime] BIGINT NOT NULL,
> [capability] VARCHAR(6000) NOT NULL,
> CONSTRAINT [pk_subClusterId] PRIMARY KEY
> (
> [subClusterId]
> ), -- missing comma
> CONSTRAINT [uc_lastStartTime] UNIQUE
> (
> [lastStartTime]
> )
> )
> SET ANSI_PADDING OFF
> PRINT 'Table membership created.'
> END
> ELSE
> PRINT 'Table membership exists, no operation required...'
> GO
> GO {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17602779#comment-17602779
]
ASF GitHub Bot commented on YARN-11296:
---
hadoop-yetus commented on PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#issuecomment-1242800826
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 51s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 22s | | trunk passed |
| +1 :green_heart: | compile | 10m 42s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 9m 7s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 2s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 8s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 7s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 3m 25s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 14m 28s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 40s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 24m 2s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 5m 22s | | the patch passed |
| +1 :green_heart: | compile | 9m 58s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 9m 58s | | the patch passed |
| +1 :green_heart: | compile | 9m 8s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 9m 8s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 50s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4858/7/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 20 new + 0 unchanged -
1 fixed = 20 total (was 1) |
| +1 :green_heart: | mvnsite | 5m 54s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 46s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 3m 14s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 14m 41s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 21s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 242m 42s | | hadoop-yarn in the patch
passed. |
| +1 :green_heart: | unit | 3m 22s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | asflicense | 1m 12s | | The patch does not
generate ASF License warnings. |
| | | 444m 54s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4858/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4858 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac
javadoc mvninstall mvnsite unit shadedclient xmllint spotbugs checkstyle |
| uname | Linux e7132f04b383 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4b894b270b87cb9498f4ede3260cd7fda9245565 |
| Default Java | Private Build
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17602892#comment-17602892
]
ASF GitHub Bot commented on YARN-11296:
---
hadoop-yetus commented on PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#issuecomment-1243010705
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 48s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 40s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 44s | | trunk passed |
| +1 :green_heart: | compile | 9m 53s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 8m 46s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 9s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 21s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 11s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 3m 31s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 13m 39s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 41s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 21m 9s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 34s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 5m 17s | | the patch passed |
| +1 :green_heart: | compile | 9m 16s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 9m 16s | | the patch passed |
| +1 :green_heart: | compile | 8m 41s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 8m 41s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 49s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4858/8/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 20 new + 0 unchanged -
1 fixed = 20 total (was 1) |
| +1 :green_heart: | mvnsite | 6m 0s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 45s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 3m 21s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 13m 37s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 59s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 226m 35s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4858/8/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn in the patch failed. |
| +1 :green_heart: | unit | 3m 31s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | asflicense | 1m 29s | | The patch does not
generate ASF License warnings. |
| | | 417m 25s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4858/8/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4858 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac
javadoc mvninstall mvnsite unit shadedclient xmllint spotbugs checkstyle |
| uname | Linux de66d5be8fd1 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17602993#comment-17602993
]
ASF GitHub Bot commented on YARN-11296:
---
slfan1989 commented on code in PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#discussion_r967601401
##
hadoop-yarn-project/hadoop-yarn/bin/FederationStateStore/MySQL/FederationStateStoreTables.sql:
##
@@ -36,7 +36,7 @@ CREATE TABLE membership(
state varchar(32) NOT NULL,
lastStartTime bigint NULL,
capability varchar(6000),
- CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId)
+ CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId),
Review Comment:
Thank you very much for your help in reviewing the code. This part of the
modification has been verified. I used the compatibility mode of HSQLDB's Mysql
and SqlServer to verify the table scripts respectively.
> Fix SQLFederationStateStore#Sql script bug
> --
>
> Key: YARN-11296
> URL: https://issues.apache.org/jira/browse/YARN-11296
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
>
> *MySQL:*
> FederationStateStoreTables.sql cannot create *membership* table, both in
> MySQL 5.7 and MySQL 8.0.
>
> {code:java}
> CREATE TABLE membership(
>subClusterId varchar(256) NOT NULL,
>amRMServiceAddress varchar(256) NOT NULL,
>clientRMServiceAddress varchar(256) NOT NULL,
>rmAdminServiceAddress varchar(256) NOT NULL,
>rmWebServiceAddress varchar(256) NOT NULL,
>lastHeartBeat datetime NOT NULL,
>state varchar(32) NOT NULL,
>lastStartTime bigint NULL,
>capability varchar(6000),
>CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId), -- missing comma
> UNIQUE(lastStartTime)
> ); {code}
>
> *SQLServer:*
> FederationStateStoreTables.sql
>
> {code:java}
> IF NOT EXISTS ( SELECT * FROM [FederationStateStore].sys.tables
> WHERE name = 'membership'
> AND schema_id = SCHEMA_ID('dbo'))
> BEGIN
> PRINT 'Table membership does not exist, create it...'
> SET ANSI_NULLS ON
> SET QUOTED_IDENTIFIER ON
> SET ANSI_PADDING ON
> CREATE TABLE [dbo].[membership](
> [subClusterId]VARCHAR(256) COLLATE
> Latin1_General_100_BIN2 NOT NULL,
> [amRMServiceAddress] VARCHAR(256) NOT NULL,
> [clientRMServiceAddress] VARCHAR(256) NOT NULL,
> [rmAdminServiceAddress] VARCHAR(256) NOT NULL,
> [rmWebServiceAddress] VARCHAR(256) NOT NULL,
> [lastHeartBeat] DATETIME2 NOT NULL,
> [state] VARCHAR(32) NOT NULL,
> [lastStartTime] BIGINT NOT NULL,
> [capability] VARCHAR(6000) NOT NULL,
> CONSTRAINT [pk_subClusterId] PRIMARY KEY
> (
> [subClusterId]
> ), -- missing comma
> CONSTRAINT [uc_lastStartTime] UNIQUE
> (
> [lastStartTime]
> )
> )
> SET ANSI_PADDING OFF
> PRINT 'Table membership created.'
> END
> ELSE
> PRINT 'Table membership exists, no operation required...'
> GO
> GO {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17602998#comment-17602998
]
ASF GitHub Bot commented on YARN-11296:
---
slfan1989 commented on code in PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#discussion_r968145834
##
hadoop-yarn-project/hadoop-yarn/bin/FederationStateStore/MySQL/FederationStateStoreTables.sql:
##
@@ -36,7 +36,7 @@ CREATE TABLE membership(
state varchar(32) NOT NULL,
lastStartTime bigint NULL,
capability varchar(6000),
- CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId)
+ CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId),
Review Comment:
I have tested locally and I found that HSQLDB's compatibility mode for Mysql
and SqlServer is not fully compatible with all syntaxes. The verification of
this part of the stored procedure may only be verified by providing a test
report. I will execute the SQL statement in the database and provide the test
report.
> Fix SQLFederationStateStore#Sql script bug
> --
>
> Key: YARN-11296
> URL: https://issues.apache.org/jira/browse/YARN-11296
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
>
> *MySQL:*
> FederationStateStoreTables.sql cannot create *membership* table, both in
> MySQL 5.7 and MySQL 8.0.
>
> {code:java}
> CREATE TABLE membership(
>subClusterId varchar(256) NOT NULL,
>amRMServiceAddress varchar(256) NOT NULL,
>clientRMServiceAddress varchar(256) NOT NULL,
>rmAdminServiceAddress varchar(256) NOT NULL,
>rmWebServiceAddress varchar(256) NOT NULL,
>lastHeartBeat datetime NOT NULL,
>state varchar(32) NOT NULL,
>lastStartTime bigint NULL,
>capability varchar(6000),
>CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId), -- missing comma
> UNIQUE(lastStartTime)
> ); {code}
>
> *SQLServer:*
> FederationStateStoreTables.sql
>
> {code:java}
> IF NOT EXISTS ( SELECT * FROM [FederationStateStore].sys.tables
> WHERE name = 'membership'
> AND schema_id = SCHEMA_ID('dbo'))
> BEGIN
> PRINT 'Table membership does not exist, create it...'
> SET ANSI_NULLS ON
> SET QUOTED_IDENTIFIER ON
> SET ANSI_PADDING ON
> CREATE TABLE [dbo].[membership](
> [subClusterId]VARCHAR(256) COLLATE
> Latin1_General_100_BIN2 NOT NULL,
> [amRMServiceAddress] VARCHAR(256) NOT NULL,
> [clientRMServiceAddress] VARCHAR(256) NOT NULL,
> [rmAdminServiceAddress] VARCHAR(256) NOT NULL,
> [rmWebServiceAddress] VARCHAR(256) NOT NULL,
> [lastHeartBeat] DATETIME2 NOT NULL,
> [state] VARCHAR(32) NOT NULL,
> [lastStartTime] BIGINT NOT NULL,
> [capability] VARCHAR(6000) NOT NULL,
> CONSTRAINT [pk_subClusterId] PRIMARY KEY
> (
> [subClusterId]
> ), -- missing comma
> CONSTRAINT [uc_lastStartTime] UNIQUE
> (
> [lastStartTime]
> )
> )
> SET ANSI_PADDING OFF
> PRINT 'Table membership created.'
> END
> ELSE
> PRINT 'Table membership exists, no operation required...'
> GO
> GO {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17603030#comment-17603030
]
ASF GitHub Bot commented on YARN-11296:
---
hadoop-yetus commented on PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#issuecomment-1243516304
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 46s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 35s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 41s | | trunk passed |
| +1 :green_heart: | compile | 23m 22s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 20m 44s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 17s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 15s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 58s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 4m 16s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 47s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 21m 4s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 21m 32s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 33s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 5m 33s | | the patch passed |
| +1 :green_heart: | compile | 22m 39s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 22m 39s | | the patch passed |
| +1 :green_heart: | compile | 20m 38s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 20m 38s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 4m 3s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4858/9/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 11 new + 0 unchanged - 1 fixed = 11 total (was 1)
|
| +1 :green_heart: | mvnsite | 7m 18s | | the patch passed |
| +1 :green_heart: | javadoc | 4m 44s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 4m 6s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 32s | | hadoop-project has no data from
spotbugs |
| +1 :green_heart: | shadedclient | 22m 54s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 35s | | hadoop-project in the patch
passed. |
| -1 :x: | unit | 213m 17s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4858/9/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn in the patch failed. |
| +1 :green_heart: | unit | 3m 40s | | hadoop-yarn-server-common in
the patch passed. |
| -1 :x: | asflicense | 1m 28s |
[/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4858/9/artifact/out/results-asflicense.txt)
| The patch generated 12 ASF License warnings. |
| | | 471m 55s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4858/9/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4858 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell d
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17603236#comment-17603236
]
ASF GitHub Bot commented on YARN-11296:
---
goiri commented on code in PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#discussion_r968858363
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/sql/TestFederationSQLServerScriptAccuracy.java:
##
@@ -0,0 +1,62 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.federation.store.sql;
+
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import
org.apache.hadoop.yarn.server.federation.store.impl.SQLServerFederationStateStore;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.SQLException;
+import java.util.List;
+
+public class TestFederationSQLServerScriptAccuracy extends
FederationSQLAccuracyTest {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(TestFederationSQLServerScriptAccuracy.class);
+
+ private final String DATABASE_URL_SQLSERVER =
"jdbc:hsqldb:mem-sqlserver:state-sqlserver";
+ private final String SQLSERVER_COMPATIBILITY = ";sql.syntax_mss=true";
+
+ @Override
+ protected SQLServerFederationStateStore createStateStore() {
+YarnConfiguration conf = new YarnConfiguration();
+conf.set(YarnConfiguration.FEDERATION_STATESTORE_SQL_JDBC_CLASS,
HSQLDB_DRIVER);
+conf.set(YarnConfiguration.FEDERATION_STATESTORE_SQL_USERNAME,
DATABASE_USERNAME);
+conf.set(YarnConfiguration.FEDERATION_STATESTORE_SQL_PASSWORD,
DATABASE_PASSWORD);
+conf.set(YarnConfiguration.FEDERATION_STATESTORE_SQL_URL,
+DATABASE_URL_SQLSERVER + System.currentTimeMillis() +
SQLSERVER_COMPATIBILITY);
+super.setConf(conf);
+return new SQLServerFederationStateStore();
+ }
+
+ @Test
+ public void checkSqlServerScriptAccuracy() throws SQLException {
+SQLServerFederationStateStore federationStateStore =
this.createStateStore();
+federationStateStore.init(getConf());
+
+// get a list of tables
+List tables = federationStateStore.getTables();
+for (String table : tables) {
+ federationStateStore.getConn().prepareStatement(table).execute();
+}
+
+LOG.info("[SqlServer] - FederationStateStore create table.");
Review Comment:
Do we need this [SqlServer] tag? Logger should already give the class name.
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/sql/TestFederationSQLServerScriptAccuracy.java:
##
@@ -0,0 +1,62 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.federation.store.sql;
+
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import
org.apache.hadoop.yarn.server.federation.store.impl.SQLServerFederationStateStore;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.SQLException;
+import java.util.List;
+
+public class TestFederationSQLServerScriptAccuracy extends
FederationSQLAccuracyTest {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(TestFederationSQLServerScriptAccuracy.class);
+
+ private final String DATABASE_URL_SQLSERVER =
"
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17603289#comment-17603289
]
ASF GitHub Bot commented on YARN-11296:
---
slfan1989 commented on code in PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#discussion_r969053578
##
hadoop-project/pom.xml:
##
@@ -185,7 +185,7 @@
1.0-beta-1
900
1.12.262
-2.3.4
+2.5.2
Review Comment:
Thank you very much for helping to review the code!
I have submitted a separate pr for upgrading hsqldb version, can you help to
review it?
HADOOP-18451. Update hsqldb.version from 2.3.4 to 2.5.2.
PR: https://github.com/apache/hadoop/pull/4880
> Fix SQLFederationStateStore#Sql script bug
> --
>
> Key: YARN-11296
> URL: https://issues.apache.org/jira/browse/YARN-11296
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
> Attachments: Improve Query Condition of
> FederationStateStore#getApplicationsHomeSubCluster Test Report.pdf
>
>
> *MySQL:*
> FederationStateStoreTables.sql cannot create *membership* table, both in
> MySQL 5.7 and MySQL 8.0.
>
> {code:java}
> CREATE TABLE membership(
>subClusterId varchar(256) NOT NULL,
>amRMServiceAddress varchar(256) NOT NULL,
>clientRMServiceAddress varchar(256) NOT NULL,
>rmAdminServiceAddress varchar(256) NOT NULL,
>rmWebServiceAddress varchar(256) NOT NULL,
>lastHeartBeat datetime NOT NULL,
>state varchar(32) NOT NULL,
>lastStartTime bigint NULL,
>capability varchar(6000),
>CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId), -- missing comma
> UNIQUE(lastStartTime)
> ); {code}
>
> *SQLServer:*
> FederationStateStoreTables.sql
>
> {code:java}
> IF NOT EXISTS ( SELECT * FROM [FederationStateStore].sys.tables
> WHERE name = 'membership'
> AND schema_id = SCHEMA_ID('dbo'))
> BEGIN
> PRINT 'Table membership does not exist, create it...'
> SET ANSI_NULLS ON
> SET QUOTED_IDENTIFIER ON
> SET ANSI_PADDING ON
> CREATE TABLE [dbo].[membership](
> [subClusterId]VARCHAR(256) COLLATE
> Latin1_General_100_BIN2 NOT NULL,
> [amRMServiceAddress] VARCHAR(256) NOT NULL,
> [clientRMServiceAddress] VARCHAR(256) NOT NULL,
> [rmAdminServiceAddress] VARCHAR(256) NOT NULL,
> [rmWebServiceAddress] VARCHAR(256) NOT NULL,
> [lastHeartBeat] DATETIME2 NOT NULL,
> [state] VARCHAR(32) NOT NULL,
> [lastStartTime] BIGINT NOT NULL,
> [capability] VARCHAR(6000) NOT NULL,
> CONSTRAINT [pk_subClusterId] PRIMARY KEY
> (
> [subClusterId]
> ), -- missing comma
> CONSTRAINT [uc_lastStartTime] UNIQUE
> (
> [lastStartTime]
> )
> )
> SET ANSI_PADDING OFF
> PRINT 'Table membership created.'
> END
> ELSE
> PRINT 'Table membership exists, no operation required...'
> GO
> GO {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17603290#comment-17603290
]
ASF GitHub Bot commented on YARN-11296:
---
slfan1989 commented on code in PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#discussion_r969056894
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/sql/TestFederationSQLServerScriptAccuracy.java:
##
@@ -0,0 +1,62 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.federation.store.sql;
+
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import
org.apache.hadoop.yarn.server.federation.store.impl.SQLServerFederationStateStore;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.SQLException;
+import java.util.List;
+
+public class TestFederationSQLServerScriptAccuracy extends
FederationSQLAccuracyTest {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(TestFederationSQLServerScriptAccuracy.class);
+
+ private final String DATABASE_URL_SQLSERVER =
"jdbc:hsqldb:mem-sqlserver:state-sqlserver";
+ private final String SQLSERVER_COMPATIBILITY = ";sql.syntax_mss=true";
+
+ @Override
+ protected SQLServerFederationStateStore createStateStore() {
+YarnConfiguration conf = new YarnConfiguration();
+conf.set(YarnConfiguration.FEDERATION_STATESTORE_SQL_JDBC_CLASS,
HSQLDB_DRIVER);
+conf.set(YarnConfiguration.FEDERATION_STATESTORE_SQL_USERNAME,
DATABASE_USERNAME);
+conf.set(YarnConfiguration.FEDERATION_STATESTORE_SQL_PASSWORD,
DATABASE_PASSWORD);
+conf.set(YarnConfiguration.FEDERATION_STATESTORE_SQL_URL,
+DATABASE_URL_SQLSERVER + System.currentTimeMillis() +
SQLSERVER_COMPATIBILITY);
+super.setConf(conf);
+return new SQLServerFederationStateStore();
+ }
+
+ @Test
+ public void checkSqlServerScriptAccuracy() throws SQLException {
+SQLServerFederationStateStore federationStateStore =
this.createStateStore();
+federationStateStore.init(getConf());
+
+// get a list of tables
+List tables = federationStateStore.getTables();
+for (String table : tables) {
+ federationStateStore.getConn().prepareStatement(table).execute();
+}
+
+LOG.info("[SqlServer] - FederationStateStore create table.");
Review Comment:
Thank you very much for your suggestion, I will modify the code, the [Sql
Server] tag is not needed.
> Fix SQLFederationStateStore#Sql script bug
> --
>
> Key: YARN-11296
> URL: https://issues.apache.org/jira/browse/YARN-11296
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
> Attachments: Improve Query Condition of
> FederationStateStore#getApplicationsHomeSubCluster Test Report.pdf
>
>
> *MySQL:*
> FederationStateStoreTables.sql cannot create *membership* table, both in
> MySQL 5.7 and MySQL 8.0.
>
> {code:java}
> CREATE TABLE membership(
>subClusterId varchar(256) NOT NULL,
>amRMServiceAddress varchar(256) NOT NULL,
>clientRMServiceAddress varchar(256) NOT NULL,
>rmAdminServiceAddress varchar(256) NOT NULL,
>rmWebServiceAddress varchar(256) NOT NULL,
>lastHeartBeat datetime NOT NULL,
>state varchar(32) NOT NULL,
>lastStartTime bigint NULL,
>capability varchar(6000),
>CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId), -- missing comma
> UNIQUE(lastStartTime)
> ); {code}
>
> *SQLServer:*
> FederationStateStoreTables.sql
>
> {code:java}
> IF NOT EXISTS ( SELECT * FROM [FederationStateStore].sys.tables
> WHERE name = 'membership'
> AND schema_id = SCHEMA_ID('dbo'))
> BEGIN
> PRINT 'Table membership does not exist, create it...'
> SET ANSI_NULLS ON
> SET QUOTED_IDENTIFIER ON
> SET ANSI_PADDING ON
> C
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17603291#comment-17603291
]
ASF GitHub Bot commented on YARN-11296:
---
slfan1989 commented on code in PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#discussion_r969057224
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/sql/TestFederationSQLServerScriptAccuracy.java:
##
@@ -0,0 +1,62 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.federation.store.sql;
+
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import
org.apache.hadoop.yarn.server.federation.store.impl.SQLServerFederationStateStore;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.SQLException;
+import java.util.List;
+
+public class TestFederationSQLServerScriptAccuracy extends
FederationSQLAccuracyTest {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(TestFederationSQLServerScriptAccuracy.class);
+
+ private final String DATABASE_URL_SQLSERVER =
"jdbc:hsqldb:mem-sqlserver:state-sqlserver";
+ private final String SQLSERVER_COMPATIBILITY = ";sql.syntax_mss=true";
+
+ @Override
+ protected SQLServerFederationStateStore createStateStore() {
+YarnConfiguration conf = new YarnConfiguration();
+conf.set(YarnConfiguration.FEDERATION_STATESTORE_SQL_JDBC_CLASS,
HSQLDB_DRIVER);
+conf.set(YarnConfiguration.FEDERATION_STATESTORE_SQL_USERNAME,
DATABASE_USERNAME);
+conf.set(YarnConfiguration.FEDERATION_STATESTORE_SQL_PASSWORD,
DATABASE_PASSWORD);
+conf.set(YarnConfiguration.FEDERATION_STATESTORE_SQL_URL,
Review Comment:
I will refactor this part of the code.
> Fix SQLFederationStateStore#Sql script bug
> --
>
> Key: YARN-11296
> URL: https://issues.apache.org/jira/browse/YARN-11296
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
> Attachments: Improve Query Condition of
> FederationStateStore#getApplicationsHomeSubCluster Test Report.pdf
>
>
> *MySQL:*
> FederationStateStoreTables.sql cannot create *membership* table, both in
> MySQL 5.7 and MySQL 8.0.
>
> {code:java}
> CREATE TABLE membership(
>subClusterId varchar(256) NOT NULL,
>amRMServiceAddress varchar(256) NOT NULL,
>clientRMServiceAddress varchar(256) NOT NULL,
>rmAdminServiceAddress varchar(256) NOT NULL,
>rmWebServiceAddress varchar(256) NOT NULL,
>lastHeartBeat datetime NOT NULL,
>state varchar(32) NOT NULL,
>lastStartTime bigint NULL,
>capability varchar(6000),
>CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId), -- missing comma
> UNIQUE(lastStartTime)
> ); {code}
>
> *SQLServer:*
> FederationStateStoreTables.sql
>
> {code:java}
> IF NOT EXISTS ( SELECT * FROM [FederationStateStore].sys.tables
> WHERE name = 'membership'
> AND schema_id = SCHEMA_ID('dbo'))
> BEGIN
> PRINT 'Table membership does not exist, create it...'
> SET ANSI_NULLS ON
> SET QUOTED_IDENTIFIER ON
> SET ANSI_PADDING ON
> CREATE TABLE [dbo].[membership](
> [subClusterId]VARCHAR(256) COLLATE
> Latin1_General_100_BIN2 NOT NULL,
> [amRMServiceAddress] VARCHAR(256) NOT NULL,
> [clientRMServiceAddress] VARCHAR(256) NOT NULL,
> [rmAdminServiceAddress] VARCHAR(256) NOT NULL,
> [rmWebServiceAddress] VARCHAR(256) NOT NULL,
> [lastHeartBeat] DATETIME2 NOT NULL,
> [state] VARCHAR(32) NOT NULL,
> [lastStartTime] BIGINT NOT NULL,
> [capability] VARCHAR(6000) NOT NULL,
> CONSTRAINT [pk_subClusterId] PRIMARY KEY
> (
>
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17603482#comment-17603482
]
ASF GitHub Bot commented on YARN-11296:
---
hadoop-yetus commented on PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#issuecomment-1245137721
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 44s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 33s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 48s | | trunk passed |
| +1 :green_heart: | compile | 23m 2s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 20m 41s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 12s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 16s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 57s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 4m 7s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 47s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 20m 58s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 21m 24s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 5m 37s | | the patch passed |
| +1 :green_heart: | compile | 22m 27s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 22m 27s | | the patch passed |
| +1 :green_heart: | compile | 20m 35s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 20m 35s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 3m 57s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4858/10/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 11 new + 0 unchanged - 1 fixed = 11 total (was 1)
|
| +1 :green_heart: | mvnsite | 7m 14s | | the patch passed |
| +1 :green_heart: | javadoc | 4m 49s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 4m 2s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 32s | | hadoop-project has no data from
spotbugs |
| +1 :green_heart: | shadedclient | 20m 55s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 42s | | hadoop-project in the patch
passed. |
| -1 :x: | unit | 210m 54s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4858/10/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn in the patch failed. |
| +1 :green_heart: | unit | 3m 43s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | asflicense | 1m 27s | | The patch does not
generate ASF License warnings. |
| | | 466m 26s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4858/10/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4858 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle |
| uname | Linux 2e292edaa551 4.15.0-191-generic #202-Ubuntu SMP Thu
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17603732#comment-17603732
]
ASF GitHub Bot commented on YARN-11296:
---
hadoop-yetus commented on PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#issuecomment-1245851811
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 41s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 53s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 29m 41s | | trunk passed |
| +1 :green_heart: | compile | 25m 44s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 26m 49s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 11s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 23s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 55s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 4m 12s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 46s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 20m 59s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 21m 28s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 35s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 5m 37s | | the patch passed |
| +1 :green_heart: | compile | 22m 27s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 22m 27s | | the patch passed |
| +1 :green_heart: | compile | 20m 36s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 20m 36s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 4m 3s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4858/11/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 4 new + 0 unchanged - 1 fixed = 4 total (was 1) |
| +1 :green_heart: | mvnsite | 7m 2s | | the patch passed |
| +1 :green_heart: | javadoc | 4m 36s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 4m 11s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 39s | | hadoop-project has no data from
spotbugs |
| +1 :green_heart: | shadedclient | 21m 14s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 37s | | hadoop-project in the patch
passed. |
| +1 :green_heart: | unit | 236m 24s | | hadoop-yarn in the patch
passed. |
| +1 :green_heart: | unit | 3m 53s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | asflicense | 1m 38s | | The patch does not
generate ASF License warnings. |
| | | 504m 50s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4858/11/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4858 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle |
| uname | Linux b2288332249a 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 62fc9322743
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17604009#comment-17604009
]
ASF GitHub Bot commented on YARN-11296:
---
hadoop-yetus commented on PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#issuecomment-1246560423
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 13s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 33s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 40s | | trunk passed |
| +1 :green_heart: | compile | 26m 0s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 23m 26s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 16s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 35s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 0s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 4m 15s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 44s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 21m 10s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 21m 37s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 38s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 6m 7s | | the patch passed |
| +1 :green_heart: | compile | 25m 50s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 25m 50s | | the patch passed |
| +1 :green_heart: | compile | 23m 16s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 23m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 4m 12s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4858/12/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 0 unchanged - 1 fixed = 1 total (was 1) |
| +1 :green_heart: | mvnsite | 7m 29s | | the patch passed |
| +1 :green_heart: | javadoc | 4m 53s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 4m 24s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 34s | | hadoop-project has no data from
spotbugs |
| +1 :green_heart: | shadedclient | 20m 58s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 30s | | hadoop-project in the patch
passed. |
| +1 :green_heart: | unit | 238m 17s | | hadoop-yarn in the patch
passed. |
| +1 :green_heart: | unit | 3m 34s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | asflicense | 1m 26s | | The patch does not
generate ASF License warnings. |
| | | 511m 33s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4858/12/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4858 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle |
| uname | Linux bac311eb18c0 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 99ea0332b4e
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17604010#comment-17604010
]
ASF GitHub Bot commented on YARN-11296:
---
hadoop-yetus commented on PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#issuecomment-1246562711
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 41s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 37s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 36s | | trunk passed |
| +1 :green_heart: | compile | 26m 4s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 23m 3s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 11s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 43s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 5s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 4m 15s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 47s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 21m 27s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 21m 54s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 35s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 6m 3s | | the patch passed |
| +1 :green_heart: | compile | 25m 26s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 25m 26s | | the patch passed |
| +1 :green_heart: | compile | 23m 17s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 23m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 4m 6s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4858/13/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 0 unchanged - 1 fixed = 1 total (was 1) |
| +1 :green_heart: | mvnsite | 7m 24s | | the patch passed |
| +1 :green_heart: | javadoc | 4m 51s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 4m 20s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 35s | | hadoop-project has no data from
spotbugs |
| +1 :green_heart: | shadedclient | 21m 10s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 33s | | hadoop-project in the patch
passed. |
| +1 :green_heart: | unit | 239m 0s | | hadoop-yarn in the patch
passed. |
| +1 :green_heart: | unit | 3m 40s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | asflicense | 1m 29s | | The patch does not
generate ASF License warnings. |
| | | 511m 35s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4858/13/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4858 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle |
| uname | Linux c098b10d1935 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 99ea0332b4e
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17604831#comment-17604831
]
ASF GitHub Bot commented on YARN-11296:
---
goiri commented on code in PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#discussion_r971047522
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/impl/HSQLDBFederationStateStore.java:
##
@@ -356,4 +376,31 @@ public void closeConnection() {
}
}
+ /**
+ * Extract The Create Table Sql From The Script.
+ *
+ * @param dbIdentifier database identifier, Like Mysql / SqlServer
+ * @param regex the regex
+ * @throws IOException IO exception.
+ */
+ protected void extractCreateTableSQL(String dbIdentifier, String regex)
throws IOException {
+String createTableScriptPath = "." + File.separator + "target" +
File.separator +
+"test-classes" + File.separator + dbIdentifier +
"/FederationStateStoreTables.sql";
+String createTableSQL =
+FileUtils.readFileToString(new File(createTableScriptPath),
StandardCharsets.UTF_8);
+Pattern p = Pattern.compile(regex);
+Matcher m = p.matcher(createTableSQL);
+while(m!=null && m.find()) {
Review Comment:
space after while
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/impl/HSQLDBFederationStateStore.java:
##
@@ -301,14 +309,17 @@ public class HSQLDBFederationStateStore extends
SQLFederationStateStore {
+ " WHERE reservationId = reservationId_IN;"
+ " SET rowCount_OUT = 2; END";
+ private List tables = new ArrayList<>();
@Override
public void init(Configuration conf) {
+
Review Comment:
Avoid
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/impl/HSQLDBFederationStateStore.java:
##
@@ -342,11 +353,20 @@ public void init(Configuration conf) {
conn.prepareStatement(SP_UPDATERESERVATIONHOMESUBCLUSTER).execute();
LOG.info("Database Init: Complete");
-} catch (SQLException e) {
+} catch (Exception e) {
LOG.error("ERROR: failed to inizialize HSQLDB " + e.getMessage());
Review Comment:
Use logger {}
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/impl/HSQLDBFederationStateStore.java:
##
@@ -301,14 +309,17 @@ public class HSQLDBFederationStateStore extends
SQLFederationStateStore {
+ " WHERE reservationId = reservationId_IN;"
+ " SET rowCount_OUT = 2; END";
+ private List tables = new ArrayList<>();
@Override
public void init(Configuration conf) {
+
try {
super.init(conf);
} catch (YarnException e1) {
LOG.error("ERROR: failed to init HSQLDB " + e1.getMessage());
}
+
Review Comment:
Avoid
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/impl/HSQLDBFederationStateStore.java:
##
@@ -342,11 +353,20 @@ public void init(Configuration conf) {
conn.prepareStatement(SP_UPDATERESERVATIONHOMESUBCLUSTER).execute();
LOG.info("Database Init: Complete");
-} catch (SQLException e) {
+} catch (Exception e) {
LOG.error("ERROR: failed to inizialize HSQLDB " + e.getMessage());
}
}
+ public void initConnection(Configuration conf) {
+try {
+ super.init(conf);
+ conn = super.conn;
+} catch (YarnException e1) {
+ LOG.error("ERROR: failed to init HSQLDB " + e1.getMessage());
Review Comment:
Use {}
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/impl/HSQLDBFederationStateStore.java:
##
@@ -356,4 +376,31 @@ public void closeConnection() {
}
}
+ /**
+ * Extract The Create Table Sql From The Script.
+ *
+ * @param dbIdentifier database identifier, Like Mysql / SqlServer
+ * @param regex the regex
+ * @throws IOException IO exception.
+ */
+ protected void extractCreateTableSQL(String dbIdentifier, String regex)
throws IOException {
+String createTableScriptPath = "." + File.separator + "target" +
File.separator +
Review Comment:
It might be cleaner to use String.format()
> Fix SQLFederationStateStore#Sql script bug
> --
>
> Key: YARN-11296
> URL: https://issues.apache.org/jira/browse/YARN-11296
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.4.0
>
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17605000#comment-17605000
]
ASF GitHub Bot commented on YARN-11296:
---
slfan1989 commented on code in PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#discussion_r971358259
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/impl/HSQLDBFederationStateStore.java:
##
@@ -301,14 +309,17 @@ public class HSQLDBFederationStateStore extends
SQLFederationStateStore {
+ " WHERE reservationId = reservationId_IN;"
+ " SET rowCount_OUT = 2; END";
+ private List tables = new ArrayList<>();
@Override
public void init(Configuration conf) {
+
Review Comment:
Thanks for your help reviewing the code, I will fix it!
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/impl/HSQLDBFederationStateStore.java:
##
@@ -301,14 +309,17 @@ public class HSQLDBFederationStateStore extends
SQLFederationStateStore {
+ " WHERE reservationId = reservationId_IN;"
+ " SET rowCount_OUT = 2; END";
+ private List tables = new ArrayList<>();
@Override
public void init(Configuration conf) {
+
Review Comment:
Thanks for your help reviewing the code, I will fix it.
> Fix SQLFederationStateStore#Sql script bug
> --
>
> Key: YARN-11296
> URL: https://issues.apache.org/jira/browse/YARN-11296
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
> Attachments: Improve Query Condition of
> FederationStateStore#getApplicationsHomeSubCluster Test Report.pdf
>
>
> *MySQL:*
> FederationStateStoreTables.sql cannot create *membership* table, both in
> MySQL 5.7 and MySQL 8.0.
>
> {code:java}
> CREATE TABLE membership(
>subClusterId varchar(256) NOT NULL,
>amRMServiceAddress varchar(256) NOT NULL,
>clientRMServiceAddress varchar(256) NOT NULL,
>rmAdminServiceAddress varchar(256) NOT NULL,
>rmWebServiceAddress varchar(256) NOT NULL,
>lastHeartBeat datetime NOT NULL,
>state varchar(32) NOT NULL,
>lastStartTime bigint NULL,
>capability varchar(6000),
>CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId), -- missing comma
> UNIQUE(lastStartTime)
> ); {code}
>
> *SQLServer:*
> FederationStateStoreTables.sql
>
> {code:java}
> IF NOT EXISTS ( SELECT * FROM [FederationStateStore].sys.tables
> WHERE name = 'membership'
> AND schema_id = SCHEMA_ID('dbo'))
> BEGIN
> PRINT 'Table membership does not exist, create it...'
> SET ANSI_NULLS ON
> SET QUOTED_IDENTIFIER ON
> SET ANSI_PADDING ON
> CREATE TABLE [dbo].[membership](
> [subClusterId]VARCHAR(256) COLLATE
> Latin1_General_100_BIN2 NOT NULL,
> [amRMServiceAddress] VARCHAR(256) NOT NULL,
> [clientRMServiceAddress] VARCHAR(256) NOT NULL,
> [rmAdminServiceAddress] VARCHAR(256) NOT NULL,
> [rmWebServiceAddress] VARCHAR(256) NOT NULL,
> [lastHeartBeat] DATETIME2 NOT NULL,
> [state] VARCHAR(32) NOT NULL,
> [lastStartTime] BIGINT NOT NULL,
> [capability] VARCHAR(6000) NOT NULL,
> CONSTRAINT [pk_subClusterId] PRIMARY KEY
> (
> [subClusterId]
> ), -- missing comma
> CONSTRAINT [uc_lastStartTime] UNIQUE
> (
> [lastStartTime]
> )
> )
> SET ANSI_PADDING OFF
> PRINT 'Table membership created.'
> END
> ELSE
> PRINT 'Table membership exists, no operation required...'
> GO
> GO {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17605001#comment-17605001
]
ASF GitHub Bot commented on YARN-11296:
---
slfan1989 commented on code in PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#discussion_r971358346
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/impl/HSQLDBFederationStateStore.java:
##
@@ -301,14 +309,17 @@ public class HSQLDBFederationStateStore extends
SQLFederationStateStore {
+ " WHERE reservationId = reservationId_IN;"
+ " SET rowCount_OUT = 2; END";
+ private List tables = new ArrayList<>();
@Override
public void init(Configuration conf) {
+
try {
super.init(conf);
} catch (YarnException e1) {
LOG.error("ERROR: failed to init HSQLDB " + e1.getMessage());
}
+
Review Comment:
I will fix it.
> Fix SQLFederationStateStore#Sql script bug
> --
>
> Key: YARN-11296
> URL: https://issues.apache.org/jira/browse/YARN-11296
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
> Attachments: Improve Query Condition of
> FederationStateStore#getApplicationsHomeSubCluster Test Report.pdf
>
>
> *MySQL:*
> FederationStateStoreTables.sql cannot create *membership* table, both in
> MySQL 5.7 and MySQL 8.0.
>
> {code:java}
> CREATE TABLE membership(
>subClusterId varchar(256) NOT NULL,
>amRMServiceAddress varchar(256) NOT NULL,
>clientRMServiceAddress varchar(256) NOT NULL,
>rmAdminServiceAddress varchar(256) NOT NULL,
>rmWebServiceAddress varchar(256) NOT NULL,
>lastHeartBeat datetime NOT NULL,
>state varchar(32) NOT NULL,
>lastStartTime bigint NULL,
>capability varchar(6000),
>CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId), -- missing comma
> UNIQUE(lastStartTime)
> ); {code}
>
> *SQLServer:*
> FederationStateStoreTables.sql
>
> {code:java}
> IF NOT EXISTS ( SELECT * FROM [FederationStateStore].sys.tables
> WHERE name = 'membership'
> AND schema_id = SCHEMA_ID('dbo'))
> BEGIN
> PRINT 'Table membership does not exist, create it...'
> SET ANSI_NULLS ON
> SET QUOTED_IDENTIFIER ON
> SET ANSI_PADDING ON
> CREATE TABLE [dbo].[membership](
> [subClusterId]VARCHAR(256) COLLATE
> Latin1_General_100_BIN2 NOT NULL,
> [amRMServiceAddress] VARCHAR(256) NOT NULL,
> [clientRMServiceAddress] VARCHAR(256) NOT NULL,
> [rmAdminServiceAddress] VARCHAR(256) NOT NULL,
> [rmWebServiceAddress] VARCHAR(256) NOT NULL,
> [lastHeartBeat] DATETIME2 NOT NULL,
> [state] VARCHAR(32) NOT NULL,
> [lastStartTime] BIGINT NOT NULL,
> [capability] VARCHAR(6000) NOT NULL,
> CONSTRAINT [pk_subClusterId] PRIMARY KEY
> (
> [subClusterId]
> ), -- missing comma
> CONSTRAINT [uc_lastStartTime] UNIQUE
> (
> [lastStartTime]
> )
> )
> SET ANSI_PADDING OFF
> PRINT 'Table membership created.'
> END
> ELSE
> PRINT 'Table membership exists, no operation required...'
> GO
> GO {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17605016#comment-17605016
]
ASF GitHub Bot commented on YARN-11296:
---
slfan1989 commented on code in PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#discussion_r971414849
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/impl/HSQLDBFederationStateStore.java:
##
@@ -342,11 +353,20 @@ public void init(Configuration conf) {
conn.prepareStatement(SP_UPDATERESERVATIONHOMESUBCLUSTER).execute();
LOG.info("Database Init: Complete");
-} catch (SQLException e) {
+} catch (Exception e) {
LOG.error("ERROR: failed to inizialize HSQLDB " + e.getMessage());
Review Comment:
I will fix it.
> Fix SQLFederationStateStore#Sql script bug
> --
>
> Key: YARN-11296
> URL: https://issues.apache.org/jira/browse/YARN-11296
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
> Attachments: Improve Query Condition of
> FederationStateStore#getApplicationsHomeSubCluster Test Report.pdf
>
>
> *MySQL:*
> FederationStateStoreTables.sql cannot create *membership* table, both in
> MySQL 5.7 and MySQL 8.0.
>
> {code:java}
> CREATE TABLE membership(
>subClusterId varchar(256) NOT NULL,
>amRMServiceAddress varchar(256) NOT NULL,
>clientRMServiceAddress varchar(256) NOT NULL,
>rmAdminServiceAddress varchar(256) NOT NULL,
>rmWebServiceAddress varchar(256) NOT NULL,
>lastHeartBeat datetime NOT NULL,
>state varchar(32) NOT NULL,
>lastStartTime bigint NULL,
>capability varchar(6000),
>CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId), -- missing comma
> UNIQUE(lastStartTime)
> ); {code}
>
> *SQLServer:*
> FederationStateStoreTables.sql
>
> {code:java}
> IF NOT EXISTS ( SELECT * FROM [FederationStateStore].sys.tables
> WHERE name = 'membership'
> AND schema_id = SCHEMA_ID('dbo'))
> BEGIN
> PRINT 'Table membership does not exist, create it...'
> SET ANSI_NULLS ON
> SET QUOTED_IDENTIFIER ON
> SET ANSI_PADDING ON
> CREATE TABLE [dbo].[membership](
> [subClusterId]VARCHAR(256) COLLATE
> Latin1_General_100_BIN2 NOT NULL,
> [amRMServiceAddress] VARCHAR(256) NOT NULL,
> [clientRMServiceAddress] VARCHAR(256) NOT NULL,
> [rmAdminServiceAddress] VARCHAR(256) NOT NULL,
> [rmWebServiceAddress] VARCHAR(256) NOT NULL,
> [lastHeartBeat] DATETIME2 NOT NULL,
> [state] VARCHAR(32) NOT NULL,
> [lastStartTime] BIGINT NOT NULL,
> [capability] VARCHAR(6000) NOT NULL,
> CONSTRAINT [pk_subClusterId] PRIMARY KEY
> (
> [subClusterId]
> ), -- missing comma
> CONSTRAINT [uc_lastStartTime] UNIQUE
> (
> [lastStartTime]
> )
> )
> SET ANSI_PADDING OFF
> PRINT 'Table membership created.'
> END
> ELSE
> PRINT 'Table membership exists, no operation required...'
> GO
> GO {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17605019#comment-17605019
]
ASF GitHub Bot commented on YARN-11296:
---
slfan1989 commented on code in PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#discussion_r971425926
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/impl/HSQLDBFederationStateStore.java:
##
@@ -356,4 +376,31 @@ public void closeConnection() {
}
}
+ /**
+ * Extract The Create Table Sql From The Script.
+ *
+ * @param dbIdentifier database identifier, Like Mysql / SqlServer
+ * @param regex the regex
+ * @throws IOException IO exception.
+ */
+ protected void extractCreateTableSQL(String dbIdentifier, String regex)
throws IOException {
+String createTableScriptPath = "." + File.separator + "target" +
File.separator +
Review Comment:
Thanks for your suggestion, I will modify the code.
I found that if I use string.format, the code readability doesn't seem to be
very good
```
String.format(".%starget%stest-classes%s%s%sFederationStateStoreTables.sql");
```
I think it might be better to write the following.
```
String[] createTableScriptPathItem = new String[] {
".", "target", "test-classes", dbIdentifier,
"FederationStateStoreTables.sql" };
String createTableScriptPath = StringUtils.join(createTableScriptPathItem,
File.separator);
```
> Fix SQLFederationStateStore#Sql script bug
> --
>
> Key: YARN-11296
> URL: https://issues.apache.org/jira/browse/YARN-11296
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
> Attachments: Improve Query Condition of
> FederationStateStore#getApplicationsHomeSubCluster Test Report.pdf
>
>
> *MySQL:*
> FederationStateStoreTables.sql cannot create *membership* table, both in
> MySQL 5.7 and MySQL 8.0.
>
> {code:java}
> CREATE TABLE membership(
>subClusterId varchar(256) NOT NULL,
>amRMServiceAddress varchar(256) NOT NULL,
>clientRMServiceAddress varchar(256) NOT NULL,
>rmAdminServiceAddress varchar(256) NOT NULL,
>rmWebServiceAddress varchar(256) NOT NULL,
>lastHeartBeat datetime NOT NULL,
>state varchar(32) NOT NULL,
>lastStartTime bigint NULL,
>capability varchar(6000),
>CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId), -- missing comma
> UNIQUE(lastStartTime)
> ); {code}
>
> *SQLServer:*
> FederationStateStoreTables.sql
>
> {code:java}
> IF NOT EXISTS ( SELECT * FROM [FederationStateStore].sys.tables
> WHERE name = 'membership'
> AND schema_id = SCHEMA_ID('dbo'))
> BEGIN
> PRINT 'Table membership does not exist, create it...'
> SET ANSI_NULLS ON
> SET QUOTED_IDENTIFIER ON
> SET ANSI_PADDING ON
> CREATE TABLE [dbo].[membership](
> [subClusterId]VARCHAR(256) COLLATE
> Latin1_General_100_BIN2 NOT NULL,
> [amRMServiceAddress] VARCHAR(256) NOT NULL,
> [clientRMServiceAddress] VARCHAR(256) NOT NULL,
> [rmAdminServiceAddress] VARCHAR(256) NOT NULL,
> [rmWebServiceAddress] VARCHAR(256) NOT NULL,
> [lastHeartBeat] DATETIME2 NOT NULL,
> [state] VARCHAR(32) NOT NULL,
> [lastStartTime] BIGINT NOT NULL,
> [capability] VARCHAR(6000) NOT NULL,
> CONSTRAINT [pk_subClusterId] PRIMARY KEY
> (
> [subClusterId]
> ), -- missing comma
> CONSTRAINT [uc_lastStartTime] UNIQUE
> (
> [lastStartTime]
> )
> )
> SET ANSI_PADDING OFF
> PRINT 'Table membership created.'
> END
> ELSE
> PRINT 'Table membership exists, no operation required...'
> GO
> GO {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17605022#comment-17605022
]
ASF GitHub Bot commented on YARN-11296:
---
slfan1989 commented on code in PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#discussion_r971426352
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/impl/HSQLDBFederationStateStore.java:
##
@@ -342,11 +353,20 @@ public void init(Configuration conf) {
conn.prepareStatement(SP_UPDATERESERVATIONHOMESUBCLUSTER).execute();
LOG.info("Database Init: Complete");
-} catch (SQLException e) {
+} catch (Exception e) {
LOG.error("ERROR: failed to inizialize HSQLDB " + e.getMessage());
}
}
+ public void initConnection(Configuration conf) {
+try {
+ super.init(conf);
+ conn = super.conn;
+} catch (YarnException e1) {
+ LOG.error("ERROR: failed to init HSQLDB " + e1.getMessage());
Review Comment:
I will fix it.
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/impl/HSQLDBFederationStateStore.java:
##
@@ -356,4 +376,31 @@ public void closeConnection() {
}
}
+ /**
+ * Extract The Create Table Sql From The Script.
+ *
+ * @param dbIdentifier database identifier, Like Mysql / SqlServer
+ * @param regex the regex
+ * @throws IOException IO exception.
+ */
+ protected void extractCreateTableSQL(String dbIdentifier, String regex)
throws IOException {
+String createTableScriptPath = "." + File.separator + "target" +
File.separator +
+"test-classes" + File.separator + dbIdentifier +
"/FederationStateStoreTables.sql";
+String createTableSQL =
+FileUtils.readFileToString(new File(createTableScriptPath),
StandardCharsets.UTF_8);
+Pattern p = Pattern.compile(regex);
+Matcher m = p.matcher(createTableSQL);
+while(m!=null && m.find()) {
Review Comment:
I will fix it.
> Fix SQLFederationStateStore#Sql script bug
> --
>
> Key: YARN-11296
> URL: https://issues.apache.org/jira/browse/YARN-11296
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
> Attachments: Improve Query Condition of
> FederationStateStore#getApplicationsHomeSubCluster Test Report.pdf
>
>
> *MySQL:*
> FederationStateStoreTables.sql cannot create *membership* table, both in
> MySQL 5.7 and MySQL 8.0.
>
> {code:java}
> CREATE TABLE membership(
>subClusterId varchar(256) NOT NULL,
>amRMServiceAddress varchar(256) NOT NULL,
>clientRMServiceAddress varchar(256) NOT NULL,
>rmAdminServiceAddress varchar(256) NOT NULL,
>rmWebServiceAddress varchar(256) NOT NULL,
>lastHeartBeat datetime NOT NULL,
>state varchar(32) NOT NULL,
>lastStartTime bigint NULL,
>capability varchar(6000),
>CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId), -- missing comma
> UNIQUE(lastStartTime)
> ); {code}
>
> *SQLServer:*
> FederationStateStoreTables.sql
>
> {code:java}
> IF NOT EXISTS ( SELECT * FROM [FederationStateStore].sys.tables
> WHERE name = 'membership'
> AND schema_id = SCHEMA_ID('dbo'))
> BEGIN
> PRINT 'Table membership does not exist, create it...'
> SET ANSI_NULLS ON
> SET QUOTED_IDENTIFIER ON
> SET ANSI_PADDING ON
> CREATE TABLE [dbo].[membership](
> [subClusterId]VARCHAR(256) COLLATE
> Latin1_General_100_BIN2 NOT NULL,
> [amRMServiceAddress] VARCHAR(256) NOT NULL,
> [clientRMServiceAddress] VARCHAR(256) NOT NULL,
> [rmAdminServiceAddress] VARCHAR(256) NOT NULL,
> [rmWebServiceAddress] VARCHAR(256) NOT NULL,
> [lastHeartBeat] DATETIME2 NOT NULL,
> [state] VARCHAR(32) NOT NULL,
> [lastStartTime] BIGINT NOT NULL,
> [capability] VARCHAR(6000) NOT NULL,
> CONSTRAINT [pk_subClusterId] PRIMARY KEY
> (
> [subClusterId]
> ), -- missing comma
> CONSTRAINT [uc_lastStartTime] UNIQUE
> (
> [lastStartTime]
> )
> )
> SET ANSI_PADDING OFF
> PRINT 'Table membership created.'
> END
> ELSE
> PRINT 'Table membership exists, no operation required...'
> GO
> GO {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17605237#comment-17605237
]
ASF GitHub Bot commented on YARN-11296:
---
hadoop-yetus commented on PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#issuecomment-1247862385
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 1s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 17m 1s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 31s | | trunk passed |
| +1 :green_heart: | compile | 25m 36s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 22m 7s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 59s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 45s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 3m 54s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 43s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 24m 14s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 24m 37s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 5m 39s | | the patch passed |
| +1 :green_heart: | compile | 24m 51s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 24m 51s | | the patch passed |
| +1 :green_heart: | compile | 22m 12s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 22m 12s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 4m 16s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4858/14/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 0 unchanged - 1 fixed = 1 total (was 1) |
| +1 :green_heart: | mvnsite | 6m 59s | | the patch passed |
| +1 :green_heart: | javadoc | 4m 34s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 3m 56s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 28s | | hadoop-project has no data from
spotbugs |
| +1 :green_heart: | shadedclient | 24m 42s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 29s | | hadoop-project in the patch
passed. |
| +1 :green_heart: | unit | 243m 29s | | hadoop-yarn in the patch
passed. |
| +1 :green_heart: | unit | 3m 28s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | asflicense | 1m 20s | | The patch does not
generate ASF License warnings. |
| | | 517m 51s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4858/14/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4858 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle |
| uname | Linux d798eeab891c 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 5803124ef6b
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17607520#comment-17607520
]
ASF GitHub Bot commented on YARN-11296:
---
hadoop-yetus commented on PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#issuecomment-1253233651
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 53s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 55s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 36s | | trunk passed |
| +1 :green_heart: | compile | 10m 39s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 9m 8s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 5s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 14s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 9s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 3m 25s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 14m 36s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 55s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 24m 17s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 51s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 5m 33s | | the patch passed |
| +1 :green_heart: | compile | 10m 8s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 10m 8s | | the patch passed |
| +1 :green_heart: | compile | 9m 5s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 9m 5s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4858/15/artifact/out/blanks-eol.txt)
| The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 1m 52s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4858/15/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 0 unchanged -
1 fixed = 1 total (was 1) |
| +1 :green_heart: | mvnsite | 5m 55s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 49s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 3m 13s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 14m 39s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 8s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 243m 37s | | hadoop-yarn in the patch
passed. |
| +1 :green_heart: | unit | 3m 23s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | asflicense | 1m 13s | | The patch does not
generate ASF License warnings. |
| | | 447m 42s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4858/15/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4858 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac
javadoc mvninstall mvnsite unit shadedclient xmllint spotbugs checkstyle |
| uname | Linux 5a7433a2eadd 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17607818#comment-17607818
]
ASF GitHub Bot commented on YARN-11296:
---
hadoop-yetus commented on PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#issuecomment-1253864996
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 57s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 13s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 30m 7s | | trunk passed |
| +1 :green_heart: | compile | 11m 24s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 10m 15s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 15s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 55s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 19s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 3m 31s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 16m 8s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 7s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 25m 29s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 5m 36s | | the patch passed |
| +1 :green_heart: | compile | 10m 47s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 10m 47s | | the patch passed |
| +1 :green_heart: | compile | 9m 35s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 9m 35s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 54s | | the patch passed |
| +1 :green_heart: | mvnsite | 6m 6s | | the patch passed |
| +1 :green_heart: | javadoc | 4m 3s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 3m 23s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 14m 41s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 15s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 244m 25s | | hadoop-yarn in the patch
passed. |
| +1 :green_heart: | unit | 3m 22s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | asflicense | 1m 12s | | The patch does not
generate ASF License warnings. |
| | | 457m 36s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4858/16/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4858 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac
javadoc mvninstall mvnsite unit shadedclient xmllint spotbugs checkstyle |
| uname | Linux 35da3629aafb 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b6dbb16771ec31ad3d459662219e8f5fc6c19e7b |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoo
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17607836#comment-17607836
]
ASF GitHub Bot commented on YARN-11296:
---
slfan1989 commented on PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#issuecomment-1253912362
@goiri Can you help merge this pr into trunk branch? Thank you very much!
> Fix SQLFederationStateStore#Sql script bug
> --
>
> Key: YARN-11296
> URL: https://issues.apache.org/jira/browse/YARN-11296
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
> Attachments: Improve Query Condition of
> FederationStateStore#getApplicationsHomeSubCluster Test Report.pdf
>
>
> *MySQL:*
> FederationStateStoreTables.sql cannot create *membership* table, both in
> MySQL 5.7 and MySQL 8.0.
>
> {code:java}
> CREATE TABLE membership(
>subClusterId varchar(256) NOT NULL,
>amRMServiceAddress varchar(256) NOT NULL,
>clientRMServiceAddress varchar(256) NOT NULL,
>rmAdminServiceAddress varchar(256) NOT NULL,
>rmWebServiceAddress varchar(256) NOT NULL,
>lastHeartBeat datetime NOT NULL,
>state varchar(32) NOT NULL,
>lastStartTime bigint NULL,
>capability varchar(6000),
>CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId), -- missing comma
> UNIQUE(lastStartTime)
> ); {code}
>
> *SQLServer:*
> FederationStateStoreTables.sql
>
> {code:java}
> IF NOT EXISTS ( SELECT * FROM [FederationStateStore].sys.tables
> WHERE name = 'membership'
> AND schema_id = SCHEMA_ID('dbo'))
> BEGIN
> PRINT 'Table membership does not exist, create it...'
> SET ANSI_NULLS ON
> SET QUOTED_IDENTIFIER ON
> SET ANSI_PADDING ON
> CREATE TABLE [dbo].[membership](
> [subClusterId]VARCHAR(256) COLLATE
> Latin1_General_100_BIN2 NOT NULL,
> [amRMServiceAddress] VARCHAR(256) NOT NULL,
> [clientRMServiceAddress] VARCHAR(256) NOT NULL,
> [rmAdminServiceAddress] VARCHAR(256) NOT NULL,
> [rmWebServiceAddress] VARCHAR(256) NOT NULL,
> [lastHeartBeat] DATETIME2 NOT NULL,
> [state] VARCHAR(32) NOT NULL,
> [lastStartTime] BIGINT NOT NULL,
> [capability] VARCHAR(6000) NOT NULL,
> CONSTRAINT [pk_subClusterId] PRIMARY KEY
> (
> [subClusterId]
> ), -- missing comma
> CONSTRAINT [uc_lastStartTime] UNIQUE
> (
> [lastStartTime]
> )
> )
> SET ANSI_PADDING OFF
> PRINT 'Table membership created.'
> END
> ELSE
> PRINT 'Table membership exists, no operation required...'
> GO
> GO {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17608426#comment-17608426
]
ASF GitHub Bot commented on YARN-11296:
---
slfan1989 commented on PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#issuecomment-1255542864
Can you help to merge this pr into trunk branch? Thank you very much!
After this pr, I will continue to supplement the SqlServer script. The
SqlServer script lacks scripts to delete stored procedures and delete tables.
I will also add md files to explain the versions of Mysql and SqlServer that
can be used.
> Fix SQLFederationStateStore#Sql script bug
> --
>
> Key: YARN-11296
> URL: https://issues.apache.org/jira/browse/YARN-11296
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
> Attachments: Improve Query Condition of
> FederationStateStore#getApplicationsHomeSubCluster Test Report.pdf
>
>
> *MySQL:*
> FederationStateStoreTables.sql cannot create *membership* table, both in
> MySQL 5.7 and MySQL 8.0.
>
> {code:java}
> CREATE TABLE membership(
>subClusterId varchar(256) NOT NULL,
>amRMServiceAddress varchar(256) NOT NULL,
>clientRMServiceAddress varchar(256) NOT NULL,
>rmAdminServiceAddress varchar(256) NOT NULL,
>rmWebServiceAddress varchar(256) NOT NULL,
>lastHeartBeat datetime NOT NULL,
>state varchar(32) NOT NULL,
>lastStartTime bigint NULL,
>capability varchar(6000),
>CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId), -- missing comma
> UNIQUE(lastStartTime)
> ); {code}
>
> *SQLServer:*
> FederationStateStoreTables.sql
>
> {code:java}
> IF NOT EXISTS ( SELECT * FROM [FederationStateStore].sys.tables
> WHERE name = 'membership'
> AND schema_id = SCHEMA_ID('dbo'))
> BEGIN
> PRINT 'Table membership does not exist, create it...'
> SET ANSI_NULLS ON
> SET QUOTED_IDENTIFIER ON
> SET ANSI_PADDING ON
> CREATE TABLE [dbo].[membership](
> [subClusterId]VARCHAR(256) COLLATE
> Latin1_General_100_BIN2 NOT NULL,
> [amRMServiceAddress] VARCHAR(256) NOT NULL,
> [clientRMServiceAddress] VARCHAR(256) NOT NULL,
> [rmAdminServiceAddress] VARCHAR(256) NOT NULL,
> [rmWebServiceAddress] VARCHAR(256) NOT NULL,
> [lastHeartBeat] DATETIME2 NOT NULL,
> [state] VARCHAR(32) NOT NULL,
> [lastStartTime] BIGINT NOT NULL,
> [capability] VARCHAR(6000) NOT NULL,
> CONSTRAINT [pk_subClusterId] PRIMARY KEY
> (
> [subClusterId]
> ), -- missing comma
> CONSTRAINT [uc_lastStartTime] UNIQUE
> (
> [lastStartTime]
> )
> )
> SET ANSI_PADDING OFF
> PRINT 'Table membership created.'
> END
> ELSE
> PRINT 'Table membership exists, no operation required...'
> GO
> GO {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17608444#comment-17608444
]
ASF GitHub Bot commented on YARN-11296:
---
goiri merged PR #4858:
URL: https://github.com/apache/hadoop/pull/4858
> Fix SQLFederationStateStore#Sql script bug
> --
>
> Key: YARN-11296
> URL: https://issues.apache.org/jira/browse/YARN-11296
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
> Attachments: Improve Query Condition of
> FederationStateStore#getApplicationsHomeSubCluster Test Report.pdf
>
>
> *MySQL:*
> FederationStateStoreTables.sql cannot create *membership* table, both in
> MySQL 5.7 and MySQL 8.0.
>
> {code:java}
> CREATE TABLE membership(
>subClusterId varchar(256) NOT NULL,
>amRMServiceAddress varchar(256) NOT NULL,
>clientRMServiceAddress varchar(256) NOT NULL,
>rmAdminServiceAddress varchar(256) NOT NULL,
>rmWebServiceAddress varchar(256) NOT NULL,
>lastHeartBeat datetime NOT NULL,
>state varchar(32) NOT NULL,
>lastStartTime bigint NULL,
>capability varchar(6000),
>CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId), -- missing comma
> UNIQUE(lastStartTime)
> ); {code}
>
> *SQLServer:*
> FederationStateStoreTables.sql
>
> {code:java}
> IF NOT EXISTS ( SELECT * FROM [FederationStateStore].sys.tables
> WHERE name = 'membership'
> AND schema_id = SCHEMA_ID('dbo'))
> BEGIN
> PRINT 'Table membership does not exist, create it...'
> SET ANSI_NULLS ON
> SET QUOTED_IDENTIFIER ON
> SET ANSI_PADDING ON
> CREATE TABLE [dbo].[membership](
> [subClusterId]VARCHAR(256) COLLATE
> Latin1_General_100_BIN2 NOT NULL,
> [amRMServiceAddress] VARCHAR(256) NOT NULL,
> [clientRMServiceAddress] VARCHAR(256) NOT NULL,
> [rmAdminServiceAddress] VARCHAR(256) NOT NULL,
> [rmWebServiceAddress] VARCHAR(256) NOT NULL,
> [lastHeartBeat] DATETIME2 NOT NULL,
> [state] VARCHAR(32) NOT NULL,
> [lastStartTime] BIGINT NOT NULL,
> [capability] VARCHAR(6000) NOT NULL,
> CONSTRAINT [pk_subClusterId] PRIMARY KEY
> (
> [subClusterId]
> ), -- missing comma
> CONSTRAINT [uc_lastStartTime] UNIQUE
> (
> [lastStartTime]
> )
> )
> SET ANSI_PADDING OFF
> PRINT 'Table membership created.'
> END
> ELSE
> PRINT 'Table membership exists, no operation required...'
> GO
> GO {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17608460#comment-17608460
]
ASF GitHub Bot commented on YARN-11296:
---
slfan1989 commented on PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#issuecomment-1255702826
@goiri Thank you very much for helping to review the code!
> Fix SQLFederationStateStore#Sql script bug
> --
>
> Key: YARN-11296
> URL: https://issues.apache.org/jira/browse/YARN-11296
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: Improve Query Condition of
> FederationStateStore#getApplicationsHomeSubCluster Test Report.pdf
>
>
> *MySQL:*
> FederationStateStoreTables.sql cannot create *membership* table, both in
> MySQL 5.7 and MySQL 8.0.
>
> {code:java}
> CREATE TABLE membership(
>subClusterId varchar(256) NOT NULL,
>amRMServiceAddress varchar(256) NOT NULL,
>clientRMServiceAddress varchar(256) NOT NULL,
>rmAdminServiceAddress varchar(256) NOT NULL,
>rmWebServiceAddress varchar(256) NOT NULL,
>lastHeartBeat datetime NOT NULL,
>state varchar(32) NOT NULL,
>lastStartTime bigint NULL,
>capability varchar(6000),
>CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId), -- missing comma
> UNIQUE(lastStartTime)
> ); {code}
>
> *SQLServer:*
> FederationStateStoreTables.sql
>
> {code:java}
> IF NOT EXISTS ( SELECT * FROM [FederationStateStore].sys.tables
> WHERE name = 'membership'
> AND schema_id = SCHEMA_ID('dbo'))
> BEGIN
> PRINT 'Table membership does not exist, create it...'
> SET ANSI_NULLS ON
> SET QUOTED_IDENTIFIER ON
> SET ANSI_PADDING ON
> CREATE TABLE [dbo].[membership](
> [subClusterId]VARCHAR(256) COLLATE
> Latin1_General_100_BIN2 NOT NULL,
> [amRMServiceAddress] VARCHAR(256) NOT NULL,
> [clientRMServiceAddress] VARCHAR(256) NOT NULL,
> [rmAdminServiceAddress] VARCHAR(256) NOT NULL,
> [rmWebServiceAddress] VARCHAR(256) NOT NULL,
> [lastHeartBeat] DATETIME2 NOT NULL,
> [state] VARCHAR(32) NOT NULL,
> [lastStartTime] BIGINT NOT NULL,
> [capability] VARCHAR(6000) NOT NULL,
> CONSTRAINT [pk_subClusterId] PRIMARY KEY
> (
> [subClusterId]
> ), -- missing comma
> CONSTRAINT [uc_lastStartTime] UNIQUE
> (
> [lastStartTime]
> )
> )
> SET ANSI_PADDING OFF
> PRINT 'Table membership created.'
> END
> ELSE
> PRINT 'Table membership exists, no operation required...'
> GO
> GO {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
