[jira] [Updated] (RATIS-352) Persist commit index in Raft log.
[ https://issues.apache.org/jira/browse/RATIS-352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated RATIS-352: -- Attachment: (was: r352_20181023b.patch) > Persist commit index in Raft log. > - > > Key: RATIS-352 > URL: https://issues.apache.org/jira/browse/RATIS-352 > Project: Ratis > Issue Type: Improvement > Components: server >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze >Priority: Major > Attachments: r352_20181024.patch > > > Current the commit index is not persisted in the Raft log. When a server > restarts, it cannot apply its log until it has joined a Raft group. There is > no problem if there is a Raft group. > However, a server cannot recover its data when formatting a Raft group is > impossible (e.g. a majority of servers are dead). Ideally, a server can > safely apply the log up to the commit index. The JIRA is to persist the > commit index so that such recovery is possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (RATIS-352) Persist commit index in Raft log.
[ https://issues.apache.org/jira/browse/RATIS-352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated RATIS-352: -- Attachment: r352_20181024.patch > Persist commit index in Raft log. > - > > Key: RATIS-352 > URL: https://issues.apache.org/jira/browse/RATIS-352 > Project: Ratis > Issue Type: Improvement > Components: server >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze >Priority: Major > Attachments: r352_20181024.patch > > > Current the commit index is not persisted in the Raft log. When a server > restarts, it cannot apply its log until it has joined a Raft group. There is > no problem if there is a Raft group. > However, a server cannot recover its data when formatting a Raft group is > impossible (e.g. a majority of servers are dead). Ideally, a server can > safely apply the log up to the commit index. The JIRA is to persist the > commit index so that such recovery is possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RATIS-352) Persist commit index in Raft log.
[ https://issues.apache.org/jira/browse/RATIS-352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16661843#comment-16661843 ] Tsz Wo Nicholas Sze commented on RATIS-352: --- r352_20181024.patch: fixes some tests depending on log index. > Persist commit index in Raft log. > - > > Key: RATIS-352 > URL: https://issues.apache.org/jira/browse/RATIS-352 > Project: Ratis > Issue Type: Improvement > Components: server >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze >Priority: Major > Attachments: r352_20181024.patch > > > Current the commit index is not persisted in the Raft log. When a server > restarts, it cannot apply its log until it has joined a Raft group. There is > no problem if there is a Raft group. > However, a server cannot recover its data when formatting a Raft group is > impossible (e.g. a majority of servers are dead). Ideally, a server can > safely apply the log up to the commit index. The JIRA is to persist the > commit index so that such recovery is possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RATIS-352) Persist commit index in Raft log.
[
https://issues.apache.org/jira/browse/RATIS-352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16661868#comment-16661868
]
Hadoop QA commented on RATIS-352:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
2s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 16 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
6s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 46s{color}
| {color:red} root generated 1 new + 99 unchanged - 0 fixed = 100 total (was
99) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 30s{color} | {color:orange} root: The patch generated 65 new + 1139
unchanged - 48 fixed = 1204 total (was 1187) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 0s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
7s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 29s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | ratis.TestRaftServerSlownessDetection |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/ratis:date2018-10-24
|
| JIRA Issue | RATIS-352 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12945352/r352_20181024.patch |
| Optional Tests | asflicense javac javadoc unit findbugs checkstyle
compile cc |
| uname | Linux 5b371e9d417b 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-RATIS-Build/yetus-personality.sh
|
| git revision | master / 2272086 |
| Default Java | 1.8.0_181 |
| javac |
https://builds.apache.org/job/PreCommit-RATIS-Build/450/artifact/out/diff-compile-javac-root.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-RATIS-Build/450/artifact/out/diff-checkstyle-root.txt
|
| unit |
https://builds.apache.org/job/PreCommit-RATIS-Build/450/artifact/out/patch-unit-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-RATIS-Build/450/testReport/ |
| modules | C: ratis-proto ratis-server ratis-grpc ratis-netty ratis-examples
U: . |
| Console output |
https://builds.apache.org/job/PreCommit-RATIS-Build/450/console |
| Powered by | Apache Yetus 0.5.0 http://yetus.apache.org |
This message was automatically generated.
> Persist commit index in Raft log.
> -
>
> Key: RATIS-352
>
[jira] [Commented] (RATIS-369) LogService read/write path (phase 2: more UTs, etc)
[ https://issues.apache.org/jira/browse/RATIS-369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16661890#comment-16661890 ] Sergey Soldatov commented on RATIS-369: --- Why do we need the snapshot support? In theory, that would purge the log cache as well as the log itself and that's definitely not the thing we want. Another thing that we need to think about is the ratis client retries. If the execution of the query takes longer than 3 seconds it would send the same query again. > LogService read/write path (phase 2: more UTs, etc) > --- > > Key: RATIS-369 > URL: https://issues.apache.org/jira/browse/RATIS-369 > Project: Ratis > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Major > Attachments: RATIS-369-v1.patch > > > Polish implementation, implement some missing features, add more UTs -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RATIS-352) Persist commit index in Raft log.
[ https://issues.apache.org/jira/browse/RATIS-352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16661892#comment-16661892 ] Jitendra Nath Pandey commented on RATIS-352: +1 for the latest patch. > Persist commit index in Raft log. > - > > Key: RATIS-352 > URL: https://issues.apache.org/jira/browse/RATIS-352 > Project: Ratis > Issue Type: Improvement > Components: server >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze >Priority: Major > Attachments: r352_20181024.patch > > > Current the commit index is not persisted in the Raft log. When a server > restarts, it cannot apply its log until it has joined a Raft group. There is > no problem if there is a Raft group. > However, a server cannot recover its data when formatting a Raft group is > impossible (e.g. a majority of servers are dead). Ideally, a server can > safely apply the log up to the commit index. The JIRA is to persist the > commit index so that such recovery is possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RATIS-352) Persist commit index in Raft log.
[
https://issues.apache.org/jira/browse/RATIS-352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16661994#comment-16661994
]
Hadoop QA commented on RATIS-352:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
29s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
5s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 16 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
34s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
5s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 44s{color}
| {color:red} root generated 1 new + 99 unchanged - 0 fixed = 100 total (was
99) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 26s{color} | {color:orange} root: The patch generated 65 new + 1140
unchanged - 48 fixed = 1205 total (was 1188) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 23m 32s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 38s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | ratis.server.TestRaftLogMetrics |
| | ratis.server.simulation.TestServerRestartWithSimulatedRpc |
| | ratis.server.simulation.TestServerInformationWithSimulatedRpc |
| | ratis.server.simulation.TestRaftExceptionWithSimulation |
| | ratis.server.simulation.TestRaftReconfigurationWithSimulatedRpc |
| | ratis.server.simulation.TestRaftStateMachineExceptionWithSimulatedRpc |
| | ratis.TestRaftServerSlownessDetection |
| | ratis.server.simulation.TestRaftSnapshotWithSimulatedRpc |
| | ratis.server.simulation.TestRaftWithSimulatedRpc |
| | ratis.TestRaftServerLeaderElectionTimeout |
| | ratis.server.simulation.TestLeaderElectionWithSimulatedRpc |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/ratis:date2018-10-24
|
| JIRA Issue | RATIS-352 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12945352/r352_20181024.patch |
| Optional Tests | asflicense javac javadoc unit findbugs checkstyle
compile cc |
| uname | Linux 30bfe3fa5051 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10
07:31:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-RATIS-Build/yetus-personality.sh
|
| git revision | master / 2272086 |
| Default Java | 1.8.0_181 |
| javac |
https://builds.apache.org/job/PreCommit-RATIS-Build/451/artifact/out/diff-compile-javac-root.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-RATIS-Build
[jira] [Commented] (RATIS-352) Persist commit index in Raft log.
[
https://issues.apache.org/jira/browse/RATIS-352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16662115#comment-16662115
]
Hadoop QA commented on RATIS-352:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 16 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 53s{color}
| {color:red} root generated 1 new + 99 unchanged - 0 fixed = 100 total (was
99) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 33s{color} | {color:orange} root: The patch generated 65 new + 1140
unchanged - 48 fixed = 1205 total (was 1188) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 31s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
10s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 54s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | ratis.TestRaftServerSlownessDetection |
| | ratis.TestRaftServerLeaderElectionTimeout |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/ratis:date2018-10-24
|
| JIRA Issue | RATIS-352 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12945352/r352_20181024.patch |
| Optional Tests | asflicense javac javadoc unit findbugs checkstyle
compile cc |
| uname | Linux b47dbec0847f 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-RATIS-Build/yetus-personality.sh
|
| git revision | master / 2272086 |
| Default Java | 1.8.0_181 |
| javac |
https://builds.apache.org/job/PreCommit-RATIS-Build/452/artifact/out/diff-compile-javac-root.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-RATIS-Build/452/artifact/out/diff-checkstyle-root.txt
|
| unit |
https://builds.apache.org/job/PreCommit-RATIS-Build/452/artifact/out/patch-unit-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-RATIS-Build/452/testReport/ |
| modules | C: ratis-proto ratis-server ratis-grpc ratis-netty ratis-examples
U: . |
| Console output |
https://builds.apache.org/job/PreCommit-RATIS-Build/452/console |
| Powered by | Apache Yetus 0.5.0 http://yetus.apache.org |
This message was automatically generated.
> Persist commit index in Raft log.
> --
[jira] [Created] (RATIS-370) TestRaftServerSlownessDetection keep failing
Tsz Wo Nicholas Sze created RATIS-370: - Summary: TestRaftServerSlownessDetection keep failing Key: RATIS-370 URL: https://issues.apache.org/jira/browse/RATIS-370 Project: Ratis Issue Type: Bug Components: server, test Reporter: Tsz Wo Nicholas Sze TestRaftServerSlownessDetection failure keep showing up in QA reports. In my machine, it also fails every few runs. BTW, TestRaftServerSlownessDetection only check simulated rpc. The gRPC implementation is never tested. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RATIS-352) Persist commit index in Raft log.
[
https://issues.apache.org/jira/browse/RATIS-352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16662218#comment-16662218
]
Hadoop QA commented on RATIS-352:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 16 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
5s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
4s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 41s{color}
| {color:red} root generated 1 new + 99 unchanged - 0 fixed = 100 total (was
99) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 30s{color} | {color:orange} root: The patch generated 65 new + 1139
unchanged - 48 fixed = 1204 total (was 1187) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 18m 41s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
6s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 20s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
ratis.server.simulation.TestRaftReconfigurationWithSimulatedRpc |
| | ratis.server.simulation.TestServerRestartWithSimulatedRpc |
| | ratis.TestRaftServerLeaderElectionTimeout |
| | ratis.server.simulation.TestRaftStateMachineExceptionWithSimulatedRpc |
| | ratis.TestRaftServerSlownessDetection |
| | ratis.server.simulation.TestRaftWithSimulatedRpc |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/ratis:date2018-10-24
|
| JIRA Issue | RATIS-352 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12945352/r352_20181024.patch |
| Optional Tests | asflicense javac javadoc unit findbugs checkstyle
compile cc |
| uname | Linux 6d92f2b8bb57 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-RATIS-Build/yetus-personality.sh
|
| git revision | master / 2272086 |
| Default Java | 1.8.0_181 |
| javac |
https://builds.apache.org/job/PreCommit-RATIS-Build/454/artifact/out/diff-compile-javac-root.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-RATIS-Build/454/artifact/out/diff-checkstyle-root.txt
|
| unit |
https://builds.apache.org/job/PreCommit-RATIS-Build/454/artifact/out/patch-unit-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-RATIS-Build/454/testReport/ |
| modules | C: ratis-proto ratis-server ratis-grpc ratis-netty rati
[jira] [Commented] (RATIS-352) Persist commit index in Raft log.
[
https://issues.apache.org/jira/browse/RATIS-352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16662221#comment-16662221
]
Hadoop QA commented on RATIS-352:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
3s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 16 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
6s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 49s{color}
| {color:red} root generated 1 new + 99 unchanged - 0 fixed = 100 total (was
99) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 35s{color} | {color:orange} root: The patch generated 65 new + 1139
unchanged - 48 fixed = 1204 total (was 1187) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 22m 20s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 30s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | ratis.server.simulation.TestRaftWithSimulatedRpc |
| | ratis.TestRaftServerLeaderElectionTimeout |
| | ratis.server.simulation.TestRaftStateMachineExceptionWithSimulatedRpc |
| | ratis.server.simulation.TestServerRestartWithSimulatedRpc |
| | ratis.server.simulation.TestRaftReconfigurationWithSimulatedRpc |
| | ratis.server.TestRaftLogMetrics |
| | ratis.server.simulation.TestLeaderElectionWithSimulatedRpc |
| | ratis.TestRaftServerSlownessDetection |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/ratis:date2018-10-24
|
| JIRA Issue | RATIS-352 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12945352/r352_20181024.patch |
| Optional Tests | asflicense javac javadoc unit findbugs checkstyle
compile cc |
| uname | Linux b4efb8ca72c0 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-RATIS-Build/yetus-personality.sh
|
| git revision | master / 2272086 |
| Default Java | 1.8.0_181 |
| javac |
https://builds.apache.org/job/PreCommit-RATIS-Build/453/artifact/out/diff-compile-javac-root.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-RATIS-Build/453/artifact/out/diff-checkstyle-root.txt
|
| unit |
https://builds.apache.org/job/PreCommit-RATIS-Build/453/artifact/out/patch-unit-root.txt
|
| Test Results |
https://builds.apache.org/job/P
[jira] [Commented] (RATIS-352) Persist commit index in Raft log.
[ https://issues.apache.org/jira/browse/RATIS-352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16662354#comment-16662354 ] Tsz Wo Nicholas Sze commented on RATIS-352: --- Thanks [~jnp] for reviewing the patch. Since this feature add Metadata entries to the Raft log, the tests expecting consecutive log index fail intermittently. It seems better to first fix the tests (so that they won't depends on consecutive log index) and then commit this patch. Let me file a JIRA. > Persist commit index in Raft log. > - > > Key: RATIS-352 > URL: https://issues.apache.org/jira/browse/RATIS-352 > Project: Ratis > Issue Type: Improvement > Components: server >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze >Priority: Major > Attachments: r352_20181024.patch > > > Current the commit index is not persisted in the Raft log. When a server > restarts, it cannot apply its log until it has joined a Raft group. There is > no problem if there is a Raft group. > However, a server cannot recover its data when formatting a Raft group is > impossible (e.g. a majority of servers are dead). Ideally, a server can > safely apply the log up to the commit index. The JIRA is to persist the > commit index so that such recovery is possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (RATIS-371) Change tests so that they do not depend on consecutive log indices
Tsz Wo Nicholas Sze created RATIS-371: - Summary: Change tests so that they do not depend on consecutive log indices Key: RATIS-371 URL: https://issues.apache.org/jira/browse/RATIS-371 Project: Ratis Issue Type: Improvement Components: test Reporter: Tsz Wo Nicholas Sze Assignee: Tsz Wo Nicholas Sze RATIS-352 tries to add metadata entries to the raft log. However, there are tests expecting consecutive log indices so that they fail intermittently. This JIRA is to fix the (most of) tests so that they won't depends on consecutive log indices. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RATIS-371) Change tests so that they do not depend on consecutive log indices
[ https://issues.apache.org/jira/browse/RATIS-371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16662376#comment-16662376 ] Tsz Wo Nicholas Sze commented on RATIS-371: --- r371_20181024.patch: split the test change from the patch in RATIS-352. > Change tests so that they do not depend on consecutive log indices > -- > > Key: RATIS-371 > URL: https://issues.apache.org/jira/browse/RATIS-371 > Project: Ratis > Issue Type: Improvement > Components: test >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze >Priority: Major > Attachments: r371_20181024.patch > > > RATIS-352 tries to add metadata entries to the raft log. However, there are > tests expecting consecutive log indices so that they fail intermittently. > This JIRA is to fix the (most of) tests so that they won't depends on > consecutive log indices. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (RATIS-371) Change tests so that they do not depend on consecutive log indices
[ https://issues.apache.org/jira/browse/RATIS-371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated RATIS-371: -- Attachment: r371_20181024.patch > Change tests so that they do not depend on consecutive log indices > -- > > Key: RATIS-371 > URL: https://issues.apache.org/jira/browse/RATIS-371 > Project: Ratis > Issue Type: Improvement > Components: test >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze >Priority: Major > Attachments: r371_20181024.patch > > > RATIS-352 tries to add metadata entries to the raft log. However, there are > tests expecting consecutive log indices so that they fail intermittently. > This JIRA is to fix the (most of) tests so that they won't depends on > consecutive log indices. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RATIS-352) Persist commit index in Raft log.
[ https://issues.apache.org/jira/browse/RATIS-352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16662378#comment-16662378 ] Tsz Wo Nicholas Sze commented on RATIS-352: --- Filed RATIS-371. > Persist commit index in Raft log. > - > > Key: RATIS-352 > URL: https://issues.apache.org/jira/browse/RATIS-352 > Project: Ratis > Issue Type: Improvement > Components: server >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze >Priority: Major > Attachments: r352_20181024.patch > > > Current the commit index is not persisted in the Raft log. When a server > restarts, it cannot apply its log until it has joined a Raft group. There is > no problem if there is a Raft group. > However, a server cannot recover its data when formatting a Raft group is > impossible (e.g. a majority of servers are dead). Ideally, a server can > safely apply the log up to the commit index. The JIRA is to persist the > commit index so that such recovery is possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RATIS-371) Change tests so that they do not depend on consecutive log indices
[
https://issues.apache.org/jira/browse/RATIS-371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16662412#comment-16662412
]
Hadoop QA commented on RATIS-371:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 18 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
5s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 24m
58s{color} | {color:green} root in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
10s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 33s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/ratis:date2018-10-24
|
| JIRA Issue | RATIS-371 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12945416/r371_20181024.patch |
| Optional Tests | asflicense javac javadoc unit findbugs checkstyle
compile |
| uname | Linux 4c61a395012a 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-RATIS-Build/yetus-personality.sh
|
| git revision | master / 2272086 |
| Default Java | 1.8.0_181 |
| Test Results |
https://builds.apache.org/job/PreCommit-RATIS-Build/455/testReport/ |
| modules | C: ratis-server ratis-grpc ratis-netty ratis-examples U: . |
| Console output |
https://builds.apache.org/job/PreCommit-RATIS-Build/455/console |
| Powered by | Apache Yetus 0.5.0 http://yetus.apache.org |
This message was automatically generated.
> Change tests so that they do not depend on consecutive log indices
> --
>
> Key: RATIS-371
> URL: https://issues.apache.org/jira/browse/RATIS-371
> Project: Ratis
> Issue Type: Improvement
> Components: test
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
>Priority: Major
> Attachments: r371_20181024.patch
>
>
> RATIS-352 tries to add metadata entries to the raft log. However, there are
> tests expecting consecutive log indices so that they fail intermittently.
> This JIRA is to fix the (most of) tests so that they won't depends on
> consecutive log indices.
--
This message wa
[jira] [Updated] (RATIS-334) Implement server membership for LogService Metadata Service
[ https://issues.apache.org/jira/browse/RATIS-334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated RATIS-334: - Component/s: LogService > Implement server membership for LogService Metadata Service > --- > > Key: RATIS-334 > URL: https://issues.apache.org/jira/browse/RATIS-334 > Project: Ratis > Issue Type: Sub-task > Components: LogService >Reporter: Josh Elser >Assignee: Sergey Soldatov >Priority: Major > Attachments: RATIS-334-v1.patch, RATIS-334-v3.patch, > RATIS-334-wip.patch > > > Half of the Metadata Service for managing LogStreams. > RATIS-279 is handling the "DDL" operations, and Rajesh suggested that we spin > out management of the servers which are available to back these LogStreams in > a second task. This is it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RATIS-334) Implement server membership for LogService Metadata Service
[
https://issues.apache.org/jira/browse/RATIS-334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16662751#comment-16662751
]
Josh Elser commented on RATIS-334:
--
{code:java}
+public class LogInfo {
{code}
{{equals(Object)}}, {{toString()}}, and {{hashCode()}}?
{code:java}
+ System.out.println("!!!");
{code}
;) should we just have an exception thrown here to prevent the "unhandled
message" case?
{code:java}
- case CREATELOG:
- return processCreateLogRequest(logServiceRequestProto);
- case ARCHIVELOG:
- return processArchiveLog(logServiceRequestProto);
+// case CREATELOG:
+// return processCreateLogRequest(logServiceRequestProto);
+// case ARCHIVELOG:
+// return processArchiveLog(logServiceRequestProto);
{code}
Just remove these from LogStateMachine instead of commenting out? Same in
{{LogServiceImpl}}.
{code:java}
+ return
LogServiceFactory.getInstance().createLogService(getRaftClient(info), null);
{code}
I know you're planning on work around this, but consolidate calls to
{{LogServiceFactory.getInstance()}} in the constructor of {{LogServiceClient}}?
Ignore if you're already re-working it.
{code:java}
public class NoEnoughWorkersException{code}
s/No/Not/ ? Also in the error message.
{code:java}
+ try {
+ reply = raftClient.sendAsync(Message.valueOf(LogServiceProtoUtil
+ .toAppendBBEntryLogRequestProto(parent.getName(), list)
+ .toByteString())).get();
+ } catch (InterruptedException e) {
+ e.printStackTrace();
+ } catch (ExecutionException e) {
+ e.printStackTrace();
+ }{code}
Re-set the interrupted exception and propagate for {{InterruptedException}}.
Unwrap and re-throw the {{ExecutionException}}?
{code:java}
+ @Parameter(names = "-port", description = "Port number")
+ private int port = ;
{code}
Doing some poking around here to make sure there's nothing obvious in the
"ecosystem" that might default to using .
I'm also wondering – ideally, we can just set {{0}} and let the service pick an
ephemeral port. I guess that would require the service to register itself
somewhere for us to find it? You have an idea about how that should work?
{code:java}
+ if (host == null) {
+ host = LogServiceUtils.getHostName();
+ }
{code}
Probably a good idea to let this be configurable too (multiple NICs with
public/private network setups).
{code:java}
+ try {
+ req =
MetaSMRequestProto.parseFrom(x.getStateMachineLogEntry().getLogData());
+ } catch (InvalidProtocolBufferException e) {
+ e.printStackTrace();
+ }{code}
Propagate. Here, and elsewhere in {{MetaStateMachine.}}
{code:java}
+ case UNREGISTERREQUEST:
+ LogServiceUnregisterLogRequestProto unregReq =
req.getUnregisterRequest();
+ logname = LogServiceProtoUtil.toLogName(unregReq.getLogname());
+ map.remove(logname);
+ break;
+ case PINGREQUEST:
+ LogServicePingRequestProto pingRequest = req.getPingRequest();
+ RaftPeer peer =
MetaServiceProtoUtil.toRaftPeer(pingRequest.getPeer());
+ if (peers.contains(peer)) {
+ //Do Nothing, that's just heartbeat
+ } else {
+ peers.add(peer);
+ avail.add(new PeerGroups(peer));
+ }
+ break;{code}
DEBUG or TRACE logging for these? Will assuredly need it later :)
{code:java}
+ if (currentGroup == null) {
+ try {
+ List x =
StreamSupport.stream(raftServer.getGroups().spliterator(), false).filter(group
-> group.getGroupId().equals(metaGroupID)).collect(Collectors.toList());
+ if (x.size() == 1) {
+ currentGroup = x.get(0);
+ }
+ } catch (IOException e) {
+ e.printStackTrace();
+ }
+ }{code}
This isn't threadsafe. What about making the global {{currentGroup}} some
{{AtomicReference}} and then copy it into a local variable when you need it?
Admittedly, I don't quite grok what this is doing yet
{code:java}
+ public static String getHostName() {
+ try (DatagramSocket socket = new DatagramSocket()) {
+ socket.connect(InetAddress.getByName("8.8.8.8"), 10002);
+ return socket.getLocalAddress().getHostName();
+ } catch (Exception e) {
+ return "localhost";
+ }{code}
Heh :), switch to {{InetAddress.getLocalHost().getHostName()}} for now. I think
we'll have to provide the ability later on to specify hostnames for quorum
members (same reason as above).
{code:java}
+ RaftServerConfigKeys.setStorageDirs(properties,
+ Collections.singletonList(new File("/tmp/logservice/" +
UUID.randomUUID().toString(
[jira] [Commented] (RATIS-334) Implement server membership for LogService Metadata Service
[ https://issues.apache.org/jira/browse/RATIS-334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16662762#comment-16662762 ] Vladimir Rodionov commented on RATIS-334: - np, I will rebase RATIS-369 on top of this. > Implement server membership for LogService Metadata Service > --- > > Key: RATIS-334 > URL: https://issues.apache.org/jira/browse/RATIS-334 > Project: Ratis > Issue Type: Sub-task > Components: LogService >Reporter: Josh Elser >Assignee: Sergey Soldatov >Priority: Major > Attachments: RATIS-334-v1.patch, RATIS-334-v3.patch, > RATIS-334-wip.patch > > > Half of the Metadata Service for managing LogStreams. > RATIS-279 is handling the "DDL" operations, and Rajesh suggested that we spin > out management of the servers which are available to back these LogStreams in > a second task. This is it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RATIS-371) Change tests so that they do not depend on consecutive log indices
[ https://issues.apache.org/jira/browse/RATIS-371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16662801#comment-16662801 ] Jitendra Nath Pandey commented on RATIS-371: Thanks for fixing these. As I understand, the logIndex is still sequential, but log-entries have additional metadata entries in between that add to the count. +1 for the patch. > Change tests so that they do not depend on consecutive log indices > -- > > Key: RATIS-371 > URL: https://issues.apache.org/jira/browse/RATIS-371 > Project: Ratis > Issue Type: Improvement > Components: test >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze >Priority: Major > Attachments: r371_20181024.patch > > > RATIS-352 tries to add metadata entries to the raft log. However, there are > tests expecting consecutive log indices so that they fail intermittently. > This JIRA is to fix the (most of) tests so that they won't depends on > consecutive log indices. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RATIS-334) Implement server membership for LogService Metadata Service
[ https://issues.apache.org/jira/browse/RATIS-334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16662854#comment-16662854 ] Sergey Soldatov commented on RATIS-334: --- bq. equals(Object), toString(), and hashCode()? This class is used only to pass the pair between client and meta. Can add it later, but not sure what for. bq. should we just have an exception thrown here to prevent the "unhandled message" case? accidentally added that to the git :( Was trying to figure out why my test for read/write doesn't work (that was the problem that ratis keeps support messages in the same raft log, so the first was some configuration message with empty body) bq. I know you're planning on work around this, but consolidate calls to LogServiceFactory.getInstance() in the constructor of LogServiceClient? Ignore if you're already re-working it. I don't want to instantiate LogService nor use the factory at all. My suggestion is to move raft client to the LogStream and instantiate the instances using raft group that returned by meta service. But would like to get an approval from [~vrodionov] bq. Re-set the interrupted exception and propagate for InterruptedException. Unwrap and re-throw the ExecutionException? Proper error handling is in my TODO list bq. Doing some poking around here to make sure there's nothing obvious in the "ecosystem" that might default to using . That's for tests only. I was going to discuss the default ports/etc. bq. I'm also wondering – ideally, we can just set 0 and let the service pick an ephemeral port. I guess that would require the service to register itself somewhere for us to find it? You have an idea about how that should work? Exactly like zk works. For masters we have to specify the quorum (list of the host:port pairs). Workers would register themselves in meta. bq. This isn't threadsafe. What about making the global currentGroup some AtomicReference and then copy it into a local variable when you need it? Admittedly, I don't quite grok what this is doing yet That was a workaround for a deadlock in ratis itself. Actually was planning to fix dead lock rather than keep this code :) bq. Probably a good idea to let this be configurable too (multiple NICs with public/private network setups). That's in my TODO as well ( all builder params will be configurable). bq. Heh , switch to InetAddress.getLocalHost().getHostName() for now. I think we'll have to provide the ability later on to specify hostnames for quorum members (same reason as above). heh, that part has a long story :) We have the list of the quorum members, but for each node, we need to identify itself in this list to properly create RaftPeer with a public address. The idea is to have a common configuration without node specific elements. So that was added as a workaround to get everything working in real cluster scenario with fewer efforts. I'm planning to enhance that with resolving quorum ips and check which belongs to the current node. bq. Can we push down the log dir into LogServiceWorker? Of course, that will be configurable as well as other params for the workers. I already mentioned that earlier in the list of things to do. bq. Does it make sense to have some common configuration across the DML and DDL services? well, at least meta quorum id should be common. We also may do it for working dir. For ports we may have some default ports for masters and workers and ability to specify those values in the common configuration. > Implement server membership for LogService Metadata Service > --- > > Key: RATIS-334 > URL: https://issues.apache.org/jira/browse/RATIS-334 > Project: Ratis > Issue Type: Sub-task > Components: LogService >Reporter: Josh Elser >Assignee: Sergey Soldatov >Priority: Major > Attachments: RATIS-334-v1.patch, RATIS-334-v3.patch, > RATIS-334-wip.patch > > > Half of the Metadata Service for managing LogStreams. > RATIS-279 is handling the "DDL" operations, and Rajesh suggested that we spin > out management of the servers which are available to back these LogStreams in > a second task. This is it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RATIS-334) Implement server membership for LogService Metadata Service
[ https://issues.apache.org/jira/browse/RATIS-334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16662883#comment-16662883 ] Sergey Soldatov commented on RATIS-334: --- bq. Just remove these from LogStateMachine instead of commenting out? Ah. Forgot about this one. I commented to make it easier to resolve conflicts during rebase and to track what exactly should be removed from the existing code in case if someone has objections. Just in case, the list of functionality/improvements I'm working on at the moment: 1. Restore quorum in case of node failure. When one of the nodes goes down we will be able to restore quorum with a new peer automatically. At the moment that would happen on slow node message that leader receive if one of the followers was unable to proceed transactions for 60 seconds (possible we need to adjust that default timeout for something less). 2. Experimenting with the retry policy for meta queries. The default one is very aggressive (resend the query if the previous call takes longer than 3 seconds). Not sure how to better handle that. Yet. After that, I will start working on the correctness of all responses and error/exception handling during internal operations. > Implement server membership for LogService Metadata Service > --- > > Key: RATIS-334 > URL: https://issues.apache.org/jira/browse/RATIS-334 > Project: Ratis > Issue Type: Sub-task > Components: LogService >Reporter: Josh Elser >Assignee: Sergey Soldatov >Priority: Major > Attachments: RATIS-334-v1.patch, RATIS-334-v3.patch, > RATIS-334-wip.patch > > > Half of the Metadata Service for managing LogStreams. > RATIS-279 is handling the "DDL" operations, and Rajesh suggested that we spin > out management of the servers which are available to back these LogStreams in > a second task. This is it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RATIS-334) Implement server membership for LogService Metadata Service
[ https://issues.apache.org/jira/browse/RATIS-334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16662888#comment-16662888 ] Vladimir Rodionov commented on RATIS-334: - Someone (me or Sergey) should stop and wait until the other patch is committed. We have 2 conflicting patches (both will require rebasing) at the time. > Implement server membership for LogService Metadata Service > --- > > Key: RATIS-334 > URL: https://issues.apache.org/jira/browse/RATIS-334 > Project: Ratis > Issue Type: Sub-task > Components: LogService >Reporter: Josh Elser >Assignee: Sergey Soldatov >Priority: Major > Attachments: RATIS-334-v1.patch, RATIS-334-v3.patch, > RATIS-334-wip.patch > > > Half of the Metadata Service for managing LogStreams. > RATIS-279 is handling the "DDL" operations, and Rajesh suggested that we spin > out management of the servers which are available to back these LogStreams in > a second task. This is it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RATIS-334) Implement server membership for LogService Metadata Service
[ https://issues.apache.org/jira/browse/RATIS-334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16662909#comment-16662909 ] Josh Elser commented on RATIS-334: -- Thanks for the quick feedback, Sergey. Let's just plan to leave the comments there for now to aid rebase as you say. [~vrodionov], since you said it was OK earlier, I'll commit this one and let you rebase on top of it. Let's plan to get the other things you're working on captured in Jira, Sergey. > Implement server membership for LogService Metadata Service > --- > > Key: RATIS-334 > URL: https://issues.apache.org/jira/browse/RATIS-334 > Project: Ratis > Issue Type: Sub-task > Components: LogService >Reporter: Josh Elser >Assignee: Sergey Soldatov >Priority: Major > Attachments: RATIS-334-v1.patch, RATIS-334-v3.patch, > RATIS-334-wip.patch > > > Half of the Metadata Service for managing LogStreams. > RATIS-279 is handling the "DDL" operations, and Rajesh suggested that we spin > out management of the servers which are available to back these LogStreams in > a second task. This is it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RATIS-371) Change tests so that they do not depend on consecutive log indices
[ https://issues.apache.org/jira/browse/RATIS-371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16662958#comment-16662958 ] Tsz Wo Nicholas Sze commented on RATIS-371: --- [~jnp], thanks for review the patch. > ... As I understand, the logIndex is still sequential, but log-entries have > additional metadata entries in between that add to the count. Yes, that is true. RATIS-352 will add metadata entries so that the state machine log entries becomes non-consecutive. The log index is still sequential. > Change tests so that they do not depend on consecutive log indices > -- > > Key: RATIS-371 > URL: https://issues.apache.org/jira/browse/RATIS-371 > Project: Ratis > Issue Type: Improvement > Components: test >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze >Priority: Major > Attachments: r371_20181024.patch > > > RATIS-352 tries to add metadata entries to the raft log. However, there are > tests expecting consecutive log indices so that they fail intermittently. > This JIRA is to fix the (most of) tests so that they won't depends on > consecutive log indices. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RATIS-334) Implement server membership for LogService Metadata Service
[
https://issues.apache.org/jira/browse/RATIS-334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16662965#comment-16662965
]
Josh Elser commented on RATIS-334:
--
Had to tweak the protobuf-compiler pom stuff to account for the move (keep
generated code out of {{src/main/java}}).
TestLogServiceWithGrpc and TestLogServiceWithNetty were failing because of a
stubbed out implementation. Fixed those by commenting them out.
TestMetaServer is failing for me on OSX. I ran it on a linux box and I think
Maven killed it after 10 minutes. Not sure if this is working for you, Sergey..
> Implement server membership for LogService Metadata Service
> ---
>
> Key: RATIS-334
> URL: https://issues.apache.org/jira/browse/RATIS-334
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Assignee: Sergey Soldatov
>Priority: Major
> Attachments: RATIS-334-v1.patch, RATIS-334-v3.patch,
> RATIS-334-wip.patch
>
>
> Half of the Metadata Service for managing LogStreams.
> RATIS-279 is handling the "DDL" operations, and Rajesh suggested that we spin
> out management of the servers which are available to back these LogStreams in
> a second task. This is it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (RATIS-334) Implement server membership for LogService Metadata Service
[ https://issues.apache.org/jira/browse/RATIS-334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser resolved RATIS-334. -- Resolution: Fixed Fix Version/s: 0.3.0 Thanks for the patch, [~sergey.soldatov]. This is a nice starting point. It's cool to see this coming together. Please spin out follow-on Jira issues to do the cleanup mentioned in the code-review here. > Implement server membership for LogService Metadata Service > --- > > Key: RATIS-334 > URL: https://issues.apache.org/jira/browse/RATIS-334 > Project: Ratis > Issue Type: Sub-task > Components: LogService >Reporter: Josh Elser >Assignee: Sergey Soldatov >Priority: Major > Fix For: 0.3.0 > > Attachments: RATIS-334-v1.patch, RATIS-334-v3.patch, > RATIS-334-wip.patch > > > Half of the Metadata Service for managing LogStreams. > RATIS-279 is handling the "DDL" operations, and Rajesh suggested that we spin > out management of the servers which are available to back these LogStreams in > a second task. This is it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (RATIS-352) Persist commit index in Raft log.
[ https://issues.apache.org/jira/browse/RATIS-352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated RATIS-352: -- Attachment: r352_20181025.patch > Persist commit index in Raft log. > - > > Key: RATIS-352 > URL: https://issues.apache.org/jira/browse/RATIS-352 > Project: Ratis > Issue Type: Improvement > Components: server >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze >Priority: Major > Attachments: r352_20181025.patch > > > Current the commit index is not persisted in the Raft log. When a server > restarts, it cannot apply its log until it has joined a Raft group. There is > no problem if there is a Raft group. > However, a server cannot recover its data when formatting a Raft group is > impossible (e.g. a majority of servers are dead). Ideally, a server can > safely apply the log up to the commit index. The JIRA is to persist the > commit index so that such recovery is possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (RATIS-352) Persist commit index in Raft log.
[ https://issues.apache.org/jira/browse/RATIS-352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated RATIS-352: -- Attachment: (was: r352_20181024.patch) > Persist commit index in Raft log. > - > > Key: RATIS-352 > URL: https://issues.apache.org/jira/browse/RATIS-352 > Project: Ratis > Issue Type: Improvement > Components: server >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze >Priority: Major > Attachments: r352_20181025.patch > > > Current the commit index is not persisted in the Raft log. When a server > restarts, it cannot apply its log until it has joined a Raft group. There is > no problem if there is a Raft group. > However, a server cannot recover its data when formatting a Raft group is > impossible (e.g. a majority of servers are dead). Ideally, a server can > safely apply the log up to the commit index. The JIRA is to persist the > commit index so that such recovery is possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RATIS-352) Persist commit index in Raft log.
[ https://issues.apache.org/jira/browse/RATIS-352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16662985#comment-16662985 ] Tsz Wo Nicholas Sze commented on RATIS-352: --- r352_20181025.patch: sync'ed after RATIS-371. > Persist commit index in Raft log. > - > > Key: RATIS-352 > URL: https://issues.apache.org/jira/browse/RATIS-352 > Project: Ratis > Issue Type: Improvement > Components: server >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze >Priority: Major > Attachments: r352_20181025.patch > > > Current the commit index is not persisted in the Raft log. When a server > restarts, it cannot apply its log until it has joined a Raft group. There is > no problem if there is a Raft group. > However, a server cannot recover its data when formatting a Raft group is > impossible (e.g. a majority of servers are dead). Ideally, a server can > safely apply the log up to the commit index. The JIRA is to persist the > commit index so that such recovery is possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RATIS-369) LogService read/write path (phase 2: more UTs, etc)
[ https://issues.apache.org/jira/browse/RATIS-369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16662988#comment-16662988 ] Josh Elser commented on RATIS-369: -- [~vrodionov], you should be good to rebase this on top of Sergey's latest. > LogService read/write path (phase 2: more UTs, etc) > --- > > Key: RATIS-369 > URL: https://issues.apache.org/jira/browse/RATIS-369 > Project: Ratis > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Major > Attachments: RATIS-369-v1.patch > > > Polish implementation, implement some missing features, add more UTs -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RATIS-369) LogService read/write path (phase 2: more UTs, etc)
[ https://issues.apache.org/jira/browse/RATIS-369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16662997#comment-16662997 ] Vladimir Rodionov commented on RATIS-369: - Hmm, this is what I usually try to avoid : working on a same (almost) code in parallel. Rebasing RATIS-369 on top of RATIS-334 is harder than rewriting patch from scratch. > LogService read/write path (phase 2: more UTs, etc) > --- > > Key: RATIS-369 > URL: https://issues.apache.org/jira/browse/RATIS-369 > Project: Ratis > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Major > Attachments: RATIS-369-v1.patch > > > Polish implementation, implement some missing features, add more UTs -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RATIS-369) LogService read/write path (phase 2: more UTs, etc)
[
https://issues.apache.org/jira/browse/RATIS-369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16662999#comment-16662999
]
Josh Elser commented on RATIS-369:
--
bq. Hmm, this is what I usually try to avoid : working on a same (almost) code
in parallel. Rebasing RATIS-369 on top of RATIS-334 is harder than rewriting
patch from scratch.
I'm sorry, Vlad. I don't know of a better way to do this while we get it off
the ground other than having one person do everything (which we don't have the
time to allow). Really sorry if there's extra pain now. I hope it's brief.
Some general comments from your v1
{code:java}
+ public static enum State {
+OPEN, CLOSED, UNDEFINED
+ }{code}
What's the {{UNDEFINED}} for?
{code:java}
diff --git
a/ratis-logservice/src/test/java/org/apache/ratis/logservice/LogServiceBaseTest.java
b/ratis-logservice/src/test/java/org/apache/ratis/logservice/TestLogServiceAPIBase.java
similarity index 95%
rename from
ratis-logservice/src/test/java/org/apache/ratis/logservice/LogServiceBaseTest.java
rename to
ratis-logservice/src/test/java/org/apache/ratis/logservice/TestLogServiceAPIBase.java
index 82553511..346bf68f 100644
---
a/ratis-logservice/src/test/java/org/apache/ratis/logservice/LogServiceBaseTest.java
+++
b/ratis-logservice/src/test/java/org/apache/ratis/logservice/TestLogServiceAPIBase.java
@@ -39,10 +39,10 @@ import org.junit.Test;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-public abstract class LogServiceBaseTest
+public abstract class TestLogServiceAPIBase
extends BaseTest
implements MiniRaftCluster.Factory.Get {
- public static final Logger LOG =
LoggerFactory.getLogger(LogServiceBaseTest.class);
+ public static final Logger LOG =
LoggerFactory.getLogger(TestLogServiceAPIBase.class);{code}
Do we want {{Test}} in the file name? We don't want to run this class directly,
right?
> LogService read/write path (phase 2: more UTs, etc)
> ---
>
> Key: RATIS-369
> URL: https://issues.apache.org/jira/browse/RATIS-369
> Project: Ratis
> Issue Type: Sub-task
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Major
> Attachments: RATIS-369-v1.patch
>
>
> Polish implementation, implement some missing features, add more UTs
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (RATIS-369) LogService read/write path (phase 2: more UTs, etc)
[ https://issues.apache.org/jira/browse/RATIS-369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16662997#comment-16662997 ] Vladimir Rodionov edited comment on RATIS-369 at 10/24/18 11:37 PM: Deleted was (Author: vrodionov): Hmm, this is what I usually try to avoid : working on a same (almost) code in parallel. Rebasing RATIS-369 on top of RATIS-334 is harder than rewriting patch from scratch. > LogService read/write path (phase 2: more UTs, etc) > --- > > Key: RATIS-369 > URL: https://issues.apache.org/jira/browse/RATIS-369 > Project: Ratis > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Major > Attachments: RATIS-369-v1.patch > > > Polish implementation, implement some missing features, add more UTs -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RATIS-369) LogService read/write path (phase 2: more UTs, etc)
[
https://issues.apache.org/jira/browse/RATIS-369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16663001#comment-16663001
]
Vladimir Rodionov commented on RATIS-369:
-
{quote}
Do we want Test in the file name? We don't want to run this class directly,
right?
{quote}
Yeah, will fix it.
> LogService read/write path (phase 2: more UTs, etc)
> ---
>
> Key: RATIS-369
> URL: https://issues.apache.org/jira/browse/RATIS-369
> Project: Ratis
> Issue Type: Sub-task
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Major
> Attachments: RATIS-369-v1.patch
>
>
> Polish implementation, implement some missing features, add more UTs
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-369) LogService read/write path (phase 2: more UTs, etc)
[
https://issues.apache.org/jira/browse/RATIS-369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16663004#comment-16663004
]
Vladimir Rodionov commented on RATIS-369:
-
{quote}
What's the UNDEFINED for?
{quote}
Binary logic meets tertiary logic :) Will remove it.
> LogService read/write path (phase 2: more UTs, etc)
> ---
>
> Key: RATIS-369
> URL: https://issues.apache.org/jira/browse/RATIS-369
> Project: Ratis
> Issue Type: Sub-task
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Major
> Attachments: RATIS-369-v1.patch
>
>
> Polish implementation, implement some missing features, add more UTs
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-369) LogService read/write path (phase 2: more UTs, etc)
[ https://issues.apache.org/jira/browse/RATIS-369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16663015#comment-16663015 ] Josh Elser commented on RATIS-369: -- Hahah. Thanks Vlad! :) > LogService read/write path (phase 2: more UTs, etc) > --- > > Key: RATIS-369 > URL: https://issues.apache.org/jira/browse/RATIS-369 > Project: Ratis > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Major > Attachments: RATIS-369-v1.patch > > > Polish implementation, implement some missing features, add more UTs -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RATIS-352) Persist commit index in Raft log.
[
https://issues.apache.org/jira/browse/RATIS-352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16663022#comment-16663022
]
Hadoop QA commented on RATIS-352:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
5s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 44s{color}
| {color:red} root generated 1 new + 90 unchanged - 0 fixed = 91 total (was 90)
{color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 31s{color} | {color:orange} root: The patch generated 65 new + 1139
unchanged - 48 fixed = 1204 total (was 1187) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 19m 43s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 0s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
ratis.server.simulation.TestRaftReconfigurationWithSimulatedRpc |
| | ratis.server.simulation.TestLeaderElectionWithSimulatedRpc |
| | ratis.server.simulation.TestServerRestartWithSimulatedRpc |
| | ratis.TestRaftServerLeaderElectionTimeout |
| | ratis.TestRaftServerSlownessDetection |
| | ratis.server.TestRaftLogMetrics |
| | ratis.server.simulation.TestRaftWithSimulatedRpc |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/ratis:date2018-10-24
|
| JIRA Issue | RATIS-352 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12945492/r352_20181025.patch |
| Optional Tests | asflicense cc unit javac javadoc findbugs checkstyle
compile |
| uname | Linux 9579f12a9c88 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-RATIS-Build/yetus-personality.sh
|
| git revision | master / 85d8e02 |
| Default Java | 1.8.0_181 |
| javac |
https://builds.apache.org/job/PreCommit-RATIS-Build/456/artifact/out/diff-compile-javac-root.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-RATIS-Build/456/artifact/out/diff-checkstyle-root.txt
|
| unit |
https://builds.apache.org/job/PreCommit-RATIS-Build/456/artifact/out/patch-unit-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-RATIS-Build/456/testReport/ |
| modules | C: ratis-proto ratis-server
[jira] [Created] (RATIS-372) Basic test harness for LogService
Josh Elser created RATIS-372: Summary: Basic test harness for LogService Key: RATIS-372 URL: https://issues.apache.org/jira/browse/RATIS-372 Project: Ratis Issue Type: Task Components: LogService Reporter: Josh Elser Assignee: Rajeshbabu Chintaguntla We should have something that can stand up a logservice in a pseudo-distributed manner. Docker is all the rage right now, and would make it easy to deploy onto something like k8s in the future. Using [docker-compose|https://docs.docker.com/compose/] would provide us a nice way to have one docker image for the metadata service daemons, another for logservice daemons (if needed), and then create a network that connects them all together. The final docker-compose yaml would be something like: It would be nice to provide a "client" container in which we show a basic create/write/read/delete example to give folks a starting point. * 1 network * 3 instances of metadata service statemachines * 3 instances of log service statemachines * 1 image with the example client. [~chrajeshbab...@gmail.com], make sense to you? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RATIS-370) TestRaftServerSlownessDetection keep failing
[ https://issues.apache.org/jira/browse/RATIS-370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16663241#comment-16663241 ] Tsz Wo Nicholas Sze commented on RATIS-370: --- Found two problems: # Minor: The RequestHandler thread in simulated rpc may not terminate so that the test will timeout in such case. # Real bug: StateMachine.notifySlowness(..) is never called even retrying for 5 seconds. > TestRaftServerSlownessDetection keep failing > > > Key: RATIS-370 > URL: https://issues.apache.org/jira/browse/RATIS-370 > Project: Ratis > Issue Type: Bug > Components: server, test >Reporter: Tsz Wo Nicholas Sze >Priority: Major > > TestRaftServerSlownessDetection failure keep showing up in QA reports. In my > machine, it also fails every few runs. > BTW, TestRaftServerSlownessDetection only check simulated rpc. The gRPC > implementation is never tested. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (RATIS-370) TestRaftServerSlownessDetection keep failing
[ https://issues.apache.org/jira/browse/RATIS-370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated RATIS-370: -- Attachment: r370_20181025.patch > TestRaftServerSlownessDetection keep failing > > > Key: RATIS-370 > URL: https://issues.apache.org/jira/browse/RATIS-370 > Project: Ratis > Issue Type: Bug > Components: server, test >Reporter: Tsz Wo Nicholas Sze >Priority: Major > Attachments: r370_20181025.patch > > > TestRaftServerSlownessDetection failure keep showing up in QA reports. In my > machine, it also fails every few runs. > BTW, TestRaftServerSlownessDetection only check simulated rpc. The gRPC > implementation is never tested. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RATIS-370) TestRaftServerSlownessDetection keep failing
[ https://issues.apache.org/jira/browse/RATIS-370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16663270#comment-16663270 ] Tsz Wo Nicholas Sze commented on RATIS-370: --- r370_20181025.patch: fixes #1 and adds multiple attempts but TestRaftServerSlownessDetection still may fail. > TestRaftServerSlownessDetection keep failing > > > Key: RATIS-370 > URL: https://issues.apache.org/jira/browse/RATIS-370 > Project: Ratis > Issue Type: Bug > Components: server, test >Reporter: Tsz Wo Nicholas Sze >Priority: Major > Attachments: r370_20181025.patch > > > TestRaftServerSlownessDetection failure keep showing up in QA reports. In my > machine, it also fails every few runs. > BTW, TestRaftServerSlownessDetection only check simulated rpc. The gRPC > implementation is never tested. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (RATIS-373) Tolerate half written log entry
Tsz Wo Nicholas Sze created RATIS-373: - Summary: Tolerate half written log entry Key: RATIS-373 URL: https://issues.apache.org/jira/browse/RATIS-373 Project: Ratis Issue Type: Improvement Components: server Reporter: Tsz Wo Nicholas Sze Assignee: Tsz Wo Nicholas Sze When a server is killed, the last log entry may be half written. RaftLog will then consider the log as corrupted and the server will fail to start up. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (RATIS-352) Persist commit index in Raft log.
[ https://issues.apache.org/jira/browse/RATIS-352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated RATIS-352: -- Attachment: (was: r352_20181025.patch) > Persist commit index in Raft log. > - > > Key: RATIS-352 > URL: https://issues.apache.org/jira/browse/RATIS-352 > Project: Ratis > Issue Type: Improvement > Components: server >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze >Priority: Major > Attachments: r352_20181025b.patch > > > Current the commit index is not persisted in the Raft log. When a server > restarts, it cannot apply its log until it has joined a Raft group. There is > no problem if there is a Raft group. > However, a server cannot recover its data when formatting a Raft group is > impossible (e.g. a majority of servers are dead). Ideally, a server can > safely apply the log up to the commit index. The JIRA is to persist the > commit index so that such recovery is possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (RATIS-352) Persist commit index in Raft log.
[ https://issues.apache.org/jira/browse/RATIS-352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated RATIS-352: -- Attachment: r352_20181025b.patch > Persist commit index in Raft log. > - > > Key: RATIS-352 > URL: https://issues.apache.org/jira/browse/RATIS-352 > Project: Ratis > Issue Type: Improvement > Components: server >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze >Priority: Major > Attachments: r352_20181025b.patch > > > Current the commit index is not persisted in the Raft log. When a server > restarts, it cannot apply its log until it has joined a Raft group. There is > no problem if there is a Raft group. > However, a server cannot recover its data when formatting a Raft group is > impossible (e.g. a majority of servers are dead). Ideally, a server can > safely apply the log up to the commit index. The JIRA is to persist the > commit index so that such recovery is possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (RATIS-373) Tolerate the last half written log entry
[ https://issues.apache.org/jira/browse/RATIS-373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated RATIS-373: -- Summary: Tolerate the last half written log entry (was: Tolerate half written log entry) > Tolerate the last half written log entry > > > Key: RATIS-373 > URL: https://issues.apache.org/jira/browse/RATIS-373 > Project: Ratis > Issue Type: Improvement > Components: server >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze >Priority: Major > > When a server is killed, the last log entry may be half written. RaftLog > will then consider the log as corrupted and the server will fail to start up. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RATIS-352) Persist commit index in Raft log.
[
https://issues.apache.org/jira/browse/RATIS-352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16663300#comment-16663300
]
Hadoop QA commented on RATIS-352:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 9 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
5s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 47s{color}
| {color:red} root generated 1 new + 90 unchanged - 0 fixed = 91 total (was 90)
{color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 30s{color} | {color:orange} root: The patch generated 65 new + 1234
unchanged - 48 fixed = 1299 total (was 1282) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 13m 36s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
10s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 6s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | ratis.netty.TestRaftReconfigurationWithNetty |
| | ratis.grpc.TestRaftWithGrpc |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/ratis:date2018-10-25
|
| JIRA Issue | RATIS-352 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12945535/r352_20181025b.patch |
| Optional Tests | asflicense cc unit javac javadoc findbugs checkstyle
compile |
| uname | Linux 0891f8305dff 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-RATIS-Build/yetus-personality.sh
|
| git revision | master / 85d8e02 |
| Default Java | 1.8.0_181 |
| javac |
https://builds.apache.org/job/PreCommit-RATIS-Build/457/artifact/out/diff-compile-javac-root.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-RATIS-Build/457/artifact/out/diff-checkstyle-root.txt
|
| unit |
https://builds.apache.org/job/PreCommit-RATIS-Build/457/artifact/out/patch-unit-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-RATIS-Build/457/testReport/ |
| modules | C: ratis-proto ratis-server U: . |
| Console output |
https://builds.apache.org/job/PreCommit-RATIS-Build/457/console |
| Powered by | Apache Yetus 0.5.0 http://yetus.apache.org |
This message was automatically generated.
> Persist commit index in Raft log.
> -
>
> Key: RATIS-352
