[jira] [Commented] (RATIS-253) Segmented Raft log rolls over segment frequently

2018-10-17 Thread Tsz Wo Nicholas Sze (JIRA) 


[ 
https://issues.apache.org/jira/browse/RATIS-253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16653090#comment-16653090
 ] 

Tsz Wo Nicholas Sze commented on RATIS-253:
---

Filed RATIS-355 for the bug.

> Segmented Raft log rolls over segment frequently
> 
>
> Key: RATIS-253
> URL: https://issues.apache.org/jira/browse/RATIS-253
> Project: Ratis
>  Issue Type: Bug
>  Components: server
>Affects Versions: 0.2.0
>Reporter: Mukul Kumar Singh
>Assignee: Rajeshbabu Chintaguntla
>Priority: Major
>  Labels: ozone
> Fix For: 0.3.0
>
> Attachments: RATIS-253.patch
>
>
> With statemachine implementation which abstract out the state machine data 
> from the append entries, frequent log rollovers can be observed.
> Looking into the code, this seems to be because of 2 reasons.
> a) LogSegment#append, updates the total size. However the size which needs to 
> be considered should not include the statemachine data.
> b) Also SegmentRaftLog#isSegmentFull also considers the total proto object. 
> Here too, only the entry to be written to the log should be considered.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (RATIS-253) Segmented Raft log rolls over segment frequently

2018-10-16 Thread Tsz Wo Nicholas Sze (JIRA) 


[ 
https://issues.apache.org/jira/browse/RATIS-253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16652997#comment-16652997
 ] 

Tsz Wo Nicholas Sze commented on RATIS-253:
---

The formula is to calculate the LogEntryProto size after stateMachineData is 
removed so that we may implement as below.  TestRestartRaftPeer can pass with 
it.
{code}
  static long getEntrySize(LogEntryProto entry) {
final int serialized = 
ProtoUtils.removeStateMachineData(entry).getSerializedSize();
return serialized + CodedOutputStream.computeUInt32SizeNoTag(serialized) + 
4;
  }
{code}
However, the above method may unnecessarily copy data.  Since we are going to  
refactor LogEntryProto (RATIS-353), let use it as a workaround.  We will 
improve it in RATIS-353 or its subtasks.


> Segmented Raft log rolls over segment frequently
> 
>
> Key: RATIS-253
> URL: https://issues.apache.org/jira/browse/RATIS-253
> Project: Ratis
>  Issue Type: Bug
>  Components: server
>Affects Versions: 0.2.0
>Reporter: Mukul Kumar Singh
>Assignee: Rajeshbabu Chintaguntla
>Priority: Major
>  Labels: ozone
> Fix For: 0.3.0
>
> Attachments: RATIS-253.patch
>
>
> With statemachine implementation which abstract out the state machine data 
> from the append entries, frequent log rollovers can be observed.
> Looking into the code, this seems to be because of 2 reasons.
> a) LogSegment#append, updates the total size. However the size which needs to 
> be considered should not include the statemachine data.
> b) Also SegmentRaftLog#isSegmentFull also considers the total proto object. 
> Here too, only the entry to be written to the log should be considered.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (RATIS-253) Segmented Raft log rolls over segment frequently

2018-10-16 Thread Tsz Wo Nicholas Sze (JIRA) 


[ 
https://issues.apache.org/jira/browse/RATIS-253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16652991#comment-16652991
 ] 

Tsz Wo Nicholas Sze commented on RATIS-253:
---

This probably is the cause of the bug in the formula -- In 
ProtoUtils.removeStateMachineData(..), it also calls 
setStateMachineDataAttached(..) and setSerializedProtobufSize(..).  The formula 
only subtracts the stateMachineData size but not adds the size for 
serializedProtobufSize.  Let me see if I could fix it.

> Segmented Raft log rolls over segment frequently
> 
>
> Key: RATIS-253
> URL: https://issues.apache.org/jira/browse/RATIS-253
> Project: Ratis
>  Issue Type: Bug
>  Components: server
>Affects Versions: 0.2.0
>Reporter: Mukul Kumar Singh
>Assignee: Rajeshbabu Chintaguntla
>Priority: Major
>  Labels: ozone
> Fix For: 0.3.0
>
> Attachments: RATIS-253.patch
>
>
> With statemachine implementation which abstract out the state machine data 
> from the append entries, frequent log rollovers can be observed.
> Looking into the code, this seems to be because of 2 reasons.
> a) LogSegment#append, updates the total size. However the size which needs to 
> be considered should not include the statemachine data.
> b) Also SegmentRaftLog#isSegmentFull also considers the total proto object. 
> Here too, only the entry to be written to the log should be considered.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (RATIS-253) Segmented Raft log rolls over segment frequently

2018-10-16 Thread Tsz Wo Nicholas Sze (JIRA) 


[ 
https://issues.apache.org/jira/browse/RATIS-253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16652113#comment-16652113
 ] 

Tsz Wo Nicholas Sze commented on RATIS-253:
---

[~rajeshbabu], just found that TestRestartRaftPeer failed after this commit 
(6369d4e65eb3d733d101786b6c79ca600bdace77).  If I reset the head to right 
before this (2ce10dc1a88b7657c7d63e02774459afd8724c4d), the test can pass.
{code}
commit 6369d4e65eb3d733d101786b6c79ca600bdace77 (HEAD -> master)
Author: Tsz Wo Nicholas Sze 
Date:   Tue Sep 11 12:03:57 2018 -0700

RATIS-253. Segmented Raft log rolls over segment frequently.  Contributed 
by Rajeshbabu Chintaguntla

commit 2ce10dc1a88b7657c7d63e02774459afd8724c4d
Author: Tsz Wo Nicholas Sze 
Date:   Tue Sep 11 11:04:22 2018 -0700

RATIS-310. Add support for Retry Policy in Ratis.  Contributed by 
Shashikant Banerjee
{code}
Could you take a look?

> Segmented Raft log rolls over segment frequently
> 
>
> Key: RATIS-253
> URL: https://issues.apache.org/jira/browse/RATIS-253
> Project: Ratis
>  Issue Type: Bug
>  Components: server
>Affects Versions: 0.2.0
>Reporter: Mukul Kumar Singh
>Assignee: Rajeshbabu Chintaguntla
>Priority: Major
>  Labels: ozone
> Fix For: 0.3.0
>
> Attachments: RATIS-253.patch
>
>
> With statemachine implementation which abstract out the state machine data 
> from the append entries, frequent log rollovers can be observed.
> Looking into the code, this seems to be because of 2 reasons.
> a) LogSegment#append, updates the total size. However the size which needs to 
> be considered should not include the statemachine data.
> b) Also SegmentRaftLog#isSegmentFull also considers the total proto object. 
> Here too, only the entry to be written to the log should be considered.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (RATIS-253) Segmented Raft log rolls over segment frequently

2018-09-11 Thread Rajeshbabu Chintaguntla (JIRA) 


[ 
https://issues.apache.org/jira/browse/RATIS-253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16611493#comment-16611493
 ] 

Rajeshbabu Chintaguntla commented on RATIS-253:
---

Thanks [~szetszwo] for reviewing and committing.

> Segmented Raft log rolls over segment frequently
> 
>
> Key: RATIS-253
> URL: https://issues.apache.org/jira/browse/RATIS-253
> Project: Ratis
>  Issue Type: Bug
>  Components: server
>Affects Versions: 0.2.0
>Reporter: Mukul Kumar Singh
>Assignee: Rajeshbabu Chintaguntla
>Priority: Major
>  Labels: ozone
> Fix For: 0.3.0
>
> Attachments: RATIS-253.patch
>
>
> With statemachine implementation which abstract out the state machine data 
> from the append entries, frequent log rollovers can be observed.
> Looking into the code, this seems to be because of 2 reasons.
> a) LogSegment#append, updates the total size. However the size which needs to 
> be considered should not include the statemachine data.
> b) Also SegmentRaftLog#isSegmentFull also considers the total proto object. 
> Here too, only the entry to be written to the log should be considered.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (RATIS-253) Segmented Raft log rolls over segment frequently

2018-09-11 Thread Hadoop QA (JIRA) 


[ 
https://issues.apache.org/jira/browse/RATIS-253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16611071#comment-16611071
 ] 

Hadoop QA commented on RATIS-253:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
33s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  
7s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 15s{color} | {color:orange} root: The patch generated 3 new + 121 unchanged 
- 0 fixed = 124 total (was 121) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
33s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 20m 47s{color} 
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
 8s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 32s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | ratis.grpc.TestRaftAsyncWithGrpc |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/ratis:date2018-09-11 
|
| JIRA Issue | RATIS-253 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12936117/RATIS-253.patch |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  checkstyle  
compile  |
| uname | Linux 6b8ed39cbb4b 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27 
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-RATIS-Build/yetus-personality.sh
 |
| git revision | master / 2ce10dc |
| Default Java | 1.8.0_181 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-RATIS-Build/336/artifact/out/diff-checkstyle-root.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-RATIS-Build/336/artifact/out/patch-unit-root.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-RATIS-Build/336/testReport/ |
| modules | C: ratis-server U: ratis-server |
| Console output | 
https://builds.apache.org/job/PreCommit-RATIS-Build/336/console |
| Powered by | Apache Yetus 0.5.0   http://yetus.apache.org |


This message was automatically generated.



> Segmented Raft log rolls over segment frequently
> 
>
> Key: RATIS-253
> URL: https://issues.apache.org/jira/browse/RATIS-253
> Project: Ratis
>  Issue Type: Bug
>  Components: server
>Affects Versions: 0.2.0
>Reporter: Mukul Kumar Singh
>Assignee: Rajeshbabu Chintaguntla
>Priority: Major
>  Labels: ozone
> Attachments: RATIS-253.patch
>
>
> With statemachine implementation which abstract out the state machine data 
> from the append entri

[jira] [Commented] (RATIS-253) Segmented Raft log rolls over segment frequently

2018-09-11 Thread Hadoop QA (JIRA) 


[ 
https://issues.apache.org/jira/browse/RATIS-253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16611067#comment-16611067
 ] 

Hadoop QA commented on RATIS-253:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
35s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
12s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 22s{color} | {color:orange} root: The patch generated 3 new + 121 unchanged 
- 0 fixed = 124 total (was 121) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  7m 21s{color} 
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
 6s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 34s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | ratis.TestRaftServerSlownessDetection |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/ratis:date2018-09-11 
|
| JIRA Issue | RATIS-253 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12936117/RATIS-253.patch |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  checkstyle  
compile  |
| uname | Linux 4abc0d6cc8f7 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14 
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-RATIS-Build/yetus-personality.sh
 |
| git revision | master / 2ce10dc |
| Default Java | 1.8.0_181 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-RATIS-Build/337/artifact/out/diff-checkstyle-root.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-RATIS-Build/337/artifact/out/patch-unit-root.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-RATIS-Build/337/testReport/ |
| modules | C: ratis-server U: ratis-server |
| Console output | 
https://builds.apache.org/job/PreCommit-RATIS-Build/337/console |
| Powered by | Apache Yetus 0.5.0   http://yetus.apache.org |


This message was automatically generated.



> Segmented Raft log rolls over segment frequently
> 
>
> Key: RATIS-253
> URL: https://issues.apache.org/jira/browse/RATIS-253
> Project: Ratis
>  Issue Type: Bug
>  Components: server
>Affects Versions: 0.2.0
>Reporter: Mukul Kumar Singh
>Assignee: Rajeshbabu Chintaguntla
>Priority: Major
>  Labels: ozone
> Attachments: RATIS-253.patch
>
>
> With statemachine implementation which abstract out the state machine data 
> from the append

[jira] [Commented] (RATIS-253) Segmented Raft log rolls over segment frequently

2018-09-11 Thread Tsz Wo Nicholas Sze (JIRA) 


[ 
https://issues.apache.org/jira/browse/RATIS-253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16611044#comment-16611044
 ] 

Tsz Wo Nicholas Sze commented on RATIS-253:
---

+1 patch looks good.  Pending Jenkins.

> Segmented Raft log rolls over segment frequently
> 
>
> Key: RATIS-253
> URL: https://issues.apache.org/jira/browse/RATIS-253
> Project: Ratis
>  Issue Type: Bug
>  Components: server
>Affects Versions: 0.2.0
>Reporter: Mukul Kumar Singh
>Assignee: Rajeshbabu Chintaguntla
>Priority: Major
>  Labels: ozone
> Attachments: RATIS-253.patch
>
>
> With statemachine implementation which abstract out the state machine data 
> from the append entries, frequent log rollovers can be observed.
> Looking into the code, this seems to be because of 2 reasons.
> a) LogSegment#append, updates the total size. However the size which needs to 
> be considered should not include the statemachine data.
> b) Also SegmentRaftLog#isSegmentFull also considers the total proto object. 
> Here too, only the entry to be written to the log should be considered.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (RATIS-253) Segmented Raft log rolls over segment frequently

2018-08-17 Thread Rajeshbabu Chintaguntla (JIRA) 


[ 
https://issues.apache.org/jira/browse/RATIS-253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16584542#comment-16584542
 ] 

Rajeshbabu Chintaguntla commented on RATIS-253:
---

[~msingh][~szetszwo] Here is the patch not including the statemachine data size 
while calculating entry size.

> Segmented Raft log rolls over segment frequently
> 
>
> Key: RATIS-253
> URL: https://issues.apache.org/jira/browse/RATIS-253
> Project: Ratis
>  Issue Type: Bug
>  Components: server
>Affects Versions: 0.2.0
>Reporter: Mukul Kumar Singh
>Assignee: Rajeshbabu Chintaguntla
>Priority: Major
>  Labels: ozone
> Attachments: RATIS-253.patch
>
>
> With statemachine implementation which abstract out the state machine data 
> from the append entries, frequent log rollovers can be observed.
> Looking into the code, this seems to be because of 2 reasons.
> a) LogSegment#append, updates the total size. However the size which needs to 
> be considered should not include the statemachine data.
> b) Also SegmentRaftLog#isSegmentFull also considers the total proto object. 
> Here too, only the entry to be written to the log should be considered.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (RATIS-253) Segmented Raft log rolls over segment frequently

2018-06-12 Thread Tsz Wo Nicholas Sze (JIRA) 


[ 
https://issues.apache.org/jira/browse/RATIS-253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16509460#comment-16509460
 ] 

Tsz Wo Nicholas Sze commented on RATIS-253:
---

Good catch!  We should not include state machine data in 
LogSegment.getEntrySize(..).

> Segmented Raft log rolls over segment frequently
> 
>
> Key: RATIS-253
> URL: https://issues.apache.org/jira/browse/RATIS-253
> Project: Ratis
>  Issue Type: Bug
>Affects Versions: 0.2.0-alpha
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
>Priority: Major
> Fix For: 0.2.0-alpha
>
>
> With statemachine implementation which abstract out the state machine data 
> from the append entries, frequent log rollovers can be observed.
> Looking into the code, this seems to be because of 2 reasons.
> a) LogSegment#append, updates the total size. However the size which needs to 
> be considered should not include the statemachine data.
> b) Also SegmentRaftLog#isSegmentFull also considers the total proto object. 
> Here too, only the entry to be written to the log should be considered.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)