[jira] [Commented] (RATIS-477) Improve LogStateMachine.processReadRequest()
[
https://issues.apache.org/jira/browse/RATIS-477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812946#comment-16812946
]
Tsz Wo Nicholas Sze commented on RATIS-477:
---
> ... Looking like it's going to miss the 0.3.0 bus.
[~elserj], if this is important to LogService, please free feel to discuss it
on the mailing list. We probably still have time if this can be committed soon.
> Improve LogStateMachine.processReadRequest()
>
>
> Key: RATIS-477
> URL: https://issues.apache.org/jira/browse/RATIS-477
> Project: Ratis
> Issue Type: Improvement
> Components: LogService
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 0.3.0
>
> Attachments: RATIS-477.001.patch
>
>
> [~vrodionov] and [~rajeshbabu] were asking about this in RATIS-470. Can we
> improve the logic that reads through the "head" of the RaftLog to find the
> "LogService offset"?
> Vlad suggested:
> {quote}
> Josh, can we simplify this a little bit? Client gets Raft log ID one every
> append, for multi-get calls we can use this LogID (RAFT) as our start log ID
> to avoid iterating through all logs, then you can unwind log entries by using
> new logic, see above:
> int numRecordsInAppend = append.getDataCount();
> There is no need to iterate from the very beginning
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-477) Improve LogStateMachine.processReadRequest()
[
https://issues.apache.org/jira/browse/RATIS-477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812728#comment-16812728
]
Josh Elser commented on RATIS-477:
--
Any comments on this? Looking like it's going to miss the 0.3.0 bus.
> Improve LogStateMachine.processReadRequest()
>
>
> Key: RATIS-477
> URL: https://issues.apache.org/jira/browse/RATIS-477
> Project: Ratis
> Issue Type: Improvement
> Components: LogService
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 0.3.0
>
> Attachments: RATIS-477.001.patch
>
>
> [~vrodionov] and [~rajeshbabu] were asking about this in RATIS-470. Can we
> improve the logic that reads through the "head" of the RaftLog to find the
> "LogService offset"?
> Vlad suggested:
> {quote}
> Josh, can we simplify this a little bit? Client gets Raft log ID one every
> append, for multi-get calls we can use this LogID (RAFT) as our start log ID
> to avoid iterating through all logs, then you can unwind log entries by using
> new logic, see above:
> int numRecordsInAppend = append.getDataCount();
> There is no need to iterate from the very beginning
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-477) Improve LogStateMachine.processReadRequest()
[
https://issues.apache.org/jira/browse/RATIS-477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16801707#comment-16801707
]
Josh Elser commented on RATIS-477:
--
{quote}
1. Memory consumption in InMemoryLogServiceRaftLogIndex - to keep pair of 8byte
values you create 2 objects along with internal object of CSLM itself - we are
looking into 100-200 bytes overhead
2. Snapshotting of in memory index can take seconds (maybe 10s of seconds) - is
it safe? State machine snapshot is a synchronous operation
3. Scalability of an in-memory approach? I think it's good for testing only.
CSLM in pre-Java11 world does not scale beyond low 10s millions KVs.
{quote}
I think I would say that these are all satisfied with an upper-bound on the
size of a log that we expect the system to handle today (e.g. what have we
tried and know to work), and a default index "interval" that avoids creating
too many of these in-memory entries. I think we are both in agreement that an
in-memory solution may not be the final solution for "tomorrow", but my hope
was that this would be adoption a little easier "today".
bq. 4. How and when to clean index? No code for that yet.
Right. Logs are immutable so we don't need clean up for that. I think I had
mentioned before that we could automatically "re-index" to clean up elements as
the log grows big, but punting on that for now (for the same reasons as above).
bq. 5. There should range delete API for index or delete all up to recordId
To serve what purpose?
bq. 6. Any other other than in-memory implementation on your mind?
One "improvement" over this version today might be to try to use off-heap
memory, but I'm not sure how necessary that would be yet. Another thought is
that we could push an update for an index entry we create into the log itself,
and, as each follower gets that update, they update their in-memory state. I'm
not sure if it is safe to do this from within the state machine though.
> Improve LogStateMachine.processReadRequest()
>
>
> Key: RATIS-477
> URL: https://issues.apache.org/jira/browse/RATIS-477
> Project: Ratis
> Issue Type: Improvement
> Components: LogService
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 0.3.0
>
> Attachments: RATIS-477.001.patch
>
>
> [~vrodionov] and [~rajeshbabu] were asking about this in RATIS-470. Can we
> improve the logic that reads through the "head" of the RaftLog to find the
> "LogService offset"?
> Vlad suggested:
> {quote}
> Josh, can we simplify this a little bit? Client gets Raft log ID one every
> append, for multi-get calls we can use this LogID (RAFT) as our start log ID
> to avoid iterating through all logs, then you can unwind log entries by using
> new logic, see above:
> int numRecordsInAppend = append.getDataCount();
> There is no need to iterate from the very beginning
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-477) Improve LogStateMachine.processReadRequest()
[
https://issues.apache.org/jira/browse/RATIS-477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16801213#comment-16801213
]
Vladimir Rodionov commented on RATIS-477:
-
[~elserj], there are some open questions which need to be addressed before
considering next version of the patch:
1. Memory consumption in InMemoryLogServiceRaftLogIndex - to keep pair of 8byte
values you create 2 objects along with internal object of CSLM itself - we are
looking into 100-200 bytes overhead
2. Snapshotting of in memory index can take seconds (maybe 10s of seconds) - is
it safe? State machine snapshot is a synchronous operation
3. Scalability of an in-memory approach? I think it's good for testing only.
CSLM in pre-Java11 world does not scale beyond low 10s millions KVs.
4. How and when to clean index? No code for that yet.
5. There should range delete API for index or delete all up to recordId
6. Any other other than in-memory implementation on your mind?
> Improve LogStateMachine.processReadRequest()
>
>
> Key: RATIS-477
> URL: https://issues.apache.org/jira/browse/RATIS-477
> Project: Ratis
> Issue Type: Improvement
> Components: LogService
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 0.3.0
>
> Attachments: RATIS-477.001.patch
>
>
> [~vrodionov] and [~rajeshbabu] were asking about this in RATIS-470. Can we
> improve the logic that reads through the "head" of the RaftLog to find the
> "LogService offset"?
> Vlad suggested:
> {quote}
> Josh, can we simplify this a little bit? Client gets Raft log ID one every
> append, for multi-get calls we can use this LogID (RAFT) as our start log ID
> to avoid iterating through all logs, then you can unwind log entries by using
> new logic, see above:
> int numRecordsInAppend = append.getDataCount();
> There is no need to iterate from the very beginning
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-477) Improve LogStateMachine.processReadRequest()
[
https://issues.apache.org/jira/browse/RATIS-477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789914#comment-16789914
]
Hadoop QA commented on RATIS-477:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 11s{color} | {color:orange} root: The patch generated 12 new + 21 unchanged
- 3 fixed = 33 total (was 24) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 6 line(s) with tabs. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
32s{color} | {color:red} root generated 6 new + 94 unchanged - 6 fixed = 100
total (was 100) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 12m 51s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 38s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | ratis.netty.TestRaftReconfigurationWithNetty |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/ratis:date2019-03-11
|
| JIRA Issue | RATIS-477 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12962022/RATIS-477.001.patch |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
checkstyle compile cc |
| uname | Linux 501f105b6fc2 4.4.0-139-generic #165~14.04.1-Ubuntu SMP Wed Oct
31 10:55:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-RATIS-Build/yetus-personality.sh
|
| git revision | master / 35838f0 |
| maven | version: Apache Maven 3.6.0
(97c98ec64a1fdfee7767ce5ffb20918da4f719f3; 2018-10-24T18:41:47Z) |
| Default Java | 1.8.0_191 |
| checkstyle |
https://builds.apache.org/job/PreCommit-RATIS-Build/672/artifact/out/diff-checkstyle-root.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-RATIS-Build/672/artifact/out/whitespace-eol.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-RATIS-Build/672/artifact/out/whitespace-tabs.txt
|
| javadoc |
https://builds.apache.org/job/PreCommit-RATIS-Build/672/artifact/out/diff-javadoc-javadoc-root.txt
|
| unit |
https://builds.apache.org/job/PreCommit-RATIS-Build/672/artifact/out/patch-unit-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-RATIS-Build/672/testReport/ |
| Max. process+thread count | 2052 (vs. ulimit of 5000) |
| modules | C: ratis-logservice U: ratis-logservice |
| Console output |
[jira] [Commented] (RATIS-477) Improve LogStateMachine.processReadRequest()
[
https://issues.apache.org/jira/browse/RATIS-477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763999#comment-16763999
]
Josh Elser commented on RATIS-477:
--
Filed RATIS-481 because I was getting stuck trying to work on this. Cleaning up
processReadRequest into its own class should help with what I was trying to do
here.
> Improve LogStateMachine.processReadRequest()
>
>
> Key: RATIS-477
> URL: https://issues.apache.org/jira/browse/RATIS-477
> Project: Ratis
> Issue Type: Improvement
> Components: LogService
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 0.4.0
>
>
> [~vrodionov] and [~rajeshbabu] were asking about this in RATIS-470. Can we
> improve the logic that reads through the "head" of the RaftLog to find the
> "LogService offset"?
> Vlad suggested:
> {quote}
> Josh, can we simplify this a little bit? Client gets Raft log ID one every
> append, for multi-get calls we can use this LogID (RAFT) as our start log ID
> to avoid iterating through all logs, then you can unwind log entries by using
> new logic, see above:
> int numRecordsInAppend = append.getDataCount();
> There is no need to iterate from the very beginning
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-477) Improve LogStateMachine.processReadRequest()
[
https://issues.apache.org/jira/browse/RATIS-477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758598#comment-16758598
]
Vladimir Rodionov commented on RATIS-477:
-
I think that we have to avoid future technical debt from the very beginning.
We provide read log entry by Id only for one use case - to resume scanning log
entries from a last known position (replication use case). Some inefficiency in
locating this position is OK, as since this is going to be non frequent
operation. Keeping in memory index for that? Not sure it is worth it.
{code}
/**
* Reads (potentially) multiple log entries starting from a given startId
* @param buffer
* @param startId
* @return next logId to read next log entries from or
the same startId if available space in a buffer is not sufficient
**/
long nextId read(ByteBuffer buffer, long startId)
{code}
This method always reads at least one whole Raft log entry, it skips Ratis
internal log entries and never stops reading in the middle
of a Raft log entry (if it is multi-log entry - all log entries MUST be read)
We can provide utility class which can extract log records blobs from a buffer,
as well as number of records
{code}
long nextId = read(buffer, startId);
if (nextId != startId) {
// successful read
int numRead = Utils.numEntries(buffer);
Iterator entries = Utils.iterator(buffer);
while(entries.hasNext()) {
byte[] entry = entries.next();
}
{code}
This is raw access low level.
> Improve LogStateMachine.processReadRequest()
>
>
> Key: RATIS-477
> URL: https://issues.apache.org/jira/browse/RATIS-477
> Project: Ratis
> Issue Type: Improvement
> Components: LogService
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 0.4.0
>
>
> [~vrodionov] and [~rajeshbabu] were asking about this in RATIS-470. Can we
> improve the logic that reads through the "head" of the RaftLog to find the
> "LogService offset"?
> Vlad suggested:
> {quote}
> Josh, can we simplify this a little bit? Client gets Raft log ID one every
> append, for multi-get calls we can use this LogID (RAFT) as our start log ID
> to avoid iterating through all logs, then you can unwind log entries by using
> new logic, see above:
> int numRecordsInAppend = append.getDataCount();
> There is no need to iterate from the very beginning
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-477) Improve LogStateMachine.processReadRequest()
[
https://issues.apache.org/jira/browse/RATIS-477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16757616#comment-16757616
]
Josh Elser commented on RATIS-477:
--
I think 3 & 4 would be tricky but not impediments. Moreso, I think this would
be something trivial we could put into place now and revisit when we have
better tooling.
"Shrinking" the index (making it less granular) would be possible, at worse,
re-reading the log. We might be able to just sample the index (e.g. remove
every-other index entry) if it gets too big?
My thinking here is that even a sub-optimal (cost of maintenance and GC costs)
index would be better than the non-performant solution in the code now. WDYT?
> Improve LogStateMachine.processReadRequest()
>
>
> Key: RATIS-477
> URL: https://issues.apache.org/jira/browse/RATIS-477
> Project: Ratis
> Issue Type: Improvement
> Components: LogService
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 0.4.0
>
>
> [~vrodionov] and [~rajeshbabu] were asking about this in RATIS-470. Can we
> improve the logic that reads through the "head" of the RaftLog to find the
> "LogService offset"?
> Vlad suggested:
> {quote}
> Josh, can we simplify this a little bit? Client gets Raft log ID one every
> append, for multi-get calls we can use this LogID (RAFT) as our start log ID
> to avoid iterating through all logs, then you can unwind log entries by using
> new logic, see above:
> int numRecordsInAppend = append.getDataCount();
> There is no need to iterate from the very beginning
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-477) Improve LogStateMachine.processReadRequest()
[
https://issues.apache.org/jira/browse/RATIS-477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16757601#comment-16757601
]
Vladimir Rodionov commented on RATIS-477:
-
Hmm:
1. you will need to keep new data structure in sync in the cluster, hence it
should be part of a snapshot. Cost?
2. Size of the index?
3. Maintenance cost?
4. GC if size is quite big and it will be big, yes?
> Improve LogStateMachine.processReadRequest()
>
>
> Key: RATIS-477
> URL: https://issues.apache.org/jira/browse/RATIS-477
> Project: Ratis
> Issue Type: Improvement
> Components: LogService
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 0.4.0
>
>
> [~vrodionov] and [~rajeshbabu] were asking about this in RATIS-470. Can we
> improve the logic that reads through the "head" of the RaftLog to find the
> "LogService offset"?
> Vlad suggested:
> {quote}
> Josh, can we simplify this a little bit? Client gets Raft log ID one every
> append, for multi-get calls we can use this LogID (RAFT) as our start log ID
> to avoid iterating through all logs, then you can unwind log entries by using
> new logic, see above:
> int numRecordsInAppend = append.getDataCount();
> There is no need to iterate from the very beginning
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-477) Improve LogStateMachine.processReadRequest()
[
https://issues.apache.org/jira/browse/RATIS-477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16757485#comment-16757485
]
Josh Elser commented on RATIS-477:
--
{quote}Keeping index in memory is very bad idea. If you will think about this,
Josh.
{quote}
Actually, I don't follow. Why do you think this is a bad idea? This could be as
simple as tracking the recordId mapping from each RaftLog segment on disk,
which we are already doing insisde of LogSegment.
> Improve LogStateMachine.processReadRequest()
>
>
> Key: RATIS-477
> URL: https://issues.apache.org/jira/browse/RATIS-477
> Project: Ratis
> Issue Type: Improvement
> Components: LogService
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 0.4.0
>
>
> [~vrodionov] and [~rajeshbabu] were asking about this in RATIS-470. Can we
> improve the logic that reads through the "head" of the RaftLog to find the
> "LogService offset"?
> Vlad suggested:
> {quote}
> Josh, can we simplify this a little bit? Client gets Raft log ID one every
> append, for multi-get calls we can use this LogID (RAFT) as our start log ID
> to avoid iterating through all logs, then you can unwind log entries by using
> new logic, see above:
> int numRecordsInAppend = append.getDataCount();
> There is no need to iterate from the very beginning
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-477) Improve LogStateMachine.processReadRequest()
[
https://issues.apache.org/jira/browse/RATIS-477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756760#comment-16756760
]
Vladimir Rodionov commented on RATIS-477:
-
Keeping index in memory is very bad idea.
> Improve LogStateMachine.processReadRequest()
>
>
> Key: RATIS-477
> URL: https://issues.apache.org/jira/browse/RATIS-477
> Project: Ratis
> Issue Type: Improvement
> Components: LogService
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 0.4.0
>
>
> [~vrodionov] and [~rajeshbabu] were asking about this in RATIS-470. Can we
> improve the logic that reads through the "head" of the RaftLog to find the
> "LogService offset"?
> Vlad suggested:
> {quote}
> Josh, can we simplify this a little bit? Client gets Raft log ID one every
> append, for multi-get calls we can use this LogID (RAFT) as our start log ID
> to avoid iterating through all logs, then you can unwind log entries by using
> new logic, see above:
> int numRecordsInAppend = append.getDataCount();
> There is no need to iterate from the very beginning
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-477) Improve LogStateMachine.processReadRequest()
[
https://issues.apache.org/jira/browse/RATIS-477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756711#comment-16756711
]
Josh Elser commented on RATIS-477:
--
{quote}how are you going to resolve read(3) call? You will need to iterate
through *all* log entries again. Correct?
{quote}
I don't think I will have to. In the memory of LogStateMachine, I think I can
keep an in-memory representation (an index) over the RaftLog of RaftLog's ID to
the recordId (LogService ID). This will help me avoid the large penalty of not
knowing where to seek. We can construct this memory when we are starting the
state machine (if we have previous state) and we can maintain it for every
operation we apply. At least, I think we can do this.
{quote}Read can return multiple entries if corresponding write was batch write
for this particular *logId*
{quote}
I worry about this approach because it will make the read API less like a
simple List. If you read one entry, sometimes you get one value, sometimes you
get many values. I think we can hide this detail.
> Improve LogStateMachine.processReadRequest()
>
>
> Key: RATIS-477
> URL: https://issues.apache.org/jira/browse/RATIS-477
> Project: Ratis
> Issue Type: Improvement
> Components: LogService
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 0.4.0
>
>
> [~vrodionov] and [~rajeshbabu] were asking about this in RATIS-470. Can we
> improve the logic that reads through the "head" of the RaftLog to find the
> "LogService offset"?
> Vlad suggested:
> {quote}
> Josh, can we simplify this a little bit? Client gets Raft log ID one every
> append, for multi-get calls we can use this LogID (RAFT) as our start log ID
> to avoid iterating through all logs, then you can unwind log entries by using
> new logic, see above:
> int numRecordsInAppend = append.getDataCount();
> There is no need to iterate from the very beginning
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-477) Improve LogStateMachine.processReadRequest()
[
https://issues.apache.org/jira/browse/RATIS-477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756700#comment-16756700
]
Vladimir Rodionov commented on RATIS-477:
-
Josh, how are you going to resolve read(3) call? You will need to iterate
through *all* log entries again. Correct? What I suggest:
1. Change read log API to accommodate possible multiple return log entries.
For example:
{code}
long logId = write(List entries)
Then you can call read:
List entries = read(logId);
{code}
Read can return multiple entries if corresponding write was batch write for
this particular *logId*. I would change read API by keeping only one method:
{code}
long nextId = read(List buffer, int logId)
{code}
The API call returns nextId to read, *buffer* list size can be increased by
implementation if a given size can't accomodate all log entries in this
particular raft log entry (side effect), if there are no log service entries in
this Raft ID entry, then we iterate until we get real log entries.
> Improve LogStateMachine.processReadRequest()
>
>
> Key: RATIS-477
> URL: https://issues.apache.org/jira/browse/RATIS-477
> Project: Ratis
> Issue Type: Improvement
> Components: LogService
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 0.4.0
>
>
> [~vrodionov] and [~rajeshbabu] were asking about this in RATIS-470. Can we
> improve the logic that reads through the "head" of the RaftLog to find the
> "LogService offset"?
> Vlad suggested:
> {quote}
> Josh, can we simplify this a little bit? Client gets Raft log ID one every
> append, for multi-get calls we can use this LogID (RAFT) as our start log ID
> to avoid iterating through all logs, then you can unwind log entries by using
> new logic, see above:
> int numRecordsInAppend = append.getDataCount();
> There is no need to iterate from the very beginning
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-477) Improve LogStateMachine.processReadRequest()
[
https://issues.apache.org/jira/browse/RATIS-477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756683#comment-16756683
]
Josh Elser commented on RATIS-477:
--
{quote}we can use this LogID (RAFT) as our start log ID to avoid iterating
through all logs
{quote}
You're right that we're returning back the RaftLog index via the append(), but
I think that's a bug right now. For example, take this sequence:
# write 'a', returns id=1
# write ['b', 'c', 'd'], returns id=2
If you asked for id=3 (trying to read the value 'c'), we would read off the end
of the RaftLog. I think for all DML operations, we want to use our own index
that is disjoint from the RaftLog index. I think this gives a much easier
client implementation and user experience, and gives us the chance to hide this
complexity server-side.
For example, I bet we could maintain a sorted-map of recordId->RaftLog ID and
then binary-search over it to make the RaftLog accesses fast.
Does this make sense? I'm not sure if my idea for design is clear.
> Improve LogStateMachine.processReadRequest()
>
>
> Key: RATIS-477
> URL: https://issues.apache.org/jira/browse/RATIS-477
> Project: Ratis
> Issue Type: Improvement
> Components: LogService
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 0.4.0
>
>
> [~vrodionov] and [~rajeshbabu] were asking about this in RATIS-470. Can we
> improve the logic that reads through the "head" of the RaftLog to find the
> "LogService offset"?
> Vlad suggested:
> {quote}
> Josh, can we simplify this a little bit? Client gets Raft log ID one every
> append, for multi-get calls we can use this LogID (RAFT) as our start log ID
> to avoid iterating through all logs, then you can unwind log entries by using
> new logic, see above:
> int numRecordsInAppend = append.getDataCount();
> There is no need to iterate from the very beginning
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
