[jira] [Commented] (RATIS-272) Design ideal API
[ https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16607615#comment-16607615 ] Tsz Wo Nicholas Sze commented on RATIS-272: --- +1 patch looks good. > Design ideal API > > > Key: RATIS-272 > URL: https://issues.apache.org/jira/browse/RATIS-272 > Project: Ratis > Issue Type: Sub-task > Components: LogService >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > Fix For: 0.3.0 > > Attachments: RATIS-272.001.patch > > > With influence from Apache DistributedLog, Kafka, and BookKeeper, design an > API that balances the ideal notion of what a distribute log system should > look like, but also considers the needs of HBase to replace a WAL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16599348#comment-16599348
]
Hadoop QA commented on RATIS-272:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 17s{color} | {color:orange} root: The patch generated 178 new + 0 unchanged
- 0 fixed = 178 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 15m 31s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
7s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 35s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | ratis.TestRaftServerLeaderElectionTimeout |
| | ratis.server.simulation.TestRaftWithSimulatedRpc |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/ratis:date2018-08-31
|
| JIRA Issue | RATIS-272 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12937966/RATIS-272.001.patch |
| Optional Tests | asflicense javac javadoc unit xml compile findbugs
checkstyle |
| uname | Linux 2dd9a82ddd25 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-RATIS-Build/yetus-personality.sh
|
| git revision | master / c6c9ddf |
| Default Java | 1.8.0_181 |
| checkstyle |
https://builds.apache.org/job/PreCommit-RATIS-Build/316/artifact/out/diff-checkstyle-root.txt
|
| unit |
https://builds.apache.org/job/PreCommit-RATIS-Build/316/artifact/out/patch-unit-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-RATIS-Build/316/testReport/ |
| modules | C: ratis-logservice . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-RATIS-Build/316/console |
| Powered by | Apache Yetus 0.5.0 http://yetus.apache.org |
This message was automatically generated.
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 0.3.0
>
> At
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16599344#comment-16599344
]
ASF GitHub Bot commented on RATIS-272:
--
Github user liuml07 commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r214489468
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogName.java ---
@@ -0,0 +1,48 @@
+package org.apache.ratis.logservice.api;
+
+import static java.util.Objects.requireNonNull;
+
+import java.util.Objects;
+
+/**
+ * Identifier to uniquely identify a {@link LogStream}.
+ */
+public class LogName {
+ // It's pretty likely that what uniquely defines a LogStream
+ // to change over time. We should account for this by making an
+ // API which can naturally evolve.
+ private final String name;
+
+ private LogName(String name) {
+this.name = requireNonNull(name);
+ }
+
+ // Implementation detail -- we want uses to use the LogName as
identifiable, not to
--- End diff --
I see. This seems not enabled by default. I found a change in other
projects to enable it via `maven-javadoc-plugin` see:
https://github.com/CodeFX-org/demo-javadoc-8-tags/blob/master/pom.xml?ts=4#L110-L133
It's nice to have. No intention to block this PR. :)
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 0.3.0
>
> Attachments: RATIS-272.001.patch
>
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16599203#comment-16599203
]
ASF GitHub Bot commented on RATIS-272:
--
Github user joshelser commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r214457041
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogWriter.java
---
@@ -0,0 +1,59 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+
+/**
+ * Synchronous client interface to write to a LogStream.
+ */
+public interface LogWriter extends AutoCloseable {
+
+ /**
+ * Appends the given data as a record in the LogStream.
+ *
+ * @param data The record to append
+ * @return The recordId for the record just written
+ */
+ long write(ByteBuffer data) throws IOException;
+
+ /**
+ * Appends each entry of data as a new record in the LogStream. If this
method returns
+ * successfully, all records can be considered persisted. Otherwise,
none can be assumed
+ * to have been written.
+ *
+ * @param records Records to append
+ * @return The largest recordId assigned to the records written
+ */
+ long writeMany(List records) throws IOException;
+
+ /**
+ * Guarantees that all previous data appended by {@link
#write(ByteBuffer)} are persisted
+ * and durable in the LogStream.
+ *
+ * @return The recordId prior to which all records are durable
+ */
+ long sync() throws IOException;
--- End diff --
I'm not 100% sure, but, given what I remember from the paper, I don't think
we'd have to take a new snapshot. I would have to re-read.
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[ https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16599202#comment-16599202 ] ASF GitHub Bot commented on RATIS-272: -- Github user joshelser commented on the issue: https://github.com/apache/incubator-ratis/pull/4 > C: Still think that adopting hierarchical naming scheme for log directory is a right thing. Let's keep this in mind as we work on the LogStream administrative/management side. I think some hierarchy would make sense -- the notion of a namespace to separate different LogStreams using the same "storage directory" would be nice (DistributedLog has this concept). Right now we just have a String (inside of `LogName`) to uniquely identify one LogStream from another. This was intentional to let us either change how `LogName`s compute uniqueness, or overlay some "logical" hierarchy using that arbitrary String. > Design ideal API > > > Key: RATIS-272 > URL: https://issues.apache.org/jira/browse/RATIS-272 > Project: Ratis > Issue Type: Sub-task > Components: LogService >Reporter: Josh Elser >Priority: Major > > With influence from Apache DistributedLog, Kafka, and BookKeeper, design an > API that balances the ideal notion of what a distribute log system should > look like, but also considers the needs of HBase to replace a WAL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[ https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16599200#comment-16599200 ] ASF GitHub Bot commented on RATIS-272: -- Github user joshelser commented on the issue: https://github.com/apache/incubator-ratis/pull/4 > Q: May be I missed that, but I did not find truncate log method. The only way to say to Log Service that we do need anymore records with ID <= X. Nope, you're right. The API only lets you archive (or delete) an entire LogStream. I think this flows a bit more naturally with what we're doing inside of HBase already. We _could_ support intra-LogStream truncation and archival, but I think that would complicate things quite a lot. > Design ideal API > > > Key: RATIS-272 > URL: https://issues.apache.org/jira/browse/RATIS-272 > Project: Ratis > Issue Type: Sub-task > Components: LogService >Reporter: Josh Elser >Priority: Major > > With influence from Apache DistributedLog, Kafka, and BookKeeper, design an > API that balances the ideal notion of what a distribute log system should > look like, but also considers the needs of HBase to replace a WAL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16599194#comment-16599194
]
ASF GitHub Bot commented on RATIS-272:
--
Github user joshelser commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r214455091
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogWriter.java
---
@@ -0,0 +1,59 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+
+/**
+ * Synchronous client interface to write to a LogStream.
+ */
+public interface LogWriter extends AutoCloseable {
+
+ /**
+ * Appends the given data as a record in the LogStream.
+ *
+ * @param data The record to append
+ * @return The recordId for the record just written
+ */
+ long write(ByteBuffer data) throws IOException;
+
+ /**
+ * Appends each entry of data as a new record in the LogStream. If this
method returns
+ * successfully, all records can be considered persisted. Otherwise,
none can be assumed
+ * to have been written.
+ *
+ * @param records Records to append
+ * @return The largest recordId assigned to the records written
+ */
+ long writeMany(List records) throws IOException;
--- End diff --
Right, not intrinsically, but I think that's OK. Raft is making sure that
when the method returns successfully, a majority of the nodes has the writes
which were acked.
For any other case, the client would have to re-read the log to see which
were actually applied. Specifically for HBase, I think that's acceptable -- if
the write to the HBase WAL fails, that flows back to the client and the client
needs to just resubmit the update.
I'm open to suggestions on how we could make this API better; my only
hesitation is that I think I know too little to give the problem adequate
consideration right now. I remember reading a blog by Jay Kreps about how hard
exactly once is.
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16599191#comment-16599191
]
ASF GitHub Bot commented on RATIS-272:
--
Github user VladRodionov commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r214454880
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogStreamConfiguration.java
---
@@ -0,0 +1,47 @@
+package org.apache.ratis.logservice.api;
+
+import java.util.Map;
+import java.util.Map.Entry;
+
+/**
+ * An encapsulation of configuration for a LogStream.
+ */
+public interface LogStreamConfiguration {
--- End diff --
OK, got it.
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[ https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16599186#comment-16599186 ] ASF GitHub Bot commented on RATIS-272: -- Github user VladRodionov commented on the issue: https://github.com/apache/incubator-ratis/pull/4 Overall looks good for the first cut. Just one question and one concern: Q: May be I missed that, but I did not find truncate log method. The only way to say to Log Service that we do need anymore records with ID <= X. C: Still think that adopting hierarchical naming scheme for log directory is a right thing. > Design ideal API > > > Key: RATIS-272 > URL: https://issues.apache.org/jira/browse/RATIS-272 > Project: Ratis > Issue Type: Sub-task > Components: LogService >Reporter: Josh Elser >Priority: Major > > With influence from Apache DistributedLog, Kafka, and BookKeeper, design an > API that balances the ideal notion of what a distribute log system should > look like, but also considers the needs of HBase to replace a WAL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16599185#comment-16599185
]
ASF GitHub Bot commented on RATIS-272:
--
Github user joshelser commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r214453779
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogName.java ---
@@ -0,0 +1,48 @@
+package org.apache.ratis.logservice.api;
+
+import static java.util.Objects.requireNonNull;
+
+import java.util.Objects;
+
+/**
+ * Identifier to uniquely identify a {@link LogStream}.
+ */
+public class LogName {
+ // It's pretty likely that what uniquely defines a LogStream
+ // to change over time. We should account for this by making an
+ // API which can naturally evolve.
+ private final String name;
+
+ private LogName(String name) {
+this.name = requireNonNull(name);
+ }
+
+ // Implementation detail -- we want uses to use the LogName as
identifiable, not to
--- End diff --
Hrm, seems like these are proposed but not yet implemented? The
maven-javadoc-plugin warns me against defining them.
```
Note: Custom tags that could override future standard tags: @implSpec,
@implNote. To avoid potential overrides, use at least one period character (.)
in custom tag names.
```
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16599182#comment-16599182
]
ASF GitHub Bot commented on RATIS-272:
--
Github user joshelser commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r214453065
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogStreamConfiguration.java
---
@@ -0,0 +1,47 @@
+package org.apache.ratis.logservice.api;
+
+import java.util.Map;
+import java.util.Map.Entry;
+
+/**
+ * An encapsulation of configuration for a LogStream.
+ */
+public interface LogStreamConfiguration {
--- End diff --
I definitely don't want to reuse Hadoop's Configuration because
hadoop-common pulls in tons of crap we don't want or need just to reuse
Configuration. Ratis does have the notion of configuration, but I see this
object sitting on top of that. The RaftClient has configuration controls how
that would be set up, but the LogService would have more on top of that.
My goal was to just keep this as minimal as possible and only using
JRE-provided classes until we have a better idea of how we'll use the
configuration.
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16599180#comment-16599180
]
ASF GitHub Bot commented on RATIS-272:
--
Github user ss77892 commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r214452753
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogWriter.java
---
@@ -0,0 +1,59 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+
+/**
+ * Synchronous client interface to write to a LogStream.
+ */
+public interface LogWriter extends AutoCloseable {
+
+ /**
+ * Appends the given data as a record in the LogStream.
+ *
+ * @param data The record to append
+ * @return The recordId for the record just written
+ */
+ long write(ByteBuffer data) throws IOException;
+
+ /**
+ * Appends each entry of data as a new record in the LogStream. If this
method returns
+ * successfully, all records can be considered persisted. Otherwise,
none can be assumed
+ * to have been written.
+ *
+ * @param records Records to append
+ * @return The largest recordId assigned to the records written
+ */
+ long writeMany(List records) throws IOException;
--- End diff --
that means that we don't support 'exactly once delivery'?
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16599181#comment-16599181
]
ASF GitHub Bot commented on RATIS-272:
--
Github user VladRodionov commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r214452765
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogWriter.java
---
@@ -0,0 +1,59 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+
+/**
+ * Synchronous client interface to write to a LogStream.
+ */
+public interface LogWriter extends AutoCloseable {
+
+ /**
+ * Appends the given data as a record in the LogStream.
+ *
+ * @param data The record to append
+ * @return The recordId for the record just written
+ */
+ long write(ByteBuffer data) throws IOException;
+
+ /**
+ * Appends each entry of data as a new record in the LogStream. If this
method returns
+ * successfully, all records can be considered persisted. Otherwise,
none can be assumed
+ * to have been written.
+ *
+ * @param records Records to append
+ * @return The largest recordId assigned to the records written
+ */
+ long writeMany(List records) throws IOException;
+
+ /**
+ * Guarantees that all previous data appended by {@link
#write(ByteBuffer)} are persisted
+ * and durable in the LogStream.
+ *
+ * @return The recordId prior to which all records are durable
+ */
+ long sync() throws IOException;
--- End diff --
Q: sync will perform snapshot?
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16599179#comment-16599179
]
ASF GitHub Bot commented on RATIS-272:
--
Github user VladRodionov commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r214452653
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogWriter.java
---
@@ -0,0 +1,59 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+
+/**
+ * Synchronous client interface to write to a LogStream.
+ */
+public interface LogWriter extends AutoCloseable {
+
+ /**
+ * Appends the given data as a record in the LogStream.
+ *
+ * @param data The record to append
+ * @return The recordId for the record just written
+ */
+ long write(ByteBuffer data) throws IOException;
+
+ /**
+ * Appends each entry of data as a new record in the LogStream. If this
method returns
+ * successfully, all records can be considered persisted. Otherwise,
none can be assumed
+ * to have been written.
+ *
+ * @param records Records to append
+ * @return The largest recordId assigned to the records written
+ */
+ long writeMany(List records) throws IOException;
--- End diff --
Just naming convention, please drop Many.
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16599178#comment-16599178
]
ASF GitHub Bot commented on RATIS-272:
--
Github user VladRodionov commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r214452525
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogStreamConfiguration.java
---
@@ -0,0 +1,47 @@
+package org.apache.ratis.logservice.api;
+
+import java.util.Map;
+import java.util.Map.Entry;
+
+/**
+ * An encapsulation of configuration for a LogStream.
+ */
+public interface LogStreamConfiguration {
--- End diff --
Can we reuse Hadoop Configuration? Or Ratis configuration (if they have
one)?
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16599173#comment-16599173
]
ASF GitHub Bot commented on RATIS-272:
--
Github user joshelser commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r214450694
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogWriter.java
---
@@ -0,0 +1,59 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+
+/**
+ * Synchronous client interface to write to a LogStream.
+ */
+public interface LogWriter extends AutoCloseable {
+
+ /**
+ * Appends the given data as a record in the LogStream.
+ *
+ * @param data The record to append
+ * @return The recordId for the record just written
+ */
+ long write(ByteBuffer data) throws IOException;
+
+ /**
+ * Appends each entry of data as a new record in the LogStream. If this
method returns
+ * successfully, all records can be considered persisted. Otherwise,
none can be assumed
+ * to have been written.
+ *
+ * @param records Records to append
+ * @return The largest recordId assigned to the records written
+ */
+ long writeMany(List records) throws IOException;
--- End diff --
Exactly.
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16599171#comment-16599171
]
ASF GitHub Bot commented on RATIS-272:
--
Github user liuml07 commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r214449799
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogWriter.java
---
@@ -0,0 +1,59 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+
+/**
+ * Synchronous client interface to write to a LogStream.
+ */
+public interface LogWriter extends AutoCloseable {
+
+ /**
+ * Appends the given data as a record in the LogStream.
+ *
+ * @param data The record to append
+ * @return The recordId for the record just written
+ */
+ long write(ByteBuffer data) throws IOException;
+
+ /**
+ * Appends each entry of data as a new record in the LogStream. If this
method returns
+ * successfully, all records can be considered persisted. Otherwise,
none can be assumed
+ * to have been written.
+ *
+ * @param records Records to append
+ * @return The largest recordId assigned to the records written
+ */
+ long writeMany(List records) throws IOException;
--- End diff --
Yes. I assume we don't provide any transaction support, i.e. if one record
writing fails, previous written record in the batch will have no guarantee
(succeeded or reverted).
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16599168#comment-16599168
]
ASF GitHub Bot commented on RATIS-272:
--
Github user joshelser commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r214448422
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogName.java ---
@@ -0,0 +1,48 @@
+package org.apache.ratis.logservice.api;
+
+import static java.util.Objects.requireNonNull;
+
+import java.util.Objects;
+
+/**
+ * Identifier to uniquely identify a {@link LogStream}.
+ */
+public class LogName {
+ // It's pretty likely that what uniquely defines a LogStream
+ // to change over time. We should account for this by making an
+ // API which can naturally evolve.
+ private final String name;
+
+ private LogName(String name) {
+this.name = requireNonNull(name);
+ }
+
+ // Implementation detail -- we want uses to use the LogName as
identifiable, not to
--- End diff --
Nice! Didn't know about these. Happy to add that.
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16599167#comment-16599167
]
ASF GitHub Bot commented on RATIS-272:
--
Github user joshelser commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r214447775
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogWriter.java
---
@@ -0,0 +1,59 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+
+/**
+ * Synchronous client interface to write to a LogStream.
+ */
+public interface LogWriter extends AutoCloseable {
+
+ /**
+ * Appends the given data as a record in the LogStream.
+ *
+ * @param data The record to append
+ * @return The recordId for the record just written
+ */
+ long write(ByteBuffer data) throws IOException;
+
+ /**
+ * Appends each entry of data as a new record in the LogStream. If this
method returns
+ * successfully, all records can be considered persisted. Otherwise,
none can be assumed
+ * to have been written.
+ *
+ * @param records Records to append
+ * @return The largest recordId assigned to the records written
+ */
+ long writeMany(List records) throws IOException;
--- End diff --
Just calling doing a `for (ByteBuffer record : records) write(record);`,
you mean? Could do that.
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[ https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16599166#comment-16599166 ] ASF GitHub Bot commented on RATIS-272: -- Github user joshelser commented on a diff in the pull request: https://github.com/apache/incubator-ratis/pull/4#discussion_r214447536 --- Diff: ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogStreamConfiguration.java --- @@ -0,0 +1,47 @@ +package org.apache.ratis.logservice.api; --- End diff -- oops! :D > Design ideal API > > > Key: RATIS-272 > URL: https://issues.apache.org/jira/browse/RATIS-272 > Project: Ratis > Issue Type: Sub-task > Components: LogService >Reporter: Josh Elser >Priority: Major > > With influence from Apache DistributedLog, Kafka, and BookKeeper, design an > API that balances the ideal notion of what a distribute log system should > look like, but also considers the needs of HBase to replace a WAL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16599164#comment-16599164
]
ASF GitHub Bot commented on RATIS-272:
--
Github user joshelser commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r214447351
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/RecordListener.java
---
@@ -0,0 +1,18 @@
+package org.apache.ratis.logservice.api;
+
+import java.nio.ByteBuffer;
+
+/**
+ * Interface that, when registered with a {@link LogStream}, will receive
all records written
+ * to that LogStream until it is removed.
+ */
+public interface RecordListener {
--- End diff --
Good point. Given what we know now, I don't think we have more complex
requirements. Switching this over to a consumer is a good idea, I think.
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16599122#comment-16599122
]
ASF GitHub Bot commented on RATIS-272:
--
Github user liuml07 commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r214436589
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/RecordListener.java
---
@@ -0,0 +1,18 @@
+package org.apache.ratis.logservice.api;
+
+import java.nio.ByteBuffer;
+
+/**
+ * Interface that, when registered with a {@link LogStream}, will receive
all records written
+ * to that LogStream until it is removed.
+ */
+public interface RecordListener {
--- End diff --
Will more methods to be added? If not, is a simple `Consumer` sufficient as
listeners (consumers)?
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[ https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16599123#comment-16599123 ] ASF GitHub Bot commented on RATIS-272: -- Github user liuml07 commented on a diff in the pull request: https://github.com/apache/incubator-ratis/pull/4#discussion_r214439383 --- Diff: ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogName.java --- @@ -0,0 +1,48 @@ +package org.apache.ratis.logservice.api; --- End diff -- Apache License > Design ideal API > > > Key: RATIS-272 > URL: https://issues.apache.org/jira/browse/RATIS-272 > Project: Ratis > Issue Type: Sub-task > Components: LogService >Reporter: Josh Elser >Priority: Major > > With influence from Apache DistributedLog, Kafka, and BookKeeper, design an > API that balances the ideal notion of what a distribute log system should > look like, but also considers the needs of HBase to replace a WAL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16599125#comment-16599125
]
ASF GitHub Bot commented on RATIS-272:
--
Github user liuml07 commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r214439890
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogName.java ---
@@ -0,0 +1,48 @@
+package org.apache.ratis.logservice.api;
+
+import static java.util.Objects.requireNonNull;
+
+import java.util.Objects;
+
+/**
+ * Identifier to uniquely identify a {@link LogStream}.
+ */
+public class LogName {
+ // It's pretty likely that what uniquely defines a LogStream
+ // to change over time. We should account for this by making an
+ // API which can naturally evolve.
+ private final String name;
+
+ private LogName(String name) {
+this.name = requireNonNull(name);
+ }
+
+ // Implementation detail -- we want uses to use the LogName as
identifiable, not to
--- End diff --
s/Implementation detail/`@implSpec`?
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[ https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16599124#comment-16599124 ] ASF GitHub Bot commented on RATIS-272: -- Github user liuml07 commented on a diff in the pull request: https://github.com/apache/incubator-ratis/pull/4#discussion_r214439207 --- Diff: ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogStreamConfiguration.java --- @@ -0,0 +1,47 @@ +package org.apache.ratis.logservice.api; --- End diff -- Apache License > Design ideal API > > > Key: RATIS-272 > URL: https://issues.apache.org/jira/browse/RATIS-272 > Project: Ratis > Issue Type: Sub-task > Components: LogService >Reporter: Josh Elser >Priority: Major > > With influence from Apache DistributedLog, Kafka, and BookKeeper, design an > API that balances the ideal notion of what a distribute log system should > look like, but also considers the needs of HBase to replace a WAL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[ https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16599126#comment-16599126 ] ASF GitHub Bot commented on RATIS-272: -- Github user liuml07 commented on a diff in the pull request: https://github.com/apache/incubator-ratis/pull/4#discussion_r214439467 --- Diff: ratis-logservice/src/main/java/org/apache/ratis/logservice/LogServiceFactory.java --- @@ -0,0 +1,27 @@ +package org.apache.ratis.logservice; --- End diff -- Apache License > Design ideal API > > > Key: RATIS-272 > URL: https://issues.apache.org/jira/browse/RATIS-272 > Project: Ratis > Issue Type: Sub-task > Components: LogService >Reporter: Josh Elser >Priority: Major > > With influence from Apache DistributedLog, Kafka, and BookKeeper, design an > API that balances the ideal notion of what a distribute log system should > look like, but also considers the needs of HBase to replace a WAL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16599121#comment-16599121
]
ASF GitHub Bot commented on RATIS-272:
--
Github user liuml07 commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r214439058
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogWriter.java
---
@@ -0,0 +1,59 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+
+/**
+ * Synchronous client interface to write to a LogStream.
+ */
+public interface LogWriter extends AutoCloseable {
+
+ /**
+ * Appends the given data as a record in the LogStream.
+ *
+ * @param data The record to append
+ * @return The recordId for the record just written
+ */
+ long write(ByteBuffer data) throws IOException;
+
+ /**
+ * Appends each entry of data as a new record in the LogStream. If this
method returns
+ * successfully, all records can be considered persisted. Otherwise,
none can be assumed
+ * to have been written.
+ *
+ * @param records Records to append
+ * @return The largest recordId assigned to the records written
+ */
+ long writeMany(List records) throws IOException;
--- End diff --
Is default implementation (via for-write) possible/necessary?
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[ https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16598745#comment-16598745 ] ASF GitHub Bot commented on RATIS-272: -- Github user joshelser commented on the issue: https://github.com/apache/incubator-ratis/pull/4 I think this is close. @mukul1987 @szetszwo @anuengineer any input from y'all? Maybe a merge if you think this is OK :) > Design ideal API > > > Key: RATIS-272 > URL: https://issues.apache.org/jira/browse/RATIS-272 > Project: Ratis > Issue Type: Sub-task > Components: LogService >Reporter: Josh Elser >Priority: Major > > With influence from Apache DistributedLog, Kafka, and BookKeeper, design an > API that balances the ideal notion of what a distribute log system should > look like, but also considers the needs of HBase to replace a WAL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16597889#comment-16597889
]
ASF GitHub Bot commented on RATIS-272:
--
Github user joshelser commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r214166342
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogReader.java
---
@@ -0,0 +1,53 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+
+/**
+ * Synchronous client interface to read from a LogStream.
+ */
+public interface LogReader extends AutoCloseable {
+
+ /**
+ * Seeks to the position before the record at the provided {@code
offset} in the LogStream.
+ *
+ * @param offset A non-negative, offset in the LogStream
+ * @return A future for when the operation is completed.
+ */
+ void seek(long recordId) throws IOException;
--- End diff --
Ratis has a directory in which is stores many LogSegments. Each of these
are ordered with an Index, per the Raft protocol. Ratis can find the LogSegment
containing the record at an Index using binary search and then does a linear
search over that LogSegment to find the specific ID. I think that's sufficient
for what we need in a first-pass.
I don't think we need to support a "recordId" that is different than the
Raft Index. Just trying to give it a better name.
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16597856#comment-16597856
]
ASF GitHub Bot commented on RATIS-272:
--
Github user ss77892 commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r214160302
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogService.java
---
@@ -0,0 +1,53 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.util.Iterator;
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * Entry point for interacting with the Ratis LogService.
+ */
+public interface LogService {
+
+ /**
+ * Creates a new {@link LogStream} identified by the given name. Throws
+ * an exception if a {@link LogStream} with the given name already
exists.
+ *
+ * @param name Unique name for this LogStream.
+ */
+ CompletableFuture createLog(LogName name);
+
+ /**
+ * Fetches the {@link LogStream} identified by the given name.
+ *
+ * @param name The name of the LogStream
+ */
+ CompletableFuture getLog(LogName name);
+
+ /**
+ * Lists all {@link LogStream} instances known by this LogService.
+ */
+ CompletableFuture> listLogs();
--- End diff --
@VladRodionov I completely agree with you that it's possible to implement
more than one log using a single state machine. I was talking about the current
Ratis implementation which has a single log per state machine. The benefit we
have is that the replication of this log is a part of the implementation and we
shouldn't implement anything on our own. If I understand correctly you suggest
implementing multiple logs separately. That would mean 1) double writes (still
all that data will go to RatisLog) 2) Special handling for those records to
write it to the correct log. From my point of view, the performance of the
write path is more critical for us compared to the read path.
Just in case there is a concept of 'multi-raft' already implemented by a
number of projects. We may get it implemented on Ratis level, but I'm not sure
how it's suitable in the embedded library concept and how much efforts it would
require.
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[ https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16597853#comment-16597853 ] ASF GitHub Bot commented on RATIS-272: -- Github user joshelser commented on the issue: https://github.com/apache/incubator-ratis/pull/4 Rebased on master, updated javadoc, added a factory for LogService creation, and fixed a couple of other trivial cleanup things. Good for a first draft, @VladRodionov @chrajeshbabu ? > Design ideal API > > > Key: RATIS-272 > URL: https://issues.apache.org/jira/browse/RATIS-272 > Project: Ratis > Issue Type: Sub-task > Components: LogService >Reporter: Josh Elser >Priority: Major > > With influence from Apache DistributedLog, Kafka, and BookKeeper, design an > API that balances the ideal notion of what a distribute log system should > look like, but also considers the needs of HBase to replace a WAL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16597813#comment-16597813
]
ASF GitHub Bot commented on RATIS-272:
--
Github user VladRodionov commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r214150795
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogService.java
---
@@ -0,0 +1,53 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.util.Iterator;
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * Entry point for interacting with the Ratis LogService.
+ */
+public interface LogService {
+
+ /**
+ * Creates a new {@link LogStream} identified by the given name. Throws
+ * an exception if a {@link LogStream} with the given name already
exists.
+ *
+ * @param name Unique name for this LogStream.
+ */
+ CompletableFuture createLog(LogName name);
+
+ /**
+ * Fetches the {@link LogStream} identified by the given name.
+ *
+ * @param name The name of the LogStream
+ */
+ CompletableFuture getLog(LogName name);
+
+ /**
+ * Lists all {@link LogStream} instances known by this LogService.
+ */
+ CompletableFuture> listLogs();
--- End diff --
Sergej, state machine can track more than one log. I have mentioned already
very simple design of a STATE: it is a key-value map, where keys are logs
(fully qualified names), and values are current recordId or offset. In this
case, we will have to separate actual logs from a Raft log, otherwise all the
data will be in a single Raft log, which makes everything more complex during
RS failover. Raft protocol and implementation is not a Bible, we can add our
own bits and pieces which suite our goals.
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16597811#comment-16597811
]
ASF GitHub Bot commented on RATIS-272:
--
Github user joshelser commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r214150276
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogReader.java
---
@@ -0,0 +1,82 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+
+/**
+ * Synchronous client interface to read from a LogStream.
+ */
+public interface LogReader extends AutoCloseable {
+
+ /**
+ * Seeks to the position before the record at the provided {@code
recordId} in the LogStream.
+ *
+ * @param offset A non-negative, offset in the LogStream
+ * @return A future for when the operation is completed.
+ */
+ void seek(long recordId) throws IOException;
+
+ /**
+ * Reads the next record from the LogStream at the current position and
advances the current position
+ * to after the record which was just returned.
+ *
+ * @return The data for the next record.
+ */
+ ByteBuffer readNext() throws IOException;
+
+ /**
+ * Reads the next record from the LogStream at the current position into
the provided {@link buffer} and
+ * advances the current position to the point after the record just read.
+ *
+ * @param buffer A buffer to read the record into
+ */
+ void readNext(ByteBuffer buffer) throws IOException;
--- End diff --
Ok, sounds good. Let me update the javadoc at least to call this out.
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16597799#comment-16597799
]
ASF GitHub Bot commented on RATIS-272:
--
Github user VladRodionov commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r214148013
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogReader.java
---
@@ -0,0 +1,82 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+
+/**
+ * Synchronous client interface to read from a LogStream.
+ */
+public interface LogReader extends AutoCloseable {
+
+ /**
+ * Seeks to the position before the record at the provided {@code
recordId} in the LogStream.
+ *
+ * @param offset A non-negative, offset in the LogStream
+ * @return A future for when the operation is completed.
+ */
+ void seek(long recordId) throws IOException;
+
+ /**
+ * Reads the next record from the LogStream at the current position and
advances the current position
+ * to after the record which was just returned.
+ *
+ * @return The data for the next record.
+ */
+ ByteBuffer readNext() throws IOException;
+
+ /**
+ * Reads the next record from the LogStream at the current position into
the provided {@link buffer} and
+ * advances the current position to the point after the record just read.
+ *
+ * @param buffer A buffer to read the record into
+ */
+ void readNext(ByteBuffer buffer) throws IOException;
--- End diff --
No problem, exception should be thrown. It is up to user to decide what is
the minimum safe buffer size is.
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16597795#comment-16597795
]
ASF GitHub Bot commented on RATIS-272:
--
Github user VladRodionov commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r214147512
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogReader.java
---
@@ -0,0 +1,53 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+
+/**
+ * Synchronous client interface to read from a LogStream.
+ */
+public interface LogReader extends AutoCloseable {
+
+ /**
+ * Seeks to the position before the record at the provided {@code
offset} in the LogStream.
+ *
+ * @param offset A non-negative, offset in the LogStream
+ * @return A future for when the operation is completed.
+ */
+ void seek(long recordId) throws IOException;
--- End diff --
>> Moving to a specific point in a LogStream is absolutely a necessary API
call
I think everyone here agree with you. The issue is keeping separate indexes
for records. I do not know how this was implemented in Ratis, but in any case,
maintaining index eats CPU and IO and slow everything down. If it is there
already - fine, we can reuse it. To resume HBase replication - offset in a log
stream (not record id) would suffice (in our use case, of course).
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16597743#comment-16597743
]
ASF GitHub Bot commented on RATIS-272:
--
Github user joshelser commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r214132445
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogReader.java
---
@@ -0,0 +1,82 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+
+/**
+ * Synchronous client interface to read from a LogStream.
+ */
+public interface LogReader extends AutoCloseable {
+
+ /**
+ * Seeks to the position before the record at the provided {@code
recordId} in the LogStream.
+ *
+ * @param offset A non-negative, offset in the LogStream
+ * @return A future for when the operation is completed.
+ */
+ void seek(long recordId) throws IOException;
+
+ /**
+ * Reads the next record from the LogStream at the current position and
advances the current position
+ * to after the record which was just returned.
+ *
+ * @return The data for the next record.
+ */
+ ByteBuffer readNext() throws IOException;
+
+ /**
+ * Reads the next record from the LogStream at the current position into
the provided {@link buffer} and
+ * advances the current position to the point after the record just read.
+ *
+ * @param buffer A buffer to read the record into
+ */
+ void readNext(ByteBuffer buffer) throws IOException;
--- End diff --
@VladRodionov one concern I just realized is: what if the buffer's size is
less than the size of the record we need to read? As written now, we would have
to throw an Exception. I don't know if we have flexibility within ByteBuffer to
fix this.
One way around it would be to build in some kind of ByteBufferPool to Ratis
itself that would reuse buffers and reduce the work a client would have to do.
WDYT?
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[ https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16596805#comment-16596805 ] ASF GitHub Bot commented on RATIS-272: -- Github user joshelser commented on the issue: https://github.com/apache/incubator-ratis/pull/4 Dropped a big update that tries to account for the implementation decisions. > Design ideal API > > > Key: RATIS-272 > URL: https://issues.apache.org/jira/browse/RATIS-272 > Project: Ratis > Issue Type: Sub-task > Components: LogService >Reporter: Josh Elser >Priority: Major > > With influence from Apache DistributedLog, Kafka, and BookKeeper, design an > API that balances the ideal notion of what a distribute log system should > look like, but also considers the needs of HBase to replace a WAL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16596735#comment-16596735
]
ASF GitHub Bot commented on RATIS-272:
--
Github user joshelser commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r213797238
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/RecordListener.java
---
@@ -0,0 +1,19 @@
+package org.apache.ratis.logservice.api;
+
+import java.nio.ByteBuffer;
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * Interface that, when registered with a {@link LogStream}, will receive
all records written
+ * to that LogStream until it is removed.
+ */
+public interface RecordListener {
+
+ /**
+ * Processes the written record from the LogStream.
+ *
+ * @param record The record
+ */
+ CompletableFuture receiveRecord(ByteBuffer record);
--- End diff --
Dropping this with the other async stuff.
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16596737#comment-16596737
]
ASF GitHub Bot commented on RATIS-272:
--
Github user joshelser commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r213797305
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogStream.java
---
@@ -0,0 +1,109 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * A distributed log with "infinite" length that supports reads and writes.
+ */
+public interface LogStream {
+
+ /**
+ * Returns the unique name to identify this log.
+ */
+ LogName getName();
+
+ /**
+ * Returns the size of this LogStream in bytes.
+ */
+ long getSizeInBytes();
+
+ /**
+ * Returns the number of records in this LogStream.
+ */
+ long getSizeInRecords();
+
+ /**
+ * Creates a reader to read this LogStream asynchronously.
+ *
+ * @return An asynchronous reader
+ */
+ AsyncLogReader createAsyncReader();
+
+ /**
+ * Creates a writer to write to this LogStream asynchronously.
+ *
+ * @return An asynchronous writer
+ */
+ AsyncLogWriter createAsyncWriter();
+
+ /**
+ * Creates a reader to read this LogStream synchronously.
+ *
+ * @return A synchronous reader
+ */
+ LogReader createReader();
+
+ /**
+ * Creates a write to write to this LogStream synchronously.
+ *
+ * @return A synchronous writer
+ */
+ LogWriter createWriter();
+
+ /**
+ * Removes the elements in this LogStream prior to the given recordId.
+ *
+ * @param recordId A non-negative recordId for this LogStream
+ */
+ CompletableFuture truncateBefore(long recordId);
+
+ /**
+ * Returns the recordId which is the start of the LogStream. When there
are records which were truncated
+ * from the LogStream, this will return a value larger than {@code 0}.
+ */
+ CompletableFuture getFirstRecordId();
+
+ /**
+ * Copies all records from the beginning of the LogStream until the
given {@code recordId}
+ * to the configured archival storage.
+ *
+ * @param recordId A non-negative recordId for this LogStream
+ */
+ CompletableFuture archiveBefore(long recordId);
+
+ /**
+ * Returns the recordId, prior to which, all records in the LogStream
are archived.
+ */
+ CompletableFuture getArchivalPoint();
--- End diff --
Yeah, need to rework this based on the doc I updated on RATIS-271 -- thanks!
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16596734#comment-16596734
]
ASF GitHub Bot commented on RATIS-272:
--
Github user joshelser commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r213797086
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogReader.java
---
@@ -0,0 +1,53 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+
+/**
+ * Synchronous client interface to read from a LogStream.
+ */
+public interface LogReader extends AutoCloseable {
+
+ /**
+ * Seeks to the position before the record at the provided {@code
offset} in the LogStream.
+ *
+ * @param offset A non-negative, offset in the LogStream
+ * @return A future for when the operation is completed.
+ */
+ void seek(long recordId) throws IOException;
--- End diff --
Moving to a specific point in a LogStream is absolutely a necessary API
call. We must be able to resume reading from a known point in the Log. HBase
Replication needs that functionality.
A naive implementation can read all records from the LogStream to know
where to start reading. A slightly more advanced implementation can use the
underlying segment files to find the right segment file to read from. An
advance implementation could consult an index which we explicitly maintain over
all segment files.
We haven't touched on implementation of how the LogReader class would
negotiate this call with the back-end StateMachine (that depends on what the
StateMachine and/or Ratis APIs look like, I think).
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16596726#comment-16596726
]
ASF GitHub Bot commented on RATIS-272:
--
Github user joshelser commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r213795159
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/ArchivedLog.java
---
@@ -0,0 +1,43 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+/**
+ * A {@link LogStream} which has been archived in some external
+ * system. This interface is parameterized to allow for implementations
+ * to use their own class to encapsulate how to find the archived log.
+ *
+ * In the majority of cases, this should be transparent to end-users, as
+ * the {@link LogStream} should hide the fact that this even exists.
+ * TODO maybe that means this should be client-facing at all?
+ *
+ * @param A referent to the log on the external system.
+ */
+public interface ArchivedLog extends AutoCloseable {
+
+ /**
+ * Creates an asynchronous reader over this archived log.
+ */
+ AsyncLogReader createAsyncReader();
--- End diff --
Dropping all of the async API for now. Might help later but we should do
that with purpose.
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590818#comment-16590818
]
ASF GitHub Bot commented on RATIS-272:
--
Github user joshelser commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r212458341
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogStream.java
---
@@ -0,0 +1,109 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * A distributed log with "infinite" length that supports reads and writes.
+ */
+public interface LogStream {
+
+ /**
+ * Returns the unique name to identify this log.
+ */
+ LogName getName();
+
+ /**
+ * Returns the size of this LogStream in bytes.
+ */
+ long getSizeInBytes();
+
+ /**
+ * Returns the number of records in this LogStream.
+ */
+ long getSizeInRecords();
+
+ /**
+ * Creates a reader to read this LogStream asynchronously.
+ *
+ * @return An asynchronous reader
+ */
+ AsyncLogReader createAsyncReader();
+
+ /**
+ * Creates a writer to write to this LogStream asynchronously.
+ *
+ * @return An asynchronous writer
+ */
+ AsyncLogWriter createAsyncWriter();
+
+ /**
+ * Creates a reader to read this LogStream synchronously.
+ *
+ * @return A synchronous reader
+ */
+ LogReader createReader();
+
+ /**
+ * Creates a write to write to this LogStream synchronously.
+ *
+ * @return A synchronous writer
+ */
+ LogWriter createWriter();
+
+ /**
+ * Removes the elements in this LogStream prior to the given recordId.
+ *
+ * @param recordId A non-negative recordId for this LogStream
+ */
+ CompletableFuture truncateBefore(long recordId);
--- End diff --
A LogStream is a sequence of entries: and RecordId is
monotonically increasing. Truncation is the removal of entries from the "head"
of the stream.
e.g. if we have a log with recordIds from `[0,100]`, we might say
`truncateBefore(10, false)` which would remove entries 0, 1, 2, ..., 9. The
"head" of the log would then be `recordId=10`
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16590538#comment-16590538
]
ASF GitHub Bot commented on RATIS-272:
--
Github user VladRodionov commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r212385674
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogReader.java
---
@@ -0,0 +1,53 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+
+/**
+ * Synchronous client interface to read from a LogStream.
+ */
+public interface LogReader extends AutoCloseable {
+
+ /**
+ * Seeks to the position before the record at the provided {@code
offset} in the LogStream.
+ *
+ * @param offset A non-negative, offset in the LogStream
+ * @return A future for when the operation is completed.
+ */
+ void seek(long recordId) throws IOException;
--- End diff --
I meant what I meant: logs are not a message queues. Period. There is no
need for random access in log file to grab a particular log record, but there
is such a need in message queue. As for seeking support in Raft log - there is
no such requirement to implement this efficiently, quite contrary, afair,
protocol describes very stupid and lazy algo of synching two logs: leader goes
backward one record at a time and compares it with a lagging follower records.
Do not bring please functionality we (Log service users) are not asking
for.
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589806#comment-16589806
]
ASF GitHub Bot commented on RATIS-272:
--
Github user ss77892 commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r212202468
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogService.java
---
@@ -0,0 +1,53 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.util.Iterator;
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * Entry point for interacting with the Ratis LogService.
+ */
+public interface LogService {
+
+ /**
+ * Creates a new {@link LogStream} identified by the given name. Throws
+ * an exception if a {@link LogStream} with the given name already
exists.
+ *
+ * @param name Unique name for this LogStream.
+ */
+ CompletableFuture createLog(LogName name);
+
+ /**
+ * Fetches the {@link LogStream} identified by the given name.
+ *
+ * @param name The name of the LogStream
+ */
+ CompletableFuture getLog(LogName name);
+
+ /**
+ * Lists all {@link LogStream} instances known by this LogService.
+ */
+ CompletableFuture> listLogs();
--- End diff --
I would better ask whether this method should be here. Ratis is an embedded
library that allows creating a SINGLE distributed state machine. That means a
SINGLE log. A distributed log service that would allow creating a number of
logs (i.e. state machines) sounds like an application that would include state
machines creation/management. That still can be a part of the embedded library,
but it would make everything more complicated and less flexible (i.e. different
use cases may require special rules for quorum members like rack awareness,
latency measurement, RS Groups in case of HBase, etc).
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589792#comment-16589792
]
ASF GitHub Bot commented on RATIS-272:
--
Github user ss77892 commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r212199898
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogReader.java
---
@@ -0,0 +1,53 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+
+/**
+ * Synchronous client interface to read from a LogStream.
+ */
+public interface LogReader extends AutoCloseable {
+
+ /**
+ * Seeks to the position before the record at the provided {@code
offset} in the LogStream.
+ *
+ * @param offset A non-negative, offset in the LogStream
+ * @return A future for when the operation is completed.
+ */
+ void seek(long recordId) throws IOException;
--- End diff --
One more thing. Actually, such seek operation should be already implemented
under the hood because it's required for quick sync up for lagging quorum nodes
due network problems/restarts ( didn't dig into the code yet, but almost 100%
sure why segments were implemented there). All we need is to provide a facade
that can be used outside.
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589774#comment-16589774
]
ASF GitHub Bot commented on RATIS-272:
--
Github user ss77892 commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r212197492
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogReader.java
---
@@ -0,0 +1,53 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+
+/**
+ * Synchronous client interface to read from a LogStream.
+ */
+public interface LogReader extends AutoCloseable {
+
+ /**
+ * Seeks to the position before the record at the provided {@code
offset} in the LogStream.
+ *
+ * @param offset A non-negative, offset in the LogStream
+ * @return A future for when the operation is completed.
+ */
+ void seek(long recordId) throws IOException;
--- End diff --
That's why most of the implementations of distributed logs (pub/sub model)
actually are using segments/slices/whatever they call it and why usually they
are relatively small. Usually, there is a separate index that maintains the map
between id and segment OR it keeps start id for the segment in its name. What
do you mean by 'separate logs from message queues'?
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589535#comment-16589535
]
ASF GitHub Bot commented on RATIS-272:
--
Github user VladRodionov commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r212155946
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogService.java
---
@@ -0,0 +1,53 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.util.Iterator;
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * Entry point for interacting with the Ratis LogService.
+ */
+public interface LogService {
+
+ /**
+ * Creates a new {@link LogStream} identified by the given name. Throws
+ * an exception if a {@link LogStream} with the given name already
exists.
+ *
+ * @param name Unique name for this LogStream.
+ */
+ CompletableFuture createLog(LogName name);
+
+ /**
+ * Fetches the {@link LogStream} identified by the given name.
+ *
+ * @param name The name of the LogStream
+ */
+ CompletableFuture getLog(LogName name);
+
+ /**
+ * Lists all {@link LogStream} instances known by this LogService.
+ */
+ CompletableFuture> listLogs();
--- End diff --
Josh, how are you going to implement this w/o FileSystem-like API?
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589527#comment-16589527
]
ASF GitHub Bot commented on RATIS-272:
--
Github user VladRodionov commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r212154701
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/RecordListener.java
---
@@ -0,0 +1,19 @@
+package org.apache.ratis.logservice.api;
+
+import java.nio.ByteBuffer;
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * Interface that, when registered with a {@link LogStream}, will receive
all records written
+ * to that LogStream until it is removed.
+ */
+public interface RecordListener {
+
+ /**
+ * Processes the written record from the LogStream.
+ *
+ * @param record The record
+ */
+ CompletableFuture receiveRecord(ByteBuffer record);
--- End diff --
Again, do not break Listener contract - its synchronous.
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589524#comment-16589524
]
ASF GitHub Bot commented on RATIS-272:
--
Github user VladRodionov commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r212154511
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogWriter.java
---
@@ -0,0 +1,43 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+
+/**
+ * Synchronous client interface to write to a LogStream.
+ */
+public interface LogWriter extends AutoCloseable {
+
+ /**
+ * Appends the given data as a record in the LogStream.
+ *
+ * @param data The record to append
+ * @return The recordId for the record just written
+ */
+ long write(ByteBuffer data) throws IOException;
+
+ /**
+ * Guarantees that all previous data appended by {@link
#write(ByteBuffer)} are persisted
+ * and durable in the LogStream.
+ *
+ * @return TODO Unknown?
+ */
--- End diff --
Offset
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589523#comment-16589523
]
ASF GitHub Bot commented on RATIS-272:
--
Github user VladRodionov commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r212154446
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogStream.java
---
@@ -0,0 +1,109 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * A distributed log with "infinite" length that supports reads and writes.
+ */
+public interface LogStream {
+
+ /**
+ * Returns the unique name to identify this log.
+ */
+ LogName getName();
+
+ /**
+ * Returns the size of this LogStream in bytes.
+ */
+ long getSizeInBytes();
+
+ /**
+ * Returns the number of records in this LogStream.
+ */
+ long getSizeInRecords();
+
+ /**
+ * Creates a reader to read this LogStream asynchronously.
+ *
+ * @return An asynchronous reader
+ */
+ AsyncLogReader createAsyncReader();
+
+ /**
+ * Creates a writer to write to this LogStream asynchronously.
+ *
+ * @return An asynchronous writer
+ */
+ AsyncLogWriter createAsyncWriter();
+
+ /**
+ * Creates a reader to read this LogStream synchronously.
+ *
+ * @return A synchronous reader
+ */
+ LogReader createReader();
+
+ /**
+ * Creates a write to write to this LogStream synchronously.
+ *
+ * @return A synchronous writer
+ */
+ LogWriter createWriter();
+
+ /**
+ * Removes the elements in this LogStream prior to the given recordId.
+ *
+ * @param recordId A non-negative recordId for this LogStream
+ */
+ CompletableFuture truncateBefore(long recordId);
+
+ /**
+ * Returns the recordId which is the start of the LogStream. When there
are records which were truncated
+ * from the LogStream, this will return a value larger than {@code 0}.
+ */
+ CompletableFuture getFirstRecordId();
+
+ /**
+ * Copies all records from the beginning of the LogStream until the
given {@code recordId}
+ * to the configured archival storage.
+ *
+ * @param recordId A non-negative recordId for this LogStream
+ */
+ CompletableFuture archiveBefore(long recordId);
+
+ /**
+ * Returns the recordId, prior to which, all records in the LogStream
are archived.
+ */
+ CompletableFuture getArchivalPoint();
--- End diff --
What does concept of log archive give us? We either need log (regardless
its status: active/archive) or we do not.
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589520#comment-16589520
]
ASF GitHub Bot commented on RATIS-272:
--
Github user VladRodionov commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r212154050
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogStream.java
---
@@ -0,0 +1,109 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * A distributed log with "infinite" length that supports reads and writes.
+ */
+public interface LogStream {
+
+ /**
+ * Returns the unique name to identify this log.
+ */
+ LogName getName();
+
+ /**
+ * Returns the size of this LogStream in bytes.
+ */
+ long getSizeInBytes();
+
+ /**
+ * Returns the number of records in this LogStream.
+ */
+ long getSizeInRecords();
+
+ /**
+ * Creates a reader to read this LogStream asynchronously.
+ *
+ * @return An asynchronous reader
+ */
+ AsyncLogReader createAsyncReader();
+
+ /**
+ * Creates a writer to write to this LogStream asynchronously.
+ *
+ * @return An asynchronous writer
+ */
+ AsyncLogWriter createAsyncWriter();
+
+ /**
+ * Creates a reader to read this LogStream synchronously.
+ *
+ * @return A synchronous reader
+ */
+ LogReader createReader();
+
+ /**
+ * Creates a write to write to this LogStream synchronously.
+ *
+ * @return A synchronous writer
+ */
+ LogWriter createWriter();
+
+ /**
+ * Removes the elements in this LogStream prior to the given recordId.
+ *
+ * @param recordId A non-negative recordId for this LogStream
+ */
+ CompletableFuture truncateBefore(long recordId);
+
+ /**
+ * Returns the recordId which is the start of the LogStream. When there
are records which were truncated
+ * from the LogStream, this will return a value larger than {@code 0}.
+ */
+ CompletableFuture getFirstRecordId();
+
--- End diff --
Do we really need this? What is the use case?
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589519#comment-16589519
]
ASF GitHub Bot commented on RATIS-272:
--
Github user VladRodionov commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r212153951
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogStream.java
---
@@ -0,0 +1,109 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * A distributed log with "infinite" length that supports reads and writes.
+ */
+public interface LogStream {
+
+ /**
+ * Returns the unique name to identify this log.
+ */
+ LogName getName();
+
+ /**
+ * Returns the size of this LogStream in bytes.
+ */
+ long getSizeInBytes();
+
+ /**
+ * Returns the number of records in this LogStream.
+ */
+ long getSizeInRecords();
+
+ /**
+ * Creates a reader to read this LogStream asynchronously.
+ *
+ * @return An asynchronous reader
+ */
+ AsyncLogReader createAsyncReader();
+
+ /**
+ * Creates a writer to write to this LogStream asynchronously.
+ *
+ * @return An asynchronous writer
+ */
+ AsyncLogWriter createAsyncWriter();
+
+ /**
+ * Creates a reader to read this LogStream synchronously.
+ *
+ * @return A synchronous reader
+ */
+ LogReader createReader();
+
+ /**
+ * Creates a write to write to this LogStream synchronously.
+ *
+ * @return A synchronous writer
+ */
+ LogWriter createWriter();
+
+ /**
+ * Removes the elements in this LogStream prior to the given recordId.
+ *
+ * @param recordId A non-negative recordId for this LogStream
+ */
+ CompletableFuture truncateBefore(long recordId);
--- End diff --
Do we really need this one? Use case?
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589516#comment-16589516
]
ASF GitHub Bot commented on RATIS-272:
--
Github user VladRodionov commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r212153843
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogService.java
---
@@ -0,0 +1,53 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.util.Iterator;
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * Entry point for interacting with the Ratis LogService.
+ */
+public interface LogService {
+
+ /**
+ * Creates a new {@link LogStream} identified by the given name. Throws
+ * an exception if a {@link LogStream} with the given name already
exists.
+ *
+ * @param name Unique name for this LogStream.
+ */
+ CompletableFuture createLog(LogName name);
--- End diff --
Again, why is this async operation and why we expose this to a client?
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589514#comment-16589514
]
ASF GitHub Bot commented on RATIS-272:
--
Github user VladRodionov commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r212153677
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogReader.java
---
@@ -0,0 +1,53 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+
+/**
+ * Synchronous client interface to read from a LogStream.
+ */
+public interface LogReader extends AutoCloseable {
+
+ /**
+ * Seeks to the position before the record at the provided {@code
offset} in the LogStream.
+ *
+ * @param offset A non-negative, offset in the LogStream
+ * @return A future for when the operation is completed.
+ */
+ void seek(long recordId) throws IOException;
--- End diff --
Seeking by record id is expensive to implement. One has to support separate
index {recordid -> file offset}. This should be not ID, but generic "offset",
which maps 1 -> 1 to file offset. The only use case, where we need positional
reads (at least in logs) is to resume sequential reading from the last read
offset (HBase replication as an example).
We have to separate logs from message queues - they are totally different
by their purpose.
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589511#comment-16589511
]
ASF GitHub Bot commented on RATIS-272:
--
Github user VladRodionov commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r212153063
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/ArchivedLog.java
---
@@ -0,0 +1,43 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+/**
+ * A {@link LogStream} which has been archived in some external
+ * system. This interface is parameterized to allow for implementations
+ * to use their own class to encapsulate how to find the archived log.
+ *
+ * In the majority of cases, this should be transparent to end-users, as
+ * the {@link LogStream} should hide the fact that this even exists.
+ * TODO maybe that means this should be client-facing at all?
+ *
+ * @param A referent to the log on the external system.
+ */
+public interface ArchivedLog extends AutoCloseable {
+
+ /**
+ * Creates an asynchronous reader over this archived log.
+ */
+ AsyncLogReader createAsyncReader();
--- End diff --
It would be nice to have a clarification on all async stuff
(readers/writers).
1. Do they improve performance?
2. Do they improve latencies? and can we measure this improvement?
3. Do they simplify programming model?
My guess on all three is they don't. Async model (not based on OS kernel
support) is broken in Java.
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589434#comment-16589434
]
ASF GitHub Bot commented on RATIS-272:
--
Github user VladRodionov commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r212136412
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/ArchivedLog.java
---
@@ -0,0 +1,43 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+/**
+ * A {@link LogStream} which has been archived in some external
+ * system. This interface is parameterized to allow for implementations
+ * to use their own class to encapsulate how to find the archived log.
+ *
+ * In the majority of cases, this should be transparent to end-users, as
+ * the {@link LogStream} should hide the fact that this even exists.
+ * TODO maybe that means this should be client-facing at all?
+ *
+ * @param A referent to the log on the external system.
+ */
+public interface ArchivedLog extends AutoCloseable {
--- End diff --
That is something different. Raft vs Zab/Chubby. Log Service is based on
Raft, so we are kosher here :) - we do not build distributed system on top of
FS abstraction, just we need something similar to use Log Service at its
fullest. No, we are not providing Posix-compatible API nor even Hadoop -
compatible API, although I think, potentially, this would be a nice feature:
mount/unmount log service as a file system.
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[ https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589427#comment-16589427 ] ASF GitHub Bot commented on RATIS-272: -- Github user VladRodionov commented on the issue: https://github.com/apache/incubator-ratis/pull/4 Hmm, read, write, create, open, seek, close, sync, list, delete etc. What else? Is not this already File I/O? They are not streams as long as you provide positional access. They are files. In any case you will need API, which closely matches File I/O. > Design ideal API > > > Key: RATIS-272 > URL: https://issues.apache.org/jira/browse/RATIS-272 > Project: Ratis > Issue Type: Sub-task > Components: LogService >Reporter: Josh Elser >Priority: Major > > With influence from Apache DistributedLog, Kafka, and BookKeeper, design an > API that balances the ideal notion of what a distribute log system should > look like, but also considers the needs of HBase to replace a WAL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589366#comment-16589366
]
ASF GitHub Bot commented on RATIS-272:
--
Github user anuengineer commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r212115543
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/ArchivedLog.java
---
@@ -0,0 +1,43 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+/**
+ * A {@link LogStream} which has been archived in some external
+ * system. This interface is parameterized to allow for implementations
+ * to use their own class to encapsulate how to find the archived log.
+ *
+ * In the majority of cases, this should be transparent to end-users, as
+ * the {@link LogStream} should hide the fact that this even exists.
+ * TODO maybe that means this should be client-facing at all?
+ *
+ * @param A referent to the log on the external system.
+ */
+public interface ArchivedLog extends AutoCloseable {
--- End diff --
While I may not agree completely with the write up, A file system like API
for Replicated State Machines have some criticisms. Here is a blog written from
the HyderDex people.
http://hackingdistributed.com/2013/12/26/introducing-replicant/
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[ https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589359#comment-16589359 ] ASF GitHub Bot commented on RATIS-272: -- Github user joshelser commented on the issue: https://github.com/apache/incubator-ratis/pull/4 Breaking this out to a top-level thread instead of being anchored on a code segment: > We are not talking about full fledged FileSystem API here, but some API would help in adoption, taking into account familiar file system abstraction. I had a chat with Ankit and Sergey about this one, trying to weigh the pros and cons. A FileSystem-like API would give users some familiarity in working with the LogService, but the general feeling was that we may be over-burdening ourselves in the first implementation. There would be many things we could potentially implement via a FileSystem like API which we wouldn't necessarily need for HBase as an initial consumer. I think we could explore using a FileSystem-like API in the future to supplement a more minimal core API (closer to what is here already). Something higher-level could also be done as we get closer to HBase's rearchitecting. I think it's becoming pretty clear that HBase's new WAL API won't be written using the notion of a File on a FileSystem (and instead use a "stream-like" API at the core). I could see us integrating the Ratis LogService into HBase via a FileSystem facade. How do you feel about that, @VladRodionov? Are there any immediate concerns which would limit how we're thinking about building out the LogService if we didn't have a FileSystem facade? > Design ideal API > > > Key: RATIS-272 > URL: https://issues.apache.org/jira/browse/RATIS-272 > Project: Ratis > Issue Type: Sub-task > Components: LogService >Reporter: Josh Elser >Priority: Major > > With influence from Apache DistributedLog, Kafka, and BookKeeper, design an > API that balances the ideal notion of what a distribute log system should > look like, but also considers the needs of HBase to replace a WAL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589305#comment-16589305
]
ASF GitHub Bot commented on RATIS-272:
--
Github user VladRodionov commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r212087778
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogStream.java
---
@@ -0,0 +1,109 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * A distributed log with "infinite" length that supports reads and writes.
+ */
+public interface LogStream {
+
+ /**
+ * Returns the unique name to identify this log.
+ */
+ LogName getName();
+
+ /**
+ * Returns the size of this LogStream in bytes.
+ */
+ long getSizeInBytes();
+
+ /**
+ * Returns the number of records in this LogStream.
+ */
+ long getSizeInRecords();
+
+ /**
+ * Creates a reader to read this LogStream asynchronously.
+ *
+ * @return An asynchronous reader
+ */
+ AsyncLogReader createAsyncReader();
+
+ /**
+ * Creates a writer to write to this LogStream asynchronously.
+ *
+ * @return An asynchronous writer
+ */
+ AsyncLogWriter createAsyncWriter();
+
+ /**
+ * Creates a reader to read this LogStream synchronously.
+ *
+ * @return A synchronous reader
+ */
+ LogReader createReader();
+
+ /**
+ * Creates a write to write to this LogStream synchronously.
+ *
+ * @return A synchronous writer
+ */
+ LogWriter createWriter();
+
--- End diff --
Multiple writers - multiple readers is a MUST for generic Log Service (not
HBase), but close() must be provided. On commit() - I would replace it with
sync() (sync to durable storage). Commit implies rollback :)
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589301#comment-16589301
]
ASF GitHub Bot commented on RATIS-272:
--
Github user VladRodionov commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r212086396
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/AsyncLogReader.java
---
@@ -0,0 +1,54 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * Asynchronous client interface to read from a LogStream.
+ */
+public interface AsyncLogReader extends AutoCloseable {
+
+ /**
+ * Seeks to the position before the record at the provided {@code
recordId} in the LogStream.
+ *
+ * @param recordId A non-negative, recordId in the LogStream
+ * @return A future for when the operation is completed.
+ */
+ CompletableFuture seek(long recordId) throws IOException;
+
+ /**
+ * Reads the next record from the LogStream at the current position and
advances the current position
+ * to after the record which was just returned.
+ *
+ * @return A future providing the data for the next record.
+ */
+ CompletableFuture readNext() throws IOException;
+
+ /**
+ * Reads the next {@code numRecords} records from the LogStream,
starting at the current position. This method
+ * may return fewer than requested records if the LogStream does not
have sufficient records to return.
+ *
+ * @param numRecords The number of records to return
+ * @return A future providing the records, no more than the requested
{@code numRecords} amount.
+ */
+ CompletableFuture> readBulk(int numRecords) throws
IOException;
--- End diff --
Sure, let us do not breed buffers - they are expensive :)
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589299#comment-16589299
]
ASF GitHub Bot commented on RATIS-272:
--
Github user VladRodionov commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r212086058
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/ArchivedLog.java
---
@@ -0,0 +1,43 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+/**
+ * A {@link LogStream} which has been archived in some external
+ * system. This interface is parameterized to allow for implementations
+ * to use their own class to encapsulate how to find the archived log.
+ *
+ * In the majority of cases, this should be transparent to end-users, as
+ * the {@link LogStream} should hide the fact that this even exists.
+ * TODO maybe that means this should be client-facing at all?
+ *
+ * @param A referent to the log on the external system.
+ */
+public interface ArchivedLog extends AutoCloseable {
--- End diff --
We are not talking about full fledged FileSystem API here, but some API
would help in adoption, taking into account familiar file system abstraction.
Besides this, as a HBase developer I have my own requirements (see my comments
above) to this Log Service and they can be fulfilled only by adopting FS-like
API.
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589291#comment-16589291
]
ASF GitHub Bot commented on RATIS-272:
--
Github user joshelser commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r212083394
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/ArchivedLog.java
---
@@ -0,0 +1,43 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+/**
+ * A {@link LogStream} which has been archived in some external
+ * system. This interface is parameterized to allow for implementations
+ * to use their own class to encapsulate how to find the archived log.
+ *
+ * In the majority of cases, this should be transparent to end-users, as
+ * the {@link LogStream} should hide the fact that this even exists.
+ * TODO maybe that means this should be client-facing at all?
+ *
+ * @param A referent to the log on the external system.
+ */
+public interface ArchivedLog extends AutoCloseable {
--- End diff --
Looking more closely at java.nio.file.Path, java.nio.file.WatchService,
java.nio.file.FileSystem, and java.nio.file.spi.FileSystemProvider, I think
that there are definitely pieces which make sense for us to use, but I think we
also get a lot of baggage that we don't want (that we would just always have
implemented with a RuntimeException).
The WatchService is basically what the RecordListener was meant to be. I
think using a "filesystem path" (e.g. /a/b/c) does make sense and gives us
flexibility for the LogService overall (instead of just "log name").
What about something in between? Take some of the good parts of Path and
FileSystem, and pull them into LogService directly?
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589277#comment-16589277
]
ASF GitHub Bot commented on RATIS-272:
--
Github user ss77892 commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r212078301
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/ArchivedLog.java
---
@@ -0,0 +1,43 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+/**
+ * A {@link LogStream} which has been archived in some external
+ * system. This interface is parameterized to allow for implementations
+ * to use their own class to encapsulate how to find the archived log.
+ *
+ * In the majority of cases, this should be transparent to end-users, as
+ * the {@link LogStream} should hide the fact that this even exists.
+ * TODO maybe that means this should be client-facing at all?
+ *
+ * @param A referent to the log on the external system.
+ */
+public interface ArchivedLog extends AutoCloseable {
--- End diff --
Using Path just to identify the Log is abusing the public API. Or that
would mean that we need to really build a file system on top of that, adding
File, FileSystem, FileStore, etc implementations with a corresponding
functionality. I think that we should think about an implementation of a
classic distributed log system which is close to 'pub-sub'/messaging rather
than a file system.
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589274#comment-16589274
]
ASF GitHub Bot commented on RATIS-272:
--
Github user joshelser commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r212077861
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/AsyncLogReader.java
---
@@ -0,0 +1,54 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * Asynchronous client interface to read from a LogStream.
+ */
+public interface AsyncLogReader extends AutoCloseable {
+
+ /**
+ * Seeks to the position before the record at the provided {@code
recordId} in the LogStream.
+ *
+ * @param recordId A non-negative, recordId in the LogStream
+ * @return A future for when the operation is completed.
+ */
+ CompletableFuture seek(long recordId) throws IOException;
+
+ /**
+ * Reads the next record from the LogStream at the current position and
advances the current position
+ * to after the record which was just returned.
+ *
+ * @return A future providing the data for the next record.
+ */
+ CompletableFuture readNext() throws IOException;
+
+ /**
+ * Reads the next {@code numRecords} records from the LogStream,
starting at the current position. This method
+ * may return fewer than requested records if the LogStream does not
have sufficient records to return.
+ *
+ * @param numRecords The number of records to return
+ * @return A future providing the records, no more than the requested
{@code numRecords} amount.
+ */
+ CompletableFuture> readBulk(int numRecords) throws
IOException;
--- End diff --
@VladRodionov given your suggestion to have a `readNext(ByteBuffer)`
variant, do you think that we should have a similar addition for `readBulk`,
e.g. `readBulk(List)` where we fill each provided ByteBuffer with
the next record? For example, if the caller gave a `List` of length
5, we would return back 5 records.
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589270#comment-16589270
]
ASF GitHub Bot commented on RATIS-272:
--
Github user joshelser commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r212074817
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogStream.java
---
@@ -0,0 +1,109 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * A distributed log with "infinite" length that supports reads and writes.
+ */
+public interface LogStream {
+
+ /**
+ * Returns the unique name to identify this log.
+ */
+ LogName getName();
+
+ /**
+ * Returns the size of this LogStream in bytes.
+ */
+ long getSizeInBytes();
+
+ /**
+ * Returns the number of records in this LogStream.
+ */
+ long getSizeInRecords();
+
+ /**
+ * Creates a reader to read this LogStream asynchronously.
+ *
+ * @return An asynchronous reader
+ */
+ AsyncLogReader createAsyncReader();
+
+ /**
+ * Creates a writer to write to this LogStream asynchronously.
+ *
+ * @return An asynchronous writer
+ */
+ AsyncLogWriter createAsyncWriter();
+
+ /**
+ * Creates a reader to read this LogStream synchronously.
+ *
+ * @return A synchronous reader
+ */
+ LogReader createReader();
+
+ /**
+ * Creates a write to write to this LogStream synchronously.
+ *
+ * @return A synchronous writer
+ */
+ LogWriter createWriter();
+
+ /**
+ * Removes the elements in this LogStream prior to the given recordId.
+ *
+ * @param recordId A non-negative recordId for this LogStream
+ */
+ CompletableFuture truncateBefore(long recordId);
+
+ /**
+ * Returns the recordId which is the start of the LogStream. When there
are records which were truncated
+ * from the LogStream, this will return a value larger than {@code 0}.
+ */
+ CompletableFuture getFirstRecordId();
+
+ /**
+ * Copies all records from the beginning of the LogStream until the
given {@code recordId}
+ * to the configured archival storage.
+ *
+ * @param recordId A non-negative recordId for this LogStream
+ */
+ CompletableFuture archiveBefore(long recordId);
--- End diff --
On second thought "archiveBefore" would still be correct. I still like the
inclusivity.
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589056#comment-16589056
]
ASF GitHub Bot commented on RATIS-272:
--
Github user joshelser commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r212010047
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogService.java
---
@@ -0,0 +1,53 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.util.Iterator;
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * Entry point for interacting with the Ratis LogService.
+ */
+public interface LogService {
+
+ /**
+ * Creates a new {@link LogStream} identified by the given name. Throws
+ * an exception if a {@link LogStream} with the given name already
exists.
+ *
+ * @param name Unique name for this LogStream.
+ */
+ CompletableFuture createLog(LogName name);
+
+ /**
+ * Fetches the {@link LogStream} identified by the given name.
+ *
+ * @param name The name of the LogStream
+ */
+ CompletableFuture getLog(LogName name);
+
+ /**
+ * Lists all {@link LogStream} instances known by this LogService.
+ */
+ CompletableFuture> listLogs();
--- End diff --
If we go towards Vlad's suggestion of a FileSystem-like API, we would give
a `PathFilter` that would give the same kind of functionality. Having the
notion of directories actually gets infinitely easier to find just logs you
want too :)
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589052#comment-16589052
]
ASF GitHub Bot commented on RATIS-272:
--
Github user joshelser commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r212009732
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogStream.java
---
@@ -0,0 +1,109 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * A distributed log with "infinite" length that supports reads and writes.
+ */
+public interface LogStream {
+
+ /**
+ * Returns the unique name to identify this log.
+ */
+ LogName getName();
+
+ /**
+ * Returns the size of this LogStream in bytes.
+ */
+ long getSizeInBytes();
+
+ /**
+ * Returns the number of records in this LogStream.
+ */
+ long getSizeInRecords();
+
+ /**
+ * Creates a reader to read this LogStream asynchronously.
+ *
+ * @return An asynchronous reader
+ */
+ AsyncLogReader createAsyncReader();
+
+ /**
+ * Creates a writer to write to this LogStream asynchronously.
+ *
+ * @return An asynchronous writer
+ */
+ AsyncLogWriter createAsyncWriter();
+
+ /**
+ * Creates a reader to read this LogStream synchronously.
+ *
+ * @return A synchronous reader
+ */
+ LogReader createReader();
+
+ /**
+ * Creates a write to write to this LogStream synchronously.
+ *
+ * @return A synchronous writer
+ */
+ LogWriter createWriter();
+
--- End diff --
My hope was that we could support multiple, independent writers. The writer
implementation would be able to hide most details (I hope).
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589048#comment-16589048
]
ASF GitHub Bot commented on RATIS-272:
--
Github user joshelser commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r212009110
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/AsyncLogReader.java
---
@@ -0,0 +1,54 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * Asynchronous client interface to read from a LogStream.
+ */
+public interface AsyncLogReader extends AutoCloseable {
+
+ /**
+ * Seeks to the position before the record at the provided {@code
recordId} in the LogStream.
+ *
+ * @param recordId A non-negative, recordId in the LogStream
+ * @return A future for when the operation is completed.
+ */
+ CompletableFuture seek(long recordId) throws IOException;
+
--- End diff --
If you think that would be helpful, we can. I am wondering about the
frequency of use of such an API. Does it make sense to have that "wrapper" when
we have something like `seek(getFirstRecordId())` already present? I don't feel
strongly either way :)
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589046#comment-16589046
]
ASF GitHub Bot commented on RATIS-272:
--
Github user joshelser commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r212008406
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogService.java
---
@@ -0,0 +1,53 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.util.Iterator;
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * Entry point for interacting with the Ratis LogService.
+ */
+public interface LogService {
+
+ /**
+ * Creates a new {@link LogStream} identified by the given name. Throws
+ * an exception if a {@link LogStream} with the given name already
exists.
+ *
+ * @param name Unique name for this LogStream.
+ */
+ CompletableFuture createLog(LogName name);
--- End diff --
> we may need to add some metadata for the stream, for eg:- Details of
SERDE classes, compression etc.
Good point. I can make a basic configuration class which just wraps a
`Map` for now? How does that sound while we flesh out what kind
of metadata we'll have?
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589043#comment-16589043
]
ASF GitHub Bot commented on RATIS-272:
--
Github user joshelser commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r212007747
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogStream.java
---
@@ -0,0 +1,109 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * A distributed log with "infinite" length that supports reads and writes.
+ */
+public interface LogStream {
+
+ /**
+ * Returns the unique name to identify this log.
+ */
+ LogName getName();
+
+ /**
+ * Returns the size of this LogStream in bytes.
+ */
+ long getSizeInBytes();
+
+ /**
+ * Returns the number of records in this LogStream.
+ */
+ long getSizeInRecords();
+
+ /**
+ * Creates a reader to read this LogStream asynchronously.
+ *
+ * @return An asynchronous reader
+ */
+ AsyncLogReader createAsyncReader();
+
+ /**
+ * Creates a writer to write to this LogStream asynchronously.
+ *
+ * @return An asynchronous writer
+ */
+ AsyncLogWriter createAsyncWriter();
+
+ /**
+ * Creates a reader to read this LogStream synchronously.
+ *
+ * @return A synchronous reader
+ */
+ LogReader createReader();
+
+ /**
+ * Creates a write to write to this LogStream synchronously.
+ *
+ * @return A synchronous writer
+ */
+ LogWriter createWriter();
+
+ /**
+ * Removes the elements in this LogStream prior to the given recordId.
+ *
+ * @param recordId A non-negative recordId for this LogStream
+ */
+ CompletableFuture truncateBefore(long recordId);
+
+ /**
+ * Returns the recordId which is the start of the LogStream. When there
are records which were truncated
+ * from the LogStream, this will return a value larger than {@code 0}.
+ */
+ CompletableFuture getFirstRecordId();
+
+ /**
+ * Copies all records from the beginning of the LogStream until the
given {@code recordId}
+ * to the configured archival storage.
+ *
+ * @param recordId A non-negative recordId for this LogStream
+ */
+ CompletableFuture archiveBefore(long recordId);
--- End diff --
Can certainly expand this to be `archive(long recordId, boolean
inclusive)`. I like "archive" much more than "archiveBefore" ;)
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589045#comment-16589045
]
ASF GitHub Bot commented on RATIS-272:
--
Github user joshelser commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r212008014
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/ArchivedLog.java
---
@@ -0,0 +1,43 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+/**
+ * A {@link LogStream} which has been archived in some external
+ * system. This interface is parameterized to allow for implementations
+ * to use their own class to encapsulate how to find the archived log.
+ *
+ * In the majority of cases, this should be transparent to end-users, as
+ * the {@link LogStream} should hide the fact that this even exists.
+ * TODO maybe that means this should be client-facing at all?
+ *
--- End diff --
Yeah, Vlad brought up the same point. Let me drop it client API -- this is
something we probably want as internal API.
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589042#comment-16589042
]
ASF GitHub Bot commented on RATIS-272:
--
Github user joshelser commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r212007378
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/ArchivedLog.java
---
@@ -0,0 +1,43 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+/**
+ * A {@link LogStream} which has been archived in some external
+ * system. This interface is parameterized to allow for implementations
+ * to use their own class to encapsulate how to find the archived log.
+ *
+ * In the majority of cases, this should be transparent to end-users, as
+ * the {@link LogStream} should hide the fact that this even exists.
+ * TODO maybe that means this should be client-facing at all?
+ *
+ * @param A referent to the log on the external system.
+ */
+public interface ArchivedLog extends AutoCloseable {
--- End diff --
Yeah, I was mulling over whether or not to even include it in this initial
drop.
I like the idea of a consistent API to read from Log, regardless of it
being "active" (still in the state machine) or "archived" (exported to some
remote storage).
> Log archiving is the feature HBase requires
I think that this is a generally reusable feature that users would like. It
would help keep the state machine "small" and consistent performance.
> HBase client will create archive subdirectory and will be moving old
files to this subdirectory
I think automated archival should be a feature that LogService provides.
HBase should be unaware that we are moving "segments" of data around.
> I want to enhance LogService with FileSystem-like API:
I have changes that have been sitting locally for too long. I'm
prioritizing getting them finished and then update this. I think being able to
think of the LogService like a FileSystem is nice. I'm curious if you have put
more thought about how the notion of "directories" might be used :)
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[ https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589035#comment-16589035 ] ASF GitHub Bot commented on RATIS-272: -- Github user joshelser commented on the issue: https://github.com/apache/incubator-ratis/pull/4 > We should adopt standard Unix convention - everything is a file, Path in our case. java.io.Path gives us direct way to LogService file system. Good point. That makes sense. > Design ideal API > > > Key: RATIS-272 > URL: https://issues.apache.org/jira/browse/RATIS-272 > Project: Ratis > Issue Type: Sub-task > Components: LogService >Reporter: Josh Elser >Priority: Major > > With influence from Apache DistributedLog, Kafka, and BookKeeper, design an > API that balances the ideal notion of what a distribute log system should > look like, but also considers the needs of HBase to replace a WAL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16587767#comment-16587767
]
ASF GitHub Bot commented on RATIS-272:
--
Github user VladRodionov commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r211689322
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/ArchivedLog.java
---
@@ -0,0 +1,43 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+/**
+ * A {@link LogStream} which has been archived in some external
+ * system. This interface is parameterized to allow for implementations
+ * to use their own class to encapsulate how to find the archived log.
+ *
+ * In the majority of cases, this should be transparent to end-users, as
+ * the {@link LogStream} should hide the fact that this even exists.
+ * TODO maybe that means this should be client-facing at all?
+ *
+ * @param A referent to the log on the external system.
+ */
+public interface ArchivedLog extends AutoCloseable {
--- End diff --
I think we do not need this at all. Log archiving is the feature HBase
requires. From HBase point of view its the same log file only in different
directory. I want to enhance LogService with FileSystem-like API:
1. create/delete dir/files(logs)
2. Use java.io.Path to identify Log streams
3. rename support
4. delete (including recursive)
Client can manipulate directories and files. HBase client will create
archive subdirectory and will be moving old files to this subdirectory
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16587755#comment-16587755
]
ASF GitHub Bot commented on RATIS-272:
--
Github user VladRodionov commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r211685307
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogReader.java
---
@@ -0,0 +1,53 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+
+/**
+ * Synchronous client interface to read from a LogStream.
+ */
+public interface LogReader extends AutoCloseable {
+
+ /**
+ * Seeks to the position before the record at the provided {@code
offset} in the LogStream.
+ *
+ * @param offset A non-negative, offset in the LogStream
+ * @return A future for when the operation is completed.
+ */
+ void seek(long recordId) throws IOException;
+
+ /**
+ * Reads the next record from the LogStream at the current position and
advances the current position
+ * to after the record which was just returned.
+ *
+ * @return The data for the next record.
+ */
+ ByteBuffer readNext() throws IOException;
+
+ /**
+ * Reads the next {@code numRecords} records from the LogStream,
starting at the current position. This method
+ * may return fewer than requested records if the LogStream does not
have sufficient records to return.
+ *
+ * @param numRecords The number of records to return
+ * @return The records, no more than the requested {@code numRecords}
amount.
+ */
+ List readBulk(int numRecords) throws IOException;
--- End diff --
The same here as well. API call to reuse existing buffers
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16587751#comment-16587751
]
ASF GitHub Bot commented on RATIS-272:
--
Github user VladRodionov commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r211684524
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogName.java ---
@@ -0,0 +1,28 @@
+package org.apache.ratis.logservice.api;
+
+import static java.util.Objects.requireNonNull;
+
+/**
+ * Identifier to uniquely identify a {@link LogStream}.
+ */
+public class LogName {
--- End diff --
LogName -> java.io.Path. Let us adopt file system for log service
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16587753#comment-16587753
]
ASF GitHub Bot commented on RATIS-272:
--
Github user VladRodionov commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r211684885
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogReader.java
---
@@ -0,0 +1,53 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+
+/**
+ * Synchronous client interface to read from a LogStream.
+ */
+public interface LogReader extends AutoCloseable {
+
+ /**
+ * Seeks to the position before the record at the provided {@code
offset} in the LogStream.
+ *
+ * @param offset A non-negative, offset in the LogStream
+ * @return A future for when the operation is completed.
+ */
+ void seek(long recordId) throws IOException;
+
+ /**
+ * Reads the next record from the LogStream at the current position and
advances the current position
+ * to after the record which was just returned.
+ *
+ * @return The data for the next record.
+ */
+ ByteBuffer readNext() throws IOException;
--- End diff --
readNext(ByteBuffer) - to reuse existing buffer
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[ https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16587747#comment-16587747 ] ASF GitHub Bot commented on RATIS-272: -- Github user VladRodionov commented on the issue: https://github.com/apache/incubator-ratis/pull/4 LogName is kind of artificial thing. We should adopt standard Unix convention - everything is a file, Path in our case. java.io.Path gives us direct way to LogService file system. > Design ideal API > > > Key: RATIS-272 > URL: https://issues.apache.org/jira/browse/RATIS-272 > Project: Ratis > Issue Type: Sub-task > Components: LogService >Reporter: Josh Elser >Priority: Major > > With influence from Apache DistributedLog, Kafka, and BookKeeper, design an > API that balances the ideal notion of what a distribute log system should > look like, but also considers the needs of HBase to replace a WAL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[ https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16586568#comment-16586568 ] ASF GitHub Bot commented on RATIS-272: -- Github user joshelser commented on the issue: https://github.com/apache/incubator-ratis/pull/4 Thanks @ankitsinghal and @chrajeshbabu. I have some more changes staged locally, too. Will update what I have (along with some "noop" implementations to show an e2e example) with your feedback! > Design ideal API > > > Key: RATIS-272 > URL: https://issues.apache.org/jira/browse/RATIS-272 > Project: Ratis > Issue Type: Sub-task > Components: LogService >Reporter: Josh Elser >Priority: Major > > With influence from Apache DistributedLog, Kafka, and BookKeeper, design an > API that balances the ideal notion of what a distribute log system should > look like, but also considers the needs of HBase to replace a WAL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16586163#comment-16586163
]
ASF GitHub Bot commented on RATIS-272:
--
Github user chrajeshbabu commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r211319590
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogStream.java
---
@@ -0,0 +1,109 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * A distributed log with "infinite" length that supports reads and writes.
+ */
+public interface LogStream {
+
+ /**
+ * Returns the unique name to identify this log.
+ */
+ LogName getName();
+
+ /**
+ * Returns the size of this LogStream in bytes.
+ */
+ long getSizeInBytes();
+
+ /**
+ * Returns the number of records in this LogStream.
+ */
+ long getSizeInRecords();
+
+ /**
+ * Creates a reader to read this LogStream asynchronously.
+ *
+ * @return An asynchronous reader
+ */
+ AsyncLogReader createAsyncReader();
+
+ /**
+ * Creates a writer to write to this LogStream asynchronously.
+ *
+ * @return An asynchronous writer
+ */
+ AsyncLogWriter createAsyncWriter();
+
+ /**
+ * Creates a reader to read this LogStream synchronously.
+ *
+ * @return A synchronous reader
+ */
+ LogReader createReader();
+
+ /**
+ * Creates a write to write to this LogStream synchronously.
+ *
+ * @return A synchronous writer
+ */
+ LogWriter createWriter();
+
+ /**
+ * Removes the elements in this LogStream prior to the given recordId.
+ *
+ * @param recordId A non-negative recordId for this LogStream
+ */
+ CompletableFuture truncateBefore(long recordId);
+
+ /**
+ * Returns the recordId which is the start of the LogStream. When there
are records which were truncated
+ * from the LogStream, this will return a value larger than {@code 0}.
+ */
+ CompletableFuture getFirstRecordId();
+
+ /**
+ * Copies all records from the beginning of the LogStream until the
given {@code recordId}
+ * to the configured archival storage.
+ *
+ * @param recordId A non-negative recordId for this LogStream
+ */
+ CompletableFuture archiveBefore(long recordId);
--- End diff --
May be we can have a way to say wether the passed id is exclusive or
inclusive.
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16586162#comment-16586162
]
ASF GitHub Bot commented on RATIS-272:
--
Github user chrajeshbabu commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r211321060
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogStream.java
---
@@ -0,0 +1,109 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * A distributed log with "infinite" length that supports reads and writes.
+ */
+public interface LogStream {
+
+ /**
+ * Returns the unique name to identify this log.
+ */
+ LogName getName();
+
+ /**
+ * Returns the size of this LogStream in bytes.
+ */
+ long getSizeInBytes();
+
+ /**
+ * Returns the number of records in this LogStream.
+ */
+ long getSizeInRecords();
+
+ /**
+ * Creates a reader to read this LogStream asynchronously.
+ *
+ * @return An asynchronous reader
+ */
+ AsyncLogReader createAsyncReader();
+
+ /**
+ * Creates a writer to write to this LogStream asynchronously.
+ *
+ * @return An asynchronous writer
+ */
+ AsyncLogWriter createAsyncWriter();
+
+ /**
+ * Creates a reader to read this LogStream synchronously.
+ *
+ * @return A synchronous reader
+ */
+ LogReader createReader();
+
+ /**
+ * Creates a write to write to this LogStream synchronously.
+ *
+ * @return A synchronous writer
+ */
+ LogWriter createWriter();
+
+ /**
+ * Removes the elements in this LogStream prior to the given recordId.
+ *
+ * @param recordId A non-negative recordId for this LogStream
+ */
+ CompletableFuture truncateBefore(long recordId);
--- End diff --
Here also we should have a way whether to include the passed record id or
not. If we don't want to include the passed recordid then we client of log
service need to increment it.
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16584626#comment-16584626
]
ASF GitHub Bot commented on RATIS-272:
--
Github user ankitsinghal commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r211064777
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/ArchivedLog.java
---
@@ -0,0 +1,43 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+/**
+ * A {@link LogStream} which has been archived in some external
+ * system. This interface is parameterized to allow for implementations
+ * to use their own class to encapsulate how to find the archived log.
+ *
+ * In the majority of cases, this should be transparent to end-users, as
+ * the {@link LogStream} should hide the fact that this even exists.
+ * TODO maybe that means this should be client-facing at all?
+ *
--- End diff --
Yes, it should not be client-facing at all. Just knowing from logStream
that if I'm reading from archived or active location should be
fine(recordId Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16584623#comment-16584623
]
ASF GitHub Bot commented on RATIS-272:
--
Github user ankitsinghal commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r211064158
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogService.java
---
@@ -0,0 +1,53 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.util.Iterator;
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * Entry point for interacting with the Ratis LogService.
+ */
+public interface LogService {
+
+ /**
+ * Creates a new {@link LogStream} identified by the given name. Throws
+ * an exception if a {@link LogStream} with the given name already
exists.
+ *
+ * @param name Unique name for this LogStream.
+ */
+ CompletableFuture createLog(LogName name);
+
+ /**
+ * Fetches the {@link LogStream} identified by the given name.
+ *
+ * @param name The name of the LogStream
+ */
+ CompletableFuture getLog(LogName name);
+
+ /**
+ * Lists all {@link LogStream} instances known by this LogService.
+ */
+ CompletableFuture> listLogs();
--- End diff --
can accept a regular expression to avoid large data requested from the
service.
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16584625#comment-16584625
]
ASF GitHub Bot commented on RATIS-272:
--
Github user ankitsinghal commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r211064553
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/AsyncLogReader.java
---
@@ -0,0 +1,54 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * Asynchronous client interface to read from a LogStream.
+ */
+public interface AsyncLogReader extends AutoCloseable {
+
+ /**
+ * Seeks to the position before the record at the provided {@code
recordId} in the LogStream.
+ *
+ * @param recordId A non-negative, recordId in the LogStream
+ * @return A future for when the operation is completed.
+ */
+ CompletableFuture seek(long recordId) throws IOException;
+
--- End diff --
Can add reset() also to reset according to the logStream getFirstRecordId().
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16584627#comment-16584627
]
ASF GitHub Bot commented on RATIS-272:
--
Github user ankitsinghal commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r21106
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogStream.java
---
@@ -0,0 +1,109 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * A distributed log with "infinite" length that supports reads and writes.
+ */
+public interface LogStream {
+
+ /**
+ * Returns the unique name to identify this log.
+ */
+ LogName getName();
+
+ /**
+ * Returns the size of this LogStream in bytes.
+ */
+ long getSizeInBytes();
+
+ /**
+ * Returns the number of records in this LogStream.
+ */
+ long getSizeInRecords();
+
+ /**
+ * Creates a reader to read this LogStream asynchronously.
+ *
+ * @return An asynchronous reader
+ */
+ AsyncLogReader createAsyncReader();
+
+ /**
+ * Creates a writer to write to this LogStream asynchronously.
+ *
+ * @return An asynchronous writer
+ */
+ AsyncLogWriter createAsyncWriter();
+
+ /**
+ * Creates a reader to read this LogStream synchronously.
+ *
+ * @return A synchronous reader
+ */
+ LogReader createReader();
+
+ /**
+ * Creates a write to write to this LogStream synchronously.
+ *
+ * @return A synchronous writer
+ */
+ LogWriter createWriter();
+
--- End diff --
I believe the plan is to support a single writer(otherwise commit() would
get tedious to manage at server), right ? can we make it getWriter()?
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16584622#comment-16584622
]
ASF GitHub Bot commented on RATIS-272:
--
Github user ankitsinghal commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r211064260
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogWriter.java
---
@@ -0,0 +1,43 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+
+/**
+ * Synchronous client interface to write to a LogStream.
+ */
+public interface LogWriter extends AutoCloseable {
+
+ /**
+ * Appends the given data as a record in the LogStream.
+ *
+ * @param data The record to append
+ * @return The recordId for the record just written
+ */
+ long write(ByteBuffer data) throws IOException;
+
+ /**
+ * Guarantees that all previous data appended by {@link
#write(ByteBuffer)} are persisted
+ * and durable in the LogStream.
+ *
+ * @return TODO Unknown?
+ */
--- End diff --
Return last recordId committed.
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16584628#comment-16584628
]
ASF GitHub Bot commented on RATIS-272:
--
Github user ankitsinghal commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r211064684
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogService.java
---
@@ -0,0 +1,53 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.util.Iterator;
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * Entry point for interacting with the Ratis LogService.
+ */
+public interface LogService {
+
+ /**
+ * Creates a new {@link LogStream} identified by the given name. Throws
+ * an exception if a {@link LogStream} with the given name already
exists.
+ *
+ * @param name Unique name for this LogStream.
+ */
+ CompletableFuture createLog(LogName name);
--- End diff --
While creating log , we may need to add some metadata for the stream, for
eg:- Details of SERDE classes, compression etc.
And need the corresponding API in LogStream to get the MetaData.
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16584624#comment-16584624
]
ASF GitHub Bot commented on RATIS-272:
--
Github user ankitsinghal commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r211064459
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogStream.java
---
@@ -0,0 +1,109 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * A distributed log with "infinite" length that supports reads and writes.
+ */
+public interface LogStream {
+
+ /**
+ * Returns the unique name to identify this log.
+ */
+ LogName getName();
+
+ /**
+ * Returns the size of this LogStream in bytes.
+ */
+ long getSizeInBytes();
+
+ /**
+ * Returns the number of records in this LogStream.
+ */
+ long getSizeInRecords();
+
+ /**
+ * Creates a reader to read this LogStream asynchronously.
+ *
+ * @return An asynchronous reader
+ */
+ AsyncLogReader createAsyncReader();
+
+ /**
+ * Creates a writer to write to this LogStream asynchronously.
+ *
+ * @return An asynchronous writer
+ */
+ AsyncLogWriter createAsyncWriter();
+
+ /**
+ * Creates a reader to read this LogStream synchronously.
+ *
+ * @return A synchronous reader
+ */
+ LogReader createReader();
+
+ /**
+ * Creates a write to write to this LogStream synchronously.
+ *
+ * @return A synchronous writer
+ */
+ LogWriter createWriter();
+
+ /**
+ * Removes the elements in this LogStream prior to the given recordId.
+ *
+ * @param recordId A non-negative recordId for this LogStream
+ */
+ CompletableFuture truncateBefore(long recordId);
+
+ /**
+ * Returns the recordId which is the start of the LogStream. When there
are records which were truncated
+ * from the LogStream, this will return a value larger than {@code 0}.
+ */
+ CompletableFuture getFirstRecordId();
+
--- End diff --
Similarly getlastRecordId() would be helpfull.
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16584611#comment-16584611
]
ASF GitHub Bot commented on RATIS-272:
--
Github user ankitsinghal commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r211064058
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogReader.java
---
@@ -0,0 +1,53 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+
+/**
+ * Synchronous client interface to read from a LogStream.
+ */
+public interface LogReader extends AutoCloseable {
+
--- End diff --
Need a way to get the recordId of the current record read
(getCurrentRecordId()?), it will be needed to keep a track so that we can start
from where we left after any failure (using seek)
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16584375#comment-16584375
]
Josh Elser commented on RATIS-272:
--
{quote}can you update API doc? It is hard to discuss API using source files :)
{quote}
Heh, the first document was essentially java code, so I figured source would
help. Let me try to distill what I wrote in code back to human readable and
attach here.
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[
https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16583388#comment-16583388
]
ASF GitHub Bot commented on RATIS-272:
--
Github user chrajeshbabu commented on a diff in the pull request:
https://github.com/apache/incubator-ratis/pull/4#discussion_r210808962
--- Diff:
ratis-logservice/src/main/java/org/apache/ratis/logservice/api/LogWriter.java
---
@@ -0,0 +1,43 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ratis.logservice.api;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+
+/**
+ * Synchronous client interface to write to a LogStream.
+ */
+public interface LogWriter extends AutoCloseable {
+
+ /**
+ * Appends the given data as a record in the LogStream.
+ *
+ * @param data The record to append
+ * @return The recordId for the record just written
+ */
+ long write(ByteBuffer data) throws IOException;
+
+ /**
+ * Guarantees that all previous data appended by {@link
#write(ByteBuffer)} are persisted
+ * and durable in the LogStream.
+ *
+ * @return TODO Unknown?
+ */
+ long commit() throws IOException;
--- End diff --
@joshelser This need to be changed to sync.
> Design ideal API
>
>
> Key: RATIS-272
> URL: https://issues.apache.org/jira/browse/RATIS-272
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
>Reporter: Josh Elser
>Priority: Major
>
> With influence from Apache DistributedLog, Kafka, and BookKeeper, design an
> API that balances the ideal notion of what a distribute log system should
> look like, but also considers the needs of HBase to replace a WAL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[ https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16583131#comment-16583131 ] Vladimir Rodionov commented on RATIS-272: - [~elserj], can you update API doc? It is hard to discuss API using source files :) > Design ideal API > > > Key: RATIS-272 > URL: https://issues.apache.org/jira/browse/RATIS-272 > Project: Ratis > Issue Type: Sub-task > Components: LogService >Reporter: Josh Elser >Priority: Major > > With influence from Apache DistributedLog, Kafka, and BookKeeper, design an > API that balances the ideal notion of what a distribute log system should > look like, but also considers the needs of HBase to replace a WAL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[ https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16583106#comment-16583106 ] Josh Elser commented on RATIS-272: -- [~vrodionov], [~rajeshbabu], first stab at an API. I need to keep going through this to think some more about what I missed, but feedback very welcome at this point. > Design ideal API > > > Key: RATIS-272 > URL: https://issues.apache.org/jira/browse/RATIS-272 > Project: Ratis > Issue Type: Sub-task > Components: LogService >Reporter: Josh Elser >Priority: Major > > With influence from Apache DistributedLog, Kafka, and BookKeeper, design an > API that balances the ideal notion of what a distribute log system should > look like, but also considers the needs of HBase to replace a WAL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RATIS-272) Design ideal API
[ https://issues.apache.org/jira/browse/RATIS-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16583104#comment-16583104 ] ASF GitHub Bot commented on RATIS-272: -- GitHub user joshelser opened a pull request: https://github.com/apache/incubator-ratis/pull/4 RATIS-272 logservice api WIP. Based on Rajeshbabu's first API suggestions. Also adds: * LogStream and LogService interfaces * LogName to uniquely identify a LogStream in a LogService * Listener (Tail) API for a LogStream You can merge this pull request into a Git repository by running: $ git pull https://github.com/joshelser/incubator-ratis 272-logservice-api Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-ratis/pull/4.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4 commit 2a0465c06bdbe718824d6130426b352445b36f0a Author: Josh Elser Date: 2018-08-16T16:34:23Z RATIS-272 First wag at an API for the LogService > Design ideal API > > > Key: RATIS-272 > URL: https://issues.apache.org/jira/browse/RATIS-272 > Project: Ratis > Issue Type: Sub-task > Components: LogService >Reporter: Josh Elser >Priority: Major > > With influence from Apache DistributedLog, Kafka, and BookKeeper, design an > API that balances the ideal notion of what a distribute log system should > look like, but also considers the needs of HBase to replace a WAL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
