Re: [DISCUSSION] First Release

[Sijie Guo]

Flavio,

I totally agreed that not using an official Apache  version is not great
for the community. We had a bookkeeper meetup last night. We've discussed
the current situation with the community. We came to a commitment to merge
Twitter's branch back into bookkeeper 4.5. After that we won't maintain our
own branch and switch to 4.5.

Hope this make sense.

Sijie

On Nov 16, 2016 8:04 PM, "Flavio Junqueira"  wrote:

It isn't great that DistributedLog is not using the Apache BookKeeper
release. Essentially anyone using DistributedLog today needs to use the
Twitter branch of BookKeeper, which has diverged from Apache BookKeeper.
I'm sure the changes in the Twitter branch are all great, but I'd be more
comfortable being able to rely on the Apache BookKeeper releases, which are
community driven.

I'm not going to block the release on this alone because it is important
for this project to get a first release out soon, but we need to fix DL-2.

-Flavio

> On 15 Nov 2016, at 19:02, Leigh Stewart 
wrote:
>
> +1
>
> On Tue, Nov 15, 2016 at 7:01 PM, Franck Cuny 
wrote:
>
>> +1 and I agree to not make DL-2 a blocker.
>>
>> On Tue, Nov 15, 2016 at 2:02 AM, Xi Liu  wrote:
>>
>>> +1 especially on DL-23.
>>>
>>> - Xi
>>>
>>> On Sat, Nov 12, 2016 at 2:22 AM, Khurrum Nasim 
>>> wrote:
>>>
 +1 (non-binding)

 I am also interested in participating.

 - kn

 On Tue, Nov 8, 2016 at 11:08 PM, Sijie Guo  wrote:

> I'd like to start the discussion about the first release. There are
 still a
> few discussions and pull requests outstanding. I think we need to
>> pick
 up a
> few items and cut the first release and then iterate from there. Here
>>> is
 a
> list of items that I think we should include:
>
> - DL-4 : Repackaging
 namespace
> to org.apache (the pull request is out and under reviewing)
> - DL-49 : support scala
 2.10
> and 2.11 (the review is done, need to be merged)
> - DL-23 : Move DL to
>>> depend
> on
> central maven repo. The main blocker is about the libthrift version,
 which
> is only hosted at twtter's maven repo. There is a pull request out.
>
> I am kind of thinking to not make DL-2
>  (using the official
> bookkeeper
> version) the blocker for the first release. We can cut a new release
>>> once
> that change is ready. So to decouple the release procedure between DL
>>> and
> BK.
>
> Please feel free to add any jiras that you believe it should be
>>> included
 in
> the first release.
>
> Also, is there anyone interested in being the release manager for
>> first
> release?
>
> - Sijie
>

>>>
>>
>>
>>
>> --
>> -franck
>>

[GitHub] incubator-distributedlog pull request #44: Add documentation for distributed...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-distributedlog/pull/44


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

incubator-distributedlog git commit: Add documentation for distributedlog-benchmark

[fcuny]

Repository: incubator-distributedlog
Updated Branches:
  refs/heads/master 878ae87be -> 50f6bd40a


Add documentation for distributedlog-benchmark

Add documentation for distributedlog-benchmark - quick overview of how to run 
dbench.

Author: Leigh Stewart 

Reviewers: Sijie Guo 

Closes #44 from leighst/lstewart/docs_benchmark


Project: http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/commit/50f6bd40
Tree: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/tree/50f6bd40
Diff: 
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/diff/50f6bd40

Branch: refs/heads/master
Commit: 50f6bd40a629800723a50feb66ff3de27b664ef3
Parents: 878ae87
Author: Leigh Stewart 
Authored: Wed Nov 16 20:29:19 2016 -0800
Committer: Franck Cuny 
Committed: Wed Nov 16 20:29:19 2016 -0800

--
 docs/admin_guide/loadtest.rst | 100 +
 1 file changed, 100 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/50f6bd40/docs/admin_guide/loadtest.rst
--
diff --git a/docs/admin_guide/loadtest.rst b/docs/admin_guide/loadtest.rst
new file mode 100644
index 000..e7a1d6b
--- /dev/null
+++ b/docs/admin_guide/loadtest.rst
@@ -0,0 +1,100 @@
+---
+layout: default
+
+# Top navigation
+top-nav-group: admin-guide
+top-nav-pos: 2
+top-nav-title: Load Test
+
+# Sub-level navigation
+sub-nav-group: admin-guide
+sub-nav-parent: admin-guide
+sub-nav-id: performance
+sub-nav-pos: 2
+sub-nav-title: Load Test
+---
+
+.. contents:: Load Test
+
+Load Test
+=
+
+Overview
+
+
+Under distributedlog-benchmark you will find a set of applications intended 
for generating large amounts of load in a distributedlog custer. These 
applications are suitable for load testing, performance testing, benchmarking, 
or even simply smoke testing a distributedlog cluster.
+
+The dbench script can run in several modes:
+
+1. bkwrite - Benchmark the distributedlog write path using the core library
+
+2. write - Benchmark the distributedlog write path, via write proxy, using the 
thin client
+
+3. read - Benchmark the distributedlog read path using the core library
+
+
+Running Dbench
+--
+
+The distributedlog-benchmark binary dbench is intended to be run 
simultaneously from many machines with identical settings. Together, all 
instances of dbench comprise a benchmark job. How you launch a benchmark job 
will depend on your operating environment. We recommend using a cluster 
scheduler like aurora or kubernetes to simplify the process, but tools like 
capistrano can also simplify this process greatly.
+
+The benchmark script can be found at
+
+::
+
+distributedlog-benchmark/bin/dbench
+
+Arguments may be passed to this script via environment variables. The 
available arguments depend on the execution mode. For an up to date list, check 
the script itself.
+
+
+Write to Proxy with Thin Client
+---
+
+The proxy write test (mode = `write`) can be used to send writes to a proxy 
cluster to be written to a set of streams.
+
+For example to use the proxy write test to generate 1 requests per second 
across 10 streams using 50 machines, run the following command on each machine.
+
+::
+
+STREAM_NAME_PREFIX=loadtest_
+BENCHMARK_DURATION=3600 # seconds
+DL_NAMESPACE=
+NUM_STREAMS=10
+INITIAL_RATE=200
+distributedlog-benchmark/bin/dbench write
+
+
+Write to BookKeeper with Core Library
+-
+
+The core library write test (mode = `bkwrite`) can be used to send writes to 
directly to bookkeeper using the core library.
+
+For example to use the core library write test to generate 100MBps across 10 
streams using 100 machines, run the following command on each machine.
+
+::
+
+STREAM_NAME_PREFIX=loadtest_
+BENCHMARK_DURATION=3600 # seconds
+DL_NAMESPACE=
+NUM_STREAMS=10
+INITIAL_RATE=1024
+MSG_SIZE=1024
+distributedlog-benchmark/bin/dbench bkwrite
+
+
+Read from BookKeeper with Core Library
+--
+
+The core library read test (mode = `read`) can be used to read directly from 
bookkeeper using the core library.
+
+For example to use the core library read test to read from 10 streams on 100 
instances, run the following command on each machine.
+
+::
+
+STREAM_NAME_PREFIX=loadtest_
+BENCHMARK_DURATION=3600 # seconds
+DL_NAMESPACE=
+MAX_STREAM_ID=9
+NUM_READERS_PER_STREAM=5
+TRUNCATION_INTERVAL=60 # seconds
+distributedlog-benchmark/bin/dbench read

[GitHub] incubator-distributedlog pull request #46: Update a couple of scripts

[franckcuny]

GitHub user franckcuny opened a pull request:

https://github.com/apache/incubator-distributedlog/pull/46

Update a couple of scripts

Mostly style.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/franckcuny/incubator-distributedlog 
fcuny/update-scripts

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-distributedlog/pull/46.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #46


commit 4f81ba23af8867af8b5f5afeaca86e5b2e255489
Author: Franck Cuny 
Date:   2016-11-17T04:11:57Z

Rely on git to find the root directory.

Using `rev-parse --show-toplevel` we can find the top directory. Since
we've set the '-e' option for bash, if we're not in a git repository
this will fail (which is what we want).

Add a few quotes and upper case a variable.

commit 85aa8a789c3b99f71d6fd64c960dd7bd4ffa438d
Author: Franck Cuny 
Date:   2016-11-17T04:13:39Z

Make the shell style more consistent.

The style was inconsistent, using sometimes 4 spaces or 2 spaces for
indentation; using 'then' on the same line or another line; some
variables were quoted correctly; etc.

The style is now consistent and it's easier to read the script.

Instead of blindly running `sysctl`, only do it if we're on Linux.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Proxy Client - Batch Ordering / Commit

[Sijie Guo]

Cameron,

I just granted you the permissions. You should be able to edit the wiki
pages now. Let me know if you encountered any issues.

- Sijie

On Wed, Nov 16, 2016 at 12:20 PM, Cameron Hatfield  wrote:

> I believe it is:
> https://cwiki.apache.org/confluence/display/~cameron.hatfield
>
> On Wed, Nov 16, 2016 at 12:14 PM, Sijie Guo 
> wrote:
>
> > Cameron,
> >
> > Can you send me your wiki account name? I can grant you the permission to
> > edit it.
> >
> > - Sijie
> >
> > On Wed, Nov 16, 2016 at 12:11 PM, Cameron Hatfield 
> > wrote:
> >
> > > Also, would it be possible for me to get wiki access so I will be able
> to
> > > update it / etc?
> > >
> > > -Cameron
> > >
> > > On Wed, Nov 16, 2016 at 11:59 AM, Cameron Hatfield 
> > > wrote:
> > >
> > > > "A couple of questions" is what I originally wrote, and then the
> > > following
> > > > happened. Sorry about the large swath of them, making sure my
> > > understanding
> > > > of the code base, as well as the DL/Bookkeeper/ZK ecosystem
> > interaction,
> > > > makes sense.
> > > >
> > > > ==General:
> > > > What is an exclusive session? What is it providing over a regular
> > > session?
> > > >
> > > >
> > > > ==Proxy side:
> > > > Should a new streamop be added for the fencing operation, or does it
> > make
> > > > sense to piggyback on an existing one (such as write)?
> > > >
> > > > getLastDLSN:
> > > > What should be the return result for:
> > > > A new stream
> > > > A new session, after successful fencing
> > > > A new session, after a change in ownership / first starting up
> > > >
> > > > What is the main use case for getLastDLSN(, false)? Is this
> to
> > > > guarantee that the recovery process has happened in case of ownership
> > > > failure (I don't have a good understanding of what causes the
> recovery
> > > > process to happen, especially from the reader side)? Or is it to
> handle
> > > the
> > > > lost-ack problem? Since all the rest of the read related things go
> > > through
> > > > the read client, I'm not sure if I see the use case, but it seems
> like
> > > > there would be a large potential for confusion on which to use. What
> > > about
> > > > just a fenceSession op, that always fences, returning the DLSN of the
> > > > fence, and leave the normal getLastDLSN for the regular read client.
> > > >
> > > > Fencing:
> > > > When a fence session occurs, what call needs to be made to make sure
> > any
> > > > outstanding writes are flushed and committed (so that we guarantee
> the
> > > > client will be able to read anything that was in the write queue)?
> > > > Is there a guaranteed ordering for things written in the future queue
> > for
> > > > AsyncLogWriter (I'm not quite confident that I was able to accurately
> > > > follow the logic, as their are many parts of the code that write,
> have
> > > > queues, heartbeat, etc)?
> > > >
> > > > SessionID:
> > > > What is the default sessionid / transactionid for a new stream? I
> > assume
> > > > this would just be the first control record
> > > >
> > > > ==Should all streams have a sessionid by default, regardless if
> it
> > is
> > > > never used by a client (aka, everytime ownership changes, a new
> control
> > > > record is generated, and a sessionid is stored)?
> > > > Main edge case that would have to be handled is if a client writes
> with
> > > an
> > > > old sessionid, but the owner has changed and has yet to create a
> > > sessionid.
> > > > This should be handled by the "non-matching sessionid" rule, since
> the
> > > > invalid sessionid wouldn't match the passed sessionid, which should
> > cause
> > > > the client to get a new sessionid.
> > > >
> > > > ==Where in the code does it make sense to own the session, the
> > stream
> > > > interfaces / classes? Should they pass that information down to the
> > ops,
> > > or
> > > > do the sessionid check within?
> > > > My first thought would be Stream owns the sessionid, passes it into
> the
> > > > ops (as either a nullable value, or an invalid default value), which
> > then
> > > > do the sessionid check if they care. The main issue is updating the
> > > > sessionid is a bit backwards, as either every op has the ability to
> > > update
> > > > it through some type of return value / direct stream access / etc, or
> > > there
> > > > is a special case in the stream for the fence operation / any other
> > > > operation that can update the session.
> > > >
> > > > ==For "the owner of the log stream will first advance the
> > transaction
> > > > id generator to claim a new transaction id and write a control record
> > to
> > > > the log stream. ":
> > > > Should "DistributedLogConstants.CONTROL_RECORD_CONTENT" be the type
> of
> > > > control record written?
> > > > Should the "writeControlRecord" on the BKAsyncLogWriter be exposed in
> > the
> > > > AsyncLogWriter interface be exposed?  Or even in the one within the
> > > 

Re: Proxy Client - Batch Ordering / Commit

[Cameron Hatfield]

I believe it is:
https://cwiki.apache.org/confluence/display/~cameron.hatfield

On Wed, Nov 16, 2016 at 12:14 PM, Sijie Guo 
wrote:

> Cameron,
>
> Can you send me your wiki account name? I can grant you the permission to
> edit it.
>
> - Sijie
>
> On Wed, Nov 16, 2016 at 12:11 PM, Cameron Hatfield 
> wrote:
>
> > Also, would it be possible for me to get wiki access so I will be able to
> > update it / etc?
> >
> > -Cameron
> >
> > On Wed, Nov 16, 2016 at 11:59 AM, Cameron Hatfield 
> > wrote:
> >
> > > "A couple of questions" is what I originally wrote, and then the
> > following
> > > happened. Sorry about the large swath of them, making sure my
> > understanding
> > > of the code base, as well as the DL/Bookkeeper/ZK ecosystem
> interaction,
> > > makes sense.
> > >
> > > ==General:
> > > What is an exclusive session? What is it providing over a regular
> > session?
> > >
> > >
> > > ==Proxy side:
> > > Should a new streamop be added for the fencing operation, or does it
> make
> > > sense to piggyback on an existing one (such as write)?
> > >
> > > getLastDLSN:
> > > What should be the return result for:
> > > A new stream
> > > A new session, after successful fencing
> > > A new session, after a change in ownership / first starting up
> > >
> > > What is the main use case for getLastDLSN(, false)? Is this to
> > > guarantee that the recovery process has happened in case of ownership
> > > failure (I don't have a good understanding of what causes the recovery
> > > process to happen, especially from the reader side)? Or is it to handle
> > the
> > > lost-ack problem? Since all the rest of the read related things go
> > through
> > > the read client, I'm not sure if I see the use case, but it seems like
> > > there would be a large potential for confusion on which to use. What
> > about
> > > just a fenceSession op, that always fences, returning the DLSN of the
> > > fence, and leave the normal getLastDLSN for the regular read client.
> > >
> > > Fencing:
> > > When a fence session occurs, what call needs to be made to make sure
> any
> > > outstanding writes are flushed and committed (so that we guarantee the
> > > client will be able to read anything that was in the write queue)?
> > > Is there a guaranteed ordering for things written in the future queue
> for
> > > AsyncLogWriter (I'm not quite confident that I was able to accurately
> > > follow the logic, as their are many parts of the code that write, have
> > > queues, heartbeat, etc)?
> > >
> > > SessionID:
> > > What is the default sessionid / transactionid for a new stream? I
> assume
> > > this would just be the first control record
> > >
> > > ==Should all streams have a sessionid by default, regardless if it
> is
> > > never used by a client (aka, everytime ownership changes, a new control
> > > record is generated, and a sessionid is stored)?
> > > Main edge case that would have to be handled is if a client writes with
> > an
> > > old sessionid, but the owner has changed and has yet to create a
> > sessionid.
> > > This should be handled by the "non-matching sessionid" rule, since the
> > > invalid sessionid wouldn't match the passed sessionid, which should
> cause
> > > the client to get a new sessionid.
> > >
> > > ==Where in the code does it make sense to own the session, the
> stream
> > > interfaces / classes? Should they pass that information down to the
> ops,
> > or
> > > do the sessionid check within?
> > > My first thought would be Stream owns the sessionid, passes it into the
> > > ops (as either a nullable value, or an invalid default value), which
> then
> > > do the sessionid check if they care. The main issue is updating the
> > > sessionid is a bit backwards, as either every op has the ability to
> > update
> > > it through some type of return value / direct stream access / etc, or
> > there
> > > is a special case in the stream for the fence operation / any other
> > > operation that can update the session.
> > >
> > > ==For "the owner of the log stream will first advance the
> transaction
> > > id generator to claim a new transaction id and write a control record
> to
> > > the log stream. ":
> > > Should "DistributedLogConstants.CONTROL_RECORD_CONTENT" be the type of
> > > control record written?
> > > Should the "writeControlRecord" on the BKAsyncLogWriter be exposed in
> the
> > > AsyncLogWriter interface be exposed?  Or even in the one within the
> > segment
> > > writer? Or should the code be duplicated / pulled out into a helper /
> > etc?
> > > (Not a big Java person, so any suggestions on the "Java Way", or at
> least
> > > the DL way, to do it would be appreciated)
> > >
> > > ==Transaction ID:
> > > The BKLogSegmentWriter ignores the transaction ids from control records
> > > when it records the "LastTXId." Would that be an issue here for
> anything?
> > > It looks like it may do that because it assumes you're calling 

Re: Proxy Client - Batch Ordering / Commit

[Cameron Hatfield]

Also, would it be possible for me to get wiki access so I will be able to
update it / etc?

-Cameron

On Wed, Nov 16, 2016 at 11:59 AM, Cameron Hatfield  wrote:

> "A couple of questions" is what I originally wrote, and then the following
> happened. Sorry about the large swath of them, making sure my understanding
> of the code base, as well as the DL/Bookkeeper/ZK ecosystem interaction,
> makes sense.
>
> ==General:
> What is an exclusive session? What is it providing over a regular session?
>
>
> ==Proxy side:
> Should a new streamop be added for the fencing operation, or does it make
> sense to piggyback on an existing one (such as write)?
>
> getLastDLSN:
> What should be the return result for:
> A new stream
> A new session, after successful fencing
> A new session, after a change in ownership / first starting up
>
> What is the main use case for getLastDLSN(, false)? Is this to
> guarantee that the recovery process has happened in case of ownership
> failure (I don't have a good understanding of what causes the recovery
> process to happen, especially from the reader side)? Or is it to handle the
> lost-ack problem? Since all the rest of the read related things go through
> the read client, I'm not sure if I see the use case, but it seems like
> there would be a large potential for confusion on which to use. What about
> just a fenceSession op, that always fences, returning the DLSN of the
> fence, and leave the normal getLastDLSN for the regular read client.
>
> Fencing:
> When a fence session occurs, what call needs to be made to make sure any
> outstanding writes are flushed and committed (so that we guarantee the
> client will be able to read anything that was in the write queue)?
> Is there a guaranteed ordering for things written in the future queue for
> AsyncLogWriter (I'm not quite confident that I was able to accurately
> follow the logic, as their are many parts of the code that write, have
> queues, heartbeat, etc)?
>
> SessionID:
> What is the default sessionid / transactionid for a new stream? I assume
> this would just be the first control record
>
> ==Should all streams have a sessionid by default, regardless if it is
> never used by a client (aka, everytime ownership changes, a new control
> record is generated, and a sessionid is stored)?
> Main edge case that would have to be handled is if a client writes with an
> old sessionid, but the owner has changed and has yet to create a sessionid.
> This should be handled by the "non-matching sessionid" rule, since the
> invalid sessionid wouldn't match the passed sessionid, which should cause
> the client to get a new sessionid.
>
> ==Where in the code does it make sense to own the session, the stream
> interfaces / classes? Should they pass that information down to the ops, or
> do the sessionid check within?
> My first thought would be Stream owns the sessionid, passes it into the
> ops (as either a nullable value, or an invalid default value), which then
> do the sessionid check if they care. The main issue is updating the
> sessionid is a bit backwards, as either every op has the ability to update
> it through some type of return value / direct stream access / etc, or there
> is a special case in the stream for the fence operation / any other
> operation that can update the session.
>
> ==For "the owner of the log stream will first advance the transaction
> id generator to claim a new transaction id and write a control record to
> the log stream. ":
> Should "DistributedLogConstants.CONTROL_RECORD_CONTENT" be the type of
> control record written?
> Should the "writeControlRecord" on the BKAsyncLogWriter be exposed in the
> AsyncLogWriter interface be exposed?  Or even in the one within the segment
> writer? Or should the code be duplicated / pulled out into a helper / etc?
> (Not a big Java person, so any suggestions on the "Java Way", or at least
> the DL way, to do it would be appreciated)
>
> ==Transaction ID:
> The BKLogSegmentWriter ignores the transaction ids from control records
> when it records the "LastTXId." Would that be an issue here for anything?
> It looks like it may do that because it assumes you're calling it's local
> function for writing a controlrecord, which uses the lastTxId.
>
>
> ==Thrift Interface:
> Should the write response be split out for different calls?
> It seems odd to have a single struct with many optional items that are
> filled depending on the call made for every rpc call. This is mostly a
> curiosity question, since I assume it comes from the general practices from
> using thrift for a while. Would it at least make sense for the
> getLastDLSN/fence endpoint to have a new struct?
>
> Any particular error code that makes sense for session fenced? If we
> want to be close to the HTTP errors, looks like 412 (PRECONDITION FAILED)
> might make the most sense, if a bit generic.
>
> 412 def:
> "The precondition given in one or more of the