Re: Is there a page like "who is using BK/DL"?
Sounds a good idea! Let's create one for that purpose. Sijie On Dec 5, 2016 6:00 PM, "Jia Zhai" wrote: > Yes. That is what I mean. By it, we may get more direct info about who is > intresting and want to try on it, not only who has used it for big case. > > On Tue, Dec 6, 2016 at 4:21 AM, Sijie Guo wrote: > > > I think both BK and DL have the poweredby cwiki page. Do you mean we > open a > > dummy github issue for people to comment to register themselves? > > > > - Sijie > > > > On Sun, Dec 4, 2016 at 6:50 PM, Jia Zhai wrote: > > > > > Hi guys, > > > I found that in project RocketMQ, there is an open issue: > > > https://github.com/alibaba/RocketMQ/issues/1 , in which a lot of user > / > > > developer registered themself. This may bring benifit of connecting > > > between usr, developor and community. > > > > > > Is there a need for us to have such a similar issue or page related to > > > BK/DL? > > > > > > Regards. > > > -Jia > > > > > >
Re: Is there a page like "who is using BK/DL"?
Yes. That is what I mean. By it, we may get more direct info about who is intresting and want to try on it, not only who has used it for big case. On Tue, Dec 6, 2016 at 4:21 AM, Sijie Guo wrote: > I think both BK and DL have the poweredby cwiki page. Do you mean we open a > dummy github issue for people to comment to register themselves? > > - Sijie > > On Sun, Dec 4, 2016 at 6:50 PM, Jia Zhai wrote: > > > Hi guys, > > I found that in project RocketMQ, there is an open issue: > > https://github.com/alibaba/RocketMQ/issues/1 , in which a lot of user / > > developer registered themself. This may bring benifit of connecting > > between usr, developor and community. > > > > Is there a need for us to have such a similar issue or page related to > > BK/DL? > > > > Regards. > > -Jia > > >
[jira] [Assigned] (DL-45) DL should allow ByteBuffer based API and should avoid copying of arrays
[ https://issues.apache.org/jira/browse/DL-45?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Zhai reassigned DL-45: -- Assignee: Jia Zhai > DL should allow ByteBuffer based API and should avoid copying of arrays > --- > > Key: DL-45 > URL: https://issues.apache.org/jira/browse/DL-45 > Project: DistributedLog > Issue Type: Improvement > Components: distributedlog-core, distributedlog-protocol >Reporter: Arvind Kandhare >Assignee: Jia Zhai > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (DL-46) DL should have an API which supports atomic merging of two logs
[ https://issues.apache.org/jira/browse/DL-46?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Zhai reassigned DL-46: -- Assignee: Jia Zhai > DL should have an API which supports atomic merging of two logs > --- > > Key: DL-46 > URL: https://issues.apache.org/jira/browse/DL-46 > Project: DistributedLog > Issue Type: Improvement >Reporter: Arvind Kandhare >Assignee: Jia Zhai > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Is there a page like "who is using BK/DL"?
I think both BK and DL have the poweredby cwiki page. Do you mean we open a dummy github issue for people to comment to register themselves? - Sijie On Sun, Dec 4, 2016 at 6:50 PM, Jia Zhai wrote: > Hi guys, > I found that in project RocketMQ, there is an open issue: > https://github.com/alibaba/RocketMQ/issues/1 , in which a lot of user / > developer registered themself. This may bring benifit of connecting > between usr, developor and community. > > Is there a need for us to have such a similar issue or page related to > BK/DL? > > Regards. > -Jia >
Re: Proxy Client - Batch Ordering / Commit
Great discussion here :) Have you started any work for this? I just updated the proposal page - > https://cwiki.apache.org/confluence/display/DL/DP-2+-+Epoch+Write+Support > Maybe we can work together with this. Looks good Xi. If I understand the proposal correctly, this gets us, in thin client: a) ability to achieve a very strict form of consistency, supporting ex. exactly once updates b) exclusive ownership I think there may also be a need for some kind of large atomic update functionality, which is a different way of looking at consistent/transactional updates. Cameron earlier Sijie asked if you need > 1MB writes - is that a requirement for you? If so, epoch write may not meet all of your requirements above. We actually have a fairly urgent business need to support something like this. Our two use cases are: 1. apply a set of logically separate writes as an atomic batch (we want them all to be visible at the same time, or not at all). might be useful to be able to aggregate updates and apply at a time of our choosing. 2. large writes: dl write now has a 1MB write limit. we will need to apply updates which are larger than 1MB atomically. On Fri, Nov 18, 2016 at 2:37 PM, Sijie Guo wrote: > On Thu, Nov 17, 2016 at 2:30 AM, Xi Liu wrote: > > > Cameron, > > > > Thank you for your comments. It's very helpful. My replies are inline. > > > > On Wed, Nov 16, 2016 at 11:59 AM, Cameron Hatfield > > wrote: > > > > > "A couple of questions" is what I originally wrote, and then the > > following > > > happened. Sorry about the large swath of them, making sure my > > understanding > > > of the code base, as well as the DL/Bookkeeper/ZK ecosystem > interaction, > > > makes sense. > > > > > > ==General: > > > What is an exclusive session? What is it providing over a regular > > session? > > > > > > > The idea here is to provide exclusive writer semantic for the > > distributedlog (thin) client use case. > > So it can have similar behavior as using the storage > (distributedlog-core) > > library directly. > > > > +1 for an exclusive writer feature. Leigh and me were talking about this > before. It is glad to see it is happening now. However it might be worth to > separate the exclusive writer feature into a separate task once we have > 'fencing' feature available. > > > > > > > > > > > > > > > ==Proxy side: > > > Should a new streamop be added for the fencing operation, or does it > make > > > sense to piggyback on an existing one (such as write)? > > > > > > > getLastDLSN: > > > What should be the return result for: > > > A new stream > > > A new session, after successful fencing > > > A new session, after a change in ownership / first starting up > > > > > > What is the main use case for getLastDLSN(, false)? Is this to > > > guarantee that the recovery process has happened in case of ownership > > > failure (I don't have a good understanding of what causes the recovery > > > process to happen, especially from the reader side)? Or is it to handle > > the > > > lost-ack problem? Since all the rest of the read related things go > > through > > > the read client, I'm not sure if I see the use case, but it seems like > > > there would be a large potential for confusion on which to use. What > > about > > > just a fenceSession op, that always fences, returning the DLSN of the > > > fence, and leave the normal getLastDLSN for the regular read client. > > > > > > > > > Ah, your point is valid. I was following the style of bookkeeper. The > > bookkeeper client supplies a fencing flag on readLastAddConfirmed request > > during recovery. > > > > But you are right. It is clear to have just a fenceSessionOp and return > the > > DLSN of the fence request. > > > > > Correct. In bookkeeper client, the fencing flag is set with a read op. > However it is part of the recovery procedure and internal to the client. I > agree with Cameron that we should hide the details from public. Otherwise, > it will cause confusion. > > > > > > > > > > > > > > Fencing: > > > When a fence session occurs, what call needs to be made to make sure > any > > > outstanding writes are flushed and committed (so that we guarantee the > > > client will be able to read anything that was in the write queue)? > > > Is there a guaranteed ordering for things written in the future queue > for > > > AsyncLogWriter (I'm not quite confident that I was able to accurately > > > follow the logic, as their are many parts of the code that write, have > > > queues, heartbeat, etc)? > > > > > > > I believed that when calling AsyncLogWriter asyncWrite in order, the > > records will be written in order. Sijie or Leigh can confirm that. > > > > That's write. The writes are guaranteed to write in the order of how they > are issues. > > > > > > Since the writes are in order, when a fence session occurs, the control > > record written successfully by the writer will guarantee the writes > called > > before the fence session write are flushed and committed. > > > >
[jira] [Commented] (DL-84) Reduce HashedWheelTimer instance number
[
https://issues.apache.org/jira/browse/DL-84?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15722632#comment-15722632
]
Leigh Stewart commented on DL-84:
-
First two are client, second two are server. Client and server are separate
processes - however you could *probably* reuse the timer in client and server
to get to one on either side :)
> Reduce HashedWheelTimer instance number
> ---
>
> Key: DL-84
> URL: https://issues.apache.org/jira/browse/DL-84
> Project: DistributedLog
> Issue Type: Improvement
> Components: distributedlog-client, distributedlog-core,
> distributedlog-service
>Affects Versions: 0.4.0, 0.5.0
>Reporter: Liang Xie
>Assignee: Liang Xie
>
> Reading code and finding that we use HashedWheelTimer in OwnershipCache
> Class, from netty HashedWheelTimer java doc we know:
> {quote}
> Do not create many instances.
> HashedWheelTimer creates a new thread whenever it is instantiated and
> started. Therefore, you should make sure to create only one instance and
> share it across your application
> {quote}
> then i grep the code base and found there're several HashedWheelTimer
> instantces be created seems. i think the current code could be improved (to
> reduce the thread num at least?)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (DL-84) Reduce HashedWheelTimer instance number
[
https://issues.apache.org/jira/browse/DL-84?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15721557#comment-15721557
]
Liang Xie commented on DL-84:
-
grep -R "new HashedWheelTimer" .|grep -v "test"
./distributedlog-client/src/main/java/com/twitter/distributedlog/client/routing/ConsistentHashRoutingService.java:
this.hashedWheelTimer = new HashedWheelTimer(new ThreadFactoryBuilder()
./distributedlog-client/src/main/java/com/twitter/distributedlog/client/DistributedLogClientImpl.java:
this.dlTimer = new HashedWheelTimer(
./distributedlog-core/src/main/java/com/twitter/distributedlog/BKDistributedLogNamespace.java:
this.requestTimer = new HashedWheelTimer(
./distributedlog-service/src/main/java/com/twitter/distributedlog/service/DistributedLogServiceImpl.java:
this.requestTimer = new HashedWheelTimer(
> Reduce HashedWheelTimer instance number
> ---
>
> Key: DL-84
> URL: https://issues.apache.org/jira/browse/DL-84
> Project: DistributedLog
> Issue Type: Improvement
> Components: distributedlog-client, distributedlog-core,
> distributedlog-service
>Affects Versions: 0.4.0, 0.5.0
>Reporter: Liang Xie
>Assignee: Liang Xie
>
> Reading code and finding that we use HashedWheelTimer in OwnershipCache
> Class, from netty HashedWheelTimer java doc we know:
> {quote}
> Do not create many instances.
> HashedWheelTimer creates a new thread whenever it is instantiated and
> started. Therefore, you should make sure to create only one instance and
> share it across your application
> {quote}
> then i grep the code base and found there're several HashedWheelTimer
> instantces be created seems. i think the current code could be improved (to
> reduce the thread num at least?)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
