Re: [DISCUSS] using protobuf than thrift
one question - bk is using protobuf 2.x while gRPC is using 3.x. IMO, they are not backward compatible. Are you also considering moving bk's protobuf to 3.x? On Thu, Jan 5, 2017 at 10:29 PM, Gerrit Sundaram wrote: > Hello all, > > for the comment in > https://github.com/apache/incubator-distributedlog/pull/99, I am starting > this email thread for discussing using protobuf to store metadata for ease > extension. > > I have a few reasons for using protobuf rather than using thrift: > > - bookkeeper is using protobuf for storing metadata. so there is no extra > dependency. and it will make things consistent. > - the thrift version that DL is using now is 0.5.0-1, which is an > out-of-date thrift version and seems to be a special version that Twitter > customized for finagle. it makes me impossible to build a c++ client to > access DL. > - using protobuf, I can easily write a gRPC request handler for current > proxy service to support c++. > > Any thoughts? > > - Sijie >
Re: vote process for proposals?
I think there is not a lot of activities on proposals. A 'lazy approval' might be just good enough - a proposal with lazy approval is implicitly allowed/accepted unless a -1 vote is received. That's probably the best for now. what do you think? On Wed, Jan 4, 2017 at 12:50 AM, Sijie Guo wrote: > Ping? > > Xi, Jon, any updates about this? Do any of you want to drive this? > > - Sijie > > > > On Thu, Dec 15, 2016 at 11:18 AM, Sijie Guo > wrote: > > > Xi, Jon, are any of you interested in making a draft about about the > > proposal workflow? > > > > On Wed, Dec 14, 2016 at 6:14 PM, Jon Derrick < > jonathan.derri...@gmail.com> > > wrote: > > > > > I think it really worth having a voting proposal, as sometime I might > > lose > > > track of if a proposal is accepted or not and whether it is under > > > development. > > > > > > Beam's process looks promising. You can try to start with that. > > > > > > Another suggestion is it would be awesome if the DL jira queue can have > > new > > > type, called 'Proposal'. Then we can enforce the proposal workflow in > the > > > jira. > > > > > > > I think it is possible to ask INFRA team to create a new jira > type/workflow > > for us, if we can come up with more details. Can you tell us more about > > your thoughts? > > > > - Sijie > > > > > > > > > > On Tue, Dec 13, 2016 at 1:10 AM, Xi Liu wrote: > > > > > > > Thank you Sijie. I feel it is good to have a voting process, so that > it > > > > would be good to track if a proposal is accepted for developing or > > > > discarded due to any reasons. I will start with my proposal and see > how > > > it > > > > is going with the community. > > > > > > > > - Xi > > > > > > > > On Thu, Dec 8, 2016 at 9:11 PM, Sijie Guo wrote: > > > > > > > > > Xi, thank you for raising this up. I don't think we have a formal > > > process > > > > > for track proposals. I think we can learn the proposals from other > > > apache > > > > > projects. For example, beam has very nice documentation on > > contribution > > > > > guide (http://beam.incubator.apache.org/contribute/contribution- > > guide/ > > > ). > > > > > We > > > > > probably can adopt it. > > > > > > > > > > I don't feel strong about the voting process. If it is easier for > > > making > > > > > conclusion on the proposal discussion, let's vote for any discussed > > > > > proposal. > > > > > > > > > > - Sijie > > > > > > > > > > > > > > > > > > > > On Thu, Dec 8, 2016 at 9:10 AM, Xi Liu > wrote: > > > > > > > > > > > Hi all, > > > > > > > > > > > > It is great that we have a process to track/discuss proposals. > but > > > the > > > > > > process is still a bit unclear to me. do we need a vote phase to > > > adopt > > > > > the > > > > > > proposals? and shall we document the process in wiki page? > > > > > > > > > > > > my basic understand about the process is: > > > > > > > > > > > > - create a proposal wiki page to describe the proposal > > > > > > - start the '[discussion]' email thread for the proposal > > > > > > - conversation will happen in the '[discussion]' email thread and > > the > > > > > wiki > > > > > > page will be refined > > > > > > > > > > > > I feel there will be a phase to decide whether this proposal will > > be > > > > > > accepted or discarded and update the state of the proposals. > shall > > I > > > > vote > > > > > > DP-2? > > > > > > > > > > > > - Xi > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > - jderrick > > > > > >
Re: [Review] The first release of Apache DistributedLog
It would be great to include any performance numbers. On Thu, Jan 5, 2017 at 5:51 PM, Sijie Guo wrote: > Cool to see you here, Enrico. And thank you for your suggestion. > > I will try to write a separate one for DL and BK. Try to keep this one > focus on a short release post. > > - Sijie > > On Thu, Jan 5, 2017 at 1:43 PM, Enrico Olivelli > wrote: > > > Hi Sijie, > > I am following this release and the great work DL comunity is doing. > > Maybe it would be worth to write some paragraph about the difference from > > BookKeeper and/or the relation with it. > > > > > > Enrico > > > > Il gio 5 gen 2017, 19:13 Asko Kauppi ha > scritto: > > > > > Hi Sijie, > > > > > > most readers will likely not know about DistributedLog. A short > > comparison > > > - or a link to one - e.g. with Kafka might help set the stage for them. > > > i.e. why does it exist? > > > > > > This is even more important since Uber recently also publicized their > > > persistent message bus solution. If these start dropping down, there > > needs > > > to be more (technical) reason than just another vendor opening their > > chest. > > > :) > > > > > > Other than that, the structure seemed nice but it can be slightly > > shorter. > > > > > > Just my 2c > > > - asko > > > > > > > > > On 4 January 2017 at 10:38, Sijie Guo wrote: > > > > > > > I drafted a blog post for announcing the first apache release. Here > is > > > the > > > > draft. Please help review it :D > > > > > > > > https://docs.google.com/document/d/1IXVmP2cHkf4ydeUHUJN9p5ZWTpA1a > > > > uwBhfqMnYBu4A0/edit > > > > > > > > - Sijie > > > > > > > > > -- > > > > > > -- Enrico Olivelli > > >
Re: [Discuss] Transaction Support
Asko and Sijie, Thank you so much for your feedbacks. We are not targeting at building a general XA transaction coordinator. The feature we want is be able to write data to multiple log streams in an atomic way. I totally agreed with you about building minimal logic. We also don't want to enforce this feature to all the users of DL. Building the TC as a separated service sounds clear to me. We will do it follow your suggestion. I am also replying the comments to you and Leigh on the doc. Hopefully we can come to an agreement so that our changes can be accepted. - Xi On Wed, Jan 4, 2017 at 1:14 AM, Asko Kauppi wrote: > > Beside that, I have one general question - What is the major goal for > this > > feature? Are you targeting on building a general XA transaction > coordinator > > or just for supporting things like `copy-modify-write' style workflow? > > The use case I would have for transactions - at some level of the stack - > is supporting dynamic configurations. > > If a config changes in e.g. three lines, some of the changes may logically > belong together. E.g. changing both “host” and “port” (if separate > entries). One shouldn’t be able to read a state, even temporarily, that has > new host but old port. > > I can do this in the application level - it does not need to be part of > the DL protocol. > > > Asko Kauppi > Zalando Tech Helsinki > > > On 4 Jan 2017, at 9.18, Sijie Guo wrote: > > > > Sorry for late response. I think Leigh and you already had some very > > valuable discussions in the doc. I will try to add some of my questions > to > > the discussion. > > > > Beside that, I had a discussion with Leigh today about this. first of > all, > > I think it is very good to add transaction support in distributedlog. It > is > > one of the primitives that would help building distributed service. But > we > > have a concern about making this system become complicated and introduce > > operational overhead when it runs in the large scale system on > production. > > There are two major suggestions that I have for this feature - > > > > Build the 'minimum' logic in core - I think the minimum logic that need > to > > be added to the core is - the special control records (begin, commit and > > abort) and make the reader be able to detect those special control > records > > and know what do they mean and how to interrupt with them. Since they are > > special control records, there is not overhead to other readers that > > doesn't require this feature. > > > > Build the transaction coordinator as a separated proxy service - I think > > the major concern that we have is putting more complexities into the > 'write > > proxy' service. We architected distributedlog in a more microservice-like > > way - we have the core as the stream store, the proxy for serving write > and > > read traffic. It would be good that the transaction feature can be done > in > > a similar way. So the architecture would be like this - > > > > *[ write service ] [ read service ] [ transaction coordinator ]* > > *[ stream store > >]* > > > > if people doesn't need the transaction feature, they can turn if off > > completely without any operational overhead. > > > > Beside that, I have one general question - What is the major goal for > this > > feature? Are you targeting on building a general XA transaction > coordinator > > or just for supporting things like `copy-modify-write' style workflow? > > > > > > Thanks, > > Sijie > > > > > > > > > > > > On Wed, Dec 28, 2016 at 1:12 PM, Xi Liu wrote: > > > >> Ping? > >> > >> On Mon, Dec 19, 2016 at 8:28 AM, Xi Liu wrote: > >> > >>> Sijie, > >>> > >>> No. I thought it might be easier for people to comment on a google doc > to > >>> gather the initial feedback. I will put the content back to wiki page > >> once > >>> addressing the comments. Does that sound good to you? > >>> > >>> And thank you in advance. > >>> > >>> - Xi > >>> > >>> > >>> > >>> On Sun, Dec 18, 2016 at 8:48 AM, Sijie Guo wrote: > >>> > Hi Xi, > > sorry for late response. I will review it soon. > > regarding this, a separate question "are we going to use google doc > instead > of email thread for any discussion"? I am a bit worried that the > discussion > will become lost after moving to google doc. No idea on how other > apache > projects are doing. > > - Sijie > > On Wed, Dec 14, 2016 at 11:41 PM, Xi Liu > wrote: > > > Hi all, > > > > I finalized the first version of the design. This time I used a > google > doc > > so that it is easier for commenting and add a link the wiki page. I > >> will > > update this to the wiki page once we come to the finalized design. > > > > https://docs.google.com/document/d/14Ns05M8Z5a6DF6fHmWQwISyD5jjeK > > bSIGgSzXuTI5BA/edit > > > > Let me know if you have any questions. Appreciate your reviews! > > > > - Xi > > > > > > > > >
Re: [DISCUSS] using protobuf than thrift
Sijie, sorry. I typed your name in the wrong line. was planning to mention since you raised the comment in the pull request. On Thu, Jan 5, 2017 at 10:29 PM, Gerrit Sundaram wrote: > Hello all, > > for the comment in https://github.com/apache/ > incubator-distributedlog/pull/99, I am starting this email thread for > discussing using protobuf to store metadata for ease extension. > > I have a few reasons for using protobuf rather than using thrift: > > - bookkeeper is using protobuf for storing metadata. so there is no extra > dependency. and it will make things consistent. > - the thrift version that DL is using now is 0.5.0-1, which is an > out-of-date thrift version and seems to be a special version that Twitter > customized for finagle. it makes me impossible to build a c++ client to > access DL. > - using protobuf, I can easily write a gRPC request handler for current > proxy service to support c++. > > Any thoughts? > > - Sijie >
[DISCUSS] using protobuf than thrift
Hello all, for the comment in https://github.com/apache/incubator-distributedlog/pull/99, I am starting this email thread for discussing using protobuf to store metadata for ease extension. I have a few reasons for using protobuf rather than using thrift: - bookkeeper is using protobuf for storing metadata. so there is no extra dependency. and it will make things consistent. - the thrift version that DL is using now is 0.5.0-1, which is an out-of-date thrift version and seems to be a special version that Twitter customized for finagle. it makes me impossible to build a c++ client to access DL. - using protobuf, I can easily write a gRPC request handler for current proxy service to support c++. Any thoughts? - Sijie
[GitHub] incubator-distributedlog issue #99: DL-172: Provide a IDL definition for str...
Github user gerritsundaram commented on the issue: https://github.com/apache/incubator-distributedlog/pull/99 @sijie will start a discussion thread for the protobuf. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-distributedlog issue #99: DL-172: Provide a IDL definition for str...
Github user gerritsundaram commented on the issue: https://github.com/apache/incubator-distributedlog/pull/99 @sijie sorry my bad. exclude the generated files from findbugs --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-distributedlog pull request #105: DL-152: removed symlink distribu...
GitHub user adamtracymartin opened a pull request: https://github.com/apache/incubator-distributedlog/pull/105 DL-152: removed symlink distributedlog-service/conf and copied files from distributedlog-core/conf to distributed-service/conf â¦tributedlog-core/conf to distributedlog-service/conf You can merge this pull request into a Git repository by running: $ git pull https://github.com/adamtracymartin/incubator-distributedlog DL-152 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-distributedlog/pull/105.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #105 commit 0b933493863487060876a875ccfdc892fa6d0a55 Author: adamtracymartin Date: 2017-01-06T03:19:08Z removed symlink distributedlog-service/conf and copied files from distributedlog-core/conf to distributedlog-service/conf --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-distributedlog pull request #104: DL-174: added getParent method t...
GitHub user adamtracymartin opened a pull request: https://github.com/apache/incubator-distributedlog/pull/104 DL-174: added getParent method to Utils to replace usage of File.getParent which changes / to \ on Windows â¦ich changes / to \ in Windows. You can merge this pull request into a Git repository by running: $ git pull https://github.com/adamtracymartin/incubator-distributedlog DL-174 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-distributedlog/pull/104.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #104 commit ba70dec41ae7cdac839f01048459f7eef893f6f8 Author: adamtracymartin Date: 2017-01-06T03:03:01Z added getParent method to Utils to replace usage of File.getParent which changes / to \ in Windows. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-distributedlog pull request #103: DL-173: changed FileUtils.delete...
GitHub user adamtracymartin opened a pull request: https://github.com/apache/incubator-distributedlog/pull/103 DL-173: changed FileUtils.deleteDirectory to FileUtils.forceDeleteOnExit so directory will delete on Windows. You can merge this pull request into a Git repository by running: $ git pull https://github.com/adamtracymartin/incubator-distributedlog DL-173 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-distributedlog/pull/103.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #103 commit db04e15ef18ce27fe240303d3325127941e99aec Author: adamtracymartin Date: 2017-01-05T21:05:48Z changed // comments to /* */ commit 2783d26325378def9c8d4f77de1a3498709dfcc4 Author: adamtracymartin Date: 2017-01-06T02:28:41Z changed FileUtils.deleteDirectory to FileUtils.forceDeleteOnExit so directory will delete on Windows. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: [Review] The first release of Apache DistributedLog
Cool to see you here, Enrico. And thank you for your suggestion. I will try to write a separate one for DL and BK. Try to keep this one focus on a short release post. - Sijie On Thu, Jan 5, 2017 at 1:43 PM, Enrico Olivelli wrote: > Hi Sijie, > I am following this release and the great work DL comunity is doing. > Maybe it would be worth to write some paragraph about the difference from > BookKeeper and/or the relation with it. > > > Enrico > > Il gio 5 gen 2017, 19:13 Asko Kauppi ha scritto: > > > Hi Sijie, > > > > most readers will likely not know about DistributedLog. A short > comparison > > - or a link to one - e.g. with Kafka might help set the stage for them. > > i.e. why does it exist? > > > > This is even more important since Uber recently also publicized their > > persistent message bus solution. If these start dropping down, there > needs > > to be more (technical) reason than just another vendor opening their > chest. > > :) > > > > Other than that, the structure seemed nice but it can be slightly > shorter. > > > > Just my 2c > > - asko > > > > > > On 4 January 2017 at 10:38, Sijie Guo wrote: > > > > > I drafted a blog post for announcing the first apache release. Here is > > the > > > draft. Please help review it :D > > > > > > https://docs.google.com/document/d/1IXVmP2cHkf4ydeUHUJN9p5ZWTpA1a > > > uwBhfqMnYBu4A0/edit > > > > > > - Sijie > > > > > > -- > > > -- Enrico Olivelli >
Re: [Review] The first release of Apache DistributedLog
Thank you for your suggestion, Asko. Try to make the announcement post shorter and keep it focused on release. Will try to prepare another post for education. - Sijie On Thu, Jan 5, 2017 at 10:13 AM, Asko Kauppi wrote: > Hi Sijie, > > most readers will likely not know about DistributedLog. A short comparison > - or a link to one - e.g. with Kafka might help set the stage for them. > i.e. why does it exist? > > This is even more important since Uber recently also publicized their > persistent message bus solution. If these start dropping down, there needs > to be more (technical) reason than just another vendor opening their chest. > :) > > Other than that, the structure seemed nice but it can be slightly shorter. > > Just my 2c > - asko > > > On 4 January 2017 at 10:38, Sijie Guo wrote: > > > I drafted a blog post for announcing the first apache release. Here is > the > > draft. Please help review it :D > > > > https://docs.google.com/document/d/1IXVmP2cHkf4ydeUHUJN9p5ZWTpA1a > > uwBhfqMnYBu4A0/edit > > > > - Sijie > > >
[GitHub] incubator-distributedlog pull request #102: DL-176: Rename the DL artifact f...
GitHub user sijie opened a pull request: https://github.com/apache/incubator-distributedlog/pull/102 DL-176: Rename the DL artifact from com.twitter to org.apache.distributedlog You can merge this pull request into a Git repository by running: $ git pull https://github.com/sijie/incubator-distributedlog sijie/fix_pom_file_layout Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-distributedlog/pull/102.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #102 commit bcbeaf1bb701256f59206962250b484706398e2d Author: Sijie Guo Date: 2017-01-06T01:43:06Z DL-176: Rename the DL artifact from com.twitter to org.apache.distributedlog --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-distributedlog issue #95: DL-4: Add repackaging script
Github user sijie commented on the issue: https://github.com/apache/incubator-distributedlog/pull/95 oops I think we need to change the artifact name too. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-distributedlog pull request #101: DL-173: changed FileUtiles.delet...
Github user adamtracymartin closed the pull request at: https://github.com/apache/incubator-distributedlog/pull/101 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-distributedlog pull request #101: DL-173: changed FileUtiles.delet...
GitHub user adamtracymartin opened a pull request: https://github.com/apache/incubator-distributedlog/pull/101 DL-173: changed FileUtiles.deleteDirectory to FileUtiles.forceDeleteOnExit so files would delete on Windows You can merge this pull request into a Git repository by running: $ git pull https://github.com/adamtracymartin/incubator-distributedlog DL-173 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-distributedlog/pull/101.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #101 commit 4d4153c7bc8f021fd141ab1dbae6f4a4732c9569 Author: adamtracymartin Date: 2017-01-05T21:49:54Z changed FileUtiles.deleteDirectory to FileUtiles.forceDeleteOnExit so files would delete on Windows commit c80a4d82603c1553121cf48d9447da7945f2d899 Author: Tracy Martin Date: 2017-01-05T21:55:01Z I did not mean to check in the .checkstyle files commit 6e8e23ab065114d0d5cdad97759f2d0b5918de46 Author: Tracy Martin Date: 2017-01-05T21:55:28Z I did not mean to check in the .checkstyle file. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: [Review] The first release of Apache DistributedLog
Hi Sijie, I am following this release and the great work DL comunity is doing. Maybe it would be worth to write some paragraph about the difference from BookKeeper and/or the relation with it. Enrico Il gio 5 gen 2017, 19:13 Asko Kauppi ha scritto: > Hi Sijie, > > most readers will likely not know about DistributedLog. A short comparison > - or a link to one - e.g. with Kafka might help set the stage for them. > i.e. why does it exist? > > This is even more important since Uber recently also publicized their > persistent message bus solution. If these start dropping down, there needs > to be more (technical) reason than just another vendor opening their chest. > :) > > Other than that, the structure seemed nice but it can be slightly shorter. > > Just my 2c > - asko > > > On 4 January 2017 at 10:38, Sijie Guo wrote: > > > I drafted a blog post for announcing the first apache release. Here is > the > > draft. Please help review it :D > > > > https://docs.google.com/document/d/1IXVmP2cHkf4ydeUHUJN9p5ZWTpA1a > > uwBhfqMnYBu4A0/edit > > > > - Sijie > > > -- -- Enrico Olivelli
[GitHub] incubator-distributedlog pull request #100: DL-28: changed // comments to /*...
GitHub user adamtracymartin opened a pull request: https://github.com/apache/incubator-distributedlog/pull/100 DL-28: changed // comments to /* */ You can merge this pull request into a Git repository by running: $ git pull https://github.com/adamtracymartin/incubator-distributedlog DL-28 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-distributedlog/pull/100.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #100 commit db04e15ef18ce27fe240303d3325127941e99aec Author: adamtracymartin Date: 2017-01-05T21:05:48Z changed // comments to /* */ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: [Discuss] Transaction Support
Xi, I added more comments. We are looking forward to your reply and seeing this happen. - Sijie On Tue, Jan 3, 2017 at 11:18 PM, Sijie Guo wrote: > Sorry for late response. I think Leigh and you already had some very > valuable discussions in the doc. I will try to add some of my questions to > the discussion. > > Beside that, I had a discussion with Leigh today about this. first of all, > I think it is very good to add transaction support in distributedlog. It is > one of the primitives that would help building distributed service. But we > have a concern about making this system become complicated and introduce > operational overhead when it runs in the large scale system on production. > There are two major suggestions that I have for this feature - > > Build the 'minimum' logic in core - I think the minimum logic that need to > be added to the core is - the special control records (begin, commit and > abort) and make the reader be able to detect those special control records > and know what do they mean and how to interrupt with them. Since they are > special control records, there is not overhead to other readers that > doesn't require this feature. > > Build the transaction coordinator as a separated proxy service - I think > the major concern that we have is putting more complexities into the 'write > proxy' service. We architected distributedlog in a more microservice-like > way - we have the core as the stream store, the proxy for serving write and > read traffic. It would be good that the transaction feature can be done in > a similar way. So the architecture would be like this - > > *[ write service ] [ read service ] [ transaction coordinator ]* > *[ stream store > ]* > > if people doesn't need the transaction feature, they can turn if off > completely without any operational overhead. > > Beside that, I have one general question - What is the major goal for this > feature? Are you targeting on building a general XA transaction coordinator > or just for supporting things like `copy-modify-write' style workflow? > > > Thanks, > Sijie > > > > > > On Wed, Dec 28, 2016 at 1:12 PM, Xi Liu wrote: > >> Ping? >> >> On Mon, Dec 19, 2016 at 8:28 AM, Xi Liu wrote: >> >> > Sijie, >> > >> > No. I thought it might be easier for people to comment on a google doc >> to >> > gather the initial feedback. I will put the content back to wiki page >> once >> > addressing the comments. Does that sound good to you? >> > >> > And thank you in advance. >> > >> > - Xi >> > >> > >> > >> > On Sun, Dec 18, 2016 at 8:48 AM, Sijie Guo wrote: >> > >> >> Hi Xi, >> >> >> >> sorry for late response. I will review it soon. >> >> >> >> regarding this, a separate question "are we going to use google doc >> >> instead >> >> of email thread for any discussion"? I am a bit worried that the >> >> discussion >> >> will become lost after moving to google doc. No idea on how other >> apache >> >> projects are doing. >> >> >> >> - Sijie >> >> >> >> On Wed, Dec 14, 2016 at 11:41 PM, Xi Liu wrote: >> >> >> >> > Hi all, >> >> > >> >> > I finalized the first version of the design. This time I used a >> google >> >> doc >> >> > so that it is easier for commenting and add a link the wiki page. I >> will >> >> > update this to the wiki page once we come to the finalized design. >> >> > >> >> > https://docs.google.com/document/d/14Ns05M8Z5a6DF6fHmWQwISyD5jjeK >> >> > bSIGgSzXuTI5BA/edit >> >> > >> >> > Let me know if you have any questions. Appreciate your reviews! >> >> > >> >> > - Xi >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > On Fri, Oct 28, 2016 at 7:58 AM, Leigh Stewart >> >> > > >> > > wrote: >> >> > >> >> > > Interesting proposal. A couple quick notes while you continue to >> flesh >> >> > this >> >> > > out. >> >> > > >> >> > > a. just to be sure - does this eliminate the need to save seqno >> with >> >> > > checkpoint? >> >> > > >> >> > > b. i.e. another way to describe this kind of improvement is >> "support >> >> > > records (atomic writes) larger than 1MB", iiuc. the advantage >> being it >> >> > > avoids the baggage of transactions. disadvantages include inability >> >> to do >> >> > > cross stream transactions, and flexibility (interleaving, etc) (are >> >> there >> >> > > others?). >> >> > > >> >> > > c. proxy use case is for supporting multiple writers - have you >> >> thought >> >> > > about how this would work with multiple writers? >> >> > > >> >> > > Thanks! >> >> > > >> >> > > >> >> > > On Tue, Oct 18, 2016 at 6:45 PM, Sijie Guo >> > >> > >> >> > > wrote: >> >> > > >> >> > > > Sound good to me. look forward to the detailed proposal. >> >> > > > >> >> > > > (I don't mind the format if it makes things easier to you) >> >> > > > >> >> > > > Sijie >> >> > > > >> >> > > > On Friday, October 14, 2016, Xi Liu >> wrote: >> >> > > > >> >> > > > > Thank you, Sijie >> >> > > > > >> >> > > > > We have some internal discussions to sort out some details. We >> are >> >> > > ready >> >> > > > to >> >> > > > > col
Re: [Discuss] Transaction Support
On Wed, Jan 4, 2017 at 1:14 AM, Asko Kauppi wrote: > > Beside that, I have one general question - What is the major goal for > this > > feature? Are you targeting on building a general XA transaction > coordinator > > or just for supporting things like `copy-modify-write' style workflow? > > The use case I would have for transactions - at some level of the stack - > is supporting dynamic configurations. > > If a config changes in e.g. three lines, some of the changes may logically > belong together. E.g. changing both “host” and “port” (if separate > entries). One shouldn’t be able to read a state, even temporarily, that has > new host but old port. > > I can do this in the application level - it does not need to be part of > the DL protocol. > Yeah, I can see 'transaction' as a large atomic write in DL is very useful. Currently DL limits the record size to 1MB. If people wants to write a record larger than 1MB, it will potentially produce a `partial` write if application breaks their record into multiple records. Your use case falls into this category. I think the minimal support is large atomic write. That is good enough for most of the log use cases. Having a separated TC (transaction coordinator) is cool. but it can be an opt-in solution. > > > Asko Kauppi > Zalando Tech Helsinki > > > On 4 Jan 2017, at 9.18, Sijie Guo wrote: > > > > Sorry for late response. I think Leigh and you already had some very > > valuable discussions in the doc. I will try to add some of my questions > to > > the discussion. > > > > Beside that, I had a discussion with Leigh today about this. first of > all, > > I think it is very good to add transaction support in distributedlog. It > is > > one of the primitives that would help building distributed service. But > we > > have a concern about making this system become complicated and introduce > > operational overhead when it runs in the large scale system on > production. > > There are two major suggestions that I have for this feature - > > > > Build the 'minimum' logic in core - I think the minimum logic that need > to > > be added to the core is - the special control records (begin, commit and > > abort) and make the reader be able to detect those special control > records > > and know what do they mean and how to interrupt with them. Since they are > > special control records, there is not overhead to other readers that > > doesn't require this feature. > > > > Build the transaction coordinator as a separated proxy service - I think > > the major concern that we have is putting more complexities into the > 'write > > proxy' service. We architected distributedlog in a more microservice-like > > way - we have the core as the stream store, the proxy for serving write > and > > read traffic. It would be good that the transaction feature can be done > in > > a similar way. So the architecture would be like this - > > > > *[ write service ] [ read service ] [ transaction coordinator ]* > > *[ stream store > >]* > > > > if people doesn't need the transaction feature, they can turn if off > > completely without any operational overhead. > > > > Beside that, I have one general question - What is the major goal for > this > > feature? Are you targeting on building a general XA transaction > coordinator > > or just for supporting things like `copy-modify-write' style workflow? > > > > > > Thanks, > > Sijie > > > > > > > > > > > > On Wed, Dec 28, 2016 at 1:12 PM, Xi Liu wrote: > > > >> Ping? > >> > >> On Mon, Dec 19, 2016 at 8:28 AM, Xi Liu wrote: > >> > >>> Sijie, > >>> > >>> No. I thought it might be easier for people to comment on a google doc > to > >>> gather the initial feedback. I will put the content back to wiki page > >> once > >>> addressing the comments. Does that sound good to you? > >>> > >>> And thank you in advance. > >>> > >>> - Xi > >>> > >>> > >>> > >>> On Sun, Dec 18, 2016 at 8:48 AM, Sijie Guo wrote: > >>> > Hi Xi, > > sorry for late response. I will review it soon. > > regarding this, a separate question "are we going to use google doc > instead > of email thread for any discussion"? I am a bit worried that the > discussion > will become lost after moving to google doc. No idea on how other > apache > projects are doing. > > - Sijie > > On Wed, Dec 14, 2016 at 11:41 PM, Xi Liu > wrote: > > > Hi all, > > > > I finalized the first version of the design. This time I used a > google > doc > > so that it is easier for commenting and add a link the wiki page. I > >> will > > update this to the wiki page once we come to the finalized design. > > > > https://docs.google.com/document/d/14Ns05M8Z5a6DF6fHmWQwISyD5jjeK > > bSIGgSzXuTI5BA/edit > > > > Let me know if you have any questions. Appreciate your reviews! > > > > - Xi > > > > > > > > > > > > On Fri, Oct 28, 2016 at 7:58 AM, Leigh Stewart >
[GitHub] incubator-distributedlog issue #97: DL-170: All dependency versions should b...
Github user sijie commented on the issue: https://github.com/apache/incubator-distributedlog/pull/97 I liked the change of managing the versions in a central place. great contribution! @gerritsundaram +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-distributedlog issue #96: DL-83: Add javadoc to website
Github user sijie commented on the issue: https://github.com/apache/incubator-distributedlog/pull/96 @mgodave @leighst need this for generating the javadoc for the release. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-distributedlog issue #98: DL-171 : adding a short sleep to let the...
Github user sijie commented on the issue: https://github.com/apache/incubator-distributedlog/pull/98 looks good to me. it is okay for a short term fix. moving forward, we might consider how to improve the test in general to make them time independent. +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-distributedlog issue #99: DL-172: Provide a IDL definition for str...
Github user sijie commented on the issue: https://github.com/apache/incubator-distributedlog/pull/99 @gerritsundaram looks like a findbugs warming. you might consider excluding the generated code. regarding whether using thrift or protobuf, can you raise a discussion email thread for that? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: [Review] The first release of Apache DistributedLog
Hi Sijie, most readers will likely not know about DistributedLog. A short comparison - or a link to one - e.g. with Kafka might help set the stage for them. i.e. why does it exist? This is even more important since Uber recently also publicized their persistent message bus solution. If these start dropping down, there needs to be more (technical) reason than just another vendor opening their chest. :) Other than that, the structure seemed nice but it can be slightly shorter. Just my 2c - asko On 4 January 2017 at 10:38, Sijie Guo wrote: > I drafted a blog post for announcing the first apache release. Here is the > draft. Please help review it :D > > https://docs.google.com/document/d/1IXVmP2cHkf4ydeUHUJN9p5ZWTpA1a > uwBhfqMnYBu4A0/edit > > - Sijie >
[GitHub] incubator-distributedlog pull request #99: DL-172: Provide a IDL definition ...
GitHub user gerritsundaram opened a pull request: https://github.com/apache/incubator-distributedlog/pull/99 DL-172: Provide a IDL definition for stream metadata and segment meta⦠https://issues.apache.org/jira/browse/DL-172 You can merge this pull request into a Git repository by running: $ git pull https://github.com/gerritsundaram/incubator-distributedlog DL-73 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-distributedlog/pull/99.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #99 commit c9224cb28e486c56a87f411f894e6bcab9d32b14 Author: Gerrit Sundaram Date: 2017-01-05T09:38:15Z DL-172: Provide a IDL definition for stream metadata and segment metadata --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-distributedlog pull request #98: DL-171 : adding a short sleep to ...
GitHub user xieliang opened a pull request: https://github.com/apache/incubator-distributedlog/pull/98 DL-171 : adding a short sleep to let the WriteCompleteListener have time to run before the final position be requested once the "writer.write" is done, if "writer.position()" be invoked easier than the WriteCompleteListener onSuccess callback, due to the "synchronized", the position result will be 0, not the expected 33. we can just add a short sleep to avoid this test issue. You can merge this pull request into a Git repository by running: $ git pull https://github.com/xieliang/incubator-distributedlog DL-171-Fix-TestAppendOnlyStreamWriter Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-distributedlog/pull/98.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #98 commit 9de4d177d2c4d41aaaeaa69012be01fa4e0104ad Author: xieliang Date: 2017-01-05T08:56:37Z adding a short sleep to let the WriteCompleteListener have time to run before the final position be requested --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-distributedlog pull request #97: DL-170: All dependency versions s...
GitHub user gerritsundaram opened a pull request: https://github.com/apache/incubator-distributedlog/pull/97 DL-170: All dependency versions should be defined in the parent pom make all dependencies' version be defined in the parent pom file. You can merge this pull request into a Git repository by running: $ git pull https://github.com/gerritsundaram/incubator-distributedlog DL-170 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-distributedlog/pull/97.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #97 commit a2038f4f65b34eaca3af1565eecdf47291bcc8b2 Author: Gerrit Sundaram Date: 2017-01-05T08:05:22Z DL-170: All dependency versions should be defined in the parent pom --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
