Re: [RESULT] [VOTE] Release 0.4.0, release candidate #2
Is there a URL / newsfeed that I could share about the release with my organization? Asko Kauppi Zalando Tech Helsinki For Zalando internal messages, I prefer HipChat > On 25 Jan 2017, at 8.24, Sijie Guo <si...@apache.org> wrote: > > I am happy to announce that we have unanimously approved this release. > > There are 8 +1 votes and no disapproving votes. > > Thanks everyone! > > - Sijie > > > > > > On Tue, Jan 10, 2017 at 11:56 PM, Sijie Guo <si...@apache.org> wrote: > >> Hi all, >> >> Please review and vote on the release candidate #2 for the version 0.4.0, >> as follows: >> >> [ ] +1, Approve the release >> [ ] -1, Do not approve the release (please provide specific comments) >> >> The complete staging area is available for your review, which includes: >> >>* JIRA release notes [1], >>* the official Apache source release to be deployed to dist.apache.org >> [2], >>* all artifacts to be deployed to the Maven Central Repository [3][4], >>* source code tag "v0.4.0-incubating-RC1_2.11" (for scala 2.11) and >> "v0.4.0-incubating-RC1_2.10" (for scala 2.10) [5][6], >>* website pull request listing the release [7] and publishing the API >> reference manual. >> >> A simple instruction for validation the source and binary packages. >> >> - source package: building the package with "*mvn clean apache-rat:check >> package findbugs:check -DskipTests*" >> >> The vote will be open for at least 72 hours. It is adopted by majority >> approval, with at least 3 PPMC affirmative votes. >> >> Thanks, >> Sijie >> >> [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?p >> rojectId=12320620=12337980 >> [2] https://dist.apache.org/repos/dist/dev/incubator/distributedlog/0.4.0- >> incubating-RC2/ >> [3] https://repository.apache.org/content/repositories/ >> orgapachedistributedlog-1003/ >> [4] https://repository.apache.org/content/repositories/ >> orgapachedistributedlog-1004/ >> [5] https://github.com/apache/incubator-distributedlog/tree/ >> v0.4.0-incubating-RC1_2.11 >> [6] https://github.com/apache/incubator-distributedlog/tree/ >> v0.4.0-incubating-RC1_2.10 >> [7] https://github.com/apache/incubator-distributedlog/pull/109 >>
Re: [Review] The first release of Apache DistributedLog
Hi Sijie, most readers will likely not know about DistributedLog. A short comparison - or a link to one - e.g. with Kafka might help set the stage for them. i.e. why does it exist? This is even more important since Uber recently also publicized their persistent message bus solution. If these start dropping down, there needs to be more (technical) reason than just another vendor opening their chest. :) Other than that, the structure seemed nice but it can be slightly shorter. Just my 2c - asko On 4 January 2017 at 10:38, Sijie Guowrote: > I drafted a blog post for announcing the first apache release. Here is the > draft. Please help review it :D > > https://docs.google.com/document/d/1IXVmP2cHkf4ydeUHUJN9p5ZWTpA1a > uwBhfqMnYBu4A0/edit > > - Sijie >
Re: [Discuss] Transaction Support
> Beside that, I have one general question - What is the major goal for this > feature? Are you targeting on building a general XA transaction coordinator > or just for supporting things like `copy-modify-write' style workflow? The use case I would have for transactions - at some level of the stack - is supporting dynamic configurations. If a config changes in e.g. three lines, some of the changes may logically belong together. E.g. changing both “host” and “port” (if separate entries). One shouldn’t be able to read a state, even temporarily, that has new host but old port. I can do this in the application level - it does not need to be part of the DL protocol. Asko Kauppi Zalando Tech Helsinki > On 4 Jan 2017, at 9.18, Sijie Guo <si...@apache.org> wrote: > > Sorry for late response. I think Leigh and you already had some very > valuable discussions in the doc. I will try to add some of my questions to > the discussion. > > Beside that, I had a discussion with Leigh today about this. first of all, > I think it is very good to add transaction support in distributedlog. It is > one of the primitives that would help building distributed service. But we > have a concern about making this system become complicated and introduce > operational overhead when it runs in the large scale system on production. > There are two major suggestions that I have for this feature - > > Build the 'minimum' logic in core - I think the minimum logic that need to > be added to the core is - the special control records (begin, commit and > abort) and make the reader be able to detect those special control records > and know what do they mean and how to interrupt with them. Since they are > special control records, there is not overhead to other readers that > doesn't require this feature. > > Build the transaction coordinator as a separated proxy service - I think > the major concern that we have is putting more complexities into the 'write > proxy' service. We architected distributedlog in a more microservice-like > way - we have the core as the stream store, the proxy for serving write and > read traffic. It would be good that the transaction feature can be done in > a similar way. So the architecture would be like this - > > *[ write service ] [ read service ] [ transaction coordinator ]* > *[ stream store >]* > > if people doesn't need the transaction feature, they can turn if off > completely without any operational overhead. > > Beside that, I have one general question - What is the major goal for this > feature? Are you targeting on building a general XA transaction coordinator > or just for supporting things like `copy-modify-write' style workflow? > > > Thanks, > Sijie > > > > > > On Wed, Dec 28, 2016 at 1:12 PM, Xi Liu <xi.liu@gmail.com> wrote: > >> Ping? >> >> On Mon, Dec 19, 2016 at 8:28 AM, Xi Liu <xi.liu@gmail.com> wrote: >> >>> Sijie, >>> >>> No. I thought it might be easier for people to comment on a google doc to >>> gather the initial feedback. I will put the content back to wiki page >> once >>> addressing the comments. Does that sound good to you? >>> >>> And thank you in advance. >>> >>> - Xi >>> >>> >>> >>> On Sun, Dec 18, 2016 at 8:48 AM, Sijie Guo <si...@apache.org> wrote: >>> >>>> Hi Xi, >>>> >>>> sorry for late response. I will review it soon. >>>> >>>> regarding this, a separate question "are we going to use google doc >>>> instead >>>> of email thread for any discussion"? I am a bit worried that the >>>> discussion >>>> will become lost after moving to google doc. No idea on how other apache >>>> projects are doing. >>>> >>>> - Sijie >>>> >>>> On Wed, Dec 14, 2016 at 11:41 PM, Xi Liu <xi.liu@gmail.com> wrote: >>>> >>>>> Hi all, >>>>> >>>>> I finalized the first version of the design. This time I used a google >>>> doc >>>>> so that it is easier for commenting and add a link the wiki page. I >> will >>>>> update this to the wiki page once we come to the finalized design. >>>>> >>>>> https://docs.google.com/document/d/14Ns05M8Z5a6DF6fHmWQwISyD5jjeK >>>>> bSIGgSzXuTI5BA/edit >>>>> >>>>> Let me know if you have any questions. Appreciate your reviews! >>>>> >>>>> - Xi >>>>> >>>>> >>>&
What's the 'name' parameter for opening a Write Proxy Client?
In http://distributedlog.incubator.apache.org/docs/latest/tutorials/basic-2.html <http://distributedlog.incubator.apache.org/docs/latest/tutorials/basic-2.html> DistributedLogClientBuilder builder = DistributedLogClientBuilder.newBuilder(); .clientId(ClientId.apply("console-proxy-writer")) .name("console-proxy-writer”); What’s the role of the client id (is it only for logging/debugging)? What’s the role of the name? How is it different from client id? Where do I provide the DistributedLog namespace that I wish to write to? Asko Kauppi Zalando Tech Helsinki
Is there a REST API for Write and Read proxies?
I’m reading http://distributedlog.incubator.apache.org/docs/latest/user_guide/api/proxy.html <http://distributedlog.incubator.apache.org/docs/latest/user_guide/api/proxy.html> Ideally, I wouldn’t need to use a library to talk to a proxy service, right? Is there documentation on how to access the proxies as REST endpoints / are they such? My preferred environment is Scala and akka-http. Asko Kauppi Zalando Tech Helsinki
[jira] [Commented] (DL-148) Not providing net-util 0.0.100 via 'mvn clean install -DskipTests'
[ https://issues.apache.org/jira/browse/DL-148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15765450#comment-15765450 ] Asko Kauppi commented on DL-148: I think `com.twitter.common.zookeeper:client:0.0.76` is the one bringing the faulty 0.0.100 in. > Not providing net-util 0.0.100 via 'mvn clean install -DskipTests' > -- > > Key: DL-148 > URL: https://issues.apache.org/jira/browse/DL-148 > Project: DistributedLog > Issue Type: Bug > Components: builds > Reporter: Asko Kauppi >Priority: Minor > > I was building a Scala wrapper for DistributedLog 0.4.0, and wanted to get > the JVM artefacts via local Maven repo. > It works, for most of them, but the > `com.twitter.common#net-util;0.0.100!net-util.jar` is not available. > Looking at `~/.m2/repository` shows that 0.0.97 has a jar but 0.0.100 doesn't. > << > $ find /Users/akauppi/.m2/repository/ -name "*net-util*" > /Users/akauppi/.m2/repository//com/twitter/common/net-util > /Users/akauppi/.m2/repository//com/twitter/common/net-util/0.0.100/net-util-0.0.100.pom > /Users/akauppi/.m2/repository//com/twitter/common/net-util/0.0.100/net-util-0.0.100.pom.lastUpdated > /Users/akauppi/.m2/repository//com/twitter/common/net-util/0.0.100/net-util-0.0.100.pom.sha1 > /Users/akauppi/.m2/repository//com/twitter/common/net-util/0.0.97/net-util-0.0.97.jar > /Users/akauppi/.m2/repository//com/twitter/common/net-util/0.0.97/net-util-0.0.97.jar.lastUpdated > /Users/akauppi/.m2/repository//com/twitter/common/net-util/0.0.97/net-util-0.0.97.jar.sha1 > /Users/akauppi/.m2/repository//com/twitter/common/net-util/0.0.97/net-util-0.0.97.pom > /Users/akauppi/.m2/repository//com/twitter/common/net-util/0.0.97/net-util-0.0.97.pom.lastUpdated > /Users/akauppi/.m2/repository//com/twitter/common/net-util/0.0.97/net-util-0.0.97.pom.sha1 > << > My error on customer project / Scala side: > << > $ sbt update > [info] Loading global plugins from /Users/akauppi/.sbt/0.13/plugins > [info] Loading project definition from > /Users/akauppi/Source/sleeves-distributedlog/project > [info] Set current project to sleeves-distributedlog (in build > file:/Users/akauppi/Source/sleeves-distributedlog/) > [info] Updating > {file:/Users/akauppi/Source/sleeves-distributedlog/}sleeves-distributedlog... > [info] Resolving jline#jline;2.12.1 ... > [info] downloading > https://repo1.maven.org/maven2/org/apache/zookeeper/zookeeper/3.5.1-alpha/zookeeper-3.5.1-alpha.jar > ... > [info][SUCCESSFUL ] > org.apache.zookeeper#zookeeper;3.5.1-alpha!zookeeper.jar (680ms) > [info] downloading > https://repo1.maven.org/maven2/com/twitter/util-core_2.11/6.34.0/util-core_2.11-6.34.0.jar > ... > [info][SUCCESSFUL ] > com.twitter#util-core_2.11;6.34.0!util-core_2.11.jar (787ms) > [info] downloading > https://repo1.maven.org/maven2/com/twitter/scrooge-core_2.11/4.6.0/scrooge-core_2.11-4.6.0.jar > ... > [info][SUCCESSFUL ] > com.twitter#scrooge-core_2.11;4.6.0!scrooge-core_2.11.jar (177ms) > [info] downloading > https://repo1.maven.org/maven2/commons-cli/commons-cli/1.1/commons-cli-1.1.jar > ... > [info][SUCCESSFUL ] commons-cli#commons-cli;1.1!commons-cli.jar (69ms) > [info] downloading > https://repo1.maven.org/maven2/log4j/log4j/1.2.16/log4j-1.2.16.jar ... > [info][SUCCESSFUL ] log4j#log4j;1.2.16!log4j.jar(bundle) (269ms) > [info] downloading > https://repo1.maven.org/maven2/net/java/dev/javacc/javacc/5.0/javacc-5.0.jar > ... > [info][SUCCESSFUL ] net.java.dev.javacc#javacc;5.0!javacc.jar (200ms) > [info] downloading > https://repo1.maven.org/maven2/com/google/inject/guice/3.0/guice-3.0.jar ... > [info][SUCCESSFUL ] com.google.inject#guice;3.0!guice.jar (431ms) > [info] downloading > https://repo1.maven.org/maven2/org/sonatype/sisu/inject/cglib/2.2.1-v20090111/cglib-2.2.1-v20090111.jar > ... > [info][SUCCESSFUL ] > org.sonatype.sisu.inject#cglib;2.2.1-v20090111!cglib.jar (247ms) > [info] downloading https://repo1.maven.org/maven2/asm/asm/3.1/asm-3.1.jar ... > [info][SUCCESSFUL ] asm#asm;3.1!asm.jar (82ms) > [info] downloading > https://repo1.maven.org/maven2/com/twitter/util-function_2.11/6.34.0/util-function_2.11-6.34.0.jar > ... > [info][SUCCESSFUL ] > com.twitter#util-function_2.11;6.34.0!util-function_2.11.jar (160ms) > [info] downloading > https://repo1.maven.org/maven2/com/twitter/jsr166e/1.0.0/jsr166e-1.0.0.jar ... > [info][SUCCESSFUL ] com.twitter#jsr166e;1.0.0!jsr166e.jar (57ms) > [info] downloading > https://rep
[jira] [Created] (DL-148) Not providing net-util 0.0.100 via 'mvn clean install -DskipTests'
Asko Kauppi created DL-148: -- Summary: Not providing net-util 0.0.100 via 'mvn clean install -DskipTests' Key: DL-148 URL: https://issues.apache.org/jira/browse/DL-148 Project: DistributedLog Issue Type: Bug Components: builds Reporter: Asko Kauppi Priority: Minor I was building a Scala wrapper for DistributedLog 0.4.0, and wanted to get the JVM artefacts via local Maven repo. It works, for most of them, but the `com.twitter.common#net-util;0.0.100!net-util.jar` is not available. Looking at `~/.m2/repository` shows that 0.0.97 has a jar but 0.0.100 doesn't. << $ find /Users/akauppi/.m2/repository/ -name "*net-util*" /Users/akauppi/.m2/repository//com/twitter/common/net-util /Users/akauppi/.m2/repository//com/twitter/common/net-util/0.0.100/net-util-0.0.100.pom /Users/akauppi/.m2/repository//com/twitter/common/net-util/0.0.100/net-util-0.0.100.pom.lastUpdated /Users/akauppi/.m2/repository//com/twitter/common/net-util/0.0.100/net-util-0.0.100.pom.sha1 /Users/akauppi/.m2/repository//com/twitter/common/net-util/0.0.97/net-util-0.0.97.jar /Users/akauppi/.m2/repository//com/twitter/common/net-util/0.0.97/net-util-0.0.97.jar.lastUpdated /Users/akauppi/.m2/repository//com/twitter/common/net-util/0.0.97/net-util-0.0.97.jar.sha1 /Users/akauppi/.m2/repository//com/twitter/common/net-util/0.0.97/net-util-0.0.97.pom /Users/akauppi/.m2/repository//com/twitter/common/net-util/0.0.97/net-util-0.0.97.pom.lastUpdated /Users/akauppi/.m2/repository//com/twitter/common/net-util/0.0.97/net-util-0.0.97.pom.sha1 << My error on customer project / Scala side: << $ sbt update [info] Loading global plugins from /Users/akauppi/.sbt/0.13/plugins [info] Loading project definition from /Users/akauppi/Source/sleeves-distributedlog/project [info] Set current project to sleeves-distributedlog (in build file:/Users/akauppi/Source/sleeves-distributedlog/) [info] Updating {file:/Users/akauppi/Source/sleeves-distributedlog/}sleeves-distributedlog... [info] Resolving jline#jline;2.12.1 ... [info] downloading https://repo1.maven.org/maven2/org/apache/zookeeper/zookeeper/3.5.1-alpha/zookeeper-3.5.1-alpha.jar ... [info] [SUCCESSFUL ] org.apache.zookeeper#zookeeper;3.5.1-alpha!zookeeper.jar (680ms) [info] downloading https://repo1.maven.org/maven2/com/twitter/util-core_2.11/6.34.0/util-core_2.11-6.34.0.jar ... [info] [SUCCESSFUL ] com.twitter#util-core_2.11;6.34.0!util-core_2.11.jar (787ms) [info] downloading https://repo1.maven.org/maven2/com/twitter/scrooge-core_2.11/4.6.0/scrooge-core_2.11-4.6.0.jar ... [info] [SUCCESSFUL ] com.twitter#scrooge-core_2.11;4.6.0!scrooge-core_2.11.jar (177ms) [info] downloading https://repo1.maven.org/maven2/commons-cli/commons-cli/1.1/commons-cli-1.1.jar ... [info] [SUCCESSFUL ] commons-cli#commons-cli;1.1!commons-cli.jar (69ms) [info] downloading https://repo1.maven.org/maven2/log4j/log4j/1.2.16/log4j-1.2.16.jar ... [info] [SUCCESSFUL ] log4j#log4j;1.2.16!log4j.jar(bundle) (269ms) [info] downloading https://repo1.maven.org/maven2/net/java/dev/javacc/javacc/5.0/javacc-5.0.jar ... [info] [SUCCESSFUL ] net.java.dev.javacc#javacc;5.0!javacc.jar (200ms) [info] downloading https://repo1.maven.org/maven2/com/google/inject/guice/3.0/guice-3.0.jar ... [info] [SUCCESSFUL ] com.google.inject#guice;3.0!guice.jar (431ms) [info] downloading https://repo1.maven.org/maven2/org/sonatype/sisu/inject/cglib/2.2.1-v20090111/cglib-2.2.1-v20090111.jar ... [info] [SUCCESSFUL ] org.sonatype.sisu.inject#cglib;2.2.1-v20090111!cglib.jar (247ms) [info] downloading https://repo1.maven.org/maven2/asm/asm/3.1/asm-3.1.jar ... [info] [SUCCESSFUL ] asm#asm;3.1!asm.jar (82ms) [info] downloading https://repo1.maven.org/maven2/com/twitter/util-function_2.11/6.34.0/util-function_2.11-6.34.0.jar ... [info] [SUCCESSFUL ] com.twitter#util-function_2.11;6.34.0!util-function_2.11.jar (160ms) [info] downloading https://repo1.maven.org/maven2/com/twitter/jsr166e/1.0.0/jsr166e-1.0.0.jar ... [info] [SUCCESSFUL ] com.twitter#jsr166e;1.0.0!jsr166e.jar (57ms) [info] downloading https://repo1.maven.org/maven2/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar ... [info] [SUCCESSFUL ] commons-configuration#commons-configuration;1.6!commons-configuration.jar (207ms) [info] downloading https://repo1.maven.org/maven2/net/java/dev/jna/jna/3.2.7/jna-3.2.7.jar ... [info] [SUCCESSFUL ] net.java.dev.jna#jna;3.2.7!jna.jar (587ms) [info] downloading https://repo1.maven.org/maven2/commons-digester/commons-digester/1.8/commons-digester-1.8.jar ... [info] [SUCCESSFUL ] commons-digester#commons-digester;1.8!commons-digester.jar (124ms) [info] downloading https://repo1.maven.org/maven2/commons-beanutils/commons-beanutils-core/1.8.0/commons-beanutils-core-1.8.0.jar ... [info] [SUCCESSFUL ] commons-beanutils#commons-beanutils-core;1.8.
Re: Hello
Hi Arun, I am also a newcomer to DistributedLog, and spent some time reading the documentation. I'd like to share some tips with you, to hopefully make your journey faster. First of all, I think 0.4.0 has not been released but the documentation I am following is based on that. My goal is to get a proper Docker-based setup running within this week. I did manage, today, to run the stack locally on macOS. - I found this page the best: http://distributedlog.incubator.apache.org/docs/latest/deployment/cluster.html It has almost all the information needed to set things up. At places, there seems to be mistakes, but with a little trying one gets past those. To check that everything actually works, the write tutorial pages (like http://distributedlog.incubator.apache.org/docs/latest/tutorials/basic-1.html ) have good one-liners. Those two pages are all I really needed. As a third pointer, there's a Docker page http://distributedlog.incubator.apache.org/docs/latest/deployment/docker.html that is likely the fastest way to get going. I plan to make a similar, but maybe clearer setup during the week... There are some aspects of DistributedLog that took me a while to get a hang on. For one, both Zookeeper and BookKeeper are delivered as part of the dependencies, and can be run from the versions in the lib/ folder. That's the best, or at least safest way to progress I guess. The documentation seems to be there, but it's like blown into multiple pieces and doesn't currently offer a good linear reading experience. Some pages can be entered from multiple menu positions in the docs page, which I find distracting. I'm hoping to get things up, understanding the vision and technical layout of DistributedLog, so that I can be helpful in creating an "outside in" documentation for diving into DistributedLog. I think the project has a lot of potential. Nice to have you join it! - asko On 17 December 2016 at 11:20, Arun Khetarpalwrote: > Hi, > > My name is Arun Khetarpal and I am from India. Just read about > “Distributed log” being inducted to apache incubator projects, congrats! > > I have around 5 yrs on development experience in building scalable, > distributed systems and would love to be a part of this community. Let me > know how if there is a newbie guide on how to get started with the project > > Regards, > Arun
[jira] [Created] (DL-144) Scala 2.12 support???
Asko Kauppi created DL-144: -- Summary: Scala 2.12 support??? Key: DL-144 URL: https://issues.apache.org/jira/browse/DL-144 Project: DistributedLog Issue Type: Improvement Components: builds, distributedlog-benchmark, distributedlog-client, distributedlog-core, distributedlog-protocol, distributedlog-service, distributedlog-tutorials, documentation Reporter: Asko Kauppi Priority: Minor The current relation of distributedlog to Scala is strange, and I think can be improved. The project itself is coded in Java, but uses two Scala dependencies (finagle https://github.com/twitter/finagle and scrooge https://github.com/twitter/scrooge). For this reason, also the output artefacts of distributedlog itself have the `_2.10` or `_2.11` postfixes appended to them. What about keeping the source Java, but moving the build handling from Maven to sbt. Sbt provides cross-compilation for various Scala versions, out of the box, and supports compiling Java source code as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Quick start download link is down
http://distributedlog.incubator.apache.org/docs/latest/download on page http://distributedlog.incubator.apache.org/docs/latest/start/quickstart I get: The requested URL /docs/latest/download was not found on this server.
Re: [discuss] Official Slack or Gitter channel for Distributedlog
Either works for me, too. > On 14 Dec 2016, at 10.32, Stevo Slavićwrote: > > Either works for me, as long as it's documented in project website and > readme file.
Introduction - wanting to work on the documentation
Hi all, my name is Asko Kauppi and I work as a technical Agility Coach at the Zalando Helsinki site we started in 2015. I'm interested in persistent message buses in general, because I see them offering a new kind of way to do microservices. Instead of each service being responsible for its state we can now store the state "in between" and have stateless filters. I know Kafka, Akka Streams and RxJS rather well. DistributedLog to me looks like a simpler, maybe lighter Kafka - I'm not for the normal big data use cases but simply stream programming, in small and medium size range (MBs to maybe 1 GB). The documentation is something I'd like to help with - it also allows me to learn DistributedLog well I hope. Instead of starting to make PRs I'd like to discuss goals and tools first, with some people already looking into the documentation. I'm very keen on detail and optimizing text for the reading experience (good flow). During next week (week 51), we are running a Hackweek and I might try out DistributedLog on a Raspberry Pi cluster, if all goes well. - Asko Kauppi