Re: Nightly build!
I found http://www.apache.org/legal/release-policy.html#host-rc which looks like the policy we should follow if we start doing nightly builds. I guess we shouldn't archive nightly builds, since there would be too many. On Thu, Aug 9, 2018 at 6:02 PM Jihoon Son wrote: > Hi all, > > Nightly build would be useful for folks who want to stay on the bleeding > edge of Druid. I'm thinking to add a Jenkins job to > https://builds.apache.org/ which checks every hour that there are changes > in the master branch and builds a new build. Once the build succeeds, the > binary is archived, so that we can add a link to the binary to our web > page. If the build fails, the notification would be sent to > comm...@druid.apache.org. > > Welcome any thoughts. > > Best, > Jihoon >
Nightly build!
Hi all, Nightly build would be useful for folks who want to stay on the bleeding edge of Druid. I'm thinking to add a Jenkins job to https://builds.apache.org/ which checks every hour that there are changes in the master branch and builds a new build. Once the build succeeds, the binary is archived, so that we can add a link to the binary to our web page. If the build fails, the notification would be sent to comm...@druid.apache.org. Welcome any thoughts. Best, Jihoon
Re: Druid 0.12.2 release vote
Hi all, thank you for the vote! The vote has passed with 8 positive votes (+1). I'm going to release 0.12.2. Best, Jihoon On Thu, Aug 9, 2018 at 10:40 AM Charles Allen wrote: > +1 > > Still rolling out but looks good for what I've seen so far > > On Thu, Aug 9, 2018 at 10:39 AM Charles Allen > wrote: > > > Oh probably. Long story short: approximately equivalent, new release has > > slightly tucked in long tail on the bad side of query time (but also few > > samples in the long tail overall). > > > > > > > > On Thu, Aug 9, 2018 at 10:37 AM Gian Merlino wrote: > > > > > Nice!! > > > > > > Although I don't see the graphic attached, maybe the mailing list ate > it? > > > > > > On Wed, Aug 8, 2018 at 4:15 PM Charles Allen > > .invalid> > > > wrote: > > > > > > > Blue is 0.12.2 with some minor backports not perf related. Red is > from > > > the > > > > 0.11.x series. This is effectively a bucketed PDF of the query times > > for > > > a > > > > live cluster with Timeseries queries as self-reported by historical > > > nodes. > > > > I mentioned elsewhere I'm not convinced query/time is a good proxy > for > > > user > > > > experience, but it does provide a good baseline for comparisons > between > > > > versions. Low query times are suspected due to some aggressive > caching > > or > > > > complete node misses (node very little data for that time range for > > that > > > > datasource). And high query time outliers are often the result of bad > > GC. > > > > > > > > On our side there is a new java version going out with the 0.12.2 > > > > deployment so it is unclear how much is attributed to the new java > > > version > > > > and how much is attributed to the druid jars or other config changes. > > > > Overall things seem to consistently display a small % improvement in > > the > > > > mean with our internal 0.12.2 release. This is good! > > > > > > > > Cheers, > > > > Charles Allen > > > > > > > > [image: Screen Shot 2018-08-08 at 4.01.24 PM.png] > > > > > > > > > > > > On Wed, Aug 8, 2018 at 3:11 PM David Lim > wrote: > > > > > > > >> +1, thank you! > > > >> > > > >> On Wed, Aug 8, 2018 at 3:16 PM Jonathan Wei > > wrote: > > > >> > > > >> > +1, thanks Jihoon! > > > >> > > > > >> > On Wed, Aug 8, 2018 at 1:18 PM, Jihoon Son > > > >> wrote: > > > >> > > > > >> > > Awesome! Thanks Charles! > > > >> > > > > > >> > > Jihoon > > > >> > > > > > >> > > On Wed, Aug 8, 2018 at 1:16 PM Gian Merlino > > > wrote: > > > >> > > > > > >> > > > Thanks, it will be nice to see! > > > >> > > > > > > >> > > > On Wed, Aug 8, 2018 at 1:15 PM Charles Allen < > > > >> charles.al...@snap.com > > > >> > > > .invalid> > > > >> > > > wrote: > > > >> > > > > > > >> > > > > I don't think it should be a blocker to release, but I have > to > > > run > > > >> > perf > > > >> > > > > tests for rollouts anyways so I figured I'd publish what I > > find > > > >> :-P > > > >> > > > > > > > >> > > > > Cheers, > > > >> > > > > Charles Allen > > > >> > > > > > > > >> > > > > > > > >> > > > > On Wed, Aug 8, 2018 at 12:33 PM Gian Merlino < > g...@apache.org > > > > > > >> > wrote: > > > >> > > > > > > > >> > > > > > That being said, Charles I am definitely looking forward > to > > > your > > > >> > > report > > > >> > > > > of > > > >> > > > > > what the upgrade from 0.11 -> 0.12.2-rc1 is like in your > > > >> cluster! > > > >> > > > > > > > > >> > > > > > On Wed, Aug 8, 2018 at 12:30 PM Gian Merlino < > > g...@apache.org > > > > > > > >> > > wrote: > > > >> > > > > > > > > >> > > > > > > My thought is that recently we have started doing small > > > >> bug-fix > > > >> > > > > releases > > > >> > > > > > > more often (0.12.1 and 0.12.2 were both small releases) > > and > > > I > > > >> > think > > > >> > > > it > > > >> > > > > > > makes sense to continue this practice. It makes sense to > > get > > > >> them > > > >> > > out > > > >> > > > > > > quickly, since shipping bug fixes is good. IMO trying to > > > >> validate > > > >> > > bug > > > >> > > > > fix > > > >> > > > > > > releases within the customary Apache style 72 hour > voting > > > >> period > > > >> > > is a > > > >> > > > > > good > > > >> > > > > > > goal. > > > >> > > > > > > > > > >> > > > > > > On the other hand we do strive to put out high quality > > > >> releases, > > > >> > > and > > > >> > > > we > > > >> > > > > > > don't want bug fix releases to introduce regressions. > > > Testing > > > >> > every > > > >> > > > > > single > > > >> > > > > > > patch in real clusters is an important part of that. > All I > > > >> can do > > > >> > > is > > > >> > > > > > > encourage people running real clusters to deploy RCs as > > fast > > > >> as > > > >> > > they > > > >> > > > > can! > > > >> > > > > > > Fwiw, we have already incorporated all the 0.12.2 > patches > > > into > > > >> > our > > > >> > > > > Imply > > > >> > > > > > > distro of Druid and already have a good number users > > running > > > >> > them. > > > >> > > So > > > >> > > > > my > > > >> > > > > > +1 > > > >> > >
Podling Report Reminder - August 2018
Dear podling, This email was sent by an automated system on behalf of the Apache Incubator PMC. It is an initial reminder to give you plenty of time to prepare your quarterly board report. The board meeting is scheduled for Wed, 15 August 2018, 10:30 am PDT. The report for your podling will form a part of the Incubator PMC report. The Incubator PMC requires your report to be submitted 2 weeks before the board meeting, to allow sufficient time for review and submission (Wed, August 01). Please submit your report with sufficient time to allow the Incubator PMC, and subsequently board members to review and digest. Again, the very latest you should submit your report is 2 weeks prior to the board meeting. Candidate names should not be made public before people are actually elected, so please do not include the names of potential committers or PPMC members in your report. Thanks, The Apache Incubator PMC Submitting your Report -- Your report should contain the following: * Your project name * A brief description of your project, which assumes no knowledge of the project or necessarily of its field * A list of the three most important issues to address in the move towards graduation. * Any issues that the Incubator PMC or ASF Board might wish/need to be aware of * How has the community developed since the last report * How has the project developed since the last report. * How does the podling rate their own maturity. This should be appended to the Incubator Wiki page at: https://wiki.apache.org/incubator/August2018 Note: This is manually populated. You may need to wait a little before this page is created from a template. Mentors --- Mentors should review reports for their project(s) and sign them off on the Incubator wiki page. Signing off reports shows that you are following the project - projects that are not signed may raise alarms for the Incubator PMC. Incubator PMC - To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org For additional commands, e-mail: dev-h...@druid.apache.org
Re: Druid 0.12.2 release vote
Oh probably. Long story short: approximately equivalent, new release has slightly tucked in long tail on the bad side of query time (but also few samples in the long tail overall). On Thu, Aug 9, 2018 at 10:37 AM Gian Merlino wrote: > Nice!! > > Although I don't see the graphic attached, maybe the mailing list ate it? > > On Wed, Aug 8, 2018 at 4:15 PM Charles Allen .invalid> > wrote: > > > Blue is 0.12.2 with some minor backports not perf related. Red is from > the > > 0.11.x series. This is effectively a bucketed PDF of the query times for > a > > live cluster with Timeseries queries as self-reported by historical > nodes. > > I mentioned elsewhere I'm not convinced query/time is a good proxy for > user > > experience, but it does provide a good baseline for comparisons between > > versions. Low query times are suspected due to some aggressive caching or > > complete node misses (node very little data for that time range for that > > datasource). And high query time outliers are often the result of bad GC. > > > > On our side there is a new java version going out with the 0.12.2 > > deployment so it is unclear how much is attributed to the new java > version > > and how much is attributed to the druid jars or other config changes. > > Overall things seem to consistently display a small % improvement in the > > mean with our internal 0.12.2 release. This is good! > > > > Cheers, > > Charles Allen > > > > [image: Screen Shot 2018-08-08 at 4.01.24 PM.png] > > > > > > On Wed, Aug 8, 2018 at 3:11 PM David Lim wrote: > > > >> +1, thank you! > >> > >> On Wed, Aug 8, 2018 at 3:16 PM Jonathan Wei wrote: > >> > >> > +1, thanks Jihoon! > >> > > >> > On Wed, Aug 8, 2018 at 1:18 PM, Jihoon Son > >> wrote: > >> > > >> > > Awesome! Thanks Charles! > >> > > > >> > > Jihoon > >> > > > >> > > On Wed, Aug 8, 2018 at 1:16 PM Gian Merlino > wrote: > >> > > > >> > > > Thanks, it will be nice to see! > >> > > > > >> > > > On Wed, Aug 8, 2018 at 1:15 PM Charles Allen < > >> charles.al...@snap.com > >> > > > .invalid> > >> > > > wrote: > >> > > > > >> > > > > I don't think it should be a blocker to release, but I have to > run > >> > perf > >> > > > > tests for rollouts anyways so I figured I'd publish what I find > >> :-P > >> > > > > > >> > > > > Cheers, > >> > > > > Charles Allen > >> > > > > > >> > > > > > >> > > > > On Wed, Aug 8, 2018 at 12:33 PM Gian Merlino > >> > wrote: > >> > > > > > >> > > > > > That being said, Charles I am definitely looking forward to > your > >> > > report > >> > > > > of > >> > > > > > what the upgrade from 0.11 -> 0.12.2-rc1 is like in your > >> cluster! > >> > > > > > > >> > > > > > On Wed, Aug 8, 2018 at 12:30 PM Gian Merlino > > >> > > wrote: > >> > > > > > > >> > > > > > > My thought is that recently we have started doing small > >> bug-fix > >> > > > > releases > >> > > > > > > more often (0.12.1 and 0.12.2 were both small releases) and > I > >> > think > >> > > > it > >> > > > > > > makes sense to continue this practice. It makes sense to get > >> them > >> > > out > >> > > > > > > quickly, since shipping bug fixes is good. IMO trying to > >> validate > >> > > bug > >> > > > > fix > >> > > > > > > releases within the customary Apache style 72 hour voting > >> period > >> > > is a > >> > > > > > good > >> > > > > > > goal. > >> > > > > > > > >> > > > > > > On the other hand we do strive to put out high quality > >> releases, > >> > > and > >> > > > we > >> > > > > > > don't want bug fix releases to introduce regressions. > Testing > >> > every > >> > > > > > single > >> > > > > > > patch in real clusters is an important part of that. All I > >> can do > >> > > is > >> > > > > > > encourage people running real clusters to deploy RCs as fast > >> as > >> > > they > >> > > > > can! > >> > > > > > > Fwiw, we have already incorporated all the 0.12.2 patches > into > >> > our > >> > > > > Imply > >> > > > > > > distro of Druid and already have a good number users running > >> > them. > >> > > So > >> > > > > my > >> > > > > > +1 > >> > > > > > > earlier incorporated knowledge that the patches have been > >> > validated > >> > > > in > >> > > > > > that > >> > > > > > > way. > >> > > > > > > > >> > > > > > > I agree with Jihoon that we will probably end up doing an > >> 0.12.3 > >> > > > soon, > >> > > > > to > >> > > > > > > fix the issues he mentioned and a couple of others as well. > >> > > > > > > > >> > > > > > > On Wed, Aug 8, 2018 at 12:07 PM Jihoon Son < > >> jihoon...@apache.org > >> > > > >> > > > > wrote: > >> > > > > > > > >> > > > > > >> Charles, thank you for doing performance evaluation! > >> Performance > >> > > > > numbers > >> > > > > > >> are always good and helpful. > >> > > > > > >> > >> > > > > > >> However, IMO, any kind of performance degradation shouldn't > >> be a > >> > > > > blocker > >> > > > > > >> for this release. 0.12.2 is a minor release and contains > only > >> > bug > >> > > > > fixes. > >> > > > > > >>
Re: Druid 0.12.2 release vote
Nice!! Although I don't see the graphic attached, maybe the mailing list ate it? On Wed, Aug 8, 2018 at 4:15 PM Charles Allen wrote: > Blue is 0.12.2 with some minor backports not perf related. Red is from the > 0.11.x series. This is effectively a bucketed PDF of the query times for a > live cluster with Timeseries queries as self-reported by historical nodes. > I mentioned elsewhere I'm not convinced query/time is a good proxy for user > experience, but it does provide a good baseline for comparisons between > versions. Low query times are suspected due to some aggressive caching or > complete node misses (node very little data for that time range for that > datasource). And high query time outliers are often the result of bad GC. > > On our side there is a new java version going out with the 0.12.2 > deployment so it is unclear how much is attributed to the new java version > and how much is attributed to the druid jars or other config changes. > Overall things seem to consistently display a small % improvement in the > mean with our internal 0.12.2 release. This is good! > > Cheers, > Charles Allen > > [image: Screen Shot 2018-08-08 at 4.01.24 PM.png] > > > On Wed, Aug 8, 2018 at 3:11 PM David Lim wrote: > >> +1, thank you! >> >> On Wed, Aug 8, 2018 at 3:16 PM Jonathan Wei wrote: >> >> > +1, thanks Jihoon! >> > >> > On Wed, Aug 8, 2018 at 1:18 PM, Jihoon Son >> wrote: >> > >> > > Awesome! Thanks Charles! >> > > >> > > Jihoon >> > > >> > > On Wed, Aug 8, 2018 at 1:16 PM Gian Merlino wrote: >> > > >> > > > Thanks, it will be nice to see! >> > > > >> > > > On Wed, Aug 8, 2018 at 1:15 PM Charles Allen < >> charles.al...@snap.com >> > > > .invalid> >> > > > wrote: >> > > > >> > > > > I don't think it should be a blocker to release, but I have to run >> > perf >> > > > > tests for rollouts anyways so I figured I'd publish what I find >> :-P >> > > > > >> > > > > Cheers, >> > > > > Charles Allen >> > > > > >> > > > > >> > > > > On Wed, Aug 8, 2018 at 12:33 PM Gian Merlino >> > wrote: >> > > > > >> > > > > > That being said, Charles I am definitely looking forward to your >> > > report >> > > > > of >> > > > > > what the upgrade from 0.11 -> 0.12.2-rc1 is like in your >> cluster! >> > > > > > >> > > > > > On Wed, Aug 8, 2018 at 12:30 PM Gian Merlino >> > > wrote: >> > > > > > >> > > > > > > My thought is that recently we have started doing small >> bug-fix >> > > > > releases >> > > > > > > more often (0.12.1 and 0.12.2 were both small releases) and I >> > think >> > > > it >> > > > > > > makes sense to continue this practice. It makes sense to get >> them >> > > out >> > > > > > > quickly, since shipping bug fixes is good. IMO trying to >> validate >> > > bug >> > > > > fix >> > > > > > > releases within the customary Apache style 72 hour voting >> period >> > > is a >> > > > > > good >> > > > > > > goal. >> > > > > > > >> > > > > > > On the other hand we do strive to put out high quality >> releases, >> > > and >> > > > we >> > > > > > > don't want bug fix releases to introduce regressions. Testing >> > every >> > > > > > single >> > > > > > > patch in real clusters is an important part of that. All I >> can do >> > > is >> > > > > > > encourage people running real clusters to deploy RCs as fast >> as >> > > they >> > > > > can! >> > > > > > > Fwiw, we have already incorporated all the 0.12.2 patches into >> > our >> > > > > Imply >> > > > > > > distro of Druid and already have a good number users running >> > them. >> > > So >> > > > > my >> > > > > > +1 >> > > > > > > earlier incorporated knowledge that the patches have been >> > validated >> > > > in >> > > > > > that >> > > > > > > way. >> > > > > > > >> > > > > > > I agree with Jihoon that we will probably end up doing an >> 0.12.3 >> > > > soon, >> > > > > to >> > > > > > > fix the issues he mentioned and a couple of others as well. >> > > > > > > >> > > > > > > On Wed, Aug 8, 2018 at 12:07 PM Jihoon Son < >> jihoon...@apache.org >> > > >> > > > > wrote: >> > > > > > > >> > > > > > >> Charles, thank you for doing performance evaluation! >> Performance >> > > > > numbers >> > > > > > >> are always good and helpful. >> > > > > > >> >> > > > > > >> However, IMO, any kind of performance degradation shouldn't >> be a >> > > > > blocker >> > > > > > >> for this release. 0.12.2 is a minor release and contains only >> > bug >> > > > > fixes. >> > > > > > >> https://github.com/apache/incubator-druid/pull/5878/files is >> > the >> > > > only >> > > > > > one >> > > > > > >> tagged with 'Performance', but it can be regarded as a more >> > like a >> > > > > code >> > > > > > >> bug >> > > > > > >> rather than architectural performance issue. >> > > > > > >> >> > > > > > >> Instead, those kinds of performance tests should be performed >> > per >> > > > > major >> > > > > > >> release to catch any kinds of unexpected performance change. >> > They >> > > > can >> > > > > > be a >> > > > > > >> blocker if we find any performance regression. >> > > > > > >>
