Re: Proposal for 1.3 branch and release
+1 Thank you for taking care of the RM as well. I don't think it's the best time for me to do RM this time, sorry. Best regards, Alexey On Tue, Feb 21, 2017 at 10:32 AM, Todd Lipcon wrote: > Hi folks, > > I don't think the topic of Kudu 1.3 has been brought up on the dev list yet > (though I just hinted at it on a user@ thread). I wanted to open this > topic > to discussion by proposing a release plan. > > Back in December, the discussion around the 1.2 release was to hold off on > the security features, and push them to 1.3. So, obviously security has > been a big focus of development over the last month or so. I've been pretty > involved with this development and it seems from my perspective that the > initial phase of development (authentication without authorization) is > wrapping up, and would be good to get out to users. > > Aside from security, there are a bunch of good bug fixes and improvements. > From a quick scan of the git log, a few things of note are: > > - fixes to various cases where more disk space was consumed than actually > used for data > - support for using Breakpad for crash analysis > - new default encodings/compression to save space and increase performance > - compression of the WAL and deltafiles (performance and space) > - support for IS NULL and IS NOT NULL predicates > - improved hash-partition pruning for IN-list predicates > > ...plus a few nice bug fixes and stability improvements. > > All of the above seems like it should make for a compelling 1.3 release. > > Following the pattern of 1.2, I think we should plan to branch this week, > and then spend a week or two on stabilization and testing before making the > release candidate. > > I'll volunteer to be the RM again unles anyone else feels like taking a > turn. > > Any comments or concerns? > > -Todd > > -- > Todd Lipcon > Software Engineer, Cloudera >
Re: Proposal for 1.3 branch and release
+1 On Tue, Feb 21, 2017 at 11:22 AM, Dan Burkert wrote: > +1 SGTM, thanks! > > - Dan > > On Tue, Feb 21, 2017 at 11:19 AM, Hao Hao wrote: > >> +1 Thanks a lot Todd for putting this together and volunteered to be the >> RM. >> >> Best, >> Hao >> >> On Tue, Feb 21, 2017 at 10:40 AM, Dave Wang wrote: >> >> > Strong +1 on your proposal Todd. >> > >> > - Dave >> > >> > On Tue, Feb 21, 2017 at 10:39 AM, Jean-Daniel Cryans < >> jdcry...@apache.org> >> > wrote: >> > >> > > +1 on the plan and +1 on you being the RM since you're so close to all >> > the >> > > security-related patches. >> > > >> > > J-D >> > > >> > > On Tue, Feb 21, 2017 at 10:32 AM, Todd Lipcon wrote: >> > > >> > > > Hi folks, >> > > > >> > > > I don't think the topic of Kudu 1.3 has been brought up on the dev >> list >> > > yet >> > > > (though I just hinted at it on a user@ thread). I wanted to open >> this >> > > > topic >> > > > to discussion by proposing a release plan. >> > > > >> > > > Back in December, the discussion around the 1.2 release was to hold >> off >> > > on >> > > > the security features, and push them to 1.3. So, obviously security >> has >> > > > been a big focus of development over the last month or so. I've been >> > > pretty >> > > > involved with this development and it seems from my perspective that >> > the >> > > > initial phase of development (authentication without authorization) >> is >> > > > wrapping up, and would be good to get out to users. >> > > > >> > > > Aside from security, there are a bunch of good bug fixes and >> > > improvements. >> > > > From a quick scan of the git log, a few things of note are: >> > > > >> > > > - fixes to various cases where more disk space was consumed than >> > actually >> > > > used for data >> > > > - support for using Breakpad for crash analysis >> > > > - new default encodings/compression to save space and increase >> > > performance >> > > > - compression of the WAL and deltafiles (performance and space) >> > > > - support for IS NULL and IS NOT NULL predicates >> > > > - improved hash-partition pruning for IN-list predicates >> > > > >> > > > ...plus a few nice bug fixes and stability improvements. >> > > > >> > > > All of the above seems like it should make for a compelling 1.3 >> > release. >> > > > >> > > > Following the pattern of 1.2, I think we should plan to branch this >> > week, >> > > > and then spend a week or two on stabilization and testing before >> making >> > > the >> > > > release candidate. >> > > > >> > > > I'll volunteer to be the RM again unles anyone else feels like >> taking a >> > > > turn. >> > > > >> > > > Any comments or concerns? >> > > > >> > > > -Todd >> > > > >> > > > -- >> > > > Todd Lipcon >> > > > Software Engineer, Cloudera >> > > > >> > > >> > >>
Tweaked some gerrit httpd settings
I always have slowness issues with gerrit, so I just took a quick look at the httpd config and noticed that KeepAlive was off. I turned it on, with a low KeepAlive timeout, and also bumped the max number of workers (since keepalive connections hold workers open). Ideally we'd switch to a better server than httpd for SSL termination, but that's more work than I have time for. Hoping that the keepalive setting will make things a bit snappier going forward, since we'll avoid a bunch of round trips and small TCP window sizes when fetching page assets, etc. LMK if you hit any issues and I'll revert the change. -Todd -- Todd Lipcon Software Engineer, Cloudera
Re: Proposal for 1.3 branch and release
+1 SGTM, thanks! - Dan On Tue, Feb 21, 2017 at 11:19 AM, Hao Hao wrote: > +1 Thanks a lot Todd for putting this together and volunteered to be the > RM. > > Best, > Hao > > On Tue, Feb 21, 2017 at 10:40 AM, Dave Wang wrote: > > > Strong +1 on your proposal Todd. > > > > - Dave > > > > On Tue, Feb 21, 2017 at 10:39 AM, Jean-Daniel Cryans < > jdcry...@apache.org> > > wrote: > > > > > +1 on the plan and +1 on you being the RM since you're so close to all > > the > > > security-related patches. > > > > > > J-D > > > > > > On Tue, Feb 21, 2017 at 10:32 AM, Todd Lipcon wrote: > > > > > > > Hi folks, > > > > > > > > I don't think the topic of Kudu 1.3 has been brought up on the dev > list > > > yet > > > > (though I just hinted at it on a user@ thread). I wanted to open > this > > > > topic > > > > to discussion by proposing a release plan. > > > > > > > > Back in December, the discussion around the 1.2 release was to hold > off > > > on > > > > the security features, and push them to 1.3. So, obviously security > has > > > > been a big focus of development over the last month or so. I've been > > > pretty > > > > involved with this development and it seems from my perspective that > > the > > > > initial phase of development (authentication without authorization) > is > > > > wrapping up, and would be good to get out to users. > > > > > > > > Aside from security, there are a bunch of good bug fixes and > > > improvements. > > > > From a quick scan of the git log, a few things of note are: > > > > > > > > - fixes to various cases where more disk space was consumed than > > actually > > > > used for data > > > > - support for using Breakpad for crash analysis > > > > - new default encodings/compression to save space and increase > > > performance > > > > - compression of the WAL and deltafiles (performance and space) > > > > - support for IS NULL and IS NOT NULL predicates > > > > - improved hash-partition pruning for IN-list predicates > > > > > > > > ...plus a few nice bug fixes and stability improvements. > > > > > > > > All of the above seems like it should make for a compelling 1.3 > > release. > > > > > > > > Following the pattern of 1.2, I think we should plan to branch this > > week, > > > > and then spend a week or two on stabilization and testing before > making > > > the > > > > release candidate. > > > > > > > > I'll volunteer to be the RM again unles anyone else feels like > taking a > > > > turn. > > > > > > > > Any comments or concerns? > > > > > > > > -Todd > > > > > > > > -- > > > > Todd Lipcon > > > > Software Engineer, Cloudera > > > > > > > > > >
Re: Proposal for 1.3 branch and release
+1 Thanks a lot Todd for putting this together and volunteered to be the RM. Best, Hao On Tue, Feb 21, 2017 at 10:40 AM, Dave Wang wrote: > Strong +1 on your proposal Todd. > > - Dave > > On Tue, Feb 21, 2017 at 10:39 AM, Jean-Daniel Cryans > wrote: > > > +1 on the plan and +1 on you being the RM since you're so close to all > the > > security-related patches. > > > > J-D > > > > On Tue, Feb 21, 2017 at 10:32 AM, Todd Lipcon wrote: > > > > > Hi folks, > > > > > > I don't think the topic of Kudu 1.3 has been brought up on the dev list > > yet > > > (though I just hinted at it on a user@ thread). I wanted to open this > > > topic > > > to discussion by proposing a release plan. > > > > > > Back in December, the discussion around the 1.2 release was to hold off > > on > > > the security features, and push them to 1.3. So, obviously security has > > > been a big focus of development over the last month or so. I've been > > pretty > > > involved with this development and it seems from my perspective that > the > > > initial phase of development (authentication without authorization) is > > > wrapping up, and would be good to get out to users. > > > > > > Aside from security, there are a bunch of good bug fixes and > > improvements. > > > From a quick scan of the git log, a few things of note are: > > > > > > - fixes to various cases where more disk space was consumed than > actually > > > used for data > > > - support for using Breakpad for crash analysis > > > - new default encodings/compression to save space and increase > > performance > > > - compression of the WAL and deltafiles (performance and space) > > > - support for IS NULL and IS NOT NULL predicates > > > - improved hash-partition pruning for IN-list predicates > > > > > > ...plus a few nice bug fixes and stability improvements. > > > > > > All of the above seems like it should make for a compelling 1.3 > release. > > > > > > Following the pattern of 1.2, I think we should plan to branch this > week, > > > and then spend a week or two on stabilization and testing before making > > the > > > release candidate. > > > > > > I'll volunteer to be the RM again unles anyone else feels like taking a > > > turn. > > > > > > Any comments or concerns? > > > > > > -Todd > > > > > > -- > > > Todd Lipcon > > > Software Engineer, Cloudera > > > > > >
Re: Proposal for 1.3 branch and release
Strong +1 on your proposal Todd. - Dave On Tue, Feb 21, 2017 at 10:39 AM, Jean-Daniel Cryans wrote: > +1 on the plan and +1 on you being the RM since you're so close to all the > security-related patches. > > J-D > > On Tue, Feb 21, 2017 at 10:32 AM, Todd Lipcon wrote: > > > Hi folks, > > > > I don't think the topic of Kudu 1.3 has been brought up on the dev list > yet > > (though I just hinted at it on a user@ thread). I wanted to open this > > topic > > to discussion by proposing a release plan. > > > > Back in December, the discussion around the 1.2 release was to hold off > on > > the security features, and push them to 1.3. So, obviously security has > > been a big focus of development over the last month or so. I've been > pretty > > involved with this development and it seems from my perspective that the > > initial phase of development (authentication without authorization) is > > wrapping up, and would be good to get out to users. > > > > Aside from security, there are a bunch of good bug fixes and > improvements. > > From a quick scan of the git log, a few things of note are: > > > > - fixes to various cases where more disk space was consumed than actually > > used for data > > - support for using Breakpad for crash analysis > > - new default encodings/compression to save space and increase > performance > > - compression of the WAL and deltafiles (performance and space) > > - support for IS NULL and IS NOT NULL predicates > > - improved hash-partition pruning for IN-list predicates > > > > ...plus a few nice bug fixes and stability improvements. > > > > All of the above seems like it should make for a compelling 1.3 release. > > > > Following the pattern of 1.2, I think we should plan to branch this week, > > and then spend a week or two on stabilization and testing before making > the > > release candidate. > > > > I'll volunteer to be the RM again unles anyone else feels like taking a > > turn. > > > > Any comments or concerns? > > > > -Todd > > > > -- > > Todd Lipcon > > Software Engineer, Cloudera > > >
Re: Proposal for 1.3 branch and release
+1 on the plan and +1 on you being the RM since you're so close to all the security-related patches. J-D On Tue, Feb 21, 2017 at 10:32 AM, Todd Lipcon wrote: > Hi folks, > > I don't think the topic of Kudu 1.3 has been brought up on the dev list yet > (though I just hinted at it on a user@ thread). I wanted to open this > topic > to discussion by proposing a release plan. > > Back in December, the discussion around the 1.2 release was to hold off on > the security features, and push them to 1.3. So, obviously security has > been a big focus of development over the last month or so. I've been pretty > involved with this development and it seems from my perspective that the > initial phase of development (authentication without authorization) is > wrapping up, and would be good to get out to users. > > Aside from security, there are a bunch of good bug fixes and improvements. > From a quick scan of the git log, a few things of note are: > > - fixes to various cases where more disk space was consumed than actually > used for data > - support for using Breakpad for crash analysis > - new default encodings/compression to save space and increase performance > - compression of the WAL and deltafiles (performance and space) > - support for IS NULL and IS NOT NULL predicates > - improved hash-partition pruning for IN-list predicates > > ...plus a few nice bug fixes and stability improvements. > > All of the above seems like it should make for a compelling 1.3 release. > > Following the pattern of 1.2, I think we should plan to branch this week, > and then spend a week or two on stabilization and testing before making the > release candidate. > > I'll volunteer to be the RM again unles anyone else feels like taking a > turn. > > Any comments or concerns? > > -Todd > > -- > Todd Lipcon > Software Engineer, Cloudera >
Proposal for 1.3 branch and release
Hi folks, I don't think the topic of Kudu 1.3 has been brought up on the dev list yet (though I just hinted at it on a user@ thread). I wanted to open this topic to discussion by proposing a release plan. Back in December, the discussion around the 1.2 release was to hold off on the security features, and push them to 1.3. So, obviously security has been a big focus of development over the last month or so. I've been pretty involved with this development and it seems from my perspective that the initial phase of development (authentication without authorization) is wrapping up, and would be good to get out to users. Aside from security, there are a bunch of good bug fixes and improvements. >From a quick scan of the git log, a few things of note are: - fixes to various cases where more disk space was consumed than actually used for data - support for using Breakpad for crash analysis - new default encodings/compression to save space and increase performance - compression of the WAL and deltafiles (performance and space) - support for IS NULL and IS NOT NULL predicates - improved hash-partition pruning for IN-list predicates ...plus a few nice bug fixes and stability improvements. All of the above seems like it should make for a compelling 1.3 release. Following the pattern of 1.2, I think we should plan to branch this week, and then spend a week or two on stabilization and testing before making the release candidate. I'll volunteer to be the RM again unles anyone else feels like taking a turn. Any comments or concerns? -Todd -- Todd Lipcon Software Engineer, Cloudera
Re: TSAN false positives with std::call_once
To close the loop on this, I put up a patch here: http://gerrit.cloudera.org:8080/6095 -Todd On Tue, Feb 14, 2017 at 6:32 PM, Todd Lipcon wrote: > On Tue, Feb 14, 2017 at 6:22 PM, Adar Dembo wrote: > >> Any idea why this only surfaced recently? There's a call to >> std::call_once in util/logging.cc that's been there for almost a year. >> > > I'm guessing that one is only very rarely called (on crash) and would only > be called by multiple threads at or near the same time in an even rarer > case (two threads crash simultaneously). > > >> In any case, I'm not a huge fan of #2. #1 and #3 both require a patch, >> but I think #1 is more likely to get noticed (and reverted) upon an >> LLVM 4.0 upgrade. So I think I vote for #1, though #3 is reasonable >> too. >> > > OK, I'll go with #1. I tested that the patch applies cleanly and fixes a > manufactured test case I wrote. > > >> >> On Tue, Feb 14, 2017 at 6:09 PM, Todd Lipcon wrote: >> > BTW, I should add that libstdcxx as included in gcc 4.9.2 (devtoolset on >> > el6) doesn't have this bug, since it just wraps pthread_once. So, it's >> > probably not a _real_ issue since we don't use libc++ for production >> builds. >> > >> > -Todd >> > >> > On Tue, Feb 14, 2017 at 6:05 PM, Todd Lipcon wrote: >> > >> >> I noticed that a lot of our tests have gotten flaky with a TSAN race >> >> reported around SSL initialization. After digging into a bit, I found >> that >> >> our version of libcxx has a bug in std::once where it's using a relaxed >> >> load instead of an acquire load to check the flag of whether it has >> run yet. >> >> >> >> I went to go report this upstream and discovered it's actually already >> >> fixed: >> >> https://github.com/llvm-mirror/libcxx/commit/ >> >> 4dbd4fcf8626453949625bf36976e668094cfbb1 >> >> >> >> I'd still like to resolve the race, though. A couple options: >> >> 1) apply this patch to our libcxx >> >> 2) use GoogleOnce instead of std::call_once >> >> 3) suppress it until we upgrade to llvm 4.0 with the newer libcxx, >> which >> >> should likely be released in a few weeks >> >> >> >> Any opinions? >> >> -Todd >> >> -- >> >> Todd Lipcon >> >> Software Engineer, Cloudera >> >> >> > >> > >> > >> > -- >> > Todd Lipcon >> > Software Engineer, Cloudera >> > > > > -- > Todd Lipcon > Software Engineer, Cloudera > -- Todd Lipcon Software Engineer, Cloudera
