I have merged all pending Jiras (including blockers) and will start building RC0 shortly.
*Note for the committers:* *until the release is not over and Drill version is not changed to 1.12.0-SNAPSHOT, please do not push any changes into Drill master. * Kind regards Arina On Wed, Jul 19, 2017 at 12:15 PM, Arina Yelchiyeva < [email protected]> wrote: > Thanks, Boaz! > > Unfortunately, I still see the failures connected to DRILL-5669. > I'll share the details. > > Kind regards > Arina > > On Wed, Jul 19, 2017 at 2:00 AM, Boaz Ben-Zvi <[email protected]> wrote: > >> Hi Arina, >> >> The three Jira’s are ready (+1, squashed, rebased, etc.): >> >> • DRILL-5616: Add memory checks, plus minor metrics changes >> (Commit ID: c7ad56b7e075e84761896c9ce660b18b09d9e49a ) >> • DRILL-5665: Add the option planner.force_2phase_aggr to override small >> inputs >> (Commit ID: 1917ec084bdef74962370fbe5b253172c634c86e ) >> • DRILL-5669: Add a configurable option for minimum memory allocation to >> buffered ops >> (Commit ID: 4d7a3fe3e7ae52db57b0fd1f0619c6d1d64308c8 ) >> >> Thanks, >> >> Boaz >> >> On 7/18/17, 3:04 AM, "Arina Yelchiyeva" <[email protected]> >> wrote: >> >> Hi Parth, >> >> no problem, I did not start preparing the release since there is one >> blocker has left: >> >> *DRILL-5669 - Boaz Ben-Zvi (fix will be ready in a couple of days)* >> >> >> Kind regards >> Arina >> >> On Mon, Jul 17, 2017 at 11:26 PM, Parth Chandra <[email protected]> >> wrote: >> >> > Hi Arina, >> > >> > I just realized that I did not push the C++ client fix ( >> DRILL-5659) to >> > master. I have just done so. >> > >> > Sorry if I messed things up for the release, but we should get >> this fix >> > in. >> > >> > Parth >> > >> > On Fri, Jul 14, 2017 at 8:15 AM, Paul Rogers <[email protected]> >> wrote: >> > >> > > I looked into one NPE on CSV. Turns out it is already fixed by >> previous >> > > 1.11 PRs, so we’re good. >> > > >> > > There is another reported by a user on the user list, but I’ve >> not heard >> > > back about getting sample data. Let’s not hold the release for >> that. >> > > >> > > - Paul >> > > >> > > > On Jul 14, 2017, at 2:38 AM, Arina Yelchiyeva < >> > > [email protected]> wrote: >> > > > >> > > > *Blockers* >> > > > DRILL-5659 - Parth (done) >> > > > DRILL-5669 - Boaz Ben-Zvi (fix will be ready in a couple of >> days) >> > > > >> > > > *Possible blockers* >> > > > NPE on CSV - Paul can you please provide an update? >> > > > >> > > > Kind regards >> > > > Arina >> > > > >> > > > On Thu, Jul 13, 2017 at 2:46 PM, Arina Yelchiyeva < >> > > > [email protected]> wrote: >> > > > >> > > >> *Blockers* >> > > >> DRILL-5659 - problem is identified, Parth is working on the >> fix. >> > > >> >> > > >> *Not blockers* >> > > >> DRILL-5660 - Vitalii >> > > >> DRILL-5468 - Jinfeng >> > > >> >> > > >> *Possible blockers* >> > > >> DRILL-5669 - Boaz Ben-Zvi / Paul >> > > >> NPE on CSV - Paul >> > > >> >> > > >> Kind regards >> > > >> Arina >> > > >> >> > > >> >> > > >> On Thu, Jul 13, 2017 at 1:28 AM, Paul Rogers <[email protected] >> > >> > wrote: >> > > >> >> > > >>> I withdraw my suggestion to fix DRILL-5660. Was a big hassle >> for us >> > to >> > > >>> track down the incompatibility the issue internally, but >> production >> > > users >> > > >>> probably don’t attempt to use one build of Drill with >> metadata files >> > > >>> created from another. We can worry about the issue later only >> if a >> > user >> > > >>> actually runs into it. >> > > >>> >> > > >>> As suggested earlier, let’s just document this obscure case >> in a >> > > release >> > > >>> note. >> > > >>> >> > > >>> - Paul >> > > >>> >> > > >>>> On Jul 12, 2017, at 3:04 PM, Jinfeng Ni <[email protected]> >> wrote: >> > > >>>> >> > > >>>> Upgrade from 1.10 to 1.11 is irreversible only when parquet >> metadata >> > > >>> cache >> > > >>>> file is involved. >> > > >>>> >> > > >>>> I might be wrong, but I thought that if someone is running on >> > 1.10.0, >> > > >>> and >> > > >>>> he wants to go back to 1.0.0, where the first version of >> parquet >> > > >>> meatadata >> > > >>>> cache was introduced, can we guarantee that Drillbit would >> work >> > > >>> correctly? >> > > >>>> I'm trying to understand what DRILL-5660 asks for is a new >> > > requirement, >> > > >>> or >> > > >>>> an requirement being there starting from Drill 1.0.0? >> > > >>>> >> > > >>>> Regression failures are normally regarded as release blocker. >> > > >>>> >> > > >>>> >> > > >>>> >> > > >>>> On Wed, Jul 12, 2017 at 2:54 PM, Paul Rogers < >> [email protected]> >> > > wrote: >> > > >>>> >> > > >>>>> If we don’t do DRILL-5660 for 1.11 then we don’t need to do >> it at >> > > all, >> > > >>> so >> > > >>>>> that is a time savings. We could handle this via a release >> note >> > item >> > > >>> that >> > > >>>>> simply states that an upgrade from 1.10 to 1.11 is >> irreversible: we >> > > do >> > > >>> not >> > > >>>>> support the ability to fall back to the old version. >> Everyone OK >> > with >> > > >>> that? >> > > >>>>> >> > > >>>>> We are seeing some stress test regression failures that, >> I’m told, >> > > will >> > > >>>>> show up as JIRA entries shortly. >> > > >>>>> >> > > >>>>> - Paul >> > > >>>>> >> > > >>>>>> On Jul 12, 2017, at 11:05 AM, Jinfeng Ni <[email protected]> >> wrote: >> > > >>>>>> >> > > >>>>>> I put some analysis in DRILL-5468. >> > > >>>>>> >> > > >>>>>> Also, IMHO, DRILL-5660 is not a release blocker. >> > > >>>>>> >> > > >>>>>> >> > > >>>>>> >> > > >>>>>> On Tue, Jul 11, 2017 at 11:27 AM, Parth Chandra < >> > [email protected]> >> > > >>>>> wrote: >> > > >>>>>> >> > > >>>>>>> Re DRILL-5634: >> > > >>>>>>> From the Apache commons web page [1] - >> > > >>>>>>> >> > > >>>>>>> Please note that Apache Commons Crypto doesn't implement >> the >> > > >>>>> cryptographic >> > > >>>>>>> algorithm such as AES directly.* It wraps to Openssl or >> JCE which >> > > >>>>> implement >> > > >>>>>>> the algorithms.* >> > > >>>>>>> >> > > >>>>>>> >> > > >>>>>>> Since we are cleared for using Openssl and JCE, we are OK >> to >> > > include >> > > >>> the >> > > >>>>>>> AES code in the UDFs. >> > > >>>>>>> >> > > >>>>>>> >> > > >>>>>>> Parth >> > > >>>>>>> >> > > >>>>>>> [1] https://commons.apache.org/proper/commons-crypto/ >> > > >>>>>>> >> > > >>>>>>> On Tue, Jul 11, 2017 at 10:56 AM, Arina Yelchiyeva < >> > > >>>>>>> [email protected]> wrote: >> > > >>>>>>> >> > > >>>>>>>> During today hangouts we agreed on approximate cut-off >> date - 14 >> > > >>> July, >> > > >>>>>>> 2017 >> > > >>>>>>>> (date may be shifted due to the blockers). >> > > >>>>>>>> I'll start preparing test RC on Thursday to solve any >> release >> > > >>>>> preparation >> > > >>>>>>>> issues beforehand. >> > > >>>>>>>> >> > > >>>>>>>> *Blockers* >> > > >>>>>>>> DRILL-5660 - Vitalii >> > > >>>>>>>> >> > > >>>>>>>> *Possible blockers* >> > > >>>>>>>> DRILL-5468 - Jinfeng >> > > >>>>>>>> NPE on CSV source - Paul >> > > >>>>>>>> DRILL-5659 - Parth >> > > >>>>>>>> >> > > >>>>>>>> *Will be included if done before the cut-off date* >> > > >>>>>>>> DRILL-5616 / DRILL-5665 - Boaz >> > > >>>>>>>> DRILL-5634 - Parth will take a look at Apache commons >> > > implementation >> > > >>>>> and >> > > >>>>>>>> update. >> > > >>>>>>>> >> > > >>>>>>>> Kind regards >> > > >>>>>>>> Arina >> > > >>>>>>>> >> > > >>>>>>>> On Tue, Jul 11, 2017 at 3:14 PM, Charles Givre < >> > [email protected]> >> > > >>>>> wrote: >> > > >>>>>>>> >> > > >>>>>>>>> Hi Arina, >> > > >>>>>>>>> I can’t make the hangout, but we have a few options. >> > Basically, >> > > >>> since >> > > >>>>>>>> all >> > > >>>>>>>>> the hashing algorithms have no export restrictions, we >> could: >> > > >>>>>>>>> 1. Remove the AES encrypt/decrypt until I can figure >> out what >> > we >> > > >>> have >> > > >>>>>>> to >> > > >>>>>>>>> do (and do it) >> > > >>>>>>>>> 2. I can try to research it today or tomorrow to >> figure out >> > > >>> exactly >> > > >>>>>>> what >> > > >>>>>>>>> we have to do to include AES. >> > > >>>>>>>>> 3. Remove the entire package >> > > >>>>>>>>> >> > > >>>>>>>>> I’d like to see the hashing functions at least included >> because >> > > >>> we’ve >> > > >>>>>>> had >> > > >>>>>>>>> a lot of requests for that functionality. I also >> don’t want >> > > this >> > > >>> to >> > > >>>>>>>> hold >> > > >>>>>>>>> things up, so please let me know your preference. >> Regardless, >> > > I’m >> > > >>>>>>> going >> > > >>>>>>>> to >> > > >>>>>>>>> start trying to figure out the restrictions so we’ll >> know for >> > the >> > > >>>>>>> future. >> > > >>>>>>>>> >> > > >>>>>>>>> Best, >> > > >>>>>>>>> — C >> > > >>>>>>>>> >> > > >>>>>>>>> >> > > >>>>>>>>> >> > > >>>>>>>>>> On Jul 11, 2017, at 05:58, Arina Yelchiyeva < >> > > >>>>>>>> [email protected]> >> > > >>>>>>>>> wrote: >> > > >>>>>>>>>> >> > > >>>>>>>>>> @Jinfeng, >> > > >>>>>>>>>> Let's discuss today on Drill hangout final cut-off >> date. >> > > >>>>>>>>>> >> > > >>>>>>>>>> @Charles >> > > >>>>>>>>>> Since there were questions about AES in DRILL-5634, >> should we >> > > >>> exclude >> > > >>>>>>>> it >> > > >>>>>>>>>> from the candidates for 1.11.0 release? >> > > >>>>>>>>>> >> > > >>>>>>>>>> @Boaz >> > > >>>>>>>>>> Do we have any ETA when DRILL-5616 and DRILL-5665 will >> pass >> > code >> > > >>>>>>>> review? >> > > >>>>>>>>>> >> > > >>>>>>>>>> Also I believe we should include DRILL-5660 where we >> need to >> > > bump >> > > >>> up >> > > >>>>>>>>>> parquet metadata version. >> > > >>>>>>>>>> @Vitalii >> > > >>>>>>>>>> Do you have any ETA for the fix? >> > > >>>>>>>>>> >> > > >>>>>>>>>> Kind regards >> > > >>>>>>>>>> Arina >> > > >>>>>>>>>> >> > > >>>>>>>>>> On Tue, Jul 11, 2017 at 6:02 AM, Boaz Ben-Zvi < >> > > [email protected]> >> > > >>>>>>>> wrote: >> > > >>>>>>>>>> >> > > >>>>>>>>>>> Hi Arina, >> > > >>>>>>>>>>> >> > > >>>>>>>>>>> Two PR have just been submitted, relating to the >> Hash Aggr >> > > >>>>>>> Spill: >> > > >>>>>>>>>>> DRILL-5616 and DRILL-5665 . >> > > >>>>>>>>>>> Can these be considered for 1.11 as well ? >> > > >>>>>>>>>>> >> > > >>>>>>>>>>> Thanks, >> > > >>>>>>>>>>> >> > > >>>>>>>>>>> Boaz >> > > >>>>>>>>>>> >> > > >>>>>>>>>>> On 7/10/17, 1:49 PM, "Paul Rogers" <[email protected]> >> wrote: >> > > >>>>>>>>>>> >> > > >>>>>>>>>>> DRILL-5601 is no longer a blocker for 1.11; we’ll >> defer that >> > > >>>>>>> until >> > > >>>>>>>>>>> after the release to allow others more time to work >> on other >> > > >>>>>>> features >> > > >>>>>>>>>>> rather than doing code reviews. >> > > >>>>>>>>>>> >> > > >>>>>>>>>>> - Paul >> > > >>>>>>>>>>> >> > > >>>>>>>>>>>> On Jul 10, 2017, at 1:43 PM, Jinfeng Ni < >> [email protected]> >> > > wrote: >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> Hi Arina, >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> It has been a while since we discussed 1.11.0 >> release. Any >> > > >>> update >> > > >>>>>>>>>>> on the >> > > >>>>>>>>>>>> new release plan? Do we have a cut-off date ? >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> In the past, I believe Drill release is a >> time-bounded, not >> > > >>> feature >> > > >>>>>>>>>>>> bounded. If certain fixes/features could not make it >> within >> > a >> > > >>>>>>> certain >> > > >>>>>>>>>>>> time-frame, we had better defer them into future >> release, >> > > unless >> > > >>>>>>>>>>> those >> > > >>>>>>>>>>>> fixes/features are regarded as blocking issues for a >> new >> > > >>> release. >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>> On Tue, Jul 4, 2017 at 5:14 AM, Arina Yelchiyeva < >> > > >>>>>>>>>>> [email protected] >> > > >>>>>>>>>>>>> wrote: >> > > >>>>>>>>>>>> >> > > >>>>>>>>>>>>> Latest update: >> > > >>>>>>>>>>>>> >> > > >>>>>>>>>>>>> We wait for the following Jiras before final cut >> off. >> > > >>>>>>>>>>>>> 1. DRILL-5601 - Paul >> > > >>>>>>>>>>>>> 2. DRILL-5420 - Kunal >> > > >>>>>>>>>>>>> 3. DRILL-5634 - Charles >> > > >>>>>>>>>>>>> >> > > >>>>>>>>>>>>> Kind regards >> > > >>>>>>>>>>>>> Arina >> > > >>>>>>>>>>>>> >> > > >>>>>>>>>>>>> On Fri, Jun 30, 2017 at 9:29 PM, Kunal Khatua < >> > > >>> [email protected]> >> > > >>>>>>>>>>> wrote: >> > > >>>>>>>>>>>>> >> > > >>>>>>>>>>>>>> Parth's already +1'd the process with a minor >> update. So, >> > > >>> we're >> > > >>>>>>>>>>> good to >> > > >>>>>>>>>>>>> go >> > > >>>>>>>>>>>>>> on DRILL-5420. I'll change the label on that to >> > > >>> ready-to-commit. >> > > >>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>> <http://www.mapr.com/> >> > > >>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>> ________________________________ >> > > >>>>>>>>>>>>>> From: Parth Chandra <[email protected]> >> > > >>>>>>>>>>>>>> Sent: Thursday, June 29, 2017 9:44:39 AM >> > > >>>>>>>>>>>>>> To: [email protected] >> > > >>>>>>>>>>>>>> Subject: Re: [DISCUSS] Drill 1.11.0 release >> > > >>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>> I'm Ok with the pcap format plugin if Pauls >> comments are >> > > also >> > > >>>>>>>>>>> addressed. >> > > >>>>>>>>>>>>> If >> > > >>>>>>>>>>>>>> the changes are done before your cutoff >> deadline,we can >> > > merge >> > > >>> it >> > > >>>>>>>>>>> in. >> > > >>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>> On Thu, Jun 29, 2017 at 12:26 AM, Arina Yelchiyeva >> < >> > > >>>>>>>>>>>>>> [email protected]> wrote: >> > > >>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>> Hi all, >> > > >>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>> here is the list of Jira we are waiting before >> making >> > final >> > > >>> cut >> > > >>>>>>>>>>> off: >> > > >>>>>>>>>>>>>>> 1. DRILL-5601 - Paul >> > > >>>>>>>>>>>>>>> 2. DRILL-3640 - Kunal >> > > >>>>>>>>>>>>>>> 3. DRILL-5420 - Kunal >> > > >>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>> Please let me know if there are any others Jiras >> we >> > should >> > > >>> wait >> > > >>>>>>>>>>> for. >> > > >>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>> @Charles, >> > > >>>>>>>>>>>>>>> you have mentioned that you want to include some >> UDFs in >> > > >>> Drill >> > > >>>>>>>>>>> 1.11, do >> > > >>>>>>>>>>>>>> you >> > > >>>>>>>>>>>>>>> have any ETA for the PR? >> > > >>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>> @Paul / Parth >> > > >>>>>>>>>>>>>>> I see there is active CR for pcap format PR, do >> we want >> > to >> > > >>>>>>>>>>> include it >> > > >>>>>>>>>>>>> in >> > > >>>>>>>>>>>>>>> this release as well? >> > > >>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>> Kind regards >> > > >>>>>>>>>>>>>>> Arina >> > > >>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>> On Thu, Jun 29, 2017 at 10:05 AM, Arina >> Yelchiyeva < >> > > >>>>>>>>>>>>>>> [email protected]> wrote: >> > > >>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>> Parth, thanks a lot the instructions. >> > > >>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>> On Wed, Jun 28, 2017 at 8:24 PM, Parth Chandra < >> > > >>>>>>>>>>> [email protected]> >> > > >>>>>>>>>>>>>>> wrote: >> > > >>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>>> FWIW, I put release instructions culled from >> gists >> > > written >> > > >>> by >> > > >>>>>>>>>>>>> previous >> > > >>>>>>>>>>>>>>>>> release managers into an updated gist here [1] >> > > >>>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>>> HTH >> > > >>>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>>> Parth >> > > >>>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>>> [1] https://github.com/parthchandr >> a/drill/wiki/Drill- >> > > >>>>>>>>>>>>> Release-Process >> > > >>>>>>>>>>>>>>>>> <https://github.com/parthchand >> ra/drill/wiki/Drill- >> > > >>>>>>>>>>> Release-Process> >> > > >>>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>>> On Tue, Jun 27, 2017 at 5:24 PM, Paul Rogers < >> > > >>>>>>> [email protected]> >> > > >>>>>>>>>>>>>> wrote: >> > > >>>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>>>> Hi Arina, >> > > >>>>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>>>> I have two projects in progress: one is a >> candidate >> > for >> > > >>> 1.11, >> > > >>>>>>>>>>> the >> > > >>>>>>>>>>>>>>> other >> > > >>>>>>>>>>>>>>>>> is >> > > >>>>>>>>>>>>>>>>>> not. >> > > >>>>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>>>> The “managed external sort” is the good >> candidate for >> > > >>> 1.11: >> > > >>>>>>>>>>> The PR >> > > >>>>>>>>>>>>>> for >> > > >>>>>>>>>>>>>>>>>> DRILL-5601 is being reviewed. Getting that in >> will >> > allow >> > > >>> us >> > > >>>>>>> to >> > > >>>>>>>>>>>>>> enable >> > > >>>>>>>>>>>>>>>>> the >> > > >>>>>>>>>>>>>>>>>> “new” external sort by default. >> > > >>>>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>>>> The other project is to address Drill’s memory >> > > >>> fragmentation >> > > >>>>>>>>>>> issue >> > > >>>>>>>>>>>>>>> with >> > > >>>>>>>>>>>>>>>>>> vectors over 16 MB in size. Two PRs have been >> open >> > for a >> > > >>>>>>> while. >> > > >>>>>>>>>>>>> They >> > > >>>>>>>>>>>>>>> are >> > > >>>>>>>>>>>>>>>>>> the first of several. Each takes a while to >> review. >> > So, >> > > >>> the >> > > >>>>>>>>>>>>> project >> > > >>>>>>>>>>>>>>> does >> > > >>>>>>>>>>>>>>>>>> not fit 1.11. The work can go into the next >> release >> > > >>> instead. >> > > >>>>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>>>> Thanks, >> > > >>>>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>>>> - Paul >> > > >>>>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>>> >> > > >>>>>>>>>>>>>> >> > > >>>>>>>>>>>>> >> > > >>>>>>>>>>> >> > > >>>>>>>>>>> >> > > >>>>>>>>>>> >> > > >>>>>>>>>>> >> > > >>>>>>>>> >> > > >>>>>>>>> >> > > >>>>>>>> >> > > >>>>>>> >> > > >>>>> >> > > >>>>> >> > > >>> >> > > >>> >> > > >> >> > > >> > > >> > >> >> >> >
