*Blockers* DRILL-5659 - Parth (done) DRILL-5669 - Boaz Ben-Zvi (fix will be ready in a couple of days)
*Possible blockers* NPE on CSV - Paul can you please provide an update? Kind regards Arina On Thu, Jul 13, 2017 at 2:46 PM, Arina Yelchiyeva < [email protected]> wrote: > *Blockers* > DRILL-5659 - problem is identified, Parth is working on the fix. > > *Not blockers* > DRILL-5660 - Vitalii > DRILL-5468 - Jinfeng > > *Possible blockers* > DRILL-5669 - Boaz Ben-Zvi / Paul > NPE on CSV - Paul > > Kind regards > Arina > > > On Thu, Jul 13, 2017 at 1:28 AM, Paul Rogers <[email protected]> wrote: > >> I withdraw my suggestion to fix DRILL-5660. Was a big hassle for us to >> track down the incompatibility the issue internally, but production users >> probably don’t attempt to use one build of Drill with metadata files >> created from another. We can worry about the issue later only if a user >> actually runs into it. >> >> As suggested earlier, let’s just document this obscure case in a release >> note. >> >> - Paul >> >> > On Jul 12, 2017, at 3:04 PM, Jinfeng Ni <[email protected]> wrote: >> > >> > Upgrade from 1.10 to 1.11 is irreversible only when parquet metadata >> cache >> > file is involved. >> > >> > I might be wrong, but I thought that if someone is running on 1.10.0, >> and >> > he wants to go back to 1.0.0, where the first version of parquet >> meatadata >> > cache was introduced, can we guarantee that Drillbit would work >> correctly? >> > I'm trying to understand what DRILL-5660 asks for is a new requirement, >> or >> > an requirement being there starting from Drill 1.0.0? >> > >> > Regression failures are normally regarded as release blocker. >> > >> > >> > >> > On Wed, Jul 12, 2017 at 2:54 PM, Paul Rogers <[email protected]> wrote: >> > >> >> If we don’t do DRILL-5660 for 1.11 then we don’t need to do it at all, >> so >> >> that is a time savings. We could handle this via a release note item >> that >> >> simply states that an upgrade from 1.10 to 1.11 is irreversible: we do >> not >> >> support the ability to fall back to the old version. Everyone OK with >> that? >> >> >> >> We are seeing some stress test regression failures that, I’m told, will >> >> show up as JIRA entries shortly. >> >> >> >> - Paul >> >> >> >>> On Jul 12, 2017, at 11:05 AM, Jinfeng Ni <[email protected]> wrote: >> >>> >> >>> I put some analysis in DRILL-5468. >> >>> >> >>> Also, IMHO, DRILL-5660 is not a release blocker. >> >>> >> >>> >> >>> >> >>> On Tue, Jul 11, 2017 at 11:27 AM, Parth Chandra <[email protected]> >> >> wrote: >> >>> >> >>>> Re DRILL-5634: >> >>>> From the Apache commons web page [1] - >> >>>> >> >>>> Please note that Apache Commons Crypto doesn't implement the >> >> cryptographic >> >>>> algorithm such as AES directly.* It wraps to Openssl or JCE which >> >> implement >> >>>> the algorithms.* >> >>>> >> >>>> >> >>>> Since we are cleared for using Openssl and JCE, we are OK to include >> the >> >>>> AES code in the UDFs. >> >>>> >> >>>> >> >>>> Parth >> >>>> >> >>>> [1] https://commons.apache.org/proper/commons-crypto/ >> >>>> >> >>>> On Tue, Jul 11, 2017 at 10:56 AM, Arina Yelchiyeva < >> >>>> [email protected]> wrote: >> >>>> >> >>>>> During today hangouts we agreed on approximate cut-off date - 14 >> July, >> >>>> 2017 >> >>>>> (date may be shifted due to the blockers). >> >>>>> I'll start preparing test RC on Thursday to solve any release >> >> preparation >> >>>>> issues beforehand. >> >>>>> >> >>>>> *Blockers* >> >>>>> DRILL-5660 - Vitalii >> >>>>> >> >>>>> *Possible blockers* >> >>>>> DRILL-5468 - Jinfeng >> >>>>> NPE on CSV source - Paul >> >>>>> DRILL-5659 - Parth >> >>>>> >> >>>>> *Will be included if done before the cut-off date* >> >>>>> DRILL-5616 / DRILL-5665 - Boaz >> >>>>> DRILL-5634 - Parth will take a look at Apache commons implementation >> >> and >> >>>>> update. >> >>>>> >> >>>>> Kind regards >> >>>>> Arina >> >>>>> >> >>>>> On Tue, Jul 11, 2017 at 3:14 PM, Charles Givre <[email protected]> >> >> wrote: >> >>>>> >> >>>>>> Hi Arina, >> >>>>>> I can’t make the hangout, but we have a few options. Basically, >> since >> >>>>> all >> >>>>>> the hashing algorithms have no export restrictions, we could: >> >>>>>> 1. Remove the AES encrypt/decrypt until I can figure out what we >> have >> >>>> to >> >>>>>> do (and do it) >> >>>>>> 2. I can try to research it today or tomorrow to figure out >> exactly >> >>>> what >> >>>>>> we have to do to include AES. >> >>>>>> 3. Remove the entire package >> >>>>>> >> >>>>>> I’d like to see the hashing functions at least included because >> we’ve >> >>>> had >> >>>>>> a lot of requests for that functionality. I also don’t want this >> to >> >>>>> hold >> >>>>>> things up, so please let me know your preference. Regardless, I’m >> >>>> going >> >>>>> to >> >>>>>> start trying to figure out the restrictions so we’ll know for the >> >>>> future. >> >>>>>> >> >>>>>> Best, >> >>>>>> — C >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>>> On Jul 11, 2017, at 05:58, Arina Yelchiyeva < >> >>>>> [email protected]> >> >>>>>> wrote: >> >>>>>>> >> >>>>>>> @Jinfeng, >> >>>>>>> Let's discuss today on Drill hangout final cut-off date. >> >>>>>>> >> >>>>>>> @Charles >> >>>>>>> Since there were questions about AES in DRILL-5634, should we >> exclude >> >>>>> it >> >>>>>>> from the candidates for 1.11.0 release? >> >>>>>>> >> >>>>>>> @Boaz >> >>>>>>> Do we have any ETA when DRILL-5616 and DRILL-5665 will pass code >> >>>>> review? >> >>>>>>> >> >>>>>>> Also I believe we should include DRILL-5660 where we need to bump >> up >> >>>>>>> parquet metadata version. >> >>>>>>> @Vitalii >> >>>>>>> Do you have any ETA for the fix? >> >>>>>>> >> >>>>>>> Kind regards >> >>>>>>> Arina >> >>>>>>> >> >>>>>>> On Tue, Jul 11, 2017 at 6:02 AM, Boaz Ben-Zvi <[email protected]> >> >>>>> wrote: >> >>>>>>> >> >>>>>>>> Hi Arina, >> >>>>>>>> >> >>>>>>>> Two PR have just been submitted, relating to the Hash Aggr >> >>>> Spill: >> >>>>>>>> DRILL-5616 and DRILL-5665 . >> >>>>>>>> Can these be considered for 1.11 as well ? >> >>>>>>>> >> >>>>>>>> Thanks, >> >>>>>>>> >> >>>>>>>> Boaz >> >>>>>>>> >> >>>>>>>> On 7/10/17, 1:49 PM, "Paul Rogers" <[email protected]> wrote: >> >>>>>>>> >> >>>>>>>> DRILL-5601 is no longer a blocker for 1.11; we’ll defer that >> >>>> until >> >>>>>>>> after the release to allow others more time to work on other >> >>>> features >> >>>>>>>> rather than doing code reviews. >> >>>>>>>> >> >>>>>>>> - Paul >> >>>>>>>> >> >>>>>>>>> On Jul 10, 2017, at 1:43 PM, Jinfeng Ni <[email protected]> wrote: >> >>>>>>>>> >> >>>>>>>>> Hi Arina, >> >>>>>>>>> >> >>>>>>>>> It has been a while since we discussed 1.11.0 release. Any >> update >> >>>>>>>> on the >> >>>>>>>>> new release plan? Do we have a cut-off date ? >> >>>>>>>>> >> >>>>>>>>> In the past, I believe Drill release is a time-bounded, not >> feature >> >>>>>>>>> bounded. If certain fixes/features could not make it within a >> >>>> certain >> >>>>>>>>> time-frame, we had better defer them into future release, unless >> >>>>>>>> those >> >>>>>>>>> fixes/features are regarded as blocking issues for a new >> release. >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> On Tue, Jul 4, 2017 at 5:14 AM, Arina Yelchiyeva < >> >>>>>>>> [email protected] >> >>>>>>>>>> wrote: >> >>>>>>>>> >> >>>>>>>>>> Latest update: >> >>>>>>>>>> >> >>>>>>>>>> We wait for the following Jiras before final cut off. >> >>>>>>>>>> 1. DRILL-5601 - Paul >> >>>>>>>>>> 2. DRILL-5420 - Kunal >> >>>>>>>>>> 3. DRILL-5634 - Charles >> >>>>>>>>>> >> >>>>>>>>>> Kind regards >> >>>>>>>>>> Arina >> >>>>>>>>>> >> >>>>>>>>>> On Fri, Jun 30, 2017 at 9:29 PM, Kunal Khatua < >> [email protected]> >> >>>>>>>> wrote: >> >>>>>>>>>> >> >>>>>>>>>>> Parth's already +1'd the process with a minor update. So, >> we're >> >>>>>>>> good to >> >>>>>>>>>> go >> >>>>>>>>>>> on DRILL-5420. I'll change the label on that to >> ready-to-commit. >> >>>>>>>>>>> >> >>>>>>>>>>> <http://www.mapr.com/> >> >>>>>>>>>>> >> >>>>>>>>>>> ________________________________ >> >>>>>>>>>>> From: Parth Chandra <[email protected]> >> >>>>>>>>>>> Sent: Thursday, June 29, 2017 9:44:39 AM >> >>>>>>>>>>> To: [email protected] >> >>>>>>>>>>> Subject: Re: [DISCUSS] Drill 1.11.0 release >> >>>>>>>>>>> >> >>>>>>>>>>> I'm Ok with the pcap format plugin if Pauls comments are also >> >>>>>>>> addressed. >> >>>>>>>>>> If >> >>>>>>>>>>> the changes are done before your cutoff deadline,we can merge >> it >> >>>>>>>> in. >> >>>>>>>>>>> >> >>>>>>>>>>> On Thu, Jun 29, 2017 at 12:26 AM, Arina Yelchiyeva < >> >>>>>>>>>>> [email protected]> wrote: >> >>>>>>>>>>> >> >>>>>>>>>>>> Hi all, >> >>>>>>>>>>>> >> >>>>>>>>>>>> here is the list of Jira we are waiting before making final >> cut >> >>>>>>>> off: >> >>>>>>>>>>>> 1. DRILL-5601 - Paul >> >>>>>>>>>>>> 2. DRILL-3640 - Kunal >> >>>>>>>>>>>> 3. DRILL-5420 - Kunal >> >>>>>>>>>>>> >> >>>>>>>>>>>> Please let me know if there are any others Jiras we should >> wait >> >>>>>>>> for. >> >>>>>>>>>>>> >> >>>>>>>>>>>> @Charles, >> >>>>>>>>>>>> you have mentioned that you want to include some UDFs in >> Drill >> >>>>>>>> 1.11, do >> >>>>>>>>>>> you >> >>>>>>>>>>>> have any ETA for the PR? >> >>>>>>>>>>>> >> >>>>>>>>>>>> @Paul / Parth >> >>>>>>>>>>>> I see there is active CR for pcap format PR, do we want to >> >>>>>>>> include it >> >>>>>>>>>> in >> >>>>>>>>>>>> this release as well? >> >>>>>>>>>>>> >> >>>>>>>>>>>> Kind regards >> >>>>>>>>>>>> Arina >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> On Thu, Jun 29, 2017 at 10:05 AM, Arina Yelchiyeva < >> >>>>>>>>>>>> [email protected]> wrote: >> >>>>>>>>>>>> >> >>>>>>>>>>>>> Parth, thanks a lot the instructions. >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> On Wed, Jun 28, 2017 at 8:24 PM, Parth Chandra < >> >>>>>>>> [email protected]> >> >>>>>>>>>>>> wrote: >> >>>>>>>>>>>>> >> >>>>>>>>>>>>>> FWIW, I put release instructions culled from gists written >> by >> >>>>>>>>>> previous >> >>>>>>>>>>>>>> release managers into an updated gist here [1] >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> HTH >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Parth >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> [1] https://github.com/parthchandra/drill/wiki/Drill- >> >>>>>>>>>> Release-Process >> >>>>>>>>>>>>>> <https://github.com/parthchandra/drill/wiki/Drill- >> >>>>>>>> Release-Process> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> On Tue, Jun 27, 2017 at 5:24 PM, Paul Rogers < >> >>>> [email protected]> >> >>>>>>>>>>> wrote: >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> Hi Arina, >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> I have two projects in progress: one is a candidate for >> 1.11, >> >>>>>>>> the >> >>>>>>>>>>>> other >> >>>>>>>>>>>>>> is >> >>>>>>>>>>>>>>> not. >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> The “managed external sort” is the good candidate for >> 1.11: >> >>>>>>>> The PR >> >>>>>>>>>>> for >> >>>>>>>>>>>>>>> DRILL-5601 is being reviewed. Getting that in will allow >> us >> >>>> to >> >>>>>>>>>>> enable >> >>>>>>>>>>>>>> the >> >>>>>>>>>>>>>>> “new” external sort by default. >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> The other project is to address Drill’s memory >> fragmentation >> >>>>>>>> issue >> >>>>>>>>>>>> with >> >>>>>>>>>>>>>>> vectors over 16 MB in size. Two PRs have been open for a >> >>>> while. >> >>>>>>>>>> They >> >>>>>>>>>>>> are >> >>>>>>>>>>>>>>> the first of several. Each takes a while to review. So, >> the >> >>>>>>>>>> project >> >>>>>>>>>>>> does >> >>>>>>>>>>>>>>> not fit 1.11. The work can go into the next release >> instead. >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> Thanks, >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> - Paul >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>> >> >>>>>> >> >>>>> >> >>>> >> >> >> >> >> >> >
