I'll throw my .02 here... As a user of Drill, I've only had the occasion to use the Union once. However, when I used it, it consumed so much memory, we ended up finding a workaround anyway and stopped using it. Honestly, since we improved the implicit casting rules, I think Drill is a lot smarter about how it reads data anyway. Bottom line, I do think we could drop the union and repeated union.
The repeated lists and maps however are unfortunately something that does come up a bit. Honestly, I'm not sure what work is remaining here but TBH Drill works pretty well at the moment with most of the data I'm using it for. This would include some really nasty nested JSON objects. -- C > On Dec 31, 2023, at 01:38, Paul Rogers <par0...@gmail.com> wrote: > > Hi Luoc, > > Thanks for reminding me about the EVF V2 work. I got mostly done adding > projection for complex types, then got busy on other projects. I've yet to > tackle the hard cases: unions, repeated unions and repeated lists (which > are, in fact, repeated repeated unions). > > The code to handle unprojected fields in these areas is getting awfully > complicated. In doing that work, and then seeing a trick that Druid uses, > I'm tempted to rework the projection bits of the code to use a cleaner > approach. However, it might be better to commit the work done thus far so > folks can use it before I wander off to take another approach. > > Then, I wondered if anyone actually still uses this stuff. Do you still > need the code to handle non-projection of complex types? > > Of course, perhaps no one will ever need the hard cases: I've never been > convinced that unions, repeated lists, or arrays of repeated lists are > things that any sane data engineer will want to use -- or use more than > once. > > Thanks, > > - Paul > > > On Sat, Dec 30, 2023 at 10:26 PM James Turton <dz...@apache.org> wrote: > >> Hi Luoc and Drill devs! >> >> It's best to email Paul directly since he doesn't follow these lists >> closely. In the meantime I've prepared a PR of backported fixes for >> 1.21.2 to the 1.21 branch [1]. I think we can try to get the Netty >> upgrade that Maksym is working on, and which looks close to done, >> included? There's at least one CVE applicable to our current version of >> Netty... >> >> Regards >> James >> >> >> 1. https://github.com/apache/drill/pull/2860 >> >> On 2023/12/11 04:41, luoc wrote: >>> Hello all, >>> 1.22 will be a more stable version. This is a digression: Is Paul >> still interested in participating in the EVF V2 refactoring in the >> framework? I would like to offer time to assist him. >>> >>> luoc >>> >>>> 2023年12月9日 01:01，Charles Givre <cgi...@gmail.com> 写道： >>>> >>>> Hello all, >>>> Happy Friday everyone! I wanted to raise the topic of getting a Drill >> minor release out the door before the end of the year. My opinion is that >> I'd really like to release Drill 1.22 once the integration with Apache >> Daffodil is complete, but it sounds like that is still a few weeks away. >>>> >>>> What does everyone think about issuing a maintenance release before the >> end of the year? There are a number of singificant fixes including some >> security updates and a major bug in the ES plugin that basically makes it >> unusable. >>>> Best, >>>> -- C >> >>
