Thanks for the quick turnaround Charles! I compiled the latest snapshot build this morning. I didn't see the allowDiskUse parameter listed in the mongo storage plugin configuration, so I added it manually under the "batchSize" parameter as following:
"batchSize": 100, "allowDiskUse": true, "enabled": true Updating the configuration is successful, but if I go back to update the mongo configuration again, I don't see the parameter listed. On Sat, Jan 29, 2022 at 6:46 PM Charles Givre <[email protected]> wrote: > Hi Daniel, > I just submitted DRILL-8118 (https://github.com/apache/drill/issues/2440 < > https://github.com/apache/drill/issues/2440>). Please take a look at let > us know if this solves your issue. > Best, > -- C > > > On Jan 28, 2022, at 12:35 PM, Daniel Clark <[email protected]> wrote: > > > > Sounds good. Thank you Charles. > > > > On Fri, Jan 28, 2022 at 12:30 PM Charles Givre <[email protected] > <mailto:[email protected]>> wrote: > > > >> Hey Daniel, > >> I looked at the mongo docs, and I'm going to add a config option to add > >> the "spill to disk" for mongo. I'll do it over the weekend and submit > a PR > >> to Drill. This is a very minor fix. > >> Best, > >> -- C > >> > >> > >> > >>> On Jan 28, 2022, at 12:28 PM, Daniel Clark <[email protected]> wrote: > >>> > >>> I've attached the json version of the profile. Is that what you are > >> requesting? The latest run ended in another error. > >>> > >>> On Fri, Jan 28, 2022 at 9:47 AM luoc <[email protected] <mailto: > [email protected]> <mailto: > >> [email protected] <mailto:[email protected]>>> wrote: > >>> > >>> Hi Daniel, > >>> The 1.20 is actually included a feature to improve performance using > >> the new push down framework. > >>> For describe the specific things, could you please post the profile > >> file of query? > >>> > >>>> On Jan 28, 2022, at 22:32, Daniel Clark <[email protected] <mailto: > [email protected]> <mailto: > >> [email protected]>> wrote: > >>>> > >>>> Hello, > >>>> > >>>> While evaluating 1.20.0-SNAPSHOT release performance, I ran a mongo > >> query that runs in 15 minutes in the 1.19 release (below). > >>>> > >>>> SELECT `Elements_Efforts`.`EffortTypeName` AS `EffortTypeName`, > >>>> `Elements`.`ElementSubTypeName` AS `ElementSubTypeName`, > >>>> `Elements`.`ElementTypeName` AS `ElementTypeName`, > >>>> `Elements`.`PlanID` AS `PlanID` > >>>> FROM `mongo.grounds`.`Elements` `Elements` > >>>> INNER JOIN `mongo.grounds`.`Elements_Efforts` `Elements_Efforts` ON > >> (`Elements`.`_id` = `Elements_Efforts`.`_id`) > >>>> WHERE (`Elements`.`PlanID` = '1623263140') > >>>> GROUP BY `Elements_Efforts`.`EffortTypeName`, > >>>> `Elements`.`ElementSubTypeName`, > >>>> `Elements`.`ElementTypeName`, > >>>> `Elements`.`PlanID` > >>>> > >>>> The query runs for 34 minutes before returning this error; "Sort > >> exceeded memory limit of 104857600 bytes, but did not opt in to external > >> sorting. Aborting operation. Pass allowDiskUse:true to opt in.' on > server > >> localhost:27017." Any ideas? I realize that it's a mongodb error, but > the > >> mongo database doesn't raise this error with the 1.19 release. I was > >> expecting improved performance with the mongo storage plugin in the > >> upcoming 1.20 release. Nothing in my environment has changed. I've > attached > >> the full stacktrace. > >>>> > >>>> <stacktrace.txt> > >>> <profile.json> > >
