If Druid isn't creating off-heap memory, what are BufferAggregators for? And what about the aggregation buffers that are preallocated? I thought that was all off-heap memory.
Will <http://www.verizonmedia.com> Will Lauer Senior Principal Architect, Audience & Advertising Reporting Data Platforms & Systems Engineering M 508 561 6427 1908 S. First St Champaign, IL 61822 <http://www.facebook.com/verizonmedia> <http://twitter.com/verizonmedia> <https://www.linkedin.com/company/verizon-media/> <http://www.instagram.com/verizonmedia> On Mon, Nov 9, 2020 at 8:27 PM leerho <lee...@gmail.com> wrote: > Well, partly. As long as Druid doesn't actually create off-heap Memory, it > works OK with JDK11. Currently compiling with JDK11 will show errors, but > those errors can be ignored as long as they don't create off-heap Memory, > which they do not do. > > This is an issue we will begin addressing soon, we just haven't had the > resources to devote to it. > > Lee. > > On Mon, Nov 9, 2020 at 5:53 PM Will Lauer <wla...@verizonmedia.com > .invalid> > wrote: > > > My understanding was that the Apache Datasketch project was still having > > issues with Java 11 and direct memory support in their sketch > > implementation. That would impact Druid's use of datasketches, and > possibly > > Druid's use of direct memory as well. Does anyone know whether the > > Datasketch issues have been resolved yet? > > > > Will > > > > <http://www.verizonmedia.com> > > > > Will Lauer > > > > Senior Principal Architect, Audience & Advertising Reporting > > Data Platforms & Systems Engineering > > > > M 508 561 6427 > > 1908 S. First St > > Champaign, IL 61822 > > > > < > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.facebook.com_verizonmedia&d=DwIFaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=30cwQjI_tqpOsvBcWhnY-ZftONuafrczjKp_ncVpW0M&s=-tGMOH_OrEiWdOQ86LBnuU2qMSfXIS5hP-9VpQieV6s&e= > > < > https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.com_verizonmedia&d=DwIFaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=30cwQjI_tqpOsvBcWhnY-ZftONuafrczjKp_ncVpW0M&s=_IGT6oamCmA-1OIb2AoR35JkHzKVLlU2VB11khMrM3A&e= > > > > < > https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_verizon-2Dmedia_&d=DwIFaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=30cwQjI_tqpOsvBcWhnY-ZftONuafrczjKp_ncVpW0M&s=rT_p44rQm49vbvQbe1LVN7bNzawOkiGC2q19B0SI12k&e= > > > > < > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.instagram.com_verizonmedia&d=DwIFaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=30cwQjI_tqpOsvBcWhnY-ZftONuafrczjKp_ncVpW0M&s=6UhWWfy4257r45sCNSepsm1KfzAA1vZI4U9O0A1tLUM&e= > > > > > > > > > > On Mon, Nov 9, 2020 at 5:39 PM Himanshu <g.himan...@gmail.com> wrote: > > > > > There are many projects that have Druid integration and use jars > created > > by > > > Druid releases. Druid itself has hadoop based batch ingestion feature > > where > > > it ships its own jars over to hadoop and the code is executed in hadoop > > > cluster. For these integrations to continue working, it is essential > that > > > Druid jars continue to work with Java8 JVMs or that all other related > > > projects also provide support for JDK-11 . I know this could be a > > > chicken-or-the-egg problem, so just giving another justification to why > > it > > > makes sense to continue supporting JDK8 . > > > > > > -- Himanshu > > > > > > On Mon, Nov 9, 2020 at 3:25 PM Julian Hyde <jh...@apache.org> wrote: > > > > > > > I have no objections to dropping support for JDK 8, but why only > > support > > > > JDK 11? What would it take to support JDK 11 and all newer JDKs? > > > > > > > > There's an analogy with starting and stopping a train. Once you have > > > > stopped a train (settled on a particular JDK release for a number of > > > years) > > > > starting it is hard, because of static friction. So it's easier to > just > > > > keep the train rolling. > > > > > > > > In Calcite we have a policy to support all JDK versions, and it means > > > that > > > > we have never become dug into one version. Supporting a new version > > > becomes > > > > a small incremental effort. Sometimes we have had to build shims to > > > protect > > > > us against changes, but it was never too onerous in terms of > > maintenance > > > or > > > > performance. Obviously Druid is a very different project to Calcite, > > but > > > > "keeping the train rolling" has worked well for us. > > > > > > > > Also, remember that Druid is a project, not a product. It's fine if > > Druid > > > > compiles, and builds, and passes the basic test suite, on JDK 12 even > > > > though you declare that is not 'supported' in production. > > > > > > > > Julian > > > > > > > > > > > > On Mon, Nov 9, 2020 at 2:41 PM Suneet Saldanha < > > suneet.salda...@imply.io > > > > > > > > wrote: > > > > > > > > > Hi Druids! > > > > > > > > > > I've been thinking about what it would take for us to remove > support > > > for > > > > > JDK 8 in the 0.21 release, and officially add support for JDK 11. > > > > > > > > > > I see that unit tests for jdk11 were added about 1+ year ago in Aug > > > 2019 > > > > - > > > > > > > > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_druid_pull_8400&d=DwIBaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=oTTYxCRAYdLG9vX5nRES2_H05PLf_g6-6O07b4RijaI&s=P8_diPbQbYLZ-mpgzW0bj2b4np3XzqtxJNaAKPbJip0&e= > > > > > And integration tests were added ~ 8 months ago in Feb 2020 - > > > > > > > > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_druid_pull_9249&d=DwIBaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=oTTYxCRAYdLG9vX5nRES2_H05PLf_g6-6O07b4RijaI&s=EQAynkpyeUQFocV4cb39geHJ87UXU8SZcDHXlhNL8uA&e= > > > > > > > > > > Since the jdk11 integration tests were added, I can't remember > seeing > > > any > > > > > tests fail for a jdk11 test, but pass for jdk 8 or vice versa. > Given > > > that > > > > > it's been about 1 year that these tests have been running, is there > > any > > > > > opposition to officially removing support for JDK 8 and adding > > support > > > > for > > > > > JDK 11? > > > > > > > > > > There are a few big advantages I can think of for this: > > > > > > > > > > 1. Java 8 is end of life, so officially supporting java 11 will > > > allow > > > > > users to run Druid with at least java 11 which is slotted for > > > support > > > > till > > > > > September 2023 according to > > > > > > > > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__www.oracle.com_java_technologies_java-2Dse-2Dsupport-2Droadmap.html&d=DwIBaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=oTTYxCRAYdLG9vX5nRES2_H05PLf_g6-6O07b4RijaI&s=WsB-nZyjMgEFbJE0pz2LYHAEOen0781bgDwEbqkCaro&e= > > > > > 2. We can reduce our Travis usage by almost half, and lighten > the > > > load > > > > > on ASF's Travis account - today we are the second highest > consumer > > > of > > > > > travis resources by project > > > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__infra-2Dreports.apache.org_cistats_&d=DwIBaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=oTTYxCRAYdLG9vX5nRES2_H05PLf_g6-6O07b4RijaI&s=RSK7N5Mti_x-eG7WQxNIM5zMgfulpJey2eh8dLkQJJs&e= > > > > > [image: Screen Shot 2020-11-09 at 2.34.45 PM.png] > > > > > > > > > > The only limiting factor I can think of are full scale performance > > > tests > > > > > that prove there is no major regression in moving from jdk 8 to jdk > > 11. > > > > Are > > > > > there any other issues that should be considered before adding > > support > > > > for > > > > > java 11? > > > > > > > > > > > > > > >
