I think data-sketches related integrations are fine as of now, as they get tested in Druid's integration test builds which are running with JDK11 as well as JDK8.
On Tue, Nov 10, 2020 at 8:58 AM leerho <lee...@gmail.com> wrote: > They are not using Memory to create their off-heap spaces, they are using > ByteBuffer to create them. The issue with JDK11 is it has removed access > to some classes like Cleaner that Memory used when creating off-heap > memory. Even if they create their off-heap spaces with ByteBuffer, we can > still use Memory to access them. > > On Tue, Nov 10, 2020 at 7:24 AM Will Lauer <wla...@verizonmedia.com > .invalid> > wrote: > > > If Druid isn't creating off-heap memory, what are BufferAggregators for? > > And what about the aggregation buffers that are preallocated? I thought > > that was all off-heap memory. > > > > Will > > > > <http://www.verizonmedia.com> > > > > Will Lauer > > > > Senior Principal Architect, Audience & Advertising Reporting > > Data Platforms & Systems Engineering > > > > M 508 561 6427 > > 1908 S. First St > > Champaign, IL 61822 > > > > <http://www.facebook.com/verizonmedia> < > http://twitter.com/verizonmedia> > > <https://www.linkedin.com/company/verizon-media/> > > <http://www.instagram.com/verizonmedia> > > > > > > > > On Mon, Nov 9, 2020 at 8:27 PM leerho <lee...@gmail.com> wrote: > > > > > Well, partly. As long as Druid doesn't actually create off-heap > Memory, > > it > > > works OK with JDK11. Currently compiling with JDK11 will show errors, > > but > > > those errors can be ignored as long as they don't create off-heap > Memory, > > > which they do not do. > > > > > > This is an issue we will begin addressing soon, we just haven't had the > > > resources to devote to it. > > > > > > Lee. > > > > > > On Mon, Nov 9, 2020 at 5:53 PM Will Lauer <wla...@verizonmedia.com > > > .invalid> > > > wrote: > > > > > > > My understanding was that the Apache Datasketch project was still > > having > > > > issues with Java 11 and direct memory support in their sketch > > > > implementation. That would impact Druid's use of datasketches, and > > > possibly > > > > Druid's use of direct memory as well. Does anyone know whether the > > > > Datasketch issues have been resolved yet? > > > > > > > > Will > > > > > > > > <http://www.verizonmedia.com> > > > > > > > > Will Lauer > > > > > > > > Senior Principal Architect, Audience & Advertising Reporting > > > > Data Platforms & Systems Engineering > > > > > > > > M 508 561 6427 > > > > 1908 S. First St > > > > Champaign, IL 61822 > > > > > > > > < > > > > > > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.facebook.com_verizonmedia&d=DwIFaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=30cwQjI_tqpOsvBcWhnY-ZftONuafrczjKp_ncVpW0M&s=-tGMOH_OrEiWdOQ86LBnuU2qMSfXIS5hP-9VpQieV6s&e= > > > > < > > > > > > https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.com_verizonmedia&d=DwIFaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=30cwQjI_tqpOsvBcWhnY-ZftONuafrczjKp_ncVpW0M&s=_IGT6oamCmA-1OIb2AoR35JkHzKVLlU2VB11khMrM3A&e= > > > > > > > > < > > > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_verizon-2Dmedia_&d=DwIFaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=30cwQjI_tqpOsvBcWhnY-ZftONuafrczjKp_ncVpW0M&s=rT_p44rQm49vbvQbe1LVN7bNzawOkiGC2q19B0SI12k&e= > > > > > > > > < > > > > > > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.instagram.com_verizonmedia&d=DwIFaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=30cwQjI_tqpOsvBcWhnY-ZftONuafrczjKp_ncVpW0M&s=6UhWWfy4257r45sCNSepsm1KfzAA1vZI4U9O0A1tLUM&e= > > > > > > > > > > > > > > > > > > > > On Mon, Nov 9, 2020 at 5:39 PM Himanshu <g.himan...@gmail.com> > wrote: > > > > > > > > > There are many projects that have Druid integration and use jars > > > created > > > > by > > > > > Druid releases. Druid itself has hadoop based batch ingestion > feature > > > > where > > > > > it ships its own jars over to hadoop and the code is executed in > > hadoop > > > > > cluster. For these integrations to continue working, it is > essential > > > that > > > > > Druid jars continue to work with Java8 JVMs or that all other > related > > > > > projects also provide support for JDK-11 . I know this could be a > > > > > chicken-or-the-egg problem, so just giving another justification to > > why > > > > it > > > > > makes sense to continue supporting JDK8 . > > > > > > > > > > -- Himanshu > > > > > > > > > > On Mon, Nov 9, 2020 at 3:25 PM Julian Hyde <jh...@apache.org> > wrote: > > > > > > > > > > > I have no objections to dropping support for JDK 8, but why only > > > > support > > > > > > JDK 11? What would it take to support JDK 11 and all newer JDKs? > > > > > > > > > > > > There's an analogy with starting and stopping a train. Once you > > have > > > > > > stopped a train (settled on a particular JDK release for a number > > of > > > > > years) > > > > > > starting it is hard, because of static friction. So it's easier > to > > > just > > > > > > keep the train rolling. > > > > > > > > > > > > In Calcite we have a policy to support all JDK versions, and it > > means > > > > > that > > > > > > we have never become dug into one version. Supporting a new > version > > > > > becomes > > > > > > a small incremental effort. Sometimes we have had to build shims > to > > > > > protect > > > > > > us against changes, but it was never too onerous in terms of > > > > maintenance > > > > > or > > > > > > performance. Obviously Druid is a very different project to > > Calcite, > > > > but > > > > > > "keeping the train rolling" has worked well for us. > > > > > > > > > > > > Also, remember that Druid is a project, not a product. It's fine > if > > > > Druid > > > > > > compiles, and builds, and passes the basic test suite, on JDK 12 > > even > > > > > > though you declare that is not 'supported' in production. > > > > > > > > > > > > Julian > > > > > > > > > > > > > > > > > > On Mon, Nov 9, 2020 at 2:41 PM Suneet Saldanha < > > > > suneet.salda...@imply.io > > > > > > > > > > > > wrote: > > > > > > > > > > > > > Hi Druids! > > > > > > > > > > > > > > I've been thinking about what it would take for us to remove > > > support > > > > > for > > > > > > > JDK 8 in the 0.21 release, and officially add support for JDK > 11. > > > > > > > > > > > > > > I see that unit tests for jdk11 were added about 1+ year ago in > > Aug > > > > > 2019 > > > > > > - > > > > > > > > > > > > > > > > > > > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_druid_pull_8400&d=DwIBaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=oTTYxCRAYdLG9vX5nRES2_H05PLf_g6-6O07b4RijaI&s=P8_diPbQbYLZ-mpgzW0bj2b4np3XzqtxJNaAKPbJip0&e= > > > > > > > And integration tests were added ~ 8 months ago in Feb 2020 - > > > > > > > > > > > > > > > > > > > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_druid_pull_9249&d=DwIBaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=oTTYxCRAYdLG9vX5nRES2_H05PLf_g6-6O07b4RijaI&s=EQAynkpyeUQFocV4cb39geHJ87UXU8SZcDHXlhNL8uA&e= > > > > > > > > > > > > > > Since the jdk11 integration tests were added, I can't remember > > > seeing > > > > > any > > > > > > > tests fail for a jdk11 test, but pass for jdk 8 or vice versa. > > > Given > > > > > that > > > > > > > it's been about 1 year that these tests have been running, is > > there > > > > any > > > > > > > opposition to officially removing support for JDK 8 and adding > > > > support > > > > > > for > > > > > > > JDK 11? > > > > > > > > > > > > > > There are a few big advantages I can think of for this: > > > > > > > > > > > > > > 1. Java 8 is end of life, so officially supporting java 11 > > will > > > > > allow > > > > > > > users to run Druid with at least java 11 which is slotted > for > > > > > support > > > > > > till > > > > > > > September 2023 according to > > > > > > > > > > > > > > > > > > > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__www.oracle.com_java_technologies_java-2Dse-2Dsupport-2Droadmap.html&d=DwIBaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=oTTYxCRAYdLG9vX5nRES2_H05PLf_g6-6O07b4RijaI&s=WsB-nZyjMgEFbJE0pz2LYHAEOen0781bgDwEbqkCaro&e= > > > > > > > 2. We can reduce our Travis usage by almost half, and > lighten > > > the > > > > > load > > > > > > > on ASF's Travis account - today we are the second highest > > > consumer > > > > > of > > > > > > > travis resources by project > > > > > > > > > > > > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__infra-2Dreports.apache.org_cistats_&d=DwIBaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=oTTYxCRAYdLG9vX5nRES2_H05PLf_g6-6O07b4RijaI&s=RSK7N5Mti_x-eG7WQxNIM5zMgfulpJey2eh8dLkQJJs&e= > > > > > > > [image: Screen Shot 2020-11-09 at 2.34.45 PM.png] > > > > > > > > > > > > > > The only limiting factor I can think of are full scale > > performance > > > > > tests > > > > > > > that prove there is no major regression in moving from jdk 8 to > > jdk > > > > 11. > > > > > > Are > > > > > > > there any other issues that should be considered before adding > > > > support > > > > > > for > > > > > > > java 11? > > > > > > > > > > > > > > > > > > > > > > > > > > > >
