The memory repository is the one that needs the most conversion help. Everything using Unsafe in datasketches-java runs through the memory package.
I'm not sure off-hand if the characterization repo has low level tests of Memory (on my phone at the moment), but we'd be looking to minimize the speed impact in the process since it very directly impacts sketch speed. jon On Thu, Oct 17, 2019, 4:43 PM Christopher <[email protected]> wrote: > I've converted several other projects to using JDK 11, and can help > with this if there's still a need for help. I've actually been looking > at ways I can contribute to DataSketches since ApacheCon, and this > seems to be something I can probably do. > > On Wed, Oct 16, 2019 at 9:47 PM leerho <[email protected]> wrote: > > > > Gian, > > > > If I were to supply you with a link to a snapshot or a Release Candidate > > version of the datasketches jar (and Memory jar) could you test in your > > environment under JDK 11 and see if it works? > > > > Lee > > > > > > On Mon, Oct 14, 2019 at 9:19 PM leerho <[email protected]> wrote: > > > > > Holy Cow! I have never seen such a complex travis config! > > > > > > On Mon, Oct 14, 2019 at 6:54 PM Gian Merlino <[email protected]> wrote: > > > > > >> I should have linked Druid's Travis config file for reference. Here > it is: > > >> https://github.com/apache/incubator-druid/blob/master/.travis.yml > > >> > > >> On Mon, Oct 14, 2019 at 6:53 PM Gian Merlino <[email protected]> wrote: > > >> > > >> > > So what exactly is the requirement? Our DataSketches code does > > >> compile > > >> > > under JDK8 and we have lots of customers and users using JDK 8. > > >> > > > >> > It depends on who you ask, I guess :) > > >> > > > >> > From Druid's perspective, we aren't going to be compiling your code, > > >> only > > >> > running it. So the only requirement we would ask for is that the > code > > >> > _runs_ under a Java 11 runtime. Being able to compile under Java 11 > is > > >> good > > >> > for future-proofing your code, though. It is starting to become more > > >> > prevalent. > > >> > > > >> > > It sounds like (as far as Unsafe is concerned) converting all the > > >> direct > > >> > > unsafe.blah() calls to static MethodHandle.invoke calls is what is > > >> > required. > > >> > > > > >> > > Is this correct? > > >> > > > >> > That is certainly part of it, I'm not 100% sure if that's all that's > > >> > needed. Btw, CI can help verify. In Druid's Travis config, we run > our > > >> tests > > >> > on both JDK 8 and JDK 11. In order to make this automated testing > > >> easier, > > >> > we chose to do the work needed to compile Druid on JDK 11, even > though > > >> we > > >> > still compile release builds using JDK 8 only. (If we hadn't done > this > > >> > work, then we'd have to compile on 8 and test on 11, which Travis > > >> doesn't > > >> > easily support.) > > >> > > > >> > On Mon, Oct 14, 2019 at 5:51 PM leerho <[email protected]> wrote: > > >> > > > >> >> Thanks, Gian, this is VERY helpful! > > >> >> > > >> >> From my cursory glance, it appears that UnsafeUtil acquires Unsafe > and > > >> >> Unsafe-class the usual way using reflection (I didn't think this > would > > >> >> work > > >> >> with 9+ ), and then defines some method-handles that the other > classes > > >> >> use. > > >> >> > > >> >> Will this approach allow code to compile under JDK11 and JDK8? > > >> >> > > >> >> It also seems that compiling with JDK11 is still a work in > progress as > > >> the > > >> >> pom.xml clearly specifies JDK8. > > >> >> > > >> >> So what exactly is the requirement? Our DataSketches code does > compile > > >> >> under JDK8 and we have lots of customers and users using JDK 8. > > >> >> > > >> >> It sounds like (as far as Unsafe is concerned) converting all the > > >> direct > > >> >> unsafe.blah() calls to static MethodHandle.invoke calls is what is > > >> >> required. > > >> >> > > >> >> Is this correct? > > >> >> > > >> >> Lee. > > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > > >> >> On Mon, Oct 14, 2019 at 2:07 PM Gian Merlino <[email protected]> > wrote: > > >> >> > > >> >> > And nobody should be using the Oracle JDK without a commercial > > >> >> relationship > > >> >> > with Oracle, its license is a nightmare. > > >> >> > > > >> >> > On Mon, Oct 14, 2019 at 2:04 PM Gian Merlino <[email protected]> > > >> wrote: > > >> >> > > > >> >> > > Oracle and OpenJDK builds behave the same as each other now. > The > > >> >> > > differences are in terms of licensing, length of support, and > > >> release > > >> >> > > cadence. IMO most Java devs should be using one of the > third-party > > >> >> > OpenJDK > > >> >> > > distributions (e.g. Corretto, Zulu, AdoptOpenJDK) rather than > > >> vanilla > > >> >> > > OpenJDK these days, because of the new OpenJDK policy that > public > > >> >> updates > > >> >> > > will cease for major versions after 6 months. > > >> >> > > > > >> >> > > On Mon, Oct 14, 2019 at 11:07 AM leerho <[email protected]> > wrote: > > >> >> > > > > >> >> > >> Thanks, Gian, > > >> >> > >> > > >> >> > >> Your references are helpful and I am studying them. > > >> >> > >> > > >> >> > >> I didn't see any references to OpenJDK vs Oracle's JDKs. > > >> >> > >> > > >> >> > >> Are there any differences in the way Unsafe calls (or other > hidden > > >> >> > >> classes) > > >> >> > >> are handled by OpenJDK? Or is this a "don't care" as far as > > >> Druid is > > >> >> > >> concerned? Are the version numbers between Oracle and OpenJDK > > >> always > > >> >> > >> aligned? > > >> >> > >> > > >> >> > >> > > >> >> > >> Lee. > > >> >> > >> > > >> >> > >> > > >> >> > >> > > >> >> > >> On Fri, Oct 11, 2019 at 5:45 PM Gian Merlino <[email protected] > > > > >> >> wrote: > > >> >> > >> > > >> >> > >> > You didn't address this message to me, but in Druid most of > the > > >> >> work > > >> >> > for > > >> >> > >> > Java 9+ compatibility is mentioned in this master issue, > which > > >> you > > >> >> > might > > >> >> > >> > find helpful: > > >> >> https://github.com/apache/incubator-druid/issues/5589. > > >> >> > >> > Skimming the list, these might be particularly relevant to > > >> >> > DataSketches: > > >> >> > >> > > > >> >> > >> > - https://github.com/apache/incubator-druid/pull/7487 > (cleaner > > >> >> > >> operations) > > >> >> > >> > - https://github.com/apache/incubator-druid/pull/7466 > > >> (ByteBuffer > > >> >> > unmap > > >> >> > >> > operation) > > >> >> > >> > - https://github.com/apache/incubator-druid/pull/7576 > (Remove > > >> >> direct > > >> >> > >> > references to Unsafe) > > >> >> > >> > > > >> >> > >> > Btw, it would be better, I think, to address emails to the > list > > >> at > > >> >> > >> large. > > >> >> > >> > It encourages more people to participate. If Roman is a > > >> subscriber > > >> >> he > > >> >> > >> would > > >> >> > >> > get a copy anyway. If not, you could encourage him to > subscribe. > > >> >> > >> > > > >> >> > >> > On Fri, Oct 11, 2019 at 3:13 PM leerho <[email protected]> > > >> wrote: > > >> >> > >> > > > >> >> > >> > > Roman, > > >> >> > >> > > > > >> >> > >> > > I hope you are doing well. > > >> >> > >> > > > > >> >> > >> > > I really appreciate the contributions you made to our > Memory > > >> >> > >> component! > > >> >> > >> > > > > >> >> > >> > > Given that Druid has made its code base compatible with > > >> OpenJDK > > >> >> 11, > > >> >> > we > > >> >> > >> > > could use your help in what changes do we need to make to > make > > >> >> that > > >> >> > >> > happen > > >> >> > >> > > for DataSketches. As I recall, our DataSketches library > was > > >> not > > >> >> the > > >> >> > >> only > > >> >> > >> > > Druid dependency that took advantage of Unsafe. So I > assume > > >> you > > >> >> > have > > >> >> > >> > been > > >> >> > >> > > down this path :) > > >> >> > >> > > > > >> >> > >> > > Currently, I have a check in the static initializer in > > >> >> > >> Memory/UnsafeUtil: > > >> >> > >> > > parseJavaVersion(...) that checks the string returned from > > >> >> > >> > > System.getProperty("java.version"). If that string does > not > > >> >> contain > > >> >> > >> > "1.8" > > >> >> > >> > > or "8" it will throw an error. Given that we have not > tested > > >> >> with > > >> >> > >> JDK 9, > > >> >> > >> > > 10, 11 or 12, I felt it was safer to explicitly throw > rather > > >> than > > >> >> > >> having > > >> >> > >> > > the users experience some weird failure later that may be > > >> >> difficult > > >> >> > to > > >> >> > >> > > debug. > > >> >> > >> > > > > >> >> > >> > > With only one minor exception (which we could easily fix) > the > > >> >> Memory > > >> >> > >> > > component is the only place where Unsafe is used. > However, > > >> we do > > >> >> > use > > >> >> > >> > > reflection to gain internal access to a number of other > > >> classes > > >> >> > >> including > > >> >> > >> > > ByteBuffer, Cleaner, and Bits. > > >> >> > >> > > > > >> >> > >> > > Looking forward to hearing from you. > > >> >> > >> > > > > >> >> > >> > > Cheers, > > >> >> > >> > > > > >> >> > >> > > Lee. > > >> >> > >> > > > > >> >> > >> > > > >> >> > >> > > >> >> > > > > >> >> > > > >> >> > > >> > > > >> > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
