I had a go at this, and it seems to be much easier than I thought. I'm going to open a ticket and PR for this soon.
Istvan On Fri, Aug 26, 2022 at 5:15 PM Andrew Purtell <andrew.purt...@gmail.com> wrote: > I agree with Geoffrey that OMID should align with Phoenix on default > dependency versions but based on this you don’t have an immediate > integration problem. I don’t believe the Configuration API has changed in a > long time. > > The security posture of Hadoop 2 in general is a problem, though. I know > Hadoop released 2.10.2 to address some CVE worthy problems but it is > unclear if 2.10.2 addresses all known issues, unlike 3.3.4. Also as you > know Hadoop 2 has unpatchable dependencies on org.codehaus versions of > Jackson and Jetty, which themselves have high scoring CVEs that will never > be fixed because they are EOL, and other similar issues. Hadoop 3 doesn’t > completely solve such problems but is the only realistic place we can hope > they can be addressed as required. For organizations that implement or > require a top to bottom security audit of their software bill of materials, > it seems possible to avoid some user pain here by aligning build defaults. > > I also see Geoffrey started a discussion on dev@hbase about HBase > producing Hadoop 3 variant builds that could be consumed by downstreamers. > I have responded on that thread today to express my intention to sponsor > that effort, for what it’s worth. > > > > On Aug 25, 2022, at 10:41 PM, Istvan Toth <st...@cloudera.com.invalid> > wrote: > > > > The only non-hbase Hadoop APIs referred in Omid are > > org.apache.hadoop.conf.Configuration, and a few classes from of > > org.apache.hadoop.security > > (the latter is not referenced directly from the coprocessors) . > > > > Also, the coprocessor JARs don't have any Hbase or Hadoop code shaded in. > > > > > > > >> On Fri, Aug 26, 2022 at 5:45 AM Andrew Purtell < > andrew.purt...@gmail.com> > >> wrote: > >> > >> Is any OMID coprocessor code using Hadoop APIs? > >> > >>> On Aug 25, 2022, at 9:41 AM, Istvan Toth <st...@cloudera.com.invalid> > >> wrote: > >>> > >>> We dependencyManage every Hadoop component in Phoenix, so going to > >> Hadoop3 > >>> in Omid is not strictly necessary. > >>> Hadoop is also added to the TSO libraries in the assembly, but AFAIK > >>> Hadoop2 and Hadoop3 are wire compatible > >>> (and I'm not sure if Omid even calls Hadoop endpoints directly), so > that > >> is > >>> not a show stopper either. > >>> > >>> If we go with the assumption that there is no significant usage of Omid > >>> outside HBase, then updating to Hadoop3 is a no-brainer, if only to > avoid > >>> juggling > >>> the Hadoop2 and Hadoop3 compiled HBases when developing. > >>> Unfortunately it's quite a bit of work with all the dependency > >> management, > >>> build docs, and updating the CI pipelines to rebuild HBase > >>> like we do in Phoenix. > >>> > >>> Istvan > >>> > >>> > >>>> On Wed, Aug 24, 2022 at 9:24 PM Geoffrey Jacoby <gjac...@apache.org> > >> wrote: > >>>> > >>>> Thanks for volunteering to be RM for the next Omid release. > >>>> > >>>> I agree we should have a release soon (I think we'll want Phoenix 5.2 > to > >>>> use the new Omid). In addition to OMID-226, I suggest we also upgrade > >>>> Omid's internal Hadoop version to align with whatever default Hadoop > we > >>>> choose for Phoenix 5.2, to keep Phoenix's transitive dependencies (a > >>>> little) simpler. Right now Omid appears to depend on Hadoop 2.10, > which > >>>> since Phoenix 5 is Hadoop 3-based, seems incorrect. > >>>> > >>>> Geoffrey > >>>> > >>>>> On Wed, Aug 24, 2022 at 12:46 AM Istvan Toth <st...@apache.org> > wrote: > >>>>> > >>>>> Hi! > >>>>> > >>>>> Most of the planned OMID changes for Phoenix 5.2 have landed. > >>>>> The only outstanding ticket that I'm aware of is OMID-226 which I > also > >>>>> expect to land soon. > >>>>> > >>>>> Unless someone has more changes targeted for the next release, I > >> propose > >>>>> that we release the next Omid version soon after OMID-226. > >>>>> > >>>>> I also propose bumping the version to 1.1.0, though because of the > >> HBase > >>>>> 1.x compatibility removal, and maven artifact changes we could also > >> argue > >>>>> for 2.0.0 > >>>>> > >>>>> If there are no other volunteers, I also volunteer to be the RM for > the > >>>>> release. > >>>>> > >>>>> regards > >>>>> Istvan > >>>>> > >>>> > >>> > >>> > >>> -- > >>> *István Tóth* | Staff Software Engineer > >>> st...@cloudera.com <https://www.cloudera.com> > >>> [image: Cloudera] <https://www.cloudera.com/> > >>> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > >>> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: > >> Cloudera > >>> on LinkedIn] <https://www.linkedin.com/company/cloudera> > >>> <https://www.cloudera.com/> > >>> ------------------------------ > >> > > > > > > -- > > *István Tóth* | Staff Software Engineer > > st...@cloudera.com <https://www.cloudera.com> > > [image: Cloudera] <https://www.cloudera.com/> > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: > Cloudera > > on LinkedIn] <https://www.linkedin.com/company/cloudera> > > <https://www.cloudera.com/> > > ------------------------------ > -- *István Tóth* | Staff Software Engineer st...@cloudera.com <https://www.cloudera.com> [image: Cloudera] <https://www.cloudera.com/> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image: Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera on LinkedIn] <https://www.linkedin.com/company/cloudera> <https://www.cloudera.com/> ------------------------------
