Hi Zhenyu, Until HBase dependency resolved, without effecting the existing code on X86 i suggest create a separate profile with "os72" repo.
Down the line we should have common version for both X86 and ARM. Hope It Helps, Chinna On Wed, Mar 11, 2020 at 8:39 AM Zhenyu Zheng <zhengzhenyul...@gmail.com> wrote: > Hi Chinna, David and others might interested, > > Thanks for bring this up, we are currently working on improving enabling > big-data software on the ARM platform, > we have already done fixes and providing CIs to some of the well-know > projects like: > 1. Hadoop: > > https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-qbt-linux-ARM-trunk/ > 2. Spark: https://amplab.cs.berkeley.edu/jenkins/label/spark-arm/ > 3. HBase: > https://builds.apache.org/view/H-L/view/HBase/job/HBase-Nightly-ARM/ > > And we are now working on projects including Hive, Kudu, etc. > > Regarding to the protobuf upgrades in Hive, except upgrading to 3.x and > break dependency for HBase, there can > be some possible short-term plan(or walk-arounds), doing thes can make Hive > work on ARM without break any > dependencies, and then we can interact with Hbase project to see how can we > both upgrade to 3.x(since this > make take some time). > > Those possible solutions can be: > 1. Using pre-patched protobuf 2.5.0 with ARM support > from org.openlabtesting repo, some projects(HBase did > this: https://github.com/apache/hbase/pull/959, and we will add a profile > for this, it will detact the host arch and > will only use this pre-patched protobuf when the host arch is ARM so that > nothing will be affected for existing x86 > users; Spark and Hadoop used some other package from this repo as they have > already upgraded protobuf). > > 2. Using pre-patched protobuf 2.6.1-build3 from os72's repo: > https://mvnrepository.com/artifact/com.github.os72/protoc/2.6.1-build3 > Hive already used ``protoc-jar-maven-plugin`` from this repo: > > https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/pom.xml#L484 > > > https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/pom.xml#L472 > so it is a minor update. > > I've tested both ways, worked OK on the ARM machine. > > So my ideas will be that we make this a two-step-work, first step we use > one of the short-term plan to enable > Hive on ARM, and 2nd Step, we look for the long run to discuss with HBase > to find out a fesiable solution. > > BR, > > Zhenyu Zheng > > On Tue, Mar 10, 2020 at 9:29 PM Chinna Rao Lalam < > lalamchinnara...@gmail.com> > wrote: > > > Thanks David for the quick reply. > > Yes currently HBase protobuf version mismatch is the issue. > > > > We are investigating any common version of protobuf is there to support > for > > HBase an HIVE. > > > > > > > > > > On Tue, Mar 10, 2020 at 6:08 PM David Lavati > <dlav...@cloudera.com.invalid > > > > > wrote: > > > > > Hi, > > > > > > I recently investigated this in > > > https://issues.apache.org/jira/browse/HIVE-20359 and it didn't look > > > feasible, as protobuf has to be in sync across the whole stack, but the > > > external protobuf version for hbase is on 2.5.0 for hbase-2.x, see my > > > comment on the jira and > > > https://github.com/apache/hbase/blob/branch-2.2/pom.xml#L1414. > > > > > > Kind Regards, > > > David > > > > > > On Tue, Mar 10, 2020 at 1:11 PM Chinna Rao Lalam < > > > lalamchinnara...@gmail.com> > > > wrote: > > > > > > > Hi All, > > > > > > > > We are working on HIVE building and running on ARM architecture. As > > part > > > of > > > > this i want to add one periodic HIVE JENKINS JOB on ARM machines. > Which > > > > helps to identity gaps for ARM, If any issues identified we will keep > > on > > > > working to fix those issues. > > > > > > > > Currently to run HIVE on aarch64, we need to upgrade to > protobuf-3.7.1. > > > > HIVE-21939 <https://issues.apache.org/jira/browse/HIVE-21939> > upgraded > > > > protobuf-2.5.0 to protobuf-3.7.1, It have mainly 2 failures. We are > > > working > > > > on this. > > > > > > > > Thanks, > > > > Chinna Rao Lalam > > > > > > > > > > > > > -- > > Hope It Helps, > > Chinna > > > -- Hope It Helps, Chinna
