Hi Folks,
ARM is becoming famous lately in its processing capability and has got the
potential to run Bigdata workloads.
Many users have been moving to ARM machines due to its low cost.
In the past there were attempts to compile Hadoop on ARM (Rasberry PI) for
experimental purposes. Today ARM architecture is taking some of the
serverside processing as well. So there will be/is a real need of Hadoop to
support ARM architecture as well.
There are bunch of users who are trying out building Hadoop on ARM, trying
to add ARM CI to hadoop and facing issues[1]. Also some
As of today, Hadoop does not compile on ARM due to below issues, found from
testing done in openlab in [2].
1. Protobuf :
-------------------
Hadoop project (also some downstream projects) stuck to protobuf 2.5.0
version, due to backward compatibility reasons. Protobuf-2.5.0 is not being
maintained in the community. While protobuf 3.x is being actively adopted
widely, still protobuf 3.x provides wire compatibility for proto2 messages.
Due to some compilation issues in the generated java code, which can induce
problems in downstream. Due to this reason protobuf upgrade from 2.5.0 was
not taken up.
In 3.0.0 onwards, hadoop supports shading of libraries to avoid classpath
problem in downstream projects.
There are patches available to fix compilation in Hadoop. But need to
find a way to upgrade protobuf to latest version and still maintain the
downstream's classpath using shading feature of Hadoop build.
There is a Jira for protobuf upgrade[3] created even before shade
support was added to Hadoop. Now need to revisit the Jira and continue
explore possibilities.
2. leveldbjni:
---------------
Current leveldbjni used in YARN doesnot support ARM architecture, need
to check whether any of the future versions support ARM and can hadoop
upgrade to that version.
3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1'
-------------------------
'protoc-gen-grpc-java:1.15.1' does not provide ARM executable by default in
the maven repository. Workaround is to build it locally and keep in local
maven repository.
Need to check whether any future versions of 'protoc-gen-grpc-java' is
having ARM executable and whether hadoop-yarn-csi can upgrade it?
Once the compilation issues are solved, then there might be many native
code related issues due to different architectures.
So to explore everything, need to join hands together and proceed.
Let us discuss and check, whether any body else out there who also need the
support of Hadoop on ARM architectures and ready to lend their hands and
time in this work.
[1] https://issues.apache.org/jira/browse/HADOOP-16358
[2]
https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
[3] https://issues.apache.org/jira/browse/HADOOP-13363
-Vinay
