PeterPtroc commented on PR #7903: URL: https://github.com/apache/hadoop/pull/7903#issuecomment-3235836592
@pan3793 Thanks for the suggestion! Below is a concise doc to verify the correctness of the crc32riscv implementation: I mainly verify on RISC‑V by using QEMU together with the openEuler RISC‑V image. Download the image - https://dl-cdn.openeuler.openatom.cn/openEuler-25.03/virtual_machine_img/riscv64/ For me, from the above link, download these four files: RISCV_VIRT_CODE.fd, RISCV_VIRT_VARS.fd, openEuler-25.03-riscv64.qcow2.xz, and start_vm.sh; then log in as root with the password: openEuler12#$. Install required packages ````bash yum install -y gcc gcc-c++ gcc-gfortran libgcc cmake yum install -y wget openssl openssl-devel zlib zlib-devel automake libtool make libstdc++-static glibc-static git snappy snappy-devel fuse fuse-devel doxygen clang cyrus-sasl cyrus-sasl-devel libtirpc libtirpc-devel yum install -y java-17-openjdk.riscv64 java-17-openjdk-devel.riscv64 java-17-openjdk-headless.riscv64 ```` Install Protobuf 2.5.0 (with RISC‑V patches) ````bash mkdir protobuf && cd protobuf # Fetch sources git clone https://gitee.com/src-openeuler/protobuf2.git cd protobuf2 tar -xjf protobuf-2.5.0.tar.bz2 cp *.patch protobuf-2.5.0 && cd protobuf-2.5.0 # Apply patches (adds riscv64 support and build fixes) patch -p1 < 0001-Add-generic-GCC-support-for-atomic-operations.patch patch -p1 < protobuf-2.5.0-gtest.patch patch -p1 < protobuf-2.5.0-java-fixes.patch patch -p1 < protobuf-2.5.0-makefile.patch patch -p1 < add-riscv64-support.patch # Autotools setup libtoolize yum install -y automake automake-1.17 -a chmod +x configure # Configure, build, install ./configure --build=riscv64-unknown-linux --prefix=/usr/local/protobuf-2.5.0 make make check make install ldconfig # Publish protoc 2.5.0 into local Maven repo (riscv64 classifier) mvn install:install-file \ -DgroupId=com.google.protobuf \ -DartifactId=protoc \ -Dversion=2.5.0 \ -Dclassifier=linux-riscv64 \ -Dpackaging=exe \ -Dfile=/usr/local/protobuf-2.5.0/bin/protoc cd .. ```` Install Protobuf 3.25.5 ````bash # Download and unpack wget -c https://github.com/protocolbuffers/protobuf/releases/download/v25.5/protobuf-25.5.tar.gz tar -xzf protobuf-25.5.tar.gz cd protobuf-25.5 # Abseil dependency git clone https://github.com/abseil/abseil-cpp third_party/abseil-cpp # Configure and build cmake ./ \ -DCMAKE_BUILD_TYPE=RELEASE \ -Dprotobuf_BUILD_TESTS=off \ -DCMAKE_CXX_STANDARD=20 \ -DCMAKE_INSTALL_PREFIX=/usr/local/protobuf-3.25.5 make install -j "$(nproc)" # Publish protoc 3.25.5 into local Maven repo (riscv64 classifier) mvn install:install-file \ -DgroupId=com.google.protobuf \ -DartifactId=protoc \ -Dversion=3.25.5 \ -Dclassifier=linux-riscv64 \ -Dpackaging=exe \ -Dfile=/usr/local/protobuf-3.25.5/bin/protoc # Make protoc available on PATH and verify sudo ln -sfn /usr/local/protobuf-3.25.5/bin/protoc /usr/local/bin/protoc protoc --version ```` Verify CRC32 using Hadoop native ````bash # Clone Hadoop git clone https://github.com/apache/hadoop.git cd hadoop # Increase Maven memory export MAVEN_OPTS="-Xmx8g -Xms6g" # Build Hadoop Common (native enabled) nohup mvn -pl hadoop-common-project/hadoop-common -am -Pnative -DskipTests clean install > build.log 2>&1 & # Point to built native library directory export HADOOP_COMMON_LIB_NATIVE_DIR="$PWD/hadoop-common-project/hadoop-common/target/native/target/usr/local/lib" export LD_LIBRARY_PATH="$HADOOP_COMMON_LIB_NATIVE_DIR:$LD_LIBRARY_PATH" # Run the CRC32 native test nohup mvn -Pnative -Dtest=org.apache.hadoop.util.TestNativeCrc32 \ -Djava.library.path="$HADOOP_COMMON_LIB_NATIVE_DIR" test > test.log 2>&1 & -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
