[
https://issues.apache.org/jira/browse/KUDU-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18101389#comment-18101389
]
ASF subversion and git services commented on KUDU-3788:
-------------------------------------------------------
Commit 30501ee9e33a2799d3eca6816405b4132d3e0d14 in kudu's branch
refs/heads/master from Alexey Serbin
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=30501ee9e ]
KUDU-3788 an option to use pre-built thirdparty artifacts
With this changelist, now it's possible to fetch and use pre-built
artifacts for 3rd-party components instead of building them from
scratch when running $KUDU_HOME/thirdparty/build-if-necessary.sh
or $KUDU_HOME/thirdparty/build-thirdparty.sh <component_name>.
The following attributes are used to find a match for a particular
3rd-party component when installing on a host machine:
* name of the component (cmake, protobuf, etc.)
* version of the component as in $KUDU_HOME/thirdparty/vars.sh
* Kudu patch version of the component as in $KUDU_HOME/thirdparty/vars.sh
* OS name/flavor (redhat, ubuntu, macos, etc.)
* OS major version (9, 24, 26, etc.) -- this assumes the newer minor
versions of the same major release are backwards compatible
* CPU architecture (x86_64, aarch64, etc.)
* build toolchain info:
** compiler family (gcc, clang, etc.)
** C++ compiler version in form <major>.<minor>' (13.3, 17.0, etc.)
The usage of the pre-built 3rd-party artifacts is enabled by default:
use the USE_PREBUILT_THIRDPARTY env/shell variable to control this
feature. If a pre-built archive isn't available at the designated
S3 bucket, it's built from source and put into the local cache
directory: $KUDU_HOME/thirdparty/prebuilt-cache. After that, it's
possible to upload the result into a sub-directory of the designated
S3 bucket s3://cloudera-thirdparty-libs/prebuilt. I didn't implement
automatic uploading into the bucket because:
* it makes sense to limit upload and refresh of the pre-built
archives, making sure we distribute only vetted and verified ones
* as for CI, our current upstream CI pipeline isn't supposed to keep
valid S3 credentials in the build environment
Set REBUILD_PREBUILT_THIRDPARTY=1 when it's necessary to force
rebuilding pre-built artifacts even if corresponding archive is
available either in $KUDU_HOME/thirdparty/prebuilt-cache or in the
S3 bucket.
To upload all the pre-built archives from the cache when necessary
credentials are present in the environment, run the following command
from the $KUDU_HOME/thirdparty directory:
aws s3 cp prebuilt-cache s3://cloudera-thirdparty-libs/prebuilt --recursive
The prefix is set to /opt/kudu/thirdparty/{common,uninstrumented,tsan}
when building pre-built artifacts for a 3rd-party component. This is
uniform and independent from the layout of the local Kudu git workspace:
compare with $KUDU_HOME/thirdparty/installed/{...} in the legacy
approach. This allows for using pre-built 3rd-party artifacts at any
machine regardless of the actual layout of local Kudu git workspace.
Upon installation, symbolic links are created in /opt/kudu/thirdparty:
they point to the actual location of pre-built artifacts under one of
the sub-directories of $KUDU_HOME/thirdparty/installed. When building
3rd-party components using super-user account (UID 0), the prefix
directory /opt/kudu/thirdparty directory is automatically created.
Otherwise, the script prompts a regular user to create the directory
and grant necessary permissions on the directory before commencing
the build.
Since ninja doesn't support DESTDIR notation for the 'install' target,
it's no longer used for building 3rd-party components, even if it's
available. Only CMake/GNUmake combination is now used for building
3rd-party components that rely on CMake to bootstrap build environment.
Below a list of TODOs to address in follow-up changelists:
* address the issue with trace-viewer: it should be packed into
pre-built archive as well, and links should be established
from $KUDU_HOME/www into somewhere under thirdparty/installed
instead of copying files directly into $KUDU_HOME
* avoid unconditionally downloading and unpacking source tarballs:
as of this version, the build script is still downloading and
unpacking the source archives for all the 3rd-party components
when it doesn't find corresponding sub-directory in
$KUDU_HOME/thirdparty/src
* add CPU features that are auto-detected and enabled for building
particular 3rd-party components, e.g., AVX2 instructions support
when building bitshuffle
* introduce matching of any CPU architecture and any OS: this
should help with de-duplication of packed JARs and similar content
(e.g., interpreted scripts, etc.).
CLEAN_THIRDPARTY
Change-Id: I55b5b99fcbe60eec3f6ae19fa4f2e335f361d143
Reviewed-on: http://gerrit.cloudera.org:8080/24583
Tested-by: Alexey Serbin <[email protected]>
Reviewed-by: Marton Greber <[email protected]>
Reviewed-by: Ashwani Raina <[email protected]>
> Allow fethcing and using pre-built 3rd-party components when building Kudu
> --------------------------------------------------------------------------
>
> Key: KUDU-3788
> URL: https://issues.apache.org/jira/browse/KUDU-3788
> Project: Kudu
> Issue Type: Improvement
> Reporter: Alexey Serbin
> Assignee: Alexey Serbin
> Priority: Major
>
> It would be nice to allow to fetch and use pre-built 3rd-party components
> when building Kudu. Each of the 3rd-party components should be checked
> against a presence of pre-built version that matches the machine architecture
> and OS, and the matching rule should compare at least the following
> attributes:
> * Component name (e.g., gflags)
> * Component version (minor, major, patch, or just source hash if no semantic
> versioning is available)
> * The machine's CPU architecture that binaries/libraries are built for (e.g.,
> x86_64, arm64)
> * OS name or flavor/distro name if applicable (e.g., macOS, Ubuntu, Debian,
> RHEL, etc.)
> * OS version: major, minor (e.g., 9.1 for RHEL, 24.04 for Ubuntu, 15.7 for
> macOS, etc.)
> * The toolchain that the bits are built with: compiler name, major and minor
> versions, and the standard C++ library flavor where applicable (e.g.,
> gcc-10.5-libstdc++, clang-15.0)
> * Sanitizer option: TSAN if built with tread sanitizer support, or none if a
> regular build
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
