[
https://issues.apache.org/jira/browse/BIGTOP-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sanjay Radia updated BIGTOP-1422:
---------------------------------
Attachment: RU-Packaginglayout.pdf
At Hortonworks we have addressed some of the problems raised above using what
we call versioned-rpms and versioned-debs. Ie using the native packaging
mechanism of Linux, but where the version number is in the name of the rpm/deb
- this seems to be along the lines of what Roman seems to be suggesting.
Our goal is rolling upgrades where one needs to be able to have side-by-side
installs of multiple versions of the Hadoop stack. So our goal wasn't a
universal packaging format, though we did strongly consider it. The actual bits
that gets laid out are fully self contained and fully relocatable (as proposed
by this Jira)
Further we wanted rolling upgrade to work with both Apache Amabri (the Apache
Hadoop installer that deploys and manages a Hadoop cluster) and also with a
customer's custom installation scripts.
Turns out there are many subtle issues to make side-by-side installs and
rolling upgrades to actually work in practice. (i.e. even if you have
successfully done side by side installs, you may find that rolling upgrades
may not work correctly). Some of these are small details to get right, but
others are subtle and more complex. For example, one must make sure that cross
component dependencies in the stack work correctly (e.g. HBase must use the
matching version of HDFS's client libraries), Another issue concernes
Yarn/MR/Tez so that one does not mix a running job's context of its libraries
and configs with those of the compute node on which it executes: keeping a
running job's context consistent is important because the job may have been
submitted with version N of the Hadoop but the compute node on which it
executes may have been rolled-forward to version N+1 of Hadoop.
We would be willing to share what we have done; some of the lessons we learned
would apply equally well to universal packaging. I have attached a short doc
describing what we have done. The fixes we have made to make Hadoop work
correctly during a rolling upgrade are of course all being pushed to Apache.
> Introduce Universal Packaging System for Hadoop distribution
> ------------------------------------------------------------
>
> Key: BIGTOP-1422
> URL: https://issues.apache.org/jira/browse/BIGTOP-1422
> Project: Bigtop
> Issue Type: New Feature
> Components: build, deployment
> Affects Versions: 0.8.0
> Reporter: Konstantin Boudnik
> Priority: Blocker
> Fix For: 1.0.0
>
> Attachments: RU-Packaginglayout.pdf
>
>
> Looks like the time comes where Bigtop needs to step up and champion new
> revolutionary format for packaging of the Hadoop.
> New UPS provides completely relocatable archive-like media of distribution
> for the content and the services alike. Preparation of an UPS deliverables
> shouldn't require any special development tools nor particular SDL: only
> cross-platform archives and widely available command-line interpreters would
> be a per-requisite for the creation and installation.
> In our opinion, the separation of the services and content of the
> applications, e.g. .deb and .rpm, are an outdated paradigm, which complicates
> the deployment processes, upgrades, and overall system orchestration.
> Architectural proposal will follow shortly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
