[
https://issues.apache.org/jira/browse/HAWQ-706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Boudnik updated HAWQ-706:
------------------------------------
Description:
This ticket is to list and track the required steps to finally enable the
integration of HAWQ into Bigtop.
All relevant resources are linked below, and here's the overview of the
remaining steps and the overall status of the integration work.
# External dependencies
- the biggest issue was and remains the use of libthrift, which isn't packaged,
provided nor supported by anyone. Right now, Bigtop-HAWQ integration branch
uses my own pre-built version of the library, hosted [here|
# Packaging
- overall, the packaging code is complete and is pushed to the Bigtop branch
(see link below). Considering that the work has been completed about 5 weeks
ago and was aimed at the state of trunk back in the March, there might be some
minor changes, which would require additional tweaks
- libhdfs library code (if already included into HAWQ project) might require
additional changes to the packaging code, so the library can be produces and
properly set in the installation phase
- Bigtop CI has jobs to create CentOS and Ubuntu packages (linked from the
BIGTOP-2320 below)
# Tests
- smoke tests need to be created (as per BIGTOP-2322), but that seems to be a
minor undertaking once the rest of the work is finished
- packaging tests are required to be integrated into Bigtop stack BIGTOP-2324
# Deployment
- deployment code is completed. However, it needs to be extended to property
support cluster roles and to be linked to the main {{site.pp}} recipe
- because real-life deployment can not rely on in-house python wrappers using
passwordless-ssh, the lifecycle management and initial bootstrap are done
directly by calling into HAWQ scripts, providing such functionality. It is
possible that some of these interfaces were updated in the last 6 weeks, so
additional testing would be needed.
- it should be responsibility of the HAWQ to provide a concise way of
initializing a master, segment, and so on without a need for password-less ssh,
which is suboptimal and won't be accepted by Bigtop community as it is breaks
the deployment model
# Toolchains
- toolchain code is completed in the bigtop branch. This will allow to build
HAWQ in the standard Bigtop container available for the CI and 3rd party users
- toolchain code needs to be rebased on top of current Bigtop master. and
possible conflicts would have to be resolved
- once the integration is finished, Bigtop slave images will have to be updated
to enable automatic CI runs
was:
This ticket is to list and track the required steps to finally enable the
integration of HAWQ into Bigtop.
All relevant resources are linked below, and here's the overview of the
remaining steps and the overall status of the integration work.
# Packaging
- overall, the packaging code is complete and could be found on the Bigtop
branch (see link below). Considering, that the work has been completed about 5
weeks ago and was aimed at the state of trunk back in the March, there might be
some minor changes, which would require additional tweaks
- libhdfs library code (if already included into HAWQ project) might require
additional changes to the packaging code, so the library can be produces and
properly set in the installation phase
- Bigtop CI has jobs to create CentOS and Ubuntu packages (linked from the
BIGTOP-2320 below)
# Tests
- smoke tests need to be created (as per BIGTOP-2322
- packaging tests are required to be integrated into Bigtop stack BIGTOP-2324
# Deployment
- deployment code is completed. However, it needs to be extended to property
support cluster roles and to be linked to the main {{site.pp}} recipe
# Toolchains
- toolchain code is completed in the bigtop branch. This will allow to build
HAWQ in the standard Bigtop container available for the CI and 3rd party users
- toolchain code needs to be rebased on top of current Bigtop master. and
possible conflicts would have to be resolved
> Final steps to bring HAWQ in compliance with Bigtop reqs
> --------------------------------------------------------
>
> Key: HAWQ-706
> URL: https://issues.apache.org/jira/browse/HAWQ-706
> Project: Apache HAWQ
> Issue Type: Task
> Components: Build
> Affects Versions: 2.0.0-beta-incubating
> Reporter: Konstantin Boudnik
> Assignee: Konstantin Boudnik
>
> This ticket is to list and track the required steps to finally enable the
> integration of HAWQ into Bigtop.
> All relevant resources are linked below, and here's the overview of the
> remaining steps and the overall status of the integration work.
> # External dependencies
> - the biggest issue was and remains the use of libthrift, which isn't
> packaged, provided nor supported by anyone. Right now, Bigtop-HAWQ
> integration branch uses my own pre-built version of the library, hosted [here|
> # Packaging
> - overall, the packaging code is complete and is pushed to the Bigtop branch
> (see link below). Considering that the work has been completed about 5 weeks
> ago and was aimed at the state of trunk back in the March, there might be
> some minor changes, which would require additional tweaks
> - libhdfs library code (if already included into HAWQ project) might require
> additional changes to the packaging code, so the library can be produces and
> properly set in the installation phase
> - Bigtop CI has jobs to create CentOS and Ubuntu packages (linked from the
> BIGTOP-2320 below)
> # Tests
> - smoke tests need to be created (as per BIGTOP-2322), but that seems to be a
> minor undertaking once the rest of the work is finished
> - packaging tests are required to be integrated into Bigtop stack BIGTOP-2324
> # Deployment
> - deployment code is completed. However, it needs to be extended to property
> support cluster roles and to be linked to the main {{site.pp}} recipe
> - because real-life deployment can not rely on in-house python wrappers using
> passwordless-ssh, the lifecycle management and initial bootstrap are done
> directly by calling into HAWQ scripts, providing such functionality. It is
> possible that some of these interfaces were updated in the last 6 weeks, so
> additional testing would be needed.
> - it should be responsibility of the HAWQ to provide a concise way of
> initializing a master, segment, and so on without a need for password-less
> ssh, which is suboptimal and won't be accepted by Bigtop community as it is
> breaks the deployment model
> # Toolchains
> - toolchain code is completed in the bigtop branch. This will allow to build
> HAWQ in the standard Bigtop container available for the CI and 3rd party users
> - toolchain code needs to be rebased on top of current Bigtop master. and
> possible conflicts would have to be resolved
> - once the integration is finished, Bigtop slave images will have to be
> updated to enable automatic CI runs
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
