[
https://issues.apache.org/jira/browse/THRIFT-4802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thibault Piana updated THRIFT-4802:
-----------------------------------
Description:
Hello everyone,
This thread follows some other threads dealing with concerns about online
libraries (THRIFT-4688, THRIFT-4687), as well as emails being sent to the user
mailing list about the Docker container on DockerHub.
If I understand the situation correctly, at the moment, with each new version
of the software, you must prepare and update each library on each website
independently.
In addition, each of those libraries is managed by different people, with whom
contact is not always easy.
Also, I would like to propose here a system that would offer the following
advantages:
- No more worrying about who is maintaining each platform/CDN/Repositories...
- No more manual uploading on each platform at each release
- Allow anyone to develop an extension to put a Thrift package online on a
particular site (For example, a version for Yocto as I requested in a recent
email), and be sure this package will be updated at each new release.
I will try to explain my idea in the following.
h2. Vocabulary
* *CDN :* Content delivery network
* *Manager :* An algorithm to upload content to a CDN/online repository
* *Orchestrator :* The algorithm that orchestrates the execution of managers,
provides credentials, manages manager feedback, and generates the final report
* *Package* : The library/component we want to upload on the CDN
In the following, all online resources providers/repositories (like Pypi,
NPMJs, DockerHub ...) will be called "CDN"
h2. General principle
The idea would be to develop a single interface for module deployment, with an
orchestrator to automatically configure and perform these deployments.
The organization would be like this :
* A Keepass database will store all passwords/identifiers required for each
platform (NPMJS, Pypi, DockerHub, ...) :
** Each module will specify what credentials it needs from this database.
** This database could be committed directly on the repo, to be sure each
committers have the most recent version.
** The master password of this database will only be known by committers (or
project chair).
I think KeePass is quite perfect for this, Keepass databases are very
lightweight, highly secure, and almost impossible to brute-force.
* For each CDN, a manager is in charge of bundling (create the package to
upload from Thrift current release sources), uploading and checking the package
on the CDN :
** First, the orchestrator load required IDs from the KeePass database
** The orchestrator creates a temporary directory where the manager will be
able to bundle everything it needs.
*** The manager receives credentials, a path to a temporary directory and a
path to Thrift sources
*** The manager checks online if the current version of the package already
exists online. If not, it will upload it
*** The manager prepares the bundled package, upload it and check if
everything went well.
All these steps could be executed in a dedicated Docker/Vagrant environment
containing all needed libraries to bundle the package (So that the
person/commiter executing the script does not have to download 500
dependencies).
The algorithm diagram is provided in attached (it's a draft)
The directory structure of this solution could be something like this (I chose
Python arbitrarily here):
{code:java}
root
└── deployment/
├── access.kdbx
├── deploy.py
├── libraries # Some managers
│ ├── javascript
│ │ ├── npm.py
│ │ ├── bower.py
│ │ └── a_random_cdn.py
│ ├── nodejs
│ │ └── npm.py
│ └── python
│ └── pypi.js
│ └── ...
└── miscellaneous # Some other managers
├── docker
│ └── docker.py
└── yocto
└── yocto.py
└── ...{code}
I am not a regular contributor on open source projects. So I don't know if I'm
making my proposal in the best way. But I really like this project and I would
like to help improve it
I had this idea based on what had been put in place in a project I was recently
leading. It is not perfect but does the job properly, and it's a solution I
find quite elegant because we don't have to worry about the credentials of each
site.
If you find this idea interesting, I could propose a prototype in a personal
repository with some examples of managers for Npm, Pypi, and DockerHub (for
example) in several weeks.
was:
Hello everyone,
This thread follows some other threads dealing with concerns about online
libraries (THRIFT-4688, THRIFT-4687), as well as emails being sent to the user
mailing list about the Docker container on DockerHub.
If I understand the situation correctly, at the moment, with each new version
of the software, you must prepare and update each library on each website
independently.
In addition, each of those libraries is managed by different people, with whom
contact is not always easy.
Also, I would like to propose here a system that would offer the following
advantages:
- No more worrying about who is maintaining each platform
- No more manual uploading on each platform at each release
- Allow anyone to develop an extension to put a Thrift package online on a
particular site (For example, a version for Yocto as I requested in a recent
email), and be sure this package will be updated at each new release.
I will try to explain my idea in the following.
h2. Vocabulary
* *CDN :* Content delivery network
* *Manager :* An algorithm to upload content to a CDN/online repository
* *Orchestrator :* The algorithm that orchestrates the execution of managers,
provides credentials, manages manager feedback, and generates the final report
* *Package* : The library/component we want to upload on the CDN
In the following, all online resources providers/repositories (like Pypi,
NPMJs, DockerHub ...) will be called "CDN"
h2. General principle
The idea would be to develop a single interface for module deployment, with an
orchestrator to automatically configure and perform these deployments.
The organization would be like this :
* A Keepass database will store all passwords/identifiers required for each
platform (NPMJS, Pypi, DockerHub, ...) :
** Each module will specify what credentials it needs from this database.
** This database could be committed directly on the repo, to be sure each
committers have the most recent version.
** The master password of this database will only be known by committers (or
project chair).
I think KeePass is quite perfect for this, Keepass databases are very
lightweight, highly secure, and almost impossible to brute-force.
* For each CDN, a manager is in charge of bundling (create the package to
upload from Thrift current release sources), uploading and checking the package
on the CDN :
** First, the orchestrator load required IDs from the KeePass database
** The orchestrator creates a temporary directory where the manager will be
able to bundle everything it needs.
*** The manager receives credentials, a path to a temporary directory and a
path to Thrift sources
*** The manager checks online if the current version of the package already
exists online. If not, it will upload it
*** The manager prepares the bundled package, upload it and check if
everything went well.
All these steps could be executed in a dedicated Docker/Vagrant environment
containing all needed libraries to bundle the package (So that the
person/commiter executing the script does not have to download 500
dependencies).
The algorithm diagram is provided in attached (it's a draft)
The directory structure of this solution could be something like this (I chose
Python arbitrarily here):
{code:java}
root
└── deployment/
├── access.kdbx
├── deploy.py
├── libraries # Some managers
│ ├── javascript
│ │ ├── npm.py
│ │ ├── bower.py
│ │ └── a_random_cdn.py
│ ├── nodejs
│ │ └── npm.py
│ └── python
│ └── pypi.js
│ └── ...
└── miscellaneous # Some other managers
├── docker
│ └── docker.py
└── yocto
└── yocto.py
└── ...{code}
I am not a regular contributor on open source projects. So I don't know if I'm
making my proposal in the best way. But I really like this project and I would
like to help improve it
I had this idea based on what had been put in place in a project I was recently
leading. It is not perfect but does the job properly, and it's a solution I
find quite elegant because we don't have to worry about the credentials of each
site.
If you find this idea interesting, I could propose a prototype in a personal
repository with some examples of managers for Npm, Pypi, and DockerHub (for
example) in several weeks.
> Automatic library deployment on online CDN/Repository services
> --------------------------------------------------------------
>
> Key: THRIFT-4802
> URL: https://issues.apache.org/jira/browse/THRIFT-4802
> Project: Thrift
> Issue Type: Improvement
> Components: Deployment
> Affects Versions: 0.12.0
> Reporter: Thibault Piana
> Priority: Critical
> Labels: cdn, deploy,deployment, library
> Attachments: Workflow.png
>
>
> Hello everyone,
> This thread follows some other threads dealing with concerns about online
> libraries (THRIFT-4688, THRIFT-4687), as well as emails being sent to the
> user mailing list about the Docker container on DockerHub.
> If I understand the situation correctly, at the moment, with each new version
> of the software, you must prepare and update each library on each website
> independently.
> In addition, each of those libraries is managed by different people, with
> whom contact is not always easy.
> Also, I would like to propose here a system that would offer the following
> advantages:
> - No more worrying about who is maintaining each platform/CDN/Repositories...
> - No more manual uploading on each platform at each release
> - Allow anyone to develop an extension to put a Thrift package online on a
> particular site (For example, a version for Yocto as I requested in a recent
> email), and be sure this package will be updated at each new release.
> I will try to explain my idea in the following.
> h2. Vocabulary
> * *CDN :* Content delivery network
> * *Manager :* An algorithm to upload content to a CDN/online repository
> * *Orchestrator :* The algorithm that orchestrates the execution of
> managers, provides credentials, manages manager feedback, and generates the
> final report
> * *Package* : The library/component we want to upload on the CDN
> In the following, all online resources providers/repositories (like Pypi,
> NPMJs, DockerHub ...) will be called "CDN"
> h2. General principle
> The idea would be to develop a single interface for module deployment, with
> an orchestrator to automatically configure and perform these deployments.
> The organization would be like this :
> * A Keepass database will store all passwords/identifiers required for each
> platform (NPMJS, Pypi, DockerHub, ...) :
> ** Each module will specify what credentials it needs from this database.
> ** This database could be committed directly on the repo, to be sure each
> committers have the most recent version.
> ** The master password of this database will only be known by committers (or
> project chair).
> I think KeePass is quite perfect for this, Keepass databases are very
> lightweight, highly secure, and almost impossible to brute-force.
> * For each CDN, a manager is in charge of bundling (create the package to
> upload from Thrift current release sources), uploading and checking the
> package on the CDN :
> ** First, the orchestrator load required IDs from the KeePass database
> ** The orchestrator creates a temporary directory where the manager will be
> able to bundle everything it needs.
> *** The manager receives credentials, a path to a temporary directory and a
> path to Thrift sources
> *** The manager checks online if the current version of the package already
> exists online. If not, it will upload it
> *** The manager prepares the bundled package, upload it and check if
> everything went well.
> All these steps could be executed in a dedicated Docker/Vagrant environment
> containing all needed libraries to bundle the package (So that the
> person/commiter executing the script does not have to download 500
> dependencies).
> The algorithm diagram is provided in attached (it's a draft)
> The directory structure of this solution could be something like this (I
> chose Python arbitrarily here):
> {code:java}
> root
> └── deployment/
> ├── access.kdbx
> ├── deploy.py
> ├── libraries # Some managers
> │ ├── javascript
> │ │ ├── npm.py
> │ │ ├── bower.py
> │ │ └── a_random_cdn.py
> │ ├── nodejs
> │ │ └── npm.py
> │ └── python
> │ └── pypi.js
> │ └── ...
> └── miscellaneous # Some other managers
> ├── docker
> │ └── docker.py
> └── yocto
> └── yocto.py
> └── ...{code}
> I am not a regular contributor on open source projects. So I don't know if
> I'm making my proposal in the best way. But I really like this project and I
> would like to help improve it
> I had this idea based on what had been put in place in a project I was
> recently leading. It is not perfect but does the job properly, and it's a
> solution I find quite elegant because we don't have to worry about the
> credentials of each site.
> If you find this idea interesting, I could propose a prototype in a personal
> repository with some examples of managers for Npm, Pypi, and DockerHub (for
> example) in several weeks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
