I agree that we need to work on a more comprehensive documentation-building process. Perhaps let's start with removing old code https://github.com/apache/pulsar-site/pull/215. I'd say it needs a code bash to sort current logic - it's somewhat brittle depending on relative path, coupled shell scripts, etc. And yes, it's better for a separate proposal.
Best, tison. tison <wander4...@gmail.com> 于2022年9月20日周二 16:35写道： > Hi Yunze, > > > Just wondering if there is a way to retain the git history in the > pulsar-client-cpp directory? > > Matteo's proposal already write: > > > git filter-repo --subdirectory-filter pulsar-client-cpp > > So you will retain the git history. > > Best, > tison. > > > Yunze Xu <y...@streamnative.io.invalid> 于2022年9月20日周二 16:27写道： > >> LGTM. I also listed the related files outside the pulsar-client-cpp >> directory recently: >> >> - pulsar-common/src/main/proto/PulsarApi.proto: the Pulsar binary >> proto file >> - src/gen-pulsar-version-macro.py: generate the internal version info >> - pulsar-client/src/test/proto/*.proto: test the protobuf native >> schema feature >> >> It would not be a complicated job for that. Just wondering if there is >> a way to retain the git history in the pulsar-client-cpp directory? >> >> Thanks, >> Yunze >> >> >> >> >> > On Sep 20, 2022, at 07:25, Matteo Merli <mme...@apache.org> wrote: >> > >> > https://github.com/apache/pulsar/issues/17724 >> > >> > >> > >> > ## Motivation >> > >> > Pulsar C++ code base is in the same main repository for the Pulsar >> project. >> > >> > While the decision was the right one at the time, there is a >> > considerable overhead >> > in keeping the C++ client in its current position. >> > >> > ### Issues with the current approach >> > >> > The Pulsar repository has grown a lot in size and number of active >> developers. >> > >> > 1. The frequency of changes in various parts of the codebase has >> increased to a >> > point where the amount of resources dedicated to CI is very >> significant. >> > >> > Every change in Java code will trigger the CI jobs for the C++ >> > client and every >> > change in the C++ client will do the same. >> > >> > During a CI job we are building the C++ client multiple times: >> > 1. For C++ and Python client tests >> > 2. To build Python wheels to be included in the pulsar Docker >> > images (for supporting >> > Pulsar functions) >> > >> > 2. The release process for Pulsar has become very complex and >> > requires building a >> > large number of binaries for C++ and Python clients. This has >> > become too much >> > of a burden during the course of a Pulsar release. >> > >> > >> > ## Goal >> > >> > Decouple the development of C++ and Python client libraries from the >> development >> > of the core components of Pulsar in Java. >> > >> > >> > ## Changes >> > >> > ### Repositories >> > >> > 1. Move the C++ client code to a new repository >> > `github.com/apache/pulsar-client-c++` >> <http://github.com/apache/pulsar-client-c++> >> > 2. Move the Python client code to a new repository >> > `github.com/apache/pulsar-client-python` >> <http://github.com/apache/pulsar-client-python> >> > >> > The change will be done without losing any history, extracting a >> > sub-directory into >> > a new Git repository. >> > >> > ``` >> > git filter-repo --subdirectory-filter pulsar-client-cpp >> > ``` >> > >> > ### Release process >> > >> > The release process will be split in multiple parts: >> > >> > 1. the main Pulsar release will only contain the Java parts (server >> > distribution >> > and Java client library) >> > 2. The C++ client will have its own release schedule and versioning >> > 3. The Python client will have its own release schedule and versioning >> > >> > #### Versioning >> > >> > Both C++ and Python clients will continue with their own individual >> versioning. >> > >> > In order to not break anything or cause more confusion, we would need >> to use >> > a new version that is bigger than the current version (2.11.x). >> > >> > The suggestion is to start the new releases for both C++ and Python >> from 3.0.0. >> > >> > >> > #### Existing branches >> > >> > Existing branches of Pulsar, where the C++ client will still be in the >> same main >> > the repository and will be receiving bug fixes in their current >> location. >> > >> > The different location of the new C++ code will make the cherry-picking >> process >> > slightly more painful in the short term, though it will even out in >> long term. >> > >> > >> > ### Projects dependencies >> > >> > #### C++/Python --> Pulsar >> > >> > Both C++ and Python unit/integration tests are designed to run against >> > a standalone >> > instance of Pulsar broker. In the current form, they're using the >> `master` code >> > that is built to run the tests. >> > >> > After the split, the unit tests will use a Docker image of Pulsar. We >> > can use a few >> > different images to test for compatibility >> > 1. Latest stable (eg: 2.10.1) >> > 2. Nightly (Pulsar Docker image published every day from master branch) >> > >> > #### Pulsar --> Python >> > >> > To create a Pulsar image, we are now building the Python client wheel >> > file and then >> > installing it at build time. >> > >> > Instead, we are going to include a wheel file for a version of the >> Python client >> > that has been already released. >> > >> > #### Python --> C++ >> > >> > The Python client library is just a wrapper on top of the C++ client. >> > Today these >> > are built together, with Python wrapper code residing in a >> > sub-directory of C++ client >> > code, and compiled using the same CMake build script. >> > >> > By separating the Python client into a different repository, we are >> going to >> > depend on an already released version of the C++ client. >> > >> > >> > #### Automated documentation in the website >> > >> > On the Pulsar website we are auto-generating C++ documentation with the >> Doxygen >> > tool and the Python one with Pdoc. >> > >> > Instead of just fetching the main repo code, the website build job >> should be >> > also fetching the new repos to run the tooling. >> > >> > >> > >> > >> > >> > >> > -- >> > Matteo Merli >> > <mme...@apache.org> >> >>
