Re: Proposal for migrating web content to git
Hi Perfect! Thanks a lot for the hard/good work!! -- Olivier On Wed, 12 Dec 2018 at 07:02, Martin Stockhammer wrote: > Hi together, > > after my last request about the status on the JIRA ticket, the > infrastructure guys changed the publish process from svn to git publish > without further asking. > So I will not ask for any votes, just raise your hand, if you have still > concerns about it. > > I checked the publish process and it worked fine. > > Changes for the publish process (maven part, scripts, repository urls) > are commited to master now. > > > Some notes about the publish process: > - archiva-web-content.git is like a mirror to the web site > archiva.apache.org >Anything that is pushed to the master branch on this will be > available on the web site. > - the archiva-web-content.git is fed by different maven configurations > into subfolders: >- archiva/archiva-doc >- archiva/archiva-modules/... >- archiva-site >- archiva-redback-site >- archiva-redback-core >- archiva-redback-components/... > > - There are README files and scripts on the corresponding directories >that tells how to build, check and deploy the site documents. > > I did not run a complete publish on all this components, just checked > the publish process to a local repo. So the site has not changed by this > migration. > > > Regards > > Martin > > > On 08.11.18 08:52, Martin wrote: > > Hi, > > > > ok, so I will prepare the redback changes too. > > And yes, I will provide some documentation, for sure. > > > > I will start the vote in parallel, in the hope that some other > registered developers are listening here. > > > > Regards > > > > Martin > > > > > > > > Am Dienstag, 6. November 2018, 01:02:55 CET schrieb Olivier Lamy: > >> Hi > >> Looks good to me. Thanks for the hard work! > >> Maybe add documentation somewhere? > >> as a README file in the git repos (build/deploy site section)? > >> > >> On Mon, 5 Nov 2018 at 00:33, Martin wrote: > >> > >>> Hi together, > >>> > >>> sorry for the long mail, but it's a bit complicated ... > >>> > >>> The archiva webcontent repo (site-content) is the last one that still > >>> resides on SVN. > >>> I would like to move this one to git using the gitpubsub mechanism for > >>> deployment to the HTTP server. > >>> > >>> There is a ticket at INFRA open already > >>> https://issues.apache.org/jira/browse/INFRA-16387 to get more > information > >>> about > >>> the migration process. > >>> > >>> This repo is still not migrated to git, because: > >>> - The size is really a concern, because the old ref docs of previous > >>> versions are stored there > >>> - there are some specialities in the site generation process that are > >>> bound to the svnpubsub and svn to avoid checking out > >>>the whole repo > >>> - The archiva-modules and archiva-docs site content is stored in a > >>> subdirectory which is working fine with SVN but not so easy > >>>with git > >>> - redback site publish is using the same repository > >>> > >>> So for the migration, we have to make sure that: > >>> 1.) the content of the git repo is the same as of the SVN repo > >>> 2.) we are still able to publish site changes from our source tree to > the > >>> public archiva site: > >>> - archiva-site -> The main site pages > >>> - archiva/archiva-docs -> The user and admin documentation > >>> - archiva/archiva-modules/... -> the reports and module > specific > >>> documentation > >>> - redback -> The redback documentation > >>> 3.) the publish process does not need to checkout the whole content > >>> repository to publish one of the documentations above > >>> > >>> I think I have a proof of concept for these tasks (not for redback now, > >>> but it will be the similar). You can check it out from the > >>> Branch feature/site-git-migration > >>> > >>> for 1.) There is a migrated repo > >>> https://gitbox.apache.org/repos/asf/archiva-web-content.git with the > >>> current status of the SVN repository > >>> for 2.) > >>> I'm still using the maven-scm-publish-plugin which is > capable of > >>> using git repositories > >>> The part with subdirectories and submodules is a bit tricky, > >>> because you cannot specifiy a subdirectory of a git repository in the > >>> publish url. > >>> I change the path for the staging directory used by the site > >>> plugin (e.g. to target/staging/docs/${project.version}) and reference > the > >>> staging > >>> root directory in the scm-publish plugin. > >>> For submodules we need a path to the archiva-modules project > as > >>> staging dir, therefore the new property entries in the submodules (if > >>> somebody knows a > >>> better solution, please tell me) > >>> for 3.) I'm using git sparse checkouts which allows to checkout only > >>> subtrees of a repository. The git repository index is still needed for > each > >>> site checkout, but > >>> its about 86MB and that is sti
Re: Proposal for migrating web content to git
Hi together, after my last request about the status on the JIRA ticket, the infrastructure guys changed the publish process from svn to git publish without further asking. So I will not ask for any votes, just raise your hand, if you have still concerns about it. I checked the publish process and it worked fine. Changes for the publish process (maven part, scripts, repository urls) are commited to master now. Some notes about the publish process: - archiva-web-content.git is like a mirror to the web site archiva.apache.org Anything that is pushed to the master branch on this will be available on the web site. - the archiva-web-content.git is fed by different maven configurations into subfolders: - archiva/archiva-doc - archiva/archiva-modules/... - archiva-site - archiva-redback-site - archiva-redback-core - archiva-redback-components/... - There are README files and scripts on the corresponding directories that tells how to build, check and deploy the site documents. I did not run a complete publish on all this components, just checked the publish process to a local repo. So the site has not changed by this migration. Regards Martin On 08.11.18 08:52, Martin wrote: Hi, ok, so I will prepare the redback changes too. And yes, I will provide some documentation, for sure. I will start the vote in parallel, in the hope that some other registered developers are listening here. Regards Martin Am Dienstag, 6. November 2018, 01:02:55 CET schrieb Olivier Lamy: Hi Looks good to me. Thanks for the hard work! Maybe add documentation somewhere? as a README file in the git repos (build/deploy site section)? On Mon, 5 Nov 2018 at 00:33, Martin wrote: Hi together, sorry for the long mail, but it's a bit complicated ... The archiva webcontent repo (site-content) is the last one that still resides on SVN. I would like to move this one to git using the gitpubsub mechanism for deployment to the HTTP server. There is a ticket at INFRA open already https://issues.apache.org/jira/browse/INFRA-16387 to get more information about the migration process. This repo is still not migrated to git, because: - The size is really a concern, because the old ref docs of previous versions are stored there - there are some specialities in the site generation process that are bound to the svnpubsub and svn to avoid checking out the whole repo - The archiva-modules and archiva-docs site content is stored in a subdirectory which is working fine with SVN but not so easy with git - redback site publish is using the same repository So for the migration, we have to make sure that: 1.) the content of the git repo is the same as of the SVN repo 2.) we are still able to publish site changes from our source tree to the public archiva site: - archiva-site -> The main site pages - archiva/archiva-docs -> The user and admin documentation - archiva/archiva-modules/... -> the reports and module specific documentation - redback -> The redback documentation 3.) the publish process does not need to checkout the whole content repository to publish one of the documentations above I think I have a proof of concept for these tasks (not for redback now, but it will be the similar). You can check it out from the Branch feature/site-git-migration for 1.) There is a migrated repo https://gitbox.apache.org/repos/asf/archiva-web-content.git with the current status of the SVN repository for 2.) I'm still using the maven-scm-publish-plugin which is capable of using git repositories The part with subdirectories and submodules is a bit tricky, because you cannot specifiy a subdirectory of a git repository in the publish url. I change the path for the staging directory used by the site plugin (e.g. to target/staging/docs/${project.version}) and reference the staging root directory in the scm-publish plugin. For submodules we need a path to the archiva-modules project as staging dir, therefore the new property entries in the submodules (if somebody knows a better solution, please tell me) for 3.) I'm using git sparse checkouts which allows to checkout only subtrees of a repository. The git repository index is still needed for each site checkout, but its about 86MB and that is still manageable. There is a shell script in the archiva-docs and archiva-modules folder that runs the clone and sparse checkout and which is executed by a exec plugin. It is linux/bash only, if we need windows as well, I may work on some platform independent way. I did not checkin a valid publish url for the archiva-web-content.git repository, to avoid littering it with too much testing commits, but you can test it with a local git clone (must be a bare clone): git clone --bare https://gitbox.apache.org/repos/asf/archiva-web-content.git ${HOME}/git/archiva-web-content.git With this local clone you can test the publish
Re: Proposal for migrating web content to git
Hi, ok, so I will prepare the redback changes too. And yes, I will provide some documentation, for sure. I will start the vote in parallel, in the hope that some other registered developers are listening here. Regards Martin Am Dienstag, 6. November 2018, 01:02:55 CET schrieb Olivier Lamy: > Hi > Looks good to me. Thanks for the hard work! > Maybe add documentation somewhere? > as a README file in the git repos (build/deploy site section)? > > On Mon, 5 Nov 2018 at 00:33, Martin wrote: > > > Hi together, > > > > sorry for the long mail, but it's a bit complicated ... > > > > The archiva webcontent repo (site-content) is the last one that still > > resides on SVN. > > I would like to move this one to git using the gitpubsub mechanism for > > deployment to the HTTP server. > > > > There is a ticket at INFRA open already > > https://issues.apache.org/jira/browse/INFRA-16387 to get more information > > about > > the migration process. > > > > This repo is still not migrated to git, because: > > - The size is really a concern, because the old ref docs of previous > > versions are stored there > > - there are some specialities in the site generation process that are > > bound to the svnpubsub and svn to avoid checking out > > the whole repo > > - The archiva-modules and archiva-docs site content is stored in a > > subdirectory which is working fine with SVN but not so easy > > with git > > - redback site publish is using the same repository > > > > So for the migration, we have to make sure that: > > 1.) the content of the git repo is the same as of the SVN repo > > 2.) we are still able to publish site changes from our source tree to the > > public archiva site: > > - archiva-site -> The main site pages > > - archiva/archiva-docs -> The user and admin documentation > > - archiva/archiva-modules/... -> the reports and module specific > > documentation > > - redback -> The redback documentation > > 3.) the publish process does not need to checkout the whole content > > repository to publish one of the documentations above > > > > I think I have a proof of concept for these tasks (not for redback now, > > but it will be the similar). You can check it out from the > > Branch feature/site-git-migration > > > > for 1.) There is a migrated repo > > https://gitbox.apache.org/repos/asf/archiva-web-content.git with the > > current status of the SVN repository > > for 2.) > > I'm still using the maven-scm-publish-plugin which is capable of > > using git repositories > > The part with subdirectories and submodules is a bit tricky, > > because you cannot specifiy a subdirectory of a git repository in the > > publish url. > > I change the path for the staging directory used by the site > > plugin (e.g. to target/staging/docs/${project.version}) and reference the > > staging > > root directory in the scm-publish plugin. > > For submodules we need a path to the archiva-modules project as > > staging dir, therefore the new property entries in the submodules (if > > somebody knows a > > better solution, please tell me) > > for 3.) I'm using git sparse checkouts which allows to checkout only > > subtrees of a repository. The git repository index is still needed for each > > site checkout, but > > its about 86MB and that is still manageable. > > There is a shell script in the archiva-docs and archiva-modules > > folder that runs the clone and sparse checkout and which is executed by a > > exec plugin. > > It is linux/bash only, if we need windows as well, I may work on > > some platform independent way. > > > > > > I did not checkin a valid publish url for the archiva-web-content.git > > repository, to avoid littering it with too much testing commits, but you > > can test it with a > > local git clone (must be a bare clone): > > git clone --bare > > https://gitbox.apache.org/repos/asf/archiva-web-content.git > > ${HOME}/git/archiva-web-content.git > > > > With this local clone you can test the publish process: > > > > archiva-site: > > == > > - Checkout the branch feature/site-git-migration > > - Run > > mvn > > -DsiteRepositoryUrl=scm:git:file:///${HOME}/git/archiva-web-content.git > > clean site > > # if you want to look at the staging area in target/staging > > mvn > > -DsiteRepositoryUrl=scm:git:file:///${HOME}/git/archiva-web-content.git > > site:stage > > # Deploy: > > mvn > > -DsiteRepositoryUrl=scm:git:file:///${HOME}/git/archiva-web-content.git > > site-deploy > > # You can verify the changes in your archiva-web-content.git clone > > > > archiva-docs: > > == > > - Checkout the branch feature/site-git-migration of archiva repo > > - cd archiva-docs > > - Run > > mvn > > -DsiteRepositoryUrl=scm:git:file:///${HOME}/git/archiva-web-content.git > > clean site > > # if you want to look at the staging area in target/staging > > mvn > > -DsiteRepositoryUrl=scm:git:file:///${HOME}/git/archiva-web
Re: Proposal for migrating web content to git
Hi Looks good to me. Thanks for the hard work! Maybe add documentation somewhere? as a README file in the git repos (build/deploy site section)? On Mon, 5 Nov 2018 at 00:33, Martin wrote: > Hi together, > > sorry for the long mail, but it's a bit complicated ... > > The archiva webcontent repo (site-content) is the last one that still > resides on SVN. > I would like to move this one to git using the gitpubsub mechanism for > deployment to the HTTP server. > > There is a ticket at INFRA open already > https://issues.apache.org/jira/browse/INFRA-16387 to get more information > about > the migration process. > > This repo is still not migrated to git, because: > - The size is really a concern, because the old ref docs of previous > versions are stored there > - there are some specialities in the site generation process that are > bound to the svnpubsub and svn to avoid checking out > the whole repo > - The archiva-modules and archiva-docs site content is stored in a > subdirectory which is working fine with SVN but not so easy > with git > - redback site publish is using the same repository > > So for the migration, we have to make sure that: > 1.) the content of the git repo is the same as of the SVN repo > 2.) we are still able to publish site changes from our source tree to the > public archiva site: > - archiva-site -> The main site pages > - archiva/archiva-docs -> The user and admin documentation > - archiva/archiva-modules/... -> the reports and module specific > documentation > - redback -> The redback documentation > 3.) the publish process does not need to checkout the whole content > repository to publish one of the documentations above > > I think I have a proof of concept for these tasks (not for redback now, > but it will be the similar). You can check it out from the > Branch feature/site-git-migration > > for 1.) There is a migrated repo > https://gitbox.apache.org/repos/asf/archiva-web-content.git with the > current status of the SVN repository > for 2.) > I'm still using the maven-scm-publish-plugin which is capable of > using git repositories > The part with subdirectories and submodules is a bit tricky, > because you cannot specifiy a subdirectory of a git repository in the > publish url. > I change the path for the staging directory used by the site > plugin (e.g. to target/staging/docs/${project.version}) and reference the > staging > root directory in the scm-publish plugin. > For submodules we need a path to the archiva-modules project as > staging dir, therefore the new property entries in the submodules (if > somebody knows a > better solution, please tell me) > for 3.) I'm using git sparse checkouts which allows to checkout only > subtrees of a repository. The git repository index is still needed for each > site checkout, but > its about 86MB and that is still manageable. > There is a shell script in the archiva-docs and archiva-modules > folder that runs the clone and sparse checkout and which is executed by a > exec plugin. > It is linux/bash only, if we need windows as well, I may work on > some platform independent way. > > > I did not checkin a valid publish url for the archiva-web-content.git > repository, to avoid littering it with too much testing commits, but you > can test it with a > local git clone (must be a bare clone): > git clone --bare > https://gitbox.apache.org/repos/asf/archiva-web-content.git > ${HOME}/git/archiva-web-content.git > > With this local clone you can test the publish process: > > archiva-site: > == > - Checkout the branch feature/site-git-migration > - Run > mvn > -DsiteRepositoryUrl=scm:git:file:///${HOME}/git/archiva-web-content.git > clean site > # if you want to look at the staging area in target/staging > mvn > -DsiteRepositoryUrl=scm:git:file:///${HOME}/git/archiva-web-content.git > site:stage > # Deploy: > mvn > -DsiteRepositoryUrl=scm:git:file:///${HOME}/git/archiva-web-content.git > site-deploy > # You can verify the changes in your archiva-web-content.git clone > > archiva-docs: > == > - Checkout the branch feature/site-git-migration of archiva repo > - cd archiva-docs > - Run > mvn > -DsiteRepositoryUrl=scm:git:file:///${HOME}/git/archiva-web-content.git > clean site > # if you want to look at the staging area in target/staging > mvn > -DsiteRepositoryUrl=scm:git:file:///${HOME}/git/archiva-web-content.git > site:stage > # Deploy: > mvn > -DsiteRepositoryUrl=scm:git:file:///${HOME}/git/archiva-web-content.git > site-deploy > # You can verify the changes in your archiva-web-content.git clone > > > archiva-modules: > = > I reduced the generated reports to speedup the build process. > - Checkout the branch feature/site-git-migration of archiva repo > - Run > cd archiva-modules > ./deploySite.sh > -DsiteRepositoryUrl=scm:git:file:///${HOME}/git/archiva-web-content.git > site-deploy > # You can verify the