Yes. There are few options for publishing your ASF website, described here: https://www.apache.org/dev/project-site.html. We can publish from a Git repo, SVN, or a UI-based CMS interface.
On Wed, Sep 26, 2018 at 9:45 AM Robert Bradshaw <rober...@google.com> wrote: > I am also definitely in favor of a single repository. Perhaps I'm just > misunderstanding why the generated must be put in a git repository at > all--is it because that's the easiest way to serve them? > > On Wed, Sep 26, 2018 at 3:39 PM Scott Wegner <sc...@apache.org> wrote: > >> Alan found the place where website publishing is configured [1], which >> has examples of project sites being configured with more than one git root. >> This is great for us because it allows us to leave generated >> javadocs/pydocs in the beam-site repository and publish website markdown >> content from the main repo. >> >> Alan has a PR ready to publish generated HTML in a post-commit job [2]. >> Once that goes through the last step is to upgrade the publishing config. >> >> [1] >> https://github.com/apache/infrastructure-puppet/blob/deployment/modules/gitwcsub/files/config/gitwcsub.cfg >> [2] https://github.com/apache/beam/pull/6431 >> >> On Mon, Sep 24, 2018 at 4:35 PM Scott Wegner <sweg...@google.com> wrote: >> >>> > We could add a new default branch (master?) and keep all the >>> non-generated files (src/) there, and put generated files (content/) in the >>> asf-site branch (like we already do). >>> >>> I'm strongly in favor of having sources in a single repository. We have >>> significant process and infrastructure built up for the apache/beam repo >>> (for build, PR, CI, release, etc.) that we can take advantage of by putting >>> website sources in the same repo. The current beam-site repo PR automation >>> is flaky because it was custom-built and not given the same level of >>> attention as the main repo. >>> >>> The caveat to consolidating website sources in the main repo is that it >>> incentivizes putting the generated sources branch on the same repo. I've >>> documented a few of the reasons in the Appendix of the design doc [1]: >>> - It's easier to maintain a single repository; easily apply existing >>> tooling/infrastructure >>> - Jenkins tooling for publishing generated HTML may not work cross-repo >>> [2] >>> >>> My preference is to move forward with the migration of sources to >>> apache/beam [master], and website generated HTML to apache/beam [asf-site]. >>> I like the idea of separating the publishing/hosting of generated >>> javadocs/pydocs since they add so much cruft, but it should not hold up the >>> migration. >>> >>> [1] >>> https://docs.google.com/document/d/1lfbMhdIyDzIaBTgc9OUByhSwR94kfOzS_ozwKWTVl5U/edit#heading=h.wqwi2jpoiiuc >>> >>> [2] >>> https://stackoverflow.com/questions/14843696/checkout-multiple-git-repos-into-same-jenkins-workspace >>> >>> On Mon, Sep 24, 2018 at 2:33 PM Udi Meiri <eh...@google.com> wrote: >>> >>>> Staying on beam-site SGTM. We could add a new default branch (master?) >>>> and keep all the non-generated files (src/) there, and put generated files >>>> (content/) in the asf-site branch (like we already do). >>>> That way there's no confusion as to which files you should update. >>>> (This is of course assuming we still place generated docs in git repos.) >>>> >>>> On Mon, Sep 24, 2018 at 11:23 AM Thomas Weise <t...@apache.org> wrote: >>>> >>>>> My thought was to leave the asf-site branch in the beam-site >>>>> repository, add generated docs to that branch (until we have a better >>>>> solution), and have only sources in the beam repo. >>>>> >>>>> Scott had filed https://issues.apache.org/jira/browse/BEAM-5459 - >>>>> it would eliminate the need to place generated docs into git repos. >>>>> >>>>> On Mon, Sep 24, 2018 at 11:06 AM Udi Meiri <eh...@google.com> wrote: >>>>> >>>>>> I believe that beam.apache.org is populated from the asf-site branch >>>>>> of the apache/beam-site repo. (gitpubsub: >>>>>> https://www.apache.org/dev/project-site.html#intro) >>>>>> If we move the markdown-based docs to apache/beam, leave generated >>>>>> javadoc and pydoc in apache/beam-site, and point gitpubsub to >>>>>> apache/beam, >>>>>> then javadoc and pydoc will not get pushed to the website. >>>>>> >>>>>> Is there some place where we can push javadoc and pydoc files? Or >>>>>> perhaps there an alternative way to push updates to beam.apache.org? >>>>>> (not requiring the asf-site branch) >>>>>> >>>>>> On Fri, Sep 21, 2018 at 6:40 PM Thomas Weise <t...@apache.org> wrote: >>>>>> >>>>>>> Hi Scott, >>>>>>> >>>>>>> Thanks for bringing the discussion back here. >>>>>>> >>>>>>> I agree that we should separate the changes for hosting of generated >>>>>>> java/pydocs from the rest of website automation so that we can make the >>>>>>> switch and fix the contributor headache soon. >>>>>>> >>>>>>> But perhaps we can avoid adding 4m lines of generated code to the >>>>>>> main beam repository (and keep on adding with every release) if we >>>>>>> continue >>>>>>> to serve the site from the old beam-site repo? (I left a comment the >>>>>>> doc.) >>>>>>> >>>>>>> About trying buildbot, as mentioned earlier I would be happy to help >>>>>>> with it. I prefer a setup that keeps the docs separate from the web >>>>>>> site. >>>>>>> >>>>>>> Thomas >>>>>>> >>>>>>> >>>>>>> On Fri, Sep 21, 2018 at 10:28 AM Scott Wegner <sc...@apache.org> >>>>>>> wrote: >>>>>>> >>>>>>>> Re-opening this thread as it came up today in the discussion for >>>>>>>> PR#6458 [1]. This PR is part of the work for Beam-Site Automation >>>>>>>> Reliability improvements; design doc here: >>>>>>>> https://s.apache.org/beam-site-automation >>>>>>>> >>>>>>>> The current plan is to keep generated javadoc/pydoc sources only on >>>>>>>> the asf-site branch, which is necessary for the current githubpubsub >>>>>>>> publishing mechanism. This maintains our current approach, the only >>>>>>>> change >>>>>>>> being that we're moving the asf-site branch from the retiring >>>>>>>> apache/beam-site repository into a new apache/beam repo branch. >>>>>>>> >>>>>>>> The concern for committing generated content is the extra overhead >>>>>>>> during git fetch. I did some analysis to measure the impact [2], and >>>>>>>> found >>>>>>>> that fetching a week of source + generated content history from >>>>>>>> apache/beam-site took 0.39 seconds. >>>>>>>> >>>>>>>> I like the idea of publishing javadoc/pydoc snapshots to an >>>>>>>> external location like Flink does with buildbot, but that work is >>>>>>>> separable >>>>>>>> and shouldn't be a prerequisite for this effort. The goal of this work >>>>>>>> is >>>>>>>> to improve the reliability of automation for contributing website >>>>>>>> changes. >>>>>>>> At last measure, only about half of beam-site PR merges use Mergebot >>>>>>>> without experiencing some reliability issue [3]. >>>>>>>> >>>>>>>> I've opened BEAM-5459 [4] to track moving our generated docs out of >>>>>>>> git. Thomas, would you have bandwidth to look into this? >>>>>>>> >>>>>>>> [1] https://github.com/apache/beam/pull/6458#issuecomment-423406643 >>>>>>>> >>>>>>>> [2] >>>>>>>> https://docs.google.com/document/d/1lfbMhdIyDzIaBTgc9OUByhSwR94kfOzS_ozwKWTVl5U/edit#heading=h.uqzivheohd7j >>>>>>>> [3] >>>>>>>> https://docs.google.com/document/d/1lfbMhdIyDzIaBTgc9OUByhSwR94kfOzS_ozwKWTVl5U/edit#heading=h.a208cwi78xmu >>>>>>>> [4] https://issues.apache.org/jira/browse/BEAM-5459 >>>>>>>> >>>>>>>> On Fri, Aug 24, 2018 at 11:48 AM Thomas Weise <t...@apache.org> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Udi, >>>>>>>>> >>>>>>>>> Good to know you will continue this work. >>>>>>>>> >>>>>>>>> Let me know if you want to try the buildbot route (which does not >>>>>>>>> require generated documentation to be checked into the repo). Happy >>>>>>>>> to help >>>>>>>>> with that. >>>>>>>>> >>>>>>>>> Thomas >>>>>>>>> >>>>>>>>> On Fri, Aug 24, 2018 at 11:36 AM Udi Meiri <eh...@google.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> I'm picking up the website migration. The plan is to not include >>>>>>>>>> generated files in the master branch. >>>>>>>>>> >>>>>>>>>> However, I've been told that even putting generated files a >>>>>>>>>> separate branch could blow up the git repository for all (e.g. make >>>>>>>>>> git >>>>>>>>>> pulls a lot longer?). >>>>>>>>>> Not sure if this is a real issue or not. >>>>>>>>>> >>>>>>>>>> On Mon, Aug 20, 2018 at 2:53 AM Robert Bradshaw < >>>>>>>>>> rober...@google.com> wrote: >>>>>>>>>> >>>>>>>>>>> On Sun, Aug 5, 2018 at 5:28 AM Thomas Weise <t...@apache.org> >>>>>>>>>>> wrote: >>>>>>>>>>> > >>>>>>>>>>> > Yes, I think the separation of generated code will need to >>>>>>>>>>> occur prior to completing the merge and switching the web site to >>>>>>>>>>> the main >>>>>>>>>>> repo. >>>>>>>>>>> > >>>>>>>>>>> > There should be no reason to check generated documentation >>>>>>>>>>> into either of the repos/branches. >>>>>>>>>>> >>>>>>>>>>> Huge +1 to this. Thomas, would have time to set something like >>>>>>>>>>> this up >>>>>>>>>>> for Beam? If not, could anyone else pick this up? >>>>>>>>>>> >>>>>>>>>>> > Please see as an example how this was solved in Flink, using >>>>>>>>>>> the ASF buildbot infrastructure. >>>>>>>>>>> > >>>>>>>>>>> > Documentation per version/release, for example: >>>>>>>>>>> > >>>>>>>>>>> > https://ci.apache.org/projects/flink/flink-docs-release-1.5/ >>>>>>>>>>> > >>>>>>>>>>> > The buildbot configuration is here (requires committer access): >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> https://svn.apache.org/repos/infra/infrastructure/buildbot/aegis/buildmaster/master1/projects/flink.conf >>>>>>>>>>> > >>>>>>>>>>> > Thanks, >>>>>>>>>>> > Thomas >>>>>>>>>>> > >>>>>>>>>>> > On Thu, Aug 2, 2018 at 6:46 PM Mikhail Gryzykhin < >>>>>>>>>>> mig...@google.com> wrote: >>>>>>>>>>> >> >>>>>>>>>>> >> Last time I talked with Scott I brought this idea in. I >>>>>>>>>>> believe the plan was either to publish compiled site to website >>>>>>>>>>> directly, >>>>>>>>>>> or keep it in separate storage from apache/beam repo. >>>>>>>>>>> >> >>>>>>>>>>> >> One of the main reasons not to check in compiled version of >>>>>>>>>>> website is that every developer will have to pull all the versions >>>>>>>>>>> of >>>>>>>>>>> website every time they clone repo, which is not that good of an >>>>>>>>>>> idea to do. >>>>>>>>>>> >> >>>>>>>>>>> >> Regards, >>>>>>>>>>> >> --Mikhail >>>>>>>>>>> >> >>>>>>>>>>> >> Have feedback? >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> >> On Thu, Aug 2, 2018 at 6:42 PM Udi Meiri <eh...@google.com> >>>>>>>>>>> wrote: >>>>>>>>>>> >>> >>>>>>>>>>> >>> Pablo, the docs are generated into versioned paths, e.g., >>>>>>>>>>> https://beam.apache.org/documentation/sdks/javadoc/2.5.0/ so >>>>>>>>>>> tags are not necessary? >>>>>>>>>>> >>> Also, once apache/beam-site is merged with apache/beam the >>>>>>>>>>> release branch should have the relevant docs (although perhaps it's >>>>>>>>>>> better >>>>>>>>>>> to put them in a different repo or storage system). >>>>>>>>>>> >>> >>>>>>>>>>> >>> Thomas, I would very much like to not have javadoc/pydoc >>>>>>>>>>> generation be part of the website review process, as it takes up a >>>>>>>>>>> lot of >>>>>>>>>>> time when changes are staged (10s of thousands of files), >>>>>>>>>>> especially when a >>>>>>>>>>> PR is updated and existing staged files need to be deleted. >>>>>>>>>>> >>> >>>>>>>>>>> >>> >>>>>>>>>>> >>> On Thu, Aug 2, 2018 at 1:15 PM Mikhail Gryzykhin < >>>>>>>>>>> mig...@google.com> wrote: >>>>>>>>>>> >>>> >>>>>>>>>>> >>>> +1 For removing old documentation. >>>>>>>>>>> >>>> >>>>>>>>>>> >>>> @Thomas: Migration work is in backlog and will be picked up >>>>>>>>>>> in near time. >>>>>>>>>>> >>>> >>>>>>>>>>> >>>> --Mikhail >>>>>>>>>>> >>>> >>>>>>>>>>> >>>> Have feedback? >>>>>>>>>>> >>>> >>>>>>>>>>> >>>> >>>>>>>>>>> >>>> On Thu, Aug 2, 2018 at 12:54 PM Thomas Weise < >>>>>>>>>>> t...@apache.org> wrote: >>>>>>>>>>> >>>>> >>>>>>>>>>> >>>>> +1 for removing pre 2.0 documentation (as well as the >>>>>>>>>>> entries from https://beam.apache.org/get-started/downloads/) >>>>>>>>>>> >>>>> >>>>>>>>>>> >>>>> Isn't it part of the beam-site changes that we will no >>>>>>>>>>> longer check in generated documentation into the repository? Those >>>>>>>>>>> can be >>>>>>>>>>> generated and deployed independently (when a commit to a branch >>>>>>>>>>> occurs), >>>>>>>>>>> such as done in the Apex and Flink projects. >>>>>>>>>>> >>>>> >>>>>>>>>>> >>>>> I was told that Scott who was working in the beam-site >>>>>>>>>>> changes is on leave now and the migration is still pending (see >>>>>>>>>>> note at >>>>>>>>>>> https://github.com/apache/beam/tree/master/website). Is anyone >>>>>>>>>>> else going to pick it up? >>>>>>>>>>> >>>>> >>>>>>>>>>> >>>>> Thanks, >>>>>>>>>>> >>>>> Thomas >>>>>>>>>>> >>>>> >>>>>>>>>>> >>>>> >>>>>>>>>>> >>>>> On Thu, Aug 2, 2018 at 12:33 PM Pablo Estrada < >>>>>>>>>>> pabl...@google.com> wrote: >>>>>>>>>>> >>>>>> >>>>>>>>>>> >>>>>> Is it worth adding a tag / branch to the repositories >>>>>>>>>>> every time we make a release, so that people are able to dive in >>>>>>>>>>> and find >>>>>>>>>>> the docs? >>>>>>>>>>> >>>>>> Best >>>>>>>>>>> >>>>>> -P. >>>>>>>>>>> >>>>>> >>>>>>>>>>> >>>>>> On Thu, Aug 2, 2018 at 12:09 PM Ahmet Altay < >>>>>>>>>>> al...@google.com> wrote: >>>>>>>>>>> >>>>>>> >>>>>>>>>>> >>>>>>> I would guess that users are still using some of these >>>>>>>>>>> old releases. It is unclear from Beam website which releases are >>>>>>>>>>> still >>>>>>>>>>> supported or not. It probably makes sense to drop documentation for >>>>>>>>>>> releases < 2.0. (I would suggest keeping docs for 2.0). For the >>>>>>>>>>> future I >>>>>>>>>>> can work on updating the Beam website to clarify the state of each >>>>>>>>>>> release. >>>>>>>>>>> >>>>>>> >>>>>>>>>>> >>>>>>> On Thu, Aug 2, 2018 at 12:06 PM, Udi Meiri < >>>>>>>>>>> eh...@google.com> wrote: >>>>>>>>>>> >>>>>>>> >>>>>>>>>>> >>>>>>>> The older docs are not directly linked to and are in >>>>>>>>>>> Github commit history. >>>>>>>>>>> >>>>>>>> >>>>>>>>>>> >>>>>>>> If there are no objections I'm going to delete javadocs >>>>>>>>>>> and pydocs for releases older than 1 year, >>>>>>>>>>> >>>>>>>> meaning 2.0.0 and older (going by the dates here). >>>>>>>>>>> >>>>>>>> >>>>>>>>>>> >>>>>>>> On Thu, Aug 2, 2018 at 11:51 AM Daniel Oliveira < >>>>>>>>>>> danolive...@google.com> wrote: >>>>>>>>>>> >>>>>>>>> >>>>>>>>>>> >>>>>>>>> The older docs should be recorded in the commit >>>>>>>>>>> history of the website repository, right? If they're not currently >>>>>>>>>>> used in >>>>>>>>>>> the website and they're in the commit history then I don't see a >>>>>>>>>>> reason to >>>>>>>>>>> save them. >>>>>>>>>>> >>>>>>>>> >>>>>>>>>>> >>>>>>>>> On Tue, Jul 31, 2018 at 1:51 PM Udi Meiri < >>>>>>>>>>> eh...@google.com> wrote: >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> Hi all, >>>>>>>>>>> >>>>>>>>>> I'm writing a PR for apache/beam-site and >>>>>>>>>>> beam_PreCommit_Website_Stage is timing out after 100 minutes, >>>>>>>>>>> because it's >>>>>>>>>>> trying to deletes 22k files and then copy 22k files (warning large >>>>>>>>>>> file). >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> It seems that we could save a lot of time by deleting >>>>>>>>>>> the older javadoc and pydoc files for older versions. Is there a >>>>>>>>>>> good >>>>>>>>>>> reason to keep around this kind of documentation for older versions >>>>>>>>>>> (say 1 >>>>>>>>>>> year back)? >>>>>>>>>>> >>>>>>> >>>>>>>>>>> >>>>>>> >>>>>>>>>>> >>>>>> -- >>>>>>>>>>> >>>>>> Got feedback? go/pabloem-feedback >>>>>>>>>>> <https://goto.google.com/pabloem-feedback> >>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Got feedback? tinyurl.com/swegner-feedback >>>>>>>> >>>>>>> >>> >>> -- >>> >>> >>> >>> >>> Got feedback? tinyurl.com/swegner-feedback >>> >> >> >> -- >> >> >> >> >> Got feedback? tinyurl.com/swegner-feedback >> > -- Got feedback? tinyurl.com/swegner-feedback
