Sure I understand that need. If there's a way of pulling the document sources directly from the web I can parse the content to look for snippet markers and make those available to the template system. That's pretty much how www.apache.org is generated.
Sent from my iPhone > On Mar 19, 2014, at 5:28 AM, Henrique Mendonça <henri...@apache.org> wrote: > > Hi Joe, > > Thanks for your effort. I just wanted to add that the code snippets are a > crucial feature for this project, which has way more target languages than > active contributors. In our case, reducing manual maintenance costs is much > more important than the total site build time. Perhaps we can also solve > this in the CMS, though. > > - Henrique > > >> On 19 March 2014 03:17, Joe Schaefer <joe_schae...@yahoo.com> wrote: >> >> A couple more thoughts regarding the background on the CMS... >> the bulk of the focus of the CMS up until now has been to >> support really big sites like maven and openoffice, both of >> which have websites well over 5GB worth of text files. The >> build system was hollowed out and revamped to support external >> builds in other programming languages, as well as scaling up >> the perl builds to provide about 8 child processes that build >> the site in parallel by default. Full site build times for >> openoffice went from over 30 minutes to between 5 to 10 minutes >> depending on disk activity, which made experimentation with site-wide >> changes feasible on a reasonable timeframe. At first the project was in >> the habit of running the builds locally before checking in their changes >> because the delay in feedback post-commit was unreasonable, but with smart >> use of SSI the were able to cut down on the number of changes that required >> full site builds and hence no longer bother with the whole pre-build mumbo >> jumbo before checking in changes. The CMS is designed to only build pages >> that were altered, and if you just edit a page or two only those pages and >> the pages that depend on them will be built. >> >> Now that I've turned my attention to thrift, there are several aspects >> that are better off being retuned based on the modest size of the site. >> First of all you have a sitemap url that lists all of your markdown-based >> pages, which would be a nonstarter for a site like openoffice. The >> implications of the sitemap url are that every page needs to be built in >> memory just to generate that page, which leads to a lot of redundancy in a >> build system designed to run as forked child processes. We can do better... >> >> I've done two things to make this work well- first I made the number of >> child processes "aka runners" tunable on a per-project basis, and since the >> sitemap is going to build every page anyway as the slowest page to build, I >> memoized the main view that builds the markdown pages for thrift. The best >> results are when there's only one runner so we get the full benefit of >> memoization: typical site build times are consistently under 2 seconds now. >> >> You'll appreciate why I am such a performance stickler for the CMS when >> you start using the bookmarklet, since builds are the only asynchronous >> part of the process from making a change to a page and publishing the >> results. Ideally when the system is working well the builds happen before >> you have time to go looking for them, but you do need to be mindful of not >> publishing before the built pages have been committed back to svn by >> buildbot. >> >> >> >>> On Monday, March 17, 2014 11:59 PM, Joe Schaefer <joe_schae...@yahoo.com> >> wrote: >>>> Ok there's enough of an idea about how to port the rest of the site's >>> pages over based on the porting work I've already done that it's time >>> for me to step back and let you guys catch up. Basically the idea is >> that while >>> there is templating support in the markdown sources, it's fairly limited >>> compared to what nanoc offers, but instead of writing a lot of complex >> template >>> code into your sources, with the cms you just add code to lib/view.pmand/or >>> lib/path.pm to keep the document sources simple and just add more >> template >>> variables. What I just finished in lib/view.pm and lib/path.pm is >> directory >>> listing support for /docs.html but it can be carried over to similar >> pages. >>> >>> >>> >>> >>>> On Monday, March 17, 2014 9:16 PM, Joe Schaefer >>> <joe_schae...@yahoo.com> wrote: >>>>> Interesting ideas, thanks. FWIW I think that sort >>>> of thing might be best supported as a periodic cron >>>> if we can write something stable enough, because pulling >>>> stuff out of the git repo is something the CMS isn't going >>>> to do all that well being an svn application. >>>> >>>> The CMS build results are available at >> http://thrift.staging.apache.org/ >>>> feel free to play with the sources in repos/asf/thrift/cms-site/trunk. >>>> It's very fast, probably an order of magnitude faster than your current >>> >>>> tech. >>>> Whole site builds in two seconds; fractions of a second for typical >> mods to >>> >>>> content/ markdown pages. >>>> >>>> The CMS bookmarklet is compatible with the staging site FWIW. >>>> >>>> >>>> >>>> >>>>> On Monday, March 17, 2014 7:40 PM, Roger Meier >>>> <ro...@bufferoverflow.ch> wrote: >>>>>> Hi Joe & Co >>>>> >>>>> thanks for this test run with Apache CMS! >>>>> >>>>> What I would love to see is this: >>>>> >>>>> Source file: => Web Site URL: >>>>> test/README.md => thrift.apache.org/test >>>>> test/STATUS.md => thrift.apache.org/test/status >>>>> test/keys/README.md => thrift.apache.org/test/keys >>>>> lib/<lang>/README.md => thrift.apache.org/lib/<lang> >>>>> lib/<lang>/test/README.md => >>>> thrift.apache.org/lib/<lang>/test >>>>> lib/<lang>/examples/README.md => >>>>> thrift.apache.org/lib/<lang>/examples >>>>> compiler/cpp/README.md => thrift.apache.org/compiler >>>>> debian/README.md => thrift.apache.org/debian >>>>> contrib/<topic>/README.md => thrift.apache.org/<topic> >>>>> >>>>> just made the following patch to rename all file to .md within our >>> source >>>>> tree: https://issues.apache.org/jira/browse/THRIFT-2407 >>>>> ... we have *41* md's within the source tree and that's the >>> place >>>> where >>>>> people look for documentation first. => simple to manage >>>>> Would be user friendly to have all of this online with each release >>>>> with the URL layout mentioned above. >>>>> >>>>> just received the buildboot... >>>>> Do you have kind of a preview of your prototype? >>>>> >>>>> all the best! >>>>> -roger >>>>> ;-r >>>>> >>>>> >>>>> Quoting Jake Farrell <jfarr...@apache.org>: >>>>> >>>>>> No objections from me >>>>>> >>>>>> -Jake >>>>>> >>>>>> >>>>>> On Mon, Mar 17, 2014 at 12:04 PM, Joe Schaefer >>>>> <joe_schae...@yahoo.com>wrote: >>>>>> >>>>>>> Any objections to me making a copy of the site >>>>>>> tree alongside it called cms-site? I'd like to >>>>>>> get cracking on knocking up the supporting perl >>>>>>> for this so we can continue to brainstorm... >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> On Sunday, March 16, 2014 1:53 PM, Joe Schaefer >>>>> <joe_schae...@yahoo.com> >>>>>>> wrote: >>>>>>>>> On Sunday, March 16, 2014 1:40 PM, Jake Farrell >>>>> <jfarr...@apache.org> >>>>>>>> wrote: >>>>>>>> >>>>>>>> Hey Joe >>>>>>>>> Thanks for reaching out, we chose to go with a site >>>> generator >>>>> over the >>>>>>>>> Apache CMS due to it initially not meeting all of >>> our >>>> needs. >>>>> Our current >>>>>>>>> needs for the site are >>>>>>>>> >>>>>>>>> - Markdown to html conversion >>>>>>>> >>>>>>>> >>>>>>>> That one's easy- its the default for the cms. >>>>>>>> >>>>>>>> >>>>>>>>> - Global variables/config usable throughout in a >>> template >>>> >>>>> based layout >>>>>>>> >>>>>>>> >>>>>>>> Easy too- just create a perl hash somewhere and make it >>>> available >>>>> to >>>>>>> your views. >>>>>>>> Code your views to pass that hash along to your >>> templates >>>> and you >>>>> are >>>>>>> all set. >>>>>>>> Note if your hash contains objects you can make method >>> calls >>>> on >>>>> them in >>>>>>> your >>>>>>>> templates. Note: while I wouldn't recommend this in >>> >>>> general, >>>>> for smaller >>>>>>>> projects like thrift that prefer convenience over >>> separation >>>> of >>>>> concerns >>>>>>> you can >>>>>>>> have the django template engine do a pass over your >>> markdown >>>>> before >>>>>>> passing it >>>>>>>> along to your actual template page, just as is seemingly >>> >>>> common to >>>>> other >>>>>>> popular >>>>>>>> site generation software. >>>>>>>> >>>>>>>> >>>>>>>>> - Syntax highlighting >>>>>>>> >>>>>>>> >>>>>>>> Comes with python's markdown support, but some >>> people >>>> prefer >>>>> the look of >>>>>>>> javascript highlighters. >>>>>>>> >>>>>>>>> - Easily include code snippets from our source code >>> base >>>>>>>> >>>>>>>> >>>>>>>> Statically I hope. No idea how to pull snippets out of >>> your >>>> git >>>>> repo >>>>>>> via the >>>>>>>> cms. >>>>>>>> >>>>>>>>> - Ability to locally test before committing >>>>>>>> >>>>>>>> >>>>>>>> Belt and suspenders with the cms- you can build your >>> changes >>>>> before >>>>>>> committing >>>>>>>> with the build scripts, browse your changes before >>> committing >>>> via >>>>> the cms >>>>>>>> webgui, or simply commit your changes and view the >>> staging >>>> site >>>>> prior to >>>>>>> live >>>>>>>> publication (which is a separate step). >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> -Jake >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Sun, Mar 16, 2014 at 12:44 PM, Joe Schaefer >>>>>>>> <joe_schae...@yahoo.com>wrote: >>>>>>>>> >>>>>>>>>> As it so happens I am interested in working on >>>> supporting >>>>>>>>>> thrift's use case as a potential user of >>> the >>>> Apache >>>>> CMS. >>>>>>>>>> While the CMS works well for massive projects >>> there >>>> are >>>>>>>>>> things it could do better at supporting for >>> more >>>> moderate >>>>>>>>>> sized sites like thrift's. >>>>>>>>>> >>>>>>>>>> I'd therefore like to engage you folks in a >>> >>>>> brainstorming session >>>>>>>>>> about what sorts of features you want for your >>> site >>>> and >>>>> to find >>>>>>>>>> simple ways of supporting those features for >>> you. >>>>>>>>>> >>>>>>>>>> Thanks. >>
