While I don't yet know the url magic to support branches and tags in git, I can say that the snippet support is ready to play with. I suggest just focusing on the /index.md page first to see how it works in practice because once it's fully enabled the build will break until all the snippet sources have been altered to conform with the new style.
To turn it on for index.md testing, there are a couple of things that need to happen. First the [snippet:...] marker should be edited to be [XXXsnippet:...] to enable testing without breaking the build. Second each marker contains a "token" argument instead of line numbers for more stability under edits to the source code. The token argument is an arbitrary string that is used to denote the boundaries of the snippet within the source code. ASF::Value::Snippet will pull out the lines from the source file between the lines that contain the token argument and format them as a code snippet embedded into markdown. To get started I'll copy over the code highlighting css from www.apache.org but note that this is all based on python pygments and there are a variety of online css files to choose from if this one is not to the group's liking. > On Wednesday, March 19, 2014 4:16 PM, Joe Schaefer <joe_schae...@yahoo.com> > wrote: > > Sorry I need to clarify with an example: look at > http://thrift.staging.apache.org/tutorial/ versus > http://thrift.staging.apache.org/tutorial.html. The > former is generated from content/tutorial/index.html > and the latter from content/tutorial.md. The latter file > needs to be removed from the tree now that it's content > has been ported to content/tutorial/index.html. This > uses django template inheritance to maintain simplicity. > > Other directories that follow a similar pattern should > also be ported in the same fashion. > > > >> On Wednesday, March 19, 2014 9:46 AM, Joe Schaefer > <joe_schae...@yahoo.com> wrote: >> > If someone's looking for a way to help with the porting >> effort, consider tackling the creation of an index.html >> page in each subdir of the content/ tree (other than the >> base which has an /index.md). The contents of the file >> should be just >> >> {% include 'default' %} >> >> (see the docs dir for an example page that can be copied) >> and the build system will take care of generating the page >> in a consistent fashion with the rest of the site. Once >> that is done there are a lot of pages that function like >> index pages but are written in markdown that can be deleted, >> like docs.md and its ilk. Nanoc transforms that file into >> /docs/index.html but that's kinda crufty and not supported >> by the CMS, which only lets you alter file extensions not paths. >> >> >> >> >>> On Wednesday, March 19, 2014 8:54 AM, Joe Schaefer >> <joe_schae...@yahoo.com> wrote: >>> > I just pencilled in snippet preprocessing support into >>> the thrift builds. Once Jake and I have settled on the >>> feature set of ASF::Value::Snippet, and we have ported >>> the snippet calling text within the markdown pages >>> to the new format, things will just work (knock on wood). >>> >>> >>> >>> >>>> On Wednesday, March 19, 2014 6:48 AM, Joseph Schaefer >>> <joe_schae...@yahoo.com> wrote: >>>> > Sure I understand that need. If there's a way of > pulling the >> >>> document >>>> sources directly from the web I can parse the content to look for > >> snippet >>>> markers and make those available to the template system. > That's >> pretty >>> much >>>> how www.apache.org is generated. >>>> >>>> Sent from my iPhone >>>> >>>> >>>>> On Mar 19, 2014, at 5:28 AM, Henrique Mendonça >>> <henri...@apache.org> >>>> wrote: >>>>> >>>>> Hi Joe, >>>>> >>>>> Thanks for your effort. I just wanted to add that the code >> snippets >>> are a >>>>> crucial feature for this project, which has way more target >> languages >>> than >>>>> active contributors. In our case, reducing manual > maintenance >> costs is >>> much >>>>> more important than the total site build time. Perhaps we > can >> also >>> solve >>>>> this in the CMS, though. >>>>> >>>>> - Henrique >>>>> >>>>> >>>>>> On 19 March 2014 03:17, Joe Schaefer >>> <joe_schae...@yahoo.com> >>>> wrote: >>>>>> >>>>>> A couple more thoughts regarding the background on the > CMS... >>>>>> the bulk of the focus of the CMS up until now has been > to >>>>>> support really big sites like maven and openoffice, both > of >>>>>> which have websites well over 5GB worth of text files. > The >>>>>> build system was hollowed out and revamped to support >> external >>>>>> builds in other programming languages, as well as > scaling up >>>>>> the perl builds to provide about 8 child processes that > build >>>>>> the site in parallel by default. Full site build times > for >>>>>> openoffice went from over 30 minutes to between 5 to 10 >> minutes >>>>>> depending on disk activity, which made experimentation > with >>> site-wide >>>>>> changes feasible on a reasonable timeframe. At first > the >> project >>> was >>>> in >>>>>> the habit of running the builds locally before checking > in >> their >>>> changes >>>>>> because the delay in feedback post-commit was > unreasonable, >> but >>> with >>>> smart >>>>>> use of SSI the were able to cut down on the number of > changes >> that >>> >>>> required >>>>>> full site builds and hence no longer bother with the > whole >>> pre-build >>>> mumbo >>>>>> jumbo before checking in changes. The CMS is designed > to >> only >>> build >>>> pages >>>>>> that were altered, and if you just edit a page or two > only >> those >>> pages >>>> and >>>>>> the pages that depend on them will be built. >>>>>> >>>>>> Now that I've turned my attention to thrift, there > are >> several >>> >>>> aspects >>>>>> that are better off being retuned based on the modest > size of >> the >>> site. >>>>>> First of all you have a sitemap url that lists all of > your >>>> markdown-based >>>>>> pages, which would be a nonstarter for a site like >> openoffice. >>> The >>>>>> implications of the sitemap url are that every page > needs to >> be >>> built >>>> in >>>>>> memory just to generate that page, which leads to a lot > of >>> redundancy >>>> in a >>>>>> build system designed to run as forked child processes. > We >> can do >>> >>>> better... >>>>>> >>>>>> I've done two things to make this work well- first I > made >> the >>>> number of >>>>>> child processes "aka runners" tunable on a >> per-project >>> basis, >>>> and since the >>>>>> sitemap is going to build every page anyway as the > slowest >> page to >>> >>>> build, I >>>>>> memoized the main view that builds the markdown pages > for >> thrift. >>> The >>>> best >>>>>> results are when there's only one runner so we get > the >> full >>> benefit >>>> of >>>>>> memoization: typical site build times are consistently > under >> 2 >>> seconds >>>> now. >>>>>> >>>>>> You'll appreciate why I am such a performance > stickler >> for the >>> CMS >>>> when >>>>>> you start using the bookmarklet, since builds are the > only >>> asynchronous >>>>>> part of the process from making a change to a page and >> publishing >>> the >>>>>> results. Ideally when the system is working well the > builds >>> happen >>>> before >>>>>> you have time to go looking for them, but you do need to > be >>> mindful of >>>> not >>>>>> publishing before the built pages have been committed > back to >> svn >>> by >>>>>> buildbot. >>>>>> >>>>>> >>>>>> >>>>>>> On Monday, March 17, 2014 11:59 PM, Joe Schaefer >>>> <joe_schae...@yahoo.com> >>>>>> wrote: >>>>>>>> Ok there's enough of an idea about how to > port >> the >>> rest of >>>> the site's >>>>>>> pages over based on the porting work I've > already >> done >>> that >>>> it's time >>>>>>> for me to step back and let you guys catch up. > Basically >> the >>> idea >>>> is >>>>>> that while >>>>>>> there is templating support in the markdown sources, > >> it's >>>> fairly limited >>>>>>> compared to what nanoc offers, but instead of > writing a >> lot of >>> >>>> complex >>>>>> template >>>>>>> code into your sources, with the cms you just add > code to >> >>>> lib/view.pmand/or >>>>>>> lib/path.pm to keep the document sources simple and > just >> add >>> more >>>>>> template >>>>>>> variables. What I just finished in lib/view.pm and >>> lib/path.pm is >>>>>> directory >>>>>>> listing support for /docs.html but it can be carried > over >> to >>>> similar >>>>>> pages. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> On Monday, March 17, 2014 9:16 PM, Joe Schaefer >>>>>>> <joe_schae...@yahoo.com> wrote: >>>>>>>>> Interesting ideas, thanks. FWIW I think > that >> sort >>>>>>>> of thing might be best supported as a periodic > cron >>>>>>>> if we can write something stable enough, because > >> pulling >>>>>>>> stuff out of the git repo is something the CMS >> isn't >>> going >>>>>>>> to do all that well being an svn application. >>>>>>>> >>>>>>>> The CMS build results are available at >>>>>> http://thrift.staging.apache.org/ >>>>>>>> feel free to play with the sources in >>>> repos/asf/thrift/cms-site/trunk. >>>>>>>> It's very fast, probably an order of > magnitude >> faster >>> than >>>> your current >>>>>>> >>>>>>>> tech. >>>>>>>> Whole site builds in two seconds; fractions of a > >> second >>> for >>>> typical >>>>>> mods to >>>>>>> >>>>>>>> content/ markdown pages. >>>>>>>> >>>>>>>> The CMS bookmarklet is compatible with the > staging >> site >>> FWIW. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> On Monday, March 17, 2014 7:40 PM, Roger > Meier >>>>>>>> <ro...@bufferoverflow.ch> wrote: >>>>>>>>>> Hi Joe & Co >>>>>>>>> >>>>>>>>> thanks for this test run with Apache CMS! >>>>>>>>> >>>>>>>>> What I would love to see is this: >>>>>>>>> >>>>>>>>> Source file: => Web Site URL: >>>>>>>>> test/README.md => thrift.apache.org/test >>>>>>>>> test/STATUS.md => >> thrift.apache.org/test/status >>>>>>>>> test/keys/README.md => >> thrift.apache.org/test/keys >>>>>>>>> lib/<lang>/README.md => >>>> thrift.apache.org/lib/<lang> >>>>>>>>> lib/<lang>/test/README.md => >>>>>>>> thrift.apache.org/lib/<lang>/test >>>>>>>>> lib/<lang>/examples/README.md => >>>>>>>>> thrift.apache.org/lib/<lang>/examples >>>>>>>>> compiler/cpp/README.md => >>> thrift.apache.org/compiler >>>>>>>>> debian/README.md => > thrift.apache.org/debian >>>>>>>>> contrib/<topic>/README.md => >>>> thrift.apache.org/<topic> >>>>>>>>> >>>>>>>>> just made the following patch to rename all > file >> to >>> .md >>>> within our >>>>>>> source >>>>>>>>> tree: >>> https://issues.apache.org/jira/browse/THRIFT-2407 >>>>>>>>> ... we have *41* md's within the source > tree >> and >>>> that's the >>>>>>> place >>>>>>>> where >>>>>>>>> people look for documentation first. => >> simple to >>>> manage >>>>>>>>> Would be user friendly to have all of this >> online >>> with >>>> each release >>>>>>>>> with the URL layout mentioned above. >>>>>>>>> >>>>>>>>> just received the buildboot... >>>>>>>>> Do you have kind of a preview of your > prototype? >>>>>>>>> >>>>>>>>> all the best! >>>>>>>>> -roger >>>>>>>>> ;-r >>>>>>>>> >>>>>>>>> >>>>>>>>> Quoting Jake Farrell >> <jfarr...@apache.org>: >>>>>>>>> >>>>>>>>>> No objections from me >>>>>>>>>> >>>>>>>>>> -Jake >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mon, Mar 17, 2014 at 12:04 PM, Joe >> Schaefer >>>>>>>>> <joe_schae...@yahoo.com>wrote: >>>>>>>>>> >>>>>>>>>>> Any objections to me making a copy > of >> the >>> site >>>>>>>>>>> tree alongside it called > cms-site? >> I'd >>> like >>>> to >>>>>>>>>>> get cracking on knocking up the >> supporting >>> perl >>>>>>>>>>> for this so we can continue to >> brainstorm... >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> On Sunday, March 16, 2014 1:53 > PM, >> Joe >>> Schaefer >>>>>>>>> <joe_schae...@yahoo.com> >>>>>>>>>>> wrote: >>>>>>>>>>>>> On Sunday, March 16, 2014 > 1:40 >> PM, >>> Jake >>>> Farrell >>>>>>>>> <jfarr...@apache.org> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hey Joe >>>>>>>>>>>>> Thanks for reaching out, we > chose >> to >>> go >>>> with a site >>>>>>>> generator >>>>>>>>> over the >>>>>>>>>>>>> Apache CMS due to it > initially >> not >>> meeting >>>> all of >>>>>>> our >>>>>>>> needs. >>>>>>>>> Our current >>>>>>>>>>>>> needs for the site are >>>>>>>>>>>>> >>>>>>>>>>>>> - Markdown to html > conversion >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> That one's easy- its the > default >> for >>> the >>>> cms. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> - Global variables/config > usable >>> throughout >>>> in a >>>>>>> template >>>>>>>> >>>>>>>>> based layout >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Easy too- just create a perl > hash >>> somewhere and >>>> make it >>>>>>>> available >>>>>>>>> to >>>>>>>>>>> your views. >>>>>>>>>>>> Code your views to pass that > hash >> along to >>> your >>>>>>> templates >>>>>>>> and you >>>>>>>>> are >>>>>>>>>>> all set. >>>>>>>>>>>> Note if your hash contains > objects >> you can >>> make >>>> method >>>>>>> calls >>>>>>>> on >>>>>>>>> them in >>>>>>>>>>> your >>>>>>>>>>>> templates. Note: while I >> wouldn't >>>> recommend this in >>>>>>> >>>>>>>> general, >>>>>>>>> for smaller >>>>>>>>>>>> projects like thrift that prefer > >>> convenience >>>> over >>>>>>> separation >>>>>>>> of >>>>>>>>> concerns >>>>>>>>>>> you can >>>>>>>>>>>> have the django template engine > do a >> pass >>> over >>>> your >>>>>>> markdown >>>>>>>>> before >>>>>>>>>>> passing it >>>>>>>>>>>> along to your actual template > page, >> just >>> as is >>>> seemingly >>>>>>> >>>>>>>> common to >>>>>>>>> other >>>>>>>>>>> popular >>>>>>>>>>>> site generation software. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> - Syntax highlighting >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Comes with python's markdown > >> support, >>> but >>>> some >>>>>>> people >>>>>>>> prefer >>>>>>>>> the look of >>>>>>>>>>>> javascript highlighters. >>>>>>>>>>>> >>>>>>>>>>>>> - Easily include code > snippets >> from >>> our >>>> source code >>>>>>> base >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Statically I hope. No idea how > to >> pull >>>> snippets out of >>>>>>> your >>>>>>>> git >>>>>>>>> repo >>>>>>>>>>> via the >>>>>>>>>>>> cms. >>>>>>>>>>>> >>>>>>>>>>>>> - Ability to locally test > before >>> committing >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Belt and suspenders with the > cms- you >> can >>> build >>>> your >>>>>>> changes >>>>>>>>> before >>>>>>>>>>> committing >>>>>>>>>>>> with the build scripts, browse > your >>> changes >>>> before >>>>>>> committing >>>>>>>> via >>>>>>>>> the cms >>>>>>>>>>>> webgui, or simply commit your > changes >> and >>> view >>>> the >>>>>>> staging >>>>>>>> site >>>>>>>>> prior to >>>>>>>>>>> live >>>>>>>>>>>> publication (which is a separate > >> step). >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -Jake >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Sun, Mar 16, 2014 at > 12:44 PM, >> Joe >>>> Schaefer >>>>>>>>>>>> > <joe_schae...@yahoo.com>wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> As it so happens I am >> interested >>> in >>>> working on >>>>>>>> supporting >>>>>>>>>>>>>> thrift's use case as > a >>> potential >>>> user of >>>>>>> the >>>>>>>> Apache >>>>>>>>> CMS. >>>>>>>>>>>>>> While the CMS works well > for >>> massive >>>> projects >>>>>>> there >>>>>>>> are >>>>>>>>>>>>>> things it could do > better at >>> supporting >>>> for >>>>>>> more >>>>>>>> moderate >>>>>>>>>>>>>> sized sites like >> thrift's. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I'd therefore like > to >> engage >>> you >>>> folks in a >>>>>>> >>>>>>>>> brainstorming session >>>>>>>>>>>>>> about what sorts of > features >> you >>> want >>>> for your >>>>>>> site >>>>>>>> and >>>>>>>>> to find >>>>>>>>>>>>>> simple ways of > supporting >> those >>>> features for >>>>>>> you. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks. >>>>>> >>>> >>> >> >
