Modulo some bizarre content truncation out of git-wip-us.apache.org the snippet support is now active. At this point I think all the build support you need in this project to support every type of brainstorming activity we've discussed is now there, so it's just up to you guys to now familiarize yourselves with the build technology and how to apply it to your needs. The site builds with snippet support are running around 10 seconds, which is to be expected with the network interactions involved.
I'm always willing to answer any questions that come up in the interim, but I wish you guys the best of luck in the remaining porting work to be done. All I see really are some links that need to have the trailing slashes removed and whatever <%= %> template constructs that remain need to be ported to django. So it should be smooth sailing, but as I said if you hit any snags holler. I'd really like to see the live site tech switched over to the cms stuff before Apachecon ends in early April, hint hint. > On Wednesday, March 19, 2014 10:41 PM, Joe Schaefer <[email protected]> > wrote: > > While I don't yet know the url magic to support branches and tags in git, > I can say that the snippet support is ready to play with. I suggest just > focusing on the /index.md page first to see how it works in practice because > once it's fully enabled the build will break until all the snippet sources > have been altered to conform with the new style. > > To turn it on for index.md testing, there are a couple of things that need to > happen. First the [snippet:...] marker should be edited to be > [XXXsnippet:...] > to enable testing without breaking the build. Second each marker contains a > "token" argument instead of line numbers for more stability under > edits to the source code. The token argument is an arbitrary string that is > used to denote the boundaries of the snippet within the source code. > ASF::Value::Snippet will pull out the lines from the source file between the > lines that contain the token argument and format them as a code snippet > embedded > into markdown. To get started I'll copy over the code highlighting css from > www.apache.org but note that this is all based on python pygments and there > are > a variety of online css files to choose from if this one is not to the > group's liking. > > > >> On Wednesday, March 19, 2014 4:16 PM, Joe Schaefer > <[email protected]> wrote: >> > Sorry I need to clarify with an example: look at >> http://thrift.staging.apache.org/tutorial/ versus >> http://thrift.staging.apache.org/tutorial.html. The >> former is generated from content/tutorial/index.html >> and the latter from content/tutorial.md. The latter file >> needs to be removed from the tree now that it's content >> has been ported to content/tutorial/index.html. This >> uses django template inheritance to maintain simplicity. >> >> Other directories that follow a similar pattern should >> also be ported in the same fashion. >> >> >> >>> On Wednesday, March 19, 2014 9:46 AM, Joe Schaefer >> <[email protected]> wrote: >>> > If someone's looking for a way to help with the porting >>> effort, consider tackling the creation of an index.html >>> page in each subdir of the content/ tree (other than the >>> base which has an /index.md). The contents of the file >>> should be just >>> >>> {% include 'default' %} >>> >>> (see the docs dir for an example page that can be copied) >>> and the build system will take care of generating the page >>> in a consistent fashion with the rest of the site. Once >>> that is done there are a lot of pages that function like >>> index pages but are written in markdown that can be deleted, >>> like docs.md and its ilk. Nanoc transforms that file into >>> /docs/index.html but that's kinda crufty and not supported >>> by the CMS, which only lets you alter file extensions not paths. >>> >>> >>> >>> >>>> On Wednesday, March 19, 2014 8:54 AM, Joe Schaefer >>> <[email protected]> wrote: >>>> > I just pencilled in snippet preprocessing support into >>>> the thrift builds. Once Jake and I have settled on the >>>> feature set of ASF::Value::Snippet, and we have ported >>>> the snippet calling text within the markdown pages >>>> to the new format, things will just work (knock on wood). >>>> >>>> >>>> >>>> >>>>> On Wednesday, March 19, 2014 6:48 AM, Joseph Schaefer >>>> <[email protected]> wrote: >>>>> > Sure I understand that need. If there's a way of >> pulling the >>> >>>> document >>>>> sources directly from the web I can parse the content to > look for >> >>> snippet >>>>> markers and make those available to the template system. >> That's >>> pretty >>>> much >>>>> how www.apache.org is generated. >>>>> >>>>> Sent from my iPhone >>>>> >>>>> >>>>>> On Mar 19, 2014, at 5:28 AM, Henrique Mendonça >>>> <[email protected]> >>>>> wrote: >>>>>> >>>>>> Hi Joe, >>>>>> >>>>>> Thanks for your effort. I just wanted to add that the > code >>> snippets >>>> are a >>>>>> crucial feature for this project, which has way more > target >>> languages >>>> than >>>>>> active contributors. In our case, reducing manual >> maintenance >>> costs is >>>> much >>>>>> more important than the total site build time. Perhaps > we >> can >>> also >>>> solve >>>>>> this in the CMS, though. >>>>>> >>>>>> - Henrique >>>>>> >>>>>> >>>>>>> On 19 March 2014 03:17, Joe Schaefer >>>> <[email protected]> >>>>> wrote: >>>>>>> >>>>>>> A couple more thoughts regarding the background on > the >> CMS... >>>>>>> the bulk of the focus of the CMS up until now has > been >> to >>>>>>> support really big sites like maven and openoffice, > both >> of >>>>>>> which have websites well over 5GB worth of text > files. >> The >>>>>>> build system was hollowed out and revamped to > support >>> external >>>>>>> builds in other programming languages, as well as >> scaling up >>>>>>> the perl builds to provide about 8 child processes > that >> build >>>>>>> the site in parallel by default. Full site build > times >> for >>>>>>> openoffice went from over 30 minutes to between 5 > to 10 >>> minutes >>>>>>> depending on disk activity, which made > experimentation >> with >>>> site-wide >>>>>>> changes feasible on a reasonable timeframe. At > first >> the >>> project >>>> was >>>>> in >>>>>>> the habit of running the builds locally before > checking >> in >>> their >>>>> changes >>>>>>> because the delay in feedback post-commit was >> unreasonable, >>> but >>>> with >>>>> smart >>>>>>> use of SSI the were able to cut down on the number > of >> changes >>> that >>>> >>>>> required >>>>>>> full site builds and hence no longer bother with > the >> whole >>>> pre-build >>>>> mumbo >>>>>>> jumbo before checking in changes. The CMS is > designed >> to >>> only >>>> build >>>>> pages >>>>>>> that were altered, and if you just edit a page or > two >> only >>> those >>>> pages >>>>> and >>>>>>> the pages that depend on them will be built. >>>>>>> >>>>>>> Now that I've turned my attention to thrift, > there >> are >>> several >>>> >>>>> aspects >>>>>>> that are better off being retuned based on the > modest >> size of >>> the >>>> site. >>>>>>> First of all you have a sitemap url that lists all > of >> your >>>>> markdown-based >>>>>>> pages, which would be a nonstarter for a site like >>> openoffice. >>>> The >>>>>>> implications of the sitemap url are that every page > >> needs to >>> be >>>> built >>>>> in >>>>>>> memory just to generate that page, which leads to a > lot >> of >>>> redundancy >>>>> in a >>>>>>> build system designed to run as forked child > processes. >> We >>> can do >>>> >>>>> better... >>>>>>> >>>>>>> I've done two things to make this work well- > first I >> made >>> the >>>>> number of >>>>>>> child processes "aka runners" tunable on > a >>> per-project >>>> basis, >>>>> and since the >>>>>>> sitemap is going to build every page anyway as the >> slowest >>> page to >>>> >>>>> build, I >>>>>>> memoized the main view that builds the markdown > pages >> for >>> thrift. >>>> The >>>>> best >>>>>>> results are when there's only one runner so we > get >> the >>> full >>>> benefit >>>>> of >>>>>>> memoization: typical site build times are > consistently >> under >>> 2 >>>> seconds >>>>> now. >>>>>>> >>>>>>> You'll appreciate why I am such a performance >> stickler >>> for the >>>> CMS >>>>> when >>>>>>> you start using the bookmarklet, since builds are > the >> only >>>> asynchronous >>>>>>> part of the process from making a change to a page > and >>> publishing >>>> the >>>>>>> results. Ideally when the system is working well > the >> builds >>>> happen >>>>> before >>>>>>> you have time to go looking for them, but you do > need to >> be >>>> mindful of >>>>> not >>>>>>> publishing before the built pages have been > committed >> back to >>> svn >>>> by >>>>>>> buildbot. >>>>>>> >>>>>>> >>>>>>> >>>>>>>> On Monday, March 17, 2014 11:59 PM, Joe > Schaefer >>>>> <[email protected]> >>>>>>> wrote: >>>>>>>>> Ok there's enough of an idea about how > to >> port >>> the >>>> rest of >>>>> the site's >>>>>>>> pages over based on the porting work I've >> already >>> done >>>> that >>>>> it's time >>>>>>>> for me to step back and let you guys catch up. > >> Basically >>> the >>>> idea >>>>> is >>>>>>> that while >>>>>>>> there is templating support in the markdown > sources, >> >>> it's >>>>> fairly limited >>>>>>>> compared to what nanoc offers, but instead of >> writing a >>> lot of >>>> >>>>> complex >>>>>>> template >>>>>>>> code into your sources, with the cms you just > add >> code to >>> >>>>> lib/view.pmand/or >>>>>>>> lib/path.pm to keep the document sources simple > and >> just >>> add >>>> more >>>>>>> template >>>>>>>> variables. What I just finished in lib/view.pm > and >>>> lib/path.pm is >>>>>>> directory >>>>>>>> listing support for /docs.html but it can be > carried >> over >>> to >>>>> similar >>>>>>> pages. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> On Monday, March 17, 2014 9:16 PM, Joe > Schaefer >>>>>>>> <[email protected]> wrote: >>>>>>>>>> Interesting ideas, thanks. FWIW I > think >> that >>> sort >>>>>>>>> of thing might be best supported as a > periodic >> cron >>>>>>>>> if we can write something stable enough, > because >> >>> pulling >>>>>>>>> stuff out of the git repo is something the > CMS >>> isn't >>>> going >>>>>>>>> to do all that well being an svn > application. >>>>>>>>> >>>>>>>>> The CMS build results are available at >>>>>>> http://thrift.staging.apache.org/ >>>>>>>>> feel free to play with the sources in >>>>> repos/asf/thrift/cms-site/trunk. >>>>>>>>> It's very fast, probably an order of >> magnitude >>> faster >>>> than >>>>> your current >>>>>>>> >>>>>>>>> tech. >>>>>>>>> Whole site builds in two seconds; fractions > of a >> >>> second >>>> for >>>>> typical >>>>>>> mods to >>>>>>>> >>>>>>>>> content/ markdown pages. >>>>>>>>> >>>>>>>>> The CMS bookmarklet is compatible with the >> staging >>> site >>>> FWIW. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> On Monday, March 17, 2014 7:40 PM, > Roger >> Meier >>>>>>>>> <[email protected]> wrote: >>>>>>>>>>> Hi Joe & Co >>>>>>>>>> >>>>>>>>>> thanks for this test run with Apache > CMS! >>>>>>>>>> >>>>>>>>>> What I would love to see is this: >>>>>>>>>> >>>>>>>>>> Source file: => Web Site URL: >>>>>>>>>> test/README.md => > thrift.apache.org/test >>>>>>>>>> test/STATUS.md => >>> thrift.apache.org/test/status >>>>>>>>>> test/keys/README.md => >>> thrift.apache.org/test/keys >>>>>>>>>> lib/<lang>/README.md => >>>>> thrift.apache.org/lib/<lang> >>>>>>>>>> lib/<lang>/test/README.md => >>>>>>>>> thrift.apache.org/lib/<lang>/test >>>>>>>>>> lib/<lang>/examples/README.md > => >>>>>>>>>> > thrift.apache.org/lib/<lang>/examples >>>>>>>>>> compiler/cpp/README.md => >>>> thrift.apache.org/compiler >>>>>>>>>> debian/README.md => >> thrift.apache.org/debian >>>>>>>>>> contrib/<topic>/README.md => >>>>> thrift.apache.org/<topic> >>>>>>>>>> >>>>>>>>>> just made the following patch to > rename all >> file >>> to >>>> .md >>>>> within our >>>>>>>> source >>>>>>>>>> tree: >>>> https://issues.apache.org/jira/browse/THRIFT-2407 >>>>>>>>>> ... we have *41* md's within the > source >> tree >>> and >>>>> that's the >>>>>>>> place >>>>>>>>> where >>>>>>>>>> people look for documentation first. > => >>> simple to >>>>> manage >>>>>>>>>> Would be user friendly to have all of > this >>> online >>>> with >>>>> each release >>>>>>>>>> with the URL layout mentioned above. >>>>>>>>>> >>>>>>>>>> just received the buildboot... >>>>>>>>>> Do you have kind of a preview of your >> prototype? >>>>>>>>>> >>>>>>>>>> all the best! >>>>>>>>>> -roger >>>>>>>>>> ;-r >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Quoting Jake Farrell >>> <[email protected]>: >>>>>>>>>> >>>>>>>>>>> No objections from me >>>>>>>>>>> >>>>>>>>>>> -Jake >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Mon, Mar 17, 2014 at 12:04 PM, > Joe >>> Schaefer >>>>>>>>>> <[email protected]>wrote: >>>>>>>>>>> >>>>>>>>>>>> Any objections to me making a > copy >> of >>> the >>>> site >>>>>>>>>>>> tree alongside it called >> cms-site? >>> I'd >>>> like >>>>> to >>>>>>>>>>>> get cracking on knocking up > the >>> supporting >>>> perl >>>>>>>>>>>> for this so we can continue > to >>> brainstorm... >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> On Sunday, March 16, 2014 > 1:53 >> PM, >>> Joe >>>> Schaefer >>>>>>>>>> <[email protected]> >>>>>>>>>>>> wrote: >>>>>>>>>>>>>> On Sunday, March 16, > 2014 >> 1:40 >>> PM, >>>> Jake >>>>> Farrell >>>>>>>>>> <[email protected]> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Hey Joe >>>>>>>>>>>>>> Thanks for reaching > out, we >> chose >>> to >>>> go >>>>> with a site >>>>>>>>> generator >>>>>>>>>> over the >>>>>>>>>>>>>> Apache CMS due to it >> initially >>> not >>>> meeting >>>>> all of >>>>>>>> our >>>>>>>>> needs. >>>>>>>>>> Our current >>>>>>>>>>>>>> needs for the site are >>>>>>>>>>>>>> >>>>>>>>>>>>>> - Markdown to html >> conversion >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> That one's easy- its > the >> default >>> for >>>> the >>>>> cms. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> - Global > variables/config >> usable >>>> throughout >>>>> in a >>>>>>>> template >>>>>>>>> >>>>>>>>>> based layout >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Easy too- just create a > perl >> hash >>>> somewhere and >>>>> make it >>>>>>>>> available >>>>>>>>>> to >>>>>>>>>>>> your views. >>>>>>>>>>>>> Code your views to pass > that >> hash >>> along to >>>> your >>>>>>>> templates >>>>>>>>> and you >>>>>>>>>> are >>>>>>>>>>>> all set. >>>>>>>>>>>>> Note if your hash contains >> objects >>> you can >>>> make >>>>> method >>>>>>>> calls >>>>>>>>> on >>>>>>>>>> them in >>>>>>>>>>>> your >>>>>>>>>>>>> templates. Note: while I >>> wouldn't >>>>> recommend this in >>>>>>>> >>>>>>>>> general, >>>>>>>>>> for smaller >>>>>>>>>>>>> projects like thrift that > prefer >> >>>> convenience >>>>> over >>>>>>>> separation >>>>>>>>> of >>>>>>>>>> concerns >>>>>>>>>>>> you can >>>>>>>>>>>>> have the django template > engine >> do a >>> pass >>>> over >>>>> your >>>>>>>> markdown >>>>>>>>>> before >>>>>>>>>>>> passing it >>>>>>>>>>>>> along to your actual > template >> page, >>> just >>>> as is >>>>> seemingly >>>>>>>> >>>>>>>>> common to >>>>>>>>>> other >>>>>>>>>>>> popular >>>>>>>>>>>>> site generation software. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> - Syntax highlighting >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Comes with python's > markdown >> >>> support, >>>> but >>>>> some >>>>>>>> people >>>>>>>>> prefer >>>>>>>>>> the look of >>>>>>>>>>>>> javascript highlighters. >>>>>>>>>>>>> >>>>>>>>>>>>>> - Easily include code >> snippets >>> from >>>> our >>>>> source code >>>>>>>> base >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Statically I hope. No idea > how >> to >>> pull >>>>> snippets out of >>>>>>>> your >>>>>>>>> git >>>>>>>>>> repo >>>>>>>>>>>> via the >>>>>>>>>>>>> cms. >>>>>>>>>>>>> >>>>>>>>>>>>>> - Ability to locally > test >> before >>>> committing >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Belt and suspenders with > the >> cms- you >>> can >>>> build >>>>> your >>>>>>>> changes >>>>>>>>>> before >>>>>>>>>>>> committing >>>>>>>>>>>>> with the build scripts, > browse >> your >>>> changes >>>>> before >>>>>>>> committing >>>>>>>>> via >>>>>>>>>> the cms >>>>>>>>>>>>> webgui, or simply commit > your >> changes >>> and >>>> view >>>>> the >>>>>>>> staging >>>>>>>>> site >>>>>>>>>> prior to >>>>>>>>>>>> live >>>>>>>>>>>>> publication (which is a > separate >> >>> step). >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -Jake >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Sun, Mar 16, 2014 at > >> 12:44 PM, >>> Joe >>>>> Schaefer >>>>>>>>>>>>> >> <[email protected]>wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> As it so happens I > am >>> interested >>>> in >>>>> working on >>>>>>>>> supporting >>>>>>>>>>>>>>> thrift's use > case as >> a >>>> potential >>>>> user of >>>>>>>> the >>>>>>>>> Apache >>>>>>>>>> CMS. >>>>>>>>>>>>>>> While the CMS works > well >> for >>>> massive >>>>> projects >>>>>>>> there >>>>>>>>> are >>>>>>>>>>>>>>> things it could do >> better at >>>> supporting >>>>> for >>>>>>>> more >>>>>>>>> moderate >>>>>>>>>>>>>>> sized sites like >>> thrift's. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I'd therefore > like >> to >>> engage >>>> you >>>>> folks in a >>>>>>>> >>>>>>>>>> brainstorming session >>>>>>>>>>>>>>> about what sorts of > >> features >>> you >>>> want >>>>> for your >>>>>>>> site >>>>>>>>> and >>>>>>>>>> to find >>>>>>>>>>>>>>> simple ways of >> supporting >>> those >>>>> features for >>>>>>>> you. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks. >>>>>>> >>>>> >>>> >>> >> >
