Ok I've now done the basic necessities involved in porting the site over. Links have been adjusted, templates ported, files created, snippet feature activated. At this point I'm happy to declare victory and migrate the thrift live site over, as anything broken that still remains can be dealt with on a per-page basis with the bookmarklet.
> On Thursday, March 20, 2014 11:19 AM, Joe Schaefer <joe_schae...@yahoo.com> > wrote: > > Modulo some bizarre content truncation out of git-wip-us.apache.org > the snippet support is now active. At this point I think all the build > support you need in this project to support every type of brainstorming > activity we've discussed is now there, so it's just up to you guys to > now familiarize yourselves with the build technology and how to apply > it to your needs. The site builds with snippet support are running around > 10 seconds, which is to be expected with the network interactions involved. > > I'm always willing to answer any questions that come up in the interim, > but I wish you guys the best of luck in the remaining porting work to be done. > All I see really are some links that need to have the trailing slashes removed > and whatever <%= %> template constructs that remain need to be ported to > django. So it should be smooth sailing, but as I said if you hit any snags > holler. I'd really like to see the live site tech switched over to the cms > stuff before Apachecon ends in early April, hint hint. > > > > > >> On Wednesday, March 19, 2014 10:41 PM, Joe Schaefer > <joe_schae...@yahoo.com> wrote: >> > While I don't yet know the url magic to support branches and tags > in git, >> I can say that the snippet support is ready to play with. I suggest just >> focusing on the /index.md page first to see how it works in practice > because >> once it's fully enabled the build will break until all the snippet > sources >> have been altered to conform with the new style. >> >> To turn it on for index.md testing, there are a couple of things that need > to >> happen. First the [snippet:...] marker should be edited to be > [XXXsnippet:...] >> to enable testing without breaking the build. Second each marker contains > a >> "token" argument instead of line numbers for more stability under > >> edits to the source code. The token argument is an arbitrary string that > is >> used to denote the boundaries of the snippet within the source code. >> ASF::Value::Snippet will pull out the lines from the source file between > the >> lines that contain the token argument and format them as a code snippet > embedded >> into markdown. To get started I'll copy over the code highlighting css > from >> www.apache.org but note that this is all based on python pygments and there > are >> a variety of online css files to choose from if this one is not to the >> group's liking. >> >> >> >>> On Wednesday, March 19, 2014 4:16 PM, Joe Schaefer >> <joe_schae...@yahoo.com> wrote: >>> > Sorry I need to clarify with an example: look at >>> http://thrift.staging.apache.org/tutorial/ versus >>> http://thrift.staging.apache.org/tutorial.html. The >>> former is generated from content/tutorial/index.html >>> and the latter from content/tutorial.md. The latter file >>> needs to be removed from the tree now that it's content >>> has been ported to content/tutorial/index.html. This >>> uses django template inheritance to maintain simplicity. >>> >>> Other directories that follow a similar pattern should >>> also be ported in the same fashion. >>> >>> >>> >>>> On Wednesday, March 19, 2014 9:46 AM, Joe Schaefer >>> <joe_schae...@yahoo.com> wrote: >>>> > If someone's looking for a way to help with the porting >>>> effort, consider tackling the creation of an index.html >>>> page in each subdir of the content/ tree (other than the >>>> base which has an /index.md). The contents of the file >>>> should be just >>>> >>>> {% include 'default' %} >>>> >>>> (see the docs dir for an example page that can be copied) >>>> and the build system will take care of generating the page >>>> in a consistent fashion with the rest of the site. Once >>>> that is done there are a lot of pages that function like >>>> index pages but are written in markdown that can be deleted, >>>> like docs.md and its ilk. Nanoc transforms that file into >>>> /docs/index.html but that's kinda crufty and not supported >>>> by the CMS, which only lets you alter file extensions not paths. >>>> >>>> >>>> >>>> >>>>> On Wednesday, March 19, 2014 8:54 AM, Joe Schaefer >>>> <joe_schae...@yahoo.com> wrote: >>>>> > I just pencilled in snippet preprocessing support into >>>>> the thrift builds. Once Jake and I have settled on the >>>>> feature set of ASF::Value::Snippet, and we have ported >>>>> the snippet calling text within the markdown pages >>>>> to the new format, things will just work (knock on wood). >>>>> >>>>> >>>>> >>>>> >>>>>> On Wednesday, March 19, 2014 6:48 AM, Joseph Schaefer >>>>> <joe_schae...@yahoo.com> wrote: >>>>>> > Sure I understand that need. If there's a way > of >>> pulling the >>>> >>>>> document >>>>>> sources directly from the web I can parse the content > to >> look for >>> >>>> snippet >>>>>> markers and make those available to the template > system. >>> That's >>>> pretty >>>>> much >>>>>> how www.apache.org is generated. >>>>>> >>>>>> Sent from my iPhone >>>>>> >>>>>> >>>>>>> On Mar 19, 2014, at 5:28 AM, Henrique Mendonça >>>>> <henri...@apache.org> >>>>>> wrote: >>>>>>> >>>>>>> Hi Joe, >>>>>>> >>>>>>> Thanks for your effort. I just wanted to add that > the >> code >>>> snippets >>>>> are a >>>>>>> crucial feature for this project, which has way > more >> target >>>> languages >>>>> than >>>>>>> active contributors. In our case, reducing manual >>> maintenance >>>> costs is >>>>> much >>>>>>> more important than the total site build time. > Perhaps >> we >>> can >>>> also >>>>> solve >>>>>>> this in the CMS, though. >>>>>>> >>>>>>> - Henrique >>>>>>> >>>>>>> >>>>>>>> On 19 March 2014 03:17, Joe Schaefer >>>>> <joe_schae...@yahoo.com> >>>>>> wrote: >>>>>>>> >>>>>>>> A couple more thoughts regarding the > background on >> the >>> CMS... >>>>>>>> the bulk of the focus of the CMS up until now > has >> been >>> to >>>>>>>> support really big sites like maven and > openoffice, >> both >>> of >>>>>>>> which have websites well over 5GB worth of > text >> files. >>> The >>>>>>>> build system was hollowed out and revamped to >> support >>>> external >>>>>>>> builds in other programming languages, as well > as >>> scaling up >>>>>>>> the perl builds to provide about 8 child > processes >> that >>> build >>>>>>>> the site in parallel by default. Full site > build >> times >>> for >>>>>>>> openoffice went from over 30 minutes to > between 5 >> to 10 >>>> minutes >>>>>>>> depending on disk activity, which made >> experimentation >>> with >>>>> site-wide >>>>>>>> changes feasible on a reasonable timeframe. > At >> first >>> the >>>> project >>>>> was >>>>>> in >>>>>>>> the habit of running the builds locally before > >> checking >>> in >>>> their >>>>>> changes >>>>>>>> because the delay in feedback post-commit was >>> unreasonable, >>>> but >>>>> with >>>>>> smart >>>>>>>> use of SSI the were able to cut down on the > number >> of >>> changes >>>> that >>>>> >>>>>> required >>>>>>>> full site builds and hence no longer bother > with >> the >>> whole >>>>> pre-build >>>>>> mumbo >>>>>>>> jumbo before checking in changes. The CMS is >> designed >>> to >>>> only >>>>> build >>>>>> pages >>>>>>>> that were altered, and if you just edit a page > or >> two >>> only >>>> those >>>>> pages >>>>>> and >>>>>>>> the pages that depend on them will be built. >>>>>>>> >>>>>>>> Now that I've turned my attention to > thrift, >> there >>> are >>>> several >>>>> >>>>>> aspects >>>>>>>> that are better off being retuned based on the > >> modest >>> size of >>>> the >>>>> site. >>>>>>>> First of all you have a sitemap url that lists > all >> of >>> your >>>>>> markdown-based >>>>>>>> pages, which would be a nonstarter for a site > like >>>> openoffice. >>>>> The >>>>>>>> implications of the sitemap url are that every > page >> >>> needs to >>>> be >>>>> built >>>>>> in >>>>>>>> memory just to generate that page, which leads > to a >> lot >>> of >>>>> redundancy >>>>>> in a >>>>>>>> build system designed to run as forked child >> processes. >>> We >>>> can do >>>>> >>>>>> better... >>>>>>>> >>>>>>>> I've done two things to make this work > well- >> first I >>> made >>>> the >>>>>> number of >>>>>>>> child processes "aka runners" > tunable on >> a >>>> per-project >>>>> basis, >>>>>> and since the >>>>>>>> sitemap is going to build every page anyway as > the >>> slowest >>>> page to >>>>> >>>>>> build, I >>>>>>>> memoized the main view that builds the > markdown >> pages >>> for >>>> thrift. >>>>> The >>>>>> best >>>>>>>> results are when there's only one runner > so we >> get >>> the >>>> full >>>>> benefit >>>>>> of >>>>>>>> memoization: typical site build times are >> consistently >>> under >>>> 2 >>>>> seconds >>>>>> now. >>>>>>>> >>>>>>>> You'll appreciate why I am such a > performance >>> stickler >>>> for the >>>>> CMS >>>>>> when >>>>>>>> you start using the bookmarklet, since builds > are >> the >>> only >>>>> asynchronous >>>>>>>> part of the process from making a change to a > page >> and >>>> publishing >>>>> the >>>>>>>> results. Ideally when the system is working > well >> the >>> builds >>>>> happen >>>>>> before >>>>>>>> you have time to go looking for them, but you > do >> need to >>> be >>>>> mindful of >>>>>> not >>>>>>>> publishing before the built pages have been >> committed >>> back to >>>> svn >>>>> by >>>>>>>> buildbot. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> On Monday, March 17, 2014 11:59 PM, Joe >> Schaefer >>>>>> <joe_schae...@yahoo.com> >>>>>>>> wrote: >>>>>>>>>> Ok there's enough of an idea about > how >> to >>> port >>>> the >>>>> rest of >>>>>> the site's >>>>>>>>> pages over based on the porting work > I've >>> already >>>> done >>>>> that >>>>>> it's time >>>>>>>>> for me to step back and let you guys catch > up. >> >>> Basically >>>> the >>>>> idea >>>>>> is >>>>>>>> that while >>>>>>>>> there is templating support in the > markdown >> sources, >>> >>>> it's >>>>>> fairly limited >>>>>>>>> compared to what nanoc offers, but instead > of >>> writing a >>>> lot of >>>>> >>>>>> complex >>>>>>>> template >>>>>>>>> code into your sources, with the cms you > just >> add >>> code to >>>> >>>>>> lib/view.pmand/or >>>>>>>>> lib/path.pm to keep the document sources > simple >> and >>> just >>>> add >>>>> more >>>>>>>> template >>>>>>>>> variables. What I just finished in > lib/view.pm >> and >>>>> lib/path.pm is >>>>>>>> directory >>>>>>>>> listing support for /docs.html but it can > be >> carried >>> over >>>> to >>>>>> similar >>>>>>>> pages. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> On Monday, March 17, 2014 9:16 PM, Joe > >> Schaefer >>>>>>>>> <joe_schae...@yahoo.com> wrote: >>>>>>>>>>> Interesting ideas, thanks. FWIW I > >> think >>> that >>>> sort >>>>>>>>>> of thing might be best supported as a >> periodic >>> cron >>>>>>>>>> if we can write something stable > enough, >> because >>> >>>> pulling >>>>>>>>>> stuff out of the git repo is something > the >> CMS >>>> isn't >>>>> going >>>>>>>>>> to do all that well being an svn >> application. >>>>>>>>>> >>>>>>>>>> The CMS build results are available at >>>>>>>> http://thrift.staging.apache.org/ >>>>>>>>>> feel free to play with the sources in >>>>>> repos/asf/thrift/cms-site/trunk. >>>>>>>>>> It's very fast, probably an order > of >>> magnitude >>>> faster >>>>> than >>>>>> your current >>>>>>>>> >>>>>>>>>> tech. >>>>>>>>>> Whole site builds in two seconds; > fractions >> of a >>> >>>> second >>>>> for >>>>>> typical >>>>>>>> mods to >>>>>>>>> >>>>>>>>>> content/ markdown pages. >>>>>>>>>> >>>>>>>>>> The CMS bookmarklet is compatible with > the >>> staging >>>> site >>>>> FWIW. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> On Monday, March 17, 2014 7:40 > PM, >> Roger >>> Meier >>>>>>>>>> <ro...@bufferoverflow.ch> wrote: >>>>>>>>>>>> Hi Joe & Co >>>>>>>>>>> >>>>>>>>>>> thanks for this test run with > Apache >> CMS! >>>>>>>>>>> >>>>>>>>>>> What I would love to see is this: >>>>>>>>>>> >>>>>>>>>>> Source file: => Web Site URL: >>>>>>>>>>> test/README.md => >> thrift.apache.org/test >>>>>>>>>>> test/STATUS.md => >>>> thrift.apache.org/test/status >>>>>>>>>>> test/keys/README.md => >>>> thrift.apache.org/test/keys >>>>>>>>>>> lib/<lang>/README.md => >>>>>> thrift.apache.org/lib/<lang> >>>>>>>>>>> lib/<lang>/test/README.md > => >>>>>>>>>> > thrift.apache.org/lib/<lang>/test >>>>>>>>>>> > lib/<lang>/examples/README.md >> => >>>>>>>>>>> >> thrift.apache.org/lib/<lang>/examples >>>>>>>>>>> compiler/cpp/README.md => >>>>> thrift.apache.org/compiler >>>>>>>>>>> debian/README.md => >>> thrift.apache.org/debian >>>>>>>>>>> contrib/<topic>/README.md > => >>>>>> thrift.apache.org/<topic> >>>>>>>>>>> >>>>>>>>>>> just made the following patch to >> rename all >>> file >>>> to >>>>> .md >>>>>> within our >>>>>>>>> source >>>>>>>>>>> tree: >>>>> https://issues.apache.org/jira/browse/THRIFT-2407 >>>>>>>>>>> ... we have *41* md's within > the >> source >>> tree >>>> and >>>>>> that's the >>>>>>>>> place >>>>>>>>>> where >>>>>>>>>>> people look for documentation > first. >> => >>>> simple to >>>>>> manage >>>>>>>>>>> Would be user friendly to have > all of >> this >>>> online >>>>> with >>>>>> each release >>>>>>>>>>> with the URL layout mentioned > above. >>>>>>>>>>> >>>>>>>>>>> just received the buildboot... >>>>>>>>>>> Do you have kind of a preview of > your >>> prototype? >>>>>>>>>>> >>>>>>>>>>> all the best! >>>>>>>>>>> -roger >>>>>>>>>>> ;-r >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Quoting Jake Farrell >>>> <jfarr...@apache.org>: >>>>>>>>>>> >>>>>>>>>>>> No objections from me >>>>>>>>>>>> >>>>>>>>>>>> -Jake >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Mar 17, 2014 at > 12:04 PM, >> Joe >>>> Schaefer >>>>>>>>>>> > <joe_schae...@yahoo.com>wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Any objections to me > making a >> copy >>> of >>>> the >>>>> site >>>>>>>>>>>>> tree alongside it called > >>> cms-site? >>>> I'd >>>>> like >>>>>> to >>>>>>>>>>>>> get cracking on knocking > up >> the >>>> supporting >>>>> perl >>>>>>>>>>>>> for this so we can > continue >> to >>>> brainstorm... >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> On Sunday, March 16, > 2014 >> 1:53 >>> PM, >>>> Joe >>>>> Schaefer >>>>>>>>>>> <joe_schae...@yahoo.com> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> On Sunday, March > 16, >> 2014 >>> 1:40 >>>> PM, >>>>> Jake >>>>>> Farrell >>>>>>>>>>> <jfarr...@apache.org> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hey Joe >>>>>>>>>>>>>>> Thanks for > reaching >> out, we >>> chose >>>> to >>>>> go >>>>>> with a site >>>>>>>>>> generator >>>>>>>>>>> over the >>>>>>>>>>>>>>> Apache CMS due to > it >>> initially >>>> not >>>>> meeting >>>>>> all of >>>>>>>>> our >>>>>>>>>> needs. >>>>>>>>>>> Our current >>>>>>>>>>>>>>> needs for the site > are >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Markdown to html > >>> conversion >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> That one's easy- > its >> the >>> default >>>> for >>>>> the >>>>>> cms. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Global >> variables/config >>> usable >>>>> throughout >>>>>> in a >>>>>>>>> template >>>>>>>>>> >>>>>>>>>>> based layout >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Easy too- just create > a >> perl >>> hash >>>>> somewhere and >>>>>> make it >>>>>>>>>> available >>>>>>>>>>> to >>>>>>>>>>>>> your views. >>>>>>>>>>>>>> Code your views to > pass >> that >>> hash >>>> along to >>>>> your >>>>>>>>> templates >>>>>>>>>> and you >>>>>>>>>>> are >>>>>>>>>>>>> all set. >>>>>>>>>>>>>> Note if your hash > contains >>> objects >>>> you can >>>>> make >>>>>> method >>>>>>>>> calls >>>>>>>>>> on >>>>>>>>>>> them in >>>>>>>>>>>>> your >>>>>>>>>>>>>> templates. Note: > while I >>>> wouldn't >>>>>> recommend this in >>>>>>>>> >>>>>>>>>> general, >>>>>>>>>>> for smaller >>>>>>>>>>>>>> projects like thrift > that >> prefer >>> >>>>> convenience >>>>>> over >>>>>>>>> separation >>>>>>>>>> of >>>>>>>>>>> concerns >>>>>>>>>>>>> you can >>>>>>>>>>>>>> have the django > template >> engine >>> do a >>>> pass >>>>> over >>>>>> your >>>>>>>>> markdown >>>>>>>>>>> before >>>>>>>>>>>>> passing it >>>>>>>>>>>>>> along to your actual >> template >>> page, >>>> just >>>>> as is >>>>>> seemingly >>>>>>>>> >>>>>>>>>> common to >>>>>>>>>>> other >>>>>>>>>>>>> popular >>>>>>>>>>>>>> site generation > software. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Syntax > highlighting >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Comes with > python's >> markdown >>> >>>> support, >>>>> but >>>>>> some >>>>>>>>> people >>>>>>>>>> prefer >>>>>>>>>>> the look of >>>>>>>>>>>>>> javascript > highlighters. >>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Easily include > code >>> snippets >>>> from >>>>> our >>>>>> source code >>>>>>>>> base >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Statically I hope. No > idea >> how >>> to >>>> pull >>>>>> snippets out of >>>>>>>>> your >>>>>>>>>> git >>>>>>>>>>> repo >>>>>>>>>>>>> via the >>>>>>>>>>>>>> cms. >>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Ability to > locally >> test >>> before >>>>> committing >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Belt and suspenders > with >> the >>> cms- you >>>> can >>>>> build >>>>>> your >>>>>>>>> changes >>>>>>>>>>> before >>>>>>>>>>>>> committing >>>>>>>>>>>>>> with the build > scripts, >> browse >>> your >>>>> changes >>>>>> before >>>>>>>>> committing >>>>>>>>>> via >>>>>>>>>>> the cms >>>>>>>>>>>>>> webgui, or simply > commit >> your >>> changes >>>> and >>>>> view >>>>>> the >>>>>>>>> staging >>>>>>>>>> site >>>>>>>>>>> prior to >>>>>>>>>>>>> live >>>>>>>>>>>>>> publication (which is > a >> separate >>> >>>> step). >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -Jake >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Sun, Mar 16, > 2014 at >> >>> 12:44 PM, >>>> Joe >>>>>> Schaefer >>>>>>>>>>>>>> >>> <joe_schae...@yahoo.com>wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> As it so > happens I >> am >>>> interested >>>>> in >>>>>> working on >>>>>>>>>> supporting >>>>>>>>>>>>>>>> thrift's > use >> case as >>> a >>>>> potential >>>>>> user of >>>>>>>>> the >>>>>>>>>> Apache >>>>>>>>>>> CMS. >>>>>>>>>>>>>>>> While the CMS > works >> well >>> for >>>>> massive >>>>>> projects >>>>>>>>> there >>>>>>>>>> are >>>>>>>>>>>>>>>> things it > could do >>> better at >>>>> supporting >>>>>> for >>>>>>>>> more >>>>>>>>>> moderate >>>>>>>>>>>>>>>> sized sites > like >>>> thrift's. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I'd > therefore >> like >>> to >>>> engage >>>>> you >>>>>> folks in a >>>>>>>>> >>>>>>>>>>> brainstorming session >>>>>>>>>>>>>>>> about what > sorts of >> >>> features >>>> you >>>>> want >>>>>> for your >>>>>>>>> site >>>>>>>>>> and >>>>>>>>>>> to find >>>>>>>>>>>>>>>> simple ways of > >>> supporting >>>> those >>>>>> features for >>>>>>>>> you. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks. >>>>>>>> >>>>>> >>>>> >>>> >>> >> >
