On Thursday, March 20, 2014 11:19 AM, Joe Schaefer <joe_schae...@yahoo.com>
wrote:
Modulo some bizarre content truncation out of git-wip-us.apache.org
the snippet support is now active. At this point I think all the build
support you need in this project to support every type of brainstorming
activity we've discussed is now there, so it's just up to you guys to
now familiarize yourselves with the build technology and how to apply
it to your needs. The site builds with snippet support are running around
10 seconds, which is to be expected with the network interactions involved.
I'm always willing to answer any questions that come up in the interim,
but I wish you guys the best of luck in the remaining porting work to be done.
All I see really are some links that need to have the trailing slashes removed
and whatever <%= %> template constructs that remain need to be ported to
django. So it should be smooth sailing, but as I said if you hit any snags
holler. I'd really like to see the live site tech switched over to the cms
stuff before Apachecon ends in early April, hint hint.
On Wednesday, March 19, 2014 10:41 PM, Joe Schaefer
<joe_schae...@yahoo.com> wrote:
> While I don't yet know the url magic to support branches and tags
in git,
I can say that the snippet support is ready to play with. I suggest just
focusing on the /index.md page first to see how it works in practice
because
once it's fully enabled the build will break until all the snippet
sources
have been altered to conform with the new style.
To turn it on for index.md testing, there are a couple of things that need
to
happen. First the [snippet:...] marker should be edited to be
[XXXsnippet:...]
to enable testing without breaking the build. Second each marker contains
a
"token" argument instead of line numbers for more stability under
edits to the source code. The token argument is an arbitrary string that
is
used to denote the boundaries of the snippet within the source code.
ASF::Value::Snippet will pull out the lines from the source file between
the
lines that contain the token argument and format them as a code snippet
embedded
into markdown. To get started I'll copy over the code highlighting css
from
www.apache.org but note that this is all based on python pygments and there
are
a variety of online css files to choose from if this one is not to the
group's liking.
On Wednesday, March 19, 2014 4:16 PM, Joe Schaefer
<joe_schae...@yahoo.com> wrote:
> Sorry I need to clarify with an example: look at
http://thrift.staging.apache.org/tutorial/ versus
http://thrift.staging.apache.org/tutorial.html. The
former is generated from content/tutorial/index.html
and the latter from content/tutorial.md. The latter file
needs to be removed from the tree now that it's content
has been ported to content/tutorial/index.html. This
uses django template inheritance to maintain simplicity.
Other directories that follow a similar pattern should
also be ported in the same fashion.
On Wednesday, March 19, 2014 9:46 AM, Joe Schaefer
<joe_schae...@yahoo.com> wrote:
> If someone's looking for a way to help with the porting
effort, consider tackling the creation of an index.html
page in each subdir of the content/ tree (other than the
base which has an /index.md). The contents of the file
should be just
{% include 'default' %}
(see the docs dir for an example page that can be copied)
and the build system will take care of generating the page
in a consistent fashion with the rest of the site. Once
that is done there are a lot of pages that function like
index pages but are written in markdown that can be deleted,
like docs.md and its ilk. Nanoc transforms that file into
/docs/index.html but that's kinda crufty and not supported
by the CMS, which only lets you alter file extensions not paths.
On Wednesday, March 19, 2014 8:54 AM, Joe Schaefer
<joe_schae...@yahoo.com> wrote:
> I just pencilled in snippet preprocessing support into
the thrift builds. Once Jake and I have settled on the
feature set of ASF::Value::Snippet, and we have ported
the snippet calling text within the markdown pages
to the new format, things will just work (knock on wood).
On Wednesday, March 19, 2014 6:48 AM, Joseph Schaefer
<joe_schae...@yahoo.com> wrote:
> Sure I understand that need. If there's a way
of
pulling the
document
sources directly from the web I can parse the content
to
look for
snippet
markers and make those available to the template
system.
That's
pretty
much
how www.apache.org is generated.
Sent from my iPhone
On Mar 19, 2014, at 5:28 AM, Henrique Mendonça
<henri...@apache.org>
wrote:
Hi Joe,
Thanks for your effort. I just wanted to add that
the
code
snippets
are a
crucial feature for this project, which has way
more
target
languages
than
active contributors. In our case, reducing manual
maintenance
costs is
much
more important than the total site build time.
Perhaps
we
can
also
solve
this in the CMS, though.
- Henrique
On 19 March 2014 03:17, Joe Schaefer
<joe_schae...@yahoo.com>
wrote:
A couple more thoughts regarding the
background on
the
CMS...
the bulk of the focus of the CMS up until now
has
been
to
support really big sites like maven and
openoffice,
both
of
which have websites well over 5GB worth of
text
files.
The
build system was hollowed out and revamped to
support
external
builds in other programming languages, as well
as
scaling up
the perl builds to provide about 8 child
processes
that
build
the site in parallel by default. Full site
build
times
for
openoffice went from over 30 minutes to
between 5
to 10
minutes
depending on disk activity, which made
experimentation
with
site-wide
changes feasible on a reasonable timeframe.
At
first
the
project
was
in
the habit of running the builds locally before
checking
in
their
changes
because the delay in feedback post-commit was
unreasonable,
but
with
smart
use of SSI the were able to cut down on the
number
of
changes
that
required
full site builds and hence no longer bother
with
the
whole
pre-build
mumbo
jumbo before checking in changes. The CMS is
designed
to
only
build
pages
that were altered, and if you just edit a page
or
two
only
those
pages
and
the pages that depend on them will be built.
Now that I've turned my attention to
thrift,
there
are
several
aspects
that are better off being retuned based on the
modest
size of
the
site.
First of all you have a sitemap url that lists
all
of
your
markdown-based
pages, which would be a nonstarter for a site
like
openoffice.
The
implications of the sitemap url are that every
page
needs to
be
built
in
memory just to generate that page, which leads
to a
lot
of
redundancy
in a
build system designed to run as forked child
processes.
We
can do
better...
I've done two things to make this work
well-
first I
made
the
number of
child processes "aka runners"
tunable on
a
per-project
basis,
and since the
sitemap is going to build every page anyway as
the
slowest
page to
build, I
memoized the main view that builds the
markdown
pages
for
thrift.
The
best
results are when there's only one runner
so we
get
the
full
benefit
of
memoization: typical site build times are
consistently
under
2
seconds
now.
You'll appreciate why I am such a
performance
stickler
for the
CMS
when
you start using the bookmarklet, since builds
are
the
only
asynchronous
part of the process from making a change to a
page
and
publishing
the
results. Ideally when the system is working
well
the
builds
happen
before
you have time to go looking for them, but you
do
need to
be
mindful of
not
publishing before the built pages have been
committed
back to
svn
by
buildbot.
On Monday, March 17, 2014 11:59 PM, Joe
Schaefer
<joe_schae...@yahoo.com>
wrote:
Ok there's enough of an idea about
how
to
port
the
rest of
the site's
pages over based on the porting work
I've
already
done
that
it's time
for me to step back and let you guys catch
up.
Basically
the
idea
is
that while
there is templating support in the
markdown
sources,
it's
fairly limited
compared to what nanoc offers, but instead
of
writing a
lot of
complex
template
code into your sources, with the cms you
just
add
code to
lib/view.pmand/or
lib/path.pm to keep the document sources
simple
and
just
add
more
template
variables. What I just finished in
lib/view.pm
and
lib/path.pm is
directory
listing support for /docs.html but it can
be
carried
over
to
similar
pages.
On Monday, March 17, 2014 9:16 PM, Joe
Schaefer
<joe_schae...@yahoo.com> wrote:
Interesting ideas, thanks. FWIW I
think
that
sort
of thing might be best supported as a
periodic
cron
if we can write something stable
enough,
because
pulling
stuff out of the git repo is something
the
CMS
isn't
going
to do all that well being an svn
application.
The CMS build results are available at
http://thrift.staging.apache.org/
feel free to play with the sources in
repos/asf/thrift/cms-site/trunk.
It's very fast, probably an order
of
magnitude
faster
than
your current
tech.
Whole site builds in two seconds;
fractions
of a
second
for
typical
mods to
content/ markdown pages.
The CMS bookmarklet is compatible with
the
staging
site
FWIW.
On Monday, March 17, 2014 7:40
PM,
Roger
Meier
<ro...@bufferoverflow.ch> wrote:
Hi Joe & Co
thanks for this test run with
Apache
CMS!
What I would love to see is this:
Source file: => Web Site URL:
test/README.md =>
thrift.apache.org/test
test/STATUS.md =>
thrift.apache.org/test/status
test/keys/README.md =>
thrift.apache.org/test/keys
lib/<lang>/README.md =>
thrift.apache.org/lib/<lang>
lib/<lang>/test/README.md
=>
thrift.apache.org/lib/<lang>/test
lib/<lang>/examples/README.md
=>
thrift.apache.org/lib/<lang>/examples
compiler/cpp/README.md =>
thrift.apache.org/compiler
debian/README.md =>
thrift.apache.org/debian
contrib/<topic>/README.md
=>
thrift.apache.org/<topic>
just made the following patch to
rename all
file
to
.md
within our
source
tree:
https://issues.apache.org/jira/browse/THRIFT-2407
... we have *41* md's within
the
source
tree
and
that's the
place
where
people look for documentation
first.
=>
simple to
manage
Would be user friendly to have
all of
this
online
with
each release
with the URL layout mentioned
above.
just received the buildboot...
Do you have kind of a preview of
your
prototype?
all the best!
-roger
;-r
Quoting Jake Farrell
<jfarr...@apache.org>:
No objections from me
-Jake
On Mon, Mar 17, 2014 at
12:04 PM,
Joe
Schaefer
<joe_schae...@yahoo.com>wrote:
Any objections to me
making a
copy
of
the
site
tree alongside it called
cms-site?
I'd
like
to
get cracking on knocking
up
the
supporting
perl
for this so we can
continue
to
brainstorm...
On Sunday, March 16,
2014
1:53
PM,
Joe
Schaefer
<joe_schae...@yahoo.com>
wrote:
On Sunday, March
16,
2014
1:40
PM,
Jake
Farrell
<jfarr...@apache.org>
wrote:
Hey Joe
Thanks for
reaching
out, we
chose
to
go
with a site
generator
over the
Apache CMS due to
it
initially
not
meeting
all of
our
needs.
Our current
needs for the site
are
- Markdown to html
conversion
That one's easy-
its
the
default
for
the
cms.
- Global
variables/config
usable
throughout
in a
template
based layout
Easy too- just create
a
perl
hash
somewhere and
make it
available
to
your views.
Code your views to
pass
that
hash
along to
your
templates
and you
are
all set.
Note if your hash
contains
objects
you can
make
method
calls
on
them in
your
templates. Note:
while I
wouldn't
recommend this in
general,
for smaller
projects like thrift
that
prefer
convenience
over
separation
of
concerns
you can
have the django
template
engine
do a
pass
over
your
markdown
before
passing it
along to your actual
template
page,
just
as is
seemingly
common to
other
popular
site generation
software.
- Syntax
highlighting
Comes with
python's
markdown
support,
but
some
people
prefer
the look of
javascript
highlighters.
- Easily include
code
snippets
from
our
source code
base
Statically I hope. No
idea
how
to
pull
snippets out of
your
git
repo
via the
cms.
- Ability to
locally
test
before
committing
Belt and suspenders
with
the
cms- you
can
build
your
changes
before
committing
with the build
scripts,
browse
your
changes
before
committing
via
the cms
webgui, or simply
commit
your
changes
and
view
the
staging
site
prior to
live
publication (which is
a
separate
step).
-Jake
On Sun, Mar 16,
2014 at
12:44 PM,
Joe
Schaefer
<joe_schae...@yahoo.com>wrote:
As it so
happens I
am
interested
in
working on
supporting
thrift's
use
case as
a
potential
user of
the
Apache
CMS.
While the CMS
works
well
for
massive
projects
there
are
things it
could do
better at
supporting
for
more
moderate
sized sites
like
thrift's.
I'd
therefore
like
to
engage
you
folks in a
brainstorming session
about what
sorts of
features
you
want
for your
site
and
to find
simple ways of
supporting
those
features for
you.
Thanks.
