Thanks, Jon! I’m working on characterizing exactly how to fix the two main
issues. I think I’ve got a script that will auto-fix the triple-backtick
problem. The bullet list problem will require hand-editing, so I want to make
sure I’ve got the right recommendation.
The larger issue is going thru and making the doc content better and more
usable.
But that can occur over time, and will be motivated by having the book there to
gripe about :-)
On 1/19/17, 9:05 AM, "zeo...@gmail.com" <zeo...@gmail.com> wrote:
Looking at the screenshot, that would be an incredible improvement on what
we currently have. I'd be happy to help out with any markdown
modifications and documentation cleanup, if necessary, to fix the problems
you outlined above.
Jon
On Thu, Jan 19, 2017 at 11:22 AM Matt Foley <ma...@apache.org> wrote:
> Sorry, I forgot text-only messages won’t accept attachments. Please see
>
>
https://issues.apache.org/jira/secure/attachment/12848335/Metron-book-screenshot.png
>
> Thanks,
> --Matt
>
>
> On 1/19/17, 6:03 AM, "Otto Fowler" <ottobackwa...@gmail.com> wrote:
>
> Not seeing the attachment, is it attached to a jira?
>
>
> On January 19, 2017 at 04:19:02, Matt Foley (ma...@apache.org) wrote:
>
> Here’s a screen shot, attached :-)
>
> On 1/19/17, 1:04 AM, "Matt Foley" <ma...@apache.org> wrote:
>
> Hi all,
> I’ve put together a prototype doc book, along the lines we discussed,
> and
> it looks pretty worthwhile.
> Many thanks to Mike M. who whipped the pom.xml file into shape, and
> helped
> me find the right site.xml file to imitate.
>
> If you’re interested, please do a single-branch clone as follows:
> git clone --single-branch -b METRON-660
> https://github.com/mattf-horton/incubator-metron.git [clonename]
> (or whatever git command pleases you :-)
>
> In this branch, there’s a new top-level subdirectory named site-book/.
> This
> is not necessarily how we want to integrate stuff, it was just
> convenient
> to do it separate from the existing site/ directory for now. To build
> the
> book, do these three commands:
> cd site-book/
> bin/generate-md.sh #This gathers all the *.md files into
> site-book/src/site/markdown/**, and generates the menu tree into
> site-book/src/site/site.xml
> mvn clean site:site #This builds the book with the maven site plugin
> and
> doxia-markdown plugin
>
> If both those steps are successful, you can then go to a browser and
> open
>
> file:///Users/yourname/yourpath/clonename/site-book/target/site/index.html
> and see the book, with the nav menu on the LHS.
> It’s important to note that a very usable (not perfect) nav hierarchy
> has
> been auto-generated; this is not hardwired nor hand-edited.
> I do plan to add some overrides that allow individual items in the
> menu to
> be tweaked.
>
> While it already looks fairly nice, and clearly illustrates the value
> of
> building a book, there are two glaring issues.
> • Doxia-markdown doesn’t process the triple back-tick (```) the same
> way as
> Github Markdown. It seems to color-code it as <code>, but doesn’t
> preserve
> line breaks, which is really bad.
> • Similarly, it only processes bullet lists in isolation, and it
> doesn’t
> correctly combine bullet lists subordinate to a numbered list.
>
> The upshot is that
> • both code and bullet lists often lose their linebreaks, and get
> mushed
> into run-on paragraphs, usually combined with the preceding paragraph,
> and
> • bullet lists interrupt numbered lists and make them start over at
#1.
>
> Perhaps 80-90% of these issues can be fixed by editing the markdown
> files
> to put blank lines around the list formats. I started doing this, but
I
> didn’t want to obscure the proto by editing tons of .md files. As of
> this
> proto, only the half dozen actually broken files (that caused maven
> site
> build errors) have been fixed.
> The other 10-20% will just require simplification of the markdown
used,
> unless we can get an updated version of the plugins.
>
> Anyway, please take a look and share your thoughts.
>
> Thanks,
> --Matt
>
>
> On 1/16/17, 1:02 PM, "Michael Miklavcic" <michael.miklav...@gmail.com>
> wrote:
>
> Hey Matt, feel free to ping me.
>
> On Mon, Jan 16, 2017 at 1:39 PM, Matt Foley <ma...@apache.org> wrote:
>
> > I looked into the Falcon website and doxia over the weekend, and I’m
> > convinced that using the doxia-markdown plugin should make it dirt
> simple
> > to do what’s been discussed in this thread, with no overhead on the
> part
> of
> > people writing the README.md files.
> >
> > I fiddled with trying to do a POC, and unfortunately concluded
> (again)
> > that I don’t really know maven very well :-)
> > Are there any maven experts out there who would be willing to give
me
> some
> > pointers (offline) on how to make use of this apparently simple
maven
> > plug-in?
> >
> > I can do the bit of scripting needed to gather the docs. I’ve opened
> > https://issues.apache.org/jira/browse/METRON-660 with some
> sub-tasks for
> > this work.
> > --Matt
> >
> > On 1/13/17, 12:04 PM, "zeo...@gmail.com" <zeo...@gmail.com> wrote:
> >
> > +1 on any improvement to documentation and more consistency. At this
> > point, I think getting rid of or hiding some of the pages on the
wiki
> > (at
> > least for the short term) would be better than leaving them around
> > because
> > there's a lot of misinformation.
> >
> > Jon
> >
> > On Fri, Jan 13, 2017 at 10:13 AM Nick Allen <n...@nickallen.org>
> > wrote:
> >
> > > +1 I think it is sorely needed.
> > >
> > > If we can come up with a really slick solution like Spark, then
> > great. I am
> > > also not against a half-baked solution that can later evolve into
> > something
> > > else. For example, create an index README.md that links together
> > all the
> > > existing READMEs and run Pandoc on it. Not ideal, but way better
> > than what
> > > we have.
> > >
> > >
> > >
> > > On Fri, Jan 13, 2017 at 9:53 AM, Otto Fowler <
> > ottobackwa...@gmail.com>
> > > wrote:
> > >
> > > > I think something that does what you have laid out here, no
> matter
> > the
> > > > implementation details would be ideal
> > > >
> > > >
> > > > On January 12, 2017 at 18:05:24, Matt Foley (ma...@apache.org)
> > wrote:
> > > >
> > > > We currently have three forms of documentation, with the
> following
> > > > advantages and disadvantages:
> > > >
> > > > || Docs || Pro || Con ||
> > > > | CWiki |
> > > > Easy to edit, no special tools required, don't have to be a
> > developer to
> > > > contribute, google and wiki search |
> > > > Not versioned, no review process, distant from the code,
obsolete
> > content
> > > > tends to accumulate |
> > > > | Site |
> > > > Versioned and reviewed, only committers can edit, google search
|
> > > > Yet another arcane toolset must be learned, only web programmers
> > feel
> > > > comfortable contributing, "asf-site" branch not related to code
> > versions,
> > > > distant from the code, tends to go obsolete due to
> non-maintenance
> > |
> > > > | README.md |
> > > > Versioned and reviewed, only committers can edit, tied to code
> > versions,
> > > > highly local to the code being documented |
> > > > Non-developers don't know about them, may be scared by github,
> poor
> > > scoring
> > > > in google search, no high-level presentation |
> > > >
> > > > Various discussion threads indicate the developer community
likes
> > > > README-based docs, and it's easy to see why from the above. I
> > propose
> > > this
> > > > extension to the README-based documentation, to address their
> > > > disadvantages:
> > > >
> > > > 1. Produce a script that gathers the README.md files from all
> code
> > > > subdirectories into a hierarchical list. The script would have
an
> > > exclusion
> > > > list for non-user-content, which at this point would consist of
> > [site/*,
> > > > build_utils/*]. The hierarchy would be sorted depth-first. The
> > resulting
> > > > hierarchical list at this time (with six added README files to
> > complete
> > > the
> > > > hierarchy) would be:
> > > >
> > > > ./README.md
> > > > ./metron-analytics/README.md <== (need file here)
> > > > ./metron-analytics/metron-maas-service/README.md
> > > > ./metron-analytics/metron-profiler/README.md
> > > > ./metron-analytics/metron-profiler-client/README.md
> > > > ./metron-analytics/metron-statistics/README.md
> > > > ./metron-deployment/README.md
> > > > ./metron-deployment/amazon-ec2/README.md
> > > > ./metron-deployment/packaging/README.md <== (need file here)
> > > > ./metron-deployment/packaging/ambari/README.md <== (need file
> > here)
> > > > ./metron-deployment/packaging/docker/ansible-docker/README.md
> > > > ./metron-deployment/packaging/docker/rpm-docker/README.md
> > > > ./metron-deployment/packer-build/README.md
> > > > ./metron-deployment/roles/ <== (need file here)
> > > > ./metron-deployment/roles/kibana/README.md
> > > > ./metron-deployment/roles/monit/README.md
> > > > ./metron-deployment/roles/opentaxii/README.md
> > > > ./metron-deployment/roles/pcap_replay/README.md
> > > > ./metron-deployment/roles/sensor-test-mode/README.md
> > > > ./metron-deployment/vagrant/README.md <== (need file here)
> > > > ./metron-deployment/vagrant/codelab-platform/README.md
> > > > ./metron-deployment/vagrant/fastcapa-test-platform/README.md
> > > > ./metron-deployment/vagrant/full-dev-platform/README.md
> > > > ./metron-deployment/vagrant/quick-dev-platform/README.md
> > > > ./metron-platform/README.md
> > > > ./metron-platform/metron-api/README.md
> > > > ./metron-platform/metron-common/README.md
> > > > ./metron-platform/metron-data-management/README.md
> > > > ./metron-platform/metron-enrichment/README.md
> > > > ./metron-platform/metron-indexing/README.md
> > > > ./metron-platform/metron-management/README.md
> > > > ./metron-platform/metron-parsers/README.md
> > > > ./metron-platform/metron-pcap-backend/README.md
> > > > ./metron-sensors/README.md <== (need file here)
> > > > ./metron-sensors/fastcapa/README.md
> > > > ./metron-sensors/pycapa/README.md
> > > >
> > > > 2. Arrange to run this script as part of the build process, and
> > commit
> > > the
> > > > resulting hierarchy list to someplace in the versioned and
> branched
> > > ./site/
> > > > subdirectory.
> > > >
> > > > 3. Produce a "doc reader" web page that takes in this hierarchy
> of
> > .md
> > > > pages, and presents a LHS doc tree of links, and a main display
> > area for
> > > a
> > > > currently selected file. If we want to get fancy, this page
would
> > also
> > > > provide: (a) telescoping (collapse/expand) of the doc tree; (b)
> > floating
> > > > next/prev/up/home buttons in the display area.
> > > >
> > > > #4. Add to this web page a pull-down menu that selects among all
> > the
> > > > release versions of Metron, and (if not running in the Apache
> > site) a
> > > > SNAPSHOT version for the current filesystem version (for
> developer
> > > > preview). Let it re-write the file paths per release version to
> the
> > > proper
> > > > release tag in github. This web page will therefore be
> > > version-independent.
> > > > Put it in the asf-site branch of the Apache site, as the new
> "docs"
> > > > sub-site from the home web page. Update the list of releases at
> > each
> > > > release, or if we want to get fancy, teach it to read the
release
> > tags
> > > from
> > > > github.
> > > >
> > > > 5. As part of the release process, the release manager (a)
> assures
> > the
> > > > release is tagged in github with a consistent naming convention,
> > and (b)
> > > > submits the new hierarchy of links to google search (there's an
> > api for
> > > > that).
> > > >
> > > > 6. Deprecate the use of cwiki for anything but long-lived
> > > > demonstrations/tutorials that are unlikely to go obsolete.
> > > >
> > > >
> > > > Do folks feel this would be a good contribution to the
> visibility,
> > > > timeliness, and usability of our docs?
> > > > Is this an adequate solution for the current problems?
> > > >
> > > > Thanks,
> > > > --Matt
> > > >
> > >
> > >
> > >
> > > --
> > > Nick Allen <n...@nickallen.org>
> > >
> > --
> >
> > Jon
> >
> > Sent from my mobile device
> >
> >
> >
> >
> >
>
>
>
> --
Jon
Sent from my mobile device
