Re: [PROPOSAL] up-to-date versioned documentation

[James Sirota]

This is fantastic.  Great job, Matt 

29.01.2017, 22:13, "Matt Foley" :
> Hi all, please take a look at 
> https://github.com/apache/incubator-metron/pull/429
> I think all major issues are resolved, as best I can tell from an eyeball 
> scan of the result.
> Thanks,
> --Matt
>
> On 1/19/17, 12:06 PM, "Justin Leet"  wrote:
>
> Yeah, this looks like a huge leap forward, and I'm thrilled that you made
> such good progress. Great job, Matt.
>
> On Thu, Jan 19, 2017 at 1:55 PM, Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > Agreed! Matt, thanks for taking this on and glad I could help.
> >
> > M
> >
> > On Thu, Jan 19, 2017 at 11:42 AM, Casey Stella  
> wrote:
> >
> > > Oh wow, I really like the looks of that. I was skeptical before, but 
> if
> > > you got that far in a couple of days, I think this is a worth-while
> > > endeavor! Thanks so much Matt!
> > >
> > > Casey
> > >
> > > On Thu, Jan 19, 2017 at 12:39 PM, Matt Foley  wrote:
> > >
> > > > Thanks, Jon! I’m working on characterizing exactly how to fix the 
> two
> > > > main issues. I think I’ve got a script that will auto-fix the
> > > > triple-backtick problem. The bullet list problem will require
> > > > hand-editing, so I want to make sure I’ve got the right 
> recommendation.
> > > >
> > > > The larger issue is going thru and making the doc content better and
> > more
> > > > usable.
> > > > But that can occur over time, and will be motivated by having the 
> book
> > > > there to gripe about :-)
> > > >
> > > > On 1/19/17, 9:05 AM, "zeo...@gmail.com"  wrote:
> > > >
> > > > Looking at the screenshot, that would be an incredible improvement
> > on
> > > > what
> > > > we currently have. I'd be happy to help out with any markdown
> > > > modifications and documentation cleanup, if necessary, to fix the
> > > > problems
> > > > you outlined above.
> > > >
> > > > Jon
> > > >
> > > > On Thu, Jan 19, 2017 at 11:22 AM Matt Foley 
> > > wrote:
> > > >
> > > > > Sorry, I forgot text-only messages won’t accept attachments.
> > > Please
> > > > see
> > > > >
> > > > > https://issues.apache.org/jira/secure/attachment/
> > > > 12848335/Metron-book-screenshot.png
> > > > >
> > > > > Thanks,
> > > > > --Matt
> > > > >
> > > > >
> > > > > On 1/19/17, 6:03 AM, "Otto Fowler" 
> > > wrote:
> > > > >
> > > > > Not seeing the attachment, is it attached to a jira?
> > > > >
> > > > >
> > > > > On January 19, 2017 at 04:19:02, Matt Foley (
> > ma...@apache.org)
> > > > wrote:
> > > > >
> > > > > Here’s a screen shot, attached :-)
> > > > >
> > > > > On 1/19/17, 1:04 AM, "Matt Foley"  wrote:
> > > > >
> > > > > Hi all,
> > > > > I’ve put together a prototype doc book, along the lines we
> > > > discussed,
> > > > > and
> > > > > it looks pretty worthwhile.
> > > > > Many thanks to Mike M. who whipped the pom.xml file into
> > shape,
> > > > and
> > > > > helped
> > > > > me find the right site.xml file to imitate.
> > > > >
> > > > > If you’re interested, please do a single-branch clone as
> > > follows:
> > > > > git clone --single-branch -b METRON-660
> > > > > https://github.com/mattf-horton/incubator-metron.git
> > > [clonename]
> > > > > (or whatever git command pleases you :-)
> > > > >
> > > > > In this branch, there’s a new top-level subdirectory named
> > > > site-book/.
> > > > > This
> > > > > is not necessarily how we want to integrate stuff, it was
> > just
> > > > > convenient
> > > > > to do it separate from the existing site/ directory for now.
> > To
> > > > build
> > > > > the
> > > > > book, do these three commands:
> > > > > cd site-book/
> > > > > bin/generate-md.sh #This gathers all the *.md files into
> > > > > site-book/src/site/markdown/**, and generates the menu tree
> > > into
> > > > > site-book/src/site/site.xml
> > > > > mvn clean site:site #This builds the book with the maven site
> > > > plugin
> > > > > and
> > > > > doxia-markdown plugin
> > > > >
> > > > > If both those steps are successful, you can then go to a
> > > browser
> > > > and
> > > > > open
> > > > >
> > > > > file:///Users/yourname/yourpath/clonename/site-book/
> > > > target/site/index.html
> > > > > and see the book, with the nav menu on the LHS.
> > > > > It’s important to note that a very usable (not perfect) nav
> > > > hierarchy
> > > > > has
> > > > > been auto-generated; this is not hardwired nor hand-edited.
> > > > 

Re: [PROPOSAL] up-to-date versioned documentation

[Matt Foley]

Hi all, please take a look at 
https://github.com/apache/incubator-metron/pull/429
I think all major issues are resolved, as best I can tell from an eyeball scan 
of the result.
Thanks,
--Matt

On 1/19/17, 12:06 PM, "Justin Leet"  wrote:

Yeah, this looks like a huge leap forward, and I'm thrilled that you made
such good progress.  Great job, Matt.

On Thu, Jan 19, 2017 at 1:55 PM, Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Agreed! Matt, thanks for taking this on and glad I could help.
>
> M
>
> On Thu, Jan 19, 2017 at 11:42 AM, Casey Stella  wrote:
>
> > Oh wow, I really like the looks of that.  I was skeptical before, but if
> > you got that far in a couple of days, I think this is a worth-while
> > endeavor!  Thanks so much Matt!
> >
> > Casey
> >
> > On Thu, Jan 19, 2017 at 12:39 PM, Matt Foley  wrote:
> >
> > > Thanks, Jon!  I’m working on characterizing exactly how to fix the two
> > > main issues.  I think I’ve got a script that will auto-fix the
> > > triple-backtick problem.  The bullet list problem will require
> > > hand-editing, so I want to make sure I’ve got the right 
recommendation.
> > >
> > > The larger issue is going thru and making the doc content better and
> more
> > > usable.
> > > But that can occur over time, and will be motivated by having the book
> > > there to gripe about :-)
> > >
> > > On 1/19/17, 9:05 AM, "zeo...@gmail.com"  wrote:
> > >
> > > Looking at the screenshot, that would be an incredible improvement
> on
> > > what
> > > we currently have.  I'd be happy to help out with any markdown
> > > modifications and documentation cleanup, if necessary, to fix the
> > > problems
> > > you outlined above.
> > >
> > > Jon
> > >
> > > On Thu, Jan 19, 2017 at 11:22 AM Matt Foley 
> > wrote:
> > >
> > > > Sorry, I forgot text-only messages won’t accept attachments.
> > Please
> > > see
> > > >
> > > > https://issues.apache.org/jira/secure/attachment/
> > > 12848335/Metron-book-screenshot.png
> > > >
> > > > Thanks,
> > > > --Matt
> > > >
> > > >
> > > > On 1/19/17, 6:03 AM, "Otto Fowler" 
> > wrote:
> > > >
> > > > Not seeing the attachment, is it attached to a jira?
> > > >
> > > >
> > > > On January 19, 2017 at 04:19:02, Matt Foley (
> ma...@apache.org)
> > > wrote:
> > > >
> > > > Here’s a screen shot, attached :-)
> > > >
> > > > On 1/19/17, 1:04 AM, "Matt Foley"  wrote:
> > > >
> > > > Hi all,
> > > > I’ve put together a prototype doc book, along the lines we
> > > discussed,
> > > > and
> > > > it looks pretty worthwhile.
> > > > Many thanks to Mike M. who whipped the pom.xml file into
> shape,
> > > and
> > > > helped
> > > > me find the right site.xml file to imitate.
> > > >
> > > > If you’re interested, please do a single-branch clone as
> > follows:
> > > > git clone --single-branch -b METRON-660
> > > > https://github.com/mattf-horton/incubator-metron.git
> > [clonename]
> > > > (or whatever git command pleases you :-)
> > > >
> > > > In this branch, there’s a new top-level subdirectory named
> > > site-book/.
> > > > This
> > > > is not necessarily how we want to integrate stuff, it was
> just
> > > > convenient
> > > > to do it separate from the existing site/ directory for now.
> To
> > > build
> > > > the
> > > > book, do these three commands:
> > > > cd site-book/
> > > > bin/generate-md.sh #This gathers all the *.md files into
> > > > site-book/src/site/markdown/**, and generates the menu tree
> > into
> > > > site-book/src/site/site.xml
> > > > mvn clean site:site #This builds the book with the maven 
site
> > > plugin
> > > > and
> > > > doxia-markdown plugin
> > > >
> > > > If both those steps are successful, you can then go to a
> > browser
> > > and
> > > > open
> > > >
> > > > file:///Users/yourname/yourpath/clonename/site-book/
> > > target/site/index.html
> > > > and see the book, with the nav menu on the LHS.
> > > > It’s important to note that a very usable (not perfect) nav
> > > hierarchy
> > > > has
> > > > been auto-generated; this is not hardwired nor hand-edited.
> 

Re: [PROPOSAL] up-to-date versioned documentation

[Justin Leet]

Yeah, this looks like a huge leap forward, and I'm thrilled that you made
such good progress.  Great job, Matt.

On Thu, Jan 19, 2017 at 1:55 PM, Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Agreed! Matt, thanks for taking this on and glad I could help.
>
> M
>
> On Thu, Jan 19, 2017 at 11:42 AM, Casey Stella  wrote:
>
> > Oh wow, I really like the looks of that.  I was skeptical before, but if
> > you got that far in a couple of days, I think this is a worth-while
> > endeavor!  Thanks so much Matt!
> >
> > Casey
> >
> > On Thu, Jan 19, 2017 at 12:39 PM, Matt Foley  wrote:
> >
> > > Thanks, Jon!  I’m working on characterizing exactly how to fix the two
> > > main issues.  I think I’ve got a script that will auto-fix the
> > > triple-backtick problem.  The bullet list problem will require
> > > hand-editing, so I want to make sure I’ve got the right recommendation.
> > >
> > > The larger issue is going thru and making the doc content better and
> more
> > > usable.
> > > But that can occur over time, and will be motivated by having the book
> > > there to gripe about :-)
> > >
> > > On 1/19/17, 9:05 AM, "zeo...@gmail.com"  wrote:
> > >
> > > Looking at the screenshot, that would be an incredible improvement
> on
> > > what
> > > we currently have.  I'd be happy to help out with any markdown
> > > modifications and documentation cleanup, if necessary, to fix the
> > > problems
> > > you outlined above.
> > >
> > > Jon
> > >
> > > On Thu, Jan 19, 2017 at 11:22 AM Matt Foley 
> > wrote:
> > >
> > > > Sorry, I forgot text-only messages won’t accept attachments.
> > Please
> > > see
> > > >
> > > > https://issues.apache.org/jira/secure/attachment/
> > > 12848335/Metron-book-screenshot.png
> > > >
> > > > Thanks,
> > > > --Matt
> > > >
> > > >
> > > > On 1/19/17, 6:03 AM, "Otto Fowler" 
> > wrote:
> > > >
> > > > Not seeing the attachment, is it attached to a jira?
> > > >
> > > >
> > > > On January 19, 2017 at 04:19:02, Matt Foley (
> ma...@apache.org)
> > > wrote:
> > > >
> > > > Here’s a screen shot, attached :-)
> > > >
> > > > On 1/19/17, 1:04 AM, "Matt Foley"  wrote:
> > > >
> > > > Hi all,
> > > > I’ve put together a prototype doc book, along the lines we
> > > discussed,
> > > > and
> > > > it looks pretty worthwhile.
> > > > Many thanks to Mike M. who whipped the pom.xml file into
> shape,
> > > and
> > > > helped
> > > > me find the right site.xml file to imitate.
> > > >
> > > > If you’re interested, please do a single-branch clone as
> > follows:
> > > > git clone --single-branch -b METRON-660
> > > > https://github.com/mattf-horton/incubator-metron.git
> > [clonename]
> > > > (or whatever git command pleases you :-)
> > > >
> > > > In this branch, there’s a new top-level subdirectory named
> > > site-book/.
> > > > This
> > > > is not necessarily how we want to integrate stuff, it was
> just
> > > > convenient
> > > > to do it separate from the existing site/ directory for now.
> To
> > > build
> > > > the
> > > > book, do these three commands:
> > > > cd site-book/
> > > > bin/generate-md.sh #This gathers all the *.md files into
> > > > site-book/src/site/markdown/**, and generates the menu tree
> > into
> > > > site-book/src/site/site.xml
> > > > mvn clean site:site #This builds the book with the maven site
> > > plugin
> > > > and
> > > > doxia-markdown plugin
> > > >
> > > > If both those steps are successful, you can then go to a
> > browser
> > > and
> > > > open
> > > >
> > > > file:///Users/yourname/yourpath/clonename/site-book/
> > > target/site/index.html
> > > > and see the book, with the nav menu on the LHS.
> > > > It’s important to note that a very usable (not perfect) nav
> > > hierarchy
> > > > has
> > > > been auto-generated; this is not hardwired nor hand-edited.
> > > > I do plan to add some overrides that allow individual items
> in
> > > the
> > > > menu to
> > > > be tweaked.
> > > >
> > > > While it already looks fairly nice, and clearly illustrates
> the
> > > value
> > > > of
> > > > building a book, there are two glaring issues.
> > > > • Doxia-markdown doesn’t process the triple back-tick (```)
> the
> > > same
> > > > way as
> > > > Github Markdown. It seems to color-code it as , but
> > doesn’t
> > > > preserve
> > > > line breaks, which is really bad.
> > > > • Similarly, it only processes bullet lists in isolation, and
> > it
> > > > doesn’t
> > > > correctly combine bullet lists subordinate to a 

Re: [PROPOSAL] up-to-date versioned documentation

[Michael Miklavcic]

Agreed! Matt, thanks for taking this on and glad I could help.

M

On Thu, Jan 19, 2017 at 11:42 AM, Casey Stella  wrote:

> Oh wow, I really like the looks of that.  I was skeptical before, but if
> you got that far in a couple of days, I think this is a worth-while
> endeavor!  Thanks so much Matt!
>
> Casey
>
> On Thu, Jan 19, 2017 at 12:39 PM, Matt Foley  wrote:
>
> > Thanks, Jon!  I’m working on characterizing exactly how to fix the two
> > main issues.  I think I’ve got a script that will auto-fix the
> > triple-backtick problem.  The bullet list problem will require
> > hand-editing, so I want to make sure I’ve got the right recommendation.
> >
> > The larger issue is going thru and making the doc content better and more
> > usable.
> > But that can occur over time, and will be motivated by having the book
> > there to gripe about :-)
> >
> > On 1/19/17, 9:05 AM, "zeo...@gmail.com"  wrote:
> >
> > Looking at the screenshot, that would be an incredible improvement on
> > what
> > we currently have.  I'd be happy to help out with any markdown
> > modifications and documentation cleanup, if necessary, to fix the
> > problems
> > you outlined above.
> >
> > Jon
> >
> > On Thu, Jan 19, 2017 at 11:22 AM Matt Foley 
> wrote:
> >
> > > Sorry, I forgot text-only messages won’t accept attachments.
> Please
> > see
> > >
> > > https://issues.apache.org/jira/secure/attachment/
> > 12848335/Metron-book-screenshot.png
> > >
> > > Thanks,
> > > --Matt
> > >
> > >
> > > On 1/19/17, 6:03 AM, "Otto Fowler" 
> wrote:
> > >
> > > Not seeing the attachment, is it attached to a jira?
> > >
> > >
> > > On January 19, 2017 at 04:19:02, Matt Foley (ma...@apache.org)
> > wrote:
> > >
> > > Here’s a screen shot, attached :-)
> > >
> > > On 1/19/17, 1:04 AM, "Matt Foley"  wrote:
> > >
> > > Hi all,
> > > I’ve put together a prototype doc book, along the lines we
> > discussed,
> > > and
> > > it looks pretty worthwhile.
> > > Many thanks to Mike M. who whipped the pom.xml file into shape,
> > and
> > > helped
> > > me find the right site.xml file to imitate.
> > >
> > > If you’re interested, please do a single-branch clone as
> follows:
> > > git clone --single-branch -b METRON-660
> > > https://github.com/mattf-horton/incubator-metron.git
> [clonename]
> > > (or whatever git command pleases you :-)
> > >
> > > In this branch, there’s a new top-level subdirectory named
> > site-book/.
> > > This
> > > is not necessarily how we want to integrate stuff, it was just
> > > convenient
> > > to do it separate from the existing site/ directory for now. To
> > build
> > > the
> > > book, do these three commands:
> > > cd site-book/
> > > bin/generate-md.sh #This gathers all the *.md files into
> > > site-book/src/site/markdown/**, and generates the menu tree
> into
> > > site-book/src/site/site.xml
> > > mvn clean site:site #This builds the book with the maven site
> > plugin
> > > and
> > > doxia-markdown plugin
> > >
> > > If both those steps are successful, you can then go to a
> browser
> > and
> > > open
> > >
> > > file:///Users/yourname/yourpath/clonename/site-book/
> > target/site/index.html
> > > and see the book, with the nav menu on the LHS.
> > > It’s important to note that a very usable (not perfect) nav
> > hierarchy
> > > has
> > > been auto-generated; this is not hardwired nor hand-edited.
> > > I do plan to add some overrides that allow individual items in
> > the
> > > menu to
> > > be tweaked.
> > >
> > > While it already looks fairly nice, and clearly illustrates the
> > value
> > > of
> > > building a book, there are two glaring issues.
> > > • Doxia-markdown doesn’t process the triple back-tick (```) the
> > same
> > > way as
> > > Github Markdown. It seems to color-code it as , but
> doesn’t
> > > preserve
> > > line breaks, which is really bad.
> > > • Similarly, it only processes bullet lists in isolation, and
> it
> > > doesn’t
> > > correctly combine bullet lists subordinate to a numbered list.
> > >
> > > The upshot is that
> > > • both code and bullet lists often lose their linebreaks, and
> get
> > > mushed
> > > into run-on paragraphs, usually combined with the preceding
> > paragraph,
> > > and
> > > • bullet lists interrupt numbered lists and make them start
> over
> > at #1.
> > >
> > > Perhaps 80-90% of these issues can be fixed by editing the
> > markdown
> > > files
> > > to 

Re: [PROPOSAL] up-to-date versioned documentation

[Matt Foley]

Thanks, Jon!  I’m working on characterizing exactly how to fix the two main 
issues.  I think I’ve got a script that will auto-fix the triple-backtick 
problem.  The bullet list problem will require hand-editing, so I want to make 
sure I’ve got the right recommendation.

The larger issue is going thru and making the doc content better and more 
usable.
But that can occur over time, and will be motivated by having the book there to 
gripe about :-)

On 1/19/17, 9:05 AM, "zeo...@gmail.com"  wrote:

Looking at the screenshot, that would be an incredible improvement on what
we currently have.  I'd be happy to help out with any markdown
modifications and documentation cleanup, if necessary, to fix the problems
you outlined above.

Jon

On Thu, Jan 19, 2017 at 11:22 AM Matt Foley  wrote:

> Sorry, I forgot text-only messages won’t accept attachments.  Please see
>
> 
https://issues.apache.org/jira/secure/attachment/12848335/Metron-book-screenshot.png
>
> Thanks,
> --Matt
>
>
> On 1/19/17, 6:03 AM, "Otto Fowler"  wrote:
>
> Not seeing the attachment, is it attached to a jira?
>
>
> On January 19, 2017 at 04:19:02, Matt Foley (ma...@apache.org) wrote:
>
> Here’s a screen shot, attached :-)
>
> On 1/19/17, 1:04 AM, "Matt Foley"  wrote:
>
> Hi all,
> I’ve put together a prototype doc book, along the lines we discussed,
> and
> it looks pretty worthwhile.
> Many thanks to Mike M. who whipped the pom.xml file into shape, and
> helped
> me find the right site.xml file to imitate.
>
> If you’re interested, please do a single-branch clone as follows:
> git clone --single-branch -b METRON-660
> https://github.com/mattf-horton/incubator-metron.git [clonename]
> (or whatever git command pleases you :-)
>
> In this branch, there’s a new top-level subdirectory named site-book/.
> This
> is not necessarily how we want to integrate stuff, it was just
> convenient
> to do it separate from the existing site/ directory for now. To build
> the
> book, do these three commands:
> cd site-book/
> bin/generate-md.sh #This gathers all the *.md files into
> site-book/src/site/markdown/**, and generates the menu tree into
> site-book/src/site/site.xml
> mvn clean site:site #This builds the book with the maven site plugin
> and
> doxia-markdown plugin
>
> If both those steps are successful, you can then go to a browser and
> open
>
> file:///Users/yourname/yourpath/clonename/site-book/target/site/index.html
> and see the book, with the nav menu on the LHS.
> It’s important to note that a very usable (not perfect) nav hierarchy
> has
> been auto-generated; this is not hardwired nor hand-edited.
> I do plan to add some overrides that allow individual items in the
> menu to
> be tweaked.
>
> While it already looks fairly nice, and clearly illustrates the value
> of
> building a book, there are two glaring issues.
> • Doxia-markdown doesn’t process the triple back-tick (```) the same
> way as
> Github Markdown. It seems to color-code it as , but doesn’t
> preserve
> line breaks, which is really bad.
> • Similarly, it only processes bullet lists in isolation, and it
> doesn’t
> correctly combine bullet lists subordinate to a numbered list.
>
> The upshot is that
> • both code and bullet lists often lose their linebreaks, and get
> mushed
> into run-on paragraphs, usually combined with the preceding paragraph,
> and
> • bullet lists interrupt numbered lists and make them start over at 
#1.
>
> Perhaps 80-90% of these issues can be fixed by editing the markdown
> files
> to put blank lines around the list formats. I started doing this, but 
I
> didn’t want to obscure the proto by editing tons of .md files. As of
> this
> proto, only the half dozen actually broken files (that caused maven
> site
> build errors) have been fixed.
> The other 10-20% will just require simplification of the markdown 
used,
> unless we can get an updated version of the plugins.
>
> Anyway, please take a look and share your thoughts.
>
> Thanks,
> --Matt
>
>
> On 1/16/17, 1:02 PM, "Michael Miklavcic" 
> wrote:
>
> Hey Matt, feel free to ping me.
>
> On Mon, Jan 16, 2017 at 1:39 PM, Matt Foley  wrote:
>
> > I looked into the Falcon website and doxia over the weekend, and I’m
> > 

Re: [PROPOSAL] up-to-date versioned documentation

[zeo...@gmail.com]

Looking at the screenshot, that would be an incredible improvement on what
we currently have.  I'd be happy to help out with any markdown
modifications and documentation cleanup, if necessary, to fix the problems
you outlined above.

Jon

On Thu, Jan 19, 2017 at 11:22 AM Matt Foley  wrote:

> Sorry, I forgot text-only messages won’t accept attachments.  Please see
>
> https://issues.apache.org/jira/secure/attachment/12848335/Metron-book-screenshot.png
>
> Thanks,
> --Matt
>
>
> On 1/19/17, 6:03 AM, "Otto Fowler"  wrote:
>
> Not seeing the attachment, is it attached to a jira?
>
>
> On January 19, 2017 at 04:19:02, Matt Foley (ma...@apache.org) wrote:
>
> Here’s a screen shot, attached :-)
>
> On 1/19/17, 1:04 AM, "Matt Foley"  wrote:
>
> Hi all,
> I’ve put together a prototype doc book, along the lines we discussed,
> and
> it looks pretty worthwhile.
> Many thanks to Mike M. who whipped the pom.xml file into shape, and
> helped
> me find the right site.xml file to imitate.
>
> If you’re interested, please do a single-branch clone as follows:
> git clone --single-branch -b METRON-660
> https://github.com/mattf-horton/incubator-metron.git [clonename]
> (or whatever git command pleases you :-)
>
> In this branch, there’s a new top-level subdirectory named site-book/.
> This
> is not necessarily how we want to integrate stuff, it was just
> convenient
> to do it separate from the existing site/ directory for now. To build
> the
> book, do these three commands:
> cd site-book/
> bin/generate-md.sh #This gathers all the *.md files into
> site-book/src/site/markdown/**, and generates the menu tree into
> site-book/src/site/site.xml
> mvn clean site:site #This builds the book with the maven site plugin
> and
> doxia-markdown plugin
>
> If both those steps are successful, you can then go to a browser and
> open
>
> file:///Users/yourname/yourpath/clonename/site-book/target/site/index.html
> and see the book, with the nav menu on the LHS.
> It’s important to note that a very usable (not perfect) nav hierarchy
> has
> been auto-generated; this is not hardwired nor hand-edited.
> I do plan to add some overrides that allow individual items in the
> menu to
> be tweaked.
>
> While it already looks fairly nice, and clearly illustrates the value
> of
> building a book, there are two glaring issues.
> • Doxia-markdown doesn’t process the triple back-tick (```) the same
> way as
> Github Markdown. It seems to color-code it as , but doesn’t
> preserve
> line breaks, which is really bad.
> • Similarly, it only processes bullet lists in isolation, and it
> doesn’t
> correctly combine bullet lists subordinate to a numbered list.
>
> The upshot is that
> • both code and bullet lists often lose their linebreaks, and get
> mushed
> into run-on paragraphs, usually combined with the preceding paragraph,
> and
> • bullet lists interrupt numbered lists and make them start over at #1.
>
> Perhaps 80-90% of these issues can be fixed by editing the markdown
> files
> to put blank lines around the list formats. I started doing this, but I
> didn’t want to obscure the proto by editing tons of .md files. As of
> this
> proto, only the half dozen actually broken files (that caused maven
> site
> build errors) have been fixed.
> The other 10-20% will just require simplification of the markdown used,
> unless we can get an updated version of the plugins.
>
> Anyway, please take a look and share your thoughts.
>
> Thanks,
> --Matt
>
>
> On 1/16/17, 1:02 PM, "Michael Miklavcic" 
> wrote:
>
> Hey Matt, feel free to ping me.
>
> On Mon, Jan 16, 2017 at 1:39 PM, Matt Foley  wrote:
>
> > I looked into the Falcon website and doxia over the weekend, and I’m
> > convinced that using the doxia-markdown plugin should make it dirt
> simple
> > to do what’s been discussed in this thread, with no overhead on the
> part
> of
> > people writing the README.md files.
> >
> > I fiddled with trying to do a POC, and unfortunately concluded
> (again)
> > that I don’t really know maven very well :-)
> > Are there any maven experts out there who would be willing to give me
> some
> > pointers (offline) on how to make use of this apparently simple maven
> > plug-in?
> >
> > I can do the bit of scripting needed to gather the docs. I’ve opened
> > https://issues.apache.org/jira/browse/METRON-660 with some
> sub-tasks for
> > this work.
> > --Matt
> >
> > On 1/13/17, 12:04 PM, "zeo...@gmail.com"  wrote:
> >
> > +1 on any improvement to documentation and more consistency. At this
> > point, I think getting rid of or hiding some of the pages on the 

Re: [PROPOSAL] up-to-date versioned documentation

[Otto Fowler]

Not seeing the attachment, is it attached to a jira?


On January 19, 2017 at 04:19:02, Matt Foley (ma...@apache.org) wrote:

Here’s a screen shot, attached :-)

On 1/19/17, 1:04 AM, "Matt Foley"  wrote:

Hi all,
I’ve put together a prototype doc book, along the lines we discussed, and
it looks pretty worthwhile.
Many thanks to Mike M. who whipped the pom.xml file into shape, and helped
me find the right site.xml file to imitate.

If you’re interested, please do a single-branch clone as follows:
git clone --single-branch -b METRON-660
https://github.com/mattf-horton/incubator-metron.git [clonename]
(or whatever git command pleases you :-)

In this branch, there’s a new top-level subdirectory named site-book/. This
is not necessarily how we want to integrate stuff, it was just convenient
to do it separate from the existing site/ directory for now. To build the
book, do these three commands:
cd site-book/
bin/generate-md.sh #This gathers all the *.md files into
site-book/src/site/markdown/**, and generates the menu tree into
site-book/src/site/site.xml
mvn clean site:site #This builds the book with the maven site plugin and
doxia-markdown plugin

If both those steps are successful, you can then go to a browser and open
file:///Users/yourname/yourpath/clonename/site-book/target/site/index.html
and see the book, with the nav menu on the LHS.
It’s important to note that a very usable (not perfect) nav hierarchy has
been auto-generated; this is not hardwired nor hand-edited.
I do plan to add some overrides that allow individual items in the menu to
be tweaked.

While it already looks fairly nice, and clearly illustrates the value of
building a book, there are two glaring issues.
• Doxia-markdown doesn’t process the triple back-tick (```) the same way as
Github Markdown. It seems to color-code it as , but doesn’t preserve
line breaks, which is really bad.
• Similarly, it only processes bullet lists in isolation, and it doesn’t
correctly combine bullet lists subordinate to a numbered list.

The upshot is that
• both code and bullet lists often lose their linebreaks, and get mushed
into run-on paragraphs, usually combined with the preceding paragraph, and
• bullet lists interrupt numbered lists and make them start over at #1.

Perhaps 80-90% of these issues can be fixed by editing the markdown files
to put blank lines around the list formats. I started doing this, but I
didn’t want to obscure the proto by editing tons of .md files. As of this
proto, only the half dozen actually broken files (that caused maven site
build errors) have been fixed.
The other 10-20% will just require simplification of the markdown used,
unless we can get an updated version of the plugins.

Anyway, please take a look and share your thoughts.

Thanks,
--Matt


On 1/16/17, 1:02 PM, "Michael Miklavcic" 
wrote:

Hey Matt, feel free to ping me.

On Mon, Jan 16, 2017 at 1:39 PM, Matt Foley  wrote:

> I looked into the Falcon website and doxia over the weekend, and I’m
> convinced that using the doxia-markdown plugin should make it dirt simple
> to do what’s been discussed in this thread, with no overhead on the part
of
> people writing the README.md files.
>
> I fiddled with trying to do a POC, and unfortunately concluded (again)
> that I don’t really know maven very well :-)
> Are there any maven experts out there who would be willing to give me
some
> pointers (offline) on how to make use of this apparently simple maven
> plug-in?
>
> I can do the bit of scripting needed to gather the docs. I’ve opened
> https://issues.apache.org/jira/browse/METRON-660 with some sub-tasks for
> this work.
> --Matt
>
> On 1/13/17, 12:04 PM, "zeo...@gmail.com"  wrote:
>
> +1 on any improvement to documentation and more consistency. At this
> point, I think getting rid of or hiding some of the pages on the wiki
> (at
> least for the short term) would be better than leaving them around
> because
> there's a lot of misinformation.
>
> Jon
>
> On Fri, Jan 13, 2017 at 10:13 AM Nick Allen 
> wrote:
>
> > +1 I think it is sorely needed.
> >
> > If we can come up with a really slick solution like Spark, then
> great. I am
> > also not against a half-baked solution that can later evolve into
> something
> > else. For example, create an index README.md that links together
> all the
> > existing READMEs and run Pandoc on it. Not ideal, but way better
> than what
> > we have.
> >
> >
> >
> > On Fri, Jan 13, 2017 at 9:53 AM, Otto Fowler <
> ottobackwa...@gmail.com>
> > wrote:
> >
> > > I think something that does what you have laid out here, no matter
> the
> > > implementation details would be ideal
> > >
> > >
> > > On January 12, 2017 at 18:05:24, Matt Foley (ma...@apache.org)
> wrote:
> > >
> > > We currently have three forms of documentation, with the following
> > > advantages and disadvantages:
> > >
> > > || Docs || Pro || Con ||
> > > | CWiki |
> > > Easy to 

Re: [PROPOSAL] up-to-date versioned documentation

[Matt Foley]

Hi all,
I’ve put together a prototype doc book, along the lines we discussed, and it 
looks pretty worthwhile.
Many thanks to Mike M. who whipped the pom.xml file into shape, and helped me 
find the right site.xml file to imitate.

If you’re interested, please do a single-branch clone as follows:
git clone --single-branch -b METRON-660 
https://github.com/mattf-horton/incubator-metron.git  [clonename]
(or whatever git command pleases you :-)

In this branch, there’s a new top-level subdirectory named site-book/.  This is 
not necessarily how we want to integrate stuff, it was just convenient to do it 
separate from the existing site/ directory for now.  To build the book, do 
these three commands:
cd site-book/
bin/generate-md.sh   #This gathers all the *.md files into 
site-book/src/site/markdown/**, and generates the menu tree into 
site-book/src/site/site.xml
mvn clean site:site #This builds the book with the maven site plugin 
and doxia-markdown plugin

If both those steps are successful, you can then go to a browser and open
file:///Users/yourname/yourpath/clonename/site-book/target/site/index.html
and see the book, with the nav menu on the LHS.
It’s important to note that a very usable (not perfect) nav hierarchy has been 
auto-generated; this is not hardwired nor hand-edited.
I do plan to add some overrides that allow individual items in the menu to be 
tweaked.

While it already looks fairly nice, and clearly illustrates the value of 
building a book, there are two glaring issues.
• Doxia-markdown doesn’t process the triple back-tick (```) the same way as 
Github Markdown.  It seems to color-code it as , but doesn’t preserve 
line breaks, which is really bad.
• Similarly, it only processes bullet lists in isolation, and it doesn’t 
correctly combine bullet lists subordinate to a numbered list.

The upshot is that 
• both code and bullet lists often lose their linebreaks, and get mushed into 
run-on paragraphs, usually combined with the preceding paragraph, and
• bullet lists interrupt numbered lists and make them start over at #1.

Perhaps 80-90% of these issues can be fixed by editing the markdown files to 
put blank lines around the list formats.  I started doing this, but I didn’t 
want to obscure the proto by editing tons of .md files.  As of this proto, only 
the half dozen actually broken files (that caused maven site build errors) have 
been fixed.
The other 10-20% will just require simplification of the markdown used, unless 
we can get an updated version of the plugins.

Anyway, please take a look and share your thoughts.

Thanks,
--Matt


On 1/16/17, 1:02 PM, "Michael Miklavcic"  wrote:

Hey Matt, feel free to ping me.

On Mon, Jan 16, 2017 at 1:39 PM, Matt Foley  wrote:

> I looked into the Falcon website and doxia over the weekend, and I’m
> convinced that using the doxia-markdown plugin should make it dirt simple
> to do what’s been discussed in this thread, with no overhead on the part 
of
> people writing the README.md files.
>
> I fiddled with trying to do a POC, and unfortunately concluded (again)
> that I don’t really know maven very well :-)
> Are there any maven experts out there who would be willing to give me some
> pointers (offline) on how to make use of this apparently simple maven
> plug-in?
>
> I can do the bit of scripting needed to gather the docs.  I’ve opened
> https://issues.apache.org/jira/browse/METRON-660 with some sub-tasks for
> this work.
> --Matt
>
> On 1/13/17, 12:04 PM, "zeo...@gmail.com"  wrote:
>
> +1 on any improvement to documentation and more consistency.  At this
> point, I think getting rid of or hiding some of the pages on the wiki
> (at
> least for the short term) would be better than leaving them around
> because
> there's a lot of misinformation.
>
> Jon
>
> On Fri, Jan 13, 2017 at 10:13 AM Nick Allen 
> wrote:
>
> > +1 I think it is sorely needed.
> >
> > If we can come up with a really slick solution like Spark, then
> great. I am
> > also not against a half-baked solution that can later evolve into
> something
> > else.  For example, create an index README.md that links together
> all the
> > existing READMEs and run Pandoc on it.  Not ideal, but way better
> than what
> > we have.
> >
> >
> >
> > On Fri, Jan 13, 2017 at 9:53 AM, Otto Fowler <
> ottobackwa...@gmail.com>
> > wrote:
> >
> > > I think something that does what you have laid out here, no matter
> the
> > > implementation details would be ideal
> > >
> > >
> > > On January 12, 2017 at 18:05:24, Matt Foley (ma...@apache.org)
> wrote:
> > >

Re: [PROPOSAL] up-to-date versioned documentation

[Michael Miklavcic]

Hey Matt, feel free to ping me.

On Mon, Jan 16, 2017 at 1:39 PM, Matt Foley  wrote:

> I looked into the Falcon website and doxia over the weekend, and I’m
> convinced that using the doxia-markdown plugin should make it dirt simple
> to do what’s been discussed in this thread, with no overhead on the part of
> people writing the README.md files.
>
> I fiddled with trying to do a POC, and unfortunately concluded (again)
> that I don’t really know maven very well :-)
> Are there any maven experts out there who would be willing to give me some
> pointers (offline) on how to make use of this apparently simple maven
> plug-in?
>
> I can do the bit of scripting needed to gather the docs.  I’ve opened
> https://issues.apache.org/jira/browse/METRON-660 with some sub-tasks for
> this work.
> --Matt
>
> On 1/13/17, 12:04 PM, "zeo...@gmail.com"  wrote:
>
> +1 on any improvement to documentation and more consistency.  At this
> point, I think getting rid of or hiding some of the pages on the wiki
> (at
> least for the short term) would be better than leaving them around
> because
> there's a lot of misinformation.
>
> Jon
>
> On Fri, Jan 13, 2017 at 10:13 AM Nick Allen 
> wrote:
>
> > +1 I think it is sorely needed.
> >
> > If we can come up with a really slick solution like Spark, then
> great. I am
> > also not against a half-baked solution that can later evolve into
> something
> > else.  For example, create an index README.md that links together
> all the
> > existing READMEs and run Pandoc on it.  Not ideal, but way better
> than what
> > we have.
> >
> >
> >
> > On Fri, Jan 13, 2017 at 9:53 AM, Otto Fowler <
> ottobackwa...@gmail.com>
> > wrote:
> >
> > > I think something that does what you have laid out here, no matter
> the
> > > implementation details would be ideal
> > >
> > >
> > > On January 12, 2017 at 18:05:24, Matt Foley (ma...@apache.org)
> wrote:
> > >
> > > We currently have three forms of documentation, with the following
> > > advantages and disadvantages:
> > >
> > > || Docs || Pro || Con ||
> > > | CWiki |
> > > Easy to edit, no special tools required, don't have to be a
> developer to
> > > contribute, google and wiki search |
> > > Not versioned, no review process, distant from the code, obsolete
> content
> > > tends to accumulate |
> > > | Site |
> > > Versioned and reviewed, only committers can edit, google search |
> > > Yet another arcane toolset must be learned, only web programmers
> feel
> > > comfortable contributing, "asf-site" branch not related to code
> versions,
> > > distant from the code, tends to go obsolete due to non-maintenance
> |
> > > | README.md |
> > > Versioned and reviewed, only committers can edit, tied to code
> versions,
> > > highly local to the code being documented |
> > > Non-developers don't know about them, may be scared by github, poor
> > scoring
> > > in google search, no high-level presentation |
> > >
> > > Various discussion threads indicate the developer community likes
> > > README-based docs, and it's easy to see why from the above. I
> propose
> > this
> > > extension to the README-based documentation, to address their
> > > disadvantages:
> > >
> > > 1. Produce a script that gathers the README.md files from all code
> > > subdirectories into a hierarchical list. The script would have an
> > exclusion
> > > list for non-user-content, which at this point would consist of
> [site/*,
> > > build_utils/*]. The hierarchy would be sorted depth-first. The
> resulting
> > > hierarchical list at this time (with six added README files to
> complete
> > the
> > > hierarchy) would be:
> > >
> > > ./README.md
> > > ./metron-analytics/README.md <== (need file here)
> > > ./metron-analytics/metron-maas-service/README.md
> > > ./metron-analytics/metron-profiler/README.md
> > > ./metron-analytics/metron-profiler-client/README.md
> > > ./metron-analytics/metron-statistics/README.md
> > > ./metron-deployment/README.md
> > > ./metron-deployment/amazon-ec2/README.md
> > > ./metron-deployment/packaging/README.md <== (need file here)
> > > ./metron-deployment/packaging/ambari/README.md <== (need file
> here)
> > > ./metron-deployment/packaging/docker/ansible-docker/README.md
> > > ./metron-deployment/packaging/docker/rpm-docker/README.md
> > > ./metron-deployment/packer-build/README.md
> > > ./metron-deployment/roles/ <== (need file here)
> > > ./metron-deployment/roles/kibana/README.md
> > > ./metron-deployment/roles/monit/README.md
> > > ./metron-deployment/roles/opentaxii/README.md
> > > ./metron-deployment/roles/pcap_replay/README.md
> > > ./metron-deployment/roles/sensor-test-mode/README.md
>

Re: [PROPOSAL] up-to-date versioned documentation

[Matt Foley]

I looked into the Falcon website and doxia over the weekend, and I’m convinced 
that using the doxia-markdown plugin should make it dirt simple to do what’s 
been discussed in this thread, with no overhead on the part of people writing 
the README.md files.

I fiddled with trying to do a POC, and unfortunately concluded (again) that I 
don’t really know maven very well :-)
Are there any maven experts out there who would be willing to give me some 
pointers (offline) on how to make use of this apparently simple maven plug-in?

I can do the bit of scripting needed to gather the docs.  I’ve opened 
https://issues.apache.org/jira/browse/METRON-660 with some sub-tasks for this 
work.
--Matt

On 1/13/17, 12:04 PM, "zeo...@gmail.com"  wrote:

+1 on any improvement to documentation and more consistency.  At this
point, I think getting rid of or hiding some of the pages on the wiki (at
least for the short term) would be better than leaving them around because
there's a lot of misinformation.

Jon

On Fri, Jan 13, 2017 at 10:13 AM Nick Allen  wrote:

> +1 I think it is sorely needed.
>
> If we can come up with a really slick solution like Spark, then great. I 
am
> also not against a half-baked solution that can later evolve into 
something
> else.  For example, create an index README.md that links together all the
> existing READMEs and run Pandoc on it.  Not ideal, but way better than 
what
> we have.
>
>
>
> On Fri, Jan 13, 2017 at 9:53 AM, Otto Fowler 
> wrote:
>
> > I think something that does what you have laid out here, no matter the
> > implementation details would be ideal
> >
> >
> > On January 12, 2017 at 18:05:24, Matt Foley (ma...@apache.org) wrote:
> >
> > We currently have three forms of documentation, with the following
> > advantages and disadvantages:
> >
> > || Docs || Pro || Con ||
> > | CWiki |
> > Easy to edit, no special tools required, don't have to be a developer to
> > contribute, google and wiki search |
> > Not versioned, no review process, distant from the code, obsolete 
content
> > tends to accumulate |
> > | Site |
> > Versioned and reviewed, only committers can edit, google search |
> > Yet another arcane toolset must be learned, only web programmers feel
> > comfortable contributing, "asf-site" branch not related to code 
versions,
> > distant from the code, tends to go obsolete due to non-maintenance |
> > | README.md |
> > Versioned and reviewed, only committers can edit, tied to code versions,
> > highly local to the code being documented |
> > Non-developers don't know about them, may be scared by github, poor
> scoring
> > in google search, no high-level presentation |
> >
> > Various discussion threads indicate the developer community likes
> > README-based docs, and it's easy to see why from the above. I propose
> this
> > extension to the README-based documentation, to address their
> > disadvantages:
> >
> > 1. Produce a script that gathers the README.md files from all code
> > subdirectories into a hierarchical list. The script would have an
> exclusion
> > list for non-user-content, which at this point would consist of [site/*,
> > build_utils/*]. The hierarchy would be sorted depth-first. The resulting
> > hierarchical list at this time (with six added README files to complete
> the
> > hierarchy) would be:
> >
> > ./README.md
> > ./metron-analytics/README.md <== (need file here)
> > ./metron-analytics/metron-maas-service/README.md
> > ./metron-analytics/metron-profiler/README.md
> > ./metron-analytics/metron-profiler-client/README.md
> > ./metron-analytics/metron-statistics/README.md
> > ./metron-deployment/README.md
> > ./metron-deployment/amazon-ec2/README.md
> > ./metron-deployment/packaging/README.md <== (need file here)
> > ./metron-deployment/packaging/ambari/README.md <== (need file here)
> > ./metron-deployment/packaging/docker/ansible-docker/README.md
> > ./metron-deployment/packaging/docker/rpm-docker/README.md
> > ./metron-deployment/packer-build/README.md
> > ./metron-deployment/roles/ <== (need file here)
> > ./metron-deployment/roles/kibana/README.md
> > ./metron-deployment/roles/monit/README.md
> > ./metron-deployment/roles/opentaxii/README.md
> > ./metron-deployment/roles/pcap_replay/README.md
> > ./metron-deployment/roles/sensor-test-mode/README.md
> > ./metron-deployment/vagrant/README.md <== (need file here)
> > ./metron-deployment/vagrant/codelab-platform/README.md
> > ./metron-deployment/vagrant/fastcapa-test-platform/README.md
> > ./metron-deployment/vagrant/full-dev-platform/README.md
> > 

Re: [PROPOSAL] up-to-date versioned documentation

[Nick Allen]

+1 I think it is sorely needed.

If we can come up with a really slick solution like Spark, then great. I am
also not against a half-baked solution that can later evolve into something
else.  For example, create an index README.md that links together all the
existing READMEs and run Pandoc on it.  Not ideal, but way better than what
we have.



On Fri, Jan 13, 2017 at 9:53 AM, Otto Fowler 
wrote:

> I think something that does what you have laid out here, no matter the
> implementation details would be ideal
>
>
> On January 12, 2017 at 18:05:24, Matt Foley (ma...@apache.org) wrote:
>
> We currently have three forms of documentation, with the following
> advantages and disadvantages:
>
> || Docs || Pro || Con ||
> | CWiki |
> Easy to edit, no special tools required, don't have to be a developer to
> contribute, google and wiki search |
> Not versioned, no review process, distant from the code, obsolete content
> tends to accumulate |
> | Site |
> Versioned and reviewed, only committers can edit, google search |
> Yet another arcane toolset must be learned, only web programmers feel
> comfortable contributing, "asf-site" branch not related to code versions,
> distant from the code, tends to go obsolete due to non-maintenance |
> | README.md |
> Versioned and reviewed, only committers can edit, tied to code versions,
> highly local to the code being documented |
> Non-developers don't know about them, may be scared by github, poor scoring
> in google search, no high-level presentation |
>
> Various discussion threads indicate the developer community likes
> README-based docs, and it's easy to see why from the above. I propose this
> extension to the README-based documentation, to address their
> disadvantages:
>
> 1. Produce a script that gathers the README.md files from all code
> subdirectories into a hierarchical list. The script would have an exclusion
> list for non-user-content, which at this point would consist of [site/*,
> build_utils/*]. The hierarchy would be sorted depth-first. The resulting
> hierarchical list at this time (with six added README files to complete the
> hierarchy) would be:
>
> ./README.md
> ./metron-analytics/README.md <== (need file here)
> ./metron-analytics/metron-maas-service/README.md
> ./metron-analytics/metron-profiler/README.md
> ./metron-analytics/metron-profiler-client/README.md
> ./metron-analytics/metron-statistics/README.md
> ./metron-deployment/README.md
> ./metron-deployment/amazon-ec2/README.md
> ./metron-deployment/packaging/README.md <== (need file here)
> ./metron-deployment/packaging/ambari/README.md <== (need file here)
> ./metron-deployment/packaging/docker/ansible-docker/README.md
> ./metron-deployment/packaging/docker/rpm-docker/README.md
> ./metron-deployment/packer-build/README.md
> ./metron-deployment/roles/ <== (need file here)
> ./metron-deployment/roles/kibana/README.md
> ./metron-deployment/roles/monit/README.md
> ./metron-deployment/roles/opentaxii/README.md
> ./metron-deployment/roles/pcap_replay/README.md
> ./metron-deployment/roles/sensor-test-mode/README.md
> ./metron-deployment/vagrant/README.md <== (need file here)
> ./metron-deployment/vagrant/codelab-platform/README.md
> ./metron-deployment/vagrant/fastcapa-test-platform/README.md
> ./metron-deployment/vagrant/full-dev-platform/README.md
> ./metron-deployment/vagrant/quick-dev-platform/README.md
> ./metron-platform/README.md
> ./metron-platform/metron-api/README.md
> ./metron-platform/metron-common/README.md
> ./metron-platform/metron-data-management/README.md
> ./metron-platform/metron-enrichment/README.md
> ./metron-platform/metron-indexing/README.md
> ./metron-platform/metron-management/README.md
> ./metron-platform/metron-parsers/README.md
> ./metron-platform/metron-pcap-backend/README.md
> ./metron-sensors/README.md <== (need file here)
> ./metron-sensors/fastcapa/README.md
> ./metron-sensors/pycapa/README.md
>
> 2. Arrange to run this script as part of the build process, and commit the
> resulting hierarchy list to someplace in the versioned and branched ./site/
> subdirectory.
>
> 3. Produce a "doc reader" web page that takes in this hierarchy of .md
> pages, and presents a LHS doc tree of links, and a main display area for a
> currently selected file. If we want to get fancy, this page would also
> provide: (a) telescoping (collapse/expand) of the doc tree; (b) floating
> next/prev/up/home buttons in the display area.
>
> #4. Add to this web page a pull-down menu that selects among all the
> release versions of Metron, and (if not running in the Apache site) a
> SNAPSHOT version for the current filesystem version (for developer
> preview). Let it re-write the file paths per release version to the proper
> release tag in github. This web page will therefore be version-independent.
> Put it in the asf-site branch of the Apache site, as the new "docs"
> sub-site from the home web page. Update the list of releases at each
> release, or if we want to get fancy, 

Re: [PROPOSAL] up-to-date versioned documentation

[Otto Fowler]

I think something that does what you have laid out here, no matter the
implementation details would be ideal


On January 12, 2017 at 18:05:24, Matt Foley (ma...@apache.org) wrote:

We currently have three forms of documentation, with the following
advantages and disadvantages:

|| Docs || Pro || Con ||
| CWiki |
Easy to edit, no special tools required, don't have to be a developer to
contribute, google and wiki search |
Not versioned, no review process, distant from the code, obsolete content
tends to accumulate |
| Site |
Versioned and reviewed, only committers can edit, google search |
Yet another arcane toolset must be learned, only web programmers feel
comfortable contributing, "asf-site" branch not related to code versions,
distant from the code, tends to go obsolete due to non-maintenance |
| README.md |
Versioned and reviewed, only committers can edit, tied to code versions,
highly local to the code being documented |
Non-developers don't know about them, may be scared by github, poor scoring
in google search, no high-level presentation |

Various discussion threads indicate the developer community likes
README-based docs, and it's easy to see why from the above. I propose this
extension to the README-based documentation, to address their
disadvantages:

1. Produce a script that gathers the README.md files from all code
subdirectories into a hierarchical list. The script would have an exclusion
list for non-user-content, which at this point would consist of [site/*,
build_utils/*]. The hierarchy would be sorted depth-first. The resulting
hierarchical list at this time (with six added README files to complete the
hierarchy) would be:

./README.md
./metron-analytics/README.md <== (need file here)
./metron-analytics/metron-maas-service/README.md
./metron-analytics/metron-profiler/README.md
./metron-analytics/metron-profiler-client/README.md
./metron-analytics/metron-statistics/README.md
./metron-deployment/README.md
./metron-deployment/amazon-ec2/README.md
./metron-deployment/packaging/README.md <== (need file here)
./metron-deployment/packaging/ambari/README.md <== (need file here)
./metron-deployment/packaging/docker/ansible-docker/README.md
./metron-deployment/packaging/docker/rpm-docker/README.md
./metron-deployment/packer-build/README.md
./metron-deployment/roles/ <== (need file here)
./metron-deployment/roles/kibana/README.md
./metron-deployment/roles/monit/README.md
./metron-deployment/roles/opentaxii/README.md
./metron-deployment/roles/pcap_replay/README.md
./metron-deployment/roles/sensor-test-mode/README.md
./metron-deployment/vagrant/README.md <== (need file here)
./metron-deployment/vagrant/codelab-platform/README.md
./metron-deployment/vagrant/fastcapa-test-platform/README.md
./metron-deployment/vagrant/full-dev-platform/README.md
./metron-deployment/vagrant/quick-dev-platform/README.md
./metron-platform/README.md
./metron-platform/metron-api/README.md
./metron-platform/metron-common/README.md
./metron-platform/metron-data-management/README.md
./metron-platform/metron-enrichment/README.md
./metron-platform/metron-indexing/README.md
./metron-platform/metron-management/README.md
./metron-platform/metron-parsers/README.md
./metron-platform/metron-pcap-backend/README.md
./metron-sensors/README.md <== (need file here)
./metron-sensors/fastcapa/README.md
./metron-sensors/pycapa/README.md

2. Arrange to run this script as part of the build process, and commit the
resulting hierarchy list to someplace in the versioned and branched ./site/
subdirectory.

3. Produce a "doc reader" web page that takes in this hierarchy of .md
pages, and presents a LHS doc tree of links, and a main display area for a
currently selected file. If we want to get fancy, this page would also
provide: (a) telescoping (collapse/expand) of the doc tree; (b) floating
next/prev/up/home buttons in the display area.

#4. Add to this web page a pull-down menu that selects among all the
release versions of Metron, and (if not running in the Apache site) a
SNAPSHOT version for the current filesystem version (for developer
preview). Let it re-write the file paths per release version to the proper
release tag in github. This web page will therefore be version-independent.
Put it in the asf-site branch of the Apache site, as the new "docs"
sub-site from the home web page. Update the list of releases at each
release, or if we want to get fancy, teach it to read the release tags from
github.

5. As part of the release process, the release manager (a) assures the
release is tagged in github with a consistent naming convention, and (b)
submits the new hierarchy of links to google search (there's an api for
that).

6. Deprecate the use of cwiki for anything but long-lived
demonstrations/tutorials that are unlikely to go obsolete.


Do folks feel this would be a good contribution to the visibility,
timeliness, and usability of our docs?
Is this an adequate solution for the current problems?

Thanks,
--Matt

Re: [PROPOSAL] up-to-date versioned documentation

[Matt Foley]

The Spark docs sure are pretty.  I suspect there’s a lot of person-weeks of 
work behind the content.  I don’t know how hard it was to set up the 
infrastructure, but the instructions for generating the site mention an 
impressive list of tools needed.

The Falcon docs site seems much more straightforward, and reasonably pretty 
too.  I can take a little time to understand it better.

Thanks,
--Matt


On 1/12/17, 6:19 PM, "Kyle Richardson"  wrote:

Matt, thanks for pulling this together. I completely agree that we need to
go all in on either cwiki or the README.md's. I think the wiki is poorly
updated and can cause confusion for new users and devs. My preference is
certainly for the README.md's.

I like your approach but also agree that we shouldn't need to roll our own
here. I really like the Spark documentation that Mike pointed out. Any way
we can duplicate/adapt their approach?

-Kyle

On Thu, Jan 12, 2017 at 7:19 PM, Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Casey, Matt - These guys are using doxia
> https://github.com/apache/falcon/tree/master/docs
>
> Honestly, I kind of like Spark's approach -
> https://github.com/apache/spark/tree/master/docs
>
> Mike
>
> On Thu, Jan 12, 2017 at 4:48 PM, Matt Foley  wrote:
>
> > I’m ambivalent; I think we’d end up tied to the doxia processing
> pipeline,
> > which is “yet another arcane toolset” to learn.  Using .md as the input
> > format decreases the dependency, but we’d still be dependent on it.
> >
> > I had anticipated that the web page would be a write-once thing that
> would
> > be only a couple days for an experienced Web developer. But I was going
> to
> > get an estimate from some co-workers before actually trying to get it
> > implemented. And the script is a few hours of work with find and awk.
> >
> > On the other hand, doxia is certainly an expectable solution.  Is 
setting
> > up that infrastructure less work than developing the web page?  Or is it
> > actually just a matter of a few lines in pom.xml?
> >
> >
> > On 1/12/17, 3:24 PM, "Casey Stella"  wrote:
> >
> > Just a followup thought that's a bit more constructive, maybe we
> could
> > migrate the README.md's into a site directory and use doxia markdown
> > (example here ) to
> > generate the site as part of the build to resolve 1 through 3?
> >
> > On Thu, Jan 12, 2017 at 6:19 PM, Casey Stella 
> > wrote:
> >
> > > So, I do think this would be better than what we currently do.  I
> > like a
> > > few things in particular:
> > >
> > >- I don't like the wiki one bit.
> > >- We have a LOT of documentation in the README.md's and it's
> > sometimes
> > >poorly organized
> > >- I like a documentation preprocessing pipeline to be present.
> > For
> > >instance, a major ask is all of the stellar functions in one
> > place.  That's
> > >solved by updating an index manually in the READMEs and keeping
> > it in sync
> > >with the annotation.  I'd like to make a stellar annotation ->
> > markdown
> > >generator as part of the build and this would be nice for such 
a
> > task.
> > >
> > > My only concern is that the html generation/viewer seems like a
> fair
> > > amount of engineering.  Are you sure there isn't something easier
> > that we
> > > could conform to?  I'm sure we aren't the only project in the 
world
> > that
> > > has this particular issue.  Is there something like a maven site
> > plugin or
> > > something?  Just a thought.  I'll come back with more :)
> > >
> > > Great ideas!  Keep them coming!
> > >
> > > Casey
> > >
> > > On Thu, Jan 12, 2017 at 6:05 PM, Matt Foley 
> > wrote:
> > >
> > >> We currently have three forms of documentation, with the 
following
> > >> advantages and disadvantages:
> > >>
> > >> || Docs || Pro || Con ||
> > >> | CWiki |
> > >>   Easy to edit, no special tools required, don't have to be a
> > >> developer to contribute, google and wiki search |
> > >> Not versioned, no review process, distant from the code, obsolete
> > content
> > >> tends to accumulate |
> > >> | Site |
> > >>   Versioned and reviewed, only committers can edit, google
> > search |
> > >>   Yet another arcane toolset must be learned, only web
> > programmers
> > >> feel comfortable contributing, "asf-site" 

Re: [PROPOSAL] up-to-date versioned documentation

[Kyle Richardson]

Matt, thanks for pulling this together. I completely agree that we need to
go all in on either cwiki or the README.md's. I think the wiki is poorly
updated and can cause confusion for new users and devs. My preference is
certainly for the README.md's.

I like your approach but also agree that we shouldn't need to roll our own
here. I really like the Spark documentation that Mike pointed out. Any way
we can duplicate/adapt their approach?

-Kyle

On Thu, Jan 12, 2017 at 7:19 PM, Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Casey, Matt - These guys are using doxia
> https://github.com/apache/falcon/tree/master/docs
>
> Honestly, I kind of like Spark's approach -
> https://github.com/apache/spark/tree/master/docs
>
> Mike
>
> On Thu, Jan 12, 2017 at 4:48 PM, Matt Foley  wrote:
>
> > I’m ambivalent; I think we’d end up tied to the doxia processing
> pipeline,
> > which is “yet another arcane toolset” to learn.  Using .md as the input
> > format decreases the dependency, but we’d still be dependent on it.
> >
> > I had anticipated that the web page would be a write-once thing that
> would
> > be only a couple days for an experienced Web developer. But I was going
> to
> > get an estimate from some co-workers before actually trying to get it
> > implemented. And the script is a few hours of work with find and awk.
> >
> > On the other hand, doxia is certainly an expectable solution.  Is setting
> > up that infrastructure less work than developing the web page?  Or is it
> > actually just a matter of a few lines in pom.xml?
> >
> >
> > On 1/12/17, 3:24 PM, "Casey Stella"  wrote:
> >
> > Just a followup thought that's a bit more constructive, maybe we
> could
> > migrate the README.md's into a site directory and use doxia markdown
> > (example here ) to
> > generate the site as part of the build to resolve 1 through 3?
> >
> > On Thu, Jan 12, 2017 at 6:19 PM, Casey Stella 
> > wrote:
> >
> > > So, I do think this would be better than what we currently do.  I
> > like a
> > > few things in particular:
> > >
> > >- I don't like the wiki one bit.
> > >- We have a LOT of documentation in the README.md's and it's
> > sometimes
> > >poorly organized
> > >- I like a documentation preprocessing pipeline to be present.
> > For
> > >instance, a major ask is all of the stellar functions in one
> > place.  That's
> > >solved by updating an index manually in the READMEs and keeping
> > it in sync
> > >with the annotation.  I'd like to make a stellar annotation ->
> > markdown
> > >generator as part of the build and this would be nice for such a
> > task.
> > >
> > > My only concern is that the html generation/viewer seems like a
> fair
> > > amount of engineering.  Are you sure there isn't something easier
> > that we
> > > could conform to?  I'm sure we aren't the only project in the world
> > that
> > > has this particular issue.  Is there something like a maven site
> > plugin or
> > > something?  Just a thought.  I'll come back with more :)
> > >
> > > Great ideas!  Keep them coming!
> > >
> > > Casey
> > >
> > > On Thu, Jan 12, 2017 at 6:05 PM, Matt Foley 
> > wrote:
> > >
> > >> We currently have three forms of documentation, with the following
> > >> advantages and disadvantages:
> > >>
> > >> || Docs || Pro || Con ||
> > >> | CWiki |
> > >>   Easy to edit, no special tools required, don't have to be a
> > >> developer to contribute, google and wiki search |
> > >> Not versioned, no review process, distant from the code, obsolete
> > content
> > >> tends to accumulate |
> > >> | Site |
> > >>   Versioned and reviewed, only committers can edit, google
> > search |
> > >>   Yet another arcane toolset must be learned, only web
> > programmers
> > >> feel comfortable contributing, "asf-site" branch not related to
> code
> > >> versions, distant from the code, tends to go obsolete due to
> > >> non-maintenance |
> > >> | README.md |
> > >>   Versioned and reviewed, only committers can edit, tied to
> code
> > >> versions, highly local to the code being documented |
> > >>   Non-developers don't know about them, may be scared by
> > github, poor
> > >> scoring in google search, no high-level presentation |
> > >>
> > >> Various discussion threads indicate the developer community likes
> > >> README-based docs, and it's easy to see why from the above.  I
> > propose this
> > >> extension to the README-based documentation, to address their
> > disadvantages:
> > >>
> > >> 1. Produce a script that gathers the README.md files from all code
> > >> subdirectories into a hierarchical list.  The script would have an
> > 

Re: [PROPOSAL] up-to-date versioned documentation

[Michael Miklavcic]

Casey, Matt - These guys are using doxia
https://github.com/apache/falcon/tree/master/docs

Honestly, I kind of like Spark's approach -
https://github.com/apache/spark/tree/master/docs

Mike

On Thu, Jan 12, 2017 at 4:48 PM, Matt Foley  wrote:

> I’m ambivalent; I think we’d end up tied to the doxia processing pipeline,
> which is “yet another arcane toolset” to learn.  Using .md as the input
> format decreases the dependency, but we’d still be dependent on it.
>
> I had anticipated that the web page would be a write-once thing that would
> be only a couple days for an experienced Web developer. But I was going to
> get an estimate from some co-workers before actually trying to get it
> implemented. And the script is a few hours of work with find and awk.
>
> On the other hand, doxia is certainly an expectable solution.  Is setting
> up that infrastructure less work than developing the web page?  Or is it
> actually just a matter of a few lines in pom.xml?
>
>
> On 1/12/17, 3:24 PM, "Casey Stella"  wrote:
>
> Just a followup thought that's a bit more constructive, maybe we could
> migrate the README.md's into a site directory and use doxia markdown
> (example here ) to
> generate the site as part of the build to resolve 1 through 3?
>
> On Thu, Jan 12, 2017 at 6:19 PM, Casey Stella 
> wrote:
>
> > So, I do think this would be better than what we currently do.  I
> like a
> > few things in particular:
> >
> >- I don't like the wiki one bit.
> >- We have a LOT of documentation in the README.md's and it's
> sometimes
> >poorly organized
> >- I like a documentation preprocessing pipeline to be present.
> For
> >instance, a major ask is all of the stellar functions in one
> place.  That's
> >solved by updating an index manually in the READMEs and keeping
> it in sync
> >with the annotation.  I'd like to make a stellar annotation ->
> markdown
> >generator as part of the build and this would be nice for such a
> task.
> >
> > My only concern is that the html generation/viewer seems like a fair
> > amount of engineering.  Are you sure there isn't something easier
> that we
> > could conform to?  I'm sure we aren't the only project in the world
> that
> > has this particular issue.  Is there something like a maven site
> plugin or
> > something?  Just a thought.  I'll come back with more :)
> >
> > Great ideas!  Keep them coming!
> >
> > Casey
> >
> > On Thu, Jan 12, 2017 at 6:05 PM, Matt Foley 
> wrote:
> >
> >> We currently have three forms of documentation, with the following
> >> advantages and disadvantages:
> >>
> >> || Docs || Pro || Con ||
> >> | CWiki |
> >>   Easy to edit, no special tools required, don't have to be a
> >> developer to contribute, google and wiki search |
> >> Not versioned, no review process, distant from the code, obsolete
> content
> >> tends to accumulate |
> >> | Site |
> >>   Versioned and reviewed, only committers can edit, google
> search |
> >>   Yet another arcane toolset must be learned, only web
> programmers
> >> feel comfortable contributing, "asf-site" branch not related to code
> >> versions, distant from the code, tends to go obsolete due to
> >> non-maintenance |
> >> | README.md |
> >>   Versioned and reviewed, only committers can edit, tied to code
> >> versions, highly local to the code being documented |
> >>   Non-developers don't know about them, may be scared by
> github, poor
> >> scoring in google search, no high-level presentation |
> >>
> >> Various discussion threads indicate the developer community likes
> >> README-based docs, and it's easy to see why from the above.  I
> propose this
> >> extension to the README-based documentation, to address their
> disadvantages:
> >>
> >> 1. Produce a script that gathers the README.md files from all code
> >> subdirectories into a hierarchical list.  The script would have an
> >> exclusion list for non-user-content, which at this point would
> consist of
> >> [site/*, build_utils/*].  The hierarchy would be sorted
> depth-first.  The
> >> resulting hierarchical list at this time (with six added README
> files to
> >> complete the hierarchy) would be:
> >>
> >> ./README.md
> >> ./metron-analytics/README.md  <== (need file here)
> >> ./metron-analytics/metron-maas-service/README.md
> >> ./metron-analytics/metron-profiler/README.md
> >> ./metron-analytics/metron-profiler-client/README.md
> >> ./metron-analytics/metron-statistics/README.md
> >> ./metron-deployment/README.md
> >> ./metron-deployment/amazon-ec2/README.md
> >> ./metron-deployment/packaging/README.md  <== 

Re: [PROPOSAL] up-to-date versioned documentation

[Matt Foley]

I’m ambivalent; I think we’d end up tied to the doxia processing pipeline, 
which is “yet another arcane toolset” to learn.  Using .md as the input format 
decreases the dependency, but we’d still be dependent on it.

I had anticipated that the web page would be a write-once thing that would be 
only a couple days for an experienced Web developer. But I was going to get an 
estimate from some co-workers before actually trying to get it implemented. And 
the script is a few hours of work with find and awk.

On the other hand, doxia is certainly an expectable solution.  Is setting up 
that infrastructure less work than developing the web page?  Or is it actually 
just a matter of a few lines in pom.xml?


On 1/12/17, 3:24 PM, "Casey Stella"  wrote:

Just a followup thought that's a bit more constructive, maybe we could
migrate the README.md's into a site directory and use doxia markdown
(example here ) to
generate the site as part of the build to resolve 1 through 3?

On Thu, Jan 12, 2017 at 6:19 PM, Casey Stella  wrote:

> So, I do think this would be better than what we currently do.  I like a
> few things in particular:
>
>- I don't like the wiki one bit.
>- We have a LOT of documentation in the README.md's and it's sometimes
>poorly organized
>- I like a documentation preprocessing pipeline to be present.  For
>instance, a major ask is all of the stellar functions in one place.  
That's
>solved by updating an index manually in the READMEs and keeping it in 
sync
>with the annotation.  I'd like to make a stellar annotation -> markdown
>generator as part of the build and this would be nice for such a task.
>
> My only concern is that the html generation/viewer seems like a fair
> amount of engineering.  Are you sure there isn't something easier that we
> could conform to?  I'm sure we aren't the only project in the world that
> has this particular issue.  Is there something like a maven site plugin or
> something?  Just a thought.  I'll come back with more :)
>
> Great ideas!  Keep them coming!
>
> Casey
>
> On Thu, Jan 12, 2017 at 6:05 PM, Matt Foley  wrote:
>
>> We currently have three forms of documentation, with the following
>> advantages and disadvantages:
>>
>> || Docs || Pro || Con ||
>> | CWiki |
>>   Easy to edit, no special tools required, don't have to be a
>> developer to contribute, google and wiki search |
>> Not versioned, no review process, distant from the code, obsolete content
>> tends to accumulate |
>> | Site |
>>   Versioned and reviewed, only committers can edit, google search |
>>   Yet another arcane toolset must be learned, only web programmers
>> feel comfortable contributing, "asf-site" branch not related to code
>> versions, distant from the code, tends to go obsolete due to
>> non-maintenance |
>> | README.md |
>>   Versioned and reviewed, only committers can edit, tied to code
>> versions, highly local to the code being documented |
>>   Non-developers don't know about them, may be scared by github, poor
>> scoring in google search, no high-level presentation |
>>
>> Various discussion threads indicate the developer community likes
>> README-based docs, and it's easy to see why from the above.  I propose 
this
>> extension to the README-based documentation, to address their 
disadvantages:
>>
>> 1. Produce a script that gathers the README.md files from all code
>> subdirectories into a hierarchical list.  The script would have an
>> exclusion list for non-user-content, which at this point would consist of
>> [site/*, build_utils/*].  The hierarchy would be sorted depth-first.  The
>> resulting hierarchical list at this time (with six added README files to
>> complete the hierarchy) would be:
>>
>> ./README.md
>> ./metron-analytics/README.md  <== (need file here)
>> ./metron-analytics/metron-maas-service/README.md
>> ./metron-analytics/metron-profiler/README.md
>> ./metron-analytics/metron-profiler-client/README.md
>> ./metron-analytics/metron-statistics/README.md
>> ./metron-deployment/README.md
>> ./metron-deployment/amazon-ec2/README.md
>> ./metron-deployment/packaging/README.md  <== (need file here)
>> ./metron-deployment/packaging/ambari/README.md <== (need file here)
>> ./metron-deployment/packaging/docker/ansible-docker/README.md
>> ./metron-deployment/packaging/docker/rpm-docker/README.md
>> ./metron-deployment/packer-build/README.md
>> ./metron-deployment/roles/  <== (need file here)
>> ./metron-deployment/roles/kibana/README.md
>> ./metron-deployment/roles/monit/README.md
>> 

Re: [PROPOSAL] up-to-date versioned documentation

[Casey Stella]

Just a followup thought that's a bit more constructive, maybe we could
migrate the README.md's into a site directory and use doxia markdown
(example here ) to
generate the site as part of the build to resolve 1 through 3?

On Thu, Jan 12, 2017 at 6:19 PM, Casey Stella  wrote:

> So, I do think this would be better than what we currently do.  I like a
> few things in particular:
>
>- I don't like the wiki one bit.
>- We have a LOT of documentation in the README.md's and it's sometimes
>poorly organized
>- I like a documentation preprocessing pipeline to be present.  For
>instance, a major ask is all of the stellar functions in one place.  That's
>solved by updating an index manually in the READMEs and keeping it in sync
>with the annotation.  I'd like to make a stellar annotation -> markdown
>generator as part of the build and this would be nice for such a task.
>
> My only concern is that the html generation/viewer seems like a fair
> amount of engineering.  Are you sure there isn't something easier that we
> could conform to?  I'm sure we aren't the only project in the world that
> has this particular issue.  Is there something like a maven site plugin or
> something?  Just a thought.  I'll come back with more :)
>
> Great ideas!  Keep them coming!
>
> Casey
>
> On Thu, Jan 12, 2017 at 6:05 PM, Matt Foley  wrote:
>
>> We currently have three forms of documentation, with the following
>> advantages and disadvantages:
>>
>> || Docs || Pro || Con ||
>> | CWiki |
>>   Easy to edit, no special tools required, don't have to be a
>> developer to contribute, google and wiki search |
>> Not versioned, no review process, distant from the code, obsolete content
>> tends to accumulate |
>> | Site |
>>   Versioned and reviewed, only committers can edit, google search |
>>   Yet another arcane toolset must be learned, only web programmers
>> feel comfortable contributing, "asf-site" branch not related to code
>> versions, distant from the code, tends to go obsolete due to
>> non-maintenance |
>> | README.md |
>>   Versioned and reviewed, only committers can edit, tied to code
>> versions, highly local to the code being documented |
>>   Non-developers don't know about them, may be scared by github, poor
>> scoring in google search, no high-level presentation |
>>
>> Various discussion threads indicate the developer community likes
>> README-based docs, and it's easy to see why from the above.  I propose this
>> extension to the README-based documentation, to address their disadvantages:
>>
>> 1. Produce a script that gathers the README.md files from all code
>> subdirectories into a hierarchical list.  The script would have an
>> exclusion list for non-user-content, which at this point would consist of
>> [site/*, build_utils/*].  The hierarchy would be sorted depth-first.  The
>> resulting hierarchical list at this time (with six added README files to
>> complete the hierarchy) would be:
>>
>> ./README.md
>> ./metron-analytics/README.md  <== (need file here)
>> ./metron-analytics/metron-maas-service/README.md
>> ./metron-analytics/metron-profiler/README.md
>> ./metron-analytics/metron-profiler-client/README.md
>> ./metron-analytics/metron-statistics/README.md
>> ./metron-deployment/README.md
>> ./metron-deployment/amazon-ec2/README.md
>> ./metron-deployment/packaging/README.md  <== (need file here)
>> ./metron-deployment/packaging/ambari/README.md <== (need file here)
>> ./metron-deployment/packaging/docker/ansible-docker/README.md
>> ./metron-deployment/packaging/docker/rpm-docker/README.md
>> ./metron-deployment/packer-build/README.md
>> ./metron-deployment/roles/  <== (need file here)
>> ./metron-deployment/roles/kibana/README.md
>> ./metron-deployment/roles/monit/README.md
>> ./metron-deployment/roles/opentaxii/README.md
>> ./metron-deployment/roles/pcap_replay/README.md
>> ./metron-deployment/roles/sensor-test-mode/README.md
>> ./metron-deployment/vagrant/README.md  <== (need file here)
>> ./metron-deployment/vagrant/codelab-platform/README.md
>> ./metron-deployment/vagrant/fastcapa-test-platform/README.md
>> ./metron-deployment/vagrant/full-dev-platform/README.md
>> ./metron-deployment/vagrant/quick-dev-platform/README.md
>> ./metron-platform/README.md
>> ./metron-platform/metron-api/README.md
>> ./metron-platform/metron-common/README.md
>> ./metron-platform/metron-data-management/README.md
>> ./metron-platform/metron-enrichment/README.md
>> ./metron-platform/metron-indexing/README.md
>> ./metron-platform/metron-management/README.md
>> ./metron-platform/metron-parsers/README.md
>> ./metron-platform/metron-pcap-backend/README.md
>> ./metron-sensors/README.md  <== (need file here)
>> ./metron-sensors/fastcapa/README.md
>> ./metron-sensors/pycapa/README.md
>>
>> 2. Arrange to run this script as part of the build process, and commit
>> the resulting hierarchy list to someplace 

Re: [PROPOSAL] up-to-date versioned documentation

[Casey Stella]

So, I do think this would be better than what we currently do.  I like a
few things in particular:

   - I don't like the wiki one bit.
   - We have a LOT of documentation in the README.md's and it's sometimes
   poorly organized
   - I like a documentation preprocessing pipeline to be present.  For
   instance, a major ask is all of the stellar functions in one place.  That's
   solved by updating an index manually in the READMEs and keeping it in sync
   with the annotation.  I'd like to make a stellar annotation -> markdown
   generator as part of the build and this would be nice for such a task.

My only concern is that the html generation/viewer seems like a fair amount
of engineering.  Are you sure there isn't something easier that we could
conform to?  I'm sure we aren't the only project in the world that has this
particular issue.  Is there something like a maven site plugin or
something?  Just a thought.  I'll come back with more :)

Great ideas!  Keep them coming!

Casey

On Thu, Jan 12, 2017 at 6:05 PM, Matt Foley  wrote:

> We currently have three forms of documentation, with the following
> advantages and disadvantages:
>
> || Docs || Pro || Con ||
> | CWiki |
>   Easy to edit, no special tools required, don't have to be a
> developer to contribute, google and wiki search |
> Not versioned, no review process, distant from the code, obsolete content
> tends to accumulate |
> | Site |
>   Versioned and reviewed, only committers can edit, google search |
>   Yet another arcane toolset must be learned, only web programmers
> feel comfortable contributing, "asf-site" branch not related to code
> versions, distant from the code, tends to go obsolete due to
> non-maintenance |
> | README.md |
>   Versioned and reviewed, only committers can edit, tied to code
> versions, highly local to the code being documented |
>   Non-developers don't know about them, may be scared by github, poor
> scoring in google search, no high-level presentation |
>
> Various discussion threads indicate the developer community likes
> README-based docs, and it's easy to see why from the above.  I propose this
> extension to the README-based documentation, to address their disadvantages:
>
> 1. Produce a script that gathers the README.md files from all code
> subdirectories into a hierarchical list.  The script would have an
> exclusion list for non-user-content, which at this point would consist of
> [site/*, build_utils/*].  The hierarchy would be sorted depth-first.  The
> resulting hierarchical list at this time (with six added README files to
> complete the hierarchy) would be:
>
> ./README.md
> ./metron-analytics/README.md  <== (need file here)
> ./metron-analytics/metron-maas-service/README.md
> ./metron-analytics/metron-profiler/README.md
> ./metron-analytics/metron-profiler-client/README.md
> ./metron-analytics/metron-statistics/README.md
> ./metron-deployment/README.md
> ./metron-deployment/amazon-ec2/README.md
> ./metron-deployment/packaging/README.md  <== (need file here)
> ./metron-deployment/packaging/ambari/README.md <== (need file here)
> ./metron-deployment/packaging/docker/ansible-docker/README.md
> ./metron-deployment/packaging/docker/rpm-docker/README.md
> ./metron-deployment/packer-build/README.md
> ./metron-deployment/roles/  <== (need file here)
> ./metron-deployment/roles/kibana/README.md
> ./metron-deployment/roles/monit/README.md
> ./metron-deployment/roles/opentaxii/README.md
> ./metron-deployment/roles/pcap_replay/README.md
> ./metron-deployment/roles/sensor-test-mode/README.md
> ./metron-deployment/vagrant/README.md  <== (need file here)
> ./metron-deployment/vagrant/codelab-platform/README.md
> ./metron-deployment/vagrant/fastcapa-test-platform/README.md
> ./metron-deployment/vagrant/full-dev-platform/README.md
> ./metron-deployment/vagrant/quick-dev-platform/README.md
> ./metron-platform/README.md
> ./metron-platform/metron-api/README.md
> ./metron-platform/metron-common/README.md
> ./metron-platform/metron-data-management/README.md
> ./metron-platform/metron-enrichment/README.md
> ./metron-platform/metron-indexing/README.md
> ./metron-platform/metron-management/README.md
> ./metron-platform/metron-parsers/README.md
> ./metron-platform/metron-pcap-backend/README.md
> ./metron-sensors/README.md  <== (need file here)
> ./metron-sensors/fastcapa/README.md
> ./metron-sensors/pycapa/README.md
>
> 2. Arrange to run this script as part of the build process, and commit the
> resulting hierarchy list to someplace in the versioned and branched ./site/
> subdirectory.
>
> 3. Produce a "doc reader" web page that takes in this hierarchy of .md
> pages, and presents a LHS doc tree of links, and a main display area for a
> currently selected file.  If we want to get fancy, this page would also
> provide: (a) telescoping (collapse/expand) of the doc tree; (b) floating
> next/prev/up/home buttons in the display area.
>
> #4. Add to this web page a pull-down menu that 

[PROPOSAL] up-to-date versioned documentation

[Matt Foley]

We currently have three forms of documentation, with the following advantages 
and disadvantages:

|| Docs || Pro || Con ||
| CWiki | 
  Easy to edit, no special tools required, don't have to be a developer to 
contribute, google and wiki search | 
Not versioned, no review process, distant from the code, obsolete content tends 
to accumulate |
| Site | 
  Versioned and reviewed, only committers can edit, google search | 
  Yet another arcane toolset must be learned, only web programmers feel 
comfortable contributing, "asf-site" branch not related to code versions, 
distant from the code, tends to go obsolete due to non-maintenance |
| README.md | 
  Versioned and reviewed, only committers can edit, tied to code versions, 
highly local to the code being documented | 
  Non-developers don't know about them, may be scared by github, poor 
scoring in google search, no high-level presentation |

Various discussion threads indicate the developer community likes README-based 
docs, and it's easy to see why from the above.  I propose this extension to the 
README-based documentation, to address their disadvantages:

1. Produce a script that gathers the README.md files from all code 
subdirectories into a hierarchical list.  The script would have an exclusion 
list for non-user-content, which at this point would consist of [site/*, 
build_utils/*].  The hierarchy would be sorted depth-first.  The resulting 
hierarchical list at this time (with six added README files to complete the 
hierarchy) would be:

./README.md
./metron-analytics/README.md  <== (need file here)
./metron-analytics/metron-maas-service/README.md
./metron-analytics/metron-profiler/README.md
./metron-analytics/metron-profiler-client/README.md
./metron-analytics/metron-statistics/README.md
./metron-deployment/README.md
./metron-deployment/amazon-ec2/README.md
./metron-deployment/packaging/README.md  <== (need file here)
./metron-deployment/packaging/ambari/README.md <== (need file here)
./metron-deployment/packaging/docker/ansible-docker/README.md
./metron-deployment/packaging/docker/rpm-docker/README.md
./metron-deployment/packer-build/README.md
./metron-deployment/roles/  <== (need file here)
./metron-deployment/roles/kibana/README.md
./metron-deployment/roles/monit/README.md
./metron-deployment/roles/opentaxii/README.md
./metron-deployment/roles/pcap_replay/README.md
./metron-deployment/roles/sensor-test-mode/README.md
./metron-deployment/vagrant/README.md  <== (need file here)
./metron-deployment/vagrant/codelab-platform/README.md
./metron-deployment/vagrant/fastcapa-test-platform/README.md
./metron-deployment/vagrant/full-dev-platform/README.md
./metron-deployment/vagrant/quick-dev-platform/README.md
./metron-platform/README.md
./metron-platform/metron-api/README.md
./metron-platform/metron-common/README.md
./metron-platform/metron-data-management/README.md
./metron-platform/metron-enrichment/README.md
./metron-platform/metron-indexing/README.md
./metron-platform/metron-management/README.md
./metron-platform/metron-parsers/README.md
./metron-platform/metron-pcap-backend/README.md
./metron-sensors/README.md  <== (need file here)
./metron-sensors/fastcapa/README.md
./metron-sensors/pycapa/README.md

2. Arrange to run this script as part of the build process, and commit the 
resulting hierarchy list to someplace in the versioned and branched ./site/ 
subdirectory.

3. Produce a "doc reader" web page that takes in this hierarchy of .md pages, 
and presents a LHS doc tree of links, and a main display area for a currently 
selected file.  If we want to get fancy, this page would also provide: (a) 
telescoping (collapse/expand) of the doc tree; (b) floating next/prev/up/home 
buttons in the display area.

#4. Add to this web page a pull-down menu that selects among all the release 
versions of Metron, and (if not running in the Apache site) a SNAPSHOT version 
for the current filesystem version (for developer preview).  Let it re-write 
the file paths per release version to the proper release tag in github.  This 
web page will therefore be version-independent.  Put it in the asf-site branch 
of the Apache site, as the new "docs" sub-site from the home web page.  Update 
the list of releases at each release, or if we want to get fancy, teach it to 
read the release tags from github.

5. As part of the release process, the release manager (a) assures the release 
is tagged in github with a consistent naming convention, and (b) submits the 
new hierarchy of links to google search (there's an api for that).

6. Deprecate the use of cwiki for anything but long-lived 
demonstrations/tutorials that are unlikely to go obsolete.


Do folks feel this would be a good contribution to the visibility, timeliness, 
and usability of our docs?
Is this an adequate solution for the current problems?

Thanks, 
--Matt