Hi all, please take a look at
https://github.com/apache/incubator-metron/pull/429
I think all major issues are resolved, as best I can tell from an eyeball scan
of the result.
Thanks,
--Matt
On 1/19/17, 12:06 PM, "Justin Leet" <justinjl...@gmail.com> wrote:
Yeah, this looks like a huge leap forward, and I'm thrilled that you made
such good progress. Great job, Matt.
On Thu, Jan 19, 2017 at 1:55 PM, Michael Miklavcic <
michael.miklav...@gmail.com> wrote:
> Agreed! Matt, thanks for taking this on and glad I could help.
>
> M
>
> On Thu, Jan 19, 2017 at 11:42 AM, Casey Stella <ceste...@gmail.com> wrote:
>
> > Oh wow, I really like the looks of that. I was skeptical before, but if
> > you got that far in a couple of days, I think this is a worth-while
> > endeavor! Thanks so much Matt!
> >
> > Casey
> >
> > On Thu, Jan 19, 2017 at 12:39 PM, Matt Foley <ma...@apache.org> wrote:
> >
> > > Thanks, Jon! I’m working on characterizing exactly how to fix the two
> > > main issues. I think I’ve got a script that will auto-fix the
> > > triple-backtick problem. The bullet list problem will require
> > > hand-editing, so I want to make sure I’ve got the right
recommendation.
> > >
> > > The larger issue is going thru and making the doc content better and
> more
> > > usable.
> > > But that can occur over time, and will be motivated by having the book
> > > there to gripe about :-)
> > >
> > > On 1/19/17, 9:05 AM, "zeo...@gmail.com" <zeo...@gmail.com> wrote:
> > >
> > > Looking at the screenshot, that would be an incredible improvement
> on
> > > what
> > > we currently have. I'd be happy to help out with any markdown
> > > modifications and documentation cleanup, if necessary, to fix the
> > > problems
> > > you outlined above.
> > >
> > > Jon
> > >
> > > On Thu, Jan 19, 2017 at 11:22 AM Matt Foley <ma...@apache.org>
> > wrote:
> > >
> > > > Sorry, I forgot text-only messages won’t accept attachments.
> > Please
> > > see
> > > >
> > > > https://issues.apache.org/jira/secure/attachment/
> > > 12848335/Metron-book-screenshot.png
> > > >
> > > > Thanks,
> > > > --Matt
> > > >
> > > >
> > > > On 1/19/17, 6:03 AM, "Otto Fowler" <ottobackwa...@gmail.com>
> > wrote:
> > > >
> > > > Not seeing the attachment, is it attached to a jira?
> > > >
> > > >
> > > > On January 19, 2017 at 04:19:02, Matt Foley (
> ma...@apache.org)
> > > wrote:
> > > >
> > > > Here’s a screen shot, attached :-)
> > > >
> > > > On 1/19/17, 1:04 AM, "Matt Foley" <ma...@apache.org> wrote:
> > > >
> > > > Hi all,
> > > > I’ve put together a prototype doc book, along the lines we
> > > discussed,
> > > > and
> > > > it looks pretty worthwhile.
> > > > Many thanks to Mike M. who whipped the pom.xml file into
> shape,
> > > and
> > > > helped
> > > > me find the right site.xml file to imitate.
> > > >
> > > > If you’re interested, please do a single-branch clone as
> > follows:
> > > > git clone --single-branch -b METRON-660
> > > > https://github.com/mattf-horton/incubator-metron.git
> > [clonename]
> > > > (or whatever git command pleases you :-)
> > > >
> > > > In this branch, there’s a new top-level subdirectory named
> > > site-book/.
> > > > This
> > > > is not necessarily how we want to integrate stuff, it was
> just
> > > > convenient
> > > > to do it separate from the existing site/ directory for now.
> To
> > > build
> > > > the
> > > > book, do these three commands:
> > > > cd site-book/
> > > > bin/generate-md.sh #This gathers all the *.md files into
> > > > site-book/src/site/markdown/**, and generates the menu tree
> > into
> > > > site-book/src/site/site.xml
> > > > mvn clean site:site #This builds the book with the maven
site
> > > plugin
> > > > and
> > > > doxia-markdown plugin
> > > >
> > > > If both those steps are successful, you can then go to a
> > browser
> > > and
> > > > open
> > > >
> > > > file:///Users/yourname/yourpath/clonename/site-book/
> > > target/site/index.html
> > > > and see the book, with the nav menu on the LHS.
> > > > It’s important to note that a very usable (not perfect) nav
> > > hierarchy
> > > > has
> > > > been auto-generated; this is not hardwired nor hand-edited.
> > > > I do plan to add some overrides that allow individual items
> in
> > > the
> > > > menu to
> > > > be tweaked.
> > > >
> > > > While it already looks fairly nice, and clearly illustrates
> the
> > > value
> > > > of
> > > > building a book, there are two glaring issues.
> > > > • Doxia-markdown doesn’t process the triple back-tick (```)
> the
> > > same
> > > > way as
> > > > Github Markdown. It seems to color-code it as <code>, but
> > doesn’t
> > > > preserve
> > > > line breaks, which is really bad.
> > > > • Similarly, it only processes bullet lists in isolation,
and
> > it
> > > > doesn’t
> > > > correctly combine bullet lists subordinate to a numbered
> list.
> > > >
> > > > The upshot is that
> > > > • both code and bullet lists often lose their linebreaks,
and
> > get
> > > > mushed
> > > > into run-on paragraphs, usually combined with the preceding
> > > paragraph,
> > > > and
> > > > • bullet lists interrupt numbered lists and make them start
> > over
> > > at #1.
> > > >
> > > > Perhaps 80-90% of these issues can be fixed by editing the
> > > markdown
> > > > files
> > > > to put blank lines around the list formats. I started doing
> > > this, but I
> > > > didn’t want to obscure the proto by editing tons of .md
> files.
> > > As of
> > > > this
> > > > proto, only the half dozen actually broken files (that
caused
> > > maven
> > > > site
> > > > build errors) have been fixed.
> > > > The other 10-20% will just require simplification of the
> > > markdown used,
> > > > unless we can get an updated version of the plugins.
> > > >
> > > > Anyway, please take a look and share your thoughts.
> > > >
> > > > Thanks,
> > > > --Matt
> > > >
> > > >
> > > > On 1/16/17, 1:02 PM, "Michael Miklavcic" <
> > > michael.miklav...@gmail.com>
> > > > wrote:
> > > >
> > > > Hey Matt, feel free to ping me.
> > > >
> > > > On Mon, Jan 16, 2017 at 1:39 PM, Matt Foley <
> ma...@apache.org>
> > > wrote:
> > > >
> > > > > I looked into the Falcon website and doxia over the
> weekend,
> > > and I’m
> > > > > convinced that using the doxia-markdown plugin should make
> it
> > > dirt
> > > > simple
> > > > > to do what’s been discussed in this thread, with no
> overhead
> > > on the
> > > > part
> > > > of
> > > > > people writing the README.md files.
> > > > >
> > > > > I fiddled with trying to do a POC, and unfortunately
> > concluded
> > > > (again)
> > > > > that I don’t really know maven very well :-)
> > > > > Are there any maven experts out there who would be willing
> to
> > > give me
> > > > some
> > > > > pointers (offline) on how to make use of this apparently
> > > simple maven
> > > > > plug-in?
> > > > >
> > > > > I can do the bit of scripting needed to gather the docs.
> I’ve
> > > opened
> > > > > https://issues.apache.org/jira/browse/METRON-660 with some
> > > > sub-tasks for
> > > > > this work.
> > > > > --Matt
> > > > >
> > > > > On 1/13/17, 12:04 PM, "zeo...@gmail.com" <zeo...@gmail.com
> >
> > > wrote:
> > > > >
> > > > > +1 on any improvement to documentation and more
> consistency.
> > > At this
> > > > > point, I think getting rid of or hiding some of the pages
> on
> > > the wiki
> > > > > (at
> > > > > least for the short term) would be better than leaving
them
> > > around
> > > > > because
> > > > > there's a lot of misinformation.
> > > > >
> > > > > Jon
> > > > >
> > > > > On Fri, Jan 13, 2017 at 10:13 AM Nick Allen <
> > > n...@nickallen.org>
> > > > > wrote:
> > > > >
> > > > > > +1 I think it is sorely needed.
> > > > > >
> > > > > > If we can come up with a really slick solution like
> Spark,
> > > then
> > > > > great. I am
> > > > > > also not against a half-baked solution that can later
> > evolve
> > > into
> > > > > something
> > > > > > else. For example, create an index README.md that links
> > > together
> > > > > all the
> > > > > > existing READMEs and run Pandoc on it. Not ideal, but
way
> > > better
> > > > > than what
> > > > > > we have.
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Fri, Jan 13, 2017 at 9:53 AM, Otto Fowler <
> > > > > ottobackwa...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > I think something that does what you have laid out
> here,
> > no
> > > > matter
> > > > > the
> > > > > > > implementation details would be ideal
> > > > > > >
> > > > > > >
> > > > > > > On January 12, 2017 at 18:05:24, Matt Foley (
> > > ma...@apache.org)
> > > > > wrote:
> > > > > > >
> > > > > > > We currently have three forms of documentation, with
> the
> > > > following
> > > > > > > advantages and disadvantages:
> > > > > > >
> > > > > > > || Docs || Pro || Con ||
> > > > > > > | CWiki |
> > > > > > > Easy to edit, no special tools required, don't have to
> > be a
> > > > > developer to
> > > > > > > contribute, google and wiki search |
> > > > > > > Not versioned, no review process, distant from the
> code,
> > > obsolete
> > > > > content
> > > > > > > tends to accumulate |
> > > > > > > | Site |
> > > > > > > Versioned and reviewed, only committers can edit,
> google
> > > search |
> > > > > > > Yet another arcane toolset must be learned, only web
> > > programmers
> > > > > feel
> > > > > > > comfortable contributing, "asf-site" branch not
related
> > to
> > > code
> > > > > versions,
> > > > > > > distant from the code, tends to go obsolete due to
> > > > non-maintenance
> > > > > |
> > > > > > > | README.md |
> > > > > > > Versioned and reviewed, only committers can edit, tied
> to
> > > code
> > > > > versions,
> > > > > > > highly local to the code being documented |
> > > > > > > Non-developers don't know about them, may be scared by
> > > github,
> > > > poor
> > > > > > scoring
> > > > > > > in google search, no high-level presentation |
> > > > > > >
> > > > > > > Various discussion threads indicate the developer
> > > community likes
> > > > > > > README-based docs, and it's easy to see why from the
> > > above. I
> > > > > propose
> > > > > > this
> > > > > > > extension to the README-based documentation, to
address
> > > their
> > > > > > > disadvantages:
> > > > > > >
> > > > > > > 1. Produce a script that gathers the README.md files
> from
> > > all
> > > > code
> > > > > > > subdirectories into a hierarchical list. The script
> would
> > > have an
> > > > > > exclusion
> > > > > > > list for non-user-content, which at this point would
> > > consist of
> > > > > [site/*,
> > > > > > > build_utils/*]. The hierarchy would be sorted
> > depth-first.
> > > The
> > > > > resulting
> > > > > > > hierarchical list at this time (with six added README
> > > files to
> > > > > complete
> > > > > > the
> > > > > > > hierarchy) would be:
> > > > > > >
> > > > > > > ./README.md
> > > > > > > ./metron-analytics/README.md <== (need file here)
> > > > > > > ./metron-analytics/metron-maas-service/README.md
> > > > > > > ./metron-analytics/metron-profiler/README.md
> > > > > > > ./metron-analytics/metron-profiler-client/README.md
> > > > > > > ./metron-analytics/metron-statistics/README.md
> > > > > > > ./metron-deployment/README.md
> > > > > > > ./metron-deployment/amazon-ec2/README.md
> > > > > > > ./metron-deployment/packaging/README.md <== (need file
> > > here)
> > > > > > > ./metron-deployment/packaging/ambari/README.md <==
> (need
> > > file
> > > > > here)
> > > > > > > ./metron-deployment/packaging/
> > > docker/ansible-docker/README.md
> > > > > > > ./metron-deployment/packaging/
> > docker/rpm-docker/README.md
> > > > > > > ./metron-deployment/packer-build/README.md
> > > > > > > ./metron-deployment/roles/ <== (need file here)
> > > > > > > ./metron-deployment/roles/kibana/README.md
> > > > > > > ./metron-deployment/roles/monit/README.md
> > > > > > > ./metron-deployment/roles/opentaxii/README.md
> > > > > > > ./metron-deployment/roles/pcap_replay/README.md
> > > > > > > ./metron-deployment/roles/sensor-test-mode/README.md
> > > > > > > ./metron-deployment/vagrant/README.md <== (need file
> > here)
> > > > > > > ./metron-deployment/vagrant/codelab-platform/README.md
> > > > > > > ./metron-deployment/vagrant/
> > fastcapa-test-platform/README.
> > > md
> > > > > > > ./metron-deployment/vagrant/
> full-dev-platform/README.md
> > > > > > > ./metron-deployment/vagrant/
> quick-dev-platform/README.md
> > > > > > > ./metron-platform/README.md
> > > > > > > ./metron-platform/metron-api/README.md
> > > > > > > ./metron-platform/metron-common/README.md
> > > > > > > ./metron-platform/metron-data-management/README.md
> > > > > > > ./metron-platform/metron-enrichment/README.md
> > > > > > > ./metron-platform/metron-indexing/README.md
> > > > > > > ./metron-platform/metron-management/README.md
> > > > > > > ./metron-platform/metron-parsers/README.md
> > > > > > > ./metron-platform/metron-pcap-backend/README.md
> > > > > > > ./metron-sensors/README.md <== (need file here)
> > > > > > > ./metron-sensors/fastcapa/README.md
> > > > > > > ./metron-sensors/pycapa/README.md
> > > > > > >
> > > > > > > 2. Arrange to run this script as part of the build
> > > process, and
> > > > > commit
> > > > > > the
> > > > > > > resulting hierarchy list to someplace in the versioned
> > and
> > > > branched
> > > > > > ./site/
> > > > > > > subdirectory.
> > > > > > >
> > > > > > > 3. Produce a "doc reader" web page that takes in this
> > > hierarchy
> > > > of
> > > > > .md
> > > > > > > pages, and presents a LHS doc tree of links, and a
main
> > > display
> > > > > area for
> > > > > > a
> > > > > > > currently selected file. If we want to get fancy, this
> > > page would
> > > > > also
> > > > > > > provide: (a) telescoping (collapse/expand) of the doc
> > > tree; (b)
> > > > > floating
> > > > > > > next/prev/up/home buttons in the display area.
> > > > > > >
> > > > > > > #4. Add to this web page a pull-down menu that selects
> > > among all
> > > > > the
> > > > > > > release versions of Metron, and (if not running in the
> > > Apache
> > > > > site) a
> > > > > > > SNAPSHOT version for the current filesystem version
> (for
> > > > developer
> > > > > > > preview). Let it re-write the file paths per release
> > > version to
> > > > the
> > > > > > proper
> > > > > > > release tag in github. This web page will therefore be
> > > > > > version-independent.
> > > > > > > Put it in the asf-site branch of the Apache site, as
> the
> > > new
> > > > "docs"
> > > > > > > sub-site from the home web page. Update the list of
> > > releases at
> > > > > each
> > > > > > > release, or if we want to get fancy, teach it to read
> the
> > > release
> > > > > tags
> > > > > > from
> > > > > > > github.
> > > > > > >
> > > > > > > 5. As part of the release process, the release manager
> > (a)
> > > > assures
> > > > > the
> > > > > > > release is tagged in github with a consistent naming
> > > convention,
> > > > > and (b)
> > > > > > > submits the new hierarchy of links to google search
> > > (there's an
> > > > > api for
> > > > > > > that).
> > > > > > >
> > > > > > > 6. Deprecate the use of cwiki for anything but
> long-lived
> > > > > > > demonstrations/tutorials that are unlikely to go
> > obsolete.
> > > > > > >
> > > > > > >
> > > > > > > Do folks feel this would be a good contribution to the
> > > > visibility,
> > > > > > > timeliness, and usability of our docs?
> > > > > > > Is this an adequate solution for the current problems?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > --Matt
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Nick Allen <n...@nickallen.org>
> > > > > >
> > > > > --
> > > > >
> > > > > Jon
> > > > >
> > > > > Sent from my mobile device
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > >
> > > Jon
> > >
> > > Sent from my mobile device
> > >
> > >
> > >
> > >
> >
>
