Hello All,
For a while now, we have been exploring different tools to enable the creation of a dazzling, professionally-impressive documentation library. I am glad Andy mentioned *Confluence*, because much of my points are centered on this solution (you will soon know why). What we need is to initiate a docs organization in minutes, and subsequently, establish formal processes to enable an optimal document development lifecycle (DDLC). It is noteworthy that the *Apache community already maintains a Confluence wiki *for many of its projects. See here: https://cwiki.apache.org/confluence/dashboard.action Confluence looks like the way to go, looking ahead. *What Confluence does * Confluence supports ALL OF THE GOALS THAT ANDY LISTED AND A LOT MORE. Here are the top reasons why I am in favor of Confluence: - Simple, uncomplicated documentation process A documentation platform should be ready within minutes. With Confluence, documentation (assuming that doc strategy and structure are in place) involves creating a page, writing on to this page, and saving the page. Much like a blog experience! - A documentation tool MUST HAVE superior style support. For example, pre-defined styles for a variety of code blocks (JSON, Python, Shell, the works). Confluence has supports styling using master CSS and page-level CSS. Additionally, from an author's perspective, it has a very powerful WYSIWYG editor, which is much, much smarter than what you see in Word 2013. It also has a “backend” HTML editor. - A very powerful feature in Confluence is the ability to create plug-ins (for example, a plug-in for creating a filter), and integrate it with your Confluence instance. Confluence has a library of thousands of such plug-ins created by contributors. Many are available freely. See here: https://marketplace.atlassian.com/plugins/app/confluence/popular I will truly urge you to go through the list to see just how enabling a couple can make your Confluence instance smart as hell. - A documentation tool MUST support complete abstraction of backend complexities. Think of our platform and how it handles operability. Authors SHOULD NEVER BE required to style manually, use scripts for content control, and much less, use markdown (or markup) for actual content. The tool should be able to AUTOMATICALLY TRANSLATE content to a markdown or markup flavor (the source content), while making the source content available, whenever. Confluence stores all your content, written as plain text, into an XML chunk that contains text, as well as all the style specifications. At any time, you can export this XML, import it elsewhere, and get going with your documentation. - Customization is a concern as far as professional quality documentation goes. A doc portal MUST sit in with an organization’s style constraints while matching in terms of themes, fonts, paragraph styles, and others, with what the organization has already adopted. Confluence can be tweaked as much as you want to best match your color/style themes. It also has a plug-in for customization. See: http://www.adaptavist.com/w/products-plugins/premium/themebuilder/ - The Confluence search logic depends on Lucene. See this article: https://wiki.nci.nih.gov/display/WikiTrainFAQsTips/How+Confluence+search+works Additionally, Confluence permits rigging of search results. For example, you can tweak Confluence such that a term (for example, operator) will throw up specific pages as initial hits (think of this as a powerful influencer that can pique your readers interests in reading release notes, ultimately translating to an upgrade decision). - A documentation tool should be SEO-compliant. Documentation is a selling point for products, especially new ones. That is why the tool should support SEO techniques inherently, without making it the headache of the tool users. See this macro in Confluence (or plug-in): https://marketplace.atlassian.com/plugins/com.playsql.seo-manager Also see: http://www.adaptavist.com/w/products-plugins/premium/themebuilder/ - Permission settings ensure that you can control the availability of content to readers. This means that you can choose to hide some pages, make some visible, some others available for editing, and so on. What this means is, we can hide pages that are WIP, and at the time of a release, make them available. All of this takes only a couple clicks in Confluence. - Context-sensitive help is another plus point. See this blog: http://blogs.atlassian.com/2007/12/using_a_wiki_fo/ *Here are the caveats that Andy listed, with my comments in line* - Any updates to the docs in the repos require a manual build and push to the website. With Confluence, we need none of this. Documenting is as easy as blogging. - This would be a custom solution, and while I will rely on existing modules as much as possible, there will be a little glue-code. You require no code-specific assistance to build a documentation portal, if you use Confluence. - The JSON files used by the search feature may become very large if the docs become extremely large. This may hurt browser performance. I doubt this will ever be a problem though. The search intricacies are entirely handled by Confluence. If assistance is ever felt, they have a team ready 24*7. *Here are the “cons” that Andy listed, with my comments in line*: - Docs not versioned along with the code (readthedocs is the exception here) You do not require code, and you need not do anything special to create versions. You only need to specify the “front-end” version that matches with your release version and create supported documentation. Confluence handles multiple revisions, and also supports one-click reverts to earlier versions. - Most solutions are not very user-friendly (no real-time search) - Not so for Confluence. I explained earlier how Confluence treats search. - CSS-skinning is difficult or impossible Again, not with Confluence. If customization is an issue, we can use the theme builder plug-in for this. - System exists outside of current projects/website While the actual server might reside elsewhere, you can host all your content in a place that you want. All you need to do is to change the base URL of your Confluence server to something such as help.datatorrent.com. *Here are some other cons that Andy noticed, along with my comments*: - Separately hosted Confluence can be hosted on-premise, in cloud, or at a colo (such as Contegix). While the physical location might be different from that of our server, all the content can be available at a URL we specify. - Learning curve for usage Supremely simple to use, at a basic level. Confluence can be developed to a very powerful platform that can ultimately transcend to a collaborative platform. - Difficulty for skinning See my comments on the Theme Builder ( http://www.adaptavist.com/w/products-plugins/premium/themebuilder/). - Difficulty extending/adding functionality Not Confluence! See my earlier comment on plug-ins. - Slow search/frustrating left-hand navigation Not at all Confluence! See my earlier comments on search and Confluence. I understand that you will also want to know how a typical documentation process looks like in a Confluence environment. This email is long, and I am refraining from adding it here. I will however, describe the process in brief in another email shortly. Please let me know if you have more questions around Confluence, and the documentation approach in the wiki world. I will be happy to answer them. Thanks kindly, ~ Amruta On Thu, Nov 19, 2015 at 2:45 AM, Andy Perlitch <[email protected]> wrote: > Sorry, seems like the images didn't get through. > > - index page for the docs: http://i.imgur.com/ZB8bzhO.png > - individual doc example: http://i.imgur.com/BcGaIEQ.png > - realtime searching (taken from angular docs as example) > http://i.imgur.com/K6247Q9.gif > > > Andy > > On Mon, Nov 16, 2015 at 7:46 PM, Andy Perlitch <[email protected]> > wrote: > >> Hello Apexers, >> >> I'd like to propose a process for creating, storing, updating, and >> displaying documentation of the apex and malhar projects. >> >> >> *## Goals* >> The goals for docs that I had in mind are: >> >> - Versioned with the code base >> - Easy to contribute to by anyone >> - Displayed on apex.incubator.apache.org >> - Simple deploy process >> - Indexable by search engines >> - Easily navigable; table of contents, support for nested "groups" of >> docs, responsive search >> >> *## Proposal* >> My proposal is as follows: >> >> 1. Source files for docs will live in their respective git repositories, >> under a `docs` folder. Format for said files will be github-flavored >> markdown <https://help.github.com/articles/github-flavored-markdown/> (In >> the future, other formats could be supported). This takes care of the first >> two goals listed above. >> 2. In addition to the docs themselves, a single json file acting as an >> index for the rendered docs will exist in `docs/index.json`. This will be >> used to render a left navigation bar in the docs section. >> 3. Create a gulp <http://gulpjs.com/> task which: >> a. pulls down these files for each released version of apex and malhar >> (using the Github API) >> b. creates HTML versions of each project/version/doc, each with the >> same left-side navigation >> c. creates a JSON file (per version) containing the raw markdown >> content that can be used by Javascript on the page to power a fast search >> function in the left-side navigation. >> 4. As part of the release process, run the aforementioned gulp task and >> push to the site using the current `build.sh` script >> >> >> *## Mockups* >> This would result in an index page on >> *apex.incubator.apache.org/docs/index.html >> <http://apex.incubator.apache.org/docs/index.html>*: >> [image: Inline image 1] >> >> And this is a mockup of looking at an individual doc, e.g. >> *apex.incubator.apache.org/docs/v3.2.0-incubating/contributing.html >> <http://apex.incubator.apache.org/docs/v3.2.0-incubating/contributing.html>* >> : >> [image: Inline image 2] >> >> The search feature could filter the docs shown in the side bar in >> real-time: >> [image: Inline image 3] >> >> *## Caveats* >> No plan is perfect, and this one is no exception. A few caveats to this >> plan are: >> >> - Any updates to the docs in the repos require a manual build and push to >> the website. >> - This would be a custom solution, and while I will rely on existing >> modules as much as possible, there will be a little glue-code >> - The JSON files used by the search feature may become very large if the >> docs become extremely large. This may hurt browser performance. I doubt >> this will ever be a problem though. >> >> *## Alternatives* >> We could approach the problem of docs in a few different ways. Most >> notably are solutions hosted by others (i.e., Documentation as a Service). >> Some examples of this solution: Atlassian Confluence >> <https://developer.atlassian.com/opensource/>, Readme.io, and Readthedocs >> <https://readthedocs.org/>. I personally take issue with these for the >> following reasons: >> >> - docs not versioned along with the code (readthedocs is the exception >> here) >> - most solutions are not very user-friendly (no real-time search) >> - css-skinning is difficult or impossible >> - system exists outside of current projects/website >> >> If the community rejects the proposal put forth here, my choice of >> alternative would be ReadTheDocs (see the features >> <http://read-the-docs.readthedocs.org/en/latest/features.html>). >> >> *Pros of ReadTheDocs:* >> - docs are also versioned with codebase in git repo >> - git hook automatically updates docs without any manual step >> - out-of-the-box support for showing different versions of the docs >> - decent default theme >> >> *Cons of ReadTheDocs are: * >> - separately hosted >> - learning curve for usage >> - difficulty for skinning >> - difficulty extending/adding functionality >> - Slow search/frustrating left-hand navigation >> >> >> *## Conclusions* >> Documentation is one of the first places potential new community >> members/developers will go when vetting Apex for use. Ideally, it should >> dazzle and inspire. I think the status-quo for documentation on open-source >> projects is a low bar, and I am willing to put in the bit of work to lay >> the foundation for excellent docs. Again, if my custom proposal seems too >> problematic, my vote is for using ReadTheDocs. >> >> Please share your thoughts. >> >> -Andy >> >> >> -- >> Regards, >> Andy Perlitch >> Software Engineer >> DataTorrent Inc >> (408)829-9319 >> > > > > -- > Regards, > Andy Perlitch > Software Engineer > DataTorrent Inc > (408)829-9319 >
