For now, I am thinking about adding usage examples as long as they are correct .. even if they are very simple. I would like to make the contributions very easy so if any contributor has some expertise in some domain, they could easily add some examples. And maybe later, based on the examples, we could host a site after categorizing it.
Basically I would like to have the swarm bunch of correct, recommended examples first, and leverage it. The logic behind this is that basically agents would think those examples in the official repo as correct and recommended ones so I hope this eventually makes agents work better with users and dev. I am still figuring out the details :-). On Mon, 10 Aug 2026 at 22:26, Nicholas Chammas <[email protected]> wrote: > Are you describing more narrative documentation, or simply adding usage > examples to the reference documentation? > > I don’t know if these are universal terms, but to clarify what I mean: > - Narrative documentation: tutorials, guides, etc. that walk a user > through what to do and why; often includes general advice and use-case > specific guidance; e.g. Programming Guides > <https://spark.apache.org/docs/latest/sql-programming-guide.html> > - Reference documentation: comprehensive documentation of what exists and > how to use it; usually more neutral in tone with a consistent structure; > e.g. API Docs > <https://spark.apache.org/docs/latest/api/python/reference/index.html> > > In any case, I think that’s a good idea and I look forward to the > discussion and/or SPIP. > > > On Aug 9, 2026, at 6:47 PM, Hyukjin Kwon <[email protected]> wrote: > > Hey Nicholas, thanks for taking a look at this. > > I was thinking about writing some, not only for dev but also writing best > practices for users. > For example, avoid repeating withColumns in a loop (it is similar to your > code smell suggestion but the instructions for users), and I was actually > thinking about proposing an SPIP. > > Wonder if we can prepare this in two tracks, one for dev, and one for > users. > > On Sat, 8 Aug 2026 at 05:08, Xiao Li <[email protected]> wrote: > >> Thank you for contributing to it, Nicholas! >> >> I also recently discussed best practices and code examples with @Hyukjin >> Kwon <[email protected]> . We should provide more official examples >> for our APIs and features to make them easier for both humans and AI to >> understand and adopt. Hyukjin is also planning to share more thoughts with >> the community soon. >> >> Nicholas Chammas <[email protected]> 于2026年8月7日周五 09:05写道： >> >>> I'd like to share some general documentation nitpicks in the hopes that >>> we can address them as a community. >>> >>> None of these issues are a big problem -- again, they are nitpicks! -- >>> but they do have a notable impact on our documentation when considered all >>> together. >>> >>> I will submit some patches to chip away at these problems. Hopefully, >>> some of you reading this will join me. >>> Agent instructions should not compensate for documentation deficiencies >>> >>> We should instead correct the deficiencies "upstream" in the main >>> documentation so that both humans and AI agents can benefit. >>> >>> Some instructions in AGENTS.md >>> <https://github.com/apache/spark/blob/1226c5ccc1cf87362136d4e10332e35e80640cd9/AGENTS.md> >>> look >>> like documentation smells <https://c2.com/xp/CodeSmell.html> to me. For >>> example, the guidance on how to select a test base class or how to run >>> tests seems like it should already be captured well for humans, no? And if >>> it isn't, then shouldn't we fix the documentation "for humans" first and >>> just point agents to it? >>> >>> Likewise, some of the guidance under "Pull Request Workflow" seems like >>> it should instead live in our general contributing guide >>> <https://spark.apache.org/contributing.html>. The instructions on how >>> to label PRs with components, in particular, seem more up-to-date for >>> agents than they are for humans! >>> A page should only have one title >>> >>> That means the rendered HTML on any given page should only contain one >>> h1 tag >>> <https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/Heading_Elements#avoid_using_multiple_h1_elements_on_one_page>. >>> This is something we get wrong on many pages. We often use # as a >>> section heading, which translates to h1. We should instead use ##. >>> >>> Another way we get this wrong is by adding a title to pages that already >>> have a title specified in the YAML frontmatter >>> <https://github.com/apache/spark/blob/1226c5ccc1cf87362136d4e10332e35e80640cd9/docs/hardware-provisioning.md?plain=1#L3>. >>> Jekyll generates a title automatically using this information, so we don't >>> need to manually add a title on top of that. >>> >>> I don't plan to try fixing all of our headings, but I would like us to >>> be aware of this issue so we stop adding to it. I am looking into adding a >>> Markdown linter that can help us with this. >>> Don't use Liquid tags for syntax highlighting >>> >>> Instead, use the Markdown code fences that we're all already familiar >>> with. >>> >>> For some reason, perhaps simply due to inertia, we have many uses of Liquid >>> tags for syntax highlighting >>> <https://github.com/apache/spark/blob/1226c5ccc1cf87362136d4e10332e35e80640cd9/docs/mllib-data-types.md?plain=1#L61-L74>. >>> In addition to being more verbose than Markdown code fences, Liquid tags >>> confuse most editors and IDEs. Note how GitHub interprets the Python code >>> comments in that example >>> <https://github.com/apache/spark/blob/1226c5ccc1cf87362136d4e10332e35e80640cd9/docs/mllib-data-types.md?plain=1#L61-L74> >>> as >>> headings because it doesn't recognize Liquid tags. My IDE does the same >>> thing. >>> >> >
