Are you describing more narrative documentation, or simply adding usage examples to the reference documentation?
I don’t know if these are universal terms, but to clarify what I mean: - Narrative documentation: tutorials, guides, etc. that walk a user through what to do and why; often includes general advice and use-case specific guidance; e.g. Programming Guides <https://spark.apache.org/docs/latest/sql-programming-guide.html> - Reference documentation: comprehensive documentation of what exists and how to use it; usually more neutral in tone with a consistent structure; e.g. API Docs <https://spark.apache.org/docs/latest/api/python/reference/index.html> In any case, I think that’s a good idea and I look forward to the discussion and/or SPIP. > On Aug 9, 2026, at 6:47 PM, Hyukjin Kwon <[email protected]> wrote: > > Hey Nicholas, thanks for taking a look at this. > > I was thinking about writing some, not only for dev but also writing best > practices for users. > For example, avoid repeating withColumns in a loop (it is similar to your > code smell suggestion but the instructions for users), and I was actually > thinking about proposing an SPIP. > > Wonder if we can prepare this in two tracks, one for dev, and one for users. > > On Sat, 8 Aug 2026 at 05:08, Xiao Li <[email protected] > <mailto:[email protected]>> wrote: >> Thank you for contributing to it, Nicholas! >> >> I also recently discussed best practices and code examples with @Hyukjin >> Kwon <mailto:[email protected]> . We should provide more official >> examples for our APIs and features to make them easier for both humans and >> AI to understand and adopt. Hyukjin is also planning to share more thoughts >> with the community soon. >> >> Nicholas Chammas <[email protected] >> <mailto:[email protected]>> 于2026年8月7日周五 09:05写道： >>> I'd like to share some general documentation nitpicks in the hopes that we >>> can address them as a community. >>> >>> None of these issues are a big problem -- again, they are nitpicks! -- but >>> they do have a notable impact on our documentation when considered all >>> together. >>> >>> I will submit some patches to chip away at these problems. Hopefully, some >>> of you reading this will join me. >>> >>> Agent instructions should not compensate for documentation deficiencies >>> >>> We should instead correct the deficiencies "upstream" in the main >>> documentation so that both humans and AI agents can benefit. >>> >>> Some instructions in AGENTS.md >>> <https://github.com/apache/spark/blob/1226c5ccc1cf87362136d4e10332e35e80640cd9/AGENTS.md> >>> look like documentation smells <https://c2.com/xp/CodeSmell.html> to me. >>> For example, the guidance on how to select a test base class or how to run >>> tests seems like it should already be captured well for humans, no? And if >>> it isn't, then shouldn't we fix the documentation "for humans" first and >>> just point agents to it? >>> >>> Likewise, some of the guidance under "Pull Request Workflow" seems like it >>> should instead live in our general contributing guide >>> <https://spark.apache.org/contributing.html>. The instructions on how to >>> label PRs with components, in particular, seem more up-to-date for agents >>> than they are for humans! >>> >>> A page should only have one title >>> >>> That means the rendered HTML on any given page should only contain one h1 >>> tag >>> <https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/Heading_Elements#avoid_using_multiple_h1_elements_on_one_page>. >>> This is something we get wrong on many pages. We often use # as a section >>> heading, which translates to h1. We should instead use ##. >>> >>> Another way we get this wrong is by adding a title to pages that already >>> have a title specified in the YAML frontmatter >>> <https://github.com/apache/spark/blob/1226c5ccc1cf87362136d4e10332e35e80640cd9/docs/hardware-provisioning.md?plain=1#L3>. >>> Jekyll generates a title automatically using this information, so we don't >>> need to manually add a title on top of that. >>> >>> I don't plan to try fixing all of our headings, but I would like us to be >>> aware of this issue so we stop adding to it. I am looking into adding a >>> Markdown linter that can help us with this. >>> >>> Don't use Liquid tags for syntax highlighting >>> >>> Instead, use the Markdown code fences that we're all already familiar with. >>> >>> For some reason, perhaps simply due to inertia, we have many uses of Liquid >>> tags for syntax highlighting >>> <https://github.com/apache/spark/blob/1226c5ccc1cf87362136d4e10332e35e80640cd9/docs/mllib-data-types.md?plain=1#L61-L74>. >>> In addition to being more verbose than Markdown code fences, Liquid tags >>> confuse most editors and IDEs. Note how GitHub interprets the Python code >>> comments in that example >>> <https://github.com/apache/spark/blob/1226c5ccc1cf87362136d4e10332e35e80640cd9/docs/mllib-data-types.md?plain=1#L61-L74> >>> as headings because it doesn't recognize Liquid tags. My IDE does the same >>> thing. >>>
