Hey Nicholas, thanks for taking a look at this. I was thinking about writing some, not only for dev but also writing best practices for users. For example, avoid repeating withColumns in a loop (it is similar to your code smell suggestion but the instructions for users), and I was actually thinking about proposing an SPIP.
Wonder if we can prepare this in two tracks, one for dev, and one for users. On Sat, 8 Aug 2026 at 05:08, Xiao Li <[email protected]> wrote: > Thank you for contributing to it, Nicholas! > > I also recently discussed best practices and code examples with @Hyukjin > Kwon <[email protected]> . We should provide more official examples > for our APIs and features to make them easier for both humans and AI to > understand and adopt. Hyukjin is also planning to share more thoughts with > the community soon. > > Nicholas Chammas <[email protected]> 于2026年8月7日周五 09:05写道： > >> I'd like to share some general documentation nitpicks in the hopes that >> we can address them as a community. >> >> None of these issues are a big problem -- again, they are nitpicks! -- >> but they do have a notable impact on our documentation when considered all >> together. >> >> I will submit some patches to chip away at these problems. Hopefully, >> some of you reading this will join me. >> Agent instructions should not compensate for documentation deficiencies >> >> We should instead correct the deficiencies "upstream" in the main >> documentation so that both humans and AI agents can benefit. >> >> Some instructions in AGENTS.md >> <https://github.com/apache/spark/blob/1226c5ccc1cf87362136d4e10332e35e80640cd9/AGENTS.md> >> look >> like documentation smells <https://c2.com/xp/CodeSmell.html> to me. For >> example, the guidance on how to select a test base class or how to run >> tests seems like it should already be captured well for humans, no? And if >> it isn't, then shouldn't we fix the documentation "for humans" first and >> just point agents to it? >> >> Likewise, some of the guidance under "Pull Request Workflow" seems like >> it should instead live in our general contributing guide >> <https://spark.apache.org/contributing.html>. The instructions on how to >> label PRs with components, in particular, seem more up-to-date for agents >> than they are for humans! >> A page should only have one title >> >> That means the rendered HTML on any given page should only contain one h1 >> tag >> <https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/Heading_Elements#avoid_using_multiple_h1_elements_on_one_page>. >> This is something we get wrong on many pages. We often use # as a >> section heading, which translates to h1. We should instead use ##. >> >> Another way we get this wrong is by adding a title to pages that already >> have a title specified in the YAML frontmatter >> <https://github.com/apache/spark/blob/1226c5ccc1cf87362136d4e10332e35e80640cd9/docs/hardware-provisioning.md?plain=1#L3>. >> Jekyll generates a title automatically using this information, so we don't >> need to manually add a title on top of that. >> >> I don't plan to try fixing all of our headings, but I would like us to be >> aware of this issue so we stop adding to it. I am looking into adding a >> Markdown linter that can help us with this. >> Don't use Liquid tags for syntax highlighting >> >> Instead, use the Markdown code fences that we're all already familiar >> with. >> >> For some reason, perhaps simply due to inertia, we have many uses of Liquid >> tags for syntax highlighting >> <https://github.com/apache/spark/blob/1226c5ccc1cf87362136d4e10332e35e80640cd9/docs/mllib-data-types.md?plain=1#L61-L74>. >> In addition to being more verbose than Markdown code fences, Liquid tags >> confuse most editors and IDEs. Note how GitHub interprets the Python code >> comments in that example >> <https://github.com/apache/spark/blob/1226c5ccc1cf87362136d4e10332e35e80640cd9/docs/mllib-data-types.md?plain=1#L61-L74> >> as >> headings because it doesn't recognize Liquid tags. My IDE does the same >> thing. >> >
