In any event, what you initially proposed sounds like scoped and localized? I think you can just go ahead with a PR
On Tue, 11 Aug 2026 at 08:52, Hyukjin Kwon <[email protected]> wrote: > For now, I am thinking about adding usage examples as long as they are > correct .. even if they are very simple. > I would like to make the contributions very easy so if any contributor has > some expertise in some domain, they could easily add some examples. > And maybe later, based on the examples, we could host a site after > categorizing it. > > Basically I would like to have the swarm bunch of correct, > recommended examples first, and leverage it. The logic behind this is that > basically agents would think those examples in the official repo as correct > and recommended ones so I hope this eventually makes agents work better > with users and dev. > > I am still figuring out the details :-). > > > On Mon, 10 Aug 2026 at 22:26, Nicholas Chammas <[email protected]> > wrote: > >> Are you describing more narrative documentation, or simply adding usage >> examples to the reference documentation? >> >> I don’t know if these are universal terms, but to clarify what I mean: >> - Narrative documentation: tutorials, guides, etc. that walk a user >> through what to do and why; often includes general advice and use-case >> specific guidance; e.g. Programming Guides >> <https://spark.apache.org/docs/latest/sql-programming-guide.html> >> - Reference documentation: comprehensive documentation of what exists and >> how to use it; usually more neutral in tone with a consistent structure; >> e.g. API Docs >> <https://spark.apache.org/docs/latest/api/python/reference/index.html> >> >> In any case, I think that’s a good idea and I look forward to the >> discussion and/or SPIP. >> >> >> On Aug 9, 2026, at 6:47 PM, Hyukjin Kwon <[email protected]> wrote: >> >> Hey Nicholas, thanks for taking a look at this. >> >> I was thinking about writing some, not only for dev but also writing best >> practices for users. >> For example, avoid repeating withColumns in a loop (it is similar to your >> code smell suggestion but the instructions for users), and I was actually >> thinking about proposing an SPIP. >> >> Wonder if we can prepare this in two tracks, one for dev, and one for >> users. >> >> On Sat, 8 Aug 2026 at 05:08, Xiao Li <[email protected]> wrote: >> >>> Thank you for contributing to it, Nicholas! >>> >>> I also recently discussed best practices and code examples with @Hyukjin >>> Kwon <[email protected]> . We should provide more official examples >>> for our APIs and features to make them easier for both humans and AI to >>> understand and adopt. Hyukjin is also planning to share more thoughts with >>> the community soon. >>> >>> Nicholas Chammas <[email protected]> 于2026年8月7日周五 09:05写道： >>> >>>> I'd like to share some general documentation nitpicks in the hopes that >>>> we can address them as a community. >>>> >>>> None of these issues are a big problem -- again, they are nitpicks! -- >>>> but they do have a notable impact on our documentation when considered all >>>> together. >>>> >>>> I will submit some patches to chip away at these problems. Hopefully, >>>> some of you reading this will join me. >>>> Agent instructions should not compensate for documentation deficiencies >>>> >>>> We should instead correct the deficiencies "upstream" in the main >>>> documentation so that both humans and AI agents can benefit. >>>> >>>> Some instructions in AGENTS.md >>>> <https://github.com/apache/spark/blob/1226c5ccc1cf87362136d4e10332e35e80640cd9/AGENTS.md> >>>> look >>>> like documentation smells <https://c2.com/xp/CodeSmell.html> to me. >>>> For example, the guidance on how to select a test base class or how to run >>>> tests seems like it should already be captured well for humans, no? And if >>>> it isn't, then shouldn't we fix the documentation "for humans" first and >>>> just point agents to it? >>>> >>>> Likewise, some of the guidance under "Pull Request Workflow" seems like >>>> it should instead live in our general contributing guide >>>> <https://spark.apache.org/contributing.html>. The instructions on how >>>> to label PRs with components, in particular, seem more up-to-date for >>>> agents than they are for humans! >>>> A page should only have one title >>>> >>>> That means the rendered HTML on any given page should only contain one >>>> h1 tag >>>> <https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Elements/Heading_Elements#avoid_using_multiple_h1_elements_on_one_page>. >>>> This is something we get wrong on many pages. We often use # as a >>>> section heading, which translates to h1. We should instead use ##. >>>> >>>> Another way we get this wrong is by adding a title to pages that >>>> already have a title specified in the YAML frontmatter >>>> <https://github.com/apache/spark/blob/1226c5ccc1cf87362136d4e10332e35e80640cd9/docs/hardware-provisioning.md?plain=1#L3>. >>>> Jekyll generates a title automatically using this information, so we don't >>>> need to manually add a title on top of that. >>>> >>>> I don't plan to try fixing all of our headings, but I would like us to >>>> be aware of this issue so we stop adding to it. I am looking into adding a >>>> Markdown linter that can help us with this. >>>> Don't use Liquid tags for syntax highlighting >>>> >>>> Instead, use the Markdown code fences that we're all already familiar >>>> with. >>>> >>>> For some reason, perhaps simply due to inertia, we have many uses of Liquid >>>> tags for syntax highlighting >>>> <https://github.com/apache/spark/blob/1226c5ccc1cf87362136d4e10332e35e80640cd9/docs/mllib-data-types.md?plain=1#L61-L74>. >>>> In addition to being more verbose than Markdown code fences, Liquid tags >>>> confuse most editors and IDEs. Note how GitHub interprets the Python code >>>> comments in that example >>>> <https://github.com/apache/spark/blob/1226c5ccc1cf87362136d4e10332e35e80640cd9/docs/mllib-data-types.md?plain=1#L61-L74> >>>> as >>>> headings because it doesn't recognize Liquid tags. My IDE does the same >>>> thing. >>>> >>> >>
