andygrove opened a new pull request, #6322: URL: https://github.com/apache/datafusion-comet/pull/6322
## Which issue does this PR close? No issue; this is documentation only. ## Rationale for this change The architecture diagram on the home page, under "Tight integration with Apache DataFusion", is old and hard to follow. It draws the JVM and native operators as two parallel stacks joined by solid, dashed and dotted arrows that each mean something different, with callouts attached by leader lines. It also doesn't show what the paragraph above it says it shows: the Comet plugin intercepting Spark's physical plan, translating the supported operators to protobuf, and handing them to DataFusion. This replaces it with a deliberately simple diagram of that flow, in the same style as the executor memory diagram from #6237. It is not meant to be complete. For example, "Anything else keeps running in Spark" reads as falling back one operator at a time, while a query stage that contains an unsupported operator falls back as a whole (see [Spark Operator Support](https://datafusion.apache.org/comet/user-guide/latest/operators.html)). The "How Comet works" page linked under the diagram carries the precise version. ## What changes are included in this PR? - `docs/source/_static/images/comet-overview.svg`: a new hand-written SVG ([rendered](https://github.com/andygrove/datafusion-comet/blob/a8f25f63b437a07a25d6248b4afe946eac1a1e61/docs/source/_static/images/comet-overview.svg)). It has three layers. Apache Spark plans the query as usual, the Comet plugin replaces the operators it supports with native ones, and the Apache DataFusion native engine runs them, reading Parquet files and Iceberg tables directly. The plan goes down the left side, first as Spark's physical plan and then as the protobuf native plan, and the results come back up the right side to Spark as Arrow columnar batches. The colours follow the existing site diagrams, yellow for Spark and green for Comet, and each layer is tagged as JVM or native. - `docs/source/index.md`: the home page figure now uses the SVG, with its dimensions and alt text updated. The SVG has a transparent background, like the PNG it replaces. The home page figure card is white in both site themes, and the theme dims every image in dark mode, so an opaque white background would show up there as a grey box inside the card. The old `comet-overview.png` is no longer referenced but is left in the tree; it can be removed in a follow-up. ## How are these changes tested? Documentation only, no code paths touched. - Built the site locally with Sphinx from main, with and without this change. Both builds report the same 63 warnings, all of which main already has. - `npx prettier@latest --check docs/source/index.md` passes. - Rendered the SVG in headless Chrome, including with a deliberately wide fallback font: an SVG embedded as an image cannot use the site's web fonts, so most readers get a system font. I also screenshotted the built home page in the light and dark themes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
