seanpavlak opened a new pull request, #59839:
URL: https://github.com/apache/spark/pull/59839
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR:
https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g.,
'[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a
faster review.
7. If you want to add a new configuration, please read the guideline first
for naming configurations in
'common/utils/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the
guideline first in
'common/utils/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
Add a built-in fixed-width file data source, format "fwf", with
spark.read.fwf and DataFrame.write.fwf. It follows the existing CSV, JSON, and
XML shortcuts and includes the V1 and V2 file source paths, Scala, Java,
Python, and Spark Connect APIs, docs, and examples.
Reading supports explicit colspecs or widths, optional inference of column
positions, header, comment, skipRows, nullValue, and encoding, plus PERMISSIVE,
DROPMALFORMED, and FAILFAST parse modes. Writing requires explicit colspecs or
widths. A value longer than its field width is truncated; a shorter value is
padded on the right with spaces.
Geometry, geography, variant, interval, and nested types are rejected when
the schema is resolved. Filter pushdown is enabled by default through
spark.sql.fwf.filterPushdown.enabled.
### Why are the changes needed?
Fixed-width files are a common input format and Spark has no built-in data
source for them. The use case is the same as spark.read.csv: load a file or
directory of line-oriented records without first converting them to CSV.
SPARK-36392 is pandas API on Spark read_fwf only. This ticket, SPARK-60589,
is the SQL data source.
### Does this PR introduce _any_ user-facing change?
Yes. This adds a new data source on unreleased master (5.0.0).
spark.read.fwf and DataFrame.write.fwf are new APIs, as are format "fwf" and
the fixed-width options documented in docs/sql-data-sources-fwf.md. Existing
formats are unchanged.
Example:
```scala
val df = spark.read.option("widths", "10,4").option("header",
"true").fwf(path)
df.write.option("widths", "10,4").fwf("output")
```
### How was this patch tested?
Added FixedWidthOptionsSuite, FixedWidthUtilsSuite, and FixedWidthSuite. On
current master those passed: 27 catalyst tests and 24 FixedWidthSuite tests.
Coverage includes read, write, round-trip, schema inference, parse modes,
corrupt records, skipRows, column pruning, filter pushdown, and rejection of
geometry, geography, variant, and interval types.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor Grok 4.7
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
