Hi all, We would like to start a discussion on introducing a new SparkSQL data type called FILE for providing better support for unstructured or large data in Spark.
Jira: SPARK-59132 <https://issues.apache.org/jira/browse/SPARK-59132> Brief context: We want to make it easy and cheap to work with files, such as PDFs, images, audio and video, from tables using Spark. Today you either copy the bytes of a file into a table as a binary column, or you keep the path to the file as a string and hope everything downstream knows what to do with it. We propose a new data type, FileType (spelled FILE in SQL), which holds the path of a file and a small amount of metadata describing it. The engine, the file format, and user defined functions all understand this type, so a query can filter, join and project files without reading their contents, and load the bytes only where it actually needs them. Please find more information in the SPIP <https://docs.google.com/document/d/1pPof896ZwcZ-2Yn-YhC4TbyYWn1umiQJykGDkAxzCWc/edit?tab=t.0> . We've recently introduced a FILE type in Parquet <https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#file> to provide a consistent representation across engines and table formats to access large blobs. This SPIP provides the engine interfaces and implementation to leverage this representation. Looking forward to your feedback! Best regards, Burak Yavuz and Dejan Krakovic
