[ https://issues.apache.org/jira/browse/SPARK-41791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-41791: ------------------------------------ Assignee: (was: Apache Spark) > Create distinct metadata attributes for metadata that is constant or file and > metadata that is generated during the scan > ------------------------------------------------------------------------------------------------------------------------ > > Key: SPARK-41791 > URL: https://issues.apache.org/jira/browse/SPARK-41791 > Project: Spark > Issue Type: Improvement > Components: Optimizer > Affects Versions: 3.3.1 > Reporter: Jan-Ole Sasse > Priority: Major > Fix For: 3.4.0 > > > There are two types or Metadata in Spark > * Metadata that is constant per file (file_name, file_size, ...) > * Metadata that is not contant (currently only row_index) > The two types are generated differently > * File constant metadata is appended to the output after scan > * non-constant metadata is generated during the scan > The proposal here is to create different metadata attributes to distinguish > those different types throughout the code -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org