Messages by Date
-
2026/11/18
[jira] [Updated] (SPARK-62070) Fix CHECK constraints accepting rows that violate them when they compare a CHAR column with a string literal
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62070) Fix CHECK constraints accepting rows that violate them when they compare a CHAR column with a string literal
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62070) Fix CHECK constraints accepting rows that violate them when they compare a CHAR column with a string literal
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62070) Fix CHECK constraints accepting rows that violate them when they compare a CHAR column with a string literal
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62070) Fix CHECK constraints accepting rows that violate them when they compare a CHAR column with a string literal
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62069) Fix query failures when ReorderJoin evaluates a condition after a NULL condition of a lower join
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62069) Fix query failures when ReorderJoin evaluates a condition after a NULL condition of a lower join
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-62066) [Connect][Python] Drop-in: take local rows apart by the cached names of a StructType changed in place, raise PySpark's errors of the types that proto_schema_to_pyspark_data_type does not convert, and raise the MALFORMED conditions of no parameters
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-62060) Fix CSV reader re-analyzing every existence default value for each missing column in each row
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-62060) Fix CSV reader re-analyzing every existence default value for each missing column in each row
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-62066) [Connect][Python] Drop-in: take local rows apart by the cached names of a StructType changed in place, raise PySpark's errors of the types that proto_schema_to_pyspark_data_type does not convert, and raise the MALFORMED conditions of no parameters
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-61938) Fix query errors by BooleanSimplification when an operand of AND or OR guards a throwable expression
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-61938) Fix query errors by BooleanSimplification when an operand of AND or OR guards a throwable expression
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-60775) getOrCreateDataFrame for DataFrame to allow to resuming after a failed stage
Amogh Desai (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62068) Support SparkSession.getOrCreateDataFrame in Spark Connect
Amogh Desai (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-60775) getOrCreate for Dataset / DataFrame to allow to resuming after a failed stage
Amogh Desai (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-62056) Take the arguments of the methods of the maps and repeated containers of the Rust protobuf runtime as upb's, with upb's errors
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-62056) Take the arguments of the methods of the maps and repeated containers of the Rust protobuf runtime as upb's, with upb's errors
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-62064) Send the relations of DataFrames that the extensions of expressions hold, and with the changes a plan makes to them, and resolve the messages of extensions in a time of their length, as PySpark sends its plans
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-62064) Send the relations of DataFrames that the extensions of expressions hold, and with the changes a plan makes to them, and resolve the messages of extensions in a time of their length, as PySpark sends its plans
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-62055) Fix vectorized Parquet and ORC readers re-analyzing every existence default value for each column
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-62055) Fix vectorized Parquet and ORC readers re-analyzing every existence default value for each column
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62050) Tag more DSv2-only test suites with `DataSourceV2Test`
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-62050) Tag more DSv2-only test suites with `DataSourceV2Test`
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62050) Tag more DSv2-only test suites with `DataSourceV2Test`
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62050) Tag more DSv2-only test suites with `DataSourceV2Test`
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-62050) Tag more DSv2-only test suites with `DataSourceV2Test`
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62050) Tag more DSv2-only test suites with `DataSourceV2Test`
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-59869) Update GitHub Actions to the latest approved versions
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62067) Report Python worker timing metrics in microseconds
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62067) Report Python worker timing metrics in microseconds
Haotian Sun (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-62063) Report the option keys and values that are objects, and the arguments that the writers keep, to Python's garbage collector in the readers, writers and session builder, also of their shallow copies
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-62063) Report the option keys and values that are objects, and the arguments that the writers keep, to Python's garbage collector in the readers, writers and session builder, also of their shallow copies
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-59869) Update GitHub Actions to the latest approved versions
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-59869) Update GitHub Actions to the latest approved versions
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62066) [Connect][Python] Drop-in: take local rows apart by the cached names of a StructType changed in place, raise PySpark's errors of the types that proto_schema_to_pyspark_data_type does not convert, and raise the MALFORMED conditions of no parameters
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62066) [Connect][Python] Drop-in: take local rows apart by the cached names of a StructType changed in place, raise PySpark's errors of the types that proto_schema_to_pyspark_data_type does not convert, and raise the MALFORMED conditions of no parameters
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62065) [Connect][Python] Drop-in: hold PySpark's user-defined type objects in the cached schemas of set operations and pickles, raise the conversion errors of the schema of the server where PySpark reads it, and make DataType.simpleString the typeName
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62065) [Connect][Python] Drop-in: hold PySpark's user-defined type objects in the cached schemas of set operations and pickles, raise the conversion errors of the schema of the server where PySpark reads it, and make DataType.simpleString the typeName
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-59869) Update GitHub Actions to the latest approved versions
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62064) Send the relations of DataFrames that the extensions of expressions hold, and with the changes a plan makes to them, and resolve the messages of extensions in a time of their length, as PySpark sends its plans
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62064) Send the relations of DataFrames that the extensions of expressions hold, and with the changes a plan makes to them, and resolve the messages of extensions in a time of their length, as PySpark sends its plans
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-59869) Update GitHub Actions to the latest approved versions
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-60528) Fix `ExpressionEncoder` assertion failure on nested empty structs
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-60528) Fix `ExpressionEncoder` assertion failure on nested empty structs
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-61704) Fix ConstantPropagation propagating an EqualTo with a NULL literal under a NOT
Kousuke Saruta (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-61704) Fix ConstantPropagation propagating an EqualTo with a NULL literal under a NOT
Kousuke Saruta (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62063) Report the option keys and values that are objects, and the arguments that the writers keep, to Python's garbage collector in the readers, writers and session builder, also of their shallow copies
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62063) Report the option keys and values that are objects, and the arguments that the writers keep, to Python's garbage collector in the readers, writers and session builder, also of their shallow copies
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-62061) SparkSession.addArtifacts reads a path that is not a str, and pyspark.sql.connect.client exports its names, as PySpark Connect
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-62061) SparkSession.addArtifacts reads a path that is not a str, and pyspark.sql.connect.client exports its names, as PySpark Connect
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-62049) Use `splitMovable` in `CombineFilters`
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-62049) Use `splitMovable` in `CombineFilters`
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-62058) Log the events of a pipeline run at the exact times of their timestamps, as PySpark's Timestamp.ToDatetime() gives them
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-62058) Log the events of a pipeline run at the exact times of their timestamps, as PySpark's Timestamp.ToDatetime() gives them
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62057) Stream-stream left semi join over a streaming union fails with a missing state file when a union input gets its first data after the first micro-batch
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62054) Fix wrong results and query failures when cost-based join reorder drops or moves join conditions
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62054) Fix wrong results and query failures when cost-based join reorder drops or moves join conditions
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62054) Fix wrong results and query failures when cost-based join reorder drops or moves join conditions
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62062) Fix JSON and row-based Parquet and ORC readers re-analyzing every existence default value for each row
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62062) Fix JSON and row-based Parquet and ORC readers re-analyzing every existence default value for each row
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62062) Fix JSON reader and row-based Parquet and ORC readers re-analyzing every existence default value for each row
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62044) Stream-stream join evicts state bounded through a CAST to TIMESTAMP by the event time, losing matches
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-62047) Pickle the Any and Timestamp messages of the drop-in's protobuf runtime, and fail each request of a user context extension that is not an Any, as PySpark
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-62047) Pickle the Any and Timestamp messages of the drop-in's protobuf runtime, and fail each request of a user context extension that is not an Any, as PySpark
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62061) SparkSession.addArtifacts reads a path that is not a str, and pyspark.sql.connect.client exports its names, as PySpark Connect
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62061) SparkSession.addArtifacts reads a path that is not a str, and pyspark.sql.connect.client exports its names, as PySpark Connect
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62060) Fix CSV reader re-analyzing every existence default value for each missing column in each row
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-61558) Fix broadcasting the large side of a join under OFFSET when the child has no row count
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62060) Fix CSV reader re-analyzing every existence default value for each missing column in each row
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62059) Bound transpiler early-return normalization and emit the NULL-guard advisory once
Holden Karau (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62058) Log the events of a pipeline run at the exact times of their timestamps, as PySpark's Timestamp.ToDatetime() gives them
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62058) Log the events of a pipeline run at the exact times of their timestamps, as PySpark's Timestamp.ToDatetime() gives them
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62057) Stream-stream left semi join over a streaming union fails with a missing state file when a union input gets its first data after the first micro-batch
Jungtaek Lim (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62057) Stream-stream left semi join over a streaming union fails with a missing state file when a union input gets its first data after the first micro-batch
Jungtaek Lim (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62057) Stream-stream left semi join over a streaming union fails with a missing state file when a union input gets its first data after the first micro-batch
Jungtaek Lim (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62056) Take the arguments of the methods of the maps and repeated containers of the Rust protobuf runtime as upb's, with upb's errors
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62056) Take the arguments of the methods of the maps and repeated containers of the Rust protobuf runtime as upb's, with upb's errors
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62055) Fix vectorized Parquet and ORC readers re-analyzing every existence default value for each column
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62055) Fix vectorized Parquet and ORC readers re-analyzing every existence default value for each column
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-60301) Spark doesn't respect session timezone when casting nested variant to string
Wenchen Fan (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-60301) Spark doesn't respect session timezone when casting nested variant to string
Wenchen Fan (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-60755) Fix codegen compilation failure of `element_at` and array index when the result is not nullable
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-61669) Fix CALL failing with INTERNAL_ERROR when an argument has a Python UDF or a nested IN subquery
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Commented] (SPARK-59718) Add opt-in in-process Python UDF execution via Arrow C Data Interface
Yicong Huang (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-61377) Compute each input column only before the first condition reading it in the subexpression elimination path of FilterExec
Kousuke Saruta (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-61377) Compute each input column only before the first condition reading it in the subexpression elimination path of FilterExec
Kousuke Saruta (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-61685) Add `DataSourceV2Test` test tag
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-61685) Add `DataSourceV2Test` test tag
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62022) NullPointerException for a NULL window watermark column in stream-stream join state format V4
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62038) Stream-stream join state format V4 never evicts rows that an older version's cutoff left in state after a restart
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62054) Fix wrong results and query failures when cost-based join reorder drops or moves join conditions
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62054) Fix wrong results and query failures when cost-based join reorder drops or moves join conditions
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62053) Remove outdated unsupported `ArrayType of TimestampType` note from `spark.sql.execution.arrow.pyspark.enabled` documentation
Zhou Yifan (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-62041) Warn of a window in turn with the picklings of UDFs, send the clauses of a merge as they were built, keep the execution info of a failed toPandas, and pickle subquery Columns and Column._expr, as PySpark
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-62041) Warn of a window in turn with the picklings of UDFs, send the clauses of a merge as they were built, keep the execution info of a failed toPandas, and pickle subquery Columns and Column._expr, as PySpark
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62047) Pickle the Any and Timestamp messages of the drop-in's protobuf runtime, and fail each request of a user context extension that is not an Any, as PySpark
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62051) Fix wrong results by Dataset.unpivot when an outer join makes a value column nullable
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62051) Fix wrong results by Dataset.unpivot when an outer join makes a value column nullable
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62052) Fix DPP evaluating a pruning key's Python or external UDF twice without pruning any partition
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62051) Fix wrong results by Dataset.unpivot when an outer join makes a value column nullable
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62052) Fix DPP evaluating a pruning key's Python or external UDF twice without pruning any partition
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62051) Fix wrong results by Dataset.unpivot when an outer join makes a value column nullable
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62051) Fix wrong results by Dataset.unpivot when an outer join makes a value column nullable
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-61935) Fix DSv2 writes failing or writing NULL into NOT NULL columns for a Scala UDF with a primitive input
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-61935) Fix DSv2 writes failing or writing NULL into NOT NULL columns for a Scala UDF with a primitive input
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62042) Exclude non-deterministic nested field accesses from nested column aliasing
Xi Liang (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-61301) Improve the FILE type error messages and their builders
Daniel Tenedorio (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-62037) Share the options and hooks of a builder's shallow copies, hold PySpark's session and builder locks, let SparkSession, Catalog and DataSourceRegistration be subclassed, and load PySpark's pickles of readers
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-62037) Share the options and hooks of a builder's shallow copies, hold PySpark's session and builder locks, let SparkSession, Catalog and DataSourceRegistration be subclassed, and load PySpark's pickles of readers
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62050) Tag more DSv2-only test suites with `DataSourceV2Test`
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62050) Tag more DSv2-only test suites with `DataSourceV2Test`
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62050) Tag more DSv2-only test suites with `DataSourceV2Test`
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-61930) Fix query failures when `ReorderJoin` moves a throwable condition, or one that can fail after a non-deterministic or throwable one, into a lower join
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62049) Use `splitMovable` in `CombineFilters`
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62048) MemoryStore.putIteratorAsBytes leaks the compression stream when unrolling a block fails
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62049) Use `splitMovable` in `CombineFilters`
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62043) Remove test tags that ScalaTest never reads
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-62043) Remove test tags that ScalaTest never reads
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-59816) Transpile str upper, lower, startswith, and endswith
Holden Karau (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62032) Operators that pair data with an initial state by key do not check that the two sides' grouping keys have one type
Anil K (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-62043) Remove test tags that ScalaTest never reads
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62047) Pickle the Any and Timestamp messages of the drop-in's protobuf runtime, and fail each request of a user context extension that is not an Any, as PySpark
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-62013) Make the result objects of a DataFrame PySpark's: the bytes WKB of spatial values, the selfDestruct copy of toPandas, one executionInfo object and the Arrow metadata of _to_table
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62048) MemoryStore.putIteratorAsBytes leaks the compression stream when unrolling a block fails
Ivan Sadikov (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-61656) Fix CALL failing with INTERNAL_ERROR when an argument has an IN subquery
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62040) Improve hypothesis testing by batching examples when it makes sense through df creation
Holden Karau (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62046) Stream-stream outer join never outputs a row whose watermarked equality key is NULL, on the input that has that watermark
Jungtaek Lim (Jira) via issues
-
2026/11/18
[jira] [Commented] (SPARK-61382) Stream-stream full outer join joins rows whose equality keys are both NULL
Jungtaek Lim (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62046) Stream-stream outer join never outputs a row whose watermarked equality key is NULL, on the input that has that watermark
Jungtaek Lim (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62046) Stream-stream outer join never outputs a row whose watermarked equality key is NULL, on the input that has that watermark
Jungtaek Lim (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-60533) Make spark.sql.streaming.checkpoint.renamedFileCheck.enabled public and document it for object stores
Jungtaek Lim (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-60533) Make spark.sql.streaming.checkpoint.renamedFileCheck.enabled public and document it for object stores
Jungtaek Lim (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62045) Driver-side Python workers of a streaming query started through StreamingPythonRunner can't see the session's artifacts
Jungtaek Lim (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62042) Exclude non-deterministic nested field accesses from nested column aliasing
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-61930) Fix query failures when `ReorderJoin` moves a throwable condition, or one that can fail after a non-deterministic or throwable one, into a lower join
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-62012) Send local rows and pandas string columns of PySpark's large Arrow types under spark.sql.execution.arrow.useLargeVarTypes, and compare the cache threshold with pyarrow's nbytes
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62043) Remove test tags that ScalaTest never reads
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62044) Stream-stream join evicts state bounded through a CAST to TIMESTAMP by the event time, losing matches
Jungtaek Lim (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62043) Remove test tags that ScalaTest never reads
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62044) Stream-stream join evicts state bounded through a CAST to TIMESTAMP by the event time, losing matches
Jungtaek Lim (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62044) Stream-stream join evicts state bounded through a CAST to TIMESTAMP by the event time, losing matches
Jungtaek Lim (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62042) Exclude non-deterministic nested field accesses from nested column aliasing
Xi Liang (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62042) Exclude non-deterministic nested field accesses from nested column aliasing
Xi Liang (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62042) Exclude non-deterministic nested field accesses from nested column aliasing
Xi Liang (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-60979) Fix wrong results by nested schema pruning when column names differ only by case
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-60650) pandas-on-Spark named size aggregation is mapped to SQL size and fails on an integer column
Holden Karau (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-59816) Transpile str upper, lower, startswith, and endswith
Holden Karau (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-62023) Make SparkSession.client PySpark's SparkConnectClient: its state per session, a threading.local of live tags and extensions, PySpark's argument conversions and error order in its registrations and executions, and the members PySpark's tests read
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-62023) Make SparkSession.client PySpark's SparkConnectClient: its state per session, a threading.local of live tags and extensions, PySpark's argument conversions and error order in its registrations and executions, and the members PySpark's tests read
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-60650) pandas-on-Spark named size aggregation is mapped to SQL size and fails on an integer column
Holden Karau (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-59816) Transpile str upper, lower, startswith, and endswith
Holden Karau (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-62039) Split the Rust migration notes into the 4.3 line and the unreleased 4.4 line, and qualify the claim that the Rust API mirrors PySpark's surface
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-62039) Split the Rust migration notes into the 4.3 line and the unreleased 4.4 line, and qualify the claim that the Rust API mirrors PySpark's surface
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-61557) Fix CALL failing with IllegalArgumentException when an argument is not foldable
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-59828) Support `skipFixedLenByteArray` in `VectorizedDeltaByteArrayReader`
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62041) Warn of a window in turn with the picklings of UDFs, send the clauses of a merge as they were built, keep the execution info of a failed toPandas, and pickle subquery Columns and Column._expr, as PySpark
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62041) Warn of a window in turn with the picklings of UDFs, send the clauses of a merge as they were built, keep the execution info of a failed toPandas, and pickle subquery Columns and Column._expr, as PySpark
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-62017) Report the types, flags, metadata and fields that the array, map, field and struct types hold to Python's garbage collector
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Commented] (SPARK-61833) Set up CI for branch-4.4
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-61833) Set up CI for branch-4.4
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-62017) Report the types, flags, metadata and fields that the array, map, field and struct types hold to Python's garbage collector
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62039) Split the Rust migration notes into the 4.3 line and the unreleased 4.4 line, and qualify the claim that the Rust API mirrors PySpark's surface
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62039) Split the Rust migration notes into the 4.3 line and the unreleased 4.4 line, and qualify the claim that the Rust API mirrors PySpark's surface
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62014) Keep the state of Python subclasses of the native ML classes in their pickles and copies, give pyspark.ml.stat PySpark's docs, and make the ML objects' pickles, uids, metadata and attributes PySpark's
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62034) Fix vectorized Parquet/ORC reads of existence defaults with wide decimals or nested NULLs
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-59791) Add optional pre-push checks to the merge script's backports
Holden Karau (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62038) Stream-stream join state format V4 never evicts rows that an older version's cutoff left in state after a restart
Jungtaek Lim (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62038) Stream-stream join state format V4 never evicts rows that an older version's cutoff left in state after a restart
Jungtaek Lim (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-59791) Add optional pre-push checks to the merge script's backports
Holden Karau (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62037) Share the options and hooks of a builder's shallow copies, hold PySpark's session and builder locks, let SparkSession, Catalog and DataSourceRegistration be subclassed, and load PySpark's pickles of readers
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-59791) Add optional pre-push checks to the merge script's backports
Holden Karau (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62037) Share the options and hooks of a builder's shallow copies, hold PySpark's session and builder locks, let SparkSession, Catalog and DataSourceRegistration be subclassed, and load PySpark's pickles of readers
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62036) Support pinning a SparkApplication attempt to a single topology zone
Alex Zhilko (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-60465) Late-materialization storage-filter pushdown for data source V2 scans
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-62021) Make the relation of df._plan.plan(client) hold the relation of the DataFrame, built at the call, so that it is sent after the DataFrame is gone, as PySpark's
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-60541) Per-partition runtime bloom filters for storage-partitioned joins
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-61826) Projects generated by spark-pipelines init lose their tables after the run and fail the second run with LOCATION_ALREADY_EXISTS
Sudhanva Huruli (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62036) Support pinning a SparkApplication attempt to a single topology zone
Alex Zhilko (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-62021) Make the relation of df._plan.plan(client) hold the relation of the DataFrame, built at the call, so that it is sent after the DataFrame is gone, as PySpark's
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-61534) Estimate the size of an existence join from its left side
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62022) NullPointerException for a NULL window watermark column in stream-stream join state format V4
Jungtaek Lim (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62035) Add spark.sql.execution.maxScanBytes to cancel queries that read too much data
Gengliang Wang (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62035) Add spark.sql.execution.maxScanBytes to cancel queries that read too much data
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62033) COMMENT ON TABLE clears the default collation, and ALTER TABLE ... DEFAULT COLLATION clears the comment, on the session catalog
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62034) Fix vectorized Parquet/ORC reads of existence defaults with wide decimals or nested NULLs
Daniel Tenedorio (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-62017) Report the types, flags, metadata and fields that the array, map, field and struct types hold to Python's garbage collector
ASF GitHub Bot (Jira) via issues
-
2026/11/18
[jira] [Commented] (SPARK-61654) Place a cogroup key stored like the layout's column by the column's own transform in the one-side shuffle
Anil K (Jira) via issues
-
2026/11/18
[jira] [Updated] (SPARK-60978) Fix wrong results by DSv2 DELETE, UPDATE and MERGE when column names differ only by case
Dongjoon Hyun (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-62007) Make patches of pyspark.sql.DataFrame reach native frames as PySpark's, take PySpark's Arrow path for pandas data of struct subclasses and NUL names, and fail names that UTF-8 cannot encode at the request
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-62019) Compare the messages of the Rust protobuf runtime as upb's: floats of their bits, unknown fields of their contents, maps as mappings, and take only protobuf's messages of its default pool
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-62018) Keep in the type stubs the docstrings that PySpark's objects have at run time, so that editors show PySpark's documentation as with pyspark-client
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-62016) Treat user-defined type objects as PySpark does: keep them in the types that hold them until a use needs their JSON, make new ones for the schemas of the server, and render casts by their simpleString()
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-62018) Keep in the type stubs the docstrings that PySpark's objects have at run time, so that editors show PySpark's documentation as with pyspark-client
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-62019) Compare the messages of the Rust protobuf runtime as upb's: floats of their bits, unknown fields of their contents, maps as mappings, and take only protobuf's messages of its default pool
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-62016) Treat user-defined type objects as PySpark does: keep them in the types that hold them until a use needs their JSON, make new ones for the schemas of the server, and render casts by their simpleString()
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Assigned] (SPARK-62030) Connect directly to the hosts that no_proxy excludes despite an invalid proxy, end a connection whose server sends no HTTP/2 settings in the connect deadline, and write grpcio's errors of the proxy and the TLS roots to stderr
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Resolved] (SPARK-62030) Connect directly to the hosts that no_proxy excludes despite an invalid proxy, end a connection whose server sends no HTTP/2 settings in the connect deadline, and write grpcio's errors of the proxy and the TLS roots to stderr
Hyukjin Kwon (Jira) via issues
-
2026/11/18
[jira] [Comment Edited] (SPARK-61654) Place a cogroup key stored like the layout's column by the column's own transform in the one-side shuffle
Anil K (Jira) via issues
-
2026/11/18
[jira] [Created] (SPARK-62033) COMMENT ON TABLE clears the default collation, and ALTER TABLE ... DEFAULT COLLATION clears the comment, on the session catalog
Sudhanva Huruli (Jira) via issues