srielau commented on PR #58549: URL: https://github.com/apache/spark/pull/58549#issuecomment-5603635938
Addressed the remaining summary findings in af66d51d5fc: - Python DataSource now rejects recursive CHAR/VARCHAR return schemas before Arrow conversion. - Python scalar UDF validation rejects CHAR/VARCHAR nested inside UDT storage schemas. - Python TransformWithState rejects constrained grouping-key, value, list, map-key, and map-value schemas; the incidental state-server policy plumbing was removed. - Both fixed-schema and analyze-derived row Python UDTFs reject recursive CHAR/VARCHAR return schemas. - The Arrow RDD branch, mixed-policy batch output, normalized map-key collision, lambda lifting, legacy schema, and true Arrow-columnar paths now have focused regressions. Validation: the focused EvaluatePythonSuite SPARK-59275 tests pass, which also compiled the changed SQL main and test sources. Python compileall passes. The focused PySpark runtime attempt could not start SparkSubmit from this checkout because the local assembled classpath is incomplete; Arrow/HOF tests are additionally skipped here because pandas is unavailable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
