This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new 135bb497eda [SPARK-44479][CONNECT][PYTHON] Fix protobuf conversion
from an empty struct type
135bb497eda is described below
commit 135bb497edad8d132323257547c86bd405ecba8e
Author: Takuya UESHIN <ues...@databricks.com>
AuthorDate: Thu Jul 27 10:10:44 2023 +0900
[SPARK-44479][CONNECT][PYTHON] Fix protobuf conversion from an empty struct
type
### What changes were proposed in this pull request?
This is a partial backport of #42161.
Fixes protobuf conversion from an empty struct type.
### Why are the changes needed?
The empty struct type was not properly converted to the protobuf message.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing tests.
Closes #42179 from ueshin/issues/SPARK-44479/3.4/empty_schema.
Authored-by: Takuya UESHIN <ues...@databricks.com>
Signed-off-by: Hyukjin Kwon <gurwls...@apache.org>
---
python/pyspark/sql/connect/types.py | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/python/pyspark/sql/connect/types.py
b/python/pyspark/sql/connect/types.py
index dfb0fb5303f..be8d62c805f 100644
--- a/python/pyspark/sql/connect/types.py
+++ b/python/pyspark/sql/connect/types.py
@@ -155,6 +155,7 @@ def pyspark_types_to_proto_types(data_type: DataType) ->
pb2.DataType:
ret.day_time_interval.start_field = data_type.startField
ret.day_time_interval.end_field = data_type.endField
elif isinstance(data_type, StructType):
+ struct = pb2.DataType.Struct()
for field in data_type.fields:
struct_field = pb2.DataType.StructField()
struct_field.name = field.name
@@ -162,7 +163,8 @@ def pyspark_types_to_proto_types(data_type: DataType) ->
pb2.DataType:
struct_field.nullable = field.nullable
if field.metadata is not None and len(field.metadata) > 0:
struct_field.metadata = json.dumps(field.metadata)
- ret.struct.fields.append(struct_field)
+ struct.fields.append(struct_field)
+ ret.struct.CopyFrom(struct)
elif isinstance(data_type, MapType):
ret.map.key_type.CopyFrom(pyspark_types_to_proto_types(data_type.keyType))
ret.map.value_type.CopyFrom(pyspark_types_to_proto_types(data_type.valueType))
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
