Kristin Cowalcijk created SEDONA-497:
----------------------------------------
Summary: SpatialRDD read from multiple Shapefiles has incorrect
fieldName property
Key: SEDONA-497
URL: https://issues.apache.org/jira/browse/SEDONA-497
Project: Apache Sedona
Issue Type: Bug
Affects Versions: 1.5.1
Reporter: Kristin Cowalcijk
Fix For: 1.6.0
Attachments: debug_shapefiles.zip
A user reported this issue on Discord. It could be easily reproduced using the
following shapefiles provided by the user: [^debug_shapefiles.zip]
The following code loads a directory containing multiple shapefiles to
SpatialRDD at once, and then use {{Adapter.toDF}} to convert the SpatialRDD to
a Spark DataFrame:
{code:python}
parcel_rdd = ShapefileReader.readToGeometryRDD(sc, parcel_path)
parcel_df = Adapter.toDf(parcel_rdd, sedona)
parcel_df.printSchema()
parcel_df.show()
{code}
The above code yields the following output:
{code}
root
|-- geometry: geometry (nullable = true)
|-- id: string (nullable = true)
|-- name id: string (nullable = true)
|-- name id: string (nullable = true)
|-- name: string (nullable = true)
24/01/31 14:09:24 WARN TaskSetManager: Lost task 0.0 in stage 32.0 (TID 43)
(172.20.0.130 executor 0): org.apache.spark.SparkRuntimeException: Error while
encoding: java.lang.ArrayIndexOutOfBoundsException: Index 3 out of bounds for
length 3
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else
newInstance(class org.apache.spark.sql.sedona_sql.UDT.GeometryUDT).serialize AS
geometry#275
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else
staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType,
fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0,
org.apache.spark.sql.Row, true]), 1, id), StringType, ObjectType(class
java.lang.String)), true, false, true) AS id#276
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else
staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType,
fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0,
org.apache.spark.sql.Row, true]), 2, name id), StringType, ObjectType(class
java.lang.String)), true, false, true) AS name id#277
{code}
The reason why {{Adapter.toDf}} returns a dataframe with weird schema is
because the {{fieldNames}} property of {{parcel_rdd}} is incorrect:
{code}
>>> parcel_rdd.fieldNames
['id', 'name id', 'name id', 'name']
{code}
The schema of the shapefiles should be ['id', 'name'], but it was strangely
duplicated 3 times.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
