hi all,
i noticed a weird behavior to when spark parses nested json with schema
conflict.
i also just noticed that spark "fixed" this in the most recent release
3.5.0 but since i'm working with AWS services being:
* EMR 6: spark 3.3.* spark3.4.*
* Glue 3: spark3.1.1
* Glue 4: spark 3.3.0
https://docs.aws.amazon.com/glue/latest/dg/release-notes.html
https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-app-versions-6.x.html
..we're still facing this issue company-wide.
the problem was that spark is silently dropping records (or even whole
files) when there's an schema conflict or empty string values (
https://kb.databricks.com/notebooks/json-reader-parses-value-as-null).
My whole concern here is that spark is not even echoing a warn or error or
exception when such cases occurs.
To reproduce:
i'm using amaznlinux2 with Python 3.7.16 and pyspark==3.4.1.
echo
'{"Records":[{"evtid":"1","requestParameters":{"DescribeHostsRequest":{"MaxResults":500}}}]}'
> one.json
echo
'{"Records":[{"evtid":"2","requestParameters":{"lol":{},"lol2":{}}},{"evtid":"3","requestParameters":{"DescribeHostsRequest":""}}]}'
> two.json
Output of command:
> spark.read.json(["one.json", "two.json"]).show()
using 3.1.0 and 3.3.0
+--+
| Records|
+--+
| null|
|[{1, {{500}}}]|
+--+
drops the second (two.json) file
using Spark 3.4.0
+--+
| Records|
+--+
| null|
|[{1, {{500}}}]|
+--+
it completely drops the second (two.json) file.
Spark 3.5.0
++
| Records|
++
|[{2, {NULL}}, {3,...|
| [{1, {{500}}}]|
++
it reads both files but completely drops the "requestParameters" content of
all the records in the second (two.json) file.
{"evtid":"2","requestParameters":{}} <-- not good
{"evtid":"3","requestParameters":{}} <-- not good
{"evtid":"1","requestParameters":{"DescribeHostsRequest":{"MaxResults":500}}}
enabling spark.conf.set("spark.sql.legacy.json.allowEmptyString.enabled",
True) as suggested by
https://kb.databricks.com/notebooks/json-reader-parses-value-as-null in
spark 3.1 and 3.3 yields the same result seen in spark 3.5. which is not
ideal if one wants the later fetch the records as is.
to this. the only solution I found was to explicitly enforce the schema
when reading.
that said.
does anyone the exact thread or changelog where this issue was handled?
i've checked it on the links below but was non conclusive:
https://spark.apache.org/docs/latest/sql-migration-guide.html
https://spark.apache.org/releases/spark-release-3-5-0.html
another question.
how would you guys handle this scenario?
I could not see a clue even after enabling full verbose.
I could maybe force spark to issue an exception when such a case is
encountered.
or maybe send those bad/broken records to another file or bucket (dlq-ish)
best regards,c.
