Okay. So what should I do if I get such data?
On Wed, Feb 23, 2022 at 11:59 PM Sean Owen <sro...@gmail.com> wrote:
> There is no record "345" here it seems, right? it's not that it exists and
> has null fields; it's invalid w.r.t. the schema that the rest suggests.
>
> On Wed, Feb 23, 2022 at 11:57 AM Sid <flinkbyhe...@gmail.com> wrote:
>
>> Hello experts,
>>
>> I have a JSON data like below:
>>
>> [
>> {
>> "123": {
>> "Party1": {
>> "FIRSTNAMEBEN": "ABC",
>> "ALIASBEN": "",
>> "RELATIONSHIPTYPE": "ABC, FGHIJK LMN",
>> "DATEOFBIRTH": "7/Oct/1969"
>> },
>> "Party2": {
>> "FIRSTNAMEBEN": "ABCC",
>> "ALIASBEN": "",
>> "RELATIONSHIPTYPE": "ABC, FGHIJK LMN",
>> "DATEOFBIRTH": "7/Oct/1969"
>> }
>> },
>> "GeneratedTime": "2022-01-30 03:09:26"
>> },
>> {
>> "456": {
>> "Party1": {
>> "FIRSTNAMEBEN": "ABCD",
>> "ALIASBEN": "",
>> "RELATIONSHIPTYPE": "ABC, FGHIJK LMN",
>> "DATEOFBIRTH": "7/Oct/1969"
>> },
>> "Party2": {
>> "FIRSTNAMEBEN": "ABCDD",
>> "ALIASBEN": "",
>> "RELATIONSHIPTYPE": "ABC, FGHIJK LMN",
>> "DATEOFBIRTH": "7/Oct/1969"
>> },
>> "Party3": {
>> "FIRSTNAMEBEN": "ABCDDE",
>> "ALIASBEN": "",
>> "RELATIONSHIPTYPE": "ABC, FGHIJK LMN",
>> "DATEOFBIRTH": "7/Oct/1969"
>> }
>> },
>> "GeneratedTime": "2022-01-30 03:09:26"
>> },
>> {
>> "345": {
>>
>>
>> },
>> "GeneratedTime": "2022-01-30 03:09:26"
>> }
>> ]
>>
>> However, when I try to display this JSON using below code, it doesn't
>> show the blank records. In my case I don't get any records for 345 since it
>> is null but I want to display it in the final flattened dataset.
>>
>> val df = spark.read.option("multiline",
>> true).json("/home/siddhesh/Documents/nested_json.json")
>>
>> Spark version:3.1.1
>>
>> Thanks,
>> Sid
>>
>
