im reading below json in spark {"bucket": "B01", "actionType": "A1", "preaction": "NULL", "postaction": "NULL"} {"bucket": "B02", "actionType": "A2", "preaction": "NULL", "postaction": "NULL"} {"bucket": "B03", "actionType": "A3", "preaction": "NULL", "postaction": "NULL"}
val df=spark.read.json("actions.json").toDF() Now im writing the same to a json output as below df.write. format("json"). mode("append"). partitionBy("bucket","actionType"). save("output.json") and the output.json is as below {"preaction":"NULL","postaction":"NULL"} bucket,actionType columns are missing in the json output, i need partitionby columns as well in the output