Hi,
We are using something like the following to write data to files in Structured Streaming and we seem to get file names as part* as mentioned in https://stackoverflow.com/questions/51056764/how-to-define-a-spark-structured-streaming-file-sink-file-path-or-file-name. How to get file names of our choice for each row in the dataframe? Like say /day/month/id/log.txt? df.writeStream .format("parquet") // can be "orc", "json", "csv", etc. .option("path", "/path/to/save/") .partitionBy("year", "month", "day", "hour") .start() Thanks for the help!!! -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
