Hi NA function will replace null with some default value and not all my columns are of type string, so for some other data types (long/int etc) I have to provide some default value
But ideally those values should be null Actually this null column drop is happening in this step df.selectExpr( "to_json(struct(*)) AS value") Is it possible to retain those columns where all the values are null in this step without using na functions ? Regards, Snehasish On Tue, Apr 30, 2019, 4:58 AM Jason Nerothin <jasonnerot...@gmail.com> wrote: > See also here: > https://stackoverflow.com/questions/44671597/how-to-replace-null-values-with-a-specific-value-in-dataframe-using-spark-in-jav > > On Mon, Apr 29, 2019 at 5:27 PM Jason Nerothin <jasonnerot...@gmail.com> > wrote: > >> Spark SQL has had an na.fill function on it since at least 2.1. Would >> that work for you? >> >> >> https://spark.apache.org/docs/2.1.0/api/java/org/apache/spark/sql/DataFrameNaFunctions.html >> >> On Mon, Apr 29, 2019 at 4:57 PM Shixiong(Ryan) Zhu < >> shixi...@databricks.com> wrote: >> >>> Hey Snehasish, >>> >>> Do you have a reproducer for this issue? >>> >>> Best Regards, >>> Ryan >>> >>> >>> On Wed, Apr 24, 2019 at 7:24 AM SNEHASISH DUTTA < >>> info.snehas...@gmail.com> wrote: >>> >>>> Hi, >>>> >>>> While writing to kafka using spark structured streaming , if all the >>>> values in certain column are Null it gets dropped >>>> Is there any way to override this , other than using na.fill functions >>>> >>>> Regards, >>>> Snehasish >>>> >>> >> >> -- >> Thanks, >> Jason >> > > > -- > Thanks, > Jason >
