Glad to help! On Sat, Jul 13, 2019 at 12:17 PM Gourav Sengupta <gourav.sengu...@gmail.com> wrote:
> Hi Swetha, > I always look into the source code a lot but it never occured to me to > look into the test suite, thank a ton for the tip. Does definitely give > quite a few ideas - thanks a ton. > > Thanks and Regards, > Gourav > > On Fri, Jul 12, 2019 at 6:51 AM Swetha Ramaiah <swetha.rama...@gmail.com> > wrote: > >> Hi Anil >> >> That was an example. You can replace quote with what double quotes. But >> these options should give you an idea on how you want treat nulls, empty >> values and quotes. >> >> When I faced this issues, I forked Spark repo and looked at the test >> suite. This definitely helped me solve my issue. >> >> https://github.com/apache/spark/blob/v2.4.3/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala >> >> Regards >> Swetha >> >> On Jul 11, 2019, at 4:09 PM, Anil Kulkarni <anil...@gmail.com> wrote: >> >> Hi Swetha, >> >> Thank you. >> But we need the data to be quoted with ". >> and when a field is null, we dont need the quotes around it. >> >> Example: >> "A",,"B","C" >> Thanks >> Anil >> >> On Thu, Jul 11, 2019, 1:51 PM Swetha Ramaiah <swetha.rama...@gmail.com> >> wrote: >> >>> If you are using Spark 2.4.0, I think you can try something like this: >>> >>> .option("quote", "\u0000") >>> .option("emptyValue", “”) >>> >>> .option("nullValue", null) >>> >>> Regards >>> Swetha >>> >>> >>> >>> On Jul 11, 2019, at 1:45 PM, Anil Kulkarni <anil...@gmail.com> wrote: >>> >>> Hi Spark users, >>> >>> My question is : >>> I am writing a Dataframe to csv. >>> Option i am using as >>> .option("quoteAll","true"). >>> >>> This is quoting even null values and making them appear as an empty >>> string. >>> >>> How do i make sure that quotes are enabled only for non null values? >>> >>> -- >>> Cheers, >>> Anil Kulkarni >>> about.me/anilkulkarni >>> [image: Anil Kulkarni on about.me] >>> http://anilkulkarni.com/ <http://about.me/anilkulkarni> >>> >>> >>> >> -- Regards, Swetha
