that is what i would expect: escaping only works if quoted On Thu, Oct 27, 2016 at 1:24 PM, Jain, Nishit <nja...@underarmour.com> wrote:
> Interesting finding: Escaping works if data is quoted but not otherwise. > > From: "Jain, Nishit" <nja...@underarmour.com> > Date: Thursday, October 27, 2016 at 10:54 AM > To: "user@spark.apache.org" <user@spark.apache.org> > Subject: CSV escaping not working > > I am using spark-core version 2.0.1 with Scala 2.11. I have simple code to > read a csv file which has \ escapes. > > val myDA = spark.read > .option("quote",null) > .schema(mySchema) > .csv(filePath) > > As per documentation \ is default escape for csv reader. But it does not > work. Spark is reading \ as part of my data. For Ex: City column in csv > file is *north rocks\,au* . I am expecting city column should read in > code as *northrocks,au*. But instead spark reads it as *northrocks\* and > moves *au* to next column. > > I have tried following but did not work: > > - Explicitly defined escape .option("escape",”\\") > - Changed escape to | or : in file and in code > - I have tried using spark-csv library > > Any one facing same issue? Am I missing something? > > Thanks >