Hello, My name is Mario Fernández and I´m a Big Data developer, I usually program in Apache Spark in Java, and we have a big problem to read properly a csv file. The issue is that:
When I want to read csv file, for instance, with semicolon delimiter, the dataframe take semicolon like delimiter ant that´s correct, but also take comma like delimiter and that´s the problem. I check this problem in Apache Spark 2.10, 1.6.2 DataFrame and also in Apache Spark 2.11 2.0.2 Dataset, and troubles are the same. dfFile1 = sqlContext.read() .format("com.databricks.spark.csv") .schema(customSchema) .option("charset", "Cp1252") .option("header", "true") .option("delimiter", ";") .load(path); When I read a csv file like that, dataFrame take like delimiter the yellow letters: Number;Name;Surname;Category 129.363;Mathew, Thomas;Johnson;Centers Technician And the comma between Mathew and Thomas, shouldn´t be take like delimiter. I would like to know if that´s problem is a bug and you are going to correct or the way to read simply is like that. Thank you so much in advance. Kind Regards. ________________________________ AVISO DE CONFIDENCIALIDAD. Este correo y la información contenida o adjunta al mismo es privada y confidencial y va dirigida exclusivamente a su destinatario. everis informa a quien pueda haber recibido este correo por error que contiene información confidencial cuyo uso, copia, reproducción o distribución está expresamente prohibida. Si no es Vd. el destinatario del mismo y recibe este correo por error, le rogamos lo ponga en conocimiento del emisor y proceda a su eliminación sin copiarlo, imprimirlo o utilizarlo de ningún modo. CONFIDENTIALITY WARNING. This message and the information contained in or attached to it are private and confidential and intended exclusively for the addressee. everis informs to whom it may receive it in error that it contains privileged information and its use, copy, reproduction or distribution is prohibited. If you are not an intended recipient of this E-mail, please notify the sender, delete it and do not read, act upon, print, disclose, copy, retain or redistribute any portion of this E-mail.