I am trying to read csv in spark dataframe . My Os = Ubuntu 18.04,
spark-version 2.3.1, python -version 2.7.15
My code :
from pyspark import SparkConf
from pyspark import SparkContext
from pyspark import SQLContext
from pyspark.sql import SparkSession
conf = SparkConf()
sc = SparkContext(conf = conf)
spark =
SparkSession.builder.config(conf=conf).appName('FinancialRecon').getOrCreate()
sqlContext =SQLContext(sc)
df=spark.read.csv('/home/iconnect4/finrecon/test2.csv')
df.show()
Error :
Traceback (most recent call last):
File "/home/iconnect4/finrecon/scratch_3.py", line 43, in <module>
df=sqlContext.read.csv('/home/iconnect4/finrecon/test2.csv')
File "/usr/local/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py",
line 441, in csv
File
"/usr/local/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line
1257, in __call__
File "/usr/local/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line
79, in deco
pyspark.sql.utils.IllegalArgumentException: 'Illegal pattern component: XXX'
Please help
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
