As a tempary fix, it works when I convert field six to a list manually. That
is:
def generateRecords(line):
# input : the row stored in parquet file
# output : a python dictionary with all the key value pairs
field1 = line.field1
summary = {}
summary['field2'] = line.field2
summary['field3'] = line.field3
summary['field4'] = line.field4
summary['field5'] = line.field5
*summary['field6'] = list(line.field6) *
return (field1,summary)
profiles = sqc.parquetFile(path)
profileRecords = profiles.map(lambda line: generateRecords(line))
Works!!
But I am not convinced this is the best I could do :)
Cheers
SahanB
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Error-when-mapping-a-schema-RDD-when-converting-lists-tp20577p20579.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
