Thanks for the response Sean.
As a correction. The code I provided actually ended up working. I tried to
reduce my code down but I was being overzealous and running count actually
works.
The minimal code that triggers the problem is this:
val userProfiles = lines.map(line => {parse(line)}).map(json => {
val user = (json \ "user")
val given_name = (user \ "given_name").extract[String]
(user, given_name)
})
this also doesn't work using what you suggested and still has the same
serialization issue. It seems like a weird bug since doing lines.count works
and even removing given_name in the closure works as well. :-/
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Serialize-input-path-tp13519p13530.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
