Hi, Yes, that's ForeachWriter.
Yes, it works with element by element. You're looking for mapPartition and ForeachWriter has partitionId that you could use to implement a similar thing. Pozdrawiam, Jacek Laskowski ---- https://medium.com/@jaceklaskowski/ Mastering Apache Spark 2.0 https://bit.ly/mastering-apache-spark Follow me at https://twitter.com/jaceklaskowski On Thu, Feb 9, 2017 at 3:55 AM, Egor Pahomov <pahomov.e...@gmail.com> wrote: > Jacek, you mean > http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.ForeachWriter > ? I do not understand how to use it, since it passes every value separately, > not every partition. And addding to table value by value would not work > > 2017-02-07 12:10 GMT-08:00 Jacek Laskowski <ja...@japila.pl>: >> >> Hi, >> >> Have you considered foreach sink? >> >> Jacek >> >> On 6 Feb 2017 8:39 p.m., "Egor Pahomov" <pahomov.e...@gmail.com> wrote: >>> >>> Hi, I'm thinking of using Structured Streaming instead of old streaming, >>> but I need to be able to save results to Hive table. Documentation for file >>> sink >>> says(http://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#output-sinks): >>> "Supports writes to partitioned tables. ". But being able to write to >>> partitioned directories is not enough to write to the table: someone needs >>> to write to Hive metastore. How can I use Structured Streaming and write to >>> Hive table? >>> >>> -- >>> Sincerely yours >>> Egor Pakhomov > > > > > -- > Sincerely yours > Egor Pakhomov --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
