We see that all the time. For example, in SQL, people can write their user-defined function in Scala/Java and use it from SQL/python/anywhere. That is the recommended way to get the best combo of performance and ease-of-use from non-jvm languages.
On Wed, Jul 26, 2017 at 11:49 AM, Priyank Shrivastava < priy...@asperasoft.com> wrote: > Thanks TD. I am going to try the python-scala hybrid approach by using > scala only for custom redis sink and python for the rest of the app . I > understand it might not be as efficient as purely writing the app in scala > but unfortunately I am constrained on scala resources. Have you come > across other use cases where people have resided to such python-scala > hybrid approach? > > Regards, > Priyank > > > > On Wed, Jul 26, 2017 at 1:46 AM, Tathagata Das < > tathagata.das1...@gmail.com> wrote: > >> Hello Priyank >> >> Writing something purely in Scale/Java would be the most efficient. Even >> if we expose python APIs that allow writing custom sinks in pure Python, it >> wont be as efficient as Scala/Java foreach as the data would have to go >> through JVM / PVM boundary which has significant overheads. So Scala/Java >> foreach is always going to be the best option. >> >> TD >> >> On Tue, Jul 25, 2017 at 6:05 PM, Priyank Shrivastava < >> priy...@asperasoft.com> wrote: >> >>> I am trying to write key-values to redis using a DataStreamWriter object >>> using pyspark structured streaming APIs. I am using Spark 2.2 >>> >>> Since the Foreach Sink is not supported for python; here >>> <http://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#using-foreach>, >>> I am trying to find out some alternatives. >>> >>> One alternative is to write a separate Scala module only to push data >>> into redis using foreach; ForeachWriter >>> <http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.ForeachWriter> >>> is >>> supported in Scala. BUT this doesn't seem like an efficient approach and >>> adds deployment overhead because now I will have to support Scala in my app. >>> >>> Another approach is obviously to use Scala instead of python, which is >>> fine but I want to make sure that I absolutely cannot use python for this >>> problem before I take this path. >>> >>> Would appreciate some feedback and alternative design approaches for >>> this problem. >>> >>> Thanks. >>> >>> >>> >>> >> >
