Re: Why do we need Java-Friendly APIs in Spark ?

2019-05-15 Thread Jason Nerothin
I did a quick google search for "Java/Scala interoperability" and was surprised to find very few recent results on the topic. (Has the world given up?) It's easy to use Java library code from Scala, but the opposite is not true. I would think about the problem this way: Do *YOU* need to provide

Re: Why do we need Java-Friendly APIs in Spark ?

2019-05-15 Thread Jean-Georges Perrin
I see… Did you consider Structure Streaming? Otherwise, you could create a factory that will build your higher level object, that will return an interface defining your API, but the implementation may vary based on the context. And English is not my native language as well... Jean -Georges

Re: Why do we need Java-Friendly APIs in Spark ?

2019-05-14 Thread Gary Gao
Thanks for reply, Jean In my project , I'm working on higher abstraction layer of spark streaming to build a data processing product and trying to provide a common api for java and scala developers. You can see the abstract class defined here:

Re: Why do we need Java-Friendly APIs in Spark ?

2019-05-14 Thread Jean Georges Perrin
There are a little bit more than the list you specified nevertheless, some data types are not directly compatible between Scala and Java and requires conversion, so it’s good to not pollute your code with plenty of conversion and focus on using the straight API. I don’t remember from the top