Adding flink-scala as a dependency to flink-streaming-core

2015-06-10 Thread Gábor Gévay
Hello, I would like to ask if it would be OK if I added flink-scala as a dependency to flink-streaming-core. An alternative would be to move the Scala typeutils to flink-core (to where the Java typeutils are). Why I need this: While I am implementing the fast median calculation for windows as par

Re: Adding flink-scala as a dependency to flink-streaming-core

2015-06-10 Thread Till Rohrmann
I see the reason why you want to add flink-scala as a dependency to flink-streaming-core. However, it does not feel right to add an API package to a core package IMHO. But I noticed that flink-streaming-core also depends on flink-java. Which seems odd to me as well. I'm not a streaming expert and

Re: Adding flink-scala as a dependency to flink-streaming-core

2015-06-10 Thread Till Rohrmann
Btw: I noticed that all streaming modules depend on flink-core, flink-runtime, flink-clients and flink-java. Is there a particular reason why the streaming connectors depend on flink-clients and flink-java? On Wed, Jun 10, 2015 at 3:41 PM Till Rohrmann wrote: > I see the reason why you want to a

Re: Adding flink-scala as a dependency to flink-streaming-core

2015-06-10 Thread Márton Balassi
As for the current streaming dependencies: flink-java is not needed at all, flink-clients is only needed in specific cases. Thanks for spotting it, I am cleaning it. As for Gabor's problem: maybe we should move the CaseClassTypeInfo to flink-core then given Till's comment. Any opinions on that? O

Re: Adding flink-scala as a dependency to flink-streaming-core

2015-06-10 Thread Gábor Gévay
> "it does not feel right to add an API package to a core package Yes, that makes sense. I just tried removing the flink-java dependency from flink-streaming to see what needs it, and it builds fine without it :) What do you think about the second option? (to move the Scala typeutils (or just Cas

Re: Adding flink-scala as a dependency to flink-streaming-core

2015-06-10 Thread Aljoscha Krettek
If they can be easily converted to Java code, that would be a good solution. On Wed, 10 Jun 2015 at 15:56 Gábor Gévay wrote: > > "it does not feel right to add an API package to a core package > > Yes, that makes sense. I just tried removing the flink-java dependency > from flink-streaming to se

Re: Adding flink-scala as a dependency to flink-streaming-core

2015-06-10 Thread Gábor Gévay
I see four options to solve this without adding the dependency: 1. Move CaseClassTypeInfo and CaseClassComparator to flink-core. Till said that we want to avoid mixed Scala and Java modules, which rules this out. 2. Create a new toplevel maven project scala-core, and move things there. 3. Hacky sol

Re: Adding flink-scala as a dependency to flink-streaming-core

2015-06-20 Thread Robert Metzger
I like option 1 the most ("move to flink-core"), however, it would scatter the type extractor / type information classes accross multiple projects. Why are we not moving the entire type extractor system into "flink-core"? There were some thoughts in the past to restructure the maven module layout:

Re: Adding flink-scala as a dependency to flink-streaming-core

2015-06-21 Thread Chiwan Park
+1 for moving the whole type extractor and type information classes to "flink-core". The user cannot see "flink-core” because it is a hidden package from user’s perspective. So the package could be deployed with Scala version variation safely. I think that we can mix “flink-core” and some Scala

Re: Adding flink-scala as a dependency to flink-streaming-core

2015-06-21 Thread Stephan Ewen
I like moving the type system classes to core. But one blocker so far was that the TupleX classes are in flink-java, and no one moved them. We have this long discussion pending whether we want to merge flink-core and flink-java, actually... On Sun, Jun 21, 2015 at 12:29 AM, Chiwan Park wrote: