Re: Streaming with Java: Expected ReduceByWindow to Return JavaDStream

2015-01-19 Thread Jeff Nadler 
For anyone who finds this later, looks like Jerry already took care of this
here:  https://issues.apache.org/jira/browse/SPARK-5315

Thanks!

On Sun, Jan 18, 2015 at 10:28 PM, Shao, Saisai saisai.s...@intel.com
wrote:

  Hi Jeff,



 From my understanding it seems more like a bug, since JavaDStreamLike is
 used for Java code, return a Scala DStream is not reasonable. You can fix
 this by submitting a PR, or I can help you to fix this.



 Thanks

 Jerry



 *From:* Jeff Nadler [mailto:jnad...@srcginc.com]
 *Sent:* Monday, January 19, 2015 2:04 PM
 *To:* user@spark.apache.org
 *Subject:* Streaming with Java: Expected ReduceByWindow to Return
 JavaDStream





 Can anyone tell me if my expectations are sane?



 I'm trying to do a reduceByWindow using the 3-arg signature (not providing
 an inverse reduce function):

 JavaDStreamwhatevs reducedStream = messages.reduceByWindow((x, y) - 
 reduce(x, y), Durations.*seconds*(5), Durations.*seconds*(5));

 This isn't building; looks like it's returning DStream not JavaDStream.

 From JavaDStreamLike.scala, looks like this sig returns DStream, the 4-arg 
 sig with the inverse reduce returns JavaDStream.

 *def *reduceByWindow(
 reduceFunc: (T, T) = T,
 windowDuration: Duration,
 slideDuration: Duration
   ): DStream[T] = {
   dstream.reduceByWindow(reduceFunc, windowDuration, slideDuration)
 }

 So I'm just a noob.  Is this a bug or am I missing something?

 Thanks!

 Jeff Nadler

RE: Streaming with Java: Expected ReduceByWindow to Return JavaDStream

2015-01-18 Thread Shao, Saisai 
Hi Jeff,

From my understanding it seems more like a bug, since JavaDStreamLike is used 
for Java code, return a Scala DStream is not reasonable. You can fix this by 
submitting a PR, or I can help you to fix this.

Thanks
Jerry

From: Jeff Nadler [mailto:jnad...@srcginc.com]
Sent: Monday, January 19, 2015 2:04 PM
To: user@spark.apache.org
Subject: Streaming with Java: Expected ReduceByWindow to Return JavaDStream


Can anyone tell me if my expectations are sane?

I'm trying to do a reduceByWindow using the 3-arg signature (not providing an 
inverse reduce function):

JavaDStreamwhatevs reducedStream = messages.reduceByWindow((x, y) - 
reduce(x, y), Durations.seconds(5), Durations.seconds(5));

This isn't building; looks like it's returning DStream not JavaDStream.

From JavaDStreamLike.scala, looks like this sig returns DStream, the 4-arg sig 
with the inverse reduce returns JavaDStream.

def reduceByWindow(
reduceFunc: (T, T) = T,
windowDuration: Duration,
slideDuration: Duration
  ): DStream[T] = {
  dstream.reduceByWindow(reduceFunc, windowDuration, slideDuration)
}

So I'm just a noob.  Is this a bug or am I missing something?

Thanks!

Jeff Nadler