Re: Apache Spark - Spark Structured Streaming - Watermark usage

2018-02-05 Thread M Singh
Just checking if anyone has more details on how watermark works in cases where event time is earlier than processing time stamp. On Friday, February 2, 2018 8:47 AM, M Singh wrote: Hi Vishu/Jacek: Thanks for your responses. Jacek - At the moment, the current time for my use case is proc

Re: Apache Spark - Exception on adding column to Structured Streaming DataFrame

2018-02-05 Thread M Singh
Hi TD: Just wondering if you have any insight for me or need more info. Thanks On Thursday, February 1, 2018 7:43 AM, M Singh wrote: Hi TD: Here is the udpated code with explain and full stack trace. Please let me know what could be the issue and what to look for in the explain output.

high TFIDF value terms

2018-02-05 Thread Donni Khan
Hi, anyone knows how I can get the high TFIDF value terms by using Spark(Java)? IDF idf = *new* IDF().setInputCol("TF").setOutputCol("IDF"); IDFModel idfModel = idf.fit(featurizedData); DataFrame tfidf = idfModel.transform(featurizedData); Thanks; Donni