Re: queryable state & streaming

https://issues.apache.org/jira/browse/SPARK-16738 I don't believe anyone is working on it yet. I think the most useful thing is to start enumerating requirements and use cases and then we can talk about how to build it. On Fri, Dec 8, 2017 at 10:47 AM, Stavros Kontopoulos <

Re: queryable state & streaming

Hi Stavros, Queryable state is definitely on the roadmap! We will revamp the StateStore API a bit, and a queryable StateStore is definitely one of the things we are thinking about during that revamp. Best, Burak On Dec 8, 2017 9:57 AM, "Stavros Kontopoulos" wrote: >

Re: queryable state & streaming

Just to re-phrase my question: Would query-able state make a viable SPIP? Regards, Stavros On Thu, Dec 7, 2017 at 1:34 PM, Stavros Kontopoulos < st.kontopou...@gmail.com> wrote: > Hi, > > Maybe this has been discussed before. Given the fact that many streaming > apps out there use state

Re: BUILD FAILURE due to...not found: value AnalysisBarrier in spark-catalyst_2.11?

Build is fine for me, and on Jenkins. Try a clean build? On Fri, Dec 8, 2017 at 11:04 AM Jacek Laskowski wrote: > Hi, > > Just got BUILD FAILURE and have been wondering if it's just me or is this > a known issue that's being worked on? > > (Sorry if that's just my local setup

Re: Leveraging S3 select

Hey Steve, Happen to have a link to the TPC-DS benchmark data w/random S3 reads? I've done a decent amount of digging, but all I've found is a reference in a slide deck and some jira tickets. From: Steve Loughran Date: Tuesday, December 5, 2017 at 09:44 To: "Lalwani,

BUILD FAILURE due to...not found: value AnalysisBarrier in spark-catalyst_2.11?

Hi, Just got BUILD FAILURE and have been wondering if it's just me or is this a known issue that's being worked on? (Sorry if that's just my local setup that I got broken) [INFO] --- scala-maven-plugin:3.2.2:compile (scala-compile-first) @ spark-catalyst_2.11 --- [INFO] Using zinc server for

Deprecating UserDefinedGenerator logical operator?

Hi, I've just ran across UserDefinedGenerator logical operator [1] that is used exclusively for Dataset.explode operator [2][3] that is... @deprecated("use flatMap() or select() with functions.explode() instead", "2.0.0") Could that also "trigger" deprecating UserDefinedGenerator? [1]