Re: [k8s] Spark operator (the Java one)

Hi all, I also left a comment on the PR with more details. I dont see why the java operator should be maintained by the Spark project. This is an interesting project and could thrive on its own as an external operator project. Best, Stavros On Thu, Oct 10, 2019 at 7:51 PM Sean Owen wrote: >

Re: Thoughts on Spark 3 release, or a preview release

+1 as a contributor and as a user. Given the amount of testing required for all the new cool stuff like java 11 support, major refactorings/deprecations etc, a preview version would help a lot the community making adoption smoother long term. I would also add to the list of issues, Scala 2.13

Re: Python API for mapGroupsWithState

indication that a Python version of the API is coming. > > Is this something that will likely be added in the near future, or is it a > major undertaking? Can someone briefly describe the problem? > > Nick > > -- Stavros Kontopoulos *Principal Engineer* *Lightbend Platform <h

Re: Welcoming some new committers and PMC members

Congrats! Well deserved. On Tue, Sep 10, 2019 at 1:20 PM Driesprong, Fokko wrote: > Congrats all, well deserved! > > > Cheers, Fokko > > Op di 10 sep. 2019 om 10:21 schreef Gabor Somogyi < > gabor.g.somo...@gmail.com>: > >> Congrats Guys! >> >> G >> >> >> On Tue, Sep 10, 2019 at 2:32 AM Matei

Re: Release Apache Spark 2.4.4 before 3.0.0

Hi Dongjoon, Should we also consider fixing https://issues.apache.org/jira/browse/SPARK-27812 before the cut? Best, Stavros On Mon, Jul 15, 2019 at 7:04 PM Dongjoon Hyun wrote: > Hi, Apache Spark PMC members. > > Can we cut Apache Spark 2.4.4 next Monday (22nd July)? > > Bests, > Dongjoon. >

Re: Contribution help needed for sub-tasks of an umbrella JIRA - port *.sql tests to improve coverage of Python, Pandas, Scala UDF cases

I can try one and see how it goes, although not familiar with the area. Stavros On Tue, Jul 9, 2019 at 6:17 AM Hyukjin Kwon wrote: > Hi all, > > I am currently targeting to improve Python, Pandas UDFs Scala UDF test > cases by integrating our existing *.sql files at >

Re: Support SqlStreaming in spark

Hi all, >From what I read there is an effort here to globally standardize SQL Streaming (Flink people, Google at others are working with SQL standardization body) https://arxiv.org/abs/1905.12133v1

Re: dynamic allocation manager in SS

> On Fri, May 24, 2019 at 9:40 PM Stavros Kontopoulos < > stavros.kontopou...@lightbend.com> wrote: > >> Btw the heuristics for batch mode ( >> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala#L289) >> vs

Re: dynamic allocation manager in SS

). PS. this has been discussed, not in depth, in the past on the list ( https://mail-archives.apache.org/mod_mbox/spark-user/201708.mbox/%3c1503626484779-29104.p...@n3.nabble.com%3E ) On Fri, May 24, 2019 at 9:22 PM Stavros Kontopoulos < stavros.kontopou...@lightbend.com> wrote: > I

Re: dynamic allocation manager in SS

Somogyi < gabor.g.somo...@gmail.com> έγραψε: > It scales down with yarn. Not sure how you've tested. > > On Fri, 24 May 2019, 19:10 Stavros Kontopoulos, < > stavros.kontopou...@lightbend.com> wrote: > >> Yes nothing happens. In this case it could propagate info to

Re: dynamic allocation manager in SS

a arrives no tasks are > executed (just had a case in this area). > > BR, > G > > > On Fri, 24 May 2019, 18:14 Stavros Kontopoulos, < > stavros.kontopou...@lightbend.com> wrote: > >> Hi, >> >> Some while ago the streaming dynamic allocation part wa

dynamic allocation manager in SS

Hi, Some while ago the streaming dynamic allocation part was added in DStreams( https://issues.apache.org/jira/browse/SPARK-12133) to improve the issues with the batch based one. Should this be ported to structured streaming? Thoughts? AFAIK there is no support in SS for it. Best, Stavros

Re: [METRICS] Metrics names inconsistent between executions

Hi, With jmx_exporter and Prometheus you can always re-write the metrics patterns on the fly. Btw if you use Grafana its easy to filter things even without the re-write. If this is a custom dashboard you can always group metrics based on the

Re: queryable state & streaming

Michael, I have listed used cases above should we proceed with a design doc? Best, Stavros Στις Δευ, 18 Μαρ 2019, 12:21 μ.μ. ο χρήστης Stavros Kontopoulos < stavros.kontopou...@lightbend.com> έγραψε: > Not really, if we agree that we want this, I can put together a design > docum

Re: JDK vs JRE in Docker Images

Hi Rob, We are using registry.redhat.io/redhat-openjdk-18/openjdk18-openshift ( https://docs.openshift.com/online/using_images/s2i_images/java.html) It looks most convenient as Red Hat leads the openjdk updates which is even more important from now on and also from a security point of view. There

Re: [VOTE] Release Apache Spark 2.4.1 (RC8)

ypically not hold the > > release unless the bug in question is a regression from the previous > > release. That being said, if there is something which is a regression > > that has not been correctly targeted please ping me or a committer to > > help target the issue. >

Re: queryable state & streaming

rence will not be >>> only reading state from non-heap, but also how to take a snapshot and store >>> delta. We may want to check snapshotting works well with small batch >>> interval, and find alternative approach when it doesn't. Sounds like it is >>> a

Re: Spark job status on Kubernetes

AFAIK completed can happen in case of failures as well, check here: https://github.com/kubernetes/kubernetes/blob/7f23a743e8c23ac6489340bbb34fa6f1d392db9d/pkg/client/conditions/conditions.go#L61 The phase of the pod should be `succeeded` to make a conclusion. This is

Re: [VOTE] Release Apache Spark 2.4.1 (RC2)

. > If it 'basically works but no guarantees' I'd favor not updating. If > it doesn't work at all, hm. That's tough. I think I'd favor updating > the client but think it's a tough call both ways. > > > > On Wed, Mar 6, 2019 at 11:14 AM Stavros Kontopoulos > wrote: >

Re: [VOTE] Release Apache Spark 2.4.1 (RC2)

Yes Shane Knapp has done the work for that already, and also tests pass, I am working on a PR now, I could submit it for the 2.4 branch . I understand that this is a major dependency update, but the problem I see is that the client version is so old that I dont think it makes much sense for

Re: [VOTE] Release Apache Spark 2.4.1 (RC2)

We need to resolve this https://issues.apache.org/jira/browse/SPARK-26742 as well for 2.4.1, to make k8s support meaningful as many people are now on 1.11+ Stavros On Tue, Mar 5, 2019 at 3:12 PM Saisai Shao wrote: > Hi DB, > > I saw that we already have 6 RCs, but the vote I can search by now

Re: DataSourceV2 sync notes - 20 Feb 2019

Thanks Ryan! On Tue, Mar 5, 2019 at 7:19 PM Ryan Blue wrote: > Everyone is welcome to join this discussion. Just send me an e-mail to get > added to the invite. > > Stavros, I'll add you. > > rb > > On Tue, Mar 5, 2019 at 5:43 AM Stavros Kontopoulos < > stavros.ko

Re: DataSourceV2 sync notes - 20 Feb 2019

Thanks for the update, is this meeting open for other people to join? Stavros On Thu, Feb 21, 2019 at 10:56 PM Ryan Blue wrote: > Here are my notes from the DSv2 sync last night. As always, if you have > corrections, please reply with them. And if you’d like to be included on > the invite to

Re: Welcome Jose Torres as a Spark committer

Congrats Jose! On Wed, Jan 30, 2019 at 10:44 AM Gabor Somogyi wrote: > Congrats Jose! > > BR, > G > > On Wed, Jan 30, 2019 at 9:05 AM Nuthan Reddy > wrote: > >> Congrats Jose, >> >> Regards, >> Nuthan Reddy >> >> >> >> On Wed, Jan 30, 2019 at 1:22 PM Marco Gaido >> wrote: >> >>> Congrats,

Re: [Discussion] Clarification regarding Stateful Aggregations over Structured Streaming

Hi, Databricks runtime as you already know has this enhancement and so it is considered a good option if you want to decouple state from the jvm. Some arguments why to do so are given by the Flink paper along with incremental snapshotting: http://www.vldb.org/pvldb/vol10/p1718-carbone.pdf. Also

Re: [ANNOUNCE] Announcing Apache Spark 2.4.0

Awesome! On Thu, Nov 8, 2018 at 9:36 PM, Jules Damji wrote: > Indeed! > > Sent from my iPhone > Pardon the dumb thumb typos :) > > On Nov 8, 2018, at 11:31 AM, Dongjoon Hyun > wrote: > > Finally, thank you all. Especially, thanks to the release manager, Wenchen! > > Bests, > Dongjoon. > > > On

Re: Test and support only LTS JDK release?

Red Hat: https://access.redhat.com/articles/1299013#OpenJDK_Lifecycle_Dates_and_RHEL_versions Stavros On Wed, Nov 7, 2018 at 12:13 PM, Kazuaki Ishizaki wrote: > This entry includes a good figure for support lifecycle. > https://www.azul.com/products/zulu-and-zulu-enterprise/zulu- >

Re: Plan on Structured Streaming in next major/minor release?

e this thread to discuss >>> about possible use cases, and if we would like to move forward to >>> individual thread we could initiate (or resurrect) its discussion thread. >>> >>> For queryable state, at least there seems no workaround in Spark to >>> prov

Re: What if anything to fix about k8s for the 2.4.0 RC5?

//github.com/apache/spark/pull/22827 is >> merged and 2.4 is unblocked. >> >> I'll cut RC5 shortly after the weekend, and it will be great to include >> the change proposed here. >> >> Thanks, >> Wenchen >> >> On Fri, Oct 26, 2018 at 12:55 AM St

Re: What if anything to fix about k8s for the 2.4.0 RC5?

the blocker tickets are resolved. > > > > Thanks, > > > > Xiao > > > > Sean Owen 于2018年10月25日周四 上午8:44写道： > >> > >> Yes, I agree, and perhaps you are best placed to do that for 2.4.0 RC5 > :) > >> > >> On Thu, Oct 25, 201

Re: What if anything to fix about k8s for the 2.4.0 RC5?

I agree these tests should be manual for now but should be run somehow before a release to make sure things are working right? For the other issue: https://issues.apache.org/jira/browse/SPARK-25835 . On Thu, Oct 25, 2018 at 6:29 PM, Stavros Kontopoulos < stavros.kontopou...@lightbend.com>

Re: What if anything to fix about k8s for the 2.4.0 RC5?

I will open a jira for the profile propagation issue and have a look to fix it. Stavros On Thu, Oct 25, 2018 at 6:16 PM, Erik Erlandson wrote: > > I would be comfortable making the integration testing manual for now. A > JIRA for ironing out how to make it reliable for automatic as a goal for

Re: [VOTE] SPARK 2.4.0 (RC4)

k-docker-integration-tests` at Spark > 1.6.x era and stopped since Spark 2.0.0. > >- http://central.maven.org/maven2/org/apache/spark/spark- >docker-integration-tests_2.10/ >- http://central.maven.org/maven2/org/apache/spark/spark- >docker-integration-tests_2.1

Re: [VOTE] SPARK 2.4.0 (RC4)

don't need to run with the normal suite of >>>> tests, and can be considered tests run manually by developers running these >>>> scripts? Basically, KubernetesSuite shouldn't run in a normal mvn install? >>>> > >>>> > I don't think this has to block the release even if so, just trying >>>> to get to the bottom of it. >>>> >>>> - >>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>> >>>> -- Stavros Kontopoulos *Senior Software Engineer* *Lightbend, Inc.* *p: +30 6977967274 <%2B1%20650%20678%200020>* *e: stavros.kontopou...@lightbend.com*

Re: [VOTE] SPARK 2.4.0 (RC4)

; someone would have to first build a k8s distro from source too? > >> It's not a 'must' that this be automatic but nothing else fails out of > the box. > >> I feel like I might be misunderstanding the setup here. > >> On Mon, Oct 22, 2018 at 7:25 PM Stavros

Re: [VOTE] SPARK 2.4.0 (RC4)

> tar (child): Error is not recoverable: exiting now > tar: Child returned status 2 > tar: Error is not recoverable: exiting now > scripts/setup-integration-test-env.sh: line 85: > /home/srowen/spark-2.4.0/resource-managers/kubernetes/ >

Re: Plan on Structured Streaming in next major/minor release?

and the reason why the progress got stopped? > > 2018년 10월 21일 (일) 오후 10:27, Stavros Kontopoulos < > stavros.kontopou...@lightbend.com>님이 작성: > >> That is a very interesting list thanks. I could create a design doc as a >> starting pointing for discussion if this is a feature

Re: Plan on Structured Streaming in next major/minor release?

That is a very interesting list thanks. I could create a design doc as a starting pointing for discussion if this is a feature we would like to have. Regards, Stavros On Sun, Oct 21, 2018 at 3:04 PM, JackyLee wrote: > Thanks for raising them. > > FYI, I believe this open issues could also be

Re: [DISCUSS][K8S] Local dependencies with Kubernetes

@Marcelo is correct. Mesos does not have something similar. Only Yarn does due to the distributed cache thing. I have described most of the above in the the jira also there are some other options. Best, Stavros On Fri, Oct 5, 2018 at 8:28 PM, Marcelo Vanzin wrote: > On Fri, Oct 5, 2018 at 7:54

Re: [DISCUSS][K8S] Local dependencies with Kubernetes

Hi Rob, Interesting topic and affects UX a lot. I provided my thoughts in the related jira. Best, Stavros On Fri, Oct 5, 2018 at 5:53 PM, Rob Vesse wrote: > Folks > > > > One of the big limitations of the current Spark on K8S implementation is > that it isn’t possible to use local

Re: welcome a new batch of committers

Congrats! On Wednesday, October 3, 2018, sujith chacko wrote: > Great news Congrats all for achieving the feat !!! > > On Wed, 3 Oct 2018 at 2:29 PM, Reynold Xin wrote: > >> Hi all, >> >> The Apache Spark PMC has recently voted to add several new committers to >> the project, for their

Re: Python friendly API for Spark 3.0

Regarding Python 3.x upgrade referenced earlier. Some people already gone down that path of upgrading: https://blogs.dropbox.com/tech/2018/09/how-we-rolled-out-one-of-the-largest-python-3-migrations-ever They describe some good reasons. Stavros On Tue, Sep 18, 2018 at 6:35 PM, Erik Erlandson

Re: [VOTE] SPARK 2.4.0 (RC2)

+1 Stavros On Sat, Sep 29, 2018 at 5:59 AM, Sean Owen wrote: > +1, with comments: > > There are 5 critical issues for 2.4, and no blockers: > SPARK-25378 ArrayData.toArray(StringType) assume UTF8String in 2.4 > SPARK-25325 ML, Graph 2.4 QA: Update user guide for new features & APIs >

Re: [DISCUSS][K8S] Supporting advanced pod customisation

There is a design document that covers a lot of concerns: https://docs.google.com/document/d/1pcyH5f610X2jyJW9WbWHnj8jktQPLlbbmmUwdeK4fJk, validation included. We had a discussion about validation (validate before we hit the api server) and was considered too much. In general regarding Rob's

Re: [VOTE] SPARK 2.4.0 (RC1)

t;> >> > FAQ >>>> >> >> > >>>> >> >> > = >>>> >> >> > How can I help test this release? >>>> >> >> > = >>>> >> >> > >>>> >> >> > If you are a Spark user, you can help us test this release by >>>> taking >>>> >> >> > an existing Spark workload and running on this release >>>> candidate, then >>>> >> >> > reporting any regressions. >>>> >> >> > >>>> >> >> > If you're working in PySpark you can set up a virtual env and >>>> install >>>> >> >> > the current RC and see if anything important breaks, in the >>>> Java/Scala >>>> >> >> > you can add the staging repository to your projects resolvers >>>> and test >>>> >> >> > with the RC (make sure to clean up the artifact cache >>>> before/after so >>>> >> >> > you don't end up building with a out of date RC going forward). >>>> >> >> > >>>> >> >> > === >>>> >> >> > What should happen to JIRA tickets still targeting 2.4.0? >>>> >> >> > === >>>> >> >> > >>>> >> >> > The current list of open tickets targeted at 2.4.0 can be found >>>> at: >>>> >> >> > https://issues.apache.org/jira/projects/SPARK and search for >>>> "Target Version/s" = 2.4.0 >>>> >> >> > >>>> >> >> > Committers should look at those and triage. Extremely important >>>> bug >>>> >> >> > fixes, documentation, and API tweaks that impact compatibility >>>> should >>>> >> >> > be worked on immediately. Everything else please retarget to an >>>> >> >> > appropriate release. >>>> >> >> > >>>> >> >> > == >>>> >> >> > But my bug isn't fixed? >>>> >> >> > == >>>> >> >> > >>>> >> >> > In order to make timely releases, we will typically not hold the >>>> >> >> > release unless the bug in question is a regression from the >>>> previous >>>> >> >> > release. That being said, if there is something which is a >>>> regression >>>> >> >> > that has not been correctly targeted please ping me or a >>>> committer to >>>> >> >> > help target the issue. >>>> >>>> >>>> >>>> -- >>>> Marcelo >>>> >>>> - >>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>> >>>> -- Stavros Kontopoulos *Senior Software Engineer* *Lightbend, Inc.* *p: +30 6977967274 <%2B1%20650%20678%200020>* *e: stavros.kontopou...@lightbend.com*

Re: [VOTE] SPARK 2.4.0 (RC1)

> I believe -1 votes are merited only for correctness bugs and regressions > since the previous release. > > Does SPARK-23200 count as either? > > 2018년 9월 17일 (월) 오전 9:40, Stavros Kontopoulos lightbend.com>님이 작성: > >> -1 >> >> I would like to see: https://g

Re: [VOTE] SPARK 2.4.0 (RC1)

-1 I would like to see: https://github.com/apache/spark/pull/22392 in, as discussed here: https://issues.apache.org/jira/browse/SPARK-23200. It is important IMHO for streaming on K8s. I just started testing it btw. Also 2.12.7(https://contributors.scala-lang.org/t/2-12-7-release/2301,

[DISCUSS][CORE] Exposing application status metrics via a source

Hi all, I have a PR https://github.com/apache/spark/pull/22381 that exposes application status metrics (related jira: SPARK-25394). So far metrics tooling needs to scrape the metrics rest api to get metrics like job delay, stages failed, stages completed etc. >From devops perspective it is good

custom sink & model transformation

Hi, Just copying form users, since got no response. Is it unsfate to do model prediction within a custom sink eg. model.transform(df)? I see that the only transformation done is adding a prediction column AFAIK, does that change the execution plan? Thanks, Stavros

Re: code freeze and branch cut for Apache Spark 2.4

+1 that would be great Sean, also you put a lot of effort in there, would make sense to wait a bit. Stavros On Fri, Aug 31, 2018 at 12:00 AM, Sean Owen wrote: > I know it's famous last words, but we really might be down to the last > fix: https://github.com/apache/spark/pull/22264 More a

Re: Set up Scala 2.12 test build in Jenkins

leaner- if this was not caught by suite, perhaps we should > augment it ... > > Regards > Mridul > > On Mon, Aug 6, 2018 at 1:08 AM Stavros Kontopoulos lightbend.com> wrote: > >> Closure cleaner's initial purpose AFAIK is to clean the dependencies >> brought in w

Re: Set up Scala 2.12 test build in Jenkins

t case defines a serializable class inside the test case, and the > class doesn't refer to anything in the outer class. Ideally it can be > serialized after cleaning up the closure. > > > > This is somehow a very weird way to define a class, so I'm not sure how > serious the problem

Re: Set up Scala 2.12 test build in Jenkins

just > help catch these things. > > On Sun, Aug 5, 2018 at 2:16 PM Stavros Kontopoulos lightbend.com> wrote: > >> Hi Sean, >> >> I run a quick build so the failing tests seem to be: >> >> - SPARK-17644: After one stage is aborted for too many failed attempts

Re: Set up Scala 2.12 test build in Jenkins

Hi Sean, I run a quick build so the failing tests seem to be: - SPARK-17644: After one stage is aborted for too many failed attempts, subsequent stagesstill behave correctly on fetch failures *** FAILED *** A job with one fetch failure should eventually succeed (DAGSchedulerSuite.scala:2422)

Re: code freeze and branch cut for Apache Spark 2.4

I have a PR out for SPARK-14540 (Support Scala 2.12 closures and Java 8 lambdas in ClosureCleaner). This should allows us to add support for Scala 2.12, I think we can resolve this long standing issue with 2.4. Best, Stavros On Tue, Jul 31, 2018 at 4:07 PM, Tomasz Gawęda wrote: > Hi, > > what

Re: code freeze and branch cut for Apache Spark 2.4

st of the major features targeted for the 2.4 release. > I also have a lot of free time during this release timeframe and should be > able to figure out problems that may appear during the release. > > Thanks, > Wenchen > > On Fri, Jul 27, 2018 at 11:27 PM Stavros Kontopoulos light

Re: code freeze and branch cut for Apache Spark 2.4

possible to have an >> extension of code freeze for a week? >> >> Thanks, >> >> Xingbo >> >> 2018-07-07 0:47 GMT+08:00 Reynold Xin : >> >>> FYI 6 mo is coming up soon since the last release. We will cut the >>> branch and code freeze

Re: Time for 2.3.2?

ream >>>>>> > join. Users can hit this bug if one of the join side is partitioned >>>>>> by a >>>>>> > subset of the join keys. >>>>>> > >>>>>> > SPARK-24552: Task attempt numbers are reused when stage

Re: Scala 2.12 support

the work so we can move forward with this long-standing issue. PS. I think my previous msg didnt reach the list... Best, Stavros On Thu, Jun 21, 2018 at 3:37 PM, Stavros Kontopoulos < stavros.kontopou...@lightbend.com> wrote: > Hi all, > > Scala team @Lightbend (Lukas, Adriaan, Jas

Re: queryable state & streaming

ri, Dec 8, 2017 at 10:47 AM, Stavros Kontopoulos < > st.kontopou...@gmail.com> wrote: > >> Cool Burak do you have a pointer, should I take the initiative for a >> first design document or Databricks is working on it? >> >> Best, >> Stavros >> >>

Re: queryable state & streaming

Just to re-phrase my question: Would query-able state make a viable SPIP? Regards, Stavros On Thu, Dec 7, 2017 at 1:34 PM, Stavros Kontopoulos < st.kontopou...@gmail.com> wrote: > Hi, > > Maybe this has been discussed before. Given the fact that many streaming > apps o

queryable state & streaming

Hi, Maybe this has been discussed before. Given the fact that many streaming apps out there use state extensively, could be a good idea to make Spark expose streaming state with an external API like other systems do (Kafka streams, Flink etc), in order to facilitate interactive queries? Regards,

Re: SparkOscope: Enabling Spark Optimization through Cross-stack Monitoring and Visualization

Cool work! I will have a look to the project. Cheers On Fri, Feb 5, 2016 at 11:09 AM, Pete Robbins wrote: > Yiannis, > > I'm interested in what you've done here as I was looking for ways to allow > the Spark UI to display custom metrics in a pluggable way without having to

Re: Using spark MLlib without installing Spark

how it to anyone; please delete/destroy and inform the >> sender immediately. >> >> >> On Sunday, 22 November 2015 at 00:38, bowen zhang wrote: >> >> Hi folks, >> I am a big fan of Spark's Mllib package. I have a java web app where I >> want to run some ml jobs inside the web app. My question is: is there a way >> to just import spark-core and spark-mllib jars to invoke my ML jobs without >> installing the entire Spark package? All the tutorials related Spark seems >> to indicate installing Spark is a pre-condition for this. >> >> Thanks, >> Bowen >> >> >> >> >> >> > -- Stavros Kontopoulos <http://www.typesafe.com> <http://www.typesafe.com>

[VOTE] Release Apache Spark 1.5.2 (RC1)

+1 (non binding) I tested several of the examples on mesos latest version (fine and coarse-grained modes) and they work fine. Hope not too late...though.. -- Stavros Kontopoulos <http://www.typesafe.com> <http://www.typesafe.com>