Re: Jenkins is down

[Takeshi Yamamuro]

Great! Thanks, Shane!

On Mon, Jul 6, 2020 at 10:21 AM Hyukjin Kwon  wrote:

> Thanks Shane!
>
> 2020년 7월 6일 (월) 오전 9:30, shane knapp ☠ 님이 작성:
>
>> hey all, i was out of town for the weekend and noticed it was down this
>> morning and restarted the service.  it's been pretty flaky recently, so
>> i'll take a much closer look at things this coming week.
>>
>> On Sun, Jul 5, 2020 at 1:14 PM Dongjoon Hyun 
>> wrote:
>>
>>> Hi, All.
>>>
>>> Now, AmpLab Jenkins farm came back online.
>>>
>>>
>>> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/
>>>
>>> Also, many PRBuilder jobs were re-started 10 minutes ago.
>>>
>>> Bests,
>>> Dongjoon.
>>>
>>>
>>> On Fri, Jul 3, 2020 at 4:43 AM Hyukjin Kwon  wrote:
>>>
 Hi all and Shane,

 Is there something wrong with the Jenkins machines? Seems they are down.

>>>
>>
>> --
>> Shane Knapp
>> Computer Guy / Voice of Reason
>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>> https://rise.cs.berkeley.edu
>>
>

-- 
---
Takeshi Yamamuro

Re: m2 cache issues in Jenkins?

[Hyukjin Kwon]

Shane, can we remove .m2 in worker machine 4?

2020년 7월 3일 (금) 오전 8:18, Jungtaek Lim 님이 작성:

> Looks like Jenkins service itself becomes unstable. It took considerable
> time to just open the test report for a specific build, and Jenkins doesn't
> pick the request on rebuild (retest this, please) in Github comment.
>
> On Thu, Jul 2, 2020 at 2:12 PM Hyukjin Kwon  wrote:
>
>> Ah, okay. Actually there already is -
>> https://issues.apache.org/jira/browse/SPARK-31693. I am reopening.
>>
>> 2020년 7월 2일 (목) 오후 2:06, Holden Karau 님이 작성:
>>
>>> We don't I didn't file one originally, but Shane reminded me to in the
>>> future.
>>>
>>> On Wed, Jul 1, 2020 at 9:44 PM Hyukjin Kwon  wrote:
>>>
 Nope, do we have an existing ticket? I think we can reopen if there is.

 2020년 7월 2일 (목) 오후 1:43, Holden Karau 님이 작성:

> Huh interesting that it’s the same worker. Have you filed a ticket to
> Shane?
>
> On Wed, Jul 1, 2020 at 8:50 PM Hyukjin Kwon 
> wrote:
>
>> Hm .. seems this is happening again in amp-jenkins-worker-04 ;(.
>>
>> 2020년 6월 25일 (목) 오전 3:15, shane knapp ☠ 님이 작성:
>>
>>> done:
>>> -bash-4.1$ cd .m2
>>> -bash-4.1$ ls
>>> repository
>>> -bash-4.1$ time rm -rf *
>>>
>>> real17m4.607s
>>> user0m0.950s
>>> sys 0m18.816s
>>> -bash-4.1$
>>>
>>> On Wed, Jun 24, 2020 at 10:50 AM shane knapp ☠ 
>>> wrote:
>>>
 ok, i've taken that worker offline and once the job running on it
 finishes, i'll wipe the cache.

 in the future, please file a JIRA and assign it to me so i don't
 have to track my work through emails to the dev@ list.  ;)

 thanks!

 shane

 On Wed, Jun 24, 2020 at 10:48 AM Holden Karau 
 wrote:

> The most recent one I noticed was
> https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124437/console
>  which
> was run on  amp-jenkins-worker-04.
>
> On Wed, Jun 24, 2020 at 10:44 AM shane knapp ☠ <
> skn...@berkeley.edu> wrote:
>
>> for those weird failures, it's super helpful to provide which
>> workers are showing these issues.  :)
>>
>> i'd rather not wipe all of the m2 caches on all of the workers,
>> as we'll then potentially get blacklisted again if we download too 
>> many
>> packages from apache.org.
>>
>> On Tue, Jun 23, 2020 at 5:58 PM Holden Karau <
>> hol...@pigscanfly.ca> wrote:
>>
>>> Hi Folks,
>>>
>>> I've been see some weird failures on Jenkins and it looks like
>>> it might be from the m2 cache. Would it be OK to clean it out? Or 
>>> is it
>>> important?
>>>
>>> Cheers,
>>>
>>> Holden
>>>
>>> --
>>> Twitter: https://twitter.com/holdenkarau
>>> Books (Learning Spark, High Performance Spark, etc.):
>>> https://amzn.to/2MaRAG9  
>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>
>>
>>
>> --
>> Shane Knapp
>> Computer Guy / Voice of Reason
>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>> https://rise.cs.berkeley.edu
>>
>
>
> --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9  
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>


 --
 Shane Knapp
 Computer Guy / Voice of Reason
 UC Berkeley EECS Research / RISELab Staff Technical Lead
 https://rise.cs.berkeley.edu

>>>
>>>
>>> --
>>> Shane Knapp
>>> Computer Guy / Voice of Reason
>>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>>> https://rise.cs.berkeley.edu
>>>
>> --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9  
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>

>>>
>>> --
>>> Twitter: https://twitter.com/holdenkarau
>>> Books (Learning Spark, High Performance Spark, etc.):
>>> https://amzn.to/2MaRAG9  
>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>
>>

Re: Jenkins is down

[Hyukjin Kwon]

Thanks Shane!

2020년 7월 6일 (월) 오전 9:30, shane knapp ☠ 님이 작성:

> hey all, i was out of town for the weekend and noticed it was down this
> morning and restarted the service.  it's been pretty flaky recently, so
> i'll take a much closer look at things this coming week.
>
> On Sun, Jul 5, 2020 at 1:14 PM Dongjoon Hyun 
> wrote:
>
>> Hi, All.
>>
>> Now, AmpLab Jenkins farm came back online.
>>
>>
>> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/
>>
>> Also, many PRBuilder jobs were re-started 10 minutes ago.
>>
>> Bests,
>> Dongjoon.
>>
>>
>> On Fri, Jul 3, 2020 at 4:43 AM Hyukjin Kwon  wrote:
>>
>>> Hi all and Shane,
>>>
>>> Is there something wrong with the Jenkins machines? Seems they are down.
>>>
>>
>
> --
> Shane Knapp
> Computer Guy / Voice of Reason
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu
>

Re: Jenkins is down

[shane knapp ☠]

hey all, i was out of town for the weekend and noticed it was down this
morning and restarted the service.  it's been pretty flaky recently, so
i'll take a much closer look at things this coming week.

On Sun, Jul 5, 2020 at 1:14 PM Dongjoon Hyun 
wrote:

> Hi, All.
>
> Now, AmpLab Jenkins farm came back online.
>
>
> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/
>
> Also, many PRBuilder jobs were re-started 10 minutes ago.
>
> Bests,
> Dongjoon.
>
>
> On Fri, Jul 3, 2020 at 4:43 AM Hyukjin Kwon  wrote:
>
>> Hi all and Shane,
>>
>> Is there something wrong with the Jenkins machines? Seems they are down.
>>
>

-- 
Shane Knapp
Computer Guy / Voice of Reason
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

Re: Apache Spark 3.1 Feature Expectation (Dec. 2020)

[Dongjoon Hyun]

Thank you for sharing your opinions, Jacky, Maxim, Holden, Jungtaek, Yi,
Tom, Gabor, Felix.

I also want to include both `New Features` and `Improvements` together
according to the above discussion.

When I checked the item status as of today, it looked like the following.
In short, I removed K8s GA and DSv2 Stabilization explicitly from ON-TRACK
list according to the given concerns. For those items, we can try to build
a consensus for Apache Spark 3.2 (June 2021) or later.

ON-TRACK
1. Support Scala 2.13 (SPARK-25075)
2. Use Apache Hadoop 3.2 by default for better cloud support (SPARK-32058)
3. Stage Level Scheduling (SPARK-27495)
4. Support filter pushdown more (CSV is already shipped by SPARK-30323 in
3.0)
- Support filters pushdown to JSON (SPARK-30648 in 3.1)
- Support filters pushdown to Avro (SPARK-XXX in 3.1)
- Support nested attributes of filters pushed down to JSON
5. Support JDBC Kerberos w/ keytab (SPARK-12312)

NICE TO HAVE OR DEFERRED TO APACHE SPARK 3.2
1. Declaring Kubernetes Scheduler GA
- Should we also consider the shuffle service refactoring to support
pluggable storage engines as targeting the 3.1 release? (Holden)
- I think pluggable storage in shuffle is essential for k8s GA (Felix)
- Use remote storage for persisting shuffle data (SPARK-25299)
2. DSv2 Stabilization? (The followings and more)
- SPARK-31357 Catalog API for view metadata
- SPARK-31694 Add SupportsPartitions Catalog APIs on DataSourceV2

As we know, we work willingly and voluntarily. If something lands on the
`master` branch before the feature freeze (November), it will be a part of
Apache Spark 3.1, of course.

Thanks,
Dongjoon.

On Sun, Jul 5, 2020 at 12:21 PM Felix Cheung 
wrote:

> I think pluggable storage in shuffle is essential for k8s GA
>
> --
> *From:* Holden Karau 
> *Sent:* Monday, June 29, 2020 9:33 AM
> *To:* Maxim Gekk
> *Cc:* Dongjoon Hyun; dev
> *Subject:* Re: Apache Spark 3.1 Feature Expectation (Dec. 2020)
>
> Should we also consider the shuffle service refactoring to support
> pluggable storage engines as targeting the 3.1 release?
>
> On Mon, Jun 29, 2020 at 9:31 AM Maxim Gekk 
> wrote:
>
>> Hi Dongjoon,
>>
>> I would add:
>> - Filters pushdown to JSON (https://github.com/apache/spark/pull/27366)
>> - Filters pushdown to other datasources like Avro
>> - Support nested attributes of filters pushed down to JSON
>>
>> Maxim Gekk
>>
>> Software Engineer
>>
>> Databricks, Inc.
>>
>>
>> On Mon, Jun 29, 2020 at 7:07 PM Dongjoon Hyun 
>> wrote:
>>
>>> Hi, All.
>>>
>>> After a short celebration of Apache Spark 3.0, I'd like to ask you the
>>> community opinion on Apache Spark 3.1 feature expectations.
>>>
>>> First of all, Apache Spark 3.1 is scheduled for December 2020.
>>> - https://spark.apache.org/versioning-policy.html
>>>
>>> I'm expecting the following items:
>>>
>>> 1. Support Scala 2.13
>>> 2. Use Apache Hadoop 3.2 by default for better cloud support
>>> 3. Declaring Kubernetes Scheduler GA
>>> In my perspective, the last main missing piece was Dynamic
>>> allocation and
>>> - Dynamic allocation with shuffle tracking is already shipped at 3.0.
>>> - Dynamic allocation with worker decommission/data migration is
>>> targeting 3.1. (Thanks, Holden)
>>> 4. DSv2 Stabilization
>>>
>>> I'm aware of some more features which are on the way currently, but I
>>> love to hear the opinions from the main developers and more over the main
>>> users who need those features.
>>>
>>> Thank you in advance. Welcome for any comments.
>>>
>>> Bests,
>>> Dongjoon.
>>>
>>
>
> --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9  
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>

Re: Jenkins is down

[Dongjoon Hyun]

Hi, All.

Now, AmpLab Jenkins farm came back online.

https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/

Also, many PRBuilder jobs were re-started 10 minutes ago.

Bests,
Dongjoon.


On Fri, Jul 3, 2020 at 4:43 AM Hyukjin Kwon  wrote:

> Hi all and Shane,
>
> Is there something wrong with the Jenkins machines? Seems they are down.
>

Re: Apache Spark 3.1 Feature Expectation (Dec. 2020)

[Felix Cheung]

I think pluggable storage in shuffle is essential for k8s GA


From: Holden Karau 
Sent: Monday, June 29, 2020 9:33 AM
To: Maxim Gekk
Cc: Dongjoon Hyun; dev
Subject: Re: Apache Spark 3.1 Feature Expectation (Dec. 2020)

Should we also consider the shuffle service refactoring to support pluggable 
storage engines as targeting the 3.1 release?

On Mon, Jun 29, 2020 at 9:31 AM Maxim Gekk 
mailto:maxim.g...@databricks.com>> wrote:
Hi Dongjoon,

I would add:
- Filters pushdown to JSON (https://github.com/apache/spark/pull/27366)
- Filters pushdown to other datasources like Avro
- Support nested attributes of filters pushed down to JSON

Maxim Gekk

Software Engineer

Databricks, Inc.


On Mon, Jun 29, 2020 at 7:07 PM Dongjoon Hyun 
mailto:dongjoon.h...@gmail.com>> wrote:
Hi, All.

After a short celebration of Apache Spark 3.0, I'd like to ask you the 
community opinion on Apache Spark 3.1 feature expectations.

First of all, Apache Spark 3.1 is scheduled for December 2020.
- https://spark.apache.org/versioning-policy.html

I'm expecting the following items:

1. Support Scala 2.13
2. Use Apache Hadoop 3.2 by default for better cloud support
3. Declaring Kubernetes Scheduler GA
In my perspective, the last main missing piece was Dynamic allocation and
- Dynamic allocation with shuffle tracking is already shipped at 3.0.
- Dynamic allocation with worker decommission/data migration is targeting 
3.1. (Thanks, Holden)
4. DSv2 Stabilization

I'm aware of some more features which are on the way currently, but I love to 
hear the opinions from the main developers and more over the main users who 
need those features.

Thank you in advance. Welcome for any comments.

Bests,
Dongjoon.


--
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 

YouTube Live Streams: https://www.youtube.com/user/holdenkarau

Re: [DISCUSS] Apache Spark 3.0.1 Release

[wuyi]

Ok, after having another look, I think it only affects local cluster deploy
mode, which is for testing only. 


wuyi wrote
> Please also includes https://issues.apache.org/jira/browse/SPARK-32120 in
> Spark 3.0.1. It's a regression compares to Spark 3.0.0-preview2.
> 
> Thanks,
> Yi Wu
> 
> 
> Yuanjian Li wrote
>> Hi dev-list,
>> 
>> I’m writing this to raise the discussion about Spark 3.0.1 feasibility
>> since 4 blocker issues were found after Spark 3.0.0:
>> 
>> 
>>1.
>> 
>>[SPARK-31990]
>> ;
>> The
>>state store compatibility broken will cause a correctness issue when
>>Streaming query with `dropDuplicate` uses the checkpoint written by
>> the
>> old
>>Spark version.
>>2.
>> 
>>[SPARK-32038]
>> ;
>> The
>>regression bug in handling NaN values in COUNT(DISTINCT)
>>3.
>> 
>>[SPARK-31918]
>> [WIP]
>>CRAN requires to make it working with the latest R 4.0. It makes the
>> 3.0
>>release unavailable on CRAN, and only supports R [3.5, 4.0)
>>4.
>> 
>>[SPARK-31967]
>> ;
>>Downgrade vis.js to fix Jobs UI loading time regression
>> 
>> 
>> I also noticed branch-3.0 already has 39 commits
>> ;
>> after Spark 3.0.0. I think it would be great if we have Spark 3.0.1 to
>> deliver the critical fixes.
>> 
>> Any comments are appreciated.
>> 
>> Best,
>> 
>> Yuanjian
> 
> 
> 
> 
> 
> --
> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
> 
> -
> To unsubscribe e-mail: 

> dev-unsubscribe@.apache





--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [DISCUSS] Apache Spark 3.0.1 Release

[wuyi]

Please also includes https://issues.apache.org/jira/browse/SPARK-32120 in
Spark 3.0.1. It's a regression compares to Spark 3.0.0-preview2.

Thanks,
Yi Wu


Yuanjian Li wrote
> Hi dev-list,
> 
> I’m writing this to raise the discussion about Spark 3.0.1 feasibility
> since 4 blocker issues were found after Spark 3.0.0:
> 
> 
>1.
> 
>[SPARK-31990] ;
> The
>state store compatibility broken will cause a correctness issue when
>Streaming query with `dropDuplicate` uses the checkpoint written by the
> old
>Spark version.
>2.
> 
>[SPARK-32038] ;
> The
>regression bug in handling NaN values in COUNT(DISTINCT)
>3.
> 
>[SPARK-31918]
> [WIP]
>CRAN requires to make it working with the latest R 4.0. It makes the
> 3.0
>release unavailable on CRAN, and only supports R [3.5, 4.0)
>4.
> 
>[SPARK-31967] ;
>Downgrade vis.js to fix Jobs UI loading time regression
> 
> 
> I also noticed branch-3.0 already has 39 commits
> ;
> after Spark 3.0.0. I think it would be great if we have Spark 3.0.1 to
> deliver the critical fixes.
> 
> Any comments are appreciated.
> 
> Best,
> 
> Yuanjian





--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org