Re: 4.0.0-preview1 test report: running on Yarn

Thank you all so much for the kind words of encouragement on my first test report. As a follow up, I ran all my HDFS and Yarn nodes on Java 8 - including my Nodemanagers. I then modified Spark's conf/spark-defaults.conf according to Mr. Pan's prior post, and it worked: I was able

Re: 4.0.0-preview1 test report: running on Yarn

FYI, I have submitted SPARK-48651(https://github.com/apache/spark/pull/47010) to update the Spark on YARN docs for JDK configuration, looking forward to your feedback. Thanks, Cheng Pan > On Jun 18, 2024, at 02:00, George Magiros wrote: > > I successfully submitted and ran org.apache.spark.e

Re: 4.0.0-preview1 test report: running on Yarn

You don’t need to upgrade Java for HDFS and YARN. Just keep using Java 8 for Hadoop and set JAVA_HOME to Java 17 for Spark applications[1]. 0. Install Java 17 on all nodes, for example, under /opt/openjdk-17 1. Modify $SPARK_CONF_DIR/spark-env.sh export JAVA_HOME=/opt/openjdk-17 2. Modify $SPAR

Re: 4.0.0-preview1 test report: running on Yarn

Thanks for sharing! Yea Spark 4.0 is built using Java 17. On Tue, Jun 18, 2024 at 5:07 AM George Magiros wrote: > I successfully submitted and ran org.apache.spark.examples.SparkPi on Yarn > using 4.0.0-preview1. However I got it to work only after fixing an issue > with the Yarn nodemanagers (

4.0.0-preview1 test report: running on Yarn

I successfully submitted and ran org.apache.spark.examples.SparkPi on Yarn using 4.0.0-preview1. However I got it to work only after fixing an issue with the Yarn nodemanagers (Hadoop v3.3.6 and v3.4.0). Namely the issue was: 1. If the nodemanagers used java 11, Yarn threw an error about not find

Re: [FYI] SPARK-45981: Improve Python language test coverage

Awesome! On Sat, Dec 2, 2023 at 2:33 PM Dongjoon Hyun wrote: > Hi, All. > > As a part of Apache Spark 4.0.0 (SPARK-44111), the Apache Spark community > starts to have test coverage for all supported Python versions from Today. > > - https://github.com/apache/spark/actio

[FYI] SPARK-45981: Improve Python language test coverage

Hi, All. As a part of Apache Spark 4.0.0 (SPARK-44111), the Apache Spark community starts to have test coverage for all supported Python versions from Today. - https://github.com/apache/spark/actions/runs/7061665420 Here is a summary. 1. Main CI: All PRs and commits on `master` branch are

[VOTE][RESULT] PySpark Test Framework

The vote passes with 10 +1s (nine binding +1s) and one +0. Thank you all for your participation and comments! (* = binding) +1: - Holden Karau (*) - Reynold Xin (*) - Mich Talebzadeh - Maciej Szymkiewicz (*) - Hyukjin Kwon (*) - Dongjoon Hyun (*) - Ruifeng Zheng (*) - Xinrong Meng (*) - Liang-Chi

Re: [VOTE][SPIP] PySpark Test Framework

i > >>>>> > >>>>> "The Internals Of" Online Books > >>>>> Follow me on https://twitter.com/jaceklaskowski > >>>>> > >>>>> > >>>>> > >>>>> On Wed, Jun

Re: [VOTE][SPIP] PySpark Test Framework

;>>> >>>>> Pozdrawiam, >>>>> Jacek Laskowski >>>>> >>>>> "The Internals Of" Online Books >>>>> Follow me on https://twitter.com/jaceklaskowski >>>>> >>>>> >>>>

Re: [VOTE][SPIP] PySpark Test Framework

://books.japila.pl/> >>>> Follow me on https://twitter.com/jaceklaskowski >>>> >>>> <https://twitter.com/jaceklaskowski> >>>> >>>> >>>> On Wed, Jun 21, 2023 at 5:11 PM Amanda Liu < >>>> amandastephanie.

Re: [VOTE][SPIP] PySpark Test Framework

iam, >>> Jacek Laskowski >>> >>> "The Internals Of" Online Books <https://books.japila.pl/> >>> Follow me on https://twitter.com/jaceklaskowski >>> >>> <https://twitter.com/jaceklaskowski> >>> >>> &g

Re: [VOTE][SPIP] PySpark Test Framework

apila.pl/> >> Follow me on https://twitter.com/jaceklaskowski >> >> <https://twitter.com/jaceklaskowski> >> >> >> On Wed, Jun 21, 2023 at 5:11 PM Amanda Liu >> wrote: >> >>> Hi all, >>> >>> I'd like to start

Re: [VOTE][SPIP] PySpark Test Framework

skowski> > > > On Wed, Jun 21, 2023 at 5:11 PM Amanda Liu > wrote: > >> Hi all, >> >> I'd like to start the vote for SPIP: PySpark Test Framework. >> >> The high-level summary for the SPIP is that it proposes an official test >> framework

Re: [VOTE][SPIP] PySpark Test Framework

to start the vote for SPIP: PySpark Test Framework. > > The high-level summary for the SPIP is that it proposes an official test > framework for PySpark. Currently, there are only disparate open-source > repos and blog posts for PySpark testing resources. We can streamline and > simplify

Re: [VOTE][SPIP] PySpark Test Framework

t;> I’d like to start with a +1, better Python testing tools integrated into >>> the project make sense. >>> >>> On Wed, Jun 21, 2023 at 8:11 AM Amanda Liu >>> wrote: >>> >>>> Hi all, >>>> >>>> I'd like to

Re: [VOTE][SPIP] PySpark Test Framework

to start with a +1, better Python testing tools integrated into the project make sense. On Wed, Jun 21, 2023 at 8:11 AM Amanda Liu wrote: Hi all, I'd like to start the vote for SPIP: PySpark Test Framework. The high-level summar

Re: [VOTE][SPIP] PySpark Test Framework

;> On Wed, Jun 21, 2023 at 8:11 AM Amanda Liu >> wrote: >> >>> Hi all, >>> >>> I'd like to start the vote for SPIP: PySpark Test Framework. >>> >>> The high-level summary for the SPIP is that it proposes an official test >>&

Re: [VOTE][SPIP] PySpark Test Framework

ion. On Wed, 21 Jun 2023 at 16:30, Holden Karau wrote: > I’d like to start with a +1, better Python testing tools integrated into > the project make sense. > > On Wed, Jun 21, 2023 at 8:11 AM Amanda Liu > wrote: > >> Hi all, >> >> I'd like to start

Re: [VOTE][SPIP] PySpark Test Framework

stephanieliu@ gmail. com > ( amandastephanie...@gmail.com ) > wrote: > > >> Hi all, >> >> I'd like to start the vote for SPIP: PySpark Test Framework. >> >> The high-level summary for the SPIP is that it proposes an official test >> framework for PyS

Re: [VOTE][SPIP] PySpark Test Framework

I’d like to start with a +1, better Python testing tools integrated into the project make sense. On Wed, Jun 21, 2023 at 8:11 AM Amanda Liu wrote: > Hi all, > > I'd like to start the vote for SPIP: PySpark Test Framework. > > The high-level summary for the SPIP is that it

[VOTE][SPIP] PySpark Test Framework

Hi all, I'd like to start the vote for SPIP: PySpark Test Framework. The high-level summary for the SPIP is that it proposes an official test framework for PySpark. Currently, there are only disparate open-source repos and blog posts for PySpark testing resources. We can streamline and sim

Re: [DISCUSS] SPIP: Add PySpark Test Framework

ary damages arising from such loss, damage or destruction. On Wed, 14 Jun 2023 at 00:10, Amanda Liu wrote: > Hi all, > > I'd like to start a discussion about implementing an official PySpark test > framework. Currently, there's no official test framework, but only various &

Re: [DISCUSS] SPIP: Add PySpark Test Framework

+1 from my side sounds good, it will be helpful to both users and contributors to improve the test coverage On Wed, Jun 14, 2023 at 8:27 AM Hyukjin Kwon wrote: > Yeah, I have been thinking about this too, and Holden did some work here > that this SPIP will reuse. I support this. > >

Re: [DISCUSS] SPIP: Add PySpark Test Framework

Yeah, I have been thinking about this too, and Holden did some work here that this SPIP will reuse. I support this. On Wed, 14 Jun 2023 at 08:10, Amanda Liu wrote: > Hi all, > > I'd like to start a discussion about implementing an official PySpark test > framework. Curr

[DISCUSS] SPIP: Add PySpark Test Framework

Hi all, I'd like to start a discussion about implementing an official PySpark test framework. Currently, there's no official test framework, but only various open-source repos and blog posts. Many of these open-source resources are very popular, which demonstrates user-demand for PySpa

Observed consistent test failure in master (ParquetIOSuite)

Hi, I just observed the test failure in ParquetIOSuite which I can consistently reproduce with IntelliJ. Haven't had a chance to run a test with maven/sbt. I filed SPARK-39622 <https://issues.apache.org/jira/browse/SPARK-39622> for this failure. It'd be awesome if someone havi

Maven Test blocks with TransportCipherSuite

Hi, team. I run the maven command to run unit test, and have a NPE. command: ./build/mvn test refer to https://spark.apache.org/docs/latest/building-spark.html#running-tests NPE is as follow: 22/05/20 16:32:45.450 main WARN AbstractChannelHandlerContext: Failed to mark a promise as failure

Re: Skip single integration test case in Spark on K8s

-user@spark For cloud backend, you need to exclude minikube specific tests and local-only test (SparkRemoteFileTest). -Dtest.exclude.tags=minikube,local You can find more options including SBT commands here. https://github.com/apache/spark/tree/master/resource-managers/kubernetes

Skip single integration test case in Spark on K8s

Hi Spark team I am running Spark kubernetes integration test suite on cloud. build/mvn install \ -f pom.xml \ -pl resource-managers/kubernetes/integration-tests -am -Pscala-2.12 -Phadoop-3.1.1 -Phive -Phive-thriftserver -Pyarn -Pkubernetes -Pkubernetes-integration-tests \ -Djava.version=8

Re: [How To] run test suites for specific module

stOnly org.apache.spark.scheduler.DAGSchedulerSuite Hope this helps Best regards, Qian Sun Fangjia Shen 于2022年1月25日周二 07:44写道： > Hello all, > > How do you run Spark's test suites when you want to test the correctness > of your code? Is there a way to run a specific test suite for Spark? For > e

Re: [How To] run test suites for specific module

Hi, Please check the relevant section of the developer tools docs: https://spark.apache.org/developer-tools.html#running-individual-tests On 1/25/22 00:44, Fangjia Shen wrote: > Hello all, > > How do you run Spark's test suites when you want to test the correctness > of your

[How To] run test suites for specific module

Hello all, How do you run Spark's test suites when you want to test the correctness of your code? Is there a way to run a specific test suite for Spark? For example, running test suite XXXSuite alone, instead of every class under the test/ directories. Here's some background info

Re: ivy unit test case filing for Spark

You would have to make it available? This doesn't seem like a spark issue. On Tue, Dec 21, 2021, 10:48 AM Pralabh Kumar wrote: > Hi Spark Team > > I am building a spark in VPN . But the unit test case below is failing. > This is pointing to ivy location which cannot be reached

ivy unit test case filing for Spark

Hi Spark Team I am building a spark in VPN . But the unit test case below is failing. This is pointing to ivy location which cannot be reached within VPN . Any help would be appreciated test("SPARK-33084: Add jar support Ivy URI -- default transitive = true") { *sc *= new SparkC

Re: [PSA] Please read: PR builder now runs test and build in your forked repository

seems too strict ( >>> https://github.com/apache/spark/pull/32168#issuecomment-819736508). >>> Thanks Yikun. >>> I think we should make it less strict. I can create a PR right away but >>> would like to encourage Yikun or Kent to do it in order to keep the cre

Re: [PSA] Please read: PR builder now runs test and build in your forked repository

right away but would like to encourage Yikun or Kent to do it in order to keep the credits of their investigation.2021년 4월 15일 (목) 오전 7:21, Dongjoon Hyun <dongjoon.h...@gmail.com>님이 작성:Hi, Kent.I checked (1) in your PR, but those test result comments look correct to me.Please note that bo

Re: [PSA] Please read: PR builder now runs test and build in your forked repository

>> >> 2021년 4월 15일 (목) 오전 7:21, Dongjoon Hyun 님이 작성: >> >>> Hi, Kent. >>> >>> I checked (1) in your PR, but those test result comments look correct to >>> me. >>> Please note that both Jenkins and GitHub Action leave the same number of &g

Re: [PSA] Please read: PR builder now runs test and build in your forked repository

o it in order to keep the credits > of their investigation. > > 2021년 4월 15일 (목) 오전 7:21, Dongjoon Hyun 님이 작성: > >> Hi, Kent. >> >> I checked (1) in your PR, but those test result comments look correct to >> me. >> Please note that both Jenkins and GitHub

Re: [PSA] Please read: PR builder now runs test and build in your forked repository

away but would like to encourage Yikun or Kent to do it in order to keep the credits of their investigation. 2021년 4월 15일 (목) 오전 7:21, Dongjoon Hyun 님이 작성: > Hi, Kent. > > I checked (1) in your PR, but those test result comments look correct to > me. > Please note that both Jen

Re: [PSA] Please read: PR builder now runs test and build in your forked repository

Hi, Kent. I checked (1) in your PR, but those test result comments look correct to me. Please note that both Jenkins and GitHub Action leave the same number of comments on the same GitHash. Given that, there are not fake comments. It looks like a real result of your commits on that PR. GitHash

Re: [PSA] Please read: PR builder now runs test and build in your forked repository

/apache/spark/pull/32144#issuecomment-819679970 2. New Forks have to turn on GitHub action by the fork owner manually 3. `Notify test workflow` keeps waiting when the build flow canceled or the whole fork gone 4. After refreshed master or even re-forked :(, I still got failures and seems not alone

Re: [PSA] Please read: PR builder now runs test and build in your forked repository

Thank you again, Hyukjin. Bests, Dongjoon. On Wed, Apr 14, 2021 at 5:25 AM Kent Yao wrote: > Cool, thanks! > > Hyukjin Kwon 于2021年4月14日周三 下午8:19写道： > >> Good point! I had to clarify. >> Once is enough. The sync is needed for your branch to include the changes >> of https://github.com/apache/sp

Re: [PSA] Please read: PR builder now runs test and build in your forked repository

Cool, thanks! Hyukjin Kwon 于2021年4月14日周三 下午8:19写道： > Good point! I had to clarify. > Once is enough. The sync is needed for your branch to include the changes > of https://github.com/apache/spark/pull/32092. > > > 2021년 4월 14일 (수) 오후 9:11, Kent Yao 님이 작성: > >> Hi Hyukjin, >> >> > Please sync you

Re: [PSA] Please read: PR builder now runs test and build in your forked repository

Good point! I had to clarify. Once is enough. The sync is needed for your branch to include the changes of https://github.com/apache/spark/pull/32092. 2021년 4월 14일 (수) 오후 9:11, Kent Yao 님이 작성: > Hi Hyukjin, > > > Please sync your branch to the latest master branch in Apache Spark in > order for

Re: [PSA] Please read: PR builder now runs test and build in your forked repository

Hi Hyukjin,> Please sync your branch to the latest master branch in Apache Spark in order for the main repository to run the workflow and detect it. Do we need to sync master for every PR or just one-time cost to 

Re: [PSA] Please read: PR builder now runs test and build in your forked repository

Cool~Thanks, Hyukjin Yuanjian Li 于2021年4月14日周三 下午3:39写道： > Awesome! Thanks for making this happen, Hyukjin! > > Yi Wu 于2021年4月14日周三 下午2:51写道： > >> Thanks for the great work, Hyukjin! >> >> On Wed, Apr 14, 2021 at 1:00 PM Gengliang Wang wrote: >> >>> Thanks for the amazing work, Hyukjin! >>> I

Re: [PSA] Please read: PR builder now runs test and build in your forked repository

Awesome! Thanks for making this happen, Hyukjin! Yi Wu 于2021年4月14日周三 下午2:51写道： > Thanks for the great work, Hyukjin! > > On Wed, Apr 14, 2021 at 1:00 PM Gengliang Wang wrote: > >> Thanks for the amazing work, Hyukjin! >> I created a PR for trial and it looks well so far: >> https://github.com/a

Re: [PSA] Please read: PR builder now runs test and build in your forked repository

Thanks for the great work, Hyukjin! On Wed, Apr 14, 2021 at 1:00 PM Gengliang Wang wrote: > Thanks for the amazing work, Hyukjin! > I created a PR for trial and it looks well so far: > https://github.com/apache/spark/pull/32158 > > On Wed, Apr 14, 2021 at 12:47 PM Hyukjin Kwon wrote: > >> Hi al

Re: [PSA] Please read: PR builder now runs test and build in your forked repository

Thanks for the amazing work, Hyukjin! I created a PR for trial and it looks well so far: https://github.com/apache/spark/pull/32158 On Wed, Apr 14, 2021 at 12:47 PM Hyukjin Kwon wrote: > Hi all, > > After https://github.com/apache/spark/pull/32092 merged, now we run the > GitHub Actions > workfl

Re: [PSA] Please read: PR builder now runs test and build in your forked repository

Hi all, After https://github.com/apache/spark/pull/32092 merged, now we run the GitHub Actions workflows in your forked repository. In short, please see this example HyukjinKwon#34 1. You create a PR and your repository triggers the workflow. You

[PSA] Please read: PR builder now runs test and build in your forked repository

Hi all, After https://github.com/apache/spark/pull/32092 merged, now we run the GitHub Actions workflows in your forked repository. In short, please see this example HyukjinKwon#34 1. You create a PR and your repository triggers the workflow. You

Re: K8s Integration test is unable to run because of the unavailable libs

34820 [2] https://github.com/apache/spark/pull/31923 Regards, Yikun Yi Wu 于2021年3月22日周一 下午2:15写道： > Hi devs, > > It seems like the K8s Integration test is unable to run recently because > of the unavailable libs: > > Err:20 http://security.debian.org/debian-security buster/upd

K8s Integration test is unable to run because of the unavailable libs

Hi devs, It seems like the K8s Integration test is unable to run recently because of the unavailable libs: Err:20 http://security.debian.org/debian-security buster/updates/main amd64 libldap-common all 2.4.47+dfsg-3+deb10u4 404 Not Found [IP: 151.101.194.132 80] Err:21 http

Re: K8s integration test failure ("credentials Jenkins is using is probably wrong...")

single worker. >> >> On Tue, Feb 23, 2021 at 8:46 AM Phillip Henry >> wrote: >> >>> >>> Hi, >>> >>> Silly question: the Jenkins build for my PR is failing but it seems >>> outside of my control. What must I do to remedy this? >>> >&

Re: K8s integration test failure ("credentials Jenkins is using is probably wrong...")

t; >> https://github.com/apache/spark/pull/31535 >> >> but Spark QA is telling me "Kubernetes integration test status failure". >> >> The Jenkins job says "SUCCESS" but also barfs with: >> >> FileNotFoundException means that the cre

Re: K8s integration test failure ("credentials Jenkins is using is probably wrong...")

ted > > https://github.com/apache/spark/pull/31535 > > but Spark QA is telling me "Kubernetes integration test status failure". > > The Jenkins job says "SUCCESS" but also barfs with: > > FileNotFoundException means that the credentials Jenkins is using is pr

K8s integration test failure ("credentials Jenkins is using is probably wrong...")

Hi, Silly question: the Jenkins build for my PR is failing but it seems outside of my control. What must I do to remedy this? I've submitted https://github.com/apache/spark/pull/31535 but Spark QA is telling me "Kubernetes integration test status failure". The Jenkins job say

Re: Unit test failure in spark-core

tephen Coy mailto:s...@infomedia.com.au.INVALID>> wrote: Hi all, When trying to build current master with a simple: mvn clean install I get a consistent unit test failure in core: [ERROR] Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 5.403 s &

Unit test failure in spark-core

Hi all, When trying to build current master with a simple: mvn clean install I get a consistent unit test failure in core: [ERROR] Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 5.403 s <<< FAILURE! - in org.apache.spark.launcher.SparkLauncherSui

Test Failures on macOS 10.15.4

Hi, I am new in the development of Spark. When I tried to run unit tests locally on macOS 10.15.4, everything went smoothly except a single testcase - SPARK-6330 regression test. After a few hours struggling with it, I moved to Linux and it passed magically. My OS is Ubuntu 18.0.4. Digging into

Re: SQL test failures in PR builder?

an -fdx" > git clean -fdx > > # Configure per-build-executor Ivy caches to avoid SBT Ivy lock contention > export HOME="/home/sparkivy/per-executor-caches/$EXECUTOR_NUMBER" > mkdir -p "$HOME" > export SBT_OPTS="-Duser.home=$HOME -Dsbt.ivy.home=$HOME/.ivy2&

Re: SQL test failures in PR builder?

or-caches/$EXECUTOR_NUMBER" mkdir -p "$HOME" export SBT_OPTS="-Duser.home=$HOME -Dsbt.ivy.home=$HOME/.ivy2" export SPARK_VERSIONS_SUITE_IVY_PATH="$HOME/.ivy2" ./dev/run-tests-jenkins # Hack to ensure that at least one JVM suite always runs in order to prevent spu

Re: SQL test failures in PR builder?

tps://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4960/testReport/ > >> > >> > >> org.apache.spark.sql.hive.thriftserver.SparkMetadataOperationSuite.Spark's > >> own GetSchemasOperation(SparkGetSchemasOperation)14 ms2 > >> > >> org.apache.spark.sql.hive.thriftserver.Thrift

Re: SQL test failures in PR builder?

m seeing consistent failures in the PR builder when touching SQL code: >> >> https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4960/testReport/ >> >> org.apache.spark.sql.hive.thriftserver.SparkMetadataOperationSuite.Spark's >> own GetSchemasOp

Re: SQL test failures in PR builder?

uilder/4960/testReport/ > > org.apache.spark.sql.hive.thriftserver.SparkMetadataOperationSuite.Spark's > own GetSchemasOperation(SparkGetSchemasOperation)14 ms2 > org.apache.spark.sql.hive.thriftserver.ThriftServerWithSparkContextSuite.(It > is not a test it is a sbt.testing.SuiteSelector) > > Looks like this has failed

SQL test failures in PR builder?

ion)14 ms2 org.apache.spark.sql.hive.thriftserver.ThriftServerWithSparkContextSuite.(It is not a test it is a sbt.testing.SuiteSelector) Looks like this has failed about 6 builds in the past few days. Has anyone seen this / has a clue what's causing it? errors are like ... java.sql.SQLExcepti

Re: Adding JIRA ID as the prefix for the test case name

I opened a PR - https://github.com/apache/spark-website/pull/232 2019년 11월 19일 (화) 오전 9:22, Hyukjin Kwon 님이 작성: > Let me document as below in few days: > > 1. For Python and Java, write a single comment that starts with JIRA ID > and short description, e.g. (SPARK-X: test blah bl

Re: Adding JIRA ID as the prefix for the test case name

Let me document as below in few days: 1. For Python and Java, write a single comment that starts with JIRA ID and short description, e.g. (SPARK-X: test blah blah) 2. For R, use JIRA ID as a prefix for its test name. assuming everybody is happy. 2019년 11월 18일 (월) 오전 11:36, Hyukjin Kwon 님이

Re: Adding JIRA ID as the prefix for the test case name

Actually there are not so many Java test cases in Spark (because Scala runs on JVM as everybody knows)[1]. Given that, I think we can avoid to put some efforts on this for now .. I don't mind if somebody wants to give a shot since it looks good anyway but to me I wouldn't spend so mu

Re: Adding JIRA ID as the prefix for the test case name

Test reporters do often contain some assumptions about the characters in the test methods. Historically JUnit XML reporters have never sanitised the method names so XML injection attacks have been fairly trivial. Haven't tried this for a while. That whole JUnit XML report "standard&quo

Re: Adding JIRA ID as the prefix for the test case name

e for non Scala tests? Other languages >> (Java, Python, R) don't support using string as a test name. >> >> Best Regards, >> Ryan >> >> >> On Thu, Nov 14, 2019 at 4:04 AM Hyukjin Kwon wrote: >> >>> I opened a PR - https://github.com/

Re: Adding JIRA ID as the prefix for the test case name

7;t support using string as a test name. > > Best Regards, > Ryan > > > On Thu, Nov 14, 2019 at 4:04 AM Hyukjin Kwon wrote: > >> I opened a PR - https://github.com/apache/spark-website/pull/231 >> >> 2019년 11월 13일 (수) 오전 10:43, Hyukjin Kwon 님이 작성: >>

Re: Adding JIRA ID as the prefix for the test case name

this is about test description and not test file name right? if yes I don’t see a problem. From: Hyukjin Kwon Sent: Thursday, November 14, 2019 6:03:02 PM To: Shixiong(Ryan) Zhu Cc: dev ; Felix Cheung ; Shivaram Venkataraman Subject: Re: Adding JIRA ID as the

Re: Adding JIRA ID as the prefix for the test case name

Yeah, sounds good to have it. In case of R, it seems not quite common to write down JIRA ID [1] but looks some have the prefix in its test name in general. In case of Python and Java, seems we time to time write a JIRA ID in the comment right under the test method [2][3]. Given this pattern, I

Re: Adding JIRA ID as the prefix for the test case name

Should we also add a guideline for non Scala tests? Other languages (Java, Python, R) don't support using string as a test name. Best Regards, Ryan On Thu, Nov 14, 2019 at 4:04 AM Hyukjin Kwon wrote: > I opened a PR - https://github.com/apache/spark-website/pull/231 > > 2019년

Re: Adding JIRA ID as the prefix for the test case name

I opened a PR - https://github.com/apache/spark-website/pull/231 2019년 11월 13일 (수) 오전 10:43, Hyukjin Kwon 님이 작성: > > In general a test should be self descriptive and I don't think we should > be adding JIRA ticket references wholesale. Any action that the reader has > to take to

Re: Adding JIRA ID as the prefix for the test case name

> In general a test should be self descriptive and I don't think we should be adding JIRA ticket references wholesale. Any action that the reader has to take to understand why a test was introduced is one too many. However in some cases the thing we are trying to test is very subtle and

Re: Adding JIRA ID as the prefix for the test case name

Let's suggest "SPARK-12345:" but not go back and change a bunch of test cases. I'd add this only when a test specifically targets a certain issue. It's a nice-to-have, not super essential, just because in the rare case you need to understand why a test asserts something,

Re: Adding JIRA ID as the prefix for the test case name

+1 Two confusions to clarify: 1. what if multiple JIRA IDs relating to the same test? we just take the very first JIRA ID? 2. are we going to have a full scan of all existing tests and attach a JIRA ID to it? Thank you Hyukjin :) On Tue, Nov 12, 2019 at 1:47 PM Dongjoon Hyun wrote: > Th

Re: Adding JIRA ID as the prefix for the test case name

Thank you for the suggestion, Hyukjin. Previously, we added Jira IDs for the bug fix PR test cases as Gabor said. For the new features (and improvements), we didn't add them because all test cases in the newly added test suite share the same prefix JIRA ID in that case. It might

Re: Adding JIRA ID as the prefix for the test case name

+1 for having that consistent rule in test names. +1 for making it a guideline. +1 defining exact guides in general. Until now I've followed the alternative (only add the prefix when the JIRA's type is bug) and that way I knew that such tests contain edge cases. In case of new features

Re: Adding JIRA ID as the prefix for the test case name

In few days, I will wrote this in our guidelines probably after rewording it a bit better: 1. Add a prefix into a test name when a PR adds a couple of tests. 2. Uses "SPARK-XXXX: test name" format. Please let me know if you have any different opinion about what/when to write the JIRA

Re: Adding JIRA ID as the prefix for the test case name

+1 for making it a guideline. This is helpful when the test cases are moved to a different file. On Mon, Nov 11, 2019 at 3:23 PM Takeshi Yamamuro wrote: > +1 for having that consistent rule in test names. > This is a trivial problem though, I think documenting this rule in the > con

Re: Adding JIRA ID as the prefix for the test case name

+1 for having that consistent rule in test names. This is a trivial problem though, I think documenting this rule in the contribution guide might be able to make reviewer overhead a little smaller. Bests, Takeshi On Tue, Nov 12, 2019 at 1:46 AM Hyukjin Kwon wrote: > Hi all, > > Maybe

Adding JIRA ID as the prefix for the test case name

Hi all, Maybe it's not a big deal but it brought some confusions time to time into Spark dev and community. I think it's time to discuss about when/which format to add a JIRA ID as a prefix for the test case name in Scala test cases. Currently we have many test case names with prefixe

Re: Standardizing test build config

amplab. josh rosen and patrick wendell would have a LOT more context about some of the decisions made. > See https://issues.apache.org/jira/browse/SPARK-28900 for a summary of > how the different build configs are set up, and why we might need to > standardize them to fully test with

Standardizing test build config

I'm surfacing this to dev@ as the right answers may depend on a lot of historical decisions that I don't know about. See https://issues.apache.org/jira/browse/SPARK-28900 for a summary of how the different build configs are set up, and why we might need to standardize them to fully tes

Re: displaying "Test build" in PR

"Can one of the admins verify this patch?" is a corrected message, as Jenkins won't test your PR until an admin approves it. BTW I think "5 minutes" is a reasonable delay for PR testing. It usually takes days to review and merge a PR, so I don't think seeing test pr

Re: displaying "Test build" in PR

es and will trigger jobs based on creation/approval to test/code > updates/etc. > > On Mon, Aug 12, 2019 at 11:25 AM Younggyu Chun > wrote: > >> Hi All, >> >> I have a quick question about PR. Once I create a PR I'm not able to see >> if "Test build"

Re: displaying "Test build" in PR

when you create a PR, the jenkins pull request builder job polls every ~5 or so minutes and will trigger jobs based on creation/approval to test/code updates/etc. On Mon, Aug 12, 2019 at 11:25 AM Younggyu Chun wrote: > Hi All, > > I have a quick question about PR. Once I create a P

displaying "Test build" in PR

Hi All, I have a quick question about PR. Once I create a PR I'm not able to see if "Test build" is being processed. But I can see this after a few minutes or hours later. Is it possible to see if "Test Build" is being processed after PR is created right away? Thank you, Younggyu Chun

Re: sparkmaster-test-sbt-hadoop-2.7 failing RAT check

. > > > > see: > > > https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.7/6043/console > > > > -- > > Shane Knapp > > UC Berkeley EECS Research / RISELab Staff Technical Lead > > https://rise.cs.berkeley.edu > -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu

Re: sparkmaster-test-sbt-hadoop-2.7 failing RAT check

(We have two PRs to patch it up anyway already) On Mon, Jun 24, 2019 at 11:39 AM shane knapp wrote: > > i'm aware and will be looking in to this later today. > > see: > https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.7/6043/console > > -- &

sparkmaster-test-sbt-hadoop-2.7 failing RAT check

i'm aware and will be looking in to this later today. see: https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.7/6043/console -- Shane Knapp UC Berkeley EECS Research / RISELab Staff Technical Lead https://rise.cs.berkeley.edu

Raise Jenkins test timeout? with alternatives

I have a big PR that keeps failing because it his the 300 minute build timeout: https://github.com/apache/spark/pull/24314 https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4703/console It's because it touches so much code that all tests run including things like Kinesis. It l

Re: Run a specific PySpark test or group of tests

b.com/apache/spark/compare/master…nchammas:pytest >>>> <https://github.com/apache/spark/compare/master...nchammas:pytest> >>>> >>>> For example I was able to do this from that branch and it did the right >>>> thing, running only the tests with

Re: Run a specific PySpark test or group of tests

le to do this from that branch and it did the right >>> thing, running only the tests with string in their name: >>> >>> python [pytest *]$ ../bin/spark-submit ./pytest-run-tests.py >>> ./pyspark/sql/tests.py -v -k string >>> >>> However, looking more

Re: Run a specific PySpark test or group of tests

ithub.com/apache/spark/compare/master...nchammas:pytest> >> >> For example I was able to do this from that branch and it did the right >> thing, running only the tests with string in their name: >> >> python [pytest *]$ ../bin/spark-submit ./pytest-run-tests.py >&

Re: Run a specific PySpark test or group of tests

yspark/sql/tests.py -v -k string > > However, looking more closely at the whole test setup, I’m hesitant to > work any further on this. > > My intention was to see if we could leverage pytest, tox, and other test > tools that are standard in the Python ecosystem to replace some of

  1   2   3   4   5   >