testing

Dr Mich Talebzadeh, Architect | Data Science | Financial Crime | Forensic Analysis | GDPR view my Linkedin profile

[RESULT][VOTE] SPIP: Testing Framework for Spark UI Javascript files

Hi Spark dev, The vote[1] has now closed. The results are: +1 Votes(*=binding): - Mridul Muralidharan* - Ye Zhou - Dongjoon Hyun* - Reynold Xin* - Yang Jie - Gengliang Wang* - Ruifeng Zheng* - Binjie Yang - Kent Yao 0 Votes: None -1 Votes: None The vote is successful with 5 binding +1 votes.

Re: [VOTE] SPIP: Testing Framework for Spark UI Javascript files

t; > >> *日期**: *2023年11月25日 星期六 14:35 > > >> *收件人**: *Dongjoon Hyun > > >> *抄送**: *Ye Zhou , Mridul Muralidharan < > > >> mri...@gmail.com>, Kent Yao , dev > > >> *主题**: *Re: [VOTE] SPIP: Testing Framework for

Re: [VOTE] SPIP: Testing Framework for Spark UI Javascript files

t; > >> > >> *发件人**: *Reynold Xin > >> *日期**: *2023年11月25日 星期六 14:35 > >> *收件人**: *Dongjoon Hyun > >> *抄送**: *Ye Zhou , Mridul Muralidharan < > >> mri...@gmail.com>, Kent Yao , dev > >> *主题**: *Re: [VOTE] SPIP: Testing Framework f

Re: [VOTE] SPIP: Testing Framework for Spark UI Javascript files

*Ye Zhou , Mridul Muralidharan < >> mri...@gmail.com>, Kent Yao , dev >> *主题**: *Re: [VOTE] SPIP: Testing Framework for Spark UI Javascript files >> >> >> >> +1 >> >> [image: 图像已被发件人删除。] >> >> >> >> >> >> On

Re: [VOTE] SPIP: Testing Framework for Spark UI Javascript files

+1 On Sat, Nov 25, 2023 at 2:50 AM yangjie01 wrote: > +1 > > > > *发件人**: *Reynold Xin > *日期**: *2023年11月25日 星期六 14:35 > *收件人**: *Dongjoon Hyun > *抄送**: *Ye Zhou , Mridul Muralidharan < > mri...@gmail.com>, Kent Yao , dev > *主题**: *Re: [VOTE] SPIP: Testi

Re: [VOTE] SPIP: Testing Framework for Spark UI Javascript files

+1 发件人: Reynold Xin 日期: 2023年11月25日 星期六 14:35 收件人: Dongjoon Hyun 抄送: Ye Zhou , Mridul Muralidharan , Kent Yao , dev 主题: Re: [VOTE] SPIP: Testing Framework for Spark UI Javascript files +1 On Fri, Nov 24, 2023 at 10:19 PM, Dongjoon Hyun mailto:dongjoon.h...@gmail.com>> wrote: +1

Re: [VOTE] SPIP: Testing Framework for Spark UI Javascript files

+1 On Fri, Nov 24, 2023 at 10:19 PM, Dongjoon Hyun < dongjoon.h...@gmail.com > wrote: > > +1 > > > Thanks, > Dongjoon. > > On Fri, Nov 24, 2023 at 7:14 PM Ye Zhou < zhouyejoe@ gmail. com ( > zhouye...@gmail.com ) > wrote: > > >> +1(non-binding) >> >> On Fri, Nov 24, 2023 at 11:16 Mridul M

Re: [VOTE] SPIP: Testing Framework for Spark UI Javascript files

+1 Thanks, Dongjoon. On Fri, Nov 24, 2023 at 7:14 PM Ye Zhou wrote: > +1(non-binding) > > On Fri, Nov 24, 2023 at 11:16 Mridul Muralidharan > wrote: > >> >> +1 >> >> Regards, >> Mridul >> >> On Fri, Nov 24, 2023 at 8:21 AM Kent Yao wrote: >> >>> Hi Spark Dev, >>> >>> Following the discussion

Re: [VOTE] SPIP: Testing Framework for Spark UI Javascript files

+1(non-binding) On Fri, Nov 24, 2023 at 11:16 Mridul Muralidharan wrote: > > +1 > > Regards, > Mridul > > On Fri, Nov 24, 2023 at 8:21 AM Kent Yao wrote: > >> Hi Spark Dev, >> >> Following the discussion [1], I'd like to start the vote for the SPIP [2]. >> >> The SPIP aims to improve the test c

Re: [VOTE] SPIP: Testing Framework for Spark UI Javascript files

+1 Regards, Mridul On Fri, Nov 24, 2023 at 8:21 AM Kent Yao wrote: > Hi Spark Dev, > > Following the discussion [1], I'd like to start the vote for the SPIP [2]. > > The SPIP aims to improve the test coverage and develop experience for > Spark UI-related javascript codes. > > This thread will b

[VOTE] SPIP: Testing Framework for Spark UI Javascript files

Hi Spark Dev, Following the discussion [1], I'd like to start the vote for the SPIP [2]. The SPIP aims to improve the test coverage and develop experience for Spark UI-related javascript codes. This thread will be open for at least the next 72 hours. Please vote accordingly, [ ] +1: Accept the

Re: [DISCUSS] SPIP: Testing Framework for Spark UI Javascript files

posing a new UI test framework for Apache Spark 4.0. > > > > It looks very useful. > > > > Thanks, > > Dongjoon. > > > > > > On Tue, Nov 21, 2023 at 1:51 AM Kent Yao wrote: > > > >> Hi Spark Dev, > >> > >> This is a cal

Re: [DISCUSS] SPIP: Testing Framework for Spark UI Javascript files

AM Kent Yao wrote: > >> Hi Spark Dev, >> >> This is a call to discuss a new SPIP: Testing Framework for >> Spark UI Javascript files [1]. The SPIP aims to improve the test >> coverage and develop experience for Spark UI-related javascript >> codes. >> The

Re: [DISCUSS] SPIP: Testing Framework for Spark UI Javascript files

i Spark Dev, >> >> This is a call to discuss a new SPIP: Testing Framework for >> Spark UI Javascript files [1]. The SPIP aims to improve the test >> coverage and develop experience for Spark UI-related javascript >> codes. >> The Jest [2], a JavaScript Testing Frame

Help for testing Windows specific fix (SPARK-23015)

Hi all, I used to have my Windows environment in another laptop but that laptop is broken now so I don't have Windows env to test Windows PRs out (e.g., https://github.com/apache/spark/pull/43706). If anyone has a Windows env, would appreciate it if you take a look at this. Thanks.

Re: [DISCUSS] SPIP: Testing Framework for Spark UI Javascript files

Thank you for proposing a new UI test framework for Apache Spark 4.0. It looks very useful. Thanks, Dongjoon. On Tue, Nov 21, 2023 at 1:51 AM Kent Yao wrote: > Hi Spark Dev, > > This is a call to discuss a new SPIP: Testing Framework for > Spark UI Javascript files [1]. The

[DISCUSS] SPIP: Testing Framework for Spark UI Javascript files

Hi Spark Dev, This is a call to discuss a new SPIP: Testing Framework for Spark UI Javascript files [1]. The SPIP aims to improve the test coverage and develop experience for Spark UI-related javascript codes. The Jest [2], a JavaScript Testing Framework licensed under MIT, will be used to build

Re: Tools for regression testing

At the wikipedia regression testing page https://en.wikipedia.org/wiki/Regression_testing <https://en.wikipedia.org/wiki/Regression_testing> Under use " Regression tests can be broadly categorized as functional tests <https://en.wikipedia.org/wiki/Functional_test> or

Re: Tools for regression testing

likely impact. Integration testing can be achieved through CI/CD which I believe Spark relied on Jenkins until recently. HTH view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* Use it at yo

Re: Tools for regression testing

Hm, then what are you looking for besides all the tests in Spark? On Thu, Mar 24, 2022, 2:34 PM Mich Talebzadeh wrote: > Thanks > > I know what unit testing is. The question was not about unit testing. it > was specific to regression testing > <https://katalon.com/res

Re: Tools for regression testing

Thanks I know what unit testing is. The question was not about unit testing. it was specific to regression testing <https://katalon.com/resources-center/blog/regression-testing#:~:text=Regression%20testing%20is%20a%20software,functionality%20of%20the%20existing%20features.> artifacts .

Re: Tools for regression testing

Yes, Spark uses unit tests. https://app.codecov.io/gh/apache/spark https://en.wikipedia.org/wiki/Unit_testing man. 21. mar. 2022 kl. 15:46 skrev Mich Talebzadeh < mich.talebza...@gmail.com>: > Hi, > > As a matter of interest do Spark releases deploy a specific regression

Tools for regression testing

Hi, As a matter of interest do Spark releases deploy a specific regression testing tool? Thanks view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* Use it at your own risk. Any a

Re: Migration onto error classes and their testing

n > - QueryCompilationErrorsSuite ... query compilation or eagerly executing > commands > - QueryParsingErrorsSuite ... parsing errors > > Here is an example https://github.com/apache/spark/pull/35157 of how an > existing Java exception can be replaced, and testing of related > error c

Migration onto error classes and their testing

eagerly executing commands - QueryParsingErrorsSuite ... parsing errors Here is an example https://github.com/apache/spark/pull/35157 of how an existing Java exception can be replaced, and testing of related error classes. If there will be no objections to the approach, I would create an umbrella

Re: PR testing and flaky tests (triggering executions separately)

AFAIK, Github actions triggered checks are almost the same as SparkPullRequestBuilder except that it has one more Scala 2.13 check. So at least we don't have to wait for both SparkPullRequestBuilder and Github actions to merge PR. On Fri, Mar 26, 2021 at 6:09 PM Attila Zsolt Piros < piros.attila.z

PR testing and flaky tests (triggering executions separately)

Hello everybody, A commit for a PR triggers three different test executions: - SparkPullRequestBuilder - SparkPullRequestBuilder-K8s - Github actions triggered checks Each one of these could contain a flaky test. *Is it already possible to trigger these executions separately?* If not

Re: minikube and kubernetes cluster versions for integration testing

Thanks Shane! As I promised: - the PR about documenting the change <https://github.com/apache/spark-website/pull/321> - my Spark PR <https://github.com/apache/spark/pull/31829> with checking Minikube versions and using a simpler way to configure kubernetes client for integration t

Re: minikube and kubernetes cluster versions for integration testing

fwiw, upgrading minikube and the associated VM drivers is potentially a PITA. your PR will absolutely be tested before merging. :) On Thu, Mar 4, 2021 at 10:13 AM attilapiros wrote: > Thanks Shane! > > I can do the documentation task and the Minikube version check can be > incorporated into my

Re: minikube and kubernetes cluster versions for integration testing

Thanks Shane! I can do the documentation task and the Minikube version check can be incorporated into my PR. When my PR is finalized (probably next week) I will create a jira for you and you can set up the test systems and you can even test my PR before merging it. Is this possible / fine for you

Re: minikube and kubernetes cluster versions for integration testing

> > On Tue, Mar 2, 2021 at 5:03 AM Attila Zsolt Piros < > piros.attila.zs...@gmail.com> wrote: > >> Hi All, >> >> I am working on PR to change kubernetes integration testing and use the >> `minikube kubectl -- config view --minify` output to build the kuberne

Re: minikube and kubernetes cluster versions for integration testing

rsion checking. Thanks, Dongjoon. On Tue, Mar 2, 2021 at 5:03 AM Attila Zsolt Piros < piros.attila.zs...@gmail.com> wrote: > Hi All, > > I am working on PR to change kubernetes integration testing and use the > `minikube kubectl -- config view --minify` output to build the kub

minikube and kubernetes cluster versions for integration testing

Hi All, I am working on PR to change kubernetes integration testing and use the `minikube kubectl -- config view --minify` output to build the kubernetes client config. This solution has the advantage of not using hardcoded values like 8443 for server port (which is wrong when the vm-driver is

Status of Spark testing on ARM64

Hi all, I will give you some informations about ARM CI of Spark: Our team and community are working on build/test Spark master on ARM64 server, after find and fix some issues[1], we have integrated two ARM testing jobs[2] to community CI(AMPLAB Jenkins), they run as daily job and have been

[GitHub] HyukjinKwon closed pull request #162: Add a note about Spark build requirement at PySpark testing guide in Developer Tools

HyukjinKwon closed pull request #162: Add a note about Spark build requirement at PySpark testing guide in Developer Tools URL: https://github.com/apache/spark-website/pull/162 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below

[GitHub] HyukjinKwon commented on issue #162: Add a note about Spark build requirement at PySpark testing guide in Developer Tools

HyukjinKwon commented on issue #162: Add a note about Spark build requirement at PySpark testing guide in Developer Tools URL: https://github.com/apache/spark-website/pull/162#issuecomment-448164740 Thanks guys! This is an

[GitHub] HyukjinKwon commented on issue #162: Add a note about Spark build requirement at PySpark testing guide in Developer Tools

HyukjinKwon commented on issue #162: Add a note about Spark build requirement at PySpark testing guide in Developer Tools URL: https://github.com/apache/spark-website/pull/162#issuecomment-448075651 adding @squito as well FYI

[GitHub] HyukjinKwon opened a new pull request #162: Add a note about Spark build requirement at PySpark testing guide in Developer Tools

HyukjinKwon opened a new pull request #162: Add a note about Spark build requirement at PySpark testing guide in Developer Tools URL: https://github.com/apache/spark-website/pull/162 I received some feedback about running PySpark tests via private emails. Unlike SBT or Maven testing

[GitHub] HyukjinKwon commented on issue #162: Add a note about Spark build requirement at PySpark testing guide in Developer Tools

HyukjinKwon commented on issue #162: Add a note about Spark build requirement at PySpark testing guide in Developer Tools URL: https://github.com/apache/spark-website/pull/162#issuecomment-448075198 adding @cloud-fan and @srowen

Re: [MLlib][Test] Smoke and Metamorphic Testing of MLlib

" for a potentially productive use by projects. Relaxing > the assertions to only react if the differences are significant would be > another possible change. This could be a command line option to allow > different strictness of testing. > > Best, > Steffen > > > Am 22

Re: [MLlib][Test] Smoke and Metamorphic Testing of MLlib

Behaviors at this level of detail, across different ML implementations, are highly unlikely to ever align exactly. Statistically small changes in logic, such as "<" versus "<=", or differences in random number generators, etc, (to say nothing of different implementation languages) will accumulate o

Re: [MLlib][Test] Smoke and Metamorphic Testing of MLlib

seems to be a "must" for a potentially productive use by projects. Relaxing the assertions to only react if the differences are significant would be another possible change. This could be a command line option to allow different strictness of testing. Best, Steffen Am 22.08.201

Re: [MLlib][Test] Smoke and Metamorphic Testing of MLlib

Hi Steffen, Thanks for sharing your results about MLlib — this sounds like a useful tool. However, I wanted to point out that some of the results may be expected for certain machine learning algorithms, so it might be good to design those tests with that in mind. For example: > - The classific

Re: [MLlib][Test] Smoke and Metamorphic Testing of MLlib

ther tests, they're not as useful, but tests covering new cases should probably be added. On Wed, Aug 22, 2018 at 6:14 AM Steffen Herbold < herb...@cs.uni-goettingen.de> wrote: > Dear developers, > > I am writing you because I applied an approach for the automated testing > of c

[MLlib][Test] Smoke and Metamorphic Testing of MLlib

Dear developers, I am writing you because I applied an approach for the automated testing of classification algorithms to Spark MLlib and would like to forward the results to you. The approach is a combination of smoke testing and metamorphic testing. The smoke tests try to find problems by

Re: Integration testing and Scheduler Backends

If we can actually get our acts together and have integration tests in Jenkins (perhaps not run on every commit but can be run weekly or pre-release smoke tests), that'd be great. Then it relies less on contributors manually testing. On Tue, Jan 9, 2018 at 8:09 AM, Timothy Chen wrote:

Re: Integration testing and Scheduler Backends

re handy, they're not nearly as good at giving us > confidence as a successful run of our integration tests against > single/multi-node k8s clusters. Currently, we have integration testing setup > at https://github.com/apache-spark-on-k8s/spark-integration and it's running > c

Re: Integration testing and Scheduler Backends

s discussion sooner than later. This is more of a post-2.3 question - >> not something we're looking to solve right away. >> >> While unit tests are handy, they're not nearly as good at giving us >> confidence as a successful run of our integration tests aga

Re: Integration testing and Scheduler Backends

ntegration tests against > single/multi-node k8s clusters. Currently, we have integration testing > setup at https://github.com/apache-spark-on-k8s/spark-integration and > it's running continuously against apache/spark:master in > pepperdata-jenkins > <http://spark-k8s-jenk

Integration testing and Scheduler Backends

- not something we're looking to solve right away. While unit tests are handy, they're not nearly as good at giving us confidence as a successful run of our integration tests against single/multi-node k8s clusters. Currently, we have integration testing setup at https://github.com/apache-spa

Re: Testing Apache Spark with JDK 9 Early Access builds

t Oracle is reaching out to see whether anyone > wants to test with JDK 9 and give them feedback. Just contact them > directly if you'd like to. > > > -- Forwarded message -- > From: dalibor topic > Date: Wed, Jul 12, 2017 at 3:16 AM > Subject: Testing

Fwd: Testing Apache Spark with JDK 9 Early Access builds

FYI, the JDK group at Oracle is reaching out to see whether anyone wants to test with JDK 9 and give them feedback. Just contact them directly if you'd like to. -- Forwarded message -- From: dalibor topic Date: Wed, Jul 12, 2017 at 3:16 AM Subject: Testing Apache Spark wit

Re: Testing --supervise flag

Widening to dev@spark On Mon, Aug 1, 2016 at 4:21 PM, Noorul Islam K M wrote: > > Hi all, > > I was trying to test --supervise flag of spark-submit. > > The documentation [1] says that, the flag helps in restarting your > application automatically if it exited with non-zero exit code. > > I am lo

Spark streaming connectors available for testing

While we are continuing to work towards a release to support Spark 2.0.0, we appreciate your help around testing the release and the current Spark Streaming connectors. To add the connectors to your scala application, the best way is to build the source of Bahir with 'mvn clean install' which

testing the kafka 0.10 connector

Cody has graciously worked on a new connector for dstream for Kafka 0.10. Can people that use Kafka test this connector out? The patch is at https://github.com/apache/spark/pull/11863 Although we have stopped merging new features into branch-2.0, this connector is very decoupled from rest of Spark

Re: Using local-cluster mode for testing Spark-related projects

y database multiple times. On Sun, Apr 17, 2016 at 9:51 AM, Jon Maurer wrote: > Take a look at spark testing base. > https://github.com/holdenk/spark-testing-base/blob/master/README.md > > On Apr 17, 2016 10:28 AM, "Evan Chan" wrote: >> >> What I want to fi

Re: Using local-cluster mode for testing Spark-related projects

Take a look at spark testing base. https://github.com/holdenk/spark-testing-base/blob/master/README.md On Apr 17, 2016 10:28 AM, "Evan Chan" wrote: > What I want to find out is how to run tests like Spark's with > local-cluster, just like that suite, but in your own projects

Re: Using local-cluster mode for testing Spark-related projects

a separate project (I'm using Scala 2.10 and ScalaTest) is >> nontrivial. Does anyone have any suggestions to get up and running? >> >> This is what I've observed so far (I'm testing against 1.5.1, but >> suspect this would apply equally to 1.6.x): >&g

Re: Using local-cluster mode for testing Spark-related projects

d > running in a separate project (I'm using Scala 2.10 and ScalaTest) is > nontrivial. Does anyone have any suggestions to get up and running? > > This is what I've observed so far (I'm testing against 1.5.1, but > suspect this would apply equally to 1.6.x): &g

Using local-cluster mode for testing Spark-related projects

in a separate project (I'm using Scala 2.10 and ScalaTest) is nontrivial. Does anyone have any suggestions to get up and running? This is what I've observed so far (I'm testing against 1.5.1, but suspect this would apply equally to 1.6.x): - One needs to have a real Spark di

Re: security testing on spark ?

Best Regards On Wed, Dec 16, 2015 at 6:46 AM, Judy Nash wrote: > Hi all, > > > > Does anyone know of any effort from the community on security testing > spark clusters. > > I.e. > > Static source code analysis to find security flaws > > Penetration testing to

security testing on spark ?

Hi all, Does anyone know of any effort from the community on security testing spark clusters. I.e. Static source code analysis to find security flaws Penetration testing to identify ways to compromise spark cluster Fuzzing to crash spark Thanks, Judy

Re: Spark doesn't unset HADOOP_CONF_DIR when testing ?

>> I try to do test on HiveSparkSubmitSuite on local box, but fails. The >> cause is that spark is still using my local single node cluster hadoop when >> doing the unit test. I don't think it make sense to do that. These >> environment variable should be unset before

Re: Spark doesn't unset HADOOP_CONF_DIR when testing ?

p when > doing the unit test. I don't think it make sense to do that. These > environment variable should be unset before the testing. And I suspect > dev/run-tests also > didn't do that either. > > Here's the error message: > > Cause: java.lang.RuntimeEx

Spark doesn't unset HADOOP_CONF_DIR when testing ?

I try to do test on HiveSparkSubmitSuite on local box, but fails. The cause is that spark is still using my local single node cluster hadoop when doing the unit test. I don't think it make sense to do that. These environment variable should be unset before the testing. And I suspect dev/run-

Re: Adding Spark Testing functionality

So here is a quick description of the current testing bits (I can expand on it if people are interested) http://bit.ly/pandaPandaPanda . On Tue, Oct 6, 2015 at 3:49 PM, Holden Karau wrote: > I'll put together a google doc and send that out (in the meantime a quick > guide of sort

Re: Adding Spark Testing functionality

I'll put together a google doc and send that out (in the meantime a quick guide of sort of how the current package can be used is in the blog post I did at http://blog.cloudera.com/blog/2015/09/making-apache-spark-testing-easy-with-spark-testing-base/ ) If people think its better to keep

Re: Adding Spark Testing functionality

merging modules into Spark that can exist outside of the project. A testing utility package that is based on Spark's public API's seems like a really useful thing for the community, but it does seem like a good fit for a package library. At least, this is my first question after taking

Adding Spark Testing functionality

Hi Spark Devs, So this has been brought up a few times before, and generally on the user list people get directed to use spark-testing-base. I'd like to start moving some of spark-testing-base's functionality into Spark so that people don't need a library to do what is (hopef

[ANNOUNCE] New testing capabilities for pull requests

] - test using older hadoop versions (can use 1.0, 2.0, 2.2, and 2.3). The relevant source code is here: https://github.com/apache/spark/blob/master/dev/run-tests-jenkins#L193 This is useful because it allows up-front testing of build changes to avoid breaks once a patch has already been merged

Re: [pyspark] What is the best way to run a minimum unit testing related to our developing module?

ark.ml API in Python, >> I'd >> like to run a minimum unit testing related to the developing module again >> and again. >> In the previous version, that was easy with commenting out unrelated >> modules >> in the ./python/run-tests script. So what is the best wa

Re: [pyspark] What is the best way to run a minimum unit testing related to our developing module?

Thanks! --Yu - -- Yu Ishikawa -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/pyspark-What-is-the-best-way-to-run-a-minimum-unit-testing-related-to-our-developing-module-tp12987p12989.html Sent from the Apache Spark Developers List mailing list

Re: [pyspark] What is the best way to run a minimum unit testing related to our developing module?

Run ./python/run-tests --help and you will see. :) On Wed, Jul 1, 2015 at 9:10 PM, Yu Ishikawa wrote: > Hi all, > > When I develop pyspark modules, such as adding a spark.ml API in Python, > I'd > like to run a minimum unit testing related to the developing module again &

[pyspark] What is the best way to run a minimum unit testing related to our developing module?

Hi all, When I develop pyspark modules, such as adding a spark.ml API in Python, I'd like to run a minimum unit testing related to the developing module again and again. In the previous version, that was easy with commenting out unrelated modules in the ./python/run-tests script. So what i

Re: Testing spark applications

I think that @holdenk's *spark-testing-base* project publishes some of these test classes as well as some helper classes for testing streaming jobs: https://github.com/holdenk/spark-testing-base On Thu, May 21, 2015 at 10:39 PM, Reynold Xin wrote: > It is just 15 lines of code to copy,

Re: Testing spark applications

andard way of writing unit tests that > require a SparkContext? > > We've ended up copying out the code of SharedSparkContext to our own > testing hierarchy, but it occurs to me someone would have published a test > jar by now if that was the best way. > > -Nathan > >

Testing spark applications

> > see discussions about Spark not really liking multiple contexts in the > same JVM > Speaking of this - is there a standard way of writing unit tests that require a SparkContext? We've ended up copying out the code of SharedSparkContext to our own testing hierarchy, bu

RE: testing HTML email

Testing too. Recently I got few undelivered mails to dev-list. From: Reynold Xin [mailto:r...@databricks.com] Sent: Thursday, May 14, 2015 3:39 PM To: dev@spark.apache.org Subject: testing HTML email Testing html emails ... Hello This is bold This is a link<http://databricks.com/>

testing HTML email

Testing html emails ... Hello *This is bold* This is a link <http://databricks.com/>

Re: Automatic testing for Spark App developed in Python

opers-list.1001551.n3.nabble.com/Automatic-testing-for-Spark-App-developed-in-Python-tp12099p12103.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. - To unsubscribe, e-mail: dev-unsubscr...@spark.

Automatic testing for Spark App developed in Python

s the pyworker is not aware of project structure.There is a more detailed explanation of my problem here <http://stackoverflow.com/questions/30079934/pyspark-automatic-testing-using-nosetests> Thanks! -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.co

Re: Spurious test failures, testing best practices

bly post-clean just to continue running one > > specific test case feels especially wasteful, and the failure output when > > naively attempting to run a specific test without having built an > assembly > > jar is not always clear about what the issue is or how to fix it; even > the

Re: Spurious test failures, testing best practices

Thanks Marcelo, "this is just how Maven works (unfortunately)" answers my question. Another related question: I tried to use `mvn scala:cc` and discovered that it only seems to work scan src/main and src/test directories (according to its docs

Re: Spurious test failures, testing best practices

On Tue, Dec 2, 2014 at 4:40 PM, Ryan Williams wrote: >> But you only need to compile the others once. > > once... every time I rebase off master, or am obliged to `mvn clean` by some > other build-correctness bug, as I said before. In my experience this works > out to a few times per week. No, yo

Re: Spurious test failures, testing best practices

st everything > else that depends on it. > Last response applies. > > So, do this ONCE: > again, s/ONCE/several times a week/, in my experience. > > mvn install -DskipTests > > Then do this as many times as you want: > > mvn -pl spark-core_2.10 something &

Re: Spurious test failures, testing best practices

especially if you work in core - you should still, at some point, compile and test everything else that depends on it. So, do this ONCE: mvn install -DskipTests Then do this as many times as you want: mvn -pl spark-core_2.10 something That doesn't seem too bad to me. (Be aware of the &q

Re: Spurious test failures, testing best practices

Marcelo: by my count, there are 19 maven modules in the codebase. I am typically only concerned with "core" (and therefore its two dependencies as well, `network/{shuffle,common}`). The `mvn package` workflow (and its sbt equivalent) that most people apparently use involves (for me) compiling+pack

Re: Spurious test failures, testing best practices

henomenon cost me a few >> > > hours of running `./dev/run-tests` in different ways before deciding >> > that, >> > > as usual, there was no way around it and that I'd need to run `mvn >> clean` >> > > and start running tests from scratch. >> &g

Re: Spurious test failures, testing best practices

On Tue, Dec 2, 2014 at 2:40 PM, Ryan Williams wrote: > Following on Mark's Maven examples, here is another related issue I'm > having: > > I'd like to compile just the `core` module after a `mvn clean`, without > building an assembly JAR first. Is this possible? Out of curiosity, may I ask why? W

Re: Spurious test failures, testing best practices

specially befuddled by them when I was seeing a disjoint > > set > > > of reproducible failures on my mac [19] (the triaging of which involved > > > dozens of runs of `./dev/run-tests`). > > > > > > While I'm interested in digging into each of these issues,

Re: Spurious test failures, testing best practices

Hi Ilya - you can just submit a pull request and the way we test them is to run it through jenkins. You don't need to do anything special. On Sun, Nov 30, 2014 at 8:57 PM, Ganelin, Ilya wrote: > Hi, Patrick - with regards to testing on Jenkins, is the process for this > to submit a p

Re: Spurious test failures, testing best practices

Hi, Patrick - with regards to testing on Jenkins, is the process for this to submit a pull request for the branch or is there another interface we can use to submit a build to Jenkins for testing? On 11/30/14, 6:49 PM, "Patrick Wendell" wrote: >Hey Ryan, > >A few more thin

Re: Spurious test failures, testing best practices

o; I was mostly responding to a perceived implication that I should have been getting building/testing-spark advice from the github .md files instead of from /latest. I agree that neither one works very well currently, and that docs-snapshots-via-jenkins is the right solution. Per my other email, leavin

Re: Spurious test failures, testing best practices

pilation-with-zinc> page and will try using it. Even so, finding those commands took a nontrivial amount of trial and error, I've not seen them very-well-documented outside of this list (your and Matei's emails (and previous emails to this list) each have more info about building/testing

Re: Spurious test failures, testing best practices

the previous runs >>>> >> used), >>>> >> > and it passed [16], so the failure seems specific to my linux >>>> >> machine/arch. >>>> >> > >>>> >> > At this point I believe that my changes don't break any te

Re: Spurious test failures, testing best practices

nd did, >>> >> > hopefully) fix once and for all. It cost me an ~hour this time >>> >> (approximate >>> >> > time of running ./dev/run-tests) and a few hours other times when I >>> >> didn't >>> >> > fully un

Re: Spurious test failures, testing best practices

ere; I don't know why I'm >> still >> >> > running into it. This cost me a few hours of running >> `./dev/run-tests` >> >> > multiple times to see if it was transient, plus some time >> researching and >> >> > working around i

Re: Spurious test failures, testing best practices

r running dev/run-tests locally, or just let > Jenkins > > do it during the CR process? > > - Needing to run a full assembly post-clean just to continue running one > > specific test case feels especially wasteful, and the failure output when > > naively attempting to run a specif

Re: Spurious test failures, testing best practices

out whether they're my fault. I've seen them many > times > >> > over the past weeks/months, plus or minus other failures that have > come > >> and > >> > gone, and was especially befuddled by them when I was seeing a > disjoint > >> set &

  1   2   >