Re: [FYI] SPARK-45981: Improve Python language test coverage

Awesome! On Sat, Dec 2, 2023 at 2:33 PM Dongjoon Hyun wrote: > Hi, All. > > As a part of Apache Spark 4.0.0 (SPARK-44111), the Apache Spark community > starts to have test coverage for all supported Python versions from Today. > > - https://github.com/apache/spark/actio

[FYI] SPARK-45981: Improve Python language test coverage

Hi, All. As a part of Apache Spark 4.0.0 (SPARK-44111), the Apache Spark community starts to have test coverage for all supported Python versions from Today. - https://github.com/apache/spark/actions/runs/7061665420 Here is a summary. 1. Main CI: All PRs and commits on `master` branch

Re: IDEA compile fail but sbt test succeed

Dear AlphaBetaGo, First of all, there are not only guys here, but also women. Second, you didn't give a context that would allow to understand the connection with Spark. From what I see, it's more likely that it's an issue in Spark/sbt support in IDEA. Feel free to create an issue in the

Re: Hive 3 has big performance improvement from my test

What bothers me is that you are making sweeping statements about Spark inability to handle quote " ... the key weakness of Spark is 1) its poor performance when executing concurrent queries and 2) its poor resource utilization when executing multiple Spark applications concurrently" and conversely

Re: Hive 3 has big performance improvement from my test

ive 3.1.3 has much better performance than > the old ones. > It's even as fast as Spark by using the default mr engine. > My test process and dataset, > https://blog.crypt.pw/Another-10-million-dataset-testing-for-Spark-and-Hive > > Thanks. >

Re: The Dataset unit test is much slower than the RDD unit test (in Scala)

Which Spark version are you using? SPARK-36444[1] and SPARK-38138[2] may be related, please test w/ the patched version or disable DPP by setting spark.sql.optimizer.dynamicPartitionPruning.enabled=false to see if it helps. [1] https://issues.apache.org/jira/browse/SPARK-36444 [2] https

Re: The Dataset unit test is much slower than the RDD unit test (in Scala)

Hi Tanin, running your test with option "spark.sql.planChangeLog.level" set to "info" or "warn" (depending on your Spark log level) will show you insights into the planning (which rules are applied, how long rules take, how many iterations are done). Hoping th

The Dataset unit test is much slower than the RDD unit test (in Scala)

Hi All, Our data job is very complex (e.g. 100+ joins), and we have switched from RDD to Dataset recently. We've found that the unit test takes much longer. We profiled it and have found that it's the planning phase that is slow, not execution. I wonder if anyone has encountered this issue

Skip single integration test case in Spark on K8s

Hi Spark team I am running Spark kubernetes integration test suite on cloud. build/mvn install \ -f pom.xml \ -pl resource-managers/kubernetes/integration-tests -am -Pscala-2.12 -Phadoop-3.1.1 -Phive -Phive-thriftserver -Pyarn -Pkubernetes -Pkubernetes-integration-tests \ -Djava.version=8

Re: ivy unit test case filing for Spark

Are you using IvyVPN which causes this problem? If the VPN software changes the network URL silently you should avoid using them. Regards. On Wed, Dec 22, 2021 at 1:48 AM Pralabh Kumar wrote: > Hi Spark Team > > I am building a spark in VPN . But the unit test case below i

Re: ivy unit test case filing for Spark

You would have to make it available? This doesn't seem like a spark issue. On Tue, Dec 21, 2021, 10:48 AM Pralabh Kumar wrote: > Hi Spark Team > > I am building a spark in VPN . But the unit test case below is failing. > This is pointing to ivy location which cannot be reached with

ivy unit test case filing for Spark

Hi Spark Team I am building a spark in VPN . But the unit test case below is failing. This is pointing to ivy location which cannot be reached within VPN . Any help would be appreciated test("SPARK-33084: Add jar support Ivy URI -- default transitive = true") { *sc *= new SparkC

Re: Need Unit test complete reference for Pyspark

to create dataframe on the fly, perhaps from a string. you can have a look at pandas, it will have methods for it - a method to test dataframe equality. you can use df1.subtract(df2) I am assuming you are into dataframes - rather than RDDs, for which the two packages you mention should have everything

Need Unit test complete reference for Pyspark

Hi Users, I have to write Unit Test cases for PySpark. I think pytest-spark and "spark testing base" are good test libraries. Can anyone please provide full reference for writing the test cases in Python using these? Kind Regards, Sachit Murarka

Re: test

Yes, your emails are getting through. On Mon, Jul 27, 2020 at 6:31 PM Suat Toksöz wrote: > user@spark.apache.org > > -- > > Best regards, > > *Suat Toksoz* > -- Kustoms On Silver

test

user@spark.apache.org -- Best regards, *Suat Toksoz*

Re: find failed test

Srsly? On Sat, 7 Mar 2020 at 03:28, Koert Kuipers wrote: > i just ran: > mvn test -fae > log.txt > > at the end of log.txt i find it says there are failures: > [INFO] Spark Project SQL .. FAILURE [47:55 > min] > > that is not very

find failed test

i just ran: mvn test -fae > log.txt at the end of log.txt i find it says there are failures: [INFO] Spark Project SQL .. FAILURE [47:55 min] that is not very helpful. what tests failed? i could go scroll up but the file has 21,517 lines. ok let's skip that. s

Test mail

test

[Spark SQL] dependencies to use test helpers

I have a Scala application in which I have added some extra rules to Catalyst. While adding some unit tests, I am trying to use some existing functions from Catalyst's test code: Specifically comparePlans() and normalizePlan() under PlanTestBase <https://github.com/apache/spark/b

spark ./build/mvn test failed on aarch64

Hi all, Recently I run './build/mvn test' of spark on aarch64, and master and branch-2.4 are all failled, the log pieces as below: .. [INFO] T E S T S [INFO] --- [INFO] Running org.apache.spark.util.kvstore.LevelDBTypeInfoSuite [INFO] Tests

spark ./build/mvn test failed on aarch64

Hi all, Recently I run './build/mvn test' of spark on aarch64, and master and branch-2.4 are all failled, the log pieces as below: .. [INFO] T E S T S [INFO] --- [INFO] Running org.apache.spark.util.kvstore.LevelDBTypeInfoSuite [INFO] Tests

Testing with spark-base-test

I'm using spark-unit-test and I don't get to compile the code. test("Testging") { val inputInsert = A("data2") val inputDelete = A("data1") val outputInsert = B(1) val outputDelete = C(1) val input = List(List(inputInsert), List(inputDelete

Re: SparkR test script issue: unable to run run-tests.h on spark 2.2

an earlier version with devtools? will > follow up for a fix. > > _ > From: Hyukjin Kwon <gurwls...@gmail.com> > Sent: Wednesday, February 14, 2018 6:49 PM > Subject: Re: SparkR test script issue: unable to run run-tests.h on spark > 2.2 > To: chand

Re: SparkR test script issue: unable to run run-tests.h on spark 2.2

Yes it is issue with the newer release of testthat. To workaround could you install an earlier version with devtools? will follow up for a fix. _ From: Hyukjin Kwon <gurwls...@gmail.com> Sent: Wednesday, February 14, 2018 6:49 PM Subject: Re: SparkR test script

Re: SparkR test script issue: unable to run run-tests.h on spark 2.2

<chandanbaran...@gmail.com> wrote: > Hi All, > I am trying to run test script of R under ./R/run-tests.sh but hitting > same ERROR everytime. > I tried running on mac as well as centos machine, same issue coming up. > I am using spark 2.2 (branch-2.2) > I followed from apache doc a

SparkR test script issue: unable to run run-tests.h on spark 2.2

Hi All, I am trying to run test script of R under ./R/run-tests.sh but hitting same ERROR everytime. I tried running on mac as well as centos machine, same issue coming up. I am using spark 2.2 (branch-2.2) I followed from apache doc and followed the steps: 1. installed R 2. installed packages

not able to read git info from Scala Test Suite

Hello - I'm writing a scala unittest for my Spark project which checks the git information, and somehow it is not working from the Unit Test Added in pom.xml -- pl.project13.maven git-commit-id-plugin 2.2.4

Re: Collecting matrix's entries raises an error only when run inside a test

are no hints to possible workarounds. > > the error is the following: > > Traceback (most recent call last): > File > "/home/simone/motionlogic/trip-labeler/test/trip_labeler_test/model_test.py", > line 43, in test_make_trip_matrix > entries = trip_matr

Collecting matrix's entries raises an error only when run inside a test

Hello, I have this problem and Google is not helping. Instead, it looks like an unreported bug and there are no hints to possible workarounds. the error is the following: Traceback (most recent call last): File "/home/simone/motionlogic/trip-labeler/test/trip_labeler_test/model_test.py&q

Re: test mail

test email recieved ;p On 4 Jul 2017 7:40 am, "Sudha KS" <sudha...@fuzzylogix.com> wrote: -- *Disclaimer: The information in this email is confidential and may be legally privileged. Access to this email by anyone other than the intended addressee is unauthorized. If you are

test mail

(Spark-ml) java.util.NosuchElementException: key not found exception on doing prediction and computing test error.

Thanks. Its working now. My test data had some labels which were not there in training set. On Wednesday, June 28, 2017, Pralabh Kumar <pralabhku...@gmail.com <javascript:_e(%7B%7D,'cvml','pralabhku...@gmail.com');>> wrote: > Hi Neha > > This generally occurred when , you t

Re: (Spark-ml) java.util.NosuchElementException: key not found exception on doing prediction and computing test error.

Hi Neha This generally occurred when , you training data set have some value of categorical variable ,which in not there in your testing data. For e.g you have column DAYS ,with value M,T,W in training data . But when your test data contains F ,then it say no key found exception . Please look

Fwd: (Spark-ml) java.util.NosuchElementException: key not found exception on doing prediction and computing test error.

Hi, I am using Apache spark 2.0.2 randomforest ml (standalone mode) for text classification. TF-IDF feature extractor is also used. The training part runs without any issues and returns 100% accuracy. But when I am trying to do prediction using trained model and compute test error, it fails

(Spark-ml) java.util.NosuchElementException: key not found exception on doing prediction and computing test error.

Hi, I am using Apache spark 2.0.2 randomforest ml (standalone mode) for text classification. TF-IDF feature extractor is also used. The training part runs without any issues and returns 100% accuracy. But when I am trying to do prediction using trained model and compute test error, it fails

Test

Test - To unsubscribe e-mail: user-unsubscr...@spark.apache.org

NPE in UDF yet no nulls in data because analyzer runs test with nulls

added filters for nulls and the NPE still happened. then we also saw the NPE when just doing dataframe.explain instead of running our job. turns out the issue is in EliminateOuterJoin.canFilterOutNull where a row with all nulls ifs fed into the expression as a test. its the line: val v = boundE.eval

Re: scala test is unable to initialize spark context.

Seems it is caused by your log4j file *Caused by: java.lang.IllegalStateException: FileNamePattern [-.log] does not contain a valid date format specifier* <psw...@in.imshealth.com>于2017年4月6日周四 下午4:03写道： > Hi All , > > > >I am just trying to use scala test for testi

scala test is unable to initialize spark context.

Hi All , I am just trying to use scala test for testing a small spark code . But spark context is not getting initialized , while I am running test file . I have given code, pom and exception I am getting in mail , please help me to understand what mistake I am doing , so that Spark context

Re: how do i force unit test to do whole stage codegen

e days. >> >> Jacek >> >> On 5 Apr 2017 6:08 a.m., "Kazuaki Ishizaki" <ishiz...@jp.ibm.com> wrote: >>> >>> Hi, >>> The page in the URL explains the old style of physical plan output. >>> The current style adds "*" as a prefi

Re: how do i force unit test to do whole stage codegen

gt;> The page in the URL explains the old style of physical plan output. >> The current style adds "*" as a prefix of each operation that the >> whole-stage codegen can be apply to. >> >> So, in your test case, whole-stage codegen has been already enabled!! >> >&g

Re: how do i force unit test to do whole stage codegen

p.ibm.com> wrote: > Hi, > The page in the URL explains the old style of physical plan output. > The current style adds "*" as a prefix of each operation that the > whole-stage codegen can be apply to. > > So, in your test case, whole-stage codegen has been already en

Re: how do i force unit test to do whole stage codegen

e-stage codegen can be apply to. > > So, in your test case, whole-stage codegen has been already enabled!! > > FYI. I think that it is a good topic for d...@spark.apache.org. > > Kazuaki Ishizaki > > > > From:Koert Kuipers <ko...@tresata.com> > To:

Re: how do i force unit test to do whole stage codegen

Hi, The page in the URL explains the old style of physical plan output. The current style adds "*" as a prefix of each operation that the whole-stage codegen can be apply to. So, in your test case, whole-stage codegen has been already enabled!! FYI. I think that it is a good t

how do i force unit test to do whole stage codegen

i wrote my own expression with eval and doGenCode, but doGenCode never gets called in tests. also as a test i ran this in a unit test: spark.range(10).select('id as 'asId).where('id === 4).explain according to https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-sql-whole-stage

This is a test mail, please ignore!

Sending plain text mail to test whether my mail appear in the list. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/This-is-a-test-mail-please-ignore-tp28538.html Sent from the Apache Spark User List mailing list archive at Nabble.com

Re: How to unit test spark streaming?

Agreed with the statement in quotes below whether one wants to do unit tests or not It is a good practice to write code that way. But I think the more painful and tedious task is to mock/emulate all the nodes such as spark workers/master/hdfs/input source stream and all that. I wish there is

Re: How to unit test spark streaming?

> > Basically you abstract your transformations to take in a dataframe and > return one, then you assert on the returned df > +1 to this suggestion. This is why we wanted streaming and batch dataframes to share the same API.

Re: How to unit test spark streaming?

ali <kanth...@gmail.com> wrote: > > Hi All, > > How to unit test spark streaming or spark in general? How do I test the > results of my transformations? Also, more importantly don't we need to spawn > master and worker JVM's either in one or multiple

Re: How to unit test spark streaming?

Hey kant You can use holdens spark test base Have a look at some of the specs I wrote here to give you an idea https://github.com/samelamin/spark-bigquery/blob/master/src/test/scala/com/samelamin/spark/bigquery/BigQuerySchemaSpecs.scala Basically you abstract your transformations to take

How to unit test spark streaming?

Hi All, How to unit test spark streaming or spark in general? How do I test the results of my transformations? Also, more importantly don't we need to spawn master and worker JVM's either in one or multiple nodes? Thanks! kant

Spark test error in ProactiveClosureSerializationSuite.scala

hello all, I am building Spark1.6.2 and I meet a problem when doing mvn test The command is mvn -e -Pyarn -Phive -Phive-thriftserver -DwildcardSuites=org.apache.spark.serializer.ProactiveClosureSerializationSuite test and the test error is ProactiveClosureSerializationSuite: - throws

Spark test error

I tried to run the tests in 'GeneralizedLinearRegressionSuite', and all tests passed except for test("read/write") which yielded the following error message. Any suggestion on why this happened and how to fix it? Thanks. BTW, I ran the test in IntelliJ. The default jsonEncode onl

How to clean the cache when i do performance test in spark

Hi all: When I test my spark application, I found that the second round(application_1481153226569_0002) is more faster than first round(application_1481153226569_0001). Actually the configuration is same. I guess the second round is improved a lot by cache. So how can I clean the cache

Re: Want to test spark-sql-kafka but get unresolved dependency error

kafka-integration.md >> > >> > But I get a unresolved dependency error ("unresolved dependency: >> > org.apache.spark#spark-sql-kafka-0-10_2.11;2.0.0: not found"). So it >> > seems >> > not to be available via maven or spark-packages. >> > >

Re: Want to test spark-sql-kafka but get unresolved dependency error

e via maven or spark-packages. > > > > How can I accesss this package? Or am I doing something wrong/missing? > > > > Thank you for you help. > > > > > > > > -- > > View this message in context: http://apache-spark-user-list.

Re: Want to test spark-sql-kafka but get unresolved dependency error

ound"). So it seems > not to be available via maven or spark-packages. > > How can I accesss this package? Or am I doing something wrong/missing? > > Thank you for you help. > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabb

Re: Want to test spark-sql-kafka but get unresolved dependency error

r am I doing something wrong/missing? > > Thank you for you help. > > > > -- > View this message in context: http://apache-spark-user-list. > 1001560.n3.nabble.com/Want-to-test-spark-sql-kafka-but-get- > unresolved-dependency-e

Re: Want to test spark-sql-kafka but get unresolved dependency error

ccesss this package? Or am I doing something wrong/missing? > > Thank you for you help. > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Want-to-test-spark-sql-kafka-but-get-unresolved-dependency-error-tp27

Want to test spark-sql-kafka but get unresolved dependency error

ou for you help. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Want-to-test-spark-sql-kafka-but-get-unresolved-dependency-error-tp27891.html Sent from the Apache Spark User List mailing list archive at

Error in run multiple unit test that extends DataFrameSuiteBase

After I created two test case that FlatSpec with DataFrameSuiteBase. But I got errors when do sbt test. I was able to run each of them separately. My test cases does use sqlContext to read files. Here is the exception stack. Judging from the exception, I may need to unregister RpcEndpoint after

Re: build error - failing test- Error while building spark 2.0 trunk from github

Hi, Can you share what's the command to run the build? What's the OS? Java? Pozdrawiam, Jacek Laskowski https://medium.com/@jaceklaskowski/ Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark Follow me at https://twitter.com/jaceklaskowski On Sun, Jul 31, 2016 at 6:54 PM,

build error - failing test- Error while building spark 2.0 trunk from github

--- T E S T S --- Running org.apache.spark.api.java.OptionalSuite Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.052 sec - in org.apache.spark.api.java.OptionalSuite Running

Re: test - what is the wrong while adding one column in the dataframe

just for test, since it seemed that the user email system was something wrong ago, is okay now. On Friday, June 17, 2016 12:18 PM, Zhiliang Zhu <zchl.j...@yahoo.com.INVALID> wrote: On Tuesday, May 17, 2016 10:44 AM, Zhiliang Zhu <zchl.j...@yahoo.com.INVALID> wrote:

test - what is the wrong while adding one column in the dataframe

On Tuesday, May 17, 2016 10:44 AM, Zhiliang Zhu wrote: Hi All, For the given DataFrame created by hive sql, however, then it is required to add one more column based on the existing column, and should also keep the previous columns there for the result

Re: ANOVA test in Spark

If any specific algorithm is not present, perhaps you can use R/Python scikit, pipe your data to it & get the model back, I'm currently trying this, and it works fine. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/ANOVA-test-in-Spark-tp26949p27043.

How to map values read from test file to 2 different RDDs

Hi I am reading a text file with 16 fields. All the place holders for the values of this text file has been defined in say 2 different case classes: Case1 and Case2 How do i map values read from text file , so my function in scala should be able to return 2 different RDDs , with each each RDD of

Re: ANOVA test in Spark

s feature be > available in Spark ? > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/ANOVA-test-in-Spark-tp26949.html > Sent from the Apache Spark User List m

ANOVA test in Spark

Is ANOVA present in Spark Mllib if not then, when will be this feature be available in Spark ? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/ANOVA-test-in-Spark-tp26949.html Sent from the Apache Spark User List mailing list archive at Nabble.com

Hi test

Hello test

Re: [2 BUG REPORT] failed to run make-distribution.sh when a older version maven installed in system and run VersionsSuite test hang

e script stops here if the last > grep finds nothing\ > # because we use "set -o pipefail" > echo -n) > > line 170: > BUILD_COMMAND=("$MVN" *--force* clean package -DskipTests $@) > > that will force spark to use build/mvn and solve this problem. >

[2 BUG REPORT] failed to run make-distribution.sh when a older version maven installed in system and run VersionsSuite test hang

tus to 0, otherwise the script stops here if the last grep finds nothing\ # because we use "set -o pipefail" echo -n) line 170: BUILD_COMMAND=("$MVN" *--force* clean package -DskipTests $@) that will force spark to use build/mvn and solve this problem. BUG 2: When I run

test

signature.asc Description: Message signed with OpenPGP using GPGMail

RE: How this unit test passed on master trunk?

Subject: Re: How this unit test passed on master trunk? From: zzh...@hortonworks.com To: java8...@hotmail.com; gatorsm...@gmail.com CC: user@spark.apache.org Date: Sun, 24 Apr 2016 04:37:11 + There are multiple records for the DF scala> structDF.groupBy($"a").agg(min(st

Re: How this unit test passed on master trunk?

struct(1, 2). Please check how the Ordering is implemented in InterpretedOrdering. The output itself does not have any ordering. I am not sure why the unit test and the real env have different environment. Xiao, I do see the difference between unit test and local cluster run. Do you know the reaso

Re: How this unit test passed on master trunk?

"))).first() first: org.apache.spark.sql.Row = [1,[1,1]] BTW https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-hadoop-2.7/715/consoleFull shows this test passing. On Fri, Apr 22, 2016 at 11:23 AM, Yong Zhang <java8...@hotmail.com> wrote: > Hi, > > I was trying to find out why

How this unit test passed on master trunk?

Hi, I was trying to find out why this unit test can pass in Spark code. inhttps://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala for this unit test: test("Star Expansion - CreateStruct and CreateArray") { val structDf = testDa

Re: Unit test with sqlContext

If you prefer the py.test framework, I just wrote a blog post with some examples: Unit testing Apache Spark with py.test https://engblog.nextdoor.com/unit-testing-apache-spark-with-py-test-3b8970dc013b On Fri, Feb 5, 2016 at 11:43 AM, Steve Annessa <steve.anne...@gmail.com> wrote: &g

Re: Unit test with sqlContext

Thanks for all of the responses. I do have an afterAll that stops the sc. While looking over Holden's readme I noticed she mentioned "Make sure to disable parallel execution." That was what I was missing; I added the follow to my build.sbt: ``` parallelExecution in Test := false ``

Unit test with sqlContext

I'm trying to unit test a function that reads in a JSON file, manipulates the DF and then returns a Scala Map. The function has signature: def ingest(dataLocation: String, sc: SparkContext, sqlContext: SQLContext) I've created a bootstrap spec for spark jobs that instantiates the Spark Context

Re: Unit test with sqlContext

/master/src/test/1.3/scala/com/holdenkarau/spark/testing/SampleDataFrameTest.scala Thanks, Silvio From: Steve Annessa <steve.anne...@gmail.com<mailto:steve.anne...@gmail.com>> Date: Thursday, February 4, 2016 at 8:36 PM To: "user@spark.apache.org<mailto:user@spark.apache.org>&q

Re: Unit test with sqlContext

-testing-base package by Holden? It’s really >> useful for unit testing Spark apps as it handles all the bootstrapping for >> you. >> >> https://github.com/holdenk/spark-testing-base >> >> DataFrame examples are here: >> https://github.com/holdenk/spark-tes

Re: Unit test with sqlContext

Have you looked at the spark-testing-base package by Holden? It’s really > useful for unit testing Spark apps as it handles all the bootstrapping for > you. > > https://github.com/holdenk/spark-testing-base > > DataFrame examples are here: > https://github.com/holdenk/spark-testing-bas

Re: Is there a test like MiniCluster example in Spark just like hadoop ?

Please refer to the following suites: yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnClusterSuite.scala core/src/test/scala/org/apache/spark/scheduler/SparkListenerWithClusterSuite.scala Cheers On Mon, Jan 18, 2016 at 2:14 AM, zml张明磊 <mingleizh...@ctrip.com> wrote: &

Is there a test like MiniCluster example in Spark just like hadoop ?

Hello, I want to find some test file in spark which support the same function just like in Hadoop MiniCluster test environment. But I can not find them. Anyone know about that ?

livy test problem: Failed to execute goal org.scalatest:scalatest-maven-plugin:1.0:test (test) on project livy-spark_2.10: There are test failures

Livy build test from master fails with below problem. Can't track it down. YARN shows Livy Spark yarn application as running. Although attempt to connect to application master shows connection refused: HTTP ERROR 500 > Problem accessing /proxy/application_1448640910222_0046/. Rea

Re: How to test https://issues.apache.org/jira/browse/SPARK-10648 fix

> > On Thu, Dec 3, 2015 at 12:39 AM, Madabhattula Rajesh Kumar < > mrajaf...@gmail.com> wrote: > >> Hi Team, >> >> Looks like this issue is fixed in 1.6 release. How to test this fix? Is >> any jar is available? So I can add that jar in dependency and test t

How to test https://issues.apache.org/jira/browse/SPARK-10648 fix

Hi Team, Looks like this issue is fixed in 1.6 release. How to test this fix? Is any jar is available? So I can add that jar in dependency and test this fix. (Or) Any other way, I can test this fix in 1.15.2 code base. Could you please let me know the steps. Thank you for your support Regards

Re: How to test https://issues.apache.org/jira/browse/SPARK-10648 fix

Hi Rajesh, you can check codebase and build yourself in order to test: git clone https://git-wip-us.apache.org/repos/asf/spark cd spark mvn clean package -DskipTests You will have bin, sbin and conf folders to try it. Regards JB On 12/03/2015 09:39 AM, Madabhattula Rajesh Kumar wrote: Hi

Re: How to test https://issues.apache.org/jira/browse/SPARK-10648 fix

See this thread for Spark 1.6.0 RC1 http://search-hadoop.com/m/q3RTtKdUViYHH1b1=+VOTE+Release+Apache+Spark+1+6+0+RC1+ Cheers On Thu, Dec 3, 2015 at 12:39 AM, Madabhattula Rajesh Kumar < mrajaf...@gmail.com> wrote: > Hi Team, > > Looks like this issue is fixed in 1.6 relea

Re: how to run unit test for specific component only

try: mvn test -pl sql -DwildcardSuites=org.apache.spark.sql -Dtest=none On 12 Nov 2015, at 03:13, weoccc <weo...@gmail.com<mailto:weo...@gmail.com>> wrote: Hi, I am wondering how to run unit test for specific spark component only. mvn test -DwildcardSuites="org.apache.sp

Re: how to run unit test for specific component only

Have you tried the following ? build/sbt "sql/test-only *" Cheers On Wed, Nov 11, 2015 at 7:13 PM, weoccc <weo...@gmail.com> wrote: > Hi, > > I am wondering how to run unit test for specific spark component only. > > mvn test -DwildcardSuites="org.apache.sp

how to run unit test for specific component only

Hi, I am wondering how to run unit test for specific spark component only. mvn test -DwildcardSuites="org.apache.spark.sql.*" -Dtest=none The above command doesn't seem to work. I'm using spark 1.5. Thanks, Weide

Re: Meets "java.lang.IllegalArgumentException" when test spark ml pipe with DecisionTreeClassifier

$iwC$$iwC$$iwC.(:60) > >> > at > $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.(:62) > >> > at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.(:64) > >> > at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.(:66) > >> > at $iwC$$iwC$$iwC$$iwC$$iwC.(:68) &

Re: Meets "java.lang.IllegalArgumentException" when test spark ml pipe with DecisionTreeClassifier

) >> > at $iwC$$iwC$$iwC$$iwC$$iwC.(:68) >> > at $iwC$$iwC$$iwC$$iwC.(:70) >> > at $iwC$$iwC$$iwC.(:72) >> > at $iwC$$iwC.(:74) >> > at $iwC.(:76) >> > at (:78) >> > at .

Re: Meets "java.lang.IllegalArgumentException" when test spark ml pipe with DecisionTreeClassifier

the line you've not specified your label > column -- it's defaulting to "label" and it does not recognize it, or > at least not as a binary or nominal attribute. > > On Sun, Sep 6, 2015 at 5:47 AM, Terry Hole <hujie.ea...@gmail.com> wrote: > > Hi, Experts,

Re: Meets "java.lang.IllegalArgumentException" when test spark ml pipe with DecisionTreeClassifier

> I followed the guide of spark ml pipe to test DecisionTreeClassifier on > spark shell with spark 1.4.1, but always meets error like following, do you > have any idea how to fix this? > > The error stack: > java.lang.IllegalArgumentException: DecisionTreeClassifier was given

Re: Meets "java.lang.IllegalArgumentException" when test spark ml pipe with DecisionTreeClassifier

ne the line you've not specified your label >> column -- it's defaulting to "label" and it does not recognize it, or >> at least not as a binary or nominal attribute. >> >> On Sun, Sep 6, 2015 at 5:47 AM, Terry Hole <hujie.ea...@gmail.com> wrote: >> > Hi

  1   2   >