Re: Unit Test for Spark Streaming
Hi TD,
I tried some different setup on maven these days, and now I can at least get
something when running mvn test. However, it seems like scalatest cannot
find the test cases specified in the test suite.
Here is the output I get:
http://apache-spark-user-list.1001560.n3.nabble.com/file/n11825/Screen_Shot_2014-08-08_at_5.png
Could you please give me some details on how you setup the ScalaTest on your
machine? I believe there must be some other setup issue on my machine but I
cannot figure out why...
And here are the plugins and dependencies related to scalatest in my pom.xml
:
plugin
groupIdorg.apache.maven.plugins/groupId
artifactIdmaven-surefire-plugin/artifactId
version2.7/version
configuration
skipTeststrue/skipTests
/configuration
/plugin
plugin
groupIdorg.scalatest/groupId
artifactIdscalatest-maven-plugin/artifactId
version1.0/version
configuration
reportsDirectory${project.build.directory}/surefire-reports/reportsDirectory
junitxml./junitxml
filereports${project.build.directory}/SparkTestSuite.txt/filereports
tagsToIncludeATag/tagsToInclude
systemProperties
java.awt.headlesstrue/java.awt.headless
spark.test.home${session.executionRootDirectory}/spark.test.home
spark.testing1/spark.testing
/systemProperties
/configuration
executions
execution
idtest/id
goals
goaltest/goal
/goals
/execution
/executions
/plugin
dependency
groupIdjunit/groupId
artifactIdjunit/artifactId
version4.8.1/version
scopetest/scope
/dependency
dependency
groupIdorg.scalatest/groupId
artifactIdscalatest_2.10/artifactId
version2.2.1/version
scopetest/scope
/dependency
Thank you very much!
Best Regards,
Jiajia
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Unit-Test-for-Spark-Streaming-tp11394p11825.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
Re: Unit Test for Spark Streaming
Thank you TD, I have worked around that problem and now the test compiles. However, I don't actually see that test running. As when I do mvn test, it just says BUILD SUCCESS, without any TEST section on stdout. Are we suppose to use mvn test to run the test? Are there any other methods can be used to run this test? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Unit-Test-for-Spark-Streaming-tp11394p11570.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Unit Test for Spark Streaming
Does it not show the name of the testsuite on stdout, showing that it has passed? Can you try writing a small test unit-test, in the same way as your kafka unit test, and with print statements on stdout ... to see whether it works? I believe it is some configuration issue in maven, which is hard for me to guess. TD On Wed, Aug 6, 2014 at 12:53 PM, JiajiaJing jj.jing0...@gmail.com wrote: Thank you TD, I have worked around that problem and now the test compiles. However, I don't actually see that test running. As when I do mvn test, it just says BUILD SUCCESS, without any TEST section on stdout. Are we suppose to use mvn test to run the test? Are there any other methods can be used to run this test? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Unit-Test-for-Spark-Streaming-tp11394p11570.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Unit Test for Spark Streaming
That function is simply deletes a directory recursively. you can use
alternative libraries. see this discussion
http://stackoverflow.com/questions/779519/delete-files-recursively-in-java
On Tue, Aug 5, 2014 at 5:02 PM, JiajiaJing jj.jing0...@gmail.com wrote:
Hi TD,
I encountered a problem when trying to run the KafkaStreamSuite.scala unit
test.
I added scalatest-maven-plugin to my pom.xml, then ran mvn test, and got
the follow error message:
error: object Utils in package util cannot be accessed in package
org.apache.spark.util
[INFO] brokerConf.logDirs.foreach { f =
Utils.deleteRecursively(new File(f)) }
[INFO]^
I checked that Utils.scala does exists under
spark/core/src/main/scala/org/apache/spark/util/, so I have no idea about
why this access error.
Could you please help me with this?
Thank you very much!
Best Regards,
Jiajia
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Unit-Test-for-Spark-Streaming-tp11394p11505.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
Re: Unit Test for Spark Streaming
Appropriately timed question! Here is the PR that adds a real unit test for Kafka stream in Spark Streaming. Maybe this will help! https://github.com/apache/spark/pull/1751/files On Mon, Aug 4, 2014 at 6:30 PM, JiajiaJing jj.jing0...@gmail.com wrote: Hello Spark Users, I have a spark streaming program that stream data from kafka topics and output as parquet file on HDFS. Now I want to write a unit test for this program to make sure the output data is correct (i.e not missing any data from kafka). However, I have no idea about how to do this, especially how to mock a kafka topic. Can someone help me with this? Thank you very much! Best Regards, Jiajia -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Unit-Test-for-Spark-Streaming-tp11394.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Unit Test for Spark Streaming
This helps a lot!! Thank you very much! Jiajia -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Unit-Test-for-Spark-Streaming-tp11394p11396.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
