[jira] [Commented] (SPARK-1022) Add unit tests for kafka streaming
[ https://issues.apache.org/jira/browse/SPARK-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14067730#comment-14067730 ] Apache Spark commented on SPARK-1022: - User 'tdas' has created a pull request for this issue: [https://github.com/apache/spark/pull/557|https://github.com/apache/spark/pull/557] > Add unit tests for kafka streaming > -- > > Key: SPARK-1022 > URL: https://issues.apache.org/jira/browse/SPARK-1022 > Project: Spark > Issue Type: Bug >Reporter: Patrick Wendell >Assignee: Saisai Shao > > It would be nice if we could add unit tests to verify elements of kafka's > stream. Right now we do integration tests only which makes it hard to upgrade > versions of kafka. The place to start here would be to look at how kafka > tests itself and see if the functionality can be exposed to third party users. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (SPARK-1022) Add unit tests for kafka streaming
[ https://issues.apache.org/jira/browse/SPARK-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14083928#comment-14083928 ] Apache Spark commented on SPARK-1022: - User 'jerryshao' has created a pull request for this issue: https://github.com/apache/spark/pull/1751 > Add unit tests for kafka streaming > -- > > Key: SPARK-1022 > URL: https://issues.apache.org/jira/browse/SPARK-1022 > Project: Spark > Issue Type: Bug >Reporter: Patrick Wendell >Assignee: Saisai Shao > > It would be nice if we could add unit tests to verify elements of kafka's > stream. Right now we do integration tests only which makes it hard to upgrade > versions of kafka. The place to start here would be to look at how kafka > tests itself and see if the functionality can be exposed to third party users. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-1022) Add unit tests for kafka streaming
[ https://issues.apache.org/jira/browse/SPARK-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14087092#comment-14087092 ] Apache Spark commented on SPARK-1022: - User 'tdas' has created a pull request for this issue: https://github.com/apache/spark/pull/1797 > Add unit tests for kafka streaming > -- > > Key: SPARK-1022 > URL: https://issues.apache.org/jira/browse/SPARK-1022 > Project: Spark > Issue Type: Bug >Reporter: Patrick Wendell >Assignee: Saisai Shao > > It would be nice if we could add unit tests to verify elements of kafka's > stream. Right now we do integration tests only which makes it hard to upgrade > versions of kafka. The place to start here would be to look at how kafka > tests itself and see if the functionality can be exposed to third party users. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-1022) Add unit tests for kafka streaming
[ https://issues.apache.org/jira/browse/SPARK-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14087333#comment-14087333 ] Patrick Wendell commented on SPARK-1022: There was a follow up to this issue: https://github.com/apache/spark/pull/1797 > Add unit tests for kafka streaming > -- > > Key: SPARK-1022 > URL: https://issues.apache.org/jira/browse/SPARK-1022 > Project: Spark > Issue Type: Bug >Reporter: Patrick Wendell >Assignee: Saisai Shao > > It would be nice if we could add unit tests to verify elements of kafka's > stream. Right now we do integration tests only which makes it hard to upgrade > versions of kafka. The place to start here would be to look at how kafka > tests itself and see if the functionality can be exposed to third party users. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-1022) Add unit tests for kafka streaming
[ https://issues.apache.org/jira/browse/SPARK-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14087415#comment-14087415 ] Apache Spark commented on SPARK-1022: - User 'srowen' has created a pull request for this issue: https://github.com/apache/spark/pull/1804 > Add unit tests for kafka streaming > -- > > Key: SPARK-1022 > URL: https://issues.apache.org/jira/browse/SPARK-1022 > Project: Spark > Issue Type: Bug >Reporter: Patrick Wendell >Assignee: Saisai Shao > > It would be nice if we could add unit tests to verify elements of kafka's > stream. Right now we do integration tests only which makes it hard to upgrade > versions of kafka. The place to start here would be to look at how kafka > tests itself and see if the functionality can be exposed to third party users. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-1022) Add unit tests for kafka streaming
[ https://issues.apache.org/jira/browse/SPARK-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659014#comment-14659014 ] Anuj Ojha commented on SPARK-1022: -- [~tdas] and [~jerryshao2015] I was working on writing Integration test for my application, which reads data from kafka, performs transformation on the data and then writes the Dstream back to other kafka topic. To do this I had to get KafkaTestUtils and some other dependent classes locally as I needed embedded zookeeper and kafa queue. I also had to modify sendMessage to accept my specific type of ket and value. I was wondering if it is possible to make the KafkaTestUtils generic? so that others can use it for in-memory unit/integration testing. I have created generic sendMessage method hopefully it's of some use. {code} class KafkaTestUtils [T,U]{ ... // Kafka producer private var producer: Producer[T, U] = _ .. def sendMessages(topic: String, messageToFreq: JMap[T, U]): Unit = { import scala.collection.JavaConversions._ producer = new Producer[T, U](new ProducerConfig(producerConfiguration)) for ((k,v) <- messageToFreq) { var message = new KeyedMessage(topic, k,v) producer.send(message) } producer.close() producer = null } . } {code} I also had to change producer properties to provide key and value serialization class. It might be worth making possible to set these values from user test. I did not create JIRA for the request as I wanted to see if this is even a acceptable request. Please let me know. > Add unit tests for kafka streaming > -- > > Key: SPARK-1022 > URL: https://issues.apache.org/jira/browse/SPARK-1022 > Project: Spark > Issue Type: Bug >Reporter: Patrick Wendell >Assignee: Saisai Shao > > It would be nice if we could add unit tests to verify elements of kafka's > stream. Right now we do integration tests only which makes it hard to upgrade > versions of kafka. The place to start here would be to look at how kafka > tests itself and see if the functionality can be exposed to third party users. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-1022) Add unit tests for kafka streaming
[ https://issues.apache.org/jira/browse/SPARK-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659246#comment-14659246 ] Saisai Shao commented on SPARK-1022: Hi [~anujojha], {{KafkaTestUtils}} is used for Kafka and Spark integration test, it is not public to users to use it, so the API design is not so generic for everyone use. I think Spark itself will not provide such functionality for user to do integration test, if you want to do such thing, from my point it would be better for you to do it yourself, not replying on Spark. > Add unit tests for kafka streaming > -- > > Key: SPARK-1022 > URL: https://issues.apache.org/jira/browse/SPARK-1022 > Project: Spark > Issue Type: Bug >Reporter: Patrick Wendell >Assignee: Saisai Shao > > It would be nice if we could add unit tests to verify elements of kafka's > stream. Right now we do integration tests only which makes it hard to upgrade > versions of kafka. The place to start here would be to look at how kafka > tests itself and see if the functionality can be exposed to third party users. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org