[jira] [Commented] (SPARK-1022) Add unit tests for kafka streaming
[ https://issues.apache.org/jira/browse/SPARK-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14067730#comment-14067730 ] Apache Spark commented on SPARK-1022: - User 'tdas' has created a pull request for this issue: [https://github.com/apache/spark/pull/557|https://github.com/apache/spark/pull/557] > Add unit tests for kafka streaming > -- > > Key: SPARK-1022 > URL: https://issues.apache.org/jira/browse/SPARK-1022 > Project: Spark > Issue Type: Bug >Reporter: Patrick Wendell >Assignee: Saisai Shao > > It would be nice if we could add unit tests to verify elements of kafka's > stream. Right now we do integration tests only which makes it hard to upgrade > versions of kafka. The place to start here would be to look at how kafka > tests itself and see if the functionality can be exposed to third party users. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (SPARK-1022) Add unit tests for kafka streaming
[ https://issues.apache.org/jira/browse/SPARK-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14083928#comment-14083928 ] Apache Spark commented on SPARK-1022: - User 'jerryshao' has created a pull request for this issue: https://github.com/apache/spark/pull/1751 > Add unit tests for kafka streaming > -- > > Key: SPARK-1022 > URL: https://issues.apache.org/jira/browse/SPARK-1022 > Project: Spark > Issue Type: Bug >Reporter: Patrick Wendell >Assignee: Saisai Shao > > It would be nice if we could add unit tests to verify elements of kafka's > stream. Right now we do integration tests only which makes it hard to upgrade > versions of kafka. The place to start here would be to look at how kafka > tests itself and see if the functionality can be exposed to third party users. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-1022) Add unit tests for kafka streaming
[ https://issues.apache.org/jira/browse/SPARK-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14087092#comment-14087092 ] Apache Spark commented on SPARK-1022: - User 'tdas' has created a pull request for this issue: https://github.com/apache/spark/pull/1797 > Add unit tests for kafka streaming > -- > > Key: SPARK-1022 > URL: https://issues.apache.org/jira/browse/SPARK-1022 > Project: Spark > Issue Type: Bug >Reporter: Patrick Wendell >Assignee: Saisai Shao > > It would be nice if we could add unit tests to verify elements of kafka's > stream. Right now we do integration tests only which makes it hard to upgrade > versions of kafka. The place to start here would be to look at how kafka > tests itself and see if the functionality can be exposed to third party users. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-1022) Add unit tests for kafka streaming
[ https://issues.apache.org/jira/browse/SPARK-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14087333#comment-14087333 ] Patrick Wendell commented on SPARK-1022: There was a follow up to this issue: https://github.com/apache/spark/pull/1797 > Add unit tests for kafka streaming > -- > > Key: SPARK-1022 > URL: https://issues.apache.org/jira/browse/SPARK-1022 > Project: Spark > Issue Type: Bug >Reporter: Patrick Wendell >Assignee: Saisai Shao > > It would be nice if we could add unit tests to verify elements of kafka's > stream. Right now we do integration tests only which makes it hard to upgrade > versions of kafka. The place to start here would be to look at how kafka > tests itself and see if the functionality can be exposed to third party users. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-1022) Add unit tests for kafka streaming
[ https://issues.apache.org/jira/browse/SPARK-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14087415#comment-14087415 ] Apache Spark commented on SPARK-1022: - User 'srowen' has created a pull request for this issue: https://github.com/apache/spark/pull/1804 > Add unit tests for kafka streaming > -- > > Key: SPARK-1022 > URL: https://issues.apache.org/jira/browse/SPARK-1022 > Project: Spark > Issue Type: Bug >Reporter: Patrick Wendell >Assignee: Saisai Shao > > It would be nice if we could add unit tests to verify elements of kafka's > stream. Right now we do integration tests only which makes it hard to upgrade > versions of kafka. The place to start here would be to look at how kafka > tests itself and see if the functionality can be exposed to third party users. -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-1022) Add unit tests for kafka streaming
[
https://issues.apache.org/jira/browse/SPARK-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659014#comment-14659014
]
Anuj Ojha commented on SPARK-1022:
--
[~tdas] and [~jerryshao2015] I was working on writing Integration test for my
application, which reads data from kafka, performs transformation on the data
and then writes the Dstream back to other kafka topic. To do this I had to get
KafkaTestUtils and some other dependent classes locally as I needed embedded
zookeeper and kafa queue. I also had to modify sendMessage to accept my
specific type of ket and value. I was wondering if it is possible to make the
KafkaTestUtils generic? so that others can use it for in-memory
unit/integration testing. I have created generic sendMessage method hopefully
it's of some use.
{code}
class KafkaTestUtils [T,U]{
...
// Kafka producer
private var producer: Producer[T, U] = _
..
def sendMessages(topic: String, messageToFreq: JMap[T, U]): Unit = {
import scala.collection.JavaConversions._
producer = new Producer[T, U](new ProducerConfig(producerConfiguration))
for ((k,v) <- messageToFreq) {
var message = new KeyedMessage(topic, k,v)
producer.send(message)
}
producer.close()
producer = null
}
.
}
{code}
I also had to change producer properties to provide key and value serialization
class. It might be worth making possible to set these values from user test. I
did not create JIRA for the request as I wanted to see if this is even a
acceptable request. Please let me know.
> Add unit tests for kafka streaming
> --
>
> Key: SPARK-1022
> URL: https://issues.apache.org/jira/browse/SPARK-1022
> Project: Spark
> Issue Type: Bug
>Reporter: Patrick Wendell
>Assignee: Saisai Shao
>
> It would be nice if we could add unit tests to verify elements of kafka's
> stream. Right now we do integration tests only which makes it hard to upgrade
> versions of kafka. The place to start here would be to look at how kafka
> tests itself and see if the functionality can be exposed to third party users.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-1022) Add unit tests for kafka streaming
[
https://issues.apache.org/jira/browse/SPARK-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659246#comment-14659246
]
Saisai Shao commented on SPARK-1022:
Hi [~anujojha], {{KafkaTestUtils}} is used for Kafka and Spark integration
test, it is not public to users to use it, so the API design is not so generic
for everyone use. I think Spark itself will not provide such functionality for
user to do integration test, if you want to do such thing, from my point it
would be better for you to do it yourself, not replying on Spark.
> Add unit tests for kafka streaming
> --
>
> Key: SPARK-1022
> URL: https://issues.apache.org/jira/browse/SPARK-1022
> Project: Spark
> Issue Type: Bug
>Reporter: Patrick Wendell
>Assignee: Saisai Shao
>
> It would be nice if we could add unit tests to verify elements of kafka's
> stream. Right now we do integration tests only which makes it hard to upgrade
> versions of kafka. The place to start here would be to look at how kafka
> tests itself and see if the functionality can be exposed to third party users.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
