[GitHub] spark issue #18373: [SPARK-20431][SS][FOLLOWUP] Specify a schema by using a ...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18373 thanks, merging to master! @amoussoubaruch please post your question to dev list, instead of randomly picking a PR... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18373: [SPARK-20431][SS][FOLLOWUP] Specify a schema by using a ...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/18373 @amoussoubaruch, sounds unrelated with this PR. Probably, question should go to mailing list. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18373: [SPARK-20431][SS][FOLLOWUP] Specify a schema by using a ...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/18373 cc @cloud-fan, would you have some time to look into this maybe? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18373: [SPARK-20431][SS][FOLLOWUP] Specify a schema by using a ...
Github user amoussoubaruch commented on the issue: https://github.com/apache/spark/pull/18373 Hello, I want to use spark-structured-streaming to process data fetched from kafka messages, and then store as rows cassandra database. I need one clarification. The message in kafka are serialized in Avro format. How can i deserialize the message to json for processing before storing into cassandra? Is it possible with spark 2.1.0 or 2.1.1? Any advice or help could be appreciated. Thanks in advance --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18373: [SPARK-20431][SS][FOLLOWUP] Specify a schema by using a ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18373 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18373: [SPARK-20431][SS][FOLLOWUP] Specify a schema by using a ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18373 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78367/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18373: [SPARK-20431][SS][FOLLOWUP] Specify a schema by using a ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18373 **[Test build #78367 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78367/testReport)** for PR 18373 at commit [`55f1d07`](https://github.com/apache/spark/commit/55f1d075f044c716c89e9a6641c5e1341b100672). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18373: [SPARK-20431][SS][FOLLOWUP] Specify a schema by using a ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18373 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78354/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18373: [SPARK-20431][SS][FOLLOWUP] Specify a schema by using a ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18373 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18373: [SPARK-20431][SS][FOLLOWUP] Specify a schema by using a ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18373 **[Test build #78354 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78354/testReport)** for PR 18373 at commit [`8df0cd2`](https://github.com/apache/spark/commit/8df0cd2e1ac79d423510a7d66839791cd3335a6d). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18373: [SPARK-20431][SS][FOLLOWUP] Specify a schema by using a ...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/18373 Thanks for triggering tests and fixing Jenkins @shaneknapp. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18373: [SPARK-20431][SS][FOLLOWUP] Specify a schema by using a ...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/18373 I skimmed `SparkSession`, `Dataset` and `functions.scala`. I think I can add it ,in `SparkSession` , to ``` def createDataFrame(rowRDD: RDD[Row], schema: StructType) def createDataFrame(rowRDD: JavaRDD[Row], schema: StructType) def createDataFrame(rows: java.util.List[Row], schema: StructType) ``` Please let me know. If we are not sure for now, I guess it is okay as a proper follow-up. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18373: [SPARK-20431][SS][FOLLOWUP] Specify a schema by using a ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18373 **[Test build #78367 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78367/testReport)** for PR 18373 at commit [`55f1d07`](https://github.com/apache/spark/commit/55f1d075f044c716c89e9a6641c5e1341b100672). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18373: [SPARK-20431][SS][FOLLOWUP] Specify a schema by using a ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18373 **[Test build #78354 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78354/testReport)** for PR 18373 at commit [`8df0cd2`](https://github.com/apache/spark/commit/8df0cd2e1ac79d423510a7d66839791cd3335a6d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18373: [SPARK-20431][SS][FOLLOWUP] Specify a schema by using a ...
Github user shaneknapp commented on the issue: https://github.com/apache/spark/pull/18373 test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18373: [SPARK-20431][SS][FOLLOWUP] Specify a schema by using a ...
Github user maropu commented on the issue: https://github.com/apache/spark/pull/18373 Thanks for follow-up! IMHO this syntax helps much, so we could support this in other cases for defining schemas (e.g., `spark.createDataFrame`)? (I think it's impossible to support in all the cases though, is it possible to do in frequently-used APIs...?) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
