[
https://issues.apache.org/jira/browse/BAHIR-138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16295295#comment-16295295
]
ASF GitHub Bot commented on BAHIR-138:
--------------------------------------
Github user lresende commented on a diff in the pull request:
https://github.com/apache/bahir/pull/59#discussion_r157549138
--- Diff:
sql-cloudant/examples/src/main/scala/org/apache/spark/examples/sql/cloudant/CloudantStreaming.scala
---
@@ -27,59 +27,57 @@ import org.apache.bahir.cloudant.CloudantReceiver
object CloudantStreaming {
def main(args: Array[String]) {
- val sparkConf = new SparkConf().setAppName("Cloudant Spark SQL
External Datasource in Scala")
+ val sparkConf = new SparkConf().setMaster("local[*]")
+ .setAppName("Cloudant Spark SQL External Datasource in Scala")
// Create the context with a 10 seconds batch size
val ssc = new StreamingContext(sparkConf, Seconds(10))
val changes = ssc.receiverStream(new CloudantReceiver(sparkConf, Map(
- "cloudant.host" -> "ACCOUNT.cloudant.com",
- "cloudant.username" -> "USERNAME",
- "cloudant.password" -> "PASSWORD",
- "database" -> "n_airportcodemapping")))
-
+ "cloudant.host" -> "examples.cloudant.com",
+ "database" -> "sales")))
changes.foreachRDD((rdd: RDD[String], time: Time) => {
// Get the singleton instance of SparkSession
val spark =
SparkSessionSingleton.getInstance(rdd.sparkContext.getConf)
--- End diff --
I would just create the "spark" instance of SparkSession on the top of the
method (where we create SparkConf) and do the import of implicits there instead
of inside of the forEach. This would also remove the necessity of the singleton
object on the bottom.
> Fix sql-cloudant deprecation messages
> -------------------------------------
>
> Key: BAHIR-138
> URL: https://issues.apache.org/jira/browse/BAHIR-138
> Project: Bahir
> Issue Type: Task
> Components: Spark Structured Streaming Connectors
> Affects Versions: Spark-2.2.0
> Reporter: Esteban Laver
> Assignee: Esteban Laver
> Priority: Minor
> Labels: warnings
>
> Deprecation warnings in {{DefaultSource}}:
> {code}
> [INFO] --- scala-maven-plugin:3.2.2:compile (scala-compile-first) @
> spark-sql-cloudant_2.11 ---
> [INFO] Compiling 11 Scala sources to sql-cloudant/target/scala-2.11/classes...
> [WARNING]
> sql-cloudant/src/main/scala/org/apache/bahir/cloudant/DefaultSource.scala:59:
> method json in class DataFrameReader is deprecated: Use json(Dataset[String])
> instead.
> [WARNING] val df = sqlContext.read.json(cloudantRDD)
> [WARNING] ^
> [WARNING]
> sql-cloudant/src/main/scala/org/apache/bahir/cloudant/DefaultSource.scala:115:
> method json in class DataFrameReader is deprecated: Use
> json(Dataset[String]) instead.
> [WARNING] dataFrame = sqlContext.read.json(cloudantRDD)
> [WARNING] ^
> [WARNING]
> sql-cloudant/src/main/scala/org/apache/bahir/cloudant/DefaultSource.scala:121:
> method json in class DataFrameReader is deprecated: Use
> json(Dataset[String]) instead.
> [WARNING] sqlContext.read.json(aRDD)
> [WARNING] ^
> [WARNING]
> sql-cloudant/src/main/scala/org/apache/bahir/cloudant/DefaultSource.scala:152:
> method json in class DataFrameReader is deprecated: Use
> json(Dataset[String]) instead.
> [WARNING] dataFrame =
> sqlContext.sparkSession.read.json(globalRDD)
> [WARNING] ^
> [WARNING] four warnings found
> {code}
> Deprecation warnings in {{CloudantStreaming}} and
> {{CloudantStreamingSelector}} examples:
> {code}
> [INFO] --- scala-maven-plugin:3.2.2:testCompile (scala-test-compile-first) @
> spark-sql-cloudant_2.11 ---
> [INFO] Compiling 11 Scala sources to
> sql-cloudant/target/scala-2.11/test-classes...
> [WARNING]
> sql-cloudant/examples/src/main/scala/org/apache/spark/examples/sql/cloudant/CloudantStreaming.scala:46:
> method json in class DataFrameReader is deprecated: Use
> json(Dataset[String]) instead.
> [WARNING] val changesDataFrame = spark.read.json(rdd)
> [WARNING] ^
> [WARNING]
> sql-cloudant/examples/src/main/scala/org/apache/spark/examples/sql/cloudant/CloudantStreaming.scala:67:
> method registerTempTable in class Dataset is deprecated: Use
> createOrReplaceTempView(viewName) instead.
> [WARNING] changesDataFrame.registerTempTable("airportcodemapping")
> [WARNING] ^
> [WARNING]
> sql-cloudant/examples/src/main/scala/org/apache/spark/examples/sql/cloudant/CloudantStreamingSelector.scala:50:
> method json in class DataFrameReader is deprecated: Use
> json(Dataset[String]) instead.
> [WARNING] val changesDataFrame = spark.read.json(rdd)
> [WARNING] ^
> [WARNING] three warnings found
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
