[
https://issues.apache.org/jira/browse/BAHIR-110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16084166#comment-16084166
]
ASF GitHub Bot commented on BAHIR-110:
--------------------------------------
Github user emlaver commented on a diff in the pull request:
https://github.com/apache/bahir/pull/45#discussion_r126990289
--- Diff:
sql-cloudant/src/test/scala/org/apache/bahir/cloudant/CloudantSparkSQLSuite.scala
---
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.bahir.cloudant
+
+import org.apache.spark.sql.{SparkSession, SQLContext, SQLImplicits}
+
+class CloudantSparkSQLSuite extends ClientSparkFunSuite {
+ // import spark implicits
+ private object testImplicits extends SQLImplicits {
+ protected override def _sqlContext: SQLContext = spark.sqlContext
+ }
+
+ import testImplicits._
+
+ val apiReceiver = "_all_docs"
+
+ override def beforeAll() {
+ runIfTestsEnabled("Prepare Cloudant test databases") {
+ super.beforeAll()
+ spark = SparkSession.builder().config(conf)
+ .config("cloudant.protocol", TestUtils.getProtocol)
+ .config("cloudant.host", TestUtils.getHost)
+ .config("cloudant.username", TestUtils.getUsername)
+ .config("cloudant.password", TestUtils.getPassword)
+ .config("cloudant.apiReceiver", apiReceiver)
--- End diff --
Fixed in 9f8bbc5.
> Replace use of _all_docs API with _changes API in all receivers
> ---------------------------------------------------------------
>
> Key: BAHIR-110
> URL: https://issues.apache.org/jira/browse/BAHIR-110
> Project: Bahir
> Issue Type: Improvement
> Reporter: Esteban Laver
> Original Estimate: 216h
> Remaining Estimate: 216h
>
> Today we use the _changes API for Spark streaming receiver and _all_docs API
> for non-streaming receiver. _all_docs API supports parallel reads (using
> offset and range) but performance of _changes API is still better in most
> cases (even with single threaded support).
> With this ticket we want to:
> a) re-implement all receivers using _changes API
> b) compare performance between the two implementations based on _changes and
> _all_docs
> Based on the results in b) we could decide to either
> - replace _all_docs implementation with _changes based implementation OR
> - allow customers to pick one (with a solid documentation about pros and
> cons)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
