[
https://issues.apache.org/jira/browse/BAHIR-137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16311810#comment-16311810
]
ASF GitHub Bot commented on BAHIR-137:
--------------------------------------
Github user emlaver commented on a diff in the pull request:
https://github.com/apache/bahir/pull/60#discussion_r159722257
--- Diff:
sql-cloudant/src/main/scala/org/apache/bahir/cloudant/internal/ChangesReceiver.scala
---
@@ -72,6 +72,7 @@ class ChangesReceiver(config: CloudantChangesConfig)
val status = headers.getOrElse("Status", IndexedSeq.empty)
val errorMsg = "Error retrieving _changes feed " +
config.getDbname + ": " + status(0)
reportError(errorMsg, new CloudantException(errorMsg))
+ stop(errorMsg)
--- End diff --
Yes, when testing with `spark-submit` command you would see two additional
error messages. First message below is from `reportError`, and the other two
are from `stop`:
```
...
18/01/04 13:04:11 WARN ReceiverSupervisorImpl: Reported error Error
retrieving _changes feed animaldb: HTTP/1.1 401 Unauthorized -
org.apache.bahir.cloudant.common.CloudantException: Error retrieving _changes
feed animaldb: HTTP/1.1 401 Unauthorized
...
18/01/04 13:04:11 INFO ReceiverSupervisorImpl: Stopping receiver with
message: Error retrieving _changes feed animaldb: HTTP/1.1 401 Unauthorized:
...
18/01/04 13:04:11 ERROR ReceiverTracker: Deregistered receiver for stream
0: Error retrieving _changes feed animaldb: HTTP/1.1 401 Unauthorized
```
Since we already call `stop` [in
DefaultSource](https://github.com/apache/bahir/pull/60/files#diff-4a0eb687595fe14565677d020371b69eR154),
I'll remove this line.
I will also be changing [this error
message](https://github.com/apache/bahir/pull/60/files#diff-4a0eb687595fe14565677d020371b69eR165)
as it's more likely an HTTP error happened. For users running with
`spark-submit`, they will get logging in the console. There will be limited or
no logging in DSX which means it might be worth passing the HTTP error message
to `CloudantException`.
> Load performance improvements for _changes API in sql-cloudant
> --------------------------------------------------------------
>
> Key: BAHIR-137
> URL: https://issues.apache.org/jira/browse/BAHIR-137
> Project: Bahir
> Issue Type: Improvement
> Affects Versions: Spark-2.2.0
> Reporter: Esteban Laver
> Assignee: Esteban Laver
>
> Items for improving _changes feed load:
> - Make Spark streaming batch interval visible to the user for tuning based on
> type/size of document and number of docs in database
> - Merge BAHIR-128: Improve stability of _changes receiver
> - Merge BAHIR-154: refactor sql-cloudant to use java-cloudant library
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
