[
https://issues.apache.org/jira/browse/BAHIR-122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099473#comment-16099473
]
ASF GitHub Bot commented on BAHIR-122:
--------------------------------------
Github user ire7715 commented on the issue:
https://github.com/apache/bahir/pull/48
Thanks @ckadner for suggestion.
Agree with your point, just modified the `SparkGCPCredentialsBuilderSuite`
and make it read the environment variables for keyfiles defined in
`PubsubTestUtils`. It ignores the test cases if the keyfiles or email account
are not set(or file doesn't exist) and shows the hint message.
Also discovered that `ServiceAccountCredentials.getFileBuffer` didn't check
file existence before opening, just added the check.
Don't know if the force push would bother you when reviewing, so I made
some manual-diff:
----
SparkGCPCredentials.scala L66
\- `if (filePath.isEmpty) Array[Byte]()`
\+ `if (filePath.isEmpty || !Files.exists(Paths.get(filePath.get)))
Array[Byte]()`
----
SparkGCPCredentialsBuilderSuite.scala
> Deleted resource variables, let `jsonFilePath`, `p12FilePath` and
`emailAccount` read system environment.
> Implemented `jsonAssumption` and `p12Assumption`, to check the existence
of corresponding environment variables and files.
> No longer expected `credential.refreshToken` thrown an Exception,
expected it could retrieve the real token.
> [PubSub] Make "ServiceAccountCredentials" really broadcastable
> --------------------------------------------------------------
>
> Key: BAHIR-122
> URL: https://issues.apache.org/jira/browse/BAHIR-122
> Project: Bahir
> Issue Type: Improvement
> Components: Spark Streaming Connectors
> Reporter: Ire Sun
>
> The origin implementation broadcast the key file path to Spark cluster, then
> the executor read key file with the broadcasted path. Which is absurd, if you
> are using a shared Spark cluster in a group/company, you certainly not want
> to (and have no right to) put your key file on each instance of the cluster.
> If you store the key file on driver node and submit your job to a remote
> cluster. You would get the following warning:
> {{WARN ReceiverTracker: Error reported by receiver for stream 0: Failed to
> pull messages - java.io.FileNotFoundException}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
