[
https://issues.apache.org/jira/browse/SPARK-26254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16711516#comment-16711516
]
Gabor Somogyi commented on SPARK-26254:
---------------------------------------
I've reached a state where tradeoff has to be made, so interested in opinions
[~vanzin] [~ste...@apache.org]
I've created a project with token-providers name which is depending on core.
With this successfully extracted all the nasty hive + kafka dependencies + all
token providers are there. Then loaded the providers with ServiceLoader which
also works fine. Finally reached a point where kafka-sql project expects couple
of things from KafkaUtil which is in token-providers now. Here is the list of
problems:
{noformat}
[error]
/Users/gaborsomogyi/spark/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala:31:
object KafkaTokenUtil is not a member of package
org.apache.spark.deploy.security
[error] import org.apache.spark.deploy.security.KafkaTokenUtil
[error] ^
[error]
/Users/gaborsomogyi/spark/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSecurityHelper.scala:25:
object KafkaTokenUtil is not a member of package
org.apache.spark.deploy.security
[error] import org.apache.spark.deploy.security.KafkaTokenUtil
[error] ^
[error]
/Users/gaborsomogyi/spark/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSecurityHelper.scala:32:
not found: value KafkaTokenUtil
[error] KafkaTokenUtil.TOKEN_SERVICE) != null
[error] ^
[error]
/Users/gaborsomogyi/spark/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSecurityHelper.scala:37:
not found: value KafkaTokenUtil
[error] KafkaTokenUtil.TOKEN_SERVICE)
[error] ^
[error]
/Users/gaborsomogyi/spark/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala:566:
not found: value KafkaTokenUtil
[error] if (KafkaTokenUtil.isGlobalJaasConfigurationProvided) {
[error] ^
+ all isTokenAvailable tests expects TOKEN_KIND, TOKEN_SERVICE +
KafkaDelegationTokenIdentifier in KafkaSecurityHelperSuite
{noformat}
Here I see these possibilities:
* Hardcode TOKEN_KIND + TOKEN_SERVICE and duplicate
isGlobalJaasConfigurationProvided => The drawback here is we can't really test
whether the provider created token can be read in kafka-sql (we can actually
but with hardcoded strings in both sides which makes it brittle)
* As we're loading providers with ServiceLoader the kafka related one can be
moved to kafka-sql => The drawback is that providers spread around and this
code can't really be reused in DStreams.
* Add token-providers provided dependency to kafka-sql project => It's kinda'
weird but at the moment looks the least problematic
Waiting on opinions...
> Move delegation token providers into a separate project
> -------------------------------------------------------
>
> Key: SPARK-26254
> URL: https://issues.apache.org/jira/browse/SPARK-26254
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.0.0
> Reporter: Gabor Somogyi
> Priority: Major
>
> There was a discussion in
> [PR#22598|https://github.com/apache/spark/pull/22598] that there are several
> provided dependencies inside core project which shouldn't be there (for ex.
> hive and kafka). This jira is to solve this problem.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
