[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17650095#comment-17650095 ] Bhavik Patel commented on RANGER-2128: -- [~simonvanderveldt] / [~waywtdcc] let's discuss here and try to closely work on this. > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Attachments: image-2022-10-10-14-25-30-218.png, support_ranger11.tgz > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Implement SparkSQL plugin -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17629736#comment-17629736 ] Bhavik Patel commented on RANGER-2128: -- Anyone got chance to review updated patch? > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Attachments: image-2022-10-10-14-25-30-218.png, support_ranger11.tgz > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Implement SparkSQL plugin -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17628439#comment-17628439 ] Zhou Yifan commented on RANGER-2128: [~bpatel] Very glad to help. Could you provide more details about the problems in testing? > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Attachments: image-2022-10-10-14-25-30-218.png, support_ranger11.tgz > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Implement SparkSQL plugin -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17628338#comment-17628338 ] Bhavik Patel commented on RANGER-2128: -- [~zhouyifan279] I have rebased the Ranger-Sparksql(RANGER-2128) plugin patch but while testing I have observed policy download and commands are not working as expected. Will you please help on this? > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Attachments: image-2022-10-10-14-25-30-218.png, support_ranger11.tgz > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Implement SparkSQL plugin -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17624846#comment-17624846 ] Bhavik Patel commented on RANGER-2128: -- No major difference in resource wise. It is also mentioned in the attached designed doc that SparkSql is similar to HadoopSql. > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Attachments: image-2022-10-10-14-25-30-218.png, support_ranger11.tgz > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Implement SparkSQL plugin -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17624829#comment-17624829 ] kirby zhou commented on RANGER-2128: What's the difference between SparkSQL's ACL model and HadoopSQL? > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Attachments: image-2022-10-10-14-25-30-218.png, support_ranger11.tgz > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Implement SparkSQL plugin -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17614985#comment-17614985 ] Bhavik Patel commented on RANGER-2128: -- [~jles] [~kirbyzhou] seems like provided patch is based on reusing of HADOOP SQL(Hive) plugin. > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Attachments: image-2022-10-10-14-25-30-218.png, support_ranger11.tgz > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Implement SparkSQL plugin -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17614979#comment-17614979 ] Bhavik Patel commented on RANGER-2128: -- For spark plugin we can use service definition ID as *"204" or* we can use *"11"?* Below ID's are used in the service definition for the different plugins. !image-2022-10-10-14-25-30-218.png! > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Attachments: image-2022-10-10-14-25-30-218.png, support_ranger11.tgz > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Implement SparkSQL plugin -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17442537#comment-17442537 ] MinSung commented on RANGER-2128: - I would love for this to be eventually merged upstream also :) > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Attachments: support_ranger11.tgz > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Implement SparkSQL plugin -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17386314#comment-17386314 ] Bhavik Patel commented on RANGER-2128: -- [~Qin Yao] [~rmartine] [~toopt4] can this Jira be merged? > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Attachments: support_ranger11.tgz > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Implement SparkSQL plugin -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106917#comment-17106917 ] Pradeep Agrawal commented on RANGER-2128: - [~rmartine] : I dont have the env yet with this patch. I will let you know if info needed to test the patch. > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Attachments: support_ranger11.tgz > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Implement SparkSQL plugin -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17102776#comment-17102776 ] Ricardo Martinelli de Oliveira commented on RANGER-2128: [~pradeep] Any updates on this? Do you need a help? > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Attachments: support_ranger11.tgz > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Implement SparkSQL plugin -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17095382#comment-17095382 ] Pradeep Agrawal commented on RANGER-2128: - [~rmartine] : I will try to test the patch and let you know. > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Attachments: support_ranger11.tgz > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Implement SparkSQL plugin -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17093813#comment-17093813 ] Ricardo Martinelli de Oliveira commented on RANGER-2128: [~bosco] what is missing to have this jira resolved? Anything the community can help? This is a very good addition to Ranger, but the Jira is opened for 2 years and we see no visible progress here. Let me know if there's something we can help with. > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Attachments: support_ranger11.tgz > > Time Spent: 50m > Remaining Estimate: 0h > > Implement SparkSQL plugin -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17078678#comment-17078678 ] Pau Tallada Crespí commented on RANGER-2128: I would love for this to be eventually merged upstream :) > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Attachments: support_ranger11.tgz > > Time Spent: 50m > Remaining Estimate: 0h > > Implement SparkSQL plugin -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17026480#comment-17026480 ] t oo commented on RANGER-2128: -- see https://github.com/yaooqinn/spark-ranger/issues/1 and https://yaooqinn.github.io/spark-ranger/ > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Attachments: support_ranger11.tgz > > Time Spent: 40m > Remaining Estimate: 0h > > Implement SparkSQL plugin -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17026318#comment-17026318 ] Velmurugan Periasamy commented on RANGER-2128: -- [~Qin Yao] / [~toopt4] / [~bosco] - not sure where this stands. can this jira be progressed? CC [~rmani] / [~abhay] > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Attachments: support_ranger11.tgz > > Time Spent: 40m > Remaining Estimate: 0h > > Implement SparkSQL plugin -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16847745#comment-16847745 ] t oo commented on RANGER-2128: -- [~bosco] any luck with review? > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Fix For: 2.0.0 > > Attachments: support_ranger11.tgz > > Time Spent: 40m > Remaining Estimate: 0h > > Implement SparkSQL plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16847739#comment-16847739 ] t oo commented on RANGER-2128: -- [~Qin Yao] any chance for a page like [https://cwiki.apache.org/confluence/display/RANGER/Presto+Plugin] ? > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Fix For: 2.0.0 > > Attachments: support_ranger11.tgz > > Time Spent: 40m > Remaining Estimate: 0h > > Implement SparkSQL plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[
https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16843180#comment-16843180
]
Kent Yao commented on RANGER-2128:
--
{quote}What is the use case we are trying to solve here? Is it using the Spark
catalog with Ranger AuthZ? Or is this for the use case where there is no Hive
metastore and Spark has its own catalog (I believe this is the case with Hive3+
and in the more recent Spark2.3+ if I remember correctly)
{quote}
We are adding a Spark SQL Authorizer plugin here, also with row filter and data
masking functions. It uses spark's catalog to do authz, works for both so
called hive tables and spark sql datasource tables.
{quote}Where is this plugin deployed? Will it work on kerberized clusters?
{quote}
Spark has two deploy modes, client and cluster. The authz happens at Spark's
driver process, which will be the ApplicationMaster in cluster mode. In order
to adapt both modes, I shaded all jars to a uber jar which should be put into
`SPARK_HOME`/jars.
It works for kerberized clusters.
{quote}What specifically are differences in what is supported between this
plugin and the current Ranger-Hive Authorizer? What versions of Spark, Hive and
Ranger will this require?
{quote}
This plugin works for SQLs supported by Spark. We currently develop this
plugin against Spark 2.3.2, Ranger master branch. Hive version is not a problem
we need to concern.
{quote}There are pointers to another github project
[https://github.com/yaooqinn/kyuubi] in the thread above which appears to be an
enhanced version of SparkThrift Server. It would be good to understand whether
that has Apache 2 compatible licenses and whether kyuubi has to be included
into an existing Spark deployment directly or via external dependencies for
this Spark SQL Ranger plugin to work?
{quote}
Kyuubi has Apache 2 compatible licenses. Kyuubi need not be added to Spark
libs, Kyuubi can start itself with SPARK_HOME correctly set. If the spark has
implemented this plugin, Kyuubi can directly use it.
{quote}If there is a requirement to have Kyuubi vesion deployed in a cluster on
top of Spark2 then does anyone know whether there is any plan to add this
directly into Spark2 project first class? It becomes more difficult to certify
against such clones of core services in another Apache project if the
mainstream Spark2 project is not supporting this enhanced version.
{quote}
I am afraid that there is no plan for Spark PMCs to add kyuubi to Apache Spark.
Spark apps including its own SparkThrift Server are single "user" apps. Besides
Kyuubi, there are a lot of Projects can provide Spark with multi tenant
feature, such as Apache Livy, Apache Zeppelin etc. Maybe I can devote Kyuubi to
ASF too.
> Implement SparkSQL plugin
> -
>
> Key: RANGER-2128
> URL: https://issues.apache.org/jira/browse/RANGER-2128
> Project: Ranger
> Issue Type: New Feature
> Components: plugins, Ranger
>Affects Versions: 1.1.0
>Reporter: t oo
>Assignee: Kent Yao
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: support_ranger11.tgz
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Implement SparkSQL plugin
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16831868#comment-16831868 ] Srikanth Venkat commented on RANGER-2128: - I have several questions on this: 1) What is the use case we are trying to solve here? Is it using the Spark catalog with Ranger AuthZ? Or is this for the use case where there is no Hive metastore and Spark has its own catalog (I believe this is the case with Hive3+ and in the more recent Spark2.3+ if I remember correctly) 2) Where is this plugin deployed? Will it work on kerberized clusters? 3) What specifically are differences in what is supported between this plugin and the current Ranger-Hive Authorizer? What versions of Spark, Hive and Ranger will this require? 4) There are pointers to another github project [https://github.com/yaooqinn/kyuubi] in the thread above which appears to be an enhanced version of SparkThrift Server. It would be good to understand whether that has Apache 2 compatible licenses and whether kyuubi has to be included into an existing Spark deployment directly or via external dependencies for this Spark SQL Ranger plugin to work? If there is a requirement to have Kyuubi vesion deployed in a cluster on top of Spark2 then does anyone know whether there is any plan to add this directly into Spark2 project first class? It becomes more difficult to certify against such clones of core services in another Apache project if the mainstream Spark2 project is not supporting this enhanced version. Can we clarify the above items these please? > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Fix For: 2.0.0 > > Attachments: support_ranger11.tgz > > Time Spent: 20m > Remaining Estimate: 0h > > Implement SparkSQL plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16831709#comment-16831709 ] Velmurugan Periasamy commented on RANGER-2128: -- [~Qin Yao] - could you please post your patch to review board - [https://reviews.apache.org/groups/ranger/]? Thanks. > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Fix For: 2.0.0 > > Attachments: support_ranger11.tgz > > Time Spent: 20m > Remaining Estimate: 0h > > Implement SparkSQL plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16815803#comment-16815803 ] t oo commented on RANGER-2128: -- Ship it! > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Fix For: 2.0.0 > > Attachments: support_ranger11.tgz > > Time Spent: 10m > Remaining Estimate: 0h > > Implement SparkSQL plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16776733#comment-16776733 ] Kent Yao commented on RANGER-2128: -- [~pradeep.agrawal] I have added a trial doc link in this ticket, and I will add a formal doc in my pull request ASAP. thanks > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Fix For: 2.0.0 > > Attachments: support_ranger11.tgz > > Time Spent: 10m > Remaining Estimate: 0h > > Implement SparkSQL plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16776683#comment-16776683 ] Pradeep Agrawal commented on RANGER-2128: - [~Qin Yao] : Can you add a doc which explains what will be covered as part of this task and how it can be used. > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Fix For: 2.0.0 > > Attachments: support_ranger11.tgz > > Time Spent: 10m > Remaining Estimate: 0h > > Implement SparkSQL plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16776477#comment-16776477 ] Kent Yao commented on RANGER-2128: -- thanks [~bosco] [~toopt4] > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Fix For: 2.0.0 > > Attachments: support_ranger11.tgz > > Time Spent: 10m > Remaining Estimate: 0h > > Implement SparkSQL plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16775777#comment-16775777 ] Don Bosco Durai commented on RANGER-2128: - Sorry, I got pulled into other things There are few feedback, let me consolidate them and give it to [~Qin Yao] > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Fix For: 2.0.0 > > Attachments: support_ranger11.tgz > > Time Spent: 10m > Remaining Estimate: 0h > > Implement SparkSQL plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765401#comment-16765401 ] t oo commented on RANGER-2128: -- gentle ping > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Fix For: 2.0.0 > > Attachments: support_ranger11.tgz > > > Implement SparkSQL plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16744754#comment-16744754 ] Kent Yao commented on RANGER-2128: -- hi [~bosco] [~pengbo] , I have update my pull request on githup. Hope to get your feedback. Thanks. > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Fix For: 2.0.0 > > Attachments: support_ranger11.tgz > > > Implement SparkSQL plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16739219#comment-16739219 ] peng bo commented on RANGER-2128: - It seems that's exactly what i am looking for. Would you mind to give some updates please? [~bosco][~Qin Yao] > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Fix For: 2.0.0 > > Attachments: support_ranger11.tgz > > > Implement SparkSQL plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16682739#comment-16682739 ] Don Bosco Durai commented on RANGER-2128: - [~Qin Yao] thanks. Let me coordinate with you on your changes. Since I was blocked, I ended taking out all references to Hive Context, which I feel is a cleaner option. We should plan to merge both our codes. > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Fix For: 2.0.0 > > Attachments: support_ranger11.tgz > > > Implement SparkSQL plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16671073#comment-16671073 ] Kent Yao commented on RANGER-2128: -- Hi [~bosco], sorry for the delay, kind of busy lately. I also ran into the same class cast exception while the metastore client was configured into direct sql connection. I think i have found the problem and fixed it in https://github.com/yaooqinn/spark-authorizer/blob/master/src/main/scala/org/apache/spark/sql/hive/client/AuthzImpl.scala#L63 I guess we can get this done as soon as possible > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Fix For: 2.0.0 > > Attachments: support_ranger11.tgz > > > Implement SparkSQL plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[
https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593215#comment-16593215
]
Don Bosco Durai commented on RANGER-2128:
-
{quote}Hi Don Bosco Durai Kent Yao , any ideas on next steps?{quote}
[~toopt4] sorry for the delay. I was held up in other work.
I was getting class cast exception for HiveClient class in Spark 2.3 and not in
2.2. I ended up refactoring [~Qin Yao] code to remove all dependencies with
HiveClient and HiveSession. Now it is only dependent on Spark. However, I had
to copy some the Hive Plugin code into Spark Plugin.
I have to do some clean up of my code. After that, I can upload it to this JIRA
for [~Qin Yao] to review and add it to his pull request.
> Implement SparkSQL plugin
> -
>
> Key: RANGER-2128
> URL: https://issues.apache.org/jira/browse/RANGER-2128
> Project: Ranger
> Issue Type: New Feature
> Components: plugins, Ranger
>Affects Versions: 1.1.0
>Reporter: t oo
>Assignee: Kent Yao
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: support_ranger11.tgz
>
>
> Implement SparkSQL plugin
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16584540#comment-16584540 ] t oo commented on RANGER-2128: -- Hi [~bosco] [~Qin Yao] , any ideas on next steps? > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Fix For: 2.0.0 > > Attachments: support_ranger11.tgz > > > Implement SparkSQL plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[
https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547526#comment-16547526
]
Don Bosco Durai commented on RANGER-2128:
-
{quote} ran thrift server with spark2.3.1(built-in hive) on yarn against Apache
Hadoop2.7.3/Hive Metastore Server2.1/ranger0.5.3-rc3{quote}
For some reason it is not working for me. I get the same class cast exception.
I built spark branch 2.3 from github. But I using the latest Ranger. It
shouldn't matter.
I will put some debug statement in Spark to see if I can figure this out.
> Implement SparkSQL plugin
> -
>
> Key: RANGER-2128
> URL: https://issues.apache.org/jira/browse/RANGER-2128
> Project: Ranger
> Issue Type: New Feature
> Components: plugins, Ranger
>Affects Versions: 1.1.0
>Reporter: t oo
>Assignee: Kent Yao
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: support_ranger11.tgz
>
>
> Implement SparkSQL plugin
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[
https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16544855#comment-16544855
]
Kent Yao commented on RANGER-2128:
--
My env is a fully kerberized cluster, and I ran thrift server with
spark2.3.1(built-in hive) on yarn against Apache Hadoop2.7.3/Hive Metastore
Server2.1/ranger0.5.3-rc3
With the below secure options
{code:java}
hive.security.authorization.manager=org.apache.ranger.authorization.hive.authorizer.RangerHiveAuthorizerFactory
hive.security.metastore.authenticator.manager=org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator
hive.security.metastore.authorization.manager=org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
hive.server2.authentication=KERBEROS
hive.server2.enable.doAs=false
{code}
Only works for single user who start the server, fails to switch user
The exception is related to the thrift server do "use:database" action during
opening session, which will call sparksession.sql("use default"), AFAIK once it
is executed the Isolated Hive classloader will be turned off
> Implement SparkSQL plugin
> -
>
> Key: RANGER-2128
> URL: https://issues.apache.org/jira/browse/RANGER-2128
> Project: Ranger
> Issue Type: New Feature
> Components: plugins, Ranger
>Affects Versions: 1.1.0
>Reporter: t oo
>Assignee: Kent Yao
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: support_ranger11.tgz
>
>
> Implement SparkSQL plugin
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[
https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16544838#comment-16544838
]
Don Bosco Durai commented on RANGER-2128:
-
{quote}When I tried thrift server, I did not see such an exception.{quote}
How are you running your Spark/Thrift server? I am using HDP 2.6.5 because I
have setup Kerberos with it. I can try your way also.
{quote}Livy Server is multi tenant. It launches spark applications separately
in different JVMs, which is easy to work with our current work.{quote}
I have not tried with Livy Server yet. That was next in my list.
{quote}By 1-3, I guess we can supply all real multi tenant servers for spark,
for spark thrift server, we may need to make some efforts to spark community to
support this such as step 4 and some related work{quote}
I agree with you. Let's try to get as close as possible to get this working.
Then we can see how to get the rest working.
Thanks
> Implement SparkSQL plugin
> -
>
> Key: RANGER-2128
> URL: https://issues.apache.org/jira/browse/RANGER-2128
> Project: Ranger
> Issue Type: New Feature
> Components: plugins, Ranger
>Affects Versions: 1.1.0
>Reporter: t oo
>Assignee: Kent Yao
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: support_ranger11.tgz
>
>
> Implement SparkSQL plugin
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16544822#comment-16544822 ] Kent Yao commented on RANGER-2128: -- Emmm..weird, ClassCastException shouldn't occur if the hive client was correctly replaced by reflecting. When I tried thrift server, I did not see such an exception. Although I don’t want to admit it, I have to say that the current implementation of Spark thrift server is really a mess. To meet spark's architecture, it was designed as an single spark application without multi tenancy support. Livy Server is multi tenant. It launches spark applications separately in different JVMs, which is easy to work with our current work. I guess that we may implement spark sql plugin step by step, 1. Resolve the hive client and jersey incompatibility 2. Implement plugin for spark client mode 3. Implement plugin for spark cluster mode 4. Separate metastore layer from PhysicalScan layer in Spark Thrift Server 5. Implement plugin for thrift server By 1-3, I guess we can supply all real multi tenant servers for spark, for spark thrift server, we may need to make some efforts to spark community to support this such as step 4 and some related work. > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Fix For: 2.0.0 > > Attachments: support_ranger11.tgz > > > Implement SparkSQL plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[
https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16544410#comment-16544410
]
Don Bosco Durai commented on RANGER-2128:
-
{quote}You can try Spark 2.3.1, because this change
https://github.com/apache/spark/blob/branch-2.3/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala#L129
ensures the SessionState that contains the thrift user body will be
reused.{quote}
[~Qin Yao] thanks. I was able to get the user from the State. In the AuthzImpl,
the following code gave the session user. I will update the Ranger Authorizer
to use this user
{code:java}
clientImpl.state.getUserName()
{code}
Regarding the ClassCastException, "org.apache.spark.sql.AnalysisException:
org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassCastException:
org.apache.hadoop.hive.metastore.ObjectStore cannot be cast to
org.apache.hadoop.hive.metastore.RawStore", I got this error even after I
removed all HiveAuthorizer related code. Do you think this is because of
different class loader that might be used when we do
"ext.injectOptimizerRule(Authorizer)" in the case of Thrift Server? You are
familiar with Spark, any clue is appreciated. Thanks
> Implement SparkSQL plugin
> -
>
> Key: RANGER-2128
> URL: https://issues.apache.org/jira/browse/RANGER-2128
> Project: Ranger
> Issue Type: New Feature
> Components: plugins, Ranger
>Affects Versions: 1.1.0
>Reporter: t oo
>Assignee: Kent Yao
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: support_ranger11.tgz
>
>
> Implement SparkSQL plugin
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[
https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16544118#comment-16544118
]
Don Bosco Durai commented on RANGER-2128:
-
[~Qin Yao] I managed to get your code working with Ranger master branch. I have
attached the updates I have done. Can you try it on your side and see whether
it works for you?
It works with Spark Shell, but with Thrift Server I am getting an exception.
Have you seen this before? I have enabled Ranger in thrift server by adding the
following line in sparkconf.conf
{code:java}
spark.sql.extensions
org.apache.ranger.authorization.spark.authorizer.RangerSparkSQLExtension
{code}
Error opening session:
org.apache.spark.sql.AnalysisException:
org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassCastException:
org.apache.hadoop.hive.metastore.ObjectStore cannot be cast to
org.apache.hadoop.hive.metastore.RawStore;
at
org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:106)
at
org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:194)
at
org.apache.spark.sql.internal.SharedState.globalTempViewManager$lzycompute(SharedState.scala:138)
at
org.apache.spark.sql.internal.SharedState.globalTempViewManager(SharedState.scala:133)
at
org.apache.spark.sql.hive.HiveSessionStateBuilder$$anonfun$2.apply(HiveSessionStateBuilder.scala:54)
at
org.apache.spark.sql.hive.HiveSessionStateBuilder$$anonfun$2.apply(HiveSessionStateBuilder.scala:54)
at
org.apache.spark.sql.catalyst.catalog.SessionCatalog.globalTempViewManager$lzycompute(SessionCatalog.scala:91)
at
org.apache.spark.sql.catalyst.catalog.SessionCatalog.globalTempViewManager(SessionCatalog.scala:91)
at
org.apache.spark.sql.catalyst.catalog.SessionCatalog.setCurrentDatabase(SessionCatalog.scala:252)
at
org.apache.spark.sql.execution.command.SetDatabaseCommand.run(databases.scala:59)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:79)
at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:190)
at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:190)
at org.apache.spark.sql.Dataset$$anonfun$52.apply(Dataset.scala:3253)
at
org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:77)
at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3252)
at org.apache.spark.sql.Dataset.(Dataset.scala:190)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:75)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:638)
at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:694)
at
org.apache.spark.sql.hive.thriftserver.SparkSQLSessionManager.openSession(SparkSQLSessionManager.scala:70)
at
org.apache.hive.service.cli.CLIService.openSession(CLIService.java:194)
at
org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:354)
at
org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:246)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1253)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1238)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:53)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
> Implement SparkSQL plugin
> -
>
> Key: RANGER-2128
> URL: https://issues.apache.org/jira/browse/RANGER-2128
> Project: Ranger
> Issue Type: New Feature
> Components: plugins, Ranger
>Affects Versions: 1.1.0
>Reporter: t oo
>Assignee: Kent Yao
>Priority: Major
> Fix For: 2.0.0
>
>
> Implement SparkSQL plugin
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16538329#comment-16538329 ] Kent Yao commented on RANGER-2128: -- Hi [~bosco] You can try Spark 2.3.1, because this change https://github.com/apache/spark/blob/branch-2.3/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala#L129 ensures the SessionState that contains the thrift user body will be reused. But there is still a problem that the hive metastore client which contains the ugi for authorization is shared among SparkSession s inside Spark Thrift Sever. We can not reuse it for different users. We may need to manage a map to do privileges checking, which may left "show tables" / "show databases" ... statements unchecked and cause plenty of connections to the metastore server. > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Fix For: 2.0.0 > > > Implement SparkSQL plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16534375#comment-16534375 ] Kent Yao commented on RANGER-2128: -- bq. I was able to use the latest Ranger Hive Plugin with Spark That' s great! bq. I have to do a bit of hack to use Jersey2 While organizing all related jars in $SPARK_HOME/jars/ranger-hive-plugin-impl/ and using shim and classloader, it seems to have no problems of jar conflicting. Did you add all jars to spark's classloader? bq. If you know to get that somehow, then our implementation of the Spark plugin will be much simpler and not dependent on the HiveAuthorizer. My apology for not covering Spark‘s Own Thrift Server. Let me give a try please. > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Fix For: 2.0.0 > > > Implement SparkSQL plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16534345#comment-16534345 ] Don Bosco Durai commented on RANGER-2128: - Hi [~Qin Yao], I was able to use the latest Ranger Hive Plugin with Spark. I have to do a bit of hack to use Jersey2 and also comment out one line in the Hive plugin code. Next, I am trying to see if I can integrate which SparkSQL Thrift server. I will update this JIRA after that. Hopefully by tomorrow. I also wrote a standalone Spark plugin and updated your code to call it. However, when I ran it in SparkSQL Thrift server, it was called, but I was not able to get the effective user (when doAs is false). I traced the code and it seems the Hive SessionState that is created has the effective user, however, I am not able to get that outside of it. If you know to get that somehow, then our implementation of the Spark plugin will be much simpler and not dependent on the HiveAuthorizer. > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Fix For: 2.0.0 > > > Implement SparkSQL plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[
https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16528628#comment-16528628
]
Don Bosco Durai commented on RANGER-2128:
-
{quote}Hi Don Bosco Durai, the 0.6.x based ranger-hive-plugin is the pioneer
version of Apache Ranger, which is already hive2.1 based. Is there any better
option than using Hortonwork repo or inlining duplicated 0.5 based codes?{quote}
I am trying to see if we can remove the dependency with Hive and instantiate
the plugin outside of it. Let me give a try.
> Implement SparkSQL plugin
> -
>
> Key: RANGER-2128
> URL: https://issues.apache.org/jira/browse/RANGER-2128
> Project: Ranger
> Issue Type: New Feature
> Components: plugins, Ranger
>Affects Versions: 1.1.0
>Reporter: t oo
>Assignee: Kent Yao
>Priority: Major
> Fix For: 1.1.0
>
>
> Implement SparkSQL plugin
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16528620#comment-16528620 ] t oo commented on RANGER-2128: -- I've removed any references to Kylin now. # I wasn't planning on doing any work on this Jira as i'm still a java noob and getting used to the overall design of Ranger. Just wanted to provide small input that could be used by someone more experienced to tie together for their testing to start earlier. # I know there is likely to be an instance where HiveServer2 (port 1) is NOT installed/running, but just SparkThriftServer (port 10002) is running with HiveMetastore (9083) so I feel they should be independent with perhaps an option to inherit policies from Hive2 if Hive2 policies exist. # Have removed now, seem to be inherited from the kylin copy/paste. Would love to get involved with testing this on SparkThriftServer but don't feel I have much more to offer in terms of commits. > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Fix For: 1.1.0 > > > Implement SparkSQL plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16528565#comment-16528565 ] Don Bosco Durai commented on RANGER-2128: - [~toopt4] I saw that you have added the code for Spark as first class service. It is looking good, but there are still a few places you have Kylin. Few comments: # To avoid overlap with work from [~Qin Yao] , I feel you should create a dependent Jira and have your patch as part of it # Do you see a case where we would have a different meta store than Hive MetaStore? If not, we could get the policies directly from Hive Service def. So that the same policies will be enforced by Hive and Spark. # In your code, you have references to Project. E.g. getProjectResponse(). Can you clarify what it is? Just to make sure we don't step on each other work, can you list the tasks you will be working on? Also, Apache Ranger doesn't use pull request. Since this JIRA might need coordination with multiple contributors, let's get this to a working stage and then we can convert this to patch request and use review board for get comments from others. > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Fix For: 1.1.0 > > > Implement SparkSQL plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16528557#comment-16528557 ] t oo commented on RANGER-2128: -- PR [https://github.com/apache/ranger/pull/27] raised for web service, copy pasted from kylin plugin, probably needs tweaks. > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Fix For: 1.1.0 > > > Implement SparkSQL plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16527549#comment-16527549 ] Kent Yao commented on RANGER-2128: -- Hi [~bosco], the 0.6.x based [ranger-hive-plugin|https://mvnrepository.com/artifact/org.apache.ranger/ranger-hive-plugin/0.6.3] is the pioneer version of Apache Ranger, which is already hive2.1 based. Is there any better option than using Hortonwork repo or inline duplicated 0.5 based codes. > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Fix For: 1.1.0 > > > Implement SparkSQL plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525812#comment-16525812 ] Kent Yao commented on RANGER-2128: -- [~toopt4] The injection affects spark catalyst, so it might be adaptable with any kinds of spark apps which go through it. Anyway, spark thrift server has lacks for enterprise deployments, I suggest you https://github.com/yaooqinn/kyuubi as a better option > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Fix For: 1.1.0 > > > Implement SparkSQL plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16524958#comment-16524958 ] t oo commented on RANGER-2128: -- does it work with spark-thriftserver rather than spark-shell? > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Fix For: 1.1.0 > > > Implement SparkSQL plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[
https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16524935#comment-16524935
]
Kent Yao commented on RANGER-2128:
--
[~bosco] spark-authorizer works for spark2.1 and later, this pull request is
compatible with *2.2 and later * for supporting spark.sql.extensions
configuration.
I had it tested with my ranger 0.5-based personal release version against
2.2.1, and works fine.
START
{code:java}
bin/spark-shell --proxy-user hzyaoqin --conf
spark.sql.warehouse.dir=/user/hzyaoqin/warehouse --conf
spark.sql.extensions=org.apache.ranger.authorization.spark.authorizer.RangerSparkSQLExtension
{code}
TEST
{code:java}
scala> sc.setLogLevel("info");
scala> sql("show tables").show
{code}
LOG Snippet
{code:java}
18/06/27 19:11:43 INFO RangerPluginClassLoaderUtil:
getFilesInDirectory('/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl'):
adding
/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl/ranger-plugins-cred-0.5-mammut416.jar
18/06/27 19:11:43 INFO RangerPluginClassLoaderUtil:
getFilesInDirectory('/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl'):
adding
/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl/jersey-client-1.14.jar
18/06/27 19:11:43 INFO RangerPluginClassLoaderUtil:
getFilesInDirectory('/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl'):
adding
/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl/eclipselink-2.5.2.jar
18/06/27 19:11:43 INFO RangerPluginClassLoaderUtil:
getFilesInDirectory('/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl'):
adding
/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl/gson-2.2.4.jar
18/06/27 19:11:43 INFO RangerPluginClassLoaderUtil:
getFilesInDirectory('/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl'):
adding
/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl/ranger-plugins-common-0.5-mammut416.jar
18/06/27 19:11:43 INFO RangerPluginClassLoaderUtil:
getFilesInDirectory('/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl'):
adding
/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl/ranger-hive-plugin-0.5-mammut416.jar
18/06/27 19:11:43 INFO RangerPluginClassLoaderUtil:
getFilesInDirectory('/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl'):
adding
/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl/httpcore-4.4.6.jar
18/06/27 19:11:43 INFO RangerPluginClassLoaderUtil:
getFilesInDirectory('/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl'):
adding
/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl/ranger-plugins-audit-0.5-mammut416.jar
18/06/27 19:11:43 INFO RangerPluginClassLoaderUtil:
getFilesInDirectory('/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl'):
adding
/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl/solr-solrj-5.5.5.jar
18/06/27 19:11:43 INFO RangerPluginClassLoaderUtil:
getFilesInDirectory('/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl'):
adding
/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl/noggit-0.6.jar
18/06/27 19:11:43 INFO RangerPluginClassLoaderUtil:
getFilesInDirectory('/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl'):
adding
/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl/httpclient-4.5.3.jar
18/06/27 19:11:43 INFO RangerPluginClassLoaderUtil:
getFilesInDirectory('/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl'):
adding
/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl/mysql-connector-java-5.1.39.jar
18/06/27 19:11:43 INFO RangerPluginClassLoaderUtil:
getFilesInDirectory('/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl'):
adding
/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl/httpmime-4.5.3.jar
18/06/27 19:11:43 INFO RangerPluginClassLoaderUtil:
getFilesInDirectory('/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl'):
adding
/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl/ranger_solrj-0.5-mammut416.jar
18/06/27 19:11:43 INFO RangerPluginClassLoaderUtil:
getFilesInDirectory('/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl'):
adding
/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl/javax.persistence-2.1.0.jar
18/06/27 19:11:43 INFO RangerPluginClassLoaderUtil:
getFilesInDirectory('/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl'):
adding
/home/hadoop/spark-2.2.1-bin-hadoop2.7/jars/ranger-hive-plugin-impl/mybatis-3.2.8.jar
{code}
PS, I'd like to know the latest version which supports hive1.2.1 by
ranger-hive-plugin to build spark-plugin assembly
Thanks
> Implement SparkSQL plugin
> --
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16524800#comment-16524800 ] t oo commented on RANGER-2128: -- [~bosco] i saw Spark 2.2.1 in the PR > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Fix For: 1.1.0 > > > Implement SparkSQL plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16524662#comment-16524662 ] Don Bosco Durai commented on RANGER-2128: - [~Qin Yao] thanks for putting this together so quickly. I have got your changes and done the build. I will try it out and let you know how it goes. Any specific Spark version you tried this with? > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Fix For: 1.1.0 > > > Implement SparkSQL plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16523629#comment-16523629 ] Kent Yao commented on RANGER-2128: -- I submitted a pull request with basic functions to master branch for testing and further discussions > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Fix For: 1.1.0 > > > Implement SparkSQL plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[
https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16522919#comment-16522919
]
Don Bosco Durai commented on RANGER-2128:
-
{quote}It has exposed Parser/Analyzer/Optimizer/Planner, which is so great for
all users. It also makes it easier for users to call our plug-in.
1. spark-authorizer is designed as a Optimize Rule for Spark SQL and executed
after all other default rules because rules, such as column pruning, projection
push down, and so on, should be operated first.
{quote}
I was wondering if it would be difficult to migrate your extension to use the
official hook provided by Spark? If we can do that, then it might be easy to
add Ranger features like dynamic UDF and row level filtering.
{quote}2. spark-authorizer has to visit hive SessionState object which is not
accessible for spark context classloader because Spark use a isolated
classloader to load hive client jars.
2.1 spark-authorizer itself will rewrite SessionState object the first time to
do privileges checking
{quote}
I checked that. It is a pretty good hack that works :) I had to update it to
support custom authentication. The current Ranger Hive Plugin use Hadoop UGI,
which only knows Kerberos and Simple Auth.
{quote}2.2 kyuubi hacks spark and turn off that classloader.
{quote}
I went through your documentation, it seems you have added a lot of good
features. Currently, kyuubi is a custom build. Is it possible to integrate your
extensions as an addon to existing deployment? In this way, users can deploy
the default Thrift Server, but using some properties or code injections adds
your feature? We might then able to support Livy also with the same code base.
{quote}3. spark-authorizer reuses the ranger hive plugin(0.5）which contains
incompatible jersey dependencies with spark ones.
{quote}
There are few limitations with Ranger 0.5, most notably it doesn't support Tag
Based policies. I was thinking, we should just implement first class plugin for
SparkSQL using Ranger 0.7 or 1.0. It could use the same Hive
ServiceDef/Policies, but native implementation for SparkSQL. In this way, we
don't have to be dependent with Hive libraries and it's limitation.
{quote}And what are the steps I should follow to contribute Ranger?
{quote}
I have added you as a contributor to Ranger. You should be able to assign Jira
to yourself and create new ones. I was thinking of splitting the work among
those interested. Since you are familiar with the Spark code, do you want to
look into the new extensions and see how we can implement basic authorization
and advanced features like dynamic masking/UDF and Row Level filtering? I can
look into Tag based policies and also see if I can extract your current Spark
Authorizer feature into native SparkSQL Ranger Plugin.
Give me your thoughts and suggestions.
Thanks
> Implement SparkSQL plugin
> -
>
> Key: RANGER-2128
> URL: https://issues.apache.org/jira/browse/RANGER-2128
> Project: Ranger
> Issue Type: New Feature
> Components: plugins, Ranger
>Affects Versions: 1.1.0
>Reporter: t oo
>Priority: Major
> Fix For: 1.1.0
>
>
> Implement SparkSQL plugin
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16522003#comment-16522003 ] Kent Yao commented on RANGER-2128: -- [~bosco] I had a quick look on SPARK-18127 and its related PR. It has exposed Parser/Analyzer/Optimizer/Planner, which is so great for all users. It also makes it easier for users to call our plug-in. Some existing issues may be considered before we start 1. spark-authorizer is designed as a Optimize Rule for Spark SQL and executed after all other default rules because rules, such as column pruning, projection push down, and so on, should be operated first. 2. spark-authorizer has to visit hive SessionState object which is not accessible for spark context classloader because Spark use a isolated classloader to load hive client jars. 2.1 spark-authorizer itself will rewrite SessionState object the first time to do privileges checking 2.2 kyuubi hacks spark and turn off that classloader. 3. spark-authorizer reuses the ranger hive plugin(0.5）which contains incompatible jersey dependencies with spark ones. And what are the steps I should follow to contribute Ranger? Thanks > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Priority: Major > Fix For: 1.1.0 > > > Implement SparkSQL plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16521958#comment-16521958 ] Don Bosco Durai commented on RANGER-2128: - [~Qin Yao] thanks for helping us out. When I went through your spark-authorizer code, you had mapped all Spark actions to Hive actions. It was pretty impressive. Thanks for pointing out (offline) to me your work on [https://github.com/yaooqinn/kyuubi.] It seems to be what everyone wants. Since you are familiar with the Spark integrations and challenges, what is your recommendation? Can we work on a high-level design flow? E.g. Can we leverage the new Spark hook to implement some of the plugin interactions? Thanks > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Priority: Major > Fix For: 1.1.0 > > > Implement SparkSQL plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16521944#comment-16521944 ] Kent Yao commented on RANGER-2128: -- Thanks for your attention to spark-authorizer and I am willing to help > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Priority: Major > Fix For: 1.1.0 > > > Implement SparkSQL plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16521878#comment-16521878 ] Don Bosco Durai commented on RANGER-2128: - I integrated Ranger with SparkSQL ThriftServer based on [https://github.com/yaooqinn/spark-authorizer.] Since the Thrift Server was based on Hive 1.2, I had to integrate with Ranger 0.5. I have reached out to the developer from the above github to see if he can help us. Spark community has also added hooks to Spark (https://issues.apache.org/jira/browse/SPARK-18127) which can be used to implement Ranger Plugin has first-class integration. > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Priority: Major > Fix For: 1.1.0 > > > Implement SparkSQL plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519035#comment-16519035 ] t oo commented on RANGER-2128: -- Connections to HIVESERVER2 port 1 have ranger doing masking and authorization so certain users can't see some tables/columns or see masked columns. Would like same functionality for SparkThriftServer (ie SPARKSQL) port 10015 > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Priority: Major > Fix For: 1.1.0 > > > Implement SparkSQL plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16518669#comment-16518669 ] Don Bosco Durai commented on RANGER-2128: - [~toopt4] can you be more specific what you are looking for? Thanks > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Priority: Major > Fix For: 1.1.0 > > > Implement SparkSQL plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
