[jira] [Commented] (RANGER-1300) S3 support
[
https://issues.apache.org/jira/browse/RANGER-1300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16522966#comment-16522966
]
Don Bosco Durai commented on RANGER-1300:
-
{quote} * Proxy (for Ceph/RadosGW at the moment):
[https://github.com/bolkedebruin/s3gw]{quote}
Hi [~bolke], I looked into your code. Seems you have implemented Ranger policy
evaluation in golang :) I had few questions.
# Did you consider using JNI integration? I don't know much about GO, I saw
few libraries like this [https://github.com/Centny/jnigo.]
# We had a similar requirement for Apache HAWQ, which is written in 'C'. We
ended up using JNI to Ranger plugin written Java. There are multiple advantages
of using JNI approach because the Ranger community has optimized the policy
evaluation significantly, including using Trie indexing to grossly increase the
performance. Also having one code base helps in keeping all platform in the
same feature level, like supporting Tag Based Policies, custom enrichment,
conditional policies, etc. Also integrating with Ranger Audit framework will be
automatically supported if you use Java implementation
# How is Ceph configured to use the Ranger plugin? I couldn't see the
documentation for that. Is it just configuration or we need to recompile Ceph
with your plugin?
> S3 support
> --
>
> Key: RANGER-1300
> URL: https://issues.apache.org/jira/browse/RANGER-1300
> Project: Ranger
> Issue Type: New Feature
> Components: plugins
>Reporter: Jose
>Priority: Major
> Attachments: ranger-servicedef-aws-s3.json
>
>
> As more and more people are deploying hadoop into AWS and as S3 is used in
> lots of application. It'd be nice to have S3 support built into Ranger.
> It's not a trivial task. Right now Ranger Storage support (only hdfs) runs
> directly in the Namenode
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[
https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16522919#comment-16522919
]
Don Bosco Durai commented on RANGER-2128:
-
{quote}It has exposed Parser/Analyzer/Optimizer/Planner, which is so great for
all users. It also makes it easier for users to call our plug-in.
1. spark-authorizer is designed as a Optimize Rule for Spark SQL and executed
after all other default rules because rules, such as column pruning, projection
push down, and so on, should be operated first.
{quote}
I was wondering if it would be difficult to migrate your extension to use the
official hook provided by Spark? If we can do that, then it might be easy to
add Ranger features like dynamic UDF and row level filtering.
{quote}2. spark-authorizer has to visit hive SessionState object which is not
accessible for spark context classloader because Spark use a isolated
classloader to load hive client jars.
2.1 spark-authorizer itself will rewrite SessionState object the first time to
do privileges checking
{quote}
I checked that. It is a pretty good hack that works :) I had to update it to
support custom authentication. The current Ranger Hive Plugin use Hadoop UGI,
which only knows Kerberos and Simple Auth.
{quote}2.2 kyuubi hacks spark and turn off that classloader.
{quote}
I went through your documentation, it seems you have added a lot of good
features. Currently, kyuubi is a custom build. Is it possible to integrate your
extensions as an addon to existing deployment? In this way, users can deploy
the default Thrift Server, but using some properties or code injections adds
your feature? We might then able to support Livy also with the same code base.
{quote}3. spark-authorizer reuses the ranger hive plugin(0.5）which contains
incompatible jersey dependencies with spark ones.
{quote}
There are few limitations with Ranger 0.5, most notably it doesn't support Tag
Based policies. I was thinking, we should just implement first class plugin for
SparkSQL using Ranger 0.7 or 1.0. It could use the same Hive
ServiceDef/Policies, but native implementation for SparkSQL. In this way, we
don't have to be dependent with Hive libraries and it's limitation.
{quote}And what are the steps I should follow to contribute Ranger?
{quote}
I have added you as a contributor to Ranger. You should be able to assign Jira
to yourself and create new ones. I was thinking of splitting the work among
those interested. Since you are familiar with the Spark code, do you want to
look into the new extensions and see how we can implement basic authorization
and advanced features like dynamic masking/UDF and Row Level filtering? I can
look into Tag based policies and also see if I can extract your current Spark
Authorizer feature into native SparkSQL Ranger Plugin.
Give me your thoughts and suggestions.
Thanks
> Implement SparkSQL plugin
> -
>
> Key: RANGER-2128
> URL: https://issues.apache.org/jira/browse/RANGER-2128
> Project: Ranger
> Issue Type: New Feature
> Components: plugins, Ranger
>Affects Versions: 1.1.0
>Reporter: t oo
>Priority: Major
> Fix For: 1.1.0
>
>
> Implement SparkSQL plugin
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: Review Request 67697: RANGER-2140: Upgrade spring and guava libraries
> On June 22, 2018, 6:51 p.m., Velmurugan Periasamy wrote: > > reverted the changes of Guava library and test cases are passing now. we can't use the latest guava library as few third party libraries are still dependent on older version of guava library. - Pradeep --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/67697/#review205245 --- On June 25, 2018, 10:26 a.m., Pradeep Agrawal wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/67697/ > --- > > (Updated June 25, 2018, 10:26 a.m.) > > > Review request for ranger, Ankita Sinha, deepak sharma, Gautam Borad, Mehul > Parikh, suja s, and Velmurugan Periasamy. > > > Bugs: RANGER-2140 > https://issues.apache.org/jira/browse/RANGER-2140 > > > Repository: ranger > > > Description > --- > > Here I am proposing to change Guava version to only 17.0, currently there are > three different version are being used. Also proposing to change Spring > Security to 4.2.7 and Spring Framework to 4.3.18. > > > Diffs > - > > kms/pom.xml 9ca9270 > plugin-atlas/pom.xml a1f0c37 > pom.xml 0795210 > src/main/assembly/kms.xml 7fbc141 > src/main/assembly/plugin-atlas.xml 8f4a64c > src/main/assembly/plugin-kms.xml 6d15f2a > src/main/assembly/plugin-solr.xml de30bfb > ugsync/pom.xml c636f9f > unixauthclient/pom.xml f859fff > unixauthservice/pom.xml c4fe07d > > > Diff: https://reviews.apache.org/r/67697/diff/2/ > > > Testing > --- > > Tested Ranger admin installation, user login, usersync and other crud > operations on service, policy, user and group module. > > > Thanks, > > Pradeep Agrawal > >
Re: Review Request 67697: RANGER-2140: Upgrade spring and guava libraries
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/67697/ --- (Updated June 25, 2018, 10:26 a.m.) Review request for ranger, Ankita Sinha, deepak sharma, Gautam Borad, Mehul Parikh, suja s, and Velmurugan Periasamy. Changes --- roledback changes related to use of latest Guava lib as few third parties libraries are having dependency on older version of guava library. Bugs: RANGER-2140 https://issues.apache.org/jira/browse/RANGER-2140 Repository: ranger Description (updated) --- Here I am proposing to change Guava version to only 17.0, currently there are three different version are being used. Also proposing to change Spring Security to 4.2.7 and Spring Framework to 4.3.18. Diffs (updated) - kms/pom.xml 9ca9270 plugin-atlas/pom.xml a1f0c37 pom.xml 0795210 src/main/assembly/kms.xml 7fbc141 src/main/assembly/plugin-atlas.xml 8f4a64c src/main/assembly/plugin-kms.xml 6d15f2a src/main/assembly/plugin-solr.xml de30bfb ugsync/pom.xml c636f9f unixauthclient/pom.xml f859fff unixauthservice/pom.xml c4fe07d Diff: https://reviews.apache.org/r/67697/diff/2/ Changes: https://reviews.apache.org/r/67697/diff/1-2/ Testing --- Tested Ranger admin installation, user login, usersync and other crud operations on service, policy, user and group module. Thanks, Pradeep Agrawal
Re: Ranger 1.1 release
+1. Colm. On Sun, Jun 24, 2018 at 9:18 PM, Zsombor Gegesy wrote: > +1 - I believe in the "release early release often" mantra :) > > > Regards, > Zsombor > > On Sun, Jun 24, 2018 at 5:56 PM, Srikanth Venkat > wrote: > > > +1, it would be nice to have a Ranger release baseline that aligns with > > Atlas 1.0. > > Thanks Vel for taking the initiative! > > > > Thx > > Srikanth Venkat > > sven...@hortonworks.com > > > > On 6/22/18, 10:24 PM, "Balaji Ganesan" > > wrote: > > > > +1. Vel, thanks for taking the initiative. > > > > On Fri, Jun 22, 2018 at 7:51 PM Jianhua Peng > > > wrote: > > > > > +1 > > > > > > Thank you, > > > Jianhua Peng. > > > > > > On 2018/06/22 16:25:45, Velmurugan Periasamy > wrote: > > > > Rangers: > > > > > > > > Now that support for Atlas 1.0 is added ( > > > https://issues.apache.org/jira/browse/RANGER-2136 < > > > https://issues.apache.org/jira/browse/RANGER-2136>), I propose to > > release > > > Ranger 1.1 (tentative first week of July). > > > > > > > > I request the community to resolve open JIRA’s marked for 1.1 in > > the > > > next couple of weeks or move them to next release. My proposal is > to > > call > > > next release 2.0 and update master to 2.0.0-SNAPSHOT. > > > > > > > > Thank you, > > > > Vel > > > > > > > > > > -- Colm O hEigeartaigh Talend Community Coder http://coders.talend.com
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16522003#comment-16522003 ] Kent Yao commented on RANGER-2128: -- [~bosco] I had a quick look on SPARK-18127 and its related PR. It has exposed Parser/Analyzer/Optimizer/Planner, which is so great for all users. It also makes it easier for users to call our plug-in. Some existing issues may be considered before we start 1. spark-authorizer is designed as a Optimize Rule for Spark SQL and executed after all other default rules because rules, such as column pruning, projection push down, and so on, should be operated first. 2. spark-authorizer has to visit hive SessionState object which is not accessible for spark context classloader because Spark use a isolated classloader to load hive client jars. 2.1 spark-authorizer itself will rewrite SessionState object the first time to do privileges checking 2.2 kyuubi hacks spark and turn off that classloader. 3. spark-authorizer reuses the ranger hive plugin(0.5）which contains incompatible jersey dependencies with spark ones. And what are the steps I should follow to contribute Ranger? Thanks > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Priority: Major > Fix For: 1.1.0 > > > Implement SparkSQL plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16521958#comment-16521958 ] Don Bosco Durai commented on RANGER-2128: - [~Qin Yao] thanks for helping us out. When I went through your spark-authorizer code, you had mapped all Spark actions to Hive actions. It was pretty impressive. Thanks for pointing out (offline) to me your work on [https://github.com/yaooqinn/kyuubi.] It seems to be what everyone wants. Since you are familiar with the Spark integrations and challenges, what is your recommendation? Can we work on a high-level design flow? E.g. Can we leverage the new Spark hook to implement some of the plugin interactions? Thanks > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Priority: Major > Fix For: 1.1.0 > > > Implement SparkSQL plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16521944#comment-16521944 ] Kent Yao commented on RANGER-2128: -- Thanks for your attention to spark-authorizer and I am willing to help > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Priority: Major > Fix For: 1.1.0 > > > Implement SparkSQL plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
