[jira] [Commented] (RANGER-2551) Optimize obtaining the agentHostname of audit log
[
https://issues.apache.org/jira/browse/RANGER-2551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918189#comment-16918189
]
Don Bosco Durai commented on RANGER-2551:
-
[~Seymour Xu] which Ranger version are you using? The stack trace in the JIRA
is not matching the code. In the last MiscUtil.getHostname() we are already
taking care of caching the hostname.
{code:java}
public static String getHostname() {
String ret = local_hostname;
if (ret == null) {
initLocalHost();
ret = local_hostname;
if (ret == null) {
ret = "unknown";
}
}
return ret;
}
{code}
> Optimize obtaining the agentHostname of audit log
> --
>
> Key: RANGER-2551
> URL: https://issues.apache.org/jira/browse/RANGER-2551
> Project: Ranger
> Issue Type: Improvement
> Components: plugins, Ranger
>Affects Versions: 1.1.0, 2.0.0, 1.2.0
>Reporter: Haihui Xu
>Assignee: Haihui Xu
>Priority: Major
> Labels: patch
> Fix For: 2.1.0
>
> Attachments: RANGER-2551.patch
>
>
> Firstly, kafka enable ranger-kafka-plugin,. The ranger audit log event
> happens very frequently when the kafka client access kafka broker, thus
> affectting kafka performance. The jstack of kakfa broker pid logs is:
> "kafka-request-handler-23" #101 daemon prio=5 os_prio=0
> tid=0x7fbcce7d5000 nid=0x3331a runnable [0x7fb356e3000]
> java.lang.Thread.State: RUNNABLE
> at java.net.Inet4AddressImpl.getLocalHostName(Native Method)
> at java.net.InetAddress.getLocalHost(InetAddress.java:1474)
> at
> org.apache.ranger.audit.provider.MiscUtil.getHostname(MiscUtil.java 166)
> at
> org.apache.ranger.plugin.audit.RangerDefaultAuditHandler.populateDefaults(RangerDefaultAuditHandler.java:198)
> at
> org.apache.ranger.plugin.audit.RangerDefaultAuditHandler.getAuthezEvents(RangerDefaultAuditHandler.java:132)
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (RANGER-2551) Optimize obtaining the agentHostname of audit log
[ https://issues.apache.org/jira/browse/RANGER-2551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915597#comment-16915597 ] Don Bosco Durai commented on RANGER-2551: - [~Seymour Xu], the patch looks good. Can you create a Review Board review? Thanks > Optimize obtaining the agentHostname of audit log > -- > > Key: RANGER-2551 > URL: https://issues.apache.org/jira/browse/RANGER-2551 > Project: Ranger > Issue Type: Improvement > Components: plugins, Ranger >Affects Versions: 1.1.0, 2.0.0, 1.2.0 >Reporter: Haihui Xu >Assignee: Haihui Xu >Priority: Major > Labels: patch > Fix For: 2.1.0 > > Attachments: RANGER-2551.patch > > > Firstly, kafka enable ranger-kafka-plugin,. The ranger audit log event > happens very frequently when the kafka client access kafka broker, thus > affectting kafka performance. The jstack of kakfa broker pid logs is: > "kafka-request-handler-23" #101 daemon prio=5 os_prio=0 > tid=0x7fbcce7d5000 nid=0x3331a runnable [0x7fb356e3000] > java.lang.Thread.State: RUNNABLE > at java.net.Inet4AddressImpl.getLocalHostName(Native Method) > at java.net.InetAddress.getLocalHost(InetAddress.java:1474) > at > org.apache.ranger.audit.provider.MiscUtil.getHostname(MiscUtil.java 166) > at > org.apache.ranger.plugin.audit.RangerDefaultAuditHandler.populateDefaults(RangerDefaultAuditHandler.java:198) > at > org.apache.ranger.plugin.audit.RangerDefaultAuditHandler.getAuthezEvents(RangerDefaultAuditHandler.java:132) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (RANGER-2541) Upgrade Ranger to support Elasticsearch 7.x
[ https://issues.apache.org/jira/browse/RANGER-2541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16912026#comment-16912026 ] Don Bosco Durai commented on RANGER-2541: - Should we consider supporting Elastic Search from AWS also? > Upgrade Ranger to support Elasticsearch 7.x > --- > > Key: RANGER-2541 > URL: https://issues.apache.org/jira/browse/RANGER-2541 > Project: Ranger > Issue Type: New Feature > Components: plugins >Reporter: Qiang Zhang >Assignee: Qiang Zhang >Priority: Major > > We should upgrade Elasticsearch to a more recent release. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (RANGER-2494) Ranger Custom PolicyCondition for TagsNotPresent and AnyTagPresent
[ https://issues.apache.org/jira/browse/RANGER-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16875901#comment-16875901 ] Don Bosco Durai commented on RANGER-2494: - [~rmani] thanks for the clarification. These are good enhancements. Thanks > Ranger Custom PolicyCondition for TagsNotPresent and AnyTagPresent > --- > > Key: RANGER-2494 > URL: https://issues.apache.org/jira/browse/RANGER-2494 > Project: Ranger > Issue Type: Bug > Components: Ranger >Affects Versions: master, 2.0.0 >Reporter: Ramesh Mani >Assignee: Ramesh Mani >Priority: Major > Fix For: 2.0.0 > > Attachments: > 0001-RANGER-2494-Ranger-Custom-PolicyCondition-for-TagNot.patch > > > Ranger Custom PolicyCondition for TagsNotPresent and AnyTagPresent. > - Two new Custom Policy Conditions are to be created. > 1) RangerTagsAnyPresentConditionEvaluator - This condition evaluates to > "true" when "Any of the Policy condition Tag" defined is present in the > tags associated to the Resource. > 2) RangerTagsNotPresentConditionEvaluator - This condition evaluates to > "true" when "None of the Policy condition Tag" defined is present in the tags > associated to the resource. > These new custom policy conditions are in addition to the one created in > https://issues.apache.org/jira/browse/RANGER-2465 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2494) Ranger Custom PolicyCondition for TagNotPresent and AnyTagPresent
[ https://issues.apache.org/jira/browse/RANGER-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16875897#comment-16875897 ] Don Bosco Durai commented on RANGER-2494: - [~rmani] can you give additional information on this JIRA? Thanks > Ranger Custom PolicyCondition for TagNotPresent and AnyTagPresent > -- > > Key: RANGER-2494 > URL: https://issues.apache.org/jira/browse/RANGER-2494 > Project: Ranger > Issue Type: Bug > Components: Ranger >Affects Versions: master, 2.0.0 >Reporter: Ramesh Mani >Assignee: Ramesh Mani >Priority: Major > Fix For: 2.0.0 > > Attachments: > 0001-RANGER-2494-Ranger-Custom-PolicyCondition-for-TagNot.patch > > > Ranger Custom PolicyCondition for TagNotPresent and AnyTagPresent -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (RANGER-2471) Service Def only support 127 permissions
Don Bosco Durai created RANGER-2471: --- Summary: Service Def only support 127 permissions Key: RANGER-2471 URL: https://issues.apache.org/jira/browse/RANGER-2471 Project: Ranger Issue Type: Bug Components: Ranger Reporter: Don Bosco Durai Service Def contains the list of permissions it supports, e.g. Select, Insert in Hive. The DB field we are using storing the sort order (x_access_type_def::sort_order) is currently tinyint(3), which limits the number of permissions supported to around 128. While this works for most of the service definitions, but it puts a limitation for the common service def used by "Tags", which is generally a cumulative of all the permissions. We need to change the column to a DB type which can support more values. e.g. smallint -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (RANGER-2470) Support for Okta Authentication for Ranger Admin Consol
Don Bosco Durai created RANGER-2470: --- Summary: Support for Okta Authentication for Ranger Admin Consol Key: RANGER-2470 URL: https://issues.apache.org/jira/browse/RANGER-2470 Project: Ranger Issue Type: Improvement Components: Ranger Reporter: Don Bosco Durai Currently we support SSO via Knox. It might be good to also support SSO without Knox dependency. I have created this JIRA with Okta in mind, but from the design point of view, we should plan to support SAML and possibly OAuth/OpenId -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2461) ranger-kafka-plugin find resource bug
[
https://issues.apache.org/jira/browse/RANGER-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16864379#comment-16864379
]
Don Bosco Durai commented on RANGER-2461:
-
Glad your classpath issue got resolved.
1. Regarding Kerberos, just want to make sure that both Ranger and Kafka is
Kerberos enabled
2. Can I assume, without Ranger you werre able to connect to Kafka Broker
**only** after doing kinit and were able to publish/consume
> ranger-kafka-plugin find resource bug
> -
>
> Key: RANGER-2461
> URL: https://issues.apache.org/jira/browse/RANGER-2461
> Project: Ranger
> Issue Type: Bug
> Components: plugins
>Affects Versions: 1.2.0
>Reporter: Konstantin Tsypin
>Priority: Major
>
> Hi there
> Past several days i tried to implement integration apache ranger and
> ranger-kafka plugin.
> *My versions:*
> kafka: 2.12_2.2.0
> ranger: 1.2.0 , ranger-plugin 1.2.0
> *My current problem:*
> as far as i understand, kafka cant find the ranger-created audit files that i
> made with enable-kafka-plugin.sh script (ranger-kafka-audit.xml and
> ranger-kafka-security.xml). I sold this files everywhere in file system but
> this class still cant find it and cause fatal error:
>
> [2019-06-06 13:22:44,584] INFO
> getFilesInDirectory('/usr/lib/kafka/libs/ranger-kafka-plugin-impl'): adding
> /usr/lib/kafka/libs/ranger-kafka-plugin-impl/guava-17.0.jar
> (org.apache.ranger.plugin.classloader.RangerPluginClassLoaderUtil)
> [2019-06-06 13:22:44,584] INFO
> getFilesInDirectory('/usr/lib/kafka/libs/ranger-kafka-plugin-impl'): adding
> /usr/lib/kafka/libs/ranger-kafka-plugin-impl/ranger-plugins-common-1.2.0.jar
> (org.apache.ranger.plugin.classloader.RangerPluginClassLoaderUtil)
> [2019-06-06 13:22:44,615] INFO [Transaction Marker Channel Manager 0]:
> Starting (kafka.coordinator.transaction.TransactionMarkerChannelManager)
> [2019-06-06 13:22:44,648] ERROR Error getting principal.
> (org.apache.ranger.authorization.kafka.authorizer.RangerKafkaAuthorizer)
> java.lang.NoSuchMethodError:
> org.apache.kafka.common.security.JaasContext.load(Lorg/apache/kafka/common/security/JaasContext$Type;Lorg/apache/kafka/common/network/ListenerName;Ljava/util/Map;)Lorg/apache/kafka/common/security/JaasContext;
> at
> org.apache.ranger.authorization.kafka.authorizer.RangerKafkaAuthorizer.configure(RangerKafkaAuthorizer.java:98)
> at
> org.apache.ranger.authorization.kafka.authorizer.RangerKafkaAuthorizer.configure(RangerKafkaAuthorizer.java:94)
> at kafka.server.KafkaServer.$anonfun$startup$5(KafkaServer.scala:288)
> at scala.Option.map(Option.scala:163)
> at kafka.server.KafkaServer.startup(KafkaServer.scala:286)
> at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38)
> at kafka.Kafka$.main(Kafka.scala:75)
> at kafka.Kafka.main(Kafka.scala)
> [2019-06-06 13:22:44,689] INFO Calling plugin.init()
> (org.apache.ranger.authorization.kafka.authorizer.RangerKafkaAuthorizer)
> [2019-06-06 13:22:44,733] ERROR
> addResourceIfReadable(ranger-kafka-audit.xml): couldn't find resource file
> location (org.apache.ranger.authorization.hadoop.config.RangerConfiguration)
> [2019-06-06 13:22:44,734] ERROR
> addResourceIfReadable(ranger-kafka-security.xml): couldn't find resource file
> location (org.apache.ranger.authorization.hadoop.config.RangerConfiguration)
> [2019-06-06 13:22:44,758] INFO AuditProviderFactory: creating..
> (org.apache.ranger.audit.provider.AuditProviderFactory)
> [2019-06-06 13:22:44,758] INFO AuditProviderFactory: initializing..
> (org.apache.ranger.audit.provider.AuditProviderFactory)
> [2019-06-06 13:22:44,900] INFO No v3 audit configuration found. Trying v2
> audit configurations (org.apache.ranger.audit.provider.AuditProviderFactory)
> [2019-06-06 13:22:44,900] INFO AuditProviderFactory: Audit not enabled..
> (org.apache.ranger.audit.provider.AuditProviderFactory)
> [2019-06-06 13:22:44,900] INFO PolicyEngineOptions: \{ evaluatorType: auto,
> cacheAuditResult: false, disableContextEnrichers: false,
> disableCustomConditions: false, disableTrieLookupPrefilter: false,
> optimizeTrieForRetrieval: false }
> (org.apache.ranger.plugin.service.RangerBasePlugin)
> [2019-06-06 13:22:44,917] ERROR [KafkaServer id=0] Fatal error during
> KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
> java.lang.NullPointerException
>
> [root@myhost kafka]# find / -name ranger-kafka-security.xml | xargs ls -lah {}
> ls: cannot access {}: No such file or directory
> -rwxr--r-- 1 root root 2.9K Jun 5 20:41
> /opt/kafka/conf/ranger-kafka-security.xml
> -rwxr--r-- 1 root root 2.9K Jun 5 20:41 /opt/kafka/ranger-kafka-security.xml
> -rwxr-xr-x 1 kafka kafka 2.6K Jun 6 12:58 /ranger-kafka-security.xml
> -rwx-- 1 cloud-user root 2.6K Sep 27 2018
>
[jira] [Commented] (RANGER-2461) ranger-kafka-plugin find resource bug
[
https://issues.apache.org/jira/browse/RANGER-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16861915#comment-16861915
]
Don Bosco Durai commented on RANGER-2461:
-
I have to look into this more carefully. I am not sure whether you updated the
classpath
For now try this:
#bin/kafka-server-start.sh
#Add the below lines before exec $base_dir/kafka-run-class.sh $EXTRA_ARGS
kafka.Kafka "$@"
classpathmunge /home/ec2-user/kafka_2.12-2.2.0/config
#classpathmunge '/usr/hdp/current/hadoop-hdfs-client/*'
#classpathmunge '/usr/hdp/current/hadoop-hdfs-client/lib/*'
#classpathmunge '/etc/hadoop/conf'
export CLASSPATH
unset classpathmunge
#Update hdfs jar path and conf as needed.
> ranger-kafka-plugin find resource bug
> -
>
> Key: RANGER-2461
> URL: https://issues.apache.org/jira/browse/RANGER-2461
> Project: Ranger
> Issue Type: Bug
> Components: plugins
>Affects Versions: 1.2.0
>Reporter: Konstantin Tsypin
>Priority: Major
>
> Hi there
> Past several days i tried to implement integration apache ranger and
> ranger-kafka plugin.
> *My versions:*
> kafka: 2.12_2.2.0
> ranger: 1.2.0 , ranger-plugin 1.2.0
> *My current problem:*
> as far as i understand, kafka cant find the ranger-created audit files that i
> made with enable-kafka-plugin.sh script (ranger-kafka-audit.xml and
> ranger-kafka-security.xml). I sold this files everywhere in file system but
> this class still cant find it and cause fatal error:
>
> [2019-06-06 13:22:44,584] INFO
> getFilesInDirectory('/usr/lib/kafka/libs/ranger-kafka-plugin-impl'): adding
> /usr/lib/kafka/libs/ranger-kafka-plugin-impl/guava-17.0.jar
> (org.apache.ranger.plugin.classloader.RangerPluginClassLoaderUtil)
> [2019-06-06 13:22:44,584] INFO
> getFilesInDirectory('/usr/lib/kafka/libs/ranger-kafka-plugin-impl'): adding
> /usr/lib/kafka/libs/ranger-kafka-plugin-impl/ranger-plugins-common-1.2.0.jar
> (org.apache.ranger.plugin.classloader.RangerPluginClassLoaderUtil)
> [2019-06-06 13:22:44,615] INFO [Transaction Marker Channel Manager 0]:
> Starting (kafka.coordinator.transaction.TransactionMarkerChannelManager)
> [2019-06-06 13:22:44,648] ERROR Error getting principal.
> (org.apache.ranger.authorization.kafka.authorizer.RangerKafkaAuthorizer)
> java.lang.NoSuchMethodError:
> org.apache.kafka.common.security.JaasContext.load(Lorg/apache/kafka/common/security/JaasContext$Type;Lorg/apache/kafka/common/network/ListenerName;Ljava/util/Map;)Lorg/apache/kafka/common/security/JaasContext;
> at
> org.apache.ranger.authorization.kafka.authorizer.RangerKafkaAuthorizer.configure(RangerKafkaAuthorizer.java:98)
> at
> org.apache.ranger.authorization.kafka.authorizer.RangerKafkaAuthorizer.configure(RangerKafkaAuthorizer.java:94)
> at kafka.server.KafkaServer.$anonfun$startup$5(KafkaServer.scala:288)
> at scala.Option.map(Option.scala:163)
> at kafka.server.KafkaServer.startup(KafkaServer.scala:286)
> at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38)
> at kafka.Kafka$.main(Kafka.scala:75)
> at kafka.Kafka.main(Kafka.scala)
> [2019-06-06 13:22:44,689] INFO Calling plugin.init()
> (org.apache.ranger.authorization.kafka.authorizer.RangerKafkaAuthorizer)
> [2019-06-06 13:22:44,733] ERROR
> addResourceIfReadable(ranger-kafka-audit.xml): couldn't find resource file
> location (org.apache.ranger.authorization.hadoop.config.RangerConfiguration)
> [2019-06-06 13:22:44,734] ERROR
> addResourceIfReadable(ranger-kafka-security.xml): couldn't find resource file
> location (org.apache.ranger.authorization.hadoop.config.RangerConfiguration)
> [2019-06-06 13:22:44,758] INFO AuditProviderFactory: creating..
> (org.apache.ranger.audit.provider.AuditProviderFactory)
> [2019-06-06 13:22:44,758] INFO AuditProviderFactory: initializing..
> (org.apache.ranger.audit.provider.AuditProviderFactory)
> [2019-06-06 13:22:44,900] INFO No v3 audit configuration found. Trying v2
> audit configurations (org.apache.ranger.audit.provider.AuditProviderFactory)
> [2019-06-06 13:22:44,900] INFO AuditProviderFactory: Audit not enabled..
> (org.apache.ranger.audit.provider.AuditProviderFactory)
> [2019-06-06 13:22:44,900] INFO PolicyEngineOptions: \{ evaluatorType: auto,
> cacheAuditResult: false, disableContextEnrichers: false,
> disableCustomConditions: false, disableTrieLookupPrefilter: false,
> optimizeTrieForRetrieval: false }
> (org.apache.ranger.plugin.service.RangerBasePlugin)
> [2019-06-06 13:22:44,917] ERROR [KafkaServer id=0] Fatal error during
> KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
> java.lang.NullPointerException
>
> [root@myhost
[jira] [Commented] (RANGER-2395) Add presto plugin
[ https://issues.apache.org/jira/browse/RANGER-2395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16843299#comment-16843299 ] Don Bosco Durai commented on RANGER-2395: - [~bolke] I have given you the permission to the Ranger Wiki > Add presto plugin > - > > Key: RANGER-2395 > URL: https://issues.apache.org/jira/browse/RANGER-2395 > Project: Ranger > Issue Type: Improvement > Components: plugins >Reporter: Bolke de Bruin >Assignee: Bolke de Bruin >Priority: Major > Fix For: master > > Attachments: 0001-Add-Presto-plugin.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Presto (or PrestoDB) is an open source, distributed SQL query engine, > designed from the ground up for fast analytic queries against data of any > size. It supports both non-relational sources, such as the Hadoop Distributed > File System (HDFS), [Amazon S3|https://aws.amazon.com/s3/], Cassandra, > MongoDB, and [HBase|https://aws.amazon.com/emr/details/hbase/], and > relational data sources such as MySQL, PostgreSQL, [Amazon > Redshift|https://aws.amazon.com/redshift/], Microsoft SQL Server, and > Teradata. > This is to track a Ranger plugin for Presto -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2410) Build Ranger Using Docker
[ https://issues.apache.org/jira/browse/RANGER-2410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16827778#comment-16827778 ] Don Bosco Durai commented on RANGER-2410: - One time to rebuild your image and output: ./build_ranger_using_docker.sh -build_image mvn clean Subsequently: #For full build ./build_ranger_using_docker.sh #For selective build ./build_ranger_using_docker.sh mvn > Build Ranger Using Docker > - > > Key: RANGER-2410 > URL: https://issues.apache.org/jira/browse/RANGER-2410 > Project: Ranger > Issue Type: Bug > Components: Ranger >Affects Versions: 1.2.0 >Reporter: Helene Treadwell >Assignee: Don Bosco Durai >Priority: Minor > Attachments: build_ranger_using_docker.sh > > > build_ranger_user_docker.sh script throws an error due to dependency on gosu > tool > > gpg: directory `/root/.gnupg' created > gpg: new configuration file `/root/.gnupg/gpg.conf' created > gpg: WARNING: options in `/root/.gnupg/gpg.conf' are not yet active during > this run > gpg: keyring `/root/.gnupg/secring.gpg' created > gpg: keyring `/root/.gnupg/pubring.gpg' created > gpg: requesting key BF357DD4 from hkp server pool.sks-keyservers.net > gpgkeys: HTTP fetch error 7: Failed to connect to ::::c62e:cb61: > Cannot assign requested address > gpg: no valid OpenPGP data found. > gpg: Total number processed: 0 > The command '/bin/sh -c gpg --keyserver pool.sks-keyservers.net --recv-keys > B42F6819007F00F88E364FD4036A9C25BF357DD4 && curl -o /usr/local/bin/gosu > -SL "[https://github.com/tianon/gosu/releases/download/1.10/gosu-amd64" > |https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_tianon_gosu_releases_download_1.10_gosu-2Damd64-2522=DwQFaQ=7DfhQjPWzR3PmWBQVpi-kw=kcVpZ83Oz_eaC5ai7r0u5Lr9tm-XLxYP8p3M7dVqMRE=bcXk0KhABoVxKnrS1FIsgI8BKa-U4FvSrHJnzA4SFDg=RYGGYQHlDvfGkdqLPgG44BcFNK5vslmXllryWfj6LIA=] > && curl -o /usr/local/bin/gosu.asc -SL > "[https://github.com/tianon/gosu/releases/download/1.10/gosu-amd64.asc" > |https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_tianon_gosu_releases_download_1.10_gosu-2Damd64.asc-2522=DwQFaQ=7DfhQjPWzR3PmWBQVpi-kw=kcVpZ83Oz_eaC5ai7r0u5Lr9tm-XLxYP8p3M7dVqMRE=bcXk0KhABoVxKnrS1FIsgI8BKa-U4FvSrHJnzA4SFDg=uZbaTTs55_-ksH4_AXsLglep92TP23tQ4gnu8luy8DE=] > && gpg --verify /usr/local/bin/gosu.asc && rm /usr/local/bin/gosu.asc > && rm -r /root/.gnupg/ && chmod +x /usr/local/bin/gosu' returned a > non-zero code: 2 > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (RANGER-2410) Build Ranger Using Docker
[ https://issues.apache.org/jira/browse/RANGER-2410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Don Bosco Durai updated RANGER-2410: Attachment: build_ranger_using_docker.sh > Build Ranger Using Docker > - > > Key: RANGER-2410 > URL: https://issues.apache.org/jira/browse/RANGER-2410 > Project: Ranger > Issue Type: Bug > Components: Ranger >Affects Versions: 1.2.0 >Reporter: Helene Treadwell >Assignee: Don Bosco Durai >Priority: Minor > Attachments: build_ranger_using_docker.sh > > > build_ranger_user_docker.sh script throws an error due to dependency on gosu > tool > > gpg: directory `/root/.gnupg' created > gpg: new configuration file `/root/.gnupg/gpg.conf' created > gpg: WARNING: options in `/root/.gnupg/gpg.conf' are not yet active during > this run > gpg: keyring `/root/.gnupg/secring.gpg' created > gpg: keyring `/root/.gnupg/pubring.gpg' created > gpg: requesting key BF357DD4 from hkp server pool.sks-keyservers.net > gpgkeys: HTTP fetch error 7: Failed to connect to ::::c62e:cb61: > Cannot assign requested address > gpg: no valid OpenPGP data found. > gpg: Total number processed: 0 > The command '/bin/sh -c gpg --keyserver pool.sks-keyservers.net --recv-keys > B42F6819007F00F88E364FD4036A9C25BF357DD4 && curl -o /usr/local/bin/gosu > -SL "[https://github.com/tianon/gosu/releases/download/1.10/gosu-amd64" > |https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_tianon_gosu_releases_download_1.10_gosu-2Damd64-2522=DwQFaQ=7DfhQjPWzR3PmWBQVpi-kw=kcVpZ83Oz_eaC5ai7r0u5Lr9tm-XLxYP8p3M7dVqMRE=bcXk0KhABoVxKnrS1FIsgI8BKa-U4FvSrHJnzA4SFDg=RYGGYQHlDvfGkdqLPgG44BcFNK5vslmXllryWfj6LIA=] > && curl -o /usr/local/bin/gosu.asc -SL > "[https://github.com/tianon/gosu/releases/download/1.10/gosu-amd64.asc" > |https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_tianon_gosu_releases_download_1.10_gosu-2Damd64.asc-2522=DwQFaQ=7DfhQjPWzR3PmWBQVpi-kw=kcVpZ83Oz_eaC5ai7r0u5Lr9tm-XLxYP8p3M7dVqMRE=bcXk0KhABoVxKnrS1FIsgI8BKa-U4FvSrHJnzA4SFDg=uZbaTTs55_-ksH4_AXsLglep92TP23tQ4gnu8luy8DE=] > && gpg --verify /usr/local/bin/gosu.asc && rm /usr/local/bin/gosu.asc > && rm -r /root/.gnupg/ && chmod +x /usr/local/bin/gosu' returned a > non-zero code: 2 > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (RANGER-2410) Build Ranger Using Docker
[ https://issues.apache.org/jira/browse/RANGER-2410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Don Bosco Durai reassigned RANGER-2410: --- Assignee: Don Bosco Durai > Build Ranger Using Docker > - > > Key: RANGER-2410 > URL: https://issues.apache.org/jira/browse/RANGER-2410 > Project: Ranger > Issue Type: Bug > Components: Ranger >Affects Versions: 1.2.0 >Reporter: Helene Treadwell >Assignee: Don Bosco Durai >Priority: Minor > > build_ranger_user_docker.sh script throws an error due to dependency on gosu > tool > > gpg: directory `/root/.gnupg' created > gpg: new configuration file `/root/.gnupg/gpg.conf' created > gpg: WARNING: options in `/root/.gnupg/gpg.conf' are not yet active during > this run > gpg: keyring `/root/.gnupg/secring.gpg' created > gpg: keyring `/root/.gnupg/pubring.gpg' created > gpg: requesting key BF357DD4 from hkp server pool.sks-keyservers.net > gpgkeys: HTTP fetch error 7: Failed to connect to ::::c62e:cb61: > Cannot assign requested address > gpg: no valid OpenPGP data found. > gpg: Total number processed: 0 > The command '/bin/sh -c gpg --keyserver pool.sks-keyservers.net --recv-keys > B42F6819007F00F88E364FD4036A9C25BF357DD4 && curl -o /usr/local/bin/gosu > -SL "[https://github.com/tianon/gosu/releases/download/1.10/gosu-amd64" > |https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_tianon_gosu_releases_download_1.10_gosu-2Damd64-2522=DwQFaQ=7DfhQjPWzR3PmWBQVpi-kw=kcVpZ83Oz_eaC5ai7r0u5Lr9tm-XLxYP8p3M7dVqMRE=bcXk0KhABoVxKnrS1FIsgI8BKa-U4FvSrHJnzA4SFDg=RYGGYQHlDvfGkdqLPgG44BcFNK5vslmXllryWfj6LIA=] > && curl -o /usr/local/bin/gosu.asc -SL > "[https://github.com/tianon/gosu/releases/download/1.10/gosu-amd64.asc" > |https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_tianon_gosu_releases_download_1.10_gosu-2Damd64.asc-2522=DwQFaQ=7DfhQjPWzR3PmWBQVpi-kw=kcVpZ83Oz_eaC5ai7r0u5Lr9tm-XLxYP8p3M7dVqMRE=bcXk0KhABoVxKnrS1FIsgI8BKa-U4FvSrHJnzA4SFDg=uZbaTTs55_-ksH4_AXsLglep92TP23tQ4gnu8luy8DE=] > && gpg --verify /usr/local/bin/gosu.asc && rm /usr/local/bin/gosu.asc > && rm -r /root/.gnupg/ && chmod +x /usr/local/bin/gosu' returned a > non-zero code: 2 > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2341) Support for Incremental policy updates to improve performance of ranger-admin and plugins by optimal building of policy-engine
[ https://issues.apache.org/jira/browse/RANGER-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16781211#comment-16781211 ] Don Bosco Durai commented on RANGER-2341: - [~abhayk] this would be a good feature. Few questions... > Cache management in ranger-admin is enhanced to use this table to figure out > changes using a previously known version number (provided by module > requesting updated policies). Seems more like more like redo logs in database, which I feel is a good approach. > Backward compatibility is maintained with older plugins by adding another > parameter to REST API for downloading policies. Should we do the other way? New plugins should pass the addition param, so that older plugins will work without change? > Policy deltas are disabled by default. I feel, we should enable this by default. This is a good feature and let the plugins decide whether to use or not. > Policy delta table is cleared of records older than a week on restart of > ranger-admin. I not sure whether restart should be the trigger, but might be okay for now till have an inbuilt scheduler. I assume, we will make the the retention period configurable. > Support for Incremental policy updates to improve performance of ranger-admin > and plugins by optimal building of policy-engine > -- > > Key: RANGER-2341 > URL: https://issues.apache.org/jira/browse/RANGER-2341 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Affects Versions: master >Reporter: Abhay Kulkarni >Assignee: Abhay Kulkarni >Priority: Major > Fix For: master > > > Requirements: > Currently, every change to any policy causes rebuilding of policy-engine from > scratch. There are several disadvantages: > 1. Compute time for rebuilding > 2. Large traffic from ranger-admin to each of the plugins > 3. Large demand on JVM memory system resulting in frequent garbage collection > and pauses of JVM. > It will be more optimal to communicate only the changes and apply them to > existing policy-engine. > Design notes: > Policy changes are logged into a new database table. > Cache management in ranger-admin is enhanced to use this table to figure out > changes using a previously known version number (provided by module > requesting updated policies). > Policy engine supports update operation that accepts policy-deltas and > returns a new policy engine with deltas applied. > Resource Trie structures are copied from older policy-engine selectively, and > not rebuilt from scratch. > Backward compatibility is maintained with older plugins by adding another > parameter to REST API for downloading policies. > Ranger admin as well as component plugins may be configured to optionally use > policy deltas for its internal policy-engines. Policy deltas are disabled by > default. In ranger-admin, policy-deltas are enabled in the ranger-admin by > setting configuration variable 'ranger.admin.supports.policy.deltas' to true. > In individual plugins, policy-deltas are enabled by setting configuration > variable 'ranger.plugin..policy.rest.supports.policy.deltas' to > "true". > Policy delta table is cleared of records older than a week on restart of > ranger-admin. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775777#comment-16775777 ] Don Bosco Durai commented on RANGER-2128: - Sorry, I got pulled into other things There are few feedback, let me consolidate them and give it to [~Qin Yao] > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Fix For: 2.0.0 > > Attachments: support_ranger11.tgz > > Time Spent: 10m > Remaining Estimate: 0h > > Implement SparkSQL plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2324) Bootstrapping Solr in Ranger service start-up
[ https://issues.apache.org/jira/browse/RANGER-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16750238#comment-16750238 ] Don Bosco Durai commented on RANGER-2324: - [~bpatel] can we have some more detail on what you are planning to do here? Thanks > Bootstrapping Solr in Ranger service start-up > - > > Key: RANGER-2324 > URL: https://issues.apache.org/jira/browse/RANGER-2324 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Reporter: bhavik patel >Assignee: bhavik patel >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2312) Internal users should have permission to modify their personal information in User Profile page.
[ https://issues.apache.org/jira/browse/RANGER-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16728573#comment-16728573 ] Don Bosco Durai commented on RANGER-2312: - Hi [~zhangqiang2], I don't have any specific concerns for this change request. I was just curious how you were planning to use Name and email. I have given my feedback on review board for your patch. Please review > Internal users should have permission to modify their personal information in > User Profile page. > > > Key: RANGER-2312 > URL: https://issues.apache.org/jira/browse/RANGER-2312 > Project: Ranger > Issue Type: Bug > Components: admin >Affects Versions: master >Reporter: Qiang Zhang >Assignee: Qiang Zhang >Priority: Minor > Labels: patch > Fix For: 2.0.0 > > Attachments: > 0001-RANGER-2312-Internal-users-should-have-permission-to.patch > > > Auditor role users cannot modify their personal user profile. > User role and KMSAuditor role users have the same problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2312) Users should have permission to modify their personal information in User Profile page.
[ https://issues.apache.org/jira/browse/RANGER-2312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16727949#comment-16727949 ] Don Bosco Durai commented on RANGER-2312: - Hi [~zhangqiang2] There were couple of changes from your side to User management. The users in Ranger are not the real source of truth. It is just there to make it easy to create policies and also login into Ranger. The real source is AD/LDAP. Ideally, the changes done in AD/LDAP should be sync'ed automatically into Ranger. Are you envisioning that these users or their details will be used in other ways by Ranger or dependent components? Thanks > Users should have permission to modify their personal information in User > Profile page. > --- > > Key: RANGER-2312 > URL: https://issues.apache.org/jira/browse/RANGER-2312 > Project: Ranger > Issue Type: Bug > Components: admin >Affects Versions: master >Reporter: Qiang Zhang >Assignee: Qiang Zhang >Priority: Minor > Labels: patch > Fix For: 2.0.0 > > Attachments: > 0001-RANGER-2312-Users-should-have-permission-to-modify-t.patch > > > Auditor role users cannot modify their personal user profile. > User role and KMSAuditor role users have the same problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (RANGER-2284) Unable to build image using docker
[
https://issues.apache.org/jira/browse/RANGER-2284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Don Bosco Durai closed RANGER-2284.
---
> Unable to build image using docker
> --
>
> Key: RANGER-2284
> URL: https://issues.apache.org/jira/browse/RANGER-2284
> Project: Ranger
> Issue Type: Bug
> Components: build-infra
>Affects Versions: master
>Reporter: Nelson Costa
>Assignee: Don Bosco Durai
>Priority: Major
> Attachments: build_ranger_using_docker.sh
>
>
> Running `./build_ranger_using_docker.sh -build_image` fails with error:
> {noformat}
> ---> 62b699d731cb
> Step 11/27 : ADD
> https://www.apache.org/dist/maven/maven-3/3.5.3/binaries/apache-maven-3.5.3-bin.tar.gz.sha1
> /tools
> ADD failed: failed to GET
> https://www.apache.org/dist/maven/maven-3/3.5.3/binaries/apache-maven-3.5.3-bin.tar.gz.sha1
> with status 404 Not Found:
>
> 404 Not Found
>
> Not Found
> The requested URL
> /dist/maven/maven-3/3.5.3/binaries/apache-maven-3.5.3-bin.tar.gz.sha1 was not
> found on this server.
>
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (RANGER-542) Remove MPL license text from license file
[ https://issues.apache.org/jira/browse/RANGER-542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Don Bosco Durai resolved RANGER-542. Resolution: Fixed Fixed as part of other changes > Remove MPL license text from license file > - > > Key: RANGER-542 > URL: https://issues.apache.org/jira/browse/RANGER-542 > Project: Ranger > Issue Type: Bug >Affects Versions: 0.5.0 >Reporter: Don Bosco Durai >Priority: Critical > > We need to remove MPL license text from license file. > Refer to JIRA https://issues.apache.org/jira/browse/RANGER-316 and release > feedback http://s.apache.org/NBH -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (RANGER-542) Remove MPL license text from license file
[ https://issues.apache.org/jira/browse/RANGER-542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Don Bosco Durai closed RANGER-542. -- > Remove MPL license text from license file > - > > Key: RANGER-542 > URL: https://issues.apache.org/jira/browse/RANGER-542 > Project: Ranger > Issue Type: Bug >Affects Versions: 0.5.0 >Reporter: Don Bosco Durai >Priority: Critical > > We need to remove MPL license text from license file. > Refer to JIRA https://issues.apache.org/jira/browse/RANGER-316 and release > feedback http://s.apache.org/NBH -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-1986) Hive Policies doesn't access columns with spaces
[ https://issues.apache.org/jira/browse/RANGER-1986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16690714#comment-16690714 ] Don Bosco Durai commented on RANGER-1986: - [~nitin.galave] sorry for the delay. I feel we should have consistent behavior for all resources. Even HDFS files could have spaces in them. If we all spaces, we should trim white spaces before saving to DB. > Hive Policies doesn't access columns with spaces > > > Key: RANGER-1986 > URL: https://issues.apache.org/jira/browse/RANGER-1986 > Project: Ranger > Issue Type: Bug > Components: Ranger >Reporter: Don Bosco Durai >Assignee: Nitin Galave >Priority: Major > > Hive supports columns with spaces, but Ranger doesn't. When space is given > while entering the column name in the Ranger Policy UI, it automatically > splits it into 2 words/column names. > Is it possible to make "enter" has the way to accept/finalize the column > name? Or if a single or double quote is given, then wait for another single > or double quotes or enter before accepting the column name? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2284) Unable to build image using docker
[
https://issues.apache.org/jira/browse/RANGER-2284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16687322#comment-16687322
]
Don Bosco Durai commented on RANGER-2284:
-
[~nelsonc] can you try the attached script? If it works for you, I can commit.
Thanks
> Unable to build image using docker
> --
>
> Key: RANGER-2284
> URL: https://issues.apache.org/jira/browse/RANGER-2284
> Project: Ranger
> Issue Type: Bug
> Components: build-infra
>Affects Versions: master
>Reporter: Nelson Costa
>Assignee: Don Bosco Durai
>Priority: Major
> Attachments: build_ranger_using_docker.sh
>
>
> Running `./build_ranger_using_docker.sh -build_image` fails with error:
> {noformat}
> ---> 62b699d731cb
> Step 11/27 : ADD
> https://www.apache.org/dist/maven/maven-3/3.5.3/binaries/apache-maven-3.5.3-bin.tar.gz.sha1
> /tools
> ADD failed: failed to GET
> https://www.apache.org/dist/maven/maven-3/3.5.3/binaries/apache-maven-3.5.3-bin.tar.gz.sha1
> with status 404 Not Found:
>
> 404 Not Found
>
> Not Found
> The requested URL
> /dist/maven/maven-3/3.5.3/binaries/apache-maven-3.5.3-bin.tar.gz.sha1 was not
> found on this server.
>
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (RANGER-2284) Unable to build image using docker
[
https://issues.apache.org/jira/browse/RANGER-2284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Don Bosco Durai updated RANGER-2284:
Attachment: build_ranger_using_docker.sh
> Unable to build image using docker
> --
>
> Key: RANGER-2284
> URL: https://issues.apache.org/jira/browse/RANGER-2284
> Project: Ranger
> Issue Type: Bug
> Components: build-infra
>Affects Versions: master
>Reporter: Nelson Costa
>Assignee: Don Bosco Durai
>Priority: Major
> Attachments: build_ranger_using_docker.sh
>
>
> Running `./build_ranger_using_docker.sh -build_image` fails with error:
> {noformat}
> ---> 62b699d731cb
> Step 11/27 : ADD
> https://www.apache.org/dist/maven/maven-3/3.5.3/binaries/apache-maven-3.5.3-bin.tar.gz.sha1
> /tools
> ADD failed: failed to GET
> https://www.apache.org/dist/maven/maven-3/3.5.3/binaries/apache-maven-3.5.3-bin.tar.gz.sha1
> with status 404 Not Found:
>
> 404 Not Found
>
> Not Found
> The requested URL
> /dist/maven/maven-3/3.5.3/binaries/apache-maven-3.5.3-bin.tar.gz.sha1 was not
> found on this server.
>
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (RANGER-2284) Unable to build image using docker
[
https://issues.apache.org/jira/browse/RANGER-2284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Don Bosco Durai reassigned RANGER-2284:
---
Assignee: Don Bosco Durai
> Unable to build image using docker
> --
>
> Key: RANGER-2284
> URL: https://issues.apache.org/jira/browse/RANGER-2284
> Project: Ranger
> Issue Type: Bug
> Components: build-infra
>Affects Versions: master
>Reporter: Nelson Costa
>Assignee: Don Bosco Durai
>Priority: Major
>
> Running `./build_ranger_using_docker.sh -build_image` fails with error:
> {noformat}
> ---> 62b699d731cb
> Step 11/27 : ADD
> https://www.apache.org/dist/maven/maven-3/3.5.3/binaries/apache-maven-3.5.3-bin.tar.gz.sha1
> /tools
> ADD failed: failed to GET
> https://www.apache.org/dist/maven/maven-3/3.5.3/binaries/apache-maven-3.5.3-bin.tar.gz.sha1
> with status 404 Not Found:
>
> 404 Not Found
>
> Not Found
> The requested URL
> /dist/maven/maven-3/3.5.3/binaries/apache-maven-3.5.3-bin.tar.gz.sha1 was not
> found on this server.
>
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RANGER-2232) Security Zones feature in Apache Ranger
[ https://issues.apache.org/jira/browse/RANGER-2232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683145#comment-16683145 ] Don Bosco Durai commented on RANGER-2232: - [~abhayk] This is a very good feature and you have articulated it very well in the document. +1 from my side. I do have a suggestion, while we are implementing this feature, let's not make it very much tied to resource policies. We should be able to use security zones for other purposes also. E.g. In the future, we should be able to implement other features like https://issues.apache.org/jira/browse/RANGER-693 using security zones. > Security Zones feature in Apache Ranger > --- > > Key: RANGER-2232 > URL: https://issues.apache.org/jira/browse/RANGER-2232 > Project: Ranger > Issue Type: New Feature > Components: admin >Reporter: Madhan Neethiraj >Assignee: Abhay Kulkarni >Priority: Major > Attachments: Apache Ranger - Security Zones.pdf > > > This is to introduce a new abstraction in Apache Ranger that would allow > carving/bucketing of resources in a service into multiple zones, for better > administration of security policies. This would enable multiple > administrators to setup security policies for a service – based on the zones > to which they have been granted administration rights. > For example, let us consider 2 security zones ‘finance’ and ‘sales’: > - Security zone ‘finance’ includes all contents in Hive database named > ‘finance’ > - Security zone ‘sales’ includes all contents in ‘sales’ database > - Set of users and groups are designated as administrators each zone > - Users are allowed to setup policies only in zones in which they are > administrators > - Policies defined in a zone are applicable only for resources of the zone > - A zone can be extended to include resource from multiple services like > HDFS, Hive, HBase, Kafka, .., allowing administrators of a zone to setup > policies for resources owned by their organization across multiple services. > - Audit logs will include name of the zone in which the accessed resource > resides. Only users having appropriate permissions on the security zone can > view its audit logs. > Attached document has more details on various aspects of Security Zones. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16682739#comment-16682739 ] Don Bosco Durai commented on RANGER-2128: - [~Qin Yao] thanks. Let me coordinate with you on your changes. Since I was blocked, I ended taking out all references to Hive Context, which I feel is a cleaner option. We should plan to merge both our codes. > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Fix For: 2.0.0 > > Attachments: support_ranger11.tgz > > > Implement SparkSQL plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2281) Support Trusted Proxy in ranger
[ https://issues.apache.org/jira/browse/RANGER-2281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16682086#comment-16682086 ] Don Bosco Durai commented on RANGER-2281: - [~spolavarapu] can you look into this JIRA also? https://issues.apache.org/jira/browse/RANGER-2049 I had created this with the intention that RangerAdmin will support doAs. If you feel they are the same, then we can close one of them. Thanks > Support Trusted Proxy in ranger > --- > > Key: RANGER-2281 > URL: https://issues.apache.org/jira/browse/RANGER-2281 > Project: Ranger > Issue Type: Bug > Components: Ranger >Affects Versions: 2.0.0 >Reporter: Sailaja Polavarapu >Assignee: Sailaja Polavarapu >Priority: Major > Fix For: 2.0.0 > > > Ranger kerberos authentication module should support the notion of > proxy-user, who would be allowed to perform operations on behalf of other > users i.e. impersonate other users - similar to Hadoop. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (RANGER-698) Ranger policy should support variables like $user
[
https://issues.apache.org/jira/browse/RANGER-698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Don Bosco Durai closed RANGER-698.
--
> Ranger policy should support variables like $user
> -
>
> Key: RANGER-698
> URL: https://issues.apache.org/jira/browse/RANGER-698
> Project: Ranger
> Issue Type: Improvement
>Affects Versions: 0.7.0
>Reporter: Don Bosco Durai
>Assignee: Abhay Kulkarni
>Priority: Major
> Fix For: 0.7.0, 0.6.3
>
> Attachments: RANGER-698.1.patch
>
>
> It would be good to support variables in resources and users.
> E.g.
> HDFS Resource = /home/$user
> or
> Table Resource = ${user}_*
> Users allowed = $user
> Where $user will be expanded to the current user.
> I think, resource substitution will be easy. For permission, we can use key
> word like we use for all users group="public". We can use key word like
> "USER" or something like that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RANGER-729) Support multiple BaseDNs while syncing users from AD/LDAP
[ https://issues.apache.org/jira/browse/RANGER-729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16657925#comment-16657925 ] Don Bosco Durai commented on RANGER-729: [~spolavarapu] I think this has been already addressed. If so, we can close this. Thanks > Support multiple BaseDNs while syncing users from AD/LDAP > - > > Key: RANGER-729 > URL: https://issues.apache.org/jira/browse/RANGER-729 > Project: Ranger > Issue Type: Bug > Components: usersync >Affects Versions: 0.5.0 >Reporter: Don Bosco Durai >Assignee: Sailaja Polavarapu >Priority: Major > > Since AD search has limitation for not allowing OU/DN in search filter, it is > not possible to sync users from parallel OUs. > Eg: We should support synchronizing users from below search bases > OU=ServiceUsers,DC=EXAMPLE,DC=COM > and > OU=CorpUsers,DC=EXAMPLE,DC=COM > My suggestion for now will be to the take multiple BaseDNs with well known > delimiter or space -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (RANGER-720) Ldap discovery tool doesn't seem to be working as expected
[ https://issues.apache.org/jira/browse/RANGER-720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Don Bosco Durai closed RANGER-720. -- > Ldap discovery tool doesn't seem to be working as expected > -- > > Key: RANGER-720 > URL: https://issues.apache.org/jira/browse/RANGER-720 > Project: Ranger > Issue Type: Bug > Components: usersync >Affects Versions: 0.5.1 >Reporter: Don Bosco Durai >Assignee: Sailaja Polavarapu >Priority: Major > Fix For: 0.5.1, 0.6.0 > > > [~spolavarapu] > I was testing the ldap discovery tool against AD and it seems the results > were not as I expected: > input.properties: > ranger.usersync.ldap.url=ldap://ad-hello.cloud.hello.com > > ranger.usersync.ldap.binddn=CN=LDAP Access,OU=MyUsers,DC=AD-HELLO,DC=COM > ranger.usersync.ldap.ldapbindpassword= > ranger.admin.auth.sampleuser=CN=sample,OU=MyUsers,DC=AD-HELLO,DC=COM > ranger.admin.auth.samplepassword= > output: > SYNC_LDAP_USER_NAME_ATTRIBUTE=sAMAccountName > SYNC_LDAP_USER_OBJECT_CLASS=person > SYNC_LDAP_USER_GROUP_NAME_ATTRIBUTE= > SYNC_LDAP_USER_SEARCH_BASE=OU=workshop_service_users,DC=AD-HDP,DC=COM > SYNC_LDAP_USER_SEARCH_FILTER=sAMAccountName=* > ldapConfigCheck.log > INFO: No. of users from DC=AD-HELLO,DC=COM = 1 > INFO: No. of users from OU=workshop_service_users,DC=AD-HELLO,DC=COM = 12 > INFO: No. of users from OU=MyUsers,DC=AD-HELLO,DC=COM = 1 > INFO: No. of users from OU=Domain Controllers,DC=AD-HELLO,DC=COM = 1 > INFO: No. of users from CN=Users,DC=AD-HELLO,DC=COM = 5 > INFO: No. of users from DC=AD-HELLO,DC=COM = 1 > INFO: No. of users from OU=workshop_service_users,DC=AD-HELLO,DC=COM = 12 > INFO: No. of users from OU=MyUsers,DC=AD-HELLO,DC=COM = 1 > INFO: No. of users from OU=Domain Controllers,DC=AD-HELLO,DC=COM = 1 > INFO: No. of users from CN=Users,DC=AD-HELLO,DC=COM = 5 > ERROR: Connection failed: null > I was expecting the following: > SYNC_LDAP_USER_GROUP_NAME_ATTRIBUTE=sAMAccountName > SYNC_LDAP_USER_SEARCH_BASE=OU=MyUsers,DC=AD-HELLO,DC=COM > Also, there is an ERROR: Connection failed: null > Let me know if you need additional information. Thanks -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (RANGER-729) Support multiple BaseDNs while syncing users from AD/LDAP
[ https://issues.apache.org/jira/browse/RANGER-729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Don Bosco Durai reassigned RANGER-729: -- Assignee: Sailaja Polavarapu > Support multiple BaseDNs while syncing users from AD/LDAP > - > > Key: RANGER-729 > URL: https://issues.apache.org/jira/browse/RANGER-729 > Project: Ranger > Issue Type: Bug > Components: usersync >Affects Versions: 0.5.0 >Reporter: Don Bosco Durai >Assignee: Sailaja Polavarapu >Priority: Major > > Since AD search has limitation for not allowing OU/DN in search filter, it is > not possible to sync users from parallel OUs. > Eg: We should support synchronizing users from below search bases > OU=ServiceUsers,DC=EXAMPLE,DC=COM > and > OU=CorpUsers,DC=EXAMPLE,DC=COM > My suggestion for now will be to the take multiple BaseDNs with well known > delimiter or space -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (RANGER-813) Script to install Solr for Ranger Audits doesn't work in Suse
[ https://issues.apache.org/jira/browse/RANGER-813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Don Bosco Durai closed RANGER-813. -- > Script to install Solr for Ranger Audits doesn't work in Suse > - > > Key: RANGER-813 > URL: https://issues.apache.org/jira/browse/RANGER-813 > Project: Ranger > Issue Type: Bug >Reporter: Don Bosco Durai >Assignee: Don Bosco Durai >Priority: Major > Fix For: 1.0.0 > > > Seems suse OS doesn't have adduser utility method to add users. > In the ranger script to install solr, we will need to use "useradd" instead. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (RANGER-737) Need to update the current implementation for recent changes in Kafka
[ https://issues.apache.org/jira/browse/RANGER-737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Don Bosco Durai closed RANGER-737. -- > Need to update the current implementation for recent changes in Kafka > - > > Key: RANGER-737 > URL: https://issues.apache.org/jira/browse/RANGER-737 > Project: Ranger > Issue Type: Sub-task >Reporter: Don Bosco Durai >Assignee: Madhan Neethiraj >Priority: Major > Fix For: 0.5.0, 0.6.0 > > Attachments: > 0001-RANGER-737-updated-Ranger-Kakfa-plugin-for-recent-ch.patch > > > Kafka 0.9 has some changes done to their APIs (KAFKA-2210, KAFKA-2211, > KAFKA-2212). This needs to be reflected in Ranger plugin. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (RANGER-736) Apache license headers missing in files from recent commit
[ https://issues.apache.org/jira/browse/RANGER-736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Don Bosco Durai closed RANGER-736. -- > Apache license headers missing in files from recent commit > -- > > Key: RANGER-736 > URL: https://issues.apache.org/jira/browse/RANGER-736 > Project: Ranger > Issue Type: Bug > Components: admin >Affects Versions: 0.6.0 >Reporter: Don Bosco Durai >Assignee: Selvamohan Neethiraj >Priority: Blocker > Fix For: 0.6.0 > > Attachments: > 0001-RANGER-736-added-missing-apache-license-header-to-so.patch > > > > security-admin/src/main/java/org/apache/ranger/security/web/filter/RangerSSOAuthenticationFilter.java > > security-admin/src/main/java/org/apache/ranger/security/web/filter/SSOAuthentication.java > > security-admin/src/main/java/org/apache/ranger/security/web/filter/SSOAuthenticationProperties.java > Ideally, this should have been caught by our standard build options: > mvn clean compile package install assembly:assembly -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (RANGER-731) Ranger plugin for YARN doesn't seem to be able to write audit to Kerberized HDFS
[ https://issues.apache.org/jira/browse/RANGER-731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Don Bosco Durai closed RANGER-731. -- > Ranger plugin for YARN doesn't seem to be able to write audit to Kerberized > HDFS > > > Key: RANGER-731 > URL: https://issues.apache.org/jira/browse/RANGER-731 > Project: Ranger > Issue Type: Bug > Components: plugins >Affects Versions: 0.5.0 >Reporter: Don Bosco Durai >Assignee: Ramesh Mani >Priority: Major > Fix For: 0.6.0 > > > It seems the YARN plugin is missing some configuration/jars when writing to > kerberized HDFS. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (RANGER-815) In Solr managed-schema, we should set default for event_count and seq_num
[ https://issues.apache.org/jira/browse/RANGER-815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Don Bosco Durai closed RANGER-815. -- > In Solr managed-schema, we should set default for event_count and seq_num > - > > Key: RANGER-815 > URL: https://issues.apache.org/jira/browse/RANGER-815 > Project: Ranger > Issue Type: Improvement >Reporter: Don Bosco Durai >Assignee: Don Bosco Durai >Priority: Major > Fix For: 0.6.0 > > > The fields event_count gives the number of events for the audit record and > the seq_num helps in sorting audits for the same evtTime. > Currently, the values for both of these are set in the code. It would be good > if we can set the default value, so it is not dependent and mandatory for the > code to set it -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (RANGER-814) In Ranger Audit using Solr, auto create fields should be single valued
[ https://issues.apache.org/jira/browse/RANGER-814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Don Bosco Durai closed RANGER-814. -- > In Ranger Audit using Solr, auto create fields should be single valued > -- > > Key: RANGER-814 > URL: https://issues.apache.org/jira/browse/RANGER-814 > Project: Ranger > Issue Type: Bug >Reporter: Don Bosco Durai >Assignee: Don Bosco Durai >Priority: Major > Fix For: 0.6.0 > > > The managed-schema config file that is shipped with Ranger has auto field > creation enabled, but by default the values are multi valued. Multi-valued > values currently have some known limitations in Solr, example it doesn't > support sorting. > So by default, we should create any new auto discovered fields a > single-valued. And if someone needs multi-valued, then they should create or > update the field manually. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (RANGER-1065) Need to change default audit retentions in Solr 3 months and do some performance tuning
[ https://issues.apache.org/jira/browse/RANGER-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Don Bosco Durai closed RANGER-1065. --- > Need to change default audit retentions in Solr 3 months and do some > performance tuning > --- > > Key: RANGER-1065 > URL: https://issues.apache.org/jira/browse/RANGER-1065 > Project: Ranger > Issue Type: Improvement >Affects Versions: 0.5.0 >Reporter: Don Bosco Durai >Assignee: Don Bosco Durai >Priority: Major > Fix For: 0.6.0 > > > Currently, the default TTL is 3 years, which is too much. This will require a > much bigger SolrCloud and it might be necessary because, it is recommended > that long time archival should be in HDFS. > Also, need to do change the field types for date to use "docValues". This is > recommended by Solr community if the fields are used for sorting. > Another change need to be done is to the setting for useColdSearcher. The > default is "false", which will make the search to block if the searcher is > still warming up. If the index is very big, it might take a very long time > for the searcher to be fully warmed up. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (RANGER-1980) Build failure for Ranger 0.7 branch
[ https://issues.apache.org/jira/browse/RANGER-1980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Don Bosco Durai closed RANGER-1980. --- > Build failure for Ranger 0.7 branch > --- > > Key: RANGER-1980 > URL: https://issues.apache.org/jira/browse/RANGER-1980 > Project: Ranger > Issue Type: Bug > Components: build-infra >Affects Versions: 0.7.2 >Reporter: Don Bosco Durai >Assignee: Pradeep Agrawal >Priority: Major > Fix For: 1.0.0, 0.7.2 > > Attachments: > 0001-RANGER-1980-Build-failure-for-Ranger-0.7-branch.patch, > 0001-RANGER-1980-changed-Atlas-version-from-1.0.0-SNAPSHO.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > Ranger 0.7 compilation is failing with the following error. > > > [ERROR] Failed to execute goal on project ranger-tagsync: Could not resolve > dependencies for project org.apache.ranger:ranger-tagsync:jar:0.7.0.3.1.0: > The following artifacts could not be resolved: > org.apache.atlas:atlas-notification:jar:0.8.2-SNAPSHOT, > org.apache.atlas:atlas-typesystem:jar:0.8.2-SNAPSHOT, > org.apache.atlas:atlas-client-v1:jar:0.8.2-SNAPSHOT, > org.apache.atlas:atlas-client-common:jar:0.8.2-SNAPSHOT, > org.apache.atlas:atlas-client-v2:jar:0.8.2-SNAPSHOT, > org.apache.atlas:atlas-common:jar:0.8.2-SNAPSHOT, > org.apache.atlas:atlas-intg:jar:0.8.2-SNAPSHOT: Could not find artifact > org.apache.atlas:atlas-notification:jar:0.8.2-SNAPSHOT in > apache.snapshots.https > ([https://repository.apache.org/content/repositories/snapshots]) -> [Help 1] > > > > > Seems, Atlas snapshot jars are not found -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (RANGER-1283) Broken link on Wiki for 0.6 release
[ https://issues.apache.org/jira/browse/RANGER-1283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Don Bosco Durai closed RANGER-1283. --- > Broken link on Wiki for 0.6 release > --- > > Key: RANGER-1283 > URL: https://issues.apache.org/jira/browse/RANGER-1283 > Project: Ranger > Issue Type: Bug > Components: documentation >Affects Versions: 0.6.0 >Reporter: Don Bosco Durai >Assignee: Selvamohan Neethiraj >Priority: Blocker > > https://cwiki.apache.org/confluence/display/RANGER/0.6+Release+-+Apache+Ranger > Link for "Release Artifact" > Not Found > The requested URL /repos/dist/release/incubator/ranger/0.6.0-incubating/ was > not found on this server. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (RANGER-2121) Ranger build using Docker is broken
[ https://issues.apache.org/jira/browse/RANGER-2121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Don Bosco Durai closed RANGER-2121. --- > Ranger build using Docker is broken > --- > > Key: RANGER-2121 > URL: https://issues.apache.org/jira/browse/RANGER-2121 > Project: Ranger > Issue Type: Bug > Components: build-infra >Reporter: Don Bosco Durai >Assignee: Don Bosco Durai >Priority: Major > Fix For: 1.1.0 > > > The maven package used by Ranger Dockerfile is no more available in Apache. > Need to upgrade it to the latest maven version. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (RANGER-2048) Pass tag name when tag based policies are trigger
[ https://issues.apache.org/jira/browse/RANGER-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Don Bosco Durai reassigned RANGER-2048: --- Assignee: Don Bosco Durai > Pass tag name when tag based policies are trigger > - > > Key: RANGER-2048 > URL: https://issues.apache.org/jira/browse/RANGER-2048 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Reporter: Don Bosco Durai >Assignee: Don Bosco Durai >Priority: Major > > Currently, when tag policies are triggered, the tag which trigged it is not > passed downstream. This could be useful to have so that additional enrichment > can be done based on that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (RANGER-1137) Script to compile Apache Ranger using Docker
[
https://issues.apache.org/jira/browse/RANGER-1137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Don Bosco Durai closed RANGER-1137.
---
> Script to compile Apache Ranger using Docker
>
>
> Key: RANGER-1137
> URL: https://issues.apache.org/jira/browse/RANGER-1137
> Project: Ranger
> Issue Type: Bug
>Reporter: Don Bosco Durai
>Assignee: Don Bosco Durai
>Priority: Major
> Attachments: build_ranger_using_docker.sh
>
>
> By [~aneela] from @user mailing list...
> {quote}
> Hi,
> I am trying to compile ranger 0.6 release but facing following issue.
> fatal error: security/pam_appl.h: No such file or directory
> #include
> Please help me how to solve this issue?
> Thanks
> {quote}
> It might be good to use Docker to compile Apache Ranger and avoid host
> environment specific issues
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (RANGER-1137) Script to compile Apache Ranger using Docker
[
https://issues.apache.org/jira/browse/RANGER-1137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Don Bosco Durai resolved RANGER-1137.
-
Resolution: Fixed
This has been committed a while ago. Closing it
> Script to compile Apache Ranger using Docker
>
>
> Key: RANGER-1137
> URL: https://issues.apache.org/jira/browse/RANGER-1137
> Project: Ranger
> Issue Type: Bug
>Reporter: Don Bosco Durai
>Assignee: Don Bosco Durai
>Priority: Major
> Attachments: build_ranger_using_docker.sh
>
>
> By [~aneela] from @user mailing list...
> {quote}
> Hi,
> I am trying to compile ranger 0.6 release but facing following issue.
> fatal error: security/pam_appl.h: No such file or directory
> #include
> Please help me how to solve this issue?
> Thanks
> {quote}
> It might be good to use Docker to compile Apache Ranger and avoid host
> environment specific issues
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RANGER-2199) How to view the audit log from UI if the audit log stored the Kafka
[ https://issues.apache.org/jira/browse/RANGER-2199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594764#comment-16594764 ] Don Bosco Durai commented on RANGER-2199: - Honestly, I feel it would be much easy to install Solr then even trying to connect UI to Kafka. > How to view the audit log from UI if the audit log stored the Kafka > --- > > Key: RANGER-2199 > URL: https://issues.apache.org/jira/browse/RANGER-2199 > Project: Ranger > Issue Type: Improvement > Components: audit >Reporter: yinghua_zh >Priority: Major > > The ranger can store the audit log to the Kafka,But the REST API gets the > audit log from RDBMS or the Solr,How to view the audit log from UI if the > audit log stored the Kafka? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2199) How to view the audit log from UI if the audit log stored the Kafka
[
https://issues.apache.org/jira/browse/RANGER-2199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594592#comment-16594592
]
Don Bosco Durai commented on RANGER-2199:
-
Ranger Portal UI doesn't support querying audit logs from Kafka.
{quote}Or there are configurations imported from Kafka to Solr, and then UI can
query these audit logs from Solr{quote}
Ranger by default sends the audit logs to Solr. Is there any reason for not
using it? Also, Ranger supports sending audit logs to multiple destinations.
You can send to Kafka and Solr at the same time.
Alternatively, you can write a Kafka consumer to read the audits from Kafka and
add it to Solr.
> How to view the audit log from UI if the audit log stored the Kafka
> ---
>
> Key: RANGER-2199
> URL: https://issues.apache.org/jira/browse/RANGER-2199
> Project: Ranger
> Issue Type: Improvement
> Components: audit
>Reporter: yinghua_zh
>Priority: Major
>
> The ranger can store the audit log to the Kafka,But the REST API gets the
> audit log from RDBMS or the Solr,How to view the audit log from UI if the
> audit log stored the Kafka?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RANGER-2199) How to view the audit log from UI if the audit log stored the Kafka
[ https://issues.apache.org/jira/browse/RANGER-2199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593877#comment-16593877 ] Don Bosco Durai commented on RANGER-2199: - Ranger Admin portal only supports audit logs stored in Solr. It depends on Solr for listing and search capabilities. And Solr can store and index billions of records. It has good redundancy and scalability architecture. Kafka is mostly a messaging service. It is good for store and retrieve use cases. It generally has a smaller retention period (usually 1 month) and it is not possible to do search or listing/pagination from Kafka. Let me know if got your question incorrectly. > How to view the audit log from UI if the audit log stored the Kafka > --- > > Key: RANGER-2199 > URL: https://issues.apache.org/jira/browse/RANGER-2199 > Project: Ranger > Issue Type: Improvement > Components: audit >Reporter: yinghua_zh >Priority: Major > > The ranger can store the audit log to the Kafka,But the REST API gets the > audit log from RDBMS or the Solr,How to view the audit log from UI if the > audit log stored the Kafka? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[
https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593215#comment-16593215
]
Don Bosco Durai commented on RANGER-2128:
-
{quote}Hi Don Bosco Durai Kent Yao , any ideas on next steps?{quote}
[~toopt4] sorry for the delay. I was held up in other work.
I was getting class cast exception for HiveClient class in Spark 2.3 and not in
2.2. I ended up refactoring [~Qin Yao] code to remove all dependencies with
HiveClient and HiveSession. Now it is only dependent on Spark. However, I had
to copy some the Hive Plugin code into Spark Plugin.
I have to do some clean up of my code. After that, I can upload it to this JIRA
for [~Qin Yao] to review and add it to his pull request.
> Implement SparkSQL plugin
> -
>
> Key: RANGER-2128
> URL: https://issues.apache.org/jira/browse/RANGER-2128
> Project: Ranger
> Issue Type: New Feature
> Components: plugins, Ranger
>Affects Versions: 1.1.0
>Reporter: t oo
>Assignee: Kent Yao
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: support_ranger11.tgz
>
>
> Implement SparkSQL plugin
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[
https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16547526#comment-16547526
]
Don Bosco Durai commented on RANGER-2128:
-
{quote} ran thrift server with spark2.3.1(built-in hive) on yarn against Apache
Hadoop2.7.3/Hive Metastore Server2.1/ranger0.5.3-rc3{quote}
For some reason it is not working for me. I get the same class cast exception.
I built spark branch 2.3 from github. But I using the latest Ranger. It
shouldn't matter.
I will put some debug statement in Spark to see if I can figure this out.
> Implement SparkSQL plugin
> -
>
> Key: RANGER-2128
> URL: https://issues.apache.org/jira/browse/RANGER-2128
> Project: Ranger
> Issue Type: New Feature
> Components: plugins, Ranger
>Affects Versions: 1.1.0
>Reporter: t oo
>Assignee: Kent Yao
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: support_ranger11.tgz
>
>
> Implement SparkSQL plugin
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[
https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544838#comment-16544838
]
Don Bosco Durai commented on RANGER-2128:
-
{quote}When I tried thrift server, I did not see such an exception.{quote}
How are you running your Spark/Thrift server? I am using HDP 2.6.5 because I
have setup Kerberos with it. I can try your way also.
{quote}Livy Server is multi tenant. It launches spark applications separately
in different JVMs, which is easy to work with our current work.{quote}
I have not tried with Livy Server yet. That was next in my list.
{quote}By 1-3, I guess we can supply all real multi tenant servers for spark,
for spark thrift server, we may need to make some efforts to spark community to
support this such as step 4 and some related work{quote}
I agree with you. Let's try to get as close as possible to get this working.
Then we can see how to get the rest working.
Thanks
> Implement SparkSQL plugin
> -
>
> Key: RANGER-2128
> URL: https://issues.apache.org/jira/browse/RANGER-2128
> Project: Ranger
> Issue Type: New Feature
> Components: plugins, Ranger
>Affects Versions: 1.1.0
>Reporter: t oo
>Assignee: Kent Yao
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: support_ranger11.tgz
>
>
> Implement SparkSQL plugin
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[
https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544410#comment-16544410
]
Don Bosco Durai commented on RANGER-2128:
-
{quote}You can try Spark 2.3.1, because this change
https://github.com/apache/spark/blob/branch-2.3/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala#L129
ensures the SessionState that contains the thrift user body will be
reused.{quote}
[~Qin Yao] thanks. I was able to get the user from the State. In the AuthzImpl,
the following code gave the session user. I will update the Ranger Authorizer
to use this user
{code:java}
clientImpl.state.getUserName()
{code}
Regarding the ClassCastException, "org.apache.spark.sql.AnalysisException:
org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassCastException:
org.apache.hadoop.hive.metastore.ObjectStore cannot be cast to
org.apache.hadoop.hive.metastore.RawStore", I got this error even after I
removed all HiveAuthorizer related code. Do you think this is because of
different class loader that might be used when we do
"ext.injectOptimizerRule(Authorizer)" in the case of Thrift Server? You are
familiar with Spark, any clue is appreciated. Thanks
> Implement SparkSQL plugin
> -
>
> Key: RANGER-2128
> URL: https://issues.apache.org/jira/browse/RANGER-2128
> Project: Ranger
> Issue Type: New Feature
> Components: plugins, Ranger
>Affects Versions: 1.1.0
>Reporter: t oo
>Assignee: Kent Yao
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: support_ranger11.tgz
>
>
> Implement SparkSQL plugin
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[
https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544118#comment-16544118
]
Don Bosco Durai commented on RANGER-2128:
-
[~Qin Yao] I managed to get your code working with Ranger master branch. I have
attached the updates I have done. Can you try it on your side and see whether
it works for you?
It works with Spark Shell, but with Thrift Server I am getting an exception.
Have you seen this before? I have enabled Ranger in thrift server by adding the
following line in sparkconf.conf
{code:java}
spark.sql.extensions
org.apache.ranger.authorization.spark.authorizer.RangerSparkSQLExtension
{code}
Error opening session:
org.apache.spark.sql.AnalysisException:
org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassCastException:
org.apache.hadoop.hive.metastore.ObjectStore cannot be cast to
org.apache.hadoop.hive.metastore.RawStore;
at
org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:106)
at
org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:194)
at
org.apache.spark.sql.internal.SharedState.globalTempViewManager$lzycompute(SharedState.scala:138)
at
org.apache.spark.sql.internal.SharedState.globalTempViewManager(SharedState.scala:133)
at
org.apache.spark.sql.hive.HiveSessionStateBuilder$$anonfun$2.apply(HiveSessionStateBuilder.scala:54)
at
org.apache.spark.sql.hive.HiveSessionStateBuilder$$anonfun$2.apply(HiveSessionStateBuilder.scala:54)
at
org.apache.spark.sql.catalyst.catalog.SessionCatalog.globalTempViewManager$lzycompute(SessionCatalog.scala:91)
at
org.apache.spark.sql.catalyst.catalog.SessionCatalog.globalTempViewManager(SessionCatalog.scala:91)
at
org.apache.spark.sql.catalyst.catalog.SessionCatalog.setCurrentDatabase(SessionCatalog.scala:252)
at
org.apache.spark.sql.execution.command.SetDatabaseCommand.run(databases.scala:59)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:79)
at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:190)
at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:190)
at org.apache.spark.sql.Dataset$$anonfun$52.apply(Dataset.scala:3253)
at
org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:77)
at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3252)
at org.apache.spark.sql.Dataset.(Dataset.scala:190)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:75)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:638)
at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:694)
at
org.apache.spark.sql.hive.thriftserver.SparkSQLSessionManager.openSession(SparkSQLSessionManager.scala:70)
at
org.apache.hive.service.cli.CLIService.openSession(CLIService.java:194)
at
org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:354)
at
org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:246)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1253)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1238)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:53)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
> Implement SparkSQL plugin
> -
>
> Key: RANGER-2128
> URL: https://issues.apache.org/jira/browse/RANGER-2128
> Project: Ranger
> Issue Type: New Feature
> Components: plugins, Ranger
>Affects Versions: 1.1.0
>Reporter: t oo
>Assignee: Kent Yao
>Priority: Major
> Fix For: 2.0.0
>
>
> Implement SparkSQL plugin
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16534345#comment-16534345 ] Don Bosco Durai commented on RANGER-2128: - Hi [~Qin Yao], I was able to use the latest Ranger Hive Plugin with Spark. I have to do a bit of hack to use Jersey2 and also comment out one line in the Hive plugin code. Next, I am trying to see if I can integrate which SparkSQL Thrift server. I will update this JIRA after that. Hopefully by tomorrow. I also wrote a standalone Spark plugin and updated your code to call it. However, when I ran it in SparkSQL Thrift server, it was called, but I was not able to get the effective user (when doAs is false). I traced the code and it seems the Hive SessionState that is created has the effective user, however, I am not able to get that outside of it. If you know to get that somehow, then our implementation of the Spark plugin will be much simpler and not dependent on the HiveAuthorizer. > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Fix For: 2.0.0 > > > Implement SparkSQL plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2149) REST search queries make Ranger incredibly slow
[ https://issues.apache.org/jira/browse/RANGER-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16533282#comment-16533282 ] Don Bosco Durai commented on RANGER-2149: - [~pradeep.agrawal] do we have index in the DB table for policy name? In the DB world, 3000 rows is not a lot of data. Even full table scan should complete in seconds and not minutes. > REST search queries make Ranger incredibly slow > > > Key: RANGER-2149 > URL: https://issues.apache.org/jira/browse/RANGER-2149 > Project: Ranger > Issue Type: Bug > Components: Ranger >Affects Versions: 0.7.0 >Reporter: Alexander Posledov >Assignee: Pradeep Agrawal >Priority: Major > > If amount of Ranger policies is relatively big (3000-4000), then REST search > queries (e.g. search policy by name) make Ranger very slow. Example: > curl -u admin:admin -X GET > http://host:6080/service/plugins/policies?policyName=somePolicyName > 2-3 such queries (if we have ~3k policies) can cause too long delays even for > simple requests, while these searches aren't completed. In some cases, when > we ran several search queries simultaneously, this Ranger behavior caused 9 > min delay for simple adding a new policy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[
https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16528628#comment-16528628
]
Don Bosco Durai commented on RANGER-2128:
-
{quote}Hi Don Bosco Durai, the 0.6.x based ranger-hive-plugin is the pioneer
version of Apache Ranger, which is already hive2.1 based. Is there any better
option than using Hortonwork repo or inlining duplicated 0.5 based codes?{quote}
I am trying to see if we can remove the dependency with Hive and instantiate
the plugin outside of it. Let me give a try.
> Implement SparkSQL plugin
> -
>
> Key: RANGER-2128
> URL: https://issues.apache.org/jira/browse/RANGER-2128
> Project: Ranger
> Issue Type: New Feature
> Components: plugins, Ranger
>Affects Versions: 1.1.0
>Reporter: t oo
>Assignee: Kent Yao
>Priority: Major
> Fix For: 1.1.0
>
>
> Implement SparkSQL plugin
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521878#comment-16521878 ] Don Bosco Durai edited comment on RANGER-2128 at 6/30/18 7:35 AM: -- I integrated Ranger with SparkSQL ThriftServer based on [https://github.com/yaooqinn/spark-authorizer|https://github.com/yaooqinn/spark-authorizer] Since the Thrift Server was based on Hive 1.2, I had to integrate with Ranger 0.5. I have reached out to the developer from the above github to see if he can help us. Spark community has also added hooks to Spark (https://issues.apache.org/jira/browse/SPARK-18127) which can be used to implement Ranger Plugin has first-class integration. was (Author: bosco): I integrated Ranger with SparkSQL ThriftServer based on [https://github.com/yaooqinn/spark-authorizer|https://github.com/yaooqinn/spark-authorizer.] Since the Thrift Server was based on Hive 1.2, I had to integrate with Ranger 0.5. I have reached out to the developer from the above github to see if he can help us. Spark community has also added hooks to Spark (https://issues.apache.org/jira/browse/SPARK-18127) which can be used to implement Ranger Plugin has first-class integration. > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Fix For: 1.1.0 > > > Implement SparkSQL plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521878#comment-16521878 ] Don Bosco Durai edited comment on RANGER-2128 at 6/30/18 7:35 AM: -- I integrated Ranger with SparkSQL ThriftServer based on [https://github.com/yaooqinn/spark-authorizer|https://github.com/yaooqinn/spark-authorizer.] Since the Thrift Server was based on Hive 1.2, I had to integrate with Ranger 0.5. I have reached out to the developer from the above github to see if he can help us. Spark community has also added hooks to Spark (https://issues.apache.org/jira/browse/SPARK-18127) which can be used to implement Ranger Plugin has first-class integration. was (Author: bosco): I integrated Ranger with SparkSQL ThriftServer based on [https://github.com/yaooqinn/spark-authorizer.] Since the Thrift Server was based on Hive 1.2, I had to integrate with Ranger 0.5. I have reached out to the developer from the above github to see if he can help us. Spark community has also added hooks to Spark (https://issues.apache.org/jira/browse/SPARK-18127) which can be used to implement Ranger Plugin has first-class integration. > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Fix For: 1.1.0 > > > Implement SparkSQL plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16528565#comment-16528565 ] Don Bosco Durai commented on RANGER-2128: - [~toopt4] I saw that you have added the code for Spark as first class service. It is looking good, but there are still a few places you have Kylin. Few comments: # To avoid overlap with work from [~Qin Yao] , I feel you should create a dependent Jira and have your patch as part of it # Do you see a case where we would have a different meta store than Hive MetaStore? If not, we could get the policies directly from Hive Service def. So that the same policies will be enforced by Hive and Spark. # In your code, you have references to Project. E.g. getProjectResponse(). Can you clarify what it is? Just to make sure we don't step on each other work, can you list the tasks you will be working on? Also, Apache Ranger doesn't use pull request. Since this JIRA might need coordination with multiple contributors, let's get this to a working stage and then we can convert this to patch request and use review board for get comments from others. > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Fix For: 1.1.0 > > > Implement SparkSQL plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16524662#comment-16524662 ] Don Bosco Durai commented on RANGER-2128: - [~Qin Yao] thanks for putting this together so quickly. I have got your changes and done the build. I will try it out and let you know how it goes. Any specific Spark version you tried this with? > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Fix For: 1.1.0 > > > Implement SparkSQL plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (RANGER-2142) Solr installation script in Ranger should take installation package from local folder
Don Bosco Durai created RANGER-2142: --- Summary: Solr installation script in Ranger should take installation package from local folder Key: RANGER-2142 URL: https://issues.apache.org/jira/browse/RANGER-2142 Project: Ranger Issue Type: Improvement Components: Ranger Reporter: Don Bosco Durai Assignee: Don Bosco Durai With SOLR_INSTALL option enabled, security-admin/contrib/solr_for_audit_setup/setup.sh script downloads the Solr package from the internet and installs it. In some cases, the cluster might not have access to the internet. So we need an option to download the package offline and let this script install the local package copy -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-1300) S3 support
[
https://issues.apache.org/jira/browse/RANGER-1300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16524131#comment-16524131
]
Don Bosco Durai commented on RANGER-1300:
-
{quote}Although I think the policy evaluation in Ranger is complex and counter
intuitive with all the weights etc (why not use a firewall approach and user
the order by which it was entered?
{quote}
It will be a long discussion with lots of history :) There are Ranger
deployments which contains 1000s of policies, so policy evaluation time is very
critical in HDFS and other high volume components. The same will be applicable
for Object Store also. Also, the policy engine itself is a framework and
advanced policies can be built on the framework. The framework supports user
extension for context enrichment (similar to resource tags), conditional
policies (e.g. custom time-based policies), etc. Once you add Deny policy and
policy evaluation order, it further complicates the implementation. The team is
constantly updating the implementation. If you have suggestions, we should
start another thread to discuss it.
{quote}I don't know if Ranger knows a kind of events that fired off when a
policy change happens? If that exists you could manage many permissions
directly from ranger.
{quote}
I recently gave a talk in DataWorks submit on explicitly managing the policies
on S3 based according to Ranger Tag-Based policies. One open item was to
reverse sync policies from S3 to Ranger. If that is what you are mentioning,
then in S3 one option is to monitor for AWSConfig events and update Ranger or
roll it back. Currently, because of limitation on S3 on the number of policies,
it might not be practical to manage S3 resource level policies with Ranger.
> S3 support
> --
>
> Key: RANGER-1300
> URL: https://issues.apache.org/jira/browse/RANGER-1300
> Project: Ranger
> Issue Type: New Feature
> Components: plugins
>Reporter: Jose
>Priority: Major
> Attachments: ranger-servicedef-aws-s3.json
>
>
> As more and more people are deploying hadoop into AWS and as S3 is used in
> lots of application. It'd be nice to have S3 support built into Ranger.
> It's not a trivial task. Right now Ranger Storage support (only hdfs) runs
> directly in the Namenode
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RANGER-1300) S3 support
[
https://issues.apache.org/jira/browse/RANGER-1300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522966#comment-16522966
]
Don Bosco Durai commented on RANGER-1300:
-
{quote} * Proxy (for Ceph/RadosGW at the moment):
[https://github.com/bolkedebruin/s3gw]{quote}
Hi [~bolke], I looked into your code. Seems you have implemented Ranger policy
evaluation in golang :) I had few questions.
# Did you consider using JNI integration? I don't know much about GO, I saw
few libraries like this [https://github.com/Centny/jnigo.]
# We had a similar requirement for Apache HAWQ, which is written in 'C'. We
ended up using JNI to Ranger plugin written Java. There are multiple advantages
of using JNI approach because the Ranger community has optimized the policy
evaluation significantly, including using Trie indexing to grossly increase the
performance. Also having one code base helps in keeping all platform in the
same feature level, like supporting Tag Based Policies, custom enrichment,
conditional policies, etc. Also integrating with Ranger Audit framework will be
automatically supported if you use Java implementation
# How is Ceph configured to use the Ranger plugin? I couldn't see the
documentation for that. Is it just configuration or we need to recompile Ceph
with your plugin?
> S3 support
> --
>
> Key: RANGER-1300
> URL: https://issues.apache.org/jira/browse/RANGER-1300
> Project: Ranger
> Issue Type: New Feature
> Components: plugins
>Reporter: Jose
>Priority: Major
> Attachments: ranger-servicedef-aws-s3.json
>
>
> As more and more people are deploying hadoop into AWS and as S3 is used in
> lots of application. It'd be nice to have S3 support built into Ranger.
> It's not a trivial task. Right now Ranger Storage support (only hdfs) runs
> directly in the Namenode
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[
https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522919#comment-16522919
]
Don Bosco Durai commented on RANGER-2128:
-
{quote}It has exposed Parser/Analyzer/Optimizer/Planner, which is so great for
all users. It also makes it easier for users to call our plug-in.
1. spark-authorizer is designed as a Optimize Rule for Spark SQL and executed
after all other default rules because rules, such as column pruning, projection
push down, and so on, should be operated first.
{quote}
I was wondering if it would be difficult to migrate your extension to use the
official hook provided by Spark? If we can do that, then it might be easy to
add Ranger features like dynamic UDF and row level filtering.
{quote}2. spark-authorizer has to visit hive SessionState object which is not
accessible for spark context classloader because Spark use a isolated
classloader to load hive client jars.
2.1 spark-authorizer itself will rewrite SessionState object the first time to
do privileges checking
{quote}
I checked that. It is a pretty good hack that works :) I had to update it to
support custom authentication. The current Ranger Hive Plugin use Hadoop UGI,
which only knows Kerberos and Simple Auth.
{quote}2.2 kyuubi hacks spark and turn off that classloader.
{quote}
I went through your documentation, it seems you have added a lot of good
features. Currently, kyuubi is a custom build. Is it possible to integrate your
extensions as an addon to existing deployment? In this way, users can deploy
the default Thrift Server, but using some properties or code injections adds
your feature? We might then able to support Livy also with the same code base.
{quote}3. spark-authorizer reuses the ranger hive plugin(0.5）which contains
incompatible jersey dependencies with spark ones.
{quote}
There are few limitations with Ranger 0.5, most notably it doesn't support Tag
Based policies. I was thinking, we should just implement first class plugin for
SparkSQL using Ranger 0.7 or 1.0. It could use the same Hive
ServiceDef/Policies, but native implementation for SparkSQL. In this way, we
don't have to be dependent with Hive libraries and it's limitation.
{quote}And what are the steps I should follow to contribute Ranger?
{quote}
I have added you as a contributor to Ranger. You should be able to assign Jira
to yourself and create new ones. I was thinking of splitting the work among
those interested. Since you are familiar with the Spark code, do you want to
look into the new extensions and see how we can implement basic authorization
and advanced features like dynamic masking/UDF and Row Level filtering? I can
look into Tag based policies and also see if I can extract your current Spark
Authorizer feature into native SparkSQL Ranger Plugin.
Give me your thoughts and suggestions.
Thanks
> Implement SparkSQL plugin
> -
>
> Key: RANGER-2128
> URL: https://issues.apache.org/jira/browse/RANGER-2128
> Project: Ranger
> Issue Type: New Feature
> Components: plugins, Ranger
>Affects Versions: 1.1.0
>Reporter: t oo
>Priority: Major
> Fix For: 1.1.0
>
>
> Implement SparkSQL plugin
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521958#comment-16521958 ] Don Bosco Durai commented on RANGER-2128: - [~Qin Yao] thanks for helping us out. When I went through your spark-authorizer code, you had mapped all Spark actions to Hive actions. It was pretty impressive. Thanks for pointing out (offline) to me your work on [https://github.com/yaooqinn/kyuubi.] It seems to be what everyone wants. Since you are familiar with the Spark integrations and challenges, what is your recommendation? Can we work on a high-level design flow? E.g. Can we leverage the new Spark hook to implement some of the plugin interactions? Thanks > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Priority: Major > Fix For: 1.1.0 > > > Implement SparkSQL plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521878#comment-16521878 ] Don Bosco Durai commented on RANGER-2128: - I integrated Ranger with SparkSQL ThriftServer based on [https://github.com/yaooqinn/spark-authorizer.] Since the Thrift Server was based on Hive 1.2, I had to integrate with Ranger 0.5. I have reached out to the developer from the above github to see if he can help us. Spark community has also added hooks to Spark (https://issues.apache.org/jira/browse/SPARK-18127) which can be used to implement Ranger Plugin has first-class integration. > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Priority: Major > Fix For: 1.1.0 > > > Implement SparkSQL plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16518669#comment-16518669 ] Don Bosco Durai commented on RANGER-2128: - [~toopt4] can you be more specific what you are looking for? Thanks > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Priority: Major > Fix For: 1.1.0 > > > Implement SparkSQL plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2130) Ranger Admin - client-side control bypass
[
https://issues.apache.org/jira/browse/RANGER-2130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16515419#comment-16515419
]
Don Bosco Durai commented on RANGER-2130:
-
[~toopt4] , I am not sure whether this is a valid issue. What you are seeing
are the roles supported by Ranger. There is no harm in knowing all the roles
supported by Ranger. Especially, Ranger is an open source project and the
source code is available to everyone.
I would have been more concerned if by changing the UI fields values a regular
user is able to impersonate as an Admin user and able to make server-side
changes. But based on your comment, the server gives an appropriate error when
you do it.
Let me know if you still feel this is an issue.
Thanks
> Ranger Admin - client-side control bypass
> -
>
> Key: RANGER-2130
> URL: https://issues.apache.org/jira/browse/RANGER-2130
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Affects Versions: 1.0.0
>Reporter: t oo
>Assignee: Nitin Galave
>Priority: Major
> Attachments: 0001-RANGER-2130.patch, Screen Shot 2018-06-11 at
> 10.36.39 am.png, client_side_controls1.PNG, client_side_controls2.PNG
>
>
> *Risk/Issue summary finding*
> {code:java}
> Client-side Control Bypass (Ranger){code}
> *Risk/Issue summary description/detail*
> {code:java}
> The Apache Ranger application relies on client-side controls to restrict user
> access to certain information and functionality. A user can bypass these
> controls (by modifying client-side parameters or directly browsing to
> specific API requests or resources) to view information without the required
> authorisation.
> The attached screenshots show the "admin" user bypassing client-side controls
> to modify their Role from "User" to "Admin". Whilst submitting this request
> is unsuccessful and will not permanently change the user role, the GUI allows
> access to sections that were previously hidden.{code}
> *Business impact / attack scenario*
> {code:java}
> Low privilege users with restricted access are able to view information that
> is not intended for their viewing. As an example, the admin user can bypass
> client side controls to view configuration details for the HIVE_RANGER_E2E
> hive object. {code}
> *Recommendation*
> {code:java}
> Do not rely on client-side controls to restrict user access. Ensure that
> server-side controls are in place to restrict unauthorised access to
> sensitive information and APIs. {code}
>
> In the rangeradmin ui, on the users page, after clicking on a user. If you
> edit the html on the site (ie in Chrome) you can remove the 'disabled' tag so
> that the role of User becomes ungreyed out and you can change the role from
> User to Admin!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (RANGER-1974) Ranger Authorizer and Audits for AWS S3
[ https://issues.apache.org/jira/browse/RANGER-1974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Don Bosco Durai reassigned RANGER-1974: --- Assignee: Don Bosco Durai > Ranger Authorizer and Audits for AWS S3 > > > Key: RANGER-1974 > URL: https://issues.apache.org/jira/browse/RANGER-1974 > Project: Ranger > Issue Type: New Feature > Components: Ranger >Reporter: Srikanth Venkat >Assignee: Don Bosco Durai >Priority: Blocker > > As an enterprise security admin, I need to be able to define and manage > authorization policies for data stored in AWS S3 so that I can manage my > access control and authorization entitlements in hybrid and cloud > environments along with other data in platforms that Ranger currently > authorizes. This feature will should allow interoperability with AWS IAM > policies and be able to gather audits from the native cloud audit > capabilities such as via AWS CloudTrail. > Implementation considerations: > # AWS S3 IAM information: https://aws.amazon.com/documentation/iam/ > # AWS CloudTrail information: > https://aws.amazon.com/documentation/cloudtrail/ > # This could be a policy mapping or sync mechanism (either online or > offline) that will allow Ranger policy conditions, and user/group/role or > other policy elements to be mapped to what is available in AWS IAM. This > might entail having a different model where the Ranger plugin might not be > running in the cloud native service and might require a proxy or other > paradigms to be effective. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (RANGER-2121) Ranger build using Docker is broken
Don Bosco Durai created RANGER-2121: --- Summary: Ranger build using Docker is broken Key: RANGER-2121 URL: https://issues.apache.org/jira/browse/RANGER-2121 Project: Ranger Issue Type: Bug Components: build-infra Reporter: Don Bosco Durai Assignee: Don Bosco Durai The maven package used by Ranger Dockerfile is no more available in Apache. Need to upgrade it to the latest maven version. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-2086) Resource data mask policy overrides when both tag and resource datamask policies match
[ https://issues.apache.org/jira/browse/RANGER-2086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463245#comment-16463245 ] Don Bosco Durai commented on RANGER-2086: - [~abhayk] , when there are conflicting policies, which takes preference? Thanks > Resource data mask policy overrides when both tag and resource datamask > policies match > -- > > Key: RANGER-2086 > URL: https://issues.apache.org/jira/browse/RANGER-2086 > Project: Ranger > Issue Type: Bug > Components: Ranger >Affects Versions: 1.0.0, master >Reporter: Abhay Kulkarni >Assignee: Abhay Kulkarni >Priority: Major > Fix For: master, 1.1.0 > > > If both tag and resource data-mask policies match accessed hive resource, > mask-specification in the resource policy is always used. The audit log > record is inconsistent in that it contains tag-policy-id as determining > data-mask policy, but the masking specification is from resource policy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (RANGER-2050) Support non standard conf folder for hive during Hive Plugin install
[ https://issues.apache.org/jira/browse/RANGER-2050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Don Bosco Durai updated RANGER-2050: Summary: Support non standard conf folder for hive during Hive Plugin install (was: Support non standard conf folder for hive) > Support non standard conf folder for hive during Hive Plugin install > > > Key: RANGER-2050 > URL: https://issues.apache.org/jira/browse/RANGER-2050 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Reporter: Don Bosco Durai >Priority: Major > > Manual install for Hive assumes conf folder will be always in the Hive home > folder. It is more likely that the conf will be in /etc/hive/conf or other > custom folders. It would be good to take optional property with the location > of the conf folder. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (RANGER-2050) Support non standard conf folder for hive
Don Bosco Durai created RANGER-2050: --- Summary: Support non standard conf folder for hive Key: RANGER-2050 URL: https://issues.apache.org/jira/browse/RANGER-2050 Project: Ranger Issue Type: Improvement Components: Ranger Reporter: Don Bosco Durai Manual install for Hive assumes conf folder will be always in the Hive home folder. It is more likely that the conf will be in /etc/hive/conf or other custom folders. It would be good to take optional property with the location of the conf folder. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (RANGER-2049) Support doAs in Ranger Admin Portal / REST API
Don Bosco Durai created RANGER-2049: --- Summary: Support doAs in Ranger Admin Portal / REST API Key: RANGER-2049 URL: https://issues.apache.org/jira/browse/RANGER-2049 Project: Ranger Issue Type: Improvement Components: Ranger Reporter: Don Bosco Durai It will be good to support doAs in Ranger Admin and be consistent with other components within Hadoop ecosystem. This will help proxies like Knox to front Ranger Admin, but pass the calling user similar to WebHDFS in Ranger, which will enable Ranger to do fine grain authorization and auditing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (RANGER-2048) Pass tag name when tag based policies are trigger
Don Bosco Durai created RANGER-2048: --- Summary: Pass tag name when tag based policies are trigger Key: RANGER-2048 URL: https://issues.apache.org/jira/browse/RANGER-2048 Project: Ranger Issue Type: Improvement Components: Ranger Reporter: Don Bosco Durai Currently, when tag policies are triggered, the tag which trigged it is not passed downstream. This could be useful to have so that additional enrichment can be done based on that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (RANGER-2047) Option to pass parameters in predefined UDFs
Don Bosco Durai created RANGER-2047: --- Summary: Option to pass parameters in predefined UDFs Key: RANGER-2047 URL: https://issues.apache.org/jira/browse/RANGER-2047 Project: Ranger Issue Type: Improvement Components: Ranger Reporter: Don Bosco Durai For predefined UDFs there is no option to add parameters. It might be good to provide the option to give parameters so they can better utilize the pre defined UDFs. Some existing UDFs which will benefit are: Date: show only year (e.g. This could take the data format as input) Partial mask: show last 4 Partial mask: show first 4 Redact: (This could take the character to use for redaction) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (RANGER-2046) Feature to add custom predefined UDF for masking
Don Bosco Durai created RANGER-2046: --- Summary: Feature to add custom predefined UDF for masking Key: RANGER-2046 URL: https://issues.apache.org/jira/browse/RANGER-2046 Project: Ranger Issue Type: Improvement Components: Ranger Reporter: Don Bosco Durai For masking policies, Ranger comes with out of the box predefined policies like Hash, Redact, Nullify, etc. And it also provides the option to enter custom UDFs. However, for custom UDF, admins need to remember the complete syntax, which sometimes can have multiple parameters or copy paste it every time it is used. It might be good to provide an option to add frequently used custom UDF as predefined so that it comes as a dropdown for users to use it consistently and also avoid human error. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-1974) Ranger Authorizer and Audits for AWS S3
[ https://issues.apache.org/jira/browse/RANGER-1974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418062#comment-16418062 ] Don Bosco Durai commented on RANGER-1974: - Anyone interested in collaborating on this JIRA? Thanks > Ranger Authorizer and Audits for AWS S3 > > > Key: RANGER-1974 > URL: https://issues.apache.org/jira/browse/RANGER-1974 > Project: Ranger > Issue Type: New Feature > Components: Ranger >Reporter: Srikanth Venkat >Priority: Blocker > > As an enterprise security admin, I need to be able to define and manage > authorization policies for data stored in AWS S3 so that I can manage my > access control and authorization entitlements in hybrid and cloud > environments along with other data in platforms that Ranger currently > authorizes. This feature will should allow interoperability with AWS IAM > policies and be able to gather audits from the native cloud audit > capabilities such as via AWS CloudTrail. > Implementation considerations: > # AWS S3 IAM information: https://aws.amazon.com/documentation/iam/ > # AWS CloudTrail information: > https://aws.amazon.com/documentation/cloudtrail/ > # This could be a policy mapping or sync mechanism (either online or > offline) that will allow Ranger policy conditions, and user/group/role or > other policy elements to be mapped to what is available in AWS IAM. This > might entail having a different model where the Ranger plugin might not be > running in the cloud native service and might require a proxy or other > paradigms to be effective. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (RANGER-1986) Hive Policies doesn't access columns with spaces
Don Bosco Durai created RANGER-1986: --- Summary: Hive Policies doesn't access columns with spaces Key: RANGER-1986 URL: https://issues.apache.org/jira/browse/RANGER-1986 Project: Ranger Issue Type: Bug Components: Ranger Reporter: Don Bosco Durai Hive supports columns with spaces, but Ranger doesn't. When space is given while entering the column name in the Ranger Policy UI, it automatically splits it into 2 words/column names. Is it possible to make "enter" has the way to accept/finalize the column name? Or if a single or double quote is given, then wait for another single or double quotes or enter before accepting the column name? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (RANGER-1980) Build failure for Ranger 0.7 branch
Don Bosco Durai created RANGER-1980: --- Summary: Build failure for Ranger 0.7 branch Key: RANGER-1980 URL: https://issues.apache.org/jira/browse/RANGER-1980 Project: Ranger Issue Type: Bug Components: build-infra Reporter: Don Bosco Durai Ranger 0.7 compilation is failing with the following error. [ERROR] Failed to execute goal on project ranger-tagsync: Could not resolve dependencies for project org.apache.ranger:ranger-tagsync:jar:0.7.0.3.1.0: The following artifacts could not be resolved: org.apache.atlas:atlas-notification:jar:0.8.2-SNAPSHOT, org.apache.atlas:atlas-typesystem:jar:0.8.2-SNAPSHOT, org.apache.atlas:atlas-client-v1:jar:0.8.2-SNAPSHOT, org.apache.atlas:atlas-client-common:jar:0.8.2-SNAPSHOT, org.apache.atlas:atlas-client-v2:jar:0.8.2-SNAPSHOT, org.apache.atlas:atlas-common:jar:0.8.2-SNAPSHOT, org.apache.atlas:atlas-intg:jar:0.8.2-SNAPSHOT: Could not find artifact org.apache.atlas:atlas-notification:jar:0.8.2-SNAPSHOT in apache.snapshots.https ([https://repository.apache.org/content/repositories/snapshots]) -> [Help 1] Seems, Atlas snapshot jars are not found -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-1919) kerberos sync
[ https://issues.apache.org/jira/browse/RANGER-1919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346173#comment-16346173 ] Don Bosco Durai commented on RANGER-1919: - [~Ronald van de Kuil] , user sync in Ranger is generic and can support different sources. In addition to AD/LDAP, linux users, it also supports importing users from flat files. With Kerberos users, if it is not backed with AD or OpenLDAP, then you could download the principals to a file and do the translation as it is configured in HDFS auth rules and then you should be able to load into Ranger. If this works, we should be able to convert it in a scheduled job. > kerberos sync > - > > Key: RANGER-1919 > URL: https://issues.apache.org/jira/browse/RANGER-1919 > Project: Ranger > Issue Type: Improvement > Components: plugins >Reporter: Ronald van de Kuil >Priority: Minor > > In my understanding usersync, AD, LDAP configs are there to sync identities > into ranger so that access policies can be attached to them. > It would make sense then to do the same for identities in a kerberos realm. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-1968) HBase plugin support the data custom automatic desensitization function.
[ https://issues.apache.org/jira/browse/RANGER-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346105#comment-16346105 ] Don Bosco Durai commented on RANGER-1968: - Hi [~zhangqiang2] , can you elaborate on this topic? Thanks > HBase plugin support the data custom automatic desensitization function. > > > Key: RANGER-1968 > URL: https://issues.apache.org/jira/browse/RANGER-1968 > Project: Ranger > Issue Type: New Feature > Components: plugins >Reporter: Qiang Zhang >Assignee: Qiang Zhang >Priority: Major > Labels: newbie, patch > > The data desensitization is an important feature of big data security. Base > on the ranger framework, we can provide a complete automatic data > desensitization solution for Hbase. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (RANGER-1938) Solr for Audit setup doesn't use DocValues effectively
[ https://issues.apache.org/jira/browse/RANGER-1938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299544#comment-16299544 ] Don Bosco Durai commented on RANGER-1938: - bq. Does it make sense for me to open a JIRA against the Ambari project as well to fix this schema? I can link to this JIRA since they are related. Yes it would be good to create the JIRA. The Ranger members in Ambari team would be able to replicate your changes there. bq. I don't think there is a great reason to not enable at the "global" fieldType level. We can disable at the individual field level if necessary. We can do either way. We can explicitly set the docValues for each field or do the global setting. If we are going to do global changes, then let's document it in the file itself, so anyone looking into it in the future will be aware of the reason. Thanks for the details of your environment. It is very useful. > Solr for Audit setup doesn't use DocValues effectively > -- > > Key: RANGER-1938 > URL: https://issues.apache.org/jira/browse/RANGER-1938 > Project: Ranger > Issue Type: Improvement > Components: audit >Affects Versions: 0.6.0, 0.7.0, 0.6.1, 0.6.2, 0.6.3, 0.7.1 >Reporter: Kevin Risden >Assignee: Kevin Risden > Labels: performance > Fix For: 1.0.0, 0.7.2 > > Attachments: > 0001-RANGER-1938-Enable-DocValues-for-more-fields-in-Solr.patch > > > Ranger uses Ambari Infra Solr (or another Apache Solr install) for storing > Ranger Audit events for displaying in Ranger Admin. In our case, we have > noticed quite a few Ambari Infra Solr OOM due to Ranger. I've talked with a > few other people who are having very similar problems with OOM errors. > I've typed up some details about how the way Ranger is using Solr requires a > lot of heap. I've also outlined the fix for this which significantly reduced > the amount of heap memory required. I'm an Apache Lucene/Solr committer so > this optimization/usage might not be immediately obvious to those using Solr > especially version 5.x. > https://risdenk.github.io/2017/12/18/ambari-infra-solr-ranger.html -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (RANGER-1837) Enhance Ranger Audit to HDFS to support ORC file format
[ https://issues.apache.org/jira/browse/RANGER-1837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16259521#comment-16259521 ] Don Bosco Durai edited comment on RANGER-1837 at 11/20/17 5:37 PM: --- bq. current implementation uses existing AuditQueue for the data pipe into destination. If this has to be avoided we need to have a one more major refactoring on the audit framework Not sure I understood your concern. What are we currently using to write to HDFS in batches? Is it using AuditFileCacheProvider + ??? The audit framework consists of Queues and Destination. Ideally, we could just clone AuditBatchQueue and simplify it. E.g. in the new class AuditFileQueue, we can read from the consumer and write to a local file from the method log(AuditEventBase event). And at fixed intervals, we could just write the local file to the Destination (HDFS/S3). Similar to AuditFileSpool, we might have to keep track of the local files that need to be written to the destination. And I think, BaseAuditHandler should have another method called logFile( File localFile), which can be implemented by HDFSDestination to just copy the file over or convert it to ORC and write to the HDFS. Default implementation could be to read the file line by line and call logJSON(); b.q. Current framework provides 3 buffer size If possible, I would suggest using one of the current buffers. Looking at the current code, it seems AuditFileCacheProvider replaces AuditAsyncQueue at the top level. Which could be a problem, because now we are sinking into the file up front. The original design was, every destination has a queue backing it so that the queue can help regulate the output flow to the destination. In this way, the queue manages the buffer and not the destination. Even with the current implementation, I would suggest introducing something like AuditFileQueue and set the file size and time on it. This will give users to pick a larger file size e.g. 1-hour interval and write to the destination after 1 hour. bq. I see this article https://community.hortonworks.com/articles/75501/orc-creation-best-practices.html which has some details. This is good information. It seems advance users can reload the data using Hive at regular intervals to get good distribution, the sort order for audit dates, etc. And they can also analyze for better runtime performance. We can add these in best practices or recommendation section for this feature. bq. I tested it for 1 hr data for hdfs plugin and having all three 1 was fine. It didn't create multiple files for the amount, but this depends on the amount of hdfs activities. I need to check with KAFKA plugin I am worried that if we do this in memory buffer, we risk affecting the native component by hogging the memory. was (Author: bosco): bq. current implementation uses existing AuditQueue for the data pipe into destination. If this has to be avoided we need to have a one more major refactoring on the audit framework Not sure I understood your concern. What are we currently using to write to HDFS in batches? Is it using AuditFileCacheProvider + ??? The audit framework consists of Queues and Destination. Ideally, we could just clone AuditBatchQueue and simplify it. E.g. in the new class AuditFileQueue, we can read from the consumer and write to a local file from the method log(AuditEventBase event). And at fixed intervals, we could just write the local file to the Destination (HDFS/S3). Similar to AuditFileSpool, we might have to keep track of the local files that need to be written to the destination. b.q. Current framework provides 3 buffer size If possible, I would suggest using one of the current buffers. Looking at the current code, it seems AuditFileCacheProvider replaces AuditAsyncQueue at the top level. Which could be a problem, because now we are sinking into the file up front. The original design was, every destination has a queue backing it so that the queue can help regulate the output flow to the destination. In this way, the queue manages the buffer and not the destination. Even with the current implementation, I would suggest introducing something like AuditFileQueue and set the file size and time on it. This will give users to pick a larger file size e.g. 1-hour interval and write to the destination after 1 hour. And I think, BaseAuditHandler should have another method called logFile( File localFile), which can be implemented by HDFSDestination to just copy the file over or convert it to ORC and write to the HDFS. Default implementation could be to read the file line by line and call logJSON(); bq. I see this article https://community.hortonworks.com/articles/75501/orc-creation-best-practices.html which has some details. This is good information. It seems advance users can reload the data using Hive at regular intervals to get
[jira] [Commented] (RANGER-1837) Enhance Ranger Audit to HDFS to support ORC file format
[
https://issues.apache.org/jira/browse/RANGER-1837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16259521#comment-16259521
]
Don Bosco Durai commented on RANGER-1837:
-
bq. current implementation uses existing AuditQueue for the data pipe into
destination. If this has to be avoided we need to have a one more major
refactoring on the audit framework
Not sure I understood your concern. What are we currently using to write to
HDFS in batches? Is it using AuditFileCacheProvider + ???
The audit framework consists of Queues and Destination. Ideally, we could just
clone AuditBatchQueue and simplify it. E.g. in the new class AuditFileQueue, we
can read from the consumer and write to a local file from the method
log(AuditEventBase event). And at fixed intervals, we could just write the
local file to the Destination (HDFS/S3). Similar to AuditFileSpool, we might
have to keep track of the local files that need to be written to the
destination.
b.q. Current framework provides 3 buffer size
If possible, I would suggest using one of the current buffers. Looking at the
current code, it seems AuditFileCacheProvider replaces AuditAsyncQueue at the
top level. Which could be a problem, because now we are sinking into the file
up front. The original design was, every destination has a queue backing it so
that the queue can help regulate the output flow to the destination. In this
way, the queue manages the buffer and not the destination. Even with the
current implementation, I would suggest introducing something like
AuditFileQueue and set the file size and time on it. This will give users to
pick a larger file size e.g. 1-hour interval and write to the destination after
1 hour.
And I think, BaseAuditHandler should have another method called logFile( File
localFile), which can be implemented by HDFSDestination to just copy the file
over or convert it to ORC and write to the HDFS. Default implementation could
be to read the file line by line and call logJSON();
bq. I see this article
https://community.hortonworks.com/articles/75501/orc-creation-best-practices.html
which has some details.
This is good information. It seems advance users can reload the data using Hive
at regular intervals to get good distribution, the sort order for audit dates,
etc. And they can also analyze for better runtime performance. We can add these
in best practices or recommendation section for this feature.
bq. I tested it for 1 hr data for hdfs plugin and having all three 1 was
fine. It didn't create multiple files for the amount, but this depends on the
amount of hdfs activities. I need to check with KAFKA plugin
I am worried that if we do this in memory buffer, we risk affecting the native
component by hogging the memory.
> Enhance Ranger Audit to HDFS to support ORC file format
> ---
>
> Key: RANGER-1837
> URL: https://issues.apache.org/jira/browse/RANGER-1837
> Project: Ranger
> Issue Type: Improvement
> Components: audit
>Reporter: Kevin Risden
>Assignee: Ramesh Mani
> Attachments:
> 0001-RANGER-1837-Enhance-Ranger-Audit-to-HDFS-to-support-.patch,
> 0001-RANGER-1837-Enhance-Ranger-Audit-to-HDFS-to-support_001.patch
>
>
> My team has done some research and found that Ranger HDFS audits are:
> * Stored as JSON objects (one per line)
> * Not compressed
> This is currently very verbose and would benefit from compression since this
> data is not frequently accessed.
> From Bosco on the mailing list:
> {quote}You are right, currently one of the options is saving the audits in
> HDFS itself as JSON files in one folder per day. I have loaded these JSON
> files from the folder into Hive as compressed ORC format. The compressed
> files in ORC were less than 10% of the original size. So, it was significant
> decrease in size. Also, it is easier to run analytics on the Hive tables.
>
> So, there are couple of ways of doing it.
>
> Write an Oozie job which runs every night and loads the previous day worth
> audit logs into ORC or other format
> Write a AuditDestination which can write into the format you want to.
>
> Regardless which approach you take, this would be a good feature for
> Ranger.{quote}
> http://mail-archives.apache.org/mod_mbox/ranger-user/201710.mbox/%3CCAJU9nmiYzzUUX1uDEysLAcMti4iLmX7RE%3DmN2%3DdoLaaQf87njQ%40mail.gmail.com%3E
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (RANGER-1738) RangerYarnAuthorizer not compatible with Hadoop-3.0.0
[
https://issues.apache.org/jira/browse/RANGER-1738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16259441#comment-16259441
]
Don Bosco Durai commented on RANGER-1738:
-
bq. Velmurugan Periasamy, It's not possible to have RangerYarnAuthorizer work
with multiple versions, as with Hadoop 3.0.0 you have to implement methods that
take arguments that are only available in Hadoop 3.0.0.
[~coheigea], historically we have kept Ranger Admin backward compatible with
all plugin versions and the Plugin compatible with specific versions of the
component only. The reason being, plugins run within the address space of the
component process and it has to be compatible with the component signature. So
we recommend users to use the latest Ranger Admin, but for the plugins, use the
corresponding compatible Ranger Plugin version.
bq. There is a problem then when we call UserGroupInformation.createRemoteUser
in RangerStormAuthorizer, as the UGI in Hadoop 3.0.0 finds the KerberosUtil in
Storm instead of in Hadoop, and doesn't work as a result.
Since we use the shim architecture, we control what library the authorizer
should use. So ideally, we could use Hadoop 2.7 library within our plugin till
3.0 stabilizes. Do you feel, by packaging the libraries in ranger plugin, we
would be able to address your issue?
> RangerYarnAuthorizer not compatible with Hadoop-3.0.0
> -
>
> Key: RANGER-1738
> URL: https://issues.apache.org/jira/browse/RANGER-1738
> Project: Ranger
> Issue Type: Bug
> Components: plugins
>Affects Versions: 0.7.1
>Reporter: Hong Shen
>Assignee: Colm O hEigeartaigh
> Fix For: 1.0.0
>
> Attachments:
> 0001-RANGER-1738-RangerYarnAuthorizer-not-compatible-with.patch
>
>
> In the newest hadoop version 3.0.0, YarnAuthorizationProvider has changed.
> The new YarnAuthorizationProvider.java has change the methods checkPermission
> and setPermission,
> {code:title=YarnAuthorizationProvider.java|borderStyle=solid}
> /**
>* Check if user has the permission to access the target object.
>*
>* @param accessRequest
>* the request object which contains all the access context info.
>* @return true if user can access the object, otherwise false.
>*/
> public abstract boolean checkPermission(AccessRequest accessRequest);
> /**
>* Set permissions for the target object.
>*
>* @param permissions
>*A list of permissions on the target object.
>* @param ugi User who sets the permissions.
>*/
> public abstract void setPermission(List permissions,
> UserGroupInformation ugi);
> {code}
> But the RangerYarnAuthorizer extends YarnAuthorizationProvider impletement
> the old method.
> {code:title=RangerYarnAuthorizer.java|borderStyle=solid}
> @Override
> public void setPermission(PrivilegedEntity entity, Map AccessControlList> permission, UserGroupInformation ugi) {
>...
> @Override
> public boolean checkPermission(AccessType accessType, PrivilegedEntity
> entity, UserGroupInformation ugi) {
> {code}
> I think yarn plugin should also impletement the new method. I will add a
> patch for it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (RANGER-1644) Change the default Crypt Algo to use stronger cryptographic algo.
[ https://issues.apache.org/jira/browse/RANGER-1644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241180#comment-16241180 ] Don Bosco Durai commented on RANGER-1644: - [~ekovacs], thanks for clarifying. A couple of more questions: 1. Does this affect how user passwords are encrypted and stored in Ranger DB? 2. If so, what happens to those users who are already in the Ranger system? 3. Are there any other locations we might have issues decrypting/validating the password? Thanks Bosco > Change the default Crypt Algo to use stronger cryptographic algo. > -- > > Key: RANGER-1644 > URL: https://issues.apache.org/jira/browse/RANGER-1644 > Project: Ranger > Issue Type: Bug > Components: Ranger >Reporter: Selvamohan Neethiraj >Assignee: Endre Kovacs >Priority: Critical > Attachments: > 0001-RANGER-1644-replacing-MD5-DES-with-SHA512-AES128.patch > > > Change the default crypt algorithm to use a stronger cipher algorithm -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (RANGER-1837) Enhance Ranger Audit to HDFS to support ORC file format
[
https://issues.apache.org/jira/browse/RANGER-1837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16239272#comment-16239272
]
Don Bosco Durai commented on RANGER-1837:
-
[~rmani], I did a quick review of your patch and given my feedback on the
review board.
I have extracted the document feedback here:
1. We have to specify that the owner should be the same as the service process
user. E.g. for HiveServer2, the owner should be "hive"
2. We need to check with the ORC team to see what would be the ideal (raw) file
size. (This is for ORC configs)
[~risdenk], would anyone from your team be able to review and give comments.
Also, any testing of the patch after it is stable will be helpful. Thanks
> Enhance Ranger Audit to HDFS to support ORC file format
> ---
>
> Key: RANGER-1837
> URL: https://issues.apache.org/jira/browse/RANGER-1837
> Project: Ranger
> Issue Type: Improvement
> Components: audit
>Reporter: Kevin Risden
> Attachments:
> 0001-RANGER-1837-Enhance-Ranger-Audit-to-HDFS-to-support-.patch
>
>
> My team has done some research and found that Ranger HDFS audits are:
> * Stored as JSON objects (one per line)
> * Not compressed
> This is currently very verbose and would benefit from compression since this
> data is not frequently accessed.
> From Bosco on the mailing list:
> {quote}You are right, currently one of the options is saving the audits in
> HDFS itself as JSON files in one folder per day. I have loaded these JSON
> files from the folder into Hive as compressed ORC format. The compressed
> files in ORC were less than 10% of the original size. So, it was significant
> decrease in size. Also, it is easier to run analytics on the Hive tables.
>
> So, there are couple of ways of doing it.
>
> Write an Oozie job which runs every night and loads the previous day worth
> audit logs into ORC or other format
> Write a AuditDestination which can write into the format you want to.
>
> Regardless which approach you take, this would be a good feature for
> Ranger.{quote}
> http://mail-archives.apache.org/mod_mbox/ranger-user/201710.mbox/%3CCAJU9nmiYzzUUX1uDEysLAcMti4iLmX7RE%3DmN2%3DdoLaaQf87njQ%40mail.gmail.com%3E
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (RANGER-1837) HDFS Audit Compression
[
https://issues.apache.org/jira/browse/RANGER-1837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16226298#comment-16226298
]
Don Bosco Durai commented on RANGER-1837:
-
[~madhan.neethiraj], the 24 hour write period is for streaming to HDFS, right?
In this case, it would be a store and forward. In which case, it might be
better to have a shorter interval (e.g. 1 hour), so that the audit files are
sent to HDFS on regular basis.
> HDFS Audit Compression
> --
>
> Key: RANGER-1837
> URL: https://issues.apache.org/jira/browse/RANGER-1837
> Project: Ranger
> Issue Type: Improvement
> Components: audit
>Reporter: Kevin Risden
> Attachments: RANGER-1837-HDFS-Audit-Compression-0001.patch
>
>
> My team has done some research and found that Ranger HDFS audits are:
> * Stored as JSON objects (one per line)
> * Not compressed
> This is currently very verbose and would benefit from compression since this
> data is not frequently accessed.
> From Bosco on the mailing list:
> {quote}You are right, currently one of the options is saving the audits in
> HDFS itself as JSON files in one folder per day. I have loaded these JSON
> files from the folder into Hive as compressed ORC format. The compressed
> files in ORC were less than 10% of the original size. So, it was significant
> decrease in size. Also, it is easier to run analytics on the Hive tables.
>
> So, there are couple of ways of doing it.
>
> Write an Oozie job which runs every night and loads the previous day worth
> audit logs into ORC or other format
> Write a AuditDestination which can write into the format you want to.
>
> Regardless which approach you take, this would be a good feature for
> Ranger.{quote}
> http://mail-archives.apache.org/mod_mbox/ranger-user/201710.mbox/%3CCAJU9nmiYzzUUX1uDEysLAcMti4iLmX7RE%3DmN2%3DdoLaaQf87njQ%40mail.gmail.com%3E
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (RANGER-1837) HDFS Audit Compression
[
https://issues.apache.org/jira/browse/RANGER-1837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223820#comment-16223820
]
Don Bosco Durai commented on RANGER-1837:
-
I like the approach. A couple of questions:
1. Should HDFSAuditDestinationORC extend from HDFSAuditDestination? Or should
we create a common base class for batch File Output?
2. If we can create a common base class, then we can have the FileWriters
(HDFS/Text, HDFS/ORC, HDFS/Parquet, HDFS/Avro, etc.) extend from it. So all the
configuration, life cycle (batch time, rotation, recovery, etc) can be handled
by the base class and call the derived class on for writing. And the method can
take the input file and base folder for the output file.
> HDFS Audit Compression
> --
>
> Key: RANGER-1837
> URL: https://issues.apache.org/jira/browse/RANGER-1837
> Project: Ranger
> Issue Type: Improvement
> Components: audit
>Reporter: Kevin Risden
>
> My team has done some research and found that Ranger HDFS audits are:
> * Stored as JSON objects (one per line)
> * Not compressed
> This is currently very verbose and would benefit from compression since this
> data is not frequently accessed.
> From Bosco on the mailing list:
> {quote}You are right, currently one of the options is saving the audits in
> HDFS itself as JSON files in one folder per day. I have loaded these JSON
> files from the folder into Hive as compressed ORC format. The compressed
> files in ORC were less than 10% of the original size. So, it was significant
> decrease in size. Also, it is easier to run analytics on the Hive tables.
>
> So, there are couple of ways of doing it.
>
> Write an Oozie job which runs every night and loads the previous day worth
> audit logs into ORC or other format
> Write a AuditDestination which can write into the format you want to.
>
> Regardless which approach you take, this would be a good feature for
> Ranger.{quote}
> http://mail-archives.apache.org/mod_mbox/ranger-user/201710.mbox/%3CCAJU9nmiYzzUUX1uDEysLAcMti4iLmX7RE%3DmN2%3DdoLaaQf87njQ%40mail.gmail.com%3E
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (RANGER-1847) Ranger Kafka Plugin sasl.enabled.mechanisms=PLAIN
[ https://issues.apache.org/jira/browse/RANGER-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16214607#comment-16214607 ] Don Bosco Durai commented on RANGER-1847: - Yes, you should continue the discussion on the user mailing list. I personally have mostly used Kerberos with Kafka. But there might be others might have used your configuration. If not, one of can try to reproduce it. > Ranger Kafka Plugin sasl.enabled.mechanisms=PLAIN > - > > Key: RANGER-1847 > URL: https://issues.apache.org/jira/browse/RANGER-1847 > Project: Ranger > Issue Type: Test > Components: plugins >Affects Versions: 0.6.3, 0.7.1 > Environment: ubuntu stand-alone hobby environment >Reporter: Ronald van de Kuil >Priority: Minor > Fix For: 0.6.3 > > > I am such a NOOB hobby-ing away. And I like it. ;) > I figured I would give it a try to setup Kafka to use the > sasl.enabled.mechanisms of type PLAIN with ranger to do the authorisation and > the auditing (instead of GSSAPI). > I got it to work pretty far. KafkaServer gets into state SaslAuthenticated > with Zookeeper. > Next it loads the ranger kafka plugin. Then the RangerKafkaAuthorizer > complains about Kerberos. > I then updated the CLASSPATH and it complains about something else. > I am not sure how to classify this issue. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (RANGER-1644) Change the default Crypt Algo to use stronger cryptographic algo.
[ https://issues.apache.org/jira/browse/RANGER-1644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16214590#comment-16214590 ] Don Bosco Durai commented on RANGER-1644: - [~andrewsmith87], is there any backward compatibility concerns? Would it affect anyone with an existing installation of Ranger? > Change the default Crypt Algo to use stronger cryptographic algo. > -- > > Key: RANGER-1644 > URL: https://issues.apache.org/jira/browse/RANGER-1644 > Project: Ranger > Issue Type: Bug > Components: Ranger >Reporter: Selvamohan Neethiraj >Assignee: Endre Kovacs >Priority: Critical > Attachments: > 0001-RANGER-1644-replacing-MD5-DES-with-SHA512-AES128.patch > > > Change the default crypt algorithm to use a stronger cipher algorithm -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (RANGER-1672) Ranger supports plugin to enable, monitor and manage apache kylin
[ https://issues.apache.org/jira/browse/RANGER-1672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16214588#comment-16214588 ] Don Bosco Durai commented on RANGER-1672: - [~zhangqiang2], can you try now? > Ranger supports plugin to enable, monitor and manage apache kylin > - > > Key: RANGER-1672 > URL: https://issues.apache.org/jira/browse/RANGER-1672 > Project: Ranger > Issue Type: New Feature > Components: plugins >Reporter: Qiang Zhang >Assignee: Qiang Zhang > Labels: newbie, patch > Attachments: > 0001-RANGER-1672-Ranger-supports-plugin-to-enable-monitor.patch, > KylinAuditLog.jpg, KylinPlugins.jpg, KylinPolicies.jpg, > KylinServiceEntry.jpg, NewKylinPolicy.jpg, NewKylinService.jpg > > > Apache Kylin is an open source Distributed Analytics Engine designed to > provide SQL interface and multi-dimensional analysis (OLAP) on Hadoop > supporting extremely large datasets, original contributed from eBay Inc. > Apache Kylin lets user query massive data set at sub-second latency in 3 > steps. > 1. Identify a Star Schema on Hadoop. > 2. Build Cube from the identified tables. > 3. Query with ANSI-SQL and get results in sub-second, via ODBC, JDBC or > RESTful API. > We should support that using Ranger to control kylin's access rights for > project and cube. > Specific implementation plan is as following: > On the ranger website, administrators can configure policies to control user > access to projects and cube permissions. > Kylin provides an abstract class and authorization interfaces for use by the > ranger plugin. kylin instantiates ranger plugin’s implementation class when > starting(this class extends the abstract class provided by kylin). > Ranger plugin periodically polls ranger admin, updates the policy to the > local, and updates project and cube access rights based on policy information. > In the Kylin side： > 1. Kylin provides an abstract class that enables the ranger plugin's > implementation class to extend. > 2. Add configuration item. 1) ranger authorization switch, 2) ranger plugin > implementation class's name. > 3. Instantiate the ranger plugin implementation class when starting kylin. > 4. kylin provides authorization interfaces for ranger plugin calls. > 5. According to the ranger authorization configuration item, hide kylin's > authorization management page. > 6. Using ranger manager access rights of the kylin does not affect kylin's > existing permissions functions and logic. > In the Ranger side： > 1. Ranger plugin will periodically polls ranger admin, updates the policy to > the local. > 2. The ranger plugin invoking the authorization interfaces provided by kylin > to updates the project and cube access rights based on the policy information. > reference link:https://issues.apache.org/jira/browse/KYLIN-2703 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (RANGER-1847) Ranger Kafka Plugin sasl.enabled.mechanisms=PLAIN
[ https://issues.apache.org/jira/browse/RANGER-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16214017#comment-16214017 ] Don Bosco Durai commented on RANGER-1847: - [~rollantz], can I suggest to move this discussion to the u...@ranger.apache.org mailing list? We can close this JIRA and create one if there is an issue found which needs to be fixed. You might also have to provide additional information regarding what your overall setup is. In the case of Kafka, user level access control works only if you have Kerberos enabled, otherwise, Kafka doesn't know which user is making the call. If you don't have Kerberos enabled, then you can still use Ranger to enforce IP level permission checks. Ranger audits everything into Solr. If you are setting up manually, then I hope you followed this document https://cwiki.apache.org/confluence/display/RANGER/Install+and+Configure+Solr+for+Ranger+Audits+-+Apache+Ranger+0.5 to setup Solr and the plugins. If still not working, we might need to look into the log files. If you are just trying it out, then I recommend using Ambari to install Kafka and enable Ranger. The Ranger team has worked with the Ambari team to seamless integration between all the components in the Hadoop ecosystem. It is also very easy to update the properties centrally from Ambari and push it to Ranger or any of the components Ranger supports. > Ranger Kafka Plugin sasl.enabled.mechanisms=PLAIN > - > > Key: RANGER-1847 > URL: https://issues.apache.org/jira/browse/RANGER-1847 > Project: Ranger > Issue Type: Test > Components: plugins >Affects Versions: 0.6.3, 0.7.1 > Environment: ubuntu stand-alone hobby environment >Reporter: Ronald van de Kuil >Priority: Minor > > I am such a NOOB hobby-ing away. And I like it. ;) > I figured I would give it a try to setup Kafka to use the > sasl.enabled.mechanisms of type PLAIN with ranger to do the authorisation and > the auditing (instead of GSSAPI). > I got it to work pretty far. KafkaServer gets into state SaslAuthenticated > with Zookeeper. > Next it loads the ranger kafka plugin. Then the RangerKafkaAuthorizer > complains about Kerberos. > I then updated the CLASSPATH and it complains about something else. > I am not sure how to classify this issue. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (RANGER-1810) Ranger supports plugin to enable, monitor and manage apache Sqoop2
[ https://issues.apache.org/jira/browse/RANGER-1810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213752#comment-16213752 ] Don Bosco Durai commented on RANGER-1810: - [~zhangqiang2] I am excited to see the support for Sqoop in Ranger. Thanks for working on it. [~coheigea], thanks for all your review and feedbacks :-) > Ranger supports plugin to enable, monitor and manage apache Sqoop2 > -- > > Key: RANGER-1810 > URL: https://issues.apache.org/jira/browse/RANGER-1810 > Project: Ranger > Issue Type: New Feature > Components: plugins >Affects Versions: 1.0.0, master >Reporter: Qiang Zhang >Assignee: Qiang Zhang > Labels: newbie, patch > Attachments: > 0001-RANGER-1810-Ranger-supports-plugin-to-enable-monitor.patch, > 1_SqoopServiceManager.jpg, 2_EditSqoopService.jpg, 3_ListSqoopPolicies.jpg, > 4_EditSqoopPolicy.jpg, 5_SqoopAuditLog.jpg, 6_SqoopPlugins.jpg > > > Apache Sqoop is a tool designed for efficiently transferring bulk data > between Apache Hadoop and structured datastores such as relational databases. > You can use Sqoop to import data from external structured datastores into > Hadoop Distributed File System or related systems like Hive and HBase. > Conversely, Sqoop can be used to extract data from Hadoop and export it to > external structured datastores such as relational databases and enterprise > data warehouses.It successfully graduated from the Incubator in March of 2012 > and is now a Top-Level Apache project. > The Ranger will further expand the influence in the hadoop ecosystem if it > supports sqoop authorization. So we should develop sqoop plugin to enable, > monitor and manage apache Sqoop2. > Our test specialists have rigorously tested this feature. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (RANGER-1847) Ranger Kafka Plugin sasl.enabled.mechanisms=PLAIN
[ https://issues.apache.org/jira/browse/RANGER-1847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213751#comment-16213751 ] Don Bosco Durai commented on RANGER-1847: - [~rollantz], good it seems you were able to resolve it. Will you be able to document your experience? This will help others in the future. If you can, I can provide write access to Ranger Wiki or if you can blog somewhere, we can put that link in our Ranger Wiki. Thanks Bosco > Ranger Kafka Plugin sasl.enabled.mechanisms=PLAIN > - > > Key: RANGER-1847 > URL: https://issues.apache.org/jira/browse/RANGER-1847 > Project: Ranger > Issue Type: Test > Components: plugins >Affects Versions: 0.6.3, 0.7.1 > Environment: ubuntu stand-alone hobby environment >Reporter: Ronald van de Kuil >Priority: Minor > > I am such a NOOB hobby-ing away. And I like it. ;) > I figured I would give it a try to setup Kafka to use the > sasl.enabled.mechanisms of type PLAIN with ranger to do the authorisation and > the auditing (instead of GSSAPI). > I got it to work pretty far. KafkaServer gets into state SaslAuthenticated > with Zookeeper. > Next it loads the ranger kafka plugin. Then the RangerKafkaAuthorizer > complains about Kerberos. > I then updated the CLASSPATH and it complains about something else. > I am not sure how to classify this issue. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (RANGER-1851) Enhance Ranger Hive Plugin to support authorization for KILL QUERY command
[ https://issues.apache.org/jira/browse/RANGER-1851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213749#comment-16213749 ] Don Bosco Durai commented on RANGER-1851: - I feel this is a good thing. We should consider this as a design pattern for our other services where we need actions at non-resource level. Seems we have started having multiple cases, where the resources are at the same level (e.g. database/URL in Hive and now Service). Can we associate which "actions" are applicable at each top level? This will give a good experience for the users. This doesn't have to be part of this JIRA. > Enhance Ranger Hive Plugin to support authorization for KILL QUERY command > -- > > Key: RANGER-1851 > URL: https://issues.apache.org/jira/browse/RANGER-1851 > Project: Ranger > Issue Type: Bug > Components: Ranger >Affects Versions: master, 0.7.1 >Reporter: Ramesh Mani >Assignee: Ramesh Mani >Priority: Critical > > With the HIVE-17483 JIRA, Hive has introduced a way to kill query and > in hive its a privileged action for Hive Admin Role. In order for the Ranger > Hive Authorizer to support authorization, we need to enhance the ranger hive > authorizer. Current Hive implementation is to Kill Query in a HiveService > which can be LLAP / HIVESERVER2 , later these HIVE SERVICEs can be grouped > into NAME SPACEs and kill query can be run against them. When > HiveServer2/LLAP Ranger Plugin sends the request to Ranger for Authorization, > it will be sending the HIVE SERVICE in the context with the COMMAND that is > executed. > With all the details proposal is to have > 1) In Ranger Hive Service Definition, we will have a new Resource "Hive > Service" to authorize. > 2) In Ranger Hive Permission Model, we will have a new Permission "Service > Admin" to group Kill Query operation. > - "Service Admin" permission will enable hive ranger plugin to isolate > various admin operations in this case "Kill Query" and in future if hive > introduces other operations which are done at "HIVE SERVICE level" , group > them under this and authorize. >- "Service Admin" won't be able to do DATABASE / TABLE / COLUMN > operations as this will all be taken care by the existing > DATABASE/TABLE/COLUMN level permission model. > [~madhan.neethiraj] [~vperiasamy][~thejas][~bosco][~sneethiraj] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (RANGER-1829) RangerPolicy should use equals() to check equal for object of resources/policyItems/denyPolicyItems/allowExceptions/denyExceptions/dataMaskPolicyItems/rowFilterPolicyI
[ https://issues.apache.org/jira/browse/RANGER-1829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16204829#comment-16204829 ] Don Bosco Durai commented on RANGER-1829: - [~abhayk], it might be good to have some comments for such cases. Thanks > RangerPolicy should use equals() to check equal for object of > resources/policyItems/denyPolicyItems/allowExceptions/denyExceptions/dataMaskPolicyItems/rowFilterPolicyItems > --- > > Key: RANGER-1829 > URL: https://issues.apache.org/jira/browse/RANGER-1829 > Project: Ranger > Issue Type: Bug > Components: Ranger >Affects Versions: master >Reporter: Qiang Zhang >Assignee: Qiang Zhang >Priority: Minor > Attachments: > 0001-RANGER-1829-RangerPolicy-should-use-equals-to-check-.patch > > > When RangerPolicy check the equal for object of > resources/policyItems/denyPolicyItems/allowExceptions/denyExceptions/dataMaskPolicyItems/rowFilterPolicyItems, > it uses the "==" operator. > But it should not use "==", use "equals()" is correctly. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (RANGER-1780) Allow AuditSummaryQueue to aggregate events in the same directory
[ https://issues.apache.org/jira/browse/RANGER-1780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179914#comment-16179914 ] Don Bosco Durai commented on RANGER-1780: - [~afernandez], thanks for your extension. I have added you as a contributor to Apache Ranger and assigned this JIRA to you. Can you also follow the guidelines from [https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=55151244] to create a review board request. I have some specific questions on your requirement. Do you feel having a global "parent" level aggregation will be a bit harsh? I can understand, parent level folder for Hive table partition folders. But with globally, we might lose some granularity. Honestly, I can't think of any better way to configure this on selective resources, but I am open to suggestions. Also, this code is very generic for all applications. The resource definition for Kafka might be different than HDFS. So we might have to see we can address all of them. > Allow AuditSummaryQueue to aggregate events in the same directory > - > > Key: RANGER-1780 > URL: https://issues.apache.org/jira/browse/RANGER-1780 > Project: Ranger > Issue Type: Improvement > Components: audit >Affects Versions: 0.7.1 >Reporter: Alejandro Fernandez >Assignee: Alejandro Fernandez > Fix For: 1.0.0 > > Attachments: RANGER-1780.patch, ranger_summary.png > > > AuditSummaryQueue already has logic to enable the summarization, but it > requires 2 events to have the exact same resource path (plus a couple of > other fields such as user, access type, access result, action, client ip, > session). > This Jira is to add a config called > xasecure.audit.provider.summary.aggregate.level so that if it is set to > "directory" then 2 events can still be aggregated if they are files in the > same directory. > If the config is not specified its default value will be "file" which > preserves the existing behavior. > See [^ranger_summary.png] for screenshot on desired behavior. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (RANGER-1780) Allow AuditSummaryQueue to aggregate events in the same directory
[ https://issues.apache.org/jira/browse/RANGER-1780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Don Bosco Durai reassigned RANGER-1780: --- Assignee: Alejandro Fernandez > Allow AuditSummaryQueue to aggregate events in the same directory > - > > Key: RANGER-1780 > URL: https://issues.apache.org/jira/browse/RANGER-1780 > Project: Ranger > Issue Type: Improvement > Components: audit >Affects Versions: 0.7.1 >Reporter: Alejandro Fernandez >Assignee: Alejandro Fernandez > Fix For: 1.0.0 > > Attachments: RANGER-1780.patch, ranger_summary.png > > > AuditSummaryQueue already has logic to enable the summarization, but it > requires 2 events to have the exact same resource path (plus a couple of > other fields such as user, access type, access result, action, client ip, > session). > This Jira is to add a config called > xasecure.audit.provider.summary.aggregate.level so that if it is set to > "directory" then 2 events can still be aggregated if they are files in the > same directory. > If the config is not specified its default value will be "file" which > preserves the existing behavior. > See [^ranger_summary.png] for screenshot on desired behavior. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
