[jira] [Updated] (RANGER-4481) Add a configuration item to support Ranger client not using authentication
[ https://issues.apache.org/jira/browse/RANGER-4481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuze Yang updated RANGER-4481: -- Attachment: 0001-RANGER-4481-remove-the-matched-supported-s-entrySet-.patch > Add a configuration item to support Ranger client not using authentication > -- > > Key: RANGER-4481 > URL: https://issues.apache.org/jira/browse/RANGER-4481 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Affects Versions: 2.1.0 > Reporter: Xuze Yang >Assignee: Xuze Yang >Priority: Major > Attachments: > 0001-RANGER-4481-remove-the-matched-supported-s-entrySet-.patch, feasible > solution code.png, first http response.png, openjdk problem code.png, second > http request.png > > > As described in RANGER-3602, ranger supports downloading policies and roles > through unauthenticated http requests even if kerberos is enabled on the > server. > But in terms of the current implementation of RangerAdminRESTClient, whether > to enable authenticated HTTP requests depends on the service in which it is > located. For example, if the Hadoop service has kerberos enabled, then the > RangerAdminRESTClient in the HDFS and Yarn plugins will also use > authenticated HTTP requests. > I think this is not reasonable enough. In this case (both the Ranger server > and Hadoop are enabled for kerberos), the RangerAdminRESTClient of the HDFS > and Yarn plugins should also be allowed to download policies and roles > through unauthenticated HTTP requests. > The reason why I proposed this improvement is due to a bug I encountered in > our production environment. I will introduce the bug I encountered later. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (RANGER-4481) Add a configuration item to support Ranger client not using authentication
[
https://issues.apache.org/jira/browse/RANGER-4481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17783990#comment-17783990
]
Xuze Yang commented on RANGER-4481:
---
This problem can be reproduced everytime according to the following steps:
Environment: 2 ranger servers（called ranger0 and ranger1）, 1 kerberos
1. Restart the ranger client (to ensure that the ranger client has
interacted with only one ranger server)
2. Determine the range server that the range client reqeust to, assuming
it is range0
3. Make the kerberos unavailable
4. Make ranger0 unavailable
5. Make the kerberos available
A little explanation: In step 4, by making range0 unavailable, the ranger
client will switch to interacting with range1. After the first HTTP request
returns 401, the ranger client will request a ticket for ranger1 from kerberos.
However, due to kerberos's unavailable at this time, the ticket request failed.
And Because it's the first time for ranger client interact with ranger1,
NegotiateAuthentication#supported will put the entry: \{ranger1's hostname,
false}, and it will continue to return false in subsequent requests.
After applying the provided patch, the ranger client can pull policies after
kerberos returns to normal.
> Add a configuration item to support Ranger client not using authentication
> --
>
> Key: RANGER-4481
> URL: https://issues.apache.org/jira/browse/RANGER-4481
> Project: Ranger
> Issue Type: Improvement
> Components: Ranger
>Affects Versions: 2.1.0
> Reporter: Xuze Yang
>Assignee: Xuze Yang
>Priority: Major
> Attachments: feasible solution code.png, first http response.png,
> openjdk problem code.png, second http request.png
>
>
> As described in RANGER-3602, ranger supports downloading policies and roles
> through unauthenticated http requests even if kerberos is enabled on the
> server.
> But in terms of the current implementation of RangerAdminRESTClient, whether
> to enable authenticated HTTP requests depends on the service in which it is
> located. For example, if the Hadoop service has kerberos enabled, then the
> RangerAdminRESTClient in the HDFS and Yarn plugins will also use
> authenticated HTTP requests.
> I think this is not reasonable enough. In this case (both the Ranger server
> and Hadoop are enabled for kerberos), the RangerAdminRESTClient of the HDFS
> and Yarn plugins should also be allowed to download policies and roles
> through unauthenticated HTTP requests.
> The reason why I proposed this improvement is due to a bug I encountered in
> our production environment. I will introduce the bug I encountered later.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (RANGER-4481) Add a configuration item to support Ranger client not using authentication
[ https://issues.apache.org/jira/browse/RANGER-4481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuze Yang reassigned RANGER-4481: - Assignee: Xuze Yang > Add a configuration item to support Ranger client not using authentication > -- > > Key: RANGER-4481 > URL: https://issues.apache.org/jira/browse/RANGER-4481 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Affects Versions: 2.1.0 > Reporter: Xuze Yang >Assignee: Xuze Yang >Priority: Major > Attachments: feasible solution code.png, first http response.png, > openjdk problem code.png, second http request.png > > > As described in RANGER-3602, ranger supports downloading policies and roles > through unauthenticated http requests even if kerberos is enabled on the > server. > But in terms of the current implementation of RangerAdminRESTClient, whether > to enable authenticated HTTP requests depends on the service in which it is > located. For example, if the Hadoop service has kerberos enabled, then the > RangerAdminRESTClient in the HDFS and Yarn plugins will also use > authenticated HTTP requests. > I think this is not reasonable enough. In this case (both the Ranger server > and Hadoop are enabled for kerberos), the RangerAdminRESTClient of the HDFS > and Yarn plugins should also be allowed to download policies and roles > through unauthenticated HTTP requests. > The reason why I proposed this improvement is due to a bug I encountered in > our production environment. I will introduce the bug I encountered later. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-4481) Add a configuration item to support Ranger client not using authentication
[ https://issues.apache.org/jira/browse/RANGER-4481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuze Yang updated RANGER-4481: -- Attachment: (was: (Entête de la société).docx) > Add a configuration item to support Ranger client not using authentication > -- > > Key: RANGER-4481 > URL: https://issues.apache.org/jira/browse/RANGER-4481 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Affects Versions: 2.1.0 > Reporter: Xuze Yang >Priority: Major > Attachments: feasible solution code.png, first http response.png, > openjdk problem code.png, second http request.png > > > As described in RANGER-3602, ranger supports downloading policies and roles > through unauthenticated http requests even if kerberos is enabled on the > server. > But in terms of the current implementation of RangerAdminRESTClient, whether > to enable authenticated HTTP requests depends on the service in which it is > located. For example, if the Hadoop service has kerberos enabled, then the > RangerAdminRESTClient in the HDFS and Yarn plugins will also use > authenticated HTTP requests. > I think this is not reasonable enough. In this case (both the Ranger server > and Hadoop are enabled for kerberos), the RangerAdminRESTClient of the HDFS > and Yarn plugins should also be allowed to download policies and roles > through unauthenticated HTTP requests. > The reason why I proposed this improvement is due to a bug I encountered in > our production environment. I will introduce the bug I encountered later. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-4481) Add a configuration item to support Ranger client not using authentication
[ https://issues.apache.org/jira/browse/RANGER-4481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuze Yang updated RANGER-4481: -- Attachment: (Entête de la société).docx > Add a configuration item to support Ranger client not using authentication > -- > > Key: RANGER-4481 > URL: https://issues.apache.org/jira/browse/RANGER-4481 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Affects Versions: 2.1.0 > Reporter: Xuze Yang >Priority: Major > Attachments: (Entête de la société).docx, feasible solution > code.png, first http response.png, openjdk problem code.png, second http > request.png > > > As described in RANGER-3602, ranger supports downloading policies and roles > through unauthenticated http requests even if kerberos is enabled on the > server. > But in terms of the current implementation of RangerAdminRESTClient, whether > to enable authenticated HTTP requests depends on the service in which it is > located. For example, if the Hadoop service has kerberos enabled, then the > RangerAdminRESTClient in the HDFS and Yarn plugins will also use > authenticated HTTP requests. > I think this is not reasonable enough. In this case (both the Ranger server > and Hadoop are enabled for kerberos), the RangerAdminRESTClient of the HDFS > and Yarn plugins should also be allowed to download policies and roles > through unauthenticated HTTP requests. > The reason why I proposed this improvement is due to a bug I encountered in > our production environment. I will introduce the bug I encountered later. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (RANGER-4481) Add a configuration item to support Ranger client not using authentication
[ https://issues.apache.org/jira/browse/RANGER-4481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17776988#comment-17776988 ] Xuze Yang edited comment on RANGER-4481 at 10/26/23 2:52 AM: - Add a configuration item to enable RangerAdminRESTClient's getRolesIfUpdated()/getServicePoliciesIfUpdated()/getServiceTagsIfUpdated() use unauthenticated http request may involve a large amount of work. Because we should add this configuration item in all plugin component's configuration file. Another way, when the response code was 401, I tried to clear the supported cache through java reflection. This has been proven to be feasible. !feasible solution code.png! Now I don't know which modification method is more reasonable, or there are other better modification methods. [~madhan] [~kirbyzhou] was (Author: xuze yang): Add a configuration item to enable RangerAdminRESTClient's getRolesIfUpdated()/getServicePoliciesIfUpdated()/getServiceTagsIfUpdated() use unauthenticated http request may involve a large amount of work. Because we should add this configuration item in all plugin component's configuration file. Another way, when the response code was 401, I tried to clear the supported cache through java reflection. This has been proven to be feasible. !4.png! Now I don't know which modification method is more reasonable, or there are other better modification methods. [~madhan] [~kirbyzhou] > Add a configuration item to support Ranger client not using authentication > -- > > Key: RANGER-4481 > URL: https://issues.apache.org/jira/browse/RANGER-4481 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Affects Versions: 2.1.0 > Reporter: Xuze Yang >Priority: Major > Attachments: feasible solution code.png, first http response.png, > openjdk problem code.png, second http request.png > > > As described in RANGER-3602, ranger supports downloading policies and roles > through unauthenticated http requests even if kerberos is enabled on the > server. > But in terms of the current implementation of RangerAdminRESTClient, whether > to enable authenticated HTTP requests depends on the service in which it is > located. For example, if the Hadoop service has kerberos enabled, then the > RangerAdminRESTClient in the HDFS and Yarn plugins will also use > authenticated HTTP requests. > I think this is not reasonable enough. In this case (both the Ranger server > and Hadoop are enabled for kerberos), the RangerAdminRESTClient of the HDFS > and Yarn plugins should also be allowed to download policies and roles > through unauthenticated HTTP requests. > The reason why I proposed this improvement is due to a bug I encountered in > our production environment. I will introduce the bug I encountered later. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (RANGER-4481) Add a configuration item to support Ranger client not using authentication
[
https://issues.apache.org/jira/browse/RANGER-4481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17776958#comment-17776958
]
Xuze Yang edited comment on RANGER-4481 at 10/26/23 2:52 AM:
-
{*}1. Problem phenomenon{*}:
>From a certain moment on, the ranger plugin of resourcemanager0 has been
>unable to pull policies, and the error log is 401. The kerberos service is
>normal, and as a comparison, the ranger plugin of resourcemanager1 can pull
>policies normally. Resourcemanager0's error log:
{code:java}
2023-10-19 08:55:02,551 WARN
org.apache.ranger.admin.client.RangerAdminRESTClient: Error getting Roles.
secureMode=true,
user=hadoop/hdp-hadoop-hdp-resourcemanager-1.hdp-hadoop-hdp-resourcemanager.yangxuze.svc.cluster.lo...@dahua.com
(auth:KERBEROS), response={"httpStatusCode":401,"statusCode":0},
serviceName=default-Yarn
2023-10-19 08:55:02,552 WARN
org.apache.ranger.admin.client.RangerAdminRESTClient: Error getting policies.
secureMode=true,
user=hadoop/hdp-hadoop-hdp-resourcemanager-1.hdp-hadoop-hdp-resourcemanager.yangxuze.svc.cluster.lo...@dahua.com
(auth:KERBEROS), response={"httpStatusCode":401,"statusCode":0},
serviceName=default-Yarn
2023-10-19 08:55:12,572 WARN
org.apache.ranger.admin.client.RangerAdminRESTClient: Error getting Roles.
secureMode=true,
user=hadoop/hdp-hadoop-hdp-resourcemanager-1.hdp-hadoop-hdp-resourcemanager.yangxuze.svc.cluster.lo...@dahua.com
(auth:KERBEROS), response={"httpStatusCode":401,"statusCode":0},
serviceName=default-Yarn
2023-10-19 08:55:12,574 WARN
org.apache.ranger.admin.client.RangerAdminRESTClient: Error getting policies.
secureMode=true,
user=hadoop/hdp-hadoop-hdp-resourcemanager-1.hdp-hadoop-hdp-resourcemanager.yangxuze.svc.cluster.lo...@dahua.com
(auth:KERBEROS), response={"httpStatusCode":401,"statusCode":0},
serviceName=default-Yarn{code}
*2. Problem Cause Analysis：*
The normal process for the Ranger plugin to pull policies from the Ranger
server：
1) For the first request, the server will reply with a 401 and carry "WWW
Authenticate: Negotiate" in the HTTP header
!first http response.png!
2) The internal httpUrlConnection class of Openjdk handles this by performing a
kerberos authentication, and then placing the authentication result in the HTTP
header field Authorization to request again
!second http request.png!
3) Afterwards, the server can provide the correct response, such as a 200
response or a 304 response
abnormal:
When performing kerberos authentication, if the kdc service is abnormal, it
will be determined that the service does not support kerberos authentication,
and then this information will be stored in a global static map called
'supported'. Subsequent requests will be obtained from the 'supported' based on
the server hostname to determine whether authentication methods are supported
!openjdk problem code.png!
*Therefore, once the first failure occurs, it will be determined that the
service does not support authentication, and the second HTTP request will never
be triggered, resulting in the inability to obtain policies from the service
forever.*
was (Author: xuze yang):
{*}1. Problem phenomenon{*}:
>From a certain moment on, the ranger plugin of resourcemanager0 has been
>unable to pull policies, and the error log is 401. The kerberos service is
>normal, and as a comparison, the ranger plugin of resourcemanager1 can pull
>policies normally. Resourcemanager0's error log:
{code:java}
2023-10-19 08:55:02,551 WARN
org.apache.ranger.admin.client.RangerAdminRESTClient: Error getting Roles.
secureMode=true,
user=hadoop/hdp-hadoop-hdp-resourcemanager-1.hdp-hadoop-hdp-resourcemanager.yangxuze.svc.cluster.lo...@dahua.com
(auth:KERBEROS), response={"httpStatusCode":401,"statusCode":0},
serviceName=default-Yarn
2023-10-19 08:55:02,552 WARN
org.apache.ranger.admin.client.RangerAdminRESTClient: Error getting policies.
secureMode=true,
user=hadoop/hdp-hadoop-hdp-resourcemanager-1.hdp-hadoop-hdp-resourcemanager.yangxuze.svc.cluster.lo...@dahua.com
(auth:KERBEROS), response={"httpStatusCode":401,"statusCode":0},
serviceName=default-Yarn
2023-10-19 08:55:12,572 WARN
org.apache.ranger.admin.client.RangerAdminRESTClient: Error getting Roles.
secureMode=true,
user=hadoop/hdp-hadoop-hdp-resourcemanager-1.hdp-hadoop-hdp-resourcemanager.yangxuze.svc.cluster.lo...@dahua.com
(auth:KERBEROS), response={"httpStatusCode":401,"statusCode":0},
serviceName=default-Yarn
2023-10-19 08:55:12,574 WARN
org.apache.ranger.admin.client.RangerAdminRESTClient: Error getting policies.
secureMode=true,
user=hadoop/hdp-hadoop-hdp-resourcemanager-1.hdp-hadoop-hdp-resourcemanager.yangxuze.svc.cluster.lo...@dahua.com
(auth:KERBEROS), response={"httpS
[jira] [Updated] (RANGER-4481) Add a configuration item to support Ranger client not using authentication
[ https://issues.apache.org/jira/browse/RANGER-4481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuze Yang updated RANGER-4481: -- Attachment: (was: 2.png) > Add a configuration item to support Ranger client not using authentication > -- > > Key: RANGER-4481 > URL: https://issues.apache.org/jira/browse/RANGER-4481 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Affects Versions: 2.1.0 > Reporter: Xuze Yang >Priority: Major > Attachments: feasible solution code.png, first http response.png, > openjdk problem code.png, second http request.png > > > As described in RANGER-3602, ranger supports downloading policies and roles > through unauthenticated http requests even if kerberos is enabled on the > server. > But in terms of the current implementation of RangerAdminRESTClient, whether > to enable authenticated HTTP requests depends on the service in which it is > located. For example, if the Hadoop service has kerberos enabled, then the > RangerAdminRESTClient in the HDFS and Yarn plugins will also use > authenticated HTTP requests. > I think this is not reasonable enough. In this case (both the Ranger server > and Hadoop are enabled for kerberos), the RangerAdminRESTClient of the HDFS > and Yarn plugins should also be allowed to download policies and roles > through unauthenticated HTTP requests. > The reason why I proposed this improvement is due to a bug I encountered in > our production environment. I will introduce the bug I encountered later. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-4481) Add a configuration item to support Ranger client not using authentication
[ https://issues.apache.org/jira/browse/RANGER-4481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuze Yang updated RANGER-4481: -- Attachment: (was: 4.png) > Add a configuration item to support Ranger client not using authentication > -- > > Key: RANGER-4481 > URL: https://issues.apache.org/jira/browse/RANGER-4481 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Affects Versions: 2.1.0 > Reporter: Xuze Yang >Priority: Major > Attachments: feasible solution code.png, first http response.png, > openjdk problem code.png, second http request.png > > > As described in RANGER-3602, ranger supports downloading policies and roles > through unauthenticated http requests even if kerberos is enabled on the > server. > But in terms of the current implementation of RangerAdminRESTClient, whether > to enable authenticated HTTP requests depends on the service in which it is > located. For example, if the Hadoop service has kerberos enabled, then the > RangerAdminRESTClient in the HDFS and Yarn plugins will also use > authenticated HTTP requests. > I think this is not reasonable enough. In this case (both the Ranger server > and Hadoop are enabled for kerberos), the RangerAdminRESTClient of the HDFS > and Yarn plugins should also be allowed to download policies and roles > through unauthenticated HTTP requests. > The reason why I proposed this improvement is due to a bug I encountered in > our production environment. I will introduce the bug I encountered later. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-4481) Add a configuration item to support Ranger client not using authentication
[ https://issues.apache.org/jira/browse/RANGER-4481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuze Yang updated RANGER-4481: -- Attachment: feasible solution code.png first http response.png openjdk problem code.png second http request.png > Add a configuration item to support Ranger client not using authentication > -- > > Key: RANGER-4481 > URL: https://issues.apache.org/jira/browse/RANGER-4481 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Affects Versions: 2.1.0 > Reporter: Xuze Yang >Priority: Major > Attachments: feasible solution code.png, first http response.png, > openjdk problem code.png, second http request.png > > > As described in RANGER-3602, ranger supports downloading policies and roles > through unauthenticated http requests even if kerberos is enabled on the > server. > But in terms of the current implementation of RangerAdminRESTClient, whether > to enable authenticated HTTP requests depends on the service in which it is > located. For example, if the Hadoop service has kerberos enabled, then the > RangerAdminRESTClient in the HDFS and Yarn plugins will also use > authenticated HTTP requests. > I think this is not reasonable enough. In this case (both the Ranger server > and Hadoop are enabled for kerberos), the RangerAdminRESTClient of the HDFS > and Yarn plugins should also be allowed to download policies and roles > through unauthenticated HTTP requests. > The reason why I proposed this improvement is due to a bug I encountered in > our production environment. I will introduce the bug I encountered later. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-4481) Add a configuration item to support Ranger client not using authentication
[ https://issues.apache.org/jira/browse/RANGER-4481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuze Yang updated RANGER-4481: -- Attachment: (was: 1.png) > Add a configuration item to support Ranger client not using authentication > -- > > Key: RANGER-4481 > URL: https://issues.apache.org/jira/browse/RANGER-4481 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Affects Versions: 2.1.0 > Reporter: Xuze Yang >Priority: Major > Attachments: feasible solution code.png, first http response.png, > openjdk problem code.png, second http request.png > > > As described in RANGER-3602, ranger supports downloading policies and roles > through unauthenticated http requests even if kerberos is enabled on the > server. > But in terms of the current implementation of RangerAdminRESTClient, whether > to enable authenticated HTTP requests depends on the service in which it is > located. For example, if the Hadoop service has kerberos enabled, then the > RangerAdminRESTClient in the HDFS and Yarn plugins will also use > authenticated HTTP requests. > I think this is not reasonable enough. In this case (both the Ranger server > and Hadoop are enabled for kerberos), the RangerAdminRESTClient of the HDFS > and Yarn plugins should also be allowed to download policies and roles > through unauthenticated HTTP requests. > The reason why I proposed this improvement is due to a bug I encountered in > our production environment. I will introduce the bug I encountered later. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-4481) Add a configuration item to support Ranger client not using authentication
[ https://issues.apache.org/jira/browse/RANGER-4481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuze Yang updated RANGER-4481: -- Attachment: (was: 3.png) > Add a configuration item to support Ranger client not using authentication > -- > > Key: RANGER-4481 > URL: https://issues.apache.org/jira/browse/RANGER-4481 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Affects Versions: 2.1.0 > Reporter: Xuze Yang >Priority: Major > Attachments: feasible solution code.png, first http response.png, > openjdk problem code.png, second http request.png > > > As described in RANGER-3602, ranger supports downloading policies and roles > through unauthenticated http requests even if kerberos is enabled on the > server. > But in terms of the current implementation of RangerAdminRESTClient, whether > to enable authenticated HTTP requests depends on the service in which it is > located. For example, if the Hadoop service has kerberos enabled, then the > RangerAdminRESTClient in the HDFS and Yarn plugins will also use > authenticated HTTP requests. > I think this is not reasonable enough. In this case (both the Ranger server > and Hadoop are enabled for kerberos), the RangerAdminRESTClient of the HDFS > and Yarn plugins should also be allowed to download policies and roles > through unauthenticated HTTP requests. > The reason why I proposed this improvement is due to a bug I encountered in > our production environment. I will introduce the bug I encountered later. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (RANGER-4481) Add a configuration item to support Ranger client not using authentication
[ https://issues.apache.org/jira/browse/RANGER-4481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17776988#comment-17776988 ] Xuze Yang commented on RANGER-4481: --- Add a configuration item to enable RangerAdminRESTClient's getRolesIfUpdated()/getServicePoliciesIfUpdated()/getServiceTagsIfUpdated() use unauthenticated http request may involve a large amount of work. Because we should add this configuration item in all plugin component's configuration file. Another way, when the response code was 401, I tried to clear the supported cache through java reflection. This has been proven to be feasible. !4.png! Now I don't know which modification method is more reasonable, or there are other better modification methods. [~madhan] [~kirbyzhou] > Add a configuration item to support Ranger client not using authentication > -- > > Key: RANGER-4481 > URL: https://issues.apache.org/jira/browse/RANGER-4481 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Affects Versions: 2.1.0 > Reporter: Xuze Yang >Priority: Major > Attachments: 1.png, 2.png, 3.png, 4.png > > > As described in RANGER-3602, ranger supports downloading policies and roles > through unauthenticated http requests even if kerberos is enabled on the > server. > But in terms of the current implementation of RangerAdminRESTClient, whether > to enable authenticated HTTP requests depends on the service in which it is > located. For example, if the Hadoop service has kerberos enabled, then the > RangerAdminRESTClient in the HDFS and Yarn plugins will also use > authenticated HTTP requests. > I think this is not reasonable enough. In this case (both the Ranger server > and Hadoop are enabled for kerberos), the RangerAdminRESTClient of the HDFS > and Yarn plugins should also be allowed to download policies and roles > through unauthenticated HTTP requests. > The reason why I proposed this improvement is due to a bug I encountered in > our production environment. I will introduce the bug I encountered later. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-4481) Add a configuration item to support Ranger client not using authentication
[ https://issues.apache.org/jira/browse/RANGER-4481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuze Yang updated RANGER-4481: -- Attachment: 4.png > Add a configuration item to support Ranger client not using authentication > -- > > Key: RANGER-4481 > URL: https://issues.apache.org/jira/browse/RANGER-4481 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Affects Versions: 2.1.0 > Reporter: Xuze Yang >Priority: Major > Attachments: 1.png, 2.png, 3.png, 4.png > > > As described in RANGER-3602, ranger supports downloading policies and roles > through unauthenticated http requests even if kerberos is enabled on the > server. > But in terms of the current implementation of RangerAdminRESTClient, whether > to enable authenticated HTTP requests depends on the service in which it is > located. For example, if the Hadoop service has kerberos enabled, then the > RangerAdminRESTClient in the HDFS and Yarn plugins will also use > authenticated HTTP requests. > I think this is not reasonable enough. In this case (both the Ranger server > and Hadoop are enabled for kerberos), the RangerAdminRESTClient of the HDFS > and Yarn plugins should also be allowed to download policies and roles > through unauthenticated HTTP requests. > The reason why I proposed this improvement is due to a bug I encountered in > our production environment. I will introduce the bug I encountered later. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (RANGER-4481) Add a configuration item to support Ranger client not using authentication
[ https://issues.apache.org/jira/browse/RANGER-4481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17776964#comment-17776964 ] Xuze Yang commented on RANGER-4481: --- This is a bug in the openjdk code. Someone in the openjdk community has already raised an issue（[https://bugs.openjdk.org/browse/JDK-8208299]）, but the issue is still in an open state, and this issue still exists in the latest version of openjdk. Therefore, it is necessary for us to provide a method to avoid the problem on the ranger side. CC [~madhan] [~kirbyzhou] > Add a configuration item to support Ranger client not using authentication > -- > > Key: RANGER-4481 > URL: https://issues.apache.org/jira/browse/RANGER-4481 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Affects Versions: 2.1.0 > Reporter: Xuze Yang >Priority: Major > Attachments: 1.png, 2.png, 3.png > > > As described in RANGER-3602, ranger supports downloading policies and roles > through unauthenticated http requests even if kerberos is enabled on the > server. > But in terms of the current implementation of RangerAdminRESTClient, whether > to enable authenticated HTTP requests depends on the service in which it is > located. For example, if the Hadoop service has kerberos enabled, then the > RangerAdminRESTClient in the HDFS and Yarn plugins will also use > authenticated HTTP requests. > I think this is not reasonable enough. In this case (both the Ranger server > and Hadoop are enabled for kerberos), the RangerAdminRESTClient of the HDFS > and Yarn plugins should also be allowed to download policies and roles > through unauthenticated HTTP requests. > The reason why I proposed this improvement is due to a bug I encountered in > our production environment. I will introduce the bug I encountered later. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (RANGER-4481) Add a configuration item to support Ranger client not using authentication
[
https://issues.apache.org/jira/browse/RANGER-4481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17776958#comment-17776958
]
Xuze Yang edited comment on RANGER-4481 at 10/19/23 1:25 AM:
-
{*}1. Problem phenomenon{*}:
>From a certain moment on, the ranger plugin of resourcemanager0 has been
>unable to pull policies, and the error log is 401. The kerberos service is
>normal, and as a comparison, the ranger plugin of resourcemanager1 can pull
>policies normally. Resourcemanager0's error log:
{code:java}
2023-10-19 08:55:02,551 WARN
org.apache.ranger.admin.client.RangerAdminRESTClient: Error getting Roles.
secureMode=true,
user=hadoop/hdp-hadoop-hdp-resourcemanager-1.hdp-hadoop-hdp-resourcemanager.yangxuze.svc.cluster.lo...@dahua.com
(auth:KERBEROS), response={"httpStatusCode":401,"statusCode":0},
serviceName=default-Yarn
2023-10-19 08:55:02,552 WARN
org.apache.ranger.admin.client.RangerAdminRESTClient: Error getting policies.
secureMode=true,
user=hadoop/hdp-hadoop-hdp-resourcemanager-1.hdp-hadoop-hdp-resourcemanager.yangxuze.svc.cluster.lo...@dahua.com
(auth:KERBEROS), response={"httpStatusCode":401,"statusCode":0},
serviceName=default-Yarn
2023-10-19 08:55:12,572 WARN
org.apache.ranger.admin.client.RangerAdminRESTClient: Error getting Roles.
secureMode=true,
user=hadoop/hdp-hadoop-hdp-resourcemanager-1.hdp-hadoop-hdp-resourcemanager.yangxuze.svc.cluster.lo...@dahua.com
(auth:KERBEROS), response={"httpStatusCode":401,"statusCode":0},
serviceName=default-Yarn
2023-10-19 08:55:12,574 WARN
org.apache.ranger.admin.client.RangerAdminRESTClient: Error getting policies.
secureMode=true,
user=hadoop/hdp-hadoop-hdp-resourcemanager-1.hdp-hadoop-hdp-resourcemanager.yangxuze.svc.cluster.lo...@dahua.com
(auth:KERBEROS), response={"httpStatusCode":401,"statusCode":0},
serviceName=default-Yarn{code}
*2. Problem Cause Analysis：*
The normal process for the Ranger plugin to pull policies from the Ranger
server：
1) For the first request, the server will reply with a 401 and carry "WWW
Authenticate: Negotiate" in the HTTP header
!1.png!
2) The internal httpUrlConnection class of Openjdk handles this by performing a
kerberos authentication, and then placing the authentication result in the HTTP
header field Authorization to request again
!2.png!
3) Afterwards, the server can provide the correct response, such as a 200
response or a 304 response
abnormal:
When performing kerberos authentication, if the kdc service is abnormal, it
will be determined that the service does not support kerberos authentication,
and then this information will be stored in a global static map called
'supported'. Subsequent requests will be obtained from the 'supported' based on
the server hostname to determine whether authentication methods are supported
!3.png!
*Therefore, once the first failure occurs, it will be determined that the
service does not support authentication, and the second HTTP request will never
be triggered, resulting in the inability to obtain policies from the service
forever.*
was (Author: xuze yang):
{*}1. Problem phenomenon{*}:
>From a certain moment on, the ranger plugin of resourcemanager0 has been
>unable to pull policies, and the error log is 401. The kerberos service is
>normal, and as a comparison, the ranger plugin of resourcemanager1 can pull
>policies normally. Resourcemanager0's error log:
{code:java}
2023-10-19 08:55:02,551 WARN
org.apache.ranger.admin.client.RangerAdminRESTClient: Error getting Roles.
secureMode=true,
user=hadoop/hdp-hadoop-hdp-resourcemanager-1.hdp-hadoop-hdp-resourcemanager.yangxuze.svc.cluster.lo...@dahua.com
(auth:KERBEROS), response={"httpStatusCode":401,"statusCode":0},
serviceName=default-Yarn
2023-10-19 08:55:02,552 WARN
org.apache.ranger.admin.client.RangerAdminRESTClient: Error getting policies.
secureMode=true,
user=hadoop/hdp-hadoop-hdp-resourcemanager-1.hdp-hadoop-hdp-resourcemanager.yangxuze.svc.cluster.lo...@dahua.com
(auth:KERBEROS), response={"httpStatusCode":401,"statusCode":0},
serviceName=default-Yarn
2023-10-19 08:55:12,572 WARN
org.apache.ranger.admin.client.RangerAdminRESTClient: Error getting Roles.
secureMode=true,
user=hadoop/hdp-hadoop-hdp-resourcemanager-1.hdp-hadoop-hdp-resourcemanager.yangxuze.svc.cluster.lo...@dahua.com
(auth:KERBEROS), response={"httpStatusCode":401,"statusCode":0},
serviceName=default-Yarn
2023-10-19 08:55:12,574 WARN
org.apache.ranger.admin.client.RangerAdminRESTClient: Error getting policies.
secureMode=true,
user=hadoop/hdp-hadoop-hdp-resourcemanager-1.hdp-hadoop-hdp-resourcemanager.yangxuze.svc.cluster.lo...@dahua.com
(auth:KERBEROS), response={"httpStatusCode":401,"statusCode":0},
serviceNa
[jira] [Comment Edited] (RANGER-4481) Add a configuration item to support Ranger client not using authentication
[
https://issues.apache.org/jira/browse/RANGER-4481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17776958#comment-17776958
]
Xuze Yang edited comment on RANGER-4481 at 10/19/23 1:21 AM:
-
{*}1. Problem phenomenon{*}:
>From a certain moment on, the ranger plugin of resourcemanager0 has been
>unable to pull policies, and the error log is 401. The kerberos service is
>normal, and as a comparison, the ranger plugin of resourcemanager1 can pull
>policies normally. Resourcemanager0's error log:
{code:java}
2023-10-19 08:55:02,551 WARN
org.apache.ranger.admin.client.RangerAdminRESTClient: Error getting Roles.
secureMode=true,
user=hadoop/hdp-hadoop-hdp-resourcemanager-1.hdp-hadoop-hdp-resourcemanager.yangxuze.svc.cluster.lo...@dahua.com
(auth:KERBEROS), response={"httpStatusCode":401,"statusCode":0},
serviceName=default-Yarn
2023-10-19 08:55:02,552 WARN
org.apache.ranger.admin.client.RangerAdminRESTClient: Error getting policies.
secureMode=true,
user=hadoop/hdp-hadoop-hdp-resourcemanager-1.hdp-hadoop-hdp-resourcemanager.yangxuze.svc.cluster.lo...@dahua.com
(auth:KERBEROS), response={"httpStatusCode":401,"statusCode":0},
serviceName=default-Yarn
2023-10-19 08:55:12,572 WARN
org.apache.ranger.admin.client.RangerAdminRESTClient: Error getting Roles.
secureMode=true,
user=hadoop/hdp-hadoop-hdp-resourcemanager-1.hdp-hadoop-hdp-resourcemanager.yangxuze.svc.cluster.lo...@dahua.com
(auth:KERBEROS), response={"httpStatusCode":401,"statusCode":0},
serviceName=default-Yarn
2023-10-19 08:55:12,574 WARN
org.apache.ranger.admin.client.RangerAdminRESTClient: Error getting policies.
secureMode=true,
user=hadoop/hdp-hadoop-hdp-resourcemanager-1.hdp-hadoop-hdp-resourcemanager.yangxuze.svc.cluster.lo...@dahua.com
(auth:KERBEROS), response={"httpStatusCode":401,"statusCode":0},
serviceName=default-Yarn{code}
*2. Problem Cause Analysis：*
The normal process for the Ranger plugin to pull policies from the Ranger
server：
1) For the first request, the server will reply with a 401 and carry "WWW
Authenticate: Negotiate" in the HTTP header
!1.png!
2) The internal httpUrlConnection class of Openjdk handles this by performing a
kerberos authentication, and then placing the authentication result in the HTTP
header field Authorization to request again
!2.png!
3) Afterwards, the server can provide the correct response, such as a 200
response or a 304 response
When performing kerberos authentication, if the kdc service is abnormal, it
will be determined that the service does not support kerberos authentication,
and then this information will be stored in a global static map called
'supported'. Subsequent requests will be obtained from the 'supported' based on
the server hostname to determine whether authentication methods are supported
!3.png!
*Therefore, once the first failure occurs, it will be determined that the
service does not support authentication, and the second HTTP request will never
be triggered, resulting in the inability to obtain policies from the service
forever.*
was (Author: xuze yang):
{*}Problem phenomenon{*}:
>From a certain moment on, the ranger plugin of resourcemanager0 has been
>unable to pull policies, and the error log is 401. The kerberos service is
>normal, and as a comparison, the ranger plugin of resourcemanager1 can pull
>policies normally. Resourcemanager0's error log:
{code:java}
2023-10-19 08:55:02,551 WARN
org.apache.ranger.admin.client.RangerAdminRESTClient: Error getting Roles.
secureMode=true,
user=hadoop/hdp-hadoop-hdp-resourcemanager-1.hdp-hadoop-hdp-resourcemanager.yangxuze.svc.cluster.lo...@dahua.com
(auth:KERBEROS), response={"httpStatusCode":401,"statusCode":0},
serviceName=default-Yarn
2023-10-19 08:55:02,552 WARN
org.apache.ranger.admin.client.RangerAdminRESTClient: Error getting policies.
secureMode=true,
user=hadoop/hdp-hadoop-hdp-resourcemanager-1.hdp-hadoop-hdp-resourcemanager.yangxuze.svc.cluster.lo...@dahua.com
(auth:KERBEROS), response={"httpStatusCode":401,"statusCode":0},
serviceName=default-Yarn
2023-10-19 08:55:12,572 WARN
org.apache.ranger.admin.client.RangerAdminRESTClient: Error getting Roles.
secureMode=true,
user=hadoop/hdp-hadoop-hdp-resourcemanager-1.hdp-hadoop-hdp-resourcemanager.yangxuze.svc.cluster.lo...@dahua.com
(auth:KERBEROS), response={"httpStatusCode":401,"statusCode":0},
serviceName=default-Yarn
2023-10-19 08:55:12,574 WARN
org.apache.ranger.admin.client.RangerAdminRESTClient: Error getting policies.
secureMode=true,
user=hadoop/hdp-hadoop-hdp-resourcemanager-1.hdp-hadoop-hdp-resourcemanager.yangxuze.svc.cluster.lo...@dahua.com
(auth:KERBEROS), response={"httpStatusCode":401,"statusCode":0},
serviceName=default-Y
[jira] [Commented] (RANGER-4481) Add a configuration item to support Ranger client not using authentication
[
https://issues.apache.org/jira/browse/RANGER-4481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17776958#comment-17776958
]
Xuze Yang commented on RANGER-4481:
---
{*}Problem phenomenon{*}:
>From a certain moment on, the ranger plugin of resourcemanager0 has been
>unable to pull policies, and the error log is 401. The kerberos service is
>normal, and as a comparison, the ranger plugin of resourcemanager1 can pull
>policies normally. Resourcemanager0's error log:
{code:java}
2023-10-19 08:55:02,551 WARN
org.apache.ranger.admin.client.RangerAdminRESTClient: Error getting Roles.
secureMode=true,
user=hadoop/hdp-hadoop-hdp-resourcemanager-1.hdp-hadoop-hdp-resourcemanager.yangxuze.svc.cluster.lo...@dahua.com
(auth:KERBEROS), response={"httpStatusCode":401,"statusCode":0},
serviceName=default-Yarn
2023-10-19 08:55:02,552 WARN
org.apache.ranger.admin.client.RangerAdminRESTClient: Error getting policies.
secureMode=true,
user=hadoop/hdp-hadoop-hdp-resourcemanager-1.hdp-hadoop-hdp-resourcemanager.yangxuze.svc.cluster.lo...@dahua.com
(auth:KERBEROS), response={"httpStatusCode":401,"statusCode":0},
serviceName=default-Yarn
2023-10-19 08:55:12,572 WARN
org.apache.ranger.admin.client.RangerAdminRESTClient: Error getting Roles.
secureMode=true,
user=hadoop/hdp-hadoop-hdp-resourcemanager-1.hdp-hadoop-hdp-resourcemanager.yangxuze.svc.cluster.lo...@dahua.com
(auth:KERBEROS), response={"httpStatusCode":401,"statusCode":0},
serviceName=default-Yarn
2023-10-19 08:55:12,574 WARN
org.apache.ranger.admin.client.RangerAdminRESTClient: Error getting policies.
secureMode=true,
user=hadoop/hdp-hadoop-hdp-resourcemanager-1.hdp-hadoop-hdp-resourcemanager.yangxuze.svc.cluster.lo...@dahua.com
(auth:KERBEROS), response={"httpStatusCode":401,"statusCode":0},
serviceName=default-Yarn{code}
*Problem Cause Analysis：*
The normal process for the Ranger plugin to pull policies from the Ranger
server：
1) For the first request, the server will reply with a 401 and carry "WWW
Authenticate: Negotiate" in the HTTP header
!1.png!
2) The internal httpUrlConnection class of Openjdk handles this by performing a
kerberos authentication, and then placing the authentication result in the HTTP
header field Authorization to request again
!2.png!
3) Afterwards, the server can provide the correct response, such as a 200
response or a 304 response
When performing kerberos authentication, if the kdc service is abnormal, it
will be determined that the service does not support kerberos authentication,
and then this information will be stored in a global static map called
'supported'. Subsequent requests will be obtained from the 'supported' based on
the server hostname to determine whether authentication methods are supported
!3.png!
*Therefore, once the first failure occurs, it will be determined that the
service does not support authentication, and the second HTTP request will never
be triggered, resulting in the inability to obtain policies from the service
forever.*
> Add a configuration item to support Ranger client not using authentication
> --
>
> Key: RANGER-4481
> URL: https://issues.apache.org/jira/browse/RANGER-4481
> Project: Ranger
> Issue Type: Improvement
> Components: Ranger
>Affects Versions: 2.1.0
>Reporter: Xuze Yang
>Priority: Major
> Attachments: 1.png, 2.png, 3.png
>
>
> As described in RANGER-3602, ranger supports downloading policies and roles
> through unauthenticated http requests even if kerberos is enabled on the
> server.
> But in terms of the current implementation of RangerAdminRESTClient, whether
> to enable authenticated HTTP requests depends on the service in which it is
> located. For example, if the Hadoop service has kerberos enabled, then the
> RangerAdminRESTClient in the HDFS and Yarn plugins will also use
> authenticated HTTP requests.
> I think this is not reasonable enough. In this case (both the Ranger server
> and Hadoop are enabled for kerberos), the RangerAdminRESTClient of the HDFS
> and Yarn plugins should also be allowed to download policies and roles
> through unauthenticated HTTP requests.
> The reason why I proposed this improvement is due to a bug I encountered in
> our production environment. I will introduce the bug I encountered later.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (RANGER-4481) Add a configuration item to support Ranger client not using authentication
[ https://issues.apache.org/jira/browse/RANGER-4481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuze Yang updated RANGER-4481: -- Attachment: 3.png > Add a configuration item to support Ranger client not using authentication > -- > > Key: RANGER-4481 > URL: https://issues.apache.org/jira/browse/RANGER-4481 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Affects Versions: 2.1.0 > Reporter: Xuze Yang >Priority: Major > Attachments: 1.png, 2.png, 3.png > > > As described in RANGER-3602, ranger supports downloading policies and roles > through unauthenticated http requests even if kerberos is enabled on the > server. > But in terms of the current implementation of RangerAdminRESTClient, whether > to enable authenticated HTTP requests depends on the service in which it is > located. For example, if the Hadoop service has kerberos enabled, then the > RangerAdminRESTClient in the HDFS and Yarn plugins will also use > authenticated HTTP requests. > I think this is not reasonable enough. In this case (both the Ranger server > and Hadoop are enabled for kerberos), the RangerAdminRESTClient of the HDFS > and Yarn plugins should also be allowed to download policies and roles > through unauthenticated HTTP requests. > The reason why I proposed this improvement is due to a bug I encountered in > our production environment. I will introduce the bug I encountered later. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-4481) Add a configuration item to support Ranger client not using authentication
[ https://issues.apache.org/jira/browse/RANGER-4481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuze Yang updated RANGER-4481: -- Attachment: 1.png > Add a configuration item to support Ranger client not using authentication > -- > > Key: RANGER-4481 > URL: https://issues.apache.org/jira/browse/RANGER-4481 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Affects Versions: 2.1.0 > Reporter: Xuze Yang >Priority: Major > Attachments: 1.png, 2.png > > > As described in RANGER-3602, ranger supports downloading policies and roles > through unauthenticated http requests even if kerberos is enabled on the > server. > But in terms of the current implementation of RangerAdminRESTClient, whether > to enable authenticated HTTP requests depends on the service in which it is > located. For example, if the Hadoop service has kerberos enabled, then the > RangerAdminRESTClient in the HDFS and Yarn plugins will also use > authenticated HTTP requests. > I think this is not reasonable enough. In this case (both the Ranger server > and Hadoop are enabled for kerberos), the RangerAdminRESTClient of the HDFS > and Yarn plugins should also be allowed to download policies and roles > through unauthenticated HTTP requests. > The reason why I proposed this improvement is due to a bug I encountered in > our production environment. I will introduce the bug I encountered later. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-4481) Add a configuration item to support Ranger client not using authentication
[ https://issues.apache.org/jira/browse/RANGER-4481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuze Yang updated RANGER-4481: -- Attachment: 2.png > Add a configuration item to support Ranger client not using authentication > -- > > Key: RANGER-4481 > URL: https://issues.apache.org/jira/browse/RANGER-4481 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Affects Versions: 2.1.0 > Reporter: Xuze Yang >Priority: Major > Attachments: 1.png, 2.png > > > As described in RANGER-3602, ranger supports downloading policies and roles > through unauthenticated http requests even if kerberos is enabled on the > server. > But in terms of the current implementation of RangerAdminRESTClient, whether > to enable authenticated HTTP requests depends on the service in which it is > located. For example, if the Hadoop service has kerberos enabled, then the > RangerAdminRESTClient in the HDFS and Yarn plugins will also use > authenticated HTTP requests. > I think this is not reasonable enough. In this case (both the Ranger server > and Hadoop are enabled for kerberos), the RangerAdminRESTClient of the HDFS > and Yarn plugins should also be allowed to download policies and roles > through unauthenticated HTTP requests. > The reason why I proposed this improvement is due to a bug I encountered in > our production environment. I will introduce the bug I encountered later. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-4481) Add a configuration item to support Ranger client not using authentication
[ https://issues.apache.org/jira/browse/RANGER-4481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuze Yang updated RANGER-4481: -- Description: As described in RANGER-3602, ranger supports downloading policies and roles through unauthenticated http requests even if kerberos is enabled on the server. But in terms of the current implementation of RangerAdminRESTClient, whether to enable authenticated HTTP requests depends on the service in which it is located. For example, if the Hadoop service has kerberos enabled, then the RangerAdminRESTClient in the HDFS and Yarn plugins will also use authenticated HTTP requests. I think this is not reasonable enough. In this case (both the Ranger server and Hadoop are enabled for kerberos), the RangerAdminRESTClient of the HDFS and Yarn plugins should also be allowed to download policies and roles through unauthenticated HTTP requests. The reason why I proposed this improvement is due to a bug I encountered in our production environment. I will introduce the bug I encountered later. was: As described in [RANGER-3602|https://issues.apache.org/jira/browse/RANGER-3602], ranger supports downloading policies and roles through unauthenticated http requests even if kerberos is enabled on the server. But in terms of the current implementation of RangerAdminRESTClient, whether to enable authenticated HTTP requests depends on the service in which it is located. For example, if the Hadoop service has kerberos enabled, then the RangerAdminRESTClient in the HDFS and Yarn plugins will also use authenticated HTTP requests. I think this is not reasonable enough. In this case (both the Ranger server and Hadoop are enabled for kerberos), the RangerAdminRESTClient of the HDFS and Yarn plugins should also be allowed to download policies and roles through unauthenticated HTTP requests. > Add a configuration item to support Ranger client not using authentication > -- > > Key: RANGER-4481 > URL: https://issues.apache.org/jira/browse/RANGER-4481 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Affects Versions: 2.1.0 > Reporter: Xuze Yang >Priority: Major > > As described in RANGER-3602, ranger supports downloading policies and roles > through unauthenticated http requests even if kerberos is enabled on the > server. > But in terms of the current implementation of RangerAdminRESTClient, whether > to enable authenticated HTTP requests depends on the service in which it is > located. For example, if the Hadoop service has kerberos enabled, then the > RangerAdminRESTClient in the HDFS and Yarn plugins will also use > authenticated HTTP requests. > I think this is not reasonable enough. In this case (both the Ranger server > and Hadoop are enabled for kerberos), the RangerAdminRESTClient of the HDFS > and Yarn plugins should also be allowed to download policies and roles > through unauthenticated HTTP requests. > The reason why I proposed this improvement is due to a bug I encountered in > our production environment. I will introduce the bug I encountered later. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (RANGER-4481) Add a configuration item to support Ranger client not using authentication
Xuze Yang created RANGER-4481: - Summary: Add a configuration item to support Ranger client not using authentication Key: RANGER-4481 URL: https://issues.apache.org/jira/browse/RANGER-4481 Project: Ranger Issue Type: Improvement Components: Ranger Affects Versions: 2.1.0 Reporter: Xuze Yang As described in [RANGER-3602|https://issues.apache.org/jira/browse/RANGER-3602], ranger supports downloading policies and roles through unauthenticated http requests even if kerberos is enabled on the server. But in terms of the current implementation of RangerAdminRESTClient, whether to enable authenticated HTTP requests depends on the service in which it is located. For example, if the Hadoop service has kerberos enabled, then the RangerAdminRESTClient in the HDFS and Yarn plugins will also use authenticated HTTP requests. I think this is not reasonable enough. In this case (both the Ranger server and Hadoop are enabled for kerberos), the RangerAdminRESTClient of the HDFS and Yarn plugins should also be allowed to download policies and roles through unauthenticated HTTP requests. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-3326) get db audit log failed via rest api
[
https://issues.apache.org/jira/browse/RANGER-3326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuze Yang updated RANGER-3326:
--
Attachment: hdfs命令.pdf
> get db audit log failed via rest api
>
>
> Key: RANGER-3326
> URL: https://issues.apache.org/jira/browse/RANGER-3326
> Project: Ranger
> Issue Type: Bug
> Components: audit
>Affects Versions: 2.1.0
> Reporter: Xuze Yang
>Priority: Major
> Attachments: hdfs命令.pdf
>
>
> Using ranger2.1.0, we store audit logs in db(although log in db is deprecated
> since ranger0.6, we still use it because we don't want to introduce and
> maintain a new component like solr or elasticSearch). The audit log
> information can be successfully stored in db. But when we want to get audit
> logs via REST api(/service/assets/accessAudit), it throws following
> exceptions:
> {code:java}
> Jul 01, 2021 5:09:59 PM com.sun.jersey.spi.container.ContainerResponse
> mapMappableContainerExceptionJul 01, 2021 5:09:59 PM
> com.sun.jersey.spi.container.ContainerResponse
> mapMappableContainerExceptionSEVERE: The RuntimeException could not be mapped
> to a response, re-throwing to the HTTP containerjava.lang.ClassCastException:
> java.util.ArrayList cannot be cast to java.lang.String at
> org.apache.ranger.common.SearchUtil.buildWhereClause(SearchUtil.java:518) at
> org.apache.ranger.common.SearchUtil.buildWhereClause(SearchUtil.java:384) at
> org.apache.ranger.common.SearchUtil.createSearchQuery(SearchUtil.java:708) at
> org.apache.ranger.service.AbstractBaseResourceService.createQuery(AbstractBaseResourceService.java:519)
> at
> org.apache.ranger.service.AbstractBaseResourceService.getCountForSearchQuery(AbstractBaseResourceService.java:535)
> at
> org.apache.ranger.service.AbstractBaseResourceService.searchResources(AbstractBaseResourceService.java:564)
> at
> org.apache.ranger.service.XAccessAuditService.searchXAccessAudits(XAccessAuditService.java:172)
> at org.apache.ranger.biz.AssetMgr.getAccessLogs(AssetMgr.java:1132) at
> org.apache.ranger.rest.AssetREST.getAccessLogs(AssetREST.java:671) at
> org.apache.ranger.rest.AssetREST$$FastClassBySpringCGLIB$$8cffcb6d.invoke()
> at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
> at
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:737)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
> at
> org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:69)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
> at
> org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
> at
> org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:283)
> at
> org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
> at
> org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:672)
> at
> org.apache.ranger.rest.AssetREST$$EnhancerBySpringCGLIB$$9c5e18ed.getAccessLogs()
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPa
[jira] [Updated] (RANGER-3946) ranger-yarn-plugin missing commons-lang-2.6.jar
[
https://issues.apache.org/jira/browse/RANGER-3946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuze Yang updated RANGER-3946:
--
Attachment: 0001-RANGER-3946-ranger-yarn-plugin-missing-commons-lang-.patch
> ranger-yarn-plugin missing commons-lang-2.6.jar
> ---
>
> Key: RANGER-3946
> URL: https://issues.apache.org/jira/browse/RANGER-3946
> Project: Ranger
> Issue Type: Bug
> Components: plugins
>Affects Versions: 2.1.0
> Reporter: Xuze Yang
>Assignee: Xuze Yang
>Priority: Major
> Attachments:
> 0001-RANGER-3946-ranger-yarn-plugin-missing-commons-lang-.patch
>
>
> when implement ranger-2.1.0-yarn-plugin in hadoop-3.2.4. a error occured
> shown below:
> {code:java}
> 2022-10-14 15:12:54,561 INFO
> org.apache.ranger.plugin.classloader.RangerPluginClassLoaderUtil:
> getFilesInDirectory('/cloud/service/ranger-2.1.0-yarn-plugin/lib/ranger-yarn-plugin-impl'):
> adding
> /cloud/service/ranger-2.1.0-yarn-plugin/lib/ranger-yarn-plugin-impl/solr-solrj-7.7.1.jar
> 2022-10-14 15:12:54,582 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting
> ResourceManager
> java.lang.NoClassDefFoundError: org/apache/commons/lang/StringUtils
> at
> org.apache.ranger.authorization.hadoop.config.RangerConfiguration.getFileLocation(RangerConfiguration.java:78)
> at
> org.apache.ranger.authorization.hadoop.config.RangerConfiguration.addResourceIfReadable(RangerConfiguration.java:47)
> at
> org.apache.ranger.authorization.hadoop.config.RangerPluginConfig.addResourcesForServiceType(RangerPluginConfig.java:199)
> at
> org.apache.ranger.authorization.hadoop.config.RangerPluginConfig.(RangerPluginConfig.java:60)
> at
> org.apache.ranger.plugin.service.RangerBasePlugin.(RangerBasePlugin.java:73)
> at
> org.apache.ranger.authorization.yarn.authorizer.RangerYarnPlugin.(RangerYarnAuthorizer.java:287)
> at
> org.apache.ranger.authorization.yarn.authorizer.RangerYarnAuthorizer.init(RangerYarnAuthorizer.java:82)
> at
> org.apache.ranger.authorization.yarn.authorizer.RangerYarnAuthorizer.init(RangerYarnAuthorizer.java:95)
> at
> org.apache.hadoop.yarn.security.YarnAuthorizationProvider.getInstance(YarnAuthorizationProvider.java:57)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.QueueACLsManager.(QueueACLsManager.java:58)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createQueueACLsManager(ResourceManager.java:434)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:833)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1373)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:332)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1657)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.commons.lang.StringUtils
> at java.lang.ClassLoader.findClass(ClassLoader.java:523)
> at
> org.apache.ranger.plugin.classloader.RangerPluginClassLoader$MyClassLoader.findClass(RangerPluginClassLoader.java:290)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> at
> org.apache.ranger.plugin.classloader.RangerPluginClassLoader.loadClass(RangerPluginClassLoader.java:132)
> ... 17 more {code}
> This error shows that the commons-lang-2.6.jar cannot be found in the
> classpath.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (RANGER-3946) ranger-yarn-plugin missing commons-lang-2.6.jar
[
https://issues.apache.org/jira/browse/RANGER-3946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuze Yang reassigned RANGER-3946:
-
Assignee: Xuze Yang
> ranger-yarn-plugin missing commons-lang-2.6.jar
> ---
>
> Key: RANGER-3946
> URL: https://issues.apache.org/jira/browse/RANGER-3946
> Project: Ranger
> Issue Type: Bug
> Components: plugins
>Affects Versions: 2.1.0
> Reporter: Xuze Yang
>Assignee: Xuze Yang
>Priority: Major
>
> when implement ranger-2.1.0-yarn-plugin in hadoop-3.2.4. a error occured
> shown below:
> {code:java}
> 2022-10-14 15:12:54,561 INFO
> org.apache.ranger.plugin.classloader.RangerPluginClassLoaderUtil:
> getFilesInDirectory('/cloud/service/ranger-2.1.0-yarn-plugin/lib/ranger-yarn-plugin-impl'):
> adding
> /cloud/service/ranger-2.1.0-yarn-plugin/lib/ranger-yarn-plugin-impl/solr-solrj-7.7.1.jar
> 2022-10-14 15:12:54,582 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting
> ResourceManager
> java.lang.NoClassDefFoundError: org/apache/commons/lang/StringUtils
> at
> org.apache.ranger.authorization.hadoop.config.RangerConfiguration.getFileLocation(RangerConfiguration.java:78)
> at
> org.apache.ranger.authorization.hadoop.config.RangerConfiguration.addResourceIfReadable(RangerConfiguration.java:47)
> at
> org.apache.ranger.authorization.hadoop.config.RangerPluginConfig.addResourcesForServiceType(RangerPluginConfig.java:199)
> at
> org.apache.ranger.authorization.hadoop.config.RangerPluginConfig.(RangerPluginConfig.java:60)
> at
> org.apache.ranger.plugin.service.RangerBasePlugin.(RangerBasePlugin.java:73)
> at
> org.apache.ranger.authorization.yarn.authorizer.RangerYarnPlugin.(RangerYarnAuthorizer.java:287)
> at
> org.apache.ranger.authorization.yarn.authorizer.RangerYarnAuthorizer.init(RangerYarnAuthorizer.java:82)
> at
> org.apache.ranger.authorization.yarn.authorizer.RangerYarnAuthorizer.init(RangerYarnAuthorizer.java:95)
> at
> org.apache.hadoop.yarn.security.YarnAuthorizationProvider.getInstance(YarnAuthorizationProvider.java:57)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.QueueACLsManager.(QueueACLsManager.java:58)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createQueueACLsManager(ResourceManager.java:434)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:833)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1373)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:332)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1657)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.commons.lang.StringUtils
> at java.lang.ClassLoader.findClass(ClassLoader.java:523)
> at
> org.apache.ranger.plugin.classloader.RangerPluginClassLoader$MyClassLoader.findClass(RangerPluginClassLoader.java:290)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> at
> org.apache.ranger.plugin.classloader.RangerPluginClassLoader.loadClass(RangerPluginClassLoader.java:132)
> ... 17 more {code}
> This error shows that the commons-lang-2.6.jar cannot be found in the
> classpath.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (RANGER-3946) ranger-yarn-plugin missing commons-lang-2.6.jar
[
https://issues.apache.org/jira/browse/RANGER-3946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17617577#comment-17617577
]
Xuze Yang commented on RANGER-3946:
---
ranger-yarn-plugin can work well with hadoop-2.8.5(with some small changes) and
hadoop-2.10.1, because the two versions of hadoop rely on commons-lang-2.6.jar,
but in 3.2.4 (which may also involve other versions), the dependency on
commons-lang-2.6.jar is removed. Therefore, it is necessary to take
common-lang.jar as a dependency.
Just like in ranger-hdfs-plugin, we also need to package commons-lang in
yarn-plugin
{code:java}
lib/ranger-hdfs-plugin-impl
true
false
755
644
commons-lang:commons-lang
org.eclipse.jetty:jetty-client:jar:${jetty-client.version}
org.apache.httpcomponents:httpmime:jar:${httpcomponents.httpmime.version}
org.apache.httpcomponents:httpclient:jar:${httpcomponents.httpclient.version}
org.apache.httpcomponents:httpcore:jar:${httpcomponents.httpcore.version}
org.noggit:noggit:jar:${noggit.version}
org.apache.solr:solr-solrj:jar:${solr.version}
com.kstruct:gethostname4j:jar:${kstruct.gethostname4j.version}
net.java.dev.jna:jna:jar:${jna.version}
net.java.dev.jna:jna-platform:jar:${jna-platform.version}
org.elasticsearch:elasticsearch
org.elasticsearch:elasticsearch-core
org.elasticsearch:elasticsearch-x-content
org.elasticsearch.client:elasticsearch-rest-client
org.elasticsearch.client:elasticsearch-rest-high-level-client
org.elasticsearch.plugin:rank-eval-client
org.elasticsearch.plugin:lang-mustache-client
org.apache.httpcomponents:httpcore-nio:jar:${httpcomponents.httpcore.version}
org.apache.httpcomponents:httpasyncclient:jar:${httpcomponents.httpasyncclient.version}
org.apache.lucene:lucene-core
joda-time:joda-time
com.carrotsearch:hppc
{code}
> ranger-yarn-plugin missing commons-lang-2.6.jar
> ---
>
> Key: RANGER-3946
> URL: https://issues.apache.org/jira/browse/RANGER-3946
> Project: Ranger
> Issue Type: Bug
> Components: plugins
>Affects Versions: 2.1.0
> Reporter: Xuze Yang
>Priority: Major
>
> when implement ranger-2.1.0-yarn-plugin in hadoop-3.2.4. a error occured
> shown below:
> {code:java}
> 2022-10-14 15:12:54,561 INFO
> org.apache.ranger.plugin.classloader.RangerPluginClassLoaderUtil:
> getFilesInDirectory('/cloud/service/ranger-2.1.0-yarn-plugin/lib/ranger-yarn-plugin-impl'):
> adding
> /cloud/service/ranger-2.1.0-yarn-plugin/lib/ranger-yarn-plugin-impl/solr-solrj-7.7.1.jar
> 2022-10-14 15:12:54,582 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting
> ResourceManager
> java.lang.NoClassDefFoundError: org/apache/commons/lang/StringUtils
> at
> org.apache.ranger.authorization.hadoop.config.RangerConfiguration.getFileLocation(RangerConfiguration.java:78)
> at
> org.apache.ranger.authorization.hadoop.config.RangerConfiguration.addResourceIfReadable(RangerConfiguration.java:47)
> at
> org.apache.ranger.authorization.hadoop.config.RangerPluginConfig.addResourcesForServiceType(RangerPluginConfig.java:199)
> at
> org.apache.ranger.authorization.hadoop.config.RangerPluginConfig.(RangerPluginConfig.java:60)
> at
> org.apache.ranger.plugin.service.RangerBasePlugin.(RangerBasePlugin.java:73)
> at
> org.apache.ranger.authorization.yarn.authorizer.RangerYarnPlugin.(RangerYarnAuthorizer.java:287)
> at
> org.apache.ranger.authorization.yarn.authorizer.RangerYarnAuthorizer.init(RangerYarnAuthorizer.java:82)
> at
> org.apache.ranger.authorization.yarn.authorizer.RangerYarnAuthorizer.init(RangerYarnAuthorizer.java:95)
> at
> org.apache.hadoop.yarn.security.YarnAuthorizationProvider.getInstance(YarnAuthorizationProvider.java:57)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.QueueACLsManager.(QueueACLsManager.java:58)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createQueueACLsManager(ResourceManager.java:434)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:833)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1373)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:332)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractServi
[jira] [Created] (RANGER-3946) ranger-yarn-plugin missing commons-lang-2.6.jar
Xuze Yang created RANGER-3946:
-
Summary: ranger-yarn-plugin missing commons-lang-2.6.jar
Key: RANGER-3946
URL: https://issues.apache.org/jira/browse/RANGER-3946
Project: Ranger
Issue Type: Bug
Components: plugins
Affects Versions: 2.1.0
Reporter: Xuze Yang
when implement ranger-2.1.0-yarn-plugin in hadoop-3.2.4. a error occured shown
below:
{code:java}
2022-10-14 15:12:54,561 INFO
org.apache.ranger.plugin.classloader.RangerPluginClassLoaderUtil:
getFilesInDirectory('/cloud/service/ranger-2.1.0-yarn-plugin/lib/ranger-yarn-plugin-impl'):
adding
/cloud/service/ranger-2.1.0-yarn-plugin/lib/ranger-yarn-plugin-impl/solr-solrj-7.7.1.jar
2022-10-14 15:12:54,582 ERROR
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting
ResourceManager
java.lang.NoClassDefFoundError: org/apache/commons/lang/StringUtils
at
org.apache.ranger.authorization.hadoop.config.RangerConfiguration.getFileLocation(RangerConfiguration.java:78)
at
org.apache.ranger.authorization.hadoop.config.RangerConfiguration.addResourceIfReadable(RangerConfiguration.java:47)
at
org.apache.ranger.authorization.hadoop.config.RangerPluginConfig.addResourcesForServiceType(RangerPluginConfig.java:199)
at
org.apache.ranger.authorization.hadoop.config.RangerPluginConfig.(RangerPluginConfig.java:60)
at
org.apache.ranger.plugin.service.RangerBasePlugin.(RangerBasePlugin.java:73)
at
org.apache.ranger.authorization.yarn.authorizer.RangerYarnPlugin.(RangerYarnAuthorizer.java:287)
at
org.apache.ranger.authorization.yarn.authorizer.RangerYarnAuthorizer.init(RangerYarnAuthorizer.java:82)
at
org.apache.ranger.authorization.yarn.authorizer.RangerYarnAuthorizer.init(RangerYarnAuthorizer.java:95)
at
org.apache.hadoop.yarn.security.YarnAuthorizationProvider.getInstance(YarnAuthorizationProvider.java:57)
at
org.apache.hadoop.yarn.server.resourcemanager.security.QueueACLsManager.(QueueACLsManager.java:58)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createQueueACLsManager(ResourceManager.java:434)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:833)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1373)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:332)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1657)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.lang.StringUtils
at java.lang.ClassLoader.findClass(ClassLoader.java:523)
at
org.apache.ranger.plugin.classloader.RangerPluginClassLoader$MyClassLoader.findClass(RangerPluginClassLoader.java:290)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at
org.apache.ranger.plugin.classloader.RangerPluginClassLoader.loadClass(RangerPluginClassLoader.java:132)
... 17 more {code}
This error shows that the commons-lang-2.6.jar cannot be found in the classpath.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: Review Request 73694: RANGER-3500: Ranger policy list doesn't support sorting by field
> On 十月 11, 2022, 10:40 p.m., Madhan Neethiraj wrote:
> > security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java
> > Lines 2597 (patched)
> > <https://reviews.apache.org/r/73694/diff/2/?file=2270291#file2270291line2597>
> >
> > Consider adding a copy-constructor in SearchFilter and use it here, so
> > that other fields (startIndex/maxRows/getCount) will be copied as well.
> >
> > SearchFilter searchFilter = new SearchFilter(filter);
> >
> >
> >
> > // SearchFilter.java
> > public SearchFilter() {
> > this((Map) null);
> > }
> >
> > public SearchFilter(SearchFilter other) {
> > if (other != null) {
> > setParams(other.params != null ? new HashMap<>(other.params) :
> > null);
> > setStartIndex(other.startIndex);
> > setMaxRows(other.maxRows);
> > setGetCount(other.getCount);
> > setSortBy(other.sortBy);
> > setSortType(other.sortType);
> > } else {
> > setParams(null);
> > }
> > }
Thank you for your modification comments. I have made changes according to your
modification comments. The same verification has been done in the environment,
and everything works well
- Xuze
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73694/#review224784
---
On 九月 29, 2022, 2:37 a.m., Xuze Yang wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73694/
> ---
>
> (Updated 九月 29, 2022, 2:37 a.m.)
>
>
> Review request for ranger, Abhay Kulkarni, Madhan Neethiraj, Mahesh Bandal,
> Pradeep Agrawal, Ramesh Mani, Sailaja Polavarapu, Vishal Suvagia, and
> Velmurugan Periasamy.
>
>
> Bugs: RANGER-3500
> https://issues.apache.org/jira/browse/RANGER-3500
>
>
> Repository: ranger
>
>
> Description
> ---
>
> modify the getServicePolicies() method to support sorting by field and
> sortType(i.e. asc or desc). The changes are as following:
> 1. Add sortBy and sortType to searchFilter
> 2. Support sorting by sortType("desc" means in decreasing order, otherwise in
> increasing order)
> 3. Delete a redundant line of code
>
>
> Diffs
> -
>
>
> agents-common/src/main/java/org/apache/ranger/plugin/store/AbstractPredicateUtil.java
> 38d6b03
> agents-common/src/main/java/org/apache/ranger/plugin/util/SearchFilter.java
> c5f13b5
> security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java
> fdead07
>
>
> Diff: https://reviews.apache.org/r/73694/diff/3/
>
>
> Testing
> ---
>
> 1. compile and package the code
> 2. deploy ranger
> 3. generate hdfs policies and get the hdfs policyList via postman using url
> like
> "http://192.168.0.12:6080/service/plugins/policies/service/2?sortBy=policyName=desc=default-Hdfs;
> 4. the return policyList is sorted according our specified sortBy and sortType
>
>
> File Attachments
>
>
> 0002-RANGER-3500-Ranger-policy-list-doesn-t-support-sorti.patch
>
> https://reviews.apache.org/media/uploaded/files/2022/09/29/61ba9494-1dc8-4b67-80ed-44c7e9f313a1__0002-RANGER-3500-Ranger-policy-list-doesn-t-support-sorti.patch
>
>
> Thanks,
>
> Xuze Yang
>
>
[jira] [Updated] (RANGER-3500) Ranger policy list doesn't support sorting by field
[
https://issues.apache.org/jira/browse/RANGER-3500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuze Yang updated RANGER-3500:
--
Attachment: 0003-RANGER-3500-Ranger-policy-list-doesn-t-support-sorti.patch
> Ranger policy list doesn't support sorting by field
> ---
>
> Key: RANGER-3500
> URL: https://issues.apache.org/jira/browse/RANGER-3500
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Affects Versions: 2.1.0
> Reporter: Xuze Yang
>Assignee: Xuze Yang
>Priority: Major
> Attachments:
> 0001-RANGER-3500-Ranger-policy-list-doesn-t-support-sorti.patch,
> 0002-RANGER-3500-Ranger-policy-list-doesn-t-support-sorti.patch,
> 0003-RANGER-3500-Ranger-policy-list-doesn-t-support-sorti.patch
>
>
> When getting the ranger policy list, we may want to sort the returned policy
> list according to certain fields, such as policyId, policyName and etc. But
> case shows that adding the parameters sortBy and sortType to the url has no
> effect (eg:
> [http://192.168.0.12:6080/service/plugins/]policies/service/2?sortBy=policyName=desc=default-Hdfs).
> I look through the source code and find that code supports sorting by
> fields, but due to some code bugs, it did not really take effect.
> The main reason for the problem is that when the SearchFilter is copied
> deeply, only the params is copied, the sortBy and sortType attributes is
> omitted. The code show as follows:
> {code:java}
> // ServiceDBStore # getServicePolicies()
> Map paramsCopy = new HashMap<>(filter.getParams());
> SearchFilter searchFilter = new SearchFilter(paramsCopy); {code}
>
> {code:java}
> // SearchFilter.java
> private Map params;
> private int startIndex;
> private int maxRows= Integer.MAX_VALUE;
> private boolean getCount = true;
> private String sortBy;
> private String sortType;
> public SearchFilter(Map values) {
>setParams(values);
> }
> public void setParams(Map params) {
>this.params = params;
> } {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (RANGER-3945) enable-agent.sh can't disable sqoop and kylin's properties file
[
https://issues.apache.org/jira/browse/RANGER-3945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuze Yang reassigned RANGER-3945:
-
Assignee: Xuze Yang
review request available at:
[https://reviews.apache.org/r/74160/]
> enable-agent.sh can't disable sqoop and kylin's properties file
> ---
>
> Key: RANGER-3945
> URL: https://issues.apache.org/jira/browse/RANGER-3945
> Project: Ranger
> Issue Type: Bug
> Components: plugins
>Affects Versions: 2.1.0
> Reporter: Xuze Yang
>Assignee: Xuze Yang
>Priority: Major
> Attachments:
> 0001-RANGER-3945-enable-agent.sh-can-t-disable-sqoop-and-.patch
>
>
> Take sqoop for example, when the behaviour is disable, the authName will be
> "".
> {code:java}
> if [ "${action}" = "enable" ]
> then
>
> authName="org.apache.ranger.authorization.sqoop.authorizer.RangerSqoopAuthorizer"
> else
> authName=""
> fi {code}
> AuthName will be passed into the addOrUpdatePropertyToFile method as second
> parameter.
> {code:java}
> addOrUpdatePropertyToFile org.apache.sqoop.security.authorization.validator
> $authName ${fn} {code}
> However, when $authName is "", $authName will not be considered as a
> parameter. AddOrUpdatePropertyToFile will think there only two parameters and
> then caused a series of errors.
> {code:java}
> addOrUpdatePropertyToFile(){
> checkPropertyInFile $1 $3
> if [ $? -eq 1 ]
> then
> addPropertyToFile $1 $2 $3
> else
> updatePropertyToFile $1 $2 $3
> fi
> } {code}
> In addition, kafka also needs to modify server.properties but there is no
> corresponding implementation in this script.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (RANGER-3945) enable-agent.sh can't disable sqoop and kylin's properties file
[
https://issues.apache.org/jira/browse/RANGER-3945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuze Yang updated RANGER-3945:
--
Attachment: 0001-RANGER-3945-enable-agent.sh-can-t-disable-sqoop-and-.patch
> enable-agent.sh can't disable sqoop and kylin's properties file
> ---
>
> Key: RANGER-3945
> URL: https://issues.apache.org/jira/browse/RANGER-3945
> Project: Ranger
> Issue Type: Bug
> Components: plugins
>Affects Versions: 2.1.0
> Reporter: Xuze Yang
>Priority: Major
> Attachments:
> 0001-RANGER-3945-enable-agent.sh-can-t-disable-sqoop-and-.patch
>
>
> Take sqoop for example, when the behaviour is disable, the authName will be
> "".
> {code:java}
> if [ "${action}" = "enable" ]
> then
>
> authName="org.apache.ranger.authorization.sqoop.authorizer.RangerSqoopAuthorizer"
> else
> authName=""
> fi {code}
> AuthName will be passed into the addOrUpdatePropertyToFile method as second
> parameter.
> {code:java}
> addOrUpdatePropertyToFile org.apache.sqoop.security.authorization.validator
> $authName ${fn} {code}
> However, when $authName is "", $authName will not be considered as a
> parameter. AddOrUpdatePropertyToFile will think there only two parameters and
> then caused a series of errors.
> {code:java}
> addOrUpdatePropertyToFile(){
> checkPropertyInFile $1 $3
> if [ $? -eq 1 ]
> then
> addPropertyToFile $1 $2 $3
> else
> updatePropertyToFile $1 $2 $3
> fi
> } {code}
> In addition, kafka also needs to modify server.properties but there is no
> corresponding implementation in this script.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (RANGER-3945) enable-agent.sh can't disable sqoop and kylin's properties file
[
https://issues.apache.org/jira/browse/RANGER-3945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615602#comment-17615602
]
Xuze Yang commented on RANGER-3945:
---
if authName equals "", and it is passed in as a parameter, we need to surround
it with "". like
{code:java}
addOrUpdatePropertyToFile org.apache.sqoop.security.authorization.validator
"$authName" ${fn}{code}
In addition, I added the logic related to modifying kafka's server.properties
in this patch
> enable-agent.sh can't disable sqoop and kylin's properties file
> ---
>
> Key: RANGER-3945
> URL: https://issues.apache.org/jira/browse/RANGER-3945
> Project: Ranger
> Issue Type: Bug
> Components: plugins
> Affects Versions: 2.1.0
>Reporter: Xuze Yang
>Priority: Major
> Attachments:
> 0001-RANGER-3945-enable-agent.sh-can-t-disable-sqoop-and-.patch
>
>
> Take sqoop for example, when the behaviour is disable, the authName will be
> "".
> {code:java}
> if [ "${action}" = "enable" ]
> then
>
> authName="org.apache.ranger.authorization.sqoop.authorizer.RangerSqoopAuthorizer"
> else
> authName=""
> fi {code}
> AuthName will be passed into the addOrUpdatePropertyToFile method as second
> parameter.
> {code:java}
> addOrUpdatePropertyToFile org.apache.sqoop.security.authorization.validator
> $authName ${fn} {code}
> However, when $authName is "", $authName will not be considered as a
> parameter. AddOrUpdatePropertyToFile will think there only two parameters and
> then caused a series of errors.
> {code:java}
> addOrUpdatePropertyToFile(){
> checkPropertyInFile $1 $3
> if [ $? -eq 1 ]
> then
> addPropertyToFile $1 $2 $3
> else
> updatePropertyToFile $1 $2 $3
> fi
> } {code}
> In addition, kafka also needs to modify server.properties but there is no
> corresponding implementation in this script.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (RANGER-3945) enable-agent.sh can't disable sqoop and kylin's properties file
Xuze Yang created RANGER-3945:
-
Summary: enable-agent.sh can't disable sqoop and kylin's
properties file
Key: RANGER-3945
URL: https://issues.apache.org/jira/browse/RANGER-3945
Project: Ranger
Issue Type: Bug
Components: plugins
Affects Versions: 2.1.0
Reporter: Xuze Yang
Take sqoop for example, when the behaviour is disable, the authName will be "".
{code:java}
if [ "${action}" = "enable" ]
then
authName="org.apache.ranger.authorization.sqoop.authorizer.RangerSqoopAuthorizer"
else
authName=""
fi {code}
AuthName will be passed into the addOrUpdatePropertyToFile method as second
parameter.
{code:java}
addOrUpdatePropertyToFile org.apache.sqoop.security.authorization.validator
$authName ${fn} {code}
However, when $authName is "", $authName will not be considered as a parameter.
AddOrUpdatePropertyToFile will think there only two parameters and then caused
a series of errors.
{code:java}
addOrUpdatePropertyToFile(){
checkPropertyInFile $1 $3
if [ $? -eq 1 ]
then
addPropertyToFile $1 $2 $3
else
updatePropertyToFile $1 $2 $3
fi
} {code}
In addition, kafka also needs to modify server.properties but there is no
corresponding implementation in this script.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (RANGER-3685) hive 'show' sql produces excessive audit log
[ https://issues.apache.org/jira/browse/RANGER-3685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615529#comment-17615529 ] Xuze Yang commented on RANGER-3685: --- [~madhan] Thank you for your prompt. I missed these two jiras before. Now I have tried them in our environment. Everything works well and the audit log records meet expectations. > hive 'show' sql produces excessive audit log > > > Key: RANGER-3685 > URL: https://issues.apache.org/jira/browse/RANGER-3685 > Project: Ranger > Issue Type: Improvement > Components: audit >Affects Versions: 2.1.0 > Reporter: Xuze Yang >Assignee: Xuze Yang >Priority: Major > Attachments: > RANGER-3685-hive-show-sql-produces-excessive-audit-l.patch > > > Since ranger2.1.0. For "show databases", user needs any permission on > Database to get authorized. RangerHiveAuthorizer.filterListCmdObjects() is > implemented to filter out the database which user don't have access to. > This is a good implementation, but a problem comes with it：the method will > record an audit log for each database(each table for "show tables"). In our > production environment, There are 80,000 tables under a database of hive. A > show tables operation will generate 80001(The extra one is the verification > of USE permissions) audit logs. Unfortunately, our customers will frequently > call the show tables operation. > This brings up two problems: > # Valuable audit logs are flooded > # Take up a lot of storage resources > For problem.2, such a scenario has occurred in our environment: our audit log > destination is down. All audit logs are spooled in disk files, several days > later, the size of the disk file exceeded 800G, causing other components to > fail to provide services normally. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (RANGER-3935) In hdfs authorizer 'processResult', is accessType and action misused?
[ https://issues.apache.org/jira/browse/RANGER-3935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17610959#comment-17610959 ] Xuze Yang commented on RANGER-3935: --- [~wangning] Ranger cannot perceive the specific operation done by the hdfs side. hdfs plugin is an implementation of INodeAttributeProvider#AccessControlEnforcer. The information can be obtained through this interface is: who? which path? Required permissions (read/write/execute). > In hdfs authorizer 'processResult', is accessType and action misused? > - > > Key: RANGER-3935 > URL: https://issues.apache.org/jira/browse/RANGER-3935 > Project: Ranger > Issue Type: Improvement > Components: audit >Reporter: wangningito >Priority: Major > > [https://github.com/apache/ranger/blob/master/hdfs-agent/src/main/java/org/apache/ranger/authorization/hadoop/RangerHdfsAuthorizer.java#L1037-L1046] > I see action is filled into accessType while accessType field is filled with > action. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (RANGER-3935) In hdfs authorizer 'processResult', is accessType and action misused?
[ https://issues.apache.org/jira/browse/RANGER-3935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17610959#comment-17610959 ] Xuze Yang edited comment on RANGER-3935 at 9/29/22 11:14 AM: - [~wangning] Ranger cannot perceive the specific operation(like mkdir,delete etc.) done by the hdfs side. hdfs plugin is an implementation of INodeAttributeProvider#AccessControlEnforcer. The information can be obtained through this interface is: who? which path? Required permissions (read/write/execute). was (Author: xuze yang): [~wangning] Ranger cannot perceive the specific operation done by the hdfs side. hdfs plugin is an implementation of INodeAttributeProvider#AccessControlEnforcer. The information can be obtained through this interface is: who? which path? Required permissions (read/write/execute). > In hdfs authorizer 'processResult', is accessType and action misused? > - > > Key: RANGER-3935 > URL: https://issues.apache.org/jira/browse/RANGER-3935 > Project: Ranger > Issue Type: Improvement > Components: audit >Reporter: wangningito >Priority: Major > > [https://github.com/apache/ranger/blob/master/hdfs-agent/src/main/java/org/apache/ranger/authorization/hadoop/RangerHdfsAuthorizer.java#L1037-L1046] > I see action is filled into accessType while accessType field is filled with > action. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (RANGER-3935) In hdfs authorizer 'processResult', is accessType and action misused?
[ https://issues.apache.org/jira/browse/RANGER-3935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17610857#comment-17610857 ] Xuze Yang commented on RANGER-3935: --- I find the same situation exists in [https://github.com/apache/ranger/blob/master/agents-common/src/main/java/org/apache/ranger/plugin/audit/RangerDefaultAuditHandler.java] #L127 #130 This means that accessType and action swapping is a default behavior. Now I wonder if this is by design or a fallacy. cc [~kirbyzhou] [~madhan] > In hdfs authorizer 'processResult', is accessType and action misused? > - > > Key: RANGER-3935 > URL: https://issues.apache.org/jira/browse/RANGER-3935 > Project: Ranger > Issue Type: Improvement > Components: audit >Reporter: wangningito >Priority: Major > > [https://github.com/apache/ranger/blob/master/hdfs-agent/src/main/java/org/apache/ranger/authorization/hadoop/RangerHdfsAuthorizer.java#L1037-L1046] > I see action is filled into accessType while accessType field is filled with > action. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (RANGER-3935) In hdfs authorizer 'processResult', is accessType and action misused?
[
https://issues.apache.org/jira/browse/RANGER-3935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17610791#comment-17610791
]
Xuze Yang commented on RANGER-3935:
---
Yes, I think the code is confusing the two.
The action corresponds to FsAction, which is the real operation type of hdfs.
{code:java}
public enum FsAction {
NONE("---"),
EXECUTE("--x"),
WRITE("-w-"),
WRITE_EXECUTE("-wx"),
READ("r--"),
READ_EXECUTE("r-x"),
READ_WRITE("rw-"),
ALL("rwx");
...
}{code}
The accessType corresponds to the abstract operation type in the ranger policy,
which is defined in RangerHadoopConstants
{code:java}
public static final String READ_ACCCESS_TYPE = "read";
public static final String WRITE_ACCCESS_TYPE = "write";
public static final String EXECUTE_ACCCESS_TYPE = "execute";{code}
By the way, there is a typo in code, 'ACCCESS' should be 'ACCESS'
> In hdfs authorizer 'processResult', is accessType and action misused?
> -
>
> Key: RANGER-3935
> URL: https://issues.apache.org/jira/browse/RANGER-3935
> Project: Ranger
> Issue Type: Improvement
> Components: audit
>Reporter: wangningito
>Priority: Major
>
> [https://github.com/apache/ranger/blob/master/hdfs-agent/src/main/java/org/apache/ranger/authorization/hadoop/RangerHdfsAuthorizer.java#L1037-L1046]
> I see action is filled into accessType while accessType field is filled with
> action.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: Review Request 73694: RANGER-3500: Ranger policy list doesn't support sorting by field
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73694/
---
(Updated 九月 29, 2022, 2:37 a.m.)
Review request for ranger, Abhay Kulkarni, Madhan Neethiraj, Mahesh Bandal,
Pradeep Agrawal, Ramesh Mani, Sailaja Polavarapu, Vishal Suvagia, and
Velmurugan Periasamy.
Bugs: RANGER-3500
https://issues.apache.org/jira/browse/RANGER-3500
Repository: ranger
Description
---
modify the getServicePolicies() method to support sorting by field and
sortType(i.e. asc or desc). The changes are as following:
1. Add sortBy and sortType to searchFilter
2. Support sorting by sortType("desc" means in decreasing order, otherwise in
increasing order)
3. Delete a redundant line of code
Diffs
-
agents-common/src/main/java/org/apache/ranger/plugin/store/AbstractPredicateUtil.java
38d6b03ea
security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java
f13cef71d
Diff: https://reviews.apache.org/r/73694/diff/1/
Testing
---
1. compile and package the code
2. deploy ranger
3. generate hdfs policies and get the hdfs policyList via postman using url
like
"http://192.168.0.12:6080/service/plugins/policies/service/2?sortBy=policyName=desc=default-Hdfs;
4. the return policyList is sorted according our specified sortBy and sortType
File Attachments (updated)
0002-RANGER-3500-Ranger-policy-list-doesn-t-support-sorti.patch
https://reviews.apache.org/media/uploaded/files/2022/09/29/61ba9494-1dc8-4b67-80ed-44c7e9f313a1__0002-RANGER-3500-Ranger-policy-list-doesn-t-support-sorti.patch
Thanks,
Xuze Yang
[jira] [Commented] (RANGER-3500) Ranger policy list doesn't support sorting by field
[
https://issues.apache.org/jira/browse/RANGER-3500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17610773#comment-17610773
]
Xuze Yang commented on RANGER-3500:
---
The changes are as following:
1. Add sortBy and sortType to searchFilter
2. Support sorting by sortType("desc" means in decreasing order, otherwise in
increasing order)
3. Delete a redundant line of code(In AbstractPredicateUtil#static filed)
When the ranger web ui calls the getServicePolicies(), the sortBy and sortType
parameters are not passed by default, the change is consistent with the
previous return logic.
The patch is updated based on ranger-2.3.0.
Please find attached patch and review PR as per your convenience. Thanks.
CC: [~madhan] [~kirbyzhou] [~kulkabhay] [~rmani] [~bpatel]
> Ranger policy list doesn't support sorting by field
> ---
>
> Key: RANGER-3500
> URL: https://issues.apache.org/jira/browse/RANGER-3500
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Affects Versions: 2.1.0
>Reporter: Xuze Yang
>Assignee: Xuze Yang
>Priority: Major
> Attachments:
> 0001-RANGER-3500-Ranger-policy-list-doesn-t-support-sorti.patch,
> 0002-RANGER-3500-Ranger-policy-list-doesn-t-support-sorti.patch
>
>
> When getting the ranger policy list, we may want to sort the returned policy
> list according to certain fields, such as policyId, policyName and etc. But
> case shows that adding the parameters sortBy and sortType to the url has no
> effect (eg:
> [http://192.168.0.12:6080/service/plugins/]policies/service/2?sortBy=policyName=desc=default-Hdfs).
> I look through the source code and find that code supports sorting by
> fields, but due to some code bugs, it did not really take effect.
> The main reason for the problem is that when the SearchFilter is copied
> deeply, only the params is copied, the sortBy and sortType attributes is
> omitted. The code show as follows:
> {code:java}
> // ServiceDBStore # getServicePolicies()
> Map paramsCopy = new HashMap<>(filter.getParams());
> SearchFilter searchFilter = new SearchFilter(paramsCopy); {code}
>
> {code:java}
> // SearchFilter.java
> private Map params;
> private int startIndex;
> private int maxRows= Integer.MAX_VALUE;
> private boolean getCount = true;
> private String sortBy;
> private String sortType;
> public SearchFilter(Map values) {
>setParams(values);
> }
> public void setParams(Map params) {
>this.params = params;
> } {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (RANGER-3500) Ranger policy list doesn't support sorting by field
[
https://issues.apache.org/jira/browse/RANGER-3500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuze Yang updated RANGER-3500:
--
Attachment: 0002-RANGER-3500-Ranger-policy-list-doesn-t-support-sorti.patch
> Ranger policy list doesn't support sorting by field
> ---
>
> Key: RANGER-3500
> URL: https://issues.apache.org/jira/browse/RANGER-3500
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Affects Versions: 2.1.0
> Reporter: Xuze Yang
>Assignee: Xuze Yang
>Priority: Major
> Attachments:
> 0001-RANGER-3500-Ranger-policy-list-doesn-t-support-sorti.patch,
> 0002-RANGER-3500-Ranger-policy-list-doesn-t-support-sorti.patch
>
>
> When getting the ranger policy list, we may want to sort the returned policy
> list according to certain fields, such as policyId, policyName and etc. But
> case shows that adding the parameters sortBy and sortType to the url has no
> effect (eg:
> [http://192.168.0.12:6080/service/plugins/]policies/service/2?sortBy=policyName=desc=default-Hdfs).
> I look through the source code and find that code supports sorting by
> fields, but due to some code bugs, it did not really take effect.
> The main reason for the problem is that when the SearchFilter is copied
> deeply, only the params is copied, the sortBy and sortType attributes is
> omitted. The code show as follows:
> {code:java}
> // ServiceDBStore # getServicePolicies()
> Map paramsCopy = new HashMap<>(filter.getParams());
> SearchFilter searchFilter = new SearchFilter(paramsCopy); {code}
>
> {code:java}
> // SearchFilter.java
> private Map params;
> private int startIndex;
> private int maxRows= Integer.MAX_VALUE;
> private boolean getCount = true;
> private String sortBy;
> private String sortType;
> public SearchFilter(Map values) {
>setParams(values);
> }
> public void setParams(Map params) {
>this.params = params;
> } {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (RANGER-3500) Ranger policy list doesn't support sorting by field
[
https://issues.apache.org/jira/browse/RANGER-3500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuze Yang updated RANGER-3500:
--
Description:
When getting the ranger policy list, we may want to sort the returned policy
list according to certain fields, such as policyId, policyName and etc. But
case shows that adding the parameters sortBy and sortType to the url has no
effect (eg:
[http://192.168.0.12:6080/service/plugins/]policies/service/2?sortBy=policyName=desc=default-Hdfs).
I look through the source code and find that code supports sorting by fields,
but due to some code bugs, it did not really take effect.
The main reason for the problem is that when the SearchFilter is copied deeply,
only the params is copied, the sortBy and sortType attributes is omitted. The
code show as follows:
{code:java}
// ServiceDBStore # getServicePolicies()
Map paramsCopy = new HashMap<>(filter.getParams());
SearchFilter searchFilter = new SearchFilter(paramsCopy); {code}
{code:java}
// SearchFilter.java
private Map params;
private int startIndex;
private int maxRows= Integer.MAX_VALUE;
private boolean getCount = true;
private String sortBy;
private String sortType;
public SearchFilter(Map values) {
setParams(values);
}
public void setParams(Map params) {
this.params = params;
} {code}
was:
When getting the ranger policy list, we may want to sort the returned policy
list according to certain fields, such as policyId, policyName and etc. But
case shows that adding the parameters sortBy and sortType to the url has no
effect (eg:
[http://192.168.0.12:6080/service/plugins/]policies/service/2?sortBy=policyName=desc=default-Hdfs).
I look through the source code and find that code supports sorting by fields,
but due to some code bugs, it did not really take effect.
The main reason for the problem is that when the SearchFilter is copied
deeply, only the params is copied, the sortBy and sortType attributes is
omitted. The code show as follows:
{code:java}
Map paramsCopy = new HashMap<>(filter.getParams());
SearchFilter searchFilter = new SearchFilter(paramsCopy);
{code}
> Ranger policy list doesn't support sorting by field
> ---
>
> Key: RANGER-3500
> URL: https://issues.apache.org/jira/browse/RANGER-3500
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Affects Versions: 2.1.0
>Reporter: Xuze Yang
>Assignee: Xuze Yang
>Priority: Major
> Attachments:
> 0001-RANGER-3500-Ranger-policy-list-doesn-t-support-sorti.patch
>
>
> When getting the ranger policy list, we may want to sort the returned policy
> list according to certain fields, such as policyId, policyName and etc. But
> case shows that adding the parameters sortBy and sortType to the url has no
> effect (eg:
> [http://192.168.0.12:6080/service/plugins/]policies/service/2?sortBy=policyName=desc=default-Hdfs).
> I look through the source code and find that code supports sorting by
> fields, but due to some code bugs, it did not really take effect.
> The main reason for the problem is that when the SearchFilter is copied
> deeply, only the params is copied, the sortBy and sortType attributes is
> omitted. The code show as follows:
> {code:java}
> // ServiceDBStore # getServicePolicies()
> Map paramsCopy = new HashMap<>(filter.getParams());
> SearchFilter searchFilter = new SearchFilter(paramsCopy); {code}
>
> {code:java}
> // SearchFilter.java
> private Map params;
> private int startIndex;
> private int maxRows= Integer.MAX_VALUE;
> private boolean getCount = true;
> private String sortBy;
> private String sortType;
> public SearchFilter(Map values) {
>setParams(values);
> }
> public void setParams(Map params) {
>this.params = params;
> } {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: Review Request 74136: RANGER-3685:hive 'show' sql produces excessive audit log
> On 九月 28, 2022, 9:15 a.m., Kirby Zhou wrote: > > Why modify filterListCmdObjects() only? filterListCmdObjects() will filter out any objects that should not be shown to the user, from the list of tables or databases coming from a 'show tables' or 'show databases' command. This patch gives an option to disable record audit logs for every tables or databases when execute 'show' sql. checkPrivileges() will record one audit log about 'show' sql(accetype is USE), and this audit log is valuable, we should keep it. I would like to know what else needs to be changed, can you give more hints? Thanks - Xuze --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74136/#review224707 --- On 九月 24, 2022, 2:37 p.m., Xuze Yang wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74136/ > --- > > (Updated 九月 24, 2022, 2:37 p.m.) > > > Review request for ranger, Kirby Zhou, Abhay Kulkarni, and Madhan Neethiraj. > > > Bugs: RANGER-3685 > https://issues.apache.org/jira/browse/RANGER-3685 > > > Repository: ranger > > > Description > --- > > hive show sql produces too many audit logs, which will flood valuable audit > logs and take up a lot of storage resources. This patch aims to provide a > configuration item to support simplified hive audit logging > > > Diffs > - > > > agents-common/src/main/java/org/apache/ranger/authorization/hadoop/constants/RangerHadoopConstants.java > 6675125e1 > > hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAuthorizer.java > 8f6801be1 > > > Diff: https://reviews.apache.org/r/74136/diff/1/ > > > Testing > --- > > 1. when configuration "xasecure.hive.simplify.audit.of.hive.show.sql" is not > set or set to false. The code behaves the same as before,i.e record audit > logs for each databases or tables > 2. when configuration "xasecure.hive.simplify.audit.of.hive.show.sql" is set > to true. It will only record an audit log about USE operation > > > Thanks, > > Xuze Yang > >
[jira] [Assigned] (RANGER-3685) hive 'show' sql produces excessive audit log
[ https://issues.apache.org/jira/browse/RANGER-3685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuze Yang reassigned RANGER-3685: - Assignee: Xuze Yang > hive 'show' sql produces excessive audit log > > > Key: RANGER-3685 > URL: https://issues.apache.org/jira/browse/RANGER-3685 > Project: Ranger > Issue Type: Improvement > Components: audit >Affects Versions: 2.1.0 > Reporter: Xuze Yang >Assignee: Xuze Yang >Priority: Major > Attachments: > RANGER-3685-hive-show-sql-produces-excessive-audit-l.patch > > > Since ranger2.1.0. For "show databases", user needs any permission on > Database to get authorized. RangerHiveAuthorizer.filterListCmdObjects() is > implemented to filter out the database which user don't have access to. > This is a good implementation, but a problem comes with it：the method will > record an audit log for each database(each table for "show tables"). In our > production environment, There are 80,000 tables under a database of hive. A > show tables operation will generate 80001(The extra one is the verification > of USE permissions) audit logs. Unfortunately, our customers will frequently > call the show tables operation. > This brings up two problems: > # Valuable audit logs are flooded > # Take up a lot of storage resources > For problem.2, such a scenario has occurred in our environment: our audit log > destination is down. All audit logs are spooled in disk files, several days > later, the size of the disk file exceeded 800G, causing other components to > fail to provide services normally. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (RANGER-3685) hive 'show' sql produces excessive audit log
[ https://issues.apache.org/jira/browse/RANGER-3685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17609008#comment-17609008 ] Xuze Yang commented on RANGER-3685: --- review request available at: https://reviews.apache.org/r/74136/ > hive 'show' sql produces excessive audit log > > > Key: RANGER-3685 > URL: https://issues.apache.org/jira/browse/RANGER-3685 > Project: Ranger > Issue Type: Improvement > Components: audit >Affects Versions: 2.1.0 > Reporter: Xuze Yang >Priority: Major > Attachments: > RANGER-3685-hive-show-sql-produces-excessive-audit-l.patch > > > Since ranger2.1.0. For "show databases", user needs any permission on > Database to get authorized. RangerHiveAuthorizer.filterListCmdObjects() is > implemented to filter out the database which user don't have access to. > This is a good implementation, but a problem comes with it：the method will > record an audit log for each database(each table for "show tables"). In our > production environment, There are 80,000 tables under a database of hive. A > show tables operation will generate 80001(The extra one is the verification > of USE permissions) audit logs. Unfortunately, our customers will frequently > call the show tables operation. > This brings up two problems: > # Valuable audit logs are flooded > # Take up a lot of storage resources > For problem.2, such a scenario has occurred in our environment: our audit log > destination is down. All audit logs are spooled in disk files, several days > later, the size of the disk file exceeded 800G, causing other components to > fail to provide services normally. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-3685) hive 'show' sql produces excessive audit log
[ https://issues.apache.org/jira/browse/RANGER-3685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuze Yang updated RANGER-3685: -- Attachment: RANGER-3685-hive-show-sql-produces-excessive-audit-l.patch > hive 'show' sql produces excessive audit log > > > Key: RANGER-3685 > URL: https://issues.apache.org/jira/browse/RANGER-3685 > Project: Ranger > Issue Type: Improvement > Components: audit >Affects Versions: 2.1.0 > Reporter: Xuze Yang >Priority: Major > Attachments: > RANGER-3685-hive-show-sql-produces-excessive-audit-l.patch > > > Since ranger2.1.0. For "show databases", user needs any permission on > Database to get authorized. RangerHiveAuthorizer.filterListCmdObjects() is > implemented to filter out the database which user don't have access to. > This is a good implementation, but a problem comes with it：the method will > record an audit log for each database(each table for "show tables"). In our > production environment, There are 80,000 tables under a database of hive. A > show tables operation will generate 80001(The extra one is the verification > of USE permissions) audit logs. Unfortunately, our customers will frequently > call the show tables operation. > This brings up two problems: > # Valuable audit logs are flooded > # Take up a lot of storage resources > For problem.2, such a scenario has occurred in our environment: our audit log > destination is down. All audit logs are spooled in disk files, several days > later, the size of the disk file exceeded 800G, causing other components to > fail to provide services normally. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-3685) hive 'show' sql produces excessive audit log
[ https://issues.apache.org/jira/browse/RANGER-3685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuze Yang updated RANGER-3685: -- Attachment: (was: submit patch.pdf) > hive 'show' sql produces excessive audit log > > > Key: RANGER-3685 > URL: https://issues.apache.org/jira/browse/RANGER-3685 > Project: Ranger > Issue Type: Improvement > Components: audit >Affects Versions: 2.1.0 > Reporter: Xuze Yang >Priority: Major > Attachments: > RANGER-3685-hive-show-sql-produces-excessive-audit-l.patch > > > Since ranger2.1.0. For "show databases", user needs any permission on > Database to get authorized. RangerHiveAuthorizer.filterListCmdObjects() is > implemented to filter out the database which user don't have access to. > This is a good implementation, but a problem comes with it：the method will > record an audit log for each database(each table for "show tables"). In our > production environment, There are 80,000 tables under a database of hive. A > show tables operation will generate 80001(The extra one is the verification > of USE permissions) audit logs. Unfortunately, our customers will frequently > call the show tables operation. > This brings up two problems: > # Valuable audit logs are flooded > # Take up a lot of storage resources > For problem.2, such a scenario has occurred in our environment: our audit log > destination is down. All audit logs are spooled in disk files, several days > later, the size of the disk file exceeded 800G, causing other components to > fail to provide services normally. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-3685) hive 'show' sql produces excessive audit log
[ https://issues.apache.org/jira/browse/RANGER-3685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuze Yang updated RANGER-3685: -- Attachment: (was: 0001-1.-hive-show-sql.patch) > hive 'show' sql produces excessive audit log > > > Key: RANGER-3685 > URL: https://issues.apache.org/jira/browse/RANGER-3685 > Project: Ranger > Issue Type: Improvement > Components: audit >Affects Versions: 2.1.0 > Reporter: Xuze Yang >Priority: Major > Attachments: > RANGER-3685-hive-show-sql-produces-excessive-audit-l.patch > > > Since ranger2.1.0. For "show databases", user needs any permission on > Database to get authorized. RangerHiveAuthorizer.filterListCmdObjects() is > implemented to filter out the database which user don't have access to. > This is a good implementation, but a problem comes with it：the method will > record an audit log for each database(each table for "show tables"). In our > production environment, There are 80,000 tables under a database of hive. A > show tables operation will generate 80001(The extra one is the verification > of USE permissions) audit logs. Unfortunately, our customers will frequently > call the show tables operation. > This brings up two problems: > # Valuable audit logs are flooded > # Take up a lot of storage resources > For problem.2, such a scenario has occurred in our environment: our audit log > destination is down. All audit logs are spooled in disk files, several days > later, the size of the disk file exceeded 800G, causing other components to > fail to provide services normally. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (RANGER-3685) hive 'show' sql produces excessive audit log
[ https://issues.apache.org/jira/browse/RANGER-3685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17609007#comment-17609007 ] Xuze Yang commented on RANGER-3685: --- I uploaded a patch in which a configuration item "xasecure.hive.simplify.audit.of.hive.show.sql" was added to decide whether to simplify the audit log of hive show sql. When not specified, the default is false, that is, the hive audit log is not simplified, which is compatible with the previous behavior. Please find attached patch and review PR as per your convenience. Thanks. CC: [~madhan] [~kirbyzhou] [~kulkabhay] > hive 'show' sql produces excessive audit log > > > Key: RANGER-3685 > URL: https://issues.apache.org/jira/browse/RANGER-3685 > Project: Ranger > Issue Type: Improvement > Components: audit >Affects Versions: 2.1.0 >Reporter: Xuze Yang >Priority: Major > Attachments: 0001-1.-hive-show-sql.patch, submit patch.pdf > > > Since ranger2.1.0. For "show databases", user needs any permission on > Database to get authorized. RangerHiveAuthorizer.filterListCmdObjects() is > implemented to filter out the database which user don't have access to. > This is a good implementation, but a problem comes with it：the method will > record an audit log for each database(each table for "show tables"). In our > production environment, There are 80,000 tables under a database of hive. A > show tables operation will generate 80001(The extra one is the verification > of USE permissions) audit logs. Unfortunately, our customers will frequently > call the show tables operation. > This brings up two problems: > # Valuable audit logs are flooded > # Take up a lot of storage resources > For problem.2, such a scenario has occurred in our environment: our audit log > destination is down. All audit logs are spooled in disk files, several days > later, the size of the disk file exceeded 800G, causing other components to > fail to provide services normally. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Review Request 74136: RANGER-3685:hive 'show' sql produces excessive audit log
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74136/ --- Review request for ranger, Kirby Zhou, Abhay Kulkarni, and Madhan Neethiraj. Bugs: RANGER-3685 https://issues.apache.org/jira/browse/RANGER-3685 Repository: ranger Description --- hive show sql produces too many audit logs, which will flood valuable audit logs and take up a lot of storage resources. This patch aims to provide a configuration item to support simplified hive audit logging Diffs - agents-common/src/main/java/org/apache/ranger/authorization/hadoop/constants/RangerHadoopConstants.java 6675125e1 hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAuthorizer.java 8f6801be1 Diff: https://reviews.apache.org/r/74136/diff/1/ Testing --- 1. when configuration "xasecure.hive.simplify.audit.of.hive.show.sql" is not set or set to false. The code behaves the same as before,i.e record audit logs for each databases or tables 2. when configuration "xasecure.hive.simplify.audit.of.hive.show.sql" is set to true. It will only record an audit log about USE operation Thanks, Xuze Yang
[jira] [Updated] (RANGER-3685) hive 'show' sql produces excessive audit log
[ https://issues.apache.org/jira/browse/RANGER-3685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuze Yang updated RANGER-3685: -- Attachment: submit patch.pdf > hive 'show' sql produces excessive audit log > > > Key: RANGER-3685 > URL: https://issues.apache.org/jira/browse/RANGER-3685 > Project: Ranger > Issue Type: Improvement > Components: audit >Affects Versions: 2.1.0 > Reporter: Xuze Yang >Priority: Major > Attachments: 0001-1.-hive-show-sql.patch, submit patch.pdf > > > Since ranger2.1.0. For "show databases", user needs any permission on > Database to get authorized. RangerHiveAuthorizer.filterListCmdObjects() is > implemented to filter out the database which user don't have access to. > This is a good implementation, but a problem comes with it：the method will > record an audit log for each database(each table for "show tables"). In our > production environment, There are 80,000 tables under a database of hive. A > show tables operation will generate 80001(The extra one is the verification > of USE permissions) audit logs. Unfortunately, our customers will frequently > call the show tables operation. > This brings up two problems: > # Valuable audit logs are flooded > # Take up a lot of storage resources > For problem.2, such a scenario has occurred in our environment: our audit log > destination is down. All audit logs are spooled in disk files, several days > later, the size of the disk file exceeded 800G, causing other components to > fail to provide services normally. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-3685) hive 'show' sql produces excessive audit log
[ https://issues.apache.org/jira/browse/RANGER-3685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuze Yang updated RANGER-3685: -- Attachment: 0001-1.-hive-show-sql.patch > hive 'show' sql produces excessive audit log > > > Key: RANGER-3685 > URL: https://issues.apache.org/jira/browse/RANGER-3685 > Project: Ranger > Issue Type: Improvement > Components: audit >Affects Versions: 2.1.0 > Reporter: Xuze Yang >Priority: Major > Attachments: 0001-1.-hive-show-sql.patch > > > Since ranger2.1.0. For "show databases", user needs any permission on > Database to get authorized. RangerHiveAuthorizer.filterListCmdObjects() is > implemented to filter out the database which user don't have access to. > This is a good implementation, but a problem comes with it：the method will > record an audit log for each database(each table for "show tables"). In our > production environment, There are 80,000 tables under a database of hive. A > show tables operation will generate 80001(The extra one is the verification > of USE permissions) audit logs. Unfortunately, our customers will frequently > call the show tables operation. > This brings up two problems: > # Valuable audit logs are flooded > # Take up a lot of storage resources > For problem.2, such a scenario has occurred in our environment: our audit log > destination is down. All audit logs are spooled in disk files, several days > later, the size of the disk file exceeded 800G, causing other components to > fail to provide services normally. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (RANGER-3685) hive 'show' sql produces excessive audit log
[
https://issues.apache.org/jira/browse/RANGER-3685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17512701#comment-17512701
]
Xuze Yang commented on RANGER-3685:
---
In addition to the introduction of configuration items, we can also decide
according to the number of objects (size(objs)), if the number is less than a
set value, record the audit log. Otherwise, read the value of the configuration
item to decide whether to record the audit log or not
{code:java}
public List filterListCmdObjects(List
objs,
HiveAuthzContext
context){code}
Is this plan feasible? Any suggestions are welcome.
> hive 'show' sql produces excessive audit log
>
>
> Key: RANGER-3685
> URL: https://issues.apache.org/jira/browse/RANGER-3685
> Project: Ranger
> Issue Type: Improvement
> Components: audit
>Affects Versions: 2.1.0
> Reporter: Xuze Yang
>Priority: Major
>
> Since ranger2.1.0. For "show databases", user needs any permission on
> Database to get authorized. RangerHiveAuthorizer.filterListCmdObjects() is
> implemented to filter out the database which user don't have access to.
> This is a good implementation, but a problem comes with it：the method will
> record an audit log for each database(each table for "show tables"). In our
> production environment, There are 80,000 tables under a database of hive. A
> show tables operation will generate 80001(The extra one is the verification
> of USE permissions) audit logs. Unfortunately, our customers will frequently
> call the show tables operation.
> This brings up two problems:
> # Valuable audit logs are flooded
> # Take up a lot of storage resources
> For problem.2, such a scenario has occurred in our environment: our audit log
> destination is down. All audit logs are spooled in disk files, several days
> later, the size of the disk file exceeded 800G, causing other components to
> fail to provide services normally.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (RANGER-3685) hive 'show' sql produces excessive audit log
[ https://issues.apache.org/jira/browse/RANGER-3685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17512699#comment-17512699 ] Xuze Yang commented on RANGER-3685: --- For this problem, I think a feasible solution is to add a configuration item to decide whether to record this part of the audit log. I think the default value of this configuration item should be false, because this part of the audit log is of little significance (it is enough to keep one audit log for the show operation), and it is easy to cause problems when the scale of the hive databases or tables is large. > hive 'show' sql produces excessive audit log > > > Key: RANGER-3685 > URL: https://issues.apache.org/jira/browse/RANGER-3685 > Project: Ranger > Issue Type: Improvement > Components: audit >Affects Versions: 2.1.0 > Reporter: Xuze Yang >Priority: Major > > Since ranger2.1.0. For "show databases", user needs any permission on > Database to get authorized. RangerHiveAuthorizer.filterListCmdObjects() is > implemented to filter out the database which user don't have access to. > This is a good implementation, but a problem comes with it：the method will > record an audit log for each database(each table for "show tables"). In our > production environment, There are 80,000 tables under a database of hive. A > show tables operation will generate 80001(The extra one is the verification > of USE permissions) audit logs. Unfortunately, our customers will frequently > call the show tables operation. > This brings up two problems: > # Valuable audit logs are flooded > # Take up a lot of storage resources > For problem.2, such a scenario has occurred in our environment: our audit log > destination is down. All audit logs are spooled in disk files, several days > later, the size of the disk file exceeded 800G, causing other components to > fail to provide services normally. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (RANGER-3685) hive 'show' sql produces excessive audit log
Xuze Yang created RANGER-3685: - Summary: hive 'show' sql produces excessive audit log Key: RANGER-3685 URL: https://issues.apache.org/jira/browse/RANGER-3685 Project: Ranger Issue Type: Improvement Components: audit Affects Versions: 2.1.0 Reporter: Xuze Yang Since ranger2.1.0. For "show databases", user needs any permission on Database to get authorized. RangerHiveAuthorizer.filterListCmdObjects() is implemented to filter out the database which user don't have access to. This is a good implementation, but a problem comes with it：the method will record an audit log for each database(each table for "show tables"). In our production environment, There are 80,000 tables under a database of hive. A show tables operation will generate 80001(The extra one is the verification of USE permissions) audit logs. Unfortunately, our customers will frequently call the show tables operation. This brings up two problems: # Valuable audit logs are flooded # Take up a lot of storage resources For problem.2, such a scenario has occurred in our environment: our audit log destination is down. All audit logs are spooled in disk files, several days later, the size of the disk file exceeded 800G, causing other components to fail to provide services normally. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Comment Edited] (RANGER-3681) Ranger Database deadlock when createPolicy is running parallel
[
https://issues.apache.org/jira/browse/RANGER-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17511620#comment-17511620
]
Xuze Yang edited comment on RANGER-3681 at 3/26/22, 7:08 AM:
-
add some details：
# I wrote a test demo which calles the createPolicy() method 100 times using
two threads. The results showed that only half of the policies were
successfully created, and the other half failed to create due to deadlock.
# When I change the isolation level of mysql database session to rc
(READ-COMMITTED), there will be no deadlock exception.
# The structure of the x_policy_ref_resource table is:
{noformat}
CREATE TABLE `x_policy_ref_resource` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`guid` varchar(1024) DEFAULT NULL,
`create_time` datetime DEFAULT NULL,
`update_time` datetime DEFAULT NULL,
`added_by_id` bigint(20) DEFAULT NULL,
`upd_by_id` bigint(20) DEFAULT NULL,
`policy_id` bigint(20) NOT NULL,
`resource_def_id` bigint(20) NOT NULL,
`resource_name` varchar(4000) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `x_policy_ref_res_UK_polId_resDefId`
(`policy_id`,`resource_def_id`),
KEY `x_policy_ref_res_FK_added_by_id` (`added_by_id`),
KEY `x_policy_ref_res_FK_upd_by_id` (`upd_by_id`),
KEY `x_policy_ref_res_FK_resource_def_id` (`resource_def_id`),
CONSTRAINT `x_policy_ref_res_FK_added_by_id` FOREIGN KEY (`added_by_id`)
REFERENCES `x_portal_user` (`id`),
CONSTRAINT `x_policy_ref_res_FK_policy_id` FOREIGN KEY (`policy_id`)
REFERENCES `x_policy` (`id`),
CONSTRAINT `x_policy_ref_res_FK_resource_def_id` FOREIGN KEY
(`resource_def_id`) REFERENCES `x_resource_def` (`id`),
CONSTRAINT `x_policy_ref_res_FK_upd_by_id` FOREIGN KEY (`upd_by_id`)
REFERENCES `x_portal_user` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=7654 DEFAULT CHARSET=utf8
ROW_FORMAT=DYNAMIC{noformat}
was (Author: xuze yang):
add some details：
# I wrote a test demo which calles the createPolicy() method 100 times using
two threads. The results showed that only half of the policies were
successfully created, and the other half failed to create due to deadlock.
# When I change the isolation level of mysql database session to rc
(READ-COMMITTED), there will be no deadlock exception.
# The structure of the x_policy_ref_resource table is:
{noformat}
CREATE TABLE `x_policy_ref_resource` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`guid` varchar(1024) DEFAULT NULL,
`create_time` datetime DEFAULT NULL,
`update_time` datetime DEFAULT NULL,
`added_by_id` bigint(20) DEFAULT NULL,
`upd_by_id` bigint(20) DEFAULT NULL,
`policy_id` bigint(20) NOT NULL,
`resource_def_id` bigint(20) NOT NULL,
`resource_name` varchar(4000) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `x_policy_ref_res_UK_polId_resDefId`
(`policy_id`,`resource_def_id`),
KEY `x_policy_ref_res_FK_added_by_id` (`added_by_id`),
KEY `x_policy_ref_res_FK_upd_by_id` (`upd_by_id`),
KEY `x_policy_ref_res_FK_resource_def_id` (`resource_def_id`),
CONSTRAINT `x_policy_ref_res_FK_added_by_id` FOREIGN KEY (`added_by_id`)
REFERENCES `x_portal_user` (`id`),
CONSTRAINT `x_policy_ref_res_FK_policy_id` FOREIGN KEY (`policy_id`)
REFERENCES `x_policy` (`id`),
CONSTRAINT `x_policy_ref_res_FK_resource_def_id` FOREIGN KEY
(`resource_def_id`) REFERENCES `x_resource_def` (`id`),
CONSTRAINT `x_policy_ref_res_FK_upd_by_id` FOREIGN KEY (`upd_by_id`)
REFERENCES `x_portal_user` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=7654 DEFAULT CHARSET=utf8
ROW_FORMAT=DYNAMIC{noformat}
> Ranger Database deadlock when createPolicy is running parallel
> --
>
> Key: RANGER-3681
> URL: https://issues.apache.org/jira/browse/RANGER-3681
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Affects Versions: 2.1.0
> Environment: ranger-version: 2.1.0
> mysql-verison: 5.7.34-log
> mysql-engine: InnoDB
> mysql-isolation-level: REPEATABLE-READ
>Reporter: Xuze Yang
>Assignee: Xuze Yang
>Priority: Major
> Attachments: 0001-RANGER-3681-Ranger Database deadlock when
> createPolicy is running parallel.patch, 0002-RANGER-3681-Ranger Database
> deadlock when createPolicy is running parallel.patch,
> 0003-RANGER-3681-Ranger-Database-deadlock-when-createPolicy-is-running-parallel.patch
>
>
> h1. Error
> ranger-log：
> {noformat}
> 2022-03-21 20:14:29,685 [http-bio-6080-exec-13] ERROR
> org.apache.ranger.rest.ServiceREST (ServiceREST.java:1709) -
> createPolicy(RangerPolicy={id={null} guid={null} isEnabled={true}
> createdBy={null} updatedBy={null} createTime={null} updateTime={null}
> version={1} service={default-Hive}
> name={dcp-desensitize_a162c40cdc0140b1848b9841557
[jira] [Updated] (RANGER-3681) Ranger Database deadlock when createPolicy is running parallel
[
https://issues.apache.org/jira/browse/RANGER-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuze Yang updated RANGER-3681:
--
Attachment: 0002-RANGER-3681-Ranger Database deadlock when createPolicy is
running parallel.patch
0003-RANGER-3681-Ranger-Database-deadlock-when-createPolicy-is-running-parallel.patch
> Ranger Database deadlock when createPolicy is running parallel
> --
>
> Key: RANGER-3681
> URL: https://issues.apache.org/jira/browse/RANGER-3681
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Affects Versions: 2.1.0
> Environment: ranger-version: 2.1.0
> mysql-verison: 5.7.34-log
> mysql-engine: InnoDB
> mysql-isolation-level: REPEATABLE-READ
>Reporter: Xuze Yang
>Assignee: Xuze Yang
>Priority: Major
> Attachments: 0001-RANGER-3681-Ranger Database deadlock when
> createPolicy is running parallel.patch, 0002-RANGER-3681-Ranger Database
> deadlock when createPolicy is running parallel.patch,
> 0003-RANGER-3681-Ranger-Database-deadlock-when-createPolicy-is-running-parallel.patch
>
>
> h1. Error
> ranger-log：
> {noformat}
> 2022-03-21 20:14:29,685 [http-bio-6080-exec-13] ERROR
> org.apache.ranger.rest.ServiceREST (ServiceREST.java:1709) -
> createPolicy(RangerPolicy={id={null} guid={null} isEnabled={true}
> createdBy={null} updatedBy={null} createTime={null} updateTime={null}
> version={1} service={default-Hive}
> name={dcp-desensitize_a162c40cdc0140b1848b98415575be6c-1647864869626}
> policyType={0} policyPriority={0} description={}
> resourceSignature={4f15e3de95c81650ad869cb93a8c47a132bbec54bdf5de8c01f5075c19754cd7}
> isAuditEnabled={true} serviceType={null}
> resources={database={RangerPolicyResource={values={dcp } isExcludes={false}
> isRecursive={false} }} column={RangerPolicyResource={values={* }
> isExcludes={false} isRecursive={false} }}
> table={RangerPolicyResource={values={desensitize_a162c40cdc0140b1848b98415575be6c
> } isExcludes={false} isRecursive={false} }} } policyLabels={Consoler }
> policyConditions={}
> policyItems={RangerPolicyItem={accessTypes={RangerPolicyItemAccess={type={all}
> isAllowed={true} }} users={tangbiao2 } groups={} roles={} conditions={}
> delegateAdmin={false} }} denyPolicyItems={} allowExceptions={}
> denyExceptions={} dataMaskPolicyItems={} rowFilterPolicyItems={} options={}
> validitySchedules={, zoneName=null, isDenyAllElse={false} }}) failed
> javax.persistence.PersistenceException: Exception [EclipseLink-4002] (Eclipse
> Persistence Services - 2.5.2.v20140319-9ad6abd):
> org.eclipse.persistence.exceptions.DatabaseException
> Internal Exception:
> com.mysql.cj.jdbc.exceptions.MySQLTransactionRollbackException: Deadlock
> found when trying to get lock; try restarting transaction
> Error Code: 1213
> Call: INSERT INTO x_policy_ref_resource (ADDED_BY_ID, CREATE_TIME, policy_id,
> resource_def_id, resource_name, UPDATE_TIME, UPD_BY_ID) VALUES (?, ?, ?, ?,
> ?, ?, ?)
> bind => [7 parameters bound]
> Query: ValueReadQuery(name="x_policy_ref_resource_SEQ" sql="SELECT
> LAST_INSERT_ID()")
> at
> org.eclipse.persistence.internal.jpa.EntityManagerImpl.flush(EntityManagerImpl.java:868)
> at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.springframework.orm.jpa.SharedEntityManagerCreator$SharedEntityManagerInvocationHandler.invoke(SharedEntityManagerCreator.java:301)
> at com.sun.proxy.$Proxy30.flush(Unknown Source)
> at org.apache.ranger.common.db.BaseDao.batchCreate(BaseDao.java:102)
> ...{noformat}
> mysql-deadlock：
> {noformat}
>
> LATEST DETECTED DEADLOCK
>
> 2022-03-21 09:47:22 0x7ff3a4859700
> *** (1) TRANSACTION:
> TRANSACTION 7036760, ACTIVE 0 sec inserting
> mysql tables in use 1, locked 1
> LOCK WAIT 23 lock struct(s), heap size 1136, 12 row lock(s), undo log entries
> 2
> MySQL thread id 27293, OS thread handle 140684415063808, query id 383930
> 192.168.0.76 DHCloudBG update
> INSERT INTO x_policy_ref_resource (ADDED_BY_ID, CREATE_TIME, policy_id,
> resource_def_id, resource_name, UPDATE_TIME, UPD_BY_ID) VALUES (1,
> '2022-03-20 12:47:22.666', 13921, 5, 'database', '2022-03-20 12:47:22.681', 1)
> *** (1) WAITING FOR THIS LOCK TO BE GRANTED:
> RECORD LOCKS space id 531 page no 4
Re: Review Request 73913: RANGER-3681:Ranger Database deadlock when createPolicy is running parallel
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73913/ --- (Updated 三月 26, 2022, 5:35 a.m.) Review request for ranger, Abhay Kulkarni, Madhan Neethiraj, and Velmurugan Periasamy. Changes --- delete unnecessary commented code; remove unnecessary exception logic; correct the file path Bugs: RANGER-3681 https://issues.apache.org/jira/browse/RANGER-3681 Repository: ranger Description --- Modify delete sql to delete according to the primary key, to solve the problem of deadlock under REPEATABLE-READ isolation level Diffs (updated) - security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefAccessTypeDao.java b9a60cb security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefConditionDao.java e14bc14 security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefDataMaskTypeDao.java 7e7b8d4 security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefGroupDao.java 5f9d9ed security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefResourceDao.java 0ea7de9 security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefRoleDao.java 3ae7e7a security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefUserDao.java 518139a security-admin/src/main/resources/META-INF/jpa_named_queries.xml 5813209 Diff: https://reviews.apache.org/r/73913/diff/3/ Changes: https://reviews.apache.org/r/73913/diff/2-3/ Testing --- 1. Calles the createPolicy() method 1000 times using two threads. The results showed that all policies are successfully created without deadlock exception. 2. Calles the updatePolicy() method 1000 times using two threads. The results showed that all policies are successfully updated without deadlock exception. 3. Calles the deletePolicy() method 1000 times using two threads. The results showed that all policies are successfully deleted. Thanks, Xuze Yang
Re: Review Request 73913: RANGER-3681:Ranger Database deadlock when createPolicy is running parallel
> On 三月 25, 2022, 4:23 a.m., Madhan Neethiraj wrote: > > src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefAccessTypeDao.java > > Lines 108 (patched) > > <https://reviews.apache.org/r/73913/diff/1/?file=2266989#file2266989line108> > > > > findByPolicyId(policyId) returns XXPolicyRefAccessType objects, which > > is unnecessary here since only id is needed. Consider adding a new named > > query to return ids. > > > > Also, would replacing current query in > > "XXPolicyRefAccessType.deleteByPolicyId" with the following help avoid gap > > lock, since SELECT in sub-query might not be treated as a 'locking read'? > > > > current: > > DELETE FROM XXPolicyRefGroup obj WHERE obj.policyId = :policyId > > > > replace with: > > DELETE FROM XXPolicyRefGroup obj > >WHERE obj.id in (SELECT obj2.id FROM XXPolicyRefGroup obj2 WHERE > > obj2.policyId = :policyId) > > Xuze Yang wrote: > 1. I tried this sql in my environment. > ```sql > DELETE FROM XXPolicyRefResource obj WHERE obj.id in (SELECT obj2.id FROM > XXPolicyRefResource obj2 WHERE obj2.policyId = :policyId) > ``` >The results show that the deadlock problem still exists. >The reason is myql side will decompose this sql into 4 stages: >* create temporary table >* insert into temporary table >* delete from x_policy_ref_resource using temporary table >* delete temporary table > 2. I have updated the patch, the main change is adding a new named query > to return ids. > 3. The same tests were done as desbribed above, and everything went well. > > Madhan Neethiraj wrote: > Thank you for the details. The fix looks good! > > The patch has prefix "src/" in each file, which causes 'git apply' to > fail. Can you please create the patch from Ranger git directory and upload? > Thanks! Thanks for the reminder, I have corrected the error on the path, please check - Xuze --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73913/#review224207 --- On 三月 25, 2022, 9:33 a.m., Xuze Yang wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73913/ > --- > > (Updated 三月 25, 2022, 9:33 a.m.) > > > Review request for ranger, Abhay Kulkarni, Madhan Neethiraj, and Velmurugan > Periasamy. > > > Bugs: RANGER-3681 > https://issues.apache.org/jira/browse/RANGER-3681 > > > Repository: ranger > > > Description > --- > > Modify delete sql to delete according to the primary key, to solve the > problem of deadlock under REPEATABLE-READ isolation level > > > Diffs > - > > > src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefAccessTypeDao.java > b9a60cb > > src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefConditionDao.java > e14bc14 > > src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefDataMaskTypeDao.java > 7e7b8d4 > > src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefGroupDao.java > 5f9d9ed > > src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefResourceDao.java > 0ea7de9 > > src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefRoleDao.java > 3ae7e7a > > src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefUserDao.java > 518139a > src/security-admin/src/main/resources/META-INF/jpa_named_queries.xml > 5813209 > > > Diff: https://reviews.apache.org/r/73913/diff/2/ > > > Testing > --- > > 1. Calles the createPolicy() method 1000 times using two threads. The results > showed that all policies are successfully created without deadlock exception. > 2. Calles the updatePolicy() method 1000 times using two threads. The results > showed that all policies are successfully updated without deadlock exception. > 3. Calles the deletePolicy() method 1000 times using two threads. The results > showed that all policies are successfully deleted. > > > Thanks, > > Xuze Yang > >
Re: Review Request 73913: RANGER-3681:Ranger Database deadlock when createPolicy is running parallel
> On 三月 26, 2022, 12:45 a.m., Kirby Zhou wrote: > > src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefRoleDao.java > > Lines 124 (patched) > > <https://reviews.apache.org/r/73913/diff/2/?file=2267007#file2267007line124> > > > > It seems unnecessery to catch NoResultException when call > > getResultList, it is only happens when call getSingleResult Yeah, thank you for your correction, I have removed the relevant exception logic, please check. - Xuze --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73913/#review224216 --- On 三月 25, 2022, 9:33 a.m., Xuze Yang wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73913/ > --- > > (Updated 三月 25, 2022, 9:33 a.m.) > > > Review request for ranger, Abhay Kulkarni, Madhan Neethiraj, and Velmurugan > Periasamy. > > > Bugs: RANGER-3681 > https://issues.apache.org/jira/browse/RANGER-3681 > > > Repository: ranger > > > Description > --- > > Modify delete sql to delete according to the primary key, to solve the > problem of deadlock under REPEATABLE-READ isolation level > > > Diffs > - > > > src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefAccessTypeDao.java > b9a60cb > > src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefConditionDao.java > e14bc14 > > src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefDataMaskTypeDao.java > 7e7b8d4 > > src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefGroupDao.java > 5f9d9ed > > src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefResourceDao.java > 0ea7de9 > > src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefRoleDao.java > 3ae7e7a > > src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefUserDao.java > 518139a > src/security-admin/src/main/resources/META-INF/jpa_named_queries.xml > 5813209 > > > Diff: https://reviews.apache.org/r/73913/diff/2/ > > > Testing > --- > > 1. Calles the createPolicy() method 1000 times using two threads. The results > showed that all policies are successfully created without deadlock exception. > 2. Calles the updatePolicy() method 1000 times using two threads. The results > showed that all policies are successfully updated without deadlock exception. > 3. Calles the deletePolicy() method 1000 times using two threads. The results > showed that all policies are successfully deleted. > > > Thanks, > > Xuze Yang > >
Re: Review Request 73913: RANGER-3681:Ranger Database deadlock when createPolicy is running parallel
> On 三月 25, 2022, 4:20 p.m., Abhay Kulkarni wrote: > > src/security-admin/src/main/resources/META-INF/jpa_named_queries.xml > > Line 1669 (original), 1697 (patched) > > <https://reviews.apache.org/r/73913/diff/1-2/?file=2266996#file2266996line1697> > > > > The commented lines in this file and elsewhere (perhaps left there to > > keep history of the change) may be deleted. It will reduce the clutter in > > the source code. I have deleted the commented lines in jpa_named_queries, please check, thanks - Xuze --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73913/#review224215 --- On 三月 25, 2022, 9:33 a.m., Xuze Yang wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73913/ > --- > > (Updated 三月 25, 2022, 9:33 a.m.) > > > Review request for ranger, Abhay Kulkarni, Madhan Neethiraj, and Velmurugan > Periasamy. > > > Bugs: RANGER-3681 > https://issues.apache.org/jira/browse/RANGER-3681 > > > Repository: ranger > > > Description > --- > > Modify delete sql to delete according to the primary key, to solve the > problem of deadlock under REPEATABLE-READ isolation level > > > Diffs > - > > > src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefAccessTypeDao.java > b9a60cb > > src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefConditionDao.java > e14bc14 > > src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefDataMaskTypeDao.java > 7e7b8d4 > > src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefGroupDao.java > 5f9d9ed > > src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefResourceDao.java > 0ea7de9 > > src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefRoleDao.java > 3ae7e7a > > src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefUserDao.java > 518139a > src/security-admin/src/main/resources/META-INF/jpa_named_queries.xml > 5813209 > > > Diff: https://reviews.apache.org/r/73913/diff/2/ > > > Testing > --- > > 1. Calles the createPolicy() method 1000 times using two threads. The results > showed that all policies are successfully created without deadlock exception. > 2. Calles the updatePolicy() method 1000 times using two threads. The results > showed that all policies are successfully updated without deadlock exception. > 3. Calles the deletePolicy() method 1000 times using two threads. The results > showed that all policies are successfully deleted. > > > Thanks, > > Xuze Yang > >
Re: Review Request 73913: RANGER-3681:Ranger Database deadlock when createPolicy is running parallel
> On 三月 25, 2022, 4:20 p.m., Abhay Kulkarni wrote: > > src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefDataMaskTypeDao.java > > Line 97 (original), 95 (patched) > > <https://reviews.apache.org/r/73913/diff/1-2/?file=2266991#file2266991line97> > > > > This is not really related to this part of code. However, has this > > patch been tested with the database isolation level set to READ_COMMITTED? > > I don't expect any regression, but it will be worthwhile to document that > > as well. I set the isolation level of mysql to rc, and ran the tests described above. The results show that everything is working fine. - Xuze --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73913/#review224215 --- On 三月 25, 2022, 9:33 a.m., Xuze Yang wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73913/ > --- > > (Updated 三月 25, 2022, 9:33 a.m.) > > > Review request for ranger, Abhay Kulkarni, Madhan Neethiraj, and Velmurugan > Periasamy. > > > Bugs: RANGER-3681 > https://issues.apache.org/jira/browse/RANGER-3681 > > > Repository: ranger > > > Description > --- > > Modify delete sql to delete according to the primary key, to solve the > problem of deadlock under REPEATABLE-READ isolation level > > > Diffs > - > > > src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefAccessTypeDao.java > b9a60cb > > src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefConditionDao.java > e14bc14 > > src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefDataMaskTypeDao.java > 7e7b8d4 > > src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefGroupDao.java > 5f9d9ed > > src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefResourceDao.java > 0ea7de9 > > src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefRoleDao.java > 3ae7e7a > > src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefUserDao.java > 518139a > src/security-admin/src/main/resources/META-INF/jpa_named_queries.xml > 5813209 > > > Diff: https://reviews.apache.org/r/73913/diff/2/ > > > Testing > --- > > 1. Calles the createPolicy() method 1000 times using two threads. The results > showed that all policies are successfully created without deadlock exception. > 2. Calles the updatePolicy() method 1000 times using two threads. The results > showed that all policies are successfully updated without deadlock exception. > 3. Calles the deletePolicy() method 1000 times using two threads. The results > showed that all policies are successfully deleted. > > > Thanks, > > Xuze Yang > >
Re: Review Request 73913: RANGER-3681:Ranger Database deadlock when createPolicy is running parallel
> On 三月 25, 2022, 5:55 a.m., Kirby Zhou wrote: > > Can you add some comments to code to explain? > > I'm worried that someone will delete this code as redundant code in the > > future. Thanks for your review and suggetion, I have added some comments in this code. Please check whether it is sufficient. - Xuze --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73913/#review224208 --- On 三月 25, 2022, 9:33 a.m., Xuze Yang wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73913/ > --- > > (Updated 三月 25, 2022, 9:33 a.m.) > > > Review request for ranger, Abhay Kulkarni, Madhan Neethiraj, and Velmurugan > Periasamy. > > > Bugs: RANGER-3681 > https://issues.apache.org/jira/browse/RANGER-3681 > > > Repository: ranger > > > Description > --- > > Modify delete sql to delete according to the primary key, to solve the > problem of deadlock under REPEATABLE-READ isolation level > > > Diffs > - > > > src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefAccessTypeDao.java > b9a60cb > > src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefConditionDao.java > e14bc14 > > src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefDataMaskTypeDao.java > 7e7b8d4 > > src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefGroupDao.java > 5f9d9ed > > src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefResourceDao.java > 0ea7de9 > > src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefRoleDao.java > 3ae7e7a > > src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefUserDao.java > 518139a > src/security-admin/src/main/resources/META-INF/jpa_named_queries.xml > 5813209 > > > Diff: https://reviews.apache.org/r/73913/diff/2/ > > > Testing > --- > > 1. Calles the createPolicy() method 1000 times using two threads. The results > showed that all policies are successfully created without deadlock exception. > 2. Calles the updatePolicy() method 1000 times using two threads. The results > showed that all policies are successfully updated without deadlock exception. > 3. Calles the deletePolicy() method 1000 times using two threads. The results > showed that all policies are successfully deleted. > > > Thanks, > > Xuze Yang > >
Re: Review Request 73913: RANGER-3681:Ranger Database deadlock when createPolicy is running parallel
> On 三月 25, 2022, 4:23 a.m., Madhan Neethiraj wrote: > > src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefAccessTypeDao.java > > Lines 108 (patched) > > <https://reviews.apache.org/r/73913/diff/1/?file=2266989#file2266989line108> > > > > findByPolicyId(policyId) returns XXPolicyRefAccessType objects, which > > is unnecessary here since only id is needed. Consider adding a new named > > query to return ids. > > > > Also, would replacing current query in > > "XXPolicyRefAccessType.deleteByPolicyId" with the following help avoid gap > > lock, since SELECT in sub-query might not be treated as a 'locking read'? > > > > current: > > DELETE FROM XXPolicyRefGroup obj WHERE obj.policyId = :policyId > > > > replace with: > > DELETE FROM XXPolicyRefGroup obj > >WHERE obj.id in (SELECT obj2.id FROM XXPolicyRefGroup obj2 WHERE > > obj2.policyId = :policyId) 1. I tried this sql in my environment. ```sql DELETE FROM XXPolicyRefResource obj WHERE obj.id in (SELECT obj2.id FROM XXPolicyRefResource obj2 WHERE obj2.policyId = :policyId) ``` The results show that the deadlock problem still exists. The reason is myql side will decompose this sql into 4 stages: * create temporary table * insert into temporary table * delete from x_policy_ref_resource using temporary table * delete temporary table 2. I have updated the patch, the main change is adding a new named query to return ids. 3. The same tests were done as desbribed above, and everything went well. - Xuze --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73913/#review224207 --- On 三月 25, 2022, 9:33 a.m., Xuze Yang wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73913/ > --- > > (Updated 三月 25, 2022, 9:33 a.m.) > > > Review request for ranger, Abhay Kulkarni, Madhan Neethiraj, and Velmurugan > Periasamy. > > > Bugs: RANGER-3681 > https://issues.apache.org/jira/browse/RANGER-3681 > > > Repository: ranger > > > Description > --- > > Modify delete sql to delete according to the primary key, to solve the > problem of deadlock under REPEATABLE-READ isolation level > > > Diffs > - > > > src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefAccessTypeDao.java > b9a60cb > > src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefConditionDao.java > e14bc14 > > src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefDataMaskTypeDao.java > 7e7b8d4 > > src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefGroupDao.java > 5f9d9ed > > src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefResourceDao.java > 0ea7de9 > > src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefRoleDao.java > 3ae7e7a > > src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefUserDao.java > 518139a > src/security-admin/src/main/resources/META-INF/jpa_named_queries.xml > 5813209 > > > Diff: https://reviews.apache.org/r/73913/diff/2/ > > > Testing > --- > > 1. Calles the createPolicy() method 1000 times using two threads. The results > showed that all policies are successfully created without deadlock exception. > 2. Calles the updatePolicy() method 1000 times using two threads. The results > showed that all policies are successfully updated without deadlock exception. > 3. Calles the deletePolicy() method 1000 times using two threads. The results > showed that all policies are successfully deleted. > > > Thanks, > > Xuze Yang > >
Re: Review Request 73913: RANGER-3681:Ranger Database deadlock when createPolicy is running parallel
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73913/ --- (Updated 三月 25, 2022, 9:33 a.m.) Review request for ranger, Abhay Kulkarni, Madhan Neethiraj, and Velmurugan Periasamy. Changes --- 1. add a new named query to return ids 2. add some comments in code Bugs: RANGER-3681 https://issues.apache.org/jira/browse/RANGER-3681 Repository: ranger Description --- Modify delete sql to delete according to the primary key, to solve the problem of deadlock under REPEATABLE-READ isolation level Diffs (updated) - src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefAccessTypeDao.java b9a60cb src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefConditionDao.java e14bc14 src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefDataMaskTypeDao.java 7e7b8d4 src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefGroupDao.java 5f9d9ed src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefResourceDao.java 0ea7de9 src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefRoleDao.java 3ae7e7a src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefUserDao.java 518139a src/security-admin/src/main/resources/META-INF/jpa_named_queries.xml 5813209 Diff: https://reviews.apache.org/r/73913/diff/2/ Changes: https://reviews.apache.org/r/73913/diff/1-2/ Testing --- 1. Calles the createPolicy() method 1000 times using two threads. The results showed that all policies are successfully created without deadlock exception. 2. Calles the updatePolicy() method 1000 times using two threads. The results showed that all policies are successfully updated without deadlock exception. 3. Calles the deletePolicy() method 1000 times using two threads. The results showed that all policies are successfully deleted. Thanks, Xuze Yang
[jira] [Commented] (RANGER-3681) Ranger Database deadlock when createPolicy is running parallel
[
https://issues.apache.org/jira/browse/RANGER-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17512172#comment-17512172
]
Xuze Yang commented on RANGER-3681:
---
review request available at:
[https://reviews.apache.org/r/73913/]
> Ranger Database deadlock when createPolicy is running parallel
> --
>
> Key: RANGER-3681
> URL: https://issues.apache.org/jira/browse/RANGER-3681
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Affects Versions: 2.1.0
> Environment: ranger-version: 2.1.0
> mysql-verison: 5.7.34-log
> mysql-engine: InnoDB
> mysql-isolation-level: REPEATABLE-READ
>Reporter: Xuze Yang
>Assignee: Xuze Yang
>Priority: Major
> Attachments: 0001-RANGER-3681-Ranger Database deadlock when
> createPolicy is running parallel.patch
>
>
> h1. Error
> ranger-log：
> {noformat}
> 2022-03-21 20:14:29,685 [http-bio-6080-exec-13] ERROR
> org.apache.ranger.rest.ServiceREST (ServiceREST.java:1709) -
> createPolicy(RangerPolicy={id={null} guid={null} isEnabled={true}
> createdBy={null} updatedBy={null} createTime={null} updateTime={null}
> version={1} service={default-Hive}
> name={dcp-desensitize_a162c40cdc0140b1848b98415575be6c-1647864869626}
> policyType={0} policyPriority={0} description={}
> resourceSignature={4f15e3de95c81650ad869cb93a8c47a132bbec54bdf5de8c01f5075c19754cd7}
> isAuditEnabled={true} serviceType={null}
> resources={database={RangerPolicyResource={values={dcp } isExcludes={false}
> isRecursive={false} }} column={RangerPolicyResource={values={* }
> isExcludes={false} isRecursive={false} }}
> table={RangerPolicyResource={values={desensitize_a162c40cdc0140b1848b98415575be6c
> } isExcludes={false} isRecursive={false} }} } policyLabels={Consoler }
> policyConditions={}
> policyItems={RangerPolicyItem={accessTypes={RangerPolicyItemAccess={type={all}
> isAllowed={true} }} users={tangbiao2 } groups={} roles={} conditions={}
> delegateAdmin={false} }} denyPolicyItems={} allowExceptions={}
> denyExceptions={} dataMaskPolicyItems={} rowFilterPolicyItems={} options={}
> validitySchedules={, zoneName=null, isDenyAllElse={false} }}) failed
> javax.persistence.PersistenceException: Exception [EclipseLink-4002] (Eclipse
> Persistence Services - 2.5.2.v20140319-9ad6abd):
> org.eclipse.persistence.exceptions.DatabaseException
> Internal Exception:
> com.mysql.cj.jdbc.exceptions.MySQLTransactionRollbackException: Deadlock
> found when trying to get lock; try restarting transaction
> Error Code: 1213
> Call: INSERT INTO x_policy_ref_resource (ADDED_BY_ID, CREATE_TIME, policy_id,
> resource_def_id, resource_name, UPDATE_TIME, UPD_BY_ID) VALUES (?, ?, ?, ?,
> ?, ?, ?)
> bind => [7 parameters bound]
> Query: ValueReadQuery(name="x_policy_ref_resource_SEQ" sql="SELECT
> LAST_INSERT_ID()")
> at
> org.eclipse.persistence.internal.jpa.EntityManagerImpl.flush(EntityManagerImpl.java:868)
> at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.springframework.orm.jpa.SharedEntityManagerCreator$SharedEntityManagerInvocationHandler.invoke(SharedEntityManagerCreator.java:301)
> at com.sun.proxy.$Proxy30.flush(Unknown Source)
> at org.apache.ranger.common.db.BaseDao.batchCreate(BaseDao.java:102)
> ...{noformat}
> mysql-deadlock：
> {noformat}
>
> LATEST DETECTED DEADLOCK
>
> 2022-03-21 09:47:22 0x7ff3a4859700
> *** (1) TRANSACTION:
> TRANSACTION 7036760, ACTIVE 0 sec inserting
> mysql tables in use 1, locked 1
> LOCK WAIT 23 lock struct(s), heap size 1136, 12 row lock(s), undo log entries
> 2
> MySQL thread id 27293, OS thread handle 140684415063808, query id 383930
> 192.168.0.76 DHCloudBG update
> INSERT INTO x_policy_ref_resource (ADDED_BY_ID, CREATE_TIME, policy_id,
> resource_def_id, resource_name, UPDATE_TIME, UPD_BY_ID) VALUES (1,
> '2022-03-20 12:47:22.666', 13921, 5, 'database', '2022-03-20 12:47:22.681', 1)
> *** (1) WAITING FOR THIS LOCK TO BE GRANTED:
> RECORD LOCKS space id 531 page no 4 n bits 376 index
> x_policy_ref_res_UK_polId_resDefId of table `ranger`.`x_policy_ref_resource`
> trx id 7036760 lock_mode X insert intention waiting
> Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits > 0
> 0: len 8; hex 73757072656d756d; asc
[jira] [Updated] (RANGER-3681) Ranger Database deadlock when createPolicy is running parallel
[
https://issues.apache.org/jira/browse/RANGER-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuze Yang updated RANGER-3681:
--
Attachment: 0001-RANGER-3681-Ranger Database deadlock when createPolicy is
running parallel.patch
> Ranger Database deadlock when createPolicy is running parallel
> --
>
> Key: RANGER-3681
> URL: https://issues.apache.org/jira/browse/RANGER-3681
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Affects Versions: 2.1.0
> Environment: ranger-version: 2.1.0
> mysql-verison: 5.7.34-log
> mysql-engine: InnoDB
> mysql-isolation-level: REPEATABLE-READ
>Reporter: Xuze Yang
>Assignee: Xuze Yang
>Priority: Major
> Attachments: 0001-RANGER-3681-Ranger Database deadlock when
> createPolicy is running parallel.patch
>
>
> h1. Error
> ranger-log：
> {noformat}
> 2022-03-21 20:14:29,685 [http-bio-6080-exec-13] ERROR
> org.apache.ranger.rest.ServiceREST (ServiceREST.java:1709) -
> createPolicy(RangerPolicy={id={null} guid={null} isEnabled={true}
> createdBy={null} updatedBy={null} createTime={null} updateTime={null}
> version={1} service={default-Hive}
> name={dcp-desensitize_a162c40cdc0140b1848b98415575be6c-1647864869626}
> policyType={0} policyPriority={0} description={}
> resourceSignature={4f15e3de95c81650ad869cb93a8c47a132bbec54bdf5de8c01f5075c19754cd7}
> isAuditEnabled={true} serviceType={null}
> resources={database={RangerPolicyResource={values={dcp } isExcludes={false}
> isRecursive={false} }} column={RangerPolicyResource={values={* }
> isExcludes={false} isRecursive={false} }}
> table={RangerPolicyResource={values={desensitize_a162c40cdc0140b1848b98415575be6c
> } isExcludes={false} isRecursive={false} }} } policyLabels={Consoler }
> policyConditions={}
> policyItems={RangerPolicyItem={accessTypes={RangerPolicyItemAccess={type={all}
> isAllowed={true} }} users={tangbiao2 } groups={} roles={} conditions={}
> delegateAdmin={false} }} denyPolicyItems={} allowExceptions={}
> denyExceptions={} dataMaskPolicyItems={} rowFilterPolicyItems={} options={}
> validitySchedules={, zoneName=null, isDenyAllElse={false} }}) failed
> javax.persistence.PersistenceException: Exception [EclipseLink-4002] (Eclipse
> Persistence Services - 2.5.2.v20140319-9ad6abd):
> org.eclipse.persistence.exceptions.DatabaseException
> Internal Exception:
> com.mysql.cj.jdbc.exceptions.MySQLTransactionRollbackException: Deadlock
> found when trying to get lock; try restarting transaction
> Error Code: 1213
> Call: INSERT INTO x_policy_ref_resource (ADDED_BY_ID, CREATE_TIME, policy_id,
> resource_def_id, resource_name, UPDATE_TIME, UPD_BY_ID) VALUES (?, ?, ?, ?,
> ?, ?, ?)
> bind => [7 parameters bound]
> Query: ValueReadQuery(name="x_policy_ref_resource_SEQ" sql="SELECT
> LAST_INSERT_ID()")
> at
> org.eclipse.persistence.internal.jpa.EntityManagerImpl.flush(EntityManagerImpl.java:868)
> at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.springframework.orm.jpa.SharedEntityManagerCreator$SharedEntityManagerInvocationHandler.invoke(SharedEntityManagerCreator.java:301)
> at com.sun.proxy.$Proxy30.flush(Unknown Source)
> at org.apache.ranger.common.db.BaseDao.batchCreate(BaseDao.java:102)
> ...{noformat}
> mysql-deadlock：
> {noformat}
>
> LATEST DETECTED DEADLOCK
>
> 2022-03-21 09:47:22 0x7ff3a4859700
> *** (1) TRANSACTION:
> TRANSACTION 7036760, ACTIVE 0 sec inserting
> mysql tables in use 1, locked 1
> LOCK WAIT 23 lock struct(s), heap size 1136, 12 row lock(s), undo log entries
> 2
> MySQL thread id 27293, OS thread handle 140684415063808, query id 383930
> 192.168.0.76 DHCloudBG update
> INSERT INTO x_policy_ref_resource (ADDED_BY_ID, CREATE_TIME, policy_id,
> resource_def_id, resource_name, UPDATE_TIME, UPD_BY_ID) VALUES (1,
> '2022-03-20 12:47:22.666', 13921, 5, 'database', '2022-03-20 12:47:22.681', 1)
> *** (1) WAITING FOR THIS LOCK TO BE GRANTED:
> RECORD LOCKS space id 531 page no 4 n bits 376 index
> x_policy_ref_res_UK_polId_resDefId of table `ranger`.`x_policy_ref_resource`
> trx id 7036760 lock_mode X insert intention waiting
> Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits > 0
> 0: len 8; hex 73757072656d756d; asc
Review Request 73913: RANGER-3681:Ranger Database deadlock when createPolicy is running parallel
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73913/ --- Review request for ranger, Abhay Kulkarni, Madhan Neethiraj, and Velmurugan Periasamy. Bugs: RANGER-3681 https://issues.apache.org/jira/browse/RANGER-3681 Repository: ranger Description --- Modify delete sql to delete according to the primary key, to solve the problem of deadlock under REPEATABLE-READ isolation level Diffs - src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefAccessTypeDao.java b9a60cb src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefConditionDao.java e14bc14 src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefDataMaskTypeDao.java 7e7b8d4 src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefGroupDao.java 5f9d9ed src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefResourceDao.java 0ea7de9 src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefRoleDao.java 3ae7e7a src/security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefUserDao.java 518139a src/security-admin/src/main/resources/META-INF/jpa_named_queries.xml 5813209 Diff: https://reviews.apache.org/r/73913/diff/1/ Testing --- 1. Calles the createPolicy() method 1000 times using two threads. The results showed that all policies are successfully created without deadlock exception. 2. Calles the updatePolicy() method 1000 times using two threads. The results showed that all policies are successfully updated without deadlock exception. 3. Calles the deletePolicy() method 1000 times using two threads. The results showed that all policies are successfully deleted. Thanks, Xuze Yang
[jira] [Commented] (RANGER-3681) Ranger Database deadlock when createPolicy is running parallel
[
https://issues.apache.org/jira/browse/RANGER-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17511800#comment-17511800
]
Xuze Yang commented on RANGER-3681:
---
A little explanation on way 4:
When delete sql modify existing records on the unique index (primary key/unique
key), only row-level record locks (record key) are added, while when modifying
non-existing records on the unique index, gap locks are applied.
So in way 4, we first select 'id' according to the 'policy_id', make sure that
the record exists, and then delete it according to the primary key 'id'.
> Ranger Database deadlock when createPolicy is running parallel
> --
>
> Key: RANGER-3681
> URL: https://issues.apache.org/jira/browse/RANGER-3681
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Affects Versions: 2.1.0
> Environment: ranger-version: 2.1.0
> mysql-verison: 5.7.34-log
> mysql-engine: InnoDB
> mysql-isolation-level: REPEATABLE-READ
>Reporter: Xuze Yang
>Assignee: Xuze Yang
>Priority: Major
>
> h1. Error
> ranger-log：
> {noformat}
> 2022-03-21 20:14:29,685 [http-bio-6080-exec-13] ERROR
> org.apache.ranger.rest.ServiceREST (ServiceREST.java:1709) -
> createPolicy(RangerPolicy={id={null} guid={null} isEnabled={true}
> createdBy={null} updatedBy={null} createTime={null} updateTime={null}
> version={1} service={default-Hive}
> name={dcp-desensitize_a162c40cdc0140b1848b98415575be6c-1647864869626}
> policyType={0} policyPriority={0} description={}
> resourceSignature={4f15e3de95c81650ad869cb93a8c47a132bbec54bdf5de8c01f5075c19754cd7}
> isAuditEnabled={true} serviceType={null}
> resources={database={RangerPolicyResource={values={dcp } isExcludes={false}
> isRecursive={false} }} column={RangerPolicyResource={values={* }
> isExcludes={false} isRecursive={false} }}
> table={RangerPolicyResource={values={desensitize_a162c40cdc0140b1848b98415575be6c
> } isExcludes={false} isRecursive={false} }} } policyLabels={Consoler }
> policyConditions={}
> policyItems={RangerPolicyItem={accessTypes={RangerPolicyItemAccess={type={all}
> isAllowed={true} }} users={tangbiao2 } groups={} roles={} conditions={}
> delegateAdmin={false} }} denyPolicyItems={} allowExceptions={}
> denyExceptions={} dataMaskPolicyItems={} rowFilterPolicyItems={} options={}
> validitySchedules={, zoneName=null, isDenyAllElse={false} }}) failed
> javax.persistence.PersistenceException: Exception [EclipseLink-4002] (Eclipse
> Persistence Services - 2.5.2.v20140319-9ad6abd):
> org.eclipse.persistence.exceptions.DatabaseException
> Internal Exception:
> com.mysql.cj.jdbc.exceptions.MySQLTransactionRollbackException: Deadlock
> found when trying to get lock; try restarting transaction
> Error Code: 1213
> Call: INSERT INTO x_policy_ref_resource (ADDED_BY_ID, CREATE_TIME, policy_id,
> resource_def_id, resource_name, UPDATE_TIME, UPD_BY_ID) VALUES (?, ?, ?, ?,
> ?, ?, ?)
> bind => [7 parameters bound]
> Query: ValueReadQuery(name="x_policy_ref_resource_SEQ" sql="SELECT
> LAST_INSERT_ID()")
> at
> org.eclipse.persistence.internal.jpa.EntityManagerImpl.flush(EntityManagerImpl.java:868)
> at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.springframework.orm.jpa.SharedEntityManagerCreator$SharedEntityManagerInvocationHandler.invoke(SharedEntityManagerCreator.java:301)
> at com.sun.proxy.$Proxy30.flush(Unknown Source)
> at org.apache.ranger.common.db.BaseDao.batchCreate(BaseDao.java:102)
> ...{noformat}
> mysql-deadlock：
> {noformat}
>
> LATEST DETECTED DEADLOCK
>
> 2022-03-21 09:47:22 0x7ff3a4859700
> *** (1) TRANSACTION:
> TRANSACTION 7036760, ACTIVE 0 sec inserting
> mysql tables in use 1, locked 1
> LOCK WAIT 23 lock struct(s), heap size 1136, 12 row lock(s), undo log entries
> 2
> MySQL thread id 27293, OS thread handle 140684415063808, query id 383930
> 192.168.0.76 DHCloudBG update
> INSERT INTO x_policy_ref_resource (ADDED_BY_ID, CREATE_TIME, policy_id,
> resource_def_id, resource_name, UPDATE_TIME, UPD_BY_ID) VALUES (1,
> '2022-03-20 12:47:22.666', 13921, 5, 'database', '2022-03-20 12:47:22.681', 1)
> *** (1) WAITING FOR THIS LOCK TO BE GRANTED:
> RECORD LOCKS space id 531 page no 4 n bits 376 index
> x_policy_ref_res_UK_polId_resDefId of table `r
[jira] [Commented] (RANGER-3681) Ranger Database deadlock when createPolicy is running parallel
[
https://issues.apache.org/jira/browse/RANGER-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17511779#comment-17511779
]
Xuze Yang commented on RANGER-3681:
---
Currently, I think of four possible ways to avoid this problem:
# {-}Modify the isolation level of the database session to rc.{-}(may
introduce new problems and the scope of influence is too large)
# {-}Do not use one transaction to wrap the logic of delete and insert.{-}(not
feasible, this part needs to be made into a transaction to prevent data loss
due to insert failure)
# First determine whether the passing policy_id value exists in the
x_policy_ref_resource table. If it does not exist, skip the delete sql, insert
directly; if it exists, replace the delete and insert operations with one
update sql.
# Select the primary key of the record according to policy_id before deleting,
then delete it according to the primary key.
Both the third and fourth solutions are feasible, but the fourth solution is
less intrusive to the source code.
> Ranger Database deadlock when createPolicy is running parallel
> --
>
> Key: RANGER-3681
> URL: https://issues.apache.org/jira/browse/RANGER-3681
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Affects Versions: 2.1.0
> Environment: ranger-version: 2.1.0
> mysql-verison: 5.7.34-log
> mysql-engine: InnoDB
> mysql-isolation-level: REPEATABLE-READ
>Reporter: Xuze Yang
>Assignee: Xuze Yang
>Priority: Major
>
> h1. Error
> ranger-log：
> {noformat}
> 2022-03-21 20:14:29,685 [http-bio-6080-exec-13] ERROR
> org.apache.ranger.rest.ServiceREST (ServiceREST.java:1709) -
> createPolicy(RangerPolicy={id={null} guid={null} isEnabled={true}
> createdBy={null} updatedBy={null} createTime={null} updateTime={null}
> version={1} service={default-Hive}
> name={dcp-desensitize_a162c40cdc0140b1848b98415575be6c-1647864869626}
> policyType={0} policyPriority={0} description={}
> resourceSignature={4f15e3de95c81650ad869cb93a8c47a132bbec54bdf5de8c01f5075c19754cd7}
> isAuditEnabled={true} serviceType={null}
> resources={database={RangerPolicyResource={values={dcp } isExcludes={false}
> isRecursive={false} }} column={RangerPolicyResource={values={* }
> isExcludes={false} isRecursive={false} }}
> table={RangerPolicyResource={values={desensitize_a162c40cdc0140b1848b98415575be6c
> } isExcludes={false} isRecursive={false} }} } policyLabels={Consoler }
> policyConditions={}
> policyItems={RangerPolicyItem={accessTypes={RangerPolicyItemAccess={type={all}
> isAllowed={true} }} users={tangbiao2 } groups={} roles={} conditions={}
> delegateAdmin={false} }} denyPolicyItems={} allowExceptions={}
> denyExceptions={} dataMaskPolicyItems={} rowFilterPolicyItems={} options={}
> validitySchedules={, zoneName=null, isDenyAllElse={false} }}) failed
> javax.persistence.PersistenceException: Exception [EclipseLink-4002] (Eclipse
> Persistence Services - 2.5.2.v20140319-9ad6abd):
> org.eclipse.persistence.exceptions.DatabaseException
> Internal Exception:
> com.mysql.cj.jdbc.exceptions.MySQLTransactionRollbackException: Deadlock
> found when trying to get lock; try restarting transaction
> Error Code: 1213
> Call: INSERT INTO x_policy_ref_resource (ADDED_BY_ID, CREATE_TIME, policy_id,
> resource_def_id, resource_name, UPDATE_TIME, UPD_BY_ID) VALUES (?, ?, ?, ?,
> ?, ?, ?)
> bind => [7 parameters bound]
> Query: ValueReadQuery(name="x_policy_ref_resource_SEQ" sql="SELECT
> LAST_INSERT_ID()")
> at
> org.eclipse.persistence.internal.jpa.EntityManagerImpl.flush(EntityManagerImpl.java:868)
> at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.springframework.orm.jpa.SharedEntityManagerCreator$SharedEntityManagerInvocationHandler.invoke(SharedEntityManagerCreator.java:301)
> at com.sun.proxy.$Proxy30.flush(Unknown Source)
> at org.apache.ranger.common.db.BaseDao.batchCreate(BaseDao.java:102)
> ...{noformat}
> mysql-deadlock：
> {noformat}
>
> LATEST DETECTED DEADLOCK
>
> 2022-03-21 09:47:22 0x7ff3a4859700
> *** (1) TRANSACTION:
> TRANSACTION 7036760, ACTIVE 0 sec inserting
> mysql tables in use 1, locked 1
> LOCK WAIT 23 lock struct(s), heap size 1136, 12 row lock(s), undo log entries
> 2
> MySQL thread id 27293, OS thread handle 140684415063808, qu
[jira] [Updated] (RANGER-3681) Ranger Database deadlock when createPolicy is running parallel
[
https://issues.apache.org/jira/browse/RANGER-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuze Yang updated RANGER-3681:
--
Description:
h1. Error
ranger-log：
{noformat}
2022-03-21 20:14:29,685 [http-bio-6080-exec-13] ERROR
org.apache.ranger.rest.ServiceREST (ServiceREST.java:1709) -
createPolicy(RangerPolicy={id={null} guid={null} isEnabled={true}
createdBy={null} updatedBy={null} createTime={null} updateTime={null}
version={1} service={default-Hive}
name={dcp-desensitize_a162c40cdc0140b1848b98415575be6c-1647864869626}
policyType={0} policyPriority={0} description={}
resourceSignature={4f15e3de95c81650ad869cb93a8c47a132bbec54bdf5de8c01f5075c19754cd7}
isAuditEnabled={true} serviceType={null}
resources={database={RangerPolicyResource={values={dcp } isExcludes={false}
isRecursive={false} }} column={RangerPolicyResource={values={* }
isExcludes={false} isRecursive={false} }}
table={RangerPolicyResource={values={desensitize_a162c40cdc0140b1848b98415575be6c
} isExcludes={false} isRecursive={false} }} } policyLabels={Consoler }
policyConditions={}
policyItems={RangerPolicyItem={accessTypes={RangerPolicyItemAccess={type={all}
isAllowed={true} }} users={tangbiao2 } groups={} roles={} conditions={}
delegateAdmin={false} }} denyPolicyItems={} allowExceptions={}
denyExceptions={} dataMaskPolicyItems={} rowFilterPolicyItems={} options={}
validitySchedules={, zoneName=null, isDenyAllElse={false} }}) failed
javax.persistence.PersistenceException: Exception [EclipseLink-4002] (Eclipse
Persistence Services - 2.5.2.v20140319-9ad6abd):
org.eclipse.persistence.exceptions.DatabaseException
Internal Exception:
com.mysql.cj.jdbc.exceptions.MySQLTransactionRollbackException: Deadlock found
when trying to get lock; try restarting transaction
Error Code: 1213
Call: INSERT INTO x_policy_ref_resource (ADDED_BY_ID, CREATE_TIME, policy_id,
resource_def_id, resource_name, UPDATE_TIME, UPD_BY_ID) VALUES (?, ?, ?, ?, ?,
?, ?)
bind => [7 parameters bound]
Query: ValueReadQuery(name="x_policy_ref_resource_SEQ" sql="SELECT
LAST_INSERT_ID()")
at
org.eclipse.persistence.internal.jpa.EntityManagerImpl.flush(EntityManagerImpl.java:868)
at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.springframework.orm.jpa.SharedEntityManagerCreator$SharedEntityManagerInvocationHandler.invoke(SharedEntityManagerCreator.java:301)
at com.sun.proxy.$Proxy30.flush(Unknown Source)
at org.apache.ranger.common.db.BaseDao.batchCreate(BaseDao.java:102)
...{noformat}
mysql-deadlock：
{noformat}
LATEST DETECTED DEADLOCK
2022-03-21 09:47:22 0x7ff3a4859700
*** (1) TRANSACTION:
TRANSACTION 7036760, ACTIVE 0 sec inserting
mysql tables in use 1, locked 1
LOCK WAIT 23 lock struct(s), heap size 1136, 12 row lock(s), undo log entries 2
MySQL thread id 27293, OS thread handle 140684415063808, query id 383930
192.168.0.76 DHCloudBG update
INSERT INTO x_policy_ref_resource (ADDED_BY_ID, CREATE_TIME, policy_id,
resource_def_id, resource_name, UPDATE_TIME, UPD_BY_ID) VALUES (1, '2022-03-20
12:47:22.666', 13921, 5, 'database', '2022-03-20 12:47:22.681', 1)
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 531 page no 4 n bits 376 index
x_policy_ref_res_UK_polId_resDefId of table `ranger`.`x_policy_ref_resource`
trx id 7036760 lock_mode X insert intention waiting
Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits 0
0: len 8; hex 73757072656d756d; asc supremum;;
*** (2) TRANSACTION:
TRANSACTION 7036759, ACTIVE 0 sec inserting
mysql tables in use 1, locked 1
23 lock struct(s), heap size 1136, 12 row lock(s), undo log entries 2
MySQL thread id 27295, OS thread handle 140684413998848, query id 383932
192.168.0.76 DHCloudBG update
INSERT INTO x_policy_ref_resource (ADDED_BY_ID, CREATE_TIME, policy_id,
resource_def_id, resource_name, UPDATE_TIME, UPD_BY_ID) VALUES (1, '2022-03-20
12:47:22.666', 13920, 5, 'database', '2022-03-20 12:47:22.682', 1)
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 531 page no 4 n bits 376 index
x_policy_ref_res_UK_polId_resDefId of table `ranger`.`x_policy_ref_resource`
trx id 7036759 lock_mode X
Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits 0
0: len 8; hex 73757072656d756d; asc supremum;;
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 531 page no 4 n bits 376 index
x_policy_ref_res_UK_polId_resDefId of table `ranger`.`x_policy_ref_resource`
trx id 7036759 lock_mode X insert intention waiting
Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits 0
0: len 8; hex 73757072656d756d; asc supremum;;*** WE ROLL BACK TRANSA
[jira] [Comment Edited] (RANGER-3681) Ranger Database deadlock when createPolicy is running parallel
[
https://issues.apache.org/jira/browse/RANGER-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17511620#comment-17511620
]
Xuze Yang edited comment on RANGER-3681 at 3/24/22, 6:57 AM:
-
add some details：
# I wrote a test demo which calles the createPolicy() method 100 times using
two threads. The results showed that only half of the policies were
successfully created, and the other half failed to create due to deadlock.
# When I change the isolation level of mysql database session to rc
(READ-COMMITTED), there will be no deadlock exception.
# The structure of the x_policy_ref_resource table is:
{noformat}
CREATE TABLE `x_policy_ref_resource` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`guid` varchar(1024) DEFAULT NULL,
`create_time` datetime DEFAULT NULL,
`update_time` datetime DEFAULT NULL,
`added_by_id` bigint(20) DEFAULT NULL,
`upd_by_id` bigint(20) DEFAULT NULL,
`policy_id` bigint(20) NOT NULL,
`resource_def_id` bigint(20) NOT NULL,
`resource_name` varchar(4000) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `x_policy_ref_res_UK_polId_resDefId`
(`policy_id`,`resource_def_id`),
KEY `x_policy_ref_res_FK_added_by_id` (`added_by_id`),
KEY `x_policy_ref_res_FK_upd_by_id` (`upd_by_id`),
KEY `x_policy_ref_res_FK_resource_def_id` (`resource_def_id`),
CONSTRAINT `x_policy_ref_res_FK_added_by_id` FOREIGN KEY (`added_by_id`)
REFERENCES `x_portal_user` (`id`),
CONSTRAINT `x_policy_ref_res_FK_policy_id` FOREIGN KEY (`policy_id`)
REFERENCES `x_policy` (`id`),
CONSTRAINT `x_policy_ref_res_FK_resource_def_id` FOREIGN KEY
(`resource_def_id`) REFERENCES `x_resource_def` (`id`),
CONSTRAINT `x_policy_ref_res_FK_upd_by_id` FOREIGN KEY (`upd_by_id`)
REFERENCES `x_portal_user` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=7654 DEFAULT CHARSET=utf8
ROW_FORMAT=DYNAMIC{noformat}
was (Author: xuze yang):
add some details：
# I wrote a test demo which calles the createPolicy() method 100 times using
two threads. The results showed that only half of the policies were
successfully created, and the other half failed to create due to deadlock.
# When I change the isolation level of mysql database session to rc
(READ-COMMITTED), there will be no deadlock exception.
# The structure of the x_policy_ref_resource table is:
{noformat}
CREATE TABLE `x_policy_ref_resource` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`guid` varchar(1024) DEFAULT NULL,
`create_time` datetime DEFAULT NULL,
`update_time` datetime DEFAULT NULL,
`added_by_id` bigint(20) DEFAULT NULL,
`upd_by_id` bigint(20) DEFAULT NULL,
`policy_id` bigint(20) NOT NULL,
`resource_def_id` bigint(20) NOT NULL,
`resource_name` varchar(4000) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `x_policy_ref_res_UK_polId_resDefId`
(`policy_id`,`resource_def_id`),
KEY `x_policy_ref_res_FK_added_by_id` (`added_by_id`),
KEY `x_policy_ref_res_FK_upd_by_id` (`upd_by_id`),
KEY `x_policy_ref_res_FK_resource_def_id` (`resource_def_id`),
CONSTRAINT `x_policy_ref_res_FK_added_by_id` FOREIGN KEY (`added_by_id`)
REFERENCES `x_portal_user` (`id`),
CONSTRAINT `x_policy_ref_res_FK_policy_id` FOREIGN KEY (`policy_id`)
REFERENCES `x_policy` (`id`),
CONSTRAINT `x_policy_ref_res_FK_resource_def_id` FOREIGN KEY
(`resource_def_id`) REFERENCES `x_resource_def` (`id`),
CONSTRAINT `x_policy_ref_res_FK_upd_by_id` FOREIGN KEY (`upd_by_id`)
REFERENCES `x_portal_user` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=7654 DEFAULT CHARSET=utf8
ROW_FORMAT=DYNAMIC{noformat}
> Ranger Database deadlock when createPolicy is running parallel
> --
>
> Key: RANGER-3681
> URL: https://issues.apache.org/jira/browse/RANGER-3681
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Affects Versions: 2.1.0
> Environment: ranger-version: 2.1.0
> mysql-verison: 5.7.34-log
> mysql-engine: InnoDB
> mysql-isolation-level: REPEATABLE-READ
>Reporter: Xuze Yang
>Assignee: Xuze Yang
>Priority: Major
>
> h1. Error
> ranger-log：
> {noformat}
> 2022-03-21 20:14:29,685 [http-bio-6080-exec-13] ERROR
> org.apache.ranger.rest.ServiceREST (ServiceREST.java:1709) -
> createPolicy(RangerPolicy={id={null} guid={null} isEnabled={true}
> createdBy={null} updatedBy={null} createTime={null} updateTime={null}
> version={1} service={default-Hive}
> name={dcp-desensitize_a162c40cdc0140b1848b98415575be6c-1647864869626}
> policyType={0} policyPriority={0} description={}
> resourceSignature={4f15e3de95c81650ad869cb93a8c47a132bbec54bdf5de8c01f5075c19754cd7}
> isAuditEnabled={true} serviceType={null}
> resources={database={RangerPolicyResource={values={dcp } isExcludes={false}
> isRecursive={false} }} co
[jira] [Commented] (RANGER-3681) Ranger Database deadlock when createPolicy is running parallel
[
https://issues.apache.org/jira/browse/RANGER-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17511620#comment-17511620
]
Xuze Yang commented on RANGER-3681:
---
add some details：
# I wrote a test demo which calles the createPolicy() method 100 times using
two threads. The results showed that only half of the policies were
successfully created, and the other half failed to create due to deadlock.
# When I change the isolation level of mysql database session to rc
(READ-COMMITTED), there will be no deadlock exception.
# The structure of the x_policy_ref_resource table is:
{noformat}
CREATE TABLE `x_policy_ref_resource` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`guid` varchar(1024) DEFAULT NULL,
`create_time` datetime DEFAULT NULL,
`update_time` datetime DEFAULT NULL,
`added_by_id` bigint(20) DEFAULT NULL,
`upd_by_id` bigint(20) DEFAULT NULL,
`policy_id` bigint(20) NOT NULL,
`resource_def_id` bigint(20) NOT NULL,
`resource_name` varchar(4000) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `x_policy_ref_res_UK_polId_resDefId`
(`policy_id`,`resource_def_id`),
KEY `x_policy_ref_res_FK_added_by_id` (`added_by_id`),
KEY `x_policy_ref_res_FK_upd_by_id` (`upd_by_id`),
KEY `x_policy_ref_res_FK_resource_def_id` (`resource_def_id`),
CONSTRAINT `x_policy_ref_res_FK_added_by_id` FOREIGN KEY (`added_by_id`)
REFERENCES `x_portal_user` (`id`),
CONSTRAINT `x_policy_ref_res_FK_policy_id` FOREIGN KEY (`policy_id`)
REFERENCES `x_policy` (`id`),
CONSTRAINT `x_policy_ref_res_FK_resource_def_id` FOREIGN KEY
(`resource_def_id`) REFERENCES `x_resource_def` (`id`),
CONSTRAINT `x_policy_ref_res_FK_upd_by_id` FOREIGN KEY (`upd_by_id`)
REFERENCES `x_portal_user` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=7654 DEFAULT CHARSET=utf8
ROW_FORMAT=DYNAMIC{noformat}
> Ranger Database deadlock when createPolicy is running parallel
> --
>
> Key: RANGER-3681
> URL: https://issues.apache.org/jira/browse/RANGER-3681
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Affects Versions: 2.1.0
> Environment: ranger-version: 2.1.0
> mysql-verison: 5.7.34-log
> mysql-engine: InnoDB
> mysql-isolation-level: REPEATABLE-READ
>Reporter: Xuze Yang
>Assignee: Xuze Yang
>Priority: Major
>
> h1. Error
> ranger-log：
> {noformat}
> 2022-03-21 20:14:29,685 [http-bio-6080-exec-13] ERROR
> org.apache.ranger.rest.ServiceREST (ServiceREST.java:1709) -
> createPolicy(RangerPolicy={id={null} guid={null} isEnabled={true}
> createdBy={null} updatedBy={null} createTime={null} updateTime={null}
> version={1} service={default-Hive}
> name={dcp-desensitize_a162c40cdc0140b1848b98415575be6c-1647864869626}
> policyType={0} policyPriority={0} description={}
> resourceSignature={4f15e3de95c81650ad869cb93a8c47a132bbec54bdf5de8c01f5075c19754cd7}
> isAuditEnabled={true} serviceType={null}
> resources={database={RangerPolicyResource={values={dcp } isExcludes={false}
> isRecursive={false} }} column={RangerPolicyResource={values={* }
> isExcludes={false} isRecursive={false} }}
> table={RangerPolicyResource={values={desensitize_a162c40cdc0140b1848b98415575be6c
> } isExcludes={false} isRecursive={false} }} } policyLabels={Consoler }
> policyConditions={}
> policyItems={RangerPolicyItem={accessTypes={RangerPolicyItemAccess={type={all}
> isAllowed={true} }} users={tangbiao2 } groups={} roles={} conditions={}
> delegateAdmin={false} }} denyPolicyItems={} allowExceptions={}
> denyExceptions={} dataMaskPolicyItems={} rowFilterPolicyItems={} options={}
> validitySchedules={, zoneName=null, isDenyAllElse={false} }}) failed
> javax.persistence.PersistenceException: Exception [EclipseLink-4002] (Eclipse
> Persistence Services - 2.5.2.v20140319-9ad6abd):
> org.eclipse.persistence.exceptions.DatabaseException
> Internal Exception:
> com.mysql.cj.jdbc.exceptions.MySQLTransactionRollbackException: Deadlock
> found when trying to get lock; try restarting transaction
> Error Code: 1213
> Call: INSERT INTO x_policy_ref_resource (ADDED_BY_ID, CREATE_TIME, policy_id,
> resource_def_id, resource_name, UPDATE_TIME, UPD_BY_ID) VALUES (?, ?, ?, ?,
> ?, ?, ?)
> bind => [7 parameters bound]
> Query: ValueReadQuery(name="x_policy_ref_resource_SEQ" sql="SELECT
> LAST_INSERT_ID()")
> at
> org.eclipse.persistence.internal.jpa.EntityManagerImpl.flush(EntityManagerImpl.java:868)
> at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at
[jira] [Updated] (RANGER-3681) Ranger Database deadlock when createPolicy is running parallel
[
https://issues.apache.org/jira/browse/RANGER-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuze Yang updated RANGER-3681:
--
Description:
h1. Error
ranger-log：
{noformat}
2022-03-21 20:14:29,685 [http-bio-6080-exec-13] ERROR
org.apache.ranger.rest.ServiceREST (ServiceREST.java:1709) -
createPolicy(RangerPolicy={id={null} guid={null} isEnabled={true}
createdBy={null} updatedBy={null} createTime={null} updateTime={null}
version={1} service={default-Hive}
name={dcp-desensitize_a162c40cdc0140b1848b98415575be6c-1647864869626}
policyType={0} policyPriority={0} description={}
resourceSignature={4f15e3de95c81650ad869cb93a8c47a132bbec54bdf5de8c01f5075c19754cd7}
isAuditEnabled={true} serviceType={null}
resources={database={RangerPolicyResource={values={dcp } isExcludes={false}
isRecursive={false} }} column={RangerPolicyResource={values={* }
isExcludes={false} isRecursive={false} }}
table={RangerPolicyResource={values={desensitize_a162c40cdc0140b1848b98415575be6c
} isExcludes={false} isRecursive={false} }} } policyLabels={Consoler }
policyConditions={}
policyItems={RangerPolicyItem={accessTypes={RangerPolicyItemAccess={type={all}
isAllowed={true} }} users={tangbiao2 } groups={} roles={} conditions={}
delegateAdmin={false} }} denyPolicyItems={} allowExceptions={}
denyExceptions={} dataMaskPolicyItems={} rowFilterPolicyItems={} options={}
validitySchedules={, zoneName=null, isDenyAllElse={false} }}) failed
javax.persistence.PersistenceException: Exception [EclipseLink-4002] (Eclipse
Persistence Services - 2.5.2.v20140319-9ad6abd):
org.eclipse.persistence.exceptions.DatabaseException
Internal Exception:
com.mysql.cj.jdbc.exceptions.MySQLTransactionRollbackException: Deadlock found
when trying to get lock; try restarting transaction
Error Code: 1213
Call: INSERT INTO x_policy_ref_resource (ADDED_BY_ID, CREATE_TIME, policy_id,
resource_def_id, resource_name, UPDATE_TIME, UPD_BY_ID) VALUES (?, ?, ?, ?, ?,
?, ?)
bind => [7 parameters bound]
Query: ValueReadQuery(name="x_policy_ref_resource_SEQ" sql="SELECT
LAST_INSERT_ID()")
at
org.eclipse.persistence.internal.jpa.EntityManagerImpl.flush(EntityManagerImpl.java:868)
at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.springframework.orm.jpa.SharedEntityManagerCreator$SharedEntityManagerInvocationHandler.invoke(SharedEntityManagerCreator.java:301)
at com.sun.proxy.$Proxy30.flush(Unknown Source)
at org.apache.ranger.common.db.BaseDao.batchCreate(BaseDao.java:102)
...{noformat}
mysql-deadlock：
{noformat}
LATEST DETECTED DEADLOCK
2022-03-21 09:47:22 0x7ff3a4859700
*** (1) TRANSACTION:
TRANSACTION 7036760, ACTIVE 0 sec inserting
mysql tables in use 1, locked 1
LOCK WAIT 23 lock struct(s), heap size 1136, 12 row lock(s), undo log entries 2
MySQL thread id 27293, OS thread handle 140684415063808, query id 383930
192.168.0.76 DHCloudBG update
INSERT INTO x_policy_ref_resource (ADDED_BY_ID, CREATE_TIME, policy_id,
resource_def_id, resource_name, UPDATE_TIME, UPD_BY_ID) VALUES (1, '2022-03-20
12:47:22.666', 13921, 5, 'database', '2022-03-20 12:47:22.681', 1)
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 531 page no 4 n bits 376 index
x_policy_ref_res_UK_polId_resDefId of table `ranger`.`x_policy_ref_resource`
trx id 7036760 lock_mode X insert intention waiting
Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits 0
0: len 8; hex 73757072656d756d; asc supremum;;*** (2) TRANSACTION:
TRANSACTION 7036759, ACTIVE 0 sec inserting
mysql tables in use 1, locked 1
23 lock struct(s), heap size 1136, 12 row lock(s), undo log entries 2
MySQL thread id 27295, OS thread handle 140684413998848, query id 383932
192.168.0.76 DHCloudBG update
INSERT INTO x_policy_ref_resource (ADDED_BY_ID, CREATE_TIME, policy_id,
resource_def_id, resource_name, UPDATE_TIME, UPD_BY_ID) VALUES (1, '2022-03-20
12:47:22.666', 13920, 5, 'database', '2022-03-20 12:47:22.682', 1)
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 531 page no 4 n bits 376 index
x_policy_ref_res_UK_polId_resDefId of table `ranger`.`x_policy_ref_resource`
trx id 7036759 lock_mode X
Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits 0
0: len 8; hex 73757072656d756d; asc supremum;;*** (2) WAITING FOR THIS LOCK TO
BE GRANTED:
RECORD LOCKS space id 531 page no 4 n bits 376 index
x_policy_ref_res_UK_polId_resDefId of table `ranger`.`x_policy_ref_resource`
trx id 7036759 lock_mode X insert intention waiting
Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits 0
0: len 8; hex 73757072656d756d; asc supremum;;*** WE ROLL BACK TRANSACTION
(2){no
[jira] [Updated] (RANGER-3681) Ranger Database deadlock when createPolicy is running parallel
[
https://issues.apache.org/jira/browse/RANGER-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuze Yang updated RANGER-3681:
--
Description:
h1. Error
ranger-log：
{noformat}
2022-03-21 20:14:29,685 [http-bio-6080-exec-13] ERROR
org.apache.ranger.rest.ServiceREST (ServiceREST.java:1709) -
createPolicy(RangerPolicy={id={null} guid={null} isEnabled={true}
createdBy={null} updatedBy={null} createTime={null} updateTime={null}
version={1} service={default-Hive}
name={dcp-desensitize_a162c40cdc0140b1848b98415575be6c-1647864869626}
policyType={0} policyPriority={0} description={}
resourceSignature={4f15e3de95c81650ad869cb93a8c47a132bbec54bdf5de8c01f5075c19754cd7}
isAuditEnabled={true} serviceType={null}
resources={database={RangerPolicyResource={values={dcp } isExcludes={false}
isRecursive={false} }} column={RangerPolicyResource={values={* }
isExcludes={false} isRecursive={false} }}
table={RangerPolicyResource={values={desensitize_a162c40cdc0140b1848b98415575be6c
} isExcludes={false} isRecursive={false} }} } policyLabels={Consoler }
policyConditions={}
policyItems={RangerPolicyItem={accessTypes={RangerPolicyItemAccess={type={all}
isAllowed={true} }} users={tangbiao2 } groups={} roles={} conditions={}
delegateAdmin={false} }} denyPolicyItems={} allowExceptions={}
denyExceptions={} dataMaskPolicyItems={} rowFilterPolicyItems={} options={}
validitySchedules={, zoneName=null, isDenyAllElse={false} }}) failed
javax.persistence.PersistenceException: Exception [EclipseLink-4002] (Eclipse
Persistence Services - 2.5.2.v20140319-9ad6abd):
org.eclipse.persistence.exceptions.DatabaseException
Internal Exception:
com.mysql.cj.jdbc.exceptions.MySQLTransactionRollbackException: Deadlock found
when trying to get lock; try restarting transaction
Error Code: 1213
Call: INSERT INTO x_policy_ref_resource (ADDED_BY_ID, CREATE_TIME, policy_id,
resource_def_id, resource_name, UPDATE_TIME, UPD_BY_ID) VALUES (?, ?, ?, ?, ?,
?, ?)
bind => [7 parameters bound]
Query: ValueReadQuery(name="x_policy_ref_resource_SEQ" sql="SELECT
LAST_INSERT_ID()")
at
org.eclipse.persistence.internal.jpa.EntityManagerImpl.flush(EntityManagerImpl.java:868)
at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.springframework.orm.jpa.SharedEntityManagerCreator$SharedEntityManagerInvocationHandler.invoke(SharedEntityManagerCreator.java:301)
at com.sun.proxy.$Proxy30.flush(Unknown Source)
at org.apache.ranger.common.db.BaseDao.batchCreate(BaseDao.java:102)
...{noformat}
mysql-deadlock：
{noformat}
LATEST DETECTED DEADLOCK
2022-03-21 09:47:22 0x7ff3a4859700
*** (1) TRANSACTION:
TRANSACTION 7036760, ACTIVE 0 sec inserting
mysql tables in use 1, locked 1
LOCK WAIT 23 lock struct(s), heap size 1136, 12 row lock(s), undo log entries 2
MySQL thread id 27293, OS thread handle 140684415063808, query id 383930
192.168.0.76 DHCloudBG update
INSERT INTO x_policy_ref_resource (ADDED_BY_ID, CREATE_TIME, policy_id,
resource_def_id, resource_name, UPDATE_TIME, UPD_BY_ID) VALUES (1, '2022-03-20
12:47:22.666', 13921, 5, 'database', '2022-03-20 12:47:22.681', 1)
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 531 page no 4 n bits 376 index
x_policy_ref_res_UK_polId_resDefId of table `ranger`.`x_policy_ref_resource`
trx id 7036760 lock_mode X insert intention waiting
Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits 0
0: len 8; hex 73757072656d756d; asc supremum;;*** (2) TRANSACTION:
TRANSACTION 7036759, ACTIVE 0 sec inserting
mysql tables in use 1, locked 1
23 lock struct(s), heap size 1136, 12 row lock(s), undo log entries 2
MySQL thread id 27295, OS thread handle 140684413998848, query id 383932
192.168.0.76 DHCloudBG update
INSERT INTO x_policy_ref_resource (ADDED_BY_ID, CREATE_TIME, policy_id,
resource_def_id, resource_name, UPDATE_TIME, UPD_BY_ID) VALUES (1, '2022-03-20
12:47:22.666', 13920, 5, 'database', '2022-03-20 12:47:22.682', 1)
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 531 page no 4 n bits 376 index
x_policy_ref_res_UK_polId_resDefId of table `ranger`.`x_policy_ref_resource`
trx id 7036759 lock_mode X
Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits 0
0: len 8; hex 73757072656d756d; asc supremum;;*** (2) WAITING FOR THIS LOCK TO
BE GRANTED:
RECORD LOCKS space id 531 page no 4 n bits 376 index
x_policy_ref_res_UK_polId_resDefId of table `ranger`.`x_policy_ref_resource`
trx id 7036759 lock_mode X insert intention waiting
Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits 0
0: len 8; hex 73757072656d756d; asc supremum;;*** WE ROLL BACK TRANSACTION
(2){n
[jira] [Created] (RANGER-3681) Ranger Database deadlock when createPolicy is running parallel
Xuze Yang created RANGER-3681:
-
Summary: Ranger Database deadlock when createPolicy is running
parallel
Key: RANGER-3681
URL: https://issues.apache.org/jira/browse/RANGER-3681
Project: Ranger
Issue Type: Bug
Components: admin
Affects Versions: 2.1.0
Environment: ranger-version: 2.1.0
mysql-verison: 5.7.34-log
mysql-engine: InnoDB
mysql-isolation-level: REPEATABLE-READ
Reporter: Xuze Yang
Assignee: Xuze Yang
h1. Error
ranger-log：
{noformat}
2022-03-21 20:14:29,685 [http-bio-6080-exec-13] ERROR
org.apache.ranger.rest.ServiceREST (ServiceREST.java:1709) -
createPolicy(RangerPolicy={id={null} guid={null} isEnabled={true}
createdBy={null} updatedBy={null} createTime={null} updateTime={null}
version={1} service={default-Hive}
name={dcp-desensitize_a162c40cdc0140b1848b98415575be6c-1647864869626}
policyType={0} policyPriority={0} description={}
resourceSignature={4f15e3de95c81650ad869cb93a8c47a132bbec54bdf5de8c01f5075c19754cd7}
isAuditEnabled={true} serviceType={null}
resources={database={RangerPolicyResource={values={dcp } isExcludes={false}
isRecursive={false} }} column={RangerPolicyResource={values={* }
isExcludes={false} isRecursive={false} }}
table={RangerPolicyResource={values={desensitize_a162c40cdc0140b1848b98415575be6c
} isExcludes={false} isRecursive={false} }} } policyLabels={Consoler }
policyConditions={}
policyItems={RangerPolicyItem={accessTypes={RangerPolicyItemAccess={type={all}
isAllowed={true} }} users={tangbiao2 } groups={} roles={} conditions={}
delegateAdmin={false} }} denyPolicyItems={} allowExceptions={}
denyExceptions={} dataMaskPolicyItems={} rowFilterPolicyItems={} options={}
validitySchedules={, zoneName=null, isDenyAllElse={false} }}) failed
javax.persistence.PersistenceException: Exception [EclipseLink-4002] (Eclipse
Persistence Services - 2.5.2.v20140319-9ad6abd):
org.eclipse.persistence.exceptions.DatabaseException
Internal Exception:
com.mysql.cj.jdbc.exceptions.MySQLTransactionRollbackException: Deadlock found
when trying to get lock; try restarting transaction
Error Code: 1213
Call: INSERT INTO x_policy_ref_resource (ADDED_BY_ID, CREATE_TIME, policy_id,
resource_def_id, resource_name, UPDATE_TIME, UPD_BY_ID) VALUES (?, ?, ?, ?, ?,
?, ?)
bind => [7 parameters bound]
Query: ValueReadQuery(name="x_policy_ref_resource_SEQ" sql="SELECT
LAST_INSERT_ID()")
at
org.eclipse.persistence.internal.jpa.EntityManagerImpl.flush(EntityManagerImpl.java:868)
at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.springframework.orm.jpa.SharedEntityManagerCreator$SharedEntityManagerInvocationHandler.invoke(SharedEntityManagerCreator.java:301)
at com.sun.proxy.$Proxy30.flush(Unknown Source)
at org.apache.ranger.common.db.BaseDao.batchCreate(BaseDao.java:102)
...{noformat}
mysql-deadlock：
{noformat}
LATEST DETECTED DEADLOCK
2022-03-21 09:47:22 0x7ff3a4859700
*** (1) TRANSACTION:
TRANSACTION 7036760, ACTIVE 0 sec inserting
mysql tables in use 1, locked 1
LOCK WAIT 23 lock struct(s), heap size 1136, 12 row lock(s), undo log entries 2
MySQL thread id 27293, OS thread handle 140684415063808, query id 383930
192.168.0.76 DHCloudBG update
INSERT INTO x_policy_ref_resource (ADDED_BY_ID, CREATE_TIME, policy_id,
resource_def_id, resource_name, UPDATE_TIME, UPD_BY_ID) VALUES (1, '2022-03-20
12:47:22.666', 13921, 5, 'database', '2022-03-20 12:47:22.681', 1)
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 531 page no 4 n bits 376 index
x_policy_ref_res_UK_polId_resDefId of table `ranger`.`x_policy_ref_resource`
trx id 7036760 lock_mode X insert intention waiting
Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits 0
0: len 8; hex 73757072656d756d; asc supremum;;*** (2) TRANSACTION:
TRANSACTION 7036759, ACTIVE 0 sec inserting
mysql tables in use 1, locked 1
23 lock struct(s), heap size 1136, 12 row lock(s), undo log entries 2
MySQL thread id 27295, OS thread handle 140684413998848, query id 383932
192.168.0.76 DHCloudBG update
INSERT INTO x_policy_ref_resource (ADDED_BY_ID, CREATE_TIME, policy_id,
resource_def_id, resource_name, UPDATE_TIME, UPD_BY_ID) VALUES (1, '2022-03-20
12:47:22.666', 13920, 5, 'database', '2022-03-20 12:47:22.682', 1)
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 531 page no 4 n bits 376 index
x_policy_ref_res_UK_polId_resDefId of table `ranger`.`x_policy_ref_resource`
trx id 7036759 lock_mode X
Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits 0
0: len 8; hex 73757072656d756d; asc supremum;;*** (2) WAITING FOR THIS LOCK TO
B
[jira] [Commented] (RANGER-3500) Ranger policy list doesn't support sorting by field
[
https://issues.apache.org/jira/browse/RANGER-3500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440841#comment-17440841
]
Xuze Yang commented on RANGER-3500:
---
review request available at:
[https://reviews.apache.org/r/73694/]
> Ranger policy list doesn't support sorting by field
> ---
>
> Key: RANGER-3500
> URL: https://issues.apache.org/jira/browse/RANGER-3500
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Affects Versions: 2.1.0
> Reporter: Xuze Yang
>Assignee: Xuze Yang
>Priority: Major
> Attachments:
> 0001-RANGER-3500-Ranger-policy-list-doesn-t-support-sorti.patch
>
>
> When getting the ranger policy list, we may want to sort the returned policy
> list according to certain fields, such as policyId, policyName and etc. But
> case shows that adding the parameters sortBy and sortType to the url has no
> effect (eg:
> [http://192.168.0.12:6080/service/plugins/]policies/service/2?sortBy=policyName=desc=default-Hdfs).
> I look through the source code and find that code supports sorting by
> fields, but due to some code bugs, it did not really take effect.
> The main reason for the problem is that when the SearchFilter is copied
> deeply, only the params is copied, the sortBy and sortType attributes is
> omitted. The code show as follows:
> {code:java}
> Map paramsCopy = new HashMap<>(filter.getParams());
> SearchFilter searchFilter = new SearchFilter(paramsCopy);
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (RANGER-3500) Ranger policy list doesn't support sorting by field
[
https://issues.apache.org/jira/browse/RANGER-3500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuze Yang updated RANGER-3500:
--
Attachment: 0001-RANGER-3500-Ranger-policy-list-doesn-t-support-sorti.patch
> Ranger policy list doesn't support sorting by field
> ---
>
> Key: RANGER-3500
> URL: https://issues.apache.org/jira/browse/RANGER-3500
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Affects Versions: 2.1.0
> Reporter: Xuze Yang
>Assignee: Xuze Yang
>Priority: Major
> Attachments:
> 0001-RANGER-3500-Ranger-policy-list-doesn-t-support-sorti.patch
>
>
> When getting the ranger policy list, we may want to sort the returned policy
> list according to certain fields, such as policyId, policyName and etc. But
> case shows that adding the parameters sortBy and sortType to the url has no
> effect (eg:
> [http://192.168.0.12:6080/service/plugins/]policies/service/2?sortBy=policyName=desc=default-Hdfs).
> I look through the source code and find that code supports sorting by
> fields, but due to some code bugs, it did not really take effect.
> The main reason for the problem is that when the SearchFilter is copied
> deeply, only the params is copied, the sortBy and sortType attributes is
> omitted. The code show as follows:
> {code:java}
> Map paramsCopy = new HashMap<>(filter.getParams());
> SearchFilter searchFilter = new SearchFilter(paramsCopy);
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
Review Request 73694: RANGER-3500: Ranger policy list doesn't support sorting by field
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73694/
---
Review request for ranger.
Bugs: RANGER-3500
https://issues.apache.org/jira/browse/RANGER-3500
Repository: ranger
Description
---
modify the getServicePolicies() method to support sorting by field and
sortType(i.e. asc or desc). The changes are as following:
1. Add sortBy and sortType to searchFilter
2. Support sorting by sortType("desc" means in decreasing order, otherwise in
increasing order)
3. Delete a redundant line of code
Diffs
-
agents-common/src/main/java/org/apache/ranger/plugin/store/AbstractPredicateUtil.java
38d6b03ea
security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java
f13cef71d
Diff: https://reviews.apache.org/r/73694/diff/1/
Testing
---
1. compile and package the code
2. deploy ranger
3. generate hdfs policies and get the hdfs policyList via postman using url
like
"http://192.168.0.12:6080/service/plugins/policies/service/2?sortBy=policyName=desc=default-Hdfs;
4. the return policyList is sorted according our specified sortBy and sortType
Thanks,
Xuze Yang
[jira] [Commented] (RANGER-3500) Ranger policy list doesn't support sorting by field
[
https://issues.apache.org/jira/browse/RANGER-3500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17434853#comment-17434853
]
Xuze Yang commented on RANGER-3500:
---
In addition, in the method of sorting the policy list, only the sortBy field is
used, and the sortType field is not used. so all sorts are 'asc'. We need to
modify the sorting method to support 'desc' too. So The code can be modified as
follows:
1. When deep copying SearchFilter, add a copy of sortBy and sortType fields
2. Modify the sorting method to support 'desc'
> Ranger policy list doesn't support sorting by field
> ---
>
> Key: RANGER-3500
> URL: https://issues.apache.org/jira/browse/RANGER-3500
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Affects Versions: 2.1.0
> Reporter: Xuze Yang
>Priority: Major
>
> When getting the ranger policy list, we may want to sort the returned policy
> list according to certain fields, such as policyId, policyName and etc. But
> case shows that adding the parameters sortBy and sortType to the url has no
> effect (eg:
> [http://192.168.0.12:6080/service/plugins/]policies/service/2?sortBy=policyName=desc=default-Hdfs).
> I look through the source code and find that code supports sorting by
> fields, but due to some code bugs, it did not really take effect.
> The main reason for the problem is that when the SearchFilter is copied
> deeply, only the params is copied, the sortBy and sortType attributes is
> omitted. The code show as follows:
> {code:java}
> Map paramsCopy = new HashMap<>(filter.getParams());
> SearchFilter searchFilter = new SearchFilter(paramsCopy);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (RANGER-3500) Ranger policy list doesn't support sorting by field
[
https://issues.apache.org/jira/browse/RANGER-3500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuze Yang updated RANGER-3500:
--
Description:
When getting the ranger policy list, we may want to sort the returned policy
list according to certain fields, such as policyId, policyName and etc. But
case shows that adding the parameters sortBy and sortType to the url has no
effect (eg:
[http://192.168.0.12:6080/service/plugins/]policies/service/2?sortBy=policyName=desc=default-Hdfs).
I look through the source code and find that code supports sorting by fields,
but due to some code bugs, it did not really take effect.
The main reason for the problem is that when the SearchFilter is copied
deeply, only the params is copied, the sortBy and sortType attributes is
omitted. The code show as follows:
{code:java}
Map paramsCopy = new HashMap<>(filter.getParams());
SearchFilter searchFilter = new SearchFilter(paramsCopy);
{code}
was:
When getting the ranger policy list, we may want to sort the returned policy
list according to certain fields, such as policyId, policyName and etc. But
case shows that adding the parameters sortBy and sortType to the url has no
effect (eg: http://192.168.0.12:6080/service/plugins/
policies/service/2?sortBy=policyName=desc=default-Hdfs). I
look through the source code and find that code supports sorting by fields, but
due to some code bugs, it did not really take effect.
The main reason for the problem is that when the SearchFilter is copied deeply,
only the params is copied, the sortBy and sortType attributes is omitted. The
code show as follows:
{code:java}
Map paramsCopy = new HashMap<>(filter.getParams());
SearchFilter searchFilter = new SearchFilter(paramsCopy);
{code}
> Ranger policy list doesn't support sorting by field
> ---
>
> Key: RANGER-3500
> URL: https://issues.apache.org/jira/browse/RANGER-3500
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Affects Versions: 2.1.0
>Reporter: Xuze Yang
>Priority: Major
>
> When getting the ranger policy list, we may want to sort the returned policy
> list according to certain fields, such as policyId, policyName and etc. But
> case shows that adding the parameters sortBy and sortType to the url has no
> effect (eg:
> [http://192.168.0.12:6080/service/plugins/]policies/service/2?sortBy=policyName=desc=default-Hdfs).
> I look through the source code and find that code supports sorting by
> fields, but due to some code bugs, it did not really take effect.
> The main reason for the problem is that when the SearchFilter is copied
> deeply, only the params is copied, the sortBy and sortType attributes is
> omitted. The code show as follows:
> {code:java}
> Map paramsCopy = new HashMap<>(filter.getParams());
> SearchFilter searchFilter = new SearchFilter(paramsCopy);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (RANGER-3500) Ranger policy list doesn't support sorting by field
Xuze Yang created RANGER-3500:
-
Summary: Ranger policy list doesn't support sorting by field
Key: RANGER-3500
URL: https://issues.apache.org/jira/browse/RANGER-3500
Project: Ranger
Issue Type: Bug
Components: admin
Affects Versions: 2.1.0
Reporter: Xuze Yang
When getting the ranger policy list, we may want to sort the returned policy
list according to certain fields, such as policyId, policyName and etc. But
case shows that adding the parameters sortBy and sortType to the url has no
effect (eg: http://192.168.0.12:6080/service/plugins/
policies/service/2?sortBy=policyName=desc=default-Hdfs). I
look through the source code and find that code supports sorting by fields, but
due to some code bugs, it did not really take effect.
The main reason for the problem is that when the SearchFilter is copied deeply,
only the params is copied, the sortBy and sortType attributes is omitted. The
code show as follows:
{code:java}
Map paramsCopy = new HashMap<>(filter.getParams());
SearchFilter searchFilter = new SearchFilter(paramsCopy);
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (RANGER-3472) The createPolicy() method is not thread safe. In another word, we can create policies with same resources when creating policies concurrently
[ https://issues.apache.org/jira/browse/RANGER-3472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17429110#comment-17429110 ] Xuze Yang commented on RANGER-3472: --- [~madhan] In my understanding, high-availability means failover. Just like in hdfs, under normal circumstances, Active NameNode provides services to the outside world. When Active NameNode crash down, the Standby NameNode will transition to Active NameNode and take over everything. i.e. At the same time, only one instance will provide services. So we needn't consider the multiple instances scenario. I don’t know whether the ha of ranger is consistent with the ha of hdfs, because I haven’t found a design document about the ranger ha. Can you give more details，or can you push the design docs to me. thanks! > The createPolicy() method is not thread safe. In another word, we can create > policies with same resources when creating policies concurrently > - > > Key: RANGER-3472 > URL: https://issues.apache.org/jira/browse/RANGER-3472 > Project: Ranger > Issue Type: Bug > Components: Ranger >Affects Versions: 2.1.0 >Reporter: Xuze Yang >Priority: Major > > In our production environment, we happen to find that two policies exist with > the same resources.In this case, when we want to modify either policy, ranger > doesn't allow this operation and throws message like "*Error Code : 3010 > Another policy already exists for matching resource: policy-name=[hhh9], > service=[default-Hdfs]*". > I go through the source code about create policy, find that the > createPolicy() in class ServiceREST is not thread safe. When we create > policies concurrently, we may create several policies with the same resources. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (RANGER-3472) The createPolicy() method is not thread safe. In another word, we can create policies with same resources when creating policies concurrently
[
https://issues.apache.org/jira/browse/RANGER-3472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17429083#comment-17429083
]
Xuze Yang edited comment on RANGER-3472 at 10/15/21, 2:12 AM:
--
[~madhan], thanks for your suggestion. All we've been talking about is applying
unique-key constraint on columns. I am now wondering whether it is feasible to
add lock on the code. I think one of the roughest possible ways is as follows:
{code:java}
synchronized(this){
RangerPolicyValidator validator =
validatorFactory.getPolicyValidator(svcStore);
validator.validate(policy, Action.CREATE, bizUtil.isAdmin());
ensureAdminAccess(policy);
bizUtil.blockAuditorRoleUser();
ret = svcStore.createPolicy(policy);
}
{code}
The above code is a fragment of the createPolicy() in ServiceREST class. Of
course, this lock is too coarse-grained, we need to make the lock fine-grained.
For example, we should allow concurrently creating policies in differrent
services, etc. Do you think add locking on createPolicy is a feasible way?
Looking forward to your reply.
was (Author: xuze yang):
[~madhan], thanks for your suggestion. All we've been talking about is applying
unique-key constraint on columns. I am now wondering whether it is feasible to
add lock on the code. I think one of the roughest possible ways is as follows:
{code:java}
synchronized(this){
RangerPolicyValidator validator =
validatorFactory.getPolicyValidator(svcStore);
validator.validate(policy, Action.CREATE, bizUtil.isAdmin());
ensureAdminAccess(policy);
bizUtil.blockAuditorRoleUser();
ret = svcStore.createPolicy(policy);
}
{code}
The above code is a fragment of the createPolicy() in ServiceREST class. Of
course, this lock is too coarse-grained, we need to make the lock fine-grained.
For example, we should allow concurrently creating policies in differrent
services. Do you think add locking on createPolicy is a feasible way? Looking
forward to your reply.
> The createPolicy() method is not thread safe. In another word, we can create
> policies with same resources when creating policies concurrently
> -
>
> Key: RANGER-3472
> URL: https://issues.apache.org/jira/browse/RANGER-3472
> Project: Ranger
> Issue Type: Bug
> Components: Ranger
>Affects Versions: 2.1.0
>Reporter: Xuze Yang
>Priority: Major
>
> In our production environment, we happen to find that two policies exist with
> the same resources.In this case, when we want to modify either policy, ranger
> doesn't allow this operation and throws message like "*Error Code : 3010
> Another policy already exists for matching resource: policy-name=[hhh9],
> service=[default-Hdfs]*".
> I go through the source code about create policy, find that the
> createPolicy() in class ServiceREST is not thread safe. When we create
> policies concurrently, we may create several policies with the same resources.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (RANGER-3472) The createPolicy() method is not thread safe. In another word, we can create policies with same resources when creating policies concurrently
[
https://issues.apache.org/jira/browse/RANGER-3472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17429083#comment-17429083
]
Xuze Yang commented on RANGER-3472:
---
[~madhan], thanks for your suggestion. All we've been talking about is applying
unique-key constraint on columns. I am now wondering whether it is feasible to
add lock on the code. I think one of the roughest possible ways is as follows:
{code:java}
synchronized(this){
RangerPolicyValidator validator =
validatorFactory.getPolicyValidator(svcStore);
validator.validate(policy, Action.CREATE, bizUtil.isAdmin());
ensureAdminAccess(policy);
bizUtil.blockAuditorRoleUser();
ret = svcStore.createPolicy(policy);
}
{code}
The above code is a fragment of the createPolicy() in ServiceREST class. Of
course, this lock is too coarse-grained, we need to make the lock fine-grained.
For example, we should allow concurrently creating policies in differrent
services. Do you think add locking on createPolicy is a feasible way? Looking
forward to your reply.
> The createPolicy() method is not thread safe. In another word, we can create
> policies with same resources when creating policies concurrently
> -
>
> Key: RANGER-3472
> URL: https://issues.apache.org/jira/browse/RANGER-3472
> Project: Ranger
> Issue Type: Bug
> Components: Ranger
>Affects Versions: 2.1.0
>Reporter: Xuze Yang
>Priority: Major
>
> In our production environment, we happen to find that two policies exist with
> the same resources.In this case, when we want to modify either policy, ranger
> doesn't allow this operation and throws message like "*Error Code : 3010
> Another policy already exists for matching resource: policy-name=[hhh9],
> service=[default-Hdfs]*".
> I go through the source code about create policy, find that the
> createPolicy() in class ServiceREST is not thread safe. When we create
> policies concurrently, we may create several policies with the same resources.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (RANGER-3472) The createPolicy() method is not thread safe. In another word, we can create policies with same resources when creating policies concurrently
[
https://issues.apache.org/jira/browse/RANGER-3472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17428719#comment-17428719
]
Xuze Yang commented on RANGER-3472:
---
[~madhan] , thank you for your reply. Compared to adding locks in create policy
methods, I also think that apply unique-key constraint on column is a better
implement way. You mentioned a solution above: apply unique-key constraint on
column x_policy.resource_signature. As you mentioned above, this approach will
have problems in the following scenarios:
1. Create the same resource policy with status of disabled.
2. Create the same resource policy under different services in the same
warehouse.（eg：Two services named default-HDFS1 and default-HDFS2 can exist in
HDFS warehouse. It's reasonable to create a policy in default-HDFS1 even if the
same policy already been created in default-HDFS2 ）
3. Create the same resource policy under different zones.
I think a better way is to apply unique-key constraint on four columns -
'resource_signature', 'service', 'zone_id' and 'is_enabled', just like below
statement:
{code:java}
UNIQUE KEY `x_policy_UK_resource_service_zone_enabled`
(`resource_signature`,`service`, `zone_id`, `is_enabled`){code}
If we apply this unique-key constraint, above problems will be resolved(one
problem still exist : We can only create a disabled state policy at most). Do
you think this is a reasonable solution?
> The createPolicy() method is not thread safe. In another word, we can create
> policies with same resources when creating policies concurrently
> -
>
> Key: RANGER-3472
> URL: https://issues.apache.org/jira/browse/RANGER-3472
> Project: Ranger
> Issue Type: Bug
> Components: Ranger
>Affects Versions: 2.1.0
>Reporter: Xuze Yang
>Priority: Major
>
> In our production environment, we happen to find that two policies exist with
> the same resources.In this case, when we want to modify either policy, ranger
> doesn't allow this operation and throws message like "*Error Code : 3010
> Another policy already exists for matching resource: policy-name=[hhh9],
> service=[default-Hdfs]*".
> I go through the source code about create policy, find that the
> createPolicy() in class ServiceREST is not thread safe. When we create
> policies concurrently, we may create several policies with the same resources.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (RANGER-3472) The createPolicy() method is not thread safe. In another word, we can create policies with same resources when creating policies concurrently
[ https://issues.apache.org/jira/browse/RANGER-3472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17428615#comment-17428615 ] Xuze Yang commented on RANGER-3472: --- cc: [~bpatel] [~brahmareddy]. Do you have any suggestions? Is there any special consideration for not having concurrency protection in the code that creates the policy? > The createPolicy() method is not thread safe. In another word, we can create > policies with same resources when creating policies concurrently > - > > Key: RANGER-3472 > URL: https://issues.apache.org/jira/browse/RANGER-3472 > Project: Ranger > Issue Type: Bug > Components: Ranger >Affects Versions: 2.1.0 >Reporter: Xuze Yang >Priority: Major > > In our production environment, we happen to find that two policies exist with > the same resources.In this case, when we want to modify either policy, ranger > doesn't allow this operation and throws message like "*Error Code : 3010 > Another policy already exists for matching resource: policy-name=[hhh9], > service=[default-Hdfs]*". > I go through the source code about create policy, find that the > createPolicy() in class ServiceREST is not thread safe. When we create > policies concurrently, we may create several policies with the same resources. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (RANGER-3472) The createPolicy() method is not thread safe. In another word, we can create policies with same resources when creating policies concurrently
[ https://issues.apache.org/jira/browse/RANGER-3472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuze Yang updated RANGER-3472: -- Description: In our production environment, we happen to find that two policies exist with the same resources.In this case, when we want to modify either policy, ranger doesn't allow this operation and throws message like "*Error Code : 3010 Another policy already exists for matching resource: policy-name=[hhh9], service=[default-Hdfs]*". I go through the source code about create policy, find that the createPolicy() in class ServiceREST is not thread safe. When we create policies concurrently, we may create several policies with the same resources. was: In our production environment, we happen to find that two policies exist with the same resources.In this case, when we want to modify either policy, ranger doesn't allow this operation and throws message like "*Error Code : 3010 Another policy already exists for matching resource: policy-name=[hhh9], service=[default-Hdfs]*". I go through the source code about create policy, find that the createPolicy() in class ServiceREST is not thread safe. When we create policies concurrently, we may create several same resource's policies. > The createPolicy() method is not thread safe. In another word, we can create > policies with same resources when creating policies concurrently > - > > Key: RANGER-3472 > URL: https://issues.apache.org/jira/browse/RANGER-3472 > Project: Ranger > Issue Type: Bug > Components: Ranger > Affects Versions: 2.1.0 >Reporter: Xuze Yang >Priority: Major > > In our production environment, we happen to find that two policies exist with > the same resources.In this case, when we want to modify either policy, ranger > doesn't allow this operation and throws message like "*Error Code : 3010 > Another policy already exists for matching resource: policy-name=[hhh9], > service=[default-Hdfs]*". > I go through the source code about create policy, find that the > createPolicy() in class ServiceREST is not thread safe. When we create > policies concurrently, we may create several policies with the same resources. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (RANGER-3472) The createPolicy() method is not thread safe. In another word, we can create policies with same resources when creating policies concurrently
[ https://issues.apache.org/jira/browse/RANGER-3472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuze Yang updated RANGER-3472: -- Summary: The createPolicy() method is not thread safe. In another word, we can create policies with same resources when creating policies concurrently (was: The createPolicy() method is not thread safe. In another word, we can create same resources policies when creating policies concurrently) > The createPolicy() method is not thread safe. In another word, we can create > policies with same resources when creating policies concurrently > - > > Key: RANGER-3472 > URL: https://issues.apache.org/jira/browse/RANGER-3472 > Project: Ranger > Issue Type: Bug > Components: Ranger >Affects Versions: 2.1.0 >Reporter: Xuze Yang >Priority: Major > > In our production environment, we happen to find that two policies exist with > the same resources.In this case, when we want to modify either policy, ranger > doesn't allow this operation and throws message like "*Error Code : 3010 > Another policy already exists for matching resource: policy-name=[hhh9], > service=[default-Hdfs]*". > I go through the source code about create policy, find that the > createPolicy() in class ServiceREST is not thread safe. When we create > policies concurrently, we may create several same resource's policies. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (RANGER-3472) The createPolicy() method is not thread safe. In another word, we can create same resources policies when creating policies concurrently
Xuze Yang created RANGER-3472: - Summary: The createPolicy() method is not thread safe. In another word, we can create same resources policies when creating policies concurrently Key: RANGER-3472 URL: https://issues.apache.org/jira/browse/RANGER-3472 Project: Ranger Issue Type: Bug Components: Ranger Affects Versions: 2.1.0 Reporter: Xuze Yang In our production environment, we happen to find that two policies exist with the same resources.In this case, when we want to modify either policy, ranger doesn't allow this operation and throws message like "*Error Code : 3010 Another policy already exists for matching resource: policy-name=[hhh9], service=[default-Hdfs]*". I go through the source code about create policy, find that the createPolicy() in class ServiceREST is not thread safe. When we create policies concurrently, we may create several same resource's policies. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (RANGER-3326) get db audit log failed via rest api
Xuze Yang created RANGER-3326:
-
Summary: get db audit log failed via rest api
Key: RANGER-3326
URL: https://issues.apache.org/jira/browse/RANGER-3326
Project: Ranger
Issue Type: Bug
Components: audit
Affects Versions: 2.1.0
Reporter: Xuze Yang
Using ranger2.1.0, we store audit logs in db(although log in db is deprecated
since ranger0.6, we still use it because we don't want to introduce and
maintain a new component like solr or elasticSearch). The audit log information
can be successfully stored in db. But when we want to get audit logs via REST
api(/service/assets/accessAudit), it throws following exceptions:
{code:java}
Jul 01, 2021 5:09:59 PM com.sun.jersey.spi.container.ContainerResponse
mapMappableContainerExceptionJul 01, 2021 5:09:59 PM
com.sun.jersey.spi.container.ContainerResponse
mapMappableContainerExceptionSEVERE: The RuntimeException could not be mapped
to a response, re-throwing to the HTTP containerjava.lang.ClassCastException:
java.util.ArrayList cannot be cast to java.lang.String at
org.apache.ranger.common.SearchUtil.buildWhereClause(SearchUtil.java:518) at
org.apache.ranger.common.SearchUtil.buildWhereClause(SearchUtil.java:384) at
org.apache.ranger.common.SearchUtil.createSearchQuery(SearchUtil.java:708) at
org.apache.ranger.service.AbstractBaseResourceService.createQuery(AbstractBaseResourceService.java:519)
at
org.apache.ranger.service.AbstractBaseResourceService.getCountForSearchQuery(AbstractBaseResourceService.java:535)
at
org.apache.ranger.service.AbstractBaseResourceService.searchResources(AbstractBaseResourceService.java:564)
at
org.apache.ranger.service.XAccessAuditService.searchXAccessAudits(XAccessAuditService.java:172)
at org.apache.ranger.biz.AssetMgr.getAccessLogs(AssetMgr.java:1132) at
org.apache.ranger.rest.AssetREST.getAccessLogs(AssetREST.java:671) at
org.apache.ranger.rest.AssetREST$$FastClassBySpringCGLIB$$8cffcb6d.invoke()
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:737)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at
org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:69)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at
org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
at
org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:283)
at
org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at
org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:672)
at
org.apache.ranger.rest.AssetREST$$EnhancerBySpringCGLIB$$9c5e18ed.getAccessLogs()
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
at
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
at
com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
at
com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409
