[jira] [Updated] (RANGER-4985) GDS policy evaluation results to include all datasets and projects that allow the access
[ https://issues.apache.org/jira/browse/RANGER-4985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Madhan Neethiraj updated RANGER-4985: - Attachment: RANGER-4985.patch > GDS policy evaluation results to include all datasets and projects that allow > the access > > > Key: RANGER-4985 > URL: https://issues.apache.org/jira/browse/RANGER-4985 > Project: Ranger > Issue Type: Sub-task > Components: plugins >Reporter: Madhan Neethiraj >Assignee: Madhan Neethiraj >Priority: Major > Fix For: 3.0.0 > > Attachments: RANGER-4985.patch > > > Accessed resource can be part of multiple datasets and projects, and an > access could be allowed by more than one dataset or policy. Currently, the > result from policy evaluation includes all the datasets and projects the > resource is part of. This should be enhanced to include the datasets and > projects that allow the access. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (RANGER-4985) GDS policy evaluation results to include all datasets and projects that allow the access
Madhan Neethiraj created RANGER-4985: Summary: GDS policy evaluation results to include all datasets and projects that allow the access Key: RANGER-4985 URL: https://issues.apache.org/jira/browse/RANGER-4985 Project: Ranger Issue Type: Sub-task Components: plugins Reporter: Madhan Neethiraj Assignee: Madhan Neethiraj Accessed resource can be part of multiple datasets and projects, and an access could be allowed by more than one dataset or policy. Currently, the result from policy evaluation includes all the datasets and projects the resource is part of. This should be enhanced to include the datasets and projects that allow the access. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (RANGER-4972) Ranger User Type "federated user" should not log into Ranger for doing any operation
[ https://issues.apache.org/jira/browse/RANGER-4972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17895689#comment-17895689 ] Madhan Neethiraj commented on RANGER-4972: -- commit in ranger-2.6 branch: https://github.com/apache/ranger/commit/c4508e502ad3d0aac818983cd40e98fd6b2be990 > Ranger User Type "federated user" should not log into Ranger for doing any > operation > > > Key: RANGER-4972 > URL: https://issues.apache.org/jira/browse/RANGER-4972 > Project: Ranger > Issue Type: Task > Components: Ranger >Reporter: Dineshkumar Yadav >Assignee: Dineshkumar Yadav >Priority: Major > Fix For: 3.0.0, 2.6.0 > > > Ranger User Type "federated user" should not log into Ranger for doing any > operation. These users are external users for data-sharing, should be used > for metrics around what resources are shared, access history and audits for > data sharing features. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-4973) Enhance Ranger UI to support a new user type for external users from Data Sharing
[ https://issues.apache.org/jira/browse/RANGER-4973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Madhan Neethiraj updated RANGER-4973: - Fix Version/s: 2.6.0 > Enhance Ranger UI to support a new user type for external users from Data > Sharing > - > > Key: RANGER-4973 > URL: https://issues.apache.org/jira/browse/RANGER-4973 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Reporter: Dhaval Rajpara >Assignee: Dhaval Rajpara >Priority: Major > Fix For: 3.0.0, 2.6.0 > > Attachments: 0002-RANGER-4973.patch > > > Ranger user created by the external user registration process in Data Catalog > should have a new type "Federated User". Currently we have Internal / > External user. The existing "External User" are for users that are synced by > external sources like AD / LDAP etc and "Internal User" are for the users > created via Ranger Admin UI. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (RANGER-4973) Enhance Ranger UI to support a new user type for external users from Data Sharing
[ https://issues.apache.org/jira/browse/RANGER-4973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17895690#comment-17895690 ] Madhan Neethiraj commented on RANGER-4973: -- commit in ranger-2.6 branch: https://github.com/apache/ranger/commit/5f5887bf897ba6d21157289b5faf13a419bb5a25 > Enhance Ranger UI to support a new user type for external users from Data > Sharing > - > > Key: RANGER-4973 > URL: https://issues.apache.org/jira/browse/RANGER-4973 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Reporter: Dhaval Rajpara >Assignee: Dhaval Rajpara >Priority: Major > Fix For: 3.0.0, 2.6.0 > > Attachments: 0002-RANGER-4973.patch > > > Ranger user created by the external user registration process in Data Catalog > should have a new type "Federated User". Currently we have Internal / > External user. The existing "External User" are for users that are synced by > external sources like AD / LDAP etc and "Internal User" are for the users > created via Ranger Admin UI. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-4972) Ranger User Type "federated user" should not log into Ranger for doing any operation
[ https://issues.apache.org/jira/browse/RANGER-4972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Madhan Neethiraj updated RANGER-4972: - Fix Version/s: 2.6.0 > Ranger User Type "federated user" should not log into Ranger for doing any > operation > > > Key: RANGER-4972 > URL: https://issues.apache.org/jira/browse/RANGER-4972 > Project: Ranger > Issue Type: Task > Components: Ranger >Reporter: Dineshkumar Yadav >Assignee: Dineshkumar Yadav >Priority: Major > Fix For: 3.0.0, 2.6.0 > > > Ranger User Type "federated user" should not log into Ranger for doing any > operation. These users are external users for data-sharing, should be used > for metrics around what resources are shared, access history and audits for > data sharing features. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (RANGER-4975) DB patch fix to support for new columns in Ranger Dataset for MySql and Postrgres
[
https://issues.apache.org/jira/browse/RANGER-4975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Madhan Neethiraj resolved RANGER-4975.
--
Fix Version/s: 3.0.0
Resolution: Fixed
{noformat}
commit c97592b7173e2d6625a50a9a936999a0a3fb3dd5 (HEAD -> master, origin/master,
origin/HEAD)
Author: Radhika Kundam
Date: Tue Oct 29 13:04:57 2024 -0700
RANGER-4975: DB patch fix to support for new columns in Ranger Dataset for
MySql and Postrgres
Signed-off-by: Madhan Neethiraj
{noformat}
> DB patch fix to support for new columns in Ranger Dataset for MySql and
> Postrgres
> -
>
> Key: RANGER-4975
> URL: https://issues.apache.org/jira/browse/RANGER-4975
> Project: Ranger
> Issue Type: Bug
> Components: Ranger
>Reporter: Radhika Kundam
>Assignee: Radhika Kundam
>Priority: Major
> Fix For: 3.0.0
>
>
> Fix for RANGER-4933 introduced changes in DB patches to add new columns
> ({{{}validity_schedule{}}}, {{{}labels{}}}, {{{}keywords{}}}) to the DataSet
> table. These new columns need to be included in the core_db SQL script to
> ensure they are reflected during Ranger schema creation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (RANGER-4971) UI: condition to support validity schedule
[
https://issues.apache.org/jira/browse/RANGER-4971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Madhan Neethiraj updated RANGER-4971:
-
Description:
RANGER-4970 added support for validity schedule as a policy condition, which
can be used in allowing access to principals for specific time duration. Ranger
policy UI should be updated to support managing validity schedules in policy
conditions, similar to the UI supported for validity schedule at policy level -
as shown below. This should be supported for all conditions having
{{uiHint.valueType}} set to {{{}validitySchedule{}}}, as shown below:
{code:java}
"policyConditions": [
{
"itemId": 1,
"name": "validitySchedule",
"evaluator":
"org.apache.ranger.plugin.conditionevaluator.RangerValidityScheduleConditionEvaluator",
"uiHint": "{ \"valueType\": \"validitySchedule\" }"
"label": "Validity schedule",
"description": "Validity schedule"
}
]{code}
[~mugdha], [~brijesh]
!Policy_Validity_Schedule.png!
was:
RANGER-4970 added support for validity schedule as a policy condition, which
can be used in allowing access to principals for specific time duration. Ranger
policy UI should be updated to support managing validity schedules in policy
conditions, similar to the UI supported for validity schedule at policy level -
as shown below. This should be supported for all conditions having
{{uiHint.valueType}} set to {{validitySchedule}}.
[~mugdha], [~brijesh]
!Policy_Validity_Schedule.png!
> UI: condition to support validity schedule
> --
>
> Key: RANGER-4971
> URL: https://issues.apache.org/jira/browse/RANGER-4971
> Project: Ranger
> Issue Type: Improvement
> Components: admin
>Reporter: Madhan Neethiraj
>Assignee: Dhaval Rajpara
>Priority: Major
> Attachments: Policy_Validity_Schedule.png
>
>
> RANGER-4970 added support for validity schedule as a policy condition, which
> can be used in allowing access to principals for specific time duration.
> Ranger policy UI should be updated to support managing validity schedules in
> policy conditions, similar to the UI supported for validity schedule at
> policy level - as shown below. This should be supported for all conditions
> having {{uiHint.valueType}} set to {{{}validitySchedule{}}}, as shown below:
> {code:java}
> "policyConditions": [
> {
> "itemId": 1,
> "name": "validitySchedule",
> "evaluator":
> "org.apache.ranger.plugin.conditionevaluator.RangerValidityScheduleConditionEvaluator",
> "uiHint": "{ \"valueType\": \"validitySchedule\" }"
> "label": "Validity schedule",
> "description": "Validity schedule"
> }
> ]{code}
>
> [~mugdha], [~brijesh]
> !Policy_Validity_Schedule.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (RANGER-4971) UI: condition to support validity schedule
[
https://issues.apache.org/jira/browse/RANGER-4971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Madhan Neethiraj updated RANGER-4971:
-
Description:
RANGER-4970 added support for validity schedule as a policy condition, which
can be used in allowing access to principals for specific time duration. Ranger
policy UI should be updated to support managing validity schedules in policy
conditions, similar to the UI supported for validity schedule at policy level -
as shown below. This should be supported for all conditions having
{{uiHint.valueType}} set to {{validitySchedule}}.
[~mugdha], [~brijesh]
!Policy_Validity_Schedule.png!
was:
RANGER-4970 added support for validity schedule as a policy condition, which
can be used in allowing access to principals for specific time duration. Ranger
policy UI should be updated to support managing validity schedules in policy
conditions, similar to the UI supported for validity schedule at policy level -
as shown below. This should be supported for all conditions having
{{evaluator}} field set to
{{{}org.apache.ranger.plugin.conditionevaluator.RangerValidityScheduleConditionEvaluator{}}}.
[~mugdha], [~brijesh]
!Policy_Validity_Schedule.png!
> UI: condition to support validity schedule
> --
>
> Key: RANGER-4971
> URL: https://issues.apache.org/jira/browse/RANGER-4971
> Project: Ranger
> Issue Type: Improvement
> Components: admin
>Reporter: Madhan Neethiraj
>Assignee: Dhaval Rajpara
>Priority: Major
> Attachments: Policy_Validity_Schedule.png
>
>
> RANGER-4970 added support for validity schedule as a policy condition, which
> can be used in allowing access to principals for specific time duration.
> Ranger policy UI should be updated to support managing validity schedules in
> policy conditions, similar to the UI supported for validity schedule at
> policy level - as shown below. This should be supported for all conditions
> having {{uiHint.valueType}} set to {{validitySchedule}}.
>
> [~mugdha], [~brijesh]
> !Policy_Validity_Schedule.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (RANGER-4976) GDS policies should support validity-schedule for policy items
[ https://issues.apache.org/jira/browse/RANGER-4976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Madhan Neethiraj updated RANGER-4976: - Attachment: RANGER-4976.patch > GDS policies should support validity-schedule for policy items > -- > > Key: RANGER-4976 > URL: https://issues.apache.org/jira/browse/RANGER-4976 > Project: Ranger > Issue Type: Sub-task > Components: plugins >Reporter: Madhan Neethiraj >Assignee: Madhan Neethiraj >Priority: Major > Attachments: RANGER-4976.patch > > > One of the common use cases for data sharing is to grant access on a dataset > to principals (users/groups/roles) for a specific duration. GDS policies > should support such grants by including a policy-condition to specify > validity-schedules - see RANGER-4970. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (RANGER-4976) GDS policies should support validity-schedule for policy items
Madhan Neethiraj created RANGER-4976: Summary: GDS policies should support validity-schedule for policy items Key: RANGER-4976 URL: https://issues.apache.org/jira/browse/RANGER-4976 Project: Ranger Issue Type: Sub-task Components: plugins Reporter: Madhan Neethiraj Assignee: Madhan Neethiraj One of the common use cases for data sharing is to grant access on a dataset to principals (users/groups/roles) for a specific duration. GDS policies should support such grants by including a policy-condition to specify validity-schedules - see RANGER-4970. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-4971) UI: condition to support validity schedule
[
https://issues.apache.org/jira/browse/RANGER-4971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Madhan Neethiraj updated RANGER-4971:
-
Description:
RANGER-4970 added support for validity schedule as a policy condition, which
can be used in allowing access to principals for specific time duration. Ranger
policy UI should be updated to support managing validity schedules in policy
conditions, similar to the UI supported for validity schedule at policy level -
as shown below. This should be supported for all conditions having
{{evaluator}} field set to
{{{}org.apache.ranger.plugin.conditionevaluator.RangerValidityScheduleConditionEvaluator{}}}.
[~mugdha], [~brijesh]
!Policy_Validity_Schedule.png!
was:
RANGER-4970 added support for validity schedule as a policy condition, which
can be used in allowing access to principals for specific time duration. Ranger
policy UI should be updated to support managing validity schedules in policy
conditions, similar to the UI supported for validity schedule at policy level -
as shown below. This should be supported for all conditions having
{{evaluator}} field set to
{{{}org.apache.ranger.plugin.conditionevaluator.RangerValidityScheduleConditionEvaluator{}}}.
[~mugdha], [~brijesh]
!image-2024-10-28-20-56-20-729.png|width=911,height=397!
> UI: condition to support validity schedule
> --
>
> Key: RANGER-4971
> URL: https://issues.apache.org/jira/browse/RANGER-4971
> Project: Ranger
> Issue Type: Improvement
> Components: admin
>Reporter: Madhan Neethiraj
>Priority: Major
> Attachments: Policy_Validity_Schedule.png
>
>
> RANGER-4970 added support for validity schedule as a policy condition, which
> can be used in allowing access to principals for specific time duration.
> Ranger policy UI should be updated to support managing validity schedules in
> policy conditions, similar to the UI supported for validity schedule at
> policy level - as shown below. This should be supported for all conditions
> having {{evaluator}} field set to
> {{{}org.apache.ranger.plugin.conditionevaluator.RangerValidityScheduleConditionEvaluator{}}}.
>
> [~mugdha], [~brijesh]
> !Policy_Validity_Schedule.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (RANGER-4971) UI: condition to support validity schedule
[
https://issues.apache.org/jira/browse/RANGER-4971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Madhan Neethiraj updated RANGER-4971:
-
Attachment: Policy_Validity_Schedule.png
> UI: condition to support validity schedule
> --
>
> Key: RANGER-4971
> URL: https://issues.apache.org/jira/browse/RANGER-4971
> Project: Ranger
> Issue Type: Improvement
> Components: admin
>Reporter: Madhan Neethiraj
>Priority: Major
> Attachments: Policy_Validity_Schedule.png
>
>
> RANGER-4970 added support for validity schedule as a policy condition, which
> can be used in allowing access to principals for specific time duration.
> Ranger policy UI should be updated to support managing validity schedules in
> policy conditions, similar to the UI supported for validity schedule at
> policy level - as shown below. This should be supported for all conditions
> having {{evaluator}} field set to
> {{{}org.apache.ranger.plugin.conditionevaluator.RangerValidityScheduleConditionEvaluator{}}}.
>
> [~mugdha], [~brijesh]
> !image-2024-10-28-20-56-20-729.png|width=911,height=397!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (RANGER-4971) UI: condition to support validity schedule
Madhan Neethiraj created RANGER-4971:
Summary: UI: condition to support validity schedule
Key: RANGER-4971
URL: https://issues.apache.org/jira/browse/RANGER-4971
Project: Ranger
Issue Type: Improvement
Components: admin
Reporter: Madhan Neethiraj
RANGER-4970 added support for validity schedule as a policy condition, which
can be used in allowing access to principals for specific time duration. Ranger
policy UI should be updated to support managing validity schedules in policy
conditions, similar to the UI supported for validity schedule at policy level -
as shown below. This should be supported for all conditions having
{{evaluator}} field set to
{{{}org.apache.ranger.plugin.conditionevaluator.RangerValidityScheduleConditionEvaluator{}}}.
[~mugdha], [~brijesh]
!image-2024-10-28-20-56-20-729.png|width=911,height=397!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (RANGER-4970) Condition to support validity schedule
[ https://issues.apache.org/jira/browse/RANGER-4970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Madhan Neethiraj updated RANGER-4970: - Attachment: RANGER-4970.patch > Condition to support validity schedule > -- > > Key: RANGER-4970 > URL: https://issues.apache.org/jira/browse/RANGER-4970 > Project: Ranger > Issue Type: Improvement > Components: admin, plugins >Reporter: Madhan Neethiraj >Assignee: Madhan Neethiraj >Priority: Major > Fix For: 3.0.0, 2.6.0 > > Attachments: RANGER-4970.patch > > > Ranger policy model currently supports validity schedule at the policy level, > which allows a policy author to allow/deny permissions for specific time > periods. This should be extended at policy-item level, so that permissions > call be allowed/denied for individual principals for a specific time periods. > RANGER-3815 asked for this enhancement, which was addressed with introduction > of macros. However, supporting validity schedule, similar to the one at > policy level, will make it consistent. > > Custom conditions supported by Ranger policy model can be extended to provide > this functionality, by adding a condition that takes validity schedule as the > input. Policy UI can be enhanced to provide the same experience for policy > items as at the policy level. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (RANGER-4970) Condition to support validity schedule
Madhan Neethiraj created RANGER-4970: Summary: Condition to support validity schedule Key: RANGER-4970 URL: https://issues.apache.org/jira/browse/RANGER-4970 Project: Ranger Issue Type: Improvement Components: admin, plugins Reporter: Madhan Neethiraj Assignee: Madhan Neethiraj Ranger policy model currently supports validity schedule at the policy level, which allows a policy author to allow/deny permissions for specific time periods. This should be extended at policy-item level, so that permissions call be allowed/denied for individual principals for a specific time periods. RANGER-3815 asked for this enhancement, which was addressed with introduction of macros. However, supporting validity schedule, similar to the one at policy level, will make it consistent. Custom conditions supported by Ranger policy model can be extended to provide this functionality, by adding a condition that takes validity schedule as the input. Policy UI can be enhanced to provide the same experience for policy items as at the policy level. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-4961) Policies retrieved for a resource does not include tag policies
[
https://issues.apache.org/jira/browse/RANGER-4961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Madhan Neethiraj updated RANGER-4961:
-
Attachment: RANGER-4961.patch
> Policies retrieved for a resource does not include tag policies
> ---
>
> Key: RANGER-4961
> URL: https://issues.apache.org/jira/browse/RANGER-4961
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Affects Versions: 2.3.0, 2.4.0, 2.5.0
>Reporter: Madhan Neethiraj
>Assignee: Madhan Neethiraj
>Priority: Major
> Fix For: 3.0.0, 2.6.0
>
> Attachments: RANGER-4961.patch
>
>
> REST API at {{/service/public/v2/api/policies//for-resource}}
> should return all applicable policies for the given resource, including
> policies for the tags associated with the resource. However, the API returns
> only resource-based policies but not tag-based policies. This is a regression
> caused by updates in RANGER-3768.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (RANGER-4961) Policies retrieved for a resource does not include tag policies
Madhan Neethiraj created RANGER-4961:
Summary: Policies retrieved for a resource does not include tag
policies
Key: RANGER-4961
URL: https://issues.apache.org/jira/browse/RANGER-4961
Project: Ranger
Issue Type: Bug
Components: admin
Affects Versions: 2.5.0, 2.4.0, 2.3.0
Reporter: Madhan Neethiraj
Assignee: Madhan Neethiraj
REST API at {{/service/public/v2/api/policies//for-resource}}
should return all applicable policies for the given resource, including
policies for the tags associated with the resource. However, the API returns
only resource-based policies but not tag-based policies. This is a regression
caused by updates in RANGER-3768.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (RANGER-4956) RangerBasePlugin is not getting initialised when tag dedup feature is enabled
[
https://issues.apache.org/jira/browse/RANGER-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Madhan Neethiraj updated RANGER-4956:
-
Attachment: RANGER-4956.patch
> RangerBasePlugin is not getting initialised when tag dedup feature is enabled
> -
>
> Key: RANGER-4956
> URL: https://issues.apache.org/jira/browse/RANGER-4956
> Project: Ranger
> Issue Type: Bug
> Components: plugins
>Affects Versions: 2.5.0
>Reporter: Anand Nadar
>Assignee: Madhan Neethiraj
>Priority: Major
> Fix For: 3.0.0, 2.6.0
>
> Attachments: RANGER-4956.patch
>
>
> When ranger.admin.supports.tags.dedup is set to true, i.e tag dedup feature
> is enabled, we have found that the ranger plugin is not getting initialised
> using the below rangerbaseplugin constructor.
> public RangerBasePlugin(RangerPluginConfig pluginConfig, ServicePolicies
> policies, ServiceTags tags, RangerRoles roles, RangerUserStore userStore,
> ServiceGdsInfo gdsInfo)
> Here we have tags data as below which is provided in the above constructor
> {code:java}
> {
> "op": "add_or_update",
> "serviceName": "hive",
> "tagVersion": 7,
> "tagDefinitions": {
> "2": {
> "id": 2,
> "isEnabled": true,
> "name": "TAG2",
> "source": "Internal"
> }
> },
> "tags": {
> "10": {
> "id": 10,
> "isEnabled": true,
> "type": "TAG2"
> }
> },
> "serviceResources": [
> {
> "id": 8,
> "isEnabled": true,
> "resourceElements": {
> "database": {
> "values": [
> "db1"
> ],
> "isExcludes": false,
> "isRecursive": false
> }
> },
> "resourceSignature":
> "7ea12bf936f94ba73833373104ca818c249dfea758ed3366b675e86f42f01983"
> }
> ],
> "resourceToTagIds": {
> "8": [
> 10
> ]
> },
> "isDelta": false,
> "tagsChangeExtent": "ALL",
> "isTagsDeduped": true,
> "id": 0
> }{code}
> and therefore we set the tags in the tag enricher using the below
> tagEnricher.setServiceTags(tags);
> Now in the below method
> protected void setServiceTags(final ServiceTags serviceTags, final boolean
> rebuildOnlyIndex)
> we have this piece of code
> {code:java}
> if (!serviceTags.getIsDelta()) {
>if (serviceTags.getIsTagsDeduped()) {
>final int countOfDuplicateTags =
> serviceTags.dedupTags();
> LOG.info("Number of duplicate tags removed from the received serviceTags:[" +
> countOfDuplicateTags + "]. Number of tags in the de-duplicated serviceTags
> :[" + serviceTags.getTags().size() + "].");
> } processServiceTags(serviceTags);
> }{code}
> Here if serviceTags.getIsTagsDeduped() is true, we are deduping the tags
> again.
> And when this is being triggered, an infinite for loop is triggered
> inserviceTags serviceTags.dedupTags() method
> Below is the code which goes in infinite loop
> {code:java}
> for (Long replacerTagId = replacedIds.get(tagId); replacerTagId != null;
> replacerTagId = replacedIds.get(tagId)) {
> tagId = replacerTagId;
> } {code}
> {10, 1} was already available in cachedTags.
> Here replacedIds has \{10=10}
> tagId is 10
> and for this as data, the the for loop goes into infinite.
> {*}Note{*}: This behaviour is seen in the below case:
> 1. We initialise the plugin with the above available tag data and
> Rangerbaseplugin constructor. This works fine
> 2. Now we are trying to initialise a new Rangerbaseplugin with the same tags
> and this is when the plugin is not initialised and the for loop goes into an
> infinite state.
> Suggestion:
> {code:java}
> if (!serviceTags.getIsDelta()) {
> if (serviceTags.getIsTagsDeduped()) { final int countOfDuplicateTags =
> serviceTags.dedupTags(); LOG.info("Number of duplicate tags removed from the
> received serviceTags:[" + countOfDuplicateTags + "]. Number of tags in the
> de-duplicated serviceTags :[" + serviceTags.getTags().size() + "]."); }
> processServiceTags(serviceTags); } {code}
> Should the above piece of code in RangerTagEnricher be modified to
> {code:java}
> if (!serviceTags.getIsDelta()) {
> processServiceTags(serviceTags);
> } {code}
> Since tags are already deduped in ranger admin, I feel there is no need to
> dedup it in the plugin end.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (RANGER-4956) RangerBasePlugin is not getting initialised when tag dedup feature is enabled
[
https://issues.apache.org/jira/browse/RANGER-4956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Madhan Neethiraj reassigned RANGER-4956:
Assignee: Madhan Neethiraj
> RangerBasePlugin is not getting initialised when tag dedup feature is enabled
> -
>
> Key: RANGER-4956
> URL: https://issues.apache.org/jira/browse/RANGER-4956
> Project: Ranger
> Issue Type: Bug
> Components: plugins
>Reporter: Anand Nadar
>Assignee: Madhan Neethiraj
>Priority: Major
>
> When ranger.admin.supports.tags.dedup is set to true, i.e tag dedup feature
> is enabled, we have found that the ranger plugin is not getting initialised
> using the below rangerbaseplugin constructor.
> public RangerBasePlugin(RangerPluginConfig pluginConfig, ServicePolicies
> policies, ServiceTags tags, RangerRoles roles, RangerUserStore userStore,
> ServiceGdsInfo gdsInfo)
> Here we have tags data as below which is provided in the above constructor
> {code:java}
> {
> "op": "add_or_update",
> "serviceName": "hive",
> "tagVersion": 7,
> "tagDefinitions": {
> "2": {
> "id": 2,
> "isEnabled": true,
> "name": "TAG2",
> "source": "Internal"
> }
> },
> "tags": {
> "10": {
> "id": 10,
> "isEnabled": true,
> "type": "TAG2"
> }
> },
> "serviceResources": [
> {
> "id": 8,
> "isEnabled": true,
> "resourceElements": {
> "database": {
> "values": [
> "db1"
> ],
> "isExcludes": false,
> "isRecursive": false
> }
> },
> "resourceSignature":
> "7ea12bf936f94ba73833373104ca818c249dfea758ed3366b675e86f42f01983"
> }
> ],
> "resourceToTagIds": {
> "8": [
> 10
> ]
> },
> "isDelta": false,
> "tagsChangeExtent": "ALL",
> "isTagsDeduped": true,
> "id": 0
> }{code}
> and therefore we set the tags in the tag enricher using the below
> tagEnricher.setServiceTags(tags);
> Now in the below method
> protected void setServiceTags(final ServiceTags serviceTags, final boolean
> rebuildOnlyIndex)
> we have this piece of code
> {code:java}
> if (!serviceTags.getIsDelta()) {
>if (serviceTags.getIsTagsDeduped()) {
>final int countOfDuplicateTags =
> serviceTags.dedupTags();
> LOG.info("Number of duplicate tags removed from the received serviceTags:[" +
> countOfDuplicateTags + "]. Number of tags in the de-duplicated serviceTags
> :[" + serviceTags.getTags().size() + "].");
> } processServiceTags(serviceTags);
> }{code}
> Here if serviceTags.getIsTagsDeduped() is true, we are deduping the tags
> again.
> And when this is being triggered, an infinite for loop is triggered
> inserviceTags serviceTags.dedupTags() method
> Below is the code which goes in infinite loop
> {code:java}
> for (Long replacerTagId = replacedIds.get(tagId); replacerTagId != null;
> replacerTagId = replacedIds.get(tagId)) {
> tagId = replacerTagId;
> } {code}
> {10, 1} was already available in cachedTags.
> Here replacedIds has \{10=10}
> tagId is 10
> and for this as data, the the for loop goes into infinite.
> {*}Note{*}: This behaviour is seen in the below case:
> 1. We initialise the plugin with the above available tag data and
> Rangerbaseplugin constructor. This works fine
> 2. Now we are trying to initialise a new Rangerbaseplugin with the same tags
> and this is when the plugin is not initialised and the for loop goes into an
> infinite state.
> Suggestion:
> {code:java}
> if (!serviceTags.getIsDelta()) {
> if (serviceTags.getIsTagsDeduped()) { final int countOfDuplicateTags =
> serviceTags.dedupTags(); LOG.info("Number of duplicate tags removed from the
> received serviceTags:[" + countOfDuplicateTags + "]. Number of tags in the
> de-duplicated serviceTags :[" + serviceTags.getTags().size() + "]."); }
> processServiceTags(serviceTags); } {code}
> Should the above piece of code in RangerTagEnricher be modified to
> {code:java}
> if (!serviceTags.getIsDelta()) {
> processServiceTags(serviceTags);
> } {code}
> Since tags are already deduped in ranger admin, I feel there is no need to
> dedup it in the plugin end.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (RANGER-4959) [Ranger-Admin] Remove use of x_tag table
[
https://issues.apache.org/jira/browse/RANGER-4959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17889339#comment-17889339
]
Madhan Neethiraj commented on RANGER-4959:
--
{quote}For existing tag migration if there are multiple tags of the same type
associated with a single resource, then we would need to combine all their
attributes and store it in tags_text in the x_service_resource.
{quote}
[~anandNadar] - {{x_service_resource.tags_text}} can continue to have multiple
instances of a tag-type, just as it does currently. Is there a need to combine
attributes from all instances? That might result in attribute value in a tag
instance to be lost (i.e. overwritten by the value in another instance) - this
should be avoided.
About my earlier suggestion on repurposing {{x_tag_resource_map.tag_id}}
column, that will not work in case of rolling upgrades when 2 different
versions of Ranger admin servers can update the contents of this table. I
suggest going with a new table, {{{}x_service_resource_ref_tag{}}}, similar to
other {{_ref}} tables used in Ranger schema.
> [Ranger-Admin] Remove use of x_tag table
>
>
> Key: RANGER-4959
> URL: https://issues.apache.org/jira/browse/RANGER-4959
> Project: Ranger
> Issue Type: Improvement
> Components: admin
>Reporter: Anand Nadar
>Assignee: Anand Nadar
>Priority: Major
>
> Below is a metrics retrieved for importservicetags api to create tags and
> tag-resource association.
> |Duration|10 min|
> |Successful request |49|
> |Number of tags for each resource |6 |
> |Number of columns |100 |
> |Total resources tag mapping |6*100 = 600|
> |Total tag resource map in overall |49*600 = 29400 records in db |
> |Rate per min |2940|
> * Number of tables = 200k
> * Number of columns = 500
> * Avg tag on each column =6
> * Total resources tag mapping = 20 * 500*6 = 600,000,000
> * Time as per rate with 10 threads = 600,000,000/2940 = 204081 min =
> 141.722917 days
> Above is the performance of the importservicetags api with 2GB heap memory.
>
> Therefore to improve this performance, we are trying to do the below changes.
> # Remove usage of x_tag table
> # In the x_tag_resource_map table, the association should be between the tag
> def id and the resource id.
> # The x_tag_resource_map will have a new column "tagAttrs" to store the tag
> attributes.
> # tags_text which is stored in x_service_resource table can have the below
> data to reduce its size.
> {code:java}
> [{"id":1069576,"isEnabled":true,"name":"TAG1","attributes":{"restricted1":"true"}}]
> {code}
> id, isEnabled, name - These data will be from x_tag_def
> attributes - This will be retrieved from the x_tag_resource_map table for
> that particular resource.
> The tag owner case is not being handled here.
> ImportserviceTags flow
> - create tagDef if not exists
> - Create service resource if not exists
> - Create tag Def and resource association with tag attributes
> - Refresh the tags_text in x_service_resource (This can be handled in a
> separate thread)
> The download json structure will be maintained to minimise plugin side
> changes.
> The importservicetags json input will remain the same.
> tag delta and tag dedup will be affected, it needs to be handled accordingly
> cc: [~madhan] [~avadhavkar]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (RANGER-4959) [Ranger-Admin] Remove use of x_tag table
[
https://issues.apache.org/jira/browse/RANGER-4959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17889317#comment-17889317
]
Madhan Neethiraj commented on RANGER-4959:
--
{quote}There could be a case were there are multiple tags of the same type
associated with a resource. Need to think what needs to be done for this.
{quote}
{{x_tag_resource_map}} table will become similar to {{_ref}} tables used in
Ranger, like {{x_policy_ref_user}}. For example: a policy could reference a
user in multiple policy items; there will be one entry for all references in
{{x_policy_ref_user}} table.
> [Ranger-Admin] Remove use of x_tag table
>
>
> Key: RANGER-4959
> URL: https://issues.apache.org/jira/browse/RANGER-4959
> Project: Ranger
> Issue Type: Improvement
> Components: admin
>Reporter: Anand Nadar
>Assignee: Anand Nadar
>Priority: Major
>
> Below is a metrics retrieved for importservicetags api to create tags and
> tag-resource association.
> |Duration|10 min|
> |Successful request |49|
> |Number of tags for each resource |6 |
> |Number of columns |100 |
> |Total resources tag mapping |6*100 = 600|
> |Total tag resource map in overall |49*600 = 29400 records in db |
> |Rate per min |2940|
> * Number of tables = 200k
> * Number of columns = 500
> * Avg tag on each column =6
> * Total resources tag mapping = 20 * 500*6 = 600,000,000
> * Time as per rate with 10 threads = 600,000,000/2940 = 204081 min =
> 141.722917 days
> Above is the performance of the importservicetags api with 2GB heap memory.
>
> Therefore to improve this performance, we are trying to do the below changes.
> # Remove usage of x_tag table
> # In the x_tag_resource_map table, the association should be between the tag
> def id and the resource id.
> # The x_tag_resource_map will have a new column "tagAttrs" to store the tag
> attributes.
> # tags_text which is stored in x_service_resource table can have the below
> data to reduce its size.
> {code:java}
> [{"id":1069576,"isEnabled":true,"name":"TAG1","attributes":{"restricted1":"true"}}]
> {code}
> id, isEnabled, name - These data will be from x_tag_def
> attributes - This will be retrieved from the x_tag_resource_map table for
> that particular resource.
> The tag owner case is not being handled here.
> ImportserviceTags flow
> - create tagDef if not exists
> - Create service resource if not exists
> - Create tag Def and resource association with tag attributes
> - Refresh the tags_text in x_service_resource (This can be handled in a
> separate thread)
> The download json structure will be maintained to minimise plugin side
> changes.
> The importservicetags json input will remain the same.
> tag delta and tag dedup will be affected, it needs to be handled accordingly
> cc: [~madhan] [~avadhavkar]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (RANGER-4959) [Ranger-Admin] Remove use of x_tag table
[
https://issues.apache.org/jira/browse/RANGER-4959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17889303#comment-17889303
]
Madhan Neethiraj commented on RANGER-4959:
--
[~anandNadar] - proposed changes look good. Couple of further improvements to
consider:
# {{x_service_resource.tags_text}} already includes attributes of all tags
associated with the resource. Hence {{x_tag_resource_map.tagAttrs}} seems
unnecessary
# How do you plan to update existing foreign key
{{{}x_tag_resource_map.tag_id{}}}? Given {{x_tag}} table will effectively be
dropped, it makes sense to point this column to {{{}x_tag_def.id{}}}. If yes,
is the proposed new column, {{type}} necessary?
I don't think {{x_tag.owned_by}} field is currently used. So, it would be okay
to not carry this field in the proposed changes.
REST APIs in TagREST.java that support operations on x_tag entries should be
updated to throw an exception like UnsupportedOperationException:
* createTag(RangerTag tag)
* updateTag(RangerTag tag)
* deleteTag(Long id)
* getTag(Long id)
* getTagsByType(String type)
* getAllTags()
> [Ranger-Admin] Remove use of x_tag table
>
>
> Key: RANGER-4959
> URL: https://issues.apache.org/jira/browse/RANGER-4959
> Project: Ranger
> Issue Type: Improvement
> Components: admin
>Reporter: Anand Nadar
>Assignee: Anand Nadar
>Priority: Major
>
> Below is a metrics retrieved for importservicetags api to create tags and
> tag-resource association.
> |Duration|10 min|
> |Successful request |49|
> |Number of tags for each resource |6 |
> |Number of columns |100 |
> |Total resources tag mapping |6*100 = 600|
> |Total tag resource map in overall |49*600 = 29400 records in db |
> |Rate per min |2940|
> * Number of tables = 200k
> * Number of columns = 500
> * Avg tag on each column =6
> * Total resources tag mapping = 20 * 500*6 = 600,000,000
> * Time as per rate with 10 threads = 600,000,000/2940 = 204081 min =
> 141.722917 days
> Above is the performance of the importservicetags api with 2GB heap memory.
>
> Therefore to improve this performance, we are trying to do the below changes.
> # Remove usage of x_tag table
> # In the x_tag_resource_map table, the association should be between the tag
> def id and the resource id.
> # The x_tag_resource_map will have 2 new columns "tagAttrs" to store the
> tag attributes and "type" which will be the name of tagDef.
> # tags_text which is stored in x_service_resource table can have the below
> data to reduce its size.
> {code:java}
> [{"id":1069576,"isEnabled":true,"name":"TAG1","attributes":{"restricted1":"true"}}]
> {code}
> id, isEnabled, name - These data will be from x_tag_def
> attributes - This will be retrieved from the x_tag_resource_map table for
> that particular resource.
> The tag owner case is not being handled here.
> ImportserviceTags flow
> - create tagDef if not exists
> - Create service resource if not exists
> - Create tag Def and resource association with tag attributes
> - Refresh the tags_text in x_service_resource (This can be handled in a
> separate thread)
> The download json structure will be maintained to minimise plugin side
> changes.
> The importservicetags json input will remain the same.
> tag delta and tag dedup will be affected, it needs to be handled accordingly
> cc: [~madhan] [~avadhavkar]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (RANGER-4809) Utility to migrate admin audit logs in x_trx_log table x_trx_log_v2 table
[
https://issues.apache.org/jira/browse/RANGER-4809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17887050#comment-17887050
]
Madhan Neethiraj commented on RANGER-4809:
--
Following shows use of the tool in docker setup to migrate 300k transaction
logs (100k policies create/update/delete) from x_trx_log to x_trx_log_v2.
Migration completed in 22 seconds.
{noformat}
$ export RANGER_ADMIN_HOME=/opt/ranger/admin/
$ export RANGER_ADMIN_CONF=/opt/ranger/admin/conf/
$ export RANGER_ADMIN_LOG_DIR=/var/log/ranger/
$ /opt/ranger/admin/ews/ranger-admin-transaction-log-migrate.sh
RANGER_ADMIN_HOME : /opt/ranger/admin/
RANGER_ADMIN_CONF : /opt/ranger/admin/conf/
RANGER_ADMIN_LOG_DIR : /var/log/ranger/
Using heap size 1g
Migrating Transaction logs has started with PID - 6281
$ tail -fF /var/log/ranger/ranger_db_patch.log
...
2024-10-04 23:10:56,062 [main] INFO o.a.r.p.c.TrxLogV2MigrationUtil
(TrxLogV2MigrationUtil.java:105) ==> TrxLogV2MigrationUtil.execLoad()
2024-10-04 23:10:56,062 [Loader Monitor] INFO
o.a.r.p.c.TrxLogV2MigrationUtil$Stats (TrxLogV2MigrationUtil.java:391)
PROGRESS: 0 of 0 transactions processed. Last migrated transaction(id=null,
time=null). Counts(migrated: 0, failed: 0, already-migrated: 0)
2024-10-04 23:10:56,062 [main] INFO o.a.r.p.c.TrxLogV2MigrationUtil
(TrxLogV2MigrationUtil.java:122) ==> TrxLogV2MigrationUtil.migrateTrxLogs()
2024-10-04 23:10:57,211 [main] INFO o.a.r.p.c.TrxLogV2MigrationUtil
(TrxLogV2MigrationUtil.java:148) Found 300097 transactions to migrate
2024-10-04 23:10:57,212 [main] INFO o.a.r.p.c.TrxLogV2MigrationUtil
(TrxLogV2MigrationUtil.java:150) Starting 5 threads to migrate, commit batch
size: 25
2024-10-04 23:10:57,440 [Thread-5] INFO o.a.r.p.c.TrxLogV2MigrationUtil$Stats
(TrxLogV2MigrationUtil.java:391) PROGRESS: 1000 of 300097 transactions
processed. Last migrated transaction(id=9403, time=2024/10/04 22:41:43 +).
Counts(migrated: 1000, failed: 0, already-migrated: 0)
2024-10-04 23:10:57,575 [Thread-4] INFO o.a.r.p.c.TrxLogV2MigrationUtil$Stats
(TrxLogV2MigrationUtil.java:391) PROGRESS: 2000 of 300097 transactions
processed. Last migrated transaction(id=19420, time=2024/10/04 22:41:51 +).
Counts(migrated: 2000, failed: 0, already-migrated: 0)
2024-10-04 23:10:57,712 [Thread-8] INFO o.a.r.p.c.TrxLogV2MigrationUtil$Stats
(TrxLogV2MigrationUtil.java:391) PROGRESS: 3000 of 300097 transactions
processed. Last migrated transaction(id=29690, time=2024/10/04 22:41:57 +).
Counts(migrated: 3000, failed: 0, already-migrated: 0)
2024-10-04 23:10:57,833 [Thread-7] INFO o.a.r.p.c.TrxLogV2MigrationUtil$Stats
(TrxLogV2MigrationUtil.java:391) PROGRESS: 4000 of 300097 transactions
processed. Last migrated transaction(id=39370, time=2024/10/04 22:42:06 +).
Counts(migrated: 4002, failed: 0, already-migrated: 0)
2024-10-04 23:10:57,948 [Thread-7] INFO o.a.r.p.c.TrxLogV2MigrationUtil$Stats
(TrxLogV2MigrationUtil.java:391) PROGRESS: 5000 of 300097 transactions
processed. Last migrated transaction(id=49337, time=2024/10/04 22:42:13 +).
Counts(migrated: 5000, failed: 0, already-migrated: 0)
2024-10-04 23:10:58,073 [Thread-7] INFO o.a.r.p.c.TrxLogV2MigrationUtil$Stats
(TrxLogV2MigrationUtil.java:391) PROGRESS: 6000 of 300097 transactions
processed. Last migrated transaction(id=59323, time=2024/10/04 22:42:19 +).
Counts(migrated: 6002, failed: 0, already-migrated: 0)
2024-10-04 23:10:58,189 [Thread-5] INFO o.a.r.p.c.TrxLogV2MigrationUtil$Stats
(TrxLogV2MigrationUtil.java:391) PROGRESS: 7000 of 300097 transactions
processed. Last migrated transaction(id=69550, time=2024/10/04 22:42:26 +).
Counts(migrated: 7000, failed: 0, already-migrated: 0)
2024-10-04 23:10:58,300 [Thread-8] INFO o.a.r.p.c.TrxLogV2MigrationUtil$Stats
(TrxLogV2MigrationUtil.java:391) PROGRESS: 8000 of 300097 transactions
processed. Last migrated transaction(id=79099, time=2024/10/04 22:42:33 +).
Counts(migrated: 8000, failed: 0, already-migrated: 0)
2024-10-04 23:10:58,427 [Thread-8] INFO o.a.r.p.c.TrxLogV2MigrationUtil$Stats
(TrxLogV2MigrationUtil.java:391) PROGRESS: 9000 of 300097 transactions
processed. Last migrated transaction(id=89303, time=2024/10/04 22:42:40 +).
Counts(migrated: 9000, failed: 0, already-migrated: 0)
2024-10-04 23:10:58,543 [Thread-6] INFO o.a.r.p.c.TrxLogV2MigrationUtil$Stats
(TrxLogV2MigrationUtil.java:391) PROGRESS: 1 of 300097 transactions
processed. Last migrated transaction(id=98883, time=2024/10/04 22:42:46 +).
Counts(migrated: 10002, failed: 0, already-migrated: 0)
...
2024-10-04 23:11:17,929 [Thread-5] INFO o.a.r.p.c.TrxLogV2MigrationUtil$Stats
(TrxLogV2MigrationUtil.java:391) PROGRESS: 298000 of 300097 transactions
processed. Last migrated transaction(id=3063228, time=2024/10/04 23:04:42
+). Counts(migrated: 298001, failed: 0, already-migrated: 0)
2024-10-04 23:11:17,997 [Thread-6] INFO o.a.r.p.c.Tr
[jira] [Resolved] (RANGER-4809) Utility to migrate admin audit logs in x_trx_log table x_trx_log_v2 table
[
https://issues.apache.org/jira/browse/RANGER-4809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Madhan Neethiraj resolved RANGER-4809.
--
Resolution: Fixed
{noformat}
commit c1aaffb63fc8690d3d194f8c4d7074b1b508d067 (HEAD -> master, origin/master,
origin/HEAD)
Author: Rakesh Gupta
Date: Fri Oct 4 18:22:47 2024 +0530
RANGER-4809: Utility to migrate admin audit logs in x_trx_log table
x_trx_log_v2 table
Signed-off-by: Madhan Neethiraj
{noformat}
{noformat}
commit d06b18607415cf49e2737e36d64243c43542a931 (HEAD -> ranger-2.6,
origin/ranger-2.6)
Author: Rakesh Gupta
Date: Fri Oct 4 18:22:47 2024 +0530
RANGER-4809: Utility to migrate admin audit logs in x_trx_log table
x_trx_log_v2 table
Signed-off-by: Madhan Neethiraj
(cherry picked from commit c1aaffb63fc8690d3d194f8c4d7074b1b508d067)
{noformat}
> Utility to migrate admin audit logs in x_trx_log table x_trx_log_v2 table
> -
>
> Key: RANGER-4809
> URL: https://issues.apache.org/jira/browse/RANGER-4809
> Project: Ranger
> Issue Type: Improvement
> Components: admin
>Reporter: Madhan Neethiraj
>Assignee: Rakesh Gupta
>Priority: Major
> Fix For: 3.0.0, 2.6.0
>
>
> With updates in RANGER-4803, Ranger will store admin audit records in
> {{x_trx_log_v2}} table. Prior to this, {{x_trx_log}} table was used to store
> these records. To make earlier admin audits available to updated Ranger,
> existing entries in {{x_trx_log}} should be migrated to {{x_trx_log_v2}}
> table. Here are few notes to take into account:
> # For a given admin action, like create/update/delete of a policy, multiple
> rows are created in {{x_trx_log}} table - one row for each updated attribute.
> # For a given admin action, a single row is created in {{x_trx_log_v2}}
> table. This row captures details of all updated attributes in json format.
> # Migration utility should merge all rows for a given admin action,
> identified by column {{x_trx_log.trx_id}}, into a single record in
> {{x_trx_log_v2}} table. {{x_trx_log_v2.change_info}} column should be
> populated with contents of following columns in multiple {{x_trx_log}} rows:
> {{attr_name}}, {{prev_val}}, {{new_val}}.
> # The utility should provide an option to migrate only a subset of entries in
> {{x_trx_log}} table - for example, for a given time period (last 3 months,
> last 1 year). This will help avoid migrating older records that are of no
> interest to the customer.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (RANGER-4938) Ensure that only one instance of Ranger plugin is created in an Ozone Manager process
[ https://issues.apache.org/jira/browse/RANGER-4938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17885188#comment-17885188 ] Madhan Neethiraj edited comment on RANGER-4938 at 10/4/24 11:16 PM: Commit details: master: [https://github.com/apache/ranger/commit/4f297c35bab531877890a3e0a1c460a2eb3becd2] ranger-2.6: https://github.com/apache/ranger/commit/8904dfc06a505edfbe1c3ad97fa094083c1feb99 was (Author: abhayk): Commit details: master: https://github.com/apache/ranger/commit/4f297c35bab531877890a3e0a1c460a2eb3becd2 > Ensure that only one instance of Ranger plugin is created in an Ozone Manager > process > - > > Key: RANGER-4938 > URL: https://issues.apache.org/jira/browse/RANGER-4938 > Project: Ranger > Issue Type: Bug > Components: Ranger >Affects Versions: 2.6.0 >Reporter: Abhay Kulkarni >Assignee: Abhay Kulkarni >Priority: Major > Fix For: 2.6.0 > > > Ensure that only one instance of Ranger plugin is created in an Ozone Manager > process. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-4948) optimize use of tries in GDS policy engine
[ https://issues.apache.org/jira/browse/RANGER-4948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Madhan Neethiraj updated RANGER-4948: - Attachment: RANGER-4948.patch > optimize use of tries in GDS policy engine > --- > > Key: RANGER-4948 > URL: https://issues.apache.org/jira/browse/RANGER-4948 > Project: Ranger > Issue Type: Sub-task > Components: plugins >Affects Versions: 3.0.0 >Reporter: Madhan Neethiraj >Assignee: Madhan Neethiraj >Priority: Major > Fix For: 3.0.0 > > Attachments: RANGER-4948.patch > > > In GDS data model, a data-share consists of a bunch of resources > (databae/table/column/path/..) from a given service & security-zone. > In GDS policy engine implementation, each data-share instance has its own > resource-tries which are used to efficiently determine if a resource is part > of a data-share instance. To find data-shares for a resource, GDS policy > engine would perform as many trie lookup as the number of total data-shares. > Trie lookups can be minimized by having all resources across data-shares in a > resource-tries set. This will help improve the performance and memory > requirements, especially when working with large number of data-share > instances. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (RANGER-4948) optimize use of tries in GDS policy engine
Madhan Neethiraj created RANGER-4948: Summary: optimize use of tries in GDS policy engine Key: RANGER-4948 URL: https://issues.apache.org/jira/browse/RANGER-4948 Project: Ranger Issue Type: Sub-task Components: plugins Reporter: Madhan Neethiraj Assignee: Madhan Neethiraj In GDS data model, a data-share consists of a bunch of resources (databae/table/column/path/..) from a given service & security-zone. In GDS policy engine implementation, each data-share instance has its own resource-tries which are used to efficiently determine if a resource is part of a data-share instance. To find data-shares for a resource, GDS policy engine would perform as many trie lookup as the number of total data-shares. Trie lookups can be minimized by having all resources across data-shares in a resource-tries set. This will help improve the performance and memory requirements, especially when working with large number of data-share instances. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-4936) Feature to import and export individual policies
[ https://issues.apache.org/jira/browse/RANGER-4936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Madhan Neethiraj updated RANGER-4936: - Fix Version/s: 2.6.0 > Feature to import and export individual policies > > > Key: RANGER-4936 > URL: https://issues.apache.org/jira/browse/RANGER-4936 > Project: Ranger > Issue Type: New Feature > Components: admin >Reporter: Guru Thejus >Assignee: Guru Thejus >Priority: Minor > Fix For: 3.0.0, 2.6.0 > > Attachments: > 0001-Feature-for-download-and-upload-of-individual-polici.patch > > > Implement feature to import and export individual policies in ranger admin -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-4912) Upgrade Spring framework to 5.3.39
[ https://issues.apache.org/jira/browse/RANGER-4912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Madhan Neethiraj updated RANGER-4912: - Fix Version/s: 2.6.0 > Upgrade Spring framework to 5.3.39 > -- > > Key: RANGER-4912 > URL: https://issues.apache.org/jira/browse/RANGER-4912 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Affects Versions: 3.0.0 >Reporter: Pradeep Agrawal >Assignee: Pradeep Agrawal >Priority: Major > Fix For: 3.0.0, 2.6.0 > > Attachments: 0001-RANGER-4912-Upgrade-Spring-framework-to-5.3.39.patch > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (RANGER-4915) The default SSL Ciphers are too weak for user sync service
[ https://issues.apache.org/jira/browse/RANGER-4915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Madhan Neethiraj resolved RANGER-4915. -- Resolution: Fixed > The default SSL Ciphers are too weak for user sync service > -- > > Key: RANGER-4915 > URL: https://issues.apache.org/jira/browse/RANGER-4915 > Project: Ranger > Issue Type: Bug > Components: usersync >Affects Versions: 2.1.0, 2.2.0, 2.3.0, 2.4.0, 2.5.0 > Environment: Ranger 2.1.0 >Reporter: zhenye zhang >Priority: Minor > Labels: pull-request-available > Fix For: 3.0.0, 2.6.0 > > Time Spent: 20m > Remaining Estimate: 0h > > When we enable the ssl(ranger.usersync.ssl=true), the security tools report > the default SSL Ciphers are too weak. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (RANGER-4914) Ranger TagSync does not support ofs path parsing for Ozone-Atlas currently
[
https://issues.apache.org/jira/browse/RANGER-4914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Madhan Neethiraj resolved RANGER-4914.
--
Fix Version/s: 3.0.0
2.6.0
Resolution: Fixed
> Ranger TagSync does not support ofs path parsing for Ozone-Atlas currently
> --
>
> Key: RANGER-4914
> URL: https://issues.apache.org/jira/browse/RANGER-4914
> Project: Ranger
> Issue Type: Bug
> Components: Ranger
>Reporter: Fateh Singh
>Assignee: Fateh Singh
>Priority: Major
> Fix For: 3.0.0, 2.6.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> When trying to create hive table on top of ozone ofs path, tagsync throws
> below error
> {code:java}
> 2024-08-06 13:33:00,805 ERROR
> org.apache.ranger.tagsync.source.atlas.AtlasResourceMapperUtil: Could not get
> serviceResource for atlas entity:91faefd3-0592-400e-b452-0c9a437555db:
> java.lang.Exception: volume-name not found in attribute 'qualifiedName':
> ofs://ozone1722927741/vol1/bucket1/employee_db/meh1_hive_ozone_table_171@cm
> at
> org.apache.ranger.tagsync.source.atlas.AtlasResourceMapper.throwExceptionWithMessage(AtlasResourceMapper.java:120)
> at
> org.apache.ranger.tagsync.source.atlas.AtlasOzoneResourceMapper.buildResource(AtlasOzoneResourceMapper.java:113)
> at
> org.apache.ranger.tagsync.source.atlas.AtlasResourceMapperUtil.getRangerServiceResource(AtlasResourceMapperUtil.java:64)
> at
> org.apache.ranger.tagsync.source.atlas.AtlasNotificationMapper.buildServiceTags(AtlasNotificationMapper.java:259)
> at
> org.apache.ranger.tagsync.source.atlas.AtlasNotificationMapper.buildServiceTags(AtlasNotificationMapper.java:198)
> at
> org.apache.ranger.tagsync.source.atlas.AtlasNotificationMapper.processAtlasEntities(AtlasNotificationMapper.java:105)
> at
> org.apache.ranger.tagsync.source.atlas.AtlasTagSource$ConsumerRunnable.buildAndUploadServiceTags(AtlasTagSource.java:255)
> at
> org.apache.ranger.tagsync.source.atlas.AtlasTagSource$ConsumerRunnable.run(AtlasTagSource.java:230)
> at java.lang.Thread.run(Thread.java:748){code}
> Sample command which causes issues:
> {code:java}
> $ CREATE EXTERNAL TABLE t1 (id int, name string) row format delimited fields
> terminated by ' ' stored as textfile location
> 'ofs://ozone1/volume1/bucket1/table_oz_demo1';{code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (RANGER-4921) Fix docker compose command in CI
[ https://issues.apache.org/jira/browse/RANGER-4921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Madhan Neethiraj updated RANGER-4921: - Fix Version/s: 3.0.0 2.6.0 > Fix docker compose command in CI > > > Key: RANGER-4921 > URL: https://issues.apache.org/jira/browse/RANGER-4921 > Project: Ranger > Issue Type: Bug > Components: Ranger >Reporter: Abhishek Kumar >Assignee: Abhishek Kumar >Priority: Major > Fix For: 3.0.0, 2.6.0 > > Time Spent: 10m > Remaining Estimate: 0h > > docker-compose command changed to docker compose -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (RANGER-3801) Add support for Ozone in docker
[ https://issues.apache.org/jira/browse/RANGER-3801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Madhan Neethiraj resolved RANGER-3801. -- Fix Version/s: 3.0.0 2.6.0 Resolution: Fixed > Add support for Ozone in docker > --- > > Key: RANGER-3801 > URL: https://issues.apache.org/jira/browse/RANGER-3801 > Project: Ranger > Issue Type: Improvement > Components: build-infra >Reporter: Siyao Meng >Assignee: Abhishek Kumar >Priority: Major > Fix For: 3.0.0, 2.6.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Hi folks, > I see Ranger has dev support for hadoop, hbase, hive, solr and others. Shall > we add ozone as well? As Ozone does support using > [RangerOzoneAuthorizer|https://github.com/apache/ranger/blob/master/plugin-ozone/src/main/java/org/apache/ranger/authorization/ozone/authorizer/RangerOzoneAuthorizer.java] > as the ACL checker? > https://github.com/apache/ranger/tree/master/dev-support/ranger-docker > Thanks. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-4925) Cache downloaded archives during CI docker build
[ https://issues.apache.org/jira/browse/RANGER-4925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Madhan Neethiraj updated RANGER-4925: - Fix Version/s: 2.5.0 > Cache downloaded archives during CI docker build > > > Key: RANGER-4925 > URL: https://issues.apache.org/jira/browse/RANGER-4925 > Project: Ranger > Issue Type: Bug > Components: Ranger >Reporter: Abhishek Kumar >Assignee: Abhishek Kumar >Priority: Major > Fix For: 3.0.0, 2.5.0 > > > Cache downloaded archives (as part of docker build) in CI runs to optimize > docker build times. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (RANGER-4930) ranger
[
https://issues.apache.org/jira/browse/RANGER-4930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17881329#comment-17881329
]
Madhan Neethiraj edited comment on RANGER-4930 at 9/12/24 11:05 PM:
[~zhongwei11] - thank you for the crisp description of the issue. The fix in
RANGER-3554 should address this issue. This fix was included in 2.3.0 release.
I suggest upgrading to 2.3.0 or higher version of Ranger.
was (Author: madhan.neethiraj):
[~zhongwei11] - the fix in RANGER-3554 should address this issue. This fix was
included in 2.3.0 release.
> ranger
> --
>
> Key: RANGER-4930
> URL: https://issues.apache.org/jira/browse/RANGER-4930
> Project: Ranger
> Issue Type: Bug
> Components: admin
> Environment: ranger2.2
> x86 cnetos
>Reporter: wangzhongwei
>Priority: Major
>
> When user creates a policy through the web interface, or updates a
> policy, or creates a role or updates a role , the version of the policy or
> role will be modified. This version change is implemented through
> {{{}transactionSynchronizationAdapter.executeOnTransactionCommit{}}}. The
> database update operation occurs in the callback function
> {{{}afterCompletion{}}}. However, during debugging, it is occasionally found
> that {{afterCompletion}} is not invoked, which results in the policy or role
> being created or updated on the web interface, but the version remains
> unchanged, preventing the plugin from triggering the action to re-fetch the
> policy.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (RANGER-4930) ranger
[
https://issues.apache.org/jira/browse/RANGER-4930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17881329#comment-17881329
]
Madhan Neethiraj commented on RANGER-4930:
--
[~zhongwei11] - the fix in RANGER-3554 should address this issue. This fix was
included in 2.3.0 release.
> ranger
> --
>
> Key: RANGER-4930
> URL: https://issues.apache.org/jira/browse/RANGER-4930
> Project: Ranger
> Issue Type: Bug
> Components: admin
> Environment: ranger2.2
> x86 cnetos
>Reporter: wangzhongwei
>Priority: Major
>
> When user creates a policy through the web interface, or updates a
> policy, or creates a role or updates a role , the version of the policy or
> role will be modified. This version change is implemented through
> {{{}transactionSynchronizationAdapter.executeOnTransactionCommit{}}}. The
> database update operation occurs in the callback function
> {{{}afterCompletion{}}}. However, during debugging, it is occasionally found
> that {{afterCompletion}} is not invoked, which results in the policy or role
> being created or updated on the web interface, but the version remains
> unchanged, preventing the plugin from triggering the action to re-fetch the
> policy.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (RANGER-4809) Utility to migrate admin audit logs in x_trx_log table x_trx_log_v2 table
[
https://issues.apache.org/jira/browse/RANGER-4809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17881133#comment-17881133
]
Madhan Neethiraj commented on RANGER-4809:
--
[~rakeshgupta264] - by default, all data in x_trx_log table should be migrated
to x_trx_log_v2 table. So, I suggest "-1" as the default value of the
configuration.
> Utility to migrate admin audit logs in x_trx_log table x_trx_log_v2 table
> -
>
> Key: RANGER-4809
> URL: https://issues.apache.org/jira/browse/RANGER-4809
> Project: Ranger
> Issue Type: Improvement
> Components: admin
>Reporter: Madhan Neethiraj
>Assignee: Rakesh Gupta
>Priority: Major
> Fix For: 3.0.0, 2.6.0
>
>
> With updates in RANGER-4803, Ranger will store admin audit records in
> {{x_trx_log_v2}} table. Prior to this, {{x_trx_log}} table was used to store
> these records. To make earlier admin audits available to updated Ranger,
> existing entries in {{x_trx_log}} should be migrated to {{x_trx_log_v2}}
> table. Here are few notes to take into account:
> # For a given admin action, like create/update/delete of a policy, multiple
> rows are created in {{x_trx_log}} table - one row for each updated attribute.
> # For a given admin action, a single row is created in {{x_trx_log_v2}}
> table. This row captures details of all updated attributes in json format.
> # Migration utility should merge all rows for a given admin action,
> identified by column {{x_trx_log.trx_id}}, into a single record in
> {{x_trx_log_v2}} table. {{x_trx_log_v2.change_info}} column should be
> populated with contents of following columns in multiple {{x_trx_log}} rows:
> {{attr_name}}, {{prev_val}}, {{new_val}}.
> # The utility should provide an option to migrate only a subset of entries in
> {{x_trx_log}} table - for example, for a given time period (last 3 months,
> last 1 year). This will help avoid migrating older records that are of no
> interest to the customer.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (RANGER-4313) fix typo in DefaultSchemaRegistryClient to achieve mvn test
[
https://issues.apache.org/jira/browse/RANGER-4313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Madhan Neethiraj resolved RANGER-4313.
--
Fix Version/s: 3.0.0
2.6.0
Resolution: Fixed
[~coldestlin] - thank you for the fix. It is merged in master and ranger-2.6
branches.
{noformat}
commit de085b8b1ea3495b15cc8654fc7e818839228bfd
Author: Gee
Date: Tue Sep 10 22:43:46 2024 +0800
RANGER-4313: fix typo in DefaultSchemaRegistryClient (#272)
{noformat}
{noformat}
commit 2df63fd5f16d9216117eaa3d90951d828b687e27
Author: Gee
Date: Tue Sep 10 22:43:46 2024 +0800
RANGER-4313: fix typo in DefaultSchemaRegistryClient (#272)
(cherry picked from commit de085b8b1ea3495b15cc8654fc7e818839228bfd)
{noformat}
> fix typo in DefaultSchemaRegistryClient to achieve mvn test
> ---
>
> Key: RANGER-4313
> URL: https://issues.apache.org/jira/browse/RANGER-4313
> Project: Ranger
> Issue Type: Bug
> Components: plugins
>Reporter: geehanlin
>Priority: Minor
> Fix For: 3.0.0, 2.6.0
>
> Attachments: image-2023-07-13-12-27-40-647.png
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> fix typo in DefaultSchemaRegistryClient, otherwise `mvn test` will fail like
> below
>
> ```
> [INFO] ---
> [INFO] T E S T S
> [INFO] ---
> [INFO] Running
> org.apache.ranger.services.schema.registry.client.SchemaRegistryResourceMgrTest
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.584
> s <<< FAILURE! - in
> org.apache.ranger.services.schema.registry.client.SchemaRegistryResourceMgrTest
> [ERROR]
> org.apache.ranger.services.schema.registry.client.SchemaRegistryResourceMgrTest.getSchemaRegistryResources
> Time elapsed: 0.424 s <<< ERROR!
> java.lang.Error:
> Unresolved compilation problem:
> Syntax error on token ";", delete this token
> at
> org.apache.ranger.services.schema.registry.client.connection.DefaultSchemaRegistryClient.(DefaultSchemaRegistryClient.java:36)
> at
> org.apache.ranger.services.schema.registry.client.AutocompletionAgent.(AutocompletionAgent.java:50)
> at
> org.apache.ranger.services.schema.registry.client.util.TestAutocompletionAgent.(TestAutocompletionAgent.java:28)
> at
> org.apache.ranger.services.schema.registry.client.SchemaRegistryResourceMgrTest.getSchemaRegistryResources(SchemaRegistryResourceMgrTest.java:39)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> ```
>
>
> mvn version 3.6.3 + java 1.8
> ```
> mvn -v
> Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
> Maven home: /home/ubuntu/.sdkman/candidates/maven/current
> Java version: 1.8.0_372, vendor: Tencent, runtime:
> /home/ubuntu/.sdkman/candidates/java/8.0.372-kona/jre
> Default locale: en, platform encoding: UTF-8
> OS name: "linux", version: "5.15.0-72-generic", arch: "amd64", family: "unix"
> ```
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (RANGER-4814) Ranger - Upgrade Aircompressor to 0.27
[ https://issues.apache.org/jira/browse/RANGER-4814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Madhan Neethiraj updated RANGER-4814: - Fix Version/s: 2.6.0 > Ranger - Upgrade Aircompressor to 0.27 > -- > > Key: RANGER-4814 > URL: https://issues.apache.org/jira/browse/RANGER-4814 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Reporter: Dineshkumar Yadav >Assignee: Dineshkumar Yadav >Priority: Major > Fix For: 3.0.0, 2.6.0 > > > Ranger - Upgrade Aircompressor to 0.27 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (RANGER-4385) [Ranger UI] Querying role information by role name in the role management module does not display correct data
[
https://issues.apache.org/jira/browse/RANGER-4385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Madhan Neethiraj resolved RANGER-4385.
--
Fix Version/s: 3.0.0
2.6.0
(was: 2.2.0)
Resolution: Fixed
[~sixtofly] - thank you for the fix. It is merged in master and ranger-2.6
branches.
{noformat}
commit a463e2fb7cdc70629142bd9808f2a84c03689a99
Author: bing <369889...@qq.com>
Date: Tue Sep 10 02:16:09 2024 +0800
RANGER-4385: Fix role filtering logic in RolePredicateUtil (#280)
{noformat}
{noformat}
commit f7c74f86f6607d18c521889123aa05163380488d
Author: bing <369889...@qq.com>
Date: Tue Sep 10 02:16:09 2024 +0800
RANGER-4385: Fix role filtering logic in RolePredicateUtil (#280)
(cherry picked from commit a463e2fb7cdc70629142bd9808f2a84c03689a99)
{noformat}
> [Ranger UI] Querying role information by role name in the role management
> module does not display correct data
> --
>
> Key: RANGER-4385
> URL: https://issues.apache.org/jira/browse/RANGER-4385
> Project: Ranger
> Issue Type: Bug
> Components: Ranger
>Affects Versions: 2.2.0
>Reporter: bing
>Assignee: Dineshkumar Yadav
>Priority: Major
> Fix For: 3.0.0, 2.6.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> there is a role data:
> |Role Name|Users|Groups|Roles|
> |role_second|test|–|role_first|
> When I use a common user account to query through the role name parameter
> role_first, this piece of data is not displayed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (RANGER-4922) Reduce time to find tags associated with multi-level resource
[
https://issues.apache.org/jira/browse/RANGER-4922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Madhan Neethiraj updated RANGER-4922:
-
Fix Version/s: 3.0.0
2.6.0
> Reduce time to find tags associated with multi-level resource
> -
>
> Key: RANGER-4922
> URL: https://issues.apache.org/jira/browse/RANGER-4922
> Project: Ranger
> Issue Type: Improvement
> Components: Ranger
>Reporter: Abhay Kulkarni
>Assignee: Abhay Kulkarni
>Priority: Major
> Fix For: 3.0.0, 2.6.0
>
>
> With the following use case:
> * Service supports resource hierarchy with more than one level
> * Large number of tags are associated with the resources, with majority of
> tagged resources with values for all levels in resource hierarchy
> * Accessed resource does not have values for all levels in the resource
> hierarchy
> the time required to find the tags associated with the accessed resource is
> significant.
> When tested with a large number of tagged Ozone resources (~ 629,000) with
> approximately 20 tagged volumes and 103 tagged buckets and the rest being
> keys, the access evaluation times are:
> {code:java}
> (volume, bucket, key) : requestCount=629118, avgTimeTaken=49911ns
> (volume, bucket) : requestCount=103, avgTimeTaken=10738069ns
> (volume) :
> - requestCount=20, avgTimeTaken=21968890ns
> - requestCount=1056, avgTimeTaken=13763978ns (repeated requests in previous
> run multiple times) {code}
> This patch, using filtering and caching technique attempts to reduce this
> time.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (RANGER-4911) support scheduling of a dataset to expire at a specific time
[ https://issues.apache.org/jira/browse/RANGER-4911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Madhan Neethiraj updated RANGER-4911: - Attachment: RANGER-4911.patch > support scheduling of a dataset to expire at a specific time > > > Key: RANGER-4911 > URL: https://issues.apache.org/jira/browse/RANGER-4911 > Project: Ranger > Issue Type: Sub-task > Components: Ranger >Affects Versions: 3.0.0 >Reporter: Madhan Neethiraj >Assignee: Madhan Neethiraj >Priority: Major > Fix For: 3.0.0 > > Attachments: RANGER-4911.patch > > > Ability to automatically expire a dataset (i.e. make it inactive) can help > scenarios that require data (in a dataset) to be shared for a specific time > period. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (RANGER-4911) support scheduling of a dataset to expire at a specific time
Madhan Neethiraj created RANGER-4911: Summary: support scheduling of a dataset to expire at a specific time Key: RANGER-4911 URL: https://issues.apache.org/jira/browse/RANGER-4911 Project: Ranger Issue Type: Sub-task Components: Ranger Reporter: Madhan Neethiraj Assignee: Madhan Neethiraj Ability to automatically expire a dataset (i.e. make it inactive) can help scenarios that require data (in a dataset) to be shared for a specific time period. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-4908) update plugin library to use session cookies for all REST API calls when available
[ https://issues.apache.org/jira/browse/RANGER-4908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Madhan Neethiraj updated RANGER-4908: - Attachment: RANGER-4908.patch > update plugin library to use session cookies for all REST API calls when > available > -- > > Key: RANGER-4908 > URL: https://issues.apache.org/jira/browse/RANGER-4908 > Project: Ranger > Issue Type: Improvement > Components: plugins >Reporter: Madhan Neethiraj >Assignee: Madhan Neethiraj >Priority: Major > Fix For: 3.0.0, 2.6.0 > > Attachments: RANGER-4908.patch > > > Ranger plugins support using session cookies in REST calls to Ranger admin > server. However, this is currently used only in calls to download policies, > tags and roles. Rest of the calls, like grant/revoke don't use session > cookies, resulting in new login sessions to be created in Ranger. The plugin > library should be updated to use session cookie when available and configured. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (RANGER-4908) update plugin library to use session cookies for all REST API calls when available
Madhan Neethiraj created RANGER-4908: Summary: update plugin library to use session cookies for all REST API calls when available Key: RANGER-4908 URL: https://issues.apache.org/jira/browse/RANGER-4908 Project: Ranger Issue Type: Improvement Components: plugins Reporter: Madhan Neethiraj Assignee: Madhan Neethiraj Ranger plugins support using session cookies in REST calls to Ranger admin server. However, this is currently used only in calls to download policies, tags and roles. Rest of the calls, like grant/revoke don't use session cookies, resulting in new login sessions to be created in Ranger. The plugin library should be updated to use session cookie when available and configured. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (RANGER-4859) Update Trino service-def in Ranger for authorization changes
[ https://issues.apache.org/jira/browse/RANGER-4859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17873985#comment-17873985 ] Madhan Neethiraj commented on RANGER-4859: -- [~zhongwei11] - Trino plugin implementation was removed from Ranger git repo in 2.5.0 release and is being moved into Trino git repo - see [pull request #22675|https://github.com/trinodb/trino/pull/22675]. If you would need to use earlier version of Trino plugin from Ranger git repo, consider compiling it from [ranger-2.4 branch|https://github.com/apache/ranger/tree/ranger-2.4]. > Update Trino service-def in Ranger for authorization changes > > > Key: RANGER-4859 > URL: https://issues.apache.org/jira/browse/RANGER-4859 > Project: Ranger > Issue Type: Sub-task > Components: Ranger >Affects Versions: 3.0.0 >Reporter: Pradeep Agrawal >Assignee: Pradeep Agrawal >Priority: Major > Fix For: 3.0.0, 2.5.0 > > Attachments: image-2024-08-15-11-39-38-077.png, > image-2024-08-15-14-02-59-332.png > > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (RANGER-4904) update Hadoop version to 3.3.6
[
https://issues.apache.org/jira/browse/RANGER-4904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17873354#comment-17873354
]
Madhan Neethiraj edited comment on RANGER-4904 at 8/14/24 2:40 PM:
---
[~kokosing] - thank you for the patch. It is merged in master branch.
{noformat}
commit 4e365456f6533ee5515c5070c92e355198922c81 (HEAD -> master, origin/master,
origin/HEAD)
Author: Grzegorz Kokosiński
Date: Tue Aug 13 16:19:13 2024 -0700
RANGER-4904: updated Hadoop version to 3.3.6; removed avro dependency from
Ranger KMS assembly
Signed-off-by: Madhan Neethiraj
{noformat}
{noformat}
commit 32c677eca9f83638c1d7b840699efcf4851293a5 (HEAD -> ranger-2.6,
origin/ranger-2.6)
Author: Grzegorz Kokosiński
Date: Tue Aug 13 16:19:13 2024 -0700
RANGER-4904: updated Hadoop version to 3.3.6; removed avro dependency from
Ranger KMS assembly
Signed-off-by: Madhan Neethiraj
(cherry picked from commit 4e365456f6533ee5515c5070c92e355198922c81)
{noformat}
was (Author: madhan.neethiraj):
[~kokosing] - thank you for the patch. It is merged in master branch.
{noformat}
commit 4e365456f6533ee5515c5070c92e355198922c81 (HEAD -> master, origin/master,
origin/HEAD)
Author: Grzegorz Kokosiński
Date: Tue Aug 13 16:19:13 2024 -0700
RANGER-4904: updated Hadoop version to 3.3.6; removed avro dependency from
Ranger KMS assembly
Signed-off-by: Madhan Neethiraj
{noformat}
> update Hadoop version to 3.3.6
> --
>
> Key: RANGER-4904
> URL: https://issues.apache.org/jira/browse/RANGER-4904
> Project: Ranger
> Issue Type: Improvement
> Components: Ranger
>Reporter: Madhan Neethiraj
>Assignee: Grzegorz Kokosinski
>Priority: Major
> Fix For: 3.0.0, 2.6.0
>
>
> This Jira tracks [PR #370|https://github.com/apache/ranger/pull/370], to
> update Hadoop version to 3.3.6.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (RANGER-4904) update Hadoop version to 3.3.6
[ https://issues.apache.org/jira/browse/RANGER-4904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Madhan Neethiraj updated RANGER-4904: - Fix Version/s: 2.6.0 > update Hadoop version to 3.3.6 > -- > > Key: RANGER-4904 > URL: https://issues.apache.org/jira/browse/RANGER-4904 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Reporter: Madhan Neethiraj >Assignee: Grzegorz Kokosinski >Priority: Major > Fix For: 3.0.0, 2.6.0 > > > This Jira tracks [PR #370|https://github.com/apache/ranger/pull/370], to > update Hadoop version to 3.3.6. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (RANGER-4904) update Hadoop version to 3.3.6
[
https://issues.apache.org/jira/browse/RANGER-4904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Madhan Neethiraj resolved RANGER-4904.
--
Fix Version/s: 3.0.0
Resolution: Fixed
[~kokosing] - thank you for the patch. It is merged in master branch.
{noformat}
commit 4e365456f6533ee5515c5070c92e355198922c81 (HEAD -> master, origin/master,
origin/HEAD)
Author: Grzegorz Kokosiński
Date: Tue Aug 13 16:19:13 2024 -0700
RANGER-4904: updated Hadoop version to 3.3.6; removed avro dependency from
Ranger KMS assembly
Signed-off-by: Madhan Neethiraj
{noformat}
> update Hadoop version to 3.3.6
> --
>
> Key: RANGER-4904
> URL: https://issues.apache.org/jira/browse/RANGER-4904
> Project: Ranger
> Issue Type: Improvement
> Components: Ranger
>Reporter: Madhan Neethiraj
>Assignee: Grzegorz Kokosinski
>Priority: Major
> Fix For: 3.0.0
>
>
> This Jira tracks [PR #370|https://github.com/apache/ranger/pull/370], to
> update Hadoop version to 3.3.6.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (RANGER-4904) update Hadoop version to 3.3.6
Madhan Neethiraj created RANGER-4904: Summary: update Hadoop version to 3.3.6 Key: RANGER-4904 URL: https://issues.apache.org/jira/browse/RANGER-4904 Project: Ranger Issue Type: Improvement Components: Ranger Reporter: Madhan Neethiraj Assignee: Grzegorz Kokosinski This Jira tracks [PR #370|https://github.com/apache/ranger/pull/370], to update Hadoop version to 3.3.6. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (RANGER-4898) Docker setup to support kerberos authentication
Madhan Neethiraj created RANGER-4898: Summary: Docker setup to support kerberos authentication Key: RANGER-4898 URL: https://issues.apache.org/jira/browse/RANGER-4898 Project: Ranger Issue Type: Task Components: Ranger Reporter: Madhan Neethiraj Docker setup of Ranger services should be updated to support Kerberos authentication. Here is reference to get started: [https://www.confluent.io/blog/containerized-testing-with-kerberos-and-ssh/.|https://www.confluent.io/blog/containerized-testing-with-kerberos-and-ssh/] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-4888) The ranger setup.sh/db setup fails on Mysql ranger creation
[
https://issues.apache.org/jira/browse/RANGER-4888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Madhan Neethiraj updated RANGER-4888:
-
Fix Version/s: 2.6.0
> The ranger setup.sh/db setup fails on Mysql ranger creation
> ---
>
> Key: RANGER-4888
> URL: https://issues.apache.org/jira/browse/RANGER-4888
> Project: Ranger
> Issue Type: Bug
> Components: Ranger
>Affects Versions: 2.4.0
> Environment: Ubuntu 20.04
>Reporter: Steven Weiss
>Priority: Major
> Fix For: 2.6.0
>
>
> {quote}Running setup.sh
>
> Error:
>
> CREATE TABLE `x_trx_log_v2` ( `id` bigint(20) NOT NULL AUTO_INCREMENT,
> `create_time` datetime DEFAULT NULL, `added_by_id` bigint(20) DEFAULT NULL,
> `class_type` int(11) NOT NULL DEFAULT '0', `object_id` bigint(20) DEFAULT
> NULL, `parent_object_id` bigint(20) DEFAULT NULL,
> `parent_object_class_type` int(11) NOT NULL DEFAULT '0',
> `parent_object_name` varchar(1024) DEFAULT NULL, `object_name`
> varchar(1024) DEFAULT NULL, `change_info` MEDIUMTEXT NULL DEFAULT NULL,
> `trx_id` varchar(1024) DEFAULT NULL, `action` varchar(255) DEFAULT NULL,
> `sess_id` varchar(512) DEFAULT NULL, `req_id` varchar(30) DEFAULT NULL,
> `sess_type` varchar(30) DEFAULT NULL, PRIMARY KEY (`id`), KEY
> `x_trx_log_v2_FK_added_by_id` (`added_by_id`), KEY `x_trx_log_v2_cr_time`
> (`create_time`), KEY `x_trx_log_v2_trx_id` (`trx_id`) )ROW_FORMAT=DYNAMIC;
> java.sql.SQLSyntaxErrorException: Specified key was too long; max key length
> is 3072 bytes
> SQLException : SQL state: 42000 java.sql.SQLSyntaxErrorException: Specified
> key was too long; max key length is 3072 bytes ErrorCode: 1071
>
> Problem seems to be the trx_id is too long{quote}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (RANGER-4894) org.apache.ranger:ranger-trino-plugin
[
https://issues.apache.org/jira/browse/RANGER-4894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Madhan Neethiraj resolved RANGER-4894.
--
Resolution: Duplicate
[~phamo] - thank you for reporting the issue and subsequent updates. The
database error "key too long" for {{x_trx_log_v2.trx_id}} is tracked in
RANGER-4888.
> org.apache.ranger:ranger-trino-plugin
> --
>
> Key: RANGER-4894
> URL: https://issues.apache.org/jira/browse/RANGER-4894
> Project: Ranger
> Issue Type: Bug
> Components: plugins
>Affects Versions: 2.4.0
>Reporter: Femi
>Priority: Major
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Please can I skip org.apache.ranger:ranger-trino-plugin. Because I ran into
> this issue during compilation. or if there is any recommendation it will be
> greatly appreciated.
> [INFO] Building Ranger Security Plugin for Trino 3.0.0-SNAPSHOT
> [65/66]
> [INFO] [ jar
> ]-
> Downloading from apache.snapshots.https:
> https://repository.apache.org/content/repositories/snapshots/com/google/api/gax-bom/2.49.0/gax
> -bom-2.49.0.pom
> Downloading from apache.public.https:
> https://repository.apache.org/content/repositories/public/com/google/api/gax-bom/2.49.0/gax-bom-2
> .49.0.pom
> Downloading from google-maven-central-copy:
> https://maven-central.storage-download.googleapis.com/maven2/com/google/api/gax-bom/2.49.0/
> gax-bom-2.49.0.pom
> [INFO] I/O exception (java.net.SocketException) caught when processing
> request to \{s}->https://maven-central.storage-download.googleapi
> s.com:443: Connection timed out (Read failed)
> [INFO] Retrying request to
> \{s}->https://maven-central.storage-download.googleapis.com:443
>
> compilation issue when this command is runs mvn -DskipTests=true
> clean package
>
> Based on the retying effort it possible it firewall, since I dont need this
> plugin could this be skip during the plugin without any issue to the Ranger
> admin functionality.
> Best Regards
> Femi
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (RANGER-4897) Tests to cover REST APIs
Madhan Neethiraj created RANGER-4897: Summary: Tests to cover REST APIs Key: RANGER-4897 URL: https://issues.apache.org/jira/browse/RANGER-4897 Project: Ranger Issue Type: Test Components: Ranger Reporter: Madhan Neethiraj Test coverage for Apache Ranger REST APIs can be enhanced by having test suites that can run against a Ranger deployment - for example in docker containers. Such test suites will be of immense value in CI pipeline which already supports bringing up Ranger services in docker containers. Here are few REST API endpoints to start with: # service/public/v2 ([PublicAPIsv2.java|https://github.com/apache/ranger/blob/master/security-admin/src/main/java/org/apache/ranger/rest/PublicAPIsv2.java]) # service/xusers ([XUserREST.java|https://github.com/apache/ranger/blob/master/security-admin/src/main/java/org/apache/ranger/rest/XUserREST.java]) # service/roles ([RoleREST.java|https://github.com/apache/ranger/blob/master/security-admin/src/main/java/org/apache/ranger/rest/RoleREST.java]) # service/tags ([TagREST.java|https://github.com/apache/ranger/blob/master/security-admin/src/main/java/org/apache/ranger/rest/TagREST.java]) # service/gds ([GdsREST.java|https://github.com/apache/ranger/blob/master/security-admin/src/main/java/org/apache/ranger/rest/GdsREST.java]) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (RANGER-4810) Move Trino authorizer implementation from Ranger git repo to Trino repo
[ https://issues.apache.org/jira/browse/RANGER-4810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Madhan Neethiraj resolved RANGER-4810. -- Fix Version/s: 3.0.0 2.5.0 Resolution: Fixed > Move Trino authorizer implementation from Ranger git repo to Trino repo > --- > > Key: RANGER-4810 > URL: https://issues.apache.org/jira/browse/RANGER-4810 > Project: Ranger > Issue Type: Improvement > Components: plugins >Reporter: Madhan Neethiraj >Assignee: Madhan Neethiraj >Priority: Major > Fix For: 3.0.0, 2.5.0 > > > Moving Trino authorizer implementation from Ranger git repo to Trino repo has > several advantages, including: > # Keeping the authorizer in sync with the updates in SystemAccessControl > interface. For example, following changes in the latest Trino repo are not > compatible with the Trino authorizer in Ranger repo: > ## SystemAccessControl.checkCanAccessCatalog(): removed > ## SystemAccessControl.getRowFilter(): replaced with getRowFilters() > ## SystemAccessControl.getColumnMasks(): replaced with getColumnMask() > ## SystemAccessControl.checkCanSetSystemSessionProperty(): removed > ## SystemAccessControl.checkCanImpersonateUser(): signature changed > ## SystemAccessControl.checkCanAccessCatalog(): removed > ## SystemAccessControl.checkCanCreateSchema(): signature changed > ## SystemAccessControl.checkCanExecuteQuery(): removed > ## SystemAccessControl.checkCanViewQueryOwnedBy(): removed > ## SystemAccessControl.filterViewQueryOwnedBy(): signature changed > ## SystemAccessControl.checkCanKillQueryOwnedBy(): removed > ## SystemAccessControl.checkCanGrantExecuteFunctionPrivilege(): > removed/replaced > ## SystemAccessControl.checkCanExecuteFunction(): signature changed > ## ViewExpression(): constructor changed > ## AccessDeniedException.denyGrantExecuteFunctionPrivilege(): removed > # Trino requires more recent JDK versions (currently JDK 22) than Ranger > repo (which still supports JDK 8). Trino authorizer is built separately, as a > second phase, in Ranger repo using higher JDK versions. Moving the authorizer > to Trino repo will avoid this additional step. > > Trino seems to have a class loader isolation in place for its plugins, which > can eliminate the need for the shim layer used in Ranger plugin. This needs > to be considered along with this move. > Though the authorizer implementation would move to Trino repp, Ranger repo > will continue to have modules used in Ranger admin server for resource look > up and default policy creation (class RangerServiceTrino). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (RANGER-4748) Admin audits UI is slow when x_trx_log table has large numer of rows
[ https://issues.apache.org/jira/browse/RANGER-4748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Madhan Neethiraj resolved RANGER-4748. -- Fix Version/s: 3.0.0 2.5.0 Resolution: Fixed > Admin audits UI is slow when x_trx_log table has large numer of rows > - > > Key: RANGER-4748 > URL: https://issues.apache.org/jira/browse/RANGER-4748 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Reporter: Madhan Neethiraj >Assignee: Madhan Neethiraj >Priority: Major > Fix For: 3.0.0, 2.5.0 > > > Admin tab in Ranger audit UI lists changes performed on > policies/users/groups/security-zones/service - one row for each object. > Details of changes to an object (like old and new value of attributes) are > available in a dialog box that pops up on clicking the row. > API to retrieve list of admin audit log can take a long time when large > number of rows exists in that database table that stores change details i.e. > table named x_trx_log. This is due to the use of database view, vx_trx_log, > on top of table x_trx_log, which performs a group-by operation that would > require a full-table scan. This view is necessary since x_trx_log can have > multiple rows for one change to an object - one row for each changed > attribute. > To avoid this issue, one option to consider is store changes to all > attributes of an object in a single row (instead of one row per changed > attribute). This will eliminate the need for a view that performs group by. > > CC: [~siddheshphatak] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-4891) replace use of PrivilegedAction with PrivilegedExceptionAction in calls to UserGroupInfomation.doAs()
[
https://issues.apache.org/jira/browse/RANGER-4891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Madhan Neethiraj updated RANGER-4891:
-
Fix Version/s: 2.6.0
> replace use of PrivilegedAction with PrivilegedExceptionAction in calls to
> UserGroupInfomation.doAs()
> -
>
> Key: RANGER-4891
> URL: https://issues.apache.org/jira/browse/RANGER-4891
> Project: Ranger
> Issue Type: Improvement
> Components: plugins
>Reporter: Madhan Neethiraj
>Assignee: Madhan Neethiraj
>Priority: Major
> Fix For: 3.0.0, 2.6.0
>
> Attachments: RANGER-4891.patch
>
>
> {{UserGroupInformation.doAs(PrivilegedAction action)}} method was removed
> in Trino in a [recent
> update|https://github.com/trinodb/trino-hadoop-apache/pull/45]. This resulted
> in Ranger authorizer to fail to download policies/tags/roles from Ranger
> admin server. To address this issue, {{PrivilegedAction}} should be replaced
> with {{PrivilegedExceptionAction}} in calls to
> {{UserGroupInformation.doAs()}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (RANGER-4813) update audit UI to retrieve V2 trx log record
[
https://issues.apache.org/jira/browse/RANGER-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Madhan Neethiraj updated RANGER-4813:
-
Parent: (was: RANGER-4748)
Issue Type: Improvement (was: Sub-task)
> update audit UI to retrieve V2 trx log record
> -
>
> Key: RANGER-4813
> URL: https://issues.apache.org/jira/browse/RANGER-4813
> Project: Ranger
> Issue Type: Improvement
> Components: admin
>Reporter: Madhan Neethiraj
>Assignee: Abhishek
>Priority: Major
>
> Currently admin audit UI retrieves multiple V1 trx logs to gather details of
> an update (REST API: service/assets//report/\{transactionId}). This should be
> optimized to retrieve a single V2 trx log record (introduced in RANGER-4803)
> to render in UI. A new API should be added to retrieve V2 trx log record.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (RANGER-4813) update audit UI to retrieve V2 trx log record
[
https://issues.apache.org/jira/browse/RANGER-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Madhan Neethiraj updated RANGER-4813:
-
Fix Version/s: 3.0.0
2.6.0
> update audit UI to retrieve V2 trx log record
> -
>
> Key: RANGER-4813
> URL: https://issues.apache.org/jira/browse/RANGER-4813
> Project: Ranger
> Issue Type: Improvement
> Components: admin
>Reporter: Madhan Neethiraj
>Assignee: Abhishek
>Priority: Major
> Fix For: 3.0.0, 2.6.0
>
>
> Currently admin audit UI retrieves multiple V1 trx logs to gather details of
> an update (REST API: service/assets//report/\{transactionId}). This should be
> optimized to retrieve a single V2 trx log record (introduced in RANGER-4803)
> to render in UI. A new API should be added to retrieve V2 trx log record.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (RANGER-4809) Utility to migrate admin audit logs in x_trx_log table x_trx_log_v2 table
[
https://issues.apache.org/jira/browse/RANGER-4809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Madhan Neethiraj updated RANGER-4809:
-
Parent: (was: RANGER-4748)
Issue Type: Improvement (was: Sub-task)
> Utility to migrate admin audit logs in x_trx_log table x_trx_log_v2 table
> -
>
> Key: RANGER-4809
> URL: https://issues.apache.org/jira/browse/RANGER-4809
> Project: Ranger
> Issue Type: Improvement
> Components: admin
>Reporter: Madhan Neethiraj
>Priority: Major
>
> With updates in RANGER-4803, Ranger will store admin audit records in
> {{x_trx_log_v2}} table. Prior to this, {{x_trx_log}} table was used to store
> these records. To make earlier admin audits available to updated Ranger,
> existing entries in {{x_trx_log}} should be migrated to {{x_trx_log_v2}}
> table. Here are few notes to take into account:
> # For a given admin action, like create/update/delete of a policy, multiple
> rows are created in {{x_trx_log}} table - one row for each updated attribute.
> # For a given admin action, a single row is created in {{x_trx_log_v2}}
> table. This row captures details of all updated attributes in json format.
> # Migration utility should merge all rows for a given admin action,
> identified by column {{x_trx_log.trx_id}}, into a single record in
> {{x_trx_log_v2}} table. {{x_trx_log_v2.change_info}} column should be
> populated with contents of following columns in multiple {{x_trx_log}} rows:
> {{attr_name}}, {{prev_val}}, {{new_val}}.
> # The utility should provide an option to migrate only a subset of entries in
> {{x_trx_log}} table - for example, for a given time period (last 3 months,
> last 1 year). This will help avoid migrating older records that are of no
> interest to the customer.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (RANGER-4809) Utility to migrate admin audit logs in x_trx_log table x_trx_log_v2 table
[
https://issues.apache.org/jira/browse/RANGER-4809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Madhan Neethiraj updated RANGER-4809:
-
Fix Version/s: 3.0.0
2.6.0
> Utility to migrate admin audit logs in x_trx_log table x_trx_log_v2 table
> -
>
> Key: RANGER-4809
> URL: https://issues.apache.org/jira/browse/RANGER-4809
> Project: Ranger
> Issue Type: Improvement
> Components: admin
>Reporter: Madhan Neethiraj
>Priority: Major
> Fix For: 3.0.0, 2.6.0
>
>
> With updates in RANGER-4803, Ranger will store admin audit records in
> {{x_trx_log_v2}} table. Prior to this, {{x_trx_log}} table was used to store
> these records. To make earlier admin audits available to updated Ranger,
> existing entries in {{x_trx_log}} should be migrated to {{x_trx_log_v2}}
> table. Here are few notes to take into account:
> # For a given admin action, like create/update/delete of a policy, multiple
> rows are created in {{x_trx_log}} table - one row for each updated attribute.
> # For a given admin action, a single row is created in {{x_trx_log_v2}}
> table. This row captures details of all updated attributes in json format.
> # Migration utility should merge all rows for a given admin action,
> identified by column {{x_trx_log.trx_id}}, into a single record in
> {{x_trx_log_v2}} table. {{x_trx_log_v2.change_info}} column should be
> populated with contents of following columns in multiple {{x_trx_log}} rows:
> {{attr_name}}, {{prev_val}}, {{new_val}}.
> # The utility should provide an option to migrate only a subset of entries in
> {{x_trx_log}} table - for example, for a given time period (last 3 months,
> last 1 year). This will help avoid migrating older records that are of no
> interest to the customer.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (RANGER-4895) update ranger-2.6 branch pom.xml version to 2.6.0-SNAPSHOT
[ https://issues.apache.org/jira/browse/RANGER-4895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Madhan Neethiraj updated RANGER-4895: - Attachment: RANGER-4895.patch > update ranger-2.6 branch pom.xml version to 2.6.0-SNAPSHOT > -- > > Key: RANGER-4895 > URL: https://issues.apache.org/jira/browse/RANGER-4895 > Project: Ranger > Issue Type: Task > Components: Ranger >Affects Versions: 2.6.0 >Reporter: Madhan Neethiraj >Assignee: Madhan Neethiraj >Priority: Major > Attachments: RANGER-4895.patch > > > To get started with Apache Ranger 2.6 release, create branch ranger-2.6 and > update pom.xml to set version to 2.6.0-SNAPSHOT -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-4896) foreign key references to user table should be removed from login records table
[
https://issues.apache.org/jira/browse/RANGER-4896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Madhan Neethiraj updated RANGER-4896:
-
Attachment: RANGER-4895.patch
> foreign key references to user table should be removed from login records
> table
> ---
>
> Key: RANGER-4896
> URL: https://issues.apache.org/jira/browse/RANGER-4896
> Project: Ranger
> Issue Type: Improvement
> Components: Ranger
>Reporter: Madhan Neethiraj
>Priority: Major
>
> Ranger records all logins to admin server in {{x_auth_sess}} table. This
> table has foreign key reference to user record stored in {{x_portal_user}}
> table.
> When a user is deleted in Ranger, all related records in {{x_auth_sess}}
> table are deleted - which can take a long time if the user has large number
> of login records in {{x_auth_sess}} table.
> Removing foreign key references from {{x_auth_sess}} to {{x_portal_user}}
> table will eliminate the need to remove {{x_auth_sess}} records when a user
> is deleted - which can significantly reduce the time taken to delete users.
> In addition, this can improve performance of user login as well. Also, it
> might be desirable to retain login records even after deletion of users.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (RANGER-4896) foreign key references to user table should be removed from login records table
[
https://issues.apache.org/jira/browse/RANGER-4896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Madhan Neethiraj updated RANGER-4896:
-
Attachment: (was: RANGER-4895.patch)
> foreign key references to user table should be removed from login records
> table
> ---
>
> Key: RANGER-4896
> URL: https://issues.apache.org/jira/browse/RANGER-4896
> Project: Ranger
> Issue Type: Improvement
> Components: Ranger
>Reporter: Madhan Neethiraj
>Priority: Major
>
> Ranger records all logins to admin server in {{x_auth_sess}} table. This
> table has foreign key reference to user record stored in {{x_portal_user}}
> table.
> When a user is deleted in Ranger, all related records in {{x_auth_sess}}
> table are deleted - which can take a long time if the user has large number
> of login records in {{x_auth_sess}} table.
> Removing foreign key references from {{x_auth_sess}} to {{x_portal_user}}
> table will eliminate the need to remove {{x_auth_sess}} records when a user
> is deleted - which can significantly reduce the time taken to delete users.
> In addition, this can improve performance of user login as well. Also, it
> might be desirable to retain login records even after deletion of users.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (RANGER-4896) foreign key references to user table should be removed from login records table
[
https://issues.apache.org/jira/browse/RANGER-4896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Madhan Neethiraj updated RANGER-4896:
-
Description:
Ranger records all logins to admin server in {{x_auth_sess}} table. This table
has foreign key reference to user record stored in {{x_portal_user}} table.
When a user is deleted in Ranger, all related records in {{x_auth_sess}} table
are deleted - which can take a long time if the user has large number of login
records in {{x_auth_sess}} table.
Removing foreign key references from {{x_auth_sess}} to {{x_portal_user}} table
will eliminate the need to remove {{x_auth_sess}} records when a user is
deleted - which can significantly reduce the time taken to delete users. In
addition, this can improve performance of user login as well. Also, it might be
desirable to retain login records even after deletion of users.
was:
Ranger records all logins to admin server in {{x_auth_sess}} table. This table
has foreign key reference to user record stored in {{x_portal_user}} table.
When a user is deleted in Ranger, all related records in {{x_auth_sess}} table
are deleted - which can take a long time if the user has large number of login
records in {{x_auth_sess}} table.
Removing foreign key references from {{x_auth_sess}} to {{x_portal_user}} table
will eliminate the need to remove {{x_auth_sess}} records when a user is
deleted - which can significantly reduce the time taken to delete users. In
addition, this can improve performance of user login as well.
> foreign key references to user table should be removed from login records
> table
> ---
>
> Key: RANGER-4896
> URL: https://issues.apache.org/jira/browse/RANGER-4896
> Project: Ranger
> Issue Type: Improvement
> Components: Ranger
>Reporter: Madhan Neethiraj
>Priority: Major
>
> Ranger records all logins to admin server in {{x_auth_sess}} table. This
> table has foreign key reference to user record stored in {{x_portal_user}}
> table.
> When a user is deleted in Ranger, all related records in {{x_auth_sess}}
> table are deleted - which can take a long time if the user has large number
> of login records in {{x_auth_sess}} table.
> Removing foreign key references from {{x_auth_sess}} to {{x_portal_user}}
> table will eliminate the need to remove {{x_auth_sess}} records when a user
> is deleted - which can significantly reduce the time taken to delete users.
> In addition, this can improve performance of user login as well. Also, it
> might be desirable to retain login records even after deletion of users.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (RANGER-4896) foreign key references to user table should be removed from login records table
Madhan Neethiraj created RANGER-4896:
Summary: foreign key references to user table should be removed
from login records table
Key: RANGER-4896
URL: https://issues.apache.org/jira/browse/RANGER-4896
Project: Ranger
Issue Type: Improvement
Components: Ranger
Reporter: Madhan Neethiraj
Ranger records all logins to admin server in {{x_auth_sess}} table. This table
has foreign key reference to user record stored in {{x_portal_user}} table.
When a user is deleted in Ranger, all related records in {{x_auth_sess}} table
are deleted - which can take a long time if the user has large number of login
records in {{x_auth_sess}} table.
Removing foreign key references from {{x_auth_sess}} to {{x_portal_user}} table
will eliminate the need to remove {{x_auth_sess}} records when a user is
deleted - which can significantly reduce the time taken to delete users. In
addition, this can improve performance of user login as well.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (RANGER-4895) update ranger-2.6 branch pom.xml version to 2.6.0-SNAPSHOT
Madhan Neethiraj created RANGER-4895: Summary: update ranger-2.6 branch pom.xml version to 2.6.0-SNAPSHOT Key: RANGER-4895 URL: https://issues.apache.org/jira/browse/RANGER-4895 Project: Ranger Issue Type: Task Components: Ranger Affects Versions: 2.6.0 Reporter: Madhan Neethiraj Assignee: Madhan Neethiraj To get started with Apache Ranger 2.6 release, create branch ranger-2.6 and update pom.xml to set version to 2.6.0-SNAPSHOT -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-4894) org.apache.ranger:ranger-trino-plugin
[
https://issues.apache.org/jira/browse/RANGER-4894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Madhan Neethiraj updated RANGER-4894:
-
Fix Version/s: (was: 2.5.0)
> org.apache.ranger:ranger-trino-plugin
> --
>
> Key: RANGER-4894
> URL: https://issues.apache.org/jira/browse/RANGER-4894
> Project: Ranger
> Issue Type: Bug
> Components: plugins
>Affects Versions: 2.4.0
>Reporter: Femi
>Priority: Major
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Please can I skip org.apache.ranger:ranger-trino-plugin. Because I ran into
> this issue during compilation. or if there is any recommendation it will be
> greatly appreciated.
> [INFO] Building Ranger Security Plugin for Trino 3.0.0-SNAPSHOT
> [65/66]
> [INFO] [ jar
> ]-
> Downloading from apache.snapshots.https:
> https://repository.apache.org/content/repositories/snapshots/com/google/api/gax-bom/2.49.0/gax
> -bom-2.49.0.pom
> Downloading from apache.public.https:
> https://repository.apache.org/content/repositories/public/com/google/api/gax-bom/2.49.0/gax-bom-2
> .49.0.pom
> Downloading from google-maven-central-copy:
> https://maven-central.storage-download.googleapis.com/maven2/com/google/api/gax-bom/2.49.0/
> gax-bom-2.49.0.pom
> [INFO] I/O exception (java.net.SocketException) caught when processing
> request to \{s}->https://maven-central.storage-download.googleapi
> s.com:443: Connection timed out (Read failed)
> [INFO] Retrying request to
> \{s}->https://maven-central.storage-download.googleapis.com:443
>
> compilation issue when this command is runs mvn -DskipTests=true
> clean package
>
> Based on the retying effort it possible it firewall, since I dont need this
> plugin could this be skip during the plugin without any issue to the Ranger
> admin functionality.
> Best Regards
> Femi
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (RANGER-4792) Fix issue with creating index and import data in ElasticSearch as Audit database
[
https://issues.apache.org/jira/browse/RANGER-4792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17872612#comment-17872612
]
Madhan Neethiraj commented on RANGER-4792:
--
[~pradeep] - can you please review and resolve this issue if there are no
pending tasks?
> Fix issue with creating index and import data in ElasticSearch as Audit
> database
>
>
> Key: RANGER-4792
> URL: https://issues.apache.org/jira/browse/RANGER-4792
> Project: Ranger
> Issue Type: Bug
> Components: admin, audit
>Affects Versions: 3.0.0, 2.4.0, 2.5.0
> Environment: Container:
> - Linux: Debian buster
> - Java: openjdk- 11
> - Tested on kubernetes and openshift on AWS/Azure and on-prem
>Reporter: ognjenit
>Priority: Major
> Fix For: 3.0.0, 2.5.0
>
> Attachments: image-2024-06-27-21-30-33-630.png
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Hi all,
> I apologize in advance if I haven't adjusted this issue properly.
> Short description:
> I have deployed Trino with ranger-trino-plugin and I wanted to use
> ElasticSearch (7.10.2) as a place where I want to store the audit. When I
> configured ranger-admin to use elasticsearch (audit_store=elasticsearch and
> all other parameters audit_elasticsearch_*) I started getting errors in the
> catalina.out: java.lang.NoSuchFieldError: LUCENE_8_5_1. As I increased the
> version of Lucena, it was written in the logs that an even higher version was
> needed. So in the end, I moved it to 8.11.3 and 8.4.0 for lucene-spatial
> since it is the latest.
> After it was fixed, I tried to use https for elasticsearch protocol
> (audit_elasticsearch_protocol) however, it always showed that ranger-admin
> use http instead of https. I show in code that audit_elasticsearch_protocol
> is not configured well.
> As soon as it done, ranger admin successfully created ES index. However, I
> need to move from MiscUtil.toDate to MiscUtil.toLocalDate for evtTime
> "column" since I was getting error: Error converting value to date. Value =
> 2024-05-13T13:08:47.905Z
> As soon as I fixed it, I found an error in Trino that the plugin couldn't
> insert data into elasticsearch. After I upgraded httpcomponents bug-fix
> version, it's started inserting data.
> I opened PR with the fix 2.4.0 version, do I need to do the same on the
> master branch?
> PR: https://github.com/apache/ranger/pull/314/files
> h4. 1. Lucene version - fixed problem with writing data to ElasticSearch
> {*}Error{*}: java.lang.NoSuchFieldError: LUCENE_8_5_1
> I tried to change minor version one by one, but only latest version fit for
> me.
> Changes:
> * agents-audit/pom.xml: 311
> * pom.xml: 241
> h4. 2. Elastic search protocol - fixed problem with changing protocol
> Even though I changed ranger.audit.elasticsearch.protocol from http to https,
> audit plugin still using http protocol.
> Changes:
> * security-admin/scripts/ranger-admin-site-template.xml: 167-170
> * security-admin/scripts/setup.sh: 79, 794-797
> * security-admin/scripts/upgrade_admin.py: 116
> * security-admin/src/main/resources/conf.dist/ranger-admin-site.xml: 53-57
> *
> security-admin/src/test/java/org/apache/ranger/elasticsearch/ElasticSearchAccessAuditsServiceTest.java:
> 56
> h4. 3. Audit plugin - cannot write audit to ES
> {*}Error{*}: bootstrap method initialization exception
> After changing the version of httpcomponents I started seeing audit
> Changes:
> * pom.xml: 137, 138, 140
> h4. 4. Ranger admin console - Audit show 1-1-1970
> {*}Erro{*}: Error converting value to date. Value = 2024-05-13T13:08:47.905Z
> Even though evtTime was ok in ElasticSearch, ranger couldn't show it on GUI.
> Changes:
> *
> security-admin/src/main/java/org/apache/ranger/elasticsearch/ElasticSearchAccessAuditsService.java:
> 260
> *
> security-admin/src/main/java/org/apache/ranger/solr/SolrAccessAuditsService.java:
> 239
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (RANGER-4743) [JDK 11]: Fix ranger admin logging
[ https://issues.apache.org/jira/browse/RANGER-4743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Madhan Neethiraj updated RANGER-4743: - Fix Version/s: 3.0.0 2.6.0 (was: 2.5.0) > [JDK 11]: Fix ranger admin logging > --- > > Key: RANGER-4743 > URL: https://issues.apache.org/jira/browse/RANGER-4743 > Project: Ranger > Issue Type: Bug > Components: Ranger >Affects Versions: 2.4.0 >Reporter: Abhishek Kumar >Assignee: Abhishek Kumar >Priority: Blocker > Fix For: 3.0.0, 2.6.0 > > Attachments: 0001-RANGER-4743.patch > > > Currently, the ranger-admin logs do not appear when built and run with jdk 11. > Logback and slf4j versions need to be upgraded to fix the above issue. > More info: [https://logback.qos.ch/download.html] > CC: [~rmani] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (RANGER-3809) Implement authorizeByResourceType method of Kafka Authorizer
[
https://issues.apache.org/jira/browse/RANGER-3809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17872611#comment-17872611
]
Madhan Neethiraj commented on RANGER-3809:
--
[~rmani] - can you please review and resolve this Jira if there are no more
pending tasks?
> Implement authorizeByResourceType method of Kafka Authorizer
>
>
> Key: RANGER-3809
> URL: https://issues.apache.org/jira/browse/RANGER-3809
> Project: Ranger
> Issue Type: Improvement
> Components: plugins
>Affects Versions: 3.0.0
>Reporter: Andras Katona
>Assignee: Ramesh Mani
>Priority: Major
> Fix For: 3.0.0, 2.5.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> In Kafka 2.8 this new authorization method was introduced mainly to ease
> authorization (setup) of idempotent producers.
> The default implementation of {{authorizeByResourceType}} uses
> [acls()|https://github.com/apache/kafka/blob/a3c7017ff7e543b50f84110195690a253f19d9cf/clients/src/main/java/org/apache/kafka/server/authorizer/Authorizer.java#L154]
> which is [not
> implemented|https://github.com/apache/ranger/blob/fc7ad98fbb2ee7bb7d4cd3329abc438a73e0444a/plugin-kafka/src/main/java/org/apache/ranger/authorization/kafka/authorizer/RangerKafkaAuthorizer.java#L332-L335]
> in Kafka Ranger Plugin, but it is not really efficient and it is recommended
> to implement/override it in custom authorizer implementation (meaning Ranger
> in our case).
> {code}
> /**
> * Check if the caller is authorized to perform the given ACL operation
> on at least one
> * resource of the given type.
> *
> * Custom authorizer implementations should consider overriding this
> default implementation because:
> * 1. The default implementation iterates all AclBindings multiple times,
> without any caching
> *by principal, host, operation, permission types, and resource
> types. More efficient
> *implementations may be added in custom authorizers that directly
> access cached entries.
> * 2. The default implementation cannot integrate with any audit logging
> included in the
> *authorizer implementation.
> * 3. The default implementation does not support any custom authorizer
> configs or other access
> *rules apart from ACLs.
> *
> * @param requestContext Request context including request resourceType,
> security protocol and listener name
> * @param op The ACL operation to check
> * @param resourceType The resource type to check
> * @return Return {@link AuthorizationResult#ALLOWED} if
> the caller is authorized
> * to perform the given ACL operation on at least
> one resource of the
> * given type. Return {@link
> AuthorizationResult#DENIED} otherwise.
> */
> default AuthorizationResult
> authorizeByResourceType(AuthorizableRequestContext requestContext,
> AclOperation op, ResourceType resourceType) {
> {code}
> In Kafka 3.0.1, 3.1.1 and 3.2.0, producers are idempotent by default
> (KAFKA-13598) and the authorization of producer initialization fails, in case
> the user doesn't have the deprecated idempotent_write access, as it will call
> the {{authorizeByResourceType}} and that calls {{acls}}.
> {code}
> public Iterable acls(AclBindingFilter filter) {
> logger.error("(getting) acls is not supported by Ranger for Kafka");
> throw new UnsupportedOperationException("(getting) acls is not supported
> by Ranger for Kafka");
> }
> {code}
> With [this
> commit|https://github.com/apache/ranger/commit/0ec279474e0439f6c5e7d4497db191fb7cc99bc1]
> {{authorizeByResourceType}} got an implementation with a basic validation
> check and a (constant) denial response. It's just making
> UnsupportedOperationException disappear and having an expected response for
> initProducer authorization.
> Until the proper implementation is done, the idempotent_write access should
> be granted for the producer.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (RANGER-4844) Release Apache Ranger 2.5.0
[ https://issues.apache.org/jira/browse/RANGER-4844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Madhan Neethiraj resolved RANGER-4844. -- Resolution: Fixed > Release Apache Ranger 2.5.0 > --- > > Key: RANGER-4844 > URL: https://issues.apache.org/jira/browse/RANGER-4844 > Project: Ranger > Issue Type: Task > Components: Ranger >Affects Versions: 2.4.0 >Reporter: Bhavik Patel >Assignee: Bhavik Patel >Priority: Major > Fix For: 2.5.0 > > > Tracking 2.5.0 release tasks. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-4893) Enhance trie to support process of evaluators during traversal
[ https://issues.apache.org/jira/browse/RANGER-4893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Madhan Neethiraj updated RANGER-4893: - Attachment: RANGER-4893.patch > Enhance trie to support process of evaluators during traversal > -- > > Key: RANGER-4893 > URL: https://issues.apache.org/jira/browse/RANGER-4893 > Project: Ranger > Issue Type: Improvement > Components: plugins >Reporter: Madhan Neethiraj >Assignee: Madhan Neethiraj >Priority: Major > Fix For: 3.0.0 > > Attachments: RANGER-4893.patch > > > Ranger policy engine uses trie data structure to organize resources for > faster retrieval of policies/tags/zones associated with a given resource. > When a resource consists of multiple elements, like database/table/column, > multiple trie instances are consulted to retrieve policies/tags/zones > associated with the resource. Such multi-trie retrieval can be optimized > with a 2-pass traversal - first pass to get count and the second pass to get > the actual objects. Trie data structure used in Ranger policy engine should > be updated to support this optimization. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (RANGER-4893) Enhance trie to support process of evaluators during traversal
Madhan Neethiraj created RANGER-4893: Summary: Enhance trie to support process of evaluators during traversal Key: RANGER-4893 URL: https://issues.apache.org/jira/browse/RANGER-4893 Project: Ranger Issue Type: Improvement Components: plugins Reporter: Madhan Neethiraj Assignee: Madhan Neethiraj Ranger policy engine uses trie data structure to organize resources for faster retrieval of policies/tags/zones associated with a given resource. When a resource consists of multiple elements, like database/table/column, multiple trie instances are consulted to retrieve policies/tags/zones associated with the resource. Such multi-trie retrieval can be optimized with a 2-pass traversal - first pass to get count and the second pass to get the actual objects. Trie data structure used in Ranger policy engine should be updated to support this optimization. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-4891) replace use of PrivilegedAction with PrivilegedExceptionAction in calls to UserGroupInfomation.doAs()
[
https://issues.apache.org/jira/browse/RANGER-4891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Madhan Neethiraj updated RANGER-4891:
-
Attachment: RANGER-4891.patch
> replace use of PrivilegedAction with PrivilegedExceptionAction in calls to
> UserGroupInfomation.doAs()
> -
>
> Key: RANGER-4891
> URL: https://issues.apache.org/jira/browse/RANGER-4891
> Project: Ranger
> Issue Type: Improvement
> Components: plugins
>Reporter: Madhan Neethiraj
>Assignee: Madhan Neethiraj
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: RANGER-4891.patch
>
>
> {{UserGroupInformation.doAs(PrivilegedAction action)}} method was removed
> in Trino in a [recent
> update|https://github.com/trinodb/trino-hadoop-apache/pull/45]. This resulted
> in Ranger authorizer to fail to download policies/tags/roles from Ranger
> admin server. To address this issue, {{PrivilegedAction}} should be replaced
> with {{PrivilegedExceptionAction}} in calls to
> {{UserGroupInformation.doAs()}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (RANGER-4891) replace use of PrivilegedAction with PrivilegedExceptionAction in calls to UserGroupInfomation.doAs()
Madhan Neethiraj created RANGER-4891:
Summary: replace use of PrivilegedAction with
PrivilegedExceptionAction in calls to UserGroupInfomation.doAs()
Key: RANGER-4891
URL: https://issues.apache.org/jira/browse/RANGER-4891
Project: Ranger
Issue Type: Improvement
Components: plugins
Reporter: Madhan Neethiraj
Assignee: Madhan Neethiraj
{{UserGroupInformation.doAs(PrivilegedAction action)}} method was removed in
Trino in a [recent
update|https://github.com/trinodb/trino-hadoop-apache/pull/45]. This resulted
in Ranger authorizer to fail to download policies/tags/roles from Ranger admin
server. To address this issue, {{PrivilegedAction}} should be replaced with
{{PrivilegedExceptionAction}} in calls to {{UserGroupInformation.doAs()}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (RANGER-4889) update mem-sizing tool to support access request evaluations
[ https://issues.apache.org/jira/browse/RANGER-4889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Madhan Neethiraj updated RANGER-4889: - Attachment: RANGER-4889.patch > update mem-sizing tool to support access request evaluations > > > Key: RANGER-4889 > URL: https://issues.apache.org/jira/browse/RANGER-4889 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Reporter: Madhan Neethiraj >Assignee: Madhan Neethiraj >Priority: Major > Fix For: 3.0.0 > > Attachments: RANGER-4889.patch > > > RangerMemSizing tool is useful in finding memory requirements for a given set > of policies/tags/userstore/roles. Enhancing this tool to find time taken to > evaluate access requests, with the given policies/tags/userstore/roles can > help in understanding the performance of policy evaluation. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (RANGER-4889) update mem-sizing tool to support access request evaluations
Madhan Neethiraj created RANGER-4889: Summary: update mem-sizing tool to support access request evaluations Key: RANGER-4889 URL: https://issues.apache.org/jira/browse/RANGER-4889 Project: Ranger Issue Type: Improvement Components: Ranger Reporter: Madhan Neethiraj Assignee: Madhan Neethiraj RangerMemSizing tool is useful in finding memory requirements for a given set of policies/tags/userstore/roles. Enhancing this tool to find time taken to evaluate access requests, with the given policies/tags/userstore/roles can help in understanding the performance of policy evaluation. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-4885) upgrade from 2.4.0 to 2.5.0 fails due to missing column
[
https://issues.apache.org/jira/browse/RANGER-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Madhan Neethiraj updated RANGER-4885:
-
Attachment: RANGER-4885.patch
> upgrade from 2.4.0 to 2.5.0 fails due to missing column
> ---
>
> Key: RANGER-4885
> URL: https://issues.apache.org/jira/browse/RANGER-4885
> Project: Ranger
> Issue Type: Bug
> Components: Ranger
>Affects Versions: 2.5.0
>Reporter: Madhan Neethiraj
>Assignee: Madhan Neethiraj
>Priority: Blocker
> Fix For: 2.5.0
>
> Attachments: RANGER-4885.patch
>
>
> Apache Ranger 2.5.0 upgraded from 2.4.0 fails during startup with the
> following error:
> {noformat}
> 2024-08-01 18:24:05,193 [I] java patch
> PatchForOzoneServiceDefConfigUpdate_J10051 is being applied..
> log4j:WARN No appenders could be found for logger
> (org.apache.ranger.patch.PatchForOzoneServiceDefConfigUpdate_J10051).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
> info.
> [EL Warning]: metadata: 2024-08-01
> 18:24:06.907--ServerSession(1578026015)--You have specified multiple ids for
> the entity class [org.apache.ranger.entity.view.VXXPrincipal] without
> specifying an @IdClass. By doing this you may lose the ability to find by
> identity, distributed cache support etc. Note: You may however use
> EntityManager find operations by passing a list of primary key fields. Else,
> you will have to use JPQL queries to read your entities. For other id options
> see @PrimaryKey.
> [EL Warning]: 2024-08-01 18:24:07.888--UnitOfWork(712753515)--Exception
> [EclipseLink-4002] (Eclipse Persistence Services -
> 2.7.12.v20230209-e5c4074ef3):
> org.eclipse.persistence.exceptions.DatabaseException
> Internal Exception: org.postgresql.util.PSQLException: ERROR: column
> "category" does not exist
> Position: 25
> Error Code: 0
> Call: SELECT id, ADDED_BY_ID, category, CREATE_TIME, datamask_options,
> def_id, item_id, label, name, sort_order, rb_key_label, rowfilter_options,
> UPDATE_TIME, UPD_BY_ID FROM x_access_type_def WHERE (def_id = ?) ORDER BY
> sort_order
> bind => [1 parameter bound]
> Query: ReadAllQuery(name="XXAccessTypeDef.findByServiceDefId"
> referenceClass=XXAccessTypeDef sql="SELECT id, ADDED_BY_ID, category,
> CREATE_TIME, datamask_options, def_id, item_id, label, name, sort_order,
> rb_key_label, rowfilter_options, UPDATE_TIME, UPD_BY_ID FROM
> x_access_type_def WHERE (def_id = ?) ORDER BY sort_order")
> [EL Warning]: 2024-08-01 18:24:07.893--UnitOfWork(712753515)--Exception
> [EclipseLink-4002] (Eclipse Persistence Services -
> 2.7.12.v20230209-e5c4074ef3):
> org.eclipse.persistence.exceptions.DatabaseException
> Internal Exception: org.postgresql.util.PSQLException: ERROR: current
> transaction is aborted, commands ignored until end of transaction block
> Error Code: 0
> ...
> ...
> [EL Warning]: 2024-08-01 18:24:08.194--UnitOfWork(1608894091)--Exception
> [EclipseLink-4002] (Eclipse Persistence Services -
> 2.7.12.v20230209-e5c4074ef3):
> org.eclipse.persistence.exceptions.DatabaseException
> Internal Exception: org.postgresql.util.PSQLException: ERROR: column
> "category" does not exist
> Position: 25
> Error Code: 0
> Call: SELECT id, ADDED_BY_ID, category, CREATE_TIME, datamask_options,
> def_id, item_id, label, name, sort_order, rb_key_label, rowfilter_options,
> UPDATE_TIME, UPD_BY_ID FROM x_access_type_def WHERE (def_id = ?) ORDER BY
> sort_order
> bind => [1 parameter bound]
> Query: ReadAllQuery(name="XXAccessTypeDef.findByServiceDefId"
> referenceClass=XXAccessTypeDef sql="SELECT id, ADDED_BY_ID, category,
> CREATE_TIME, datamask_options, def_id, item_id, label, name, sort_order,
> rb_key_label, rowfilter_options, UPDATE_TIME, UPD_BY_ID FROM
> x_access_type_def WHERE (def_id = ?) ORDER BY sort_order")
> 2024-08-01 18:24:08,227 [JISQL] /usr/lib/jvm/java-8-openjdk-arm64/bin/java
> -cp /usr/share/java/postgresql.jar:/opt/ranger/ranger-2.5.0-admin/jisql/lib/*
> org.apache.util.sql.Jisql -driver postgresql -cstring
> jdbc:postgresql://ranger-db/ranger -u rangeradmin -p '' -noheader
> -trim -c \; -query "delete from x_db_version_h where version = 'J10051' and
> active = 'N' and updated_by='ranger.example.com';"
> 2024-08-01 18:24:08,308 [E] applying java patch
> PatchForOzoneServiceDefConfigUpdate_J10051 failed
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (RANGER-4885) upgrade from 2.4.0 to 2.5.0 fails due to missing column
Madhan Neethiraj created RANGER-4885:
Summary: upgrade from 2.4.0 to 2.5.0 fails due to missing column
Key: RANGER-4885
URL: https://issues.apache.org/jira/browse/RANGER-4885
Project: Ranger
Issue Type: Bug
Components: Ranger
Affects Versions: 2.5.0
Reporter: Madhan Neethiraj
Assignee: Madhan Neethiraj
Apache Ranger 2.5.0 upgraded from 2.4.0 fails during startup with the following
error:
{noformat}
2024-08-01 18:24:05,193 [I] java patch
PatchForOzoneServiceDefConfigUpdate_J10051 is being applied..
log4j:WARN No appenders could be found for logger
(org.apache.ranger.patch.PatchForOzoneServiceDefConfigUpdate_J10051).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
info.
[EL Warning]: metadata: 2024-08-01 18:24:06.907--ServerSession(1578026015)--You
have specified multiple ids for the entity class
[org.apache.ranger.entity.view.VXXPrincipal] without specifying an @IdClass. By
doing this you may lose the ability to find by identity, distributed cache
support etc. Note: You may however use EntityManager find operations by passing
a list of primary key fields. Else, you will have to use JPQL queries to read
your entities. For other id options see @PrimaryKey.
[EL Warning]: 2024-08-01 18:24:07.888--UnitOfWork(712753515)--Exception
[EclipseLink-4002] (Eclipse Persistence Services -
2.7.12.v20230209-e5c4074ef3):
org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.postgresql.util.PSQLException: ERROR: column "category"
does not exist
Position: 25
Error Code: 0
Call: SELECT id, ADDED_BY_ID, category, CREATE_TIME, datamask_options, def_id,
item_id, label, name, sort_order, rb_key_label, rowfilter_options, UPDATE_TIME,
UPD_BY_ID FROM x_access_type_def WHERE (def_id = ?) ORDER BY sort_order
bind => [1 parameter bound]
Query: ReadAllQuery(name="XXAccessTypeDef.findByServiceDefId"
referenceClass=XXAccessTypeDef sql="SELECT id, ADDED_BY_ID, category,
CREATE_TIME, datamask_options, def_id, item_id, label, name, sort_order,
rb_key_label, rowfilter_options, UPDATE_TIME, UPD_BY_ID FROM x_access_type_def
WHERE (def_id = ?) ORDER BY sort_order")
[EL Warning]: 2024-08-01 18:24:07.893--UnitOfWork(712753515)--Exception
[EclipseLink-4002] (Eclipse Persistence Services -
2.7.12.v20230209-e5c4074ef3):
org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.postgresql.util.PSQLException: ERROR: current
transaction is aborted, commands ignored until end of transaction block
Error Code: 0
...
...
[EL Warning]: 2024-08-01 18:24:08.194--UnitOfWork(1608894091)--Exception
[EclipseLink-4002] (Eclipse Persistence Services -
2.7.12.v20230209-e5c4074ef3):
org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.postgresql.util.PSQLException: ERROR: column "category"
does not exist
Position: 25
Error Code: 0
Call: SELECT id, ADDED_BY_ID, category, CREATE_TIME, datamask_options, def_id,
item_id, label, name, sort_order, rb_key_label, rowfilter_options, UPDATE_TIME,
UPD_BY_ID FROM x_access_type_def WHERE (def_id = ?) ORDER BY sort_order
bind => [1 parameter bound]
Query: ReadAllQuery(name="XXAccessTypeDef.findByServiceDefId"
referenceClass=XXAccessTypeDef sql="SELECT id, ADDED_BY_ID, category,
CREATE_TIME, datamask_options, def_id, item_id, label, name, sort_order,
rb_key_label, rowfilter_options, UPDATE_TIME, UPD_BY_ID FROM x_access_type_def
WHERE (def_id = ?) ORDER BY sort_order")
2024-08-01 18:24:08,227 [JISQL] /usr/lib/jvm/java-8-openjdk-arm64/bin/java
-cp /usr/share/java/postgresql.jar:/opt/ranger/ranger-2.5.0-admin/jisql/lib/*
org.apache.util.sql.Jisql -driver postgresql -cstring
jdbc:postgresql://ranger-db/ranger -u rangeradmin -p '' -noheader -trim
-c \; -query "delete from x_db_version_h where version = 'J10051' and active =
'N' and updated_by='ranger.example.com';"
2024-08-01 18:24:08,308 [E] applying java patch
PatchForOzoneServiceDefConfigUpdate_J10051 failed
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (RANGER-4884) updated dependent library version: hadoop, aws sdk, avro, snakeyaml
[
https://issues.apache.org/jira/browse/RANGER-4884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Madhan Neethiraj resolved RANGER-4884.
--
Fix Version/s: 3.0.0
Resolution: Fixed
{noformat}
commit f1c8f00ecb6fa444080ead3d11a1e48a7df55e3b (HEAD -> master, origin/master,
origin/HEAD)
Author: Grzegorz Kokosiński
Date: Sat Jul 27 15:35:12 2024 +0200
RANGER-4884: updated dependent library versions: Hadoop, AWS SDK, avro,
snakeyaml; excluded io.netty
Signed-off-by: Madhan Neethiraj
{noformat}
> updated dependent library version: hadoop, aws sdk, avro, snakeyaml
> ---
>
> Key: RANGER-4884
> URL: https://issues.apache.org/jira/browse/RANGER-4884
> Project: Ranger
> Issue Type: Improvement
> Components: Ranger
>Affects Versions: 3.0.0, 2.5.0
>Reporter: Madhan Neethiraj
>Assignee: Grzegorz Kokosinski
>Priority: Major
> Fix For: 3.0.0
>
>
> This Jira tracks following pull requests by [~kokosing]:
> # [#363: Update hadoop to 3.3.4|https://github.com/apache/ranger/pull/363]
> # [#364: Exclude all io.netty from hive-agent
> tests|https://github.com/apache/ranger/pull/364]
> # [#365: Update AWS SDK to
> 1.12.765|https://github.com/apache/ranger/pull/365]
> # [#366: Exclude avro dependency|https://github.com/apache/ranger/pull/366]
> # [#367: Exclude snakeyaml dependency to avoid
> CVEs|https://github.com/apache/ranger/pull/367]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (RANGER-4884) updated dependent library version: hadoop, aws sdk, avro, snakeyaml
Madhan Neethiraj created RANGER-4884: Summary: updated dependent library version: hadoop, aws sdk, avro, snakeyaml Key: RANGER-4884 URL: https://issues.apache.org/jira/browse/RANGER-4884 Project: Ranger Issue Type: Improvement Components: Ranger Affects Versions: 3.0.0, 2.5.0 Reporter: Madhan Neethiraj Assignee: Grzegorz Kokosinski This Jira tracks following pull requests by [~kokosing]: # [#363: Update hadoop to 3.3.4|https://github.com/apache/ranger/pull/363] # [#364: Exclude all io.netty from hive-agent tests|https://github.com/apache/ranger/pull/364] # [#365: Update AWS SDK to 1.12.765|https://github.com/apache/ranger/pull/365] # [#366: Exclude avro dependency|https://github.com/apache/ranger/pull/366] # [#367: Exclude snakeyaml dependency to avoid CVEs|https://github.com/apache/ranger/pull/367] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (RANGER-4883) audit to HDFS fails with error: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "hdfs"
Madhan Neethiraj created RANGER-4883:
Summary: audit to HDFS fails with error:
org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme
"hdfs"
Key: RANGER-4883
URL: https://issues.apache.org/jira/browse/RANGER-4883
Project: Ranger
Issue Type: Bug
Components: audit, plugins
Affects Versions: 3.0.0, 2.5.0
Reporter: Madhan Neethiraj
Assignee: Madhan Neethiraj
Writing audit log to HDFS fails with the following error in Kafka and Knox
plugins:
Kafka plugin:
{noformat}
[2024-08-01 01:48:50,242] ERROR Exception encountered while writing audits to
HDFS! (org.apache.ranger.audit.utils.RangerJSONAuditWriter)
org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme
"hdfs"
at
org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3443)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3466)
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
at
org.apache.ranger.audit.utils.AbstractRangerAuditWriter.createFileSystemFolders(AbstractRangerAuditWriter.java:97)
{noformat}
Knox plugin:
{noformat}
01:52:21.193 [org.apache.ranger.audit.queue.AuditBatchQueue1] ERROR
org.apache.ranger.audit.utils.RangerJSONAuditWriter - Exception encountered
while writing audits to HDFS!
org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme
"hdfs"
at
org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3443)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3466)
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
at
org.apache.ranger.audit.utils.AbstractRangerAuditWriter.createFileSystemFolders(AbstractRangerAuditWriter.java:97)
{noformat}
This is due to the plugins packaging missing library
{{{}hadoop-hdfs-client{}}}, which includes the necessary entry to handle "hdfs"
file scheme in META-INF/services/
org.apache.hadoop.hdfs.fs.FileSystem.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (RANGER-4882) update dependent library version: fasterxml.jackson, jersey
[ https://issues.apache.org/jira/browse/RANGER-4882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Madhan Neethiraj updated RANGER-4882: - Attachment: RANGER-4882.patch > update dependent library version: fasterxml.jackson, jersey > --- > > Key: RANGER-4882 > URL: https://issues.apache.org/jira/browse/RANGER-4882 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Affects Versions: 3.0.0 >Reporter: Madhan Neethiraj >Assignee: Madhan Neethiraj >Priority: Major > Fix For: 3.0.0 > > Attachments: RANGER-4882.patch > > > Update version of following dependent libraries to the latest available > version: > * fasterxml.jackson: 2.17.0 => 2.17.2 > * jersey: 1.19.3 => 1.19.4 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (RANGER-4882) update dependent library version: fasterxml.jackson, jersey
Madhan Neethiraj created RANGER-4882: Summary: update dependent library version: fasterxml.jackson, jersey Key: RANGER-4882 URL: https://issues.apache.org/jira/browse/RANGER-4882 Project: Ranger Issue Type: Improvement Components: Ranger Reporter: Madhan Neethiraj Assignee: Madhan Neethiraj Update version of following dependent libraries to the latest available version: * fasterxml.jackson: 2.17.0 => 2.17.2 * jersey: 1.19.3 => 1.19.4 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (RANGER-4874) Ranger UI inaccessible after login
[ https://issues.apache.org/jira/browse/RANGER-4874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Madhan Neethiraj reopened RANGER-4874: -- > Ranger UI inaccessible after login > --- > > Key: RANGER-4874 > URL: https://issues.apache.org/jira/browse/RANGER-4874 > Project: Ranger > Issue Type: Bug > Components: Ranger >Affects Versions: 2.5.0 >Reporter: Abhishek Kumar >Assignee: Madhan Neethiraj >Priority: Critical > Fix For: 3.0.0, 2.5.0 > > Attachments: RANGER-4874-ranger-2.5.patch, RANGER-4874.patch, > Screenshot 2024-07-26 at 7.52.48 PM.png, Screenshot 2024-07-26 at 9.22.44 > PM.png, catalina_error.out, ranger_error.log > > > Ranger UI is not accessible after login although login is successful. Errors > seen in catalina.out and ranger.log. > This is observed in docker env for ranger. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-4561) Adding the mechanism to eanble/disable Ranager Access logs based on property
[ https://issues.apache.org/jira/browse/RANGER-4561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Madhan Neethiraj updated RANGER-4561: - Fix Version/s: 2.5.0 > Adding the mechanism to eanble/disable Ranager Access logs based on property > - > > Key: RANGER-4561 > URL: https://issues.apache.org/jira/browse/RANGER-4561 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Reporter: Ramachandran >Assignee: Ramachandran >Priority: Major > Fix For: 3.0.0, 2.5.0 > > Attachments: > 0001-RANGER-4561-Adding-the-mechanism-to-eanble-disable-R.patch > > > In the current Ranger Admin, we have enabled Ranger access logs by default. > If any of the customers wants to disable, the Ranger access logs, it can not > be done without making code changes.So we need to leverage this use case so > that customers can disable the ranger access logs if they needed via setting > properties -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-4873) update zookeeper version from 3.5.5 to 3.9.2
[ https://issues.apache.org/jira/browse/RANGER-4873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Madhan Neethiraj updated RANGER-4873: - Fix Version/s: 2.5.0 > update zookeeper version from 3.5.5 to 3.9.2 > > > Key: RANGER-4873 > URL: https://issues.apache.org/jira/browse/RANGER-4873 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Reporter: Madhan Neethiraj >Assignee: Krzysztof Sobolewski >Priority: Major > Fix For: 3.0.0, 2.5.0 > > Time Spent: 10m > Remaining Estimate: 0h > > This Jira tracks [pull request > #359|https://github.com/apache/ranger/pull/359]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-4874) Ranger UI inaccessible after login
[ https://issues.apache.org/jira/browse/RANGER-4874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Madhan Neethiraj updated RANGER-4874: - Attachment: RANGER-4874-ranger-2.5.patch > Ranger UI inaccessible after login > --- > > Key: RANGER-4874 > URL: https://issues.apache.org/jira/browse/RANGER-4874 > Project: Ranger > Issue Type: Bug > Components: Ranger >Affects Versions: 2.5.0 >Reporter: Abhishek Kumar >Assignee: Madhan Neethiraj >Priority: Critical > Fix For: 3.0.0, 2.5.0 > > Attachments: RANGER-4874-ranger-2.5.patch, RANGER-4874.patch, > Screenshot 2024-07-26 at 7.52.48 PM.png, Screenshot 2024-07-26 at 9.22.44 > PM.png, catalina_error.out, ranger_error.log > > > Ranger UI is not accessible after login although login is successful. Errors > seen in catalina.out and ranger.log. > This is observed in docker env for ranger. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-4874) Ranger UI inaccessible after login
[ https://issues.apache.org/jira/browse/RANGER-4874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Madhan Neethiraj updated RANGER-4874: - Attachment: RANGER-4874.patch > Ranger UI inaccessible after login > --- > > Key: RANGER-4874 > URL: https://issues.apache.org/jira/browse/RANGER-4874 > Project: Ranger > Issue Type: Bug > Components: Ranger >Affects Versions: 2.5.0 >Reporter: Abhishek Kumar >Assignee: Madhan Neethiraj >Priority: Critical > Fix For: 3.0.0, 2.5.0 > > Attachments: RANGER-4874.patch, Screenshot 2024-07-26 at 7.52.48 > PM.png, Screenshot 2024-07-26 at 9.22.44 PM.png, catalina_error.out, > ranger_error.log > > > Ranger UI is not accessible after login although login is successful. Errors > seen in catalina.out and ranger.log. > This is observed in docker env for ranger. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (RANGER-4874) Ranger UI inaccessible after login
[ https://issues.apache.org/jira/browse/RANGER-4874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Madhan Neethiraj reassigned RANGER-4874: Assignee: Madhan Neethiraj > Ranger UI inaccessible after login > --- > > Key: RANGER-4874 > URL: https://issues.apache.org/jira/browse/RANGER-4874 > Project: Ranger > Issue Type: Bug > Components: Ranger >Affects Versions: 2.5.0 >Reporter: Abhishek Kumar >Assignee: Madhan Neethiraj >Priority: Critical > Attachments: Screenshot 2024-07-26 at 7.52.48 PM.png, Screenshot > 2024-07-26 at 9.22.44 PM.png, catalina_error.out, ranger_error.log > > > Ranger UI is not accessible after login although login is successful. Errors > seen in catalina.out and ranger.log. > This is observed in docker env for ranger. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (RANGER-4873) update zookeeper version from 3.5.5 to 3.9.2
[
https://issues.apache.org/jira/browse/RANGER-4873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17869183#comment-17869183
]
Madhan Neethiraj commented on RANGER-4873:
--
{quote}Have you validated the ranger-kms functionality? as Zookeeper 3.9.2
supports upgraded curator version
{quote}
[~bpatel] - can you please add details of the functionality you are concerned
about? CRUD operations on keys, including rollover, were validated.
> update zookeeper version from 3.5.5 to 3.9.2
>
>
> Key: RANGER-4873
> URL: https://issues.apache.org/jira/browse/RANGER-4873
> Project: Ranger
> Issue Type: Improvement
> Components: Ranger
>Reporter: Madhan Neethiraj
>Assignee: Krzysztof Sobolewski
>Priority: Major
> Fix For: 3.0.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This Jira tracks [pull request
> #359|https://github.com/apache/ranger/pull/359].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (RANGER-4874) Ranger UI inaccessible after login
[
https://issues.apache.org/jira/browse/RANGER-4874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17869127#comment-17869127
]
Madhan Neethiraj commented on RANGER-4874:
--
[~abhi_2110] - here is the failure seen in [^ranger_error.log]:
{noformat}
2024-07-27 02:39:13,754 [localhost-startStop-1] ERROR
[EmbeddedServiceDefsUtil.java:178] EmbeddedServiceDefsUtil.init(): failed
java.lang.AbstractMethodError:
javax.ws.rs.core.Response.getStatusInfo()Ljavax/ws/rs/core/Response$StatusType;
at
javax.ws.rs.WebApplicationException.computeExceptionMessage(WebApplicationException.java:188)
at
javax.ws.rs.WebApplicationException.(WebApplicationException.java:162)
{noformat}
Looking in docker container built from ranger-2.5 branch
{{javax.ws.rs.core.Response}} class is present in multiple libraries under
/opt/ranger/admin:
{noformat}
cd /opt/ranger/admin
$ find /opt/ranger/admin/ -name "*.jar" -exec fgrep -l
javax/ws/rs/core/Response {} \;
/opt/ranger/admin/ews/webapp/WEB-INF/lib/jersey-bundle-1.19.3.jar
/opt/ranger/admin/ews/webapp/WEB-INF/lib/javax.ws.rs-api-2.1.1.jar
/opt/ranger/admin/ews/webapp/WEB-INF/lib/jsr311-api-1.1.1.jar {noformat}
[jsr311-api-1.1.1.jar|https://mvnrepository.com/artifact/javax.ws.rs/jsr311-api]
is from 2009, which was replaced by
[javax.ws.rs-api|https://mvnrepository.com/artifact/javax.ws.rs/javax.ws.rs-api].
Ranger pom files need to be reviewed to remove/exclude jsr311-api-1.1.1
dependency.
[~pradeep] and [~mugdha.varadkar] - I remember you reporting this issue few
days back. Can you please confirm if your environment has multiple libraries
having class {{javax.ws.rs.core.Response}}.
> Ranger UI inaccessible after login
> ---
>
> Key: RANGER-4874
> URL: https://issues.apache.org/jira/browse/RANGER-4874
> Project: Ranger
> Issue Type: Bug
> Components: Ranger
>Affects Versions: 2.5.0
>Reporter: Abhishek Kumar
>Priority: Critical
> Attachments: Screenshot 2024-07-26 at 7.52.48 PM.png, Screenshot
> 2024-07-26 at 9.22.44 PM.png, catalina_error.out, ranger_error.log
>
>
> Ranger UI is not accessible after login although login is successful. Errors
> seen in catalina.out and ranger.log.
> This is observed in docker env for ranger.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (RANGER-4819) Proposal to Upgrade All React.js Dependent Libraries
[ https://issues.apache.org/jira/browse/RANGER-4819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Madhan Neethiraj updated RANGER-4819: - Issue Type: Improvement (was: Bug) > Proposal to Upgrade All React.js Dependent Libraries > - > > Key: RANGER-4819 > URL: https://issues.apache.org/jira/browse/RANGER-4819 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Reporter: Dhaval Rajpara >Assignee: Dhaval Rajpara >Priority: Major > Labels: ranger-react > Fix For: 2.5.0 > > Attachments: > 0001-RANGER-4819-Proposal-to-Upgrade-All-React.js-Depende.patch, > 0001-RANGER-4819-ranger-2.5.patch > > > Upgrading all dependent libraries for React.js in our project. This will > ensure we are using the latest versions, improving security, performance, and > compatibility with new features. > # babel/traverse > # axios > # braces > # follow-redirects > # json5 > # loader-utils > # minimist > # moment > # terser > # webpack > # webpack-dev-middleware -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-4604) Need to add query param createdBy for security-zone GET API
[ https://issues.apache.org/jira/browse/RANGER-4604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Madhan Neethiraj updated RANGER-4604: - Issue Type: Improvement (was: Bug) > Need to add query param createdBy for security-zone GET API > --- > > Key: RANGER-4604 > URL: https://issues.apache.org/jira/browse/RANGER-4604 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Reporter: Prashant Satam >Assignee: Prashant Satam >Priority: Major > Fix For: 3.0.0, 2.5.0 > > > We need to add a query param createdBy for security-zone GET API. It will be > useful to filter list of security-zones created by user, especially to get > the security-zone created by self -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-4618) Need to add displayName field in zoneSummary Object
[ https://issues.apache.org/jira/browse/RANGER-4618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Madhan Neethiraj updated RANGER-4618: - Issue Type: Improvement (was: Bug) > Need to add displayName field in zoneSummary Object > --- > > Key: RANGER-4618 > URL: https://issues.apache.org/jira/browse/RANGER-4618 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Reporter: Prashant Satam >Assignee: Prashant Satam >Priority: Major > Fix For: 3.0.0, 2.5.0 > > > In the zoneSummary Object services section we have name field to display > serviceName but it will be helpful if we add displayName field also to the > same -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-4354) Improve ChangePassword utility for multiple default password change request
[ https://issues.apache.org/jira/browse/RANGER-4354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Madhan Neethiraj updated RANGER-4354: - Issue Type: Improvement (was: Bug) > Improve ChangePassword utility for multiple default password change request > --- > > Key: RANGER-4354 > URL: https://issues.apache.org/jira/browse/RANGER-4354 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Affects Versions: 3.0.0, 2.4.0 >Reporter: Pradeep Agrawal >Assignee: Pradeep Agrawal >Priority: Major > Fix For: 3.0.0, 2.5.0 > > Attachments: > 0001-RANGER-4354-Improve-ChangePassword-utility-for-multi.patch > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-4248) Remove unused conf files for solr audit setup
[ https://issues.apache.org/jira/browse/RANGER-4248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Madhan Neethiraj updated RANGER-4248: - Issue Type: Improvement (was: Bug) > Remove unused conf files for solr audit setup > - > > Key: RANGER-4248 > URL: https://issues.apache.org/jira/browse/RANGER-4248 > Project: Ranger > Issue Type: Improvement > Components: audit >Affects Versions: 2.4.0 >Reporter: Abhishek Kumar >Assignee: Abhishek Kumar >Priority: Minor > Fix For: 3.0.0, 2.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > There is no content in these files and can be safely removed: > 1. security-admin/contrib/solr_for_audit_setup/conf/admin-extra.html > 2. > security-admin/contrib/solr_for_audit_setup/conf/admin-extra.menu-bottom.html > 3. security-admin/contrib/solr_for_audit_setup/conf/admin-extra.menu-top.html > This one was probably used for testing purposes and can be removed as well: > 1. security-admin/contrib/solr_for_audit_setup/conf/elevate.xml -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-4128) serviceName, if not specified in the resource, should be taken from the ServiceTags.serviceName
[
https://issues.apache.org/jira/browse/RANGER-4128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Madhan Neethiraj updated RANGER-4128:
-
Issue Type: Improvement (was: Bug)
> serviceName, if not specified in the resource, should be taken from the
> ServiceTags.serviceName
> ---
>
> Key: RANGER-4128
> URL: https://issues.apache.org/jira/browse/RANGER-4128
> Project: Ranger
> Issue Type: Improvement
> Components: Ranger
>Reporter: Fateh Singh
>Assignee: Siddhant Sontakke
>Priority: Major
> Fix For: 3.0.0, 2.4.1, 2.5.0
>
> Attachments: Screenshot 2023-04-14 at 1.20.51 PM.png, Screenshot
> 2023-04-14 at 1.41.48 PM.png, image-2023-04-14-12-36-34-803.png,
> image-2023-04-14-12-36-53-668.png, image-2023-04-14-12-40-36-377.png,
> image-2023-04-14-12-59-13-128.png, image-2023-04-14-13-01-11-416.png,
> image-2023-04-14-13-22-07-588.png, image-2023-04-14-13-43-58-983.png,
> image-2023-04-14-19-20-08-056.png, image-2023-04-14-19-21-09-315.png,
> image-2023-04-14-19-22-01-270.png, image-2023-04-14-19-22-28-641.png
>
>
> Current scenario-
> REST endpoint: "tags/importservicetags"
> Client: ranger python client (ranger_client.import_service_tags)
> Scenario: Above endpoint called multiple times with different tags but same
> set of resources gives the below error:
> {code:java}
> PUT service/public/v2/api/service/dev_hive/tags failed: expected_status=204,
> status=400, message=b'Exception [EclipseLink-4002] (Eclipse Persistence
> Services - 2.7.12.v20230209-e5c4074ef3):
> org.eclipse.persistence.exceptions.DatabaseException\nInternal Exception:
> org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique
> constraint "x_service_resource_idx_svc_id_resource_signature"\n Detail: Key
> (service_id, resource_signature)=(4,
> 688974a2b40b6536631f952c66b065ad31c8c1588bfa658953a6218ef503d38e) already
> exists.\nError Code: 0\nCall: INSERT INTO x_service_resource (id,
> ADDED_BY_ID, CREATE_TIME, guid, is_enabled, resource_signature, service_id,
> service_resource_elements_text, tags_text, UPDATE_TIME, UPD_BY_ID, version)
> VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\n\tbind => [12 parameters bound]'
> {code}
> How a serviceResource in the request look like to reproduce above
> scenario/error:
> {code:java}
> {
> "resourceElements": {
> "database": {
> ...
> },
> "column": {
> ...
> },
> "table": {
> ...
> }
> },
> "resourceSignature":
> "40c20f3a1909b0958b61451499e9a58e9ece1661f82072388f39f9685996c0dc",
> "id": 1,
> "isEnabled": true,
> "version": 2
> } {code}
> Found bug and workaround:
> serviceName, if not specified in the resource, should be taken from the
> ServiceTags.serviceName
> How a serviceResource should look like to fix above bug:
> {code:java}
> {
> "serviceName":"dev_hive",
> "resourceElements": {
> "database": {
> ...
> },
> "column": {
> ...
> },
> "table": {
> ...
> }
> },
> "resourceSignature":
> "40c20f3a1909b0958b61451499e9a58e9ece1661f82072388f39f9685996c0dc",
> "id": 1,
> "isEnabled": true,
> "version": 2
> } {code}
> Here, dev_hive is the serviceName for which service tags are being imported
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
