Re: No usersync
Hi Marc, If rangerusersync Unix user exists, then you may want to modify the unix_user and unix_group in install.properties to adapt to your case. Now for Kerberos, this applies only if you are in a kerberised environment. Otherwise you leave this blank. Best regards, Loïc Le mar. 11 juin 2024 à 14:28, Marc Hoppins a écrit : > Hi all, > > > > Sorry to appear dumb but am still trying to get this working. > > > > Why does install.properties have: > > > > #User and group for the usersync process > > unix_user=ranger > > unix_group=ranger > > > > If the rangerusersync ID exists? > > > > #change password of rangerusersync user. Please note that this password > should be as per rangerusersync user in ranger > > rangerUsersync_password= > > > > and KERBEROS > > > > #Set to run in kerberos environment > > usersync_principal= > > usersync_keytab= > > hadoop_conf=/etc/hadoop/conf > > > > What does HADOOP Kerberos have to do with LDAP – our LDAP is active > directory. > > > > Marc Hoppins >
Re: LDAP authentication issue
Vipin, Sailaja,
Following-up on this issue : is there a workaround I could use, or do I
have to wait for a fix ?
Thanks,
Loïc CHANEL
Technical leader Big Data
Capgemini (Lyon, France)
Le ven. 19 avr. 2024 à 09:50, Loïc CHANEL a
écrit :
> Hi guys,
>
> Thanks for your answers ! I created RANGER-4778
> <https://issues.apache.org/jira/browse/RANGER-4778> to track that issue.
> Now for the workaround, I'm not sure I can make it work. Here's my
> situation.
>
> The users I'm trying to authenticate are objectClass=person, and they are
> located :
> - In OU=External,OU=Company,OU=Users,DC=cmb,DC=blabla,DC=org if they are
> external users
> - In OU=Internal,OU=Company,OU=Users,DC=cmb,DC=blabla,DC=org if they are
> internal users
>
> Now the issue is that none of this is reflected in the 'memberOf'
> section. But there is a filter I know I can use for these users, it's
> memberOf=CN=usr_tool_prod,OU=Tool,OU=Groupes,DC=blabla,DC=org, so I tried
> the following :
> ranger.ldap.user.dnpattern
> DC=cmb,DC=blabla,DC=org
>
> ranger.ldap.group.searchfilter
>
> ((objectClass=person)(memberOf=CN=usr_tool_prod,OU=Tool,OU=Groupes,DC=blabla,DC=org))
>
> And it's still KO. What am I missing ? Is there a way I can make it work ?
> Thanks,
>
>
> Loïc CHANEL
> Technical leader Big Data
> Capgemini (Lyon, France)
>
>
> Le ven. 19 avr. 2024 à 05:08, Vipin Rathor a écrit :
>
>> Thank Sailaja for the reply. I was about to reply that
>> Spring LDAP API does support multiple DN but you
>> covered with complete analysis.
>> IIRC I did fix (or circumvent?) this bug back in the days
>> and I’m surprised to see that it still lingers around
>> Let’s fix it this time for good
>>
>> Regards,
>> VR
>>
>> On Apr 19, 2024, at 03:45, Sailaja Polavarapu
>> wrote:
>>
>>
>> ranger.ldap.user.dnpattern currently takes only one pattern which seems
>> to be a bug. Because the underlying spring security ldap library API
>> supports array of patterns. For now, can you try filtering based on any
>> other attributes? For example below config filters the users from group1 &
>> group2
>> ranger.ldap.group.searchbase=OU=groups,DC=rangerdev,DC=apache,DC=COM,
>> ranger.ldap.group.searchfilter=cn*, ranger.ldap.group.roleattribute=cn,
>> ranger.ldap.user.searchfilter=(&(objectclass=user)(memberOf=CN=group1,OU=groups,DC=rangerdev,DC=apache,DC=COM)(memberOf=CN=group2,OU=groups,DC=rangerdev,DC=apache,DC=COM)(samaccountname={0})),
>> ranger.ldap.user.dnpattern=dc=example,dc=com,
>> ranger.ldap.base.dn=cn=users,DC=rangerdev,DC=apache,DC=COM
>>
>> For Ldap authentication, when DN is not provided correctly, then we
>> search the users with the user search filter and retrieve the DN of the
>> user. After that we search the groups based on group search base, group
>> search filter, and the role attribute and try to match the retrieved DN of
>> the user in the list of retrieved group attributes. Hope this helps.
>>
>> For user dn pattern config to take multiple values, please file an apache
>> jira so that it can be fixed for later versions.
>>
>> - Sailaja.
>>
>> On Thu, Apr 18, 2024 at 9:25 AM j km wrote:
>>
>>> This is really old, but this is what I found.
>>>
>>> "ranger.usersync.ldap.user.searchfilter":
>>> "(|(samaaccountname=)(memberof=CN=)(...))
>>>
>>> On Thu, Apr 18, 2024 at 10:54 AM Loïc CHANEL <
>>> loic.cha...@telecomnancy.net> wrote:
>>>
>>>> That's what I was hoping indeed, but unfortunately :
>>>>
>>>> 2024-04-18 14:39:39,400 [http-nio-6080-exec-7] DEBUG
>>>> [RangerAuthenticationProvider.java:291] LDAP Authentication Failed:
>>>> org.springframework.security.authentication.InternalAuthenticationServiceException:
>>>> Failed to parse DN; nested exception is
>>>> org.springframework.ldap.core.TokenMgrError: Lexical error at line 1,
>>>> column 10. Encountered: "(" (40), after : ""
>>>>
>>>> I tried to escape the character with a backslash but got the exact same
>>>> result. Any other ideas ?
>>>> Best regards,
>>>>
>>>>
>>>> Loïc CHANEL
>>>> Technical leader Big Data
>>>> Capgemini (Lyon, France)
>>>>
>>>>
>>>> Le jeu. 18 avr. 2024 à 16:24, j km a écrit :
>>>>
>>>>> I think it is just an or
>>>>> & (&(filter1) (filter2)) AND — all conditions must be met
&
Re: LDAP authentication issue
Hi guys,
Thanks for your answers ! I created RANGER-4778
<https://issues.apache.org/jira/browse/RANGER-4778> to track that issue.
Now for the workaround, I'm not sure I can make it work. Here's my
situation.
The users I'm trying to authenticate are objectClass=person, and they are
located :
- In OU=External,OU=Company,OU=Users,DC=cmb,DC=blabla,DC=org if they are
external users
- In OU=Internal,OU=Company,OU=Users,DC=cmb,DC=blabla,DC=org if they are
internal users
Now the issue is that none of this is reflected in the 'memberOf' section.
But there is a filter I know I can use for these users, it's
memberOf=CN=usr_tool_prod,OU=Tool,OU=Groupes,DC=blabla,DC=org, so I tried
the following :
ranger.ldap.user.dnpattern
DC=cmb,DC=blabla,DC=org
ranger.ldap.group.searchfilter
((objectClass=person)(memberOf=CN=usr_tool_prod,OU=Tool,OU=Groupes,DC=blabla,DC=org))
And it's still KO. What am I missing ? Is there a way I can make it work ?
Thanks,
Loïc CHANEL
Technical leader Big Data
Capgemini (Lyon, France)
Le ven. 19 avr. 2024 à 05:08, Vipin Rathor a écrit :
> Thank Sailaja for the reply. I was about to reply that
> Spring LDAP API does support multiple DN but you
> covered with complete analysis.
> IIRC I did fix (or circumvent?) this bug back in the days
> and I’m surprised to see that it still lingers around
> Let’s fix it this time for good
>
> Regards,
> VR
>
> On Apr 19, 2024, at 03:45, Sailaja Polavarapu
> wrote:
>
>
> ranger.ldap.user.dnpattern currently takes only one pattern which seems to
> be a bug. Because the underlying spring security ldap library API supports
> array of patterns. For now, can you try filtering based on any other
> attributes? For example below config filters the users from group1 & group2
> ranger.ldap.group.searchbase=OU=groups,DC=rangerdev,DC=apache,DC=COM,
> ranger.ldap.group.searchfilter=cn*, ranger.ldap.group.roleattribute=cn,
> ranger.ldap.user.searchfilter=(&(objectclass=user)(memberOf=CN=group1,OU=groups,DC=rangerdev,DC=apache,DC=COM)(memberOf=CN=group2,OU=groups,DC=rangerdev,DC=apache,DC=COM)(samaccountname={0})),
> ranger.ldap.user.dnpattern=dc=example,dc=com,
> ranger.ldap.base.dn=cn=users,DC=rangerdev,DC=apache,DC=COM
>
> For Ldap authentication, when DN is not provided correctly, then we search
> the users with the user search filter and retrieve the DN of the user.
> After that we search the groups based on group search base, group search
> filter, and the role attribute and try to match the retrieved DN of the
> user in the list of retrieved group attributes. Hope this helps.
>
> For user dn pattern config to take multiple values, please file an apache
> jira so that it can be fixed for later versions.
>
> - Sailaja.
>
> On Thu, Apr 18, 2024 at 9:25 AM j km wrote:
>
>> This is really old, but this is what I found.
>>
>> "ranger.usersync.ldap.user.searchfilter":
>> "(|(samaaccountname=)(memberof=CN=)(...))
>>
>> On Thu, Apr 18, 2024 at 10:54 AM Loïc CHANEL <
>> loic.cha...@telecomnancy.net> wrote:
>>
>>> That's what I was hoping indeed, but unfortunately :
>>>
>>> 2024-04-18 14:39:39,400 [http-nio-6080-exec-7] DEBUG
>>> [RangerAuthenticationProvider.java:291] LDAP Authentication Failed:
>>> org.springframework.security.authentication.InternalAuthenticationServiceException:
>>> Failed to parse DN; nested exception is
>>> org.springframework.ldap.core.TokenMgrError: Lexical error at line 1,
>>> column 10. Encountered: "(" (40), after : ""
>>>
>>> I tried to escape the character with a backslash but got the exact same
>>> result. Any other ideas ?
>>> Best regards,
>>>
>>>
>>> Loïc CHANEL
>>> Technical leader Big Data
>>> Capgemini (Lyon, France)
>>>
>>>
>>> Le jeu. 18 avr. 2024 à 16:24, j km a écrit :
>>>
>>>> I think it is just an or
>>>> & (&(filter1) (filter2)) AND — all conditions must be met
>>>> | (|(filter1) (filter2)) OR — any number of conditions can be met
>>>> ! (!(filter1)) NOT — the condition must not be met
>>>>
>>>> On Thu, Apr 18, 2024 at 10:06 AM Loïc CHANEL <
>>>> loic.cha...@telecomnancy.net> wrote:
>>>>
>>>>> Just a follow-up question though : is there a way to define several
>>>>> dnpattern
>>>>> values ? Because the users are located into two different sections of
>>>>> my LDAP, so I have the following patterns :
>>>>> CN={0},OU=External,OU=Users,DC=cmb,DC=blabla,DC=org and
>>>>> CN={0},OU
Re: LDAP authentication issue
That's what I was hoping indeed, but unfortunately :
2024-04-18 14:39:39,400 [http-nio-6080-exec-7] DEBUG
[RangerAuthenticationProvider.java:291] LDAP Authentication Failed:
org.springframework.security.authentication.InternalAuthenticationServiceException:
Failed to parse DN; nested exception is
org.springframework.ldap.core.TokenMgrError: Lexical error at line 1,
column 10. Encountered: "(" (40), after : ""
I tried to escape the character with a backslash but got the exact same
result. Any other ideas ?
Best regards,
Loïc CHANEL
Technical leader Big Data
Capgemini (Lyon, France)
Le jeu. 18 avr. 2024 à 16:24, j km a écrit :
> I think it is just an or
> & (&(filter1) (filter2)) AND — all conditions must be met
> | (|(filter1) (filter2)) OR — any number of conditions can be met
> ! (!(filter1)) NOT — the condition must not be met
>
> On Thu, Apr 18, 2024 at 10:06 AM Loïc CHANEL
> wrote:
>
>> Just a follow-up question though : is there a way to define several dnpattern
>> values ? Because the users are located into two different sections of my
>> LDAP, so I have the following patterns :
>> CN={0},OU=External,OU=Users,DC=cmb,DC=blabla,DC=org and
>> CN={0},OU=Internal,OU=Users,DC=cmb,DC=blabla,DC=org and I want LDAP
>> authentication to work for both.
>> Best regards,
>>
>>
>> Loïc CHANEL
>> Technical leader Big Data
>> Capgemini (Lyon, France)
>>
>>
>> Le jeu. 18 avr. 2024 à 13:39, Loïc CHANEL
>> a écrit :
>>
>>> Hi Vipin,
>>>
>>> Already did this but didn't see the file name as I was expecting them to
>>> be before the properties but not after.
>>> By finding the loaded file in the logs I've been able to troubleshoot my
>>> issue.
>>> Thanks a lot for your help,
>>>
>>>
>>> Loïc CHANEL
>>> Technical leader Big Data
>>> Capgemini (Lyon, France)
>>>
>>>
>>> Le jeu. 18 avr. 2024 à 13:11, Vipin Rathor a
>>> écrit :
>>>
>>>> Hi Loïc,
>>>> If you turn on log4j debug for Apache Ranger, then the debug log will
>>>> tell you where the congratulations are being loaded from.
>>>>
>>>> Also, please check if you aren’t making changes in a different copy of
>>>> the actual config file (most probably you might have checked but wouldn’t
>>>> hurt to double check).
>>>>
>>>> Regards,
>>>> VR
>>>>
>>>> On Apr 18, 2024, at 14:43, Loïc CHANEL
>>>> wrote:
>>>>
>>>>
>>>> Hi everyone,
>>>>
>>>> I'm trying to configure the LDAP authentication for UI access, so I
>>>> edited the properties in ranger-admin-site.xml file and restarted, but
>>>> I saw in the logs the properties I edited are not applied. For instance,
>>>> the property ranger.ldap.url has the value ldap://cmb.mydomain.org:389
>>>> in the XML file, but when Ranger starts I can see in the logs that the
>>>> default value ( ldap:// ) is loaded by Ranger instead of what I
>>>> defined.
>>>> Is there something I'm missing ? How can I see where the values are
>>>> loaded from ?
>>>> Thanks for your help,
>>>>
>>>>
>>>> Loïc CHANEL
>>>> Technical leader Big Data
>>>> Capgemini (Lyon, France)
>>>>
>>>>
Re: LDAP authentication issue
Just a follow-up question though : is there a way to define several dnpattern
values ? Because the users are located into two different sections of my
LDAP, so I have the following patterns :
CN={0},OU=External,OU=Users,DC=cmb,DC=blabla,DC=org and
CN={0},OU=Internal,OU=Users,DC=cmb,DC=blabla,DC=org and I want LDAP
authentication to work for both.
Best regards,
Loïc CHANEL
Technical leader Big Data
Capgemini (Lyon, France)
Le jeu. 18 avr. 2024 à 13:39, Loïc CHANEL a
écrit :
> Hi Vipin,
>
> Already did this but didn't see the file name as I was expecting them to
> be before the properties but not after.
> By finding the loaded file in the logs I've been able to troubleshoot my
> issue.
> Thanks a lot for your help,
>
>
> Loïc CHANEL
> Technical leader Big Data
> Capgemini (Lyon, France)
>
>
> Le jeu. 18 avr. 2024 à 13:11, Vipin Rathor a écrit :
>
>> Hi Loïc,
>> If you turn on log4j debug for Apache Ranger, then the debug log will
>> tell you where the congratulations are being loaded from.
>>
>> Also, please check if you aren’t making changes in a different copy of
>> the actual config file (most probably you might have checked but wouldn’t
>> hurt to double check).
>>
>> Regards,
>> VR
>>
>> On Apr 18, 2024, at 14:43, Loïc CHANEL
>> wrote:
>>
>>
>> Hi everyone,
>>
>> I'm trying to configure the LDAP authentication for UI access, so I
>> edited the properties in ranger-admin-site.xml file and restarted, but I
>> saw in the logs the properties I edited are not applied. For instance, the
>> property ranger.ldap.url has the value ldap://cmb.mydomain.org:389 in
>> the XML file, but when Ranger starts I can see in the logs that the default
>> value ( ldap:// ) is loaded by Ranger instead of what I defined.
>> Is there something I'm missing ? How can I see where the values are
>> loaded from ?
>> Thanks for your help,
>>
>>
>> Loïc CHANEL
>> Technical leader Big Data
>> Capgemini (Lyon, France)
>>
>>
Re: LDAP authentication issue
Hi Vipin, Already did this but didn't see the file name as I was expecting them to be before the properties but not after. By finding the loaded file in the logs I've been able to troubleshoot my issue. Thanks a lot for your help, Loïc CHANEL Technical leader Big Data Capgemini (Lyon, France) Le jeu. 18 avr. 2024 à 13:11, Vipin Rathor a écrit : > Hi Loïc, > If you turn on log4j debug for Apache Ranger, then the debug log will tell > you where the congratulations are being loaded from. > > Also, please check if you aren’t making changes in a different copy of the > actual config file (most probably you might have checked but wouldn’t hurt > to double check). > > Regards, > VR > > On Apr 18, 2024, at 14:43, Loïc CHANEL > wrote: > > > Hi everyone, > > I'm trying to configure the LDAP authentication for UI access, so I edited > the properties in ranger-admin-site.xml file and restarted, but I saw in > the logs the properties I edited are not applied. For instance, the > property ranger.ldap.url has the value ldap://cmb.mydomain.org:389 in the > XML file, but when Ranger starts I can see in the logs that the default > value ( ldap:// ) is loaded by Ranger instead of what I defined. > Is there something I'm missing ? How can I see where the values are loaded > from ? > Thanks for your help, > > > Loïc CHANEL > Technical leader Big Data > Capgemini (Lyon, France) > >
LDAP authentication issue
Hi everyone, I'm trying to configure the LDAP authentication for UI access, so I edited the properties in ranger-admin-site.xml file and restarted, but I saw in the logs the properties I edited are not applied. For instance, the property ranger.ldap.url has the value ldap://cmb.mydomain.org:389 in the XML file, but when Ranger starts I can see in the logs that the default value ( ldap:// ) is loaded by Ranger instead of what I defined. Is there something I'm missing ? How can I see where the values are loaded from ? Thanks for your help, Loïc CHANEL Technical leader Big Data Capgemini (Lyon, France)
Re: Groups not retrieved
And now it works perfectly. Thanks ! I'm curious about that option : could you provide more details ? Why does it trigger the usage of SYNC_LDAP_USER_GROUP_NAME_ATTRIBUTE parameter to do exactly what I was trying to achieve ? And what was the previous behaviour ? Thanks a lot for your help, Loïc Le ven. 22 mars 2024 à 15:34, Sailaja Polavarapu a écrit : > Oh ok. In this case can you try setting > ranger.usersync.group.searchenabled to false? > > On Fri, Mar 22, 2024 at 1:27 AM Loïc CHANEL > wrote: > >> Hi Sailaja, >> >> Actually, the groups are not stored in the LDAP I'm querying (or at least >> I can't access them), so I'm retrieving the groups using >> the SYNC_LDAP_USER_GROUP_NAME_ATTRIBUTE parameter, which I configured but >> doesn't seem to work as I expected. >> As a matter of fact, I'm successfully retrieving users from the LDAP with >> a postOfficeBox field, but setting SYNC_LDAP_USER_GROUP_NAME_ATTRIBUTE = >> postOfficeBox does not retrieve the value of the field to create a group. >> >> Let me give you an example to clarify. From the LDAP I'm retrieving the >> following user : >> >> sn: DOE >> postOfficeBox: 9001928 >> givenName: JOHN >> displayName: DOE JOHN >> memberOf: CN=usr_tool_prd,OU=Tool,OU=Groupes,DC=blabla,DC=org >> name: FOO123 >> mail: john@blabla.com >> >> >> The field I'm really interested in for group purposes is postOfficeBox. >> So by setting SYNC_LDAP_USER_GROUP_NAME_ATTRIBUTE = postOfficeBox I expect >> Usersync to create a group named "9001928" and add John Doe to that group, >> but it doesn't work. Does Usersync only expect groups with LDAP structure >> (like the memberOf line) ? >> Thanks, >> >> >> Loïc >> >> Le jeu. 21 mars 2024 à 22:51, Sailaja Polavarapu < >> spolavar...@cloudera.com> a écrit : >> >>> Hi Loic, >>> I see that you have below config properties for group search. In this >>> case the groups are retrieved from "dc=cmb,dc=blabla,dc=org" search base. >>> Can you check if "CN=usr_tool_prd,OU=Tool,OU=Groupes,DC=blabla,DC=org" >>> group is under the configured search base? >>> groupSearchEnabled: true, groupSearchBase: [dc=cmb,dc=blabla,dc=org], >>> groupSearchScope: 2, groupObjectClass: groupofnames, >>> May be if you provide usersync logs, that can help to analyze further >>> >>> Thanks, >>> Sailaja. >>> >>> On Thu, Mar 21, 2024 at 8:00 AM Loïc CHANEL < >>> loic.cha...@telecomnancy.net> wrote: >>> >>>> Hi team, >>>> Am I the only one experiencing this issue ? >>>> Thanks, >>>> >>>> Loïc >>>> >>>> >>>> Le lun. 19 févr. 2024 à 12:38, Loïc CHANEL < >>>> loic.cha...@telecomnancy.net> a écrit : >>>> >>>>> Hi guys, >>>>> >>>>> Since 2.4, LDAP information retrieval to create groups seems broken. >>>>> My sync issues are solved for users, but I'm still unable to pull groups >>>>> from LDAP. For instance, here are the information in the LDAP from my >>>>> user : >>>>> sn: CHANEL >>>>> postOfficeBox: someValue >>>>> givenName: LOIC >>>>> displayName: CHANEL LOIC >>>>> memberOf: CN=usr_tool_prd,OU=Tool,OU=Groupes,DC=blabla,DC=org >>>>> name: LCH657 >>>>> mail: loic.cha...@telecomnancy.net >>>>> >>>>> Now here is my configuration on Ranger side : >>>>> >>>>> >>>>> ranger.usersync.ldap.user.groupnameattribute >>>>> postOfficeBox,memberOf >>>>> >>>>> >>>>> And I can even see that the retrieval is going that way : >>>>> 9 Feb 2024 12:16:56 INFO o.a.r.l.p.LdapUserGroupBuilder >>>>> [UnixUserSyncThread] - LdapUserGroupBuilder initialization completed with >>>>> -- ldapUrl: ldap://cmb.blabla.org:389, ldapBindDn: >>>>> CN=LCH657,ou=COCM,ou=utilisateurs,dc=cmb,dc=blabla,dc=org, >>>>> ldapBindPassword: * , ldapAuthenticationMechanism: simple, >>>>> searchBase: dc=cmb,dc=blabla,dc=org, userSearchBase: >>>>> [ou=COCM,ou=utilisateurs,dc=cmb,dc=blabla,dc=org], userSearchScope: 2, >>>>> userObjectClass: organizationalPerson, userSearchFilter: >>>>> (memberOf=CN=usr_tool_prd,OU=Tool,OU=Groupes,DC=blabla,DC=org), >>>>> extendedUserSear
Re: Groups not retrieved
Hi Sailaja,
Actually, the groups are not stored in the LDAP I'm querying (or at least I
can't access them), so I'm retrieving the groups using
the SYNC_LDAP_USER_GROUP_NAME_ATTRIBUTE parameter, which I configured but
doesn't seem to work as I expected.
As a matter of fact, I'm successfully retrieving users from the LDAP with a
postOfficeBox field, but setting SYNC_LDAP_USER_GROUP_NAME_ATTRIBUTE =
postOfficeBox does not retrieve the value of the field to create a group.
Let me give you an example to clarify. From the LDAP I'm retrieving the
following user :
sn: DOE
postOfficeBox: 9001928
givenName: JOHN
displayName: DOE JOHN
memberOf: CN=usr_tool_prd,OU=Tool,OU=Groupes,DC=blabla,DC=org
name: FOO123
mail: john@blabla.com
The field I'm really interested in for group purposes is postOfficeBox. So
by setting SYNC_LDAP_USER_GROUP_NAME_ATTRIBUTE = postOfficeBox I expect
Usersync to create a group named "9001928" and add John Doe to that group,
but it doesn't work. Does Usersync only expect groups with LDAP structure
(like the memberOf line) ?
Thanks,
Loïc
Le jeu. 21 mars 2024 à 22:51, Sailaja Polavarapu
a écrit :
> Hi Loic,
> I see that you have below config properties for group search. In this
> case the groups are retrieved from "dc=cmb,dc=blabla,dc=org" search base.
> Can you check if "CN=usr_tool_prd,OU=Tool,OU=Groupes,DC=blabla,DC=org"
> group is under the configured search base?
> groupSearchEnabled: true, groupSearchBase: [dc=cmb,dc=blabla,dc=org],
> groupSearchScope: 2, groupObjectClass: groupofnames,
> May be if you provide usersync logs, that can help to analyze further
>
> Thanks,
> Sailaja.
>
> On Thu, Mar 21, 2024 at 8:00 AM Loïc CHANEL
> wrote:
>
>> Hi team,
>> Am I the only one experiencing this issue ?
>> Thanks,
>>
>> Loïc
>>
>>
>> Le lun. 19 févr. 2024 à 12:38, Loïc CHANEL
>> a écrit :
>>
>>> Hi guys,
>>>
>>> Since 2.4, LDAP information retrieval to create groups seems broken. My
>>> sync issues are solved for users, but I'm still unable to pull groups from
>>> LDAP. For instance, here are the information in the LDAP from my user :
>>> sn: CHANEL
>>> postOfficeBox: someValue
>>> givenName: LOIC
>>> displayName: CHANEL LOIC
>>> memberOf: CN=usr_tool_prd,OU=Tool,OU=Groupes,DC=blabla,DC=org
>>> name: LCH657
>>> mail: loic.cha...@telecomnancy.net
>>>
>>> Now here is my configuration on Ranger side :
>>>
>>> ranger.usersync.ldap.user.groupnameattribute
>>> postOfficeBox,memberOf
>>>
>>>
>>> And I can even see that the retrieval is going that way :
>>> 9 Feb 2024 12:16:56 INFO o.a.r.l.p.LdapUserGroupBuilder
>>> [UnixUserSyncThread] - LdapUserGroupBuilder initialization completed with
>>> -- ldapUrl: ldap://cmb.blabla.org:389, ldapBindDn:
>>> CN=LCH657,ou=COCM,ou=utilisateurs,dc=cmb,dc=blabla,dc=org,
>>> ldapBindPassword: * , ldapAuthenticationMechanism: simple,
>>> searchBase: dc=cmb,dc=blabla,dc=org, userSearchBase:
>>> [ou=COCM,ou=utilisateurs,dc=cmb,dc=blabla,dc=org], userSearchScope: 2,
>>> userObjectClass: organizationalPerson, userSearchFilter:
>>> (memberOf=CN=usr_tool_prd,OU=Tool,OU=Groupes,DC=blabla,DC=org),
>>> extendedUserSearchFilter: null, userNameAttribute: name,
>>> userSearchAttributes: [postOfficeBox, uSNChanged, name, memberOf,
>>> modifytimestamp, objectid, userurincipaluame], userGroupNameAttributeSet:
>>> [postOfficeBox, memberOf], otherUserAttributes: [userurincipaluame],
>>> pagedResultsEnabled: true, pagedResultsSize: 500, groupSearchEnabled:
>>> true, groupSearchBase: [dc=cmb,dc=blabla,dc=org], groupSearchScope: 2,
>>> groupObjectClass: groupofnames, groupSearchFilter: ,
>>> extendedGroupSearchFilter: ((|(member={0})(member={1}))),
>>> extendedAllGroupsSearchFilter: null, groupMemberAttributeName: member,
>>> groupNameAttribute: cn, groupSearchAttributes: [uSNChanged, displayname,
>>> member, cn, modifytimestamp, objectid], groupSearchFirstEnabled: true,
>>> userSearchEnabled: true, ldapReferral: ignore
>>>
>>> But in Ranger, my user is created without any group. What am I missing ?
>>> Thanks,
>>>
>>>
>>> Loïc CHANEL
>>> Technical leader Big Data
>>> Capgemini (Lyon, France)
>>>
>>
Re: Groups not retrieved
Hi team,
Am I the only one experiencing this issue ?
Thanks,
Loïc
Le lun. 19 févr. 2024 à 12:38, Loïc CHANEL a
écrit :
> Hi guys,
>
> Since 2.4, LDAP information retrieval to create groups seems broken. My
> sync issues are solved for users, but I'm still unable to pull groups from
> LDAP. For instance, here are the information in the LDAP from my user :
> sn: CHANEL
> postOfficeBox: someValue
> givenName: LOIC
> displayName: CHANEL LOIC
> memberOf: CN=usr_tool_prd,OU=Tool,OU=Groupes,DC=blabla,DC=org
> name: LCH657
> mail: loic.cha...@telecomnancy.net
>
> Now here is my configuration on Ranger side :
>
> ranger.usersync.ldap.user.groupnameattribute
> postOfficeBox,memberOf
>
>
> And I can even see that the retrieval is going that way :
> 9 Feb 2024 12:16:56 INFO o.a.r.l.p.LdapUserGroupBuilder
> [UnixUserSyncThread] - LdapUserGroupBuilder initialization completed with
> -- ldapUrl: ldap://cmb.blabla.org:389, ldapBindDn:
> CN=LCH657,ou=COCM,ou=utilisateurs,dc=cmb,dc=blabla,dc=org,
> ldapBindPassword: * , ldapAuthenticationMechanism: simple,
> searchBase: dc=cmb,dc=blabla,dc=org, userSearchBase:
> [ou=COCM,ou=utilisateurs,dc=cmb,dc=blabla,dc=org], userSearchScope: 2,
> userObjectClass: organizationalPerson, userSearchFilter:
> (memberOf=CN=usr_tool_prd,OU=Tool,OU=Groupes,DC=blabla,DC=org),
> extendedUserSearchFilter: null, userNameAttribute: name,
> userSearchAttributes: [postOfficeBox, uSNChanged, name, memberOf,
> modifytimestamp, objectid, userurincipaluame], userGroupNameAttributeSet:
> [postOfficeBox, memberOf], otherUserAttributes: [userurincipaluame],
> pagedResultsEnabled: true, pagedResultsSize: 500, groupSearchEnabled:
> true, groupSearchBase: [dc=cmb,dc=blabla,dc=org], groupSearchScope: 2,
> groupObjectClass: groupofnames, groupSearchFilter: ,
> extendedGroupSearchFilter: ((|(member={0})(member={1}))),
> extendedAllGroupsSearchFilter: null, groupMemberAttributeName: member,
> groupNameAttribute: cn, groupSearchAttributes: [uSNChanged, displayname,
> member, cn, modifytimestamp, objectid], groupSearchFirstEnabled: true,
> userSearchEnabled: true, ldapReferral: ignore
>
> But in Ranger, my user is created without any group. What am I missing ?
> Thanks,
>
>
> Loïc CHANEL
> Technical leader Big Data
> Capgemini (Lyon, France)
>
Re: Usersync from file
You're right, I definitely missed this (and feel like a complete idiot). Now it works as expected. Thanks a lot for your help, Loïc CHANEL Technical leader Big Data Capgemini (Lyon, France) Le ven. 8 mars 2024 à 17:59, Sailaja Polavarapu a écrit : > I see that the below property is set to false. This property is used to > call updateSink() method > <https://github.com/apache/ranger/blob/master/ugsync/src/main/java/org/apache/ranger/usergroupsync/UserGroupSync.java#L114> > > > > ranger.usersync.enabled > false > > > > On Fri, Mar 8, 2024 at 8:35 AM Loïc CHANEL > wrote: > >> Missed the attachment before sending the email ... rookie mistake. Here >> is the actual file (sorry for the spam). >> Best regards, >> >> Loïc CHANEL >> Technical leader Big Data >> Capgemini (Lyon, France) >> >> >> Le ven. 8 mars 2024 à 17:33, Loïc CHANEL >> a écrit : >> >>> I'm using the 2.4.0 release. You will find the ranger-ugsync-site.xml config >>> file in attachment, feel free to let me know if you need more. >>> Best regards, >>> >>> >>> Loïc CHANEL >>> Technical leader Big Data >>> Capgemini (Lyon, France) >>> >>> >>> Le ven. 8 mars 2024 à 17:24, Sailaja Polavarapu < >>> spolavar...@cloudera.com> a écrit : >>> >>>> Which branch are you using? And can you share the usersync config? >>>> >>>> On Fri, Mar 8, 2024 at 8:10 AM Loïc CHANEL < >>>> loic.cha...@telecomnancy.net> wrote: >>>> >>>>> Hi Sailaja, >>>>> >>>>> I just updated the file and restarted the usersync process, I have the >>>>> exact same logs and the users are still not created. Is there a case in >>>>> which the file is printed but not synced ? >>>>> No error before, except the following warning that's quite usual : >>>>> 08 Mar 2024 16:48:50 WARN o.a.h.u.NativeCodeLoader >>>>> [UnixUserSyncThread] - Unable to load native-hadoop library for your >>>>> platform... using builtin-java classes where applicable >>>>> Still, it's weird that it shows with "UnixUserSyncThread" while it's >>>>> syncing users from a file, isn't it ? >>>>> I'm a bit lost on this, because it is clear that the usersync process >>>>> is able to read the user file and I see it communicating with Ranger Admin >>>>> to load all the users, but the users in the file are not created. Is there >>>>> some configuration missing from >>>>> https://cwiki.apache.org/confluence/display/RANGER/File+Source+User+Group+Sync+process >>>>> ? >>>>> Thanks, >>>>> >>>>> >>>>> Loïc CHANEL >>>>> Technical leader Big Data >>>>> Capgemini (Lyon, France) >>>>> >>>>> >>>>> Le ven. 8 mars 2024 à 16:00, Sailaja Polavarapu < >>>>> spolavar...@cloudera.com> a écrit : >>>>> >>>>>> This is strange as I don't see any logs from updateSink() method >>>>>> <https://github.com/apache/ranger/blob/master/ugsync/src/main/java/org/apache/ranger/unixusersync/process/FileSourceUserGroupBuilder.java#L145> >>>>>> . >>>>>> Can you check the timestamp on the file and try updating the file? and >>>>>> also >>>>>> check if there are any errors before. Looks like this file is read once >>>>>> successfully and hence you see the logs from print() method which logs >>>>>> the >>>>>> data from usersync cache. >>>>>> >>>>>> On Fri, Mar 8, 2024 at 4:40 AM Loïc CHANEL < >>>>>> loic.cha...@telecomnancy.net> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> Here are the logs on usersync side : >>>>>>> 08 Mar 2024 12:26:54 DEBUG o.a.r.u.p.PolicyMgrUserGroupBuilder >>>>>>> [UnixUserSyncThread] - PolicyMgrUserGroupBuilderOld.init()==> >>>>>>> PolMgrBaseUrl >>>>>>> : http://hdp-rec-ranger1-r.l.infra.com:6080 KeyStore File : >>>>>>> /etc/ranger/usersync/conf/cert/unixauthservice.jks TrustStore File : >>>>>>> Authentication Type : kerberos >>>>>>> 08 Mar 2024 12:26:54 INFO o.a.r.u.AbstractMapper >>>>>>> [UnixUserS
Re: Usersync from file
Missed the attachment before sending the email ... rookie mistake. Here is the actual file (sorry for the spam). Best regards, Loïc CHANEL Technical leader Big Data Capgemini (Lyon, France) Le ven. 8 mars 2024 à 17:33, Loïc CHANEL a écrit : > I'm using the 2.4.0 release. You will find the ranger-ugsync-site.xml config > file in attachment, feel free to let me know if you need more. > Best regards, > > > Loïc CHANEL > Technical leader Big Data > Capgemini (Lyon, France) > > > Le ven. 8 mars 2024 à 17:24, Sailaja Polavarapu > a écrit : > >> Which branch are you using? And can you share the usersync config? >> >> On Fri, Mar 8, 2024 at 8:10 AM Loïc CHANEL >> wrote: >> >>> Hi Sailaja, >>> >>> I just updated the file and restarted the usersync process, I have the >>> exact same logs and the users are still not created. Is there a case in >>> which the file is printed but not synced ? >>> No error before, except the following warning that's quite usual : >>> 08 Mar 2024 16:48:50 WARN o.a.h.u.NativeCodeLoader [UnixUserSyncThread] >>> - Unable to load native-hadoop library for your platform... using >>> builtin-java classes where applicable >>> Still, it's weird that it shows with "UnixUserSyncThread" while it's >>> syncing users from a file, isn't it ? >>> I'm a bit lost on this, because it is clear that the usersync process is >>> able to read the user file and I see it communicating with Ranger Admin to >>> load all the users, but the users in the file are not created. Is there >>> some configuration missing from >>> https://cwiki.apache.org/confluence/display/RANGER/File+Source+User+Group+Sync+process >>> ? >>> Thanks, >>> >>> >>> Loïc CHANEL >>> Technical leader Big Data >>> Capgemini (Lyon, France) >>> >>> >>> Le ven. 8 mars 2024 à 16:00, Sailaja Polavarapu < >>> spolavar...@cloudera.com> a écrit : >>> >>>> This is strange as I don't see any logs from updateSink() method >>>> <https://github.com/apache/ranger/blob/master/ugsync/src/main/java/org/apache/ranger/unixusersync/process/FileSourceUserGroupBuilder.java#L145> >>>> . >>>> Can you check the timestamp on the file and try updating the file? and also >>>> check if there are any errors before. Looks like this file is read once >>>> successfully and hence you see the logs from print() method which logs the >>>> data from usersync cache. >>>> >>>> On Fri, Mar 8, 2024 at 4:40 AM Loïc CHANEL < >>>> loic.cha...@telecomnancy.net> wrote: >>>> >>>>> Hi, >>>>> >>>>> Here are the logs on usersync side : >>>>> 08 Mar 2024 12:26:54 DEBUG o.a.r.u.p.PolicyMgrUserGroupBuilder >>>>> [UnixUserSyncThread] - PolicyMgrUserGroupBuilderOld.init()==> >>>>> PolMgrBaseUrl >>>>> : http://hdp-rec-ranger1-r.l.infra.com:6080 KeyStore File : >>>>> /etc/ranger/usersync/conf/cert/unixauthservice.jks TrustStore File : >>>>> Authentication Type : kerberos >>>>> 08 Mar 2024 12:26:54 INFO o.a.r.u.AbstractMapper [UnixUserSyncThread] >>>>> - Initializing for ranger.usersync.mapping.username.regex >>>>> 08 Mar 2024 12:26:54 INFO o.a.r.u.AbstractMapper [UnixUserSyncThread] >>>>> - Initializing for ranger.usersync.mapping.groupname.regex >>>>> 08 Mar 2024 12:26:54 INFO o.a.r.u.UserGroupSync [UnixUserSyncThread] >>>>> - initializing source: >>>>> org.apache.ranger.unixusersync.process.FileSourceUserGroupBuilder >>>>> 08 Mar 2024 12:26:54 INFO o.a.r.u.AbstractMapper [UnixUserSyncThread] >>>>> - Initializing for ranger.usersync.mapping.username.regex >>>>> 08 Mar 2024 12:26:54 INFO o.a.r.u.AbstractMapper [UnixUserSyncThread] >>>>> - Initializing for ranger.usersync.mapping.groupname.regex >>>>> 08 Mar 2024 12:26:54 DEBUG o.a.r.u.p.FileSourceUserGroupBuilder >>>>> [UnixUserSyncThread] - USER:user_one >>>>> 08 Mar 2024 12:26:54 DEBUG o.a.r.u.p.FileSourceUserGroupBuilder >>>>> [UnixUserSyncThread] - USER:user_two >>>>> 08 Mar 2024 12:26:54 DEBUG o.a.r.u.p.FileSourceUserGroupBuilder >>>>> [UnixUserSyncThread] - GROUP: 900021 >>>>> 08 Mar 2024 12:26:54 DEBUG o.a.r.u.p.FileSourceUserGroupBuilder >>>>> [UnixUserSyncThread] - USER:other_user >>>>> 08 Mar 2024 12:26:54 INFO o.a.r.u.
Re: Usersync from file
I'm using the 2.4.0 release. You will find the ranger-ugsync-site.xml config file in attachment, feel free to let me know if you need more. Best regards, Loïc CHANEL Technical leader Big Data Capgemini (Lyon, France) Le ven. 8 mars 2024 à 17:24, Sailaja Polavarapu a écrit : > Which branch are you using? And can you share the usersync config? > > On Fri, Mar 8, 2024 at 8:10 AM Loïc CHANEL > wrote: > >> Hi Sailaja, >> >> I just updated the file and restarted the usersync process, I have the >> exact same logs and the users are still not created. Is there a case in >> which the file is printed but not synced ? >> No error before, except the following warning that's quite usual : >> 08 Mar 2024 16:48:50 WARN o.a.h.u.NativeCodeLoader [UnixUserSyncThread] >> - Unable to load native-hadoop library for your platform... using >> builtin-java classes where applicable >> Still, it's weird that it shows with "UnixUserSyncThread" while it's >> syncing users from a file, isn't it ? >> I'm a bit lost on this, because it is clear that the usersync process is >> able to read the user file and I see it communicating with Ranger Admin to >> load all the users, but the users in the file are not created. Is there >> some configuration missing from >> https://cwiki.apache.org/confluence/display/RANGER/File+Source+User+Group+Sync+process >> ? >> Thanks, >> >> >> Loïc CHANEL >> Technical leader Big Data >> Capgemini (Lyon, France) >> >> >> Le ven. 8 mars 2024 à 16:00, Sailaja Polavarapu >> a écrit : >> >>> This is strange as I don't see any logs from updateSink() method >>> <https://github.com/apache/ranger/blob/master/ugsync/src/main/java/org/apache/ranger/unixusersync/process/FileSourceUserGroupBuilder.java#L145> >>> . >>> Can you check the timestamp on the file and try updating the file? and also >>> check if there are any errors before. Looks like this file is read once >>> successfully and hence you see the logs from print() method which logs the >>> data from usersync cache. >>> >>> On Fri, Mar 8, 2024 at 4:40 AM Loïc CHANEL >>> wrote: >>> >>>> Hi, >>>> >>>> Here are the logs on usersync side : >>>> 08 Mar 2024 12:26:54 DEBUG o.a.r.u.p.PolicyMgrUserGroupBuilder >>>> [UnixUserSyncThread] - PolicyMgrUserGroupBuilderOld.init()==> PolMgrBaseUrl >>>> : http://hdp-rec-ranger1-r.l.infra.com:6080 KeyStore File : >>>> /etc/ranger/usersync/conf/cert/unixauthservice.jks TrustStore File : >>>> Authentication Type : kerberos >>>> 08 Mar 2024 12:26:54 INFO o.a.r.u.AbstractMapper [UnixUserSyncThread] >>>> - Initializing for ranger.usersync.mapping.username.regex >>>> 08 Mar 2024 12:26:54 INFO o.a.r.u.AbstractMapper [UnixUserSyncThread] >>>> - Initializing for ranger.usersync.mapping.groupname.regex >>>> 08 Mar 2024 12:26:54 INFO o.a.r.u.UserGroupSync [UnixUserSyncThread] - >>>> initializing source: >>>> org.apache.ranger.unixusersync.process.FileSourceUserGroupBuilder >>>> 08 Mar 2024 12:26:54 INFO o.a.r.u.AbstractMapper [UnixUserSyncThread] >>>> - Initializing for ranger.usersync.mapping.username.regex >>>> 08 Mar 2024 12:26:54 INFO o.a.r.u.AbstractMapper [UnixUserSyncThread] >>>> - Initializing for ranger.usersync.mapping.groupname.regex >>>> 08 Mar 2024 12:26:54 DEBUG o.a.r.u.p.FileSourceUserGroupBuilder >>>> [UnixUserSyncThread] - USER:user_one >>>> 08 Mar 2024 12:26:54 DEBUG o.a.r.u.p.FileSourceUserGroupBuilder >>>> [UnixUserSyncThread] - USER:user_two >>>> 08 Mar 2024 12:26:54 DEBUG o.a.r.u.p.FileSourceUserGroupBuilder >>>> [UnixUserSyncThread] - GROUP: 900021 >>>> 08 Mar 2024 12:26:54 DEBUG o.a.r.u.p.FileSourceUserGroupBuilder >>>> [UnixUserSyncThread] - USER:other_user >>>> 08 Mar 2024 12:26:54 INFO o.a.r.u.UserGroupSync [UnixUserSyncThread] - >>>> Begin: initial load of user/group from source==>sink >>>> 08 Mar 2024 12:26:54 INFO o.a.r.u.UserGroupSync [UnixUserSyncThread] - >>>> End: initial load of user/group from source==>sink >>>> 08 Mar 2024 12:26:54 INFO o.a.r.u.UserGroupSync [UnixUserSyncThread] - >>>> Done initializing user/group source and sink >>>> 08 Mar 2024 12:26:54 DEBUG o.a.r.u.UserGroupSync [UnixUserSyncThread] - >>>> Sleeping for [30] milliSeconds >>>> >>>> While on the admin side, I only see the following : >>>> 10.18.1.43 - - [08/Mar/2024
Re: Usersync from file
Hi Sailaja, I just updated the file and restarted the usersync process, I have the exact same logs and the users are still not created. Is there a case in which the file is printed but not synced ? No error before, except the following warning that's quite usual : 08 Mar 2024 16:48:50 WARN o.a.h.u.NativeCodeLoader [UnixUserSyncThread] - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Still, it's weird that it shows with "UnixUserSyncThread" while it's syncing users from a file, isn't it ? I'm a bit lost on this, because it is clear that the usersync process is able to read the user file and I see it communicating with Ranger Admin to load all the users, but the users in the file are not created. Is there some configuration missing from https://cwiki.apache.org/confluence/display/RANGER/File+Source+User+Group+Sync+process ? Thanks, Loïc CHANEL Technical leader Big Data Capgemini (Lyon, France) Le ven. 8 mars 2024 à 16:00, Sailaja Polavarapu a écrit : > This is strange as I don't see any logs from updateSink() method > <https://github.com/apache/ranger/blob/master/ugsync/src/main/java/org/apache/ranger/unixusersync/process/FileSourceUserGroupBuilder.java#L145> > . > Can you check the timestamp on the file and try updating the file? and also > check if there are any errors before. Looks like this file is read once > successfully and hence you see the logs from print() method which logs the > data from usersync cache. > > On Fri, Mar 8, 2024 at 4:40 AM Loïc CHANEL > wrote: > >> Hi, >> >> Here are the logs on usersync side : >> 08 Mar 2024 12:26:54 DEBUG o.a.r.u.p.PolicyMgrUserGroupBuilder >> [UnixUserSyncThread] - PolicyMgrUserGroupBuilderOld.init()==> PolMgrBaseUrl >> : http://hdp-rec-ranger1-r.l.infra.com:6080 KeyStore File : >> /etc/ranger/usersync/conf/cert/unixauthservice.jks TrustStore File : >> Authentication Type : kerberos >> 08 Mar 2024 12:26:54 INFO o.a.r.u.AbstractMapper [UnixUserSyncThread] - >> Initializing for ranger.usersync.mapping.username.regex >> 08 Mar 2024 12:26:54 INFO o.a.r.u.AbstractMapper [UnixUserSyncThread] - >> Initializing for ranger.usersync.mapping.groupname.regex >> 08 Mar 2024 12:26:54 INFO o.a.r.u.UserGroupSync [UnixUserSyncThread] - >> initializing source: >> org.apache.ranger.unixusersync.process.FileSourceUserGroupBuilder >> 08 Mar 2024 12:26:54 INFO o.a.r.u.AbstractMapper [UnixUserSyncThread] - >> Initializing for ranger.usersync.mapping.username.regex >> 08 Mar 2024 12:26:54 INFO o.a.r.u.AbstractMapper [UnixUserSyncThread] - >> Initializing for ranger.usersync.mapping.groupname.regex >> 08 Mar 2024 12:26:54 DEBUG o.a.r.u.p.FileSourceUserGroupBuilder >> [UnixUserSyncThread] - USER:user_one >> 08 Mar 2024 12:26:54 DEBUG o.a.r.u.p.FileSourceUserGroupBuilder >> [UnixUserSyncThread] - USER:user_two >> 08 Mar 2024 12:26:54 DEBUG o.a.r.u.p.FileSourceUserGroupBuilder >> [UnixUserSyncThread] - GROUP: 900021 >> 08 Mar 2024 12:26:54 DEBUG o.a.r.u.p.FileSourceUserGroupBuilder >> [UnixUserSyncThread] - USER:other_user >> 08 Mar 2024 12:26:54 INFO o.a.r.u.UserGroupSync [UnixUserSyncThread] - >> Begin: initial load of user/group from source==>sink >> 08 Mar 2024 12:26:54 INFO o.a.r.u.UserGroupSync [UnixUserSyncThread] - >> End: initial load of user/group from source==>sink >> 08 Mar 2024 12:26:54 INFO o.a.r.u.UserGroupSync [UnixUserSyncThread] - >> Done initializing user/group source and sink >> 08 Mar 2024 12:26:54 DEBUG o.a.r.u.UserGroupSync [UnixUserSyncThread] - >> Sleeping for [30] milliSeconds >> >> While on the admin side, I only see the following : >> 10.18.1.43 - - [08/Mar/2024:12:26:52 +] "GET >> /service/xusers/users/?startIndex=4000=1000 HTTP/1.1" 200 1065581 >> 27402 "-" "Java/1.8.0_275" >> 10.18.1.43 - - [08/Mar/2024:12:26:54 +] "GET >> /service/xusers/ugsync/groupusers?startIndex=0=1000 HTTP/1.1" 200 >> 4429925 2553 "-" "Java/1.8.0_275" >> >> Looks like usersync isn't even trying to create the users. Anything I'm >> missing ? >> Best regards, >> >> >> Loïc CHANEL >> Technical leader Big Data >> Capgemini (Lyon, France) >> >> >> Le ven. 8 mars 2024 à 04:44, Bhavik Patel a >> écrit : >> >>> Hi, >>> >>> Can you kindly check Ranger admin logs and did you confirmed provided >>> rangeruseraync user password is same at Rangeradmin end. >>> >>> >>> Thanks >>> Bhavik Patel >>> +91-7208744109 >>> >>> >>> On Thu, 7 Mar 20
Re: Usersync from file
Hi, Here are the logs on usersync side : 08 Mar 2024 12:26:54 DEBUG o.a.r.u.p.PolicyMgrUserGroupBuilder [UnixUserSyncThread] - PolicyMgrUserGroupBuilderOld.init()==> PolMgrBaseUrl : http://hdp-rec-ranger1-r.l.infra.com:6080 KeyStore File : /etc/ranger/usersync/conf/cert/unixauthservice.jks TrustStore File : Authentication Type : kerberos 08 Mar 2024 12:26:54 INFO o.a.r.u.AbstractMapper [UnixUserSyncThread] - Initializing for ranger.usersync.mapping.username.regex 08 Mar 2024 12:26:54 INFO o.a.r.u.AbstractMapper [UnixUserSyncThread] - Initializing for ranger.usersync.mapping.groupname.regex 08 Mar 2024 12:26:54 INFO o.a.r.u.UserGroupSync [UnixUserSyncThread] - initializing source: org.apache.ranger.unixusersync.process.FileSourceUserGroupBuilder 08 Mar 2024 12:26:54 INFO o.a.r.u.AbstractMapper [UnixUserSyncThread] - Initializing for ranger.usersync.mapping.username.regex 08 Mar 2024 12:26:54 INFO o.a.r.u.AbstractMapper [UnixUserSyncThread] - Initializing for ranger.usersync.mapping.groupname.regex 08 Mar 2024 12:26:54 DEBUG o.a.r.u.p.FileSourceUserGroupBuilder [UnixUserSyncThread] - USER:user_one 08 Mar 2024 12:26:54 DEBUG o.a.r.u.p.FileSourceUserGroupBuilder [UnixUserSyncThread] - USER:user_two 08 Mar 2024 12:26:54 DEBUG o.a.r.u.p.FileSourceUserGroupBuilder [UnixUserSyncThread] - GROUP: 900021 08 Mar 2024 12:26:54 DEBUG o.a.r.u.p.FileSourceUserGroupBuilder [UnixUserSyncThread] - USER:other_user 08 Mar 2024 12:26:54 INFO o.a.r.u.UserGroupSync [UnixUserSyncThread] - Begin: initial load of user/group from source==>sink 08 Mar 2024 12:26:54 INFO o.a.r.u.UserGroupSync [UnixUserSyncThread] - End: initial load of user/group from source==>sink 08 Mar 2024 12:26:54 INFO o.a.r.u.UserGroupSync [UnixUserSyncThread] - Done initializing user/group source and sink 08 Mar 2024 12:26:54 DEBUG o.a.r.u.UserGroupSync [UnixUserSyncThread] - Sleeping for [30] milliSeconds While on the admin side, I only see the following : 10.18.1.43 - - [08/Mar/2024:12:26:52 +] "GET /service/xusers/users/?startIndex=4000=1000 HTTP/1.1" 200 1065581 27402 "-" "Java/1.8.0_275" 10.18.1.43 - - [08/Mar/2024:12:26:54 +] "GET /service/xusers/ugsync/groupusers?startIndex=0=1000 HTTP/1.1" 200 4429925 2553 "-" "Java/1.8.0_275" Looks like usersync isn't even trying to create the users. Anything I'm missing ? Best regards, Loïc CHANEL Technical leader Big Data Capgemini (Lyon, France) Le ven. 8 mars 2024 à 04:44, Bhavik Patel a écrit : > Hi, > > Can you kindly check Ranger admin logs and did you confirmed provided > rangeruseraync user password is same at Rangeradmin end. > > > Thanks > Bhavik Patel > +91-7208744109 > > > On Thu, 7 Mar 2024, 6:39 pm Loïc CHANEL, > wrote: > >> Hi guys, >> >> I'm currently trying to create Ranger users from a text file and followed >> this documentation to configure Usersync to work with a text file as a >> source : >> https://cwiki.apache.org/confluence/display/RANGER/File+Source+User+Group+Sync+process >> Still, even if the logs show that the CSV file containing the users is >> properly read, nothing happens : I can't see the users in Ranger UI. Here >> are the logs : >> >> 07 Mar 2024 12:22:13 DEBUG o.a.r.u.p.PolicyMgrUserGroupBuilder >> [UnixUserSyncThread] - <== >> PolicyMgrUserGroupBuilder.buildGroupUserLinkList() >> 07 Mar 2024 12:22:13 DEBUG o.a.r.u.p.PolicyMgrUserGroupBuilder >> [UnixUserSyncThread] - PolicyMgrUserGroupBuilderOld.init()==> PolMgrBaseUrl >> : http://hdp-rec-ranger1-r.l.internal.com:6080 KeyStore File : >> /etc/ranger/usersync/conf/cert/unixauthservice.jks TrustStore File : >> Authentication Type : kerberos >> 07 Mar 2024 12:22:13 INFO o.a.r.u.AbstractMapper [UnixUserSyncThread] - >> Initializing for ranger.usersync.mapping.username.regex >> 07 Mar 2024 12:22:13 INFO o.a.r.u.AbstractMapper [UnixUserSyncThread] - >> Initializing for ranger.usersync.mapping.groupname.regex >> 07 Mar 2024 12:22:13 INFO o.a.r.u.UserGroupSync [UnixUserSyncThread] - >> initializing source: >> org.apache.ranger.unixusersync.process.FileSourceUserGroupBuilder >> 07 Mar 2024 12:22:13 INFO o.a.r.u.AbstractMapper [UnixUserSyncThread] - >> Initializing for ranger.usersync.mapping.username.regex >> 07 Mar 2024 12:22:13 INFO o.a.r.u.AbstractMapper [UnixUserSyncThread] - >> Initializing for ranger.usersync.mapping.groupname.regex >> 07 Mar 2024 12:22:13 DEBUG o.a.r.u.p.FileSourceUserGroupBuilder >> [UnixUserSyncThread] - USER:user_one >> 07 Mar 2024 12:22:13 DEBUG o.a.r.u.p.FileSourceUserGroupBuilder >> [UnixUserSyncThread] - USER:user_two >> 07 Mar 2024 12:22:13 DEBUG o.a.r.u.p.FileSourceUserGroupBuilder >> [UnixUserSyncThread] - GROUP: 900021 >> 07 Mar 2024 12:22:
Usersync from file
Hi guys, I'm currently trying to create Ranger users from a text file and followed this documentation to configure Usersync to work with a text file as a source : https://cwiki.apache.org/confluence/display/RANGER/File+Source+User+Group+Sync+process Still, even if the logs show that the CSV file containing the users is properly read, nothing happens : I can't see the users in Ranger UI. Here are the logs : 07 Mar 2024 12:22:13 DEBUG o.a.r.u.p.PolicyMgrUserGroupBuilder [UnixUserSyncThread] - <== PolicyMgrUserGroupBuilder.buildGroupUserLinkList() 07 Mar 2024 12:22:13 DEBUG o.a.r.u.p.PolicyMgrUserGroupBuilder [UnixUserSyncThread] - PolicyMgrUserGroupBuilderOld.init()==> PolMgrBaseUrl : http://hdp-rec-ranger1-r.l.internal.com:6080 KeyStore File : /etc/ranger/usersync/conf/cert/unixauthservice.jks TrustStore File : Authentication Type : kerberos 07 Mar 2024 12:22:13 INFO o.a.r.u.AbstractMapper [UnixUserSyncThread] - Initializing for ranger.usersync.mapping.username.regex 07 Mar 2024 12:22:13 INFO o.a.r.u.AbstractMapper [UnixUserSyncThread] - Initializing for ranger.usersync.mapping.groupname.regex 07 Mar 2024 12:22:13 INFO o.a.r.u.UserGroupSync [UnixUserSyncThread] - initializing source: org.apache.ranger.unixusersync.process.FileSourceUserGroupBuilder 07 Mar 2024 12:22:13 INFO o.a.r.u.AbstractMapper [UnixUserSyncThread] - Initializing for ranger.usersync.mapping.username.regex 07 Mar 2024 12:22:13 INFO o.a.r.u.AbstractMapper [UnixUserSyncThread] - Initializing for ranger.usersync.mapping.groupname.regex 07 Mar 2024 12:22:13 DEBUG o.a.r.u.p.FileSourceUserGroupBuilder [UnixUserSyncThread] - USER:user_one 07 Mar 2024 12:22:13 DEBUG o.a.r.u.p.FileSourceUserGroupBuilder [UnixUserSyncThread] - USER:user_two 07 Mar 2024 12:22:13 DEBUG o.a.r.u.p.FileSourceUserGroupBuilder [UnixUserSyncThread] - GROUP: 900021 07 Mar 2024 12:22:13 DEBUG o.a.r.u.p.FileSourceUserGroupBuilder [UnixUserSyncThread] - USER:other_user 07 Mar 2024 12:22:13 INFO o.a.r.u.UserGroupSync [UnixUserSyncThread] - Begin: initial load of user/group from source==>sink 07 Mar 2024 12:22:13 INFO o.a.r.u.UserGroupSync [UnixUserSyncThread] - End: initial load of user/group from source==>sink 07 Mar 2024 12:22:13 INFO o.a.r.u.UserGroupSync [UnixUserSyncThread] - Done initializing user/group source and sink 07 Mar 2024 12:22:13 DEBUG o.a.r.u.UserGroupSync [UnixUserSyncThread] - Sleeping for [30] milliSeconds Am I missing something ? Thanks for your help, Loïc CHANEL Technical leader Big Data Capgemini (Lyon, France)
Allow external user creation via Python API
Hi guys, While I was trying to create external users via Python API I noticed this cannot be done as the only user creation allowed currently is the creation of internal users, with password verification. Still, as the Python code is fairly simple to understand, I dug into it and noticed the creation of external users would be easy to develop, so I ended up creating the following Jira describing the code modification : https://issues.apache.org/jira/browse/RANGER-4738 As I'm not a committer in the Apache project, could someone with such a role implement it ? Although the modification is minor, it would be a major improvement on my Ranger usage. Thanks for your help, Loïc CHANEL Technical leader Big Data Capgemini (Lyon, France)
Re: [Usersync] Fetch several attributes
Hi guys, I went to take a look at the code, I think I can develop the part I'm missing (pulling First Name, Last Name and Email Address fields from the LDAP/AD if the user provides the fields where to get these information in the LDAP/AD). Now this brings 2 questions : - How probable is it that my modification could be accepted in the main Ranger branch ? - When is the next Ranger release scheduled for ? To be clear, I really need that feature so if it isn't part of Ranger usersync roadmap I will have to develop a minimalistic Ranger usersync module that does it (and in my opinion it's a shame not making it available to the community). What's your opinion on this? Thanks, Loïc CHANEL Technical leader Big Data Capgemini (Lyon, France) Le ven. 16 févr. 2024 à 16:43, Loïc CHANEL a écrit : > Hi Sailaja, > > I was finally able to understand what's been developed. Actually, if the > email was in the pulled information this would have been great, but > unfortunately this doesn't bring too much information in my case. > Can we add other properties to be pulled dynamically from the LDAP ? > Thanks, > > Loïc CHANEL > Technical leader Big Data > Capgemini (Lyon, France) > > > Le jeu. 15 févr. 2024 à 09:38, Loïc CHANEL > a écrit : > >> Hi Sailaja, >> From the look of it, I'd say it could work for my use case. Do you know >> where I could find some documentation about using that feature ? >> Thanks, >> >> Loïc >> >> Le mer. 14 févr. 2024 à 18:05, Sailaja Polavarapu < >> spolavar...@cloudera.com> a écrit : >> >>> Hi Loïc CHANEL, >>> Syncing extra attributes from AD/LDAP is partly supported as part of >>> RANGER-2697 <https://issues.apache.org/jira/browse/RANGER-2697>. Can >>> you please check it out and see if this works for your use case? It is not >>> exactly what you are asking for, but in Usersync, there is an option to >>> sync other attributes which are stored as key-value pairs in >>> the otherAttributes field for each user/group. >>> - Sailaja. >>> >>> On Wed, Feb 14, 2024 at 8:00 AM Loïc CHANEL < >>> loic.cha...@telecomnancy.net> wrote: >>> >>>> Hi guys, >>>> >>>> I understand the feature is still not available at the moment, so 2 >>>> questions : >>>> - I gave a look at the current code, I don't think the development of >>>> such a feature would be too hard. If I develop it, could it be merged >>>> easily ? >>>> - Is there a way to modify the users post-import in order to enrich the >>>> first name/last name/email fields (manually or with an API) ? >>>> >>>> Thanks for your help, >>>> >>>> >>>> Loïc CHANEL >>>> Technical leader Big Data >>>> Capgemini (Lyon, France) >>>> >>>> >>>> Le jeu. 8 févr. 2024 à 12:30, Loïc CHANEL >>>> a écrit : >>>> >>>>> Hi guys, >>>>> >>>>> As I'm pulling Ranger users from LDAP, I would like not only to pull >>>>> the username from there but also the first and last name, and most >>>>> importantly the email address of the LDAP user. This led me to an old >>>>> Jira <https://issues.apache.org/jira/browse/RANGER-567> that asks for >>>>> just that feature (the person who answers has an oddly familiar first >>>>> name), which makes me think it has not been implemented yet. >>>>> Am I correct ? If so, how hard would it be to develop and >>>>> maintain that feature ? >>>>> In parallel, is there a way to modify the users post-import in order >>>>> to enrich the first name/last name/email fields ? >>>>> Thanks for your help, >>>>> >>>>> >>>>> Loïc CHANEL >>>>> Technical leader Big Data >>>>> Capgemini (Lyon, France) >>>>> >>>>
Groups not retrieved
Hi guys,
Since 2.4, LDAP information retrieval to create groups seems broken. My
sync issues are solved for users, but I'm still unable to pull groups from
LDAP. For instance, here are the information in the LDAP from my user :
sn: CHANEL
postOfficeBox: someValue
givenName: LOIC
displayName: CHANEL LOIC
memberOf: CN=usr_tool_prd,OU=Tool,OU=Groupes,DC=blabla,DC=org
name: LCH657
mail: loic.cha...@telecomnancy.net
Now here is my configuration on Ranger side :
ranger.usersync.ldap.user.groupnameattribute
postOfficeBox,memberOf
And I can even see that the retrieval is going that way :
9 Feb 2024 12:16:56 INFO o.a.r.l.p.LdapUserGroupBuilder
[UnixUserSyncThread] - LdapUserGroupBuilder initialization completed with
-- ldapUrl: ldap://cmb.blabla.org:389, ldapBindDn:
CN=LCH657,ou=COCM,ou=utilisateurs,dc=cmb,dc=blabla,dc=org,
ldapBindPassword: * , ldapAuthenticationMechanism: simple,
searchBase: dc=cmb,dc=blabla,dc=org, userSearchBase:
[ou=COCM,ou=utilisateurs,dc=cmb,dc=blabla,dc=org], userSearchScope: 2,
userObjectClass: organizationalPerson, userSearchFilter:
(memberOf=CN=usr_tool_prd,OU=Tool,OU=Groupes,DC=blabla,DC=org),
extendedUserSearchFilter: null, userNameAttribute: name,
userSearchAttributes: [postOfficeBox, uSNChanged, name, memberOf,
modifytimestamp, objectid, userurincipaluame], userGroupNameAttributeSet:
[postOfficeBox, memberOf], otherUserAttributes: [userurincipaluame],
pagedResultsEnabled: true, pagedResultsSize: 500, groupSearchEnabled:
true, groupSearchBase: [dc=cmb,dc=blabla,dc=org], groupSearchScope: 2,
groupObjectClass: groupofnames, groupSearchFilter: ,
extendedGroupSearchFilter: ((|(member={0})(member={1}))),
extendedAllGroupsSearchFilter: null, groupMemberAttributeName: member,
groupNameAttribute: cn, groupSearchAttributes: [uSNChanged, displayname,
member, cn, modifytimestamp, objectid], groupSearchFirstEnabled: true,
userSearchEnabled: true, ldapReferral: ignore
But in Ranger, my user is created without any group. What am I missing ?
Thanks,
Loïc CHANEL
Technical leader Big Data
Capgemini (Lyon, France)
Re: [Usersync] Fetch several attributes
Hi Sailaja, I was finally able to understand what's been developed. Actually, if the email was in the pulled information this would have been great, but unfortunately this doesn't bring too much information in my case. Can we add other properties to be pulled dynamically from the LDAP ? Thanks, Loïc CHANEL Technical leader Big Data Capgemini (Lyon, France) Le jeu. 15 févr. 2024 à 09:38, Loïc CHANEL a écrit : > Hi Sailaja, > From the look of it, I'd say it could work for my use case. Do you know > where I could find some documentation about using that feature ? > Thanks, > > Loïc > > Le mer. 14 févr. 2024 à 18:05, Sailaja Polavarapu < > spolavar...@cloudera.com> a écrit : > >> Hi Loïc CHANEL, >> Syncing extra attributes from AD/LDAP is partly supported as part of >> RANGER-2697 <https://issues.apache.org/jira/browse/RANGER-2697>. Can >> you please check it out and see if this works for your use case? It is not >> exactly what you are asking for, but in Usersync, there is an option to >> sync other attributes which are stored as key-value pairs in >> the otherAttributes field for each user/group. >> - Sailaja. >> >> On Wed, Feb 14, 2024 at 8:00 AM Loïc CHANEL >> wrote: >> >>> Hi guys, >>> >>> I understand the feature is still not available at the moment, so 2 >>> questions : >>> - I gave a look at the current code, I don't think the development of >>> such a feature would be too hard. If I develop it, could it be merged >>> easily ? >>> - Is there a way to modify the users post-import in order to enrich the >>> first name/last name/email fields (manually or with an API) ? >>> >>> Thanks for your help, >>> >>> >>> Loïc CHANEL >>> Technical leader Big Data >>> Capgemini (Lyon, France) >>> >>> >>> Le jeu. 8 févr. 2024 à 12:30, Loïc CHANEL >>> a écrit : >>> >>>> Hi guys, >>>> >>>> As I'm pulling Ranger users from LDAP, I would like not only to pull >>>> the username from there but also the first and last name, and most >>>> importantly the email address of the LDAP user. This led me to an old >>>> Jira <https://issues.apache.org/jira/browse/RANGER-567> that asks for >>>> just that feature (the person who answers has an oddly familiar first >>>> name), which makes me think it has not been implemented yet. >>>> Am I correct ? If so, how hard would it be to develop and maintain that >>>> feature ? >>>> In parallel, is there a way to modify the users post-import in order to >>>> enrich the first name/last name/email fields ? >>>> Thanks for your help, >>>> >>>> >>>> Loïc CHANEL >>>> Technical leader Big Data >>>> Capgemini (Lyon, France) >>>> >>>
Re: LDAP sync shows no log
Finally found the issue. By default, after the setup the property ranger.usersync.enabled is set to false in ranger-ugsync-site.xml. Setting it to true made synchronisation from LDAP work. Don't know why this behaviour has been implemented, but it seems very unnatural as this isn't even described in the install.properties file. Maybe it's a bug ? Best regards, Loïc CHANEL Technical leader Big Data Capgemini (Lyon, France) Le ven. 16 févr. 2024 à 11:53, Loïc CHANEL a écrit : > Adding more details to this : I provided usersync with a wrong password > for LDAP, but it's still showing the exact same logs. Therefore I'm > wondering where I would be able to get more details about the LDAP > connection from usersync, because the logs are clearly not enough. > Thanks, > > > Loïc CHANEL > Technical leader Big Data > Capgemini (Lyon, France) > > > Le ven. 16 févr. 2024 à 09:05, Loïc CHANEL > a écrit : > >> Hi Sailaja, >> >> I already restarted usersync since I deleted the user, but nothing >> happened. Where is this cache supposed to be stored ? >> By the way, SYNC_LDAP_DELTASYNC is set to false in my case, so I guess it >> could change the behaviour ? >> Thanks, >> >> >> Loïc CHANEL >> Technical leader Big Data >> Capgemini (Lyon, France) >> >> >> Le jeu. 15 févr. 2024 à 23:23, Sailaja Polavarapu < >> spolavar...@cloudera.com> a écrit : >> >>> Ranger Usersync caches the users and groups that are sync'd from LDAP >>> and uses this to compute delta for every sync cycle in order to update >>> ranger admin with the changes. Initially, during start up, this cache is >>> built from the users and groups that are in Ranger admin and is updated >>> only when the changes happen at the sync source (like LDAP) for every sync >>> cycle. If you delete the user in Ranger admin, usersync is notified with >>> this change and this user still exists in usersync cache. In order to >>> re-populate this cache from what is updated in ranger admin, you need to >>> restart usersync. Hope this helps. >>> >>> Thanks, >>> Sailaja. >>> >>> On Thu, Feb 15, 2024 at 9:28 AM Loïc CHANEL < >>> loic.cha...@telecomnancy.net> wrote: >>> >>>> Hi guys, >>>> >>>> I'm still working on user sync based on LDAP, I'm afraid the sync >>>> doesn't work without showing any logs. >>>> I performed an initial sync in an older version, and just upgraded to >>>> 2.4. Now, to check the usersync is still working, I deleted a user and >>>> waited for it to be created again except ... it's not, even though the user >>>> is still in the LDAP. >>>> In the logs I can see that usersync is trying to reach the LDAP, but >>>> even with DEBUG log level I can't either see the user is retrieved, nor >>>> there is an error while trying to retrieve it. Here's the only log I get : >>>> >>>> 15 Feb 2024 16:57:02 INFO o.a.r.u.UserGroupSync [UnixUserSyncThread] - >>>> initializing source: >>>> org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder >>>> 15 Feb 2024 16:57:02 INFO o.a.r.l.p.LdapUserGroupBuilder >>>> [UnixUserSyncThread] - LdapUserGroupBuilder initialization started >>>> 15 Feb 2024 16:57:02 DEBUG o.a.h.s.a.AbstractJavaKeyStoreProvider >>>> [UnixUserSyncThread] - backing jks path initialized to >>>> file:/etc/ranger/usersync/conf/rangerusersync.jceks >>>> 15 Feb 2024 16:57:02 INFO o.a.r.l.p.LdapUserGroupBuilder >>>> [UnixUserSyncThread] - LdapUserGroupBuilder initialization completed with >>>> -- ldapUrl: ldap://cmb.blabla.org:389, ldapBindDn: >>>> CN=itsme,ou=CACM,ou=utilisateurs,dc=cmb,dc=blabla,dc=org, >>>> ldapBindPassword: * , ldapAuthenticationMechanism: simple, >>>> searchBase: dc=cmb,dc=blabla,dc=org, userSearchBase: >>>> [ou=CACM,ou=utilisateurs,dc=cmb,dc=blabla,dc=org], userSearchScope: 2, >>>> userObjectClass: organizationalPerson, userSearchFilter: >>>> (memberOf=CN=usr_prd,OU=Component,OU=Groupes,DC=blabla,DC=org), >>>> extendedUserSearchFilter: null, userNameAttribute: name, >>>> userSearchAttributes: [postOfficeBox, uSNChanged, name, modifytimestamp, >>>> objectid, userurincipaluame], userGroupNameAttributeSet: [postOfficeBox], >>>> otherUserAttributes: [userurincipaluame], pagedResultsEnabled: true, >>>> pagedResultsSize: 500, groupSearchEnabled: true, groupSearchBase: >>>> [dc=cmb,dc=blabla,dc=org], groupS
Re: LDAP sync shows no log
Adding more details to this : I provided usersync with a wrong password for
LDAP, but it's still showing the exact same logs. Therefore I'm wondering
where I would be able to get more details about the LDAP connection from
usersync, because the logs are clearly not enough.
Thanks,
Loïc CHANEL
Technical leader Big Data
Capgemini (Lyon, France)
Le ven. 16 févr. 2024 à 09:05, Loïc CHANEL a
écrit :
> Hi Sailaja,
>
> I already restarted usersync since I deleted the user, but nothing
> happened. Where is this cache supposed to be stored ?
> By the way, SYNC_LDAP_DELTASYNC is set to false in my case, so I guess it
> could change the behaviour ?
> Thanks,
>
>
> Loïc CHANEL
> Technical leader Big Data
> Capgemini (Lyon, France)
>
>
> Le jeu. 15 févr. 2024 à 23:23, Sailaja Polavarapu <
> spolavar...@cloudera.com> a écrit :
>
>> Ranger Usersync caches the users and groups that are sync'd from LDAP and
>> uses this to compute delta for every sync cycle in order to update ranger
>> admin with the changes. Initially, during start up, this cache is built
>> from the users and groups that are in Ranger admin and is updated only when
>> the changes happen at the sync source (like LDAP) for every sync cycle. If
>> you delete the user in Ranger admin, usersync is notified with this change
>> and this user still exists in usersync cache. In order to re-populate this
>> cache from what is updated in ranger admin, you need to restart usersync.
>> Hope this helps.
>>
>> Thanks,
>> Sailaja.
>>
>> On Thu, Feb 15, 2024 at 9:28 AM Loïc CHANEL
>> wrote:
>>
>>> Hi guys,
>>>
>>> I'm still working on user sync based on LDAP, I'm afraid the sync
>>> doesn't work without showing any logs.
>>> I performed an initial sync in an older version, and just upgraded to
>>> 2.4. Now, to check the usersync is still working, I deleted a user and
>>> waited for it to be created again except ... it's not, even though the user
>>> is still in the LDAP.
>>> In the logs I can see that usersync is trying to reach the LDAP, but
>>> even with DEBUG log level I can't either see the user is retrieved, nor
>>> there is an error while trying to retrieve it. Here's the only log I get :
>>>
>>> 15 Feb 2024 16:57:02 INFO o.a.r.u.UserGroupSync [UnixUserSyncThread] -
>>> initializing source:
>>> org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder
>>> 15 Feb 2024 16:57:02 INFO o.a.r.l.p.LdapUserGroupBuilder
>>> [UnixUserSyncThread] - LdapUserGroupBuilder initialization started
>>> 15 Feb 2024 16:57:02 DEBUG o.a.h.s.a.AbstractJavaKeyStoreProvider
>>> [UnixUserSyncThread] - backing jks path initialized to
>>> file:/etc/ranger/usersync/conf/rangerusersync.jceks
>>> 15 Feb 2024 16:57:02 INFO o.a.r.l.p.LdapUserGroupBuilder
>>> [UnixUserSyncThread] - LdapUserGroupBuilder initialization completed with
>>> -- ldapUrl: ldap://cmb.blabla.org:389, ldapBindDn:
>>> CN=itsme,ou=CACM,ou=utilisateurs,dc=cmb,dc=blabla,dc=org,
>>> ldapBindPassword: * , ldapAuthenticationMechanism: simple,
>>> searchBase: dc=cmb,dc=blabla,dc=org, userSearchBase:
>>> [ou=CACM,ou=utilisateurs,dc=cmb,dc=blabla,dc=org], userSearchScope: 2,
>>> userObjectClass: organizationalPerson, userSearchFilter:
>>> (memberOf=CN=usr_prd,OU=Component,OU=Groupes,DC=blabla,DC=org),
>>> extendedUserSearchFilter: null, userNameAttribute: name,
>>> userSearchAttributes: [postOfficeBox, uSNChanged, name, modifytimestamp,
>>> objectid, userurincipaluame], userGroupNameAttributeSet: [postOfficeBox],
>>> otherUserAttributes: [userurincipaluame], pagedResultsEnabled: true,
>>> pagedResultsSize: 500, groupSearchEnabled: true, groupSearchBase:
>>> [dc=cmb,dc=blabla,dc=org], groupSearchScope: 2, groupObjectClass:
>>> groupofnames, groupSearchFilter: , extendedGroupSearchFilter:
>>> ((|(member={0})(member={1}))), extendedAllGroupsSearchFilter: null,
>>> groupMemberAttributeName: member, groupNameAttribute: cn,
>>> groupSearchAttributes: [uSNChanged, displayname, member, cn,
>>> modifytimestamp, objectid], groupSearchFirstEnabled: true,
>>> userSearchEnabled: true, ldapReferral: ignore
>>> 15 Feb 2024 16:57:02 INFO o.a.r.u.UserGroupSync [UnixUserSyncThread] -
>>> Begin: initial load of user/group from source==>sink
>>> 15 Feb 2024 16:57:02 INFO o.a.r.u.UserGroupSync [UnixUserSyncThread] -
>>> End: initial load of user/group from source==>sink
>>> 15 Feb 2024 16:57:02 INFO o.a.r.u.UserGroupSync [UnixUserSyncThread] -
>>> Done initializing user/group source and sink
>>> 15 Feb 2024 16:57:02 DEBUG o.a.r.u.UserGroupSync [UnixUserSyncThread] -
>>> Sleeping for [360] milliSeconds
>>>
>>> Could you help me identify what's wrong ?
>>> Thanks,
>>>
>>> Loïc CHANEL
>>> Technical leader Big Data
>>> Capgemini (Lyon, France)
>>>
>>
Re: LDAP sync shows no log
Hi Sailaja,
I already restarted usersync since I deleted the user, but nothing
happened. Where is this cache supposed to be stored ?
By the way, SYNC_LDAP_DELTASYNC is set to false in my case, so I guess it
could change the behaviour ?
Thanks,
Loïc CHANEL
Technical leader Big Data
Capgemini (Lyon, France)
Le jeu. 15 févr. 2024 à 23:23, Sailaja Polavarapu
a écrit :
> Ranger Usersync caches the users and groups that are sync'd from LDAP and
> uses this to compute delta for every sync cycle in order to update ranger
> admin with the changes. Initially, during start up, this cache is built
> from the users and groups that are in Ranger admin and is updated only when
> the changes happen at the sync source (like LDAP) for every sync cycle. If
> you delete the user in Ranger admin, usersync is notified with this change
> and this user still exists in usersync cache. In order to re-populate this
> cache from what is updated in ranger admin, you need to restart usersync.
> Hope this helps.
>
> Thanks,
> Sailaja.
>
> On Thu, Feb 15, 2024 at 9:28 AM Loïc CHANEL
> wrote:
>
>> Hi guys,
>>
>> I'm still working on user sync based on LDAP, I'm afraid the sync doesn't
>> work without showing any logs.
>> I performed an initial sync in an older version, and just upgraded to
>> 2.4. Now, to check the usersync is still working, I deleted a user and
>> waited for it to be created again except ... it's not, even though the user
>> is still in the LDAP.
>> In the logs I can see that usersync is trying to reach the LDAP, but even
>> with DEBUG log level I can't either see the user is retrieved, nor there is
>> an error while trying to retrieve it. Here's the only log I get :
>>
>> 15 Feb 2024 16:57:02 INFO o.a.r.u.UserGroupSync [UnixUserSyncThread] -
>> initializing source:
>> org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder
>> 15 Feb 2024 16:57:02 INFO o.a.r.l.p.LdapUserGroupBuilder
>> [UnixUserSyncThread] - LdapUserGroupBuilder initialization started
>> 15 Feb 2024 16:57:02 DEBUG o.a.h.s.a.AbstractJavaKeyStoreProvider
>> [UnixUserSyncThread] - backing jks path initialized to
>> file:/etc/ranger/usersync/conf/rangerusersync.jceks
>> 15 Feb 2024 16:57:02 INFO o.a.r.l.p.LdapUserGroupBuilder
>> [UnixUserSyncThread] - LdapUserGroupBuilder initialization completed with
>> -- ldapUrl: ldap://cmb.blabla.org:389, ldapBindDn:
>> CN=itsme,ou=CACM,ou=utilisateurs,dc=cmb,dc=blabla,dc=org,
>> ldapBindPassword: * , ldapAuthenticationMechanism: simple,
>> searchBase: dc=cmb,dc=blabla,dc=org, userSearchBase:
>> [ou=CACM,ou=utilisateurs,dc=cmb,dc=blabla,dc=org], userSearchScope: 2,
>> userObjectClass: organizationalPerson, userSearchFilter:
>> (memberOf=CN=usr_prd,OU=Component,OU=Groupes,DC=blabla,DC=org),
>> extendedUserSearchFilter: null, userNameAttribute: name,
>> userSearchAttributes: [postOfficeBox, uSNChanged, name, modifytimestamp,
>> objectid, userurincipaluame], userGroupNameAttributeSet: [postOfficeBox],
>> otherUserAttributes: [userurincipaluame], pagedResultsEnabled: true,
>> pagedResultsSize: 500, groupSearchEnabled: true, groupSearchBase:
>> [dc=cmb,dc=blabla,dc=org], groupSearchScope: 2, groupObjectClass:
>> groupofnames, groupSearchFilter: , extendedGroupSearchFilter:
>> ((|(member={0})(member={1}))), extendedAllGroupsSearchFilter: null,
>> groupMemberAttributeName: member, groupNameAttribute: cn,
>> groupSearchAttributes: [uSNChanged, displayname, member, cn,
>> modifytimestamp, objectid], groupSearchFirstEnabled: true,
>> userSearchEnabled: true, ldapReferral: ignore
>> 15 Feb 2024 16:57:02 INFO o.a.r.u.UserGroupSync [UnixUserSyncThread] -
>> Begin: initial load of user/group from source==>sink
>> 15 Feb 2024 16:57:02 INFO o.a.r.u.UserGroupSync [UnixUserSyncThread] -
>> End: initial load of user/group from source==>sink
>> 15 Feb 2024 16:57:02 INFO o.a.r.u.UserGroupSync [UnixUserSyncThread] -
>> Done initializing user/group source and sink
>> 15 Feb 2024 16:57:02 DEBUG o.a.r.u.UserGroupSync [UnixUserSyncThread] -
>> Sleeping for [360] milliSeconds
>>
>> Could you help me identify what's wrong ?
>> Thanks,
>>
>> Loïc CHANEL
>> Technical leader Big Data
>> Capgemini (Lyon, France)
>>
>
LDAP sync shows no log
Hi guys,
I'm still working on user sync based on LDAP, I'm afraid the sync doesn't
work without showing any logs.
I performed an initial sync in an older version, and just upgraded to 2.4.
Now, to check the usersync is still working, I deleted a user and waited
for it to be created again except ... it's not, even though the user is
still in the LDAP.
In the logs I can see that usersync is trying to reach the LDAP, but even
with DEBUG log level I can't either see the user is retrieved, nor there is
an error while trying to retrieve it. Here's the only log I get :
15 Feb 2024 16:57:02 INFO o.a.r.u.UserGroupSync [UnixUserSyncThread] -
initializing source:
org.apache.ranger.ldapusersync.process.LdapUserGroupBuilder
15 Feb 2024 16:57:02 INFO o.a.r.l.p.LdapUserGroupBuilder
[UnixUserSyncThread] - LdapUserGroupBuilder initialization started
15 Feb 2024 16:57:02 DEBUG o.a.h.s.a.AbstractJavaKeyStoreProvider
[UnixUserSyncThread] - backing jks path initialized to
file:/etc/ranger/usersync/conf/rangerusersync.jceks
15 Feb 2024 16:57:02 INFO o.a.r.l.p.LdapUserGroupBuilder
[UnixUserSyncThread] - LdapUserGroupBuilder initialization completed with
-- ldapUrl: ldap://cmb.blabla.org:389, ldapBindDn:
CN=itsme,ou=CACM,ou=utilisateurs,dc=cmb,dc=blabla,dc=org,
ldapBindPassword: * , ldapAuthenticationMechanism: simple,
searchBase: dc=cmb,dc=blabla,dc=org, userSearchBase:
[ou=CACM,ou=utilisateurs,dc=cmb,dc=blabla,dc=org], userSearchScope: 2,
userObjectClass: organizationalPerson, userSearchFilter:
(memberOf=CN=usr_prd,OU=Component,OU=Groupes,DC=blabla,DC=org),
extendedUserSearchFilter: null, userNameAttribute: name,
userSearchAttributes: [postOfficeBox, uSNChanged, name, modifytimestamp,
objectid, userurincipaluame], userGroupNameAttributeSet: [postOfficeBox],
otherUserAttributes: [userurincipaluame], pagedResultsEnabled: true,
pagedResultsSize: 500, groupSearchEnabled: true, groupSearchBase:
[dc=cmb,dc=blabla,dc=org], groupSearchScope: 2, groupObjectClass:
groupofnames, groupSearchFilter: , extendedGroupSearchFilter:
((|(member={0})(member={1}))), extendedAllGroupsSearchFilter: null,
groupMemberAttributeName: member, groupNameAttribute: cn,
groupSearchAttributes: [uSNChanged, displayname, member, cn,
modifytimestamp, objectid], groupSearchFirstEnabled: true,
userSearchEnabled: true, ldapReferral: ignore
15 Feb 2024 16:57:02 INFO o.a.r.u.UserGroupSync [UnixUserSyncThread] -
Begin: initial load of user/group from source==>sink
15 Feb 2024 16:57:02 INFO o.a.r.u.UserGroupSync [UnixUserSyncThread] -
End: initial load of user/group from source==>sink
15 Feb 2024 16:57:02 INFO o.a.r.u.UserGroupSync [UnixUserSyncThread] -
Done initializing user/group source and sink
15 Feb 2024 16:57:02 DEBUG o.a.r.u.UserGroupSync [UnixUserSyncThread] -
Sleeping for [360] milliSeconds
Could you help me identify what's wrong ?
Thanks,
Loïc CHANEL
Technical leader Big Data
Capgemini (Lyon, France)
Re: [Usersync] Fetch several attributes
Hi Sailaja, >From the look of it, I'd say it could work for my use case. Do you know where I could find some documentation about using that feature ? Thanks, Loïc Le mer. 14 févr. 2024 à 18:05, Sailaja Polavarapu a écrit : > Hi Loïc CHANEL, > Syncing extra attributes from AD/LDAP is partly supported as part of > RANGER-2697 <https://issues.apache.org/jira/browse/RANGER-2697>. Can > you please check it out and see if this works for your use case? It is not > exactly what you are asking for, but in Usersync, there is an option to > sync other attributes which are stored as key-value pairs in > the otherAttributes field for each user/group. > - Sailaja. > > On Wed, Feb 14, 2024 at 8:00 AM Loïc CHANEL > wrote: > >> Hi guys, >> >> I understand the feature is still not available at the moment, so 2 >> questions : >> - I gave a look at the current code, I don't think the development of >> such a feature would be too hard. If I develop it, could it be merged >> easily ? >> - Is there a way to modify the users post-import in order to enrich the >> first name/last name/email fields (manually or with an API) ? >> >> Thanks for your help, >> >> >> Loïc CHANEL >> Technical leader Big Data >> Capgemini (Lyon, France) >> >> >> Le jeu. 8 févr. 2024 à 12:30, Loïc CHANEL >> a écrit : >> >>> Hi guys, >>> >>> As I'm pulling Ranger users from LDAP, I would like not only to pull the >>> username from there but also the first and last name, and most importantly >>> the email address of the LDAP user. This led me to an old Jira >>> <https://issues.apache.org/jira/browse/RANGER-567> that asks for just >>> that feature (the person who answers has an oddly familiar first name), >>> which makes me think it has not been implemented yet. >>> Am I correct ? If so, how hard would it be to develop and maintain that >>> feature ? >>> In parallel, is there a way to modify the users post-import in order to >>> enrich the first name/last name/email fields ? >>> Thanks for your help, >>> >>> >>> Loïc CHANEL >>> Technical leader Big Data >>> Capgemini (Lyon, France) >>> >>
Re: [Usersync] Fetch several attributes
Hi guys, I understand the feature is still not available at the moment, so 2 questions : - I gave a look at the current code, I don't think the development of such a feature would be too hard. If I develop it, could it be merged easily ? - Is there a way to modify the users post-import in order to enrich the first name/last name/email fields (manually or with an API) ? Thanks for your help, Loïc CHANEL Technical leader Big Data Capgemini (Lyon, France) Le jeu. 8 févr. 2024 à 12:30, Loïc CHANEL a écrit : > Hi guys, > > As I'm pulling Ranger users from LDAP, I would like not only to pull the > username from there but also the first and last name, and most importantly > the email address of the LDAP user. This led me to an old Jira > <https://issues.apache.org/jira/browse/RANGER-567> that asks for just > that feature (the person who answers has an oddly familiar first name), > which makes me think it has not been implemented yet. > Am I correct ? If so, how hard would it be to develop and maintain that > feature ? > In parallel, is there a way to modify the users post-import in order to > enrich the first name/last name/email fields ? > Thanks for your help, > > > Loïc CHANEL > Technical leader Big Data > Capgemini (Lyon, France) >
[Usersync] Fetch several attributes
Hi guys, As I'm pulling Ranger users from LDAP, I would like not only to pull the username from there but also the first and last name, and most importantly the email address of the LDAP user. This led me to an old Jira <https://issues.apache.org/jira/browse/RANGER-567> that asks for just that feature (the person who answers has an oddly familiar first name), which makes me think it has not been implemented yet. Am I correct ? If so, how hard would it be to develop and maintain that feature ? In parallel, is there a way to modify the users post-import in order to enrich the first name/last name/email fields ? Thanks for your help, Loïc CHANEL Technical leader Big Data Capgemini (Lyon, France)
[Usersync] LDAP Anonymous bind
Hi guys, Has anyone been able to make LDAP sync work with an anonymous bind with latest versions of Ranger ? On the Apache documentation I made 7 years ago <https://cwiki.apache.org/confluence/display/RANGER/Configure+Ranger+UserSync+for+LDAP> I could see that anonymous bind was not supported anymore, but as I'm looking into the latest version of install.properties file <https://github.com/apache/ranger/blob/master/unixauthservice/scripts/install.properties> I can see in line 104 and 105 that a LDAP password has to be provided "unless anonymous search is allowed". The thing is, I get an installation error when I leave this field blank. Therefore I'm wondering if there is a way to make usersync work with anonymous bind. Thanks for your help, Loïc CHANEL Technical leader Big Data Capgemini (Lyon, France)
Re: trouble with hdfs plugin
Nicolas,
I haven't been on Hadoop for a while, but I remember that we could see in
the component logs that they were trying to pull the Ranger policies from
Ranger API on a regular basis (something like every 30 seconds).
So 2 things I would check :
- Is Ranger plugin activated on HDFS ? (If you use Ambari, then it's easy
to check this out in Ranger tab).
- Can it pull the Ranger policies from Ranger API ? (Take a look at HDFS
logs, both on the NameNode and on DataNodes)
What surprises me is that the Audit logs you show are not about /tmp folder
denial, so my guess is that Ranger is not aware of your activities.
Best regards,
Loïc CHANEL
System Big Data engineer
Vision 360 Degrés (Lyon, France)
Le lun. 9 mars 2020 à 15:46, Nicolas Paris a
écrit :
>
> Loïc Chanel writes:
> > Do you see in the logs that Spark is able to pull the policies from
> Ranger
> > API ?
>
> I did look to several log files. On the spark side, the previous email
> show the logs. On the ranger side, the xa_portal.log does not provide
> any mention of hdfs attempt to be read by the user.
>
> The hdfs://ranger/audit/hdfs/*.log contains such entry:
>
> ```json
> {"repoType":1,"repo":"CLUSTER_hadoop","reqUser":"nicolas","evtTime":"2020-03-09
> 13:50:08.389","access":"WRITE","resource":"/app-logs/nicolas...","resType":"path","action":"write","result":1,"policy":-1,"reason":"/app-logs/nicolas/logs-ifile/application_1583593832792_0067","enforcer":"hadoop-acl","cliIP":"IP","agentHost":"hostname","logType":"RangerAudit","id":"da76751f-af19-49f1-8d47-f52f7e68d593-6700046","seq_num":10150745,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"CLUSTER"}
> ```
> I don't find any mention of enforcer:ranger* in the audit logs
>
>
>
> > Either way, could you please share the policies you defined in Ranger for
> > your user ?
>
> The rule added are:
> Policy ID: 1
> policy type: Access
> policy name: all pth name | Enabled
> ressource Path: /*| recursive
> audit logging: yes
> select user: hdfs, rangerlookup, ambari-qa
> permission: read,write,execute
> delegate admin: yes
>
> Policy ID: 2
> policy type: Access
> policy name: kms-audit-path name | Enabled
> ressource Path: /ranger/audit/kms| recursive
> audit logging: yes
> select user: keyadmin
> permission: read,write,execute
>
> Policy ID: 3
> policy type: Access
> policy name: my policy name | Enabled
> ressource Path: /tmp| recursive
> audit logging: yes
> select user: nicolas
> permission: read,write,execute
>
> Thanks
> --
> nicolas paris
>
Re: trouble with hdfs plugin
Hi Nicolas, Do you see in the logs that Spark is able to pull the policies from Ranger API ? Either way, could you please share the policies you defined in Ranger for your user ? Thanks, Loïc CHANEL System Big Data engineer Vision 360 Degrés (Lyon, France) Le lun. 9 mars 2020 à 13:41, Nicolas Paris a écrit : > Hello, > > I am using Apache Ranger 0.7.0. IFAIK, I cannot trivially upgrade to > newer version. > > I have setup the hive plugin and both audit and authorisation works > fine. > > I tried to setup the hdfs plugin on a kerberised hdfs cluster. Most of > the setup works ok: > - the connection test to hdfs is successfull > - I can see the hdfs audit json populating the hdfs folder > - I also have added some rules (in particular read for user nicolas to > folder `/tmp/test`) > > However, the ranger rules does not look to apply: still the hdfs posix > authorisation apply or the hdfs acls. > > Here is a typical stack trace I get from apache spark when I ask for > access to an hdfs folder (revealing RangerHdfsAuthorizer is active): > > > Caused by: org.apache.hadoop.ipc.RemoteException: Permission denied: > user=nicolas, access=READ_EXECUTE, inode="/tmp/test":hdfs:hdfs:drwxrwx--- > > at > org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:353) > > at > org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:252) > > at > org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkDefaultEnforcer(RangerHdfsAuthorizer.java:427) > > at > org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkPermission(RangerHdfsAuthorizer.java:303) > > at > org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190) > > at > org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1950) > > at > org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1934) > > at > org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPathAccess(FSDirectory.java:1908) > > at > org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getListingInt(FSDirStatAndListingOp.java:77) > > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getListing(FSNamesystem.java:4820) > > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getListing(NameNodeRpcServer.java:1124) > > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getListing(ClientNamenodeProtocolServerSideTranslatorPB.java:657) > > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640) > > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) > > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351) > > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAs(Subject.java:422) > > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869) > > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2347) > > Thanks for your help, > > -- > nicolas paris >
Re: Hadoop version
Yes, this is just for Hortonworks distribution but you can assume that as far as Ranger is concerned some backward compliancy have been implemented, so Ranger 0.6.0 supports Hadoop 2.7.3 and older versions of Hadoop, and Ranger 1.0 supports Hadoop 3.1.0 and older ... but Bosco know this better than I do so I'll let him confirm. Best regards, Loïc CHANEL System Big Data engineer Vision 360 Degrés (Lyon, France) Le mer. 16 janv. 2019 à 14:58, Odon Copon a écrit : > Oh true, didn't see that Hadoop column. > But this is just for Hortonworks distribution, right? I can see per > example, in Ranger POM file on 1.2.0 release that contains a reference to > Hadoop 2.7.1, so I'm assuming Apache Ranger 1.2.0 supports Hadoop 2.7.1. > > On Wed, 16 Jan 2019 at 13:49, Loïc Chanel > wrote: > >> Hi, >> >> I think there was a misunderstanding. What you see on the left border is >> the HDP version. As HDP is a stack (a product packaged by Hortonworks), it >> comes with a lot of components (Hive, HBase, Spark ...) including Ranger >> and Hadoop (in the HDP core column, on the left side of the picture). >> Therefore, you can see on the picture that Hadoop 2.7.3 is supported by >> both Ranger 0.6.0 and Ranger 0.7.0 versions, as they are packaged in the >> same HDP stack. >> >> Hope this helps. >> >> >> Loïc CHANEL >> System Big Data engineer >> Vision 360 Degrés (Lyon, France) >> >> >> Le mer. 16 janv. 2019 à 13:38, Odon Copon a écrit : >> >>> Thanks Loic! >>> That's definitely really helpful. But let me confirm and clarify what >>> I'm seeing there: >>> - ranger 0.6 just works with hadoop 2.5.* >>> - ranger 0.7 just works with hadoop 2.6.4 and 2.6.5 >>> - ranger 1.1.0 just works with hadoop 3.0.0 >>> - ranger 1.2.0 just works with hadoop 3.1.* >>> >>> Is this correct? >>> So Hadoop 2.7.* is not supported? >>> >>> >>> On Wed, 16 Jan 2019 at 12:07, Loïc Chanel >>> wrote: >>> >>>> Hi Odon, >>>> >>>> Here it is : >>>> [image: image.png] >>>> (https://fr.hortonworks.com/products/data-platforms/hdp/ ) >>>> >>>> Best regards, >>>> >>>> >>>> Loïc CHANEL >>>> System Big Data engineer >>>> Vision 360 Degrés (Lyon, France) >>>> >>>> >>>> Le mer. 16 janv. 2019 à 11:53, Odon Copon a >>>> écrit : >>>> >>>>> Thanks! >>>>> Do you know if there's a table with Hadoop versions and the >>>>> corresponding Ranger version? >>>>> >>>>> On Wed, 16 Jan 2019 at 10:41, Don Bosco Durai >>>>> wrote: >>>>> >>>>>> Ranger implements the interface published by the service component. >>>>>> So every Ranger version can only support the components which it is built >>>>>> against. >>>>>> >>>>>> >>>>>> >>>>>> For older version of components, you will have to use older version >>>>>> of Ranger which supports it. >>>>>> >>>>>> >>>>>> >>>>>> Bosco >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> *From: *Odon Copon >>>>>> *Reply-To: * >>>>>> *Date: *Wednesday, January 16, 2019 at 2:03 AM >>>>>> *To: * >>>>>> *Subject: *Hadoop version >>>>>> >>>>>> >>>>>> >>>>>> Hi, >>>>>> >>>>>> Is there any requirement on the Hadoop version supported by Apache >>>>>> Ranger? >>>>>> I can see the pom.xml contains a 2.7.1 pinned version for Hadoop, but >>>>>> my question is, would it work on older versions like 2.6? Or is a hard >>>>>> requirement to be >= 2.7.1? >>>>>> >>>>>> Thanks. >>>>>> >>>>>
Re: Apache Ranger with Hive 2.3.2
Hello, Just to be clear, Ranger pull user data from LDAP. Therefore, the problem should be at that level. Did you configure Ranger UserSync to pull informations from your LDAP ? Regards, Loïc Loïc CHANEL System Big Data engineer Vision 360 Degrés (Lyon, France) 2018-05-15 7:51 GMT+02:00 Taher Koitawala <taher.koitaw...@gslab.com>: > Hi All, > Can someone please tell me which version of Ranger will be > compatible with hive 2.3.2? I am currently testing with Ranger 0.7.1, > however, I see a lot of warnings in hive logs. > > And also why do I see this warning in hive logs? And what does it > particularly mean? > > WARN [main([])]: authorizer.RangerHiveAuthorizer > (RangerHiveAuthorizer.java:filterListCmdObjects(455)) - > filterListCmdObjects: user information not available > > Currently, hive 2.3.2 is set up on SSL with LDAP and Kerberos as > authentication systems. > > Ranger, however, for now, is not running on Kerberos > > > Regards, > Taher Koitawala > GS Lab Pune > +91 8407979163 >
Re: Tag based policy doesn't work
Hi Fabien, Can you provide more details on waht doesn't work ? I mean, you create a policy, but it is not enforced, right ? Can you see in HiveServer logs that Hive asks Ranger for an authorization to execute your request ? Thanks, Loïc Loïc CHANEL System Big Data engineer MS - Worldline Analytics Platform - Worldline (Villeurbanne, France) 2017-06-06 14:51 GMT+02:00 fabien VIROT <fabienfo...@hotmail.fr>: > Hello, > > > I got problems with tag based policy on ranger, when I create policy based > on Atlas tag they doesn't work on Hive. > > I have no log error (in Debug mode) on ranger tagsync and in Atlas. > > I'm using beeline to send request to Hive. > > > Have you any idea of where the problem is ?? > > > Thanks for your help > > > Fabien VIROT >
