Re: [Freeipa-users] ipa migrate-ds failing when more than 1 namingcontext is available
I agree with Simo. As far as I understand I dont have a way around this but to change the source. Ok for me but perhaps a bit much for others... Btw, I haven't had time to test the proposed source changes. Rgds, Siggi -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. Simo Sorce wrote: On Mon, 2012-01-30 at 10:56 -0500, Rob Crittenden wrote: > Dmitri Pal wrote: > > On 01/30/2012 09:23 AM, Simo Sorce wrote: > >> On Mon, 2012-01-30 at 09:06 -0500, Rob Crittenden wrote: > >>> Like I said, this error is triggered before ignore is evaluated so > >>> if > >>> an unknown binary attribute is getting decoded it will cause this > >>> failure. The only solutions we have right now is to either load the > >>> schema into IPA temporarily for the migration, rremove it on the > >>> remote > >>> side or you could modify the query we make to find the remote entries > >>> to > >>> pull only certain attributes. This last one would be tricky to get > >>> right. > >>> > >>> The code looks like: > >>> > >>> (entries, truncated) = ds_ldap.find_entries( > >>> search_filter, ['*'], > >>> search_bases[ldap_obj_name], > >>> ds_ldap.SCOPE_ONELEVEL, > >>> time_limit=0, size_limit=-1, > >>> search_refs=True # migrated DS may contain > >>> search references > >>> ) > >>> > >>> You'd want to replace ['*'] with ['attr1','attr2','attr3',...]. It > >>> would > >>> be a rather long list and would need to cover both users and groups. > >>> > >> TBH I think we should turn the code around and do this by default. > >> We have no idea how to manage extra attributes anyway so we shouldn't > >> get them all, only get those we understand. And turn the exclusion list > >> into an inclusion list, so that if someone wants to import more data > >> because they added additional schema to FreeIPA they are free to do so. > >> The current way looks brittle. > >> > >> Simo. > >> > > Agree, we need to open a BZ and ticket on this one. > > > > Oh I don't know. The reason we did it this way was to specifically put > into the user's face those attributes that aren't being migrated. This > way we don't find out much after the fact that some things weren't > migrated properly forcing them to re-migrate. I'd certainly rather have > a little pain at the beginning of the process and know I have everything > I need rather than days/weeks/months later and realize something > important was missed. I understand the intent, but I think the current method causes more pain then necessary. If admins have important data they want to preserve they better explicitly include it in the import list. And check the data was properly imported. I do not think we need to make life of good admins difficult in order to try to salvage bad admins that do not even check the migration imported all data they needed. Simo. -- Simo Sorce * Red Hat, Inc * New York _ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] ipa migrate-ds failing when more than 1 namingcontext is available
On Mon, 2012-01-30 at 10:56 -0500, Rob Crittenden wrote: > Dmitri Pal wrote: > > On 01/30/2012 09:23 AM, Simo Sorce wrote: > >> On Mon, 2012-01-30 at 09:06 -0500, Rob Crittenden wrote: > >>> Like I said, this error is triggered before ignore is evaluated so > >>> if > >>> an unknown binary attribute is getting decoded it will cause this > >>> failure. The only solutions we have right now is to either load the > >>> schema into IPA temporarily for the migration, rremove it on the > >>> remote > >>> side or you could modify the query we make to find the remote entries > >>> to > >>> pull only certain attributes. This last one would be tricky to get > >>> right. > >>> > >>> The code looks like: > >>> > >>> (entries, truncated) = ds_ldap.find_entries( > >>> search_filter, ['*'], > >>> search_bases[ldap_obj_name], > >>> ds_ldap.SCOPE_ONELEVEL, > >>> time_limit=0, size_limit=-1, > >>> search_refs=True# migrated DS may contain > >>> search references > >>> ) > >>> > >>> You'd want to replace ['*'] with ['attr1','attr2','attr3',...]. It > >>> would > >>> be a rather long list and would need to cover both users and groups. > >>> > >> TBH I think we should turn the code around and do this by default. > >> We have no idea how to manage extra attributes anyway so we shouldn't > >> get them all, only get those we understand. And turn the exclusion list > >> into an inclusion list, so that if someone wants to import more data > >> because they added additional schema to FreeIPA they are free to do so. > >> The current way looks brittle. > >> > >> Simo. > >> > > Agree, we need to open a BZ and ticket on this one. > > > > Oh I don't know. The reason we did it this way was to specifically put > into the user's face those attributes that aren't being migrated. This > way we don't find out much after the fact that some things weren't > migrated properly forcing them to re-migrate. I'd certainly rather have > a little pain at the beginning of the process and know I have everything > I need rather than days/weeks/months later and realize something > important was missed. I understand the intent, but I think the current method causes more pain then necessary. If admins have important data they want to preserve they better explicitly include it in the import list. And check the data was properly imported. I do not think we need to make life of good admins difficult in order to try to salvage bad admins that do not even check the migration imported all data they needed. Simo. -- Simo Sorce * Red Hat, Inc * New York ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] ipa migrate-ds failing when more than 1 namingcontext is available
On 01/30/2012 10:56 AM, Rob Crittenden wrote: > Dmitri Pal wrote: >> On 01/30/2012 09:23 AM, Simo Sorce wrote: >>> On Mon, 2012-01-30 at 09:06 -0500, Rob Crittenden wrote: Like I said, this error is triggered before ignore is evaluated so if an unknown binary attribute is getting decoded it will cause this failure. The only solutions we have right now is to either load the schema into IPA temporarily for the migration, rremove it on the remote side or you could modify the query we make to find the remote entries to pull only certain attributes. This last one would be tricky to get right. The code looks like: (entries, truncated) = ds_ldap.find_entries( search_filter, ['*'], search_bases[ldap_obj_name], ds_ldap.SCOPE_ONELEVEL, time_limit=0, size_limit=-1, search_refs=True# migrated DS may contain search references ) You'd want to replace ['*'] with ['attr1','attr2','attr3',...]. It would be a rather long list and would need to cover both users and groups. >>> TBH I think we should turn the code around and do this by default. >>> We have no idea how to manage extra attributes anyway so we shouldn't >>> get them all, only get those we understand. And turn the exclusion list >>> into an inclusion list, so that if someone wants to import more data >>> because they added additional schema to FreeIPA they are free to do so. >>> The current way looks brittle. >>> >>> Simo. >>> >> Agree, we need to open a BZ and ticket on this one. >> > > Oh I don't know. The reason we did it this way was to specifically put > into the user's face those attributes that aren't being migrated. This > way we don't find out much after the fact that some things weren't > migrated properly forcing them to re-migrate. I'd certainly rather > have a little pain at the beginning of the process and know I have > everything I need rather than days/weeks/months later and realize > something important was missed. > > rob May be we should have a dry run option then or something like migrate-ds-test that will return two lists: List of all attributes detected and the list of attributes that we actually can migrate. For the ones we can't migrate we should explain why. Then migrate-ds should accept the second list as an input overriding the default "*". How about that? -- Thank you, Dmitri Pal Sr. Engineering Manager IPA project, Red Hat Inc. --- Looking to carve out IT costs? www.redhat.com/carveoutcosts/ ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] ipa migrate-ds failing when more than 1 namingcontext is available
Dmitri Pal wrote: On 01/30/2012 09:23 AM, Simo Sorce wrote: On Mon, 2012-01-30 at 09:06 -0500, Rob Crittenden wrote: Like I said, this error is triggered before ignore is evaluated so if an unknown binary attribute is getting decoded it will cause this failure. The only solutions we have right now is to either load the schema into IPA temporarily for the migration, rremove it on the remote side or you could modify the query we make to find the remote entries to pull only certain attributes. This last one would be tricky to get right. The code looks like: (entries, truncated) = ds_ldap.find_entries( search_filter, ['*'], search_bases[ldap_obj_name], ds_ldap.SCOPE_ONELEVEL, time_limit=0, size_limit=-1, search_refs=True# migrated DS may contain search references ) You'd want to replace ['*'] with ['attr1','attr2','attr3',...]. It would be a rather long list and would need to cover both users and groups. TBH I think we should turn the code around and do this by default. We have no idea how to manage extra attributes anyway so we shouldn't get them all, only get those we understand. And turn the exclusion list into an inclusion list, so that if someone wants to import more data because they added additional schema to FreeIPA they are free to do so. The current way looks brittle. Simo. Agree, we need to open a BZ and ticket on this one. Oh I don't know. The reason we did it this way was to specifically put into the user's face those attributes that aren't being migrated. This way we don't find out much after the fact that some things weren't migrated properly forcing them to re-migrate. I'd certainly rather have a little pain at the beginning of the process and know I have everything I need rather than days/weeks/months later and realize something important was missed. rob ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] ipa migrate-ds failing when more than 1 namingcontext is available
On 01/30/2012 09:23 AM, Simo Sorce wrote: > On Mon, 2012-01-30 at 09:06 -0500, Rob Crittenden wrote: >> Like I said, this error is triggered before ignore is evaluated so >> if >> an unknown binary attribute is getting decoded it will cause this >> failure. The only solutions we have right now is to either load the >> schema into IPA temporarily for the migration, rremove it on the >> remote >> side or you could modify the query we make to find the remote entries >> to >> pull only certain attributes. This last one would be tricky to get >> right. >> >> The code looks like: >> >> (entries, truncated) = ds_ldap.find_entries( >> search_filter, ['*'], >> search_bases[ldap_obj_name], >> ds_ldap.SCOPE_ONELEVEL, >> time_limit=0, size_limit=-1, >> search_refs=True# migrated DS may contain >> search references >> ) >> >> You'd want to replace ['*'] with ['attr1','attr2','attr3',...]. It >> would >> be a rather long list and would need to cover both users and groups. >> > TBH I think we should turn the code around and do this by default. > We have no idea how to manage extra attributes anyway so we shouldn't > get them all, only get those we understand. And turn the exclusion list > into an inclusion list, so that if someone wants to import more data > because they added additional schema to FreeIPA they are free to do so. > The current way looks brittle. > > Simo. > Agree, we need to open a BZ and ticket on this one. -- Thank you, Dmitri Pal Sr. Engineering Manager IPA project, Red Hat Inc. --- Looking to carve out IT costs? www.redhat.com/carveoutcosts/ ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] ipa migrate-ds failing when more than 1 namingcontext is available
On Mon, 2012-01-30 at 09:06 -0500, Rob Crittenden wrote: > > Like I said, this error is triggered before ignore is evaluated so > if > an unknown binary attribute is getting decoded it will cause this > failure. The only solutions we have right now is to either load the > schema into IPA temporarily for the migration, rremove it on the > remote > side or you could modify the query we make to find the remote entries > to > pull only certain attributes. This last one would be tricky to get > right. > > The code looks like: > > (entries, truncated) = ds_ldap.find_entries( > search_filter, ['*'], > search_bases[ldap_obj_name], > ds_ldap.SCOPE_ONELEVEL, > time_limit=0, size_limit=-1, > search_refs=True# migrated DS may contain > search references > ) > > You'd want to replace ['*'] with ['attr1','attr2','attr3',...]. It > would > be a rather long list and would need to cover both users and groups. > TBH I think we should turn the code around and do this by default. We have no idea how to manage extra attributes anyway so we shouldn't get them all, only get those we understand. And turn the exclusion list into an inclusion list, so that if someone wants to import more data because they added additional schema to FreeIPA they are free to do so. The current way looks brittle. Simo. -- Simo Sorce * Red Hat, Inc * New York ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] ipa migrate-ds failing when more than 1 namingcontext is available
Sigbjorn Lie wrote: On 01/27/2012 07:42 PM, Rob Crittenden wrote: Sigbjorn Lie wrote: On 01/27/2012 06:15 PM, Sigbjorn Lie wrote: On 01/27/2012 03:55 PM, Rob Crittenden wrote: Sigbjorn Lie wrote: On Fri, January 27, 2012 15:37, Rob Crittenden wrote: Stephen Gallagher wrote: On Fri, 2012-01-27 at 15:11 +0100, Sigbjorn Lie wrote: Hi The first naming context returned from the LDAP server is always chosen when using migrate-ds. This makes my import fail when I attempt to import users and groups from a previous LDAP server having more than 1 naming contexts available. The migrate-ds script should accept an option to specify what base_dn I would like to import from. Is there such an option today? I cannot find it... Not currently. I noticed this earlier in the week and opened a ticket on it, https://fedorahosted.org/freeipa/ticket/2314 Just to add to this request, if the original LDAP server has a defaultNamingContext attribute, it should be honored for auto-detecting which base to migrate. I'll update the 2314 to ensure we don't forget about this. 389-ds just added support for defaultNamingContext. Ok, thank you. Anything I can do to work around this issue today? I suppose there is just a file that need to be hacked to set a set a value instead of the auto-detected value... ? /usr/lib/python*/site-packages/ipalib/plugins/migration.py ~line 620 you'll see a block starting with the comment "retrieve DS base DN". Comment out the next 8 lines by prefixing them with # (these query to get the namingContext then pull the first value out). Add: ds_base_dn = 'dc=yourbasedn,dc=com' Alternatively you could always just add the above line to override what is detected. Commenting out just saves an LDAP lookup. Restart Apache. I already found that file and did that earlier today, however I was restarting tomcat6, not httpd... my bad. :) I have to specify --group-objectclass=posixGroup to get groups imported, that's fine. But I only get a few users imported. I see that by default it seem to be looking for objectclass=person. Only a few user accounts have that objectclass associated, so I add --user-objectclass=posixAccount as all users have this objectclass associated with their account. $ ipa migrate-ds --user-container='ou=people' --group-container='ou=group' --bind-dn='cn=directory manager' --user-objectclass=account --group-objectclass=posixGroup --schema=RFC2307 --continue ldap://ldapserver:399 ipa: ERROR: an internal error has occurred Not good. I look in the /var/log/httpd/error_log file, and I find: [Fri Jan 27 18:12:51 2012] [error] ipa: INFO: admin@NONE: ping(): SUCCESS [Fri Jan 27 18:12:52 2012] [error] ipa: ERROR: non-public: UnicodeDecodeError: 'utf8' codec can't decode byte 0xe5 in position 1: invalid continuation byte [Fri Jan 27 18:12:52 2012] [error] Traceback (most recent call last): [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipaserver/rpcserver.py", line 228, in wsgi_execute [Fri Jan 27 18:12:52 2012] [error] result = self.Command[name](*args, **options) [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/frontend.py", line 432, in __call__ [Fri Jan 27 18:12:52 2012] [error] ret = self.run(*args, **options) [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/frontend.py", line 738, in run [Fri Jan 27 18:12:52 2012] [error] return self.execute(*args, **options) [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/plugins/migration.py", line 634, in execute [Fri Jan 27 18:12:52 2012] [error] ldap, config, ds_ldap, ds_base_dn, options [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/plugins/migration.py", line 513, in migrate [Fri Jan 27 18:12:52 2012] [error] search_refs=True # migrated DS may contain search references [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 188, in new_f [Fri Jan 27 18:12:52 2012] [error] return f(*new_args, **kwargs) [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 199, in new_f [Fri Jan 27 18:12:52 2012] [error] return args[0].decode(f(*args, **kwargs)) [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 139, in decode [Fri Jan 27 18:12:52 2012] [error] return tuple(self.decode(m) for m in var) [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 139, in [Fri Jan 27 18:12:52 2012] [error] return tuple(self.decode(m) for m in var) [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 137, in decode [Fri Jan 27 18:12:52 2012] [error] return [self.decode(m) for m in var] [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 139, in decode [Fri Jan 27 18:12:52 2012] [error] return tuple(self.decode(m) for m in var) [
Re: [Freeipa-users] ipa migrate-ds failing when more than 1 namingcontext is available
Stephen Gallagher wrote: On Fri, 2012-01-27 at 13:42 -0500, Rob Crittenden wrote: This came up yesterday internally too. I don't believe a bug or ticket has been filed yet. My best guess on what is happening, based on what I saw with our own case, is this: A migrated attribute is coming in that IPA doesn't know about. We default to treating all attributes that don't require special treatment (like binary values) as utf-8. I'm guessing that an unknown binary attribute is being migrated, we try to utf-8 encode it and kablooey. There is no easy workaround for the above problem right now because that happens before the --ignore* options are considered. Why don't we just base64-encode unknown attributes instead of treating them like UTF-8? The LDAP server is *supposed* to be able to handle this implicitly. For migration we're going to have to reject them outright because the migration will fail eventually anyway. rob ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] ipa migrate-ds failing when more than 1 namingcontext is available
On 01/27/2012 07:42 PM, Rob Crittenden wrote: Sigbjorn Lie wrote: On 01/27/2012 06:15 PM, Sigbjorn Lie wrote: On 01/27/2012 03:55 PM, Rob Crittenden wrote: Sigbjorn Lie wrote: On Fri, January 27, 2012 15:37, Rob Crittenden wrote: Stephen Gallagher wrote: On Fri, 2012-01-27 at 15:11 +0100, Sigbjorn Lie wrote: Hi The first naming context returned from the LDAP server is always chosen when using migrate-ds. This makes my import fail when I attempt to import users and groups from a previous LDAP server having more than 1 naming contexts available. The migrate-ds script should accept an option to specify what base_dn I would like to import from. Is there such an option today? I cannot find it... Not currently. I noticed this earlier in the week and opened a ticket on it, https://fedorahosted.org/freeipa/ticket/2314 Just to add to this request, if the original LDAP server has a defaultNamingContext attribute, it should be honored for auto-detecting which base to migrate. I'll update the 2314 to ensure we don't forget about this. 389-ds just added support for defaultNamingContext. Ok, thank you. Anything I can do to work around this issue today? I suppose there is just a file that need to be hacked to set a set a value instead of the auto-detected value... ? /usr/lib/python*/site-packages/ipalib/plugins/migration.py ~line 620 you'll see a block starting with the comment "retrieve DS base DN". Comment out the next 8 lines by prefixing them with # (these query to get the namingContext then pull the first value out). Add: ds_base_dn = 'dc=yourbasedn,dc=com' Alternatively you could always just add the above line to override what is detected. Commenting out just saves an LDAP lookup. Restart Apache. I already found that file and did that earlier today, however I was restarting tomcat6, not httpd... my bad. :) I have to specify --group-objectclass=posixGroup to get groups imported, that's fine. But I only get a few users imported. I see that by default it seem to be looking for objectclass=person. Only a few user accounts have that objectclass associated, so I add --user-objectclass=posixAccount as all users have this objectclass associated with their account. $ ipa migrate-ds --user-container='ou=people' --group-container='ou=group' --bind-dn='cn=directory manager' --user-objectclass=account --group-objectclass=posixGroup --schema=RFC2307 --continue ldap://ldapserver:399 ipa: ERROR: an internal error has occurred Not good. I look in the /var/log/httpd/error_log file, and I find: [Fri Jan 27 18:12:51 2012] [error] ipa: INFO: admin@NONE: ping(): SUCCESS [Fri Jan 27 18:12:52 2012] [error] ipa: ERROR: non-public: UnicodeDecodeError: 'utf8' codec can't decode byte 0xe5 in position 1: invalid continuation byte [Fri Jan 27 18:12:52 2012] [error] Traceback (most recent call last): [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipaserver/rpcserver.py", line 228, in wsgi_execute [Fri Jan 27 18:12:52 2012] [error] result = self.Command[name](*args, **options) [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/frontend.py", line 432, in __call__ [Fri Jan 27 18:12:52 2012] [error] ret = self.run(*args, **options) [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/frontend.py", line 738, in run [Fri Jan 27 18:12:52 2012] [error] return self.execute(*args, **options) [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/plugins/migration.py", line 634, in execute [Fri Jan 27 18:12:52 2012] [error] ldap, config, ds_ldap, ds_base_dn, options [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/plugins/migration.py", line 513, in migrate [Fri Jan 27 18:12:52 2012] [error] search_refs=True # migrated DS may contain search references [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 188, in new_f [Fri Jan 27 18:12:52 2012] [error] return f(*new_args, **kwargs) [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 199, in new_f [Fri Jan 27 18:12:52 2012] [error] return args[0].decode(f(*args, **kwargs)) [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 139, in decode [Fri Jan 27 18:12:52 2012] [error] return tuple(self.decode(m) for m in var) [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 139, in [Fri Jan 27 18:12:52 2012] [error] return tuple(self.decode(m) for m in var) [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 137, in decode [Fri Jan 27 18:12:52 2012] [error] return [self.decode(m) for m in var] [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 139, in decode [Fri Jan 27 18:12:52 2012] [error] return tuple(self.decode(m) for m in var) [Fri
Re: [Freeipa-users] ipa migrate-ds failing when more than 1 namingcontext is available
On Fri, 2012-01-27 at 13:42 -0500, Rob Crittenden wrote: > > This came up yesterday internally too. I don't believe a bug or ticket > has been filed yet. > > My best guess on what is happening, based on what I saw with our own > case, is this: > > A migrated attribute is coming in that IPA doesn't know about. We > default to treating all attributes that don't require special treatment > (like binary values) as utf-8. I'm guessing that an unknown binary > attribute is being migrated, we try to utf-8 encode it and kablooey. > > There is no easy workaround for the above problem right now because that > happens before the --ignore* options are considered. > Why don't we just base64-encode unknown attributes instead of treating them like UTF-8? The LDAP server is *supposed* to be able to handle this implicitly. signature.asc Description: This is a digitally signed message part ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] ipa migrate-ds failing when more than 1 namingcontext is available
Sigbjorn Lie wrote: On 01/27/2012 06:15 PM, Sigbjorn Lie wrote: On 01/27/2012 03:55 PM, Rob Crittenden wrote: Sigbjorn Lie wrote: On Fri, January 27, 2012 15:37, Rob Crittenden wrote: Stephen Gallagher wrote: On Fri, 2012-01-27 at 15:11 +0100, Sigbjorn Lie wrote: Hi The first naming context returned from the LDAP server is always chosen when using migrate-ds. This makes my import fail when I attempt to import users and groups from a previous LDAP server having more than 1 naming contexts available. The migrate-ds script should accept an option to specify what base_dn I would like to import from. Is there such an option today? I cannot find it... Not currently. I noticed this earlier in the week and opened a ticket on it, https://fedorahosted.org/freeipa/ticket/2314 Just to add to this request, if the original LDAP server has a defaultNamingContext attribute, it should be honored for auto-detecting which base to migrate. I'll update the 2314 to ensure we don't forget about this. 389-ds just added support for defaultNamingContext. Ok, thank you. Anything I can do to work around this issue today? I suppose there is just a file that need to be hacked to set a set a value instead of the auto-detected value... ? /usr/lib/python*/site-packages/ipalib/plugins/migration.py ~line 620 you'll see a block starting with the comment "retrieve DS base DN". Comment out the next 8 lines by prefixing them with # (these query to get the namingContext then pull the first value out). Add: ds_base_dn = 'dc=yourbasedn,dc=com' Alternatively you could always just add the above line to override what is detected. Commenting out just saves an LDAP lookup. Restart Apache. I already found that file and did that earlier today, however I was restarting tomcat6, not httpd... my bad. :) I have to specify --group-objectclass=posixGroup to get groups imported, that's fine. But I only get a few users imported. I see that by default it seem to be looking for objectclass=person. Only a few user accounts have that objectclass associated, so I add --user-objectclass=posixAccount as all users have this objectclass associated with their account. $ ipa migrate-ds --user-container='ou=people' --group-container='ou=group' --bind-dn='cn=directory manager' --user-objectclass=account --group-objectclass=posixGroup --schema=RFC2307 --continue ldap://ldapserver:399 ipa: ERROR: an internal error has occurred Not good. I look in the /var/log/httpd/error_log file, and I find: [Fri Jan 27 18:12:51 2012] [error] ipa: INFO: admin@NONE: ping(): SUCCESS [Fri Jan 27 18:12:52 2012] [error] ipa: ERROR: non-public: UnicodeDecodeError: 'utf8' codec can't decode byte 0xe5 in position 1: invalid continuation byte [Fri Jan 27 18:12:52 2012] [error] Traceback (most recent call last): [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipaserver/rpcserver.py", line 228, in wsgi_execute [Fri Jan 27 18:12:52 2012] [error] result = self.Command[name](*args, **options) [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/frontend.py", line 432, in __call__ [Fri Jan 27 18:12:52 2012] [error] ret = self.run(*args, **options) [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/frontend.py", line 738, in run [Fri Jan 27 18:12:52 2012] [error] return self.execute(*args, **options) [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/plugins/migration.py", line 634, in execute [Fri Jan 27 18:12:52 2012] [error] ldap, config, ds_ldap, ds_base_dn, options [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/plugins/migration.py", line 513, in migrate [Fri Jan 27 18:12:52 2012] [error] search_refs=True # migrated DS may contain search references [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 188, in new_f [Fri Jan 27 18:12:52 2012] [error] return f(*new_args, **kwargs) [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 199, in new_f [Fri Jan 27 18:12:52 2012] [error] return args[0].decode(f(*args, **kwargs)) [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 139, in decode [Fri Jan 27 18:12:52 2012] [error] return tuple(self.decode(m) for m in var) [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 139, in [Fri Jan 27 18:12:52 2012] [error] return tuple(self.decode(m) for m in var) [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 137, in decode [Fri Jan 27 18:12:52 2012] [error] return [self.decode(m) for m in var] [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 139, in decode [Fri Jan 27 18:12:52 2012] [error] return tuple(self.decode(m) for m in var) [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-pack
Re: [Freeipa-users] ipa migrate-ds failing when more than 1 namingcontext is available
On 01/27/2012 06:15 PM, Sigbjorn Lie wrote: On 01/27/2012 03:55 PM, Rob Crittenden wrote: Sigbjorn Lie wrote: On Fri, January 27, 2012 15:37, Rob Crittenden wrote: Stephen Gallagher wrote: On Fri, 2012-01-27 at 15:11 +0100, Sigbjorn Lie wrote: Hi The first naming context returned from the LDAP server is always chosen when using migrate-ds. This makes my import fail when I attempt to import users and groups from a previous LDAP server having more than 1 naming contexts available. The migrate-ds script should accept an option to specify what base_dn I would like to import from. Is there such an option today? I cannot find it... Not currently. I noticed this earlier in the week and opened a ticket on it, https://fedorahosted.org/freeipa/ticket/2314 Just to add to this request, if the original LDAP server has a defaultNamingContext attribute, it should be honored for auto-detecting which base to migrate. I'll update the 2314 to ensure we don't forget about this. 389-ds just added support for defaultNamingContext. Ok, thank you. Anything I can do to work around this issue today? I suppose there is just a file that need to be hacked to set a set a value instead of the auto-detected value... ? /usr/lib/python*/site-packages/ipalib/plugins/migration.py ~line 620 you'll see a block starting with the comment "retrieve DS base DN". Comment out the next 8 lines by prefixing them with # (these query to get the namingContext then pull the first value out). Add: ds_base_dn = 'dc=yourbasedn,dc=com' Alternatively you could always just add the above line to override what is detected. Commenting out just saves an LDAP lookup. Restart Apache. I already found that file and did that earlier today, however I was restarting tomcat6, not httpd... my bad. :) I have to specify --group-objectclass=posixGroup to get groups imported, that's fine. But I only get a few users imported. I see that by default it seem to be looking for objectclass=person. Only a few user accounts have that objectclass associated, so I add --user-objectclass=posixAccount as all users have this objectclass associated with their account. $ ipa migrate-ds --user-container='ou=people' --group-container='ou=group' --bind-dn='cn=directory manager' --user-objectclass=account --group-objectclass=posixGroup --schema=RFC2307 --continue ldap://ldapserver:399 ipa: ERROR: an internal error has occurred Not good. I look in the /var/log/httpd/error_log file, and I find: [Fri Jan 27 18:12:51 2012] [error] ipa: INFO: admin@NONE: ping(): SUCCESS [Fri Jan 27 18:12:52 2012] [error] ipa: ERROR: non-public: UnicodeDecodeError: 'utf8' codec can't decode byte 0xe5 in position 1: invalid continuation byte [Fri Jan 27 18:12:52 2012] [error] Traceback (most recent call last): [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipaserver/rpcserver.py", line 228, in wsgi_execute [Fri Jan 27 18:12:52 2012] [error] result = self.Command[name](*args, **options) [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/frontend.py", line 432, in __call__ [Fri Jan 27 18:12:52 2012] [error] ret = self.run(*args, **options) [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/frontend.py", line 738, in run [Fri Jan 27 18:12:52 2012] [error] return self.execute(*args, **options) [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/plugins/migration.py", line 634, in execute [Fri Jan 27 18:12:52 2012] [error] ldap, config, ds_ldap, ds_base_dn, options [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/plugins/migration.py", line 513, in migrate [Fri Jan 27 18:12:52 2012] [error] search_refs=True# migrated DS may contain search references [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 188, in new_f [Fri Jan 27 18:12:52 2012] [error] return f(*new_args, **kwargs) [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 199, in new_f [Fri Jan 27 18:12:52 2012] [error] return args[0].decode(f(*args, **kwargs)) [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 139, in decode [Fri Jan 27 18:12:52 2012] [error] return tuple(self.decode(m) for m in var) [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 139, in [Fri Jan 27 18:12:52 2012] [error] return tuple(self.decode(m) for m in var) [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 137, in decode [Fri Jan 27 18:12:52 2012] [error] return [self.decode(m) for m in var] [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 139, in decode [Fri Jan 27 18:12:52 2012] [error] return
Re: [Freeipa-users] ipa migrate-ds failing when more than 1 namingcontext is available
On 01/27/2012 03:55 PM, Rob Crittenden wrote: Sigbjorn Lie wrote: On Fri, January 27, 2012 15:37, Rob Crittenden wrote: Stephen Gallagher wrote: On Fri, 2012-01-27 at 15:11 +0100, Sigbjorn Lie wrote: Hi The first naming context returned from the LDAP server is always chosen when using migrate-ds. This makes my import fail when I attempt to import users and groups from a previous LDAP server having more than 1 naming contexts available. The migrate-ds script should accept an option to specify what base_dn I would like to import from. Is there such an option today? I cannot find it... Not currently. I noticed this earlier in the week and opened a ticket on it, https://fedorahosted.org/freeipa/ticket/2314 Just to add to this request, if the original LDAP server has a defaultNamingContext attribute, it should be honored for auto-detecting which base to migrate. I'll update the 2314 to ensure we don't forget about this. 389-ds just added support for defaultNamingContext. Ok, thank you. Anything I can do to work around this issue today? I suppose there is just a file that need to be hacked to set a set a value instead of the auto-detected value... ? /usr/lib/python*/site-packages/ipalib/plugins/migration.py ~line 620 you'll see a block starting with the comment "retrieve DS base DN". Comment out the next 8 lines by prefixing them with # (these query to get the namingContext then pull the first value out). Add: ds_base_dn = 'dc=yourbasedn,dc=com' Alternatively you could always just add the above line to override what is detected. Commenting out just saves an LDAP lookup. Restart Apache. I already found that file and did that earlier today, however I was restarting tomcat6, not httpd... my bad. :) I have to specify --group-objectclass=posixGroup to get groups imported, that's fine. But I only get a few users imported. I see that by default it seem to be looking for objectclass=person. Only a few user accounts have that objectclass associated, so I add --user-objectclass=posixAccount as all users have this objectclass associated with their account. $ ipa migrate-ds --user-container='ou=people' --group-container='ou=group' --bind-dn='cn=directory manager' --user-objectclass=account --group-objectclass=posixGroup --schema=RFC2307 --continue ldap://ldapserver:399 ipa: ERROR: an internal error has occurred Not good. I look in the /var/log/httpd/error_log file, and I find: [Fri Jan 27 18:12:51 2012] [error] ipa: INFO: admin@NONE: ping(): SUCCESS [Fri Jan 27 18:12:52 2012] [error] ipa: ERROR: non-public: UnicodeDecodeError: 'utf8' codec can't decode byte 0xe5 in position 1: invalid continuation byte [Fri Jan 27 18:12:52 2012] [error] Traceback (most recent call last): [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipaserver/rpcserver.py", line 228, in wsgi_execute [Fri Jan 27 18:12:52 2012] [error] result = self.Command[name](*args, **options) [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/frontend.py", line 432, in __call__ [Fri Jan 27 18:12:52 2012] [error] ret = self.run(*args, **options) [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/frontend.py", line 738, in run [Fri Jan 27 18:12:52 2012] [error] return self.execute(*args, **options) [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/plugins/migration.py", line 634, in execute [Fri Jan 27 18:12:52 2012] [error] ldap, config, ds_ldap, ds_base_dn, options [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/plugins/migration.py", line 513, in migrate [Fri Jan 27 18:12:52 2012] [error] search_refs=True# migrated DS may contain search references [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 188, in new_f [Fri Jan 27 18:12:52 2012] [error] return f(*new_args, **kwargs) [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 199, in new_f [Fri Jan 27 18:12:52 2012] [error] return args[0].decode(f(*args, **kwargs)) [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 139, in decode [Fri Jan 27 18:12:52 2012] [error] return tuple(self.decode(m) for m in var) [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 139, in [Fri Jan 27 18:12:52 2012] [error] return tuple(self.decode(m) for m in var) [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 137, in decode [Fri Jan 27 18:12:52 2012] [error] return [self.decode(m) for m in var] [Fri Jan 27 18:12:52 2012] [error] File "/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 139, in decode [Fri Jan 27 18:12:52 2012] [error] return tuple(self.decode(m) for m in var) [Fri Jan 2
Re: [Freeipa-users] ipa migrate-ds failing when more than 1 namingcontext is available
Sigbjorn Lie wrote: On Fri, January 27, 2012 15:37, Rob Crittenden wrote: Stephen Gallagher wrote: On Fri, 2012-01-27 at 15:11 +0100, Sigbjorn Lie wrote: Hi The first naming context returned from the LDAP server is always chosen when using migrate-ds. This makes my import fail when I attempt to import users and groups from a previous LDAP server having more than 1 naming contexts available. The migrate-ds script should accept an option to specify what base_dn I would like to import from. Is there such an option today? I cannot find it... Not currently. I noticed this earlier in the week and opened a ticket on it, https://fedorahosted.org/freeipa/ticket/2314 Just to add to this request, if the original LDAP server has a defaultNamingContext attribute, it should be honored for auto-detecting which base to migrate. I'll update the 2314 to ensure we don't forget about this. 389-ds just added support for defaultNamingContext. Ok, thank you. Anything I can do to work around this issue today? I suppose there is just a file that need to be hacked to set a set a value instead of the auto-detected value... ? /usr/lib/python*/site-packages/ipalib/plugins/migration.py ~line 620 you'll see a block starting with the comment "retrieve DS base DN". Comment out the next 8 lines by prefixing them with # (these query to get the namingContext then pull the first value out). Add: ds_base_dn = 'dc=yourbasedn,dc=com' Alternatively you could always just add the above line to override what is detected. Commenting out just saves an LDAP lookup. Restart Apache. rob ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] ipa migrate-ds failing when more than 1 namingcontext is available
On Fri, January 27, 2012 15:37, Rob Crittenden wrote: > Stephen Gallagher wrote: > >> On Fri, 2012-01-27 at 15:11 +0100, Sigbjorn Lie wrote: >> >>> Hi >>> >>> >>> The first naming context returned from the LDAP server is always chosen >>> when using migrate-ds. This makes my import fail when I attempt to import >>> users and groups from >>> a previous LDAP server having more than 1 naming contexts available. >>> >>> The migrate-ds script should accept an option to specify what base_dn I >>> would like to import from. >>> >>> Is there such an option today? I cannot find it... >>> > > Not currently. I noticed this earlier in the week and opened a ticket on > it, https://fedorahosted.org/freeipa/ticket/2314 > >> >> Just to add to this request, if the original LDAP server has a >> defaultNamingContext attribute, it should be honored for auto-detecting >> which base to migrate. > > I'll update the 2314 to ensure we don't forget about this. 389-ds just > added support for defaultNamingContext. > Ok, thank you. Anything I can do to work around this issue today? I suppose there is just a file that need to be hacked to set a set a value instead of the auto-detected value... ? ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] ipa migrate-ds failing when more than 1 namingcontext is available
Stephen Gallagher wrote: On Fri, 2012-01-27 at 15:11 +0100, Sigbjorn Lie wrote: Hi The first naming context returned from the LDAP server is always chosen when using migrate-ds. This makes my import fail when I attempt to import users and groups from a previous LDAP server having more than 1 naming contexts available. The migrate-ds script should accept an option to specify what base_dn I would like to import from. Is there such an option today? I cannot find it... Not currently. I noticed this earlier in the week and opened a ticket on it, https://fedorahosted.org/freeipa/ticket/2314 Just to add to this request, if the original LDAP server has a defaultNamingContext attribute, it should be honored for auto-detecting which base to migrate. I'll update the 2314 to ensure we don't forget about this. 389-ds just added support for defaultNamingContext. rob ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] ipa migrate-ds failing when more than 1 namingcontext is available
On Fri, January 27, 2012 15:17, Stephen Gallagher wrote: > On Fri, 2012-01-27 at 15:11 +0100, Sigbjorn Lie wrote: > >> Hi >> >> >> The first naming context returned from the LDAP server is always chosen >> when using migrate-ds. This makes my import fail when I attempt to import >> users and groups from a >> previous LDAP server having more than 1 naming contexts available. >> >> The migrate-ds script should accept an option to specify what base_dn I >> would like to import from. >> >> Is there such an option today? I cannot find it... >> > > Just to add to this request, if the original LDAP server has a > defaultNamingContext attribute, it should be honored for auto-detecting which > base to migrate. > ___ The defaultNamingContext is not available, and I'm unable to add it. "Object class violation". This is a Sun DS 7.0. Is there a way to manually specify what base_dn to import from? Rgds, Siggi ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] ipa migrate-ds failing when more than 1 namingcontext is available
On Fri, 2012-01-27 at 15:11 +0100, Sigbjorn Lie wrote: > Hi > > The first naming context returned from the LDAP server is always chosen > when using migrate-ds. This makes my import fail when I attempt to > import users and groups from a previous LDAP server having more than 1 > naming contexts available. > > The migrate-ds script should accept an option to specify what base_dn I > would like to import from. > > Is there such an option today? I cannot find it... Just to add to this request, if the original LDAP server has a defaultNamingContext attribute, it should be honored for auto-detecting which base to migrate. signature.asc Description: This is a digitally signed message part ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
[Freeipa-users] ipa migrate-ds failing when more than 1 namingcontext is available
Hi The first naming context returned from the LDAP server is always chosen when using migrate-ds. This makes my import fail when I attempt to import users and groups from a previous LDAP server having more than 1 naming contexts available. The migrate-ds script should accept an option to specify what base_dn I would like to import from. Is there such an option today? I cannot find it... Regards, Siggi ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
