Re: [Freeipa-users] ipa migrate-ds failing when more than 1 namingcontext is available

Rob Crittenden Mon, 30 Jan 2012 06:08:20 -0800

Sigbjorn Lie wrote:

On 01/27/2012 07:42 PM, Rob Crittenden wrote:

Sigbjorn Lie wrote:

On 01/27/2012 06:15 PM, Sigbjorn Lie wrote:

On 01/27/2012 03:55 PM, Rob Crittenden wrote:

Sigbjorn Lie wrote:


On Fri, January 27, 2012 15:37, Rob Crittenden wrote:

Stephen Gallagher wrote:

On Fri, 2012-01-27 at 15:11 +0100, Sigbjorn Lie wrote:

Hi


The first naming context returned from the LDAP server is always
chosen
when using migrate-ds. This makes my import fail when I attempt
to import users and groups from
a previous LDAP server having more than 1 naming contexts
available.

The migrate-ds script should accept an option to specify what
base_dn I
would like to import from.

Is there such an option today? I cannot find it...


Not currently. I noticed this earlier in the week and opened a
ticket on
it, https://fedorahosted.org/freeipa/ticket/2314


Just to add to this request, if the original LDAP server has a
defaultNamingContext attribute, it should be honored for
auto-detecting which base to migrate.


I'll update the 2314 to ensure we don't forget about this. 389-ds
just
added support for defaultNamingContext.


Ok, thank you.

Anything I can do to work around this issue today? I suppose there
is just a file that need to be
hacked to set a set a value instead of the auto-detected value... ?


/usr/lib/python*/site-packages/ipalib/plugins/migration.py

~line 620 you'll see a block starting with the comment "retrieve DS
base DN".

Comment out the next 8 lines by prefixing them with # (these query to
get the namingContext then pull the first value out).

Add:

ds_base_dn = 'dc=yourbasedn,dc=com'

Alternatively you could always just add the above line to override
what is detected. Commenting out just saves an LDAP lookup.

Restart Apache.



I already found that file and did that earlier today, however I was
restarting tomcat6, not httpd... my bad. :)

I have to specify --group-objectclass=posixGroup to get groups
imported, that's fine. But I only get a few users imported. I see that
by default it seem to be looking for objectclass=person. Only a few
user accounts have that objectclass associated, so I add
--user-objectclass=posixAccount as all users have this objectclass
associated with their account.

$ ipa migrate-ds --user-container='ou=people'
--group-container='ou=group' --bind-dn='cn=directory manager'
--user-objectclass=account --group-objectclass=posixGroup
--schema=RFC2307 --continue ldap://ldapserver:399
ipa: ERROR: an internal error has occurred

Not good. I look in the /var/log/httpd/error_log file, and I find:

[Fri Jan 27 18:12:51 2012] [error] ipa: INFO: admin@NONE: ping():
SUCCESS
[Fri Jan 27 18:12:52 2012] [error] ipa: ERROR: non-public:
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe5 in position 1:
invalid continuation byte
[Fri Jan 27 18:12:52 2012] [error] Traceback (most recent call last):
[Fri Jan 27 18:12:52 2012] [error] File
"/usr/lib/python2.6/site-packages/ipaserver/rpcserver.py", line 228,
in wsgi_execute
[Fri Jan 27 18:12:52 2012] [error] result = self.Command[name](*args,
**options)
[Fri Jan 27 18:12:52 2012] [error] File
"/usr/lib/python2.6/site-packages/ipalib/frontend.py", line 432, in
__call__
[Fri Jan 27 18:12:52 2012] [error] ret = self.run(*args, **options)
[Fri Jan 27 18:12:52 2012] [error] File
"/usr/lib/python2.6/site-packages/ipalib/frontend.py", line 738, in run
[Fri Jan 27 18:12:52 2012] [error] return self.execute(*args,
**options)
[Fri Jan 27 18:12:52 2012] [error] File
"/usr/lib/python2.6/site-packages/ipalib/plugins/migration.py", line
634, in execute
[Fri Jan 27 18:12:52 2012] [error] ldap, config, ds_ldap, ds_base_dn,
options
[Fri Jan 27 18:12:52 2012] [error] File
"/usr/lib/python2.6/site-packages/ipalib/plugins/migration.py", line
513, in migrate
[Fri Jan 27 18:12:52 2012] [error] search_refs=True # migrated DS may
contain search references
[Fri Jan 27 18:12:52 2012] [error] File
"/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 188, in
new_f
[Fri Jan 27 18:12:52 2012] [error] return f(*new_args, **kwargs)
[Fri Jan 27 18:12:52 2012] [error] File
"/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 199, in
new_f
[Fri Jan 27 18:12:52 2012] [error] return args[0].decode(f(*args,
**kwargs))
[Fri Jan 27 18:12:52 2012] [error] File
"/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 139, in
decode
[Fri Jan 27 18:12:52 2012] [error] return tuple(self.decode(m) for m
in var)
[Fri Jan 27 18:12:52 2012] [error] File
"/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 139, in
<genexpr>
[Fri Jan 27 18:12:52 2012] [error] return tuple(self.decode(m) for m
in var)
[Fri Jan 27 18:12:52 2012] [error] File
"/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 137, in
decode
[Fri Jan 27 18:12:52 2012] [error] return [self.decode(m) for m in var]
[Fri Jan 27 18:12:52 2012] [error] File
"/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 139, in
decode
[Fri Jan 27 18:12:52 2012] [error] return tuple(self.decode(m) for m
in var)
[Fri Jan 27 18:12:52 2012] [error] File
"/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 139, in
<genexpr>
[Fri Jan 27 18:12:52 2012] [error] return tuple(self.decode(m) for m
in var)
[Fri Jan 27 18:12:52 2012] [error] File
"/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 157, in
decode
[Fri Jan 27 18:12:52 2012] [error] dct[k] = self._decode_dict_val(k, v)
[Fri Jan 27 18:12:52 2012] [error] File
"/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 64, in
_decode_dict_val
[Fri Jan 27 18:12:52 2012] [error] return self.decode(val)
[Fri Jan 27 18:12:52 2012] [error] File
"/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 137, in
decode
[Fri Jan 27 18:12:52 2012] [error] return [self.decode(m) for m in var]
[Fri Jan 27 18:12:52 2012] [error] File
"/usr/lib/python2.6/site-packages/ipalib/encoder.py", line 132, in
decode
[Fri Jan 27 18:12:52 2012] [error]
var.decode(self.encoder_settings.decode_from)
[Fri Jan 27 18:12:52 2012] [error] File
"/usr/lib64/python2.6/encodings/utf_8.py", line 16, in decode
[Fri Jan 27 18:12:52 2012] [error] return codecs.utf_8_decode(input,
errors, True)
[Fri Jan 27 18:12:52 2012] [error] UnicodeDecodeError: 'utf8' codec
can't decode byte 0xe5 in position 1: invalid continuation byte
[Fri Jan 27 18:12:52 2012] [error] ipa: INFO: admin@NONE:
migrate_ds(u'ldap://svg-p-idm02.none:389', u'********',
binddn=u'cn=directory manager', usercontainer=u'ou=people',
groupcontainer=u'ou=group', userobjectclass=(u'account',),
groupobjectclass=(u'posixGroup',), userignoreobjectclass=None,
userignoreattribute=None, groupignoreobjectclass=None,
groupignoreattribute=None, groupoverwritegid=False, schema=u'RFC2307',
continue=False, exclude_groups=None, exclude_users=None):
UnicodeDecodeError


Any suggestions?

Oh yes and of course I've already looked for accounts with any non-utf8
chars in any of the output of an ldapsearch of the same ldap tree I'm
trying to import from...


This came up yesterday internally too. I don't believe a bug or ticket
has been filed yet.

My best guess on what is happening, based on what I saw with our own
case, is this:

A migrated attribute is coming in that IPA doesn't know about. We
default to treating all attributes that don't require special
treatment (like binary values) as utf-8. I'm guessing that an unknown
binary attribute is being migrated, we try to utf-8 encode it and
kablooey.

There is no easy workaround for the above problem right now because
that happens before the --ignore* options are considered.

One idea that has come to mind but is untested is to grab the missing
schema in the form of a discrete 389-ds-compatible schema file. Stop
dirsrv, drop the file into /etc/dirsrv/slapd-YOURINSTANCE/schema,
start dirsrv

Then retry the migration.

Use --ignore* if you don't want/need these attributes, then you stop
dirsrv, remove the schema file, start dirsrv.


I did use the ignore flags, ended up with:
--user-ignore-attribute=memberOf,mail,telephoneNumber,userPassword
--user-ignore-objectclass=organizationalPerson,posixAccount,shadowAccount,inetOrgPerson,organizationalPerson


Still no luck, same error.

According to your testing: is there still some value that should have
been ignored I've overlooked ?

On the other hand, I've only been looking at entries within the ou's
that contains users and groups. Will this issue also occur if there is
non-UTF8 chars in other parts of the directory?

Like I said, this error is triggered before ignore is evaluated so ifan unknown binary attribute is getting decoded it will cause thisfailure. The only solutions we have right now is to either load theschema into IPA temporarily for the migration, rremove it on the remoteside or you could modify the query we make to find the remote entries topull only certain attributes. This last one would be tricky to get right.


The code looks like:

                (entries, truncated) = ds_ldap.find_entries(
                    search_filter, ['*'], search_bases[ldap_obj_name],
                    ds_ldap.SCOPE_ONELEVEL,
                    time_limit=0, size_limit=-1,

search_refs=True # migrated DS may containsearch references

You'd want to replace ['*'] with ['attr1','attr2','attr3',...]. It wouldbe a rather long list and would need to cover both users and groups.


rob

_______________________________________________
Freeipa-users mailing list
Freeipa-users@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-users

Re: [Freeipa-users] ipa migrate-ds failing when more than 1 namingcontext is available

Reply via email to