Re: [Freeipa-users] Unexpected IPA Crashes
- Original Message - We have been using FreeIPA since two years and were more than happy. But since two weeks we are facing unexpected crashed and can not really debug the strange behaviours. The crashes are definitely not caused by connecting a new system or changing the LDAP schema heavily. Following IPA is used: Name : ipa-server Arch : x86_64 Version : 3.3.3 Release : 28.0.1.el7.centos.3 Size : 4.1 M I have followed the troubleshooting guide http://directory.fedoraproject.org/docs/389ds/FAQ/faq.html#Troubleshooting and activated logging and activated the core dumping. Do you see the string segfault in /var/log/messages or in journalctl? Unfortunately, I cannot provide you any core dump, because it is not created after the ipa servers crashes. I'm sure the dirsrv is causing the problem, because when i restart the 389, then ipa works fine for a while. Currently I have activated the replication log level 8192. The error log shows no suspicious error or any fatal error. Following 389* versions are used: Installed Packages 389-ds-base.x86_64 1.3.3.1-15.el7_1 @/389-ds-base-1.3.3.1-15.el7_1.x86_64 389-ds-base-debuginfo.x86_64 1.3.1.6-26.el7_0 @base-debuginfo 389-ds-base-libs.x86_64 1.3.3.1-15.el7_1 Can you please provide some hint how I can debug this problem in more detail. Btw, the ipa infrastructure consist of one master and one replica. The server was also crashing, when the replica server was turned off. Do you thing an upgrade would solve the problem as the last resort? -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] Log filling up a couple of times per day
- Original Message - Hi Dimitri, I can do, we already analyzed it once. There is a loadbalancer checking the ldap protocol which seems to be seen as fail. Is there a check I can perform on the ldap ports to see if the service is available without generating the errors ? If you just need to check ldap i.e. that port 389 is listening: ldapsearch -xLLL -h ldaphost -s base -b objectclass=* 1.1 That will still log to the directory server access log. I will post a snippet later on if you have no clue. Thanks, Matt 2015-03-26 23:01 GMT+01:00 Dmitri Pal d...@redhat.com: On 03/26/2015 05:37 PM, Matt . wrote: Hi Guys, I'm facing every day a fast filling log of: /var/log/krb5kdc.log /var/log/dirsrv/slapd-DOMAIN/access* I need to remove the files and restart ipa. The kerberos log is filling up most, the access logs are quite fast on 100MB and a new one is created. Now I do some LDAP loging/logout per day, servers that chedck if they are registered for SSSD so that it logs it is normal, but I want to get rid of it I guess. I'm throwing out I think about 6GB per day of logs, all loglevels are low. Any idea ? It's 3.x on CentOS 6.6 Any idea ? Thanks Matt Do you have some services that are repeatedly doing something kerberos related and failing? I guess getting a snippet of the kerberos log would give some hint on what is it logging all the time. -- Thank you, Dmitri Pal Sr. Engineering Manager IdM portfolio Red Hat, Inc. -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] Where do I change the nsslapd-accesslog-level?
- Original Message - I am using FreeIPA 3.0.0 on RHEL 6 (ipa-server-3.0.0-37.el6.x86_64). Where do I change the verbosity of access logging? Why do you need to change the verbosity of access logging? Do you mean error logging? If so, see http://port389.org/wiki/FAQ#Troubleshooting This doc: http://docs.fedoraproject.org/en-US/Fedora/18/html/FreeIPA_Guide/server-config.html discusses turning on global debugging but doesn't help me. The same doc links to: https://access.redhat.com/site/documentation/en-US/Red_Hat_Directory_Server/8.2/html/Configuration_and_Command-Line_Tool_Reference/logs-reference.html which tells me that I need to change the nsslapd-accesslog-level but the link on that page is a 404. So what do I need to do to change the level? I would assume that setting the level to 4 would be indicated if 256 is too verbose but can someone please confirm? I tried looking in the Configuration tab of the admin GUI but I get thrown: IPA Error 4204 limits exceeded for this query Not sure what's going on there, might be symptomatic of the high load the server is under due to iowait perhaps... Thanks! ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Where do I change the nsslapd-accesslog-level?
- Original Message - On Tue, May 13, 2014 at 1:28 PM, Richard Megginson rmegg...@redhat.comwrote: - Original Message - I am using FreeIPA 3.0.0 on RHEL 6 (ipa-server-3.0.0-37.el6.x86_64). Where do I change the verbosity of access logging? Why do you need to change the verbosity of access logging? Do you mean error logging? If so, see http://port389.org/wiki/FAQ#Troubleshooting I do mean access logging. I want to change it because it's too verbose :-) . It's causing high load / iowait on the server. There isn't a way to change the access log level to make it less verbose. You can turn it off completely nsslapd-accesslog-enabled: off Note that the access log is buffered, specifically to reduce the I/O load. If that buffered load is _still_ too high, then you might want to investigate replacing the access log file with a named pipe, then writing a small bit of python code to filter out only the events you are interested in. See https://access.redhat.com/site/documentation/en-US/Red_Hat_Directory_Server/9.0/html/Administration_Guide/using-named-pipe.html Based on the link you sent if I crafted an ldif like: dn: cn=config changetype: modify replace: nsslapd-accesslog-level nsslapd-accesslog-level: 4 that would presumably get me what I want. I don't think so. The error log levels are completely different than the access log levels, in that there are no access log levels. Does it require a dirsrv restart? No, but . . . Please advise. Thanks! This doc: http://docs.fedoraproject.org/en-US/Fedora/18/html/FreeIPA_Guide/server-config.html discusses turning on global debugging but doesn't help me. The same doc links to: https://access.redhat.com/site/documentation/en-US/Red_Hat_Directory_Server/8.2/html/Configuration_and_Command-Line_Tool_Reference/logs-reference.html which tells me that I need to change the nsslapd-accesslog-level but the link on that page is a 404. So what do I need to do to change the level? I would assume that setting the level to 4 would be indicated if 256 is too verbose but can someone please confirm? I tried looking in the Configuration tab of the admin GUI but I get thrown: IPA Error 4204 limits exceeded for this query Not sure what's going on there, might be symptomatic of the high load the server is under due to iowait perhaps... Thanks! ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Where do I change the nsslapd-accesslog-level?
- Original Message - On Tue, May 13, 2014 at 2:26 PM, Richard Megginson rmegg...@redhat.comwrote: - Original Message - On Tue, May 13, 2014 at 1:28 PM, Richard Megginson rmegg...@redhat.comwrote: - Original Message - I am using FreeIPA 3.0.0 on RHEL 6 (ipa-server-3.0.0-37.el6.x86_64). Where do I change the verbosity of access logging? Why do you need to change the verbosity of access logging? Do you mean error logging? If so, see http://port389.org/wiki/FAQ#Troubleshooting I do mean access logging. I want to change it because it's too verbose :-) . It's causing high load / iowait on the server. There isn't a way to change the access log level to make it less verbose. You can turn it off completely nsslapd-accesslog-enabled: off Sorry, you've confused me. Are you saying that nsslapd-accesslog-level: 4 is just as verbose as nsslapd-accesslog-level: 256? Yes. Or that there is literally no way to change the level despite the fact that there are levels? Yes, you can change the level. You can make it much more verbose than it is already. I don't think this is what you want. https://access.redhat.com/site/documentation/en-US/Red_Hat_Directory_Server/9.0/html/Configuration_Command_and_File_Reference/Core_Server_Configuration_Reference.html#cnconfig-nsslapd_accesslog_level So you could, for example, change the level to 4, and log internal operations. 1) That may be much more verbose than the default of 256 2) That may not be particularly useful to you. If the purpose of changing the access logging level is to reduce the I/O, then no, there is no level which will reduce the verbosity but still give you some sort of useful data in the access log. If you want to reduce the verbosity, but still have some sort of useful information, then you'll have to use the named pipe log script to filter out only those events which are useful to you. Cheers Note that the access log is buffered, specifically to reduce the I/O load. If that buffered load is _still_ too high, then you might want to investigate replacing the access log file with a named pipe, then writing a small bit of python code to filter out only the events you are interested in. See https://access.redhat.com/site/documentation/en-US/Red_Hat_Directory_Server/9.0/html/Administration_Guide/using-named-pipe.html Based on the link you sent if I crafted an ldif like: dn: cn=config changetype: modify replace: nsslapd-accesslog-level nsslapd-accesslog-level: 4 that would presumably get me what I want. I don't think so. The error log levels are completely different than the access log levels, in that there are no access log levels. Does it require a dirsrv restart? No, but . . . Please advise. Thanks! This doc: http://docs.fedoraproject.org/en-US/Fedora/18/html/FreeIPA_Guide/server-config.html discusses turning on global debugging but doesn't help me. The same doc links to: https://access.redhat.com/site/documentation/en-US/Red_Hat_Directory_Server/8.2/html/Configuration_and_Command-Line_Tool_Reference/logs-reference.html which tells me that I need to change the nsslapd-accesslog-level but the link on that page is a 404. So what do I need to do to change the level? I would assume that setting the level to 4 would be indicated if 256 is too verbose but can someone please confirm? I tried looking in the Configuration tab of the admin GUI but I get thrown: IPA Error 4204 limits exceeded for this query Not sure what's going on there, might be symptomatic of the high load the server is under due to iowait perhaps... Thanks! ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] 389-ds memory usage
- Original Message - On 06/05/2012 05:55 PM, Richard Megginson wrote: - Original Message - On Mon, April 23, 2012 20:38, Rich Megginson wrote: Ok. The current theory is that the memory growth is caused by the churn of entries being added to and removed from the entry cache. It's not yet known why this growth is seen. It could be just that the memory is getting fragmented, or there is a real yet undetected memory leak. That's why entry cache sizing and monitoring is very important, to see if you are churning entries in/out of the cache, and if that is correlated with the memory growth. This memory issue is still occuring in the production environment after increasing the max entry cache size to 256MB, and it is impacting performance. See below for an output of the memory usage and the current size and hitratio of the cache on the 3 production IPA servers. How do you suggest moving forward to troubleshoot this issue? Are you seeing https://fedorahosted.org/389/ticket/386 ? I don't think so. The cache numbers described in this ticket is much higher than my cache numbers. I've set the maxentrycachesize to 256MB, and each IPA server has 8GB of memory. There should not have been a consumption of more than 1,8GB with the 7 * cachememsize as described in the ticket. Beside I don't know where the heavy modify levels would come from. About 100 new users we're moved to IPA last week, all having their passwords changed, the accounts themselves already existed. But is that considered a heavy LDAP modify level? I don't think it matters whether or not the modify level is heavy. If there is a memory leak, a heavy modify level will make it more apparent more quickly. There seem to be a memory leak somewhere. How would you recommend moving forward? If you think this leak is a completely different issue than ticket 386, please open another ticket. Regards, Siggi ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] General status of my FreeIPA servers - is there a method for cleaning them?
- Original Message -
On Tue, Apr 17, 2012 at 09:26, Rich Megginson rmegg...@redhat.com
wrote:
On 04/17/2012 07:26 AM, Dan Scott wrote:
On Fri, Apr 13, 2012 at 17:44, Rich Megginsonrmegg...@redhat.com
wrote:
On 04/13/2012 03:40 PM, Dan Scott wrote:
I cleaned up all the ruv_compare_ruv: RUV [changelog max RUV]
does
not contain element errors in the logs for each of fileservers
1, 2
and 3. The ldapsearch for
'((nsuniqueid=---)(objectclass=nstombstone))'
is still showing entries though. Is that OK?
The entry should exist, but the deleted servers should not be
present in
the
nsds50ruv attribute.
OK, so it's safe to delete replica entries which have
ldap://fileserver4.ecg.mit.edu:389 (fileserver4 is not currently a
replica) but not for the other servers?
Yes. Following the CLEANRUV procedure:
http://port389.org/wiki/Howto:CLEANRUV
Thanks. I think I'm getting there - removed the tombstones from the
main directory and the PKI-IPA directory (only one server so far
though). I still have a few strange entries though:
[root@fileserver1 ~]# ldapsearch -xLLL -D cn=directory manager -W
-b
dc=ecg,dc=mit,dc=edu
'((nsuniqueid=---)(objectclass=nstombstone))'
Enter LDAP Password:
dn:
nsuniqueid=---,dc=ecg,dc=mit,dc=edu
objectClass: top
objectClass: nsTombstone
objectClass: extensibleobject
nsds50ruv: {replicageneration} 4e7b746e0004
nsds50ruv: {replica 6 ldap://fileserver1.ecg.mit.edu:389}
4f50e685001d0006
4f8d787400020006
nsds50ruv: {replica 43 ldap://fileserver2.ecg.mit.edu:389}
4f88cf450001002b000
0 4f8d7814002b
nsds50ruv: {replica 5 ldap://fileserver3.ecg.mit.edu:389}
4f5047ad001d0005
4f8d77c30005
nsds50ruv: {replica 4 ldap://fileserver3.ecg.mit.edu:389}
nsds50ruv: {replica 9 ldap://fileserver3.ecg.mit.edu:389}
nsds50ruv: {replica 8 ldap://fileserver3.ecg.mit.edu:389}
4f7363d2001d0008
4f73640200070008
dc: ecg
nsruvReplicaLastModified: {replica 6
ldap://fileserver1.ecg.mit.edu:389} 4f8d7
806
nsruvReplicaLastModified: {replica 43
ldap://fileserver2.ecg.mit.edu:389} 4f8d
77a6
nsruvReplicaLastModified: {replica 5
ldap://fileserver3.ecg.mit.edu:389} 4f8d7
756
nsruvReplicaLastModified: {replica 4
ldap://fileserver3.ecg.mit.edu:389} 0
000
nsruvReplicaLastModified: {replica 9
ldap://fileserver3.ecg.mit.edu:389} 0
000
nsruvReplicaLastModified: {replica 8
ldap://fileserver3.ecg.mit.edu:389} 0
000
Is it safe to run CLEANRUV on IDs 4 and 9? That still leaves me with
2
entries for fileserver3. How do I know which one to delete?
Whichever one is the one currently in use.
ldapsearch -xLLL -h fileserver3 -D cn=directory manager -W -b cn=config
cn=replica
What is the replica ID? That is the one that is currently in use. You should
be able to safely delete the others.
On my PKI-IPA server, the CLEANRUV task doesn't seem to work. It
keeps
re-adding entries after I remove them. I have 3 entries for my
non-existent fileserver4 - They disappear when I remove them, but
they
come back after a few minutes.
Right, because they are being replicated from another master. You will need to
run the CLEANRUV on all masters at the same time.
Thanks,
Dan
___
Freeipa-users mailing list
Freeipa-users@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-users
