[Freeipa-users] No LDAPS for dirsrv

[Thomas Raehalme]

Hi!

As I wrote earlier we are having some serious problems with IPA right now.
dirsrv seems to hang every 15 minutes or so, but that's another post.

It seems that slapd/dirsrv is now only listening on port 389 for LDAP and
socket for LDAPI requests. Any idea what could have caused previously
available LDAPS port 636 to disappear?

Looking at the logs before this whole ordeal started port 636 was also in
use.

After the latest upgrade I have re-enabled port 389 manually because it's
used by some apps, but disabling it also doesn't bring back port 636.

Best regards,
Thomas
-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go To http://freeipa.org for more info on the project

Re: [Freeipa-users] No LDAPS for dirsrv

[Rob Crittenden]

Thomas Raehalme wrote:
> Hi!
> 
> As I wrote earlier we are having some serious problems with IPA right
> now. dirsrv seems to hang every 15 minutes or so, but that's another post.
> 
> It seems that slapd/dirsrv is now only listening on port 389 for LDAP
> and socket for LDAPI requests. Any idea what could have caused
> previously available LDAPS port 636 to disappear?
> 
> Looking at the logs before this whole ordeal started port 636 was also
> in use.
> 
> After the latest upgrade I have re-enabled port 389 manually because
> it's used by some apps, but disabling it also doesn't bring back port 636.
> 
> Best regards,
> Thomas
> 
> 

If after an upgrade you had no listeners that means that the upgrade
failed and wasn't able to restore the previous state. Look in
/etc/dirsrv/slapd-YOURREALM for dse.ldif.ipa.###. This is the copy
saved prior to the upgrade attempt. I'd diff it to dse.ldif to see what
has changed.

To enable port 636 just set nsslapd-security to on. If you do this via
dse.ldif you'll need to stop the service before editing the file.

Check /var/log/ipaupgrade.log for information on the upgrade.

rob

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go To http://freeipa.org for more info on the project

Re: [Freeipa-users] No LDAPS for dirsrv

[Chris Mohler]

On 02/17/2015 11:26 AM, Thomas Raehalme wrote:

Hi!

As I wrote earlier we are having some serious problems with IPA right 
now. dirsrv seems to hang every 15 minutes or so, but that's another post.


It seems that slapd/dirsrv is now only listening on port 389 for LDAP 
and socket for LDAPI requests. Any idea what could have caused 
previously available LDAPS port 636 to disappear?


Looking at the logs before this whole ordeal started port 636 was also 
in use.


After the latest upgrade I have re-enabled port 389 manually because 
it's used by some apps, but disabling it also doesn't bring back port 636.


Best regards,
Thomas



Hi Thomas,
I'm not an expert but just throwing out a few ideas for you.

As I wrote earlier we are having some serious problems with IPA right 
now. dirsrv seems to hang every 15 minutes or so, but that's another post.

Are you running in a VM? If so check your entropy.
cat /proc/sys/kernel/random/entropy_avail
It should be ~1k less than 50 is not great and caused me some issues in 
the past.


It seems that slapd/dirsrv is now only listening on port 389 for LDAP 
and socket for LDAPI requests. Any idea what could have caused 
previously available LDAPS port 636 to disappear? 
Did your certificates expire? I usually check the web interface and look 
at the SSL Cert in the browser to see when it expires. I bet there is a 
better way to check but I don't know it off hand.


It might help to know what OS/version you are using? and what version of 
FreeIPA you are using.


Cheers, and Good luck,
-Chris





-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go To http://freeipa.org for more info on the project

Re: [Freeipa-users] No LDAPS for dirsrv

[Thomas Raehalme]

Hi!

On Tue, Feb 17, 2015 at 6:34 PM, Rob Crittenden  wrote:

>
> If after an upgrade you had no listeners that means that the upgrade
> failed and wasn't able to restore the previous state. Look in
> /etc/dirsrv/slapd-YOURREALM for dse.ldif.ipa.###. This is the copy
> saved prior to the upgrade attempt. I'd diff it to dse.ldif to see what
> has changed.
>
> To enable port 636 just set nsslapd-security to on. If you do this via
> dse.ldif you'll need to stop the service before editing the file.
>

Thanks! For some reason the value of nsslapd-security is 'off' for the
current version although previous dse.ldif.ipa.## files have it enabled.
Changing the value back to 'on' re-enabled port 636.

Best regards,
Thomas
-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go To http://freeipa.org for more info on the project

Re: [Freeipa-users] No LDAPS for dirsrv

[Thomas Raehalme]

Hi Chris!

On Tue, Feb 17, 2015 at 6:35 PM, Chris Mohler  wrote:

>
> As I wrote earlier we are having some serious problems with IPA right now.
> dirsrv seems to hang every 15 minutes or so, but that's another post.
>
> Are you running in a VM? If so check your entropy.
> cat /proc/sys/kernel/random/entropy_avail
> It should be ~1k less than 50 is not great and caused me some issues in
> the past.
>

Yes, the server is a VM. Entropy value is 135 at the moment. Do you know
how to increase the value?

It seems that slapd/dirsrv is now only listening on port 389 for LDAP and
> socket for LDAPI requests. Any idea what could have caused previously
> available LDAPS port 636 to disappear?
>
> Did your certificates expire? I usually check the web interface and look
> at the SSL Cert in the browser to see when it expires. I bet there is a
> better way to check but I don't know it off hand.
>

No, at least for the web interface certificates expire in August.

It turned out the nsslapd-security was 'off' when it should have been 'on'.
I really don't know what had changed the value.

Now I only wish we could resolve what's causing the dirsrv process to hang
(wrote about that in another message last Sunday) about 10 minutes after
IPA services were started.

Thanks for your help!

Best regards,
Thomas
-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go To http://freeipa.org for more info on the project

Re: [Freeipa-users] No LDAPS for dirsrv

[Rob Crittenden]

Thomas Raehalme wrote:
> Hi Chris!
> 
> On Tue, Feb 17, 2015 at 6:35 PM, Chris Mohler  > wrote:
> 
> 
>> As I wrote earlier we are having some serious problems with IPA
>> right now. dirsrv seems to hang every 15 minutes or so, but that's
>> another post.
> Are you running in a VM? If so check your entropy.
> cat /proc/sys/kernel/random/entropy_avail
> It should be ~1k less than 50 is not great and caused me some issues
> in the past.
> 
> 
> Yes, the server is a VM. Entropy value is 135 at the moment. Do you know
> how to increase the value?

I don't think that's an issue. It is more a problem during initial
installation than during operation AFAIK.

>> It seems that slapd/dirsrv is now only listening on port 389 for
>> LDAP and socket for LDAPI requests. Any idea what could have
>> caused previously available LDAPS port 636 to disappear? 
> Did your certificates expire? I usually check the web interface and
> look at the SSL Cert in the browser to see when it expires. I bet
> there is a better way to check but I don't know it off hand.
> 
> 
> No, at least for the web interface certificates expire in August.
> 
> It turned out the nsslapd-security was 'off' when it should have been
> 'on'. I really don't know what had changed the value.
> 
> Now I only wish we could resolve what's causing the dirsrv process to
> hang (wrote about that in another message last Sunday) about 10 minutes
> after IPA services were started.

Evidence suggests that the last upgrade failed so I'd start there. It is
possible some plugins aren't configured properly, for example.

You can try to re-run the upgrade manually. Note that the updater will
disable all listeners while it is running. This is where things went
sideways before.

# /usr/sbin/ipa-ldap-updater --upgrade

If that succeeds:

# /usr/sbin/ipa-upgradeconfig

Then

# ipactl restart

rob

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go To http://freeipa.org for more info on the project

Re: [Freeipa-users] No LDAPS for dirsrv

[Chris Mohler]

I would agree with Rob, entropy is likely not one of your root issues.

It may still do you good to have a bit more as it can cause system 
slowdown during SSL generation loads.


It's really up to you how you go about generating entropy.
Here is a link with some suggestions
http://log.amitshah.net/2013/01/about-random-numbers-and-virtual-machines/

I would suggest you just
yum install haveged
It's worked good for me so far.

Good luck,
-Chris

On 02/17/2015 12:38 PM, Rob Crittenden wrote:

Thomas Raehalme wrote:

Hi Chris!

On Tue, Feb 17, 2015 at 6:35 PM, Chris Mohler mailto:cmoh...@oberlin.edu>> wrote:



 As I wrote earlier we are having some serious problems with IPA
 right now. dirsrv seems to hang every 15 minutes or so, but that's
 another post.

 Are you running in a VM? If so check your entropy.
 cat /proc/sys/kernel/random/entropy_avail
 It should be ~1k less than 50 is not great and caused me some issues
 in the past.


Yes, the server is a VM. Entropy value is 135 at the moment. Do you know
how to increase the value?

I don't think that's an issue. It is more a problem during initial
installation than during operation AFAIK.


 It seems that slapd/dirsrv is now only listening on port 389 for
 LDAP and socket for LDAPI requests. Any idea what could have
 caused previously available LDAPS port 636 to disappear?

 Did your certificates expire? I usually check the web interface and
 look at the SSL Cert in the browser to see when it expires. I bet
 there is a better way to check but I don't know it off hand.


No, at least for the web interface certificates expire in August.

It turned out the nsslapd-security was 'off' when it should have been
'on'. I really don't know what had changed the value.

Now I only wish we could resolve what's causing the dirsrv process to
hang (wrote about that in another message last Sunday) about 10 minutes
after IPA services were started.

Evidence suggests that the last upgrade failed so I'd start there. It is
possible some plugins aren't configured properly, for example.

You can try to re-run the upgrade manually. Note that the updater will
disable all listeners while it is running. This is where things went
sideways before.

# /usr/sbin/ipa-ldap-updater --upgrade

If that succeeds:

# /usr/sbin/ipa-upgradeconfig

Then

# ipactl restart

rob



--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go To http://freeipa.org for more info on the project

Re: [Freeipa-users] No LDAPS for dirsrv

[Thomas Raehalme]

Hi!

On Tue, Feb 17, 2015 at 7:38 PM, Rob Crittenden  wrote:

> > Now I only wish we could resolve what's causing the dirsrv process to
> > hang (wrote about that in another message last Sunday) about 10 minutes
> > after IPA services were started.
>
> Evidence suggests that the last upgrade failed so I'd start there. It is
> possible some plugins aren't configured properly, for example.
>
>
Looking at 'yum history' I performed an update after the problems had
started an hour or two before.

According to /var/log/ipaupgrade.log everything before ipa-ldap-updater ran
smoothly. But when it got to 'service dirsrv stop', it took 562 seconds to
shutdown dirsrv for EXAMPLE-COM and restarting dirsrv afterwards failed
with:

2015-02-15T12:29:58Z DEBUG   [4/8]: starting directory server
2015-02-15T12:30:09Z DEBUG args=/sbin/service dirsrv start
2015-02-15T12:30:09Z DEBUG stdout=Starting dirsrv:
PKI-IPA... already running[  OK  ]^M
PVNET-CC...[FAILED]^M
  *** Error: 1 instance(s) failed to start

2015-02-15T12:30:09Z DEBUG stderr=[15/Feb/2015:14:29:58 +0200] -
Information: Non-Secure Port Disabled

2015-02-15T12:30:09Z INFO   File
"/usr/lib/python2.6/site-packages/ipapython/admintool.py", line 152, in
execute
return_value = self.run()
  File
"/usr/lib/python2.6/site-packages/ipaserver/install/ipa_ldap_updater.py",
line 139, in run
upgrade.create_instance()
  File
"/usr/lib/python2.6/site-packages/ipaserver/install/upgradeinstance.py",
line 84, in create_instance
show_service_name=False)
  File "/usr/lib/python2.6/site-packages/ipaserver/install/service.py",
line 358, in start_creation
method()
  File
"/usr/lib/python2.6/site-packages/ipaserver/install/upgradeinstance.py",
line 67, in __start_nowait
super(IPAUpgrade, self).start(wait=False)
  File "/usr/lib/python2.6/site-packages/ipaserver/install/service.py",
line 265, in start
self.service.start(instance_name, capture_output=capture_output,
wait=wait)
  File "/usr/lib/python2.6/site-packages/ipapython/platform/redhat.py",
line 80, in start
ipautil.run(["/sbin/service", self.service_name, "start",
instance_name], capture_output=capture_output)
  File "/usr/lib/python2.6/site-packages/ipapython/ipautil.py", line 316,
in run
raise CalledProcessError(p.returncode, args)

2015-02-15T12:30:09Z INFO The ipa-ldap-updater command failed, exception:
CalledProcessError: Command '/sbin/service dirsrv start ' returned non-zero
exit status 1
2015-02-15T12:30:09Z ERROR Unexpected error - see /var/log/ipaupgrade.log
for details:
CalledProcessError: Command '/sbin/service dirsrv start ' returned non-zero
exit status 1

Best regards,
Thomas
-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go To http://freeipa.org for more info on the project

Re: [Freeipa-users] No LDAPS for dirsrv

[Thomas Raehalme]

Hi!

On Tue, Feb 17, 2015 at 8:43 PM, Thomas Raehalme <
thomas.raeha...@codecenter.fi> wrote:

> Hi!
>
> On Tue, Feb 17, 2015 at 7:38 PM, Rob Crittenden 
> wrote:
>
>> > Now I only wish we could resolve what's causing the dirsrv process to
>> > hang (wrote about that in another message last Sunday) about 10 minutes
>> > after IPA services were started.
>>
>> Evidence suggests that the last upgrade failed so I'd start there. It is
>> possible some plugins aren't configured properly, for example.
>>
>>
After having restart ipa service, the upgrade command was completed
successfully:

# ipa-ldap-updater --upgrade
Upgrading IPA:
  [1/8]: stopping directory server
  [2/8]: saving configuration
  [3/8]: disabling listeners
  [4/8]: starting directory server
  [5/8]: upgrading server
  [6/8]: stopping directory server
  [7/8]: restoring configuration
  [8/8]: starting directory server
Done.

Now dirsrv was stopped in 2 second when the previous time was over 500
seconds.

Unfortunately this still didn't resolve the issue. After the system has
been online for about 10 minutes, named starts complaining:

Feb 17 21:04:14 ipa named[31117]: LDAP query timed out. Try to adjust
"timeout" parameter

Also ldapsearch just hangs if you try to perform any queries.

Any ideas what could go wrong here?

Best regards,
Thomas
-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go To http://freeipa.org for more info on the project

Re: [Freeipa-users] No LDAPS for dirsrv

[Alexander Bokovoy]

On Tue, 17 Feb 2015, Thomas Raehalme wrote:

Hi!

On Tue, Feb 17, 2015 at 8:43 PM, Thomas Raehalme <
thomas.raeha...@codecenter.fi> wrote:


Hi!

On Tue, Feb 17, 2015 at 7:38 PM, Rob Crittenden 
wrote:


> Now I only wish we could resolve what's causing the dirsrv process to
> hang (wrote about that in another message last Sunday) about 10 minutes
> after IPA services were started.

Evidence suggests that the last upgrade failed so I'd start there. It is
possible some plugins aren't configured properly, for example.



After having restart ipa service, the upgrade command was completed
successfully:

# ipa-ldap-updater --upgrade
Upgrading IPA:
 [1/8]: stopping directory server
 [2/8]: saving configuration
 [3/8]: disabling listeners
 [4/8]: starting directory server
 [5/8]: upgrading server
 [6/8]: stopping directory server
 [7/8]: restoring configuration
 [8/8]: starting directory server
Done.

Now dirsrv was stopped in 2 second when the previous time was over 500
seconds.

Unfortunately this still didn't resolve the issue. After the system has
been online for about 10 minutes, named starts complaining:

Feb 17 21:04:14 ipa named[31117]: LDAP query timed out. Try to adjust
"timeout" parameter

Also ldapsearch just hangs if you try to perform any queries.

Any ideas what could go wrong here?

So, can you get us a backtrace again?

--
/ Alexander Bokovoy


pgppXnHyCvtvH.pgp
Description: PGP signature
-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go To http://freeipa.org for more info on the project

Re: [Freeipa-users] No LDAPS for dirsrv

[Thomas Raehalme]

On Wed, Feb 18, 2015 at 9:34 AM, Alexander Bokovoy 
wrote:

>
> Unfortunately this still didn't resolve the issue. After the system has
>> been online for about 10 minutes, named starts complaining:
>> Also ldapsearch just hangs if you try to perform any queries.
>> Any ideas what could go wrong here?
>>
> So, can you get us a backtrace again?
>

Here is a summary of what is going on:

After a fresh start of IPA master with 'ipactl start' the system goes wrong
after 5-10 minutes.

What happens first is that KDC stops responding:

kinit thomas.raehalme
kinit: Cannot contact any KDC for realm 'EXAMPLE.COM' while getting initial
credentials

LDAP is still operational, however. It can be verified with ldapsearch.

Finally, after total of 15 minutes LDAP is also not responding:

Feb 18 10:00:12 ipa named[7410]: LDAP query timed out. Try to adjust
"timeout" parameter

Now I took some stacktraces which I will e-mail to you and Rob directly.

When I then try to stop dirsrv, it responds but seems to wait indefinitely:

[18/Feb/2015:10:03:03 +0200] - slapd shutting down - signaling operation
threads
[18/Feb/2015:10:03:03 +0200] - slapd shutting down - waiting for 30 threads
to terminate

I took two stacktraces from this situation as well.

Best regards,
Thomas
-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go To http://freeipa.org for more info on the project