Re: [Freeipa-users] IPA Load Problems?
On 09/04/2013 08:01 AM, John Moyer wrote: Martin, I apologize there was a large offline conversation between Rich and myself. Rich was kind enough to help me through some of my issues. We did a lot more tests and poking and prodding. We discovered that IPA is not as efficient when dealing with large number of connections. Most of my load inefficiently reconnect to IPA over and over and over and though LDAP can deal with this fairly efficiently, IPA apparently drops to it's knees. A ticket was opened to addressed this issue. https://fedorahosted.org/freeipa/ticket/3892 Thank you for reporting this ticket. Martin is investigating it and trying to see what is the cause. The information mentioned above is missing from the the ticket, thus the question. So to summarize: you identified that the cause of the performance issue is that JIRA makes a lot of parallel connections to LDAP server and IPA is slow processing bind operations thus clients that do a lot of connections can experience a low performance. Martin, I wonder if we can have a test that would just do a lot of binds. There are a lot of plugins and one of the recent ones is the OTP one. I wonder if we do too much during bind when OTP is not enabled (by default). Thanks, _ John Moyer Director, IT Operations *Digital Reasoning Systems, Inc.* john.mo...@digitalreasoning.com mailto:john.mo...@digitalreasoning.com Office:703.678.2311 Mobile:240.460.0023 Fax:703.678.2312 www.digitalreasoning.com http://www.digitalreasoning.com/ On Sep 4, 2013, at 3:44 AM, Martin Kosek mko...@redhat.com mailto:mko...@redhat.com wrote: On 08/30/2013 11:08 PM, John Moyer wrote: Well IPA has machine entries on some test clusters that I'm rolling IPA out on (20 machines maybe) but the user base is the same (about 80 ~ 100) accounts with maybe 40 to 50 groups? I've stood up a clone of the jira server along with IPA. I cleared my logs and then did the sync and ran the log analyzer on it. These stats are pretty much ONLY for that jira sync I don't have any other connections pointed to it. Start of Log:30/Aug/2013:15:57:13 End of Log: 30/Aug/2013:16:01:14 Processed Log Time: Hours, 4 Minutes, 1 Seconds Restarts: 1 Total Connections:824 SSL Connections: 824 Peak Concurrent Connections: 6 Total Operations: 1806 Total Results:1805 Overall Performance: 99.9% Searches: 968(4.02/sec) (241.00/min) Modifications:5 (0.02/sec) (1.24/min) Adds: 0 (0.00/sec) (0.00/min) Deletes: 0 (0.00/sec) (0.00/min) Mod RDNs: 0 (0.00/sec) (0.00/min) Compares: 0 (0.00/sec) (0.00/min) Binds:833(3.46/sec) (207.39/min) Proxied Auth Operations: 0 Persistent Searches: 1 Internal Operations: 0 Entry Operations: 0 Extended Operations: 0 Abandoned Requests: 0 Smart Referrals Received: 0 VLV Operations: 0 VLV Unindexed Searches: 0 SORT Operations: 0 Entire Search Base Queries: 0 Unindexed Searches: 1 This looks like a promising way to find out the reason, thanks John. However, I see just one unindexed search. Is the access log complete? Previously I see that the sync takes 900 seconds/15 minutes, but there is only 4 minutes the access log. Note that it it may take some time until the log is dumped. I think it would be also useful to run the analyzer with -ula flags as Rob suggested earlier to find out the unindexed searches (if any). What I find interesting is that JIRA does a lot of LDAP BINDs. Can the problem be in longer BINDs than with than expected (compared to for example plain LDAP servers)? Performance-wise, it would be I think better if JIRA does just one BIND and run all the LDAP searches the established connection. But I do not know if it can be configured this way. Rich, Rob, I am wondering if the slow up is not really caused by the binds, we have several DS plugins tied to the BIND operation, it may be useful to analyze if they do not take too long. Martin ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users -- Thank you, Dmitri Pal Sr. Engineering Manager for IdM portfolio Red Hat Inc. --- Looking to carve out IT costs? www.redhat.com/carveoutcosts/ ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] IPA Load Problems?
That summary is correct. The only thing I would add is that other applications could easily bring the IPA server to it's knees as well. Our artifact server also did many connections per sec when used, and one person doing a build could bring IPA to it's knees as well. Also, not only would IPA be maxed at 100%, but users would complain that their builds were taking longer than normal (with or without the JIRA sync going, however, it was obviously worse when JIRA was running). Also, my IPA server was a larger/faster server than my LDAP server. So my LDAP server would run circles around IPA even though it was on a smaller machine. LDAP would run at about 10% maybe 15% CPU when the JIRA sync ran. IF you need any other information let me know. Thanks, _ John Moyer Director, IT Operations On Sep 4, 2013, at 8:32 AM, Dmitri Pal d...@redhat.com wrote: On 09/04/2013 08:01 AM, John Moyer wrote: Martin, I apologize there was a large offline conversation between Rich and myself. Rich was kind enough to help me through some of my issues. We did a lot more tests and poking and prodding. We discovered that IPA is not as efficient when dealing with large number of connections. Most of my load inefficiently reconnect to IPA over and over and over and though LDAP can deal with this fairly efficiently, IPA apparently drops to it's knees. A ticket was opened to addressed this issue. https://fedorahosted.org/freeipa/ticket/3892 Thank you for reporting this ticket. Martin is investigating it and trying to see what is the cause. The information mentioned above is missing from the the ticket, thus the question. So to summarize: you identified that the cause of the performance issue is that JIRA makes a lot of parallel connections to LDAP server and IPA is slow processing bind operations thus clients that do a lot of connections can experience a low performance. Martin, I wonder if we can have a test that would just do a lot of binds. There are a lot of plugins and one of the recent ones is the OTP one. I wonder if we do too much during bind when OTP is not enabled (by default). Thanks, _ John Moyer Director, IT Operations Digital Reasoning Systems, Inc. john.mo...@digitalreasoning.com Office: 703.678.2311 Mobile: 240.460.0023 Fax: 703.678.2312 www.digitalreasoning.com On Sep 4, 2013, at 3:44 AM, Martin Kosek mko...@redhat.com wrote: On 08/30/2013 11:08 PM, John Moyer wrote: Well IPA has machine entries on some test clusters that I'm rolling IPA out on (20 machines maybe) but the user base is the same (about 80 ~ 100) accounts with maybe 40 to 50 groups? I've stood up a clone of the jira server along with IPA. I cleared my logs and then did the sync and ran the log analyzer on it. These stats are pretty much ONLY for that jira sync I don't have any other connections pointed to it. Start of Log:30/Aug/2013:15:57:13 End of Log: 30/Aug/2013:16:01:14 Processed Log Time: Hours, 4 Minutes, 1 Seconds Restarts: 1 Total Connections:824 SSL Connections: 824 Peak Concurrent Connections: 6 Total Operations: 1806 Total Results:1805 Overall Performance: 99.9% Searches: 968(4.02/sec) (241.00/min) Modifications:5 (0.02/sec) (1.24/min) Adds: 0 (0.00/sec) (0.00/min) Deletes: 0 (0.00/sec) (0.00/min) Mod RDNs: 0 (0.00/sec) (0.00/min) Compares: 0 (0.00/sec) (0.00/min) Binds:833(3.46/sec) (207.39/min) Proxied Auth Operations: 0 Persistent Searches: 1 Internal Operations: 0 Entry Operations: 0 Extended Operations: 0 Abandoned Requests: 0 Smart Referrals Received: 0 VLV Operations: 0 VLV Unindexed Searches: 0 SORT Operations: 0 Entire Search Base Queries: 0 Unindexed Searches: 1 This looks like a promising way to find out the reason, thanks John. However, I see just one unindexed search. Is the access log complete? Previously I see that the sync takes 900 seconds/15 minutes, but there is only 4 minutes the access log. Note that it it may take some time until the log is dumped. I think it would be also useful to run the analyzer with -ula flags as Rob suggested earlier to find out the unindexed searches (if any). What I find interesting is that JIRA does a lot of LDAP BINDs. Can the problem be in longer BINDs than with than expected (compared to for example plain LDAP servers)? Performance-wise, it would be I think better if JIRA does just one BIND and run all the LDAP searches the established
Re: [Freeipa-users] IPA Load Problems?
Sure, just let me know what needs to be run/applied. I've already rolled back to LDAP, so if the fix looks like it works I can then roll it out again. Thanks, _ John Moyer Director, IT Operations On Sep 4, 2013, at 9:12 AM, Dmitri Pal d...@redhat.com wrote: On 09/04/2013 08:53 AM, John Moyer wrote: That summary is correct. The only thing I would add is that other applications could easily bring the IPA server to it's knees as well. Yes this is what I meant. It is not only JIRA. Any client that creates a lot of connections can cause problems. Our artifact server also did many connections per sec when used, and one person doing a build could bring IPA to it's knees as well. Also, not only would IPA be maxed at 100%, but users would complain that their builds were taking longer than normal (with or without the JIRA sync going, however, it was obviously worse when JIRA was running). Also, my IPA server was a larger/faster server than my LDAP server. So my LDAP server would run circles around IPA even though it was on a smaller machine. LDAP would run at about 10% maybe 15% CPU when the JIRA sync ran. IF you need any other information let me know. No this seems to be enough. Thank you. Would you be willing to test a fix if one is provided? Thanks Dmitri Thanks, _ John Moyer Director, IT Operations On Sep 4, 2013, at 8:32 AM, Dmitri Pal d...@redhat.com wrote: On 09/04/2013 08:01 AM, John Moyer wrote: Martin, I apologize there was a large offline conversation between Rich and myself. Rich was kind enough to help me through some of my issues. We did a lot more tests and poking and prodding. We discovered that IPA is not as efficient when dealing with large number of connections. Most of my load inefficiently reconnect to IPA over and over and over and though LDAP can deal with this fairly efficiently, IPA apparently drops to it's knees. A ticket was opened to addressed this issue. https://fedorahosted.org/freeipa/ticket/3892 Thank you for reporting this ticket. Martin is investigating it and trying to see what is the cause. The information mentioned above is missing from the the ticket, thus the question. So to summarize: you identified that the cause of the performance issue is that JIRA makes a lot of parallel connections to LDAP server and IPA is slow processing bind operations thus clients that do a lot of connections can experience a low performance. Martin, I wonder if we can have a test that would just do a lot of binds. There are a lot of plugins and one of the recent ones is the OTP one. I wonder if we do too much during bind when OTP is not enabled (by default). Thanks, _ John Moyer Director, IT Operations Digital Reasoning Systems, Inc. john.mo...@digitalreasoning.com Office: 703.678.2311 Mobile: 240.460.0023 Fax: 703.678.2312 www.digitalreasoning.com On Sep 4, 2013, at 3:44 AM, Martin Kosek mko...@redhat.com wrote: On 08/30/2013 11:08 PM, John Moyer wrote: Well IPA has machine entries on some test clusters that I'm rolling IPA out on (20 machines maybe) but the user base is the same (about 80 ~ 100) accounts with maybe 40 to 50 groups? I've stood up a clone of the jira server along with IPA. I cleared my logs and then did the sync and ran the log analyzer on it. These stats are pretty much ONLY for that jira sync I don't have any other connections pointed to it. Start of Log:30/Aug/2013:15:57:13 End of Log: 30/Aug/2013:16:01:14 Processed Log Time: Hours, 4 Minutes, 1 Seconds Restarts: 1 Total Connections:824 SSL Connections: 824 Peak Concurrent Connections: 6 Total Operations: 1806 Total Results:1805 Overall Performance: 99.9% Searches: 968(4.02/sec) (241.00/min) Modifications:5 (0.02/sec) (1.24/min) Adds: 0 (0.00/sec) (0.00/min) Deletes: 0 (0.00/sec) (0.00/min) Mod RDNs: 0 (0.00/sec) (0.00/min) Compares: 0 (0.00/sec) (0.00/min) Binds:833(3.46/sec) (207.39/min) Proxied Auth Operations: 0 Persistent Searches: 1 Internal Operations: 0 Entry Operations: 0 Extended Operations: 0 Abandoned Requests: 0 Smart Referrals Received: 0 VLV Operations: 0 VLV Unindexed Searches: 0 SORT Operations: 0 Entire Search Base Queries: 0 Unindexed Searches: 1 This looks like a promising way to find out the reason, thanks John. However, I see just one unindexed search. Is the access log
Re: [Freeipa-users] IPA Load Problems?
On Wed, 04 Sep 2013, Dmitri Pal wrote: On 09/04/2013 08:01 AM, John Moyer wrote: Martin, I apologize there was a large offline conversation between Rich and myself. Rich was kind enough to help me through some of my issues. We did a lot more tests and poking and prodding. We discovered that IPA is not as efficient when dealing with large number of connections. Most of my load inefficiently reconnect to IPA over and over and over and though LDAP can deal with this fairly efficiently, IPA apparently drops to it's knees. A ticket was opened to addressed this issue. https://fedorahosted.org/freeipa/ticket/3892 Thank you for reporting this ticket. Martin is investigating it and trying to see what is the cause. The information mentioned above is missing from the the ticket, thus the question. So to summarize: you identified that the cause of the performance issue is that JIRA makes a lot of parallel connections to LDAP server and IPA is slow processing bind operations thus clients that do a lot of connections can experience a low performance. Martin, I wonder if we can have a test that would just do a lot of binds. There are a lot of plugins and one of the recent ones is the OTP one. I wonder if we do too much during bind when OTP is not enabled (by default). This should be unrelated to OTP work as John is using CentOS with older FreeIPA build, equivalent to what is in RHEL 6.4. -- / Alexander Bokovoy ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Exporting data?
On 4.9.2013 15:04, Bret Wortman wrote: What's the right venue for making a suggestion? In particular, I'd like to toss out there that it would be really nice to be able to export, at a minimum, DNS and user data from IPA in the form of a zone file and a passwd/shadow file pair. I realize there might be security implications to the latter, and masking out passwords might be advisiable. And there's no easy way, necessarily, to get out sudo information. But having DNS and user details would at least permit a sysadmin having major issues (like I have been for the past two weeks) to get up and running in some form, using puppet or some other tool to distribute flat files with named running against a static zone file, or even to migrate off IPA if absolutely necessary. Hello, for DNS you can use normal zone transfer. Just configure IPA zone to allow zone transfer to an IP address (localhost means 'localy to IPA server') and use standard DNS tools, e.g. dig: $ ipa dnszone-mod example.com --allow-transfer='localhost;' $ dig +onesoa -t AXFR example.com /root/example.com.db That is all you need for DNS, you have the standard zone file. I believe that you can use SSSD (with enumeration enabled) to run getent passwd /root/passwd.bck. I have no idea how it works with shadow map/password. Try to ask sssd-us...@lists.fedorahosted.org. -- Petr^2 Spacek ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Exporting data?
On 09/04/2013 09:26 AM, Petr Spacek wrote: On 4.9.2013 15:04, Bret Wortman wrote: What's the right venue for making a suggestion? In particular, I'd like to toss out there that it would be really nice to be able to export, at a minimum, DNS and user data from IPA in the form of a zone file and a passwd/shadow file pair. I realize there might be security implications to the latter, and masking out passwords might be advisiable. And there's no easy way, necessarily, to get out sudo information. But having DNS and user details would at least permit a sysadmin having major issues (like I have been for the past two weeks) to get up and running in some form, using puppet or some other tool to distribute flat files with named running against a static zone file, or even to migrate off IPA if absolutely necessary. Hello, for DNS you can use normal zone transfer. Just configure IPA zone to allow zone transfer to an IP address (localhost means 'localy to IPA server') and use standard DNS tools, e.g. dig: $ ipa dnszone-mod example.com --allow-transfer='localhost;' $ dig +onesoa -t AXFR example.com /root/example.com.db That is all you need for DNS, you have the standard zone file. I believe that you can use SSSD (with enumeration enabled) to run getent passwd /root/passwd.bck. I have no idea how it works with shadow map/password. Try to ask sssd-us...@lists.fedorahosted.org. And to add to it: IPA does not keep password in clear or the hashes that are used in passwd and shadow files for security reasons so it can't generate these files as you suggest. It is unclear what the problems are that you are facing and what made you get back to local files. I agree with Petr that SSSD has a lot of bells and whistles to make your client experience smooth and help you recover from any server side problems you might have. But may be we are missing something and there is something we can do. If you can describe the problem you are facing we might be able to suggest a solution. -- Thank you, Dmitri Pal Sr. Engineering Manager for IdM portfolio Red Hat Inc. --- Looking to carve out IT costs? www.redhat.com/carveoutcosts/ ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Incorrect user information
On Wed, Sep 04, 2013 at 10:47:49AM -0400, Chris Hudson wrote: You may want to check out the sss_cache package in the sssd-tools package. It looks to be in the base channel for RHEL5 Server and optional channel for RHEL6 Server. This tool will allow you to invalidate/manipulate the sssd cache. -Chris Exactly. To invalidate user and group information and force a refresh on the next query, run sss_cache -UG. We realized that including sss_cache in the sssd-tools subpackage hid the tool from many users, so in recent versions it's included back in sssd proper. ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Incorrect user information
On Wed, Sep 04, 2013 at 10:18:13AM -0500, cbul...@gmail.com wrote: Hi Chris, Thanks for your reply!I forgot to mention that we tried sss_cache (sss_cache -u user_id and sss_cache -U) in other RH6 ipa client and it did not work...If we delete manually all /var/lib/sss/db we can see the change but it is not going to be a nice solution. This sounds really strange. Can you run a little experiment for me? Can you install the ldb-tools package and then run: 1) getent passwd $username 2) ldbsearch -H /var/lib/sss/db/cache_$domain.ldb name=$username 3) modify the entry 4) sss_cache -U 5) ldbsearch -H /var/lib/sss/db/cache_$domain.ldb name=$username 6) getent passwd $username 7) ldbsearch -H /var/lib/sss/db/cache_$domain.ldb name=$username after you run 2) you should see how the entry looks in the cache with the old attributes. After running 5) you should see the same attributes, except for dataExpireTimestamp that should be set to 1. After running 6), getent should yield the updated data and 7) should reflect ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Incorrect user information
On Wed, Sep 04, 2013 at 09:40:29AM -0500, cbul...@gmail.com wrote: Hi, We have a freeipa server (RedHat 6.3, freeipa:3.0.0-26) and freeipa client (RedHat 5.9, freeipa client 2.1.3.-5) working in our test testing scenario without further problems. We are able to use SUDO, HBAC etc. Our problem is when we change a user info (Name or Last Name) and check it using the command: getent passwd id_user it showed us the older user information. We set entry_cache_user_timeout = 0 in sssd.conf file in order to clear the cache data but it did not work. Also we tried with: use_fully_qualified_domains attribute as recommend in: https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/SSSD-Troubleshooting.html#idp26289072 but it was not helpful. If we check the user info using ldapsearch command we can see the right user info information. Changing the uid or gid we see the new change right away. Any clue about this problem? One more additional point about why changing the timestamp might not have had the effect you wanted..the cache validity that controls when the entry is refreshed from the cache is stored in the cache as a UNIX timestamp. So whenever you change the timeout, you also need to invalidate the cache using the sss_cache tool. Running the sss_cache tool sets the cache expiration timestamp to 1 (beginning of the Epoch) to force refresh on next query. Arguably this is not documented well in the sssd.conf manual page, so I've sent a patch to the upstream development list to document this behaviour better. ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Incorrect user information
On Wed, Sep 04, 2013 at 05:31:34PM +0200, Jakub Hrozek wrote: On Wed, Sep 04, 2013 at 10:18:13AM -0500, cbul...@gmail.com wrote: Hi Chris, Thanks for your reply!I forgot to mention that we tried sss_cache (sss_cache -u user_id and sss_cache -U) in other RH6 ipa client and it did not work...If we delete manually all /var/lib/sss/db we can see the change but it is not going to be a nice solution. This sounds really strange. Can you run a little experiment for me? Also are you running nscd? ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Incorrect user information
Hi Jakub, Thanks for your time and tips about sssd cache! I did the test and let me explain what I got: - After step 4 I can see dataExpireTimestamp to 1 for the user. - After step 7 dataExpireTimestamp is back to 0 but the user data have not changed. The first line after the command ldbsearch is: asq: Unable to register control with rootdse! Is it a problem? We are not using nscd service. Please let me know if you need to do some other tests. Thanks in advance! On 09/04/2013 10:31 AM, Jakub Hrozek wrote: On Wed, Sep 04, 2013 at 10:18:13AM -0500, cbul...@gmail.com wrote: Hi Chris, Thanks for your reply!I forgot to mention that we tried sss_cache (sss_cache -u user_id and sss_cache -U) in other RH6 ipa client and it did not work...If we delete manually all /var/lib/sss/db we can see the change but it is not going to be a nice solution. This sounds really strange. Can you run a little experiment for me? Can you install the ldb-tools package and then run: 1) getent passwd $username 2) ldbsearch -H /var/lib/sss/db/cache_$domain.ldb name=$username 3) modify the entry 4) sss_cache -U 5) ldbsearch -H /var/lib/sss/db/cache_$domain.ldb name=$username 6) getent passwd $username 7) ldbsearch -H /var/lib/sss/db/cache_$domain.ldb name=$username after you run 2) you should see how the entry looks in the cache with the old attributes. After running 5) you should see the same attributes, except for dataExpireTimestamp that should be set to 1. After running 6), getent should yield the updated data and 7) should reflect ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] IPA Load Problems?
Ah, ok. One of the reasons why I was poking to this thread is exactly this ticket. It does not contain much information _what exactly_ is making IPA performance poor - whether it is missing indices (which ones?) or some issue in IPA plugins during binds, etc. Without more information, we do not know what to fix, what to improve. Martin On 09/04/2013 02:01 PM, John Moyer wrote: Martin, I apologize there was a large offline conversation between Rich and myself. Rich was kind enough to help me through some of my issues. We did a lot more tests and poking and prodding. We discovered that IPA is not as efficient when dealing with large number of connections. Most of my load inefficiently reconnect to IPA over and over and over and though LDAP can deal with this fairly efficiently, IPA apparently drops to it's knees. A ticket was opened to addressed this issue. https://fedorahosted.org/freeipa/ticket/3892 Thanks, _ John Moyer Director, IT Operations Digital Reasoning Systems, Inc. john.mo...@digitalreasoning.com Office: 703.678.2311 Mobile: 240.460.0023 Fax: 703.678.2312 www.digitalreasoning.com On Sep 4, 2013, at 3:44 AM, Martin Kosek mko...@redhat.com wrote: On 08/30/2013 11:08 PM, John Moyer wrote: Well IPA has machine entries on some test clusters that I'm rolling IPA out on (20 machines maybe) but the user base is the same (about 80 ~ 100) accounts with maybe 40 to 50 groups? I've stood up a clone of the jira server along with IPA. I cleared my logs and then did the sync and ran the log analyzer on it. These stats are pretty much ONLY for that jira sync I don't have any other connections pointed to it. Start of Log:30/Aug/2013:15:57:13 End of Log: 30/Aug/2013:16:01:14 Processed Log Time: Hours, 4 Minutes, 1 Seconds Restarts: 1 Total Connections:824 SSL Connections: 824 Peak Concurrent Connections: 6 Total Operations: 1806 Total Results:1805 Overall Performance: 99.9% Searches: 968(4.02/sec) (241.00/min) Modifications:5 (0.02/sec) (1.24/min) Adds: 0 (0.00/sec) (0.00/min) Deletes: 0 (0.00/sec) (0.00/min) Mod RDNs: 0 (0.00/sec) (0.00/min) Compares: 0 (0.00/sec) (0.00/min) Binds:833 (3.46/sec) (207.39/min) Proxied Auth Operations: 0 Persistent Searches: 1 Internal Operations: 0 Entry Operations: 0 Extended Operations: 0 Abandoned Requests: 0 Smart Referrals Received: 0 VLV Operations: 0 VLV Unindexed Searches: 0 SORT Operations: 0 Entire Search Base Queries: 0 Unindexed Searches: 1 This looks like a promising way to find out the reason, thanks John. However, I see just one unindexed search. Is the access log complete? Previously I see that the sync takes 900 seconds/15 minutes, but there is only 4 minutes the access log. Note that it it may take some time until the log is dumped. I think it would be also useful to run the analyzer with -ula flags as Rob suggested earlier to find out the unindexed searches (if any). What I find interesting is that JIRA does a lot of LDAP BINDs. Can the problem be in longer BINDs than with than expected (compared to for example plain LDAP servers)? Performance-wise, it would be I think better if JIRA does just one BIND and run all the LDAP searches the established connection. But I do not know if it can be configured this way. Rich, Rob, I am wondering if the slow up is not really caused by the binds, we have several DS plugins tied to the BIND operation, it may be useful to analyze if they do not take too long. Martin ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Exporting data?
On Wed, 2013-09-04 at 09:40 -0400, Dmitri Pal wrote: On 09/04/2013 09:26 AM, Petr Spacek wrote: On 4.9.2013 15:04, Bret Wortman wrote: What's the right venue for making a suggestion? In particular, I'd like to toss out there that it would be really nice to be able to export, at a minimum, DNS and user data from IPA in the form of a zone file and a passwd/shadow file pair. I realize there might be security implications to the latter, and masking out passwords might be advisiable. And there's no easy way, necessarily, to get out sudo information. But having DNS and user details would at least permit a sysadmin having major issues (like I have been for the past two weeks) to get up and running in some form, using puppet or some other tool to distribute flat files with named running against a static zone file, or even to migrate off IPA if absolutely necessary. Hello, for DNS you can use normal zone transfer. Just configure IPA zone to allow zone transfer to an IP address (localhost means 'localy to IPA server') and use standard DNS tools, e.g. dig: $ ipa dnszone-mod example.com --allow-transfer='localhost;' $ dig +onesoa -t AXFR example.com /root/example.com.db That is all you need for DNS, you have the standard zone file. I believe that you can use SSSD (with enumeration enabled) to run getent passwd /root/passwd.bck. I have no idea how it works with shadow map/password. Try to ask sssd-us...@lists.fedorahosted.org. And to add to it: IPA does not keep password in clear or the hashes that are used in passwd and shadow files for security reasons so it can't generate these files as you suggest. We do have hashes, the default is SHA256, it is stored in userPassword and is used to validate LDAP binds, however we never let it out of LDAP, neither SSSD not the integrate NIS server expose the password hash to clients. You need Directory Manager privileges to read it. Simo. -- Simo Sorce * Red Hat, Inc * New York ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] IPA Load Problems?
On 09/04/2013 07:51 AM, Martin Kosek wrote: Ah, ok. One of the reasons why I was poking to this thread is exactly this ticket. It does not contain much information _what exactly_ is making IPA performance poor - whether it is missing indices (which ones?) or some issue in IPA plugins during binds, etc. Without more information, we do not know what to fix, what to improve. Right. It's a big, complicated problem. So we start with what we know - the JIRA LDAP sync with IPA is much, much too slow. We don't know why it's too slow. But we can at least set it up and begin profiling it. Martin On 09/04/2013 02:01 PM, John Moyer wrote: Martin, I apologize there was a large offline conversation between Rich and myself. Rich was kind enough to help me through some of my issues. We did a lot more tests and poking and prodding. We discovered that IPA is not as efficient when dealing with large number of connections. Most of my load inefficiently reconnect to IPA over and over and over and though LDAP can deal with this fairly efficiently, IPA apparently drops to it's knees. A ticket was opened to addressed this issue. https://fedorahosted.org/freeipa/ticket/3892 Thanks, _ John Moyer Director, IT Operations Digital Reasoning Systems, Inc. john.mo...@digitalreasoning.com Office: 703.678.2311 Mobile:240.460.0023 Fax:703.678.2312 www.digitalreasoning.com On Sep 4, 2013, at 3:44 AM, Martin Kosek mko...@redhat.com wrote: On 08/30/2013 11:08 PM, John Moyer wrote: Well IPA has machine entries on some test clusters that I'm rolling IPA out on (20 machines maybe) but the user base is the same (about 80 ~ 100) accounts with maybe 40 to 50 groups? I've stood up a clone of the jira server along with IPA. I cleared my logs and then did the sync and ran the log analyzer on it. These stats are pretty much ONLY for that jira sync I don't have any other connections pointed to it. Start of Log:30/Aug/2013:15:57:13 End of Log: 30/Aug/2013:16:01:14 Processed Log Time: Hours, 4 Minutes, 1 Seconds Restarts: 1 Total Connections:824 SSL Connections: 824 Peak Concurrent Connections: 6 Total Operations: 1806 Total Results:1805 Overall Performance: 99.9% Searches: 968(4.02/sec) (241.00/min) Modifications:5 (0.02/sec) (1.24/min) Adds: 0 (0.00/sec) (0.00/min) Deletes: 0 (0.00/sec) (0.00/min) Mod RDNs: 0 (0.00/sec) (0.00/min) Compares: 0 (0.00/sec) (0.00/min) Binds:833 (3.46/sec) (207.39/min) Proxied Auth Operations: 0 Persistent Searches: 1 Internal Operations: 0 Entry Operations: 0 Extended Operations: 0 Abandoned Requests: 0 Smart Referrals Received: 0 VLV Operations: 0 VLV Unindexed Searches: 0 SORT Operations: 0 Entire Search Base Queries: 0 Unindexed Searches: 1 This looks like a promising way to find out the reason, thanks John. However, I see just one unindexed search. Is the access log complete? Previously I see that the sync takes 900 seconds/15 minutes, but there is only 4 minutes the access log. Note that it it may take some time until the log is dumped. I think it would be also useful to run the analyzer with -ula flags as Rob suggested earlier to find out the unindexed searches (if any). What I find interesting is that JIRA does a lot of LDAP BINDs. Can the problem be in longer BINDs than with than expected (compared to for example plain LDAP servers)? Performance-wise, it would be I think better if JIRA does just one BIND and run all the LDAP searches the established connection. But I do not know if it can be configured this way. Rich, Rob, I am wondering if the slow up is not really caused by the binds, we have several DS plugins tied to the BIND operation, it may be useful to analyze if they do not take too long. Martin ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] IPA Load Problems?
It was our opinion that it wasn't an index issue. I cleared the logs from the IPA server, and then just ran a JIRA sync with the server. I gave Rich the log file from my IPA for that sync. I can't find the exact conversation, but we determined that JIRA was connecting to LDAP some 1000 times or so to do the sync. The logs didn't show but one search done that didn't have an index which is why we concluded it wasn't an index issue. Thanks, _ John Moyer Director, IT Operations On Sep 4, 2013, at 9:51 AM, Martin Kosek mko...@redhat.com wrote: Ah, ok. One of the reasons why I was poking to this thread is exactly this ticket. It does not contain much information _what exactly_ is making IPA performance poor - whether it is missing indices (which ones?) or some issue in IPA plugins during binds, etc. Without more information, we do not know what to fix, what to improve. Martin On 09/04/2013 02:01 PM, John Moyer wrote: Martin, I apologize there was a large offline conversation between Rich and myself. Rich was kind enough to help me through some of my issues. We did a lot more tests and poking and prodding. We discovered that IPA is not as efficient when dealing with large number of connections. Most of my load inefficiently reconnect to IPA over and over and over and though LDAP can deal with this fairly efficiently, IPA apparently drops to it's knees. A ticket was opened to addressed this issue. https://fedorahosted.org/freeipa/ticket/3892 Thanks, _ John Moyer Director, IT Operations Digital Reasoning Systems, Inc. john.mo...@digitalreasoning.com Office: 703.678.2311 Mobile: 240.460.0023 Fax: 703.678.2312 www.digitalreasoning.com On Sep 4, 2013, at 3:44 AM, Martin Kosek mko...@redhat.com wrote: On 08/30/2013 11:08 PM, John Moyer wrote: Well IPA has machine entries on some test clusters that I'm rolling IPA out on (20 machines maybe) but the user base is the same (about 80 ~ 100) accounts with maybe 40 to 50 groups? I've stood up a clone of the jira server along with IPA. I cleared my logs and then did the sync and ran the log analyzer on it. These stats are pretty much ONLY for that jira sync I don't have any other connections pointed to it. Start of Log:30/Aug/2013:15:57:13 End of Log: 30/Aug/2013:16:01:14 Processed Log Time: Hours, 4 Minutes, 1 Seconds Restarts: 1 Total Connections:824 SSL Connections: 824 Peak Concurrent Connections: 6 Total Operations: 1806 Total Results:1805 Overall Performance: 99.9% Searches: 968(4.02/sec) (241.00/min) Modifications:5 (0.02/sec) (1.24/min) Adds: 0 (0.00/sec) (0.00/min) Deletes: 0 (0.00/sec) (0.00/min) Mod RDNs: 0 (0.00/sec) (0.00/min) Compares: 0 (0.00/sec) (0.00/min) Binds:833 (3.46/sec) (207.39/min) Proxied Auth Operations: 0 Persistent Searches: 1 Internal Operations: 0 Entry Operations: 0 Extended Operations: 0 Abandoned Requests: 0 Smart Referrals Received: 0 VLV Operations: 0 VLV Unindexed Searches: 0 SORT Operations: 0 Entire Search Base Queries: 0 Unindexed Searches: 1 This looks like a promising way to find out the reason, thanks John. However, I see just one unindexed search. Is the access log complete? Previously I see that the sync takes 900 seconds/15 minutes, but there is only 4 minutes the access log. Note that it it may take some time until the log is dumped. I think it would be also useful to run the analyzer with -ula flags as Rob suggested earlier to find out the unindexed searches (if any). What I find interesting is that JIRA does a lot of LDAP BINDs. Can the problem be in longer BINDs than with than expected (compared to for example plain LDAP servers)? Performance-wise, it would be I think better if JIRA does just one BIND and run all the LDAP searches the established connection. But I do not know if it can be configured this way. Rich, Rob, I am wondering if the slow up is not really caused by the binds, we have several DS plugins tied to the BIND operation, it may be useful to analyze if they do not take too long. Martin signature.asc Description: Message signed with OpenPGP using GPGMail ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] IPA Load Problems?
On 09/04/2013 07:58 AM, John Moyer wrote: It was our opinion that it wasn't an index issue. I cleared the logs from the IPA server, and then just ran a JIRA sync with the server. I gave Rich the log file from my IPA for that sync. I can't find the exact conversation, but we determined that JIRA was connecting to LDAP some 1000 times or so to do the sync. Right. For every single entry in IPA (user and group), JIRA LDAP sync does - connect/bind/search/unbind/disconnect. This is horribly inefficient, but it is what it is, and apparently other apps work the same way (nexus? svn?), so this would be a good avenue to investigate performance. The logs didn't show but one search done that didn't have an index which is why we concluded it wasn't an index issue. Adding indexing did help, but not much, and not nearly enough to make the performance acceptable. Thanks, _ John Moyer Director, IT Operations On Sep 4, 2013, at 9:51 AM, Martin Kosek mko...@redhat.com mailto:mko...@redhat.com wrote: Ah, ok. One of the reasons why I was poking to this thread is exactly this ticket. It does not contain much information _what exactly_ is making IPA performance poor - whether it is missing indices (which ones?) or some issue in IPA plugins during binds, etc. Without more information, we do not know what to fix, what to improve. Martin On 09/04/2013 02:01 PM, John Moyer wrote: Martin, I apologize there was a large offline conversation between Rich and myself. Rich was kind enough to help me through some of my issues. We did a lot more tests and poking and prodding. We discovered that IPA is not as efficient when dealing with large number of connections. Most of my load inefficiently reconnect to IPA over and over and over and though LDAP can deal with this fairly efficiently, IPA apparently drops to it's knees. A ticket was opened to addressed this issue. https://fedorahosted.org/freeipa/ticket/3892 Thanks, _ John Moyer Director, IT Operations Digital Reasoning Systems, Inc. john.mo...@digitalreasoning.com Office:703.678.2311 Mobile:240.460.0023 Fax:703.678.2312 www.digitalreasoning.com On Sep 4, 2013, at 3:44 AM, Martin Kosek mko...@redhat.com wrote: On 08/30/2013 11:08 PM, John Moyer wrote: Well IPA has machine entries on some test clusters that I'm rolling IPA out on (20 machines maybe) but the user base is the same (about 80 ~ 100) accounts with maybe 40 to 50 groups? I've stood up a clone of the jira server along with IPA. I cleared my logs and then did the sync and ran the log analyzer on it. These stats are pretty much ONLY for that jira sync I don't have any other connections pointed to it. Start of Log:30/Aug/2013:15:57:13 End of Log: 30/Aug/2013:16:01:14 Processed Log Time: Hours, 4 Minutes, 1 Seconds Restarts: 1 Total Connections:824 SSL Connections: 824 Peak Concurrent Connections: 6 Total Operations: 1806 Total Results:1805 Overall Performance: 99.9% Searches: 968(4.02/sec) (241.00/min) Modifications:5 (0.02/sec) (1.24/min) Adds: 0 (0.00/sec) (0.00/min) Deletes: 0 (0.00/sec) (0.00/min) Mod RDNs: 0 (0.00/sec) (0.00/min) Compares: 0 (0.00/sec) (0.00/min) Binds:833 (3.46/sec) (207.39/min) Proxied Auth Operations: 0 Persistent Searches: 1 Internal Operations: 0 Entry Operations: 0 Extended Operations: 0 Abandoned Requests: 0 Smart Referrals Received: 0 VLV Operations: 0 VLV Unindexed Searches: 0 SORT Operations: 0 Entire Search Base Queries: 0 Unindexed Searches: 1 This looks like a promising way to find out the reason, thanks John. However, I see just one unindexed search. Is the access log complete? Previously I see that the sync takes 900 seconds/15 minutes, but there is only 4 minutes the access log. Note that it it may take some time until the log is dumped. I think it would be also useful to run the analyzer with -ula flags as Rob suggested earlier to find out the unindexed searches (if any). What I find interesting is that JIRA does a lot of LDAP BINDs. Can the problem be in longer BINDs than with than expected (compared to for example plain LDAP servers)? Performance-wise, it would be I think better if JIRA does just one BIND and run all the LDAP searches the established connection. But I do not know if it can be configured this way. Rich, Rob, I am wondering if the slow up is not really caused by the binds, we have several DS plugins tied to the BIND operation, it may be useful to analyze if they do not take too long. Martin ___ Freeipa-users
Re: [Freeipa-users] Incorrect user information
You may want to check out the sss_cache package in the sssd-tools package. It looks to be in the base channel for RHEL5 Server and optional channel for RHEL6 Server. This tool will allow you to invalidate/manipulate the sssd cache. -Chris - Original Message - From: cbul...@gmail.com To: freeipa-users@redhat.com Sent: Wednesday, September 4, 2013 10:40:29 AM Subject: [Freeipa-users] Incorrect user information Hi, We have a freeipa server (RedHat 6.3, freeipa:3.0.0-26) and freeipa client (RedHat 5.9, freeipa client 2.1.3.-5) working in our test testing scenario without further problems. We are able to use SUDO, HBAC etc. Our problem is when we change a user info (Name or Last Name) and check it using the command: getent passwd id_user it showed us the older user information. We set entry_cache_user_timeout = 0 in sssd.conf file in order to clear the cache data but it did not work. Also we tried with: use_fully_qualified_domains attribute as recommend in: https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/SSSD-Troubleshooting.html#idp26289072 but it was not helpful. If we check the user info using ldapsearch command we can see the right user info information. Changing the uid or gid we see the new change right away. Any clue about this problem? Thanks! CBU ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
[Freeipa-users] Incorrect user information
Hi, We have a freeipa server (RedHat 6.3, freeipa:3.0.0-26) and freeipa client (RedHat 5.9, freeipa client 2.1.3.-5) working in our test testing scenario without further problems. We are able to use SUDO, HBAC etc. Our problem is when we change a user info (Name or Last Name) and check it using the command: getent passwd id_user it showed us the older user information. We set entry_cache_user_timeout = 0 in sssd.conf file in order to clear the cache data but it did not work. Also we tried with: use_fully_qualified_domains attribute as recommend in: https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/SSSD-Troubleshooting.html#idp26289072 but it was not helpful. If we check the user info using ldapsearch command we can see the right user info information. Changing the uid or gid we see the new change right away. Any clue about this problem? Thanks! CBU ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
[Freeipa-users] Replication causing long etimes
I am experiencing some long execution times, and I'm wondering if anyone can give me some insight. We are running FreeIPA 3.0.0-26 on Redhat 6.1. We have multimaster replication running among 4 hosts. We have approv 100 users, 25 usergroups and hostgroups, and approx 2000 hosts in a single domain. We noticed that some DNS queries were timing out periodically. When I investigated further, I found several of the DNS requests in the access log [04/Sep/2013:13:42:24 -0300] conn=122491 op=3888679 SRCH base=idnsName=compute- 1.amazonaws.com,idnsname=prod.ca2.example.com,cn=dns,dc=example,dc=com scope=0 filter= (objectClass=idnsRecord) attrs=ALL [04/Sep/2013:13:42:44 -0300] conn=122491 op=3888679 RESULT err=32 tag=101 nentri es=0 etime=20 There are a lot of those, as expected, since we first noticed this issue with DNS. Then I found this ... [04/Sep/2013:13:42:23 -0300] conn=368561 op=9 EXT oid=2.16.840.1.113730.3.5.5 name=Netscape Replication End Session [04/Sep/2013:13:42:44 -0300] conn=368561 op=9 RESULT err=0 tag=120 nentries=0 etime=22 and lots of this ... [04/Sep/2013:13:42:26 -0300] conn=368604 op=0 BIND dn= method=sasl version=3 mech=GSSAPI [04/Sep/2013:13:42:44 -0300] conn=368604 op=0 RESULT err=14 tag=97 nentries=0 etime=18, SASL bind in progress So, is my SASL bind causing the replication to go long, or is the replication taking a long time and causing the hang? Is there a way I can see the details of the replication? There is not a lot of changes going on that require replication with regards to dns, users, hosts, etc, so I'm not sure why it would take so long. Also, can I remove the SASL bind and just add a replication user to the dse.ldif to remove the requirement for kerberos for replication? Terry -- Terry Soucy - Systems Engineer Salesforce MarketingCloud - http://www.salesforce.com (o) 506.631.7445 (c) 506.609.3247 | (e) tso...@salesforce.com ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Replication causing long etimes
On 09/04/2013 12:18 PM, Terry Soucy wrote: I am experiencing some long execution times, and I'm wondering if anyone can give me some insight. We are running FreeIPA 3.0.0-26 on Redhat 6.1. We have multimaster replication running among 4 hosts. We have approv 100 users, 25 usergroups and hostgroups, and approx 2000 hosts in a single domain. We noticed that some DNS queries were timing out periodically. When I investigated further, I found several of the DNS requests in the access log [04/Sep/2013:13:42:24 -0300] conn=122491 op=3888679 SRCH base=idnsName=compute- 1.amazonaws.com http://1.amazonaws.com,idnsname=prod.ca2.example.com http://prod.ca2.example.com,cn=dns,dc=example,dc=com scope=0 filter= (objectClass=idnsRecord) attrs=ALL [04/Sep/2013:13:42:44 -0300] conn=122491 op=3888679 RESULT err=32 tag=101 nentri es=0 etime=20 There are a lot of those, as expected, since we first noticed this issue with DNS. Then I found this ... [04/Sep/2013:13:42:23 -0300] conn=368561 op=9 EXT oid=2.16.840.1.113730.3.5.5 name=Netscape Replication End Session [04/Sep/2013:13:42:44 -0300] conn=368561 op=9 RESULT err=0 tag=120 nentries=0 etime=22 and lots of this ... [04/Sep/2013:13:42:26 -0300] conn=368604 op=0 BIND dn= method=sasl version=3 mech=GSSAPI [04/Sep/2013:13:42:44 -0300] conn=368604 op=0 RESULT err=14 tag=97 nentries=0 etime=18, SASL bind in progress So, is my SASL bind causing the replication to go long, or is the replication taking a long time and causing the hang? I don't know. DNS could also be related. If you can, please try to get a stack trace of the ns-slapd process during the time interval during which nothing appears to be happening. http://port389.org/wiki/FAQ#Debugging_Hangs Is there a way I can see the details of the replication? You can use the replication logging level http://port389.org/wiki/FAQ#Troubleshooting But I don't know if the problem is specifically replication related There is not a lot of changes going on that require replication with regards to dns, users, hosts, etc, so I'm not sure why it would take so long. Also, can I remove the SASL bind and just add a replication user to the dse.ldif to remove the requirement for kerberos for replication? You technically could with 389, but I don't know if that is supported with IPA. Terry -- Terry Soucy - Systems Engineer Salesforce MarketingCloud - http://www.salesforce.com (o) 506.631.7445 (c) 506.609.3247 | (e) tso...@salesforce.com mailto:tso...@salesforce.com ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
[Freeipa-users] Ldap schema
I have the radius.schema file how do I add that into my ldap schema on IPA server. I see several ldif files /etc/dirsrv/instance/schema but they are ldif files If I can extend my schema integration to free radius should be easy. Thank you. radius.schema Description: Binary data ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Ldap schema
On 09/04/2013 05:41 PM, Jason Prouty wrote: I have the radius.schema file how do I add that into my ldap schema on IPA server. I see several ldif files /etc/dirsrv/instance/schema but they are ldif files If I can extend my schema integration to free radius should be easy. Is there a reason you ignored the prior response? -- John ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Ldap schema
This is the AV-Pair I would like to implement to pass back to radius. dn: cn=priv-15,ou=cisco,ou=radius,dc=example,dc=com objectClass: radiusObjectProfile objectClass: radiusprofile cn: priv-15 radiusReplyItem: cisco-avpair = shell:priv-lvl=15 -Original Message- From: John Dennis [mailto:jden...@redhat.com] Sent: Wednesday, September 04, 2013 4:26 PM To: Jason Prouty Cc: freeipa-users@redhat.com Subject: Re: [Freeipa-users] Ldap schema On 09/04/2013 05:41 PM, Jason Prouty wrote: I have the radius.schema file how do I add that into my ldap schema on IPA server. I see several ldif files /etc/dirsrv/instance/schema but they are ldif files If I can extend my schema integration to free radius should be easy. Is there a reason you ignored the prior response? -- John ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
