Re: Provider-Consumer replication 2.4 OLC (second attempt)

[Quanah Gibson-Mount]

--On Friday, October 28, 2016 9:50 AM -0400 Ted Hyde  
wrote:



Quanah - thanks for the response. Sorry to insult if I did - but thank
you, I DID read the admin guide. Which as you have also pointed out uses
slapd.conf examples. Since I am not knee-deep in commercial OpenLdap
configuration every day (I am just a lowly IT admin, not a
paid-to-openldap-person) I would disagree in that your comment that
"conversion to cn=config" process isn't trivial, personally I get quite
swamped by it, but push through as best I can. But if you're offering to
convert my sample configs for me, I'd be happy to share them with you.


You can convert your sample configs via the slaptest command, as documented.


Or
perhaps you could help the community by providing some OLC config
examples for the admin guide, that way us peons would be able to use that
as our only official source instead of having to google to find "Random"
help.


My point was more that converting examples in the admin guide from 
slapd.conf to cn=config is fairly trivial.


For example, if we look at section 18.3.1.2 in the admin guide:

   database mdb
   maxsize 1073741824
   suffix dc=Example,dc=com
   rootdn dc=Example,dc=com
   directory /var/ldap/db
   index objectclass,entryCSN,entryUUID eq

   overlay syncprov
   syncprov-checkpoint 100 10
   syncprov-sessionlog 100

this is rather trivially converted to:

dn: olcDatabase={1}mdb
objectClass: olcDatabaseConfig
objectClass: olcMdbConfig
olcDatabase: {1}mdb
olcSuffix: dc=example,dc=com
olcRootDN: dc=example,dc=com
olcDbDirectory: /var/ldap/db
olcDbIndex: objectClass eq
olcDbIndex: entryUUID eq
olcDbIndex: entryCSN eq

dn: olcOverlay={0}syncprov,olcDatabase={1}mdb,cn=config
objectClass: olcOverlayConfig
objectClass: olcSyncProvConfig
olcSpCheckpoint: 100 10
olcSpSessionlog: 100


etc.  Converting to cn=config from slapd.conf is not particularly difficult.



I *can* move to refreshAndPersist; but the service provides two
documented options (information I got from reading the admin guide), the
description for refreshOnly best fits my scenario and needs. I didn't
read any reason as to *not* use - perhaps you're aware of a bug report
that refreshOnly is broken?


I'm aware that operating in refreshOnly is problematic, and it is advise 
not to use it.  If you want to persist in using it, I certainly can't stop 
you. ;)  If/when I find time to rewrite the admin guide, removing it from 
the examples will be one of the first steps I take.



Perhaps my research (which I'm sure isn't as broad as yours) just seemed
to point to the fact that openldap will/may be depreciating the
slapd.conf procedures, and that everyone should get on board with OLC as
soon as possible. While I can perform the setup with slapd.conf (as noted
in the admin guide), I was hoping to practice some useful technique I
could use in the future.


Again, as noted in the documentation, you can set up one time with 
slapd.conf, and then move forward with converting it to cn=config via 
slaptest, and then just use cn=config from that point forward, using ldap* 
commands to make updates as necessary.


If you want some further examples of cn=config, you may like the following:



Which has a basic cn=config layout for a standalone server with a suffix of 
"" and a few overlays loaded as a starting point.


You may also be interested in the tools I wrote for manipulating cn=config 
to use as examples:






etc.  While bits of it are specific to Zimbra, the ideas behind 
updating/modifying cn=config are universal.


On the documentation, I would note that it is a community effort, and 
anyone can contribute updates, etc, via the ITS system.  The sad fact is, 
many people complain about the documentation, but very few ever step up and 
contribute back, which means that it often languishes.


I hope the above helps.

Regards,
Quanah


--

Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:

Re: Provider-Consumer replication 2.4 OLC (second attempt)

[Ted Hyde]

  
  
On 10/28/2016 8:00 AM,
  openldap-technical-requ...@openldap.org wrote:


  Message: 10
Date: Thu, 27 Oct 2016 11:45:46 -0700
From: Quanah Gibson-Mount <qua...@symas.com>
To: Ted Hyde <laser...@gmail.com>, openldap-technical@openldap.org
Subject: Re: Provider-Consumer replication 2.4 OLC (second attempt)
Message-ID: <2B151020D2CC5C1D45A0A2C9@[192.168.1.19]>
Content-Type: text/plain; charset=us-ascii; format=flowed

--On Tuesday, October 25, 2016 11:34 AM -0400 Ted Hyde <laser...@gmail.com> 
wrote:


  
> Greets - (first post, second time. Don't know if it's being moderated or
> dropped, will keep trying).

  
  It ended up in my spam folder for some reason.


  
> After installation, I have two functioning standalone servers. During my
> research, I found two conflicting pieces of information. I prefer to
> perform "refreshOnly" instead of "refreshAndPersist", so some sources say
> ONLY the consumers need configuration, a couple said both sides need
> configuration AND a number of additional indices. This is likely where my
> problems arise.

  
  I strongly advise only using refreshAndPersist.  I'm not sure why you are 
referring to random how-not-to-do things on the interweb.  You should use 
the OpenLDAP admin guide:




  
> Any thoughts?

  
  Read the admin guide, as noted above.  The examples are still slapd.conf, 
but you can trivially translate those to what are necessary for cn=config.

--Quanah

--

Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:


Quanah - thanks for the response. Sorry to insult if I did - but
  thank you, I DID read the admin guide. Which as you have also
  pointed out uses slapd.conf examples. Since I am not knee-deep in
  commercial OpenLdap configuration every day (I am just a lowly IT
  admin, not a paid-to-openldap-person) I would disagree in that
  your comment that "conversion to cn=config" process isn't trivial,
  personally I get quite swamped by it, but push through as best I
  can. But if you're offering to convert my sample configs for me,
  I'd be happy to share them with you. Or perhaps you could help the
  community by providing some OLC config examples for the admin
  guide, that way us peons would be able to use that as our only
  official source instead of having to google to find "Random" help.
I *can* move to refreshAndPersist; but the service provides two
  documented options (information I got from reading the admin
  guide), the description for refreshOnly best fits my scenario and
  needs. I didn't read any reason as to *not* use - perhaps you're
  aware of a bug report that refreshOnly is broken?

Perhaps my research (which I'm sure isn't as broad as yours) just
  seemed to point to the fact that openldap will/may be depreciating
  the slapd.conf procedures, and that everyone should get on board
  with OLC as soon as possible. While I can perform the setup with
  slapd.conf (as noted in the admin guide), I was hoping to practice
  some useful technique I could use in the future.
So again thanks, but if it doesn't work out for me, perhaps I
  read on one of the last pages "check out 389 or ApacheDS instead".
Ted.

  



	
		
			

			
		
		
			
This email has been checked for viruses by Avast antivirus software.
www.avast.com
			
		
	

Re: Provider-Consumer replication 2.4 OLC (second attempt)

[Quanah Gibson-Mount]

--On Tuesday, October 25, 2016 11:34 AM -0400 Ted Hyde  
wrote:



Greets - (first post, second time. Don't know if it's being moderated or
dropped, will keep trying).


It ended up in my spam folder for some reason.


After installation, I have two functioning standalone servers. During my
research, I found two conflicting pieces of information. I prefer to
perform "refreshOnly" instead of "refreshAndPersist", so some sources say
ONLY the consumers need configuration, a couple said both sides need
configuration AND a number of additional indices. This is likely where my
problems arise.


I strongly advise only using refreshAndPersist.  I'm not sure why you are 
referring to random how-not-to-do things on the interweb.  You should use 
the OpenLDAP admin guide:





Any thoughts?


Read the admin guide, as noted above.  The examples are still slapd.conf, 
but you can trivially translate those to what are necessary for cn=config.


--Quanah

--

Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:

Provider-Consumer replication 2.4 OLC (second attempt)

[Ted Hyde]

  
  
Greets - (first post, second time. Don't know if it's being
moderated or dropped, will keep trying).

I have been using OpenLdap for some time - in master-slave
(provider/consumer) situations prior 2.3 (thus using slapd.conf) and
post 2.3 only as standalones (which I've found always easy). 

This is my first foray into a 2.4 replication, and it's just not
working out for me. Admittedly, I KNOW I have missed an important
step somewhere (like turning on an index or similar) - there are far
too many "I'm leaving a personal step out" tutorials on the great
web, or ones that show some really good paths, but conflict - so it
is entirely possible that my mashing together of search results has
been the cause of the problem.

I am starting on bare metal, real machines, (not VM's, not wild
hosting, etc) on a very isolated sandbox environment. Debian Jessie,
and no extra fun added in to complicate matters. (Thus a bare metal
install with ssh-server, slapd, ldap-utils from packages). Only the
default schemas are needed for my purposes.

Servers 172.18.0.1 and 172.18.0.2 are my instances (examples), where
0.1 is the "master/provider" and 0.2 is a "slave/consumer". The
intent is to have multiple consumer servers that reference the
provider; all changes/additions are to be referred back to the
provider 0.1, and although there should be good network connectivity
between all hosts, the purpose of this install is to have local
servers (the slave/consumers) acting for each local installation for
all local requests, not for speed, but for failover. IF there is a
loss of sync for a while, it is not a huge concern, as the changes
will be infrequent, and we're dealing with perhaps a record set of
100-200 accounts, mirrored to all sites.

After installation, I have two functioning standalone servers.
During my research, I found two conflicting pieces of information. I
prefer to perform "refreshOnly" instead of "refreshAndPersist", so
some sources say ONLY the consumers need configuration, a couple
said both sides need configuration AND a number of additional
indices. This is likely where my problems arise.

Here's what I popped into "sync.ldif" and loaded ONLY TO THE
CONSUMER via ldapadd:

dn: olcDatabase={1}mdb,cn=config
changetype: modify
add: olcSyncrepl
olcSyncrepl: rid=100
  provider="ldap://172.18.0.1:389/"
  type=refreshOnly
  retry="5 5 300 +"
  searchbase="dc=mydomain,dc=com"
  bindmethod=simple
  binddn="cn=replication,dc=mydomain,dc=com"
  credentials=replication
-
add: olcUpdateRef
olcUpdateRef: ldap://172.18.0.1

{end of sync.ldif-this line not included in file.}

It loads into the consumer (172.18.0.2) without error,  and change
referrals DO get forwarded to the provider (0.1). I am using the
back-mdb, and the replication user is functional as a write-write
security object on 0.1 (for simplicity, at the moment it is a copy
of admin, with all rights and privs).

At no time did the consumer's db populate with the replication
information. There was no indication that the consumer even tried.
(finding log information however was sketchy at best). There was no
indication from the provider that a consumer was calling in and was
attempting to replicate. Again, connectivity is fine, as referrals
work as expected.

If I were to backup the provider db (slapcat) and restore it to the
consumer, that db does restore, and the consumer instance starts
handling local auth requests (I intentionally disconnected 0.1 to
prove that) - and I can still add an account to the provider,
(direct or referred) but even after multiple restarts, (the retry
directive says every 5 seconds 5 times, then every 300 sec to poll),
no consumer updates occurred.

Moving forward to finding an obscure note about having to install
the syncrepl module on ONLY the provider, not the consumer (?) but
related to refreshAndPersist, I tried it anyway:
{provider syncsetup.ldif:}
dn: cn=module{0},cn=config
changetype: modify
add: olcModuleLoad
olcModuleLoad: syncprov


however still no change to things. I have not tried this on the
consumer (yet) as I've found no example that says to do so. (don't
know why you'd load syncprov on a consumer anyway).
I am quite familiar with the 2.4admin guide - particularly in how
sparse the documentation examples are for using OLC directives -
rather frustrating that even though the version is pushing OLC
(cn=config), it still holds tight to slapd.conf examples. I do not
want a slapd.conf solution attempt.

Any thoughts?

Thanks,
Ted.
  



	
		
			

			
		
		
			
This email has been checked for viruses by