On 04/12/2017 06:58 PM, William Brown wrote:
> On Wed, 2017-04-12 at 17:02 -0400, Mark Reynolds wrote:
>> Hello,
>>
>> This is a beta version of a replication diff tool written in python.
>>
>> Design page (this needs updating - I hope to get that done tonight)
>>
>> http://www.port389.org/docs/389ds/design/repl-diff-tool-design.html
>>
>>
>> Current usage:
>>
>> -v, --verbose Verbose output
>> -o FILE, --outfile=FILE
>> The output file
>> -D BINDDN, --binddn=BINDDN
>> The Bind DN (REQUIRED)
>> -w BINDPW, --bindpw=BINDPW
>> The Bind password (REQUIRED)
>> -h MHOST, --master_host=MHOST
>> The Master host (default localhost)
>> -p MPORT, --master_port=MPORT
>> The Master port (default 389)
>> -H RHOST, --replica_host=RHOST
>> The Replica host (REQUIRED)
>> -P RPORT, --replica_port=RPORT
>> The Replica port (REQUIRED)
>> -b SUFFIX, --basedn=SUFFIX
>> Replicated suffix (REQUIRED)
>> -l LAG, --lagtime=LAG
>> The amount of time to ignore inconsistencies
>> (default
>> 300 seconds)
>> -Z CERTDIR, --certdir=CERTDIR
>> The certificate database directory for startTLS
>> connections
>> -i IGNORE, --ignore=IGNORE
>> Comma separated list of attributes to ignore
>> -M MLDIF, --mldif=MLDIF
>> Master LDIF file (offline mode)
>> -R RLDIF, --rldif=RLDIF
>> Replica LDIF file (offline mode)
>>
>> |Examples: python repl-diff.py -D "cn=directory manager" -w PASSWORD -h
>> localhost -p 389 -H remotehost -P -b "dc=example,dc=com" ||python
>> repl-diff.py -D "cn=directory manager" -w PASSWORD -h localhost
>> -p 389 -H remotehost -P -b "dc=example,dc=com" -Z
>> /etc/dirsrv/slapd-localhost|
>> |python repl-diff.py -M /tmp/master.ldif -R /tmp/replica.ldif |
>>
>>
>> How long the tool takes to run depends on the number of entries per
>> database. See performance numbers below
>>
>> Entries per Replica Time
>>
>> -
>> 100k40 seconds
>> 500k3m 30secs
>> 1 million 7m 30secs
>> 2 million 14 minutes
>> 10 million ~70 minutes
>>
>>
>>
>> I'd be very interested in feedback, RFE's, and bugs.
> Hey mate,
>
> The tool looks great, awesome work on this. Really impressive that you
> got it to 70 minutes for 10 million entries.
And in reality its 20 million entries (master + replica)
>
> How responsive is the server during this process? We aren't going to
> cause some odd resource exhaustion?
Not really :) We don't have to rely on server side sorting, and it's
just a paged result search - so it breaks up the load (slightly). But,
it's still expensive because it is returning all the entries, but I
didn't see any extreme CPU usage.
>
> import optparse
>
> With python, optparse is deprecated. Can we use argparse instead? It's
> nearly identical. Lots of examples of this in dsctl.
Easy, no problem.
>
> With connect to replicas, some sites may only have ldaps (provided by a
> load balancer). So our scripts should really be taking an LDAPurl, a
> certdir, and a starttls flag. Because ldaps://localhost + certdir is a
> valid option, but if we force call start_tls_s(), we break it.
Good idea
> As well,
> someone may use ldapi:// etc. It also saves on port options and more
> flags to the cli because we can do ldap://localhost:30389 etc.
>
> Hope that helps, I'll be happy to review again later!
>
> For now, I think our strategy with this should be to add it to
> 389-ds-base, and later we can move this into lib389 when we can. How
> does that sound?
I think it should always be a standalone tool, but we can tie it in with
lib389 (move the main guts out of the tool and into lib389)
>
>
>
> ___
> 389-devel mailing list -- 389-devel@lists.fedoraproject.org
> To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org
___
389-devel mailing list -- 389-devel@lists.fedoraproject.org
To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org
