[389-devel] Re: Logging performance improvement
On Thu, 2016-06-30 at 20:53 -0600, Rich Megginson wrote: > On 06/30/2016 08:14 PM, William Brown wrote: > > On Thu, 2016-06-30 at 20:01 -0600, Rich Megginson wrote: > >> On 06/30/2016 07:52 PM, William Brown wrote: > >>> Hi, > >>> > >>> I've been thinking about this for a while, so I decided to dump my > >>> thoughts to a document. I think I won't get to implementing this for a > >>> while, but it would really help our server performance. > >>> > >>> http://www.port389.org/docs/389ds/design/logging-performance-improvement.html > >> Looks good. Can we quantify the current log overhead? > > Sure, I could probably sit down and work out a way to bench mark > > this . > > > > But without the alternative being written, hard to say. I could always > > patch out logging and drop the lock in a hacked build so we can show > > what "without logging contention" looks like? > > That's only one part of it - you'd have to figure out some way to get > rid of the overhead of the formatting and flushing in the operation > threads too. > This change would move the formatting into the thread that does the write + flush. Currently, we format the message in the operation thread, and send it to the log buffer, and every now and again, an "unlucky" operation thread has to stall to write the buffer However, I'm proposing we send unformatted time struct + unformatted message to the queue, then the actual thread which is separate does the dequeue and format. > I suppose you could just write it and see what happens. Well, a hacked up test that has slapi_log_access return 0 immediately (IE does not take the lock) showed a 6% performance improvement on my laptop. On a higher cpu count system, this would exceed that number easily, as more cpus == more cost to acquire the lock. I've put the numbers on the design doc page, but I think with this, the proof would be in the implementation. We also likely wouldn't see the benefit of this as much until we remove other serialisation points in the code IE ldbm vs lmdb, plugins that take locks, etc. So it's only a 6% possible gain now, but it may be 10% or more in the future in combination with other efforts such as nunc-stans. > > > > >>> > >>> -- > >>> 389-devel mailing list > >>> 389-devel@lists.fedoraproject.org > >>> https://lists.fedoraproject.org/admin/lists/389-devel@lists.fedoraproject.org > >> > >> -- > >> 389-devel mailing list > >> 389-devel@lists.fedoraproject.org > >> https://lists.fedoraproject.org/admin/lists/389-devel@lists.fedoraproject.org > > > > > > -- > > 389-devel mailing list > > 389-devel@lists.fedoraproject.org > > https://lists.fedoraproject.org/admin/lists/389-devel@lists.fedoraproject.org > > > -- > 389-devel mailing list > 389-devel@lists.fedoraproject.org > https://lists.fedoraproject.org/admin/lists/389-devel@lists.fedoraproject.org -- Sincerely, William Brown Software Engineer Red Hat, Brisbane signature.asc Description: This is a digitally signed message part -- 389-devel mailing list 389-devel@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/389-devel@lists.fedoraproject.org
[389-devel] Re: Logging performance improvement
On 06/30/2016 08:14 PM, William Brown wrote: On Thu, 2016-06-30 at 20:01 -0600, Rich Megginson wrote: On 06/30/2016 07:52 PM, William Brown wrote: Hi, I've been thinking about this for a while, so I decided to dump my thoughts to a document. I think I won't get to implementing this for a while, but it would really help our server performance. http://www.port389.org/docs/389ds/design/logging-performance-improvement.html Looks good. Can we quantify the current log overhead? Sure, I could probably sit down and work out a way to bench mark this . But without the alternative being written, hard to say. I could always patch out logging and drop the lock in a hacked build so we can show what "without logging contention" looks like? That's only one part of it - you'd have to figure out some way to get rid of the overhead of the formatting and flushing in the operation threads too. I suppose you could just write it and see what happens. -- 389-devel mailing list 389-devel@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/389-devel@lists.fedoraproject.org -- 389-devel mailing list 389-devel@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/389-devel@lists.fedoraproject.org -- 389-devel mailing list 389-devel@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/389-devel@lists.fedoraproject.org -- 389-devel mailing list 389-devel@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/389-devel@lists.fedoraproject.org
[389-devel] Re: Logging performance improvement
On Thu, 2016-06-30 at 20:01 -0600, Rich Megginson wrote: > On 06/30/2016 07:52 PM, William Brown wrote: > > Hi, > > > > I've been thinking about this for a while, so I decided to dump my > > thoughts to a document. I think I won't get to implementing this for a > > while, but it would really help our server performance. > > > > http://www.port389.org/docs/389ds/design/logging-performance-improvement.html > > Looks good. Can we quantify the current log overhead? Sure, I could probably sit down and work out a way to bench mark this . But without the alternative being written, hard to say. I could always patch out logging and drop the lock in a hacked build so we can show what "without logging contention" looks like? > > > > > > > -- > > 389-devel mailing list > > 389-devel@lists.fedoraproject.org > > https://lists.fedoraproject.org/admin/lists/389-devel@lists.fedoraproject.org > > > -- > 389-devel mailing list > 389-devel@lists.fedoraproject.org > https://lists.fedoraproject.org/admin/lists/389-devel@lists.fedoraproject.org -- Sincerely, William Brown Software Engineer Red Hat, Brisbane signature.asc Description: This is a digitally signed message part -- 389-devel mailing list 389-devel@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/389-devel@lists.fedoraproject.org
[389-devel] Re: Replication after full online init
On Thu, 2016-06-30 at 14:53 -0700, Noriko Hosoi wrote: > On 06/30/2016 12:45 AM, Ludwig Krispenz wrote: > > Hi William, > > > > the reason that after a total init the consumer does not have the > > latest state of the supplier RUV and is receiving updates based on the > > RUV at start of the total init is independent of the modrdn problem. > > When a supplier is performing a total init it is still accepting > > changes, the total init can take a while and there are scenarios where > > an entry which is already sent is updated before total init finishes. > > We cannot loose these changes. > OK... Then, RUV needs to be created at the time when the supplier > starts online init? > > The test case would be something like this? > 1. run online init on the supplier. > 2. do some operation like move entries against the supplier while the > online init is still running on the consumer. > 3. do some operation which depends upon the previous operation done in > the step 2. > 4. check the consumer is healthy or not. > > Isn't it a timestamp issue from which operation should be replayed after > the total update? Regardless of the way how to fix 48755, unless the > step 2 operation(s) are replayed after the online init is done, the > consumer could get broken/inconsistent? > It's not the "post init" operations I'm worried about. It's that operations that were part of the init to the consumer are replayed from the changelog. Operations that occurred after the init starts, definitely still need to be replayed, and this makes sense. Lets say we have: 1 - insert A 2 - insert ou=B 3 - modrdn A under ou=B 4 - insert C xx <<-- We start to transmit the data here. 5 - modrdn C Once the online init is complete, the master replays the log from event 1 -> 5 to the consumer, even though it should now be up to date at position 4. Previously we could not guarantee this because in the scenario above, A would have sorted before ou=B, by would not be able to be applied because the consumer hadn't seen B yet. So after the init, the consumer would have B and C, but not A, so we had to replay 1 -> 4 to fix this up. So I am suggesting that when we begin the online init we set the RUV of the consumer to match the CSN of the master at the moment we begin the transmission of data, so that we only need to replay event 5+, rather than 1->5+ Does that make sense? -- Sincerely, William Brown Software Engineer Red Hat, Brisbane signature.asc Description: This is a digitally signed message part -- 389-devel mailing list 389-devel@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/389-devel@lists.fedoraproject.org
[389-devel] Re: Logging performance improvement
On 06/30/2016 07:52 PM, William Brown wrote: Hi, I've been thinking about this for a while, so I decided to dump my thoughts to a document. I think I won't get to implementing this for a while, but it would really help our server performance. http://www.port389.org/docs/389ds/design/logging-performance-improvement.html Looks good. Can we quantify the current log overhead? -- 389-devel mailing list 389-devel@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/389-devel@lists.fedoraproject.org -- 389-devel mailing list 389-devel@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/389-devel@lists.fedoraproject.org
[389-devel] Logging performance improvement
Hi, I've been thinking about this for a while, so I decided to dump my thoughts to a document. I think I won't get to implementing this for a while, but it would really help our server performance. http://www.port389.org/docs/389ds/design/logging-performance-improvement.html -- Sincerely, William Brown Software Engineer Red Hat, Brisbane signature.asc Description: This is a digitally signed message part -- 389-devel mailing list 389-devel@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/389-devel@lists.fedoraproject.org
[389-devel] Re: Replication after full online init
On 06/30/2016 12:45 AM, Ludwig Krispenz wrote: Hi William, the reason that after a total init the consumer does not have the latest state of the supplier RUV and is receiving updates based on the RUV at start of the total init is independent of the modrdn problem. When a supplier is performing a total init it is still accepting changes, the total init can take a while and there are scenarios where an entry which is already sent is updated before total init finishes. We cannot loose these changes. OK... Then, RUV needs to be created at the time when the supplier starts online init? The test case would be something like this? 1. run online init on the supplier. 2. do some operation like move entries against the supplier while the online init is still running on the consumer. 3. do some operation which depends upon the previous operation done in the step 2. 4. check the consumer is healthy or not. Isn't it a timestamp issue from which operation should be replayed after the total update? Regardless of the way how to fix 48755, unless the step 2 operation(s) are replayed after the online init is done, the consumer could get broken/inconsistent? Thanks, --noriko Therfor the update resolution/ entry state resolution on the consumer side has to handle this, ignore changes already applied and apply new changes. And it handles it, if there are bugs they have to be fixed. Now, I am no longer sure if the fix for 48755 handles correctly all modrdns received after the id list was prepared, the parentid might change while the total init is on progress. This brings up my origimal suggestion to handle the modrdn problems also on the consumer side. Ludwig On 06/30/2016 02:34 AM, William Brown wrote: Hi, Now thathttps://fedorahosted.org/389/ticket/48755 is merged, I would like to discuss the way we handle CSN with relation to this master. As I'm not an expert on this topic, I want to get the input of everyone about this. Following this document: http://www.port389.org/docs/389ds/design/changelog-processing-in-repl-state-sending-updates.html As I understand it, after a full online init, the replica that consumed the changes does not set it's CSN to match the CSN of the master that sent the changes. As a result, after the online init, this causes a number of changes to be replicated from the sending master to the consumer. These are ignored by the URP, and we continue. However, in a number of cases these are *not* ignored, and have caused us some bugs in replication in the past. We also have some failing changes that are skipped, which could in certain circumstance lead to inconsistency in replicas. We have gone to a lot of effort to be able to skip changes, to handle the case above. The reason was is that if there was a modrdn performed, and the entry ID of the entry that was moved was less than the new parent ID, this *had* to be skipped, so that after the online init the modrdn change was replayed and applied to the consumer. Since 48755 which sorts based on the parent ID, this seems to no longer be an issue. So we don't need to have the master replay it's changelog out to the consumer, because the consumer is now a literal clone of the data. So, is there a reason for us to leave the CSN of the consumer low to allow this replay to occur? Or can we alter the behaviour of the consumer to set it's CSN to the CSN of the sending master, so that we don't need to replay these changes? -- 389-devel mailing list 389-devel@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/389-devel@lists.fedoraproject.org -- Red Hat GmbH,http://www.de.redhat.com/, Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander -- 389-devel mailing list 389-devel@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/389-devel@lists.fedoraproject.org -- 389-devel mailing list 389-devel@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/389-devel@lists.fedoraproject.org
[389-devel] please review: Ticket48907 - admin server - register-ds-admin.pl fails to find local config DS
https://fedorahosted.org/389/ticket/48907 https://fedorahosted.org/389/attachment/ticket/48907/0001-Ticket-48907-register-ds-admin-fails-to-find-local-c.patch -- 389-devel mailing list 389-devel@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/389-devel@lists.fedoraproject.org
[389-devel] please review: Ticket 48306 - Admin Server -- perl module conditional test is not conditional when checking SELinux policies
https://fedorahosted.org/389/ticket/48306 https://fedorahosted.org/389/attachment/ticket/48306/0001-Ticket-48306-perl-module-conditional-test-is-not-con.patch -- 389-devel mailing list 389-devel@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/389-devel@lists.fedoraproject.org
[389-devel] Re: Replication after full online init
Hi William, the reason that after a total init the consumer does not have the latest state of the supplier RUV and is receiving updates based on the RUV at start of the total init is independent of the modrdn problem. When a supplier is performing a total init it is still accepting changes, the total init can take a while and there are scenarios where an entry which is already sent is updated before total init finishes. We cannot loose these changes. Therfor the update resolution/ entry state resolution on the consumer side has to handle this, ignore changes already applied and apply new changes. And it handles it, if there are bugs they have to be fixed. Now, I am no longer sure if the fix for 48755 handles correctly all modrdns received after the id list was prepared, the parentid might change while the total init is on progress. This brings up my origimal suggestion to handle the modrdn problems also on the consumer side. Ludwig On 06/30/2016 02:34 AM, William Brown wrote: Hi, Now that https://fedorahosted.org/389/ticket/48755 is merged, I would like to discuss the way we handle CSN with relation to this master. As I'm not an expert on this topic, I want to get the input of everyone about this. Following this document: http://www.port389.org/docs/389ds/design/changelog-processing-in-repl-state-sending-updates.html As I understand it, after a full online init, the replica that consumed the changes does not set it's CSN to match the CSN of the master that sent the changes. As a result, after the online init, this causes a number of changes to be replicated from the sending master to the consumer. These are ignored by the URP, and we continue. However, in a number of cases these are *not* ignored, and have caused us some bugs in replication in the past. We also have some failing changes that are skipped, which could in certain circumstance lead to inconsistency in replicas. We have gone to a lot of effort to be able to skip changes, to handle the case above. The reason was is that if there was a modrdn performed, and the entry ID of the entry that was moved was less than the new parent ID, this *had* to be skipped, so that after the online init the modrdn change was replayed and applied to the consumer. Since 48755 which sorts based on the parent ID, this seems to no longer be an issue. So we don't need to have the master replay it's changelog out to the consumer, because the consumer is now a literal clone of the data. So, is there a reason for us to leave the CSN of the consumer low to allow this replay to occur? Or can we alter the behaviour of the consumer to set it's CSN to the CSN of the sending master, so that we don't need to replay these changes? -- 389-devel mailing list 389-devel@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/389-devel@lists.fedoraproject.org -- Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander -- 389-devel mailing list 389-devel@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/389-devel@lists.fedoraproject.org
