[SSSD] Re: [PATCH] SYSDB: Check changed virtual attributes before modified timestamp
On Tue, Aug 09, 2016 at 02:16:11PM +0200, Jakub Hrozek wrote: > On Fri, Aug 05, 2016 at 01:28:36PM +0200, Lukas Slebodnik wrote: > > and does not apply anymore on current master > > ACK (CI pending). > > I did some performance tests with systemtap and the performance hit was > there, but very small (the user lookup went from 2-3ms to 4-6ms). > > nonetheless, Simo had a good idea recently on IRC which I captured in > this ticket: > https://fedorahosted.org/sssd/ticket/3126 > but that's larger effort than what we can accomplish in 1.14.1. CI passed: http://sssd-ci.duckdns.org/logs/job/51/28/summary.html * master: 00f3c5cd03625357e226552084e499965512bf53 ___ sssd-devel mailing list sssd-devel@lists.fedorahosted.org https://lists.fedorahosted.org/admin/lists/sssd-devel@lists.fedorahosted.org
[SSSD] Re: [PATCH] SYSDB: Check changed virtual attributes before modified timestamp
On Fri, Aug 05, 2016 at 01:28:36PM +0200, Lukas Slebodnik wrote: > and does not apply anymore on current master ACK (CI pending). I did some performance tests with systemtap and the performance hit was there, but very small (the user lookup went from 2-3ms to 4-6ms). nonetheless, Simo had a good idea recently on IRC which I captured in this ticket: https://fedorahosted.org/sssd/ticket/3126 but that's larger effort than what we can accomplish in 1.14.1. ___ sssd-devel mailing list sssd-devel@lists.fedorahosted.org https://lists.fedorahosted.org/admin/lists/sssd-devel@lists.fedorahosted.org
[SSSD] Re: [PATCH] SYSDB: Check changed virtual attributes before modified timestamp
On (04/08/16 12:59), Lukas Slebodnik wrote: >On (04/08/16 11:38), Jakub Hrozek wrote: >>On Thu, Aug 04, 2016 at 11:26:41AM +0200, Jakub Hrozek wrote: >>> On Thu, Aug 04, 2016 at 11:19:36AM +0200, Lukas Slebodnik wrote: >>> > On (04/08/16 11:15), Jakub Hrozek wrote: >>> > >On Thu, Aug 04, 2016 at 08:45:00AM +0200, Lukas Slebodnik wrote: >>> > >> On (03/08/16 18:56), Lukas Slebodnik wrote: >>> > >> >On (29/07/16 16:41), Jakub Hrozek wrote: >>> > >> >>On Thu, Jul 28, 2016 at 01:56:50PM +0200, Jakub Hrozek wrote: >>> > >> >>> On Thu, Jul 28, 2016 at 01:33:32PM +0200, Lukas Slebodnik wrote: >>> > >> >>> > On (28/07/16 12:06), thierry bordaz wrote: >>> > >> >>> > >On 07/28/2016 09:39 AM, Jakub Hrozek wrote: >>> > >> >>> > >> On Wed, Jul 27, 2016 at 04:09:07PM +0200, thierry bordaz >>> > >> >>> > >> wrote: >>> > >> >>> > >> > >>> > >> >>> > >> > On 07/27/2016 03:36 PM, Jakub Hrozek wrote: >>> > >> >>> > >> > > On Wed, Jul 27, 2016 at 02:55:37PM +0200, thierry bordaz >>> > >> >>> > >> > > wrote: >>> > >> >>> > >> > > > On 07/27/2016 01:56 PM, Jakub Hrozek wrote: >>> > >> >>> > >> > > > > On Wed, Jul 27, 2016 at 01:03:59PM +0200, Jakub >>> > >> >>> > >> > > > > Hrozek wrote: >>> > >> >>> > >> > > > > > On Wed, Jul 27, 2016 at 12:22:46PM +0200, Lukas >>> > >> >>> > >> > > > > > Slebodnik wrote: >>> > >> >>> > >> > > > > > > On (27/07/16 12:08), Jakub Hrozek wrote: >>> > >> >>> > >> > > > > > > > On Wed, Jul 27, 2016 at 12:02:24PM +0200, Jakub >>> > >> >>> > >> > > > > > > > Hrozek wrote: >>> > >> >>> > >> > > > > > > > > On Wed, Jul 27, 2016 at 11:54:16AM +0200, >>> > >> >>> > >> > > > > > > > > Lukas Slebodnik wrote: >>> > >> >>> > >> > > > > > > > > > ehlo, >>> > >> >>> > >> > > > > > > > > > >>> > >> >>> > >> > > > > > > > > > attached patch fixes acces denied after >>> > >> >>> > >> > > > > > > > > > activating user in 389ds. >>> > >> >>> > >> > > > > > > > > > Jakub had some comments/ideas in ticket but >>> > >> >>> > >> > > > > > > > > > I think it's better to discuss >>> > >> >>> > >> > > > > > > > > > about virtual attributes and timestamp >>> > >> >>> > >> > > > > > > > > > cache on mailing list. >>> > >> >>> > >> > > > > > > > > Yes, so the comment I have is that while this >>> > >> >>> > >> > > > > > > > > works, it might break some >>> > >> >>> > >> > > > > > > > > strange LDAP servers. >>> > >> >>> > >> > > > > > > > > >>> > >> >>> > >> > > > > > > > > We use modifyTimestamp as a 'positive' >>> > >> >>> > >> > > > > > > > > indicator that the entry has not >>> > >> >>> > >> > > > > > > > > changed -- if the modifyTimestamp didn't >>> > >> >>> > >> > > > > > > > > change, we consider the cached >>> > >> >>> > >> > > > > > > > > entry the same as what is on the server and >>> > >> >>> > >> > > > > > > > > only bump the timestamp >>> > >> >>> > >> > > > > > > > > cache. If the timestamp is different, we do a >>> > >> >>> > >> > > > > > > > > deep-comparison of cached >>> > >> >>> > >> > > > > > > > > attribute values with what is on the LDAP >>> > >> >>> > >> > > > > > > > > server and write the sysdb >>> > >> >>> > >> > > > > > > > > cache entry only if the attributes differ. >>> > >> >>> > >> > > > > > > > > >>> > >> >>> > >> > > > > > > > > I was wondering if we can use the >>> > >> >>> > >> > > > > > > > > modifyTimestamp at all, then, because >>> > >> >>> > >> > > > > > > > > even if it's the same, we might want to check >>> > >> >>> > >> > > > > > > > > the attributes to see if >>> > >> >>> > >> > > > > > > > > some of the values are different because some >>> > >> >>> > >> > > > > > > > > of the attributes might be >>> > >> >>> > >> > > > > > > > > this operational/virtual attribute.. >>> > >> >>> > >> > > > > > > > Sorry, sent too soon. >>> > >> >>> > >> > > > > > > > >>> > >> >>> > >> > > > > > > > I think the questions are -- 1) can we >>> > >> >>> > >> > > > > > > > enumerate the virtual attributes? >>> > >> >>> > >> > > > > > > That might be a question for 389-ds developers. >>> > >> >>> > >> > > > > > > But it's very likely it will be different on >>> > >> >>> > >> > > > > > > other LDAP servers. >>> > >> >>> > >> > > > > > > >>> > >> >>> > >> > > > > > > > 2) Would different LDAP servers have different >>> > >> >>> > >> > > > > > > > virtual attributes. >>> > >> >>> > >> > > > > > > > >>> > >> >>> > >> > > > > > > > For 2) maybe a possible solution might be to >>> > >> >>> > >> > > > > > > > set a non-existing >>> > >> >>> > >> > > > > > > > modifyTimestamp attribute value, but I would >>> > >> >>> > >> > > > > > > > consider that only a >>> > >> >>> > >> > > > > > > > kludge, we shouldn't break existing setups.. >>> > >> >>> > >> > > > > > > I am not satisfied with this POC solution either. >>> > >> >>> > >> > > > > > > >>> > >> >>> > >> > > > > > > So should we remove usage of modifyTimestamp for >>> > >> >>> > >> > > > > > > detecting changes? >>> > >> >>> > >> > > > > > I would prefer to ask the DS developers before >>> > >> >>> > >> > > > > > removing it
[SSSD] Re: [PATCH] SYSDB: Check changed virtual attributes before modified timestamp
On (04/08/16 11:38), Jakub Hrozek wrote: >On Thu, Aug 04, 2016 at 11:26:41AM +0200, Jakub Hrozek wrote: >> On Thu, Aug 04, 2016 at 11:19:36AM +0200, Lukas Slebodnik wrote: >> > On (04/08/16 11:15), Jakub Hrozek wrote: >> > >On Thu, Aug 04, 2016 at 08:45:00AM +0200, Lukas Slebodnik wrote: >> > >> On (03/08/16 18:56), Lukas Slebodnik wrote: >> > >> >On (29/07/16 16:41), Jakub Hrozek wrote: >> > >> >>On Thu, Jul 28, 2016 at 01:56:50PM +0200, Jakub Hrozek wrote: >> > >> >>> On Thu, Jul 28, 2016 at 01:33:32PM +0200, Lukas Slebodnik wrote: >> > >> >>> > On (28/07/16 12:06), thierry bordaz wrote: >> > >> >>> > >On 07/28/2016 09:39 AM, Jakub Hrozek wrote: >> > >> >>> > >> On Wed, Jul 27, 2016 at 04:09:07PM +0200, thierry bordaz wrote: >> > >> >>> > >> > >> > >> >>> > >> > On 07/27/2016 03:36 PM, Jakub Hrozek wrote: >> > >> >>> > >> > > On Wed, Jul 27, 2016 at 02:55:37PM +0200, thierry bordaz >> > >> >>> > >> > > wrote: >> > >> >>> > >> > > > On 07/27/2016 01:56 PM, Jakub Hrozek wrote: >> > >> >>> > >> > > > > On Wed, Jul 27, 2016 at 01:03:59PM +0200, Jakub Hrozek >> > >> >>> > >> > > > > wrote: >> > >> >>> > >> > > > > > On Wed, Jul 27, 2016 at 12:22:46PM +0200, Lukas >> > >> >>> > >> > > > > > Slebodnik wrote: >> > >> >>> > >> > > > > > > On (27/07/16 12:08), Jakub Hrozek wrote: >> > >> >>> > >> > > > > > > > On Wed, Jul 27, 2016 at 12:02:24PM +0200, Jakub >> > >> >>> > >> > > > > > > > Hrozek wrote: >> > >> >>> > >> > > > > > > > > On Wed, Jul 27, 2016 at 11:54:16AM +0200, >> > >> >>> > >> > > > > > > > > Lukas Slebodnik wrote: >> > >> >>> > >> > > > > > > > > > ehlo, >> > >> >>> > >> > > > > > > > > > >> > >> >>> > >> > > > > > > > > > attached patch fixes acces denied after >> > >> >>> > >> > > > > > > > > > activating user in 389ds. >> > >> >>> > >> > > > > > > > > > Jakub had some comments/ideas in ticket but >> > >> >>> > >> > > > > > > > > > I think it's better to discuss >> > >> >>> > >> > > > > > > > > > about virtual attributes and timestamp cache >> > >> >>> > >> > > > > > > > > > on mailing list. >> > >> >>> > >> > > > > > > > > Yes, so the comment I have is that while this >> > >> >>> > >> > > > > > > > > works, it might break some >> > >> >>> > >> > > > > > > > > strange LDAP servers. >> > >> >>> > >> > > > > > > > > >> > >> >>> > >> > > > > > > > > We use modifyTimestamp as a 'positive' >> > >> >>> > >> > > > > > > > > indicator that the entry has not >> > >> >>> > >> > > > > > > > > changed -- if the modifyTimestamp didn't >> > >> >>> > >> > > > > > > > > change, we consider the cached >> > >> >>> > >> > > > > > > > > entry the same as what is on the server and >> > >> >>> > >> > > > > > > > > only bump the timestamp >> > >> >>> > >> > > > > > > > > cache. If the timestamp is different, we do a >> > >> >>> > >> > > > > > > > > deep-comparison of cached >> > >> >>> > >> > > > > > > > > attribute values with what is on the LDAP >> > >> >>> > >> > > > > > > > > server and write the sysdb >> > >> >>> > >> > > > > > > > > cache entry only if the attributes differ. >> > >> >>> > >> > > > > > > > > >> > >> >>> > >> > > > > > > > > I was wondering if we can use the >> > >> >>> > >> > > > > > > > > modifyTimestamp at all, then, because >> > >> >>> > >> > > > > > > > > even if it's the same, we might want to check >> > >> >>> > >> > > > > > > > > the attributes to see if >> > >> >>> > >> > > > > > > > > some of the values are different because some >> > >> >>> > >> > > > > > > > > of the attributes might be >> > >> >>> > >> > > > > > > > > this operational/virtual attribute.. >> > >> >>> > >> > > > > > > > Sorry, sent too soon. >> > >> >>> > >> > > > > > > > >> > >> >>> > >> > > > > > > > I think the questions are -- 1) can we enumerate >> > >> >>> > >> > > > > > > > the virtual attributes? >> > >> >>> > >> > > > > > > That might be a question for 389-ds developers. >> > >> >>> > >> > > > > > > But it's very likely it will be different on other >> > >> >>> > >> > > > > > > LDAP servers. >> > >> >>> > >> > > > > > > >> > >> >>> > >> > > > > > > > 2) Would different LDAP servers have different >> > >> >>> > >> > > > > > > > virtual attributes. >> > >> >>> > >> > > > > > > > >> > >> >>> > >> > > > > > > > For 2) maybe a possible solution might be to set >> > >> >>> > >> > > > > > > > a non-existing >> > >> >>> > >> > > > > > > > modifyTimestamp attribute value, but I would >> > >> >>> > >> > > > > > > > consider that only a >> > >> >>> > >> > > > > > > > kludge, we shouldn't break existing setups.. >> > >> >>> > >> > > > > > > I am not satisfied with this POC solution either. >> > >> >>> > >> > > > > > > >> > >> >>> > >> > > > > > > So should we remove usage of modifyTimestamp for >> > >> >>> > >> > > > > > > detecting changes? >> > >> >>> > >> > > > > > I would prefer to ask the DS developers before >> > >> >>> > >> > > > > > removing it completely. >> > >> >>> > >> > > > > > >> > >> >>> > >> > > > > > At least for large groups it might take a long time >> > >> >>> > >> > > >
[SSSD] Re: [PATCH] SYSDB: Check changed virtual attributes before modified timestamp
On Thu, Aug 04, 2016 at 11:26:41AM +0200, Jakub Hrozek wrote: > On Thu, Aug 04, 2016 at 11:19:36AM +0200, Lukas Slebodnik wrote: > > On (04/08/16 11:15), Jakub Hrozek wrote: > > >On Thu, Aug 04, 2016 at 08:45:00AM +0200, Lukas Slebodnik wrote: > > >> On (03/08/16 18:56), Lukas Slebodnik wrote: > > >> >On (29/07/16 16:41), Jakub Hrozek wrote: > > >> >>On Thu, Jul 28, 2016 at 01:56:50PM +0200, Jakub Hrozek wrote: > > >> >>> On Thu, Jul 28, 2016 at 01:33:32PM +0200, Lukas Slebodnik wrote: > > >> >>> > On (28/07/16 12:06), thierry bordaz wrote: > > >> >>> > >On 07/28/2016 09:39 AM, Jakub Hrozek wrote: > > >> >>> > >> On Wed, Jul 27, 2016 at 04:09:07PM +0200, thierry bordaz wrote: > > >> >>> > >> > > > >> >>> > >> > On 07/27/2016 03:36 PM, Jakub Hrozek wrote: > > >> >>> > >> > > On Wed, Jul 27, 2016 at 02:55:37PM +0200, thierry bordaz > > >> >>> > >> > > wrote: > > >> >>> > >> > > > On 07/27/2016 01:56 PM, Jakub Hrozek wrote: > > >> >>> > >> > > > > On Wed, Jul 27, 2016 at 01:03:59PM +0200, Jakub Hrozek > > >> >>> > >> > > > > wrote: > > >> >>> > >> > > > > > On Wed, Jul 27, 2016 at 12:22:46PM +0200, Lukas > > >> >>> > >> > > > > > Slebodnik wrote: > > >> >>> > >> > > > > > > On (27/07/16 12:08), Jakub Hrozek wrote: > > >> >>> > >> > > > > > > > On Wed, Jul 27, 2016 at 12:02:24PM +0200, Jakub > > >> >>> > >> > > > > > > > Hrozek wrote: > > >> >>> > >> > > > > > > > > On Wed, Jul 27, 2016 at 11:54:16AM +0200, Lukas > > >> >>> > >> > > > > > > > > Slebodnik wrote: > > >> >>> > >> > > > > > > > > > ehlo, > > >> >>> > >> > > > > > > > > > > > >> >>> > >> > > > > > > > > > attached patch fixes acces denied after > > >> >>> > >> > > > > > > > > > activating user in 389ds. > > >> >>> > >> > > > > > > > > > Jakub had some comments/ideas in ticket but I > > >> >>> > >> > > > > > > > > > think it's better to discuss > > >> >>> > >> > > > > > > > > > about virtual attributes and timestamp cache > > >> >>> > >> > > > > > > > > > on mailing list. > > >> >>> > >> > > > > > > > > Yes, so the comment I have is that while this > > >> >>> > >> > > > > > > > > works, it might break some > > >> >>> > >> > > > > > > > > strange LDAP servers. > > >> >>> > >> > > > > > > > > > > >> >>> > >> > > > > > > > > We use modifyTimestamp as a 'positive' > > >> >>> > >> > > > > > > > > indicator that the entry has not > > >> >>> > >> > > > > > > > > changed -- if the modifyTimestamp didn't > > >> >>> > >> > > > > > > > > change, we consider the cached > > >> >>> > >> > > > > > > > > entry the same as what is on the server and > > >> >>> > >> > > > > > > > > only bump the timestamp > > >> >>> > >> > > > > > > > > cache. If the timestamp is different, we do a > > >> >>> > >> > > > > > > > > deep-comparison of cached > > >> >>> > >> > > > > > > > > attribute values with what is on the LDAP > > >> >>> > >> > > > > > > > > server and write the sysdb > > >> >>> > >> > > > > > > > > cache entry only if the attributes differ. > > >> >>> > >> > > > > > > > > > > >> >>> > >> > > > > > > > > I was wondering if we can use the > > >> >>> > >> > > > > > > > > modifyTimestamp at all, then, because > > >> >>> > >> > > > > > > > > even if it's the same, we might want to check > > >> >>> > >> > > > > > > > > the attributes to see if > > >> >>> > >> > > > > > > > > some of the values are different because some > > >> >>> > >> > > > > > > > > of the attributes might be > > >> >>> > >> > > > > > > > > this operational/virtual attribute.. > > >> >>> > >> > > > > > > > Sorry, sent too soon. > > >> >>> > >> > > > > > > > > > >> >>> > >> > > > > > > > I think the questions are -- 1) can we enumerate > > >> >>> > >> > > > > > > > the virtual attributes? > > >> >>> > >> > > > > > > That might be a question for 389-ds developers. > > >> >>> > >> > > > > > > But it's very likely it will be different on other > > >> >>> > >> > > > > > > LDAP servers. > > >> >>> > >> > > > > > > > > >> >>> > >> > > > > > > > 2) Would different LDAP servers have different > > >> >>> > >> > > > > > > > virtual attributes. > > >> >>> > >> > > > > > > > > > >> >>> > >> > > > > > > > For 2) maybe a possible solution might be to set > > >> >>> > >> > > > > > > > a non-existing > > >> >>> > >> > > > > > > > modifyTimestamp attribute value, but I would > > >> >>> > >> > > > > > > > consider that only a > > >> >>> > >> > > > > > > > kludge, we shouldn't break existing setups.. > > >> >>> > >> > > > > > > I am not satisfied with this POC solution either. > > >> >>> > >> > > > > > > > > >> >>> > >> > > > > > > So should we remove usage of modifyTimestamp for > > >> >>> > >> > > > > > > detecting changes? > > >> >>> > >> > > > > > I would prefer to ask the DS developers before > > >> >>> > >> > > > > > removing it completely. > > >> >>> > >> > > > > > > > >> >>> > >> > > > > > At least for large groups it might take a long time > > >> >>> > >> > > > > > to compare all attribute > > >> >>> > >> > > > > > values and IIRC we don't depend on any virtual > > >> >>> > >> >
[SSSD] Re: [PATCH] SYSDB: Check changed virtual attributes before modified timestamp
On Thu, Aug 04, 2016 at 11:19:36AM +0200, Lukas Slebodnik wrote: > On (04/08/16 11:15), Jakub Hrozek wrote: > >On Thu, Aug 04, 2016 at 08:45:00AM +0200, Lukas Slebodnik wrote: > >> On (03/08/16 18:56), Lukas Slebodnik wrote: > >> >On (29/07/16 16:41), Jakub Hrozek wrote: > >> >>On Thu, Jul 28, 2016 at 01:56:50PM +0200, Jakub Hrozek wrote: > >> >>> On Thu, Jul 28, 2016 at 01:33:32PM +0200, Lukas Slebodnik wrote: > >> >>> > On (28/07/16 12:06), thierry bordaz wrote: > >> >>> > >On 07/28/2016 09:39 AM, Jakub Hrozek wrote: > >> >>> > >> On Wed, Jul 27, 2016 at 04:09:07PM +0200, thierry bordaz wrote: > >> >>> > >> > > >> >>> > >> > On 07/27/2016 03:36 PM, Jakub Hrozek wrote: > >> >>> > >> > > On Wed, Jul 27, 2016 at 02:55:37PM +0200, thierry bordaz > >> >>> > >> > > wrote: > >> >>> > >> > > > On 07/27/2016 01:56 PM, Jakub Hrozek wrote: > >> >>> > >> > > > > On Wed, Jul 27, 2016 at 01:03:59PM +0200, Jakub Hrozek > >> >>> > >> > > > > wrote: > >> >>> > >> > > > > > On Wed, Jul 27, 2016 at 12:22:46PM +0200, Lukas > >> >>> > >> > > > > > Slebodnik wrote: > >> >>> > >> > > > > > > On (27/07/16 12:08), Jakub Hrozek wrote: > >> >>> > >> > > > > > > > On Wed, Jul 27, 2016 at 12:02:24PM +0200, Jakub > >> >>> > >> > > > > > > > Hrozek wrote: > >> >>> > >> > > > > > > > > On Wed, Jul 27, 2016 at 11:54:16AM +0200, Lukas > >> >>> > >> > > > > > > > > Slebodnik wrote: > >> >>> > >> > > > > > > > > > ehlo, > >> >>> > >> > > > > > > > > > > >> >>> > >> > > > > > > > > > attached patch fixes acces denied after > >> >>> > >> > > > > > > > > > activating user in 389ds. > >> >>> > >> > > > > > > > > > Jakub had some comments/ideas in ticket but I > >> >>> > >> > > > > > > > > > think it's better to discuss > >> >>> > >> > > > > > > > > > about virtual attributes and timestamp cache on > >> >>> > >> > > > > > > > > > mailing list. > >> >>> > >> > > > > > > > > Yes, so the comment I have is that while this > >> >>> > >> > > > > > > > > works, it might break some > >> >>> > >> > > > > > > > > strange LDAP servers. > >> >>> > >> > > > > > > > > > >> >>> > >> > > > > > > > > We use modifyTimestamp as a 'positive' indicator > >> >>> > >> > > > > > > > > that the entry has not > >> >>> > >> > > > > > > > > changed -- if the modifyTimestamp didn't change, > >> >>> > >> > > > > > > > > we consider the cached > >> >>> > >> > > > > > > > > entry the same as what is on the server and only > >> >>> > >> > > > > > > > > bump the timestamp > >> >>> > >> > > > > > > > > cache. If the timestamp is different, we do a > >> >>> > >> > > > > > > > > deep-comparison of cached > >> >>> > >> > > > > > > > > attribute values with what is on the LDAP server > >> >>> > >> > > > > > > > > and write the sysdb > >> >>> > >> > > > > > > > > cache entry only if the attributes differ. > >> >>> > >> > > > > > > > > > >> >>> > >> > > > > > > > > I was wondering if we can use the modifyTimestamp > >> >>> > >> > > > > > > > > at all, then, because > >> >>> > >> > > > > > > > > even if it's the same, we might want to check the > >> >>> > >> > > > > > > > > attributes to see if > >> >>> > >> > > > > > > > > some of the values are different because some of > >> >>> > >> > > > > > > > > the attributes might be > >> >>> > >> > > > > > > > > this operational/virtual attribute.. > >> >>> > >> > > > > > > > Sorry, sent too soon. > >> >>> > >> > > > > > > > > >> >>> > >> > > > > > > > I think the questions are -- 1) can we enumerate > >> >>> > >> > > > > > > > the virtual attributes? > >> >>> > >> > > > > > > That might be a question for 389-ds developers. > >> >>> > >> > > > > > > But it's very likely it will be different on other > >> >>> > >> > > > > > > LDAP servers. > >> >>> > >> > > > > > > > >> >>> > >> > > > > > > > 2) Would different LDAP servers have different > >> >>> > >> > > > > > > > virtual attributes. > >> >>> > >> > > > > > > > > >> >>> > >> > > > > > > > For 2) maybe a possible solution might be to set a > >> >>> > >> > > > > > > > non-existing > >> >>> > >> > > > > > > > modifyTimestamp attribute value, but I would > >> >>> > >> > > > > > > > consider that only a > >> >>> > >> > > > > > > > kludge, we shouldn't break existing setups.. > >> >>> > >> > > > > > > I am not satisfied with this POC solution either. > >> >>> > >> > > > > > > > >> >>> > >> > > > > > > So should we remove usage of modifyTimestamp for > >> >>> > >> > > > > > > detecting changes? > >> >>> > >> > > > > > I would prefer to ask the DS developers before removing > >> >>> > >> > > > > > it completely. > >> >>> > >> > > > > > > >> >>> > >> > > > > > At least for large groups it might take a long time to > >> >>> > >> > > > > > compare all attribute > >> >>> > >> > > > > > values and IIRC we don't depend on any virtual > >> >>> > >> > > > > > attributes for groups. Maybe > >> >>> > >> > > > > > we could parametrize that part of the code and enable > >> >>> > >> > > > > > the fast way with > >> >>> > >> > > > > > modifyTimestamps for 'known' server types, t
[SSSD] Re: [PATCH] SYSDB: Check changed virtual attributes before modified timestamp
On (04/08/16 11:15), Jakub Hrozek wrote: >On Thu, Aug 04, 2016 at 08:45:00AM +0200, Lukas Slebodnik wrote: >> On (03/08/16 18:56), Lukas Slebodnik wrote: >> >On (29/07/16 16:41), Jakub Hrozek wrote: >> >>On Thu, Jul 28, 2016 at 01:56:50PM +0200, Jakub Hrozek wrote: >> >>> On Thu, Jul 28, 2016 at 01:33:32PM +0200, Lukas Slebodnik wrote: >> >>> > On (28/07/16 12:06), thierry bordaz wrote: >> >>> > >On 07/28/2016 09:39 AM, Jakub Hrozek wrote: >> >>> > >> On Wed, Jul 27, 2016 at 04:09:07PM +0200, thierry bordaz wrote: >> >>> > >> > >> >>> > >> > On 07/27/2016 03:36 PM, Jakub Hrozek wrote: >> >>> > >> > > On Wed, Jul 27, 2016 at 02:55:37PM +0200, thierry bordaz wrote: >> >>> > >> > > > On 07/27/2016 01:56 PM, Jakub Hrozek wrote: >> >>> > >> > > > > On Wed, Jul 27, 2016 at 01:03:59PM +0200, Jakub Hrozek >> >>> > >> > > > > wrote: >> >>> > >> > > > > > On Wed, Jul 27, 2016 at 12:22:46PM +0200, Lukas Slebodnik >> >>> > >> > > > > > wrote: >> >>> > >> > > > > > > On (27/07/16 12:08), Jakub Hrozek wrote: >> >>> > >> > > > > > > > On Wed, Jul 27, 2016 at 12:02:24PM +0200, Jakub >> >>> > >> > > > > > > > Hrozek wrote: >> >>> > >> > > > > > > > > On Wed, Jul 27, 2016 at 11:54:16AM +0200, Lukas >> >>> > >> > > > > > > > > Slebodnik wrote: >> >>> > >> > > > > > > > > > ehlo, >> >>> > >> > > > > > > > > > >> >>> > >> > > > > > > > > > attached patch fixes acces denied after >> >>> > >> > > > > > > > > > activating user in 389ds. >> >>> > >> > > > > > > > > > Jakub had some comments/ideas in ticket but I >> >>> > >> > > > > > > > > > think it's better to discuss >> >>> > >> > > > > > > > > > about virtual attributes and timestamp cache on >> >>> > >> > > > > > > > > > mailing list. >> >>> > >> > > > > > > > > Yes, so the comment I have is that while this >> >>> > >> > > > > > > > > works, it might break some >> >>> > >> > > > > > > > > strange LDAP servers. >> >>> > >> > > > > > > > > >> >>> > >> > > > > > > > > We use modifyTimestamp as a 'positive' indicator >> >>> > >> > > > > > > > > that the entry has not >> >>> > >> > > > > > > > > changed -- if the modifyTimestamp didn't change, we >> >>> > >> > > > > > > > > consider the cached >> >>> > >> > > > > > > > > entry the same as what is on the server and only >> >>> > >> > > > > > > > > bump the timestamp >> >>> > >> > > > > > > > > cache. If the timestamp is different, we do a >> >>> > >> > > > > > > > > deep-comparison of cached >> >>> > >> > > > > > > > > attribute values with what is on the LDAP server >> >>> > >> > > > > > > > > and write the sysdb >> >>> > >> > > > > > > > > cache entry only if the attributes differ. >> >>> > >> > > > > > > > > >> >>> > >> > > > > > > > > I was wondering if we can use the modifyTimestamp >> >>> > >> > > > > > > > > at all, then, because >> >>> > >> > > > > > > > > even if it's the same, we might want to check the >> >>> > >> > > > > > > > > attributes to see if >> >>> > >> > > > > > > > > some of the values are different because some of >> >>> > >> > > > > > > > > the attributes might be >> >>> > >> > > > > > > > > this operational/virtual attribute.. >> >>> > >> > > > > > > > Sorry, sent too soon. >> >>> > >> > > > > > > > >> >>> > >> > > > > > > > I think the questions are -- 1) can we enumerate the >> >>> > >> > > > > > > > virtual attributes? >> >>> > >> > > > > > > That might be a question for 389-ds developers. >> >>> > >> > > > > > > But it's very likely it will be different on other LDAP >> >>> > >> > > > > > > servers. >> >>> > >> > > > > > > >> >>> > >> > > > > > > > 2) Would different LDAP servers have different >> >>> > >> > > > > > > > virtual attributes. >> >>> > >> > > > > > > > >> >>> > >> > > > > > > > For 2) maybe a possible solution might be to set a >> >>> > >> > > > > > > > non-existing >> >>> > >> > > > > > > > modifyTimestamp attribute value, but I would consider >> >>> > >> > > > > > > > that only a >> >>> > >> > > > > > > > kludge, we shouldn't break existing setups.. >> >>> > >> > > > > > > I am not satisfied with this POC solution either. >> >>> > >> > > > > > > >> >>> > >> > > > > > > So should we remove usage of modifyTimestamp for >> >>> > >> > > > > > > detecting changes? >> >>> > >> > > > > > I would prefer to ask the DS developers before removing >> >>> > >> > > > > > it completely. >> >>> > >> > > > > > >> >>> > >> > > > > > At least for large groups it might take a long time to >> >>> > >> > > > > > compare all attribute >> >>> > >> > > > > > values and IIRC we don't depend on any virtual attributes >> >>> > >> > > > > > for groups. Maybe >> >>> > >> > > > > > we could parametrize that part of the code and enable the >> >>> > >> > > > > > fast way with >> >>> > >> > > > > > modifyTimestamps for 'known' server types, that is for >> >>> > >> > > > > > setups with AD and >> >>> > >> > > > > > IPA providers. >> >>> > >> > > > > > >> >>> > >> > > > > > For users, there is typically not as many attributes so >> >>> > >> > > > > > we should be >> >>> > >> > > > > > f
[SSSD] Re: [PATCH] SYSDB: Check changed virtual attributes before modified timestamp
On Thu, Aug 04, 2016 at 08:45:00AM +0200, Lukas Slebodnik wrote: > On (03/08/16 18:56), Lukas Slebodnik wrote: > >On (29/07/16 16:41), Jakub Hrozek wrote: > >>On Thu, Jul 28, 2016 at 01:56:50PM +0200, Jakub Hrozek wrote: > >>> On Thu, Jul 28, 2016 at 01:33:32PM +0200, Lukas Slebodnik wrote: > >>> > On (28/07/16 12:06), thierry bordaz wrote: > >>> > >On 07/28/2016 09:39 AM, Jakub Hrozek wrote: > >>> > >> On Wed, Jul 27, 2016 at 04:09:07PM +0200, thierry bordaz wrote: > >>> > >> > > >>> > >> > On 07/27/2016 03:36 PM, Jakub Hrozek wrote: > >>> > >> > > On Wed, Jul 27, 2016 at 02:55:37PM +0200, thierry bordaz wrote: > >>> > >> > > > On 07/27/2016 01:56 PM, Jakub Hrozek wrote: > >>> > >> > > > > On Wed, Jul 27, 2016 at 01:03:59PM +0200, Jakub Hrozek wrote: > >>> > >> > > > > > On Wed, Jul 27, 2016 at 12:22:46PM +0200, Lukas Slebodnik > >>> > >> > > > > > wrote: > >>> > >> > > > > > > On (27/07/16 12:08), Jakub Hrozek wrote: > >>> > >> > > > > > > > On Wed, Jul 27, 2016 at 12:02:24PM +0200, Jakub Hrozek > >>> > >> > > > > > > > wrote: > >>> > >> > > > > > > > > On Wed, Jul 27, 2016 at 11:54:16AM +0200, Lukas > >>> > >> > > > > > > > > Slebodnik wrote: > >>> > >> > > > > > > > > > ehlo, > >>> > >> > > > > > > > > > > >>> > >> > > > > > > > > > attached patch fixes acces denied after activating > >>> > >> > > > > > > > > > user in 389ds. > >>> > >> > > > > > > > > > Jakub had some comments/ideas in ticket but I > >>> > >> > > > > > > > > > think it's better to discuss > >>> > >> > > > > > > > > > about virtual attributes and timestamp cache on > >>> > >> > > > > > > > > > mailing list. > >>> > >> > > > > > > > > Yes, so the comment I have is that while this works, > >>> > >> > > > > > > > > it might break some > >>> > >> > > > > > > > > strange LDAP servers. > >>> > >> > > > > > > > > > >>> > >> > > > > > > > > We use modifyTimestamp as a 'positive' indicator > >>> > >> > > > > > > > > that the entry has not > >>> > >> > > > > > > > > changed -- if the modifyTimestamp didn't change, we > >>> > >> > > > > > > > > consider the cached > >>> > >> > > > > > > > > entry the same as what is on the server and only > >>> > >> > > > > > > > > bump the timestamp > >>> > >> > > > > > > > > cache. If the timestamp is different, we do a > >>> > >> > > > > > > > > deep-comparison of cached > >>> > >> > > > > > > > > attribute values with what is on the LDAP server and > >>> > >> > > > > > > > > write the sysdb > >>> > >> > > > > > > > > cache entry only if the attributes differ. > >>> > >> > > > > > > > > > >>> > >> > > > > > > > > I was wondering if we can use the modifyTimestamp at > >>> > >> > > > > > > > > all, then, because > >>> > >> > > > > > > > > even if it's the same, we might want to check the > >>> > >> > > > > > > > > attributes to see if > >>> > >> > > > > > > > > some of the values are different because some of the > >>> > >> > > > > > > > > attributes might be > >>> > >> > > > > > > > > this operational/virtual attribute.. > >>> > >> > > > > > > > Sorry, sent too soon. > >>> > >> > > > > > > > > >>> > >> > > > > > > > I think the questions are -- 1) can we enumerate the > >>> > >> > > > > > > > virtual attributes? > >>> > >> > > > > > > That might be a question for 389-ds developers. > >>> > >> > > > > > > But it's very likely it will be different on other LDAP > >>> > >> > > > > > > servers. > >>> > >> > > > > > > > >>> > >> > > > > > > > 2) Would different LDAP servers have different virtual > >>> > >> > > > > > > > attributes. > >>> > >> > > > > > > > > >>> > >> > > > > > > > For 2) maybe a possible solution might be to set a > >>> > >> > > > > > > > non-existing > >>> > >> > > > > > > > modifyTimestamp attribute value, but I would consider > >>> > >> > > > > > > > that only a > >>> > >> > > > > > > > kludge, we shouldn't break existing setups.. > >>> > >> > > > > > > I am not satisfied with this POC solution either. > >>> > >> > > > > > > > >>> > >> > > > > > > So should we remove usage of modifyTimestamp for > >>> > >> > > > > > > detecting changes? > >>> > >> > > > > > I would prefer to ask the DS developers before removing it > >>> > >> > > > > > completely. > >>> > >> > > > > > > >>> > >> > > > > > At least for large groups it might take a long time to > >>> > >> > > > > > compare all attribute > >>> > >> > > > > > values and IIRC we don't depend on any virtual attributes > >>> > >> > > > > > for groups. Maybe > >>> > >> > > > > > we could parametrize that part of the code and enable the > >>> > >> > > > > > fast way with > >>> > >> > > > > > modifyTimestamps for 'known' server types, that is for > >>> > >> > > > > > setups with AD and > >>> > >> > > > > > IPA providers. > >>> > >> > > > > > > >>> > >> > > > > > For users, there is typically not as many attributes so we > >>> > >> > > > > > should be > >>> > >> > > > > > fine deep-comparing all attributes. > >>> > >> > > > > I'm adding Thierry (so please reply-to-all to keep him in > >>> > >> > > > > the thread). > >>
[SSSD] Re: [PATCH] SYSDB: Check changed virtual attributes before modified timestamp
On (03/08/16 18:56), Lukas Slebodnik wrote: >On (29/07/16 16:41), Jakub Hrozek wrote: >>On Thu, Jul 28, 2016 at 01:56:50PM +0200, Jakub Hrozek wrote: >>> On Thu, Jul 28, 2016 at 01:33:32PM +0200, Lukas Slebodnik wrote: >>> > On (28/07/16 12:06), thierry bordaz wrote: >>> > >On 07/28/2016 09:39 AM, Jakub Hrozek wrote: >>> > >> On Wed, Jul 27, 2016 at 04:09:07PM +0200, thierry bordaz wrote: >>> > >> > >>> > >> > On 07/27/2016 03:36 PM, Jakub Hrozek wrote: >>> > >> > > On Wed, Jul 27, 2016 at 02:55:37PM +0200, thierry bordaz wrote: >>> > >> > > > On 07/27/2016 01:56 PM, Jakub Hrozek wrote: >>> > >> > > > > On Wed, Jul 27, 2016 at 01:03:59PM +0200, Jakub Hrozek wrote: >>> > >> > > > > > On Wed, Jul 27, 2016 at 12:22:46PM +0200, Lukas Slebodnik >>> > >> > > > > > wrote: >>> > >> > > > > > > On (27/07/16 12:08), Jakub Hrozek wrote: >>> > >> > > > > > > > On Wed, Jul 27, 2016 at 12:02:24PM +0200, Jakub Hrozek >>> > >> > > > > > > > wrote: >>> > >> > > > > > > > > On Wed, Jul 27, 2016 at 11:54:16AM +0200, Lukas >>> > >> > > > > > > > > Slebodnik wrote: >>> > >> > > > > > > > > > ehlo, >>> > >> > > > > > > > > > >>> > >> > > > > > > > > > attached patch fixes acces denied after activating >>> > >> > > > > > > > > > user in 389ds. >>> > >> > > > > > > > > > Jakub had some comments/ideas in ticket but I think >>> > >> > > > > > > > > > it's better to discuss >>> > >> > > > > > > > > > about virtual attributes and timestamp cache on >>> > >> > > > > > > > > > mailing list. >>> > >> > > > > > > > > Yes, so the comment I have is that while this works, >>> > >> > > > > > > > > it might break some >>> > >> > > > > > > > > strange LDAP servers. >>> > >> > > > > > > > > >>> > >> > > > > > > > > We use modifyTimestamp as a 'positive' indicator that >>> > >> > > > > > > > > the entry has not >>> > >> > > > > > > > > changed -- if the modifyTimestamp didn't change, we >>> > >> > > > > > > > > consider the cached >>> > >> > > > > > > > > entry the same as what is on the server and only bump >>> > >> > > > > > > > > the timestamp >>> > >> > > > > > > > > cache. If the timestamp is different, we do a >>> > >> > > > > > > > > deep-comparison of cached >>> > >> > > > > > > > > attribute values with what is on the LDAP server and >>> > >> > > > > > > > > write the sysdb >>> > >> > > > > > > > > cache entry only if the attributes differ. >>> > >> > > > > > > > > >>> > >> > > > > > > > > I was wondering if we can use the modifyTimestamp at >>> > >> > > > > > > > > all, then, because >>> > >> > > > > > > > > even if it's the same, we might want to check the >>> > >> > > > > > > > > attributes to see if >>> > >> > > > > > > > > some of the values are different because some of the >>> > >> > > > > > > > > attributes might be >>> > >> > > > > > > > > this operational/virtual attribute.. >>> > >> > > > > > > > Sorry, sent too soon. >>> > >> > > > > > > > >>> > >> > > > > > > > I think the questions are -- 1) can we enumerate the >>> > >> > > > > > > > virtual attributes? >>> > >> > > > > > > That might be a question for 389-ds developers. >>> > >> > > > > > > But it's very likely it will be different on other LDAP >>> > >> > > > > > > servers. >>> > >> > > > > > > >>> > >> > > > > > > > 2) Would different LDAP servers have different virtual >>> > >> > > > > > > > attributes. >>> > >> > > > > > > > >>> > >> > > > > > > > For 2) maybe a possible solution might be to set a >>> > >> > > > > > > > non-existing >>> > >> > > > > > > > modifyTimestamp attribute value, but I would consider >>> > >> > > > > > > > that only a >>> > >> > > > > > > > kludge, we shouldn't break existing setups.. >>> > >> > > > > > > I am not satisfied with this POC solution either. >>> > >> > > > > > > >>> > >> > > > > > > So should we remove usage of modifyTimestamp for detecting >>> > >> > > > > > > changes? >>> > >> > > > > > I would prefer to ask the DS developers before removing it >>> > >> > > > > > completely. >>> > >> > > > > > >>> > >> > > > > > At least for large groups it might take a long time to >>> > >> > > > > > compare all attribute >>> > >> > > > > > values and IIRC we don't depend on any virtual attributes >>> > >> > > > > > for groups. Maybe >>> > >> > > > > > we could parametrize that part of the code and enable the >>> > >> > > > > > fast way with >>> > >> > > > > > modifyTimestamps for 'known' server types, that is for >>> > >> > > > > > setups with AD and >>> > >> > > > > > IPA providers. >>> > >> > > > > > >>> > >> > > > > > For users, there is typically not as many attributes so we >>> > >> > > > > > should be >>> > >> > > > > > fine deep-comparing all attributes. >>> > >> > > > > I'm adding Thierry (so please reply-to-all to keep him in the >>> > >> > > > > thread). >>> > >> > > > > >>> > >> > > > > Thierry, in the latest sssd version we tried to add a >>> > >> > > > > performance >>> > >> > > > > improvement related to how we store SSSD entries in the cache. >>> > >> > > > > The short >>> > >> > > > > v
[SSSD] Re: [PATCH] SYSDB: Check changed virtual attributes before modified timestamp
On (29/07/16 16:41), Jakub Hrozek wrote: >On Thu, Jul 28, 2016 at 01:56:50PM +0200, Jakub Hrozek wrote: >> On Thu, Jul 28, 2016 at 01:33:32PM +0200, Lukas Slebodnik wrote: >> > On (28/07/16 12:06), thierry bordaz wrote: >> > >On 07/28/2016 09:39 AM, Jakub Hrozek wrote: >> > >> On Wed, Jul 27, 2016 at 04:09:07PM +0200, thierry bordaz wrote: >> > >> > >> > >> > On 07/27/2016 03:36 PM, Jakub Hrozek wrote: >> > >> > > On Wed, Jul 27, 2016 at 02:55:37PM +0200, thierry bordaz wrote: >> > >> > > > On 07/27/2016 01:56 PM, Jakub Hrozek wrote: >> > >> > > > > On Wed, Jul 27, 2016 at 01:03:59PM +0200, Jakub Hrozek wrote: >> > >> > > > > > On Wed, Jul 27, 2016 at 12:22:46PM +0200, Lukas Slebodnik >> > >> > > > > > wrote: >> > >> > > > > > > On (27/07/16 12:08), Jakub Hrozek wrote: >> > >> > > > > > > > On Wed, Jul 27, 2016 at 12:02:24PM +0200, Jakub Hrozek >> > >> > > > > > > > wrote: >> > >> > > > > > > > > On Wed, Jul 27, 2016 at 11:54:16AM +0200, Lukas >> > >> > > > > > > > > Slebodnik wrote: >> > >> > > > > > > > > > ehlo, >> > >> > > > > > > > > > >> > >> > > > > > > > > > attached patch fixes acces denied after activating >> > >> > > > > > > > > > user in 389ds. >> > >> > > > > > > > > > Jakub had some comments/ideas in ticket but I think >> > >> > > > > > > > > > it's better to discuss >> > >> > > > > > > > > > about virtual attributes and timestamp cache on >> > >> > > > > > > > > > mailing list. >> > >> > > > > > > > > Yes, so the comment I have is that while this works, it >> > >> > > > > > > > > might break some >> > >> > > > > > > > > strange LDAP servers. >> > >> > > > > > > > > >> > >> > > > > > > > > We use modifyTimestamp as a 'positive' indicator that >> > >> > > > > > > > > the entry has not >> > >> > > > > > > > > changed -- if the modifyTimestamp didn't change, we >> > >> > > > > > > > > consider the cached >> > >> > > > > > > > > entry the same as what is on the server and only bump >> > >> > > > > > > > > the timestamp >> > >> > > > > > > > > cache. If the timestamp is different, we do a >> > >> > > > > > > > > deep-comparison of cached >> > >> > > > > > > > > attribute values with what is on the LDAP server and >> > >> > > > > > > > > write the sysdb >> > >> > > > > > > > > cache entry only if the attributes differ. >> > >> > > > > > > > > >> > >> > > > > > > > > I was wondering if we can use the modifyTimestamp at >> > >> > > > > > > > > all, then, because >> > >> > > > > > > > > even if it's the same, we might want to check the >> > >> > > > > > > > > attributes to see if >> > >> > > > > > > > > some of the values are different because some of the >> > >> > > > > > > > > attributes might be >> > >> > > > > > > > > this operational/virtual attribute.. >> > >> > > > > > > > Sorry, sent too soon. >> > >> > > > > > > > >> > >> > > > > > > > I think the questions are -- 1) can we enumerate the >> > >> > > > > > > > virtual attributes? >> > >> > > > > > > That might be a question for 389-ds developers. >> > >> > > > > > > But it's very likely it will be different on other LDAP >> > >> > > > > > > servers. >> > >> > > > > > > >> > >> > > > > > > > 2) Would different LDAP servers have different virtual >> > >> > > > > > > > attributes. >> > >> > > > > > > > >> > >> > > > > > > > For 2) maybe a possible solution might be to set a >> > >> > > > > > > > non-existing >> > >> > > > > > > > modifyTimestamp attribute value, but I would consider >> > >> > > > > > > > that only a >> > >> > > > > > > > kludge, we shouldn't break existing setups.. >> > >> > > > > > > I am not satisfied with this POC solution either. >> > >> > > > > > > >> > >> > > > > > > So should we remove usage of modifyTimestamp for detecting >> > >> > > > > > > changes? >> > >> > > > > > I would prefer to ask the DS developers before removing it >> > >> > > > > > completely. >> > >> > > > > > >> > >> > > > > > At least for large groups it might take a long time to >> > >> > > > > > compare all attribute >> > >> > > > > > values and IIRC we don't depend on any virtual attributes for >> > >> > > > > > groups. Maybe >> > >> > > > > > we could parametrize that part of the code and enable the >> > >> > > > > > fast way with >> > >> > > > > > modifyTimestamps for 'known' server types, that is for setups >> > >> > > > > > with AD and >> > >> > > > > > IPA providers. >> > >> > > > > > >> > >> > > > > > For users, there is typically not as many attributes so we >> > >> > > > > > should be >> > >> > > > > > fine deep-comparing all attributes. >> > >> > > > > I'm adding Thierry (so please reply-to-all to keep him in the >> > >> > > > > thread). >> > >> > > > > >> > >> > > > > Thierry, in the latest sssd version we tried to add a >> > >> > > > > performance >> > >> > > > > improvement related to how we store SSSD entries in the cache. >> > >> > > > > The short >> > >> > > > > version is that we store the modifyTimestamp attribute in the >> > >> > > > > cache and >> > >> > > > > when we fetch an entry, we compar
[SSSD] Re: [PATCH] SYSDB: Check changed virtual attributes before modified timestamp
On Thu, Jul 28, 2016 at 01:56:50PM +0200, Jakub Hrozek wrote: > On Thu, Jul 28, 2016 at 01:33:32PM +0200, Lukas Slebodnik wrote: > > On (28/07/16 12:06), thierry bordaz wrote: > > >On 07/28/2016 09:39 AM, Jakub Hrozek wrote: > > >> On Wed, Jul 27, 2016 at 04:09:07PM +0200, thierry bordaz wrote: > > >> > > > >> > On 07/27/2016 03:36 PM, Jakub Hrozek wrote: > > >> > > On Wed, Jul 27, 2016 at 02:55:37PM +0200, thierry bordaz wrote: > > >> > > > On 07/27/2016 01:56 PM, Jakub Hrozek wrote: > > >> > > > > On Wed, Jul 27, 2016 at 01:03:59PM +0200, Jakub Hrozek wrote: > > >> > > > > > On Wed, Jul 27, 2016 at 12:22:46PM +0200, Lukas Slebodnik > > >> > > > > > wrote: > > >> > > > > > > On (27/07/16 12:08), Jakub Hrozek wrote: > > >> > > > > > > > On Wed, Jul 27, 2016 at 12:02:24PM +0200, Jakub Hrozek > > >> > > > > > > > wrote: > > >> > > > > > > > > On Wed, Jul 27, 2016 at 11:54:16AM +0200, Lukas > > >> > > > > > > > > Slebodnik wrote: > > >> > > > > > > > > > ehlo, > > >> > > > > > > > > > > > >> > > > > > > > > > attached patch fixes acces denied after activating > > >> > > > > > > > > > user in 389ds. > > >> > > > > > > > > > Jakub had some comments/ideas in ticket but I think > > >> > > > > > > > > > it's better to discuss > > >> > > > > > > > > > about virtual attributes and timestamp cache on > > >> > > > > > > > > > mailing list. > > >> > > > > > > > > Yes, so the comment I have is that while this works, it > > >> > > > > > > > > might break some > > >> > > > > > > > > strange LDAP servers. > > >> > > > > > > > > > > >> > > > > > > > > We use modifyTimestamp as a 'positive' indicator that > > >> > > > > > > > > the entry has not > > >> > > > > > > > > changed -- if the modifyTimestamp didn't change, we > > >> > > > > > > > > consider the cached > > >> > > > > > > > > entry the same as what is on the server and only bump > > >> > > > > > > > > the timestamp > > >> > > > > > > > > cache. If the timestamp is different, we do a > > >> > > > > > > > > deep-comparison of cached > > >> > > > > > > > > attribute values with what is on the LDAP server and > > >> > > > > > > > > write the sysdb > > >> > > > > > > > > cache entry only if the attributes differ. > > >> > > > > > > > > > > >> > > > > > > > > I was wondering if we can use the modifyTimestamp at > > >> > > > > > > > > all, then, because > > >> > > > > > > > > even if it's the same, we might want to check the > > >> > > > > > > > > attributes to see if > > >> > > > > > > > > some of the values are different because some of the > > >> > > > > > > > > attributes might be > > >> > > > > > > > > this operational/virtual attribute.. > > >> > > > > > > > Sorry, sent too soon. > > >> > > > > > > > > > >> > > > > > > > I think the questions are -- 1) can we enumerate the > > >> > > > > > > > virtual attributes? > > >> > > > > > > That might be a question for 389-ds developers. > > >> > > > > > > But it's very likely it will be different on other LDAP > > >> > > > > > > servers. > > >> > > > > > > > > >> > > > > > > > 2) Would different LDAP servers have different virtual > > >> > > > > > > > attributes. > > >> > > > > > > > > > >> > > > > > > > For 2) maybe a possible solution might be to set a > > >> > > > > > > > non-existing > > >> > > > > > > > modifyTimestamp attribute value, but I would consider that > > >> > > > > > > > only a > > >> > > > > > > > kludge, we shouldn't break existing setups.. > > >> > > > > > > I am not satisfied with this POC solution either. > > >> > > > > > > > > >> > > > > > > So should we remove usage of modifyTimestamp for detecting > > >> > > > > > > changes? > > >> > > > > > I would prefer to ask the DS developers before removing it > > >> > > > > > completely. > > >> > > > > > > > >> > > > > > At least for large groups it might take a long time to compare > > >> > > > > > all attribute > > >> > > > > > values and IIRC we don't depend on any virtual attributes for > > >> > > > > > groups. Maybe > > >> > > > > > we could parametrize that part of the code and enable the fast > > >> > > > > > way with > > >> > > > > > modifyTimestamps for 'known' server types, that is for setups > > >> > > > > > with AD and > > >> > > > > > IPA providers. > > >> > > > > > > > >> > > > > > For users, there is typically not as many attributes so we > > >> > > > > > should be > > >> > > > > > fine deep-comparing all attributes. > > >> > > > > I'm adding Thierry (so please reply-to-all to keep him in the > > >> > > > > thread). > > >> > > > > > > >> > > > > Thierry, in the latest sssd version we tried to add a performance > > >> > > > > improvement related to how we store SSSD entries in the cache. > > >> > > > > The short > > >> > > > > version is that we store the modifyTimestamp attribute in the > > >> > > > > cache and > > >> > > > > when we fetch an entry, we compare the entry modifyTimestamp > > >> > > > > with what > > >> > > > > is on the server. When the two are the same, we say that the > > >> > > > > entry
[SSSD] Re: [PATCH] SYSDB: Check changed virtual attributes before modified timestamp
On 07/29/2016 12:53 PM, Jakub Hrozek wrote: On Fri, Jul 29, 2016 at 12:27:24PM +0200, thierry bordaz wrote: On 07/29/2016 12:21 PM, Jakub Hrozek wrote: On Fri, Jul 29, 2016 at 11:57:54AM +0200, thierry bordaz wrote: On 07/28/2016 04:49 PM, Lukas Slebodnik wrote: On (28/07/16 16:37), thierry bordaz wrote: ... That is correct and this is the expected behavior. Using ns-inactivate.pl with a role, it inactivates all the entries in that role adding nsaccountlock virtual attibute. You are right, update (add of nsaccountlock) of regular user can be done without update of its modifytimestamp. Thank you very much for confirmation and for info that plugin is not used on IPA. So we needn't special case nsaccountlock for IPA. We had a discussion on sssd devel meeting. And we agreed that we will do some performace measurements. And if there will be significant difference then we will check modifytimestamp only with IPA and AD. and it will be disabled by default with generic LDAP. LS Hi Lukas, Just to be sure. Does SSSD currently use or intend to use ns-inactivate/ns-activate to disable/enable ipa users ? Judging by the code, we use the value of nsAccountLock as well in IPA.. (before running the HBAC rules, we check the if the user is expired by looking at nsAccountLock -> see sdap_account_expired_rhds() and us calling sdap_access_send() from ipa_pam_access_handler_send(). Hi Jakub, Thanks for your answer. You are right, freeipa is only accessing nsaccountlock and does not use ns-inactivate/ns-activate. My concern is that there can be conflict if nsaccountlock is updated as a real attribute or by COS. so IMHO sssd should not use cos approach (like ns-inactivate/ns-activate). I'm sorry, but I don't know what is COS (I only know it's a Swedish clothing brand :-)), is there some link I can take a look at? Less fun than clothing brand: https://access.redhat.com/documentation/en-US/Red_Hat_Directory_Server/10/html/Administration_Guide/Advanced_Entry_Management-Assigning_Class_of_Service.html :) In short it defines virtual attributes. ns-inactivate/ns-activate allow to add/remove a virtual attribute (nsaccountlock). The problem is that IPA is using nsaccountlock as a real attribute , but the virtual value overwrites the real value. On an other side, setting nsaccountlock as a real attribute updates modifytimestamp that can be used to detect this modification. While setting nsaccountlock as a virtual attribute is done without update of modifytimestamp. thanks thierry ___ sssd-devel mailing list sssd-devel@lists.fedorahosted.org https://lists.fedorahosted.org/admin/lists/sssd-devel@lists.fedorahosted.org
[SSSD] Re: [PATCH] SYSDB: Check changed virtual attributes before modified timestamp
On 07/29/2016 12:21 PM, Jakub Hrozek wrote: On Fri, Jul 29, 2016 at 11:57:54AM +0200, thierry bordaz wrote: On 07/28/2016 04:49 PM, Lukas Slebodnik wrote: On (28/07/16 16:37), thierry bordaz wrote: ... That is correct and this is the expected behavior. Using ns-inactivate.pl with a role, it inactivates all the entries in that role adding nsaccountlock virtual attibute. You are right, update (add of nsaccountlock) of regular user can be done without update of its modifytimestamp. Thank you very much for confirmation and for info that plugin is not used on IPA. So we needn't special case nsaccountlock for IPA. We had a discussion on sssd devel meeting. And we agreed that we will do some performace measurements. And if there will be significant difference then we will check modifytimestamp only with IPA and AD. and it will be disabled by default with generic LDAP. LS Hi Lukas, Just to be sure. Does SSSD currently use or intend to use ns-inactivate/ns-activate to disable/enable ipa users ? Judging by the code, we use the value of nsAccountLock as well in IPA.. (before running the HBAC rules, we check the if the user is expired by looking at nsAccountLock -> see sdap_account_expired_rhds() and us calling sdap_access_send() from ipa_pam_access_handler_send(). Hi Jakub, Thanks for your answer. You are right, freeipa is only accessing nsaccountlock and does not use ns-inactivate/ns-activate. My concern is that there can be conflict if nsaccountlock is updated as a real attribute or by COS. so IMHO sssd should not use cos approach (like ns-inactivate/ns-activate). thanks thierry ___ sssd-devel mailing list sssd-devel@lists.fedorahosted.org https://lists.fedorahosted.org/admin/lists/sssd-devel@lists.fedorahosted.org
[SSSD] Re: [PATCH] SYSDB: Check changed virtual attributes before modified timestamp
On 07/28/2016 04:49 PM, Lukas Slebodnik wrote: On (28/07/16 16:37), thierry bordaz wrote: ... That is correct and this is the expected behavior. Using ns-inactivate.pl with a role, it inactivates all the entries in that role adding nsaccountlock virtual attibute. You are right, update (add of nsaccountlock) of regular user can be done without update of its modifytimestamp. Thank you very much for confirmation and for info that plugin is not used on IPA. So we needn't special case nsaccountlock for IPA. We had a discussion on sssd devel meeting. And we agreed that we will do some performace measurements. And if there will be significant difference then we will check modifytimestamp only with IPA and AD. and it will be disabled by default with generic LDAP. LS Hi Lukas, Just to be sure. Does SSSD currently use or intend to use ns-inactivate/ns-activate to disable/enable ipa users ? thanks thierry ___ sssd-devel mailing list sssd-devel@lists.fedorahosted.org https://lists.fedorahosted.org/admin/lists/sssd-devel@lists.fedorahosted.org
[SSSD] Re: [PATCH] SYSDB: Check changed virtual attributes before modified timestamp
On Fri, Jul 29, 2016 at 12:27:24PM +0200, thierry bordaz wrote: > > > On 07/29/2016 12:21 PM, Jakub Hrozek wrote: > > On Fri, Jul 29, 2016 at 11:57:54AM +0200, thierry bordaz wrote: > > > > > > On 07/28/2016 04:49 PM, Lukas Slebodnik wrote: > > > > On (28/07/16 16:37), thierry bordaz wrote: > > > > > ... > > > > > That is correct and this is the expected behavior. > > > > > Using ns-inactivate.pl with a role, it inactivates all the entries in > > > > > that > > > > > role adding nsaccountlock virtual attibute. > > > > > You are right, update (add of nsaccountlock) of regular user can be > > > > > done > > > > > without update of its modifytimestamp. > > > > > > > > > Thank you very much for confirmation and for info that plugin > > > > is not used on IPA. So we needn't special case nsaccountlock for IPA. > > > > > > > > We had a discussion on sssd devel meeting. And we agreed that we will > > > > do some performace measurements. And if there will be significant > > > > difference then we will check modifytimestamp only with IPA and AD. > > > > and it will be disabled by default with generic LDAP. > > > > > > > > LS > > > Hi Lukas, > > > > > > Just to be sure. Does SSSD currently use or intend to use > > > ns-inactivate/ns-activate to disable/enable ipa users ? > > Judging by the code, we use the value of nsAccountLock as well in IPA.. > > (before running the HBAC rules, we check the if the user is expired by > > looking at nsAccountLock -> see sdap_account_expired_rhds() and us > > calling sdap_access_send() from ipa_pam_access_handler_send(). > > Hi Jakub, > > Thanks for your answer. > You are right, freeipa is only accessing nsaccountlock and does not use > ns-inactivate/ns-activate. > My concern is that there can be conflict if nsaccountlock is updated as a > real attribute or by COS. > so IMHO sssd should not use cos approach (like ns-inactivate/ns-activate). I'm sorry, but I don't know what is COS (I only know it's a Swedish clothing brand :-)), is there some link I can take a look at? ___ sssd-devel mailing list sssd-devel@lists.fedorahosted.org https://lists.fedorahosted.org/admin/lists/sssd-devel@lists.fedorahosted.org
[SSSD] Re: [PATCH] SYSDB: Check changed virtual attributes before modified timestamp
On Fri, Jul 29, 2016 at 11:57:54AM +0200, thierry bordaz wrote: > > > On 07/28/2016 04:49 PM, Lukas Slebodnik wrote: > > On (28/07/16 16:37), thierry bordaz wrote: > > > ... > > > That is correct and this is the expected behavior. > > > Using ns-inactivate.pl with a role, it inactivates all the entries in that > > > role adding nsaccountlock virtual attibute. > > > You are right, update (add of nsaccountlock) of regular user can be done > > > without update of its modifytimestamp. > > > > > Thank you very much for confirmation and for info that plugin > > is not used on IPA. So we needn't special case nsaccountlock for IPA. > > > > We had a discussion on sssd devel meeting. And we agreed that we will > > do some performace measurements. And if there will be significant > > difference then we will check modifytimestamp only with IPA and AD. > > and it will be disabled by default with generic LDAP. > > > > LS > > Hi Lukas, > > Just to be sure. Does SSSD currently use or intend to use > ns-inactivate/ns-activate to disable/enable ipa users ? Judging by the code, we use the value of nsAccountLock as well in IPA.. (before running the HBAC rules, we check the if the user is expired by looking at nsAccountLock -> see sdap_account_expired_rhds() and us calling sdap_access_send() from ipa_pam_access_handler_send(). ___ sssd-devel mailing list sssd-devel@lists.fedorahosted.org https://lists.fedorahosted.org/admin/lists/sssd-devel@lists.fedorahosted.org
[SSSD] Re: [PATCH] SYSDB: Check changed virtual attributes before modified timestamp
On (28/07/16 16:37), thierry bordaz wrote: >On 07/28/2016 12:58 PM, Lukas Slebodnik wrote: >> On (28/07/16 12:44), thierry bordaz wrote: >> > >> > On 07/28/2016 11:17 AM, Lukas Slebodnik wrote: >> > > On (28/07/16 10:24), thierry bordaz wrote: >> > > > On 07/28/2016 09:39 AM, Jakub Hrozek wrote: >> > > > > On Wed, Jul 27, 2016 at 04:09:07PM +0200, thierry bordaz wrote: >> > > > > > On 07/27/2016 03:36 PM, Jakub Hrozek wrote: >> > > > > > > On Wed, Jul 27, 2016 at 02:55:37PM +0200, thierry bordaz wrote: >> > > > > > > > On 07/27/2016 01:56 PM, Jakub Hrozek wrote: >> > > > > > > > > On Wed, Jul 27, 2016 at 01:03:59PM +0200, Jakub Hrozek wrote: >> > > > > > > > > > On Wed, Jul 27, 2016 at 12:22:46PM +0200, Lukas Slebodnik >> > > > > > > > > > wrote: >> > > > > > > > > > > On (27/07/16 12:08), Jakub Hrozek wrote: >> > > > > > > > > > > > On Wed, Jul 27, 2016 at 12:02:24PM +0200, Jakub Hrozek >> > > > > > > > > > > > wrote: >> > > > > > > > > > > > > On Wed, Jul 27, 2016 at 11:54:16AM +0200, Lukas >> > > > > > > > > > > > > Slebodnik wrote: >> > > > > > > > > > > > > > ehlo, >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > attached patch fixes acces denied after activating >> > > > > > > > > > > > > > user in 389ds. >> > > > > > > > > > > > > > Jakub had some comments/ideas in ticket but I >> > > > > > > > > > > > > > think it's better to discuss >> > > > > > > > > > > > > > about virtual attributes and timestamp cache on >> > > > > > > > > > > > > > mailing list. >> > > > > > > > > > > > > Yes, so the comment I have is that while this works, >> > > > > > > > > > > > > it might break some >> > > > > > > > > > > > > strange LDAP servers. >> > > > > > > > > > > > > >> > > > > > > > > > > > > We use modifyTimestamp as a 'positive' indicator >> > > > > > > > > > > > > that the entry has not >> > > > > > > > > > > > > changed -- if the modifyTimestamp didn't change, we >> > > > > > > > > > > > > consider the cached >> > > > > > > > > > > > > entry the same as what is on the server and only >> > > > > > > > > > > > > bump the timestamp >> > > > > > > > > > > > > cache. If the timestamp is different, we do a >> > > > > > > > > > > > > deep-comparison of cached >> > > > > > > > > > > > > attribute values with what is on the LDAP server and >> > > > > > > > > > > > > write the sysdb >> > > > > > > > > > > > > cache entry only if the attributes differ. >> > > > > > > > > > > > > >> > > > > > > > > > > > > I was wondering if we can use the modifyTimestamp at >> > > > > > > > > > > > > all, then, because >> > > > > > > > > > > > > even if it's the same, we might want to check the >> > > > > > > > > > > > > attributes to see if >> > > > > > > > > > > > > some of the values are different because some of the >> > > > > > > > > > > > > attributes might be >> > > > > > > > > > > > > this operational/virtual attribute.. >> > > > > > > > > > > > Sorry, sent too soon. >> > > > > > > > > > > > >> > > > > > > > > > > > I think the questions are -- 1) can we enumerate the >> > > > > > > > > > > > virtual attributes? >> > > > > > > > > > > That might be a question for 389-ds developers. >> > > > > > > > > > > But it's very likely it will be different on other LDAP >> > > > > > > > > > > servers. >> > > > > > > > > > > >> > > > > > > > > > > > 2) Would different LDAP servers have different virtual >> > > > > > > > > > > > attributes. >> > > > > > > > > > > > >> > > > > > > > > > > > For 2) maybe a possible solution might be to set a >> > > > > > > > > > > > non-existing >> > > > > > > > > > > > modifyTimestamp attribute value, but I would consider >> > > > > > > > > > > > that only a >> > > > > > > > > > > > kludge, we shouldn't break existing setups.. >> > > > > > > > > > > I am not satisfied with this POC solution either. >> > > > > > > > > > > >> > > > > > > > > > > So should we remove usage of modifyTimestamp for >> > > > > > > > > > > detecting changes? >> > > > > > > > > > I would prefer to ask the DS developers before removing it >> > > > > > > > > > completely. >> > > > > > > > > > >> > > > > > > > > > At least for large groups it might take a long time to >> > > > > > > > > > compare all attribute >> > > > > > > > > > values and IIRC we don't depend on any virtual attributes >> > > > > > > > > > for groups. Maybe >> > > > > > > > > > we could parametrize that part of the code and enable the >> > > > > > > > > > fast way with >> > > > > > > > > > modifyTimestamps for 'known' server types, that is for >> > > > > > > > > > setups with AD and >> > > > > > > > > > IPA providers. >> > > > > > > > > > >> > > > > > > > > > For users, there is typically not as many attributes so we >> > > > > > > > > > should be >> > > > > > > > > > fine deep-comparing all attributes. >> > > > > > > > > I'm adding Thierry (so please reply-to-all to keep him in >> > > > > > > > > the thread). >> > > > > > > > > >> > > > > > > > > Thierry, in the latest sssd version we tried to add a >> > > > > > > > > per
[SSSD] Re: [PATCH] SYSDB: Check changed virtual attributes before modified timestamp
On 07/28/2016 12:58 PM, Lukas Slebodnik wrote: On (28/07/16 12:44), thierry bordaz wrote: On 07/28/2016 11:17 AM, Lukas Slebodnik wrote: On (28/07/16 10:24), thierry bordaz wrote: On 07/28/2016 09:39 AM, Jakub Hrozek wrote: On Wed, Jul 27, 2016 at 04:09:07PM +0200, thierry bordaz wrote: On 07/27/2016 03:36 PM, Jakub Hrozek wrote: On Wed, Jul 27, 2016 at 02:55:37PM +0200, thierry bordaz wrote: On 07/27/2016 01:56 PM, Jakub Hrozek wrote: On Wed, Jul 27, 2016 at 01:03:59PM +0200, Jakub Hrozek wrote: On Wed, Jul 27, 2016 at 12:22:46PM +0200, Lukas Slebodnik wrote: On (27/07/16 12:08), Jakub Hrozek wrote: On Wed, Jul 27, 2016 at 12:02:24PM +0200, Jakub Hrozek wrote: On Wed, Jul 27, 2016 at 11:54:16AM +0200, Lukas Slebodnik wrote: ehlo, attached patch fixes acces denied after activating user in 389ds. Jakub had some comments/ideas in ticket but I think it's better to discuss about virtual attributes and timestamp cache on mailing list. Yes, so the comment I have is that while this works, it might break some strange LDAP servers. We use modifyTimestamp as a 'positive' indicator that the entry has not changed -- if the modifyTimestamp didn't change, we consider the cached entry the same as what is on the server and only bump the timestamp cache. If the timestamp is different, we do a deep-comparison of cached attribute values with what is on the LDAP server and write the sysdb cache entry only if the attributes differ. I was wondering if we can use the modifyTimestamp at all, then, because even if it's the same, we might want to check the attributes to see if some of the values are different because some of the attributes might be this operational/virtual attribute.. Sorry, sent too soon. I think the questions are -- 1) can we enumerate the virtual attributes? That might be a question for 389-ds developers. But it's very likely it will be different on other LDAP servers. 2) Would different LDAP servers have different virtual attributes. For 2) maybe a possible solution might be to set a non-existing modifyTimestamp attribute value, but I would consider that only a kludge, we shouldn't break existing setups.. I am not satisfied with this POC solution either. So should we remove usage of modifyTimestamp for detecting changes? I would prefer to ask the DS developers before removing it completely. At least for large groups it might take a long time to compare all attribute values and IIRC we don't depend on any virtual attributes for groups. Maybe we could parametrize that part of the code and enable the fast way with modifyTimestamps for 'known' server types, that is for setups with AD and IPA providers. For users, there is typically not as many attributes so we should be fine deep-comparing all attributes. I'm adding Thierry (so please reply-to-all to keep him in the thread). Thierry, in the latest sssd version we tried to add a performance improvement related to how we store SSSD entries in the cache. The short version is that we store the modifyTimestamp attribute in the cache and when we fetch an entry, we compare the entry modifyTimestamp with what is on the server. When the two are the same, we say that the entry did not change and don't update the cache. This works fine for most attributes, but not for attributes like nsAccountLock which do not change modifyTimestamp when they are modified. So when an entry was already cached but then nsAccountLock changed, we treated the entry as the same and never read the new nsAccountLock value. To fix this, I think we have several options: 1) special-case the nsAccountLock. This seems a bit dangerous, because I'm not sure we can say that some other attribute we are interested in behaves the same as nsAccountLock. 2) drop the modifyTimestamp optimization completely. Then we fall back to comparing the attribute values, which might work, but for huge objects like groups with thousands of members, this might be too expensive. 3) only use the modifyTimestamp optimization for cases where we know we don't read any virtual attributes. And my question is -- can we, in general, know if the modifyTimestamp way of detecting changes is realiable for all LDAP servers? Or do you think it should only be used for cases where we know we are not interested in any virtual attributes (that would mostly be storing groups from servers where we know exactly what is on the server side, like IPA or AD). Hello, Relying on modifytimestamp looks a good idea. Any MOD/MODRDN will update it, except I think it is unchanged when updating some password policy attributes. Regarding virtual attribute, the only one I know in IPA is nsaccountlock. nsaccountlock is an operational attribute (you need to request it to see it) and is also a virtual attribute BUT only for 'staged' and 'deleted' users. It is a stored attribute for regular users and we should update modifytimestamp when it is set.
[SSSD] Re: [PATCH] SYSDB: Check changed virtual attributes before modified timestamp
On (28/07/16 12:58), Lukas Slebodnik wrote: >On (28/07/16 12:44), thierry bordaz wrote: >> >> >>On 07/28/2016 11:17 AM, Lukas Slebodnik wrote: >>> On (28/07/16 10:24), thierry bordaz wrote: >>> > >>> > On 07/28/2016 09:39 AM, Jakub Hrozek wrote: >>> > > On Wed, Jul 27, 2016 at 04:09:07PM +0200, thierry bordaz wrote: >>> > > > On 07/27/2016 03:36 PM, Jakub Hrozek wrote: >>> > > > > On Wed, Jul 27, 2016 at 02:55:37PM +0200, thierry bordaz wrote: >>> > > > > > On 07/27/2016 01:56 PM, Jakub Hrozek wrote: >>> > > > > > > On Wed, Jul 27, 2016 at 01:03:59PM +0200, Jakub Hrozek wrote: >>> > > > > > > > On Wed, Jul 27, 2016 at 12:22:46PM +0200, Lukas Slebodnik >>> > > > > > > > wrote: >>> > > > > > > > > On (27/07/16 12:08), Jakub Hrozek wrote: >>> > > > > > > > > > On Wed, Jul 27, 2016 at 12:02:24PM +0200, Jakub Hrozek >>> > > > > > > > > > wrote: >>> > > > > > > > > > > On Wed, Jul 27, 2016 at 11:54:16AM +0200, Lukas >>> > > > > > > > > > > Slebodnik wrote: >>> > > > > > > > > > > > ehlo, >>> > > > > > > > > > > > >>> > > > > > > > > > > > attached patch fixes acces denied after activating >>> > > > > > > > > > > > user in 389ds. >>> > > > > > > > > > > > Jakub had some comments/ideas in ticket but I think >>> > > > > > > > > > > > it's better to discuss >>> > > > > > > > > > > > about virtual attributes and timestamp cache on >>> > > > > > > > > > > > mailing list. >>> > > > > > > > > > > Yes, so the comment I have is that while this works, it >>> > > > > > > > > > > might break some >>> > > > > > > > > > > strange LDAP servers. >>> > > > > > > > > > > >>> > > > > > > > > > > We use modifyTimestamp as a 'positive' indicator that >>> > > > > > > > > > > the entry has not >>> > > > > > > > > > > changed -- if the modifyTimestamp didn't change, we >>> > > > > > > > > > > consider the cached >>> > > > > > > > > > > entry the same as what is on the server and only bump >>> > > > > > > > > > > the timestamp >>> > > > > > > > > > > cache. If the timestamp is different, we do a >>> > > > > > > > > > > deep-comparison of cached >>> > > > > > > > > > > attribute values with what is on the LDAP server and >>> > > > > > > > > > > write the sysdb >>> > > > > > > > > > > cache entry only if the attributes differ. >>> > > > > > > > > > > >>> > > > > > > > > > > I was wondering if we can use the modifyTimestamp at >>> > > > > > > > > > > all, then, because >>> > > > > > > > > > > even if it's the same, we might want to check the >>> > > > > > > > > > > attributes to see if >>> > > > > > > > > > > some of the values are different because some of the >>> > > > > > > > > > > attributes might be >>> > > > > > > > > > > this operational/virtual attribute.. >>> > > > > > > > > > Sorry, sent too soon. >>> > > > > > > > > > >>> > > > > > > > > > I think the questions are -- 1) can we enumerate the >>> > > > > > > > > > virtual attributes? >>> > > > > > > > > That might be a question for 389-ds developers. >>> > > > > > > > > But it's very likely it will be different on other LDAP >>> > > > > > > > > servers. >>> > > > > > > > > >>> > > > > > > > > > 2) Would different LDAP servers have different virtual >>> > > > > > > > > > attributes. >>> > > > > > > > > > >>> > > > > > > > > > For 2) maybe a possible solution might be to set a >>> > > > > > > > > > non-existing >>> > > > > > > > > > modifyTimestamp attribute value, but I would consider >>> > > > > > > > > > that only a >>> > > > > > > > > > kludge, we shouldn't break existing setups.. >>> > > > > > > > > I am not satisfied with this POC solution either. >>> > > > > > > > > >>> > > > > > > > > So should we remove usage of modifyTimestamp for detecting >>> > > > > > > > > changes? >>> > > > > > > > I would prefer to ask the DS developers before removing it >>> > > > > > > > completely. >>> > > > > > > > >>> > > > > > > > At least for large groups it might take a long time to >>> > > > > > > > compare all attribute >>> > > > > > > > values and IIRC we don't depend on any virtual attributes for >>> > > > > > > > groups. Maybe >>> > > > > > > > we could parametrize that part of the code and enable the >>> > > > > > > > fast way with >>> > > > > > > > modifyTimestamps for 'known' server types, that is for setups >>> > > > > > > > with AD and >>> > > > > > > > IPA providers. >>> > > > > > > > >>> > > > > > > > For users, there is typically not as many attributes so we >>> > > > > > > > should be >>> > > > > > > > fine deep-comparing all attributes. >>> > > > > > > I'm adding Thierry (so please reply-to-all to keep him in the >>> > > > > > > thread). >>> > > > > > > >>> > > > > > > Thierry, in the latest sssd version we tried to add a >>> > > > > > > performance >>> > > > > > > improvement related to how we store SSSD entries in the cache. >>> > > > > > > The short >>> > > > > > > version is that we store the modifyTimestamp attribute in the >>> > > > > > > cache and >>> > > > > > > when we fetch an entry, we compare the entry modifyTimestamp >>> >
[SSSD] Re: [PATCH] SYSDB: Check changed virtual attributes before modified timestamp
On (28/07/16 13:56), Jakub Hrozek wrote: >On Thu, Jul 28, 2016 at 01:33:32PM +0200, Lukas Slebodnik wrote: >> On (28/07/16 12:06), thierry bordaz wrote: >> >On 07/28/2016 09:39 AM, Jakub Hrozek wrote: >> >> On Wed, Jul 27, 2016 at 04:09:07PM +0200, thierry bordaz wrote: >> >> > >> >> > On 07/27/2016 03:36 PM, Jakub Hrozek wrote: >> >> > > On Wed, Jul 27, 2016 at 02:55:37PM +0200, thierry bordaz wrote: >> >> > > > On 07/27/2016 01:56 PM, Jakub Hrozek wrote: >> >> > > > > On Wed, Jul 27, 2016 at 01:03:59PM +0200, Jakub Hrozek wrote: >> >> > > > > > On Wed, Jul 27, 2016 at 12:22:46PM +0200, Lukas Slebodnik wrote: >> >> > > > > > > On (27/07/16 12:08), Jakub Hrozek wrote: >> >> > > > > > > > On Wed, Jul 27, 2016 at 12:02:24PM +0200, Jakub Hrozek >> >> > > > > > > > wrote: >> >> > > > > > > > > On Wed, Jul 27, 2016 at 11:54:16AM +0200, Lukas Slebodnik >> >> > > > > > > > > wrote: >> >> > > > > > > > > > ehlo, >> >> > > > > > > > > > >> >> > > > > > > > > > attached patch fixes acces denied after activating user >> >> > > > > > > > > > in 389ds. >> >> > > > > > > > > > Jakub had some comments/ideas in ticket but I think >> >> > > > > > > > > > it's better to discuss >> >> > > > > > > > > > about virtual attributes and timestamp cache on mailing >> >> > > > > > > > > > list. >> >> > > > > > > > > Yes, so the comment I have is that while this works, it >> >> > > > > > > > > might break some >> >> > > > > > > > > strange LDAP servers. >> >> > > > > > > > > >> >> > > > > > > > > We use modifyTimestamp as a 'positive' indicator that the >> >> > > > > > > > > entry has not >> >> > > > > > > > > changed -- if the modifyTimestamp didn't change, we >> >> > > > > > > > > consider the cached >> >> > > > > > > > > entry the same as what is on the server and only bump the >> >> > > > > > > > > timestamp >> >> > > > > > > > > cache. If the timestamp is different, we do a >> >> > > > > > > > > deep-comparison of cached >> >> > > > > > > > > attribute values with what is on the LDAP server and >> >> > > > > > > > > write the sysdb >> >> > > > > > > > > cache entry only if the attributes differ. >> >> > > > > > > > > >> >> > > > > > > > > I was wondering if we can use the modifyTimestamp at all, >> >> > > > > > > > > then, because >> >> > > > > > > > > even if it's the same, we might want to check the >> >> > > > > > > > > attributes to see if >> >> > > > > > > > > some of the values are different because some of the >> >> > > > > > > > > attributes might be >> >> > > > > > > > > this operational/virtual attribute.. >> >> > > > > > > > Sorry, sent too soon. >> >> > > > > > > > >> >> > > > > > > > I think the questions are -- 1) can we enumerate the >> >> > > > > > > > virtual attributes? >> >> > > > > > > That might be a question for 389-ds developers. >> >> > > > > > > But it's very likely it will be different on other LDAP >> >> > > > > > > servers. >> >> > > > > > > >> >> > > > > > > > 2) Would different LDAP servers have different virtual >> >> > > > > > > > attributes. >> >> > > > > > > > >> >> > > > > > > > For 2) maybe a possible solution might be to set a >> >> > > > > > > > non-existing >> >> > > > > > > > modifyTimestamp attribute value, but I would consider that >> >> > > > > > > > only a >> >> > > > > > > > kludge, we shouldn't break existing setups.. >> >> > > > > > > I am not satisfied with this POC solution either. >> >> > > > > > > >> >> > > > > > > So should we remove usage of modifyTimestamp for detecting >> >> > > > > > > changes? >> >> > > > > > I would prefer to ask the DS developers before removing it >> >> > > > > > completely. >> >> > > > > > >> >> > > > > > At least for large groups it might take a long time to compare >> >> > > > > > all attribute >> >> > > > > > values and IIRC we don't depend on any virtual attributes for >> >> > > > > > groups. Maybe >> >> > > > > > we could parametrize that part of the code and enable the fast >> >> > > > > > way with >> >> > > > > > modifyTimestamps for 'known' server types, that is for setups >> >> > > > > > with AD and >> >> > > > > > IPA providers. >> >> > > > > > >> >> > > > > > For users, there is typically not as many attributes so we >> >> > > > > > should be >> >> > > > > > fine deep-comparing all attributes. >> >> > > > > I'm adding Thierry (so please reply-to-all to keep him in the >> >> > > > > thread). >> >> > > > > >> >> > > > > Thierry, in the latest sssd version we tried to add a performance >> >> > > > > improvement related to how we store SSSD entries in the cache. >> >> > > > > The short >> >> > > > > version is that we store the modifyTimestamp attribute in the >> >> > > > > cache and >> >> > > > > when we fetch an entry, we compare the entry modifyTimestamp with >> >> > > > > what >> >> > > > > is on the server. When the two are the same, we say that the >> >> > > > > entry did >> >> > > > > not change and don't update the cache. >> >> > > > > >> >> > > > > This works fine for most attributes, but not f
[SSSD] Re: [PATCH] SYSDB: Check changed virtual attributes before modified timestamp
On Thu, Jul 28, 2016 at 01:33:32PM +0200, Lukas Slebodnik wrote: > On (28/07/16 12:06), thierry bordaz wrote: > >On 07/28/2016 09:39 AM, Jakub Hrozek wrote: > >> On Wed, Jul 27, 2016 at 04:09:07PM +0200, thierry bordaz wrote: > >> > > >> > On 07/27/2016 03:36 PM, Jakub Hrozek wrote: > >> > > On Wed, Jul 27, 2016 at 02:55:37PM +0200, thierry bordaz wrote: > >> > > > On 07/27/2016 01:56 PM, Jakub Hrozek wrote: > >> > > > > On Wed, Jul 27, 2016 at 01:03:59PM +0200, Jakub Hrozek wrote: > >> > > > > > On Wed, Jul 27, 2016 at 12:22:46PM +0200, Lukas Slebodnik wrote: > >> > > > > > > On (27/07/16 12:08), Jakub Hrozek wrote: > >> > > > > > > > On Wed, Jul 27, 2016 at 12:02:24PM +0200, Jakub Hrozek wrote: > >> > > > > > > > > On Wed, Jul 27, 2016 at 11:54:16AM +0200, Lukas Slebodnik > >> > > > > > > > > wrote: > >> > > > > > > > > > ehlo, > >> > > > > > > > > > > >> > > > > > > > > > attached patch fixes acces denied after activating user > >> > > > > > > > > > in 389ds. > >> > > > > > > > > > Jakub had some comments/ideas in ticket but I think it's > >> > > > > > > > > > better to discuss > >> > > > > > > > > > about virtual attributes and timestamp cache on mailing > >> > > > > > > > > > list. > >> > > > > > > > > Yes, so the comment I have is that while this works, it > >> > > > > > > > > might break some > >> > > > > > > > > strange LDAP servers. > >> > > > > > > > > > >> > > > > > > > > We use modifyTimestamp as a 'positive' indicator that the > >> > > > > > > > > entry has not > >> > > > > > > > > changed -- if the modifyTimestamp didn't change, we > >> > > > > > > > > consider the cached > >> > > > > > > > > entry the same as what is on the server and only bump the > >> > > > > > > > > timestamp > >> > > > > > > > > cache. If the timestamp is different, we do a > >> > > > > > > > > deep-comparison of cached > >> > > > > > > > > attribute values with what is on the LDAP server and write > >> > > > > > > > > the sysdb > >> > > > > > > > > cache entry only if the attributes differ. > >> > > > > > > > > > >> > > > > > > > > I was wondering if we can use the modifyTimestamp at all, > >> > > > > > > > > then, because > >> > > > > > > > > even if it's the same, we might want to check the > >> > > > > > > > > attributes to see if > >> > > > > > > > > some of the values are different because some of the > >> > > > > > > > > attributes might be > >> > > > > > > > > this operational/virtual attribute.. > >> > > > > > > > Sorry, sent too soon. > >> > > > > > > > > >> > > > > > > > I think the questions are -- 1) can we enumerate the virtual > >> > > > > > > > attributes? > >> > > > > > > That might be a question for 389-ds developers. > >> > > > > > > But it's very likely it will be different on other LDAP > >> > > > > > > servers. > >> > > > > > > > >> > > > > > > > 2) Would different LDAP servers have different virtual > >> > > > > > > > attributes. > >> > > > > > > > > >> > > > > > > > For 2) maybe a possible solution might be to set a > >> > > > > > > > non-existing > >> > > > > > > > modifyTimestamp attribute value, but I would consider that > >> > > > > > > > only a > >> > > > > > > > kludge, we shouldn't break existing setups.. > >> > > > > > > I am not satisfied with this POC solution either. > >> > > > > > > > >> > > > > > > So should we remove usage of modifyTimestamp for detecting > >> > > > > > > changes? > >> > > > > > I would prefer to ask the DS developers before removing it > >> > > > > > completely. > >> > > > > > > >> > > > > > At least for large groups it might take a long time to compare > >> > > > > > all attribute > >> > > > > > values and IIRC we don't depend on any virtual attributes for > >> > > > > > groups. Maybe > >> > > > > > we could parametrize that part of the code and enable the fast > >> > > > > > way with > >> > > > > > modifyTimestamps for 'known' server types, that is for setups > >> > > > > > with AD and > >> > > > > > IPA providers. > >> > > > > > > >> > > > > > For users, there is typically not as many attributes so we > >> > > > > > should be > >> > > > > > fine deep-comparing all attributes. > >> > > > > I'm adding Thierry (so please reply-to-all to keep him in the > >> > > > > thread). > >> > > > > > >> > > > > Thierry, in the latest sssd version we tried to add a performance > >> > > > > improvement related to how we store SSSD entries in the cache. The > >> > > > > short > >> > > > > version is that we store the modifyTimestamp attribute in the > >> > > > > cache and > >> > > > > when we fetch an entry, we compare the entry modifyTimestamp with > >> > > > > what > >> > > > > is on the server. When the two are the same, we say that the entry > >> > > > > did > >> > > > > not change and don't update the cache. > >> > > > > > >> > > > > This works fine for most attributes, but not for attributes like > >> > > > > nsAccountLock which do not change modifyTimestamp when they are > >> > > > > modified. So when an entry was already cached but
[SSSD] Re: [PATCH] SYSDB: Check changed virtual attributes before modified timestamp
On 07/28/2016 12:06 PM, thierry bordaz wrote: On 07/28/2016 09:39 AM, Jakub Hrozek wrote: On Wed, Jul 27, 2016 at 04:09:07PM +0200, thierry bordaz wrote: On 07/27/2016 03:36 PM, Jakub Hrozek wrote: On Wed, Jul 27, 2016 at 02:55:37PM +0200, thierry bordaz wrote: On 07/27/2016 01:56 PM, Jakub Hrozek wrote: On Wed, Jul 27, 2016 at 01:03:59PM +0200, Jakub Hrozek wrote: On Wed, Jul 27, 2016 at 12:22:46PM +0200, Lukas Slebodnik wrote: On (27/07/16 12:08), Jakub Hrozek wrote: On Wed, Jul 27, 2016 at 12:02:24PM +0200, Jakub Hrozek wrote: On Wed, Jul 27, 2016 at 11:54:16AM +0200, Lukas Slebodnik wrote: ehlo, attached patch fixes acces denied after activating user in 389ds. Jakub had some comments/ideas in ticket but I think it's better to discuss about virtual attributes and timestamp cache on mailing list. Yes, so the comment I have is that while this works, it might break some strange LDAP servers. We use modifyTimestamp as a 'positive' indicator that the entry has not changed -- if the modifyTimestamp didn't change, we consider the cached entry the same as what is on the server and only bump the timestamp cache. If the timestamp is different, we do a deep-comparison of cached attribute values with what is on the LDAP server and write the sysdb cache entry only if the attributes differ. I was wondering if we can use the modifyTimestamp at all, then, because even if it's the same, we might want to check the attributes to see if some of the values are different because some of the attributes might be this operational/virtual attribute.. Sorry, sent too soon. I think the questions are -- 1) can we enumerate the virtual attributes? That might be a question for 389-ds developers. But it's very likely it will be different on other LDAP servers. 2) Would different LDAP servers have different virtual attributes. For 2) maybe a possible solution might be to set a non-existing modifyTimestamp attribute value, but I would consider that only a kludge, we shouldn't break existing setups.. I am not satisfied with this POC solution either. So should we remove usage of modifyTimestamp for detecting changes? I would prefer to ask the DS developers before removing it completely. At least for large groups it might take a long time to compare all attribute values and IIRC we don't depend on any virtual attributes for groups. Maybe we could parametrize that part of the code and enable the fast way with modifyTimestamps for 'known' server types, that is for setups with AD and IPA providers. For users, there is typically not as many attributes so we should be fine deep-comparing all attributes. I'm adding Thierry (so please reply-to-all to keep him in the thread). Thierry, in the latest sssd version we tried to add a performance improvement related to how we store SSSD entries in the cache. The short version is that we store the modifyTimestamp attribute in the cache and when we fetch an entry, we compare the entry modifyTimestamp with what is on the server. When the two are the same, we say that the entry did not change and don't update the cache. This works fine for most attributes, but not for attributes like nsAccountLock which do not change modifyTimestamp when they are modified. So when an entry was already cached but then nsAccountLock changed, we treated the entry as the same and never read the new nsAccountLock value. To fix this, I think we have several options: 1) special-case the nsAccountLock. This seems a bit dangerous, because I'm not sure we can say that some other attribute we are interested in behaves the same as nsAccountLock. 2) drop the modifyTimestamp optimization completely. Then we fall back to comparing the attribute values, which might work, but for huge objects like groups with thousands of members, this might be too expensive. 3) only use the modifyTimestamp optimization for cases where we know we don't read any virtual attributes. And my question is -- can we, in general, know if the modifyTimestamp way of detecting changes is realiable for all LDAP servers? Or do you think it should only be used for cases where we know we are not interested in any virtual attributes (that would mostly be storing groups from servers where we know exactly what is on the server side, like IPA or AD). Hello, Relying on modifytimestamp looks a good idea. Any MOD/MODRDN will update it, except I think it is unchanged when updating some password policy attributes. Regarding virtual attribute, the only one I know in IPA is nsaccountlock. nsaccountlock is an operational attribute (you need to request it to see it) and is also a virtual attribute BUT only for 'staged' and 'deleted' users. It is a stored attribute for regular users and we should update modifytimestamp when it is set. thanks thierry OK, in that case it seems like we can special-case it. But do you know about any oth
[SSSD] Re: [PATCH] SYSDB: Check changed virtual attributes before modified timestamp
On 07/28/2016 11:17 AM, Lukas Slebodnik wrote: On (28/07/16 10:24), thierry bordaz wrote: On 07/28/2016 09:39 AM, Jakub Hrozek wrote: On Wed, Jul 27, 2016 at 04:09:07PM +0200, thierry bordaz wrote: On 07/27/2016 03:36 PM, Jakub Hrozek wrote: On Wed, Jul 27, 2016 at 02:55:37PM +0200, thierry bordaz wrote: On 07/27/2016 01:56 PM, Jakub Hrozek wrote: On Wed, Jul 27, 2016 at 01:03:59PM +0200, Jakub Hrozek wrote: On Wed, Jul 27, 2016 at 12:22:46PM +0200, Lukas Slebodnik wrote: On (27/07/16 12:08), Jakub Hrozek wrote: On Wed, Jul 27, 2016 at 12:02:24PM +0200, Jakub Hrozek wrote: On Wed, Jul 27, 2016 at 11:54:16AM +0200, Lukas Slebodnik wrote: ehlo, attached patch fixes acces denied after activating user in 389ds. Jakub had some comments/ideas in ticket but I think it's better to discuss about virtual attributes and timestamp cache on mailing list. Yes, so the comment I have is that while this works, it might break some strange LDAP servers. We use modifyTimestamp as a 'positive' indicator that the entry has not changed -- if the modifyTimestamp didn't change, we consider the cached entry the same as what is on the server and only bump the timestamp cache. If the timestamp is different, we do a deep-comparison of cached attribute values with what is on the LDAP server and write the sysdb cache entry only if the attributes differ. I was wondering if we can use the modifyTimestamp at all, then, because even if it's the same, we might want to check the attributes to see if some of the values are different because some of the attributes might be this operational/virtual attribute.. Sorry, sent too soon. I think the questions are -- 1) can we enumerate the virtual attributes? That might be a question for 389-ds developers. But it's very likely it will be different on other LDAP servers. 2) Would different LDAP servers have different virtual attributes. For 2) maybe a possible solution might be to set a non-existing modifyTimestamp attribute value, but I would consider that only a kludge, we shouldn't break existing setups.. I am not satisfied with this POC solution either. So should we remove usage of modifyTimestamp for detecting changes? I would prefer to ask the DS developers before removing it completely. At least for large groups it might take a long time to compare all attribute values and IIRC we don't depend on any virtual attributes for groups. Maybe we could parametrize that part of the code and enable the fast way with modifyTimestamps for 'known' server types, that is for setups with AD and IPA providers. For users, there is typically not as many attributes so we should be fine deep-comparing all attributes. I'm adding Thierry (so please reply-to-all to keep him in the thread). Thierry, in the latest sssd version we tried to add a performance improvement related to how we store SSSD entries in the cache. The short version is that we store the modifyTimestamp attribute in the cache and when we fetch an entry, we compare the entry modifyTimestamp with what is on the server. When the two are the same, we say that the entry did not change and don't update the cache. This works fine for most attributes, but not for attributes like nsAccountLock which do not change modifyTimestamp when they are modified. So when an entry was already cached but then nsAccountLock changed, we treated the entry as the same and never read the new nsAccountLock value. To fix this, I think we have several options: 1) special-case the nsAccountLock. This seems a bit dangerous, because I'm not sure we can say that some other attribute we are interested in behaves the same as nsAccountLock. 2) drop the modifyTimestamp optimization completely. Then we fall back to comparing the attribute values, which might work, but for huge objects like groups with thousands of members, this might be too expensive. 3) only use the modifyTimestamp optimization for cases where we know we don't read any virtual attributes. And my question is -- can we, in general, know if the modifyTimestamp way of detecting changes is realiable for all LDAP servers? Or do you think it should only be used for cases where we know we are not interested in any virtual attributes (that would mostly be storing groups from servers where we know exactly what is on the server side, like IPA or AD). Hello, Relying on modifytimestamp looks a good idea. Any MOD/MODRDN will update it, except I think it is unchanged when updating some password policy attributes. Regarding virtual attribute, the only one I know in IPA is nsaccountlock. nsaccountlock is an operational attribute (you need to request it to see it) and is also a virtual attribute BUT only for 'staged' and 'deleted' users. It is a stored attribute for regular users and we should update modifytimestamp when it is set. thanks thierry OK, in that case it seems like we can special-case it. But do you know about any ot
[SSSD] Re: [PATCH] SYSDB: Check changed virtual attributes before modified timestamp
On (28/07/16 12:06), thierry bordaz wrote: >On 07/28/2016 09:39 AM, Jakub Hrozek wrote: >> On Wed, Jul 27, 2016 at 04:09:07PM +0200, thierry bordaz wrote: >> > >> > On 07/27/2016 03:36 PM, Jakub Hrozek wrote: >> > > On Wed, Jul 27, 2016 at 02:55:37PM +0200, thierry bordaz wrote: >> > > > On 07/27/2016 01:56 PM, Jakub Hrozek wrote: >> > > > > On Wed, Jul 27, 2016 at 01:03:59PM +0200, Jakub Hrozek wrote: >> > > > > > On Wed, Jul 27, 2016 at 12:22:46PM +0200, Lukas Slebodnik wrote: >> > > > > > > On (27/07/16 12:08), Jakub Hrozek wrote: >> > > > > > > > On Wed, Jul 27, 2016 at 12:02:24PM +0200, Jakub Hrozek wrote: >> > > > > > > > > On Wed, Jul 27, 2016 at 11:54:16AM +0200, Lukas Slebodnik >> > > > > > > > > wrote: >> > > > > > > > > > ehlo, >> > > > > > > > > > >> > > > > > > > > > attached patch fixes acces denied after activating user in >> > > > > > > > > > 389ds. >> > > > > > > > > > Jakub had some comments/ideas in ticket but I think it's >> > > > > > > > > > better to discuss >> > > > > > > > > > about virtual attributes and timestamp cache on mailing >> > > > > > > > > > list. >> > > > > > > > > Yes, so the comment I have is that while this works, it >> > > > > > > > > might break some >> > > > > > > > > strange LDAP servers. >> > > > > > > > > >> > > > > > > > > We use modifyTimestamp as a 'positive' indicator that the >> > > > > > > > > entry has not >> > > > > > > > > changed -- if the modifyTimestamp didn't change, we consider >> > > > > > > > > the cached >> > > > > > > > > entry the same as what is on the server and only bump the >> > > > > > > > > timestamp >> > > > > > > > > cache. If the timestamp is different, we do a >> > > > > > > > > deep-comparison of cached >> > > > > > > > > attribute values with what is on the LDAP server and write >> > > > > > > > > the sysdb >> > > > > > > > > cache entry only if the attributes differ. >> > > > > > > > > >> > > > > > > > > I was wondering if we can use the modifyTimestamp at all, >> > > > > > > > > then, because >> > > > > > > > > even if it's the same, we might want to check the attributes >> > > > > > > > > to see if >> > > > > > > > > some of the values are different because some of the >> > > > > > > > > attributes might be >> > > > > > > > > this operational/virtual attribute.. >> > > > > > > > Sorry, sent too soon. >> > > > > > > > >> > > > > > > > I think the questions are -- 1) can we enumerate the virtual >> > > > > > > > attributes? >> > > > > > > That might be a question for 389-ds developers. >> > > > > > > But it's very likely it will be different on other LDAP servers. >> > > > > > > >> > > > > > > > 2) Would different LDAP servers have different virtual >> > > > > > > > attributes. >> > > > > > > > >> > > > > > > > For 2) maybe a possible solution might be to set a non-existing >> > > > > > > > modifyTimestamp attribute value, but I would consider that >> > > > > > > > only a >> > > > > > > > kludge, we shouldn't break existing setups.. >> > > > > > > I am not satisfied with this POC solution either. >> > > > > > > >> > > > > > > So should we remove usage of modifyTimestamp for detecting >> > > > > > > changes? >> > > > > > I would prefer to ask the DS developers before removing it >> > > > > > completely. >> > > > > > >> > > > > > At least for large groups it might take a long time to compare all >> > > > > > attribute >> > > > > > values and IIRC we don't depend on any virtual attributes for >> > > > > > groups. Maybe >> > > > > > we could parametrize that part of the code and enable the fast way >> > > > > > with >> > > > > > modifyTimestamps for 'known' server types, that is for setups with >> > > > > > AD and >> > > > > > IPA providers. >> > > > > > >> > > > > > For users, there is typically not as many attributes so we should >> > > > > > be >> > > > > > fine deep-comparing all attributes. >> > > > > I'm adding Thierry (so please reply-to-all to keep him in the >> > > > > thread). >> > > > > >> > > > > Thierry, in the latest sssd version we tried to add a performance >> > > > > improvement related to how we store SSSD entries in the cache. The >> > > > > short >> > > > > version is that we store the modifyTimestamp attribute in the cache >> > > > > and >> > > > > when we fetch an entry, we compare the entry modifyTimestamp with >> > > > > what >> > > > > is on the server. When the two are the same, we say that the entry >> > > > > did >> > > > > not change and don't update the cache. >> > > > > >> > > > > This works fine for most attributes, but not for attributes like >> > > > > nsAccountLock which do not change modifyTimestamp when they are >> > > > > modified. So when an entry was already cached but then nsAccountLock >> > > > > changed, we treated the entry as the same and never read the new >> > > > > nsAccountLock value. >> > > > > >> > > > > To fix this, I think we have several options: >> > > > >1) special-case the nsAccountLock. This seems a bit dangerous, >> > > > >
[SSSD] Re: [PATCH] SYSDB: Check changed virtual attributes before modified timestamp
On (28/07/16 12:44), thierry bordaz wrote: > > >On 07/28/2016 11:17 AM, Lukas Slebodnik wrote: >> On (28/07/16 10:24), thierry bordaz wrote: >> > >> > On 07/28/2016 09:39 AM, Jakub Hrozek wrote: >> > > On Wed, Jul 27, 2016 at 04:09:07PM +0200, thierry bordaz wrote: >> > > > On 07/27/2016 03:36 PM, Jakub Hrozek wrote: >> > > > > On Wed, Jul 27, 2016 at 02:55:37PM +0200, thierry bordaz wrote: >> > > > > > On 07/27/2016 01:56 PM, Jakub Hrozek wrote: >> > > > > > > On Wed, Jul 27, 2016 at 01:03:59PM +0200, Jakub Hrozek wrote: >> > > > > > > > On Wed, Jul 27, 2016 at 12:22:46PM +0200, Lukas Slebodnik >> > > > > > > > wrote: >> > > > > > > > > On (27/07/16 12:08), Jakub Hrozek wrote: >> > > > > > > > > > On Wed, Jul 27, 2016 at 12:02:24PM +0200, Jakub Hrozek >> > > > > > > > > > wrote: >> > > > > > > > > > > On Wed, Jul 27, 2016 at 11:54:16AM +0200, Lukas >> > > > > > > > > > > Slebodnik wrote: >> > > > > > > > > > > > ehlo, >> > > > > > > > > > > > >> > > > > > > > > > > > attached patch fixes acces denied after activating >> > > > > > > > > > > > user in 389ds. >> > > > > > > > > > > > Jakub had some comments/ideas in ticket but I think >> > > > > > > > > > > > it's better to discuss >> > > > > > > > > > > > about virtual attributes and timestamp cache on >> > > > > > > > > > > > mailing list. >> > > > > > > > > > > Yes, so the comment I have is that while this works, it >> > > > > > > > > > > might break some >> > > > > > > > > > > strange LDAP servers. >> > > > > > > > > > > >> > > > > > > > > > > We use modifyTimestamp as a 'positive' indicator that >> > > > > > > > > > > the entry has not >> > > > > > > > > > > changed -- if the modifyTimestamp didn't change, we >> > > > > > > > > > > consider the cached >> > > > > > > > > > > entry the same as what is on the server and only bump >> > > > > > > > > > > the timestamp >> > > > > > > > > > > cache. If the timestamp is different, we do a >> > > > > > > > > > > deep-comparison of cached >> > > > > > > > > > > attribute values with what is on the LDAP server and >> > > > > > > > > > > write the sysdb >> > > > > > > > > > > cache entry only if the attributes differ. >> > > > > > > > > > > >> > > > > > > > > > > I was wondering if we can use the modifyTimestamp at >> > > > > > > > > > > all, then, because >> > > > > > > > > > > even if it's the same, we might want to check the >> > > > > > > > > > > attributes to see if >> > > > > > > > > > > some of the values are different because some of the >> > > > > > > > > > > attributes might be >> > > > > > > > > > > this operational/virtual attribute.. >> > > > > > > > > > Sorry, sent too soon. >> > > > > > > > > > >> > > > > > > > > > I think the questions are -- 1) can we enumerate the >> > > > > > > > > > virtual attributes? >> > > > > > > > > That might be a question for 389-ds developers. >> > > > > > > > > But it's very likely it will be different on other LDAP >> > > > > > > > > servers. >> > > > > > > > > >> > > > > > > > > > 2) Would different LDAP servers have different virtual >> > > > > > > > > > attributes. >> > > > > > > > > > >> > > > > > > > > > For 2) maybe a possible solution might be to set a >> > > > > > > > > > non-existing >> > > > > > > > > > modifyTimestamp attribute value, but I would consider that >> > > > > > > > > > only a >> > > > > > > > > > kludge, we shouldn't break existing setups.. >> > > > > > > > > I am not satisfied with this POC solution either. >> > > > > > > > > >> > > > > > > > > So should we remove usage of modifyTimestamp for detecting >> > > > > > > > > changes? >> > > > > > > > I would prefer to ask the DS developers before removing it >> > > > > > > > completely. >> > > > > > > > >> > > > > > > > At least for large groups it might take a long time to compare >> > > > > > > > all attribute >> > > > > > > > values and IIRC we don't depend on any virtual attributes for >> > > > > > > > groups. Maybe >> > > > > > > > we could parametrize that part of the code and enable the fast >> > > > > > > > way with >> > > > > > > > modifyTimestamps for 'known' server types, that is for setups >> > > > > > > > with AD and >> > > > > > > > IPA providers. >> > > > > > > > >> > > > > > > > For users, there is typically not as many attributes so we >> > > > > > > > should be >> > > > > > > > fine deep-comparing all attributes. >> > > > > > > I'm adding Thierry (so please reply-to-all to keep him in the >> > > > > > > thread). >> > > > > > > >> > > > > > > Thierry, in the latest sssd version we tried to add a performance >> > > > > > > improvement related to how we store SSSD entries in the cache. >> > > > > > > The short >> > > > > > > version is that we store the modifyTimestamp attribute in the >> > > > > > > cache and >> > > > > > > when we fetch an entry, we compare the entry modifyTimestamp >> > > > > > > with what >> > > > > > > is on the server. When the two are the same, we say that the >> > > > > > > entry did >> > > > > > > not change and don'
[SSSD] Re: [PATCH] SYSDB: Check changed virtual attributes before modified timestamp
On 07/28/2016 09:39 AM, Jakub Hrozek wrote: On Wed, Jul 27, 2016 at 04:09:07PM +0200, thierry bordaz wrote: On 07/27/2016 03:36 PM, Jakub Hrozek wrote: On Wed, Jul 27, 2016 at 02:55:37PM +0200, thierry bordaz wrote: On 07/27/2016 01:56 PM, Jakub Hrozek wrote: On Wed, Jul 27, 2016 at 01:03:59PM +0200, Jakub Hrozek wrote: On Wed, Jul 27, 2016 at 12:22:46PM +0200, Lukas Slebodnik wrote: On (27/07/16 12:08), Jakub Hrozek wrote: On Wed, Jul 27, 2016 at 12:02:24PM +0200, Jakub Hrozek wrote: On Wed, Jul 27, 2016 at 11:54:16AM +0200, Lukas Slebodnik wrote: ehlo, attached patch fixes acces denied after activating user in 389ds. Jakub had some comments/ideas in ticket but I think it's better to discuss about virtual attributes and timestamp cache on mailing list. Yes, so the comment I have is that while this works, it might break some strange LDAP servers. We use modifyTimestamp as a 'positive' indicator that the entry has not changed -- if the modifyTimestamp didn't change, we consider the cached entry the same as what is on the server and only bump the timestamp cache. If the timestamp is different, we do a deep-comparison of cached attribute values with what is on the LDAP server and write the sysdb cache entry only if the attributes differ. I was wondering if we can use the modifyTimestamp at all, then, because even if it's the same, we might want to check the attributes to see if some of the values are different because some of the attributes might be this operational/virtual attribute.. Sorry, sent too soon. I think the questions are -- 1) can we enumerate the virtual attributes? That might be a question for 389-ds developers. But it's very likely it will be different on other LDAP servers. 2) Would different LDAP servers have different virtual attributes. For 2) maybe a possible solution might be to set a non-existing modifyTimestamp attribute value, but I would consider that only a kludge, we shouldn't break existing setups.. I am not satisfied with this POC solution either. So should we remove usage of modifyTimestamp for detecting changes? I would prefer to ask the DS developers before removing it completely. At least for large groups it might take a long time to compare all attribute values and IIRC we don't depend on any virtual attributes for groups. Maybe we could parametrize that part of the code and enable the fast way with modifyTimestamps for 'known' server types, that is for setups with AD and IPA providers. For users, there is typically not as many attributes so we should be fine deep-comparing all attributes. I'm adding Thierry (so please reply-to-all to keep him in the thread). Thierry, in the latest sssd version we tried to add a performance improvement related to how we store SSSD entries in the cache. The short version is that we store the modifyTimestamp attribute in the cache and when we fetch an entry, we compare the entry modifyTimestamp with what is on the server. When the two are the same, we say that the entry did not change and don't update the cache. This works fine for most attributes, but not for attributes like nsAccountLock which do not change modifyTimestamp when they are modified. So when an entry was already cached but then nsAccountLock changed, we treated the entry as the same and never read the new nsAccountLock value. To fix this, I think we have several options: 1) special-case the nsAccountLock. This seems a bit dangerous, because I'm not sure we can say that some other attribute we are interested in behaves the same as nsAccountLock. 2) drop the modifyTimestamp optimization completely. Then we fall back to comparing the attribute values, which might work, but for huge objects like groups with thousands of members, this might be too expensive. 3) only use the modifyTimestamp optimization for cases where we know we don't read any virtual attributes. And my question is -- can we, in general, know if the modifyTimestamp way of detecting changes is realiable for all LDAP servers? Or do you think it should only be used for cases where we know we are not interested in any virtual attributes (that would mostly be storing groups from servers where we know exactly what is on the server side, like IPA or AD). Hello, Relying on modifytimestamp looks a good idea. Any MOD/MODRDN will update it, except I think it is unchanged when updating some password policy attributes. Regarding virtual attribute, the only one I know in IPA is nsaccountlock. nsaccountlock is an operational attribute (you need to request it to see it) and is also a virtual attribute BUT only for 'staged' and 'deleted' users. It is a stored attribute for regular users and we should update modifytimestamp when it is set. thanks thierry OK, in that case it seems like we can special-case it. But do you know about any other attributes in any other LDAP servers? Any LDAP server following standard should provide modifyti
[SSSD] Re: [PATCH] SYSDB: Check changed virtual attributes before modified timestamp
On (28/07/16 10:24), thierry bordaz wrote: > > >On 07/28/2016 09:39 AM, Jakub Hrozek wrote: >> On Wed, Jul 27, 2016 at 04:09:07PM +0200, thierry bordaz wrote: >> > >> > On 07/27/2016 03:36 PM, Jakub Hrozek wrote: >> > > On Wed, Jul 27, 2016 at 02:55:37PM +0200, thierry bordaz wrote: >> > > > On 07/27/2016 01:56 PM, Jakub Hrozek wrote: >> > > > > On Wed, Jul 27, 2016 at 01:03:59PM +0200, Jakub Hrozek wrote: >> > > > > > On Wed, Jul 27, 2016 at 12:22:46PM +0200, Lukas Slebodnik wrote: >> > > > > > > On (27/07/16 12:08), Jakub Hrozek wrote: >> > > > > > > > On Wed, Jul 27, 2016 at 12:02:24PM +0200, Jakub Hrozek wrote: >> > > > > > > > > On Wed, Jul 27, 2016 at 11:54:16AM +0200, Lukas Slebodnik >> > > > > > > > > wrote: >> > > > > > > > > > ehlo, >> > > > > > > > > > >> > > > > > > > > > attached patch fixes acces denied after activating user in >> > > > > > > > > > 389ds. >> > > > > > > > > > Jakub had some comments/ideas in ticket but I think it's >> > > > > > > > > > better to discuss >> > > > > > > > > > about virtual attributes and timestamp cache on mailing >> > > > > > > > > > list. >> > > > > > > > > Yes, so the comment I have is that while this works, it >> > > > > > > > > might break some >> > > > > > > > > strange LDAP servers. >> > > > > > > > > >> > > > > > > > > We use modifyTimestamp as a 'positive' indicator that the >> > > > > > > > > entry has not >> > > > > > > > > changed -- if the modifyTimestamp didn't change, we consider >> > > > > > > > > the cached >> > > > > > > > > entry the same as what is on the server and only bump the >> > > > > > > > > timestamp >> > > > > > > > > cache. If the timestamp is different, we do a >> > > > > > > > > deep-comparison of cached >> > > > > > > > > attribute values with what is on the LDAP server and write >> > > > > > > > > the sysdb >> > > > > > > > > cache entry only if the attributes differ. >> > > > > > > > > >> > > > > > > > > I was wondering if we can use the modifyTimestamp at all, >> > > > > > > > > then, because >> > > > > > > > > even if it's the same, we might want to check the attributes >> > > > > > > > > to see if >> > > > > > > > > some of the values are different because some of the >> > > > > > > > > attributes might be >> > > > > > > > > this operational/virtual attribute.. >> > > > > > > > Sorry, sent too soon. >> > > > > > > > >> > > > > > > > I think the questions are -- 1) can we enumerate the virtual >> > > > > > > > attributes? >> > > > > > > That might be a question for 389-ds developers. >> > > > > > > But it's very likely it will be different on other LDAP servers. >> > > > > > > >> > > > > > > > 2) Would different LDAP servers have different virtual >> > > > > > > > attributes. >> > > > > > > > >> > > > > > > > For 2) maybe a possible solution might be to set a non-existing >> > > > > > > > modifyTimestamp attribute value, but I would consider that >> > > > > > > > only a >> > > > > > > > kludge, we shouldn't break existing setups.. >> > > > > > > I am not satisfied with this POC solution either. >> > > > > > > >> > > > > > > So should we remove usage of modifyTimestamp for detecting >> > > > > > > changes? >> > > > > > I would prefer to ask the DS developers before removing it >> > > > > > completely. >> > > > > > >> > > > > > At least for large groups it might take a long time to compare all >> > > > > > attribute >> > > > > > values and IIRC we don't depend on any virtual attributes for >> > > > > > groups. Maybe >> > > > > > we could parametrize that part of the code and enable the fast way >> > > > > > with >> > > > > > modifyTimestamps for 'known' server types, that is for setups with >> > > > > > AD and >> > > > > > IPA providers. >> > > > > > >> > > > > > For users, there is typically not as many attributes so we should >> > > > > > be >> > > > > > fine deep-comparing all attributes. >> > > > > I'm adding Thierry (so please reply-to-all to keep him in the >> > > > > thread). >> > > > > >> > > > > Thierry, in the latest sssd version we tried to add a performance >> > > > > improvement related to how we store SSSD entries in the cache. The >> > > > > short >> > > > > version is that we store the modifyTimestamp attribute in the cache >> > > > > and >> > > > > when we fetch an entry, we compare the entry modifyTimestamp with >> > > > > what >> > > > > is on the server. When the two are the same, we say that the entry >> > > > > did >> > > > > not change and don't update the cache. >> > > > > >> > > > > This works fine for most attributes, but not for attributes like >> > > > > nsAccountLock which do not change modifyTimestamp when they are >> > > > > modified. So when an entry was already cached but then nsAccountLock >> > > > > changed, we treated the entry as the same and never read the new >> > > > > nsAccountLock value. >> > > > > >> > > > > To fix this, I think we have several options: >> > > > >1) special-case the nsAccountLock. This seems a bit dangerous, >> > > > >
[SSSD] Re: [PATCH] SYSDB: Check changed virtual attributes before modified timestamp
On 07/28/2016 09:39 AM, Jakub Hrozek wrote: On Wed, Jul 27, 2016 at 04:09:07PM +0200, thierry bordaz wrote: On 07/27/2016 03:36 PM, Jakub Hrozek wrote: On Wed, Jul 27, 2016 at 02:55:37PM +0200, thierry bordaz wrote: On 07/27/2016 01:56 PM, Jakub Hrozek wrote: On Wed, Jul 27, 2016 at 01:03:59PM +0200, Jakub Hrozek wrote: On Wed, Jul 27, 2016 at 12:22:46PM +0200, Lukas Slebodnik wrote: On (27/07/16 12:08), Jakub Hrozek wrote: On Wed, Jul 27, 2016 at 12:02:24PM +0200, Jakub Hrozek wrote: On Wed, Jul 27, 2016 at 11:54:16AM +0200, Lukas Slebodnik wrote: ehlo, attached patch fixes acces denied after activating user in 389ds. Jakub had some comments/ideas in ticket but I think it's better to discuss about virtual attributes and timestamp cache on mailing list. Yes, so the comment I have is that while this works, it might break some strange LDAP servers. We use modifyTimestamp as a 'positive' indicator that the entry has not changed -- if the modifyTimestamp didn't change, we consider the cached entry the same as what is on the server and only bump the timestamp cache. If the timestamp is different, we do a deep-comparison of cached attribute values with what is on the LDAP server and write the sysdb cache entry only if the attributes differ. I was wondering if we can use the modifyTimestamp at all, then, because even if it's the same, we might want to check the attributes to see if some of the values are different because some of the attributes might be this operational/virtual attribute.. Sorry, sent too soon. I think the questions are -- 1) can we enumerate the virtual attributes? That might be a question for 389-ds developers. But it's very likely it will be different on other LDAP servers. 2) Would different LDAP servers have different virtual attributes. For 2) maybe a possible solution might be to set a non-existing modifyTimestamp attribute value, but I would consider that only a kludge, we shouldn't break existing setups.. I am not satisfied with this POC solution either. So should we remove usage of modifyTimestamp for detecting changes? I would prefer to ask the DS developers before removing it completely. At least for large groups it might take a long time to compare all attribute values and IIRC we don't depend on any virtual attributes for groups. Maybe we could parametrize that part of the code and enable the fast way with modifyTimestamps for 'known' server types, that is for setups with AD and IPA providers. For users, there is typically not as many attributes so we should be fine deep-comparing all attributes. I'm adding Thierry (so please reply-to-all to keep him in the thread). Thierry, in the latest sssd version we tried to add a performance improvement related to how we store SSSD entries in the cache. The short version is that we store the modifyTimestamp attribute in the cache and when we fetch an entry, we compare the entry modifyTimestamp with what is on the server. When the two are the same, we say that the entry did not change and don't update the cache. This works fine for most attributes, but not for attributes like nsAccountLock which do not change modifyTimestamp when they are modified. So when an entry was already cached but then nsAccountLock changed, we treated the entry as the same and never read the new nsAccountLock value. To fix this, I think we have several options: 1) special-case the nsAccountLock. This seems a bit dangerous, because I'm not sure we can say that some other attribute we are interested in behaves the same as nsAccountLock. 2) drop the modifyTimestamp optimization completely. Then we fall back to comparing the attribute values, which might work, but for huge objects like groups with thousands of members, this might be too expensive. 3) only use the modifyTimestamp optimization for cases where we know we don't read any virtual attributes. And my question is -- can we, in general, know if the modifyTimestamp way of detecting changes is realiable for all LDAP servers? Or do you think it should only be used for cases where we know we are not interested in any virtual attributes (that would mostly be storing groups from servers where we know exactly what is on the server side, like IPA or AD). Hello, Relying on modifytimestamp looks a good idea. Any MOD/MODRDN will update it, except I think it is unchanged when updating some password policy attributes. Regarding virtual attribute, the only one I know in IPA is nsaccountlock. nsaccountlock is an operational attribute (you need to request it to see it) and is also a virtual attribute BUT only for 'staged' and 'deleted' users. It is a stored attribute for regular users and we should update modifytimestamp when it is set. thanks thierry OK, in that case it seems like we can special-case it. But do you know about any other attributes in any other LDAP servers? Any LDAP server following standard should provide modifyti
[SSSD] Re: [PATCH] SYSDB: Check changed virtual attributes before modified timestamp
On Wed, Jul 27, 2016 at 04:09:07PM +0200, thierry bordaz wrote: > > > On 07/27/2016 03:36 PM, Jakub Hrozek wrote: > > On Wed, Jul 27, 2016 at 02:55:37PM +0200, thierry bordaz wrote: > > > > > > On 07/27/2016 01:56 PM, Jakub Hrozek wrote: > > > > On Wed, Jul 27, 2016 at 01:03:59PM +0200, Jakub Hrozek wrote: > > > > > On Wed, Jul 27, 2016 at 12:22:46PM +0200, Lukas Slebodnik wrote: > > > > > > On (27/07/16 12:08), Jakub Hrozek wrote: > > > > > > > On Wed, Jul 27, 2016 at 12:02:24PM +0200, Jakub Hrozek wrote: > > > > > > > > On Wed, Jul 27, 2016 at 11:54:16AM +0200, Lukas Slebodnik wrote: > > > > > > > > > ehlo, > > > > > > > > > > > > > > > > > > attached patch fixes acces denied after activating user in > > > > > > > > > 389ds. > > > > > > > > > Jakub had some comments/ideas in ticket but I think it's > > > > > > > > > better to discuss > > > > > > > > > about virtual attributes and timestamp cache on mailing list. > > > > > > > > Yes, so the comment I have is that while this works, it might > > > > > > > > break some > > > > > > > > strange LDAP servers. > > > > > > > > > > > > > > > > We use modifyTimestamp as a 'positive' indicator that the entry > > > > > > > > has not > > > > > > > > changed -- if the modifyTimestamp didn't change, we consider > > > > > > > > the cached > > > > > > > > entry the same as what is on the server and only bump the > > > > > > > > timestamp > > > > > > > > cache. If the timestamp is different, we do a deep-comparison > > > > > > > > of cached > > > > > > > > attribute values with what is on the LDAP server and write the > > > > > > > > sysdb > > > > > > > > cache entry only if the attributes differ. > > > > > > > > > > > > > > > > I was wondering if we can use the modifyTimestamp at all, then, > > > > > > > > because > > > > > > > > even if it's the same, we might want to check the attributes to > > > > > > > > see if > > > > > > > > some of the values are different because some of the attributes > > > > > > > > might be > > > > > > > > this operational/virtual attribute.. > > > > > > > Sorry, sent too soon. > > > > > > > > > > > > > > I think the questions are -- 1) can we enumerate the virtual > > > > > > > attributes? > > > > > > That might be a question for 389-ds developers. > > > > > > But it's very likely it will be different on other LDAP servers. > > > > > > > > > > > > > 2) Would different LDAP servers have different virtual attributes. > > > > > > > > > > > > > > For 2) maybe a possible solution might be to set a non-existing > > > > > > > modifyTimestamp attribute value, but I would consider that only a > > > > > > > kludge, we shouldn't break existing setups.. > > > > > > I am not satisfied with this POC solution either. > > > > > > > > > > > > So should we remove usage of modifyTimestamp for detecting changes? > > > > > I would prefer to ask the DS developers before removing it completely. > > > > > > > > > > At least for large groups it might take a long time to compare all > > > > > attribute > > > > > values and IIRC we don't depend on any virtual attributes for groups. > > > > > Maybe > > > > > we could parametrize that part of the code and enable the fast way > > > > > with > > > > > modifyTimestamps for 'known' server types, that is for setups with AD > > > > > and > > > > > IPA providers. > > > > > > > > > > For users, there is typically not as many attributes so we should be > > > > > fine deep-comparing all attributes. > > > > I'm adding Thierry (so please reply-to-all to keep him in the thread). > > > > > > > > Thierry, in the latest sssd version we tried to add a performance > > > > improvement related to how we store SSSD entries in the cache. The short > > > > version is that we store the modifyTimestamp attribute in the cache and > > > > when we fetch an entry, we compare the entry modifyTimestamp with what > > > > is on the server. When the two are the same, we say that the entry did > > > > not change and don't update the cache. > > > > > > > > This works fine for most attributes, but not for attributes like > > > > nsAccountLock which do not change modifyTimestamp when they are > > > > modified. So when an entry was already cached but then nsAccountLock > > > > changed, we treated the entry as the same and never read the new > > > > nsAccountLock value. > > > > > > > > To fix this, I think we have several options: > > > > 1) special-case the nsAccountLock. This seems a bit dangerous, > > > > because I'm not sure we can say that some other attribute we are > > > > interested in behaves the same as nsAccountLock. > > > > 2) drop the modifyTimestamp optimization completely. Then we fall > > > > back to comparing the attribute values, which might work, but for > > > > huge objects like groups with thousands of members, this might be > > > > too expensive. > > > > 3) only use the modifyTimestamp optimization for cases where we > > > > know > > > > we don't read any
[SSSD] Re: [PATCH] SYSDB: Check changed virtual attributes before modified timestamp
On 07/27/2016 03:36 PM, Jakub Hrozek wrote: On Wed, Jul 27, 2016 at 02:55:37PM +0200, thierry bordaz wrote: On 07/27/2016 01:56 PM, Jakub Hrozek wrote: On Wed, Jul 27, 2016 at 01:03:59PM +0200, Jakub Hrozek wrote: On Wed, Jul 27, 2016 at 12:22:46PM +0200, Lukas Slebodnik wrote: On (27/07/16 12:08), Jakub Hrozek wrote: On Wed, Jul 27, 2016 at 12:02:24PM +0200, Jakub Hrozek wrote: On Wed, Jul 27, 2016 at 11:54:16AM +0200, Lukas Slebodnik wrote: ehlo, attached patch fixes acces denied after activating user in 389ds. Jakub had some comments/ideas in ticket but I think it's better to discuss about virtual attributes and timestamp cache on mailing list. Yes, so the comment I have is that while this works, it might break some strange LDAP servers. We use modifyTimestamp as a 'positive' indicator that the entry has not changed -- if the modifyTimestamp didn't change, we consider the cached entry the same as what is on the server and only bump the timestamp cache. If the timestamp is different, we do a deep-comparison of cached attribute values with what is on the LDAP server and write the sysdb cache entry only if the attributes differ. I was wondering if we can use the modifyTimestamp at all, then, because even if it's the same, we might want to check the attributes to see if some of the values are different because some of the attributes might be this operational/virtual attribute.. Sorry, sent too soon. I think the questions are -- 1) can we enumerate the virtual attributes? That might be a question for 389-ds developers. But it's very likely it will be different on other LDAP servers. 2) Would different LDAP servers have different virtual attributes. For 2) maybe a possible solution might be to set a non-existing modifyTimestamp attribute value, but I would consider that only a kludge, we shouldn't break existing setups.. I am not satisfied with this POC solution either. So should we remove usage of modifyTimestamp for detecting changes? I would prefer to ask the DS developers before removing it completely. At least for large groups it might take a long time to compare all attribute values and IIRC we don't depend on any virtual attributes for groups. Maybe we could parametrize that part of the code and enable the fast way with modifyTimestamps for 'known' server types, that is for setups with AD and IPA providers. For users, there is typically not as many attributes so we should be fine deep-comparing all attributes. I'm adding Thierry (so please reply-to-all to keep him in the thread). Thierry, in the latest sssd version we tried to add a performance improvement related to how we store SSSD entries in the cache. The short version is that we store the modifyTimestamp attribute in the cache and when we fetch an entry, we compare the entry modifyTimestamp with what is on the server. When the two are the same, we say that the entry did not change and don't update the cache. This works fine for most attributes, but not for attributes like nsAccountLock which do not change modifyTimestamp when they are modified. So when an entry was already cached but then nsAccountLock changed, we treated the entry as the same and never read the new nsAccountLock value. To fix this, I think we have several options: 1) special-case the nsAccountLock. This seems a bit dangerous, because I'm not sure we can say that some other attribute we are interested in behaves the same as nsAccountLock. 2) drop the modifyTimestamp optimization completely. Then we fall back to comparing the attribute values, which might work, but for huge objects like groups with thousands of members, this might be too expensive. 3) only use the modifyTimestamp optimization for cases where we know we don't read any virtual attributes. And my question is -- can we, in general, know if the modifyTimestamp way of detecting changes is realiable for all LDAP servers? Or do you think it should only be used for cases where we know we are not interested in any virtual attributes (that would mostly be storing groups from servers where we know exactly what is on the server side, like IPA or AD). Hello, Relying on modifytimestamp looks a good idea. Any MOD/MODRDN will update it, except I think it is unchanged when updating some password policy attributes. Regarding virtual attribute, the only one I know in IPA is nsaccountlock. nsaccountlock is an operational attribute (you need to request it to see it) and is also a virtual attribute BUT only for 'staged' and 'deleted' users. It is a stored attribute for regular users and we should update modifytimestamp when it is set. thanks thierry OK, in that case it seems like we can special-case it. But do you know about any other attributes in any other LDAP servers? Any LDAP server following standard should provide modifytimestamp that reflect the last update of the entry. Now virtual attribute values may be "attached" to the entry and its
[SSSD] Re: [PATCH] SYSDB: Check changed virtual attributes before modified timestamp
On 07/27/2016 01:56 PM, Jakub Hrozek wrote: On Wed, Jul 27, 2016 at 01:03:59PM +0200, Jakub Hrozek wrote: On Wed, Jul 27, 2016 at 12:22:46PM +0200, Lukas Slebodnik wrote: On (27/07/16 12:08), Jakub Hrozek wrote: On Wed, Jul 27, 2016 at 12:02:24PM +0200, Jakub Hrozek wrote: On Wed, Jul 27, 2016 at 11:54:16AM +0200, Lukas Slebodnik wrote: ehlo, attached patch fixes acces denied after activating user in 389ds. Jakub had some comments/ideas in ticket but I think it's better to discuss about virtual attributes and timestamp cache on mailing list. Yes, so the comment I have is that while this works, it might break some strange LDAP servers. We use modifyTimestamp as a 'positive' indicator that the entry has not changed -- if the modifyTimestamp didn't change, we consider the cached entry the same as what is on the server and only bump the timestamp cache. If the timestamp is different, we do a deep-comparison of cached attribute values with what is on the LDAP server and write the sysdb cache entry only if the attributes differ. I was wondering if we can use the modifyTimestamp at all, then, because even if it's the same, we might want to check the attributes to see if some of the values are different because some of the attributes might be this operational/virtual attribute.. Sorry, sent too soon. I think the questions are -- 1) can we enumerate the virtual attributes? That might be a question for 389-ds developers. But it's very likely it will be different on other LDAP servers. 2) Would different LDAP servers have different virtual attributes. For 2) maybe a possible solution might be to set a non-existing modifyTimestamp attribute value, but I would consider that only a kludge, we shouldn't break existing setups.. I am not satisfied with this POC solution either. So should we remove usage of modifyTimestamp for detecting changes? I would prefer to ask the DS developers before removing it completely. At least for large groups it might take a long time to compare all attribute values and IIRC we don't depend on any virtual attributes for groups. Maybe we could parametrize that part of the code and enable the fast way with modifyTimestamps for 'known' server types, that is for setups with AD and IPA providers. For users, there is typically not as many attributes so we should be fine deep-comparing all attributes. I'm adding Thierry (so please reply-to-all to keep him in the thread). Thierry, in the latest sssd version we tried to add a performance improvement related to how we store SSSD entries in the cache. The short version is that we store the modifyTimestamp attribute in the cache and when we fetch an entry, we compare the entry modifyTimestamp with what is on the server. When the two are the same, we say that the entry did not change and don't update the cache. This works fine for most attributes, but not for attributes like nsAccountLock which do not change modifyTimestamp when they are modified. So when an entry was already cached but then nsAccountLock changed, we treated the entry as the same and never read the new nsAccountLock value. To fix this, I think we have several options: 1) special-case the nsAccountLock. This seems a bit dangerous, because I'm not sure we can say that some other attribute we are interested in behaves the same as nsAccountLock. 2) drop the modifyTimestamp optimization completely. Then we fall back to comparing the attribute values, which might work, but for huge objects like groups with thousands of members, this might be too expensive. 3) only use the modifyTimestamp optimization for cases where we know we don't read any virtual attributes. And my question is -- can we, in general, know if the modifyTimestamp way of detecting changes is realiable for all LDAP servers? Or do you think it should only be used for cases where we know we are not interested in any virtual attributes (that would mostly be storing groups from servers where we know exactly what is on the server side, like IPA or AD). Hello, Relying on modifytimestamp looks a good idea. Any MOD/MODRDN will update it, except I think it is unchanged when updating some password policy attributes. Regarding virtual attribute, the only one I know in IPA is nsaccountlock. nsaccountlock is an operational attribute (you need to request it to see it) and is also a virtual attribute BUT only for 'staged' and 'deleted' users. It is a stored attribute for regular users and we should update modifytimestamp when it is set. thanks thierry ___ sssd-devel mailing list sssd-devel@lists.fedorahosted.org https://lists.fedorahosted.org/admin/lists/sssd-devel@lists.fedorahosted.org
[SSSD] Re: [PATCH] SYSDB: Check changed virtual attributes before modified timestamp
On Wed, Jul 27, 2016 at 02:55:37PM +0200, thierry bordaz wrote: > > > On 07/27/2016 01:56 PM, Jakub Hrozek wrote: > > On Wed, Jul 27, 2016 at 01:03:59PM +0200, Jakub Hrozek wrote: > > > On Wed, Jul 27, 2016 at 12:22:46PM +0200, Lukas Slebodnik wrote: > > > > On (27/07/16 12:08), Jakub Hrozek wrote: > > > > > On Wed, Jul 27, 2016 at 12:02:24PM +0200, Jakub Hrozek wrote: > > > > > > On Wed, Jul 27, 2016 at 11:54:16AM +0200, Lukas Slebodnik wrote: > > > > > > > ehlo, > > > > > > > > > > > > > > attached patch fixes acces denied after activating user in 389ds. > > > > > > > Jakub had some comments/ideas in ticket but I think it's better > > > > > > > to discuss > > > > > > > about virtual attributes and timestamp cache on mailing list. > > > > > > Yes, so the comment I have is that while this works, it might break > > > > > > some > > > > > > strange LDAP servers. > > > > > > > > > > > > We use modifyTimestamp as a 'positive' indicator that the entry has > > > > > > not > > > > > > changed -- if the modifyTimestamp didn't change, we consider the > > > > > > cached > > > > > > entry the same as what is on the server and only bump the timestamp > > > > > > cache. If the timestamp is different, we do a deep-comparison of > > > > > > cached > > > > > > attribute values with what is on the LDAP server and write the sysdb > > > > > > cache entry only if the attributes differ. > > > > > > > > > > > > I was wondering if we can use the modifyTimestamp at all, then, > > > > > > because > > > > > > even if it's the same, we might want to check the attributes to see > > > > > > if > > > > > > some of the values are different because some of the attributes > > > > > > might be > > > > > > this operational/virtual attribute.. > > > > > Sorry, sent too soon. > > > > > > > > > > I think the questions are -- 1) can we enumerate the virtual > > > > > attributes? > > > > That might be a question for 389-ds developers. > > > > But it's very likely it will be different on other LDAP servers. > > > > > > > > > 2) Would different LDAP servers have different virtual attributes. > > > > > > > > > > For 2) maybe a possible solution might be to set a non-existing > > > > > modifyTimestamp attribute value, but I would consider that only a > > > > > kludge, we shouldn't break existing setups.. > > > > I am not satisfied with this POC solution either. > > > > > > > > So should we remove usage of modifyTimestamp for detecting changes? > > > I would prefer to ask the DS developers before removing it completely. > > > > > > At least for large groups it might take a long time to compare all > > > attribute > > > values and IIRC we don't depend on any virtual attributes for groups. > > > Maybe > > > we could parametrize that part of the code and enable the fast way with > > > modifyTimestamps for 'known' server types, that is for setups with AD and > > > IPA providers. > > > > > > For users, there is typically not as many attributes so we should be > > > fine deep-comparing all attributes. > > I'm adding Thierry (so please reply-to-all to keep him in the thread). > > > > Thierry, in the latest sssd version we tried to add a performance > > improvement related to how we store SSSD entries in the cache. The short > > version is that we store the modifyTimestamp attribute in the cache and > > when we fetch an entry, we compare the entry modifyTimestamp with what > > is on the server. When the two are the same, we say that the entry did > > not change and don't update the cache. > > > > This works fine for most attributes, but not for attributes like > > nsAccountLock which do not change modifyTimestamp when they are > > modified. So when an entry was already cached but then nsAccountLock > > changed, we treated the entry as the same and never read the new > > nsAccountLock value. > > > > To fix this, I think we have several options: > > 1) special-case the nsAccountLock. This seems a bit dangerous, > > because I'm not sure we can say that some other attribute we are > > interested in behaves the same as nsAccountLock. > > 2) drop the modifyTimestamp optimization completely. Then we fall > > back to comparing the attribute values, which might work, but for > > huge objects like groups with thousands of members, this might be > > too expensive. > > 3) only use the modifyTimestamp optimization for cases where we know > > we don't read any virtual attributes. > > > > And my question is -- can we, in general, know if the modifyTimestamp > > way of detecting changes is realiable for all LDAP servers? Or do you > > think it should only be used for cases where we know we are not > > interested in any virtual attributes (that would mostly be storing > > groups from servers where we know exactly what is on the server side, > > like IPA or AD). > > Hello, > > Relying on modifytimestamp looks a good idea. Any MOD/MODRDN will update it, > except I think it is unchanged when updating some
[SSSD] Re: [PATCH] SYSDB: Check changed virtual attributes before modified timestamp
On Wed, Jul 27, 2016 at 01:03:59PM +0200, Jakub Hrozek wrote: > On Wed, Jul 27, 2016 at 12:22:46PM +0200, Lukas Slebodnik wrote: > > On (27/07/16 12:08), Jakub Hrozek wrote: > > >On Wed, Jul 27, 2016 at 12:02:24PM +0200, Jakub Hrozek wrote: > > >> On Wed, Jul 27, 2016 at 11:54:16AM +0200, Lukas Slebodnik wrote: > > >> > ehlo, > > >> > > > >> > attached patch fixes acces denied after activating user in 389ds. > > >> > Jakub had some comments/ideas in ticket but I think it's better to > > >> > discuss > > >> > about virtual attributes and timestamp cache on mailing list. > > >> > > >> Yes, so the comment I have is that while this works, it might break some > > >> strange LDAP servers. > > >> > > >> We use modifyTimestamp as a 'positive' indicator that the entry has not > > >> changed -- if the modifyTimestamp didn't change, we consider the cached > > >> entry the same as what is on the server and only bump the timestamp > > >> cache. If the timestamp is different, we do a deep-comparison of cached > > >> attribute values with what is on the LDAP server and write the sysdb > > >> cache entry only if the attributes differ. > > >> > > >> I was wondering if we can use the modifyTimestamp at all, then, because > > >> even if it's the same, we might want to check the attributes to see if > > >> some of the values are different because some of the attributes might be > > >> this operational/virtual attribute.. > > > > > >Sorry, sent too soon. > > > > > >I think the questions are -- 1) can we enumerate the virtual attributes? > > That might be a question for 389-ds developers. > > But it's very likely it will be different on other LDAP servers. > > > > >2) Would different LDAP servers have different virtual attributes. > > > > > >For 2) maybe a possible solution might be to set a non-existing > > >modifyTimestamp attribute value, but I would consider that only a > > >kludge, we shouldn't break existing setups.. > > > > I am not satisfied with this POC solution either. > > > > So should we remove usage of modifyTimestamp for detecting changes? > > I would prefer to ask the DS developers before removing it completely. > > At least for large groups it might take a long time to compare all attribute > values and IIRC we don't depend on any virtual attributes for groups. Maybe > we could parametrize that part of the code and enable the fast way with > modifyTimestamps for 'known' server types, that is for setups with AD and > IPA providers. > > For users, there is typically not as many attributes so we should be > fine deep-comparing all attributes. I'm adding Thierry (so please reply-to-all to keep him in the thread). Thierry, in the latest sssd version we tried to add a performance improvement related to how we store SSSD entries in the cache. The short version is that we store the modifyTimestamp attribute in the cache and when we fetch an entry, we compare the entry modifyTimestamp with what is on the server. When the two are the same, we say that the entry did not change and don't update the cache. This works fine for most attributes, but not for attributes like nsAccountLock which do not change modifyTimestamp when they are modified. So when an entry was already cached but then nsAccountLock changed, we treated the entry as the same and never read the new nsAccountLock value. To fix this, I think we have several options: 1) special-case the nsAccountLock. This seems a bit dangerous, because I'm not sure we can say that some other attribute we are interested in behaves the same as nsAccountLock. 2) drop the modifyTimestamp optimization completely. Then we fall back to comparing the attribute values, which might work, but for huge objects like groups with thousands of members, this might be too expensive. 3) only use the modifyTimestamp optimization for cases where we know we don't read any virtual attributes. And my question is -- can we, in general, know if the modifyTimestamp way of detecting changes is realiable for all LDAP servers? Or do you think it should only be used for cases where we know we are not interested in any virtual attributes (that would mostly be storing groups from servers where we know exactly what is on the server side, like IPA or AD). ___ sssd-devel mailing list sssd-devel@lists.fedorahosted.org https://lists.fedorahosted.org/admin/lists/sssd-devel@lists.fedorahosted.org
[SSSD] Re: [PATCH] SYSDB: Check changed virtual attributes before modified timestamp
On Wed, Jul 27, 2016 at 12:22:46PM +0200, Lukas Slebodnik wrote: > On (27/07/16 12:08), Jakub Hrozek wrote: > >On Wed, Jul 27, 2016 at 12:02:24PM +0200, Jakub Hrozek wrote: > >> On Wed, Jul 27, 2016 at 11:54:16AM +0200, Lukas Slebodnik wrote: > >> > ehlo, > >> > > >> > attached patch fixes acces denied after activating user in 389ds. > >> > Jakub had some comments/ideas in ticket but I think it's better to > >> > discuss > >> > about virtual attributes and timestamp cache on mailing list. > >> > >> Yes, so the comment I have is that while this works, it might break some > >> strange LDAP servers. > >> > >> We use modifyTimestamp as a 'positive' indicator that the entry has not > >> changed -- if the modifyTimestamp didn't change, we consider the cached > >> entry the same as what is on the server and only bump the timestamp > >> cache. If the timestamp is different, we do a deep-comparison of cached > >> attribute values with what is on the LDAP server and write the sysdb > >> cache entry only if the attributes differ. > >> > >> I was wondering if we can use the modifyTimestamp at all, then, because > >> even if it's the same, we might want to check the attributes to see if > >> some of the values are different because some of the attributes might be > >> this operational/virtual attribute.. > > > >Sorry, sent too soon. > > > >I think the questions are -- 1) can we enumerate the virtual attributes? > That might be a question for 389-ds developers. > But it's very likely it will be different on other LDAP servers. > > >2) Would different LDAP servers have different virtual attributes. > > > >For 2) maybe a possible solution might be to set a non-existing > >modifyTimestamp attribute value, but I would consider that only a > >kludge, we shouldn't break existing setups.. > > I am not satisfied with this POC solution either. > > So should we remove usage of modifyTimestamp for detecting changes? I would prefer to ask the DS developers before removing it completely. At least for large groups it might take a long time to compare all attribute values and IIRC we don't depend on any virtual attributes for groups. Maybe we could parametrize that part of the code and enable the fast way with modifyTimestamps for 'known' server types, that is for setups with AD and IPA providers. For users, there is typically not as many attributes so we should be fine deep-comparing all attributes. ___ sssd-devel mailing list sssd-devel@lists.fedorahosted.org https://lists.fedorahosted.org/admin/lists/sssd-devel@lists.fedorahosted.org
[SSSD] Re: [PATCH] SYSDB: Check changed virtual attributes before modified timestamp
On (27/07/16 12:08), Jakub Hrozek wrote: >On Wed, Jul 27, 2016 at 12:02:24PM +0200, Jakub Hrozek wrote: >> On Wed, Jul 27, 2016 at 11:54:16AM +0200, Lukas Slebodnik wrote: >> > ehlo, >> > >> > attached patch fixes acces denied after activating user in 389ds. >> > Jakub had some comments/ideas in ticket but I think it's better to discuss >> > about virtual attributes and timestamp cache on mailing list. >> >> Yes, so the comment I have is that while this works, it might break some >> strange LDAP servers. >> >> We use modifyTimestamp as a 'positive' indicator that the entry has not >> changed -- if the modifyTimestamp didn't change, we consider the cached >> entry the same as what is on the server and only bump the timestamp >> cache. If the timestamp is different, we do a deep-comparison of cached >> attribute values with what is on the LDAP server and write the sysdb >> cache entry only if the attributes differ. >> >> I was wondering if we can use the modifyTimestamp at all, then, because >> even if it's the same, we might want to check the attributes to see if >> some of the values are different because some of the attributes might be >> this operational/virtual attribute.. > >Sorry, sent too soon. > >I think the questions are -- 1) can we enumerate the virtual attributes? That might be a question for 389-ds developers. But it's very likely it will be different on other LDAP servers. >2) Would different LDAP servers have different virtual attributes. > >For 2) maybe a possible solution might be to set a non-existing >modifyTimestamp attribute value, but I would consider that only a >kludge, we shouldn't break existing setups.. I am not satisfied with this POC solution either. So should we remove usage of modifyTimestamp for detecting changes? LS ___ sssd-devel mailing list sssd-devel@lists.fedorahosted.org https://lists.fedorahosted.org/admin/lists/sssd-devel@lists.fedorahosted.org
[SSSD] Re: [PATCH] SYSDB: Check changed virtual attributes before modified timestamp
On Wed, Jul 27, 2016 at 12:02:24PM +0200, Jakub Hrozek wrote: > On Wed, Jul 27, 2016 at 11:54:16AM +0200, Lukas Slebodnik wrote: > > ehlo, > > > > attached patch fixes acces denied after activating user in 389ds. > > Jakub had some comments/ideas in ticket but I think it's better to discuss > > about virtual attributes and timestamp cache on mailing list. > > Yes, so the comment I have is that while this works, it might break some > strange LDAP servers. > > We use modifyTimestamp as a 'positive' indicator that the entry has not > changed -- if the modifyTimestamp didn't change, we consider the cached > entry the same as what is on the server and only bump the timestamp > cache. If the timestamp is different, we do a deep-comparison of cached > attribute values with what is on the LDAP server and write the sysdb > cache entry only if the attributes differ. > > I was wondering if we can use the modifyTimestamp at all, then, because > even if it's the same, we might want to check the attributes to see if > some of the values are different because some of the attributes might be > this operational/virtual attribute.. Sorry, sent too soon. I think the questions are -- 1) can we enumerate the virtual attributes? 2) Would different LDAP servers have different virtual attributes. For 2) maybe a possible solution might be to set a non-existing modifyTimestamp attribute value, but I would consider that only a kludge, we shouldn't break existing setups.. ___ sssd-devel mailing list sssd-devel@lists.fedorahosted.org https://lists.fedorahosted.org/admin/lists/sssd-devel@lists.fedorahosted.org
[SSSD] Re: [PATCH] SYSDB: Check changed virtual attributes before modified timestamp
On Wed, Jul 27, 2016 at 11:54:16AM +0200, Lukas Slebodnik wrote: > ehlo, > > attached patch fixes acces denied after activating user in 389ds. > Jakub had some comments/ideas in ticket but I think it's better to discuss > about virtual attributes and timestamp cache on mailing list. Yes, so the comment I have is that while this works, it might break some strange LDAP servers. We use modifyTimestamp as a 'positive' indicator that the entry has not changed -- if the modifyTimestamp didn't change, we consider the cached entry the same as what is on the server and only bump the timestamp cache. If the timestamp is different, we do a deep-comparison of cached attribute values with what is on the LDAP server and write the sysdb cache entry only if the attributes differ. I was wondering if we can use the modifyTimestamp at all, then, because even if it's the same, we might want to check the attributes to see if some of the values are different because some of the attributes might be this operational/virtual attribute.. ___ sssd-devel mailing list sssd-devel@lists.fedorahosted.org https://lists.fedorahosted.org/admin/lists/sssd-devel@lists.fedorahosted.org
