----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/42456/#review114998 -----------------------------------------------------------
Ship it! Ship It! - Dmitro Lisnichenko On Jan. 18, 2016, 2:15 p.m., Andrew Onischuk wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/42456/ > ----------------------------------------------------------- > > (Updated Jan. 18, 2016, 2:15 p.m.) > > > Review request for Ambari and Dmitro Lisnichenko. > > > Bugs: AMBARI-14708 > https://issues.apache.org/jira/browse/AMBARI-14708 > > > Repository: ambari > > > Description > ------- > > When performing a restart of a large cluster where LDAP is being used > indirectly by nslcd, the LDAP servers are put under heavy load. This is more > evident in LDAP organizations that are large to begin with. > > connection from pid=12345 uid=0 gid=0 > nslcd_group_all() > myldap_search(base="cn=groups,cn=accounts,dc=corp,dc=local", > filter="(objectClass=posixGroup)") > ldap_result(): end of results > > > > > > It turns out that these processes are the before-ANY hook script which > runs when a service is started, like this one I was running locally to > reproduce the query patterns. > > > > /usr/bin/python2.6 /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/before- > ANY/scripts/hook.py ANY /var/lib/ambari-agent/data/command-5950.json /var/lib > /ambari-agent/cache/stacks/HDP/2.0.6/hooks/before-ANY /var/lib/ambari- > agent/data/structured-out-5950.json INFO /var/lib/ambari-agent/data/tmp > > > > > > I tracked the issue down to this function in > {{resource_management/core/providers/accounts.py}}: > > > > @property > def user_groups(self): > return [g.gr_name for g in grp.getgrall() if self.resource.username in g.gr_me > > > > > > This property actually gets referenced at least 2 times for each user. > The call to {{grp.getgrall()}} forces a complete enumeration of groups every > time. > > What this means is for a cluster with many nodes with many processes > restarting across those nodes you are going to have many of these full > enumeration searches running at the same time. In an enterprise with a large > directory this will get very expensive, especially since this type of call is > not cached by nscd. > > I'm aware that the idiom used here to get the groups is common in python > but it's actually pretty inefficient. Commands like id and groups have more > efficient ways of discovering this. I'm not aware of the equivalent of these > in Python. > > > > @property > def user_groups(self): > ret = [] > (rc, output) = shell.checked_call(['groups', self.resource.username](https://h > sudo=True) > if rc == 0: > ret.extend(output.split(':')[1]( > ).lstrip().split()) > return ret > > This converts the full LDAP scan for groups to more efficient queries targeted > to the user. The lookups done by the groups command are also 100% cacheable. > Since it's a checked call the `rc == 0` check is probably not needed. > > An unfortunate effect of how usermod and friends work is that it always > invalidates the nscd cache after it's run. This means that Ambari could still > be a lot more efficient than it is when LDAP is in play by being pickier about > when it runs commands like useradd/usermod/groupadd/groupmod. > > We can also probably put a timed cache on the results from `grp.getgrall()` or > `groups` in memory, configurable by the agent config file. This way, we would > only call it once every hour or so. > > > Diffs > ----- > > > ambari-common/src/main/python/resource_management/core/providers/accounts.py > e7ef399 > > ambari-common/src/main/python/resource_management/core/resources/accounts.py > a75b723 > ambari-common/src/main/python/resource_management/core/system.py 228474b > ambari-common/src/main/python/resource_management/core/utils.py 247f068 > > ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml > 805aa29 > > ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-ANY/scripts/params.py > 034415a > > ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-ANY/scripts/shared_initialization.py > 10777f9 > ambari-server/src/test/python/stacks/2.0.6/configs/default.json bc40657 > > ambari-server/src/test/python/stacks/2.0.6/hooks/before-ANY/test_before_any.py > 3da58cd > > Diff: https://reviews.apache.org/r/42456/diff/ > > > Testing > ------- > > mvn clean test > > > Thanks, > > Andrew Onischuk > >
