Hi all, This is a follow-up to Clint Byrum's suggestion to add the `Map` intrinsic function[0], Zane Bitter's response[1] and Randall Burt's addendum[2].
Sorry for bringing it up again, but I'd love to reach consensus on this. The summary of the previous conversation: 1. TripleO is using some functionality currently not supported by Heat around scaled-out resources 2. Clint proposed a `map` intrinsic function that would solve it 3. Zane said Heat have historically been against a for-loop functionality 4. Randall suggested ResourceGroup's attribute passthrough may do what we need I've looked at the ResourceGroup code and experimented a bit. It does do some of what TripleO needs but not all. Here's what we're doing with our scaled-out resources (what we'd like to wrap in a ResourceGroup or similar in the future): 1. Building a coma-separated list of RabbitMQ nodes: https://github.com/openstack/tripleo-heat-templates/blob/a7f2a2c928e9c78a18defb68feb40da8c7eb95d6/overcloud-source.yaml#L642 This one is easy with ResourceGroup's inner attribute support: list_join: - ", " - {get_attr: [controller_group, name]} (controller_group is a ResourceGroup of Nova servers) 2. Get the name of the first Controller node: https://github.com/openstack/tripleo-heat-templates/blob/a7f2a2c928e9c78a18defb68feb40da8c7eb95d6/overcloud-source.yaml#L339 Possible today: {get_attr: [controller_group, resource.0.name]} 3. List of IP addresses of all controllers: https://github.com/openstack/tripleo-heat-templates/blob/a7f2a2c928e9c78a18defb68feb40da8c7eb95d6/overcloud-source.yaml#L405 We cannot do this, because resource group doesn't support extended attributes. Would need something like: {get_attr: [controller_group, networks, ctlplane, 0]} (ctlplane is the network controller_group servers are on) 4. IP address of the first node in the resource group: https://github.com/openstack/tripleo-heat-templates/blob/a7f2a2c928e9c78a18defb68feb40da8c7eb95d6/swift-deploy.yaml#L29 Can't do: extended attributes are not supported for the n-th node for the group either. This can be solved by `get_resource` working with resource IDs: get_attr: - {get_attr: [controller_group, resource.0]} - [networks, ctlplane, 0] (i.e. we get the server's ID from the ResourceGroup and change `get_attr` to work with the ID's too. Would also work if `get_resource` understood IDs). Alternatively, we could extend the ResourceGroup's get_attr behaviour: {get_attr: [controller_group, resource.0.networks.ctlplane.0]} but the former is a bit cleaner and more generic. --- That was the easy stuff, where we can get by with the current functionality (plus a few fixes). What follows are examples that really need new intrinsic functions (or seriously complicating the ResourceGroup attribute code and syntax). 5. Building a list of {ip: ..., name: ...} dictionaries to configure haproxy: https://github.com/openstack/tripleo-heat-templates/blob/a7f2a2c928e9c78a18defb68feb40da8c7eb95d6/overcloud-source.yaml#L478 This really calls for a mapping/for-each kind of functionality. Trying to invent a ResourceGroup syntax for this would be perverse. Here's what it could look like under Clint's `map` proposal: map: - ip: {get_attr: [{get_resource: "$1"}, networks, ctlplane, 0] name: {get_attr: [{get_resource: "$1"}, name]} - {get_attr: [compute_group, refs]} (this relies on `get_resource` working with resource IDs. Alternatively, we could have a `resources` attribute for ResourceGroup that returns objects that can be used with get_attr. 6. Building the /etc/hosts file https://github.com/openstack/tripleo-heat-templates/blob/a7f2a2c928e9c78a18defb68feb40da8c7eb95d6/overcloud-source.yaml#L585 Same as above, but also joining two lists together. We can use nested {list_join: ["\n", [...]} just as we're doing now, but having a `concat_list` function would make this and some other cases shorter and clearer. 7. Building the list of Swift devices: https://github.com/openstack/tripleo-heat-templates/blob/a7f2a2c928e9c78a18defb68feb40da8c7eb95d6/swift-deploy.yaml#L23 In addition to the abowe, we're adding a single element at the beginning of a list. Asking for a `cons` support is pushing it, right? ;-) We could just wrap that in a list and use `concat_list` or keep using nested `list_join`s as in the /etc/hosts case. ---- So this boils down to 4 features proposals: 1. Support extended attributes in ResourceGroup's members 2. Allow a way to use a Resource ID (e.g. what you get by {get_attr: [ResourceGroup, refs]} or {get_attr: [ResourceGroup, resource.0]}) with existing intrinsic functions (get_resource, get_attr) 3. A `map` intrinsic function that turns a list of items to another list by doing operations on each item 4. A `concat_list` intrinsic function that joins multiple lists into one. I think the first two are not controversial. What about the other two? I've shown you some examples where we would find a good use in the TripleO templates. The lack of `map` actually blocks us from going all-Heat. The alternative would be to say that this sort of stuff to be done inside the instance by os-apply-config et al. It would complicate things for TripleO, but oh well. Either way, can we come to a clear answer? I would love to convert our templates to 100% Natural Heat during Juno and I'm willing to put my time towards coding the necessary features. Tomas [0]: http://lists.openstack.org/pipermail/openstack-dev/2014-February/027536.html [1]: http://lists.openstack.org/pipermail/openstack-dev/2014-April/031944.html [2]: http://lists.openstack.org/pipermail/openstack-dev/2014-April/032074.html _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev