Hi Ali, This is a holiday in the US (Thanksgiving) and many people have a 4 day weekend. It is also common to travel for this holiday. It is possible that some of the community that know a bit more about storm will not be online during this time.
I do not have experience with the RAS, but in doing some research I can see the following: 1. Our parser topologies are built in code, and therefore would require a code change to allow for setting the cpu and memory component properties for parser topology components. It is also not clear to me how we set those properties for ambari started topologies. 2. Our enrichment and indexing topologies are built with flux, so I *think* those configurations could be edited in the field to set the cpu and memory configurations ( as well as other RAS configs ). But I have not seen any Flux examples on how to do so. 3. I am not certain how much of the node specific configurations to the yaml files can be done from ambari, but it may be. TL/DR; We (I) would need to do more research on how we could support this. Hopefully someone with more storm know how will hop on this soon. I would recommend that you open a Jira on Metron Storm Topologies Supporting Resource Aware Scheduling On November 24, 2017 at 02:56:28, Ali Nazemian (alinazem...@gmail.com) wrote: Any help regarding this question would be appreciated. On Thu, Nov 23, 2017 at 8:57 AM, Ali Nazemian <alinazem...@gmail.com> wrote: > 30 mins average of CPU load by checking Ambari. > > On 23 Nov. 2017 00:51, "Otto Fowler" <ottobackwa...@gmail.com> wrote: > > How are you measuring the utilization? > > > On November 22, 2017 at 08:12:51, Ali Nazemian (alinazem...@gmail.com) > wrote: > > Hi all, > > > One of the issues that we are dealing with is the fact that not all of > the Metron feeds have the same type of resource requirements. For example, > we have some feeds that even a single Strom slot is way more than what it > needs. We thought we could make it more utilised in total by limiting at > least the amount of available heap space per feed to the parser topology > worker. However, since Storm scheduler relies on available slots, it is > very hard and almost impossible to utilise the cluster in the scenario that > there will be lots of different topologies with different requirements > running at the same time. Therefore, on a daily basis, we can see that for > example one of the Storm hosts is 120% utilised and another is 20% > utilised! I was wondering whether we can address this situation by using > Storm Resource Aware scheduler or not. > > P.S: it would be very nice to have a functionality to tune Storm > topology-related parameters per feed in the GUI (for example in Management > UI). > > > Regards, > Ali > > > -- A.Nazemian
