Are you encountering a performance problem? If not, does the skew really matter?
Otherwise: Is the fields grouping absolutely required for your use case? If so, could you maybe split your processing into an A and B bolts where the A bolt does most of the processing by shuffle and then sends the data to a B bolt by fields that does minimal processing? You could also analyze the data skew of keys to your hash with an actual dataset and try different hashes to even out the results, possibly see if raising/lowering executor count helps/hurts the skew. On Fri, Aug 16, 2024 at 6:50 PM Karthick <ibmkarthickma...@gmail.com> wrote: > Hi Team, > > I'm using fields grouping for a bolt to maintain field-based ordering, but > I'm facing data skewness among the bolt's executors. I have 96 executors, > and I'm sending data with 500 distinct fields used in the fields grouping. > While reviewing the Storm UI, I noticed that a few executors are > underutilized while others are overutilized. > > This seems to be a hashing problem i guess. Can anyone suggest a better > hashing technique or approach to resolve this issue? > > Thanks in advance for your help. >
