Github user cestella commented on the issue:

    https://github.com/apache/incubator-metron/pull/449
  
    So, I think this is an interesting approach.  My issues with it are:
    * You seem to be sending every profile into kafka, not just the configured 
ones.
    * You seem to be assuming that one value only is being sent into the 
telemetry and it's the value that you store in HBase
    
    I'd recommend, rather, that you make the `result` field more complex by 
making it a map where the key is the source (e.g. "hbase" or "kafka").  This 
allows you to separate the storage structure by storage medium.  You may, for 
instance, want to *STORE* a stats object in Hbase, but only send along the mean 
and standard deviation.  Also, I'd recommend allowing `result`to be either a 
string (which would presume only hbase is supported) or a Map, which would 
explicitly specify the structure for just the sources you want to write to.
    
    Here's a worked example config for maximum clarity (!):
    ```
    {
      "profiles": [
        {
          "profile": "test",
          "foreach": "'global'",
          "onlyif": "source.type == 'squid'",
          "init":    { "stats": "STATS_INIT()" },
          "update":  { "stats": "STATS_ADD(stats, LENGTH(url))" },
          "result":  {
             "hbase" : "stats",
             "kafka" : "{ 'mean' : STATS_MEAN(stats), 'stddev' : 
STATS_SD(stats) }"
           }
          
        }
      ]
    }
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---

Reply via email to