Hi Sanket, Strictly speaking, there isn't a streaming join function. However, you can accomplish something reasonably similar using a streaming enrichment, as you pointed out. You would ultimately pull the enrichment based on the user id, which would then add all the additional fields from your enrichment stream to the primary stream. See this for example - https://cwiki.apache.org/confluence/display/METRON/2016/04/28/Metron+Tutorial+-+Fundamentals+Part+2%3A+Creating+a+New+Enrichment .
In that basic enrichment example I shared above, the squid data stream has a field "domain_without_subdomains." We configure it to be enriched with the "whois" enrichment type. The whois enrichment has the following fields: 1. "domain" : 0 2. ,"owner" : 1 3. ,"home_country" : 2 4. ,"registrar": 3 5. ,"domain_created_timestamp": 4 We join "domain_without_subdomains" by a lookup on "domain." When there's a match, it pulls in the additional fields (prefixed by the enrichment type) as noted at the end of the tutorial: 1. whois.owner 2. whois.domain_created_timestamp 3. whois.registrar 4. whois.home_country You could do the same thing with your use case, the only difference being that you would make yours a streaming enrichment. Be aware that there are timing considerations - ie if your enrichment doesn't exist yet, we don't fail/retry. The data will simply move along un-enriched as-is. Best, Mike Miklavcic On Mon, Jun 24, 2019 at 6:29 PM Sanket Sharma <[email protected]> wrote: > Hi, > > I have a requirement to enrich one of my event feed with another event > feed. One event feed has session logs (user id, IP address) and the other > feed has user id and other application specific fields. I would like the > two streams to be joined in real time before they are scored and triaged. > What is the best way to achieve this in Metron? > > I would have preferred to use something like KSQL but I guess its not > supported directly on HDP stack yet. Google pointed me to the below link > > > > > https://cwiki.apache.org/confluence/display/METRON/2016/06/16/Metron+Tutorial+-+Fundamentals+Part+6%3A+Streaming+Enrichment > > > > However, using the above, I can only add one field at a time. I guess > that’s workable, but just wondering if it’s still relevant and the > recommended way to merge/join streams? > > > > > > Best regards, > > Sanket >
