Thank you Mike, I’ll tackle it this morning. I must have built from HEAD so that would be my first mistake when deploying this with an eye to production.
Cheers, Tom. From: Michael Miklavcic <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Wednesday, November 6, 2019 at 9:39 AM To: "[email protected]" <[email protected]> Subject: Re: Error stemming from hbaseBolt Hi Tom, How did you build Metron? Our latest official release is 0.7.1 -> https://archive.apache.org/dist/metron/. Are you building via latest HEAD in the master branch? You might see a bland/vague exception like this if using an IDE (like Eclipse) for compilation. If you're not doing so already, I'd recommend building from the project root using "mvn clean install -DskipTests". For additional guidance, please run <your git project root dir>/metron-deployment/scripts/platform-info.sh and copy the output of running that in a reply. That should give the community some better detail to help. Thanks, Mike On Tue, Nov 5, 2019 at 5:47 PM Yerex, Tom <[email protected]> wrote: Good evening, I am working with metron 0.7.2. I have tried to implement a variation on the solution at hxxps://metron.apache.org/current-book/use-cases/geographic_login_outliers/index.html, with the modification that data is coming into a Kafka topic (JSON format) instead of an import from a CSV. No data has appeared in my Hbase profiler table. I discovered in the Storm UI an error in the hbaseBolt for the profiler topology (see below). I tried an insert at the command prompt, everything seemed okay but my experience with HBase is still limited. My plan tomorrow is to try and debug things using Stellar, but perhaps someone has seen this error somewhere and knows what is wrong? Thank you, Tom. java.lang.Error: Unresolved compilation problem: at org.apache.metron.common.utils.SerDeUtils.toBytes(SerDeUtils.java:235) at org.apache.metron.profiler.hbase.ValueOnlyColumnBuilder.columns(ValueOnlyColumnBuilder.java:52) at org.apache.metron.profiler.storm.ProfileHBaseMapper.columns(ProfileHBaseMapper.java:82) at org.apache.metron.hbase.bolt.HBaseBolt.save(HBaseBolt.java:183) at org.apache.metron.hbase.bolt.HBaseBolt.execute(HBaseBolt.java:164) at org.apache.storm.daemon.executor$fn__10195$tuple_action_fn__10197.invoke(executor.clj:735) at org.apache.storm.daemon.executor$mk_task_receiver$fn__10114.invoke(executor.clj:466) at org.apache.storm.disruptor$clojure_handler$reify__4137.onEvent(disruptor.clj:40) at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472) at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451) at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) at org.apache.storm.daemon.executor$fn__10195$fn__10208$fn__10263.invoke(executor.clj:855) at org.apache.storm.util$async_loop$fn__1221.invoke(util.clj:484) at clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745) (parser configuration) { "parserClassName": "org.apache.metron.parsers.json.JSONMapParser", "sensorTopic": "radius", "outputTopic": "indexing", "fieldTransformations": [{ "transformation": "STELLAR", "output": [ "geohash" ], "config": { "geohash": "GEOHASH_FROM_LOC(GEO_GET(client.ip))" } }] } (enrichment configuration) { "enrichment": { "fieldMap": { "stellar" : { "config" : [ "geo_locations := MULTISET_MERGE( PROFILE_GET( 'locations_by_user', user.id, PROFILE_FIXED( 15, ‘MINUTES’)))", "geo_centroid := GEOHASH_CENTROID(geo_locations)", "geo_distance := TO_INTEGER(GEOHASH_DIST(geo_centroid, hash))", "geo_locations := null" ] } } ,"fieldToTypeMap": { } }, "threatIntel": { "fieldMap": { "stellar" : { "config" : [ "geo_distance_distr:= STATS_MERGE( PROFILE_GET( 'geo_distribution_from_centroid', 'global', PROFILE_FIXED( 15, ‘MINUTES’)))", "dist_median := STATS_PERCENTILE(geo_distance_distr, 50.0)", "dist_sd := STATS_SD(geo_distance_distr)", "geo_outlier := ABS(dist_median - geo_distance) >= 5*dist_sd", "is_alert := is_alert || (geo_outlier != null && geo_outlier == true)", "geo_distance_distr := null" ] } }, "fieldToTypeMap": { }, "triageConfig" : { "riskLevelRules" : [ { "name" : "Geographic Outlier", "comment" : "Determine if the user's geographic distance from the centroid of the historic logins is an outlier as compared to all users.", "rule" : "geo_outlier != null && geo_outlier", "score" : 10, "reason" : "FORMAT('user %s has a distance (%d) from the centroid of their last login is 5 std deviations (%f) from the median (%f)', user.id, geo_distance, dist_sd, dist_median)" } ], "aggregator" : "MAX" } } } (profiler.json) { "profiles": [ { "profile": "geo_distribution_from_centroid", "foreach": "'global'", "onlyif": "geo_distance != null", "init": { "s": "STATS_INIT()" }, "update": { "s": "STATS_ADD(s, geo_distance)" }, "result": { "profile":"s" } }, { "profile": "locations_by_user", "foreach": "user.id", "onlyif": "geohash != null && LENGTH(geohash) > 0", "init": { "s": "MULTISET_INIT()" }, "update": { "s": "MULTISET_ADD(s, geohash)" }, "result": { "profile":"s" } } ] }
smime.p7s
Description: S/MIME cryptographic signature
