> In the various examples I’ve found, each profile computes only one value. Is it possible to do something like that:
Currently, each profile produces only a single value. But I've had the same thought as you. It would be nice to be able to persist multiple values in one shot. Feel free to create a JIRA if you have thoughts on how that might work. > From the Stellar CLI, it seems to works fine, but when I try during data ingest, I see no data coming into the profiler table. Please note that I’ve waited for the 15mn, and that I have deleted data in the profiler table using the “truncate_preserve” command in hbase. (1) Have you changed any of the default Profiler settings like the profile period? (2) In the Stellar REPL (CLI) did you test with the real telemetry from Kafka or just mock messages? Retrieve some of the real streaming telemetry using `KAFKA_GET` and test that against your profile definition. Maybe there is a difference between the mock data that you tested with versus the real streaming data. (3) To debug further, in the Storm UI go to the Profiler topology, then turn on DEBUG logging for the package "org.apache.metron.profiler". Leave this extra logging running for at least the length of your profile period (by default 15 minutes) and see what the logs show. > The data in the profiler table can be pulled with Stellar functions which include some advanced features like statistics, cardinality, but is it possible to access all of this from a Java / Scala / any other language? There is an internal class that the PROFILE_GET Stellar function uses to actually retrieve the profiles from HBase. You could possibly use this as a Java API. But this is definitely not ideal. I have done some proof-of-concept work to store the profile data using Phoenix in Hbase, which would allow you to access all of the profile data using SQL/JDBC/ODBC. That would allow you to retrieve that data with the tooling of your choice, even including something like a Zeppelin Notebook. When I have some cycles, I'd like to share that with the community. On Wed, Apr 24, 2019 at 4:30 AM <[email protected]> wrote: > Hello everybody, > > > > I’ve been playing with Metron for a few weeks now, it is really a very > exciting project and I’d like first to thanks all the contributors. I’m > currently investigating around the use of profiler. I’ve tested it with the > basic example of counting IP address as explained in the doc, and now it’s > time for questions: > > - In the various examples I’ve found, each profile computes only > one value. Is it possible to do something like that: > > { > > "profiles": [ > > { > > "profile": "test", > > "foreach": "ip_src_addr", > > "onlyif": "exists(ip_src_addr)", > > "init": { "count": 0 , "sum_rcvd_bytes": 0}, > > "update": { "count": "count + 1", > > "sum_rcvd_bytes": "sum_rcvd_bytes + rcvdbyte" }, > > "result": "{'count': count, 'sum_rcvd_bytes': sum_rcvd_bytes}" > > } > > ] > > } > > > > From the Stellar CLI, it seems to works fine, but when I try during data > ingest, I see no data coming into the profiler table. Please note that I’ve > waited for the 15mn, and that I have deleted data in the profiler table > using the “truncate_preserve” command in hbase. > > - In case of issue, what is the right procedure to reinitialize > all the profiler stack? > > - The data in the profiler table can be pulled with Stellar > functions which include some advanced features like statistics, > cardinality, but is it possible to access all of this from a Java / Scala / > any other language? > > - The MaaS service seems to apply to the incoming data only, how > is it possible to use it only on the aggregated profile data? > > > > > > Maybe too much questions in the same mail? > > > > Thanks, > > > > Stéphane > > _________________________________________________________________________________________________________________________ > > Ce message et ses pieces jointes peuvent contenir des informations > confidentielles ou privilegiees et ne doivent donc > pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu > ce message par erreur, veuillez le signaler > a l'expediteur et le detruire ainsi que les pieces jointes. Les messages > electroniques etant susceptibles d'alteration, > Orange decline toute responsabilite si ce message a ete altere, deforme ou > falsifie. Merci. > > This message and its attachments may contain confidential or privileged > information that may be protected by law; > they should not be distributed, used or copied without authorisation. > If you have received this email in error, please notify the sender and delete > this message and its attachments. > As emails may be altered, Orange is not liable for messages that have been > modified, changed or falsified. > Thank you. > >
