Hi Albero, trace is a keyword, please change name to data_trace. Also please have optional bool deleted as element number 2 and then you should be good to go
Thanks Megh > On May 6, 2015, at 6:57 AM, "[email protected]" <[email protected]> wrote: > > Hi anish, > > Thanks for your reply. > > In that example, aggregation type "union" is applied over all the elements of > type ProcessCpuInfo? > > I have adapted my structure following your example: > struct DataTrace { > 1: u64 val1 > 2: u64 val2 > } > > struct AlbertoAgent { > 1: string name > (key="ObjectAlberto") > 2: optional u64 cpu > 3: optional bool deleted > 4: optional list<DataTrace> trace > (tags=".val1,.val2") > } > > uve sandesh AlbertoTrace { > 1: AlbertoAgent data > } > > However there is a syntax error that I'm not able to detect (Error > message:(last token was 'trace')syntax error). > > Am I in the correct way? Where is the error? > > Thanks for your support, > Alberto. > > > Quoting Anish Mehta <[email protected]>: > >> Hi Alberto, >> >> I assume you are talking about the stats functionality, which is turned on >> using the "tags" annotation. >> I can help more if you send the sandesh file >> >> Meanwhile, I will demonstrate this feature using Analytics CPU Info. >> >> This is the sandesh file: >> >> struct AnalyticsCpuState { >> 1: string name >> (key="ObjectCollectorInfo") >> 2: optional bool deleted >> 3: optional list<cpuinfo.ProcessCpuInfo> cpu_info >> (tags=".module_id,.mem_virt,.cpu_share,.mem_res", aggtype="union") >> } >> >> uve sandesh AnalyticsCpuStateTrace { >> 1: AnalyticsCpuState data >> } >> >> >> The tags annotation will cause the "cpu_info" attribute to be recorded as >> a Stat Sample. >> This sample can then be queried via the Analytics Query API (the same API >> used for the object-log) >> The attributes listed in the tags annotation will be used for indexing. >> (the "where" part of the query) >> >> >> We implicitly index by the Source of message (Source of the Sandesh >> Generator) and the UVE Key. >> These fields are recorded as "name" and "Source". >> The other index fields (as per the tags annotation) in this case are >> cpu_info.module_id, cpu_info.mem_virt, cpu_info.cpu_share and >> cpu_info.mem_res. >> There are two other fields that will be present the sample - >> cpu_info.inst_id and cpu_info.module_id. >> These are not in the tags annotation, so we will not index by them. >> >> From the command line, you can use the "contrail-stats" command to >> exercise the Analytics Query API for stats. >> Here's an example of usage: >> >> contrail-stats --dtable AnalyticsCpuState.cpu_info --where "Source=*" >> --select "T=300" "COUNT(cpu_info)" "name" "cpu_info.module_id" >> "SUM(cpu_info.cpu_share)" --last 30m >> >> This results in the following Analytics API POST request: >> {"start_time": "now-30m", "sort_fields": [], "end_time": "now", >> "select_fields": ["T=300", "COUNT(cpu_info)", "name", >> "cpu_info.module_id", "SUM(cpu_info.cpu_share)"], "table": >> "StatTable.AnalyticsCpuState.cpu_info", "where": [[{"suffix": null, >> "value2": null, "name": "Source", "value": "", "op": 7}]]} >> >> You can get more information here: >> http://www.opencontrail.org/statistics-in-opencontrail-analytics/ >> This blog entry will also explain what "T" and "T=" mean. >> >> If you share your sandesh file, I can give you more specific pointers. >> Please reach out if you have other questions about this. >> >> Regards, >> >> Anish >> >> >>> On 5/4/15, 3:39 AM, "[email protected]" <[email protected]> wrote: >>> >>> Hi, >>> >>> Any clues about how can I make a flow like the ones from Analytics >>> node CPU info? I cannot figure how to do it with sandesh. Is it done >>> using list type? >>> >>> Best Regards, >>> Alberto. >>> >>> >>> >>> Quoting [email protected]: >>> >>>> Hi Anish, >>>> >>>> There was a mistake in the changes I did. Now it's correct and I get >>>> this: >>>> >>>> agutierrez@Mahalanobis:~/Documents/thesis/OpenContrail/query$ >>>> ./getQuery.sh /uves/alberto/lvelasco?flat >>>> % Total % Received % Xferd Average Speed Time Time >>>> Time Current >>>> Dload Upload Total Spent Left >>>> Speed >>>> 100 186 0 186 0 0 147 0 --:--:-- 0:00:01 >>>> --:--:-- 147 >>>> { >>>> "UveVirtualNetworkAgent": { >>>> "cpu": 31337, >>>> "cpu_info": [ >>>> { >>>> "StatTable.UveVirtualNetworkAgent.cpu_info": [ >>>> { >>>> "COUNT(cpu_info)": 1, >>>> "SUM(cpu_info.att1)": 2090, >>>> "SUM(cpu_info.att2)": 209 >>>> } >>>> ] >>>> } >>>> ] >>>> } >>>> } >>>> >>>> I modified slightly the structure to have a list inside. Now my >>>> question is the following: How can I make a flow like the ones from >>>> Analytics node CPU info? I have taken a look to some .sandesh files >>>> but I cannot figure out. Also, how can be "T" and "T=" specified? >>>> >>>> Thank you for your help. >>>> >>>> Best regards, >>>> Alberto. >>>> >>>> Quoting Anish Mehta <[email protected]>: >>>> >>>>> Hi Alberto. >>>>> >>>>> The schema looks correct. >>>>> >>>>> Both contrail-logs and the Query API use the _STAT_TABLES variable. >>>>> They should work if you change that variable as directed. >>>>> Please send me the output of "contrail-logs ?help?, and the contents of >>>>> your viz/constants.py file. >>>>> >>>>> >>>>> Lets work on the UVE part. >>>>> This is what you can do: >>>>> >>>>> 1. >>>>> Check if the UVE table type is available in contrail-analytics-api. >>>>> Look at http://xx.xx.xx.xx:8081/analytics/uves >>>>> Do you see an entry like this?: >>>>> >>>>> { >>>>> href: "http://xx.xx.xx.xx:8081/analytics/uves/albertos";, >>>>> name: ?albertos" >>>>> }, >>>>> >>>>> >>>>> 2. >>>>> Send the UVE from your application. >>>>> When you called InitGenerator, you can specify an Introspect port. >>>>> I?ll assume you used 5555. >>>>> After your application is up, and you have sent your UVE, check for >>>>> this: >>>>> http://xx.xx.xx.xx:5555/Snh_SandeshUVETypesReq? >>>>> >>>>> Do you see an entry for you structure ?TestAgent? >>>>> You can go onto the link for TestAgent and the the last values you >>>>> sent. >>>>> >>>>> Also, check your connection to the collector: >>>>> http://xx.xx.xx.xx:5555/Snh_CollectorInfoRequest? >>>>> >>>>> >>>>> 3. >>>>> If all this worked, you should see the UVE at >>>>> http://xx.xx.xx.xx:8081/analytics/uves/albertos >>>>> >>>>> >>>>> >>>>> Regards, >>>>> >>>>> Anish >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> On 4/24/15, 9:16 AM, "[email protected]" <[email protected]> wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> I compiled viz.sandesh using "Sandesh --gen py viz.sandesh" directly >>>>>> and I substituted the .py. I done this using the same structure as >>>>>> before but changing the name of the table and following your >>>>>> instructions, but I have problems with it. >>>>>> >>>>>> I get this schema as the table schema: >>>>>> { >>>>>> "columns": [ >>>>>> { >>>>>> "datatype": "int", >>>>>> "index": false, >>>>>> "name": "MessageTS", >>>>>> "select": null, >>>>>> "suffixes": null >>>>>> }, >>>>>> { >>>>>> "datatype": "string", >>>>>> "index": true, >>>>>> "name": "ObjectId", >>>>>> "select": null, >>>>>> "suffixes": null >>>>>> }, >>>>>> { >>>>>> "datatype": "string", >>>>>> "index": true, >>>>>> "name": "Source", >>>>>> "select": null, >>>>>> "suffixes": null >>>>>> }, >>>>>> { >>>>>> "datatype": "string", >>>>>> "index": true, >>>>>> "name": "ModuleId", >>>>>> "select": null, >>>>>> "suffixes": null >>>>>> }, >>>>>> { >>>>>> "datatype": "string", >>>>>> "index": true, >>>>>> "name": "Messagetype", >>>>>> "select": null, >>>>>> "suffixes": null >>>>>> }, >>>>>> { >>>>>> "datatype": "string", >>>>>> "index": false, >>>>>> "name": "ObjectLog", >>>>>> "select": null, >>>>>> "suffixes": null >>>>>> }, >>>>>> { >>>>>> "datatype": "string", >>>>>> "index": false, >>>>>> "name": "SystemLog", >>>>>> "select": null, >>>>>> "suffixes": null >>>>>> } >>>>>> ], >>>>>> "type": "OBJECT" >>>>>> } >>>>>> >>>>>> Does this make sense? >>>>>> >>>>>> Moreover, the following thing happens: >>>>>> contrail-logs --object-type alberto >>>>>>>> HTTP error code: 404 >>>>>> >>>>>> It seems to not be available as UVE. >>>>>> >>>>>> >>>>>> Also I have a question regarding to _STAT_TABLES variable in >>>>>> viz.sandesh. If I define the structure in this table, will it be >>>>>> available in the query part of the web? >>>>>> >>>>>> Thank you for your assistance, >>>>>> Alberto. >>>>>> >>>>>> Quoting [email protected]: >>>>>> >>>>>>> Hi Anish, >>>>>>> >>>>>>> Thank you for your reply. I'm having the following problem when >>>>>>> compiling: >>>>>>> >>>>>>> ... >>>>>>> Running virtualenv with interpreter testroot/bin/python >>>>>>> New python executable in analytics_test/bin/python >>>>>>> Installing >>>>>>> >>>>>>> >>>>>>> Setuptools............................................................ >>>>>>> ... >>>>>>> ..............................done. >>>>>>> Installing >>>>>>> >>>>>>> >>>>>>> Pip................................................................... >>>>>>> ... >>>>>>> ..............................................................done. >>>>>>> /bin/bash -c "source >>>>>>> >>>>>>> >>>>>>> /home/agutierrez/workspace/Contrail/opencontrail2/build/debug/analytic >>>>>>> s_t >>>>>>> est/bin/activate; pip install >>>>>>> --download-cache=/tmp/cache/agutierrez/systemless_test >>>>>>> greenlet==0.4.1 >>>>>>> gevent==0.13.8 eventlet==0.9.17 testtools==0.9.21 fixtures==0.3.12 >>>>>>> requests>=1.1.0 lxml==2.3.3 geventhttpclient==1.0a prettytable==0.7.2 >>>>>>> psutil==0.4.1 redis==2.7.1 xmltodict==0.2 thrift==0.8.0 >>>>>>> bottle==0.11.6 >>>>>>> mock==1.0.1 kafka-python==0.9.2 kazoo==1.3.1 >>>>>>> stevedore" >>>>>>> scons: *** [build/debug/opserver/test/greenlet] Error 1 >>>>>>> scons: building terminated because of errors. >>>>>>> >>>>>>> Do you have any idea that what can be the problem or if there is any >>>>>>> log file that could provide more information about the error? >>>>>>> >>>>>>> >>>>>>> Best regards, >>>>>>> Alberto. >>>>>>> >>>>>>> Quoting Anish Mehta <[email protected]>: >>>>>>> >>>>>>>> Hi Alberto, >>>>>>>> >>>>>>>> Adding your own table is possible. >>>>>>>> You need to add some entries to >>>>>>>> /controller/src/analytics/viz.sandesh >>>>>>>> It does not need the collector to be rebuilt, but it does require >>>>>>>> rebuilding contrail-analytics-api. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> I'm going to assume the table name "ObjectAlberto" (instead of >>>>>>>> "ObjectTableName") >>>>>>>> >>>>>>>> I'm going to make this UVE accessible via the Analytics API as >>>>>>>> follows: >>>>>>>> http://xx.xx.xx.xx:8081/analytics/uves/albertos >>>>>>>> >>>>>>>> I'm going to make the ObjectLog visible via the Analytics Query API >>>>>>>> as >>>>>>>> well (as you mentioned in your email) >>>>>>>> Based on this API, from the command line of the analytics node you >>>>>>>> will be >>>>>>>> able to do this: >>>>>>>> contrail-logs --object-type alberto >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> 1. Changes in controller/src/analytics/viz.sandesh >>>>>>>> >>>>>>>> const string ALBERTO_TABLE = "ObjectAlberto" >>>>>>>> const string ALBERTO_DISPLAY_NAME = "Alberto" >>>>>>>> ... >>>>>>>> const map<string, string> UVE_MAP = { >>>>>>>> ... >>>>>>>> "alberto" : ALBERTO_TABLE, >>>>>>>> } >>>>>>>> ... >>>>>>>> const map<string, objtable_info> _OBJECT_TABLES = { >>>>>>>> ALBERTO_TABLE : { >>>>>>>> 'objtable_display_name' : ALBERTO_DISPLAY_NAME >>>>>>>> 'log_query_name' : "alberto" >>>>>>>> } >>>>>>>> } >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> 2. Rebuild contrail-analytics-api >>>>>>>> sb/controller/src$ scons -u opserver >>>>>>>> >>>>>>>> >>>>>>>> 3. Install and test on target >>>>>>>> You should rebuild the contrail-analytics package and install it on >>>>>>>> your >>>>>>>> target. >>>>>>>> >>>>>>>> For this kind of change, there is an alternative installation >>>>>>>> method as >>>>>>>> well, given below: >>>>>>>> >>>>>>>> This is the generated file that will change when you rebuild: >>>>>>>> sb/controller/src$ ls >>>>>>>> ../../build/debug/opserver/opserver/sandesh/viz/constants.py >>>>>>>> >>>>>>>> You can copy it onto your analytics node. >>>>>>>> It will be at a location such as: >>>>>>>> /usr/lib/python2.7/dist-packages/opserver/sandesh/viz/constants.py >>>>>>>> >>>>>>>> After overwriting the file (you should take a backup first, just in >>>>>>>> case), >>>>>>>> restart contrail-analytics-api: >>>>>>>> service contrail-analytics-api restart >>>>>>>> >>>>>>>> >>>>>>>> 4. Test the UVE Analytics API and Analytics Query API after starting >>>>>>>> your >>>>>>>> application and connecting to the collector. >>>>>>>> >>>>>>>> >>>>>>>> Regards, >>>>>>>> >>>>>>>> Anish >>>>>>>> >>>>>>>> >>>>>>>> On 4/23/15, 1:34 AM, "[email protected]" <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hello Anish, >>>>>>>>> >>>>>>>>> Thanks for your reply. What we are going to do is a little >>>>>>>>> application >>>>>>>>> that sends it's own metrics to the collector so we can retrieve >>>>>>>>> that >>>>>>>>> data using de Analytics API. >>>>>>>>> >>>>>>>>> For example: >>>>>>>>> >>>>>>>>> struct TestAgent { >>>>>>>>> 1: string name(key="ObjectTableName") >>>>>>>>> 2: optional i32 valueToBeSent1 >>>>>>>>> } >>>>>>>>> >>>>>>>>> uve sandesh TestTrage { >>>>>>>>> 1: TestAgent data; >>>>>>>>> } >>>>>>>>> >>>>>>>>> This kind of message should be stored in some way in Cassandra >>>>>>>>> databases and be available to the API users. I understand that >>>>>>>>> (key="ObjectTableName") makes reference to a new table with that >>>>>>>>> name >>>>>>>>> that can be queried from the API. For me that is the concept of >>>>>>>>> "custom table", tables that are administrator/developer defined >>>>>>>>> different from base contrail tables. >>>>>>>>> >>>>>>>>> It would be retrieved with a POST query to the API similar to this: >>>>>>>>> { >>>>>>>>> "end_time": "now", >>>>>>>>> "select_fields": [ >>>>>>>>> "valueToBeSent1" >>>>>>>>> ], >>>>>>>>> "sort_fields": [], >>>>>>>>> "start_time": "now-60m", >>>>>>>>> "table": "ObjectTableName", >>>>>>>>> "where": [ >>>>>>>>> [ >>>>>>>>> { >>>>>>>>> "name": "Source", >>>>>>>>> "op": 1, >>>>>>>>> "suffix": null, >>>>>>>>> "value": "GeneratorApplication", >>>>>>>>> "value2": null >>>>>>>>> } >>>>>>>>> ] >>>>>>>>> ] >>>>>>>>> } >>>>>>>>> >>>>>>>>> Is this possible? If it is, do I have to do any step apart from >>>>>>>>> send >>>>>>>>> the structure using the code generated by Sandesh compiler? >>>>>>>>> >>>>>>>>> If it is not possible, is there any workaround? The only way that >>>>>>>>> comes to my mind is that we could query directly to message table, >>>>>>>>> filter the messages we don't want and process the XMLs but that >>>>>>>>> solution wouldn't be clean. >>>>>>>>> >>>>>>>>> >>>>>>>>> I hope I explained it correctly. Thank you for your assistance. >>>>>>>>> >>>>>>>>> Best regards, >>>>>>>>> Alberto Gutiérrez. >>>>>>>>> >>>>>>>>> Quoting Anish Mehta <[email protected]>: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> The schemas for new Sandesh types do not need to be compiled or >>>>>>>>>> loaded >>>>>>>>>> onto the Collector. >>>>>>>>>> The collector gets XML-encoded messages from the generators and >>>>>>>>>> stores >>>>>>>>>> them according to their sandesh types and the contained >>>>>>>>>> annotations. >>>>>>>>>> >>>>>>>>>> Sandesh Types and how they work are explained here: >>>>>>>>>> http://www.opencontrail.org/sandesh-a-sdn-analytics-interface/ >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I¹m not sure what you mean by ³custom tables². >>>>>>>>>> Please send us a sample sandesh file and describe what you are >>>>>>>>>> trying to >>>>>>>>>> do, and we can discuss further. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> >>>>>>>>>> Anish >>>>>>>>>> >>>>>>>>>> On 4/22/15, 6:53 AM, "[email protected]" <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Dear all, >>>>>>>>>>> >>>>>>>>>>> When creating a new sandesh structure, do I need to send somehow >>>>>>>>>>> the >>>>>>>>>>> schema to the collectors so they will understand it or it will be >>>>>>>>>>> added automatically with the first message it recieves? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Thanks for your attention, >>>>>>>>>>> Alberto. >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Dev mailing list >>>>>>>>>>> [email protected] >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> http://lists.opencontrail.org/mailman/listinfo/dev_lists.opencontr >>>>>>>>>>> ail >>>>>>>>>>> .or >>>>>>>>>>> g >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Dev mailing list >>>>>>> [email protected] >>>>>>> >>>>>>> >>>>>>> http://lists.opencontrail.org/mailman/listinfo/dev_lists.opencontrail. >>>>>>> org > > > _______________________________________________ > Dev mailing list > [email protected] > http://lists.opencontrail.org/mailman/listinfo/dev_lists.opencontrail.org _______________________________________________ Dev mailing list [email protected] http://lists.opencontrail.org/mailman/listinfo/dev_lists.opencontrail.org
