Hi Alberto,
I assume you are talking about the stats functionality, which is turned on
using the “tags” annotation.
I can help more if you send the sandesh file
Meanwhile, I will demonstrate this feature using Analytics CPU Info.
This is the sandesh file:
struct AnalyticsCpuState {
1: string name
(key="ObjectCollectorInfo")
2: optional bool deleted
3: optional list<cpuinfo.ProcessCpuInfo> cpu_info
(tags=".module_id,.mem_virt,.cpu_share,.mem_res", aggtype="union")
}
uve sandesh AnalyticsCpuStateTrace {
1: AnalyticsCpuState data
}
The tags annotation will cause the “cpu_info” attribute to be recorded as
a Stat Sample.
This sample can then be queried via the Analytics Query API (the same API
used for the object-log)
The attributes listed in the tags annotation will be used for indexing.
(the “where" part of the query)
We implicitly index by the Source of message (Source of the Sandesh
Generator) and the UVE Key.
These fields are recorded as “name” and “Source”.
The other index fields (as per the tags annotation) in this case are
cpu_info.module_id, cpu_info.mem_virt, cpu_info.cpu_share and
cpu_info.mem_res.
There are two other fields that will be present the sample -
cpu_info.inst_id and cpu_info.module_id.
These are not in the tags annotation, so we will not index by them.
From the command line, you can use the “contrail-stats” command to
exercise the Analytics Query API for stats.
Here’s an example of usage:
contrail-stats --dtable AnalyticsCpuState.cpu_info --where "Source=*"
--select "T=300" "COUNT(cpu_info)" "name" "cpu_info.module_id"
"SUM(cpu_info.cpu_share)" --last 30m
This results in the following Analytics API POST request:
{"start_time": "now-30m", "sort_fields": [], "end_time": "now",
"select_fields": ["T=300", "COUNT(cpu_info)", "name",
"cpu_info.module_id", "SUM(cpu_info.cpu_share)"], "table":
"StatTable.AnalyticsCpuState.cpu_info", "where": [[{"suffix": null,
"value2": null, "name": "Source", "value": "", "op": 7}]]}
You can get more information here:
http://www.opencontrail.org/statistics-in-opencontrail-analytics/
This blog entry will also explain what “T” and “T=“ mean.
If you share your sandesh file, I can give you more specific pointers.
Please reach out if you have other questions about this.
Regards,
Anish
On 5/4/15, 3:39 AM, "[email protected]" <[email protected]> wrote:
>Hi,
>
>Any clues about how can I make a flow like the ones from Analytics
>node CPU info? I cannot figure how to do it with sandesh. Is it done
>using list type?
>
>Best Regards,
>Alberto.
>
>
>
>Quoting [email protected]:
>
>> Hi Anish,
>>
>> There was a mistake in the changes I did. Now it's correct and I get
>>this:
>>
>> agutierrez@Mahalanobis:~/Documents/thesis/OpenContrail/query$
>> ./getQuery.sh /uves/alberto/lvelasco?flat
>> % Total % Received % Xferd Average Speed Time Time
>> Time Current
>> Dload Upload Total Spent Left
>>Speed
>> 100 186 0 186 0 0 147 0 --:--:-- 0:00:01
>> --:--:-- 147
>> {
>> "UveVirtualNetworkAgent": {
>> "cpu": 31337,
>> "cpu_info": [
>> {
>> "StatTable.UveVirtualNetworkAgent.cpu_info": [
>> {
>> "COUNT(cpu_info)": 1,
>> "SUM(cpu_info.att1)": 2090,
>> "SUM(cpu_info.att2)": 209
>> }
>> ]
>> }
>> ]
>> }
>> }
>>
>> I modified slightly the structure to have a list inside. Now my
>> question is the following: How can I make a flow like the ones from
>> Analytics node CPU info? I have taken a look to some .sandesh files
>> but I cannot figure out. Also, how can be "T" and "T=" specified?
>>
>> Thank you for your help.
>>
>> Best regards,
>> Alberto.
>>
>> Quoting Anish Mehta <[email protected]>:
>>
>>> Hi Alberto.
>>>
>>> The schema looks correct.
>>>
>>> Both contrail-logs and the Query API use the _STAT_TABLES variable.
>>> They should work if you change that variable as directed.
>>> Please send me the output of "contrail-logs ?help?, and the contents of
>>> your viz/constants.py file.
>>>
>>>
>>> Lets work on the UVE part.
>>> This is what you can do:
>>>
>>> 1.
>>> Check if the UVE table type is available in contrail-analytics-api.
>>> Look at http://xx.xx.xx.xx:8081/analytics/uves
>>> Do you see an entry like this?:
>>>
>>> {
>>> href: "http://xx.xx.xx.xx:8081/analytics/uves/albertos";,
>>> name: ?albertos"
>>> },
>>>
>>>
>>> 2.
>>> Send the UVE from your application.
>>> When you called InitGenerator, you can specify an Introspect port.
>>> I?ll assume you used 5555.
>>> After your application is up, and you have sent your UVE, check for
>>>this:
>>> http://xx.xx.xx.xx:5555/Snh_SandeshUVETypesReq?
>>>
>>> Do you see an entry for you structure ?TestAgent?
>>> You can go onto the link for TestAgent and the the last values you
>>>sent.
>>>
>>> Also, check your connection to the collector:
>>> http://xx.xx.xx.xx:5555/Snh_CollectorInfoRequest?
>>>
>>>
>>> 3.
>>> If all this worked, you should see the UVE at
>>> http://xx.xx.xx.xx:8081/analytics/uves/albertos
>>>
>>>
>>>
>>> Regards,
>>>
>>> Anish
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On 4/24/15, 9:16 AM, "[email protected]" <[email protected]> wrote:
>>>
>>>> Hi,
>>>>
>>>> I compiled viz.sandesh using "Sandesh --gen py viz.sandesh" directly
>>>> and I substituted the .py. I done this using the same structure as
>>>> before but changing the name of the table and following your
>>>> instructions, but I have problems with it.
>>>>
>>>> I get this schema as the table schema:
>>>> {
>>>> "columns": [
>>>> {
>>>> "datatype": "int",
>>>> "index": false,
>>>> "name": "MessageTS",
>>>> "select": null,
>>>> "suffixes": null
>>>> },
>>>> {
>>>> "datatype": "string",
>>>> "index": true,
>>>> "name": "ObjectId",
>>>> "select": null,
>>>> "suffixes": null
>>>> },
>>>> {
>>>> "datatype": "string",
>>>> "index": true,
>>>> "name": "Source",
>>>> "select": null,
>>>> "suffixes": null
>>>> },
>>>> {
>>>> "datatype": "string",
>>>> "index": true,
>>>> "name": "ModuleId",
>>>> "select": null,
>>>> "suffixes": null
>>>> },
>>>> {
>>>> "datatype": "string",
>>>> "index": true,
>>>> "name": "Messagetype",
>>>> "select": null,
>>>> "suffixes": null
>>>> },
>>>> {
>>>> "datatype": "string",
>>>> "index": false,
>>>> "name": "ObjectLog",
>>>> "select": null,
>>>> "suffixes": null
>>>> },
>>>> {
>>>> "datatype": "string",
>>>> "index": false,
>>>> "name": "SystemLog",
>>>> "select": null,
>>>> "suffixes": null
>>>> }
>>>> ],
>>>> "type": "OBJECT"
>>>> }
>>>>
>>>> Does this make sense?
>>>>
>>>> Moreover, the following thing happens:
>>>> contrail-logs --object-type alberto
>>>>>> HTTP error code: 404
>>>>
>>>> It seems to not be available as UVE.
>>>>
>>>>
>>>> Also I have a question regarding to _STAT_TABLES variable in
>>>> viz.sandesh. If I define the structure in this table, will it be
>>>> available in the query part of the web?
>>>>
>>>> Thank you for your assistance,
>>>> Alberto.
>>>>
>>>> Quoting [email protected]:
>>>>
>>>>> Hi Anish,
>>>>>
>>>>> Thank you for your reply. I'm having the following problem when
>>>>> compiling:
>>>>>
>>>>> ...
>>>>> Running virtualenv with interpreter testroot/bin/python
>>>>> New python executable in analytics_test/bin/python
>>>>> Installing
>>>>>
>>>>>
>>>>>Setuptools............................................................
>>>>>...
>>>>> ..............................done.
>>>>> Installing
>>>>>
>>>>>
>>>>>Pip...................................................................
>>>>>...
>>>>> ..............................................................done.
>>>>> /bin/bash -c "source
>>>>>
>>>>>
>>>>>/home/agutierrez/workspace/Contrail/opencontrail2/build/debug/analytic
>>>>>s_t
>>>>> est/bin/activate; pip install
>>>>> --download-cache=/tmp/cache/agutierrez/systemless_test
>>>>>greenlet==0.4.1
>>>>> gevent==0.13.8 eventlet==0.9.17 testtools==0.9.21 fixtures==0.3.12
>>>>> requests>=1.1.0 lxml==2.3.3 geventhttpclient==1.0a prettytable==0.7.2
>>>>> psutil==0.4.1 redis==2.7.1 xmltodict==0.2 thrift==0.8.0
>>>>>bottle==0.11.6
>>>>> mock==1.0.1 kafka-python==0.9.2 kazoo==1.3.1
>>>>> stevedore"
>>>>> scons: *** [build/debug/opserver/test/greenlet] Error 1
>>>>> scons: building terminated because of errors.
>>>>>
>>>>> Do you have any idea that what can be the problem or if there is any
>>>>> log file that could provide more information about the error?
>>>>>
>>>>>
>>>>> Best regards,
>>>>> Alberto.
>>>>>
>>>>> Quoting Anish Mehta <[email protected]>:
>>>>>
>>>>>> Hi Alberto,
>>>>>>
>>>>>> Adding your own table is possible.
>>>>>> You need to add some entries to
>>>>>>/controller/src/analytics/viz.sandesh
>>>>>> It does not need the collector to be rebuilt, but it does require
>>>>>> rebuilding contrail-analytics-api.
>>>>>>
>>>>>>
>>>>>>
>>>>>> I'm going to assume the table name "ObjectAlberto" (instead of
>>>>>> "ObjectTableName")
>>>>>>
>>>>>> I'm going to make this UVE accessible via the Analytics API as
>>>>>>follows:
>>>>>> http://xx.xx.xx.xx:8081/analytics/uves/albertos
>>>>>>
>>>>>> I'm going to make the ObjectLog visible via the Analytics Query API
>>>>>>as
>>>>>> well (as you mentioned in your email)
>>>>>> Based on this API, from the command line of the analytics node you
>>>>>> will be
>>>>>> able to do this:
>>>>>> contrail-logs --object-type alberto
>>>>>>
>>>>>>
>>>>>>
>>>>>> 1. Changes in controller/src/analytics/viz.sandesh
>>>>>>
>>>>>> const string ALBERTO_TABLE = "ObjectAlberto"
>>>>>> const string ALBERTO_DISPLAY_NAME = "Alberto"
>>>>>> ...
>>>>>> const map<string, string> UVE_MAP = {
>>>>>> ...
>>>>>> "alberto" : ALBERTO_TABLE,
>>>>>> }
>>>>>> ...
>>>>>> const map<string, objtable_info> _OBJECT_TABLES = {
>>>>>> ALBERTO_TABLE : {
>>>>>> 'objtable_display_name' : ALBERTO_DISPLAY_NAME
>>>>>> 'log_query_name' : "alberto"
>>>>>> }
>>>>>> }
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2. Rebuild contrail-analytics-api
>>>>>> sb/controller/src$ scons -u opserver
>>>>>>
>>>>>>
>>>>>> 3. Install and test on target
>>>>>> You should rebuild the contrail-analytics package and install it on
>>>>>> your
>>>>>> target.
>>>>>>
>>>>>> For this kind of change, there is an alternative installation
>>>>>>method as
>>>>>> well, given below:
>>>>>>
>>>>>> This is the generated file that will change when you rebuild:
>>>>>> sb/controller/src$ ls
>>>>>> ../../build/debug/opserver/opserver/sandesh/viz/constants.py
>>>>>>
>>>>>> You can copy it onto your analytics node.
>>>>>> It will be at a location such as:
>>>>>> /usr/lib/python2.7/dist-packages/opserver/sandesh/viz/constants.py
>>>>>>
>>>>>> After overwriting the file (you should take a backup first, just in
>>>>>> case),
>>>>>> restart contrail-analytics-api:
>>>>>> service contrail-analytics-api restart
>>>>>>
>>>>>>
>>>>>> 4. Test the UVE Analytics API and Analytics Query API after starting
>>>>>> your
>>>>>> application and connecting to the collector.
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Anish
>>>>>>
>>>>>>
>>>>>> On 4/23/15, 1:34 AM, "[email protected]" <[email protected]>
>>>>>>wrote:
>>>>>>
>>>>>>> Hello Anish,
>>>>>>>
>>>>>>> Thanks for your reply. What we are going to do is a little
>>>>>>>application
>>>>>>> that sends it's own metrics to the collector so we can retrieve
>>>>>>>that
>>>>>>> data using de Analytics API.
>>>>>>>
>>>>>>> For example:
>>>>>>>
>>>>>>> struct TestAgent {
>>>>>>> 1: string name(key="ObjectTableName")
>>>>>>> 2: optional i32 valueToBeSent1
>>>>>>> }
>>>>>>>
>>>>>>> uve sandesh TestTrage {
>>>>>>> 1: TestAgent data;
>>>>>>> }
>>>>>>>
>>>>>>> This kind of message should be stored in some way in Cassandra
>>>>>>> databases and be available to the API users. I understand that
>>>>>>> (key="ObjectTableName") makes reference to a new table with that
>>>>>>>name
>>>>>>> that can be queried from the API. For me that is the concept of
>>>>>>> "custom table", tables that are administrator/developer defined
>>>>>>> different from base contrail tables.
>>>>>>>
>>>>>>> It would be retrieved with a POST query to the API similar to this:
>>>>>>> {
>>>>>>> "end_time": "now",
>>>>>>> "select_fields": [
>>>>>>> "valueToBeSent1"
>>>>>>> ],
>>>>>>> "sort_fields": [],
>>>>>>> "start_time": "now-60m",
>>>>>>> "table": "ObjectTableName",
>>>>>>> "where": [
>>>>>>> [
>>>>>>> {
>>>>>>> "name": "Source",
>>>>>>> "op": 1,
>>>>>>> "suffix": null,
>>>>>>> "value": "GeneratorApplication",
>>>>>>> "value2": null
>>>>>>> }
>>>>>>> ]
>>>>>>> ]
>>>>>>> }
>>>>>>>
>>>>>>> Is this possible? If it is, do I have to do any step apart from
>>>>>>>send
>>>>>>> the structure using the code generated by Sandesh compiler?
>>>>>>>
>>>>>>> If it is not possible, is there any workaround? The only way that
>>>>>>> comes to my mind is that we could query directly to message table,
>>>>>>> filter the messages we don't want and process the XMLs but that
>>>>>>> solution wouldn't be clean.
>>>>>>>
>>>>>>>
>>>>>>> I hope I explained it correctly. Thank you for your assistance.
>>>>>>>
>>>>>>> Best regards,
>>>>>>> Alberto Gutiérrez.
>>>>>>>
>>>>>>> Quoting Anish Mehta <[email protected]>:
>>>>>>>
>>>>>>>>
>>>>>>>> The schemas for new Sandesh types do not need to be compiled or
>>>>>>>> loaded
>>>>>>>> onto the Collector.
>>>>>>>> The collector gets XML-encoded messages from the generators and
>>>>>>>> stores
>>>>>>>> them according to their sandesh types and the contained
>>>>>>>>annotations.
>>>>>>>>
>>>>>>>> Sandesh Types and how they work are explained here:
>>>>>>>> http://www.opencontrail.org/sandesh-a-sdn-analytics-interface/
>>>>>>>>
>>>>>>>>
>>>>>>>> I¹m not sure what you mean by ³custom tables².
>>>>>>>> Please send us a sample sandesh file and describe what you are
>>>>>>>> trying to
>>>>>>>> do, and we can discuss further.
>>>>>>>>
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> Anish
>>>>>>>>
>>>>>>>> On 4/22/15, 6:53 AM, "[email protected]" <[email protected]>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Dear all,
>>>>>>>>>
>>>>>>>>> When creating a new sandesh structure, do I need to send somehow
>>>>>>>>>the
>>>>>>>>> schema to the collectors so they will understand it or it will be
>>>>>>>>> added automatically with the first message it recieves?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks for your attention,
>>>>>>>>> Alberto.
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Dev mailing list
>>>>>>>>> [email protected]
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>http://lists.opencontrail.org/mailman/listinfo/dev_lists.opencontr
>>>>>>>>>ail
>>>>>>>>> .or
>>>>>>>>> g
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Dev mailing list
>>>>> [email protected]
>>>>>
>>>>>
>>>>>http://lists.opencontrail.org/mailman/listinfo/dev_lists.opencontrail.
>>>>>org
>>>>>
>>>>
>>>
>>>
>>
>>
>
_______________________________________________
Dev mailing list
[email protected]
http://lists.opencontrail.org/mailman/listinfo/dev_lists.opencontrail.org
