Where do you get the "Error: 'NoneType' object has no attribute ‘iterates’” reply? Were you using “contrail-stats”, or issuing a POST to the Analytics API?
After trying the query, please send these files: /var/log/contrail/contrail-analytics-api.log /var/log/contrail/contrail-analytics-api-stdout.log Also send me the output of "contrail-logs --module contrail-query-engine ―all” On 5/14/15, 4:16 AM, "[email protected]" <[email protected]> wrote: >Hi, > >I tried to do a workaround but it was somehow unsuccessful. > > >We have the following structure: >struct VMUsageObject >{ > 1: string cache_id > 2: string vm_id > 3: double cpu_used > 4: u64 mem_used >} > >struct VMServiceObject >{ > 1: string cache_id > 2: string vm_id > 3: string service_id > 4: string status > 5: u64 memory >} > >struct QoSInfoObject >{ > 1: string cache_id > 2: u64 service_quality > 3: double hit_ratio >} > >struct ChannelUsersObject >{ > 1: string cache_id > 2: string channel_id > 3: u64 num_users >} > >struct DeviceTypeUsersObject >{ > 1: string cache_id > 2: string devicetype_id > 3: u64 num_users >} > >struct CDNAgent >{ > 1: string name (key="CDNObject") > 2: optional bool deleted > 3: optional list<VMUsageObject> VMUsage > (tags=".cache_id,.vm_id") > 4: optional list<VMServiceObject> VMServices > (tags=".cache_id,.vm_id,.service_id") > 5: optional list<QoSInfoObject> QoSInfo > (tags=".cache_id") > 6: >optional list<ChannelUsersObject> ChannelUsers >(tags=".cache_id,.channel_i >d") > 7: >optional list<DeviceTypeUsersObject> DeviceTypeUsers >(tags=".cache_id,.dev >icetype_id") >} > >uve sandesh CDNTrace >{ > 1: CDNAgent data >} > >By checking the CDN uve we get the reply that is shown in the adjoint >SandeshReply.json. Except that there is a table that doesn't contain >any value (VMServices), the rest is OK. > >However, if we do the following query: > >{ > "end_time": "now", > "select_fields": [ > "name", "T", "VMUsage.cache_id", "VMUsage.vm_id", >"VMUsage.cpu_used", "VMUsage.mem_used" > ], > "sort_fields": [], > "start_time": "now-60m", > "table": "StatTable.CDNAgent.VMUsage", > "where": [ > [ > { > "name": "name", > "op": 7, > "suffix": null, > "value": "", > "value2": null > } > ] > ] >} > >We get this reply: >Error: 'NoneType' object has no attribute 'iteritems' > >Any ideas? > >Best Regards, >Alberto. > >Quoting Anish Mehta <[email protected]>: > >> Hi Alberto, >> >> For ObjectTables (Using "table":"ObjectAlberto" for queries) , the list >> and map should work fine. >> >> But for StatsTables, the answer is more complicated. >> >> Upto R2.1, we only record StatsTable samples for strings, ints and >>doubles. >> >> Starting in R2.2, we do support structs and lists-of-structs in >>StatTables. >> You would need to use the "tags" annotation against the structs and list >> of struct you want recorded in this way. >> Otherwise, the struct or list-of-structs will be ignored. >> >> Maps are not supported. >> >> For your example, it would work like this: >> >> struct ServiceInfo2 { >> 1: string sometag >> 2: u64 someval >> } >> >> struct VMInfoObject { >> 1: string cache_id >> 2: string vm_id >> 3: double cpu_used >> 4: u64 mem_used >> 6: list<ServiceInfo2> services2 (tags=".someval") >> } >> >> struct AlbertoAgent { >> >> 1: string name (key="ObjectAlberto") >> 2: optional bool deleted >> >> 3: optional list<VMInfoObject> VMInfo (tags=".cache_id,.vm_id") >> } >> >> >> >> You can query it in this way: >> >> "table": "StatTable.AlbertoAgent.VMInfo.services2", >> >> "where": [[ >> { >> "name": "name", >> "op": 1, >> "suffix": null, >> "value": "lvelascoTurbo", >> "value2": null >> }, >> { >> "name": "VMInfo.cache_id", >> "op": 1, >> "suffix": null, >> "value": "mycache", >> "value2": null >> }, >> >> { >> "name": "VMInfo.services2.sometag", >> "op": 1, >> "suffix": null, >> "value": "mytag", >> "value2": null >> } >> >> ]], >> "select_fields": [ >> "T=60", "COUNT(VMInfo.services2)", >> "SUM(VMInfo.services2.someval)", >> "VMInfo.vm_id" >> ], >> >> >> >> >> The functionally was added in this commit: >> >>https://github.com/Juniper/contrail-controller/commit/226b6230690d4ee7a65 >>48 >> 24bef0e9b667011f842 >> >> Regards, >> >> Anish >> >> On 5/13/15, 9:36 AM, "[email protected]" <[email protected]> wrote: >> >>> Hi, >>> >>> Another question: Is it possible to have lists/maps and get them in a >>> query? >>> >>> For example: >>> >>> struct VMInfoObject >>> { >>> 1: string cache_id >>> 2: string vm_id >>> 3: double cpu_used >>> 4: u64 mem_used >>> 5: map<string, ServiceInfo> services >>> 6: list<ServiceInfo2> services2 >>> } >>> >>> With: >>> 3: optional list<VMInfoObject> VMInfo >>> (tags=".cache_id,.vm_id") >>> >>> Can I get in a single query the services or the services2? We tried to >>> query for "VMInfo.services" and "VMInfo.services2" with no positive >>> result. >>> >>> >>> >>> Thanks for your support, >>> Alberto. >>> >>> Quoting Anish Mehta <[email protected]>: >>> >>>> Hi Alberto, >>>> >>>> Support for MAX and MIN is not in R2.1 or earlier release. >>>> You will see it if you use the R2.2 or mainline branches. >>>> >>>> >>>> If you aren't sure if your image has MAX and MIN, try to access this >>>> schema via the Analytics API: >>>> >>>> >>>> >>>>http://xx.xx.xx.xx:8081/analytics/table/StatTable.AnalyticsCpuState.cpu >>>>_i >>>> nf >>>> o/schema >>>> >>>> >>>> If you see MAX and MIN attributes here, they should work for your >>>>query >>>> as >>>> well. >>>> >>>> Regards, >>>> >>>> Anish >>>> >>>> >>>> On 5/12/15, 8:48 AM, "[email protected]" <[email protected]> >>>>wrote: >>>> >>>>> Hi Anish, >>>>> >>>>> Thanks for your reply. I'm now doing the following query: >>>>> >>>>> { >>>>> "end_time": "now", >>>>> "select_fields": [ >>>>> "T=30s", "COUNT(dTrace)", "SUM(dTrace.val1)", >>>>> "SUM(dTrace.val2)", "SUM(dTrace.val3)", "MAX(dTrace.val2)", >>>>> "MIN(dTrace.val3)" >>>>> ], >>>>> "sort_fields": [], >>>>> "start_time": "now-1m", >>>>> "table": "StatTable.AlbertoAgent.dTrace", >>>>> "where": [ >>>>> [ >>>>> { >>>>> "name": "name", >>>>> "op": 1, >>>>> "suffix": null, >>>>> "value": "lvelascoTurbo", >>>>> "value2": null >>>>> } >>>>> ] >>>>> ] >>>>> >>>>> } >>>>> >>>>> >>>>> And as result I get the following: >>>>> { >>>>> "value": [ >>>>> { >>>>> "COUNT(dTrace)": 22, >>>>> "SUM(dTrace.val1)": 22838019078, >>>>> "SUM(dTrace.val2)": 27778532625, >>>>> "SUM(dTrace.val3)": 726, >>>>> "T=": 1431445440000000 >>>>> }, >>>>> { >>>>> "COUNT(dTrace)": 28, >>>>> "SUM(dTrace.val1)": 30242613471, >>>>> "SUM(dTrace.val2)": 36570568510, >>>>> "SUM(dTrace.val3)": 924, >>>>> "T=": 1431445470000000 >>>>> } >>>>> ] >>>>> } >>>>> >>>>> As you can see it doesn't return neither the MAX nor the MIN. Are >>>>> those aggregations defined for type u64? >>>>> >>>>> Best regards, >>>>> Alberto. >>>>> >>>>> Quoting Anish Mehta <[email protected]>: >>>>> >>>>>> Hi Alberto, >>>>>> >>>>>> The ³aggtype² annotation in the sandesh file refers to the Analytics >>>>>> GET >>>>>> API for UVEs. >>>>>> It is unrelated to the Analytics Query API. >>>>>> >>>>>> For the Analytics GET API, we support the following aggregations: >>>>>> aggtype=³sum² (for integers or doubles) >>>>>> aggtype=³union² (for lists of strings or structures) >>>>>> >>>>>> See this blog post for details: >>>>>> >>>>>> >>>>>> >>>>>>http://www.opencontrail.org/operational-state-in-the-opencontrail-sys >>>>>>te >>>>>> m- >>>>>> uv >>>>>> e-user-visible-entities-through-analytics-api/ >>>>>> >>>>>> >>>>>> >>>>>> For the Analytics Query API on StatTables, we support the following >>>>>> aggregations for integers or doubles: >>>>>> SUM >>>>>> MAX >>>>>> MIN >>>>>> >>>>>> For your example, you are trying to use aggregations for StatTables >>>>>> queries. >>>>>> You do not need to use the ³aggtype² annotation at all. >>>>>> If you use SUM, MIN or MAX aggregations in the select part of the >>>>>> query, >>>>>> it should just work. >>>>>> >>>>>> Regards, >>>>>> >>>>>> Anish >>>>>> >>>>>> >>>>>> On 5/11/15, 2:38 AM, "[email protected]" <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> That was a silly mistake, excuse me. It works now! >>>>>>> >>>>>>> Now I have a doubt about aggregation types. I asked before about >>>>>>> which >>>>>>> kind of aggregations there are and it seems that there are SUM, >>>>>>>AVG, >>>>>>> UNION, MIN and MAX at least (correct me if I'm wrong). >>>>>>> >>>>>>> I want to have MIN and MAX along SUM in my structure. Is it >>>>>>>possible? >>>>>>> >>>>>>> I'm trying to do it with the following sandesh file with no >>>>>>>success: >>>>>>> struct DataTrace { >>>>>>> 1: u64 val1 >>>>>>> 2: u64 val2 >>>>>>> 3: u64 val3 (aggtype="sum,min,max") >>>>>>> } >>>>>>> >>>>>>> struct AlbertoAgent { >>>>>>> 1: string name >>>>>>> (key="ObjectAlberto") >>>>>>> 2: optional bool deleted >>>>>>> 3: optional u64 cpu >>>>>>> 4: optional list<DataTrace> dTrace >>>>>>> (tags=".val1,.val2") >>>>>>> } >>>>>>> >>>>>>> uve sandesh AlbertoTrace { >>>>>>> 1: AlbertoAgent data >>>>>>> } >>>>>>> >>>>>>> I haven't seen any error in the process. Where should I check if >>>>>>> there >>>>>>> is an error in this part of the environment? >>>>>>> >>>>>>> Thank you for your support. >>>>>>> >>>>>>> Best regards, >>>>>>> Alberto. >>>>>>> >>>>>>> Quoting Anish Mehta <[email protected]>: >>>>>>> >>>>>>>> Hi Alberto, >>>>>>>> >>>>>>>> You have used "dtrace" instead of "dTrace" for val1 and val2. >>>>>>>> >>>>>>>> As Raj stated in his email, we usually use T= in conjunction with >>>>>>>>at >>>>>>>> least one aggregate column like SUM(xyz) to get time series. >>>>>>>> E.g. [T=60 , SUM(dTrace.val1) , dTrace.val2] will give one time >>>>>>>> series for SUM(dTrace.val1) for each unique value of dTrace.val2. >>>>>>>> >>>>>>>> The API will allow you to specify [T=60 , dTrace.val1]. This will >>>>>>>> return all unique values of dTrace.val1 in each time period. >>>>>>>> >>>>>>>> The API will also allow you to specify [T, SUM(dTrace.val1)]. If >>>>>>>> there are multiple samples of dTrace.val1 for a given T, they will >>>>>>>> get aggregated together. Otherwise, each SUM(dTrace.val1) will >>>>>>>>only >>>>>>>> represent one dTrace.val1, and the query result will be equivalent >>>>>>>> to [T, dTrace.val1] >>>>>>>> >>>>>>>> Please see the Stats blog entry for more details. >>>>>>>> http://www.opencontrail.org/statistics-in-opencontrail-analytics/ >>>>>>>> >>>>>>>> Regards, >>>>>>>> >>>>>>>> Anish >>>>>>>> >>>>>>>> On May 7, 2015, at 6:12 AM, Raj Reddy >>>>>>>> <[email protected]<mailto:[email protected]>> wrote: >>>>>>>> >>>>>>>> You can either select >>>>>>>> "T=" & SUM(xyz) [this gives time series values] *or* >>>>>>>> "T" & XYZ [this gives raw values] >>>>>>>> >>>>>>>> One of the below will work >>>>>>>> "select_fields": [ >>>>>>>> "T=","COUNT(dTrace)", "SUM(dtrace.val1)", >>>>>>>>"SUM(dtrace.val2)" >>>>>>>> ], >>>>>>>> or >>>>>>>> "select_fields": [ >>>>>>>> "T","COUNT(dTrace)", "dtrace.val1", "dtrace.val2" >>>>>>>> ], >>>>>>>> >>>>>>>> - >>>>>>>> Raj >>>>>>>> >>>>>>>> On May 7, 2015, at 4:44 AM, >>>>>>>> <[email protected]<mailto:[email protected]>> >>>>>>>> wrote: >>>>>>>> >>>>>>>> Hi Megh and Anish, >>>>>>>> >>>>>>>> Thanks for your support. >>>>>>>> >>>>>>>> Making the query to the UVE I get the following: >>>>>>>> { >>>>>>>> "AlbertoAgent": { >>>>>>>> "cpu": 31337, >>>>>>>> "dTrace": [ >>>>>>>> { >>>>>>>> "StatTable.AlbertoAgent.dTrace": [ >>>>>>>> { >>>>>>>> "COUNT(dTrace)": 1, >>>>>>>> "SUM(dTrace.val1)": 1, >>>>>>>> "SUM(dTrace.val2)": 2 >>>>>>>> } >>>>>>>> ] >>>>>>>> } >>>>>>>> ] >>>>>>>> } >>>>>>>> } >>>>>>>> >>>>>>>> That message is produced by the following C++ code: >>>>>>>> AlbertoAgent vnAgent; >>>>>>>> >>>>>>>> vnAgent.set_name("lvelasco"); >>>>>>>> vnAgent.set_deleted(false); >>>>>>>> vnAgent.set_cpu(31337); >>>>>>>> >>>>>>>> std::vector<DataTrace> dt(1); >>>>>>>> dt[0].set_val1(1); >>>>>>>> dt[0].set_val2(2); >>>>>>>> vnAgent.set_cpu(31337); >>>>>>>> vnAgent.set_dTrace(dt); >>>>>>>> >>>>>>>> AlbertoTrace::Send(vnAgent); >>>>>>>> >>>>>>>> This code is executed in a loop fashion. >>>>>>>> >>>>>>>> Now I want to get the values produced over time by this agent so I >>>>>>>> did the following query: >>>>>>>> >>>>>>>> { >>>>>>>> "end_time": "now", >>>>>>>> "select_fields": [ >>>>>>>> "T","COUNT(dTrace)", "SUM(dtrace.val1)", "SUM(dtrace.val2)" >>>>>>>> ], >>>>>>>> "sort_fields": [], >>>>>>>> "start_time": "now-60m", >>>>>>>> "table": "StatTable.AlbertoAgent.dTrace", >>>>>>>> "where": [ >>>>>>>> [ >>>>>>>> { >>>>>>>> "name": "name", >>>>>>>> "op": 1, >>>>>>>> "suffix": null, >>>>>>>> "value": "lvelasco", >>>>>>>> "value2": null >>>>>>>> } >>>>>>>> ] >>>>>>>> ] >>>>>>>> >>>>>>>> } >>>>>>>> >>>>>>>> However it only returns me the "T","COUNT(dTrace)" part and not >>>>>>>> val1 and val2. What I'm doing wrong? >>>>>>>> >>>>>>>> Best regards, >>>>>>>> Alberto. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Quoting Megh Bhatt <[email protected]<mailto:[email protected]>>: >>>>>>>> >>>>>>>> Hi Albero, >>>>>>>> trace is a keyword, please change name to data_trace. Also please >>>>>>>> have optional bool deleted as element number 2 and then you should >>>>>>>> be good to go >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>> Megh >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On May 6, 2015, at 6:57 AM, >>>>>>>> "[email protected]<mailto:[email protected]>" >>>>>>>> <[email protected]<mailto:[email protected]>> wrote: >>>>>>>> >>>>>>>> Hi anish, >>>>>>>> >>>>>>>> Thanks for your reply. >>>>>>>> >>>>>>>> In that example, aggregation type "union" is applied over all the >>>>>>>> elements of type ProcessCpuInfo? >>>>>>>> >>>>>>>> I have adapted my structure following your example: >>>>>>>> struct DataTrace { >>>>>>>> 1: u64 val1 >>>>>>>> 2: u64 val2 >>>>>>>> } >>>>>>>> >>>>>>>> struct AlbertoAgent { >>>>>>>> 1: string name >>>>>>>> (key="ObjectAlberto") >>>>>>>> 2: optional u64 cpu >>>>>>>> 3: optional bool deleted >>>>>>>> 4: optional list<DataTrace> trace >>>>>>>> (tags=".val1,.val2") >>>>>>>> } >>>>>>>> >>>>>>>> uve sandesh AlbertoTrace { >>>>>>>> 1: AlbertoAgent data >>>>>>>> } >>>>>>>> >>>>>>>> However there is a syntax error that I'm not able to detect (Error >>>>>>>> message:(last token was 'trace')syntax error). >>>>>>>> >>>>>>>> Am I in the correct way? Where is the error? >>>>>>>> >>>>>>>> Thanks for your support, >>>>>>>> Alberto. >>>>>>>> >>>>>>>> >>>>>>>> Quoting Anish Mehta >>>>>>>><[email protected]<mailto:[email protected]>>: >>>>>>>> >>>>>>>> Hi Alberto, >>>>>>>> >>>>>>>> I assume you are talking about the stats functionality, which is >>>>>>>> turned >>>>>>>> on >>>>>>>> using the "tags" annotation. >>>>>>>> I can help more if you send the sandesh file >>>>>>>> >>>>>>>> Meanwhile, I will demonstrate this feature using Analytics CPU >>>>>>>>Info. >>>>>>>> >>>>>>>> This is the sandesh file: >>>>>>>> >>>>>>>> struct AnalyticsCpuState { >>>>>>>> 1: string name >>>>>>>> (key="ObjectCollectorInfo") >>>>>>>> 2: optional bool deleted >>>>>>>> 3: optional list<cpuinfo.ProcessCpuInfo> cpu_info >>>>>>>> (tags=".module_id,.mem_virt,.cpu_share,.mem_res", aggtype="union") >>>>>>>> } >>>>>>>> >>>>>>>> uve sandesh AnalyticsCpuStateTrace { >>>>>>>> 1: AnalyticsCpuState data >>>>>>>> } >>>>>>>> >>>>>>>> >>>>>>>> The tags annotation will cause the "cpu_info" attribute to be >>>>>>>> recorded >>>>>>>> as >>>>>>>> a Stat Sample. >>>>>>>> This sample can then be queried via the Analytics Query API (the >>>>>>>> same >>>>>>>> API >>>>>>>> used for the object-log) >>>>>>>> The attributes listed in the tags annotation will be used for >>>>>>>> indexing. >>>>>>>> (the "where" part of the query) >>>>>>>> >>>>>>>> >>>>>>>> We implicitly index by the Source of message (Source of the >>>>>>>>Sandesh >>>>>>>> Generator) and the UVE Key. >>>>>>>> These fields are recorded as "name" and "Source". >>>>>>>> The other index fields (as per the tags annotation) in this case >>>>>>>>are >>>>>>>> cpu_info.module_id, cpu_info.mem_virt, cpu_info.cpu_share and >>>>>>>> cpu_info.mem_res. >>>>>>>> There are two other fields that will be present the sample - >>>>>>>> cpu_info.inst_id and cpu_info.module_id. >>>>>>>> These are not in the tags annotation, so we will not index by >>>>>>>>them. >>>>>>>> >>>>>>>> From the command line, you can use the "contrail-stats" command to >>>>>>>> exercise the Analytics Query API for stats. >>>>>>>> Here's an example of usage: >>>>>>>> >>>>>>>> contrail-stats --dtable AnalyticsCpuState.cpu_info --where >>>>>>>> "Source=*" >>>>>>>> --select "T=300" "COUNT(cpu_info)" "name" "cpu_info.module_id" >>>>>>>> "SUM(cpu_info.cpu_share)" --last 30m >>>>>>>> >>>>>>>> This results in the following Analytics API POST request: >>>>>>>> {"start_time": "now-30m", "sort_fields": [], "end_time": "now", >>>>>>>> "select_fields": ["T=300", "COUNT(cpu_info)", "name", >>>>>>>> "cpu_info.module_id", "SUM(cpu_info.cpu_share)"], "table": >>>>>>>> "StatTable.AnalyticsCpuState.cpu_info", "where": [[{"suffix": >>>>>>>>null, >>>>>>>> "value2": null, "name": "Source", "value": "", "op": 7}]]} >>>>>>>> >>>>>>>> You can get more information here: >>>>>>>> http://www.opencontrail.org/statistics-in-opencontrail-analytics/ >>>>>>>> This blog entry will also explain what "T" and "T=" mean. >>>>>>>> >>>>>>>> If you share your sandesh file, I can give you more specific >>>>>>>> pointers. >>>>>>>> Please reach out if you have other questions about this. >>>>>>>> >>>>>>>> Regards, >>>>>>>> >>>>>>>> Anish >>>>>>>> >>>>>>>> >>>>>>>> On 5/4/15, 3:39 AM, >>>>>>>> "[email protected]<mailto:[email protected]>" >>>>>>>> <[email protected]<mailto:[email protected]>> wrote: >>>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> Any clues about how can I make a flow like the ones from Analytics >>>>>>>> node CPU info? I cannot figure how to do it with sandesh. Is it >>>>>>>>done >>>>>>>> using list type? >>>>>>>> >>>>>>>> Best Regards, >>>>>>>> Alberto. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Quoting [email protected]<mailto:[email protected]>: >>>>>>>> >>>>>>>> Hi Anish, >>>>>>>> >>>>>>>> There was a mistake in the changes I did. Now it's correct and I >>>>>>>>get >>>>>>>> this: >>>>>>>> >>>>>>>> agutierrez@Mahalanobis:~/Documents/thesis/OpenContrail/query$ >>>>>>>> ./getQuery.sh /uves/alberto/lvelasco?flat >>>>>>>> % Total % Received % Xferd Average Speed Time Time >>>>>>>> Time Current >>>>>>>> Dload Upload Total Spent >>>>>>>>Left >>>>>>>> Speed >>>>>>>> 100 186 0 186 0 0 147 0 --:--:-- 0:00:01 >>>>>>>> --:--:-- 147 >>>>>>>> { >>>>>>>> "UveVirtualNetworkAgent": { >>>>>>>> "cpu": 31337, >>>>>>>> "cpu_info": [ >>>>>>>> { >>>>>>>> "StatTable.UveVirtualNetworkAgent.cpu_info": [ >>>>>>>> { >>>>>>>> "COUNT(cpu_info)": 1, >>>>>>>> "SUM(cpu_info.att1)": 2090, >>>>>>>> "SUM(cpu_info.att2)": 209 >>>>>>>> } >>>>>>>> ] >>>>>>>> } >>>>>>>> ] >>>>>>>> } >>>>>>>> } >>>>>>>> >>>>>>>> I modified slightly the structure to have a list inside. Now my >>>>>>>> question is the following: How can I make a flow like the ones >>>>>>>>from >>>>>>>> Analytics node CPU info? I have taken a look to some .sandesh >>>>>>>>files >>>>>>>> but I cannot figure out. Also, how can be "T" and "T=" specified? >>>>>>>> >>>>>>>> Thank you for your help. >>>>>>>> >>>>>>>> Best regards, >>>>>>>> Alberto. >>>>>>>> >>>>>>>> Quoting Anish Mehta >>>>>>>><[email protected]<mailto:[email protected]>>: >>>>>>>> >>>>>>>> Hi Alberto. >>>>>>>> >>>>>>>> The schema looks correct. >>>>>>>> >>>>>>>> Both contrail-logs and the Query API use the _STAT_TABLES >>>>>>>>variable. >>>>>>>> They should work if you change that variable as directed. >>>>>>>> Please send me the output of "contrail-logs ?help?, and the >>>>>>>>contents >>>>>>>> of >>>>>>>> your viz/constants.py file. >>>>>>>> >>>>>>>> >>>>>>>> Lets work on the UVE part. >>>>>>>> This is what you can do: >>>>>>>> >>>>>>>> 1. >>>>>>>> Check if the UVE table type is available in >>>>>>>>contrail-analytics-api. >>>>>>>> Look at http://xx.xx.xx.xx:8081/analytics/uves >>>>>>>> Do you see an entry like this?: >>>>>>>> >>>>>>>> { >>>>>>>> href: "http://xx.xx.xx.xx:8081/analytics/uves/albertos";, >>>>>>>> name: ?albertos" >>>>>>>> }, >>>>>>>> >>>>>>>> >>>>>>>> 2. >>>>>>>> Send the UVE from your application. >>>>>>>> When you called InitGenerator, you can specify an Introspect port. >>>>>>>> I?ll assume you used 5555. >>>>>>>> After your application is up, and you have sent your UVE, check >>>>>>>>for >>>>>>>> this: >>>>>>>> http://xx.xx.xx.xx:5555/Snh_SandeshUVETypesReq? >>>>>>>> >>>>>>>> Do you see an entry for you structure ?TestAgent? >>>>>>>> You can go onto the link for TestAgent and the the last values you >>>>>>>> sent. >>>>>>>> >>>>>>>> Also, check your connection to the collector: >>>>>>>> http://xx.xx.xx.xx:5555/Snh_CollectorInfoRequest? >>>>>>>> >>>>>>>> >>>>>>>> 3. >>>>>>>> If all this worked, you should see the UVE at >>>>>>>> http://xx.xx.xx.xx:8081/analytics/uves/albertos >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Regards, >>>>>>>> >>>>>>>> Anish >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 4/24/15, 9:16 AM, >>>>>>>> "[email protected]<mailto:[email protected]>" >>>>>>>> <[email protected]<mailto:[email protected]>> wrote: >>>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I compiled viz.sandesh using "Sandesh --gen py viz.sandesh" >>>>>>>>directly >>>>>>>> and I substituted the .py. I done this using the same structure as >>>>>>>> before but changing the name of the table and following your >>>>>>>> instructions, but I have problems with it. >>>>>>>> >>>>>>>> I get this schema as the table schema: >>>>>>>> { >>>>>>>> "columns": [ >>>>>>>> { >>>>>>>> "datatype": "int", >>>>>>>> "index": false, >>>>>>>> "name": "MessageTS", >>>>>>>> "select": null, >>>>>>>> "suffixes": null >>>>>>>> }, >>>>>>>> { >>>>>>>> "datatype": "string", >>>>>>>> "index": true, >>>>>>>> "name": "ObjectId", >>>>>>>> "select": null, >>>>>>>> "suffixes": null >>>>>>>> }, >>>>>>>> { >>>>>>>> "datatype": "string", >>>>>>>> "index": true, >>>>>>>> "name": "Source", >>>>>>>> "select": null, >>>>>>>> "suffixes": null >>>>>>>> }, >>>>>>>> { >>>>>>>> "datatype": "string", >>>>>>>> "index": true, >>>>>>>> "name": "ModuleId", >>>>>>>> "select": null, >>>>>>>> "suffixes": null >>>>>>>> }, >>>>>>>> { >>>>>>>> "datatype": "string", >>>>>>>> "index": true, >>>>>>>> "name": "Messagetype", >>>>>>>> "select": null, >>>>>>>> "suffixes": null >>>>>>>> }, >>>>>>>> { >>>>>>>> "datatype": "string", >>>>>>>> "index": false, >>>>>>>> "name": "ObjectLog", >>>>>>>> "select": null, >>>>>>>> "suffixes": null >>>>>>>> }, >>>>>>>> { >>>>>>>> "datatype": "string", >>>>>>>> "index": false, >>>>>>>> "name": "SystemLog", >>>>>>>> "select": null, >>>>>>>> "suffixes": null >>>>>>>> } >>>>>>>> ], >>>>>>>> "type": "OBJECT" >>>>>>>> } >>>>>>>> >>>>>>>> Does this make sense? >>>>>>>> >>>>>>>> Moreover, the following thing happens: >>>>>>>> contrail-logs --object-type alberto >>>>>>>> HTTP error code: 404 >>>>>>>> >>>>>>>> It seems to not be available as UVE. >>>>>>>> >>>>>>>> >>>>>>>> Also I have a question regarding to _STAT_TABLES variable in >>>>>>>> viz.sandesh. If I define the structure in this table, will it be >>>>>>>> available in the query part of the web? >>>>>>>> >>>>>>>> Thank you for your assistance, >>>>>>>> Alberto. >>>>>>>> >>>>>>>> Quoting [email protected]<mailto:[email protected]>: >>>>>>>> >>>>>>>> Hi Anish, >>>>>>>> >>>>>>>> Thank you for your reply. I'm having the following problem when >>>>>>>> compiling: >>>>>>>> >>>>>>>> ... >>>>>>>> Running virtualenv with interpreter testroot/bin/python >>>>>>>> New python executable in analytics_test/bin/python >>>>>>>> Installing >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>Setuptools......................................................... >>>>>>>>.. >>>>>>>> . >>>>>>>> ... >>>>>>>> ..............................done. >>>>>>>> Installing >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>Pip................................................................ >>>>>>>>.. >>>>>>>> . >>>>>>>> ... >>>>>>>> >>>>>>>>..............................................................done. >>>>>>>> /bin/bash -c "source >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>/home/agutierrez/workspace/Contrail/opencontrail2/build/debug/analy >>>>>>>>ti >>>>>>>> c >>>>>>>> s_t >>>>>>>> est/bin/activate; pip install >>>>>>>> --download-cache=/tmp/cache/agutierrez/systemless_test >>>>>>>> greenlet==0.4.1 >>>>>>>> gevent==0.13.8 eventlet==0.9.17 testtools==0.9.21 fixtures==0.3.12 >>>>>>>> requests>=1.1.0 lxml==2.3.3 geventhttpclient==1.0a >>>>>>>> prettytable==0.7.2 >>>>>>>> psutil==0.4.1 redis==2.7.1 xmltodict==0.2 thrift==0.8.0 >>>>>>>> bottle==0.11.6 >>>>>>>> mock==1.0.1 kafka-python==0.9.2 kazoo==1.3.1 >>>>>>>> stevedore" >>>>>>>> scons: *** [build/debug/opserver/test/greenlet] Error 1 >>>>>>>> scons: building terminated because of errors. >>>>>>>> >>>>>>>> Do you have any idea that what can be the problem or if there is >>>>>>>>any >>>>>>>> log file that could provide more information about the error? >>>>>>>> >>>>>>>> >>>>>>>> Best regards, >>>>>>>> Alberto. >>>>>>>> >>>>>>>> Quoting Anish Mehta >>>>>>>><[email protected]<mailto:[email protected]>>: >>>>>>>> >>>>>>>> Hi Alberto, >>>>>>>> >>>>>>>> Adding your own table is possible. >>>>>>>> You need to add some entries to >>>>>>>> /controller/src/analytics/viz.sandesh >>>>>>>> It does not need the collector to be rebuilt, but it does require >>>>>>>> rebuilding contrail-analytics-api. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> I'm going to assume the table name "ObjectAlberto" (instead of >>>>>>>> "ObjectTableName") >>>>>>>> >>>>>>>> I'm going to make this UVE accessible via the Analytics API as >>>>>>>> follows: >>>>>>>> http://xx.xx.xx.xx:8081/analytics/uves/albertos >>>>>>>> >>>>>>>> I'm going to make the ObjectLog visible via the Analytics Query >>>>>>>>API >>>>>>>> as >>>>>>>> well (as you mentioned in your email) >>>>>>>> Based on this API, from the command line of the analytics node you >>>>>>>> will be >>>>>>>> able to do this: >>>>>>>> contrail-logs --object-type alberto >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> 1. Changes in controller/src/analytics/viz.sandesh >>>>>>>> >>>>>>>> const string ALBERTO_TABLE = "ObjectAlberto" >>>>>>>> const string ALBERTO_DISPLAY_NAME = "Alberto" >>>>>>>> ... >>>>>>>> const map<string, string> UVE_MAP = { >>>>>>>> ... >>>>>>>> "alberto" : ALBERTO_TABLE, >>>>>>>> } >>>>>>>> ... >>>>>>>> const map<string, objtable_info> _OBJECT_TABLES = { >>>>>>>> ALBERTO_TABLE : { >>>>>>>> 'objtable_display_name' : ALBERTO_DISPLAY_NAME >>>>>>>> 'log_query_name' : "alberto" >>>>>>>> } >>>>>>>> } >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> 2. Rebuild contrail-analytics-api >>>>>>>> sb/controller/src$ scons -u opserver >>>>>>>> >>>>>>>> >>>>>>>> 3. Install and test on target >>>>>>>> You should rebuild the contrail-analytics package and install it >>>>>>>>on >>>>>>>> your >>>>>>>> target. >>>>>>>> >>>>>>>> For this kind of change, there is an alternative installation >>>>>>>> method as >>>>>>>> well, given below: >>>>>>>> >>>>>>>> This is the generated file that will change when you rebuild: >>>>>>>> sb/controller/src$ ls >>>>>>>> ../../build/debug/opserver/opserver/sandesh/viz/constants.py >>>>>>>> >>>>>>>> You can copy it onto your analytics node. >>>>>>>> It will be at a location such as: >>>>>>>> /usr/lib/python2.7/dist-packages/opserver/sandesh/viz/constants.py >>>>>>>> >>>>>>>> After overwriting the file (you should take a backup first, just >>>>>>>>in >>>>>>>> case), >>>>>>>> restart contrail-analytics-api: >>>>>>>> service contrail-analytics-api restart >>>>>>>> >>>>>>>> >>>>>>>> 4. Test the UVE Analytics API and Analytics Query API after >>>>>>>>starting >>>>>>>> your >>>>>>>> application and connecting to the collector. >>>>>>>> >>>>>>>> >>>>>>>> Regards, >>>>>>>> >>>>>>>> Anish >>>>>>>> >>>>>>>> >>>>>>>> On 4/23/15, 1:34 AM, >>>>>>>> "[email protected]<mailto:[email protected]>" >>>>>>>> <[email protected]<mailto:[email protected]>> >>>>>>>> wrote: >>>>>>>> >>>>>>>> Hello Anish, >>>>>>>> >>>>>>>> Thanks for your reply. What we are going to do is a little >>>>>>>> application >>>>>>>> that sends it's own metrics to the collector so we can retrieve >>>>>>>> that >>>>>>>> data using de Analytics API. >>>>>>>> >>>>>>>> For example: >>>>>>>> >>>>>>>> struct TestAgent { >>>>>>>> 1: string name(key="ObjectTableName") >>>>>>>> 2: optional i32 valueToBeSent1 >>>>>>>> } >>>>>>>> >>>>>>>> uve sandesh TestTrage { >>>>>>>> 1: TestAgent data; >>>>>>>> } >>>>>>>> >>>>>>>> This kind of message should be stored in some way in Cassandra >>>>>>>> databases and be available to the API users. I understand that >>>>>>>> (key="ObjectTableName") makes reference to a new table with that >>>>>>>> name >>>>>>>> that can be queried from the API. For me that is the concept of >>>>>>>> "custom table", tables that are administrator/developer defined >>>>>>>> different from base contrail tables. >>>>>>>> >>>>>>>> It would be retrieved with a POST query to the API similar to >>>>>>>>this: >>>>>>>> { >>>>>>>> "end_time": "now", >>>>>>>> "select_fields": [ >>>>>>>> "valueToBeSent1" >>>>>>>> ], >>>>>>>> "sort_fields": [], >>>>>>>> "start_time": "now-60m", >>>>>>>> "table": "ObjectTableName", >>>>>>>> "where": [ >>>>>>>> [ >>>>>>>> { >>>>>>>> "name": "Source", >>>>>>>> "op": 1, >>>>>>>> "suffix": null, >>>>>>>> "value": "GeneratorApplication", >>>>>>>> "value2": null >>>>>>>> } >>>>>>>> ] >>>>>>>> ] >>>>>>>> } >>>>>>>> >>>>>>>> Is this possible? If it is, do I have to do any step apart from >>>>>>>> send >>>>>>>> the structure using the code generated by Sandesh compiler? >>>>>>>> >>>>>>>> If it is not possible, is there any workaround? The only way that >>>>>>>> comes to my mind is that we could query directly to message table, >>>>>>>> filter the messages we don't want and process the XMLs but that >>>>>>>> solution wouldn't be clean. >>>>>>>> >>>>>>>> >>>>>>>> I hope I explained it correctly. Thank you for your assistance. >>>>>>>> >>>>>>>> Best regards, >>>>>>>> Alberto Guti?rrez. >>>>>>>> >>>>>>>> Quoting Anish Mehta >>>>>>>><[email protected]<mailto:[email protected]>>: >>>>>>>> >>>>>>>> >>>>>>>> The schemas for new Sandesh types do not need to be compiled or >>>>>>>> loaded >>>>>>>> onto the Collector. >>>>>>>> The collector gets XML-encoded messages from the generators and >>>>>>>> stores >>>>>>>> them according to their sandesh types and the contained >>>>>>>> annotations. >>>>>>>> >>>>>>>> Sandesh Types and how they work are explained here: >>>>>>>> http://www.opencontrail.org/sandesh-a-sdn-analytics-interface/ >>>>>>>> >>>>>>>> >>>>>>>> I^1m not sure what you mean by ^3custom tables^2. >>>>>>>> Please send us a sample sandesh file and describe what you are >>>>>>>> trying to >>>>>>>> do, and we can discuss further. >>>>>>>> >>>>>>>> >>>>>>>> Regards, >>>>>>>> >>>>>>>> Anish >>>>>>>> >>>>>>>> On 4/22/15, 6:53 AM, >>>>>>>> "[email protected]<mailto:[email protected]>" >>>>>>>> <[email protected]<mailto:[email protected]>> >>>>>>>> wrote: >>>>>>>> >>>>>>>> Dear all, >>>>>>>> >>>>>>>> When creating a new sandesh structure, do I need to send somehow >>>>>>>> the >>>>>>>> schema to the collectors so they will understand it or it will be >>>>>>>> added automatically with the first message it recieves? >>>>>>>> >>>>>>>> >>>>>>>> Thanks for your attention, >>>>>>>> Alberto. >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Dev mailing list >>>>>>>> [email protected]<mailto:[email protected]> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> http://lists.opencontrail.org/mailman/listinfo/dev_lists.opencontr >>>>>>>> ail >>>>>>>> .or >>>>>>>> g >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Dev mailing list >>>>>>>> [email protected]<mailto:[email protected]> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>http://lists.opencontrail.org/mailman/listinfo/dev_lists.opencontra >>>>>>>>il >>>>>>>> . >>>>>>>> org >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Dev mailing list >>>>>>>> [email protected]<mailto:[email protected]> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>http://lists.opencontrail.org/mailman/listinfo/dev_lists.opencontra >>>>>>>>il >>>>>>>> .o >>>>>>>> rg >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Dev mailing list >>>>>>>> [email protected]<mailto:[email protected]> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>http://lists.opencontrail.org/mailman/listinfo/dev_lists.opencontra >>>>>>>>il >>>>>>>> .o >>>>>>>> rg >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Dev mailing list >>>>>>>> [email protected]<mailto:[email protected]> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>http://lists.opencontrail.org/mailman/listinfo/dev_lists.opencontra >>>>>>>>il >>>>>>>> .o >>>>>>>> rg >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>>> >>> >> >> > _______________________________________________ Dev mailing list [email protected] http://lists.opencontrail.org/mailman/listinfo/dev_lists.opencontrail.org
