On Fri, Jun 22, 2012 at 2:29 PM, Tharindu Mathew <thari...@wso2.com> wrote:
> This can be a useful feature for realtime requirements. But we need to
> follow some proper convention as this changes the event stream definition
> This is then an extensive change, and we are close to feature freezing.
>
> On Fri, Jun 22, 2012 at 2:15 PM, Amani Soysa <am...@wso2.com> wrote:
>
>>
>>
>> On Fri, Jun 22, 2012 at 2:02 PM, Deependra Ariyadewa <d...@wso2.com>wrote:
>>
>>>
>>>
>>> On Fri, Jun 22, 2012 at 1:45 PM, Amani Soysa <am...@wso2.com> wrote:
>>>
>>>>
>>>>
>>>> On Fri, Jun 22, 2012 at 1:24 PM, Tharindu Mathew <thari...@wso2.com>wrote:
>>>>
>>>>> This will be useful for folks who want real time data access, but BAM
>>>>> is not designed to be real time. I don't want the Agent API to be specific
>>>>> to Cassandra, either.
>>>>>
>>>>> There should be a clean way to do this. How did you decide to do it
>>>>> this way? Was there a discussion?
>>>>>
>>>> Yes there was a discussion on this some time back on Architecture
>>>> -"RFC: Architecture for Stratos Log Processing" where we decided to push
>>>> logs to bam event receiver through the publisher and view logs using hector
>>>> api.
>>>>
>>>
>>> Initially we tried to use flume as the Stratos log collector/manager but
>>> we stop flume evaluation because BAM's capability to cover the same use
>>> case.
>>>
>>> There are several workarounds like create relevant keyspaces in the
>>> tenant creation or create extended event receiver only for logging.
>>>
>> If we do it in the tenant creation time their will be around 10 keyspaces
>> per each tenant (given that for each server, we create a keyspace as I
>> explained earlier) So even a user doesn't use a particular server their
>> will be a keyspace for it. So if there are 1000 tenants there will be 10
>> 000 keyspace (even if some keyspaces are not used at all) So I think its
>> better to create keyspaces when ever logs are publishing to that particular
>> keyspace.
>>
> 10 keryspaces per tenant?? Are you sure that's right...
>
Yes for each sever (maybe more than 10 :) )
ie - data services,appserver,esb,mb,cep,bps,brs etc all the products we
offer for stratos deployment , because we store logs for each server in
different keyspace(if not we have to keep everything in a single keyspace
per tenant and it will be an expensive search when we filter the logs from
the server level because we give users server specific logs)
In our earlier syslog implementation we divided logs per each server for
fast result (as logs generates). Thats why its very important to not to
create keyspaces if users are not using a particular product.
>
>>> Thanks,
>>>
>>> Deependra.
>>>
>>>>
>>>>> On Fri, Jun 22, 2012 at 8:45 AM, Amani Soysa <am...@wso2.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Currently we are sending LogEvent data through bam data publisher to
>>>>>> bam event receiver using a custom log4j appender. And we retrieve logs
>>>>>> using the hector API for the carbon log viewer. However, we need to have
>>>>>> secondary indexes for several columns so that we can filter log
>>>>>> information
>>>>>> for a given column ( such as date, applicationName, priority,logger etc)
>>>>>> when creating the data publisher (keyspace). From the current Bam Data
>>>>>> publisher implementation we cannot do secondary indexing all we can do is
>>>>>> define the column name and the data type of that column, and reciver
>>>>>> creates the keyspaces for given columns with their data types.
>>>>>>
>>>>>> streamId = dataPublisher.defineEventStream("{" + "
>>>>>> 'name':'org.wso2.carbon.logging.$tenantId.$serverName',"
>>>>>> + " 'version':'1.0.0'," + " 'nickName':
>>>>>> 'Logs',"
>>>>>> + " 'description': 'Logging Event'," + "
>>>>>> 'metaData':["
>>>>>> + " {'name':'clientType','type':'STRING'}"
>>>>>> + " ],"
>>>>>> + " 'payloadData':["
>>>>>> + " {'name':'tenantID','type':'
>>>>>> STRING'},"
>>>>>> + " {'name':'serverName','type':'
>>>>>> STRING'},"
>>>>>> + " {'name':'appName','type':'
>>>>>> STRING'},"
>>>>>> + " {'name':'logTime','type':'LONG'}
>>>>>> ,"
>>>>>> + " {'name':'logger','type':'
>>>>>> STRING'},"
>>>>>> + " {'name':'priority','type':'
>>>>>> STRING'},"
>>>>>> + " {'name':'message','type':'
>>>>>> STRING'},"
>>>>>> + " {'name':'ip','type':'STRING'},"
>>>>>> + " {'name':'stacktrace','type':'
>>>>>> STRING'},"
>>>>>> + " {'name':'instance','type':'
>>>>>> STRING'}"
>>>>>> + " ]"
>>>>>> + "}");
>>>>>>
>>>>>> Is it possible to have a cassandra specific event receiver (for
>>>>>> logging purposes) so that we can create key spaces with secondary
>>>>>> indexes?[
>>>>>> 1 <https://wso2.org/jira/browse/CARBON-13468>] and it will create
>>>>>> keyspaces when ever logs are published . Or do we need to create
>>>>>> keyspaces
>>>>>> at tenant creation time?. For a given tenant we need to create several
>>>>>> keyspaces, depending on the server (and if possible for applications as
>>>>>> well so we can have better performance when viewing logs).
>>>>>> ie - keyspace1 - org_wso2_logging_tenant1_application_server (store
>>>>>> AS specific logs)
>>>>>> keyspace2 -
>>>>>> org_wso2_logging_tenant1_data_services_server (store DSS specific logs)
>>>>>>
>>>>>> Please note that we cannot use BAM analytics to view logs because we
>>>>>> need a real time log-viwer.
>>>>>>
>>>>>> [1] - https://wso2.org/jira/browse/CARBON-13468
>>>>>>
>>>>>> Regards,
>>>>>> Amani
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>>
>>>>> Tharindu
>>>>>
>>>>> blog: http://mackiemathew.com/
>>>>> M: +94777759908
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Deependra Ariyadewa
>>> WSO2, Inc. http://wso2.com/ http://wso2.org
>>>
>>> email d...@wso2.com; cell +94 71 403 5996 ;
>>> Blog http://risenfall.wordpress.com/
>>> PGP info: KeyID: 'DC627E6F'
>>>
>>>
>>
>
>
> --
> Regards,
>
> Tharindu
>
> blog: http://mackiemathew.com/
> M: +94777759908
>
>
_______________________________________________
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
