Hi All,
Due to the limitation of APIM analytics scaling, we were able to come up
with a solution based on the discussion we had in [1]. To scale APIM
analytics, which has configured alert, required following deployment.
Deployment is partitioning by analytics data, and we partitioned data by
tenant domain. So this solution works and can be scale when there are
multiple tenants in the deployment. But if it is a
single-tenant deployment and still need scaling, we can think of other
parameters like API name or provider to partition data.
[image: Untitled drawing.png]
GWs are configuring to publish events to receiver nodes. It is an
active-active deployment to support HA. Receiver nodes are
partitioned data, according to the tenant domain and forward to
the appropriate active-passive worker cluster.
To support this deployment, we need to apply configuration(siddhi app) to
receiver nodes. For each tenant(or partitioning group), there should be a
siddhi app where it filters the events by defined field name(tenant
domain). Also, the event sink configuration needs to define with its worker
cluster URLs.
The following is a sample configuration for a single tenant. Bold cloured
parameters need to change according to the tenant. So connection URLs and
tenant domains need to change in this configuration.
@App:name("APIM_EVENT_PUBLISHER_wso2_com")
@App:description("Receive the streams and passes it for inmemory sinks.")
-- org.wso2.apimgt.statistics.request:3.0.0
@sink(type='wso2event',
wso2.stream.id='org.wso2.apimgt.statistics.request:3.0.0',
*url="tcp://localhost:7613", auth.url="ssl://localhost:7713",*
protocol="thrift", username="admin", password="admin", mode="non-blocking"
, @map(type='wso2event'))
define stream OutputRequestStream (meta_clientType string,
applicationConsumerKey string, applicationName string, applicationId string,
applicationOwner string, apiContext string, apiName string, apiVersion
string, apiResourcePath string, apiResourceTemplate string,
apiMethod string, apiCreator string, apiCreatorTenantDomain string, apiTier
string, apiHostname string, username string, userTenantDomain string,
userIp string, userAgent string, requestTimestamp long, throttledOut bool,
responseTime long, serviceTime long, backendTime long, responseCacheHit
bool, responseSize long, protocol string, responseCode int, destination
string, securityLatency long, throttlingLatency long, requestMedLat long,
responseMedLat long, backendLatency long, otherLatency long, gatewayType
string, label string);
@source(type = 'inMemory' , topic = 'APIM_REQUEST')
define stream InComingRequestStream (meta_clientType string,
applicationConsumerKey string, applicationName string, applicationId string,
applicationOwner string, apiContext string, apiName string, apiVersion
string, apiResourcePath string, apiResourceTemplate string,
apiMethod string, apiCreator string, apiCreatorTenantDomain string, apiTier
string, apiHostname string, username string, userTenantDomain string,
userIp string, userAgent string, requestTimestamp long, throttledOut bool,
responseTime long, serviceTime long, backendTime long, responseCacheHit
bool, responseSize long, protocol string, responseCode int, destination
string, securityLatency long, throttlingLatency long, requestMedLat long,
responseMedLat long, backendLatency long, otherLatency long, gatewayType
string, label string);
-- org.wso2.apimgt.statistics.throttle:3.0.0
@sink(type='wso2event',
wso2.stream.id='org.wso2.apimgt.statistics.throttle:3.0.0',
*url="tcp://localhost:7613", auth.url="ssl://localhost:7713",*
protocol="thrift", username="admin", password="admin", mode="non-blocking"
, @map(type='wso2event'))
define stream OutputThrottledOutStream(meta_clientType string, username
string, userTenantDomain string, apiName string, apiVersion string,
apiContext string, apiCreator string, apiCreatorTenantDomain string,
applicationId string, applicationName string, subscriber string,
throttledOutReason string, gatewayType string, throttledOutTimestamp long,
hostname string);
@source(type = 'inMemory' , topic = 'ThrottledOutStream')
define stream InComingThrottledOutStream(meta_clientType string, username
string, userTenantDomain string, apiName string, apiVersion string,
apiContext string, apiCreator string, apiCreatorTenantDomain string,
applicationId string, applicationName string, subscriber string,
throttledOutReason string, gatewayType string, throttledOutTimestamp long,
hostname string);
-- org.wso2.apimgt.statistics.fault:3.0.0
@sink(type='wso2event',
wso2.stream.id='org.wso2.apimgt.statistics.fault:3.0.0',
*url="tcp://localhost:7613", auth.url="ssl://localhost:7713",*
protocol="thrift", username="admin", password="admin", mode="non-blocking"
, @map(type='wso2event'))
define stream OutputFaultStream(meta_clientType string,
applicationConsumerKey string, apiName string, apiVersion string,
apiContext string,
apiResourcePath string, apiMethod string, apiCreator string, username
string, userTenantDomain string, apiCreatorTenantDomain string,
hostname string, applicationId string, applicationName string, protocol
string, errorCode string, errorMessage string, requestTimestamp long);
@source(type = 'inMemory' , topic = 'FaultStream')
define stream IncomingFaultStream(meta_clientType string,
applicationConsumerKey string, apiName string, apiVersion string,
apiContext string,
apiResourcePath string, apiMethod string, apiCreator string, username
string, userTenantDomain string, apiCreatorTenantDomain string,
hostname string, applicationId string, applicationName string, protocol
string, errorCode string, errorMessage string, requestTimestamp long);
from InComingRequestStream*[apiCreatorTenantDomain == "wso2.com
<http://wso2.com>"]*
select *
insert into OutputRequestStream;
from InComingThrottledOutStream*[apiCreatorTenantDomain == "wso2.com
<http://wso2.com>"]*
select *
insert into OutputThrottledOutStream;
from IncomingFaultStream*[apiCreatorTenantDomain == "wso2.com
<http://wso2.com>"]*
select *
insert into OutputFaultStream;
Can you please review this solution and let us know if there are any
concerns.
[1] [API Analytics] Propose One Deployment Architecture That Works For The
Full Solution
Thanks and Regards
--
Rukshan C. Premathunga | Associate Technical Lead | WSO2 Inc.
(m) +94711822074 | (w) +94112145345 | Email: ruks...@wso2.com
GET INTEGRATION AGILE
Integration Agility for Digitally Driven Business
_______________________________________________
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
