[
https://issues.apache.org/jira/browse/METRON-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16680537#comment-16680537
]
ASF GitHub Bot commented on METRON-1834:
----------------------------------------
Github user anandsubbu commented on the issue:
https://github.com/apache/metron/pull/1242
> The last remaining bit before I merge this will be sharing some results
around the performance/regression testing performed by @anandsubbu.
I completed a set of performance comparison tests between the REST and
Transport clients. Here's the update.
### Note on the Testing Done
* The performance regression comparison tests are run on a multi-node
VM-based environment by:
a) Running a set of tests using the ES REST Client
b) Swap out the REST client with the Transport client, and re-run the same
set of tests.
### Observations
1) The indexing rate numbers with the REST client is similar to that of the
Transport client.
2) I had noticed storm rebalance error messages for the kafkaSpout for both
REST and Transport client runs, which required to adjust the values of
`request.timeout.ms` and `session.timeout.ms` in the ES flux file under
'kafkaProps'. The errors were resolved after making these changes.
3) For the REST client, I observed listener timeout exceptions with the
default settings. I had to increase max retry and socket timeout values in
global config as follows, in order to resolve the exceptions.
```
"es.client.settings" : {
"max.retry.timeout.millis" : "60000",
"socket.timeout.millis" : "60000"
}
```
4) For both cases, the kafkaSpout complete latency was seen to be on the
higher side (~30 secs)
5) Lastly, I noticed that there were a number of failed tuples seen in the
kafkaSpout of the RA Indexing topology. I am continuing to dig more into this,
but since this was seen for both the REST and Transport client tests, I am
suspecting it could be an anomaly with the test environment, or a different
issue common to both REST/Transport clients.
Based on the above, it is seen that there is no immediate regression in
performance b/w the REST and Transport client.
+1, great work @mmiklavc
> Migrate Elasticsearch from TransportClient to new Java REST API
> ---------------------------------------------------------------
>
> Key: METRON-1834
> URL: https://issues.apache.org/jira/browse/METRON-1834
> Project: Metron
> Issue Type: Improvement
> Reporter: Michael Miklavcic
> Assignee: Michael Miklavcic
> Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
