I think elasticsearch-hadoop has a good number of options to tune and those are documented well too. (See https://www.elastic.co/guide/en/elasticsearch/hadoop/current/configuration.html )
One thing I see in favor of the elasticsearch-hadoop is that it provides batching without Trident. This could be a very good thing for someone not using Trident. ES-Hadoop also does a node discovery of all the ES-nodes and then writes data to only primary shards (which avoids a jump from replica-shards to primary-shards). ES-Hadoop supported routing until 2.x version of ES and there is a bug to make it work in 2.x version as well. ES-Hadoop also provides a good support of time-based index-rolling which is great for logging-type use-cases. Some more features include authentication, SSL, proxy etc. I haven't looked much closely into storm-elasticsearch to see if it provides the same functionality. We should compare the above features and run some performance tests to compare the two. Disclaimer: I am just a simple user of ES-Hadoop and not associated with it in any way. Would love to switch to some other bolt if its better or more performant. On Sat, Mar 26, 2016 at 8:11 PM, Jungtaek Lim <kabh...@gmail.com> wrote: > Lakshmanan, > > We looked over elasticsearch-hadoop when we adopted storm-elasticsearch, > and elasticsearch-hadoop has only basic feature at that time. > (No trident implementation) > Please refer https://github.com/apache/storm/pull/573 to track our > discussion on this module. > > Thanks! > Jungtaek Lim (HeartSaVioR) > > 2016년 3월 27일 (일) 오전 3:12, Lakshmanan Muthuraman <lakshma...@tokbox.com>님이 > 작성: > >> We have been using elasticsearch-hadoop to write to ElasticSearch from >> Storm. It looks good so far. Any thoughts on adopting ElasticSearch Hadoop >> as part of Storm external rather than trying to write and maintain our own >> bolt in Storm Project. >> >> Any thoughts? >> >> On Tue, Mar 22, 2016 at 2:19 PM, Aaron.Dossett <aaron.doss...@target.com> >> wrote: >> >>> No, we hadn’t looked at that, but once we are streaming into elastic at >>> scale (not there yet) I would be interested to compare. >>> >>> From: Tech Id <tech.login....@gmail.com> >>> Reply-To: "user@storm.apache.org" <user@storm.apache.org> >>> Date: Monday, March 21, 2016 at 1:21 PM >>> To: "user@storm.apache.org" <user@storm.apache.org> >>> Subject: Re: external/storm-elasticsearch - upgrade requested >>> >>> Thanks Aaron, >>> >>> Did you have a chance to compare the elasticsearch-hadoop ( >>> https://www.elastic.co/guide/en/elasticsearch/hadoop/current/storm.html) >>> with the storm-elasticsearch ( >>> https://github.com/apache/storm/tree/master/external/storm-elasticsearch) >>> ? >>> >>> Former uses REST client while the latter uses TransportClient. >>> >>> It would be interesting to know some performance numbers between them. >>> >>> Thanks ! >>> >>> On Mon, Mar 21, 2016 at 5:40 AM, Aaron.Dossett <aaron.doss...@target.com >>> > wrote: >>> >>>> Hi, >>>> >>>> The best way to make feature requests like this is via the Apache Jira. >>>> (https://issues.apache.org/jira/browse/STORM/) >>>> >>>> As it happens, my team (at Target) has adapted the elastic search bolt >>>> for 2.X and we are using it in production. We opened a JIRA ( >>>> https://issues.apache.org/jira/browse/STORM-1475) to track that and >>>> contribute our change back to the project. We’re cleaning a couple of >>>> things up right now, but I’m hopeful we will contribute it back soon. >>>> >>>> Your second request could be a nice feature enhancement, can you open a >>>> JIRA for it? In my experience, these external bolts are a really nice way >>>> to get involved with the project as well. You could try adding a couple of >>>> features and contributing those. >>>> >>>> Thanks! >>>> >>>> From: Tech Id <tech.login....@gmail.com> >>>> Reply-To: "user@storm.apache.org" <user@storm.apache.org> >>>> Date: Friday, March 18, 2016 at 9:45 PM >>>> To: "user@storm.apache.org" <user@storm.apache.org> >>>> Subject: external/storm-elasticsearch - upgrade requested >>>> >>>> Hi, >>>> >>>> I see that the version of elastic-search used >>>> in external/storm-elasticsearch/pom.xml is quite old (1.6.0) while the >>>> latest elastic-search is 2.2.0 >>>> >>>> 2.x version is not compatible with 1.x version of elastic-search and so >>>> I request you to upgrade. >>>> >>>> Also, current storm-elasticsearch uses TransportClient of >>>> elastic-search but does not expose many of the useful options in the >>>> TransportClient (like routing). >>>> >>>> Request you to expose them to the users too. >>>> >>>> >>>> Thanks >>>> Tid >>>> >>> >>> >>
