I think this will NOT work, it is not supported on the most recent releases of flume and ES
On Mon, May 2, 2016 at 4:27 PM, Arif,Mubaraka <[email protected]> wrote: > Hello Iain, > > > > They are in my plugins.d dir of flume. Do I also need to have them in > flume lib folder ? > > > > Thanks, > > Muby > > > > *From:* iain wright [mailto:[email protected]] > *Sent:* Monday, May 02, 2016 4:25 PM > *To:* [email protected] > > *Subject:* Re: Flume - elasticsearch sink fails > > > > Hi Muby, > > > > Are the ES and lucene-core jars in your flume lib folder? > > > > > -- > > Iain Wright > > > > This email message is confidential, intended only for the recipient(s) > named above and may contain information that is privileged, exempt from > disclosure under applicable law. If you are not the intended recipient, do > not disclose or disseminate the message to anyone except the intended > recipient. If you have received this message in error, or are not the named > recipient(s), please immediately notify the sender by return email, and > delete all copies of this message. > > > > On Mon, May 2, 2016 at 1:47 PM, Arif,Mubaraka <[email protected]> > wrote: > > Downgraded to Elasticsearch 1.7.5 and still get the same error : > > > > RROR > > org.apache.flume.lifecycle.LifecycleSupervisor > > Unable to start SinkRunner: { > policy:org.apache.flume.sink.DefaultSinkProcessor@730d4fcf counterGroup:{ > name:null counters:{} } } - Exception follows. > > java.lang.NoSuchMethodError: > org.elasticsearch.common.transport.InetSocketTransportAddress.<init>(Ljava/lang/String;I)V > > at > org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.configureHostnames(ElasticSearchTransportClient.java:143) > > at > org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.<init>(ElasticSearchTransportClient.java:77) > > at > org.apache.flume.sink.elasticsearch.client.ElasticSearchClientFactory.getClient(ElasticSearchClientFactory.java:48) > > at > org.apache.flume.sink.elasticsearch.ElasticSearchSink.start(ElasticSearchSink.java:357) > > at > org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46) > > at org.apache.flume.SinkRunner.start(SinkRunner.java:79) > > at > org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251) > > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:745) > > > > *From:* iain wright [mailto:[email protected]] > *Sent:* Monday, May 02, 2016 2:52 PM > *To:* [email protected] > *Cc:* Mabrito,Greg > *Subject:* Re: Flume - elasticsearch sink fails > > > > Hi, > > > > These look relevant to your goals: > > > > https://github.com/elastic/elasticsearch/issues/14187 > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_elastic_elasticsearch_issues_14187&d=CwMFaQ&c=RI9dKKMRNVHr9NFa7OQiQw&r=dUN85GiSQZVDs0gTK4x1mSiAdXTZ-7F0KzGt2fcse38&m=31csM049OHrrKBoryULOX8hgtSb6qxzPl-6Z_c_-N0A&s=p6q-rN0ma3I-5GVFu0e_CASXMGrDeforSajvFopvNic&e=> > > > > https://github.com/lucidfrontier45/ElasticsearchSink2 > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_lucidfrontier45_ElasticsearchSink2&d=CwMFaQ&c=RI9dKKMRNVHr9NFa7OQiQw&r=dUN85GiSQZVDs0gTK4x1mSiAdXTZ-7F0KzGt2fcse38&m=31csM049OHrrKBoryULOX8hgtSb6qxzPl-6Z_c_-N0A&s=RSyBce-WtursnyucSVeewjWRYtSDC1dkIbXOpE3boRc&e=> > > > > HTH, > > > -- > > Iain Wright > > > > This email message is confidential, intended only for the recipient(s) > named above and may contain information that is privileged, exempt from > disclosure under applicable law. If you are not the intended recipient, do > not disclose or disseminate the message to anyone except the intended > recipient. If you have received this message in error, or are not the named > recipient(s), please immediately notify the sender by return email, and > delete all copies of this message. > > > > On Mon, May 2, 2016 at 12:44 PM, Arif,Mubaraka <[email protected]> > wrote: > > Hello, > > > > Trying to flume tweets from Tibco-JMS source into elastic search sink. > > Running Elasticsearch on our edge nodes independent of Hadoop. > > Running cloudera Hadoop 5.4.8 > > > > On running the error is : > > > > May 2, 2:15:14.047 PM > > INFO > > org.apache.flume.instrumentation.MonitoredCounterGroup > > Component type: SINK, name: k1 started > > May 2, 2:15:14.050 PM > > WARN > > org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient > > [arcapl0021707:9200] > > May 2, 2:15:14.052 PM > > ERROR > > org.apache.flume.lifecycle.LifecycleSupervisor > > Unable to start SinkRunner: { > policy:org.apache.flume.sink.DefaultSinkProcessor@50482278 counterGroup:{ > name:null counters:{} } } - Exception follows. > > java.lang.NoSuchMethodError: > org.elasticsearch.common.transport.InetSocketTransportAddress.<init>(Ljava/lang/String;I)V > > at > org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.configureHostnames(ElasticSearchTransportClient.java:143) > > at > org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.<init>(ElasticSearchTransportClient.java:77) > > at > org.apache.flume.sink.elasticsearch.client.ElasticSearchClientFactory.getClient(ElasticSearchClientFactory.java:48) > > at > org.apache.flume.sink.elasticsearch.ElasticSearchSink.start(ElasticSearchSink.java:357) > > at > org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46) > > at org.apache.flume.SinkRunner.start(SinkRunner.java:79) > > at > org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251) > > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:745) > > > > > > Flume configuration for ElasticSearch is : > > > > #confgiure elastic search sink > > test.sinks.k1.type = elasticsearch > > test.sinks.k1.hostNames = 172.17.43.207:9200 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__172.17.43.207-3A9200&d=CwMFaQ&c=RI9dKKMRNVHr9NFa7OQiQw&r=dUN85GiSQZVDs0gTK4x1mSiAdXTZ-7F0KzGt2fcse38&m=31csM049OHrrKBoryULOX8hgtSb6qxzPl-6Z_c_-N0A&s=Ofhp-XmxFUHyihMfLqzDo19Z1xzqkXNLV0T4NQnGy2Q&e=> > > test.sinks.k1.indexName = TwitterComp_index > > test.sinks.k1.indexType = TwitterComp_type > > test.sinks.k1.clusterName = twittersearch > > test.sinks.k1.batchSize = 500 > > test.sinks.k1.ttl = 5d > > test.sinks.k1.serializer = > org.apache.flume.sink.elasticsearch.ElasticSearchDynamicSerializer > > > > Any help is greatly appreciated. > > > > Thanks, > > *Muby* > > > > >
