Hello Iain, They are in my plugins.d dir of flume. Do I also need to have them in flume lib folder ?
Thanks, Muby From: iain wright [mailto:[email protected]] Sent: Monday, May 02, 2016 4:25 PM To: [email protected] Subject: Re: Flume - elasticsearch sink fails Hi Muby, Are the ES and lucene-core jars in your flume lib folder? -- Iain Wright This email message is confidential, intended only for the recipient(s) named above and may contain information that is privileged, exempt from disclosure under applicable law. If you are not the intended recipient, do not disclose or disseminate the message to anyone except the intended recipient. If you have received this message in error, or are not the named recipient(s), please immediately notify the sender by return email, and delete all copies of this message. On Mon, May 2, 2016 at 1:47 PM, Arif,Mubaraka <[email protected]<mailto:[email protected]>> wrote: Downgraded to Elasticsearch 1.7.5 and still get the same error : RROR org.apache.flume.lifecycle.LifecycleSupervisor Unable to start SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@730d4fcf counterGroup:{ name:null counters:{} } } - Exception follows. java.lang.NoSuchMethodError: org.elasticsearch.common.transport.InetSocketTransportAddress.<init>(Ljava/lang/String;I)V at org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.configureHostnames(ElasticSearchTransportClient.java:143) at org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.<init>(ElasticSearchTransportClient.java:77) at org.apache.flume.sink.elasticsearch.client.ElasticSearchClientFactory.getClient(ElasticSearchClientFactory.java:48) at org.apache.flume.sink.elasticsearch.ElasticSearchSink.start(ElasticSearchSink.java:357) at org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46) at org.apache.flume.SinkRunner.start(SinkRunner.java:79) at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) From: iain wright [mailto:[email protected]<mailto:[email protected]>] Sent: Monday, May 02, 2016 2:52 PM To: [email protected]<mailto:[email protected]> Cc: Mabrito,Greg Subject: Re: Flume - elasticsearch sink fails Hi, These look relevant to your goals: https://github.com/elastic/elasticsearch/issues/14187<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_elastic_elasticsearch_issues_14187&d=CwMFaQ&c=RI9dKKMRNVHr9NFa7OQiQw&r=dUN85GiSQZVDs0gTK4x1mSiAdXTZ-7F0KzGt2fcse38&m=31csM049OHrrKBoryULOX8hgtSb6qxzPl-6Z_c_-N0A&s=p6q-rN0ma3I-5GVFu0e_CASXMGrDeforSajvFopvNic&e=> https://github.com/lucidfrontier45/ElasticsearchSink2<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_lucidfrontier45_ElasticsearchSink2&d=CwMFaQ&c=RI9dKKMRNVHr9NFa7OQiQw&r=dUN85GiSQZVDs0gTK4x1mSiAdXTZ-7F0KzGt2fcse38&m=31csM049OHrrKBoryULOX8hgtSb6qxzPl-6Z_c_-N0A&s=RSyBce-WtursnyucSVeewjWRYtSDC1dkIbXOpE3boRc&e=> HTH, -- Iain Wright This email message is confidential, intended only for the recipient(s) named above and may contain information that is privileged, exempt from disclosure under applicable law. If you are not the intended recipient, do not disclose or disseminate the message to anyone except the intended recipient. If you have received this message in error, or are not the named recipient(s), please immediately notify the sender by return email, and delete all copies of this message. On Mon, May 2, 2016 at 12:44 PM, Arif,Mubaraka <[email protected]<mailto:[email protected]>> wrote: Hello, Trying to flume tweets from Tibco-JMS source into elastic search sink. Running Elasticsearch on our edge nodes independent of Hadoop. Running cloudera Hadoop 5.4.8 On running the error is : May 2, 2:15:14.047 PM INFO org.apache.flume.instrumentation.MonitoredCounterGroup Component type: SINK, name: k1 started May 2, 2:15:14.050 PM WARN org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient [arcapl0021707:9200] May 2, 2:15:14.052 PM ERROR org.apache.flume.lifecycle.LifecycleSupervisor Unable to start SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@50482278 counterGroup:{ name:null counters:{} } } - Exception follows. java.lang.NoSuchMethodError: org.elasticsearch.common.transport.InetSocketTransportAddress.<init>(Ljava/lang/String;I)V at org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.configureHostnames(ElasticSearchTransportClient.java:143) at org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.<init>(ElasticSearchTransportClient.java:77) at org.apache.flume.sink.elasticsearch.client.ElasticSearchClientFactory.getClient(ElasticSearchClientFactory.java:48) at org.apache.flume.sink.elasticsearch.ElasticSearchSink.start(ElasticSearchSink.java:357) at org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46) at org.apache.flume.SinkRunner.start(SinkRunner.java:79) at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Flume configuration for ElasticSearch is : #confgiure elastic search sink test.sinks.k1.type = elasticsearch test.sinks.k1.hostNames = 172.17.43.207:9200<https://urldefense.proofpoint.com/v2/url?u=http-3A__172.17.43.207-3A9200&d=CwMFaQ&c=RI9dKKMRNVHr9NFa7OQiQw&r=dUN85GiSQZVDs0gTK4x1mSiAdXTZ-7F0KzGt2fcse38&m=31csM049OHrrKBoryULOX8hgtSb6qxzPl-6Z_c_-N0A&s=Ofhp-XmxFUHyihMfLqzDo19Z1xzqkXNLV0T4NQnGy2Q&e=> test.sinks.k1.indexName = TwitterComp_index test.sinks.k1.indexType = TwitterComp_type test.sinks.k1.clusterName = twittersearch test.sinks.k1.batchSize = 500 test.sinks.k1.ttl = 5d test.sinks.k1.serializer = org.apache.flume.sink.elasticsearch.ElasticSearchDynamicSerializer Any help is greatly appreciated. Thanks, Muby
