[
https://issues.apache.org/jira/browse/FLUME-3021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939923#comment-15939923
]
tycho_yang commented on FLUME-3021:
-----------------------------------
hi Yonghao Zou
There is a certain probability that the error will be reported:
______________________________________________________________________________________
[root@centos-6-php apache-flume-1.7.0-bin]# bin/flume-ng agent --conf conf
--conf-file ./conf/elasticsearch_20160217.conf --name agent1
-Dflume.root.logger=INFO,console
Info: Sourcing environment configuration script
/data/apache-flume-1.7.0-bin/conf/flume-env.sh
Info: Including Hive libraries found via () for Hive access
+ exec /usr/java/latest/bin/java -Xms5000m -Xmx5000m
-Dcom.sun.management.jmxremote -Dflume.root.logger=INFO,console -cp
'/data/apache-flume-1.7.0-bin/conf:/data/apache-flume-1.7.0-bin/lib/*:/lib/*'
-Djava.library.path= org.apache.flume.node.Application --conf-file
./conf/elasticsearch_20160217.conf --name agent1
2017-03-24 14:38:31,657 (lifecycleSupervisor-1-0) [INFO -
org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start(PollingPropertiesFileConfigurationProvider.java:62)]
Configuration provider starting
2017-03-24 14:38:31,680 (conf-file-poller-0) [INFO -
org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:134)]
Reloading configuration file:./conf/elasticsearch_20160217.conf
2017-03-24 14:38:31,697 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)]
Processing:sink1
2017-03-24 14:38:31,722 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)]
Processing:sink1
2017-03-24 14:38:31,722 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)]
Processing:sink1
2017-03-24 14:38:31,724 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:930)]
Added sinks: sink1 Agent: agent1
2017-03-24 14:38:31,725 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)]
Processing:sink1
2017-03-24 14:38:31,725 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)]
Processing:sink1
2017-03-24 14:38:31,725 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)]
Processing:sink1
2017-03-24 14:38:31,725 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)]
Processing:sink1
2017-03-24 14:38:31,728 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)]
Processing:sink1
2017-03-24 14:38:31,728 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)]
Processing:sink1
2017-03-24 14:38:31,728 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)]
Processing:sink1
2017-03-24 14:38:31,796 (conf-file-poller-0) [INFO -
org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:140)]
Post-validation flume configuration contains configuration for agents: [agent1]
2017-03-24 14:38:31,796 (conf-file-poller-0) [INFO -
org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:147)]
Creating channels
2017-03-24 14:38:31,818 (conf-file-poller-0) [INFO -
org.apache.flume.channel.DefaultChannelFactory.create(DefaultChannelFactory.java:42)]
Creating instance of channel channel1 type memory
2017-03-24 14:38:31,826 (conf-file-poller-0) [INFO -
org.apache.flume.node.AbstractConfigurationProvider.loadChannels(AbstractConfigurationProvider.java:201)]
Created channel channel1
2017-03-24 14:38:31,827 (conf-file-poller-0) [INFO -
org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:41)]
Creating instance of source source1, type exec
2017-03-24 14:38:32,041 (conf-file-poller-0) [INFO -
org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:42)]
Creating instance of sink: sink1, type: elasticsearch
2017-03-24 14:38:32,046 (conf-file-poller-0) [INFO -
org.apache.flume.sink.elasticsearch.ElasticSearchSink.parseTTL(ElasticSearchSink.java:417)]
TTL qualifier is empty. Defaulting to day qualifier.
2017-03-24 14:38:32,094 (conf-file-poller-0) [INFO -
org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:116)]
Channel channel1 connected to [source1, sink1]
2017-03-24 14:38:32,131 (conf-file-poller-0) [INFO -
org.apache.flume.node.Application.startAllComponents(Application.java:137)]
Starting new configuration:{ sourceRunners:{source1=EventDrivenSourceRunner: {
source:org.apache.flume.source.ExecSource{name:source1,state:IDLE} }}
sinkRunners:{sink1=SinkRunner: {
policy:org.apache.flume.sink.DefaultSinkProcessor@90d4a28 counterGroup:{
name:null counters:{} } }}
channels:{channel1=org.apache.flume.channel.MemoryChannel{name: channel1}} }
2017-03-24 14:38:32,131 (conf-file-poller-0) [INFO -
org.apache.flume.node.Application.startAllComponents(Application.java:144)]
Starting Channel channel1
2017-03-24 14:38:32,132 (lifecycleSupervisor-1-0) [INFO -
org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:119)]
Monitored counter group for type: CHANNEL, name: channel1: Successfully
registered new MBean.
2017-03-24 14:38:32,132 (lifecycleSupervisor-1-0) [INFO -
org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)]
Component type: CHANNEL, name: channel1 started
2017-03-24 14:38:32,152 (conf-file-poller-0) [INFO -
org.apache.flume.node.Application.startAllComponents(Application.java:171)]
Starting Sink sink1
2017-03-24 14:38:32,153 (lifecycleSupervisor-1-1) [INFO -
org.apache.flume.sink.elasticsearch.ElasticSearchSink.start(ElasticSearchSink.java:350)]
ElasticSearch sink {} started
2017-03-24 14:38:32,153 (lifecycleSupervisor-1-1) [INFO -
org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:119)]
Monitored counter group for type: SINK, name: sink1: Successfully registered
new MBean.
2017-03-24 14:38:32,153 (lifecycleSupervisor-1-1) [INFO -
org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)]
Component type: SINK, name: sink1 started
2017-03-24 14:38:32,161 (lifecycleSupervisor-1-1) [WARN -
org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.configureHostnames(ElasticSearchTransportClient.java:139)]
[192.168.3.63:9300]
2017-03-24 14:38:32,164 (lifecycleSupervisor-1-1) [INFO -
org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.openClient(ElasticSearchTransportClient.java:199)]
Using ElasticSearch hostnames: [192.168.3.63:9300]
2017-03-24 14:38:32,169 (conf-file-poller-0) [INFO -
org.apache.flume.node.Application.startAllComponents(Application.java:182)]
Starting Source source1
2017-03-24 14:38:32,169 (lifecycleSupervisor-1-2) [INFO -
org.apache.flume.source.ExecSource.start(ExecSource.java:168)] Exec source
starting with command:tail -F /data/lb-www.access.log-20170324
2017-03-24 14:38:32,203 (lifecycleSupervisor-1-2) [INFO -
org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:119)]
Monitored counter group for type: SOURCE, name: source1: Successfully
registered new MBean.
2017-03-24 14:38:32,203 (lifecycleSupervisor-1-2) [INFO -
org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:95)]
Component type: SOURCE, name: source1 started
2017-03-24 14:38:33,315 (lifecycleSupervisor-1-1) [INFO -
org.elasticsearch.plugins.PluginsService.logPluginInfo(PluginsService.java:174)]
no modules loaded
2017-03-24 14:38:33,320 (lifecycleSupervisor-1-1) [INFO -
org.elasticsearch.plugins.PluginsService.logPluginInfo(PluginsService.java:177)]
loaded plugin [org.elasticsearch.index.reindex.ReindexPlugin]
2017-03-24 14:38:33,320 (lifecycleSupervisor-1-1) [INFO -
org.elasticsearch.plugins.PluginsService.logPluginInfo(PluginsService.java:177)]
loaded plugin [org.elasticsearch.percolator.PercolatorPlugin]
2017-03-24 14:38:33,321 (lifecycleSupervisor-1-1) [INFO -
org.elasticsearch.plugins.PluginsService.logPluginInfo(PluginsService.java:177)]
loaded plugin [org.elasticsearch.script.mustache.MustachePlugin]
2017-03-24 14:38:33,321 (lifecycleSupervisor-1-1) [INFO -
org.elasticsearch.plugins.PluginsService.logPluginInfo(PluginsService.java:177)]
loaded plugin [org.elasticsearch.transport.Netty3Plugin]
2017-03-24 14:38:33,321 (lifecycleSupervisor-1-1) [INFO -
org.elasticsearch.plugins.PluginsService.logPluginInfo(PluginsService.java:177)]
loaded plugin [org.elasticsearch.transport.Netty4Plugin]
2017-03-24 14:38:45,485 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR
-
org.apache.flume.sink.elasticsearch.ElasticSearchSink.process(ElasticSearchSink.java:221)]
Failed to commit transaction. Transaction rolled back.
NoNodeAvailableException[None of the configured nodes were available:
[{node-1}{Xb_O5doNTMuS2u4pbC4ILQ}{O4cUzwIPQ9uVMWkygbwhtw}{192.168.3.63}{192.168.3.63:9300}]];
nested: NodeNotConnectedException[[node-1][192.168.3.63:9300] Node not
connected];
at
org.elasticsearch.client.transport.TransportClientNodesService$RetryListener.onFailure(TransportClientNodesService.java:288)
at
org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:51)
at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:476)
at
org.elasticsearch.action.TransportActionNodeProxy.execute(TransportActionNodeProxy.java:51)
at
org.elasticsearch.client.transport.TransportProxyClient.lambda$execute$0(TransportProxyClient.java:59)
at
org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:247)
at
org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:59)
at
org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:356)
at
org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403)
at
org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:80)
at
org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:54)
at
org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.execute(ElasticSearchTransportClient.java:184)
at
org.apache.flume.sink.elasticsearch.ElasticSearchSink.process(ElasticSearchSink.java:205)
at
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
at java.lang.Thread.run(Thread.java:745)
Caused by: NodeNotConnectedException[[node-1][192.168.3.63:9300] Node not
connected]
at
org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:590)
at
org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:113)
at
org.elasticsearch.transport.TransportService.getConnection(TransportService.java:496)
at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:472)
... 13 more
2017-03-24 14:38:45,507 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR
- org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:158)] Unable to
deliver event. Exception follows.
NoNodeAvailableException[None of the configured nodes were available:
[{node-1}{Xb_O5doNTMuS2u4pbC4ILQ}{O4cUzwIPQ9uVMWkygbwhtw}{192.168.3.63}{192.168.3.63:9300}]];
nested: NodeNotConnectedException[[node-1][192.168.3.63:9300] Node not
connected];
at
org.elasticsearch.client.transport.TransportClientNodesService$RetryListener.onFailure(TransportClientNodesService.java:288)
at
org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:51)
at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:476)
at
org.elasticsearch.action.TransportActionNodeProxy.execute(TransportActionNodeProxy.java:51)
at
org.elasticsearch.client.transport.TransportProxyClient.lambda$execute$0(TransportProxyClient.java:59)
at
org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:247)
at
org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:59)
at
org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:356)
at
org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403)
at
org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:80)
at
org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:54)
at
org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.execute(ElasticSearchTransportClient.java:184)
at
org.apache.flume.sink.elasticsearch.ElasticSearchSink.process(ElasticSearchSink.java:205)
at
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
at java.lang.Thread.run(Thread.java:745)
Caused by: NodeNotConnectedException[[node-1][192.168.3.63:9300] Node not
connected]
at
org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:590)
at
org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:113)
at
org.elasticsearch.transport.TransportService.getConnection(TransportService.java:496)
at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:472)
... 13 more
> flume Elasticsearch 5.0 not support
> -----------------------------------
>
> Key: FLUME-3021
> URL: https://issues.apache.org/jira/browse/FLUME-3021
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: 1.7.0
> Reporter: tycho_yang
> Assignee: Yonghao Zou
> Fix For: 1.8.0
>
> Attachments: FLUME-3021-0.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
