[
https://issues.apache.org/jira/browse/FLUME-1379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jack li updated FLUME-1379:
---------------------------
Description:
hello everyone!
1, source's type is exec and sink's type is hdfs .the event can write to the
hdfs.But when I change the source's type is avro ,and no change sink's type,no
event write to hdfs.
2, flume.conf
#List sources, sinks and channels in the agent
weblog-agent.sources = s1
weblog-agent.sinks = avro-forward-sink01
weblog-agent.channels = jdbc-channel01
#define the flow
#webblog-agent sources config
weblog-agent.sources.s1.channels = jdbc-channel01
weblog-agent.sources.s1.type = avro
weblog-agent.sources.s1.port = 41414
weblog-agent.sources.s1.bind = 0.0.0.0
#avro sink properties
weblog-agent.sinks.avro-forward-sink01.channel = jdbc-channel01
weblog-agent.sinks.avro-forward-sink01.type = hdfs
weblog-agent.sinks.avro-forward-sink01.hdfs.path =
hdfs://hadoop-master.:9000/user/flume/webtest
#weblog-agent.sinks.avro-forward-sink01.hdfs.filePrefix = nginx_%p
#weblog-agent.sinks.avro-forward-sink01.hdfs.filePrefix = nginx_web001
weblog-agent.sinks.avro-forward-sink01.hdfs.kerberosPrincipal = [email protected]
weblog-agent.sinks.avro-forward-sink01.hdfs.kerberosKeytab =
/var/run/flume-ng/flume.keytab
#weblog-agent.sinks.avro-forward-sink01.hdfs.rollInterval = 43200
##### 3 hours#####
weblog-agent.sinks.avro-forward-sink01.hdfs.maxOpenFiles = 65535
weblog-agent.sinks.avro-forward-sink01.hdfs.rollSize = 0
weblog-agent.sinks.avro-forward-sink01.hdfs.rollCount = 0
weblog-agent.sinks.avro-forward-sink01.hdfs.batchSize = 30000
weblog-agent.sinks.avro-forward-sink01.hdfs.txnEventMax = 20000
weblog-agent.sinks.avro-forward-sink01.hdfs.fileType = DataStream
#channels config
weblog-agent.channels.jdbc-channel01.type = memory
weblog-agent.channels.jdbc-channel01.capacity = 200000
weblog-agent.channels.jdbc-channel01.transactionCapacity = 20000
3,flume.log
2012-07-18 23:53:31,245 INFO lifecycle.LifecycleSupervisor: Starting lifecycle
supervisor 1
2012-07-18 23:53:31,247 INFO node.FlumeNode: Flume node starting - weblog-agent
2012-07-18 23:53:31,251 INFO nodemanager.DefaultLogicalNodeManager: Node
manager starting
2012-07-18 23:53:31,252 INFO lifecycle.LifecycleSupervisor: Starting lifecycle
supervisor 10
2012-07-18 23:53:31,252 INFO properties.PropertiesFileConfigurationProvider:
Configuration provider starting
2012-07-18 23:53:31,252 DEBUG nodemanager.DefaultLogicalNodeManager: Node
manager started
2012-07-18 23:53:31,254 DEBUG properties.PropertiesFileConfigurationProvider:
Configuration provider started
2012-07-18 23:53:31,254 DEBUG properties.PropertiesFileConfigurationProvider:
Checking file:flume.conf for changes
2012-07-18 23:53:31,254 INFO properties.PropertiesFileConfigurationProvider:
Reloading configuration file:flume.conf
2012-07-18 23:53:31,261 INFO conf.FlumeConfiguration:
Processing:avro-forward-sink01
2012-07-18 23:53:31,261 DEBUG conf.FlumeConfiguration: Created context for
avro-forward-sink01: hdfs.fileType
2012-07-18 23:53:31,262 INFO conf.FlumeConfiguration:
Processing:avro-forward-sink01
2012-07-18 23:53:31,262 INFO conf.FlumeConfiguration:
Processing:avro-forward-sink01
2012-07-18 23:53:31,262 INFO conf.FlumeConfiguration:
Processing:avro-forward-sink01
2012-07-18 23:53:31,262 INFO conf.FlumeConfiguration:
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration:
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration:
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: Added sinks:
avro-forward-sink01 Agent: weblog-agent
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration:
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration:
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration:
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration:
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 DEBUG conf.FlumeConfiguration: Starting validation of
configuration for agent: weblog-agent, initial-configuration:
AgentConfiguration[weblog-agent]
SOURCES: {s1={ parameters:{port=41414, channels=jdbc-channel01, type=avro,
bind=0.0.0.0} }}
CHANNELS: {jdbc-channel01={ parameters:{transactionCapacity=20000,
capacity=200000, type=memory} }}
SINKS: {avro-forward-sink01={ parameters:{hdfs.fileType=DataStream,
[email protected], hdfs.txnEventMax=20000,
hdfs.kerberosKeytab=/var/run/flume-ng/flume.keytab,
hdfs.path=hdfs://hadoop-master.:9000/user/flume/webtest, hdfs.batchSize=30000,
hdfs.maxOpenFiles=65535, hdfs.rollSize=0, type=hdfs, channel=jdbc-channel01,
hdfs.rollCount=0} }}
2012-07-18 23:53:31,271 DEBUG conf.FlumeConfiguration: Created channel
jdbc-channel01
2012-07-18 23:53:31,289 DEBUG conf.FlumeConfiguration: Creating sink:
avro-forward-sink01 using HDFS
2012-07-18 23:53:31,292 DEBUG conf.FlumeConfiguration: Post validation
configuration for weblog-agent
AgentConfiguration created without Configuration stubs for which only basic
syntactical validation was performed[weblog-agent]
SOURCES: {s1={ parameters:{port=41414, channels=jdbc-channel01, type=avro,
bind=0.0.0.0} }}
CHANNELS: {jdbc-channel01={ parameters:{transactionCapacity=20000,
capacity=200000, type=memory} }}
SINKS: {avro-forward-sink01={ parameters:{hdfs.fileType=DataStream,
[email protected], hdfs.txnEventMax=20000,
hdfs.kerberosKeytab=/var/run/flume-ng/flume.keytab,
hdfs.path=hdfs://hadoop-master.:9000/user/flume/webtest, hdfs.batchSize=30000,
hdfs.maxOpenFiles=65535, hdfs.rollSize=0, type=hdfs, channel=jdbc-channel01,
hdfs.rollCount=0} }}
2012-07-18 23:53:31,292 DEBUG conf.FlumeConfiguration: Channels:jdbc-channel01
2012-07-18 23:53:31,292 DEBUG conf.FlumeConfiguration: Sinks avro-forward-sink01
2012-07-18 23:53:31,292 DEBUG conf.FlumeConfiguration: Sources s1
2012-07-18 23:53:31,292 INFO conf.FlumeConfiguration: Post-validation flume
configuration contains configuration for agents: [weblog-agent]
2012-07-18 23:53:31,292 INFO properties.PropertiesFileConfigurationProvider:
Creating channels
2012-07-18 23:53:31,292 DEBUG channel.DefaultChannelFactory: Creating instance
of channel jdbc-channel01 type memory
2012-07-18 23:53:31,365 INFO instrumentation.MonitoredCounterGroup: Monitoried
counter group for type: CHANNEL, name: jdbc-channel01, registered successfully.
2012-07-18 23:53:31,365 INFO properties.PropertiesFileConfigurationProvider:
created channel jdbc-channel01
2012-07-18 23:53:31,365 DEBUG source.DefaultSourceFactory: Creating instance of
source s1, type avro
2012-07-18 23:53:31,380 INFO instrumentation.MonitoredCounterGroup: Monitoried
counter group for type: SOURCE, name: s1, registered successfully.
2012-07-18 23:53:31,395 INFO sink.DefaultSinkFactory: Creating instance of
sink: avro-forward-sink01, type: hdfs
2012-07-18 23:53:31,540 DEBUG security.Groups: Creating new Groups object
2012-07-18 23:53:31,625 DEBUG security.Groups: Group mapping
impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
2012-07-18 23:53:31,640 INFO hdfs.HDFSEventSink: Hadoop Security enabled: true
2012-07-18 23:53:31,789 INFO hdfs.HDFSEventSink: { Sink type:HDFSEventSink,
name:avro-forward-sink01 }: Attempting kerberos login as principal
([email protected]) from keytab file (/var/run/flume-ng/flume.keytab)
2012-07-18 23:53:31,999 INFO security.UserGroupInformation: Login successful
for user [email protected] using keytab file /var/run/flume-ng/flume.keytab
2012-07-18 23:53:32,000 INFO hdfs.HDFSEventSink: Auth method: KERBEROS
2012-07-18 23:53:32,000 INFO hdfs.HDFSEventSink: User name: [email protected]
2012-07-18 23:53:32,000 INFO hdfs.HDFSEventSink: Using keytab: true
2012-07-18 23:53:32,000 INFO hdfs.HDFSEventSink: Logged in as user
[email protected]
2012-07-18 23:53:32,002 INFO instrumentation.MonitoredCounterGroup: Monitoried
counter group for type: SINK, name: avro-forward-sink01, registered
successfully.
2012-07-18 23:53:32,004 INFO nodemanager.DefaultLogicalNodeManager: Starting
new configuration:{ sourceRunners:{s1=EventDrivenSourceRunner: { source:Avro
source s1: { bindAddress: 0.0.0.0, port: 41414 } }}
sinkRunners:{avro-forward-sink01=SinkRunner: {
policy:org.apache.flume.sink.DefaultSinkProcessor@6c1989b5 counterGroup:{
name:null counters:{} } }}
channels:{jdbc-channel01=org.apache.flume.channel.MemoryChannel@a00185} }
2012-07-18 23:53:32,004 INFO nodemanager.DefaultLogicalNodeManager: Starting
Channel jdbc-channel01
2012-07-18 23:53:32,005 INFO nodemanager.DefaultLogicalNodeManager: Waiting for
channel: jdbc-channel01 to start. Sleeping for 500 ms
2012-07-18 23:53:32,005 INFO instrumentation.MonitoredCounterGroup: Component
type: CHANNEL, name: jdbc-channel01 started
2012-07-18 23:53:32,507 INFO nodemanager.DefaultLogicalNodeManager: Starting
Sink avro-forward-sink01
2012-07-18 23:53:32,508 INFO nodemanager.DefaultLogicalNodeManager: Starting
Source s1
2012-07-18 23:53:32,508 INFO source.AvroSource: Starting Avro source s1: {
bindAddress: 0.0.0.0, port: 41414 }...
2012-07-18 23:53:32,511 INFO instrumentation.MonitoredCounterGroup: Component
type: SINK, name: avro-forward-sink01 started
2012-07-18 23:53:32,514 DEBUG flume.SinkRunner: Polling sink runner starting
2012-07-18 23:53:32,941 INFO instrumentation.MonitoredCounterGroup: Component
type: SOURCE, name: s1 started
2012-07-18 23:53:32,941 INFO source.AvroSource: Avro source s1 started.
2012-07-18 23:54:02,514 DEBUG properties.PropertiesFileConfigurationProvider:
Checking file:flume.conf for changes
2012-07-18 23:54:32,521 DEBUG properties.PropertiesFileConfigurationProvider:
Checking file:flume.conf for changes
was:
hello everyone!
1, source's type is exec and sink's type is hdfs .the event can write to the
hdfs.But when I change the source's type is avro ,and no change sink's type,no
event write to hdfs.
2, flume.conf
#List sources, sinks and channels in the agent
weblog-agent.sources = s1
#weblog-agent.sources.s1.interceptors = ts host
#weblog-agent.sources.s1.interceptors.ts.type =
org.apache.flume.interceptor.TimestampInterceptor$Builder
#weblog-agent.sources.s1.interceptors.host.type =
org.apache.flume.interceptor.HostInterceptor$Builder
#weblog-agent.sources.s1.interceptors.host.useIP = false
#weblog-agent.sources.s1.interceptors.host.preserveExisting = true
weblog-agent.sinks = avro-forward-sink01
weblog-agent.channels = jdbc-channel01
#define the flow
#webblog-agent sources config
weblog-agent.sources.s1.channels = jdbc-channel01
weblog-agent.sources.s1.type = avro
weblog-agent.sources.s1.port = 41414
weblog-agent.sources.s1.bind = 0.0.0.0
#weblog-agent.sources.s1.restart = true
#weblog-agent.sources.s1.restartThrottle = 1000
#weblog-agent.sources.s1.command = tail -f /opt/apps/nginx/logs/access.log
#weblog-agent.sources.s1.selector.type = replicating
#avro sink properties
weblog-agent.sinks.avro-forward-sink01.channel = jdbc-channel01
weblog-agent.sinks.avro-forward-sink01.type = hdfs
#weblog-agent.sinks.avro-forward-sink01.hdfs.path =
hdfs://hadoop-master.:9000/user/flume/webtest/%Y/%m/%d
weblog-agent.sinks.avro-forward-sink01.hdfs.path =
hdfs://hadoop-master.:9000/user/flume/webtest
#weblog-agent.sinks.avro-forward-sink01.hdfs.filePrefix = nginx_%p
#weblog-agent.sinks.avro-forward-sink01.hdfs.filePrefix = nginx_web001
weblog-agent.sinks.avro-forward-sink01.hdfs.kerberosPrincipal = [email protected]
weblog-agent.sinks.avro-forward-sink01.hdfs.kerberosKeytab =
/var/run/flume-ng/flume.keytab
#weblog-agent.sinks.avro-forward-sink01.hdfs.rollInterval = 43200
##### 3 hours#####
weblog-agent.sinks.avro-forward-sink01.hdfs.maxOpenFiles = 65535
weblog-agent.sinks.avro-forward-sink01.hdfs.rollSize = 0
weblog-agent.sinks.avro-forward-sink01.hdfs.rollCount = 0
weblog-agent.sinks.avro-forward-sink01.hdfs.batchSize = 30000
weblog-agent.sinks.avro-forward-sink01.hdfs.txnEventMax = 20000
weblog-agent.sinks.avro-forward-sink01.hdfs.fileType = DataStream
#channels config
weblog-agent.channels.jdbc-channel01.type = memory
weblog-agent.channels.jdbc-channel01.capacity = 200000
weblog-agent.channels.jdbc-channel01.transactionCapacity = 20000
3,flume.log
2012-07-18 23:53:31,245 INFO lifecycle.LifecycleSupervisor: Starting lifecycle
supervisor 1
2012-07-18 23:53:31,247 INFO node.FlumeNode: Flume node starting - weblog-agent
2012-07-18 23:53:31,251 INFO nodemanager.DefaultLogicalNodeManager: Node
manager starting
2012-07-18 23:53:31,252 INFO lifecycle.LifecycleSupervisor: Starting lifecycle
supervisor 10
2012-07-18 23:53:31,252 INFO properties.PropertiesFileConfigurationProvider:
Configuration provider starting
2012-07-18 23:53:31,252 DEBUG nodemanager.DefaultLogicalNodeManager: Node
manager started
2012-07-18 23:53:31,254 DEBUG properties.PropertiesFileConfigurationProvider:
Configuration provider started
2012-07-18 23:53:31,254 DEBUG properties.PropertiesFileConfigurationProvider:
Checking file:flume.conf for changes
2012-07-18 23:53:31,254 INFO properties.PropertiesFileConfigurationProvider:
Reloading configuration file:flume.conf
2012-07-18 23:53:31,261 INFO conf.FlumeConfiguration:
Processing:avro-forward-sink01
2012-07-18 23:53:31,261 DEBUG conf.FlumeConfiguration: Created context for
avro-forward-sink01: hdfs.fileType
2012-07-18 23:53:31,262 INFO conf.FlumeConfiguration:
Processing:avro-forward-sink01
2012-07-18 23:53:31,262 INFO conf.FlumeConfiguration:
Processing:avro-forward-sink01
2012-07-18 23:53:31,262 INFO conf.FlumeConfiguration:
Processing:avro-forward-sink01
2012-07-18 23:53:31,262 INFO conf.FlumeConfiguration:
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration:
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration:
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: Added sinks:
avro-forward-sink01 Agent: weblog-agent
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration:
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration:
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration:
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration:
Processing:avro-forward-sink01
2012-07-18 23:53:31,263 DEBUG conf.FlumeConfiguration: Starting validation of
configuration for agent: weblog-agent, initial-configuration:
AgentConfiguration[weblog-agent]
SOURCES: {s1={ parameters:{port=41414, channels=jdbc-channel01, type=avro,
bind=0.0.0.0} }}
CHANNELS: {jdbc-channel01={ parameters:{transactionCapacity=20000,
capacity=200000, type=memory} }}
SINKS: {avro-forward-sink01={ parameters:{hdfs.fileType=DataStream,
[email protected], hdfs.txnEventMax=20000,
hdfs.kerberosKeytab=/var/run/flume-ng/flume.keytab,
hdfs.path=hdfs://hadoop-master.:9000/user/flume/webtest, hdfs.batchSize=30000,
hdfs.maxOpenFiles=65535, hdfs.rollSize=0, type=hdfs, channel=jdbc-channel01,
hdfs.rollCount=0} }}
2012-07-18 23:53:31,271 DEBUG conf.FlumeConfiguration: Created channel
jdbc-channel01
2012-07-18 23:53:31,289 DEBUG conf.FlumeConfiguration: Creating sink:
avro-forward-sink01 using HDFS
2012-07-18 23:53:31,292 DEBUG conf.FlumeConfiguration: Post validation
configuration for weblog-agent
AgentConfiguration created without Configuration stubs for which only basic
syntactical validation was performed[weblog-agent]
SOURCES: {s1={ parameters:{port=41414, channels=jdbc-channel01, type=avro,
bind=0.0.0.0} }}
CHANNELS: {jdbc-channel01={ parameters:{transactionCapacity=20000,
capacity=200000, type=memory} }}
SINKS: {avro-forward-sink01={ parameters:{hdfs.fileType=DataStream,
[email protected], hdfs.txnEventMax=20000,
hdfs.kerberosKeytab=/var/run/flume-ng/flume.keytab,
hdfs.path=hdfs://hadoop-master.:9000/user/flume/webtest, hdfs.batchSize=30000,
hdfs.maxOpenFiles=65535, hdfs.rollSize=0, type=hdfs, channel=jdbc-channel01,
hdfs.rollCount=0} }}
2012-07-18 23:53:31,292 DEBUG conf.FlumeConfiguration: Channels:jdbc-channel01
2012-07-18 23:53:31,292 DEBUG conf.FlumeConfiguration: Sinks avro-forward-sink01
2012-07-18 23:53:31,292 DEBUG conf.FlumeConfiguration: Sources s1
2012-07-18 23:53:31,292 INFO conf.FlumeConfiguration: Post-validation flume
configuration contains configuration for agents: [weblog-agent]
2012-07-18 23:53:31,292 INFO properties.PropertiesFileConfigurationProvider:
Creating channels
2012-07-18 23:53:31,292 DEBUG channel.DefaultChannelFactory: Creating instance
of channel jdbc-channel01 type memory
2012-07-18 23:53:31,365 INFO instrumentation.MonitoredCounterGroup: Monitoried
counter group for type: CHANNEL, name: jdbc-channel01, registered successfully.
2012-07-18 23:53:31,365 INFO properties.PropertiesFileConfigurationProvider:
created channel jdbc-channel01
2012-07-18 23:53:31,365 DEBUG source.DefaultSourceFactory: Creating instance of
source s1, type avro
2012-07-18 23:53:31,380 INFO instrumentation.MonitoredCounterGroup: Monitoried
counter group for type: SOURCE, name: s1, registered successfully.
2012-07-18 23:53:31,395 INFO sink.DefaultSinkFactory: Creating instance of
sink: avro-forward-sink01, type: hdfs
2012-07-18 23:53:31,540 DEBUG security.Groups: Creating new Groups object
2012-07-18 23:53:31,625 DEBUG security.Groups: Group mapping
impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
2012-07-18 23:53:31,640 INFO hdfs.HDFSEventSink: Hadoop Security enabled: true
2012-07-18 23:53:31,789 INFO hdfs.HDFSEventSink: { Sink type:HDFSEventSink,
name:avro-forward-sink01 }: Attempting kerberos login as principal
([email protected]) from keytab file (/var/run/flume-ng/flume.keytab)
2012-07-18 23:53:31,999 INFO security.UserGroupInformation: Login successful
for user [email protected] using keytab file /var/run/flume-ng/flume.keytab
2012-07-18 23:53:32,000 INFO hdfs.HDFSEventSink: Auth method: KERBEROS
2012-07-18 23:53:32,000 INFO hdfs.HDFSEventSink: User name: [email protected]
2012-07-18 23:53:32,000 INFO hdfs.HDFSEventSink: Using keytab: true
2012-07-18 23:53:32,000 INFO hdfs.HDFSEventSink: Logged in as user
[email protected]
2012-07-18 23:53:32,002 INFO instrumentation.MonitoredCounterGroup: Monitoried
counter group for type: SINK, name: avro-forward-sink01, registered
successfully.
2012-07-18 23:53:32,004 INFO nodemanager.DefaultLogicalNodeManager: Starting
new configuration:{ sourceRunners:{s1=EventDrivenSourceRunner: { source:Avro
source s1: { bindAddress: 0.0.0.0, port: 41414 } }}
sinkRunners:{avro-forward-sink01=SinkRunner: {
policy:org.apache.flume.sink.DefaultSinkProcessor@6c1989b5 counterGroup:{
name:null counters:{} } }}
channels:{jdbc-channel01=org.apache.flume.channel.MemoryChannel@a00185} }
2012-07-18 23:53:32,004 INFO nodemanager.DefaultLogicalNodeManager: Starting
Channel jdbc-channel01
2012-07-18 23:53:32,005 INFO nodemanager.DefaultLogicalNodeManager: Waiting for
channel: jdbc-channel01 to start. Sleeping for 500 ms
2012-07-18 23:53:32,005 INFO instrumentation.MonitoredCounterGroup: Component
type: CHANNEL, name: jdbc-channel01 started
2012-07-18 23:53:32,507 INFO nodemanager.DefaultLogicalNodeManager: Starting
Sink avro-forward-sink01
2012-07-18 23:53:32,508 INFO nodemanager.DefaultLogicalNodeManager: Starting
Source s1
2012-07-18 23:53:32,508 INFO source.AvroSource: Starting Avro source s1: {
bindAddress: 0.0.0.0, port: 41414 }...
2012-07-18 23:53:32,511 INFO instrumentation.MonitoredCounterGroup: Component
type: SINK, name: avro-forward-sink01 started
2012-07-18 23:53:32,514 DEBUG flume.SinkRunner: Polling sink runner starting
2012-07-18 23:53:32,941 INFO instrumentation.MonitoredCounterGroup: Component
type: SOURCE, name: s1 started
2012-07-18 23:53:32,941 INFO source.AvroSource: Avro source s1 started.
2012-07-18 23:54:02,514 DEBUG properties.PropertiesFileConfigurationProvider:
Checking file:flume.conf for changes
2012-07-18 23:54:32,521 DEBUG properties.PropertiesFileConfigurationProvider:
Checking file:flume.conf for changes
> source's type is avro and sink's type is hdfs but sink don't to work
> ---------------------------------------------------------------------
>
> Key: FLUME-1379
> URL: https://issues.apache.org/jira/browse/FLUME-1379
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.1.0, v1.3.0
> Environment: rhel5.7 ,jdk1.6,yum installed flume-ng
> 1.1.0+120-1.cdh4.0.0 ,mvn install 1.3.
> Reporter: jack li
> Labels: flume
> Fix For: v1.3.0
>
> Original Estimate: 504h
> Remaining Estimate: 504h
>
> hello everyone!
> 1, source's type is exec and sink's type is hdfs .the event can write to the
> hdfs.But when I change the source's type is avro ,and no change sink's
> type,no event write to hdfs.
> 2, flume.conf
> #List sources, sinks and channels in the agent
> weblog-agent.sources = s1
> weblog-agent.sinks = avro-forward-sink01
> weblog-agent.channels = jdbc-channel01
> #define the flow
> #webblog-agent sources config
> weblog-agent.sources.s1.channels = jdbc-channel01
> weblog-agent.sources.s1.type = avro
> weblog-agent.sources.s1.port = 41414
> weblog-agent.sources.s1.bind = 0.0.0.0
> #avro sink properties
> weblog-agent.sinks.avro-forward-sink01.channel = jdbc-channel01
> weblog-agent.sinks.avro-forward-sink01.type = hdfs
> weblog-agent.sinks.avro-forward-sink01.hdfs.path =
> hdfs://hadoop-master.:9000/user/flume/webtest
> #weblog-agent.sinks.avro-forward-sink01.hdfs.filePrefix = nginx_%p
> #weblog-agent.sinks.avro-forward-sink01.hdfs.filePrefix = nginx_web001
> weblog-agent.sinks.avro-forward-sink01.hdfs.kerberosPrincipal =
> [email protected]
> weblog-agent.sinks.avro-forward-sink01.hdfs.kerberosKeytab =
> /var/run/flume-ng/flume.keytab
> #weblog-agent.sinks.avro-forward-sink01.hdfs.rollInterval = 43200
> ##### 3 hours#####
> weblog-agent.sinks.avro-forward-sink01.hdfs.maxOpenFiles = 65535
> weblog-agent.sinks.avro-forward-sink01.hdfs.rollSize = 0
> weblog-agent.sinks.avro-forward-sink01.hdfs.rollCount = 0
> weblog-agent.sinks.avro-forward-sink01.hdfs.batchSize = 30000
> weblog-agent.sinks.avro-forward-sink01.hdfs.txnEventMax = 20000
> weblog-agent.sinks.avro-forward-sink01.hdfs.fileType = DataStream
> #channels config
> weblog-agent.channels.jdbc-channel01.type = memory
> weblog-agent.channels.jdbc-channel01.capacity = 200000
> weblog-agent.channels.jdbc-channel01.transactionCapacity = 20000
> 3,flume.log
> 2012-07-18 23:53:31,245 INFO lifecycle.LifecycleSupervisor: Starting
> lifecycle supervisor 1
> 2012-07-18 23:53:31,247 INFO node.FlumeNode: Flume node starting -
> weblog-agent
> 2012-07-18 23:53:31,251 INFO nodemanager.DefaultLogicalNodeManager: Node
> manager starting
> 2012-07-18 23:53:31,252 INFO lifecycle.LifecycleSupervisor: Starting
> lifecycle supervisor 10
> 2012-07-18 23:53:31,252 INFO properties.PropertiesFileConfigurationProvider:
> Configuration provider starting
> 2012-07-18 23:53:31,252 DEBUG nodemanager.DefaultLogicalNodeManager: Node
> manager started
> 2012-07-18 23:53:31,254 DEBUG properties.PropertiesFileConfigurationProvider:
> Configuration provider started
> 2012-07-18 23:53:31,254 DEBUG properties.PropertiesFileConfigurationProvider:
> Checking file:flume.conf for changes
> 2012-07-18 23:53:31,254 INFO properties.PropertiesFileConfigurationProvider:
> Reloading configuration file:flume.conf
> 2012-07-18 23:53:31,261 INFO conf.FlumeConfiguration:
> Processing:avro-forward-sink01
> 2012-07-18 23:53:31,261 DEBUG conf.FlumeConfiguration: Created context for
> avro-forward-sink01: hdfs.fileType
> 2012-07-18 23:53:31,262 INFO conf.FlumeConfiguration:
> Processing:avro-forward-sink01
> 2012-07-18 23:53:31,262 INFO conf.FlumeConfiguration:
> Processing:avro-forward-sink01
> 2012-07-18 23:53:31,262 INFO conf.FlumeConfiguration:
> Processing:avro-forward-sink01
> 2012-07-18 23:53:31,262 INFO conf.FlumeConfiguration:
> Processing:avro-forward-sink01
> 2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration:
> Processing:avro-forward-sink01
> 2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration:
> Processing:avro-forward-sink01
> 2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration: Added sinks:
> avro-forward-sink01 Agent: weblog-agent
> 2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration:
> Processing:avro-forward-sink01
> 2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration:
> Processing:avro-forward-sink01
> 2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration:
> Processing:avro-forward-sink01
> 2012-07-18 23:53:31,263 INFO conf.FlumeConfiguration:
> Processing:avro-forward-sink01
> 2012-07-18 23:53:31,263 DEBUG conf.FlumeConfiguration: Starting validation of
> configuration for agent: weblog-agent, initial-configuration:
> AgentConfiguration[weblog-agent]
> SOURCES: {s1={ parameters:{port=41414, channels=jdbc-channel01, type=avro,
> bind=0.0.0.0} }}
> CHANNELS: {jdbc-channel01={ parameters:{transactionCapacity=20000,
> capacity=200000, type=memory} }}
> SINKS: {avro-forward-sink01={ parameters:{hdfs.fileType=DataStream,
> [email protected], hdfs.txnEventMax=20000,
> hdfs.kerberosKeytab=/var/run/flume-ng/flume.keytab,
> hdfs.path=hdfs://hadoop-master.:9000/user/flume/webtest,
> hdfs.batchSize=30000, hdfs.maxOpenFiles=65535, hdfs.rollSize=0, type=hdfs,
> channel=jdbc-channel01, hdfs.rollCount=0} }}
> 2012-07-18 23:53:31,271 DEBUG conf.FlumeConfiguration: Created channel
> jdbc-channel01
> 2012-07-18 23:53:31,289 DEBUG conf.FlumeConfiguration: Creating sink:
> avro-forward-sink01 using HDFS
> 2012-07-18 23:53:31,292 DEBUG conf.FlumeConfiguration: Post validation
> configuration for weblog-agent
> AgentConfiguration created without Configuration stubs for which only basic
> syntactical validation was performed[weblog-agent]
> SOURCES: {s1={ parameters:{port=41414, channels=jdbc-channel01, type=avro,
> bind=0.0.0.0} }}
> CHANNELS: {jdbc-channel01={ parameters:{transactionCapacity=20000,
> capacity=200000, type=memory} }}
> SINKS: {avro-forward-sink01={ parameters:{hdfs.fileType=DataStream,
> [email protected], hdfs.txnEventMax=20000,
> hdfs.kerberosKeytab=/var/run/flume-ng/flume.keytab,
> hdfs.path=hdfs://hadoop-master.:9000/user/flume/webtest,
> hdfs.batchSize=30000, hdfs.maxOpenFiles=65535, hdfs.rollSize=0, type=hdfs,
> channel=jdbc-channel01, hdfs.rollCount=0} }}
> 2012-07-18 23:53:31,292 DEBUG conf.FlumeConfiguration: Channels:jdbc-channel01
> 2012-07-18 23:53:31,292 DEBUG conf.FlumeConfiguration: Sinks
> avro-forward-sink01
> 2012-07-18 23:53:31,292 DEBUG conf.FlumeConfiguration: Sources s1
> 2012-07-18 23:53:31,292 INFO conf.FlumeConfiguration: Post-validation flume
> configuration contains configuration for agents: [weblog-agent]
> 2012-07-18 23:53:31,292 INFO properties.PropertiesFileConfigurationProvider:
> Creating channels
> 2012-07-18 23:53:31,292 DEBUG channel.DefaultChannelFactory: Creating
> instance of channel jdbc-channel01 type memory
> 2012-07-18 23:53:31,365 INFO instrumentation.MonitoredCounterGroup:
> Monitoried counter group for type: CHANNEL, name: jdbc-channel01, registered
> successfully.
> 2012-07-18 23:53:31,365 INFO properties.PropertiesFileConfigurationProvider:
> created channel jdbc-channel01
> 2012-07-18 23:53:31,365 DEBUG source.DefaultSourceFactory: Creating instance
> of source s1, type avro
> 2012-07-18 23:53:31,380 INFO instrumentation.MonitoredCounterGroup:
> Monitoried counter group for type: SOURCE, name: s1, registered successfully.
> 2012-07-18 23:53:31,395 INFO sink.DefaultSinkFactory: Creating instance of
> sink: avro-forward-sink01, type: hdfs
> 2012-07-18 23:53:31,540 DEBUG security.Groups: Creating new Groups object
> 2012-07-18 23:53:31,625 DEBUG security.Groups: Group mapping
> impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping;
> cacheTimeout=300000
> 2012-07-18 23:53:31,640 INFO hdfs.HDFSEventSink: Hadoop Security enabled: true
> 2012-07-18 23:53:31,789 INFO hdfs.HDFSEventSink: { Sink type:HDFSEventSink,
> name:avro-forward-sink01 }: Attempting kerberos login as principal
> ([email protected]) from keytab file (/var/run/flume-ng/flume.keytab)
> 2012-07-18 23:53:31,999 INFO security.UserGroupInformation: Login successful
> for user [email protected] using keytab file /var/run/flume-ng/flume.keytab
> 2012-07-18 23:53:32,000 INFO hdfs.HDFSEventSink: Auth method: KERBEROS
> 2012-07-18 23:53:32,000 INFO hdfs.HDFSEventSink: User name: [email protected]
> 2012-07-18 23:53:32,000 INFO hdfs.HDFSEventSink: Using keytab: true
> 2012-07-18 23:53:32,000 INFO hdfs.HDFSEventSink: Logged in as user
> [email protected]
> 2012-07-18 23:53:32,002 INFO instrumentation.MonitoredCounterGroup:
> Monitoried counter group for type: SINK, name: avro-forward-sink01,
> registered successfully.
> 2012-07-18 23:53:32,004 INFO nodemanager.DefaultLogicalNodeManager: Starting
> new configuration:{ sourceRunners:{s1=EventDrivenSourceRunner: { source:Avro
> source s1: { bindAddress: 0.0.0.0, port: 41414 } }}
> sinkRunners:{avro-forward-sink01=SinkRunner: {
> policy:org.apache.flume.sink.DefaultSinkProcessor@6c1989b5 counterGroup:{
> name:null counters:{} } }}
> channels:{jdbc-channel01=org.apache.flume.channel.MemoryChannel@a00185} }
> 2012-07-18 23:53:32,004 INFO nodemanager.DefaultLogicalNodeManager: Starting
> Channel jdbc-channel01
> 2012-07-18 23:53:32,005 INFO nodemanager.DefaultLogicalNodeManager: Waiting
> for channel: jdbc-channel01 to start. Sleeping for 500 ms
> 2012-07-18 23:53:32,005 INFO instrumentation.MonitoredCounterGroup: Component
> type: CHANNEL, name: jdbc-channel01 started
> 2012-07-18 23:53:32,507 INFO nodemanager.DefaultLogicalNodeManager: Starting
> Sink avro-forward-sink01
> 2012-07-18 23:53:32,508 INFO nodemanager.DefaultLogicalNodeManager: Starting
> Source s1
> 2012-07-18 23:53:32,508 INFO source.AvroSource: Starting Avro source s1: {
> bindAddress: 0.0.0.0, port: 41414 }...
> 2012-07-18 23:53:32,511 INFO instrumentation.MonitoredCounterGroup: Component
> type: SINK, name: avro-forward-sink01 started
> 2012-07-18 23:53:32,514 DEBUG flume.SinkRunner: Polling sink runner starting
> 2012-07-18 23:53:32,941 INFO instrumentation.MonitoredCounterGroup: Component
> type: SOURCE, name: s1 started
> 2012-07-18 23:53:32,941 INFO source.AvroSource: Avro source s1 started.
> 2012-07-18 23:54:02,514 DEBUG properties.PropertiesFileConfigurationProvider:
> Checking file:flume.conf for changes
> 2012-07-18 23:54:32,521 DEBUG properties.PropertiesFileConfigurationProvider:
> Checking file:flume.conf for changes
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
