[
https://issues.apache.org/jira/browse/FLUME-3021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15932389#comment-15932389
]
Helmut Wahrman edited comment on FLUME-3021 at 3/20/17 9:39 AM:
----------------------------------------------------------------
I already corrected all the dependencies. I have jackson 2.8.6, the es jar and
all the transport jars in flume lib.
My conf looks like this. I am reading an avro file and my custom serializer
reads the schema from the avro file.
This works pretty fine in pre 2.0 versions of ES.
This is my config:
a1.sources = avrofld
a1.sinks = k1
a1.channels = c1
a1.sources.avrofld.type = spooldir
a1.sources.avrofld.channels = c1
a1.sources.avrofld.spoolDir = /var/export
a1.sources.avrofld.fileHeader = true
a1.sources.avrofld.deserializer = avro
a1.sinks.k1.type = org.apache.flume.sink.elasticsearch.ElasticSearchSink
a1.sinks.k1.hostNames = 192.168.1.50:9300
a1.sinks.k1.indexName = sa
a1.sinks.k1.indexType = data
a1.sinks.k1.clusterName = elasticsearch
a1.sinks.k1.batchSize = 1000
a1.sinks.k1.ttl = 2d
a1.sinks.k1.serializer = com.rsa.flume.serialization.FlumeAvroEventDeserializer
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000000
a1.channels.c1.transactionCapacity = 100000
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
Maybe you can post your config that works fine for you and i will try on my side
was (Author: hwahrmann):
I already corrected all the dependencies. I have jackson 2.8.6, the es jar and
all the transport jars in flume lib.
My conf looks like this. I am reading an avro file and my custom serializer
reads the schema from the avro file.
This works pretty fine in pre 2.0 versions of ES.
This is my config:
# Name the components on this agent
a1.sources = avrofld
a1.sinks = k1
a1.channels = c1
# Describe/configure the folder with the avro files from Warehouse Connector
a1.sources.avrofld.type = spooldir
a1.sources.avrofld.channels = c1
a1.sources.avrofld.spoolDir = /var/export
a1.sources.avrofld.fileHeader = true
a1.sources.avrofld.deserializer = avro
# Describe the elasticsearch sink
a1.sinks.k1.type = org.apache.flume.sink.elasticsearch.ElasticSearchSink
a1.sinks.k1.hostNames = 192.168.1.50:9300
a1.sinks.k1.indexName = sa
a1.sinks.k1.indexType = data
a1.sinks.k1.clusterName = elasticsearch
a1.sinks.k1.batchSize = 1000
a1.sinks.k1.ttl = 2d
a1.sinks.k1.serializer = com.rsa.flume.serialization.FlumeAvroEventDeserializer
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000000
a1.channels.c1.transactionCapacity = 100000
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
Maybe you can post your config that works fine for you and i will try on my side
> flume Elasticsearch 5.0 not support
> -----------------------------------
>
> Key: FLUME-3021
> URL: https://issues.apache.org/jira/browse/FLUME-3021
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: 1.7.0
> Reporter: tycho_yang
> Assignee: Yonghao Zou
> Fix For: 1.8.0
>
> Attachments: FLUME-3021-0.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
