[
https://issues.apache.org/jira/browse/FLUME-2653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14958778#comment-14958778
]
Ke Zhang commented on FLUME-2653:
---------------------------------
I got this error message when i was trying to update flume settings via
cloudera manager:
{quote}An error occurred while validating this configuration: Component
tsv_agent.sinks.hdfs-sink-view.hdfs.inUseSuffix: Property value missing.{quote}
As workaround I extended the HDFSEventSink like below:
{code:title=MyHDFSEventSink.java|borderStyle=solid}
public static final String IN_USE_SUFFIX_ENABLED_KEY =
"hdfs.inUseSuffixEnabled";
public static final String IN_USE_SUFFIX_KEY = "hdfs.inUseSuffix";
@Override
public void configure(Context context) {
boolean inUseSuffixEnabled = context.getBoolean(IN_USE_SUFFIX_ENABLED_KEY,
true);
if (!inUseSuffixEnabled) {
context.put(IN_USE_SUFFIX_KEY, "");
}
super.configure(context);
}
{code}
and updated flume sink type to:
{code:title=flume-conf.properties}
my_agent.sinks.my_hdfs_sink.type = com.mycompany.MyHDFSEventSink
my_agent.sinks.my_hdfs_sink.hdfs.inUseSuffixEnabled = false
# ...
{code}
> Allow inUseSuffix to be null/empty
> ----------------------------------
>
> Key: FLUME-2653
> URL: https://issues.apache.org/jira/browse/FLUME-2653
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources
> Affects Versions: v1.5.1
> Reporter: Andrew Jones
> Labels: hdfssink
>
> At the moment, it doesn't seem possible to set the null/empty. We've tried
> {{''}} which just adds the quotes to the end, and setting to nothing, which
> just uses the default {{.tmp}}.
> We want the _in use_ file to have the same name as the _closed_ file, so we
> can read from files that are in use without the file moving from underneath
> us. In our use case, we know that an in use file is still readable and
> parseable, because it is just text with a JSON document per line.
> It looks like [the HDFS sink
> code|https://github.com/apache/flume/blob/542b1695033d330eb00ae81713fdc838b88332b6/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java#L618]
> can handle this change already, but at the moment there is no way to set the
> {{bucketPath}} and {{targetPath}} to be the same.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
