[
https://issues.apache.org/jira/browse/FLUME-2653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16687844#comment-16687844
]
ASF GitHub Bot commented on FLUME-2653:
---------------------------------------
GitHub user majorendre opened a pull request:
https://github.com/apache/flume/pull/237
FLUME-2653 Allow hdfs sink inUseSuffix to be empty
This is based on the contributions for FLUME-2653 regarding a new feature
for the hdfs sink.
Added a new parameter hdfs.emptyInUseSuffix to allow the output file name
to remain unchanged.
See the user guide changes for details.
This is desired feature from the community.
I added a new junit test case for testing.
Temporarily modified old test cases in my ide to use the new flag, and they
passed. I did this just as one of test, to be on the safe side. It is not in
this PR.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/majorendre/flume FLUME-2653
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flume/pull/237.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #237
----
commit 930476595e70b2ecb5fd3a21a732b82391d351f8
Author: Endre Major <emajor@...>
Date: 2018-11-14T17:44:02Z
FLUME-2653 Allow inUseSuffix to be empty
----
> Allow inUseSuffix to be null/empty
> ----------------------------------
>
> Key: FLUME-2653
> URL: https://issues.apache.org/jira/browse/FLUME-2653
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources
> Affects Versions: 1.5.1
> Reporter: Andrew Jones
> Assignee: Endre Major
> Priority: Major
> Labels: docs-missing, hdfssink
> Fix For: 1.9.0
>
> Attachments: FLUME-2653.patch
>
>
> At the moment, it doesn't seem possible to set the null/empty. We've tried
> {{''}} which just adds the quotes to the end, and setting to nothing, which
> just uses the default {{.tmp}}.
> We want the _in use_ file to have the same name as the _closed_ file, so we
> can read from files that are in use without the file moving from underneath
> us. In our use case, we know that an in use file is still readable and
> parseable, because it is just text with a JSON document per line.
> It looks like [the HDFS sink
> code|https://github.com/apache/flume/blob/542b1695033d330eb00ae81713fdc838b88332b6/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java#L618]
> can handle this change already, but at the moment there is no way to set the
> {{bucketPath}} and {{targetPath}} to be the same.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
