[jira] [Commented] (FLUME-2273) ElasticSearchSink: Add handling for header substitution in indexName
[ https://issues.apache.org/jira/browse/FLUME-2273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14016269#comment-14016269 ] Hudson commented on FLUME-2273: --- SUCCESS: Integrated in flume-trunk #636 (See [https://builds.apache.org/job/flume-trunk/636/]) FLUME-2273 - Add handling for header substitution in ElasticSearchSink (juhani_connolly: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.gita=commith=09472ba12278a0d3696b9d2e26d6d1b0d361c830) * flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/client/ElasticSearchRestClient.java * flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/SimpleIndexNameBuilder.java * flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/AbstractElasticSearchIndexRequestBuilderFactory.java * flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/client/ElasticSearchTransportClient.java * flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/TimeBasedIndexNameBuilder.java * flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchSink.java * flume-ng-doc/sphinx/FlumeUserGuide.rst * flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/TestElasticSearchIndexRequestBuilderFactory.java ElasticSearchSink: Add handling for header substitution in indexName Key: FLUME-2273 URL: https://issues.apache.org/jira/browse/FLUME-2273 Project: Flume Issue Type: Improvement Components: Sinks+Sources Affects Versions: v1.4.0 Reporter: Paul Merry Priority: Minor Attachments: FLUME-2273.patch, new_FLUME-2273-2.patch, new_FLUME-2273-5.patch, new_FLUME-2273-6.patch, new_FLUME-2273.patch The ElasticSearchSink would be improved by allowing for header substitution in the indexName property. A use case is where the sink is an intermediate part of a chain and the index name is required to identify the message origin, at present it can only be a hardcoded value. The HDFS sink already supports header substitution so a similar format would maintain consistency. -- This message was sent by Atlassian JIRA (v6.2#6252)
a bug for flume-ng 4.5
hi ?all? ? In ?file AbstractHDFSWriter.java ?, the definition of function
?closeHDFSOutputStream ?? ? as follow ? ?which can throw IOexception on some
err.? ?? ??? ??? ??? ??? ??? ??protected void
closeHDFSOutputStream(OutputStream outputStream)?? ?? ??? ??? ??? ??? ???
??throws IOException{? ?But at the end of this function ?, it catch the
IOexception and never rethrow it !!!?? ??? ??? ??? ??? ??? ? ?catch
(IOException e) {?? ??? ??? ??? ??? ??? ??? ?? ??logger.error(Unable to close
HDFS file: ' + destPath + ', e);? ? ? ? ? ? ? ? ? ? ? ? ? }
? ? The funciton closeHDFSOutputStream ? is called by hdfsSink。 The bug ?make
? BucketWrite.close function ?always ? looks right and can?BucketWrite.rename?。
And It result in the status of ? files writed by flume in hdfs ?is openforwrite
?(but it is renamed succ ) 。 ? ??? ??? ??? ??? ??? ??? ?
namesuperw...@gmail.com
[jira] [Created] (FLUME-2394) Command line argument to disable monitoring for config changes
Rahul Ravindran created FLUME-2394: -- Summary: Command line argument to disable monitoring for config changes Key: FLUME-2394 URL: https://issues.apache.org/jira/browse/FLUME-2394 Project: Flume Issue Type: New Feature Components: Configuration Affects Versions: v1.5.0 Reporter: Rahul Ravindran Fix For: v1.6.0 Flume monitors for changes to the config file and attempts to re-initialize source/sinks/channels based on detected changes. However, this does not work for all config values, and also in undesirable in a lot of production environments where puppet/chef modifies the config, and likely restart flume. It would be good to have an optional command line argument which would disable this monitoring and require flume to be restarted for config changes. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (FLUME-2394) Command line argument to disable monitoring for config changes
[ https://issues.apache.org/jira/browse/FLUME-2394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rahul Ravindran updated FLUME-2394: --- Description: Flume monitors for changes to the config file and attempts to re-initialize source/sinks/channels based on detected changes. However, this does not work for all config values, and also in undesirable in a lot of production environments where puppet/chef modifies the config, and likely restarts flume. It would be good to have an optional command line argument which would disable this monitoring and require flume to be restarted for config changes. We can control the restart using variety of orchestration mechanisms (was: Flume monitors for changes to the config file and attempts to re-initialize source/sinks/channels based on detected changes. However, this does not work for all config values, and also in undesirable in a lot of production environments where puppet/chef modifies the config, and likely restart flume. It would be good to have an optional command line argument which would disable this monitoring and require flume to be restarted for config changes.) Command line argument to disable monitoring for config changes -- Key: FLUME-2394 URL: https://issues.apache.org/jira/browse/FLUME-2394 Project: Flume Issue Type: New Feature Components: Configuration Affects Versions: v1.5.0 Reporter: Rahul Ravindran Fix For: v1.6.0 Flume monitors for changes to the config file and attempts to re-initialize source/sinks/channels based on detected changes. However, this does not work for all config values, and also in undesirable in a lot of production environments where puppet/chef modifies the config, and likely restarts flume. It would be good to have an optional command line argument which would disable this monitoring and require flume to be restarted for config changes. We can control the restart using variety of orchestration mechanisms -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (FLUME-2395) Flume does not shutdown cleanly on sending a term signal when it is receiving events
Rahul Ravindran created FLUME-2395: -- Summary: Flume does not shutdown cleanly on sending a term signal when it is receiving events Key: FLUME-2395 URL: https://issues.apache.org/jira/browse/FLUME-2395 Project: Flume Issue Type: Bug Affects Versions: v1.5.0 Reporter: Rahul Ravindran Running flume with avrosource, file channel, avro sink Generated a reasonably high load where flume is receiving avro events via the avro source. Now, send a kill flume pid. Flume does not die. Doing the same, after using iptables to block the port on the running flume process ensures that the flume process does die gracefully on sending a TERM signal -- This message was sent by Atlassian JIRA (v6.2#6252)
