[ https://issues.apache.org/jira/browse/HDFS-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jan Filipiak updated HDFS-8836: ------------------------------- Description: Hello everyone, I recently was in the need of using the new line option -nl with getMerge because the files I needed to merge simply didn't had one. I was merging all the files from one directory and unfortunately this directory also included empty files, which effectively led to multiple newlines append after some files. I needed to remove them manually afterwards. In this situation it is maybe good to have another argument that allows skipping empty files. Thing one could try to implement this feature: The call for IOUtils.copyBytes(in, out, getConf(), false); doesn't return the number of bytes copied which would be convenient as one could skip append the new line when 0 bytes where copied or one would check the file size before. I posted this Idea on the mailing list http://mail-archives.apache.org/mod_mbox/hadoop-user/201507.mbox/%3C55B25140.3060005%40trivago.com%3E but I didn't really get many responses, so I thought I my try this way. was: Hello everyone, I recently was in the need of using the new line option -nl with getMerge because the files I needed to merge simply didn't had one. I was merging all the files from one directory and unfortunately this directory also included empty files, which effectively led to multiple newlines append after some files. I needed to remove them manually afterwards. In this situation it is maybe good to have another argument that allows skipping empty files. Thing one could try to implement this feature: The call for IOUtils.copyBytes(in, out, getConf(), false); doesn't return the number of bytes copied which would be convenient as one could skip append the new line when 0 bytes where copied or one would check the file size before. I posted this Idea on the mailing list http://mail-archives.apache.org/mod_mbox/hadoop-user/201507.mbox/%3C55B25140.3060005%40trivago.com%3E but I didn't really get many responses, so I thought I my try this way. > Skip newline on empty files with getMerge -nl > --------------------------------------------- > > Key: HDFS-8836 > URL: https://issues.apache.org/jira/browse/HDFS-8836 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client > Affects Versions: 2.6.0, 2.7.1 > Reporter: Jan Filipiak > Priority: Trivial > > Hello everyone, > I recently was in the need of using the new line option -nl with getMerge > because the files I needed to merge simply didn't had one. I was merging all > the files from one directory and unfortunately this directory also included > empty files, which effectively led to multiple newlines append after some > files. I needed to remove them manually afterwards. > In this situation it is maybe good to have another argument that allows > skipping empty files. > Thing one could try to implement this feature: > The call for IOUtils.copyBytes(in, out, getConf(), false); doesn't > return the number of bytes copied which would be convenient as one could > skip append the new line when 0 bytes where copied or one would check the > file size before. > I posted this Idea on the mailing list > http://mail-archives.apache.org/mod_mbox/hadoop-user/201507.mbox/%3C55B25140.3060005%40trivago.com%3E > but I didn't really get many responses, so I thought I my try this way. -- This message was sent by Atlassian JIRA (v6.3.4#6332)