[ https://issues.apache.org/jira/browse/NIFI-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15423640#comment-15423640 ]
ASF GitHub Bot commented on NIFI-2547: -------------------------------------- Github user mattyb149 commented on a diff in the pull request: https://github.com/apache/nifi/pull/850#discussion_r75044365 --- Diff: nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/test/java/org/apache/nifi/processors/hadoop/TestDeleteHDFS.java --- @@ -0,0 +1,187 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.hadoop; + +import static org.junit.Assert.assertEquals; +import static org.mockito.Mockito.any; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + +import java.io.IOException; +import java.util.List; +import java.util.Map; + +import org.apache.hadoop.fs.FileStatus; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.hadoop.KerberosProperties; +import org.apache.nifi.util.MockFlowFile; +import org.apache.nifi.util.NiFiProperties; +import org.apache.nifi.util.TestRunner; +import org.apache.nifi.util.TestRunners; +import org.junit.Before; +import org.junit.Test; + +import com.google.common.collect.Maps; + +public class TestDeleteHDFS { + private NiFiProperties mockNiFiProperties; + private FileSystem mockFileSystem; + private KerberosProperties kerberosProperties; + + @Before + public void setup() throws Exception { + mockNiFiProperties = mock(NiFiProperties.class); + when(mockNiFiProperties.getKerberosConfigurationFile()).thenReturn(null); + kerberosProperties = KerberosProperties.create(mockNiFiProperties); + mockFileSystem = mock(FileSystem.class); --- End diff -- Is it better to mock here or use the target/ area (since HDFS defaults to local FS) for file existence, deletion, etc.? > Add DeleteHDFS Processor > ------------------------- > > Key: NIFI-2547 > URL: https://issues.apache.org/jira/browse/NIFI-2547 > Project: Apache NiFi > Issue Type: New Feature > Reporter: Ricky Saltzer > Assignee: Ricky Saltzer > > There are times where a user may want to remove a file or directory from > HDFS. The reasons for this vary, but to provide some context, I currently > have a pipeline where I need to periodically delete files that my NiFi > pipeline is producing. In my case, it's a "Delete files after they are 7 days > old". > Currently, I have to use the {{ExecuteStreamCommand}} processor and manually > call {{hdfs dfs -rm}}, which is awful when dealing with a large amount of > files. For one, an entire JVM is spun up for each delete, and two, when > deleting directories with thousands of files, it can sometimes cause the > command to hang indefinitely. > With that being said, I am proposing we add a {{DeleteHDFS}} processor which > meets the following criteria. > * Can delete both directories and files > * Can delete directories recursively > * Supports the dynamic expression language > * Supports using glob paths (e.g. /data/for/2017/08/*) > * Capable of being a downstream processor as well as a standalone processor -- This message was sent by Atlassian JIRA (v6.3.4#6332)