I think the correct way to move forward is to go ahead and keep the file
separator,
as it is consistent with how the GetFile processor works and probably most
intuitive
for windows users. However, I do think that using a forward-slash instead does
provide
benefit as it is portable across operating systems.
At this time, we cannot really change GetFile's behavior, though, as it could
certainly break
existing flows that depend on the existing style of slashes.
With the given reviews around the ticket indicating that builds are successful
now on OS X,
Linux, and Windows and given that this is basically the only ticket preventing
the 0.4.0 release from occurring, I would like to go ahead and merge in the
patch, and we should open
up a bigger discussion for "How should NiFi handle path separators" for the
1.0.0 release, where we have
more flexibility in changing these things since it is a major release.
Thanks
-Mark
> On Dec 7, 2015, at 11:32 AM, Tony Kurc <trk...@gmail.com> wrote:
>
> Probably want a function to go from dos to unix filenames. That replace is
> a little dangerous. Although is this seriously a use case?
> On Dec 7, 2015 10:44 AM, "Mark Payne" <marka...@hotmail.com> wrote:
>
>> Mike,
>>
>> That is accurate that Linux will not recognize \ as a path separator. It
>> would have no
>> problem reading the attribute but would not be able to write to a
>> directory/file with
>> that name. If you wanted to use that as a path in linux you could
>> certainly reference it
>> via:
>>
>> ${path:replace("\\", "/")}
>>
>> However, this problem already exists today with GetFile - it is using the
>> File.separator
>> so that if you have a path like "subDir1/subDir2/myFile.txt" in Windows,
>> the path attribute
>> will be "subDir1\subDir2/" with both a forward slash and a backslash. I'm
>> suggesting that
>> with ListFile instead of using "subDir1\subDir2/" as the attribute it
>> should be "subDir1\subDir2\"
>>
>> Thanks
>> -Mark
>>
>>
>>> On Dec 7, 2015, at 10:30 AM, Michael Moser <moser...@gmail.com> wrote:
>>>
>>> I disagree Mark, mainly due to the flowfile attributes such as 'path'
>> that
>>> processors like GetFile (and I assume ListFile) create. Windows supports
>>> the forward slash but Unixes do not support the back slash
>> File.separator.
>>> What happens when a flowfile created on a Windows NiFi is sent over to a
>>> Unix NiFi? Do Expression Language expressions that expect a forward
>> slash
>>> begin to fail when reading a 'path' attribute that contains backslashes?
>>> I'm concerned ...
>>>
>>> -- Mike
>>>
>>>
>>> On Mon, Dec 7, 2015 at 10:12 AM, Mark Payne <marka...@hotmail.com>
>> wrote:
>>>
>>>> Errr, actually, after thinking about it more - operators who are running
>>>> on Windows should probably expect \ to be used instead of /
>>>>
>>>> So I think the better solution is to get rid of the "/" anywhere in the
>>>> processor and tests and always use File.separator.
>>>>
>>>> Will do so and create a patch, if nobody objects to that.
>>>>
>>>> Thanks
>>>> -Mark
>>>>
>>>>
>>>>> On Dec 7, 2015, at 9:59 AM, Mark Payne <marka...@hotmail.com> wrote:
>>>>>
>>>>> Tony,
>>>>>
>>>>> I think there are really two possible solutions to this:
>>>>>
>>>>> 1) Always use / in the path attributes instead of \ -- i generally
>>>> prefer this approach, as windows has worked with forward slashes since
>> Win
>>>> 98 (I believe?).
>>>>> 2) Have unit test look for file.separator -- benefit here is that it is
>>>> consistent with the way that GetFile works, and I'd not want to change
>> that
>>>> because it's quite likely that some people are routing based on the
>> 'path'
>>>> attribute.
>>>>>
>>>>> Normally I would tend to make consistency a high priority. However, I
>>>> see ListFile / FetchFile largely as a replacement for GetFile and am
>>>> guessing that in the future GetFile will be deprecated and removed. So
>> I am
>>>> less inclined to stay consistent between the 'old generation' and 'new
>>>> generation' of processors. So personally I'd prefer to go the first
>> route.
>>>>>
>>>>> -Mark
>>>>>
>>>>>
>>>>>> On Dec 7, 2015, at 12:35 AM, Tony Kurc <trk...@gmail.com> wrote:
>>>>>>
>>>>>> I submitted a patch to get the test to pass (NIFI-1261). Seems a bit
>>>> icky,
>>>>>> but I'll defer to Joe Skora and Mark Payne for correct behavior.
>>>>>>
>>>>>> On Mon, Dec 7, 2015 at 12:11 AM, Tony Kurc <trk...@gmail.com> wrote:
>>>>>>
>>>>>>> Joe - I'm putting a ticket in for a fix. Looks like it was introduced
>>>> by
>>>>>>> the NIFI-1246 patch.
>>>>>>>
>>>>>>> On Sun, Dec 6, 2015 at 11:36 PM, Joe Percivall <
>>>>>>> joeperciv...@yahoo.com.invalid> wrote:
>>>>>>>
>>>>>>>> Yup I saw the same behavior.
>>>>>>>>
>>>>>>>> On the second try (doing mvn clean install -rf
>>>> :nifi-standard-processors)
>>>>>>>> the tailfile error went away. The listFile error still occurred
>>>> though.
>>>>>>>>
>>>>>>>> Joe
>>>>>>>> - - - - - -
>>>>>>>> Joseph Percivall
>>>>>>>> linkedin.com/in/Percivall
>>>>>>>> e: joeperciv...@yahoo.com
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sunday, December 6, 2015 11:32 PM, Tony Kurc <trk...@gmail.com>
>>>> wrote:
>>>>>>>> Er, just the tailfile error
>>>>>>>>
>>>>>>>> On Dec 6, 2015 11:31 PM, "Tony Kurc" <trk...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Joe, I had this happen and it worked on a second try.
>>>>>>>>> On Dec 6, 2015 11:23 PM, "Joe Percivall"
>>>> <joeperciv...@yahoo.com.invalid
>>>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Windows 8 build fails with maven 3.3.3 and Java 1.8.0_65.
>>>>>>>>>>
>>>>>>>>>> I get these error messages:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> TestListFile.testRecurse:441 expected:<subdir1[/]subdir2/> but
>>>>>>>>>> was:<subdir1[\]subdir2/>
>>>>>>>>>>
>>>>>>>>
>>>>
>> TestTailFile.testMultipleRolloversAfterHavingReadAllDataWhileStillRunning:381
>>>>>>>>>> expected:<[world]> but was:<[abc
>>>>>>>>>>
>>>>>>>>>> These were not any of the same errors I saw last time testing on
>>>>>>>> Windows
>>>>>>>>>> a couple weeks ago.
>>>>>>>>>>
>>>>>>>>>> Joe
>>>>>>>>>> - - - - - -
>>>>>>>>>> Joseph Percivall
>>>>>>>>>> linkedin.com/in/Percivall
>>>>>>>>>> e: joeperciv...@yahoo.com
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Sunday, December 6, 2015 9:05 PM, Tony Kurc <trk...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>>> I've gotten confirmation of CentOS 7.1.1503 x86_64, Oracle JDK
>> 8u66
>>>>>>>>>> working
>>>>>>>>>> fine. and Fedora 23 not working with the same error that Andre
>>>>>>>> reported.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Sun, Dec 6, 2015 at 5:50 PM, Tony Kurc <trk...@gmail.com>
>> wrote:
>>>>>>>>>>
>>>>>>>>>>> I'll also try it on windows 10 (again x64_64)
>>>>>>>>>>>
>>>>>>>>>>> On Sun, Dec 6, 2015 at 5:36 PM, <joeperciv...@yahoo.com.invalid>
>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> I can run it on Windows 8 tonight if no one else has.
>>>>>>>>>>>>
>>>>>>>>>>>> Joe
>>>>>>>>>>>>
>>>>>>>>>>>> Sent from my phone
>>>>>>>>>>>>
>>>>>>>>>>>>> On Dec 6, 2015, at 4:09 PM, Tony Kurc <trk...@gmail.com>
>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Signatures and hashes look good.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Built fine on Ubuntu 14.04 x86_64. I even cursed a little bit
>>>>>>>> less at
>>>>>>>>>>>>> TestJdbcHugeStream!
>>>>>>>>>>>>>
>>>>>>>>>>>>> LICENSE, NOTICE and README look good.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Docs look good.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Binary ran successfully.
>>>>>>>>>>>>>
>>>>>>>>>>>>> +1
>>>>>>>>>>>>>
>>>>>>>>>>>>> Did anyone try building on windows?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sat, Dec 5, 2015 at 11:46 PM, Aldrin Piri <
>>>>>>>> aldrinp...@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Followed helper provided by Joe.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Keys good.
>>>>>>>>>>>>>> Signatures good.
>>>>>>>>>>>>>> Hashes good.
>>>>>>>>>>>>>> Source release builds and passes contrib
>>>>>>>>>>>>>> Required docs present and look correct.
>>>>>>>>>>>>>> Checked out copy of repo for specified commit hash and diff'd
>>>>>>>>>> against
>>>>>>>>>>>>>> source bundle. Commit is as anticipated.
>>>>>>>>>>>>>> Ran convenience binary with varying templates all
>> successfully.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Release notes and upgrade/migration guides look good.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Kudos to the community on all the efforts involved with this
>>>>>>>>>> release.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> +1, Release this package as Apache NiFi 0.4.0
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Sat, Dec 5, 2015 at 10:32 PM, Joe Witt <
>> joew...@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hello NiFi Community,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I am pleased to be calling this vote for the source release
>> of
>>>>>>>>>> Apache
>>>>>>>>>>>>>>> NiFi 0.4.0.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The source zip, including signatures, digests, and associated
>>>>>>>>>>>>>>> convenience binaries can be found at:
>>>>>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/nifi/nifi-0.4.0/
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The Git tag is nifi-0.4.0-RC1
>>>>>>>>>>>>>>> The Git commit ID is 191a56f54e3ec178f9f29e1287f23ba66dbf9e43
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>
>> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=191a56f54e3ec178f9f29e1287f23ba66dbf9e43
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Checksums of NiFi 0.4.0 Source Release:
>>>>>>>>>>>>>>> MD5: b69fd7ec632d7569906e20508058556b
>>>>>>>>>>>>>>> SHA1: 31d88ec7a8431ba5935370eb09be7a343c46411c
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Release artifacts are signed with the following key:
>>>>>>>>>>>>>>> https://people.apache.org/keys/committer/joewitt.asc
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> KEYS file available here:
>>>>>>>>>>>>>>> https://dist.apache.org/repos/dist/release/nifi/KEYS
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 152 issues were closed/resolved for this release:
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12333070
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Release note highlights:
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>
>> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version0.4.0
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Migration/Upgrade guidance:
>>>>>>>>>>>>>>>
>>>>>>>>>>
>> https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance
>>>>>>>>>>>>>>>
>>>>>>>> https://cwiki.apache.org/confluence/display/NIFI/Upgrading+NiFi
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The vote will be open for 72 hours.
>>>>>>>>>>>>>>> Please download the release candidate and evaluate the
>>>> necessary
>>>>>>>>>> items
>>>>>>>>>>>>>>> including checking hashes, signatures, build from source, and
>>>>>>>> test.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Then please vote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [ ] +1 Release this package as Apache NiFi 0.4.0
>>>>>>>>>>>>>>> [ ] +0 no opinion
>>>>>>>>>>>>>>> [ ] -1 Do not release this package because...
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>>
>>
>>
