[jira] [Commented] (NIFI-12837) Add DFS setting to smb processors
[
https://issues.apache.org/jira/browse/NIFI-12837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17835504#comment-17835504
]
ASF subversion and git services commented on NIFI-12837:
Commit f7cd8752ea10f92281bf126e60c320e5d449c72a in nifi's branch
refs/heads/support/nifi-1.x from Pierre Villard
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=f7cd8752ea ]
NIFI-12837 - Fix checkstyle issue following a manual cherrypick
> Add DFS setting to smb processors
> -
>
> Key: NIFI-12837
> URL: https://issues.apache.org/jira/browse/NIFI-12837
> Project: Apache NiFi
> Issue Type: Improvement
>Affects Versions: 1.25.0, 2.0.0-M2
>Reporter: Anders
>Assignee: Peter Turcsanyi
>Priority: Major
> Fix For: 2.0.0-M3, 1.26.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The hierynomus/smbj library has a setting for enabling DFS which is disabled
> by default:
> https://github.com/hierynomus/smbj/blob/f25d5c5478a5b73e9ba4202dcfb365974e15367e/src/main/java/com/hierynomus/smbj/SmbConfig.java#L106C17-L106C39
> This appears to cause problems in some SMB configurations.
> Patched
> https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-smb-bundle/nifi-smb-smbj-common/src/main/java/org/apache/nifi/smb/common/SmbUtils.java
> to test in my environment with:
> {code}
> $ git diff
> nifi-smb-smbj-common/src/main/java/org/apache/nifi/smb/common/SmbUtils.java
> diff --git
> a/nifi-nar-bundles/nifi-smb-bundle/nifi-smb-smbj-common/src/main/java/org/apache/nifi/smb/common/SmbUtils.java
>
> b/nifi-nar-bundles/nifi-smb-bundle/nifi-smb-smbj-common/src/main/java/org/apache/nifi/smb/common/SmbUtils.java
> index 0895abfae0..eac765 100644
> ---
> a/nifi-nar-bundles/nifi-smb-bundle/nifi-smb-smbj-common/src/main/java/org/apache/nifi/smb/common/SmbUtils.java
> +++
> b/nifi-nar-bundles/nifi-smb-bundle/nifi-smb-smbj-common/src/main/java/org/apache/nifi/smb/common/SmbUtils.java
> @@ -46,6 +46,8 @@ public final class SmbUtils {
> }
> }
> +configBuilder.withDfsEnabled(true);
> +
> if (context.getProperty(USE_ENCRYPTION).isSet()) {
>
> configBuilder.withEncryptData(context.getProperty(USE_ENCRYPTION).asBoolean());
> }
> {code}
> This appeared to resolve the issue.
> It would be very useful if this setting was available to toggle in the UI for
> all SMB processors.
> Without this setting, I get a *STATUS_PATH_NOT_COVERED* error.
> Somewhat related hierynomus/smbj github issues:
> https://github.com/hierynomus/smbj/issues/152
> https://github.com/hierynomus/smbj/issues/419
> This setting should be made available in the following processors and
> services:
> * GetSmbFile
> * PutSmbFile
> * SmbjClientProviderService
> Edit: It might require some more changes to handle the connections and
> sessions correctly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-12837) Add DFS setting to smb processors
[
https://issues.apache.org/jira/browse/NIFI-12837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17835415#comment-17835415
]
ASF subversion and git services commented on NIFI-12837:
Commit 10ce00862d1479dee1444d0da9cc48bbbd83729f in nifi's branch
refs/heads/support/nifi-1.x from Peter Turcsanyi
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=10ce00862d ]
NIFI-12837 Added DFS support in SMB processors
Signed-off-by: Pierre Villard
This closes #8527.
> Add DFS setting to smb processors
> -
>
> Key: NIFI-12837
> URL: https://issues.apache.org/jira/browse/NIFI-12837
> Project: Apache NiFi
> Issue Type: Improvement
>Affects Versions: 1.25.0, 2.0.0-M2
>Reporter: Anders
>Assignee: Peter Turcsanyi
>Priority: Major
> Fix For: 2.0.0-M3
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The hierynomus/smbj library has a setting for enabling DFS which is disabled
> by default:
> https://github.com/hierynomus/smbj/blob/f25d5c5478a5b73e9ba4202dcfb365974e15367e/src/main/java/com/hierynomus/smbj/SmbConfig.java#L106C17-L106C39
> This appears to cause problems in some SMB configurations.
> Patched
> https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-smb-bundle/nifi-smb-smbj-common/src/main/java/org/apache/nifi/smb/common/SmbUtils.java
> to test in my environment with:
> {code}
> $ git diff
> nifi-smb-smbj-common/src/main/java/org/apache/nifi/smb/common/SmbUtils.java
> diff --git
> a/nifi-nar-bundles/nifi-smb-bundle/nifi-smb-smbj-common/src/main/java/org/apache/nifi/smb/common/SmbUtils.java
>
> b/nifi-nar-bundles/nifi-smb-bundle/nifi-smb-smbj-common/src/main/java/org/apache/nifi/smb/common/SmbUtils.java
> index 0895abfae0..eac765 100644
> ---
> a/nifi-nar-bundles/nifi-smb-bundle/nifi-smb-smbj-common/src/main/java/org/apache/nifi/smb/common/SmbUtils.java
> +++
> b/nifi-nar-bundles/nifi-smb-bundle/nifi-smb-smbj-common/src/main/java/org/apache/nifi/smb/common/SmbUtils.java
> @@ -46,6 +46,8 @@ public final class SmbUtils {
> }
> }
> +configBuilder.withDfsEnabled(true);
> +
> if (context.getProperty(USE_ENCRYPTION).isSet()) {
>
> configBuilder.withEncryptData(context.getProperty(USE_ENCRYPTION).asBoolean());
> }
> {code}
> This appeared to resolve the issue.
> It would be very useful if this setting was available to toggle in the UI for
> all SMB processors.
> Without this setting, I get a *STATUS_PATH_NOT_COVERED* error.
> Somewhat related hierynomus/smbj github issues:
> https://github.com/hierynomus/smbj/issues/152
> https://github.com/hierynomus/smbj/issues/419
> This setting should be made available in the following processors and
> services:
> * GetSmbFile
> * PutSmbFile
> * SmbjClientProviderService
> Edit: It might require some more changes to handle the connections and
> sessions correctly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-12837) Add DFS setting to smb processors
[
https://issues.apache.org/jira/browse/NIFI-12837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17835372#comment-17835372
]
ASF subversion and git services commented on NIFI-12837:
Commit 928206d52f3a442b24eb5a911507af821c30ea2d in nifi's branch
refs/heads/main from Peter Turcsanyi
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=928206d52f ]
NIFI-12837 Added DFS support in SMB processors
Signed-off-by: Pierre Villard
This closes #8527.
> Add DFS setting to smb processors
> -
>
> Key: NIFI-12837
> URL: https://issues.apache.org/jira/browse/NIFI-12837
> Project: Apache NiFi
> Issue Type: Improvement
>Affects Versions: 1.25.0, 2.0.0-M2
>Reporter: Anders
>Assignee: Peter Turcsanyi
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The hierynomus/smbj library has a setting for enabling DFS which is disabled
> by default:
> https://github.com/hierynomus/smbj/blob/f25d5c5478a5b73e9ba4202dcfb365974e15367e/src/main/java/com/hierynomus/smbj/SmbConfig.java#L106C17-L106C39
> This appears to cause problems in some SMB configurations.
> Patched
> https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-smb-bundle/nifi-smb-smbj-common/src/main/java/org/apache/nifi/smb/common/SmbUtils.java
> to test in my environment with:
> {code}
> $ git diff
> nifi-smb-smbj-common/src/main/java/org/apache/nifi/smb/common/SmbUtils.java
> diff --git
> a/nifi-nar-bundles/nifi-smb-bundle/nifi-smb-smbj-common/src/main/java/org/apache/nifi/smb/common/SmbUtils.java
>
> b/nifi-nar-bundles/nifi-smb-bundle/nifi-smb-smbj-common/src/main/java/org/apache/nifi/smb/common/SmbUtils.java
> index 0895abfae0..eac765 100644
> ---
> a/nifi-nar-bundles/nifi-smb-bundle/nifi-smb-smbj-common/src/main/java/org/apache/nifi/smb/common/SmbUtils.java
> +++
> b/nifi-nar-bundles/nifi-smb-bundle/nifi-smb-smbj-common/src/main/java/org/apache/nifi/smb/common/SmbUtils.java
> @@ -46,6 +46,8 @@ public final class SmbUtils {
> }
> }
> +configBuilder.withDfsEnabled(true);
> +
> if (context.getProperty(USE_ENCRYPTION).isSet()) {
>
> configBuilder.withEncryptData(context.getProperty(USE_ENCRYPTION).asBoolean());
> }
> {code}
> This appeared to resolve the issue.
> It would be very useful if this setting was available to toggle in the UI for
> all SMB processors.
> Without this setting, I get a *STATUS_PATH_NOT_COVERED* error.
> Somewhat related hierynomus/smbj github issues:
> https://github.com/hierynomus/smbj/issues/152
> https://github.com/hierynomus/smbj/issues/419
> This setting should be made available in the following processors and
> services:
> * GetSmbFile
> * PutSmbFile
> * SmbjClientProviderService
> Edit: It might require some more changes to handle the connections and
> sessions correctly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-12837) Add DFS setting to smb processors
[
https://issues.apache.org/jira/browse/NIFI-12837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17828205#comment-17828205
]
Anders commented on NIFI-12837:
---
Been testing the PR since yesterday, and so far it seems to be working fine.
I have not had any "Write failed (BROKEN PIPE)" errors either, so fingers
crossed.
The "Temporary suffix" option still does not work though (but that is a minor
issue).
> Add DFS setting to smb processors
> -
>
> Key: NIFI-12837
> URL: https://issues.apache.org/jira/browse/NIFI-12837
> Project: Apache NiFi
> Issue Type: Improvement
>Affects Versions: 1.25.0, 2.0.0-M2
>Reporter: Anders
>Assignee: Peter Turcsanyi
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The hierynomus/smbj library has a setting for enabling DFS which is disabled
> by default:
> https://github.com/hierynomus/smbj/blob/f25d5c5478a5b73e9ba4202dcfb365974e15367e/src/main/java/com/hierynomus/smbj/SmbConfig.java#L106C17-L106C39
> This appears to cause problems in some SMB configurations.
> Patched
> https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-smb-bundle/nifi-smb-smbj-common/src/main/java/org/apache/nifi/smb/common/SmbUtils.java
> to test in my environment with:
> {code}
> $ git diff
> nifi-smb-smbj-common/src/main/java/org/apache/nifi/smb/common/SmbUtils.java
> diff --git
> a/nifi-nar-bundles/nifi-smb-bundle/nifi-smb-smbj-common/src/main/java/org/apache/nifi/smb/common/SmbUtils.java
>
> b/nifi-nar-bundles/nifi-smb-bundle/nifi-smb-smbj-common/src/main/java/org/apache/nifi/smb/common/SmbUtils.java
> index 0895abfae0..eac765 100644
> ---
> a/nifi-nar-bundles/nifi-smb-bundle/nifi-smb-smbj-common/src/main/java/org/apache/nifi/smb/common/SmbUtils.java
> +++
> b/nifi-nar-bundles/nifi-smb-bundle/nifi-smb-smbj-common/src/main/java/org/apache/nifi/smb/common/SmbUtils.java
> @@ -46,6 +46,8 @@ public final class SmbUtils {
> }
> }
> +configBuilder.withDfsEnabled(true);
> +
> if (context.getProperty(USE_ENCRYPTION).isSet()) {
>
> configBuilder.withEncryptData(context.getProperty(USE_ENCRYPTION).asBoolean());
> }
> {code}
> This appeared to resolve the issue.
> It would be very useful if this setting was available to toggle in the UI for
> all SMB processors.
> Without this setting, I get a *STATUS_PATH_NOT_COVERED* error.
> Somewhat related hierynomus/smbj github issues:
> https://github.com/hierynomus/smbj/issues/152
> https://github.com/hierynomus/smbj/issues/419
> This setting should be made available in the following processors and
> services:
> * GetSmbFile
> * PutSmbFile
> * SmbjClientProviderService
> Edit: It might require some more changes to handle the connections and
> sessions correctly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-12837) Add DFS setting to smb processors
[
https://issues.apache.org/jira/browse/NIFI-12837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17827002#comment-17827002
]
Anders commented on NIFI-12837:
---
Thank you for replying Peter!
Even if the DFS feature would be somewhat experimental, it would still be very
useful for those able to utilize ut.
Without it, I (and possibly others) can't use the SMB processors at all.
I've just patched in support for it in the PutSmbFile processor I am running,
but I'm no java coder, and it's not pretty.
I guess I might also be running into the first issue you are mentioning. After
some time (not necessarily long), I end up getting
{code}
ERROR [Timer-Driven Process Thread-X] o.apache.nifi.processors.smb.PutSmbFile
PutSmbFile[id=...] Could not establish smb connection because of error
com.hierynomus.smbj.common.SMBRuntimeException:
com.hierynomus.protocol.transport.TransportException:
java.net.SocketException: Broken pipe (Write failed)
{code}
I've also noticed that the "Temporary Suffix" option in PutSmbFile does not
work for me when I've enbled DFS and talk to the DFS-address. It gives me
STATUS_OBJECT_NOT_FOUND. I works if I talk directly to the nodes in the cluster
though.
Either way, adding the DFS options as a toggle (default false) and adding the
workaround you mention would be extremely useful.
> Add DFS setting to smb processors
> -
>
> Key: NIFI-12837
> URL: https://issues.apache.org/jira/browse/NIFI-12837
> Project: Apache NiFi
> Issue Type: Improvement
>Affects Versions: 1.25.0
>Reporter: Anders
>Assignee: Peter Turcsanyi
>Priority: Major
>
> The hierynomus/smbj library has a setting for enabling DFS which is disabled
> by default:
> https://github.com/hierynomus/smbj/blob/f25d5c5478a5b73e9ba4202dcfb365974e15367e/src/main/java/com/hierynomus/smbj/SmbConfig.java#L106C17-L106C39
> This appears to cause problems in some SMB configurations.
> Patched
> https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-smb-bundle/nifi-smb-smbj-common/src/main/java/org/apache/nifi/smb/common/SmbUtils.java
> to test in my environment with:
> {code}
> $ git diff
> nifi-smb-smbj-common/src/main/java/org/apache/nifi/smb/common/SmbUtils.java
> diff --git
> a/nifi-nar-bundles/nifi-smb-bundle/nifi-smb-smbj-common/src/main/java/org/apache/nifi/smb/common/SmbUtils.java
>
> b/nifi-nar-bundles/nifi-smb-bundle/nifi-smb-smbj-common/src/main/java/org/apache/nifi/smb/common/SmbUtils.java
> index 0895abfae0..eac765 100644
> ---
> a/nifi-nar-bundles/nifi-smb-bundle/nifi-smb-smbj-common/src/main/java/org/apache/nifi/smb/common/SmbUtils.java
> +++
> b/nifi-nar-bundles/nifi-smb-bundle/nifi-smb-smbj-common/src/main/java/org/apache/nifi/smb/common/SmbUtils.java
> @@ -46,6 +46,8 @@ public final class SmbUtils {
> }
> }
> +configBuilder.withDfsEnabled(true);
> +
> if (context.getProperty(USE_ENCRYPTION).isSet()) {
>
> configBuilder.withEncryptData(context.getProperty(USE_ENCRYPTION).asBoolean());
> }
> {code}
> This appeared to resolve the issue.
> It would be very useful if this setting was available to toggle in the UI for
> all SMB processors.
> Without this setting, I get a *STATUS_PATH_NOT_COVERED* error.
> Somewhat related hierynomus/smbj github issues:
> https://github.com/hierynomus/smbj/issues/152
> https://github.com/hierynomus/smbj/issues/419
> This setting should be made available in the following processors and
> services:
> * GetSmbFile
> * PutSmbFile
> * SmbjClientProviderService
> Edit: It might require some more changes to handle the connections and
> sessions correctly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-12837) Add DFS setting to smb processors
[
https://issues.apache.org/jira/browse/NIFI-12837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17826782#comment-17826782
]
Peter Turcsanyi commented on NIFI-12837:
[~andersns] Thanks for filing this feature request!
I did some investigation around DFS support in smbj and how it could be used in
NiFi a few months ago. You are right that turning it on is quite
straightforward. However, I ran into two bugs in the library (there may be
[others|https://github.com/hierynomus/smbj/issues?q=is%3Aissue+is%3Aopen+DFS]
too):
* [https://github.com/hierynomus/smbj/issues/796] - Stale connections in
SMBClient.connectionTable
* [https://github.com/hierynomus/smbj/issues/717] - Null Pointer when opening
DFS link
They should be fixed on the smbj side. Suggestions were provided how to handle
them, but still no answers.
Fortunately, there is a workaround to the first one that can be implemented in
NiFi.
The second one is a bit specific:
- affects the List processor only
- occurs only when the DFS root or a parent directory (not a link) is listed;
when Directory property is set to a link directly, it works
- managed to replicate on Samba DFS share and standalone Windows instances; it
seems to work in Windows AD Domain where the DFS namespace is hosted by Domain
Controller(s)
Having said that, I don't feel smbj's DFS support pretty robust but we can go
ahead and add it in the SMB processors as an "experimental" feature.
> Add DFS setting to smb processors
> -
>
> Key: NIFI-12837
> URL: https://issues.apache.org/jira/browse/NIFI-12837
> Project: Apache NiFi
> Issue Type: Improvement
>Affects Versions: 1.25.0
>Reporter: Anders
>Priority: Major
>
> The hierynomus/smbj library has a setting for enabling DFS which is disabled
> by default:
> https://github.com/hierynomus/smbj/blob/f25d5c5478a5b73e9ba4202dcfb365974e15367e/src/main/java/com/hierynomus/smbj/SmbConfig.java#L106C17-L106C39
> This appears to cause problems in some SMB configurations.
> Patched
> https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-smb-bundle/nifi-smb-smbj-common/src/main/java/org/apache/nifi/smb/common/SmbUtils.java
> to test in my environment with:
> {code}
> $ git diff
> nifi-smb-smbj-common/src/main/java/org/apache/nifi/smb/common/SmbUtils.java
> diff --git
> a/nifi-nar-bundles/nifi-smb-bundle/nifi-smb-smbj-common/src/main/java/org/apache/nifi/smb/common/SmbUtils.java
>
> b/nifi-nar-bundles/nifi-smb-bundle/nifi-smb-smbj-common/src/main/java/org/apache/nifi/smb/common/SmbUtils.java
> index 0895abfae0..eac765 100644
> ---
> a/nifi-nar-bundles/nifi-smb-bundle/nifi-smb-smbj-common/src/main/java/org/apache/nifi/smb/common/SmbUtils.java
> +++
> b/nifi-nar-bundles/nifi-smb-bundle/nifi-smb-smbj-common/src/main/java/org/apache/nifi/smb/common/SmbUtils.java
> @@ -46,6 +46,8 @@ public final class SmbUtils {
> }
> }
> +configBuilder.withDfsEnabled(true);
> +
> if (context.getProperty(USE_ENCRYPTION).isSet()) {
>
> configBuilder.withEncryptData(context.getProperty(USE_ENCRYPTION).asBoolean());
> }
> {code}
> This appeared to resolve the issue.
> It would be very useful if this setting was available to toggle in the UI for
> all SMB processors.
> Without this setting, I get a *STATUS_PATH_NOT_COVERED* error.
> Somewhat related hierynomus/smbj github issues:
> https://github.com/hierynomus/smbj/issues/152
> https://github.com/hierynomus/smbj/issues/419
> This setting should be made available in the following processors and
> services:
> * GetSmbFile
> * PutSmbFile
> * SmbjClientProviderService
> Edit: It might require some more changes to handle the connections and
> sessions correctly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
