[GitHub] nifi pull request #714: NIFI-2259: HTTP Site-to-Site can't handle DEST_FULL

[ijokarumawak]

GitHub user ijokarumawak opened a pull request:

https://github.com/apache/nifi/pull/714

NIFI-2259: HTTP Site-to-Site can't handle DEST_FULL

Hello, I found a bug related HTTP Site-to-Site push data transport, when 
there's a lots of items queued up at the remote NiFi instance.

HTTP Site-to-Site can't handle TRANSACTION_FINISHED_BUT_DESTINATION_FULL
scenario as expected.

That happens if the remote NiFi's input port destination relationship
becomes full during Site-to-Site client sends data. The data which has
already sent to the remote NiFi has to be committed successfully.
However, the remote NiFi returns 503 as a response of commit HTTP
request. Because it does check port availability.

The port availability check shouldn't be called at commit request, since
the session at source NiFi has already been committed. The remote NiFi
should commit its session as well, and return
TRANSACTION_FINISHED_BUT_DESTINATION_FULL response.

This fix makes a remote NiFi to keep the handshaken properties when it holds
transaction to be committed. Then if a transaction already has
handshaken properties, then use it, instead of doing a handshake process
again.

Please review the change, thank you!

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijokarumawak/nifi nifi-2259

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/714.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #714


commit 79521d8cd01c0675bd8bd4d6a9f9382e11ca9d6b
Author: Koji Kawamura 
Date:   2016-07-25T03:14:12Z

NIFI-2259: HTTP Site-to-Site can't handle DEST_FULL

HTTP Site-to-Site can't handle TRANSACTION_FINISHED_BUT_DESTINATION_FULL
scenario as expected.

That happens if the remote NiFi's input port destination relationship
becomes full during Site-to-Site client sends data. The data which has
already sent to the remote NiFi has to be committed successfully.
However, the remote NiFi returns 503 as a response of commit HTTP
request. Because it does check port availability.

The port availability check shouldn't be called at commit request, since
the session at source NiFi has already been committed. The remote NiFi
should commit its session as well, and return
TRANSACTION_FINISHED_BUT_DESTINATION_FULL response.

This fix makes a remote NiFi to keep the handshaken properties when it holds
transaction to be committed. Then if a transaction already has
handshaken properties, then use it, instead of doing a handshake process
again.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2259) HTTP Site-to-Site doesn't handle TRANSACTION_FINISHED _BUT_DESTINATION_FULL as expected

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15391373#comment-15391373
 ] 

ASF GitHub Bot commented on NIFI-2259:
--

Github user ijokarumawak closed the pull request at:

https://github.com/apache/nifi/pull/656


> HTTP Site-to-Site doesn't handle TRANSACTION_FINISHED _BUT_DESTINATION_FULL 
> as expected
> ---
>
> Key: NIFI-2259
> URL: https://issues.apache.org/jira/browse/NIFI-2259
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
> Fix For: 1.0.0
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> If a remote NiFi instance input-port's destination becomes full, It's 
> expected that remote NiFi instance returns 
> TRANSACTION_FINISHED_BUT_DESTINATION_FULL response.
> However, an IOException occurred due to DESTINATION_FULL handshake error when 
> org.apache.nifi.remote.Transaction.complete() is called.
> The cause of this issue is SiteToSiteResource.commitInputPortTransaction() 
> checks if all of input-port's destination is full. Which shouldn't be 
> executed at complete(). The transaction is already committed on source NiFi.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2259) HTTP Site-to-Site doesn't handle TRANSACTION_FINISHED _BUT_DESTINATION_FULL as expected

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15391374#comment-15391374
 ] 

ASF GitHub Bot commented on NIFI-2259:
--

Github user ijokarumawak commented on the issue:

https://github.com/apache/nifi/pull/656
  
Closing this PR to open a new one solely for NIFI-2259, as this one is 
getting stale and having conflicts, also, no updates about PipedInputStream 
related issue.


> HTTP Site-to-Site doesn't handle TRANSACTION_FINISHED _BUT_DESTINATION_FULL 
> as expected
> ---
>
> Key: NIFI-2259
> URL: https://issues.apache.org/jira/browse/NIFI-2259
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
> Fix For: 1.0.0
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> If a remote NiFi instance input-port's destination becomes full, It's 
> expected that remote NiFi instance returns 
> TRANSACTION_FINISHED_BUT_DESTINATION_FULL response.
> However, an IOException occurred due to DESTINATION_FULL handshake error when 
> org.apache.nifi.remote.Transaction.complete() is called.
> The cause of this issue is SiteToSiteResource.commitInputPortTransaction() 
> checks if all of input-port's destination is full. Which shouldn't be 
> executed at complete(). The transaction is already committed on source NiFi.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #656: Issue HTTP Site-to-Site with lots of queued flow-file

[ijokarumawak]

Github user ijokarumawak commented on the issue:

https://github.com/apache/nifi/pull/656
  
Closing this PR to open a new one solely for NIFI-2259, as this one is 
getting stale and having conflicts, also, no updates about PipedInputStream 
related issue.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #656: Issue HTTP Site-to-Site with lots of queued flow-fil...

[ijokarumawak]

Github user ijokarumawak closed the pull request at:

https://github.com/apache/nifi/pull/656


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-1148) Add processor to GetEmail supporting IMAP and POP3

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15391335#comment-15391335
 ] 

ASF GitHub Bot commented on NIFI-1148:
--

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/710
  
@olegz what about using the same strategy as PutEmail where the user can 
configure the socket on runtime?


https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutEmail.java#L122


> Add processor to GetEmail supporting IMAP and POP3
> --
>
> Key: NIFI-1148
> URL: https://issues.apache.org/jira/browse/NIFI-1148
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Joseph Witt
>Assignee: Oleg Zhurakousky
> Fix For: 1.0.0
>
>
> It is fairly common that users want to be able to acquire data via email.  
> This means both IMAP and POP3.  POP3 is easier as it is a sort of fire/forget 
> model whereas IMAP involves more state handling.  But in any event both modes 
> are important to support.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #710: NIFI-1148 added IMAP/POP3 support

[trixpan]

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/710
  
@olegz what about using the same strategy as PutEmail where the user can 
configure the socket on runtime?


https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutEmail.java#L122


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #710: NIFI-1148 added IMAP/POP3 support

[olegz]

Github user olegz commented on the issue:

https://github.com/apache/nifi/pull/710
  
One advise though is that there are additional JavaMail properties that one 
may need to provide to accomodate specific servers. For example in Gmail you 
need to specify 
```mail.pop3.socketFactory.class=javax.net.ssl.SSLSocketFactory``` and possibly 
few more (depending on your server settings). Those should be set as Dynamic 
properties since there is no common set. One property you should definitely set 
to get more info as to what is happening is ```mail.debug=true```. That will 
spit out all kinds of protocol messages which could explain what is going on.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #710: NIFI-1148 added IMAP/POP3 support

[olegz]

Github user olegz commented on the issue:

https://github.com/apache/nifi/pull/710
  
Digging into the source code for it looks like something is out of 
compliance with POP3 protocol
```java
if (line.startsWith("+OK"))
r.ok = true;
else if (line.startsWith("-ERR"))
r.ok = false;
else
throw new IOException("Unexpected response: " + line);
int i;
```
So, it appears that the received message is not really compliant with POP3, 
but will dig some more in the AM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-1148) Add processor to GetEmail supporting IMAP and POP3

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15391295#comment-15391295
 ] 

ASF GitHub Bot commented on NIFI-1148:
--

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/710
  
@olegz agreed. 


> Add processor to GetEmail supporting IMAP and POP3
> --
>
> Key: NIFI-1148
> URL: https://issues.apache.org/jira/browse/NIFI-1148
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Joseph Witt
>Assignee: Oleg Zhurakousky
> Fix For: 1.0.0
>
>
> It is fairly common that users want to be able to acquire data via email.  
> This means both IMAP and POP3.  POP3 is easier as it is a sort of fire/forget 
> model whereas IMAP involves more state handling.  But in any event both modes 
> are important to support.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #710: NIFI-1148 added IMAP/POP3 support

[trixpan]

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/710
  
@olegz agreed. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #710: NIFI-1148 added IMAP/POP3 support

[olegz]

Github user olegz commented on the issue:

https://github.com/apache/nifi/pull/710
  
@trixpan Well, I am not sure what that is (will be looking) and how 
reliable mailtrap.io is for such testing giving it's description "Fake smtp 
testing server. Dummy smtp email testing". I've tested it with few real servers 
(gmail, yahoo etc) and all is good. I am not dismissing that there may be a 
problem but let's make sure we are not chasing some dummy error in mailtrap.io. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-1148) Add processor to GetEmail supporting IMAP and POP3

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15391294#comment-15391294
 ] 

ASF GitHub Bot commented on NIFI-1148:
--

Github user olegz commented on the issue:

https://github.com/apache/nifi/pull/710
  
@trixpan Well, I am not sure what that is (will be looking) and how 
reliable mailtrap.io is for such testing giving it's description "Fake smtp 
testing server. Dummy smtp email testing". I've tested it with few real servers 
(gmail, yahoo etc) and all is good. I am not dismissing that there may be a 
problem but let's make sure we are not chasing some dummy error in mailtrap.io. 


> Add processor to GetEmail supporting IMAP and POP3
> --
>
> Key: NIFI-1148
> URL: https://issues.apache.org/jira/browse/NIFI-1148
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Joseph Witt
>Assignee: Oleg Zhurakousky
> Fix For: 1.0.0
>
>
> It is fairly common that users want to be able to acquire data via email.  
> This means both IMAP and POP3.  POP3 is easier as it is a sort of fire/forget 
> model whereas IMAP involves more state handling.  But in any event both modes 
> are important to support.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2028) Site-to-Site Transit URI is inconsistent

[Koji Kawamura (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Koji Kawamura updated NIFI-2028:

Description: 
Site-to-Site client and server create provenance event at both end, and those 
have Transit URI in it to record how flow files are transferred. However, the 
URI formats are inconsistent among RAW vs HTTP.

Test result as follows:

- These port numbers are configurable in nifi.properties
- 3080: Web API port (nifi.web.http.port)
- 3081: Site-to-Site RAW Socket port (nifi.remote.input.socket.port)

*Before Fix*

PUSH - RAW
|Client - SEND: | nifi://localhost:3081/flow-file-uuid |
|Server - RECEIVE: | nifi://localhost:3081/flow-file-uuid|

PULL - RAW
|Client - RECEIVE:| nifi://localhost:3081flow-file-uuid|
|Server - SEND:   | nifi://localhost:3081/flow-file-uuid|

PUSH - HTTP
|Client - SEND:   | http://127.0.0.1:3080/nifi-api/flow-file-uuid|
|Server - RECEIVE:| nifi://127.0.0.1:57390|

PULL - HTTP
|Client - RECEIVE:| http://127.0.0.1:3080/flow-file-uuid|
|Server - SEND:   | nifi://127.0.0.1:57673|

*Issues*

- PULL - RAW, Client - RECEIVE: lacks '/' in between port and flow-file uuid
- RAW uses server's host and port on both end (by transit url prefix), HTTP 
should follow this rule
- HTTP transit uri looks like REST endpoint but it is not a real endpoint. It 
should be an actual endpoint URI
- RAW uses hostname, while HTTP uses IP address

*After Fix*

PUSH - RAW
|Client - SEND: | nifi://localhost:3081/flow-file-uuid |
|Server - RECEIVE: | nifi://localhost:3081/flow-file-uuid|

PULL - RAW
|Client - RECEIVE: | nifi://localhost:3081/flow-file-uuid|
|Server - SEND:   | nifi://localhost:3081/flow-file-uuid|

PUSH - HTTP
|Client - SEND:   | 
http://localhost:3080/nifi-api/data-transfer/input-ports/$port-id/transactions/$tx-id/flow-files|
|Server - RECEIVE:| 
http://localhost:3080/nifi-api/data-transfer/input-ports/$port-id/transactions/$tx-id/flow-files|

PULL - HTTP
|Client - RECEIVE: | 
http://localhost:3080/nifi-api/data-transfer/output-ports/$port-id/transactions/$tx-id/flow-files|
|Server - SEND:   | 
http://localhost:3080/nifi-api/data-transfer/output-ports/$port-id/transactions/$tx-id/flow-files|


  was:
Site-to-Site client and server create provenance event at both end, and those 
have Transit URI in it to record how flow files are transferred. However, the 
URI formats are inconsistent among RAW vs HTTP.

Test result as follows:

- These port numbers are configurable in nifi.properties
- 3080: Web API port (nifi.web.http.port)
- 3081: Site-to-Site RAW Socket port (nifi.remote.input.socket.port)

*Before Fix*

PUSH - RAW
|Client - SEND: | nifi://localhost:3081/flow-file-uuid |
|Server - RECEIVE: | nifi://localhost:3081/flow-file-uuid|

PULL - RAW
|Client - RECEIVE:| nifi://localhost:3081flow-file-uuid|
|Server - SEND:   | nifi://localhost:3081/flow-file-uuid|

PUSH - HTTP
|Client - SEND:   | http://127.0.0.1:3080/nifi-api/flow-file-uuid|
|Server - RECEIVE:| nifi://127.0.0.1:57390|

PULL - HTTP
|Client - RECEIVE:| http://127.0.0.1:3080/flow-file-uuid|
|Server - SEND:   | nifi://127.0.0.1:57673|

*Issues*

- PULL - RAW, Client - RECEIVE: lacks '/' in between port and flow-file uuid
- RAW uses server's host and port on both end (by transit url prefix), HTTP 
should follow this rule
- HTTP transit uri looks like REST endpoint but it is not a real endpoint. It 
should be an actual endpoint URI
- RAW uses hostname, while HTTP uses IP address

*After Fix*

PUSH - RAW
|Client - SEND: | nifi://localhost:3081/flow-file-uuid |
|Server - RECEIVE: | nifi://localhost:3081/flow-file-uuid|

PULL - RAW
|Client - RECEIVE: | nifi://localhost:3081/flow-file-uuid|
|Server - SEND:   | nifi://localhost:3081/flow-file-uuid|

PUSH - HTTP
|Client - SEND:   | 
http://localhost:3080/nifi-api/site-to-site/input-ports/$port-id/transactions/$tx-id/flow-files|
|Server - RECEIVE:| 
http://localhost:3080/nifi-api/site-to-site/input-ports/$port-id/transactions/$tx-id/flow-files|

PULL - HTTP
|Client - RECEIVE: | 
http://localhost:3080/nifi-api/site-to-site/output-ports/$port-id/transactions/$tx-id/flow-files|
|Server - SEND:   | 
http://localhost:3080/nifi-api/site-to-site/output-ports/$port-id/transactions/$tx-id/flow-files|



> Site-to-Site Transit URI is inconsistent
> 
>
> Key: NIFI-2028
> URL: https://issues.apache.org/jira/browse/NIFI-2028
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core UI
>Affects Versions: 1.0.0
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
> Fix For: 1.0.0
>
> Attachments: send-to-the-same-box.png
>
>
> Site-to-Site client and server create provenance event at both end, and those 
> have Transit URI in it to record how flow files are transferred. However, the 
> URI formats are inconsistent among RAW vs HTTP.
> Test result as follows:
> - 

[jira] [Commented] (NIFI-1148) Add processor to GetEmail supporting IMAP and POP3

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15391270#comment-15391270
 ] 

ASF GitHub Bot commented on NIFI-1148:
--

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/710
  
Getting errors when trying to use mailtrap.io for POP3 testing:

```
2016-07-25 10:37:27,706 ERROR [Timer-Driven Process Thread-6] 
o.a.nifi.processors.email.ConsumePOP3
org.apache.nifi.processor.exception.ProcessException: Failed to receive 
messages from Email server: [javax.mail.Mes
sagingException - error getting size
at 
org.apache.nifi.processors.email.AbstractEmailProcessor.fillMessageQueueIfNecessary(AbstractEmailProcess
or.java:311) ~[na:na]
at 
org.apache.nifi.processors.email.AbstractEmailProcessor.receiveMessage(AbstractEmailProcessor.java:373)
~[na:na]
at 
org.apache.nifi.processors.email.AbstractEmailProcessor.onTrigger(AbstractEmailProcessor.java:182)
 ~[na:
na]
at 
org.apache.nifi.processors.email.ConsumePOP3.onTrigger(ConsumePOP3.java:31) 
~[na:na]
at 
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
 ~[nifi-api-1.0.0-SNAPSH
OT.jar:1.0.0-SNAPSHOT]
at 
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1077)
 [nifi-framew
ork-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at 
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
[nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at 
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
 [
nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at 
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:1
23) [nifi-framework-core-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
[na:1.8.0_91]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) 
[na:1.8.0_91]
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecu
tor.java:180) [na:1.8.0_91]
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.jav
a:294) [na:1.8.0_91]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[na:1.8.0_91]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
Caused by: javax.mail.MessagingException: error getting size
at com.sun.mail.pop3.POP3Message.getSize(POP3Message.java:145) 
~[mail-1.4.7.jar:1.4.7]
at javax.mail.internet.MimeMessage.(MimeMessage.java:238) 
~[mail-1.4.7.jar:1.4.7]
at 
org.springframework.integration.mail.AbstractMailReceiver$IntegrationMimeMessage.(AbstractMailRece
iver.java:557) ~[na:na]
at 
org.springframework.integration.mail.AbstractMailReceiver$IntegrationMimeMessage.(AbstractMailRece
iver.java:552) ~[na:na]
at 
org.springframework.integration.mail.AbstractMailReceiver.postProcessFilteredMessages(AbstractMailReceiv
er.java:415) ~[na:na]
at 
org.springframework.integration.mail.AbstractMailReceiver.receive(AbstractMailReceiver.java:342)
 ~[na:na
]
at 
org.apache.nifi.processors.email.AbstractEmailProcessor.fillMessageQueueIfNecessary(AbstractEmailProcess
or.java:306) ~[na:na]
... 15 common frames omitted
Caused by: java.io.IOException: Unexpected response: 1 95
at com.sun.mail.pop3.Protocol.readResponse(Protocol.java:752) 
~[mail-1.4.7.jar:1.4.7]
at com.sun.mail.pop3.Protocol.simpleCommand(Protocol.java:692) 
~[mail-1.4.7.jar:1.4.7]
at com.sun.mail.pop3.Protocol.list(Protocol.java:391) 
~[mail-1.4.7.jar:1.4.7]
at com.sun.mail.pop3.POP3Message.getSize(POP3Message.java:138) 
~[mail-1.4.7.jar:1.4.7]
... 21 common frames omitted

```



> Add processor to GetEmail supporting IMAP and POP3
> --
>
> Key: NIFI-1148
> URL: https://issues.apache.org/jira/browse/NIFI-1148
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Joseph Witt
>Assignee: Oleg Zhurakousky
> Fix For: 1.0.0
>
>
> It is fairly common that users want to be able to acquire data via email.  
> This means both IMAP and POP3.  POP3 is easier as it is a sort of fire/forget 
> model whereas IMAP involves more state handling.  But in 

[GitHub] nifi issue #675: LSB Adherence

[apiri]

Github user apiri commented on the issue:

https://github.com/apache/nifi/pull/675
  
Hey @PuspenduBanerjee, sorry for the delay.  Will scope this out in the 
next day or so.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2292) Nodes in cluster sometimes become out-of-sync with actual 'connection state' of node

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15391184#comment-15391184
 ] 

ASF GitHub Bot commented on NIFI-2292:
--

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/701
  
@markap14, this branch has conflicts with the current master, can you 
resolve them?


> Nodes in cluster sometimes become out-of-sync with actual 'connection state' 
> of node
> 
>
> Key: NIFI-2292
> URL: https://issues.apache.org/jira/browse/NIFI-2292
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.0.0
>
>
> Occasionally I'll see a node that has a different view of the cluster than 
> other nodes. Right now I'm actually seeing "node 1" think it's in 
> 'CONNECTING' state while nodes 2-5 think we have 5/5 nodes connected.
> This also can result in a node that is elected cluster coordinator and then 
> has that role revoked can continually monitor for heartbeats, even though it 
> won't receive them since it's not the coordinator anymore. This results in 
> continually logging a message like "Failed to retrieve any new heartbeat 
> information for nodes. Will not make any decisions based on heartbeats."



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #701: NIFI-2292: Funnel all cluster node status changes through t...

[JPercivall]

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/701
  
@markap14, this branch has conflicts with the current master, can you 
resolve them?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2376) Content could be prematurely archived/removed if a call to append() throws an Exception

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15391172#comment-15391172
 ] 

ASF subversion and git services commented on NIFI-2376:
---

Commit 4e08ea65254edd80d487ffc17b5046c37184467c in nifi's branch 
refs/heads/master from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=4e08ea6 ]

NIFI-2376 This closes #713. Ensure that we don't decrement claimant count more 
than once when append() throws an Exception


> Content could be prematurely archived/removed if a call to append() throws an 
> Exception
> ---
>
> Key: NIFI-2376
> URL: https://issues.apache.org/jira/browse/NIFI-2376
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Joseph Witt
>Priority: Blocker
>  Labels: releasenotes
> Fix For: 1.0.0, 0.8.0
>
>
> If ProcessSession.append() is called multiple times for a FlowFile and in one 
> of the calls other than the first call, an Exception is thrown, the claimant 
> count is decremented more than once. This can result in the content being 
> deleted or archived too soon. Consequently, this could also result in a 
> NullPointerException being thrown when attempting to write to the FlowFile.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #713: NIFI-2376: Ensure that we don't decrement claimant c...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/713


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2376) Content could be prematurely archived/removed if a call to append() throws an Exception

[Joseph Witt (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15391171#comment-15391171
 ] 

Joseph Witt commented on NIFI-2376:
---

+1.  Will merge to master.  I didn't really read the description of this but 
after reviewing the code/tests/issue came up with the same description!

"In the event of appending to a flow file and doing so more than once in the 
same session on calls after the first call to append if an exception is thrown 
we could incorrectly decrement the reference count to the original flow file 
resulting in potential data loss (when not archiving) and resulting in NPE 
which could further lead to self-supression errors in the JVM"

If this doesn't apply cleanly to 0.x will need another patch for that.

> Content could be prematurely archived/removed if a call to append() throws an 
> Exception
> ---
>
> Key: NIFI-2376
> URL: https://issues.apache.org/jira/browse/NIFI-2376
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Joseph Witt
>Priority: Blocker
>  Labels: releasenotes
> Fix For: 1.0.0, 0.8.0
>
>
> If ProcessSession.append() is called multiple times for a FlowFile and in one 
> of the calls other than the first call, an Exception is thrown, the claimant 
> count is decremented more than once. This can result in the content being 
> deleted or archived too soon. Consequently, this could also result in a 
> NullPointerException being thrown when attempting to write to the FlowFile.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2376) Content could be prematurely archived/removed if a call to append() throws an Exception

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt updated NIFI-2376:
--
Labels: releasenotes  (was: )

> Content could be prematurely archived/removed if a call to append() throws an 
> Exception
> ---
>
> Key: NIFI-2376
> URL: https://issues.apache.org/jira/browse/NIFI-2376
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Joseph Witt
>Priority: Blocker
>  Labels: releasenotes
> Fix For: 1.0.0, 0.8.0
>
>
> If ProcessSession.append() is called multiple times for a FlowFile and in one 
> of the calls other than the first call, an Exception is thrown, the claimant 
> count is decremented more than once. This can result in the content being 
> deleted or archived too soon. Consequently, this could also result in a 
> NullPointerException being thrown when attempting to write to the FlowFile.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2384) Back-pressure should use cycle detection to avoid dead lock conditions

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt updated NIFI-2384:
--
Description: 
If you have a cycle with back pressure being fed by a source with back pressure 
and the source of data is fast enough you can end up in a dead lock situation.

We should have cycle detection occur and then in the presence of a cycle which 
has backpressure on the source edge (back pressure on source edge and back 
pressure on all edges in the cycle) then we should ignore the backpressure on 
the destination edge and pull data from the back pressured cycle source edge.

Doing this generically on all components might be problematic as different 
processors can have different relationships and behaviors.  However, doing this 
on a funnel could make sense as it can have only one output relationship.  It 
might be the case that such back-pressure directed graph cycle detection and 
avoidance is only feasible when using a funnel with a single output (not 
cloned).  But then this is something which can be well documented and explained 
as the approach to avoid these cycles.

Attached is a template which can replicate this condition.  Look on the bottom 
left cycle.

  was:
If you have a cycle with back pressure being fed by a source with back pressure 
and the source of data is fast enough you can end up in a dead lock situation.

We should have cycle detection occur and then in the presence of a cycle which 
has backpressure on the source edge (back pressure on source edge and back 
pressure on all edges in the cycle) then we should ignore the backpressure on 
the destination edge and pull data from the back pressured cycle source edge.

Attached is a template which can replicate this condition.  Look on the bottom 
left cycle.


> Back-pressure should use cycle detection to avoid dead lock conditions
> --
>
> Key: NIFI-2384
> URL: https://issues.apache.org/jira/browse/NIFI-2384
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Joseph Witt
> Attachments: this_aint_right.xml
>
>
> If you have a cycle with back pressure being fed by a source with back 
> pressure and the source of data is fast enough you can end up in a dead lock 
> situation.
> We should have cycle detection occur and then in the presence of a cycle 
> which has backpressure on the source edge (back pressure on source edge and 
> back pressure on all edges in the cycle) then we should ignore the 
> backpressure on the destination edge and pull data from the back pressured 
> cycle source edge.
> Doing this generically on all components might be problematic as different 
> processors can have different relationships and behaviors.  However, doing 
> this on a funnel could make sense as it can have only one output 
> relationship.  It might be the case that such back-pressure directed graph 
> cycle detection and avoidance is only feasible when using a funnel with a 
> single output (not cloned).  But then this is something which can be well 
> documented and explained as the approach to avoid these cycles.
> Attached is a template which can replicate this condition.  Look on the 
> bottom left cycle.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2384) Back-pressure should use cycle detection to avoid dead lock conditions

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt updated NIFI-2384:
--
Attachment: this_aint_right.xml

> Back-pressure should use cycle detection to avoid dead lock conditions
> --
>
> Key: NIFI-2384
> URL: https://issues.apache.org/jira/browse/NIFI-2384
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Joseph Witt
> Attachments: this_aint_right.xml
>
>
> If you have a cycle with back pressure being fed by a source with back 
> pressure and the source of data is fast enough you can end up in a dead lock 
> situation.
> We should have cycle detection occur and then in the presence of a cycle 
> which has backpressure on the source edge (back pressure on source edge and 
> back pressure on all edges in the cycle) then we should ignore the 
> backpressure on the destination edge and pull data from the back pressured 
> cycle source edge.
> Attached is a template which can replicate this condition.  Look on the 
> bottom left cycle.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (NIFI-2384) Back-pressure should use cycle detection to avoid dead lock conditions

[Joseph Witt (JIRA)]

Joseph Witt created NIFI-2384:
-

 Summary: Back-pressure should use cycle detection to avoid dead 
lock conditions
 Key: NIFI-2384
 URL: https://issues.apache.org/jira/browse/NIFI-2384
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Core Framework
Affects Versions: 1.0.0
Reporter: Joseph Witt


If you have a cycle with back pressure being fed by a source with back pressure 
and the source of data is fast enough you can end up in a dead lock situation.

We should have cycle detection occur and then in the presence of a cycle which 
has backpressure on the source edge (back pressure on source edge and back 
pressure on all edges in the cycle) then we should ignore the backpressure on 
the destination edge and pull data from the back pressured cycle source edge.

Attached is a template which can replicate this condition.  Look on the bottom 
left cycle.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (NIFI-2376) Content could be prematurely archived/removed if a call to append() throws an Exception

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt reassigned NIFI-2376:
-

Assignee: Joseph Witt  (was: Mark Payne)

> Content could be prematurely archived/removed if a call to append() throws an 
> Exception
> ---
>
> Key: NIFI-2376
> URL: https://issues.apache.org/jira/browse/NIFI-2376
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Joseph Witt
>Priority: Blocker
> Fix For: 1.0.0, 0.8.0
>
>
> If ProcessSession.append() is called multiple times for a FlowFile and in one 
> of the calls other than the first call, an Exception is thrown, the claimant 
> count is decremented more than once. This can result in the content being 
> deleted or archived too soon. Consequently, this could also result in a 
> NullPointerException being thrown when attempting to write to the FlowFile.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2379) Add Multiple Path to "site to site" Connection

[Joseph Witt (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15391120#comment-15391120
 ] 

Joseph Witt commented on NIFI-2379:
---

Right now as it is site-to-site offers load balancing and fail-over between any 
two sites (source site and destination site).

This JIRA is to extend its function to support a given source site with one or 
more destination sites for fail-over and possibly load balancing.

The complicated part of this concept is 'when to fail-over' to the other 
cluster.  One might want this to be 'if failure occurs to all nodes in a given 
site THEN fail-over to the other site".  Another person might want 'as soon as 
failures occur to any nodes within the cluster begin some level of alternate 
routing.  A related consideration is when to start going back to the preferred 
site.  The preferred site concept means we have to possibly enable 
prioritization of sites.  This same sort of thought process probably also makes 
sense for load-balancing as well.


> Add Multiple Path to "site to site" Connection
> --
>
> Key: NIFI-2379
> URL: https://issues.apache.org/jira/browse/NIFI-2379
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: n h
>
> Add support for multiple path (IPs) to site-to-site, so that in failure cases 
> it could elect to send the data down an alternative path (IP).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2379) Enable 'site to site' to support multiple destination sites for failover and possibly load-balancing

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt updated NIFI-2379:
--
Summary: Enable 'site to site' to support multiple destination sites for 
failover and possibly load-balancing  (was: Add Multiple Path to "site to site" 
Connection)

> Enable 'site to site' to support multiple destination sites for failover and 
> possibly load-balancing
> 
>
> Key: NIFI-2379
> URL: https://issues.apache.org/jira/browse/NIFI-2379
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: n h
>
> Add support for multiple path (IPs) to site-to-site, so that in failure cases 
> it could elect to send the data down an alternative path (IP).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2163) Nifi Service does not follow LSB Sevice Spec

[Puspendu Banerjee (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15391118#comment-15391118
 ] 

Puspendu Banerjee commented on NIFI-2163:
-

[~evega] Any chance to review it again?

> Nifi Service does not follow LSB Sevice Spec
> 
>
> Key: NIFI-2163
> URL: https://issues.apache.org/jira/browse/NIFI-2163
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Configuration
>Affects Versions: 1.0.0, 0.7.0
> Environment: Centos
>Reporter: Edgardo Vega
>Assignee: Puspendu Banerjee
>Priority: Critical
>  Labels: platform-consistency
>
> Trying to use the lastest off master with nifi.sh and nifi-env.sh and they do 
> not follow the spec for services, whcih causes some configuration tools not 
> to work as they use the return codes to determine if things are running, 
> dead, or stopped.
> http://refspecs.linuxbase.org/LSB_3.1.0/LSB-Core-generic/LSB-Core-generic/iniscrptact.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (NIFI-2383) ListFile

[n h (JIRA)]

n h created NIFI-2383:
-

 Summary: ListFile
 Key: NIFI-2383
 URL: https://issues.apache.org/jira/browse/NIFI-2383
 Project: Apache NiFi
  Issue Type: Bug
  Components: Core Framework
Reporter: n h
Priority: Minor


ListFile does not detect directory updates correctly. It seems it only check 
for file name and modified date. 
Using a 2 processors flow: ListFile -> FetchFile, and configure FetchFile to 
finally delete files. Following scenario fails:
1. Create file1.txt somewhere outside of watching directory.
2. Copy file1.txt to watching directory.
3. Wait for FetchFile to fetch and delete file1.txt
4. Copy file1.txt again to watching directory.

In the 2nd time ListFile does not detect "new copy of the same file file1.txt" 
because file name and modified time are the same. Only creation time is updated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (NIFI-2382) ListFile Documentation

[n h (JIRA)]

n h created NIFI-2382:
-

 Summary: ListFile Documentation
 Key: NIFI-2382
 URL: https://issues.apache.org/jira/browse/NIFI-2382
 Project: Apache NiFi
  Issue Type: Bug
  Components: Documentation & Website
Reporter: n h
Priority: Minor


"For each file that is listed,creates a FlowFile that represents the file so 
that it can be fetched in conjunction with ListFile."
I guess it might be:
"For each file that is listed,creates a FlowFile that represents the file so 
that it can be fetched in conjunction with FetchFile."




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2380) ExtractEmailAttachments processor should support TNEF files (aka winmail.dat)

[Andre (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andre updated NIFI-2380:

Description: 
during the review of NIFI-1899 Dan Marshall highlighted some use cases for 
email processing that have not been addressed as part of the initial 
development cycle.

One of these use cases was the decoding of Microsoft Transport Neutral Encoding 
Files (TNEF). 

This type of attachments is popularly know as winmail.dat and uses a non RFC 
compliant structure to transfer attachments across different Microsoft Outlook 
clients.

Given the prevalence of outlook and the issues with winmail.dat files, it would 
be nice to be able to decode TNEF as we currently do with MIME attachments.

Permalink to Dan's comments 
http://mail-archives.apache.org/mod_mbox/nifi-dev/201607.mbox/%3C1468716836729-12827.post%40n7.nabble.com%3E

  was:
during the review of NIFI-1899 Dan Marshall highlighted some use cases for 
email processing that have not been addressed as part of the initial 
development cycle.

One of these use cases was the decoding of Microsoft Transport Neutral Encoding 
Files (TNEF). 

This type of attachments is popularly know as winmail.dat and uses a non RFC 
compliant structure to transfer attachments across different Microsoft Outlook 
clients.

Given the prevalence of outlook and the issues with winmail.dat files, it would 
be nice to be able to decode TNEF as we currently do with MIME attachments.


> ExtractEmailAttachments processor should support TNEF files (aka winmail.dat)
> -
>
> Key: NIFI-2380
> URL: https://issues.apache.org/jira/browse/NIFI-2380
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.0.0
>Reporter: Andre
>
> during the review of NIFI-1899 Dan Marshall highlighted some use cases for 
> email processing that have not been addressed as part of the initial 
> development cycle.
> One of these use cases was the decoding of Microsoft Transport Neutral 
> Encoding Files (TNEF). 
> This type of attachments is popularly know as winmail.dat and uses a non RFC 
> compliant structure to transfer attachments across different Microsoft 
> Outlook clients.
> Given the prevalence of outlook and the issues with winmail.dat files, it 
> would be nice to be able to decode TNEF as we currently do with MIME 
> attachments.
> Permalink to Dan's comments 
> http://mail-archives.apache.org/mod_mbox/nifi-dev/201607.mbox/%3C1468716836729-12827.post%40n7.nabble.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (NIFI-2379) Add Multiple Path to "site to site" Connection

[n h (JIRA)]

n h created NIFI-2379:
-

 Summary: Add Multiple Path to "site to site" Connection
 Key: NIFI-2379
 URL: https://issues.apache.org/jira/browse/NIFI-2379
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Core Framework
Reporter: n h


Add support for multiple path (IPs) to site-to-site, so that in failure cases 
it could elect to send the data down an alternative path (IP).




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)