[jira] [Commented] (NIFI-3678) NiFi may fail to startup if shutdown while FlowFile Repository partition is full

2017-04-07 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/NIFI-3678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960875#comment-15960875
 ] 

ASF subversion and git services commented on NIFI-3678:
---

Commit 292dd1d66b1726794f0d34523578727ea3a7fe08 in nifi's branch 
refs/heads/master from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=292dd1d ]

NIFI-3678: Ensure that we catch EOFException when reading header information 
from WAL Partition files; previously, we caught EOFExceptions when reading a 
'record' from the WAL but not when reading header info

NIFI-3678: If we have a transaction ID but then have no more data written to 
Partition file, we end up with a NPE. Added logic to avoid this and instead 
return null for the next record when this happens

This closes #1656.

Signed-off-by: Bryan Bende 


> NiFi may fail to startup if shutdown while FlowFile Repository partition is 
> full
> 
>
> Key: NIFI-3678
> URL: https://issues.apache.org/jira/browse/NIFI-3678
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.2.0
>
>
> A user from the users list posted that they are seeing the following stack 
> trace when trying to startup NiFi:
> {code}
> 2017-02-03 16:04:48,588 ERROR [main] o.a.nifi.controller.StandardFlowService 
> Failed to load flow from cluster due to: 
> org.apache.nifi.cluster.ConnectionException: Failed to connect node to 
> cluster due to: java.io.IOExcept 
> ion: Unable to read Record Schema from stream 
> org.apache.nifi.cluster.ConnectionException: Failed to connect node to 
> cluster due to: java.io.IOException: Unable to read Record Schema from stream 
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:901)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:770) 
> [nifi-jetty-1.1.0.jar:1.1.0] 
> at org.apache.nifi.NiFi.(NiFi.java:156) 
> [nifi-runtime-1.1.0.jar:1.1.0] 
> at org.apache.nifi.NiFi.main(NiFi.java:262) 
> [nifi-runtime-1.1.0.jar:1.1.0] 
> Caused by: java.io.IOException: Unable to read Record Schema from stream 
> at 
> org.apache.nifi.repository.schema.RecordSchema.readFrom(RecordSchema.java:117)
>  ~[nifi-schema-utils-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.repository.SchemaRepositoryRecordSerde.readHeader(SchemaRepositoryRecordSerde.java:100)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at 
> org.wali.MinimalLockingWriteAheadLog$Partition.getRecoveryStream(MinimalLockingWriteAheadLog.java:1005)
>  ~[nifi-write-ahead-log-1.1.0.jar:1.1.0] 
> at 
> org.wali.MinimalLockingWriteAheadLog$Partition.getNextRecoverableTransactionId(MinimalLockingWriteAheadLog.java:1016)
>  ~[nifi-write-ahead-log-1.1.0.jar:1.1.0] 
> at 
> org.wali.MinimalLockingWriteAheadLog.recoverFromEdits(MinimalLockingWriteAheadLog.java:430)
>  ~[nifi-write-ahead-log-1.1.0.jar:1.1.0] 
> at 
> org.wali.MinimalLockingWriteAheadLog.recoverRecords(MinimalLockingWriteAheadLog.java:301)
>  ~[nifi-write-ahead-log-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.loadFlowFiles(WriteAheadFlowFileRepository.java:346)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.FlowController.initializeFlow(FlowController.java:700)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:701)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:872)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> ... 4 common frames omitted 
> Caused by: java.io.EOFException: null 
> at java.io.DataInputStream.readFully(DataInputStream.java:197) 
> ~[na:1.8.0_111] 
> at java.io.DataInputStream.readUTF(DataInputStream.java:609) 
> ~[na:1.8.0_111] 
> at java.io.DataInputStream.readUTF(DataInputStream.java:564) 
> ~[na:1.8.0_111] 
> at 
> org.apache.nifi.repository.schema.RecordSchema.readField(RecordSchema.java:126)
>  ~[nifi-schema-utils-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.repository.schema.RecordSchema.readField(RecordSchema.java:144)
>  ~[nifi-schema-utils-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.repository.schema.RecordSchema.readField(RecordSchema.java:144)
>  ~[nifi-schema-utils-1.1.0.jar:1.1.0] 
> at 
>

[jira] [Commented] (NIFI-3678) NiFi may fail to startup if shutdown while FlowFile Repository partition is full

2017-04-07 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/NIFI-3678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960874#comment-15960874
 ] 

ASF subversion and git services commented on NIFI-3678:
---

Commit 292dd1d66b1726794f0d34523578727ea3a7fe08 in nifi's branch 
refs/heads/master from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=292dd1d ]

NIFI-3678: Ensure that we catch EOFException when reading header information 
from WAL Partition files; previously, we caught EOFExceptions when reading a 
'record' from the WAL but not when reading header info

NIFI-3678: If we have a transaction ID but then have no more data written to 
Partition file, we end up with a NPE. Added logic to avoid this and instead 
return null for the next record when this happens

This closes #1656.

Signed-off-by: Bryan Bende 


> NiFi may fail to startup if shutdown while FlowFile Repository partition is 
> full
> 
>
> Key: NIFI-3678
> URL: https://issues.apache.org/jira/browse/NIFI-3678
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.2.0
>
>
> A user from the users list posted that they are seeing the following stack 
> trace when trying to startup NiFi:
> {code}
> 2017-02-03 16:04:48,588 ERROR [main] o.a.nifi.controller.StandardFlowService 
> Failed to load flow from cluster due to: 
> org.apache.nifi.cluster.ConnectionException: Failed to connect node to 
> cluster due to: java.io.IOExcept 
> ion: Unable to read Record Schema from stream 
> org.apache.nifi.cluster.ConnectionException: Failed to connect node to 
> cluster due to: java.io.IOException: Unable to read Record Schema from stream 
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:901)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:770) 
> [nifi-jetty-1.1.0.jar:1.1.0] 
> at org.apache.nifi.NiFi.(NiFi.java:156) 
> [nifi-runtime-1.1.0.jar:1.1.0] 
> at org.apache.nifi.NiFi.main(NiFi.java:262) 
> [nifi-runtime-1.1.0.jar:1.1.0] 
> Caused by: java.io.IOException: Unable to read Record Schema from stream 
> at 
> org.apache.nifi.repository.schema.RecordSchema.readFrom(RecordSchema.java:117)
>  ~[nifi-schema-utils-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.repository.SchemaRepositoryRecordSerde.readHeader(SchemaRepositoryRecordSerde.java:100)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at 
> org.wali.MinimalLockingWriteAheadLog$Partition.getRecoveryStream(MinimalLockingWriteAheadLog.java:1005)
>  ~[nifi-write-ahead-log-1.1.0.jar:1.1.0] 
> at 
> org.wali.MinimalLockingWriteAheadLog$Partition.getNextRecoverableTransactionId(MinimalLockingWriteAheadLog.java:1016)
>  ~[nifi-write-ahead-log-1.1.0.jar:1.1.0] 
> at 
> org.wali.MinimalLockingWriteAheadLog.recoverFromEdits(MinimalLockingWriteAheadLog.java:430)
>  ~[nifi-write-ahead-log-1.1.0.jar:1.1.0] 
> at 
> org.wali.MinimalLockingWriteAheadLog.recoverRecords(MinimalLockingWriteAheadLog.java:301)
>  ~[nifi-write-ahead-log-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.loadFlowFiles(WriteAheadFlowFileRepository.java:346)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.FlowController.initializeFlow(FlowController.java:700)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:701)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:872)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> ... 4 common frames omitted 
> Caused by: java.io.EOFException: null 
> at java.io.DataInputStream.readFully(DataInputStream.java:197) 
> ~[na:1.8.0_111] 
> at java.io.DataInputStream.readUTF(DataInputStream.java:609) 
> ~[na:1.8.0_111] 
> at java.io.DataInputStream.readUTF(DataInputStream.java:564) 
> ~[na:1.8.0_111] 
> at 
> org.apache.nifi.repository.schema.RecordSchema.readField(RecordSchema.java:126)
>  ~[nifi-schema-utils-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.repository.schema.RecordSchema.readField(RecordSchema.java:144)
>  ~[nifi-schema-utils-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.repository.schema.RecordSchema.readField(RecordSchema.java:144)
>  ~[nifi-schema-utils-1.1.0.jar:1.1.0] 
> at 
>

[jira] [Commented] (NIFI-3678) NiFi may fail to startup if shutdown while FlowFile Repository partition is full

2017-04-07 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/NIFI-3678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960876#comment-15960876
 ] 

ASF GitHub Bot commented on NIFI-3678:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1656


> NiFi may fail to startup if shutdown while FlowFile Repository partition is 
> full
> 
>
> Key: NIFI-3678
> URL: https://issues.apache.org/jira/browse/NIFI-3678
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.2.0
>
>
> A user from the users list posted that they are seeing the following stack 
> trace when trying to startup NiFi:
> {code}
> 2017-02-03 16:04:48,588 ERROR [main] o.a.nifi.controller.StandardFlowService 
> Failed to load flow from cluster due to: 
> org.apache.nifi.cluster.ConnectionException: Failed to connect node to 
> cluster due to: java.io.IOExcept 
> ion: Unable to read Record Schema from stream 
> org.apache.nifi.cluster.ConnectionException: Failed to connect node to 
> cluster due to: java.io.IOException: Unable to read Record Schema from stream 
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:901)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:770) 
> [nifi-jetty-1.1.0.jar:1.1.0] 
> at org.apache.nifi.NiFi.(NiFi.java:156) 
> [nifi-runtime-1.1.0.jar:1.1.0] 
> at org.apache.nifi.NiFi.main(NiFi.java:262) 
> [nifi-runtime-1.1.0.jar:1.1.0] 
> Caused by: java.io.IOException: Unable to read Record Schema from stream 
> at 
> org.apache.nifi.repository.schema.RecordSchema.readFrom(RecordSchema.java:117)
>  ~[nifi-schema-utils-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.repository.SchemaRepositoryRecordSerde.readHeader(SchemaRepositoryRecordSerde.java:100)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at 
> org.wali.MinimalLockingWriteAheadLog$Partition.getRecoveryStream(MinimalLockingWriteAheadLog.java:1005)
>  ~[nifi-write-ahead-log-1.1.0.jar:1.1.0] 
> at 
> org.wali.MinimalLockingWriteAheadLog$Partition.getNextRecoverableTransactionId(MinimalLockingWriteAheadLog.java:1016)
>  ~[nifi-write-ahead-log-1.1.0.jar:1.1.0] 
> at 
> org.wali.MinimalLockingWriteAheadLog.recoverFromEdits(MinimalLockingWriteAheadLog.java:430)
>  ~[nifi-write-ahead-log-1.1.0.jar:1.1.0] 
> at 
> org.wali.MinimalLockingWriteAheadLog.recoverRecords(MinimalLockingWriteAheadLog.java:301)
>  ~[nifi-write-ahead-log-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.loadFlowFiles(WriteAheadFlowFileRepository.java:346)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.FlowController.initializeFlow(FlowController.java:700)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:701)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:872)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> ... 4 common frames omitted 
> Caused by: java.io.EOFException: null 
> at java.io.DataInputStream.readFully(DataInputStream.java:197) 
> ~[na:1.8.0_111] 
> at java.io.DataInputStream.readUTF(DataInputStream.java:609) 
> ~[na:1.8.0_111] 
> at java.io.DataInputStream.readUTF(DataInputStream.java:564) 
> ~[na:1.8.0_111] 
> at 
> org.apache.nifi.repository.schema.RecordSchema.readField(RecordSchema.java:126)
>  ~[nifi-schema-utils-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.repository.schema.RecordSchema.readField(RecordSchema.java:144)
>  ~[nifi-schema-utils-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.repository.schema.RecordSchema.readField(RecordSchema.java:144)
>  ~[nifi-schema-utils-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.repository.schema.RecordSchema.readFrom(RecordSchema.java:111)
>  ~[nifi-schema-utils-1.1.0.jar:1.1.0] 
> ... 13 common frames omitted 
> {code}
> It appears that we do not properly handle Exceptions that are thrown when 
> reading the header information from a Partition file.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3678) NiFi may fail to startup if shutdown while FlowFile Repository partition is full

2017-04-07 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/NIFI-3678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960872#comment-15960872
 ] 

ASF GitHub Bot commented on NIFI-3678:
--

Github user bbende commented on the issue:

https://github.com/apache/nifi/pull/1656
  
+1 looks good, will merge to master


> NiFi may fail to startup if shutdown while FlowFile Repository partition is 
> full
> 
>
> Key: NIFI-3678
> URL: https://issues.apache.org/jira/browse/NIFI-3678
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.2.0
>
>
> A user from the users list posted that they are seeing the following stack 
> trace when trying to startup NiFi:
> {code}
> 2017-02-03 16:04:48,588 ERROR [main] o.a.nifi.controller.StandardFlowService 
> Failed to load flow from cluster due to: 
> org.apache.nifi.cluster.ConnectionException: Failed to connect node to 
> cluster due to: java.io.IOExcept 
> ion: Unable to read Record Schema from stream 
> org.apache.nifi.cluster.ConnectionException: Failed to connect node to 
> cluster due to: java.io.IOException: Unable to read Record Schema from stream 
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:901)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:770) 
> [nifi-jetty-1.1.0.jar:1.1.0] 
> at org.apache.nifi.NiFi.(NiFi.java:156) 
> [nifi-runtime-1.1.0.jar:1.1.0] 
> at org.apache.nifi.NiFi.main(NiFi.java:262) 
> [nifi-runtime-1.1.0.jar:1.1.0] 
> Caused by: java.io.IOException: Unable to read Record Schema from stream 
> at 
> org.apache.nifi.repository.schema.RecordSchema.readFrom(RecordSchema.java:117)
>  ~[nifi-schema-utils-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.repository.SchemaRepositoryRecordSerde.readHeader(SchemaRepositoryRecordSerde.java:100)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at 
> org.wali.MinimalLockingWriteAheadLog$Partition.getRecoveryStream(MinimalLockingWriteAheadLog.java:1005)
>  ~[nifi-write-ahead-log-1.1.0.jar:1.1.0] 
> at 
> org.wali.MinimalLockingWriteAheadLog$Partition.getNextRecoverableTransactionId(MinimalLockingWriteAheadLog.java:1016)
>  ~[nifi-write-ahead-log-1.1.0.jar:1.1.0] 
> at 
> org.wali.MinimalLockingWriteAheadLog.recoverFromEdits(MinimalLockingWriteAheadLog.java:430)
>  ~[nifi-write-ahead-log-1.1.0.jar:1.1.0] 
> at 
> org.wali.MinimalLockingWriteAheadLog.recoverRecords(MinimalLockingWriteAheadLog.java:301)
>  ~[nifi-write-ahead-log-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.loadFlowFiles(WriteAheadFlowFileRepository.java:346)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.FlowController.initializeFlow(FlowController.java:700)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:701)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:872)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> ... 4 common frames omitted 
> Caused by: java.io.EOFException: null 
> at java.io.DataInputStream.readFully(DataInputStream.java:197) 
> ~[na:1.8.0_111] 
> at java.io.DataInputStream.readUTF(DataInputStream.java:609) 
> ~[na:1.8.0_111] 
> at java.io.DataInputStream.readUTF(DataInputStream.java:564) 
> ~[na:1.8.0_111] 
> at 
> org.apache.nifi.repository.schema.RecordSchema.readField(RecordSchema.java:126)
>  ~[nifi-schema-utils-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.repository.schema.RecordSchema.readField(RecordSchema.java:144)
>  ~[nifi-schema-utils-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.repository.schema.RecordSchema.readField(RecordSchema.java:144)
>  ~[nifi-schema-utils-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.repository.schema.RecordSchema.readFrom(RecordSchema.java:111)
>  ~[nifi-schema-utils-1.1.0.jar:1.1.0] 
> ... 13 common frames omitted 
> {code}
> It appears that we do not properly handle Exceptions that are thrown when 
> reading the header information from a Partition file.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3678) NiFi may fail to startup if shutdown while FlowFile Repository partition is full

2017-04-06 Thread Mark Payne (JIRA) 

[ 
https://issues.apache.org/jira/browse/NIFI-3678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15959769#comment-15959769
 ] 

Mark Payne commented on NIFI-3678:
--

OK [~bende] I submitted PR 1656 if you get a chance to review. It addresses the 
issue that Joe pointed out above as well. Thanks!

> NiFi may fail to startup if shutdown while FlowFile Repository partition is 
> full
> 
>
> Key: NIFI-3678
> URL: https://issues.apache.org/jira/browse/NIFI-3678
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.2.0
>
>
> A user from the users list posted that they are seeing the following stack 
> trace when trying to startup NiFi:
> {code}
> 2017-02-03 16:04:48,588 ERROR [main] o.a.nifi.controller.StandardFlowService 
> Failed to load flow from cluster due to: 
> org.apache.nifi.cluster.ConnectionException: Failed to connect node to 
> cluster due to: java.io.IOExcept 
> ion: Unable to read Record Schema from stream 
> org.apache.nifi.cluster.ConnectionException: Failed to connect node to 
> cluster due to: java.io.IOException: Unable to read Record Schema from stream 
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:901)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:770) 
> [nifi-jetty-1.1.0.jar:1.1.0] 
> at org.apache.nifi.NiFi.(NiFi.java:156) 
> [nifi-runtime-1.1.0.jar:1.1.0] 
> at org.apache.nifi.NiFi.main(NiFi.java:262) 
> [nifi-runtime-1.1.0.jar:1.1.0] 
> Caused by: java.io.IOException: Unable to read Record Schema from stream 
> at 
> org.apache.nifi.repository.schema.RecordSchema.readFrom(RecordSchema.java:117)
>  ~[nifi-schema-utils-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.repository.SchemaRepositoryRecordSerde.readHeader(SchemaRepositoryRecordSerde.java:100)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at 
> org.wali.MinimalLockingWriteAheadLog$Partition.getRecoveryStream(MinimalLockingWriteAheadLog.java:1005)
>  ~[nifi-write-ahead-log-1.1.0.jar:1.1.0] 
> at 
> org.wali.MinimalLockingWriteAheadLog$Partition.getNextRecoverableTransactionId(MinimalLockingWriteAheadLog.java:1016)
>  ~[nifi-write-ahead-log-1.1.0.jar:1.1.0] 
> at 
> org.wali.MinimalLockingWriteAheadLog.recoverFromEdits(MinimalLockingWriteAheadLog.java:430)
>  ~[nifi-write-ahead-log-1.1.0.jar:1.1.0] 
> at 
> org.wali.MinimalLockingWriteAheadLog.recoverRecords(MinimalLockingWriteAheadLog.java:301)
>  ~[nifi-write-ahead-log-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.loadFlowFiles(WriteAheadFlowFileRepository.java:346)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.FlowController.initializeFlow(FlowController.java:700)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:701)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:872)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> ... 4 common frames omitted 
> Caused by: java.io.EOFException: null 
> at java.io.DataInputStream.readFully(DataInputStream.java:197) 
> ~[na:1.8.0_111] 
> at java.io.DataInputStream.readUTF(DataInputStream.java:609) 
> ~[na:1.8.0_111] 
> at java.io.DataInputStream.readUTF(DataInputStream.java:564) 
> ~[na:1.8.0_111] 
> at 
> org.apache.nifi.repository.schema.RecordSchema.readField(RecordSchema.java:126)
>  ~[nifi-schema-utils-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.repository.schema.RecordSchema.readField(RecordSchema.java:144)
>  ~[nifi-schema-utils-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.repository.schema.RecordSchema.readField(RecordSchema.java:144)
>  ~[nifi-schema-utils-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.repository.schema.RecordSchema.readFrom(RecordSchema.java:111)
>  ~[nifi-schema-utils-1.1.0.jar:1.1.0] 
> ... 13 common frames omitted 
> {code}
> It appears that we do not properly handle Exceptions that are thrown when 
> reading the header information from a Partition file.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3678) NiFi may fail to startup if shutdown while FlowFile Repository partition is full

2017-04-06 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/NIFI-3678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15959765#comment-15959765
 ] 

ASF GitHub Bot commented on NIFI-3678:
--

GitHub user markap14 opened a pull request:

https://github.com/apache/nifi/pull/1656

NIFI-3678

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markap14/nifi NIFI-3678

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1656.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1656


commit 856767897fd91a6b08ccf75965873afce7912b88
Author: Mark Payne 
Date:   2017-04-06T19:57:11Z

NIFI-3678: Ensure that we catch EOFException when reading header 
information from WAL Partition files; previously, we caught EOFExceptions when 
reading a 'record' from the WAL but not when reading header info

commit 21db816431608be8ba859b52c2183443212d22e9
Author: Mark Payne 
Date:   2017-04-06T21:13:12Z

NIFI-3678: If we have a transaction ID but then have no more data written 
to Partition file, we end up with a NPE. Added logic to avoid this and instead 
return null for the next record when this happens




> NiFi may fail to startup if shutdown while FlowFile Repository partition is 
> full
> 
>
> Key: NIFI-3678
> URL: https://issues.apache.org/jira/browse/NIFI-3678
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.2.0
>
>
> A user from the users list posted that they are seeing the following stack 
> trace when trying to startup NiFi:
> {code}
> 2017-02-03 16:04:48,588 ERROR [main] o.a.nifi.controller.StandardFlowService 
> Failed to load flow from cluster due to: 
> org.apache.nifi.cluster.ConnectionException: Failed to connect node to 
> cluster due to: java.io.IOExcept 
> ion: Unable to read Record Schema from stream 
> org.apache.nifi.cluster.ConnectionException: Failed to connect node to 
> cluster due to: java.io.IOException: Unable to read Record Schema from stream 
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:901)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:770) 
> [nifi-jetty-1.1.0.jar:1.1.0] 
> at org.apache.nifi.NiFi.(NiFi.java:156) 
> [nifi-runtime-1.1.0.jar:1.1.0] 
> at org.apache.nifi.NiFi.main(NiFi.java:262) 
> [nifi-runtime-1.1.0.jar:1.1.0] 
> Caused by: java.io.IOException: Unable to read Record Schema from stream 
> at 
>

[jira] [Commented] (NIFI-3678) NiFi may fail to startup if shutdown while FlowFile Repository partition is full

2017-04-06 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/NIFI-3678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15959672#comment-15959672
 ] 

ASF GitHub Bot commented on NIFI-3678:
--

Github user markap14 closed the pull request at:

https://github.com/apache/nifi/pull/1654


> NiFi may fail to startup if shutdown while FlowFile Repository partition is 
> full
> 
>
> Key: NIFI-3678
> URL: https://issues.apache.org/jira/browse/NIFI-3678
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.2.0
>
>
> A user from the users list posted that they are seeing the following stack 
> trace when trying to startup NiFi:
> {code}
> 2017-02-03 16:04:48,588 ERROR [main] o.a.nifi.controller.StandardFlowService 
> Failed to load flow from cluster due to: 
> org.apache.nifi.cluster.ConnectionException: Failed to connect node to 
> cluster due to: java.io.IOExcept 
> ion: Unable to read Record Schema from stream 
> org.apache.nifi.cluster.ConnectionException: Failed to connect node to 
> cluster due to: java.io.IOException: Unable to read Record Schema from stream 
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:901)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:770) 
> [nifi-jetty-1.1.0.jar:1.1.0] 
> at org.apache.nifi.NiFi.(NiFi.java:156) 
> [nifi-runtime-1.1.0.jar:1.1.0] 
> at org.apache.nifi.NiFi.main(NiFi.java:262) 
> [nifi-runtime-1.1.0.jar:1.1.0] 
> Caused by: java.io.IOException: Unable to read Record Schema from stream 
> at 
> org.apache.nifi.repository.schema.RecordSchema.readFrom(RecordSchema.java:117)
>  ~[nifi-schema-utils-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.repository.SchemaRepositoryRecordSerde.readHeader(SchemaRepositoryRecordSerde.java:100)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at 
> org.wali.MinimalLockingWriteAheadLog$Partition.getRecoveryStream(MinimalLockingWriteAheadLog.java:1005)
>  ~[nifi-write-ahead-log-1.1.0.jar:1.1.0] 
> at 
> org.wali.MinimalLockingWriteAheadLog$Partition.getNextRecoverableTransactionId(MinimalLockingWriteAheadLog.java:1016)
>  ~[nifi-write-ahead-log-1.1.0.jar:1.1.0] 
> at 
> org.wali.MinimalLockingWriteAheadLog.recoverFromEdits(MinimalLockingWriteAheadLog.java:430)
>  ~[nifi-write-ahead-log-1.1.0.jar:1.1.0] 
> at 
> org.wali.MinimalLockingWriteAheadLog.recoverRecords(MinimalLockingWriteAheadLog.java:301)
>  ~[nifi-write-ahead-log-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.loadFlowFiles(WriteAheadFlowFileRepository.java:346)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.FlowController.initializeFlow(FlowController.java:700)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:701)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:872)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> ... 4 common frames omitted 
> Caused by: java.io.EOFException: null 
> at java.io.DataInputStream.readFully(DataInputStream.java:197) 
> ~[na:1.8.0_111] 
> at java.io.DataInputStream.readUTF(DataInputStream.java:609) 
> ~[na:1.8.0_111] 
> at java.io.DataInputStream.readUTF(DataInputStream.java:564) 
> ~[na:1.8.0_111] 
> at 
> org.apache.nifi.repository.schema.RecordSchema.readField(RecordSchema.java:126)
>  ~[nifi-schema-utils-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.repository.schema.RecordSchema.readField(RecordSchema.java:144)
>  ~[nifi-schema-utils-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.repository.schema.RecordSchema.readField(RecordSchema.java:144)
>  ~[nifi-schema-utils-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.repository.schema.RecordSchema.readFrom(RecordSchema.java:111)
>  ~[nifi-schema-utils-1.1.0.jar:1.1.0] 
> ... 13 common frames omitted 
> {code}
> It appears that we do not properly handle Exceptions that are thrown when 
> reading the header information from a Partition file.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3678) NiFi may fail to startup if shutdown while FlowFile Repository partition is full

2017-04-06 Thread Mark Payne (JIRA) 

[ 
https://issues.apache.org/jira/browse/NIFI-3678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15959671#comment-15959671
 ] 

Mark Payne commented on NIFI-3678:
--

[~joewitt] good call, I can wrap that into this JIRA/PR as well. [~bbende] 
please hold off on reviewing until i wrap that in.

> NiFi may fail to startup if shutdown while FlowFile Repository partition is 
> full
> 
>
> Key: NIFI-3678
> URL: https://issues.apache.org/jira/browse/NIFI-3678
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.2.0
>
>
> A user from the users list posted that they are seeing the following stack 
> trace when trying to startup NiFi:
> {code}
> 2017-02-03 16:04:48,588 ERROR [main] o.a.nifi.controller.StandardFlowService 
> Failed to load flow from cluster due to: 
> org.apache.nifi.cluster.ConnectionException: Failed to connect node to 
> cluster due to: java.io.IOExcept 
> ion: Unable to read Record Schema from stream 
> org.apache.nifi.cluster.ConnectionException: Failed to connect node to 
> cluster due to: java.io.IOException: Unable to read Record Schema from stream 
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:901)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:770) 
> [nifi-jetty-1.1.0.jar:1.1.0] 
> at org.apache.nifi.NiFi.(NiFi.java:156) 
> [nifi-runtime-1.1.0.jar:1.1.0] 
> at org.apache.nifi.NiFi.main(NiFi.java:262) 
> [nifi-runtime-1.1.0.jar:1.1.0] 
> Caused by: java.io.IOException: Unable to read Record Schema from stream 
> at 
> org.apache.nifi.repository.schema.RecordSchema.readFrom(RecordSchema.java:117)
>  ~[nifi-schema-utils-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.repository.SchemaRepositoryRecordSerde.readHeader(SchemaRepositoryRecordSerde.java:100)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at 
> org.wali.MinimalLockingWriteAheadLog$Partition.getRecoveryStream(MinimalLockingWriteAheadLog.java:1005)
>  ~[nifi-write-ahead-log-1.1.0.jar:1.1.0] 
> at 
> org.wali.MinimalLockingWriteAheadLog$Partition.getNextRecoverableTransactionId(MinimalLockingWriteAheadLog.java:1016)
>  ~[nifi-write-ahead-log-1.1.0.jar:1.1.0] 
> at 
> org.wali.MinimalLockingWriteAheadLog.recoverFromEdits(MinimalLockingWriteAheadLog.java:430)
>  ~[nifi-write-ahead-log-1.1.0.jar:1.1.0] 
> at 
> org.wali.MinimalLockingWriteAheadLog.recoverRecords(MinimalLockingWriteAheadLog.java:301)
>  ~[nifi-write-ahead-log-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.loadFlowFiles(WriteAheadFlowFileRepository.java:346)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.FlowController.initializeFlow(FlowController.java:700)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:701)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:872)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> ... 4 common frames omitted 
> Caused by: java.io.EOFException: null 
> at java.io.DataInputStream.readFully(DataInputStream.java:197) 
> ~[na:1.8.0_111] 
> at java.io.DataInputStream.readUTF(DataInputStream.java:609) 
> ~[na:1.8.0_111] 
> at java.io.DataInputStream.readUTF(DataInputStream.java:564) 
> ~[na:1.8.0_111] 
> at 
> org.apache.nifi.repository.schema.RecordSchema.readField(RecordSchema.java:126)
>  ~[nifi-schema-utils-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.repository.schema.RecordSchema.readField(RecordSchema.java:144)
>  ~[nifi-schema-utils-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.repository.schema.RecordSchema.readField(RecordSchema.java:144)
>  ~[nifi-schema-utils-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.repository.schema.RecordSchema.readFrom(RecordSchema.java:111)
>  ~[nifi-schema-utils-1.1.0.jar:1.1.0] 
> ... 13 common frames omitted 
> {code}
> It appears that we do not properly handle Exceptions that are thrown when 
> reading the header information from a Partition file.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3678) NiFi may fail to startup if shutdown while FlowFile Repository partition is full

2017-04-06 Thread Joseph Witt (JIRA) 

[ 
https://issues.apache.org/jira/browse/NIFI-3678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15959659#comment-15959659
 ] 

Joseph Witt commented on NIFI-3678:
---

[~markap14]  From Mike on the mailing list this weekend

2017-04-04 21:29:46,490 WARN [main] org.apache.nifi.web.server.JettyServer 
Failed to start web server... shutting down.
java.lang.NullPointerException: null
at 
org.apache.nifi.controller.repository.SchemaRepositoryRecordSerde.deserializeRecord(SchemaRepositoryRecordSerde.java:115)
 ~[nifi-framework-core-1.1.1.jar:1.1.1]
at 
org.apache.nifi.controller.repository.SchemaRepositoryRecordSerde.deserializeEdit(SchemaRepositoryRecordSerde.java:105)
 ~[nifi-framework-core-1.1.1.jar:1.1.1]
at 
org.apache.nifi.controller.repository.SchemaRepositoryRecordSerde.deserializeEdit(SchemaRepositoryRecordSerde.java:46)
 ~[nifi-framework-core-1.1.1.jar:1.1.1]
at 
org.wali.MinimalLockingWriteAheadLog$Partition.recoverNextTransaction(MinimalLockingWriteAheadLog.java:1072)
 ~[nifi-write-ahead-log-1.1.1.jar:1.1.1]
at 
org.wali.MinimalLockingWriteAheadLog.recoverFromEdits(MinimalLockingWriteAheadLog.java:459)
 ~[nifi-write-ahead-log-1.1.1.jar:1.1.1]
at 
org.wali.MinimalLockingWriteAheadLog.recoverRecords(MinimalLockingWriteAheadLog.java:301)
 ~[nifi-write-ahead-log-1.1.1.jar:1.1.1]
at 
org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.loadFlowFiles(WriteAheadFlowFileRepository.java:346)
 ~[nifi-framework-core-1.1.1.jar:1.1.1]
at 
org.apache.nifi.controller.FlowController.initializeFlow(FlowController.java:700)
 ~[nifi-framework-core-1.1.1.jar:1.1.1]
at 
org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:701)
 ~[nifi-framework-core-1.1.1.jar:1.1.1]
at 
org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:508)
 ~[nifi-framework-core-1.1.1.jar:1.1.1]
at 
org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:69)
 ~[na:na]
at 
org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:837)
 ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
at 
org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:533)
 ~[jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517]
at 
org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:810)
 ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
at 
org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:345)
 ~[jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517]
at 
org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1404) 
~[jetty-webapp-9.3.9.v20160517.jar:9.3.9.v20160517]
at 
org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1366) 
~[jetty-webapp-9.3.9.v20160517.jar:9.3.9.v20160517]
at 
org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772)
 ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
at 
org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:262)
 ~[jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517]
at 
org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:520) 
~[jetty-webapp-9.3.9.v20160517.jar:9.3.9.v20160517]
at 
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
 ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
at 
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
 ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
at 
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
 ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
at 
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
 ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
at 
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
 ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
at 
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
 ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
at 
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
 ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
at 
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
 ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:231) 
~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
at 
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
 ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]

[jira] [Commented] (NIFI-3678) NiFi may fail to startup if shutdown while FlowFile Repository partition is full

2017-04-06 Thread Joseph Witt (JIRA) 

[ 
https://issues.apache.org/jira/browse/NIFI-3678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15959657#comment-15959657
 ] 

Joseph Witt commented on NIFI-3678:
---

sweet!  Will review.

> NiFi may fail to startup if shutdown while FlowFile Repository partition is 
> full
> 
>
> Key: NIFI-3678
> URL: https://issues.apache.org/jira/browse/NIFI-3678
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.2.0
>
>
> A user from the users list posted that they are seeing the following stack 
> trace when trying to startup NiFi:
> {code}
> 2017-02-03 16:04:48,588 ERROR [main] o.a.nifi.controller.StandardFlowService 
> Failed to load flow from cluster due to: 
> org.apache.nifi.cluster.ConnectionException: Failed to connect node to 
> cluster due to: java.io.IOExcept 
> ion: Unable to read Record Schema from stream 
> org.apache.nifi.cluster.ConnectionException: Failed to connect node to 
> cluster due to: java.io.IOException: Unable to read Record Schema from stream 
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:901)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:770) 
> [nifi-jetty-1.1.0.jar:1.1.0] 
> at org.apache.nifi.NiFi.(NiFi.java:156) 
> [nifi-runtime-1.1.0.jar:1.1.0] 
> at org.apache.nifi.NiFi.main(NiFi.java:262) 
> [nifi-runtime-1.1.0.jar:1.1.0] 
> Caused by: java.io.IOException: Unable to read Record Schema from stream 
> at 
> org.apache.nifi.repository.schema.RecordSchema.readFrom(RecordSchema.java:117)
>  ~[nifi-schema-utils-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.repository.SchemaRepositoryRecordSerde.readHeader(SchemaRepositoryRecordSerde.java:100)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at 
> org.wali.MinimalLockingWriteAheadLog$Partition.getRecoveryStream(MinimalLockingWriteAheadLog.java:1005)
>  ~[nifi-write-ahead-log-1.1.0.jar:1.1.0] 
> at 
> org.wali.MinimalLockingWriteAheadLog$Partition.getNextRecoverableTransactionId(MinimalLockingWriteAheadLog.java:1016)
>  ~[nifi-write-ahead-log-1.1.0.jar:1.1.0] 
> at 
> org.wali.MinimalLockingWriteAheadLog.recoverFromEdits(MinimalLockingWriteAheadLog.java:430)
>  ~[nifi-write-ahead-log-1.1.0.jar:1.1.0] 
> at 
> org.wali.MinimalLockingWriteAheadLog.recoverRecords(MinimalLockingWriteAheadLog.java:301)
>  ~[nifi-write-ahead-log-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.loadFlowFiles(WriteAheadFlowFileRepository.java:346)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.FlowController.initializeFlow(FlowController.java:700)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:701)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:872)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> ... 4 common frames omitted 
> Caused by: java.io.EOFException: null 
> at java.io.DataInputStream.readFully(DataInputStream.java:197) 
> ~[na:1.8.0_111] 
> at java.io.DataInputStream.readUTF(DataInputStream.java:609) 
> ~[na:1.8.0_111] 
> at java.io.DataInputStream.readUTF(DataInputStream.java:564) 
> ~[na:1.8.0_111] 
> at 
> org.apache.nifi.repository.schema.RecordSchema.readField(RecordSchema.java:126)
>  ~[nifi-schema-utils-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.repository.schema.RecordSchema.readField(RecordSchema.java:144)
>  ~[nifi-schema-utils-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.repository.schema.RecordSchema.readField(RecordSchema.java:144)
>  ~[nifi-schema-utils-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.repository.schema.RecordSchema.readFrom(RecordSchema.java:111)
>  ~[nifi-schema-utils-1.1.0.jar:1.1.0] 
> ... 13 common frames omitted 
> {code}
> It appears that we do not properly handle Exceptions that are thrown when 
> reading the header information from a Partition file.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3678) NiFi may fail to startup if shutdown while FlowFile Repository partition is full

2017-04-06 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/NIFI-3678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15959658#comment-15959658
 ] 

ASF GitHub Bot commented on NIFI-3678:
--

Github user bbende commented on the issue:

https://github.com/apache/nifi/pull/1654
  
Will review...


> NiFi may fail to startup if shutdown while FlowFile Repository partition is 
> full
> 
>
> Key: NIFI-3678
> URL: https://issues.apache.org/jira/browse/NIFI-3678
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.2.0
>
>
> A user from the users list posted that they are seeing the following stack 
> trace when trying to startup NiFi:
> {code}
> 2017-02-03 16:04:48,588 ERROR [main] o.a.nifi.controller.StandardFlowService 
> Failed to load flow from cluster due to: 
> org.apache.nifi.cluster.ConnectionException: Failed to connect node to 
> cluster due to: java.io.IOExcept 
> ion: Unable to read Record Schema from stream 
> org.apache.nifi.cluster.ConnectionException: Failed to connect node to 
> cluster due to: java.io.IOException: Unable to read Record Schema from stream 
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:901)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:770) 
> [nifi-jetty-1.1.0.jar:1.1.0] 
> at org.apache.nifi.NiFi.(NiFi.java:156) 
> [nifi-runtime-1.1.0.jar:1.1.0] 
> at org.apache.nifi.NiFi.main(NiFi.java:262) 
> [nifi-runtime-1.1.0.jar:1.1.0] 
> Caused by: java.io.IOException: Unable to read Record Schema from stream 
> at 
> org.apache.nifi.repository.schema.RecordSchema.readFrom(RecordSchema.java:117)
>  ~[nifi-schema-utils-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.repository.SchemaRepositoryRecordSerde.readHeader(SchemaRepositoryRecordSerde.java:100)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at 
> org.wali.MinimalLockingWriteAheadLog$Partition.getRecoveryStream(MinimalLockingWriteAheadLog.java:1005)
>  ~[nifi-write-ahead-log-1.1.0.jar:1.1.0] 
> at 
> org.wali.MinimalLockingWriteAheadLog$Partition.getNextRecoverableTransactionId(MinimalLockingWriteAheadLog.java:1016)
>  ~[nifi-write-ahead-log-1.1.0.jar:1.1.0] 
> at 
> org.wali.MinimalLockingWriteAheadLog.recoverFromEdits(MinimalLockingWriteAheadLog.java:430)
>  ~[nifi-write-ahead-log-1.1.0.jar:1.1.0] 
> at 
> org.wali.MinimalLockingWriteAheadLog.recoverRecords(MinimalLockingWriteAheadLog.java:301)
>  ~[nifi-write-ahead-log-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.loadFlowFiles(WriteAheadFlowFileRepository.java:346)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.FlowController.initializeFlow(FlowController.java:700)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:701)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:872)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> ... 4 common frames omitted 
> Caused by: java.io.EOFException: null 
> at java.io.DataInputStream.readFully(DataInputStream.java:197) 
> ~[na:1.8.0_111] 
> at java.io.DataInputStream.readUTF(DataInputStream.java:609) 
> ~[na:1.8.0_111] 
> at java.io.DataInputStream.readUTF(DataInputStream.java:564) 
> ~[na:1.8.0_111] 
> at 
> org.apache.nifi.repository.schema.RecordSchema.readField(RecordSchema.java:126)
>  ~[nifi-schema-utils-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.repository.schema.RecordSchema.readField(RecordSchema.java:144)
>  ~[nifi-schema-utils-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.repository.schema.RecordSchema.readField(RecordSchema.java:144)
>  ~[nifi-schema-utils-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.repository.schema.RecordSchema.readFrom(RecordSchema.java:111)
>  ~[nifi-schema-utils-1.1.0.jar:1.1.0] 
> ... 13 common frames omitted 
> {code}
> It appears that we do not properly handle Exceptions that are thrown when 
> reading the header information from a Partition file.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (NIFI-3678) NiFi may fail to startup if shutdown while FlowFile Repository partition is full

2017-04-06 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/NIFI-3678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15959630#comment-15959630
 ] 

ASF GitHub Bot commented on NIFI-3678:
--

GitHub user markap14 opened a pull request:

https://github.com/apache/nifi/pull/1654

NIFI-3678: Ensure that we catch EOFException when reading header info…

…rmation from WAL Partition files; previously, we caught EOFExceptions when 
reading a 'record' from the WAL but not when reading header info

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markap14/nifi NIFI-3678

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1654.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1654


commit 5684d8f3ba5dd23c1166add7ca0a390e27425c39
Author: Mark Payne 
Date:   2017-04-06T19:57:11Z

NIFI-3678: Ensure that we catch EOFException when reading header 
information from WAL Partition files; previously, we caught EOFExceptions when 
reading a 'record' from the WAL but not when reading header info




> NiFi may fail to startup if shutdown while FlowFile Repository partition is 
> full
> 
>
> Key: NIFI-3678
> URL: https://issues.apache.org/jira/browse/NIFI-3678
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.2.0
>
>
> A user from the users list posted that they are seeing the following stack 
> trace when trying to startup NiFi:
> {code}
> 2017-02-03 16:04:48,588 ERROR [main] o.a.nifi.controller.StandardFlowService 
> Failed to load flow from cluster due to: 
> org.apache.nifi.cluster.ConnectionException: Failed to connect node to 
> cluster due to: java.io.IOExcept 
> ion: Unable to read Record Schema from stream 
> org.apache.nifi.cluster.ConnectionException: Failed to connect node to 
> cluster due to: java.io.IOException: Unable to read Record Schema from stream 
> at 
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:901)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at 
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:493)
>  ~[nifi-framework-core-1.1.0.jar:1.1.0] 
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:770) 
> [nifi-jetty-1.1.0.jar:1.1.0] 
> at org.apache.nifi.NiFi.(NiFi.java:156) 
> [nifi-runtime-1.1.0.jar:1.1.0] 
> at org.apache.nifi.NiFi.main(NiFi.java:262) 
> [nifi-runtime-1.1.0.jar:1.1.0] 
> Caused by: java.io.IOException: Unable to read Record Schema from stream 
> at 
> org.apache.nifi.repository.schema.RecordSchema.readFrom(RecordSchema.java:117)
>  ~[nifi-schema-utils-1.1.0.jar:1.1.0] 
> at 
>