[jira] [Commented] (NIFI-1180) Modify PutS3Object to enable encryption
[ https://issues.apache.org/jira/browse/NIFI-1180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15173278#comment-15173278 ] ASF GitHub Bot commented on NIFI-1180: -- Github user adamonduty commented on the pull request: https://github.com/apache/nifi/pull/246#issuecomment-190560346 @alopresto Done! Let me know if anything else needs addressed. > Modify PutS3Object to enable encryption > --- > > Key: NIFI-1180 > URL: https://issues.apache.org/jira/browse/NIFI-1180 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Elli Schwarz >Priority: Minor > > Amazon allows encryption to be enforced on objects that are put to S3. In the > PutS3Object class, the objectMetadata object contains a method > setSSEAlgorithm which can set encryption for the objects. A Nifi property > should be added to the PutS3Object class so a user can enable/disable > encryption from the processor itself. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1577) NiFi holds open too many files when using a Run Duration > 0 ms and calling session.append
[ https://issues.apache.org/jira/browse/NIFI-1577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15173034#comment-15173034 ] Joseph Witt commented on NIFI-1577: --- full clean build w/contrib check passes. Ran on-line tests using similar model to what you outlined with some tweaks. Observed completely stable open file count. +1 > NiFi holds open too many files when using a Run Duration > 0 ms and calling > session.append > -- > > Key: NIFI-1577 > URL: https://issues.apache.org/jira/browse/NIFI-1577 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 0.6.0 > > Attachments: > 0001-NIFI-1577-Close-any-streams-that-are-left-open-for-a.patch > > > If a Processor calls ProcessSession.append() and has a Run Duration scheduled > > 0 ms, we quickly end up with "Too many open files" exceptions. > This appears to be due to the fact that calling append() holds the content > repository's stream open so that the session can keep appending to it, but on > checkpoint() the session does not close these streams. It should close these > streams on checkpoint, since the Processor is no longer allowed to reference > these FlowFiles anyway at that point. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1180) Modify PutS3Object to enable encryption
[
https://issues.apache.org/jira/browse/NIFI-1180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15172985#comment-15172985
]
ASF GitHub Bot commented on NIFI-1180:
--
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/246#discussion_r54505449
--- Diff:
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/PutS3Object.java
---
@@ -407,6 +420,12 @@ public void process(final InputStream rawIn) throws
IOException {
}
}
+final String Sse =
context.getProperty(SERVER_SIDE_ENCRYPTION).getValue();
--- End diff --
Minor note -- naming convention would be `sse`, or preferably,
`serverSideEncryption`.
> Modify PutS3Object to enable encryption
> ---
>
> Key: NIFI-1180
> URL: https://issues.apache.org/jira/browse/NIFI-1180
> Project: Apache NiFi
> Issue Type: Improvement
>Reporter: Elli Schwarz
>Priority: Minor
>
> Amazon allows encryption to be enforced on objects that are put to S3. In the
> PutS3Object class, the objectMetadata object contains a method
> setSSEAlgorithm which can set encryption for the objects. A Nifi property
> should be added to the PutS3Object class so a user can enable/disable
> encryption from the processor itself.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-1180) Modify PutS3Object to enable encryption
[ https://issues.apache.org/jira/browse/NIFI-1180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15172967#comment-15172967 ] ASF GitHub Bot commented on NIFI-1180: -- Github user alopresto commented on the pull request: https://github.com/apache/nifi/pull/246#issuecomment-190467028 @adamonduty that looks good. I understand the confusion around this undocumented convention. My only outstanding request would be to combine the two commits into a single commit (via `rebase --interactive` and `push --force`) to condense the PR and then one of the committers should be able to merge it. Feel free to include my linked simple validation tests as well. > Modify PutS3Object to enable encryption > --- > > Key: NIFI-1180 > URL: https://issues.apache.org/jira/browse/NIFI-1180 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Elli Schwarz >Priority: Minor > > Amazon allows encryption to be enforced on objects that are put to S3. In the > PutS3Object class, the objectMetadata object contains a method > setSSEAlgorithm which can set encryption for the objects. A Nifi property > should be added to the PutS3Object class so a user can enable/disable > encryption from the processor itself. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1180) Modify PutS3Object to enable encryption
[ https://issues.apache.org/jira/browse/NIFI-1180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15172962#comment-15172962 ] ASF GitHub Bot commented on NIFI-1180: -- Github user adamonduty commented on the pull request: https://github.com/apache/nifi/pull/246#issuecomment-190465236 For what its worth, I already made the change to `server-side-encryption` in 33805e2e7d849ec3d1cd48b4e0d63c5c7bfc0792. Sorry if that wasn't clear. > Modify PutS3Object to enable encryption > --- > > Key: NIFI-1180 > URL: https://issues.apache.org/jira/browse/NIFI-1180 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Elli Schwarz >Priority: Minor > > Amazon allows encryption to be enforced on objects that are put to S3. In the > PutS3Object class, the objectMetadata object contains a method > setSSEAlgorithm which can set encryption for the objects. A Nifi property > should be added to the PutS3Object class so a user can enable/disable > encryption from the processor itself. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (NIFI-1558) Kafka processor clients write potentially sensitive info to the logs
[
https://issues.apache.org/jira/browse/NIFI-1558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleg Zhurakousky reassigned NIFI-1558:
--
Assignee: Oleg Zhurakousky
> Kafka processor clients write potentially sensitive info to the logs
>
>
> Key: NIFI-1558
> URL: https://issues.apache.org/jira/browse/NIFI-1558
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 0.5.0
>Reporter: Joseph Witt
>Assignee: Oleg Zhurakousky
> Fix For: 0.6.0
>
>
> I noticed the logs on startup have things like the following. This needs to
> be suppressed as it is of relatively low value but relatively high risk given
> that it appears it would write out ssl key passphrases and such.
> {quote}
> 2016-02-23 21:13:56,626 INFO [pool-29-thread-7]
> o.a.k.clients.producer.ProducerConfig ProducerConfig values:
> compression.type = none
> metric.reporters = []
> metadata.max.age.ms = 30
> metadata.fetch.timeout.ms = 3
> reconnect.backoff.ms = 50
> sasl.kerberos.ticket.renew.window.factor = 0.8
> bootstrap.servers = [172.31.8.34:9093]
> retry.backoff.ms = 100
> sasl.kerberos.kinit.cmd = /usr/bin/kinit
> buffer.memory = 1048576
> timeout.ms = 3
> key.serializer = class
> org.apache.kafka.common.serialization.ByteArraySerializer
> sasl.kerberos.service.name = null
> sasl.kerberos.ticket.renew.jitter = 0.05
> ssl.keystore.type = JKS
> ssl.trustmanager.algorithm = PKIX
> block.on.buffer.full = false
> ssl.key.password = null
> max.block.ms = 6
> sasl.kerberos.min.time.before.relogin = 6
> connections.max.idle.ms = 54
> ssl.truststore.password = null
> max.in.flight.requests.per.connection = 5
> metrics.num.samples = 2
> client.id = NiFi-2243c3f9-bd2b-4bfe-b515-09791ec25c4c
> ssl.endpoint.identification.algorithm = null
> ssl.protocol = TLS
> request.timeout.ms = 3
> ssl.provider = null
> ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
> acks = 0
> batch.size = 200
> ssl.keystore.location = null
> receive.buffer.bytes = 32768
> ssl.cipher.suites = null
> ssl.truststore.type = JKS
> security.protocol = PLAINTEXT
> retries = 0
> max.request.size = 1048576
> value.serializer = class
> org.apache.kafka.common.serialization.ByteArraySerializer
> ssl.truststore.location = null
> ssl.keystore.password = null
> ssl.keymanager.algorithm = SunX509
> metrics.sample.window.ms = 3
> partitioner.class = class
> org.apache.kafka.clients.producer.internals.DefaultPartitioner
> send.buffer.bytes = 131072
> linger.ms = 5000
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-1577) NiFi holds open too many files when using a Run Duration > 0 ms and calling session.append
[ https://issues.apache.org/jira/browse/NIFI-1577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15172613#comment-15172613 ] Joseph Witt commented on NIFI-1577: --- reviewing > NiFi holds open too many files when using a Run Duration > 0 ms and calling > session.append > -- > > Key: NIFI-1577 > URL: https://issues.apache.org/jira/browse/NIFI-1577 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 0.6.0 > > Attachments: > 0001-NIFI-1577-Close-any-streams-that-are-left-open-for-a.patch > > > If a Processor calls ProcessSession.append() and has a Run Duration scheduled > > 0 ms, we quickly end up with "Too many open files" exceptions. > This appears to be due to the fact that calling append() holds the content > repository's stream open so that the session can keep appending to it, but on > checkpoint() the session does not close these streams. It should close these > streams on checkpoint, since the Processor is no longer allowed to reference > these FlowFiles anyway at that point. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-1577) NiFi holds open too many files when using a Run Duration > 0 ms and calling session.append
[ https://issues.apache.org/jira/browse/NIFI-1577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-1577: - Fix Version/s: 0.6.0 > NiFi holds open too many files when using a Run Duration > 0 ms and calling > session.append > -- > > Key: NIFI-1577 > URL: https://issues.apache.org/jira/browse/NIFI-1577 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne > Fix For: 0.6.0 > > Attachments: > 0001-NIFI-1577-Close-any-streams-that-are-left-open-for-a.patch > > > If a Processor calls ProcessSession.append() and has a Run Duration scheduled > > 0 ms, we quickly end up with "Too many open files" exceptions. > This appears to be due to the fact that calling append() holds the content > repository's stream open so that the session can keep appending to it, but on > checkpoint() the session does not close these streams. It should close these > streams on checkpoint, since the Processor is no longer allowed to reference > these FlowFiles anyway at that point. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1577) NiFi holds open too many files when using a Run Duration > 0 ms and calling session.append
[ https://issues.apache.org/jira/browse/NIFI-1577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15172612#comment-15172612 ] Mark Payne commented on NIFI-1577: -- The easiest way that I've found to test this is to run a Processor like ListenSyslog that supports batching and calls session.append(). If the Run Duration is set to 25 ms and a fairly large amount of data is pushed to it, the logs will start being filled with errors about Too Many Open Files. Once this patch is applied, that goes away. Unfortunately, the patch does not lend itself well to unit tests, as it would require inspecting a lot of internal private state about the StandardProcessSession, which would result in very brittle unit tests. However, since checkpoint() clears the 'records' map, those streams that would be accessible will no longer be accessible anyway because the Mapping is from ContentClaim (which belongs to exactly 1 RepositoryRecord in the 'records' Map) to an OutputStream. Since the 'records' map is cleared, we cannot access the OutputStream, so they were being held open without any benefit. > NiFi holds open too many files when using a Run Duration > 0 ms and calling > session.append > -- > > Key: NIFI-1577 > URL: https://issues.apache.org/jira/browse/NIFI-1577 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne > Attachments: > 0001-NIFI-1577-Close-any-streams-that-are-left-open-for-a.patch > > > If a Processor calls ProcessSession.append() and has a Run Duration scheduled > > 0 ms, we quickly end up with "Too many open files" exceptions. > This appears to be due to the fact that calling append() holds the content > repository's stream open so that the session can keep appending to it, but on > checkpoint() the session does not close these streams. It should close these > streams on checkpoint, since the Processor is no longer allowed to reference > these FlowFiles anyway at that point. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-1577) NiFi holds open too many files when using a Run Duration > 0 ms and calling session.append
[ https://issues.apache.org/jira/browse/NIFI-1577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-1577: - Attachment: 0001-NIFI-1577-Close-any-streams-that-are-left-open-for-a.patch > NiFi holds open too many files when using a Run Duration > 0 ms and calling > session.append > -- > > Key: NIFI-1577 > URL: https://issues.apache.org/jira/browse/NIFI-1577 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne > Attachments: > 0001-NIFI-1577-Close-any-streams-that-are-left-open-for-a.patch > > > If a Processor calls ProcessSession.append() and has a Run Duration scheduled > > 0 ms, we quickly end up with "Too many open files" exceptions. > This appears to be due to the fact that calling append() holds the content > repository's stream open so that the session can keep appending to it, but on > checkpoint() the session does not close these streams. It should close these > streams on checkpoint, since the Processor is no longer allowed to reference > these FlowFiles anyway at that point. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1574) Write-Ahead log can become corrupt if NiFi is shutdown while disk is full
[
https://issues.apache.org/jira/browse/NIFI-1574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15172565#comment-15172565
]
ASF subversion and git services commented on NIFI-1574:
---
Commit 62333c9e0ae77854761cf019a65f54ccedc7af6b in nifi's branch
refs/heads/master from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=62333c9 ]
NIFI-1574: Ensure that we never flush a BufferedOutputStream's buffer on close
of the write-ahead log
> Write-Ahead log can become corrupt if NiFi is shutdown while disk is full
> -
>
> Key: NIFI-1574
> URL: https://issues.apache.org/jira/browse/NIFI-1574
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 0.6.0
>
> Attachments:
> 0001-NIFI-1574-Ensure-that-we-never-flush-a-BufferedOutpu.patch
>
>
> The FlowFile Repository, as well as the Local State Provider can exhibit this
> problem. We saw the following stack trace when attempting to restart a node
> while its disk was completely full:
> {code}
> 2016-02-27 06:43:39,249 WARN [main] org.eclipse.jetty.webapp.WebAppContext
> Failed startup of context
> o.e.j.w.WebAppContext@8b727cb{/nifi-api,file:/opt/nifi/nifi-0.5.1/work/jetty/nifi-web-api-0.5.1.war/webapp/,STARTING}{./work/nar/framework/nifi-framework-nar-0.5.1.nar-unpacked/META-INF/bundled-dependencies/nifi-web-api-0.5.1.war}
> org.apache.nifi.web.NiFiCoreException: Unable to start Flow Controller.
> at
> org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:99)
> ~[na:na]
> at
> org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:800)
> ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
> at
> org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:444)
> ~[jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
> at
> org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:791)
> ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
> at
> org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:294)
> ~[jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
> at
> org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1349)
> ~[jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529]
> at
> org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342)
> ~[jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529]
> at
> org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:741)
> ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
> at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:505)
> ~[jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529]
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
> [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
> [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
> at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
> [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
> [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
> at org.eclipse.jetty.server.Server.start(Server.java:387)
> [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
> [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
> at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
> [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
> at org.eclipse.jetty.server.Server.doStart(Server.java:354)
> [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:663)
> [nifi-jetty-0.5.1.jar:0.5.1]
> at org.apache.nifi.NiFi.(NiFi.java:137) [nifi-runtime-0.5.1.jar:0.5.1]
> at org.apache.nifi.NiFi.main(NiFi.java:227) [nifi-runtime-0.5.1.jar:0.5.1]
> Caused by: org.springframework.beans.factory.BeanCreationException: Error
> creating bean wit
[1/2] nifi git commit: NIFI-1572: Ensure that if an Exception is thrown when processing a bin, all sessions involved are rolled back or otherwise accounted for
Repository: nifi
Updated Branches:
refs/heads/master 99c7fe3b4 -> 62333c9e0
NIFI-1572: Ensure that if an Exception is thrown when processing a bin, all
sessions involved are rolled back or otherwise accounted for
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/1149bc61
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/1149bc61
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/1149bc61
Branch: refs/heads/master
Commit: 1149bc61cb3390210be4566dafba80564c4ab8d9
Parents: 99c7fe3
Author: Mark Payne
Authored: Mon Feb 29 10:43:27 2016 -0500
Committer: Mark Payne
Committed: Mon Feb 29 15:42:33 2016 -0500
--
.../java/org/apache/nifi/processor/util/bin/BinFiles.java| 8
1 file changed, 8 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/1149bc61/nifi-commons/nifi-processor-utilities/src/main/java/org/apache/nifi/processor/util/bin/BinFiles.java
--
diff --git
a/nifi-commons/nifi-processor-utilities/src/main/java/org/apache/nifi/processor/util/bin/BinFiles.java
b/nifi-commons/nifi-processor-utilities/src/main/java/org/apache/nifi/processor/util/bin/BinFiles.java
index 62e5655..7c90342 100644
---
a/nifi-commons/nifi-processor-utilities/src/main/java/org/apache/nifi/processor/util/bin/BinFiles.java
+++
b/nifi-commons/nifi-processor-utilities/src/main/java/org/apache/nifi/processor/util/bin/BinFiles.java
@@ -230,6 +230,14 @@ public abstract class BinFiles extends
AbstractSessionFactoryProcessor {
}
session.rollback();
return 1;
+} catch (final Exception e) {
+logger.error("Failed to process bundle of {} files due to {};
rolling back sessions", new Object[] {binCopy.size(), e});
+
+for (final FlowFileSessionWrapper wrapper : binCopy) {
+wrapper.getSession().rollback();
+}
+session.rollback();
+return 1;
}
// we first commit the bundle's session before the originals' sessions
because if we are restarted or crash
[2/2] nifi git commit: NIFI-1574: Ensure that we never flush a BufferedOutputStream's buffer on close of the write-ahead log
NIFI-1574: Ensure that we never flush a BufferedOutputStream's buffer on close
of the write-ahead log
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/62333c9e
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/62333c9e
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/62333c9e
Branch: refs/heads/master
Commit: 62333c9e0ae77854761cf019a65f54ccedc7af6b
Parents: 1149bc6
Author: Mark Payne
Authored: Sun Feb 28 10:08:28 2016 -0500
Committer: Mark Payne
Committed: Mon Feb 29 15:42:54 2016 -0500
--
.../org/wali/MinimalLockingWriteAheadLog.java | 58 +++---
.../wali/TestMinimalLockingWriteAheadLog.java | 178 +++
2 files changed, 208 insertions(+), 28 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/62333c9e/nifi-commons/nifi-write-ahead-log/src/main/java/org/wali/MinimalLockingWriteAheadLog.java
--
diff --git
a/nifi-commons/nifi-write-ahead-log/src/main/java/org/wali/MinimalLockingWriteAheadLog.java
b/nifi-commons/nifi-write-ahead-log/src/main/java/org/wali/MinimalLockingWriteAheadLog.java
index 501c330..f20f917 100644
---
a/nifi-commons/nifi-write-ahead-log/src/main/java/org/wali/MinimalLockingWriteAheadLog.java
+++
b/nifi-commons/nifi-write-ahead-log/src/main/java/org/wali/MinimalLockingWriteAheadLog.java
@@ -689,17 +689,43 @@ public final class MinimalLockingWriteAheadLog
implements WriteAheadRepositor
}
public void close() {
-final DataOutputStream out = dataOut;
+// Note that here we are closing fileOut and NOT dataOut.
+// This is very much intentional, not an oversight. This is done
because of
+// the way that the OutputStreams are structured. dataOut wraps a
BufferedOutputStream,
+// which then wraps the FileOutputStream. If we close 'dataOut',
then this will call
+// the flush() method of BufferedOutputStream. Under normal
conditions, this is fine.
+// However, there is a very important corner case to consider:
+//
+// If we are writing to the DataOutputStream in the update()
method and that
+// call to write() then results in the BufferedOutputStream
calling flushBuffer() -
+// or if we finish the call to update() and call flush()
ourselves - it is possible
+// that the internal buffer of the BufferedOutputStream can
get partially written to
+// to the FileOutputStream and then an IOException occurs. If
this occurs, we have
+// written a partial record to disk. This still is okay, as
we have logic to handle
+// the condition where we have a partial record and then an
unexpected End-of-File.
+// But if we then call close() on 'dataOut', this will call
the flush() method of the
+// underlying BufferedOutputStream. As a result, we will end
up again writing the internal
+// buffer of the BufferedOutputStream to the underlying file.
At this point, we are left
+// not with an unexpected/premature End-of-File but instead a
bunch of seemingly random
+// bytes that happened to be residing in that internal
buffer, and this will result in
+// a corrupt and unrecoverable Write-Ahead Log.
+//
+// Additionally, we are okay not ever calling close on the
wrapping BufferedOutputStream and
+// DataOutputStream because they don't actually hold any resources
that need to be reclaimed,
+// and after each update to the Write-Ahead Log, we call flush()
ourselves to ensure that we don't
+// leave arbitrary data in the BufferedOutputStream that hasn't
been flushed to the underlying
+// FileOutputStream.
+final OutputStream out = fileOut;
if (out != null) {
try {
out.close();
} catch (final Exception e) {
-
}
}
this.closed = true;
this.dataOut = null;
+this.fileOut = null;
}
public void blackList() {
@@ -721,32 +747,8 @@ public final class MinimalLockingWriteAheadLog
implements WriteAheadRepositor
public void rollover() throws IOException {
lock.lock();
try {
-// Note that here we are closing fileOut and NOT dataOut.
-// This is very much intentional, not an oversight. This is
done because of
-// the way that the OutputStreams are structured. dataOut
wraps a BufferedOutputStream,
-// which then wraps the FileOutp
[jira] [Commented] (NIFI-1572) Empty Queue sometimes indicates that 0 FlowFiles were dropped, even when queue contains many FlowFiles
[ https://issues.apache.org/jira/browse/NIFI-1572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15172564#comment-15172564 ] ASF subversion and git services commented on NIFI-1572: --- Commit 1149bc61cb3390210be4566dafba80564c4ab8d9 in nifi's branch refs/heads/master from [~markap14] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=1149bc6 ] NIFI-1572: Ensure that if an Exception is thrown when processing a bin, all sessions involved are rolled back or otherwise accounted for > Empty Queue sometimes indicates that 0 FlowFiles were dropped, even when > queue contains many FlowFiles > -- > > Key: NIFI-1572 > URL: https://issues.apache.org/jira/browse/NIFI-1572 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 0.6.0 > > Attachments: > 0001-NIFI-1572-Ensure-that-if-an-Exception-is-thrown-when.patch > > > Specifically, we are seeing this with MergeContent as the target of the > connection. It has about 1.2 million FlowFiles in its queue. The Processor > was stopped. So the session should be rolled back. It is unclear if the > problem is with rolling back the sessions or with Clearing the queue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1180) Modify PutS3Object to enable encryption
[ https://issues.apache.org/jira/browse/NIFI-1180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15172559#comment-15172559 ] ASF GitHub Bot commented on NIFI-1180: -- Github user mosermw commented on the pull request: https://github.com/apache/nifi/pull/246#issuecomment-190377532 When a PropertyDescriptor does not have a .displayName() set, then the .name() is used in the UI. There was a time when .displayName() did not exist and .name() displayed in the UI, so that's why you see many standard processors that do not use displayName. displayName was added so that you could change the name used in the UI without breaking backward compatibility in the flow.xml. So the original comment from @alopresto represents the latest best practice for NiFi processor development. > Modify PutS3Object to enable encryption > --- > > Key: NIFI-1180 > URL: https://issues.apache.org/jira/browse/NIFI-1180 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Elli Schwarz >Priority: Minor > > Amazon allows encryption to be enforced on objects that are put to S3. In the > PutS3Object class, the objectMetadata object contains a method > setSSEAlgorithm which can set encryption for the objects. A Nifi property > should be added to the PutS3Object class so a user can enable/disable > encryption from the processor itself. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1574) Write-Ahead log can become corrupt if NiFi is shutdown while disk is full
[
https://issues.apache.org/jira/browse/NIFI-1574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15172545#comment-15172545
]
Joseph Witt commented on NIFI-1574:
---
full clean build/seems to work. The shift in where the logic applied makes
sense.
Tough case to test to ensure it is solved .
+1
> Write-Ahead log can become corrupt if NiFi is shutdown while disk is full
> -
>
> Key: NIFI-1574
> URL: https://issues.apache.org/jira/browse/NIFI-1574
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 0.6.0
>
> Attachments:
> 0001-NIFI-1574-Ensure-that-we-never-flush-a-BufferedOutpu.patch
>
>
> The FlowFile Repository, as well as the Local State Provider can exhibit this
> problem. We saw the following stack trace when attempting to restart a node
> while its disk was completely full:
> {code}
> 2016-02-27 06:43:39,249 WARN [main] org.eclipse.jetty.webapp.WebAppContext
> Failed startup of context
> o.e.j.w.WebAppContext@8b727cb{/nifi-api,file:/opt/nifi/nifi-0.5.1/work/jetty/nifi-web-api-0.5.1.war/webapp/,STARTING}{./work/nar/framework/nifi-framework-nar-0.5.1.nar-unpacked/META-INF/bundled-dependencies/nifi-web-api-0.5.1.war}
> org.apache.nifi.web.NiFiCoreException: Unable to start Flow Controller.
> at
> org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:99)
> ~[na:na]
> at
> org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:800)
> ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
> at
> org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:444)
> ~[jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
> at
> org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:791)
> ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
> at
> org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:294)
> ~[jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
> at
> org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1349)
> ~[jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529]
> at
> org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342)
> ~[jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529]
> at
> org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:741)
> ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
> at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:505)
> ~[jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529]
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
> [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
> [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
> at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
> [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
> [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
> at org.eclipse.jetty.server.Server.start(Server.java:387)
> [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
> [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
> at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
> [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
> at org.eclipse.jetty.server.Server.doStart(Server.java:354)
> [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:663)
> [nifi-jetty-0.5.1.jar:0.5.1]
> at org.apache.nifi.NiFi.(NiFi.java:137) [nifi-runtime-0.5.1.jar:0.5.1]
> at org.apache.nifi.NiFi.main(NiFi.java:227) [nifi-runtime-0.5.1.jar:0.5.1]
> Caused by: org.springframework.beans.factory.BeanCreationException: Error
> creating bean with name 'flowService': FactoryBean threw exception on object
> creation; nested exception is
> org.springframework.beans.factory.BeanCreationException: Error creating bean
> wi
[jira] [Created] (NIFI-1577) NiFi holds open too many files when using a Run Duration > 0 ms and calling session.append
Mark Payne created NIFI-1577: Summary: NiFi holds open too many files when using a Run Duration > 0 ms and calling session.append Key: NIFI-1577 URL: https://issues.apache.org/jira/browse/NIFI-1577 Project: Apache NiFi Issue Type: Bug Components: Core Framework Reporter: Mark Payne If a Processor calls ProcessSession.append() and has a Run Duration scheduled > 0 ms, we quickly end up with "Too many open files" exceptions. This appears to be due to the fact that calling append() holds the content repository's stream open so that the session can keep appending to it, but on checkpoint() the session does not close these streams. It should close these streams on checkpoint, since the Processor is no longer allowed to reference these FlowFiles anyway at that point. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1572) Empty Queue sometimes indicates that 0 FlowFiles were dropped, even when queue contains many FlowFiles
[ https://issues.apache.org/jira/browse/NIFI-1572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15172452#comment-15172452 ] Joseph Witt commented on NIFI-1572: --- +1. Full clean build all good. Fix makes sense. Thanks > Empty Queue sometimes indicates that 0 FlowFiles were dropped, even when > queue contains many FlowFiles > -- > > Key: NIFI-1572 > URL: https://issues.apache.org/jira/browse/NIFI-1572 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 0.6.0 > > Attachments: > 0001-NIFI-1572-Ensure-that-if-an-Exception-is-thrown-when.patch > > > Specifically, we are seeing this with MergeContent as the target of the > connection. It has about 1.2 million FlowFiles in its queue. The Processor > was stopped. So the session should be rolled back. It is unclear if the > problem is with rolling back the sessions or with Clearing the queue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1180) Modify PutS3Object to enable encryption
[ https://issues.apache.org/jira/browse/NIFI-1180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15172383#comment-15172383 ] ASF GitHub Bot commented on NIFI-1180: -- Github user alopresto commented on the pull request: https://github.com/apache/nifi/pull/246#issuecomment-190332181 @adamonduty You are correct that the `.name`/`.displayName` convention is not documented well. I pinged @joewitt to look for a canonical source for the convention and we need to improve the documentation. In this case, I am ok with not changing the existing `PropertyDescriptors`, as it would break backwards compatibility as you noted. Unfortunately, this is a side-effect of the attributes not being set "properly" in the first place -- the `displayName` value is visible to the end user through the UI and may need to be changed, while the `name` value is used for object resolution in the application and so should not need to change if the messaging changes (imagine future internationalization where the `displayName` value is language-dependent). However, I do think in this case, the new `PropertyDescriptor` should follow the correct convention and a comment added noting the reason for the difference with other `PropertyDescriptor`s in the class. > Modify PutS3Object to enable encryption > --- > > Key: NIFI-1180 > URL: https://issues.apache.org/jira/browse/NIFI-1180 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Elli Schwarz >Priority: Minor > > Amazon allows encryption to be enforced on objects that are put to S3. In the > PutS3Object class, the objectMetadata object contains a method > setSSEAlgorithm which can set encryption for the objects. A Nifi property > should be added to the PutS3Object class so a user can enable/disable > encryption from the processor itself. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1572) Empty Queue sometimes indicates that 0 FlowFiles were dropped, even when queue contains many FlowFiles
[ https://issues.apache.org/jira/browse/NIFI-1572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15172203#comment-15172203 ] Joseph Witt commented on NIFI-1572: --- reviewing w/ 1574 > Empty Queue sometimes indicates that 0 FlowFiles were dropped, even when > queue contains many FlowFiles > -- > > Key: NIFI-1572 > URL: https://issues.apache.org/jira/browse/NIFI-1572 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 0.6.0 > > Attachments: > 0001-NIFI-1572-Ensure-that-if-an-Exception-is-thrown-when.patch > > > Specifically, we are seeing this with MergeContent as the target of the > connection. It has about 1.2 million FlowFiles in its queue. The Processor > was stopped. So the session should be rolled back. It is unclear if the > problem is with rolling back the sessions or with Clearing the queue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1574) Write-Ahead log can become corrupt if NiFi is shutdown while disk is full
[
https://issues.apache.org/jira/browse/NIFI-1574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15172204#comment-15172204
]
Joseph Witt commented on NIFI-1574:
---
reviewing with 1572
> Write-Ahead log can become corrupt if NiFi is shutdown while disk is full
> -
>
> Key: NIFI-1574
> URL: https://issues.apache.org/jira/browse/NIFI-1574
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 0.6.0
>
> Attachments:
> 0001-NIFI-1574-Ensure-that-we-never-flush-a-BufferedOutpu.patch
>
>
> The FlowFile Repository, as well as the Local State Provider can exhibit this
> problem. We saw the following stack trace when attempting to restart a node
> while its disk was completely full:
> {code}
> 2016-02-27 06:43:39,249 WARN [main] org.eclipse.jetty.webapp.WebAppContext
> Failed startup of context
> o.e.j.w.WebAppContext@8b727cb{/nifi-api,file:/opt/nifi/nifi-0.5.1/work/jetty/nifi-web-api-0.5.1.war/webapp/,STARTING}{./work/nar/framework/nifi-framework-nar-0.5.1.nar-unpacked/META-INF/bundled-dependencies/nifi-web-api-0.5.1.war}
> org.apache.nifi.web.NiFiCoreException: Unable to start Flow Controller.
> at
> org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:99)
> ~[na:na]
> at
> org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:800)
> ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
> at
> org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:444)
> ~[jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
> at
> org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:791)
> ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
> at
> org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:294)
> ~[jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
> at
> org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1349)
> ~[jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529]
> at
> org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342)
> ~[jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529]
> at
> org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:741)
> ~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
> at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:505)
> ~[jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529]
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
> [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
> [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
> at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
> [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
> [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
> at org.eclipse.jetty.server.Server.start(Server.java:387)
> [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
> [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
> at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
> [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
> at org.eclipse.jetty.server.Server.doStart(Server.java:354)
> [jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> [jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:663)
> [nifi-jetty-0.5.1.jar:0.5.1]
> at org.apache.nifi.NiFi.(NiFi.java:137) [nifi-runtime-0.5.1.jar:0.5.1]
> at org.apache.nifi.NiFi.main(NiFi.java:227) [nifi-runtime-0.5.1.jar:0.5.1]
> Caused by: org.springframework.beans.factory.BeanCreationException: Error
> creating bean with name 'flowService': FactoryBean threw exception on object
> creation; nested exception is
> org.springframework.beans.factory.BeanCreationException: Error creating bean
> with name 'flowController': FactoryBean threw exception on object creation;
> nested exception is java.lang.Illega
[jira] [Commented] (NIFI-615) Create a processor to extract WAV file characteristics
[ https://issues.apache.org/jira/browse/NIFI-615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15172107#comment-15172107 ] ASF GitHub Bot commented on NIFI-615: - GitHub user jskora opened a pull request: https://github.com/apache/nifi/pull/252 NIFI-615 - Create a processor to extract WAV file characteristics. * Created new ExtractMediaMetadata processor using Apache Tika Detector and Parser. * Refactored nifi-image-bundle, nifi-image-nar, and nifi-image-processors to nifi-media-bundle, nifi-media-nar, and nifi-media-processors to reflect broader media related purpose. * Preserved existing ExtractImageMetadata and ResizeImage processors as well as existing ImageViewerController components to prevent impact on existing uses. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jskora/nifi NIFI-615 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/252.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #252 commit 319d4e4643a9958e196c7226155c25c74371137c Author: Joe Skora Date: 2016-02-26T20:33:40Z NIFI-615 - Create a processor to extract WAV file characteristics. * Created new ExtractMediaMetadata processor using Apache Tika Detector and Parser. * Refactored nifi-image-bundle, nifi-image-nar, and nifi-image-processors to nifi-media-bundle, nifi-media-nar, and nifi-media-processors to reflect broader media related purpose. * Preserved existing ExtractImageMetadata and ResizeImage processors as well as existing ImageViewerController components to prevent impact on existing uses. > Create a processor to extract WAV file characteristics > -- > > Key: NIFI-615 > URL: https://issues.apache.org/jira/browse/NIFI-615 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Brandon DeVries >Assignee: Joe Skora >Priority: Minor > > Create a processor to extract information from a WAV file, including > encoding, bit rate, metadata, etc... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1575) Add ability to query DB table using "last value" from column
[ https://issues.apache.org/jira/browse/NIFI-1575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15172089#comment-15172089 ] Randy Gelhausen commented on NIFI-1575: --- I see use cases where the query needs to run not based exclusively on monotonically increasing values, but on some context provided by an external trigger. Thus a very useful (and hopefully not too complex) additional feature would be to allow the user to pull the "last value" from DistributedMapCache. This way external flows/activities may act as a more customizable and fine-grained control mechanism for the QueryDBTable processor. > Add ability to query DB table using "last value" from column > > > Key: NIFI-1575 > URL: https://issues.apache.org/jira/browse/NIFI-1575 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess > > It would be a useful feature for a processor to be able to query from a > database table for only those records that have are added/available since the > last time the query was executed. For example, if the processor could keep > the max value of a timestamp column for the last result set returned, the > next time the query was issued, it could use that timestamp value and column > to only return records whose timestamps were greater than "the last time". > This shouldn't be limited to timestamps of course; it would be useful for any > strictly-increasing value (like primary key ID) but would be up to the user > to select the table and column. The processor would simply keep the state for > the specified column's max value. > Proposed is a "QueryDBTable" processor which would have a properties to > specify the table name and the column in the table for which to keep state > information about the last maximum value retrieved. Subsequent queries of the > table use this value to filter for rows whose value for the specified column > are greater than the "last maximum value". Upon each successful query, the > "last maximum value" is updated. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-1576) Incorporate feedback/improvements to Elasticsearch processors
Matt Burgess created NIFI-1576: -- Summary: Incorporate feedback/improvements to Elasticsearch processors Key: NIFI-1576 URL: https://issues.apache.org/jira/browse/NIFI-1576 Project: Apache NiFi Issue Type: Improvement Reporter: Matt Burgess Since the release of Apache NiFi 0.5.0 containing FetchElasticsearch and PutElasticsearch processors, there have been some improvements identified that would add to the usability, flexibility, and power of the Elasticsearch processor(s). This Jira should capture those improvements for incorporation in a future release. Three initial observations: 1) FetchElasticsearch currently requires input, but it is possible to configure the processor such that no information is required from an incoming flow file. The processor should be changed to allow (but not require) input. 2) FetchElasticsearch specifies a "Document Identifier" property which may or may not contain Expression Language to determine the document ID for fetching. However PutElasticsearch only works when the IDs are unique, so it specifies a "Identifier Attribute" which forces the user to point at an existing attribute. The documentation should be very clear about what the properties are and why/how they're used. 3) Related to #2, PutElasticsearch's "Identifier Attribute" is required and a value for that attribute is currently required. Instead, if no value is present, the processor should leverage the auto-generated ID feature of the Elasticsearch API. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-1575) Add ability to query DB table using "last value" from column
Matt Burgess created NIFI-1575: -- Summary: Add ability to query DB table using "last value" from column Key: NIFI-1575 URL: https://issues.apache.org/jira/browse/NIFI-1575 Project: Apache NiFi Issue Type: Improvement Components: Extensions Reporter: Matt Burgess Assignee: Matt Burgess It would be a useful feature for a processor to be able to query from a database table for only those records that have are added/available since the last time the query was executed. For example, if the processor could keep the max value of a timestamp column for the last result set returned, the next time the query was issued, it could use that timestamp value and column to only return records whose timestamps were greater than "the last time". This shouldn't be limited to timestamps of course; it would be useful for any strictly-increasing value (like primary key ID) but would be up to the user to select the table and column. The processor would simply keep the state for the specified column's max value. Proposed is a "QueryDBTable" processor which would have a properties to specify the table name and the column in the table for which to keep state information about the last maximum value retrieved. Subsequent queries of the table use this value to filter for rows whose value for the specified column are greater than the "last maximum value". Upon each successful query, the "last maximum value" is updated. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-1572) Empty Queue sometimes indicates that 0 FlowFiles were dropped, even when queue contains many FlowFiles
[ https://issues.apache.org/jira/browse/NIFI-1572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-1572: - Attachment: 0001-NIFI-1572-Ensure-that-if-an-Exception-is-thrown-when.patch > Empty Queue sometimes indicates that 0 FlowFiles were dropped, even when > queue contains many FlowFiles > -- > > Key: NIFI-1572 > URL: https://issues.apache.org/jira/browse/NIFI-1572 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne > Fix For: 0.6.0 > > Attachments: > 0001-NIFI-1572-Ensure-that-if-an-Exception-is-thrown-when.patch > > > Specifically, we are seeing this with MergeContent as the target of the > connection. It has about 1.2 million FlowFiles in its queue. The Processor > was stopped. So the session should be rolled back. It is unclear if the > problem is with rolling back the sessions or with Clearing the queue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1558) Kafka processor clients write potentially sensitive info to the logs
[
https://issues.apache.org/jira/browse/NIFI-1558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15171969#comment-15171969
]
Oleg Zhurakousky commented on NIFI-1558:
[~joewitt], regardless if the data above is perceived to be sensitive or not
it's coming out of Kafka (not NiFi calling some toString() method) whenever
default log level is INFO. Assuming the logging settings is set to WARN when in
prod, this would be no issue, otherwise we would have to raise the issue with
Kafka. Let me know what you think.
> Kafka processor clients write potentially sensitive info to the logs
>
>
> Key: NIFI-1558
> URL: https://issues.apache.org/jira/browse/NIFI-1558
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 0.5.0
>Reporter: Joseph Witt
> Fix For: 0.6.0
>
>
> I noticed the logs on startup have things like the following. This needs to
> be suppressed as it is of relatively low value but relatively high risk given
> that it appears it would write out ssl key passphrases and such.
> {quote}
> 2016-02-23 21:13:56,626 INFO [pool-29-thread-7]
> o.a.k.clients.producer.ProducerConfig ProducerConfig values:
> compression.type = none
> metric.reporters = []
> metadata.max.age.ms = 30
> metadata.fetch.timeout.ms = 3
> reconnect.backoff.ms = 50
> sasl.kerberos.ticket.renew.window.factor = 0.8
> bootstrap.servers = [172.31.8.34:9093]
> retry.backoff.ms = 100
> sasl.kerberos.kinit.cmd = /usr/bin/kinit
> buffer.memory = 1048576
> timeout.ms = 3
> key.serializer = class
> org.apache.kafka.common.serialization.ByteArraySerializer
> sasl.kerberos.service.name = null
> sasl.kerberos.ticket.renew.jitter = 0.05
> ssl.keystore.type = JKS
> ssl.trustmanager.algorithm = PKIX
> block.on.buffer.full = false
> ssl.key.password = null
> max.block.ms = 6
> sasl.kerberos.min.time.before.relogin = 6
> connections.max.idle.ms = 54
> ssl.truststore.password = null
> max.in.flight.requests.per.connection = 5
> metrics.num.samples = 2
> client.id = NiFi-2243c3f9-bd2b-4bfe-b515-09791ec25c4c
> ssl.endpoint.identification.algorithm = null
> ssl.protocol = TLS
> request.timeout.ms = 3
> ssl.provider = null
> ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
> acks = 0
> batch.size = 200
> ssl.keystore.location = null
> receive.buffer.bytes = 32768
> ssl.cipher.suites = null
> ssl.truststore.type = JKS
> security.protocol = PLAINTEXT
> retries = 0
> max.request.size = 1048576
> value.serializer = class
> org.apache.kafka.common.serialization.ByteArraySerializer
> ssl.truststore.location = null
> ssl.keystore.password = null
> ssl.keymanager.algorithm = SunX509
> metrics.sample.window.ms = 3
> partitioner.class = class
> org.apache.kafka.clients.producer.internals.DefaultPartitioner
> send.buffer.bytes = 131072
> linger.ms = 5000
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
