[jira] [Updated] (NIFI-2654) Encrypted configs should handle login identity provider configs

[Andy LoPresto (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andy LoPresto updated NIFI-2654:

Fix Version/s: (was: 1.2.0)
   1.1.0

> Encrypted configs should handle login identity provider configs
> ---
>
> Key: NIFI-2654
> URL: https://issues.apache.org/jira/browse/NIFI-2654
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Configuration, Tools and Build
>Affects Versions: 1.0.0
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>  Labels: config, encryption, ldap, security
> Fix For: 1.1.0
>
>
> The encrypted configuration tool and internal logic to load unprotected 
> values should handle sensitive values contained in the login identity 
> providers (like LDAP Manager Password).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2654) Encrypted configs should handle login identity provider configs

[Andy LoPresto (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15658855#comment-15658855
 ] 

Andy LoPresto commented on NIFI-2654:
-

I have been working on this issue and [NIFI-2655] in parallel. Current work is 
available at [my branch|https://github.com/alopresto/nifi/tree/NIFI-2654]. I'll 
continue working this weekend, but it should make it into 1.1.0. 

NIFI-2654 work almost done
* LIP XML parsing complete
* Encryption/decryption complete
* Migration works
* Maintaining formatting/comments not yet complete
* Need to inject AES SPP into LIP context in core app

> Encrypted configs should handle login identity provider configs
> ---
>
> Key: NIFI-2654
> URL: https://issues.apache.org/jira/browse/NIFI-2654
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Configuration, Tools and Build
>Affects Versions: 1.0.0
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>  Labels: config, encryption, ldap, security
> Fix For: 1.2.0
>
>
> The encrypted configuration tool and internal logic to load unprotected 
> values should handle sensitive values contained in the login identity 
> providers (like LDAP Manager Password).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-3015) NiFi service starts from root user after installation

[Andre (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andre updated NIFI-3015:

Resolution: Not A Problem
Status: Resolved  (was: Patch Available)

While I understand the patch would streamline the config on linux environments, 
run as should be configured within bootstrap.conf as per documentation. 



> NiFi service starts from root user after installation
> -
>
> Key: NIFI-3015
> URL: https://issues.apache.org/jira/browse/NIFI-3015
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Configuration
>Affects Versions: 1.1.0
> Environment: Centos 7.2
>Reporter: Artem Yermakov
>Assignee: Andre
>Priority: Critical
>
> When install NiFi using command nifi.sh install, and then start NiFi by 
> command service nifi start, NiFi will start from user root.
> I suggest to run it from user nifi which is created during rpm installation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (NIFI-3015) NiFi service starts from root user after installation

[Andre (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andre reassigned NIFI-3015:
---

Assignee: Andre

> NiFi service starts from root user after installation
> -
>
> Key: NIFI-3015
> URL: https://issues.apache.org/jira/browse/NIFI-3015
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Configuration
>Affects Versions: 1.1.0
> Environment: Centos 7.2
>Reporter: Artem Yermakov
>Assignee: Andre
>Priority: Critical
>
> When install NiFi using command nifi.sh install, and then start NiFi by 
> command service nifi start, NiFi will start from user root.
> I suggest to run it from user nifi which is created during rpm installation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2854) Enable repositories to support upgrades and rollback in well defined scenarios

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15658585#comment-15658585
 ] 

ASF GitHub Bot commented on NIFI-2854:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1202#discussion_r87675047
  
--- Diff: 
nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/serialization/CompressableRecordReader.java
 ---
@@ -0,0 +1,281 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.provenance.serialization;
+
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.zip.GZIPInputStream;
+
+import org.apache.nifi.provenance.ProvenanceEventRecord;
+import org.apache.nifi.provenance.StandardProvenanceEventRecord;
+import org.apache.nifi.provenance.StandardRecordReader;
+import org.apache.nifi.provenance.toc.TocReader;
+import org.apache.nifi.stream.io.BufferedInputStream;
+import org.apache.nifi.stream.io.ByteCountingInputStream;
+import org.apache.nifi.stream.io.LimitingInputStream;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public abstract class CompressableRecordReader implements RecordReader {
+private static final Logger logger = 
LoggerFactory.getLogger(StandardRecordReader.class);
+
+private final ByteCountingInputStream rawInputStream;
+private final String filename;
+private final int serializationVersion;
+private final boolean compressed;
+private final TocReader tocReader;
+private final int headerLength;
+private final int maxAttributeChars;
+
+private DataInputStream dis;
+private ByteCountingInputStream byteCountingIn;
+
+public CompressableRecordReader(final InputStream in, final String 
filename, final int maxAttributeChars) throws IOException {
+this(in, filename, null, maxAttributeChars);
+}
+
+public CompressableRecordReader(final InputStream in, final String 
filename, final TocReader tocReader, final int maxAttributeChars) throws 
IOException {
+logger.trace("Creating RecordReader for {}", filename);
+
+rawInputStream = new ByteCountingInputStream(in);
+this.maxAttributeChars = maxAttributeChars;
+
+final InputStream limitedStream;
+if (tocReader == null) {
+limitedStream = rawInputStream;
+} else {
+final long offset1 = tocReader.getBlockOffset(1);
+if (offset1 < 0) {
+limitedStream = rawInputStream;
+} else {
+limitedStream = new LimitingInputStream(rawInputStream, 
offset1 - rawInputStream.getBytesConsumed());
+}
+}
+
+final InputStream readableStream;
+if (filename.endsWith(".gz")) {
+readableStream = new BufferedInputStream(new 
GZIPInputStream(limitedStream));
+compressed = true;
+} else {
+readableStream = new BufferedInputStream(limitedStream);
+compressed = false;
+}
+
+byteCountingIn = new ByteCountingInputStream(readableStream);
+dis = new DataInputStream(byteCountingIn);
+
+final String repoClassName = dis.readUTF();
+final int serializationVersion = dis.readInt();
+headerLength = 
repoClassName.getBytes(StandardCharsets.UTF_8).length + 2 + 4; // 2 bytes for 
string length, 4 for integer.
+
+if (serializationVersion < 1 || serializationVersion > 9) {
+throw new IllegalArgumentException("Unable to deserialize 
record because the version is " + serializationVersion + 

[jira] [Commented] (NIFI-2854) Enable repositories to support upgrades and rollback in well defined scenarios

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15658579#comment-15658579
 ] 

ASF GitHub Bot commented on NIFI-2854:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1202#discussion_r87674718
  
--- Diff: 
nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/serialization/CompressableRecordReader.java
 ---
@@ -0,0 +1,281 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.provenance.serialization;
+
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.zip.GZIPInputStream;
+
+import org.apache.nifi.provenance.ProvenanceEventRecord;
+import org.apache.nifi.provenance.StandardProvenanceEventRecord;
+import org.apache.nifi.provenance.StandardRecordReader;
+import org.apache.nifi.provenance.toc.TocReader;
+import org.apache.nifi.stream.io.BufferedInputStream;
+import org.apache.nifi.stream.io.ByteCountingInputStream;
+import org.apache.nifi.stream.io.LimitingInputStream;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public abstract class CompressableRecordReader implements RecordReader {
+private static final Logger logger = 
LoggerFactory.getLogger(StandardRecordReader.class);
+
+private final ByteCountingInputStream rawInputStream;
+private final String filename;
+private final int serializationVersion;
+private final boolean compressed;
+private final TocReader tocReader;
+private final int headerLength;
+private final int maxAttributeChars;
+
+private DataInputStream dis;
+private ByteCountingInputStream byteCountingIn;
+
+public CompressableRecordReader(final InputStream in, final String 
filename, final int maxAttributeChars) throws IOException {
+this(in, filename, null, maxAttributeChars);
+}
+
+public CompressableRecordReader(final InputStream in, final String 
filename, final TocReader tocReader, final int maxAttributeChars) throws 
IOException {
+logger.trace("Creating RecordReader for {}", filename);
+
+rawInputStream = new ByteCountingInputStream(in);
+this.maxAttributeChars = maxAttributeChars;
+
+final InputStream limitedStream;
+if (tocReader == null) {
+limitedStream = rawInputStream;
+} else {
+final long offset1 = tocReader.getBlockOffset(1);
+if (offset1 < 0) {
+limitedStream = rawInputStream;
+} else {
+limitedStream = new LimitingInputStream(rawInputStream, 
offset1 - rawInputStream.getBytesConsumed());
+}
+}
+
+final InputStream readableStream;
+if (filename.endsWith(".gz")) {
+readableStream = new BufferedInputStream(new 
GZIPInputStream(limitedStream));
+compressed = true;
+} else {
+readableStream = new BufferedInputStream(limitedStream);
+compressed = false;
+}
+
+byteCountingIn = new ByteCountingInputStream(readableStream);
+dis = new DataInputStream(byteCountingIn);
+
+final String repoClassName = dis.readUTF();
+final int serializationVersion = dis.readInt();
+headerLength = 
repoClassName.getBytes(StandardCharsets.UTF_8).length + 2 + 4; // 2 bytes for 
string length, 4 for integer.
+
+if (serializationVersion < 1 || serializationVersion > 9) {
+throw new IllegalArgumentException("Unable to deserialize 
record because the version is " + serializationVersion + 

[jira] [Commented] (NIFI-2854) Enable repositories to support upgrades and rollback in well defined scenarios

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15658583#comment-15658583
 ] 

ASF GitHub Bot commented on NIFI-2854:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1202#discussion_r87668398
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/repository/TestWriteAheadFlowFileRepository.java
 ---
@@ -64,6 +65,7 @@ public static void setupProperties() {
 }
 
 @Before
+@After
--- End diff --

Why does it need to be cleared before and after?


> Enable repositories to support upgrades and rollback in well defined scenarios
> --
>
> Key: NIFI-2854
> URL: https://issues.apache.org/jira/browse/NIFI-2854
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.1.0
>
>
> The flowfile, swapfile, provenance, and content repositories play a very 
> important roll in NiFi's ability to be safely upgraded and rolled back.  We 
> need to have well documented behaviors, designs, and version adherence so 
> that users can safely rely on these mechanisms.
> Once this is formalized and in place we should update our versioning guidance 
> to reflect this as well.
> The following would be true from NiFi 1.2.0 onward
> * No changes to how the repositories are persisted to disk can be made which 
> will break forward/backward compatibility and specifically this means that 
> things like the way each is serialized to disk cannot change.
> * If changes are made which impact forward or backward compatibility they 
> should be reserved for major releases only and should include a utility to 
> help users with pre-existing data convert from some older format to the newer 
> format.  It may not be feasible to have rollback on major releases.
> * The content repository should not be changed within a major release cycle 
> in any way that will harm forward or backward compatibility.
> * The flow file repository can change in that new fields can be added to 
> existing write ahead log record types but no fields can be removed nor can 
> any new types be added.  Once a field is considered required it must remain 
> required.  Changes may only be made across minor version changes - not 
> incremental.
> * Swap File storage should follow very similar rules to the flow file 
> repository.  Adding a schema to the swap file header may allow some variation 
> there but the variation should only be hints to optimize how they're 
> processed and not change their behavior otherwise. Changes are only permitted 
> during minor version releases.
> * Provenance repository changes are only permitted during minor version 
> releases.  These changes may include adding or removing fields from existing 
> event types.  If a field is considered required it must always be considered 
> required.  If a field is removed then it must not be a required field and 
> there must be a sensible default an older version could use if that value is 
> not found in new data once rolled back.  New event types may be added.  
> Fields or event types not known to older version, if seen after a rollback, 
> will simply be ignored.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2854) Enable repositories to support upgrades and rollback in well defined scenarios

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15658584#comment-15658584
 ] 

ASF GitHub Bot commented on NIFI-2854:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1202#discussion_r87661939
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/swap/SimpleSwapDeserializer.java
 ---
@@ -0,0 +1,303 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.controller.swap;
+
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+import org.apache.nifi.controller.queue.FlowFileQueue;
+import org.apache.nifi.controller.queue.QueueSize;
+import org.apache.nifi.controller.repository.FlowFileRecord;
+import org.apache.nifi.controller.repository.IncompleteSwapFileException;
+import org.apache.nifi.controller.repository.StandardFlowFileRecord;
+import org.apache.nifi.controller.repository.SwapContents;
+import org.apache.nifi.controller.repository.SwapSummary;
+import org.apache.nifi.controller.repository.claim.ResourceClaim;
+import org.apache.nifi.controller.repository.claim.ResourceClaimManager;
+import org.apache.nifi.controller.repository.claim.StandardContentClaim;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class SimpleSwapDeserializer implements SwapDeserializer {
+public static final int SWAP_ENCODING_VERSION = 10;
+private static final Logger logger = 
LoggerFactory.getLogger(SimpleSwapDeserializer.class);
+
+@Override
+public SwapSummary getSwapSummary(final DataInputStream in, final 
String swapLocation, final ResourceClaimManager claimManager) throws 
IOException {
+final int swapEncodingVersion = in.readInt();
+if (swapEncodingVersion > SWAP_ENCODING_VERSION) {
--- End diff --

Wouldn't the highest Encoding version this accepts be 9 (where 10 is this 
latest version which utilizes the new format)?


> Enable repositories to support upgrades and rollback in well defined scenarios
> --
>
> Key: NIFI-2854
> URL: https://issues.apache.org/jira/browse/NIFI-2854
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.1.0
>
>
> The flowfile, swapfile, provenance, and content repositories play a very 
> important roll in NiFi's ability to be safely upgraded and rolled back.  We 
> need to have well documented behaviors, designs, and version adherence so 
> that users can safely rely on these mechanisms.
> Once this is formalized and in place we should update our versioning guidance 
> to reflect this as well.
> The following would be true from NiFi 1.2.0 onward
> * No changes to how the repositories are persisted to disk can be made which 
> will break forward/backward compatibility and specifically this means that 
> things like the way each is serialized to disk cannot change.
> * If changes are made which impact forward or backward compatibility they 
> should be reserved for major releases only and should include a utility to 
> help users with pre-existing data convert from some older format to the newer 
> format.  It may not be feasible to have rollback on major releases.
> * The content repository should not be changed within a major release cycle 
> in any way that will harm forward or backward compatibility.
> * The flow file repository can change in that new fields can be added to 
> existing write ahead log record types but no fields can be removed nor can 
> any new 

[jira] [Commented] (NIFI-2854) Enable repositories to support upgrades and rollback in well defined scenarios

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15658572#comment-15658572
 ] 

ASF GitHub Bot commented on NIFI-2854:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1202#discussion_r87661134
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/swap/SimpleSwapSerializer.java
 ---
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.controller.swap;
+
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.nifi.controller.queue.FlowFileQueue;
+import org.apache.nifi.controller.repository.FlowFileRecord;
+import org.apache.nifi.controller.repository.claim.ContentClaim;
+import org.apache.nifi.controller.repository.claim.ResourceClaim;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class SimpleSwapSerializer implements SwapSerializer {
--- End diff --

I believe this can be marked deprecated since it will no longer be used (we 
aren't serializing the old format, only deserializing).


> Enable repositories to support upgrades and rollback in well defined scenarios
> --
>
> Key: NIFI-2854
> URL: https://issues.apache.org/jira/browse/NIFI-2854
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.1.0
>
>
> The flowfile, swapfile, provenance, and content repositories play a very 
> important roll in NiFi's ability to be safely upgraded and rolled back.  We 
> need to have well documented behaviors, designs, and version adherence so 
> that users can safely rely on these mechanisms.
> Once this is formalized and in place we should update our versioning guidance 
> to reflect this as well.
> The following would be true from NiFi 1.2.0 onward
> * No changes to how the repositories are persisted to disk can be made which 
> will break forward/backward compatibility and specifically this means that 
> things like the way each is serialized to disk cannot change.
> * If changes are made which impact forward or backward compatibility they 
> should be reserved for major releases only and should include a utility to 
> help users with pre-existing data convert from some older format to the newer 
> format.  It may not be feasible to have rollback on major releases.
> * The content repository should not be changed within a major release cycle 
> in any way that will harm forward or backward compatibility.
> * The flow file repository can change in that new fields can be added to 
> existing write ahead log record types but no fields can be removed nor can 
> any new types be added.  Once a field is considered required it must remain 
> required.  Changes may only be made across minor version changes - not 
> incremental.
> * Swap File storage should follow very similar rules to the flow file 
> repository.  Adding a schema to the swap file header may allow some variation 
> there but the variation should only be hints to optimize how they're 
> processed and not change their behavior otherwise. Changes are only permitted 
> during minor version releases.
> * Provenance repository changes are only permitted during minor version 
> releases.  These changes may include adding or removing fields from existing 
> event types.  If a field is considered required it must always be considered 
> required.  If a field is removed then it must not be a required field and 
> there must be a sensible default an older version could use if that value is 
> not found in new data once rolled back.  New event types may be added. 

[jira] [Commented] (NIFI-2854) Enable repositories to support upgrades and rollback in well defined scenarios

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15658582#comment-15658582
 ] 

ASF GitHub Bot commented on NIFI-2854:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1202#discussion_r87672258
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/swap/TestSimpleSwapSerializerDeserializer.java
 ---
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.controller.swap;
+
+import static org.junit.Assert.assertEquals;
+
+import java.io.DataInputStream;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.controller.queue.FlowFileQueue;
+import org.apache.nifi.controller.repository.FlowFileRecord;
+import org.apache.nifi.controller.repository.SwapContents;
+import org.apache.nifi.controller.repository.claim.ResourceClaimManager;
+import 
org.apache.nifi.controller.repository.claim.StandardResourceClaimManager;
+import org.apache.nifi.stream.io.NullOutputStream;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+public class TestSimpleSwapSerializerDeserializer {
+@Before
+public void setup() {
+TestFlowFile.resetIdGenerator();
+}
+
+@Test
+public void testRoundTripSerializeDeserialize() throws IOException {
+final ResourceClaimManager resourceClaimManager = new 
StandardResourceClaimManager();
+
+final List toSwap = new ArrayList<>(1);
+final Map attrs = new HashMap<>();
+for (int i = 0; i < 1; i++) {
+attrs.put("i", String.valueOf(i));
+final FlowFileRecord ff = new TestFlowFile(attrs, i, 
resourceClaimManager);
+toSwap.add(ff);
+}
+
+final FlowFileQueue flowFileQueue = 
Mockito.mock(FlowFileQueue.class);
+
Mockito.when(flowFileQueue.getIdentifier()).thenReturn("87bb99fe-412c-49f6-a441-d1b0af4e20b4");
+
+final String swapLocation = "target/testRoundTrip-" + 
UUID.randomUUID().toString() + ".swap";
+final File swapFile = new File(swapLocation);
+
+Files.deleteIfExists(swapFile.toPath());
+try {
+final SimpleSwapSerializer serializer = new 
SimpleSwapSerializer();
+try (final FileOutputStream fos = new 
FileOutputStream(swapFile)) {
+serializer.serializeFlowFiles(toSwap, flowFileQueue, 
swapLocation, fos);
+}
+
+final SimpleSwapDeserializer deserializer = new 
SimpleSwapDeserializer();
+final SwapContents swappedIn;
+try (final FileInputStream fis = new FileInputStream(swapFile);
+final DataInputStream dis = new DataInputStream(fis)) {
+swappedIn = deserializer.deserializeFlowFiles(dis, 
swapLocation, flowFileQueue, resourceClaimManager);
+}
+
+assertEquals(toSwap.size(), swappedIn.getFlowFiles().size());
+for (int i = 0; i < toSwap.size(); i++) {
+final FlowFileRecord pre = toSwap.get(i);
+final FlowFileRecord post = 
swappedIn.getFlowFiles().get(i);
+
+assertEquals(pre.getSize(), post.getSize());
+assertEquals(pre.getAttributes(), post.getAttributes());
+assertEquals(pre.getSize(), post.getSize());
+assertEquals(pre.getId(), post.getId());

[jira] [Commented] (NIFI-2854) Enable repositories to support upgrades and rollback in well defined scenarios

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15658577#comment-15658577
 ] 

ASF GitHub Bot commented on NIFI-2854:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1202#discussion_r87668539
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/swap/TestFlowFile.java
 ---
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.controller.swap;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.apache.nifi.controller.repository.FlowFileRecord;
+import org.apache.nifi.controller.repository.claim.ContentClaim;
+import org.apache.nifi.controller.repository.claim.ResourceClaim;
+import org.apache.nifi.controller.repository.claim.ResourceClaimManager;
+import org.apache.nifi.controller.repository.claim.StandardContentClaim;
+import org.apache.nifi.flowfile.FlowFile;
+
+public class TestFlowFile implements FlowFileRecord {
--- End diff --

Can this be renamed to MockFlowFile? "TestFlowFile" sounds like it is the 
unit tests for FlowFile


> Enable repositories to support upgrades and rollback in well defined scenarios
> --
>
> Key: NIFI-2854
> URL: https://issues.apache.org/jira/browse/NIFI-2854
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.1.0
>
>
> The flowfile, swapfile, provenance, and content repositories play a very 
> important roll in NiFi's ability to be safely upgraded and rolled back.  We 
> need to have well documented behaviors, designs, and version adherence so 
> that users can safely rely on these mechanisms.
> Once this is formalized and in place we should update our versioning guidance 
> to reflect this as well.
> The following would be true from NiFi 1.2.0 onward
> * No changes to how the repositories are persisted to disk can be made which 
> will break forward/backward compatibility and specifically this means that 
> things like the way each is serialized to disk cannot change.
> * If changes are made which impact forward or backward compatibility they 
> should be reserved for major releases only and should include a utility to 
> help users with pre-existing data convert from some older format to the newer 
> format.  It may not be feasible to have rollback on major releases.
> * The content repository should not be changed within a major release cycle 
> in any way that will harm forward or backward compatibility.
> * The flow file repository can change in that new fields can be added to 
> existing write ahead log record types but no fields can be removed nor can 
> any new types be added.  Once a field is considered required it must remain 
> required.  Changes may only be made across minor version changes - not 
> incremental.
> * Swap File storage should follow very similar rules to the flow file 
> repository.  Adding a schema to the swap file header may allow some variation 
> there but the variation should only be hints to optimize how they're 
> processed and not change their behavior otherwise. Changes are only permitted 
> during minor version releases.
> * Provenance repository changes are only permitted during minor version 
> releases.  These changes may include adding or removing fields from existing 
> event types.  If a field is considered required it must always be considered 
> required.  If a field is removed then it must not be a required field and 
> there must be a sensible default an older version could use if that value is 
> not found in new data once rolled back.  New event types may be added.  
> Fields or event types not known 

[jira] [Commented] (NIFI-2854) Enable repositories to support upgrades and rollback in well defined scenarios

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15658575#comment-15658575
 ] 

ASF GitHub Bot commented on NIFI-2854:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1202#discussion_r87672152
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/swap/TestSimpleSwapSerializerDeserializer.java
 ---
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.controller.swap;
+
+import static org.junit.Assert.assertEquals;
+
+import java.io.DataInputStream;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.controller.queue.FlowFileQueue;
+import org.apache.nifi.controller.repository.FlowFileRecord;
+import org.apache.nifi.controller.repository.SwapContents;
+import org.apache.nifi.controller.repository.claim.ResourceClaimManager;
+import 
org.apache.nifi.controller.repository.claim.StandardResourceClaimManager;
+import org.apache.nifi.stream.io.NullOutputStream;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+public class TestSimpleSwapSerializerDeserializer {
+@Before
+public void setup() {
+TestFlowFile.resetIdGenerator();
+}
+
+@Test
+public void testRoundTripSerializeDeserialize() throws IOException {
+final ResourceClaimManager resourceClaimManager = new 
StandardResourceClaimManager();
+
+final List toSwap = new ArrayList<>(1);
+final Map attrs = new HashMap<>();
+for (int i = 0; i < 1; i++) {
+attrs.put("i", String.valueOf(i));
+final FlowFileRecord ff = new TestFlowFile(attrs, i, 
resourceClaimManager);
+toSwap.add(ff);
+}
+
+final FlowFileQueue flowFileQueue = 
Mockito.mock(FlowFileQueue.class);
+
Mockito.when(flowFileQueue.getIdentifier()).thenReturn("87bb99fe-412c-49f6-a441-d1b0af4e20b4");
+
+final String swapLocation = "target/testRoundTrip-" + 
UUID.randomUUID().toString() + ".swap";
--- End diff --

Shouldn't the UUID in the middle of the swap location be the same as the 
flowFileQueue identifier set above?


> Enable repositories to support upgrades and rollback in well defined scenarios
> --
>
> Key: NIFI-2854
> URL: https://issues.apache.org/jira/browse/NIFI-2854
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.1.0
>
>
> The flowfile, swapfile, provenance, and content repositories play a very 
> important roll in NiFi's ability to be safely upgraded and rolled back.  We 
> need to have well documented behaviors, designs, and version adherence so 
> that users can safely rely on these mechanisms.
> Once this is formalized and in place we should update our versioning guidance 
> to reflect this as well.
> The following would be true from NiFi 1.2.0 onward
> * No changes to how the repositories are persisted to disk can be made which 
> will break forward/backward compatibility and specifically this means that 
> things like the way each is serialized to disk cannot change.
> * If changes are made which impact forward or backward compatibility they 
> should be reserved for major releases only and should include a utility to 
> 

[GitHub] nifi pull request #1202: NIFI-2854: Refactor repositories and swap files to ...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1202#discussion_r87677086
  
--- Diff: 
nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/schema/FieldSerializer.java
 ---
@@ -0,0 +1,27 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.provenance.schema;
+
+import java.io.DataOutputStream;
+import java.io.IOException;
+
+import org.apache.nifi.provenance.ProvenanceEventRecord;
+
+public interface FieldSerializer {
--- End diff --

Intellij tells me this class isn't used, remnant of design iterations?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2854) Enable repositories to support upgrades and rollback in well defined scenarios

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15658571#comment-15658571
 ] 

ASF GitHub Bot commented on NIFI-2854:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1202#discussion_r87648546
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/StandardProcessSession.java
 ---
@@ -121,8 +121,8 @@
 
 private int removedCount = 0; // number of flowfiles removed in this 
session
 private long removedBytes = 0L; // size of all flowfiles removed in 
this session
-private final AtomicLong bytesRead = new AtomicLong(0L);
-private final AtomicLong bytesWritten = new AtomicLong(0L);
+private long bytesRead = 0L;
+private long bytesWritten = 0L;
--- End diff --

Why change these from AtomicLongs? None of the places where they're 
written/read could be from multiple threads?


> Enable repositories to support upgrades and rollback in well defined scenarios
> --
>
> Key: NIFI-2854
> URL: https://issues.apache.org/jira/browse/NIFI-2854
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.1.0
>
>
> The flowfile, swapfile, provenance, and content repositories play a very 
> important roll in NiFi's ability to be safely upgraded and rolled back.  We 
> need to have well documented behaviors, designs, and version adherence so 
> that users can safely rely on these mechanisms.
> Once this is formalized and in place we should update our versioning guidance 
> to reflect this as well.
> The following would be true from NiFi 1.2.0 onward
> * No changes to how the repositories are persisted to disk can be made which 
> will break forward/backward compatibility and specifically this means that 
> things like the way each is serialized to disk cannot change.
> * If changes are made which impact forward or backward compatibility they 
> should be reserved for major releases only and should include a utility to 
> help users with pre-existing data convert from some older format to the newer 
> format.  It may not be feasible to have rollback on major releases.
> * The content repository should not be changed within a major release cycle 
> in any way that will harm forward or backward compatibility.
> * The flow file repository can change in that new fields can be added to 
> existing write ahead log record types but no fields can be removed nor can 
> any new types be added.  Once a field is considered required it must remain 
> required.  Changes may only be made across minor version changes - not 
> incremental.
> * Swap File storage should follow very similar rules to the flow file 
> repository.  Adding a schema to the swap file header may allow some variation 
> there but the variation should only be hints to optimize how they're 
> processed and not change their behavior otherwise. Changes are only permitted 
> during minor version releases.
> * Provenance repository changes are only permitted during minor version 
> releases.  These changes may include adding or removing fields from existing 
> event types.  If a field is considered required it must always be considered 
> required.  If a field is removed then it must not be a required field and 
> there must be a sensible default an older version could use if that value is 
> not found in new data once rolled back.  New event types may be added.  
> Fields or event types not known to older version, if seen after a rollback, 
> will simply be ignored.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #1202: NIFI-2854: Refactor repositories and swap files to ...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1202#discussion_r87661134
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/swap/SimpleSwapSerializer.java
 ---
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.controller.swap;
+
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.nifi.controller.queue.FlowFileQueue;
+import org.apache.nifi.controller.repository.FlowFileRecord;
+import org.apache.nifi.controller.repository.claim.ContentClaim;
+import org.apache.nifi.controller.repository.claim.ResourceClaim;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class SimpleSwapSerializer implements SwapSerializer {
--- End diff --

I believe this can be marked deprecated since it will no longer be used (we 
aren't serializing the old format, only deserializing).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1202: NIFI-2854: Refactor repositories and swap files to ...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1202#discussion_r87648546
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/StandardProcessSession.java
 ---
@@ -121,8 +121,8 @@
 
 private int removedCount = 0; // number of flowfiles removed in this 
session
 private long removedBytes = 0L; // size of all flowfiles removed in 
this session
-private final AtomicLong bytesRead = new AtomicLong(0L);
-private final AtomicLong bytesWritten = new AtomicLong(0L);
+private long bytesRead = 0L;
+private long bytesWritten = 0L;
--- End diff --

Why change these from AtomicLongs? None of the places where they're 
written/read could be from multiple threads?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1202: NIFI-2854: Refactor repositories and swap files to ...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1202#discussion_r87672152
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/swap/TestSimpleSwapSerializerDeserializer.java
 ---
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.controller.swap;
+
+import static org.junit.Assert.assertEquals;
+
+import java.io.DataInputStream;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.controller.queue.FlowFileQueue;
+import org.apache.nifi.controller.repository.FlowFileRecord;
+import org.apache.nifi.controller.repository.SwapContents;
+import org.apache.nifi.controller.repository.claim.ResourceClaimManager;
+import 
org.apache.nifi.controller.repository.claim.StandardResourceClaimManager;
+import org.apache.nifi.stream.io.NullOutputStream;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+public class TestSimpleSwapSerializerDeserializer {
+@Before
+public void setup() {
+TestFlowFile.resetIdGenerator();
+}
+
+@Test
+public void testRoundTripSerializeDeserialize() throws IOException {
+final ResourceClaimManager resourceClaimManager = new 
StandardResourceClaimManager();
+
+final List toSwap = new ArrayList<>(1);
+final Map attrs = new HashMap<>();
+for (int i = 0; i < 1; i++) {
+attrs.put("i", String.valueOf(i));
+final FlowFileRecord ff = new TestFlowFile(attrs, i, 
resourceClaimManager);
+toSwap.add(ff);
+}
+
+final FlowFileQueue flowFileQueue = 
Mockito.mock(FlowFileQueue.class);
+
Mockito.when(flowFileQueue.getIdentifier()).thenReturn("87bb99fe-412c-49f6-a441-d1b0af4e20b4");
+
+final String swapLocation = "target/testRoundTrip-" + 
UUID.randomUUID().toString() + ".swap";
--- End diff --

Shouldn't the UUID in the middle of the swap location be the same as the 
flowFileQueue identifier set above?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2854) Enable repositories to support upgrades and rollback in well defined scenarios

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15658576#comment-15658576
 ] 

ASF GitHub Bot commented on NIFI-2854:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1202#discussion_r87631014
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/WriteAheadRepositoryRecordSerde.java
 ---
@@ -0,0 +1,517 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.controller.repository;
+
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.nifi.controller.queue.FlowFileQueue;
+import org.apache.nifi.controller.repository.claim.ContentClaim;
+import org.apache.nifi.controller.repository.claim.ResourceClaim;
+import org.apache.nifi.controller.repository.claim.ResourceClaimManager;
+import org.apache.nifi.controller.repository.claim.StandardContentClaim;
+import org.apache.nifi.flowfile.FlowFile;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.wali.SerDe;
+import org.wali.UpdateType;
+
+public class WriteAheadRepositoryRecordSerde extends RepositoryRecordSerde 
implements SerDe {
+private static final Logger logger = 
LoggerFactory.getLogger(WriteAheadRepositoryRecordSerde.class);
+
+private static final int CURRENT_ENCODING_VERSION = 9;
+
+public static final byte ACTION_CREATE = 0;
+public static final byte ACTION_UPDATE = 1;
+public static final byte ACTION_DELETE = 2;
+public static final byte ACTION_SWAPPED_OUT = 3;
+public static final byte ACTION_SWAPPED_IN = 4;
+
+private long recordsRestored = 0L;
+private final ResourceClaimManager claimManager;
+
+public WriteAheadRepositoryRecordSerde(final ResourceClaimManager 
claimManager) {
+this.claimManager = claimManager;
+}
+
+@Override
+public void serializeEdit(final RepositoryRecord previousRecordState, 
final RepositoryRecord record, final DataOutputStream out) throws IOException {
+serializeEdit(previousRecordState, record, out, false);
+}
+
+public void serializeEdit(final RepositoryRecord previousRecordState, 
final RepositoryRecord record, final DataOutputStream out, final boolean 
forceAttributesWritten) throws IOException {
+if (record.isMarkedForAbort()) {
+logger.warn("Repository Record {} is marked to be aborted; it 
will be persisted in the FlowFileRepository as a DELETE record", record);
+out.write(ACTION_DELETE);
+out.writeLong(getRecordIdentifier(record));
+serializeContentClaim(record.getCurrentClaim(), 
record.getCurrentClaimOffset(), out);
+return;
+}
+
+final UpdateType updateType = getUpdateType(record);
+
+if (updateType.equals(UpdateType.DELETE)) {
+out.write(ACTION_DELETE);
+out.writeLong(getRecordIdentifier(record));
+serializeContentClaim(record.getCurrentClaim(), 
record.getCurrentClaimOffset(), out);
+return;
+}
+
+// If there's a Destination Connection, that's the one that we 
want to associated with this record.
+// However, on restart, we will restore the FlowFile and set this 
connection to its "originalConnection".
+// If we then serialize the FlowFile again before it's 
transferred, it's important to allow this to happen,
+// so we use the originalConnection instead
+FlowFileQueue associatedQueue = record.getDestination();
+if (associatedQueue == null) {
+   

[GitHub] nifi pull request #1202: NIFI-2854: Refactor repositories and swap files to ...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1202#discussion_r87654483
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FileSystemSwapManager.java
 ---
@@ -210,30 +215,36 @@ public boolean accept(final File dir, final String 
name) {
 // "--.swap". If we 
have two dashes, then we can just check if the queue ID is equal
 // to the id of the queue given and if not we can just move on.
 final String[] splits = swapFile.getName().split("-");
-if (splits.length == 3) {
-final String queueIdentifier = splits[1];
-if 
(!queueIdentifier.equals(flowFileQueue.getIdentifier())) {
-continue;
+if (splits.length > 6) {
--- End diff --

Was this broken before? The ID scheme for queue hasn't changed (still a 
UUID)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2854) Enable repositories to support upgrades and rollback in well defined scenarios

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15658574#comment-15658574
 ] 

ASF GitHub Bot commented on NIFI-2854:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1202#discussion_r87677686
  
--- Diff: 
nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/test/java/org/apache/nifi/provenance/TestPersistentProvenanceRepository.java
 ---
@@ -1914,112 +1916,6 @@ public void 
testFailureToCreateWriterDoesNotPreventSubsequentRollover() throws I
 }
 
 
-@Test
-public void testBehaviorOnOutOfMemory() throws IOException, 
InterruptedException {
--- End diff --

Why was this test removed?


> Enable repositories to support upgrades and rollback in well defined scenarios
> --
>
> Key: NIFI-2854
> URL: https://issues.apache.org/jira/browse/NIFI-2854
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.1.0
>
>
> The flowfile, swapfile, provenance, and content repositories play a very 
> important roll in NiFi's ability to be safely upgraded and rolled back.  We 
> need to have well documented behaviors, designs, and version adherence so 
> that users can safely rely on these mechanisms.
> Once this is formalized and in place we should update our versioning guidance 
> to reflect this as well.
> The following would be true from NiFi 1.2.0 onward
> * No changes to how the repositories are persisted to disk can be made which 
> will break forward/backward compatibility and specifically this means that 
> things like the way each is serialized to disk cannot change.
> * If changes are made which impact forward or backward compatibility they 
> should be reserved for major releases only and should include a utility to 
> help users with pre-existing data convert from some older format to the newer 
> format.  It may not be feasible to have rollback on major releases.
> * The content repository should not be changed within a major release cycle 
> in any way that will harm forward or backward compatibility.
> * The flow file repository can change in that new fields can be added to 
> existing write ahead log record types but no fields can be removed nor can 
> any new types be added.  Once a field is considered required it must remain 
> required.  Changes may only be made across minor version changes - not 
> incremental.
> * Swap File storage should follow very similar rules to the flow file 
> repository.  Adding a schema to the swap file header may allow some variation 
> there but the variation should only be hints to optimize how they're 
> processed and not change their behavior otherwise. Changes are only permitted 
> during minor version releases.
> * Provenance repository changes are only permitted during minor version 
> releases.  These changes may include adding or removing fields from existing 
> event types.  If a field is considered required it must always be considered 
> required.  If a field is removed then it must not be a required field and 
> there must be a sensible default an older version could use if that value is 
> not found in new data once rolled back.  New event types may be added.  
> Fields or event types not known to older version, if seen after a rollback, 
> will simply be ignored.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #1202: NIFI-2854: Refactor repositories and swap files to ...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1202#discussion_r87629214
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/WriteAheadRepositoryRecordSerde.java
 ---
@@ -0,0 +1,517 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.controller.repository;
+
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.nifi.controller.queue.FlowFileQueue;
+import org.apache.nifi.controller.repository.claim.ContentClaim;
+import org.apache.nifi.controller.repository.claim.ResourceClaim;
+import org.apache.nifi.controller.repository.claim.ResourceClaimManager;
+import org.apache.nifi.controller.repository.claim.StandardContentClaim;
+import org.apache.nifi.flowfile.FlowFile;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.wali.SerDe;
+import org.wali.UpdateType;
+
+public class WriteAheadRepositoryRecordSerde extends RepositoryRecordSerde 
implements SerDe {
+private static final Logger logger = 
LoggerFactory.getLogger(WriteAheadRepositoryRecordSerde.class);
+
+private static final int CURRENT_ENCODING_VERSION = 9;
+
+public static final byte ACTION_CREATE = 0;
+public static final byte ACTION_UPDATE = 1;
+public static final byte ACTION_DELETE = 2;
+public static final byte ACTION_SWAPPED_OUT = 3;
+public static final byte ACTION_SWAPPED_IN = 4;
+
+private long recordsRestored = 0L;
+private final ResourceClaimManager claimManager;
+
+public WriteAheadRepositoryRecordSerde(final ResourceClaimManager 
claimManager) {
+this.claimManager = claimManager;
+}
+
+@Override
+public void serializeEdit(final RepositoryRecord previousRecordState, 
final RepositoryRecord record, final DataOutputStream out) throws IOException {
+serializeEdit(previousRecordState, record, out, false);
+}
+
+public void serializeEdit(final RepositoryRecord previousRecordState, 
final RepositoryRecord record, final DataOutputStream out, final boolean 
forceAttributesWritten) throws IOException {
+if (record.isMarkedForAbort()) {
+logger.warn("Repository Record {} is marked to be aborted; it 
will be persisted in the FlowFileRepository as a DELETE record", record);
+out.write(ACTION_DELETE);
+out.writeLong(getRecordIdentifier(record));
+serializeContentClaim(record.getCurrentClaim(), 
record.getCurrentClaimOffset(), out);
+return;
+}
+
+final UpdateType updateType = getUpdateType(record);
+
+if (updateType.equals(UpdateType.DELETE)) {
+out.write(ACTION_DELETE);
+out.writeLong(getRecordIdentifier(record));
+serializeContentClaim(record.getCurrentClaim(), 
record.getCurrentClaimOffset(), out);
+return;
+}
+
+// If there's a Destination Connection, that's the one that we 
want to associated with this record.
+// However, on restart, we will restore the FlowFile and set this 
connection to its "originalConnection".
+// If we then serialize the FlowFile again before it's 
transferred, it's important to allow this to happen,
+// so we use the originalConnection instead
+FlowFileQueue associatedQueue = record.getDestination();
+if (associatedQueue == null) {
+associatedQueue = record.getOriginalQueue();
+}
+
+if (updateType.equals(UpdateType.SWAP_OUT)) {
+out.write(ACTION_SWAPPED_OUT);
+

[GitHub] nifi pull request #1202: NIFI-2854: Refactor repositories and swap files to ...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1202#discussion_r87631014
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/WriteAheadRepositoryRecordSerde.java
 ---
@@ -0,0 +1,517 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.controller.repository;
+
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.nifi.controller.queue.FlowFileQueue;
+import org.apache.nifi.controller.repository.claim.ContentClaim;
+import org.apache.nifi.controller.repository.claim.ResourceClaim;
+import org.apache.nifi.controller.repository.claim.ResourceClaimManager;
+import org.apache.nifi.controller.repository.claim.StandardContentClaim;
+import org.apache.nifi.flowfile.FlowFile;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.wali.SerDe;
+import org.wali.UpdateType;
+
+public class WriteAheadRepositoryRecordSerde extends RepositoryRecordSerde 
implements SerDe {
+private static final Logger logger = 
LoggerFactory.getLogger(WriteAheadRepositoryRecordSerde.class);
+
+private static final int CURRENT_ENCODING_VERSION = 9;
+
+public static final byte ACTION_CREATE = 0;
+public static final byte ACTION_UPDATE = 1;
+public static final byte ACTION_DELETE = 2;
+public static final byte ACTION_SWAPPED_OUT = 3;
+public static final byte ACTION_SWAPPED_IN = 4;
+
+private long recordsRestored = 0L;
+private final ResourceClaimManager claimManager;
+
+public WriteAheadRepositoryRecordSerde(final ResourceClaimManager 
claimManager) {
+this.claimManager = claimManager;
+}
+
+@Override
+public void serializeEdit(final RepositoryRecord previousRecordState, 
final RepositoryRecord record, final DataOutputStream out) throws IOException {
+serializeEdit(previousRecordState, record, out, false);
+}
+
+public void serializeEdit(final RepositoryRecord previousRecordState, 
final RepositoryRecord record, final DataOutputStream out, final boolean 
forceAttributesWritten) throws IOException {
+if (record.isMarkedForAbort()) {
+logger.warn("Repository Record {} is marked to be aborted; it 
will be persisted in the FlowFileRepository as a DELETE record", record);
+out.write(ACTION_DELETE);
+out.writeLong(getRecordIdentifier(record));
+serializeContentClaim(record.getCurrentClaim(), 
record.getCurrentClaimOffset(), out);
+return;
+}
+
+final UpdateType updateType = getUpdateType(record);
+
+if (updateType.equals(UpdateType.DELETE)) {
+out.write(ACTION_DELETE);
+out.writeLong(getRecordIdentifier(record));
+serializeContentClaim(record.getCurrentClaim(), 
record.getCurrentClaimOffset(), out);
+return;
+}
+
+// If there's a Destination Connection, that's the one that we 
want to associated with this record.
+// However, on restart, we will restore the FlowFile and set this 
connection to its "originalConnection".
+// If we then serialize the FlowFile again before it's 
transferred, it's important to allow this to happen,
+// so we use the originalConnection instead
+FlowFileQueue associatedQueue = record.getDestination();
+if (associatedQueue == null) {
+associatedQueue = record.getOriginalQueue();
+}
+
+if (updateType.equals(UpdateType.SWAP_OUT)) {
+out.write(ACTION_SWAPPED_OUT);
+

[GitHub] nifi pull request #1202: NIFI-2854: Refactor repositories and swap files to ...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1202#discussion_r87675047
  
--- Diff: 
nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/serialization/CompressableRecordReader.java
 ---
@@ -0,0 +1,281 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.provenance.serialization;
+
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.zip.GZIPInputStream;
+
+import org.apache.nifi.provenance.ProvenanceEventRecord;
+import org.apache.nifi.provenance.StandardProvenanceEventRecord;
+import org.apache.nifi.provenance.StandardRecordReader;
+import org.apache.nifi.provenance.toc.TocReader;
+import org.apache.nifi.stream.io.BufferedInputStream;
+import org.apache.nifi.stream.io.ByteCountingInputStream;
+import org.apache.nifi.stream.io.LimitingInputStream;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public abstract class CompressableRecordReader implements RecordReader {
+private static final Logger logger = 
LoggerFactory.getLogger(StandardRecordReader.class);
+
+private final ByteCountingInputStream rawInputStream;
+private final String filename;
+private final int serializationVersion;
+private final boolean compressed;
+private final TocReader tocReader;
+private final int headerLength;
+private final int maxAttributeChars;
+
+private DataInputStream dis;
+private ByteCountingInputStream byteCountingIn;
+
+public CompressableRecordReader(final InputStream in, final String 
filename, final int maxAttributeChars) throws IOException {
+this(in, filename, null, maxAttributeChars);
+}
+
+public CompressableRecordReader(final InputStream in, final String 
filename, final TocReader tocReader, final int maxAttributeChars) throws 
IOException {
+logger.trace("Creating RecordReader for {}", filename);
+
+rawInputStream = new ByteCountingInputStream(in);
+this.maxAttributeChars = maxAttributeChars;
+
+final InputStream limitedStream;
+if (tocReader == null) {
+limitedStream = rawInputStream;
+} else {
+final long offset1 = tocReader.getBlockOffset(1);
+if (offset1 < 0) {
+limitedStream = rawInputStream;
+} else {
+limitedStream = new LimitingInputStream(rawInputStream, 
offset1 - rawInputStream.getBytesConsumed());
+}
+}
+
+final InputStream readableStream;
+if (filename.endsWith(".gz")) {
+readableStream = new BufferedInputStream(new 
GZIPInputStream(limitedStream));
+compressed = true;
+} else {
+readableStream = new BufferedInputStream(limitedStream);
+compressed = false;
+}
+
+byteCountingIn = new ByteCountingInputStream(readableStream);
+dis = new DataInputStream(byteCountingIn);
+
+final String repoClassName = dis.readUTF();
+final int serializationVersion = dis.readInt();
+headerLength = 
repoClassName.getBytes(StandardCharsets.UTF_8).length + 2 + 4; // 2 bytes for 
string length, 4 for integer.
+
+if (serializationVersion < 1 || serializationVersion > 9) {
+throw new IllegalArgumentException("Unable to deserialize 
record because the version is " + serializationVersion + " and supported 
versions are 1-9");
+}
+
+this.serializationVersion = serializationVersion;
+this.filename = filename;
+this.tocReader = tocReader;
+
+readHeader(dis, 

[GitHub] nifi pull request #1202: NIFI-2854: Refactor repositories and swap files to ...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1202#discussion_r87657911
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FileSystemSwapManager.java
 ---
@@ -251,353 +262,36 @@ public SwapSummary getSwapSummary(final String 
swapLocation) throws IOException
 final InputStream bufferedIn = new 
BufferedInputStream(fis);
 final DataInputStream in = new 
DataInputStream(bufferedIn)) {
 
-final int swapEncodingVersion = in.readInt();
-if (swapEncodingVersion > SWAP_ENCODING_VERSION) {
-final String errMsg = "Cannot swap FlowFiles in from " + 
swapFile + " because the encoding version is "
-+ swapEncodingVersion + ", which is too new 
(expecting " + SWAP_ENCODING_VERSION + " or less)";
-
-eventReporter.reportEvent(Severity.ERROR, EVENT_CATEGORY, 
errMsg);
-throw new IOException(errMsg);
-}
-
-final int numRecords;
-final long contentSize;
-Long maxRecordId = null;
-try {
-in.readUTF(); // ignore Connection ID
-numRecords = in.readInt();
-contentSize = in.readLong();
-
-if (numRecords == 0) {
-return StandardSwapSummary.EMPTY_SUMMARY;
-}
-
-if (swapEncodingVersion > 7) {
-maxRecordId = in.readLong();
-}
-} catch (final EOFException eof) {
-logger.warn("Found premature End-of-File when reading Swap 
File {}. EOF occurred before any FlowFiles were encountered", swapLocation);
-return StandardSwapSummary.EMPTY_SUMMARY;
-}
-
-final QueueSize queueSize = new QueueSize(numRecords, 
contentSize);
-final SwapContents swapContents = deserializeFlowFiles(in, 
queueSize, maxRecordId, swapEncodingVersion, true, claimManager, swapLocation);
-return swapContents.getSummary();
-}
-}
-
-public static int serializeFlowFiles(final List 
toSwap, final FlowFileQueue queue, final String swapLocation, final 
OutputStream destination) throws IOException {
-if (toSwap == null || toSwap.isEmpty()) {
-return 0;
-}
-
-long contentSize = 0L;
-for (final FlowFileRecord record : toSwap) {
-contentSize += record.getSize();
-}
-
-// persist record to disk via the swap file
-final OutputStream bufferedOut = new 
BufferedOutputStream(destination);
-final DataOutputStream out = new DataOutputStream(bufferedOut);
-try {
-out.writeInt(SWAP_ENCODING_VERSION);
-out.writeUTF(queue.getIdentifier());
-out.writeInt(toSwap.size());
-out.writeLong(contentSize);
-
-// get the max record id and write that out so that we know it 
quickly for restoration
-long maxRecordId = 0L;
-for (final FlowFileRecord flowFile : toSwap) {
-if (flowFile.getId() > maxRecordId) {
-maxRecordId = flowFile.getId();
-}
-}
-
-out.writeLong(maxRecordId);
-
-for (final FlowFileRecord flowFile : toSwap) {
-out.writeLong(flowFile.getId());
-out.writeLong(flowFile.getEntryDate());
-out.writeLong(flowFile.getLineageStartDate());
-out.writeLong(flowFile.getLineageStartIndex());
-out.writeLong(flowFile.getLastQueueDate());
-out.writeLong(flowFile.getQueueDateIndex());
-out.writeLong(flowFile.getSize());
-
-final ContentClaim claim = flowFile.getContentClaim();
-if (claim == null) {
-out.writeBoolean(false);
-} else {
-out.writeBoolean(true);
-final ResourceClaim resourceClaim = 
claim.getResourceClaim();
-out.writeUTF(resourceClaim.getId());
-out.writeUTF(resourceClaim.getContainer());
-out.writeUTF(resourceClaim.getSection());
-out.writeLong(claim.getOffset());
-out.writeLong(claim.getLength());
-out.writeLong(flowFile.getContentClaimOffset());
-out.writeBoolean(resourceClaim.isLossTolerant());
-}
-
- 

[jira] [Commented] (NIFI-2854) Enable repositories to support upgrades and rollback in well defined scenarios

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15658581#comment-15658581
 ] 

ASF GitHub Bot commented on NIFI-2854:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1202#discussion_r87657911
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FileSystemSwapManager.java
 ---
@@ -251,353 +262,36 @@ public SwapSummary getSwapSummary(final String 
swapLocation) throws IOException
 final InputStream bufferedIn = new 
BufferedInputStream(fis);
 final DataInputStream in = new 
DataInputStream(bufferedIn)) {
 
-final int swapEncodingVersion = in.readInt();
-if (swapEncodingVersion > SWAP_ENCODING_VERSION) {
-final String errMsg = "Cannot swap FlowFiles in from " + 
swapFile + " because the encoding version is "
-+ swapEncodingVersion + ", which is too new 
(expecting " + SWAP_ENCODING_VERSION + " or less)";
-
-eventReporter.reportEvent(Severity.ERROR, EVENT_CATEGORY, 
errMsg);
-throw new IOException(errMsg);
-}
-
-final int numRecords;
-final long contentSize;
-Long maxRecordId = null;
-try {
-in.readUTF(); // ignore Connection ID
-numRecords = in.readInt();
-contentSize = in.readLong();
-
-if (numRecords == 0) {
-return StandardSwapSummary.EMPTY_SUMMARY;
-}
-
-if (swapEncodingVersion > 7) {
-maxRecordId = in.readLong();
-}
-} catch (final EOFException eof) {
-logger.warn("Found premature End-of-File when reading Swap 
File {}. EOF occurred before any FlowFiles were encountered", swapLocation);
-return StandardSwapSummary.EMPTY_SUMMARY;
-}
-
-final QueueSize queueSize = new QueueSize(numRecords, 
contentSize);
-final SwapContents swapContents = deserializeFlowFiles(in, 
queueSize, maxRecordId, swapEncodingVersion, true, claimManager, swapLocation);
-return swapContents.getSummary();
-}
-}
-
-public static int serializeFlowFiles(final List 
toSwap, final FlowFileQueue queue, final String swapLocation, final 
OutputStream destination) throws IOException {
-if (toSwap == null || toSwap.isEmpty()) {
-return 0;
-}
-
-long contentSize = 0L;
-for (final FlowFileRecord record : toSwap) {
-contentSize += record.getSize();
-}
-
-// persist record to disk via the swap file
-final OutputStream bufferedOut = new 
BufferedOutputStream(destination);
-final DataOutputStream out = new DataOutputStream(bufferedOut);
-try {
-out.writeInt(SWAP_ENCODING_VERSION);
-out.writeUTF(queue.getIdentifier());
-out.writeInt(toSwap.size());
-out.writeLong(contentSize);
-
-// get the max record id and write that out so that we know it 
quickly for restoration
-long maxRecordId = 0L;
-for (final FlowFileRecord flowFile : toSwap) {
-if (flowFile.getId() > maxRecordId) {
-maxRecordId = flowFile.getId();
-}
-}
-
-out.writeLong(maxRecordId);
-
-for (final FlowFileRecord flowFile : toSwap) {
-out.writeLong(flowFile.getId());
-out.writeLong(flowFile.getEntryDate());
-out.writeLong(flowFile.getLineageStartDate());
-out.writeLong(flowFile.getLineageStartIndex());
-out.writeLong(flowFile.getLastQueueDate());
-out.writeLong(flowFile.getQueueDateIndex());
-out.writeLong(flowFile.getSize());
-
-final ContentClaim claim = flowFile.getContentClaim();
-if (claim == null) {
-out.writeBoolean(false);
-} else {
-out.writeBoolean(true);
-final ResourceClaim resourceClaim = 
claim.getResourceClaim();
-out.writeUTF(resourceClaim.getId());
-out.writeUTF(resourceClaim.getContainer());
-out.writeUTF(resourceClaim.getSection());
-out.writeLong(claim.getOffset());
-  

[jira] [Commented] (NIFI-2854) Enable repositories to support upgrades and rollback in well defined scenarios

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15658573#comment-15658573
 ] 

ASF GitHub Bot commented on NIFI-2854:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1202#discussion_r87629214
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/repository/WriteAheadRepositoryRecordSerde.java
 ---
@@ -0,0 +1,517 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.controller.repository;
+
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.nifi.controller.queue.FlowFileQueue;
+import org.apache.nifi.controller.repository.claim.ContentClaim;
+import org.apache.nifi.controller.repository.claim.ResourceClaim;
+import org.apache.nifi.controller.repository.claim.ResourceClaimManager;
+import org.apache.nifi.controller.repository.claim.StandardContentClaim;
+import org.apache.nifi.flowfile.FlowFile;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.wali.SerDe;
+import org.wali.UpdateType;
+
+public class WriteAheadRepositoryRecordSerde extends RepositoryRecordSerde 
implements SerDe {
+private static final Logger logger = 
LoggerFactory.getLogger(WriteAheadRepositoryRecordSerde.class);
+
+private static final int CURRENT_ENCODING_VERSION = 9;
+
+public static final byte ACTION_CREATE = 0;
+public static final byte ACTION_UPDATE = 1;
+public static final byte ACTION_DELETE = 2;
+public static final byte ACTION_SWAPPED_OUT = 3;
+public static final byte ACTION_SWAPPED_IN = 4;
+
+private long recordsRestored = 0L;
+private final ResourceClaimManager claimManager;
+
+public WriteAheadRepositoryRecordSerde(final ResourceClaimManager 
claimManager) {
+this.claimManager = claimManager;
+}
+
+@Override
+public void serializeEdit(final RepositoryRecord previousRecordState, 
final RepositoryRecord record, final DataOutputStream out) throws IOException {
+serializeEdit(previousRecordState, record, out, false);
+}
+
+public void serializeEdit(final RepositoryRecord previousRecordState, 
final RepositoryRecord record, final DataOutputStream out, final boolean 
forceAttributesWritten) throws IOException {
+if (record.isMarkedForAbort()) {
+logger.warn("Repository Record {} is marked to be aborted; it 
will be persisted in the FlowFileRepository as a DELETE record", record);
+out.write(ACTION_DELETE);
+out.writeLong(getRecordIdentifier(record));
+serializeContentClaim(record.getCurrentClaim(), 
record.getCurrentClaimOffset(), out);
+return;
+}
+
+final UpdateType updateType = getUpdateType(record);
+
+if (updateType.equals(UpdateType.DELETE)) {
+out.write(ACTION_DELETE);
+out.writeLong(getRecordIdentifier(record));
+serializeContentClaim(record.getCurrentClaim(), 
record.getCurrentClaimOffset(), out);
+return;
+}
+
+// If there's a Destination Connection, that's the one that we 
want to associated with this record.
+// However, on restart, we will restore the FlowFile and set this 
connection to its "originalConnection".
+// If we then serialize the FlowFile again before it's 
transferred, it's important to allow this to happen,
+// so we use the originalConnection instead
+FlowFileQueue associatedQueue = record.getDestination();
+if (associatedQueue == null) {
+   

[GitHub] nifi pull request #1202: NIFI-2854: Refactor repositories and swap files to ...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1202#discussion_r87677686
  
--- Diff: 
nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/test/java/org/apache/nifi/provenance/TestPersistentProvenanceRepository.java
 ---
@@ -1914,112 +1916,6 @@ public void 
testFailureToCreateWriterDoesNotPreventSubsequentRollover() throws I
 }
 
 
-@Test
-public void testBehaviorOnOutOfMemory() throws IOException, 
InterruptedException {
--- End diff --

Why was this test removed?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2854) Enable repositories to support upgrades and rollback in well defined scenarios

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15658580#comment-15658580
 ] 

ASF GitHub Bot commented on NIFI-2854:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1202#discussion_r87677086
  
--- Diff: 
nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/schema/FieldSerializer.java
 ---
@@ -0,0 +1,27 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.provenance.schema;
+
+import java.io.DataOutputStream;
+import java.io.IOException;
+
+import org.apache.nifi.provenance.ProvenanceEventRecord;
+
+public interface FieldSerializer {
--- End diff --

Intellij tells me this class isn't used, remnant of design iterations?


> Enable repositories to support upgrades and rollback in well defined scenarios
> --
>
> Key: NIFI-2854
> URL: https://issues.apache.org/jira/browse/NIFI-2854
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.1.0
>
>
> The flowfile, swapfile, provenance, and content repositories play a very 
> important roll in NiFi's ability to be safely upgraded and rolled back.  We 
> need to have well documented behaviors, designs, and version adherence so 
> that users can safely rely on these mechanisms.
> Once this is formalized and in place we should update our versioning guidance 
> to reflect this as well.
> The following would be true from NiFi 1.2.0 onward
> * No changes to how the repositories are persisted to disk can be made which 
> will break forward/backward compatibility and specifically this means that 
> things like the way each is serialized to disk cannot change.
> * If changes are made which impact forward or backward compatibility they 
> should be reserved for major releases only and should include a utility to 
> help users with pre-existing data convert from some older format to the newer 
> format.  It may not be feasible to have rollback on major releases.
> * The content repository should not be changed within a major release cycle 
> in any way that will harm forward or backward compatibility.
> * The flow file repository can change in that new fields can be added to 
> existing write ahead log record types but no fields can be removed nor can 
> any new types be added.  Once a field is considered required it must remain 
> required.  Changes may only be made across minor version changes - not 
> incremental.
> * Swap File storage should follow very similar rules to the flow file 
> repository.  Adding a schema to the swap file header may allow some variation 
> there but the variation should only be hints to optimize how they're 
> processed and not change their behavior otherwise. Changes are only permitted 
> during minor version releases.
> * Provenance repository changes are only permitted during minor version 
> releases.  These changes may include adding or removing fields from existing 
> event types.  If a field is considered required it must always be considered 
> required.  If a field is removed then it must not be a required field and 
> there must be a sensible default an older version could use if that value is 
> not found in new data once rolled back.  New event types may be added.  
> Fields or event types not known to older version, if seen after a rollback, 
> will simply be ignored.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #1202: NIFI-2854: Refactor repositories and swap files to ...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1202#discussion_r87661939
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/swap/SimpleSwapDeserializer.java
 ---
@@ -0,0 +1,303 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.controller.swap;
+
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+import org.apache.nifi.controller.queue.FlowFileQueue;
+import org.apache.nifi.controller.queue.QueueSize;
+import org.apache.nifi.controller.repository.FlowFileRecord;
+import org.apache.nifi.controller.repository.IncompleteSwapFileException;
+import org.apache.nifi.controller.repository.StandardFlowFileRecord;
+import org.apache.nifi.controller.repository.SwapContents;
+import org.apache.nifi.controller.repository.SwapSummary;
+import org.apache.nifi.controller.repository.claim.ResourceClaim;
+import org.apache.nifi.controller.repository.claim.ResourceClaimManager;
+import org.apache.nifi.controller.repository.claim.StandardContentClaim;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class SimpleSwapDeserializer implements SwapDeserializer {
+public static final int SWAP_ENCODING_VERSION = 10;
+private static final Logger logger = 
LoggerFactory.getLogger(SimpleSwapDeserializer.class);
+
+@Override
+public SwapSummary getSwapSummary(final DataInputStream in, final 
String swapLocation, final ResourceClaimManager claimManager) throws 
IOException {
+final int swapEncodingVersion = in.readInt();
+if (swapEncodingVersion > SWAP_ENCODING_VERSION) {
--- End diff --

Wouldn't the highest Encoding version this accepts be 9 (where 10 is this 
latest version which utilizes the new format)?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2854) Enable repositories to support upgrades and rollback in well defined scenarios

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15658578#comment-15658578
 ] 

ASF GitHub Bot commented on NIFI-2854:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1202#discussion_r87654483
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FileSystemSwapManager.java
 ---
@@ -210,30 +215,36 @@ public boolean accept(final File dir, final String 
name) {
 // "--.swap". If we 
have two dashes, then we can just check if the queue ID is equal
 // to the id of the queue given and if not we can just move on.
 final String[] splits = swapFile.getName().split("-");
-if (splits.length == 3) {
-final String queueIdentifier = splits[1];
-if 
(!queueIdentifier.equals(flowFileQueue.getIdentifier())) {
-continue;
+if (splits.length > 6) {
--- End diff --

Was this broken before? The ID scheme for queue hasn't changed (still a 
UUID)


> Enable repositories to support upgrades and rollback in well defined scenarios
> --
>
> Key: NIFI-2854
> URL: https://issues.apache.org/jira/browse/NIFI-2854
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.1.0
>
>
> The flowfile, swapfile, provenance, and content repositories play a very 
> important roll in NiFi's ability to be safely upgraded and rolled back.  We 
> need to have well documented behaviors, designs, and version adherence so 
> that users can safely rely on these mechanisms.
> Once this is formalized and in place we should update our versioning guidance 
> to reflect this as well.
> The following would be true from NiFi 1.2.0 onward
> * No changes to how the repositories are persisted to disk can be made which 
> will break forward/backward compatibility and specifically this means that 
> things like the way each is serialized to disk cannot change.
> * If changes are made which impact forward or backward compatibility they 
> should be reserved for major releases only and should include a utility to 
> help users with pre-existing data convert from some older format to the newer 
> format.  It may not be feasible to have rollback on major releases.
> * The content repository should not be changed within a major release cycle 
> in any way that will harm forward or backward compatibility.
> * The flow file repository can change in that new fields can be added to 
> existing write ahead log record types but no fields can be removed nor can 
> any new types be added.  Once a field is considered required it must remain 
> required.  Changes may only be made across minor version changes - not 
> incremental.
> * Swap File storage should follow very similar rules to the flow file 
> repository.  Adding a schema to the swap file header may allow some variation 
> there but the variation should only be hints to optimize how they're 
> processed and not change their behavior otherwise. Changes are only permitted 
> during minor version releases.
> * Provenance repository changes are only permitted during minor version 
> releases.  These changes may include adding or removing fields from existing 
> event types.  If a field is considered required it must always be considered 
> required.  If a field is removed then it must not be a required field and 
> there must be a sensible default an older version could use if that value is 
> not found in new data once rolled back.  New event types may be added.  
> Fields or event types not known to older version, if seen after a rollback, 
> will simply be ignored.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #1202: NIFI-2854: Refactor repositories and swap files to ...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1202#discussion_r87672258
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/swap/TestSimpleSwapSerializerDeserializer.java
 ---
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.controller.swap;
+
+import static org.junit.Assert.assertEquals;
+
+import java.io.DataInputStream;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.controller.queue.FlowFileQueue;
+import org.apache.nifi.controller.repository.FlowFileRecord;
+import org.apache.nifi.controller.repository.SwapContents;
+import org.apache.nifi.controller.repository.claim.ResourceClaimManager;
+import 
org.apache.nifi.controller.repository.claim.StandardResourceClaimManager;
+import org.apache.nifi.stream.io.NullOutputStream;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+public class TestSimpleSwapSerializerDeserializer {
+@Before
+public void setup() {
+TestFlowFile.resetIdGenerator();
+}
+
+@Test
+public void testRoundTripSerializeDeserialize() throws IOException {
+final ResourceClaimManager resourceClaimManager = new 
StandardResourceClaimManager();
+
+final List toSwap = new ArrayList<>(1);
+final Map attrs = new HashMap<>();
+for (int i = 0; i < 1; i++) {
+attrs.put("i", String.valueOf(i));
+final FlowFileRecord ff = new TestFlowFile(attrs, i, 
resourceClaimManager);
+toSwap.add(ff);
+}
+
+final FlowFileQueue flowFileQueue = 
Mockito.mock(FlowFileQueue.class);
+
Mockito.when(flowFileQueue.getIdentifier()).thenReturn("87bb99fe-412c-49f6-a441-d1b0af4e20b4");
+
+final String swapLocation = "target/testRoundTrip-" + 
UUID.randomUUID().toString() + ".swap";
+final File swapFile = new File(swapLocation);
+
+Files.deleteIfExists(swapFile.toPath());
+try {
+final SimpleSwapSerializer serializer = new 
SimpleSwapSerializer();
+try (final FileOutputStream fos = new 
FileOutputStream(swapFile)) {
+serializer.serializeFlowFiles(toSwap, flowFileQueue, 
swapLocation, fos);
+}
+
+final SimpleSwapDeserializer deserializer = new 
SimpleSwapDeserializer();
+final SwapContents swappedIn;
+try (final FileInputStream fis = new FileInputStream(swapFile);
+final DataInputStream dis = new DataInputStream(fis)) {
+swappedIn = deserializer.deserializeFlowFiles(dis, 
swapLocation, flowFileQueue, resourceClaimManager);
+}
+
+assertEquals(toSwap.size(), swappedIn.getFlowFiles().size());
+for (int i = 0; i < toSwap.size(); i++) {
+final FlowFileRecord pre = toSwap.get(i);
+final FlowFileRecord post = 
swappedIn.getFlowFiles().get(i);
+
+assertEquals(pre.getSize(), post.getSize());
+assertEquals(pre.getAttributes(), post.getAttributes());
+assertEquals(pre.getSize(), post.getSize());
+assertEquals(pre.getId(), post.getId());
+assertEquals(pre.getContentClaim(), 
post.getContentClaim());
+assertEquals(pre.getContentClaimOffset(), 
post.getContentClaimOffset());
+assertEquals(pre.getEntryDate(), 

[GitHub] nifi pull request #1202: NIFI-2854: Refactor repositories and swap files to ...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1202#discussion_r87674718
  
--- Diff: 
nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/serialization/CompressableRecordReader.java
 ---
@@ -0,0 +1,281 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.provenance.serialization;
+
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.zip.GZIPInputStream;
+
+import org.apache.nifi.provenance.ProvenanceEventRecord;
+import org.apache.nifi.provenance.StandardProvenanceEventRecord;
+import org.apache.nifi.provenance.StandardRecordReader;
+import org.apache.nifi.provenance.toc.TocReader;
+import org.apache.nifi.stream.io.BufferedInputStream;
+import org.apache.nifi.stream.io.ByteCountingInputStream;
+import org.apache.nifi.stream.io.LimitingInputStream;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public abstract class CompressableRecordReader implements RecordReader {
+private static final Logger logger = 
LoggerFactory.getLogger(StandardRecordReader.class);
+
+private final ByteCountingInputStream rawInputStream;
+private final String filename;
+private final int serializationVersion;
+private final boolean compressed;
+private final TocReader tocReader;
+private final int headerLength;
+private final int maxAttributeChars;
+
+private DataInputStream dis;
+private ByteCountingInputStream byteCountingIn;
+
+public CompressableRecordReader(final InputStream in, final String 
filename, final int maxAttributeChars) throws IOException {
+this(in, filename, null, maxAttributeChars);
+}
+
+public CompressableRecordReader(final InputStream in, final String 
filename, final TocReader tocReader, final int maxAttributeChars) throws 
IOException {
+logger.trace("Creating RecordReader for {}", filename);
+
+rawInputStream = new ByteCountingInputStream(in);
+this.maxAttributeChars = maxAttributeChars;
+
+final InputStream limitedStream;
+if (tocReader == null) {
+limitedStream = rawInputStream;
+} else {
+final long offset1 = tocReader.getBlockOffset(1);
+if (offset1 < 0) {
+limitedStream = rawInputStream;
+} else {
+limitedStream = new LimitingInputStream(rawInputStream, 
offset1 - rawInputStream.getBytesConsumed());
+}
+}
+
+final InputStream readableStream;
+if (filename.endsWith(".gz")) {
+readableStream = new BufferedInputStream(new 
GZIPInputStream(limitedStream));
+compressed = true;
+} else {
+readableStream = new BufferedInputStream(limitedStream);
+compressed = false;
+}
+
+byteCountingIn = new ByteCountingInputStream(readableStream);
+dis = new DataInputStream(byteCountingIn);
+
+final String repoClassName = dis.readUTF();
+final int serializationVersion = dis.readInt();
+headerLength = 
repoClassName.getBytes(StandardCharsets.UTF_8).length + 2 + 4; // 2 bytes for 
string length, 4 for integer.
+
+if (serializationVersion < 1 || serializationVersion > 9) {
+throw new IllegalArgumentException("Unable to deserialize 
record because the version is " + serializationVersion + " and supported 
versions are 1-9");
--- End diff --

Wouldn't this be version 10?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not 

[GitHub] nifi pull request #1202: NIFI-2854: Refactor repositories and swap files to ...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1202#discussion_r87668398
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/repository/TestWriteAheadFlowFileRepository.java
 ---
@@ -64,6 +65,7 @@ public static void setupProperties() {
 }
 
 @Before
+@After
--- End diff --

Why does it need to be cleared before and after?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1202: NIFI-2854: Refactor repositories and swap files to ...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1202#discussion_r87668539
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/swap/TestFlowFile.java
 ---
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.controller.swap;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicLong;
+
+import org.apache.nifi.controller.repository.FlowFileRecord;
+import org.apache.nifi.controller.repository.claim.ContentClaim;
+import org.apache.nifi.controller.repository.claim.ResourceClaim;
+import org.apache.nifi.controller.repository.claim.ResourceClaimManager;
+import org.apache.nifi.controller.repository.claim.StandardContentClaim;
+import org.apache.nifi.flowfile.FlowFile;
+
+public class TestFlowFile implements FlowFileRecord {
--- End diff --

Can this be renamed to MockFlowFile? "TestFlowFile" sounds like it is the 
unit tests for FlowFile


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2854) Enable repositories to support upgrades and rollback in well defined scenarios

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15658555#comment-15658555
 ] 

ASF GitHub Bot commented on NIFI-2854:
--

Github user joshelser commented on the issue:

https://github.com/apache/nifi/pull/1202
  
Have you considered using a library such as Google Protocol Buffers to 
reduce the debt on NiFi in maintaining custom serialization logic?


> Enable repositories to support upgrades and rollback in well defined scenarios
> --
>
> Key: NIFI-2854
> URL: https://issues.apache.org/jira/browse/NIFI-2854
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.1.0
>
>
> The flowfile, swapfile, provenance, and content repositories play a very 
> important roll in NiFi's ability to be safely upgraded and rolled back.  We 
> need to have well documented behaviors, designs, and version adherence so 
> that users can safely rely on these mechanisms.
> Once this is formalized and in place we should update our versioning guidance 
> to reflect this as well.
> The following would be true from NiFi 1.2.0 onward
> * No changes to how the repositories are persisted to disk can be made which 
> will break forward/backward compatibility and specifically this means that 
> things like the way each is serialized to disk cannot change.
> * If changes are made which impact forward or backward compatibility they 
> should be reserved for major releases only and should include a utility to 
> help users with pre-existing data convert from some older format to the newer 
> format.  It may not be feasible to have rollback on major releases.
> * The content repository should not be changed within a major release cycle 
> in any way that will harm forward or backward compatibility.
> * The flow file repository can change in that new fields can be added to 
> existing write ahead log record types but no fields can be removed nor can 
> any new types be added.  Once a field is considered required it must remain 
> required.  Changes may only be made across minor version changes - not 
> incremental.
> * Swap File storage should follow very similar rules to the flow file 
> repository.  Adding a schema to the swap file header may allow some variation 
> there but the variation should only be hints to optimize how they're 
> processed and not change their behavior otherwise. Changes are only permitted 
> during minor version releases.
> * Provenance repository changes are only permitted during minor version 
> releases.  These changes may include adding or removing fields from existing 
> event types.  If a field is considered required it must always be considered 
> required.  If a field is removed then it must not be a required field and 
> there must be a sensible default an older version could use if that value is 
> not found in new data once rolled back.  New event types may be added.  
> Fields or event types not known to older version, if seen after a rollback, 
> will simply be ignored.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #1202: NIFI-2854: Refactor repositories and swap files to use sch...

[joshelser]

Github user joshelser commented on the issue:

https://github.com/apache/nifi/pull/1202
  
Have you considered using a library such as Google Protocol Buffers to 
reduce the debt on NiFi in maintaining custom serialization logic?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3015) NiFi service starts from root user after installation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15658241#comment-15658241
 ] 

ASF GitHub Bot commented on NIFI-3015:
--

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/1198
  
Have you configured bootstrap after install? The runas setting is where 
this logic happens, not within the init script 


> NiFi service starts from root user after installation
> -
>
> Key: NIFI-3015
> URL: https://issues.apache.org/jira/browse/NIFI-3015
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Configuration
>Affects Versions: 1.1.0
> Environment: Centos 7.2
>Reporter: Artem Yermakov
>Priority: Critical
>
> When install NiFi using command nifi.sh install, and then start NiFi by 
> command service nifi start, NiFi will start from user root.
> I suggest to run it from user nifi which is created during rpm installation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #1198: [NIFI-3015] Run NiFi service from nifi user instead of roo...

[trixpan]

Github user trixpan commented on the issue:

https://github.com/apache/nifi/pull/1198
  
Have you configured bootstrap after install? The runas setting is where 
this logic happens, not within the init script 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2380) ExtractEmailAttachments processor should support TNEF files (aka winmail.dat)

[Joseph Witt (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15658175#comment-15658175
 ] 

Joseph Witt commented on NIFI-2380:
---

the changes necessary for licensing/notice and conflicting deps are 
considerable and taking some time.  Hopefully will have it this weekend.

> ExtractEmailAttachments processor should support TNEF files (aka winmail.dat)
> -
>
> Key: NIFI-2380
> URL: https://issues.apache.org/jira/browse/NIFI-2380
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.0.0
>Reporter: Andre
>Assignee: Joseph Witt
> Fix For: 1.1.0
>
>
> during the review of NIFI-1899 Dan Marshall highlighted some use cases for 
> email processing that have not been addressed as part of the initial 
> development cycle.
> One of these use cases was the decoding of Microsoft Transport Neutral 
> Encoding Files (TNEF). 
> This type of attachments is popularly know as winmail.dat and uses a non RFC 
> compliant structure to transfer attachments across different Microsoft 
> Outlook clients.
> Given the prevalence of outlook and the issues with winmail.dat files, it 
> would be nice to be able to decode TNEF as we currently do with MIME 
> attachments.
> Permalink to Dan's comments 
> http://mail-archives.apache.org/mod_mbox/nifi-dev/201607.mbox/%3C1468716836729-12827.post%40n7.nabble.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2851) Improve performance of SplitText

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15658142#comment-15658142
 ] 

ASF GitHub Bot commented on NIFI-2851:
--

Github user markap14 commented on the issue:

https://github.com/apache/nifi/pull/1116
  
@olegz thanks for jumping on this. Sorry it's taken me so long to get back 
to it. I verified the changes are good now. I added an additional unit test to 
verify a corner case that was problematic in the StreamDemarcator and all is 
looking good. +1 merged to master!


> Improve performance of SplitText
> 
>
> Key: NIFI-2851
> URL: https://issues.apache.org/jira/browse/NIFI-2851
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Oleg Zhurakousky
> Fix For: 1.1.0
>
>
> SplitText is fairly CPU-intensive and quite slow. A simple flow that splits a 
> 1.4 million line text file into 5k line chunks and then splits those 5k line 
> chunks into 1 line chunks is only capable of pushing through about 10k lines 
> per second. This equates to about 10 MB/sec. JVisualVM shows that the 
> majority of the time is spent in the locateSplitPoint() method. Isolating 
> this code and inspecting how it works, and using some micro-benchmarking, it 
> appears that if we refactor the calls to InputStream.read() to instead read 
> into a byte array, we can improve performance.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2851) Improve performance of SplitText

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15658137#comment-15658137
 ] 

ASF subversion and git services commented on NIFI-2851:
---

Commit ad924745933c3e018e0d78282d9d085978b8cd2b in nifi's branch 
refs/heads/master from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=ad92474 ]

NIFI-2851: Added additional unit test to ensure correctness of demarcation when 
demarcator falls between buffered data

This closes #1116.


> Improve performance of SplitText
> 
>
> Key: NIFI-2851
> URL: https://issues.apache.org/jira/browse/NIFI-2851
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Oleg Zhurakousky
> Fix For: 1.1.0
>
>
> SplitText is fairly CPU-intensive and quite slow. A simple flow that splits a 
> 1.4 million line text file into 5k line chunks and then splits those 5k line 
> chunks into 1 line chunks is only capable of pushing through about 10k lines 
> per second. This equates to about 10 MB/sec. JVisualVM shows that the 
> majority of the time is spent in the locateSplitPoint() method. Isolating 
> this code and inspecting how it works, and using some micro-benchmarking, it 
> appears that if we refactor the calls to InputStream.read() to instead read 
> into a byte array, we can improve performance.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2851) Improve performance of SplitText

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15658138#comment-15658138
 ] 

ASF GitHub Bot commented on NIFI-2851:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1116


> Improve performance of SplitText
> 
>
> Key: NIFI-2851
> URL: https://issues.apache.org/jira/browse/NIFI-2851
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Oleg Zhurakousky
> Fix For: 1.1.0
>
>
> SplitText is fairly CPU-intensive and quite slow. A simple flow that splits a 
> 1.4 million line text file into 5k line chunks and then splits those 5k line 
> chunks into 1 line chunks is only capable of pushing through about 10k lines 
> per second. This equates to about 10 MB/sec. JVisualVM shows that the 
> majority of the time is spent in the locateSplitPoint() method. Isolating 
> this code and inspecting how it works, and using some micro-benchmarking, it 
> appears that if we refactor the calls to InputStream.read() to instead read 
> into a byte array, we can improve performance.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #1116: NIFI-2851 initial commit of perf improvements on SplitText

[markap14]

Github user markap14 commented on the issue:

https://github.com/apache/nifi/pull/1116
  
@olegz thanks for jumping on this. Sorry it's taken me so long to get back 
to it. I verified the changes are good now. I added an additional unit test to 
verify a corner case that was problematic in the StreamDemarcator and all is 
looking good. +1 merged to master!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2851) Improve performance of SplitText

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15658136#comment-15658136
 ] 

ASF subversion and git services commented on NIFI-2851:
---

Commit 41f519e84cb5d0bf8be4ce93e26a042d16cda273 in nifi's branch 
refs/heads/master from [~ozhurakousky]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=41f519e ]

NIFI-2851 initial commit of perf improvements on SplitText

- introduced org.apache.nifi.stream.io.util.TextLineDemarcator
- refactored SplitText to use org.apache.nifi.stream.io.util.TextLineDemarcator
- updated SplitText's capability discription to provide more clarity around 
splits with headers.


> Improve performance of SplitText
> 
>
> Key: NIFI-2851
> URL: https://issues.apache.org/jira/browse/NIFI-2851
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Oleg Zhurakousky
> Fix For: 1.1.0
>
>
> SplitText is fairly CPU-intensive and quite slow. A simple flow that splits a 
> 1.4 million line text file into 5k line chunks and then splits those 5k line 
> chunks into 1 line chunks is only capable of pushing through about 10k lines 
> per second. This equates to about 10 MB/sec. JVisualVM shows that the 
> majority of the time is spent in the locateSplitPoint() method. Isolating 
> this code and inspecting how it works, and using some micro-benchmarking, it 
> appears that if we refactor the calls to InputStream.read() to instead read 
> into a byte array, we can improve performance.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2851) Improve performance of SplitText

[Mark Payne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Payne updated NIFI-2851:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Improve performance of SplitText
> 
>
> Key: NIFI-2851
> URL: https://issues.apache.org/jira/browse/NIFI-2851
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Oleg Zhurakousky
> Fix For: 1.1.0
>
>
> SplitText is fairly CPU-intensive and quite slow. A simple flow that splits a 
> 1.4 million line text file into 5k line chunks and then splits those 5k line 
> chunks into 1 line chunks is only capable of pushing through about 10k lines 
> per second. This equates to about 10 MB/sec. JVisualVM shows that the 
> majority of the time is spent in the locateSplitPoint() method. Isolating 
> this code and inspecting how it works, and using some micro-benchmarking, it 
> appears that if we refactor the calls to InputStream.read() to instead read 
> into a byte array, we can improve performance.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #1116: NIFI-2851 initial commit of perf improvements on Sp...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1116


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2751) When a processor pulls a batch of FlowFiles, it keeps pulling from the same inbound connection instead of round-robin'ing

[Mark Payne (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15658131#comment-15658131
 ] 

Mark Payne commented on NIFI-2751:
--

Re-opening and setting to a blocker, as I'm seeing an Exception getting thrown 
now from StandardProcessSession on occasion:

{code}
2016-11-11 15:41:18,087 WARN [Timer-Driven Process Thread-10] 
o.a.n.c.t.ContinuallyRunProcessorTask
java.lang.ArithmeticException: / by zero
at 
org.apache.nifi.controller.repository.StandardProcessSession.get(StandardProcessSession.java:1446)
 ~[nifi-framework-core-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at 
org.apache.nifi.processor.util.bin.BinFiles.binFlowFiles(BinFiles.java:266) 
~[nifi-processor-utils-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at 
org.apache.nifi.processor.util.bin.BinFiles.onTrigger(BinFiles.java:178) 
~[nifi-processor-utils-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at 
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1064)
 ~[nifi-framework-core-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at 
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
 [nifi-framework-core-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at 
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
 [nifi-framework-core-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at 
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
 [nifi-framework-core-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
[na:1.8.0_60]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) 
[na:1.8.0_60]
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
 [na:1.8.0_60]
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
 [na:1.8.0_60]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[na:1.8.0_60]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_60]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
{code}

> When a processor pulls a batch of FlowFiles, it keeps pulling from the same 
> inbound connection instead of round-robin'ing
> -
>
> Key: NIFI-2751
> URL: https://issues.apache.org/jira/browse/NIFI-2751
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Pierre Villard
>Priority: Blocker
>  Labels: beginner, easyfix, framework, newbie
> Fix For: 1.1.0
>
>
> When a Processor calls ProcessSession.get(int) or 
> ProcessSession.get(FlowFileFilter), the FlowFiles only come from the first 
> inbound connection, unless that connection is empty or doesn't have enough 
> FlowFiles to complete the get() call. It should instead round-robin between 
> the incoming connections, just as ProcessSession.get() does.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (NIFI-2751) When a processor pulls a batch of FlowFiles, it keeps pulling from the same inbound connection instead of round-robin'ing

[Mark Payne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Payne reopened NIFI-2751:
--

> When a processor pulls a batch of FlowFiles, it keeps pulling from the same 
> inbound connection instead of round-robin'ing
> -
>
> Key: NIFI-2751
> URL: https://issues.apache.org/jira/browse/NIFI-2751
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Pierre Villard
>  Labels: beginner, easyfix, framework, newbie
> Fix For: 1.1.0
>
>
> When a Processor calls ProcessSession.get(int) or 
> ProcessSession.get(FlowFileFilter), the FlowFiles only come from the first 
> inbound connection, unless that connection is empty or doesn't have enough 
> FlowFiles to complete the get() call. It should instead round-robin between 
> the incoming connections, just as ProcessSession.get() does.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2751) When a processor pulls a batch of FlowFiles, it keeps pulling from the same inbound connection instead of round-robin'ing

[Mark Payne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Payne updated NIFI-2751:
-
Priority: Blocker  (was: Major)

> When a processor pulls a batch of FlowFiles, it keeps pulling from the same 
> inbound connection instead of round-robin'ing
> -
>
> Key: NIFI-2751
> URL: https://issues.apache.org/jira/browse/NIFI-2751
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Pierre Villard
>Priority: Blocker
>  Labels: beginner, easyfix, framework, newbie
> Fix For: 1.1.0
>
>
> When a Processor calls ProcessSession.get(int) or 
> ProcessSession.get(FlowFileFilter), the FlowFiles only come from the first 
> inbound connection, unless that connection is empty or doesn't have enough 
> FlowFiles to complete the get() call. It should instead round-robin between 
> the incoming connections, just as ProcessSession.get() does.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-3025) Bump nifi-spark-receiver's jackson version to match Spark 2.0.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15658124#comment-15658124
 ] 

ASF GitHub Bot commented on NIFI-3025:
--

GitHub user randerzander reopened a pull request:

https://github.com/apache/nifi/pull/1207

NIFI-3025: Bump nifi-spark-receiver's jackson version to match Spark 2.0.1


### For all changes:
- [X] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [X] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [X] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [X] Is your initial contribution a single, squashed commit?

### For code changes:
- [X] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?

No code or doc changes, but I ran "mvn -Pcontrib-check clean install" 
successfully anyway.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/randerzander/nifi master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1207.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1207


commit db4d10a95003e01537a666708c2ed23f1cb48068
Author: Randy Gelhausen 
Date:   2016-11-10T23:44:52Z

NiFi-3025: Bump nifi-spark-receiver's jackson version to match Spark 2.0.1




> Bump nifi-spark-receiver's jackson version to match Spark 2.0.1
> ---
>
> Key: NIFI-3025
> URL: https://issues.apache.org/jira/browse/NIFI-3025
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.0.0, 1.1.0
>Reporter: Randy Gelhausen
>Priority: Minor
>
> Apache Spark 2.0.1 uses Jackson 2.6.5.
> When trying to use nifi-spark-receiver with Spark Streaming, including the 
> NiFi artifact conflicts with Spark's included Jackson version.
> Bumping that artifact's Jackson version to 2.6.5 fixes this issue



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-3025) Bump nifi-spark-receiver's jackson version to match Spark 2.0.1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15658123#comment-15658123
 ] 

ASF GitHub Bot commented on NIFI-3025:
--

Github user randerzander closed the pull request at:

https://github.com/apache/nifi/pull/1207


> Bump nifi-spark-receiver's jackson version to match Spark 2.0.1
> ---
>
> Key: NIFI-3025
> URL: https://issues.apache.org/jira/browse/NIFI-3025
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.0.0, 1.1.0
>Reporter: Randy Gelhausen
>Priority: Minor
>
> Apache Spark 2.0.1 uses Jackson 2.6.5.
> When trying to use nifi-spark-receiver with Spark Streaming, including the 
> NiFi artifact conflicts with Spark's included Jackson version.
> Bumping that artifact's Jackson version to 2.6.5 fixes this issue



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #1207: NIFI-3025: Bump nifi-spark-receiver's jackson versi...

[randerzander]

GitHub user randerzander reopened a pull request:

https://github.com/apache/nifi/pull/1207

NIFI-3025: Bump nifi-spark-receiver's jackson version to match Spark 2.0.1


### For all changes:
- [X] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [X] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [X] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [X] Is your initial contribution a single, squashed commit?

### For code changes:
- [X] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?

No code or doc changes, but I ran "mvn -Pcontrib-check clean install" 
successfully anyway.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/randerzander/nifi master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1207.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1207


commit db4d10a95003e01537a666708c2ed23f1cb48068
Author: Randy Gelhausen 
Date:   2016-11-10T23:44:52Z

NiFi-3025: Bump nifi-spark-receiver's jackson version to match Spark 2.0.1




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1207: NIFI-3025: Bump nifi-spark-receiver's jackson versi...

[randerzander]

Github user randerzander closed the pull request at:

https://github.com/apache/nifi/pull/1207


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-2999) In NiFI Cluster, new cluster coordinator election could result in all nodes being dropped.

[Oleg Zhurakousky (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Oleg Zhurakousky updated NIFI-2999:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> In NiFI Cluster, new cluster coordinator election could result in all nodes 
> being dropped.
> --
>
> Key: NIFI-2999
> URL: https://issues.apache.org/jira/browse/NIFI-2999
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Matthew Clarke
>Assignee: Mark Payne
> Fix For: 1.1.0
>
>
> When NiFi elects a new cluster coordinator, the newly elected coordinator 
> starts accepting heartbeats from the cluster nodes. If another cluster 
> coordinator election occurs later which results in the original Node being 
> elected as the cluster coordinator once again, old heartbeat messages exist 
> which will trigger all nodes to be dropped due to their age exceeding the 
> threshold for last received heartbeat.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2999) In NiFI Cluster, new cluster coordinator election could result in all nodes being dropped.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15658061#comment-15658061
 ] 

ASF GitHub Bot commented on NIFI-2999:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1210


> In NiFI Cluster, new cluster coordinator election could result in all nodes 
> being dropped.
> --
>
> Key: NIFI-2999
> URL: https://issues.apache.org/jira/browse/NIFI-2999
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Matthew Clarke
>Assignee: Mark Payne
> Fix For: 1.1.0
>
>
> When NiFi elects a new cluster coordinator, the newly elected coordinator 
> starts accepting heartbeats from the cluster nodes. If another cluster 
> coordinator election occurs later which results in the original Node being 
> elected as the cluster coordinator once again, old heartbeat messages exist 
> which will trigger all nodes to be dropped due to their age exceeding the 
> threshold for last received heartbeat.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2999) In NiFI Cluster, new cluster coordinator election could result in all nodes being dropped.

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15658060#comment-15658060
 ] 

ASF subversion and git services commented on NIFI-2999:
---

Commit b73ba7f8d4f6319881c26b8faad121ceb12041ab in nifi's branch 
refs/heads/master from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=b73ba7f ]

NIFI-2999: When Cluster Coordinator changes, purge any old heartbeats so that 
we don't disconnect a node due to very old heartbeats

This closes #1210


> In NiFI Cluster, new cluster coordinator election could result in all nodes 
> being dropped.
> --
>
> Key: NIFI-2999
> URL: https://issues.apache.org/jira/browse/NIFI-2999
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Matthew Clarke
>Assignee: Mark Payne
> Fix For: 1.1.0
>
>
> When NiFi elects a new cluster coordinator, the newly elected coordinator 
> starts accepting heartbeats from the cluster nodes. If another cluster 
> coordinator election occurs later which results in the original Node being 
> elected as the cluster coordinator once again, old heartbeat messages exist 
> which will trigger all nodes to be dropped due to their age exceeding the 
> threshold for last received heartbeat.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #1210: NIFI-2999: When Cluster Coordinator changes, purge ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1210


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-2818) NIFI requires write access to NIFI_HOME/lib upon start

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt updated NIFI-2818:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> NIFI requires write access to NIFI_HOME/lib upon start
> --
>
> Key: NIFI-2818
> URL: https://issues.apache.org/jira/browse/NIFI-2818
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0, 0.7.0
>Reporter: Andre
>Assignee: Joseph Witt
> Fix For: 1.1.0
>
>
> As part of NIFI-1500 we noted that NiFi requires what can be described as 
> excessive filesystem privileges to be executed.
> One of the issues identified is that NiFi requires write access to 
> NIFI_HOME/lib as illustrated by the following:
> {code}
> nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-nar-utils/src/main/java/org/apache/nifi/nar/NarUnpacker.java
>  for (Path narLibraryDir : narLibraryDirs) {
> File narDir = narLibraryDir.toFile();
> FileUtils.ensureDirectoryExistAndCanAccess(narDir);
> File[] dirFiles = narDir.listFiles(NAR_FILTER);
> if (dirFiles != null) {
> List fileList = Arrays.asList(dirFiles);
> narFiles.addAll(fileList);
> }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2818) NIFI requires write access to NIFI_HOME/lib upon start

[Joseph Witt (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15658008#comment-15658008
 ] 

Joseph Witt commented on NIFI-2818:
---

ok am  +1.  I made a small change to the existsRead method to reflect exactly 
what the logic of existsReadWrite had.  This addressed a gap in the case where 
if the directory did not exist.  Tested startup with lib having only read and 
it now works just fine.  Merged to master.  thanks!

> NIFI requires write access to NIFI_HOME/lib upon start
> --
>
> Key: NIFI-2818
> URL: https://issues.apache.org/jira/browse/NIFI-2818
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0, 0.7.0
>Reporter: Andre
>Assignee: Joseph Witt
> Fix For: 1.1.0
>
>
> As part of NIFI-1500 we noted that NiFi requires what can be described as 
> excessive filesystem privileges to be executed.
> One of the issues identified is that NiFi requires write access to 
> NIFI_HOME/lib as illustrated by the following:
> {code}
> nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-nar-utils/src/main/java/org/apache/nifi/nar/NarUnpacker.java
>  for (Path narLibraryDir : narLibraryDirs) {
> File narDir = narLibraryDir.toFile();
> FileUtils.ensureDirectoryExistAndCanAccess(narDir);
> File[] dirFiles = narDir.listFiles(NAR_FILTER);
> if (dirFiles != null) {
> List fileList = Arrays.asList(dirFiles);
> narFiles.addAll(fileList);
> }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2818) NIFI requires write access to NIFI_HOME/lib upon start

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15658004#comment-15658004
 ] 

ASF GitHub Bot commented on NIFI-2818:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1059


> NIFI requires write access to NIFI_HOME/lib upon start
> --
>
> Key: NIFI-2818
> URL: https://issues.apache.org/jira/browse/NIFI-2818
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0, 0.7.0
>Reporter: Andre
>Assignee: Joseph Witt
> Fix For: 1.1.0
>
>
> As part of NIFI-1500 we noted that NiFi requires what can be described as 
> excessive filesystem privileges to be executed.
> One of the issues identified is that NiFi requires write access to 
> NIFI_HOME/lib as illustrated by the following:
> {code}
> nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-nar-utils/src/main/java/org/apache/nifi/nar/NarUnpacker.java
>  for (Path narLibraryDir : narLibraryDirs) {
> File narDir = narLibraryDir.toFile();
> FileUtils.ensureDirectoryExistAndCanAccess(narDir);
> File[] dirFiles = narDir.listFiles(NAR_FILTER);
> if (dirFiles != null) {
> List fileList = Arrays.asList(dirFiles);
> narFiles.addAll(fileList);
> }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2818) NIFI requires write access to NIFI_HOME/lib upon start

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15658002#comment-15658002
 ] 

ASF subversion and git services commented on NIFI-2818:
---

Commit f32bdf7be9d4eaa62d5fdce375e8e3fa2bfa3557 in nifi's branch 
refs/heads/master from Andre F de Miranda
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=f32bdf7 ]

NIFI-2818 - Minimise fs permission required by  NiFi

1 - Replace a r/w test over $NIFI_HOME/lib that is performed
but never utilised by a RO test
2 - Rename ensureDirectoryExistAndCanAccess to
ensureDirectoryExistAndCanReadAndWrite and deprecate the
former


> NIFI requires write access to NIFI_HOME/lib upon start
> --
>
> Key: NIFI-2818
> URL: https://issues.apache.org/jira/browse/NIFI-2818
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0, 0.7.0
>Reporter: Andre
>Assignee: Joseph Witt
> Fix For: 1.1.0
>
>
> As part of NIFI-1500 we noted that NiFi requires what can be described as 
> excessive filesystem privileges to be executed.
> One of the issues identified is that NiFi requires write access to 
> NIFI_HOME/lib as illustrated by the following:
> {code}
> nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-nar-utils/src/main/java/org/apache/nifi/nar/NarUnpacker.java
>  for (Path narLibraryDir : narLibraryDirs) {
> File narDir = narLibraryDir.toFile();
> FileUtils.ensureDirectoryExistAndCanAccess(narDir);
> File[] dirFiles = narDir.listFiles(NAR_FILTER);
> if (dirFiles != null) {
> List fileList = Arrays.asList(dirFiles);
> narFiles.addAll(fileList);
> }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #1059: NIFI-2818 - Minimise fs permission required by NiFi

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1059


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-3028) Update styling for the component details in the the Cluster Summary window

[Rob Moran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rob Moran updated NIFI-3028:

Attachment: selected-component-info.png

> Update styling for the component details in the the Cluster  
> Summary window
> --
>
> Key: NIFI-3028
> URL: https://issues.apache.org/jira/browse/NIFI-3028
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core UI
>Affects Versions: 1.0.0
>Reporter: Rob Moran
>Priority: Minor
> Attachments: selected-component-info.png
>
>
> This improvement was originally suggested in NIFI-2844.
> The improvement is to style the component details as they appear in the 
> Operate palette. See the attached image *selected-component-info.png*



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2818) NIFI requires write access to NIFI_HOME/lib upon start

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15658003#comment-15658003
 ] 

ASF subversion and git services commented on NIFI-2818:
---

Commit b9ef0fb847cd2678a62964698f17a5afb37684b9 in nifi's branch 
refs/heads/master from [~joewitt]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=b9ef0fb ]

NIFI-2818 This closes #1059. aligned read method with read/write method


> NIFI requires write access to NIFI_HOME/lib upon start
> --
>
> Key: NIFI-2818
> URL: https://issues.apache.org/jira/browse/NIFI-2818
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0, 0.7.0
>Reporter: Andre
>Assignee: Joseph Witt
> Fix For: 1.1.0
>
>
> As part of NIFI-1500 we noted that NiFi requires what can be described as 
> excessive filesystem privileges to be executed.
> One of the issues identified is that NiFi requires write access to 
> NIFI_HOME/lib as illustrated by the following:
> {code}
> nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-nar-utils/src/main/java/org/apache/nifi/nar/NarUnpacker.java
>  for (Path narLibraryDir : narLibraryDirs) {
> File narDir = narLibraryDir.toFile();
> FileUtils.ensureDirectoryExistAndCanAccess(narDir);
> File[] dirFiles = narDir.listFiles(NAR_FILTER);
> if (dirFiles != null) {
> List fileList = Arrays.asList(dirFiles);
> narFiles.addAll(fileList);
> }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (NIFI-3028) Update styling for the component details in the the Cluster Summary window

[Rob Moran (JIRA)]

Rob Moran created NIFI-3028:
---

 Summary: Update styling for the component details in the the 
Cluster  Summary window
 Key: NIFI-3028
 URL: https://issues.apache.org/jira/browse/NIFI-3028
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Core UI
Affects Versions: 1.0.0
Reporter: Rob Moran
Priority: Minor


This improvement was originally suggested in NIFi-2844.

The improvement is to style the component details as they appear in the Operate 
palette. See the attached image *selected-component-info.png*





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-3028) Update styling for the component details in the the Cluster Summary window

[Rob Moran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rob Moran updated NIFI-3028:

Description: 
This improvement was originally suggested in NIFI-2844.

The improvement is to style the component details as they appear in the Operate 
palette. See the attached image *selected-component-info.png*



  was:
This improvement was originally suggested in NIFi-2844.

The improvement is to style the component details as they appear in the Operate 
palette. See the attached image *selected-component-info.png*




> Update styling for the component details in the the Cluster  
> Summary window
> --
>
> Key: NIFI-3028
> URL: https://issues.apache.org/jira/browse/NIFI-3028
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core UI
>Affects Versions: 1.0.0
>Reporter: Rob Moran
>Priority: Minor
>
> This improvement was originally suggested in NIFI-2844.
> The improvement is to style the component details as they appear in the 
> Operate palette. See the attached image *selected-component-info.png*



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2844) The icon for the component is cut-off in the Cluster Summary window.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15657990#comment-15657990
 ] 

ASF GitHub Bot commented on NIFI-2844:
--

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/1104
  
Reviewing...


> The icon for the component is cut-off in the Cluster  Summary 
> window.
> 
>
> Key: NIFI-2844
> URL: https://issues.apache.org/jira/browse/NIFI-2844
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core UI
>Affects Versions: 1.0.0
>Reporter: Andrew Lim
>Assignee: Scott Aslan
>Priority: Minor
> Attachments: NIFI-2844_connection.png, NIFI-2844_inputPort.png, 
> NIFI-2844_outputPort.png, NIFI-2844_processor.png, selected-component-info.png
>
>
> In a clustered environment, select "Summary" from the Global Menu.
> As an example, in the Processors tab, select "View processor details" icon 
> (the 3 cube cluster icon) for one of the processors.  The icon shown for the 
> processor is cut-off at the top.
> This also occurs for the following components:
> -Input Ports
> -Output Ports
> -Connections
> Screenshots attached.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #1104: [NIFI-2844] Update CSS styles for Cluster Summary Dialog i...

[mcgilman]

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/1104
  
Reviewing...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-2898) Add Processor dialog cuts off processor descriptions

[Matt Gilman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Gilman updated NIFI-2898:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Add Processor dialog cuts off processor descriptions
> 
>
> Key: NIFI-2898
> URL: https://issues.apache.org/jira/browse/NIFI-2898
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core UI
>Affects Versions: 1.0.0
>Reporter: Jeff Storck
>Assignee: Scott Aslan
>Priority: Minor
> Fix For: 1.1.0
>
> Attachments: processor-description-scrolling.png
>
>
> In the Add Processor dialog, descriptions for processors like 
> ConvertJSONToSQL get cut off instead of showing a ellipses.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2898) Add Processor dialog cuts off processor descriptions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15657983#comment-15657983
 ] 

ASF GitHub Bot commented on NIFI-2898:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1191


> Add Processor dialog cuts off processor descriptions
> 
>
> Key: NIFI-2898
> URL: https://issues.apache.org/jira/browse/NIFI-2898
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core UI
>Affects Versions: 1.0.0
>Reporter: Jeff Storck
>Assignee: Scott Aslan
>Priority: Minor
> Fix For: 1.1.0
>
> Attachments: processor-description-scrolling.png
>
>
> In the Add Processor dialog, descriptions for processors like 
> ConvertJSONToSQL get cut off instead of showing a ellipses.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2898) Add Processor dialog cuts off processor descriptions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15657984#comment-15657984
 ] 

ASF GitHub Bot commented on NIFI-2898:
--

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/1191
  
Looks good @scottyaslan! This has been merged to master.


> Add Processor dialog cuts off processor descriptions
> 
>
> Key: NIFI-2898
> URL: https://issues.apache.org/jira/browse/NIFI-2898
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core UI
>Affects Versions: 1.0.0
>Reporter: Jeff Storck
>Assignee: Scott Aslan
>Priority: Minor
> Fix For: 1.1.0
>
> Attachments: processor-description-scrolling.png
>
>
> In the Add Processor dialog, descriptions for processors like 
> ConvertJSONToSQL get cut off instead of showing a ellipses.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #1191: [NIFI-2898] restore ellipsis for processor type, controlle...

[mcgilman]

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/1191
  
Looks good @scottyaslan! This has been merged to master.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2898) Add Processor dialog cuts off processor descriptions

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15657982#comment-15657982
 ] 

ASF subversion and git services commented on NIFI-2898:
---

Commit 52cde9ad1397e7785fbe12379cf2b50c3b275722 in nifi's branch 
refs/heads/master from [~scottyaslan]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=52cde9a ]

[NIFI-2898] restore ellipsis for processor type, controller servies type, and 
reporting task type descriptions. This closes #1191


> Add Processor dialog cuts off processor descriptions
> 
>
> Key: NIFI-2898
> URL: https://issues.apache.org/jira/browse/NIFI-2898
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core UI
>Affects Versions: 1.0.0
>Reporter: Jeff Storck
>Assignee: Scott Aslan
>Priority: Minor
> Fix For: 1.1.0
>
> Attachments: processor-description-scrolling.png
>
>
> In the Add Processor dialog, descriptions for processors like 
> ConvertJSONToSQL get cut off instead of showing a ellipses.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #1191: [NIFI-2898] restore ellipsis for processor type, co...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1191


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (NIFI-3027) ExecuteProcess supports the expression language

[Mark Payne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Payne resolved NIFI-3027.
--
Resolution: Fixed

> ExecuteProcess supports the expression language
> ---
>
> Key: NIFI-3027
> URL: https://issues.apache.org/jira/browse/NIFI-3027
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.1.0
>Reporter: Byunghwa Yun
>Priority: Minor
> Fix For: 1.1.0
>
>
> When ExecuteProcess executed with specified time argument, I needed the 
> expression language like below.
> e.g.)
> Command: /home/hadoop/hive/bin/hive
> Command Arguments: -e;ALTER TABLE TAB1 ADD IF NOT EXISTS PARTITION 
> (LOG_DATE='${now():toNumber():plus(360):format('MMdd')}', 
> LOG_HOUR='${now():toNumber():plus(360):format('HH')}')
> Argument Delimiter: ;
> Thank you.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-3027) ExecuteProcess supports the expression language

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15657974#comment-15657974
 ] 

ASF GitHub Bot commented on NIFI-3027:
--

Github user markap14 commented on the issue:

https://github.com/apache/nifi/pull/1209
  
@combineads thanks for the contribution! All looks good. I have merged to 
master.


> ExecuteProcess supports the expression language
> ---
>
> Key: NIFI-3027
> URL: https://issues.apache.org/jira/browse/NIFI-3027
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.1.0
>Reporter: Byunghwa Yun
>Priority: Minor
> Fix For: 1.1.0
>
>
> When ExecuteProcess executed with specified time argument, I needed the 
> expression language like below.
> e.g.)
> Command: /home/hadoop/hive/bin/hive
> Command Arguments: -e;ALTER TABLE TAB1 ADD IF NOT EXISTS PARTITION 
> (LOG_DATE='${now():toNumber():plus(360):format('MMdd')}', 
> LOG_HOUR='${now():toNumber():plus(360):format('HH')}')
> Argument Delimiter: ;
> Thank you.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-3027) ExecuteProcess supports the expression language

[Mark Payne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Payne updated NIFI-3027:
-
Fix Version/s: 1.1.0

> ExecuteProcess supports the expression language
> ---
>
> Key: NIFI-3027
> URL: https://issues.apache.org/jira/browse/NIFI-3027
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.1.0
>Reporter: Byunghwa Yun
>Priority: Minor
> Fix For: 1.1.0
>
>
> When ExecuteProcess executed with specified time argument, I needed the 
> expression language like below.
> e.g.)
> Command: /home/hadoop/hive/bin/hive
> Command Arguments: -e;ALTER TABLE TAB1 ADD IF NOT EXISTS PARTITION 
> (LOG_DATE='${now():toNumber():plus(360):format('MMdd')}', 
> LOG_HOUR='${now():toNumber():plus(360):format('HH')}')
> Argument Delimiter: ;
> Thank you.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-3027) ExecuteProcess supports the expression language

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15657975#comment-15657975
 ] 

ASF GitHub Bot commented on NIFI-3027:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1209


> ExecuteProcess supports the expression language
> ---
>
> Key: NIFI-3027
> URL: https://issues.apache.org/jira/browse/NIFI-3027
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.1.0
>Reporter: Byunghwa Yun
>Priority: Minor
> Fix For: 1.1.0
>
>
> When ExecuteProcess executed with specified time argument, I needed the 
> expression language like below.
> e.g.)
> Command: /home/hadoop/hive/bin/hive
> Command Arguments: -e;ALTER TABLE TAB1 ADD IF NOT EXISTS PARTITION 
> (LOG_DATE='${now():toNumber():plus(360):format('MMdd')}', 
> LOG_HOUR='${now():toNumber():plus(360):format('HH')}')
> Argument Delimiter: ;
> Thank you.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-3027) ExecuteProcess supports the expression language

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15657969#comment-15657969
 ] 

ASF subversion and git services commented on NIFI-3027:
---

Commit 91f01704165162415f5dec5c20dbb07ed03f8bf8 in nifi's branch 
refs/heads/master from [~combine]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=91f0170 ]

NIFI-3027: ExecuteProcess supports the expression language


> ExecuteProcess supports the expression language
> ---
>
> Key: NIFI-3027
> URL: https://issues.apache.org/jira/browse/NIFI-3027
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.1.0
>Reporter: Byunghwa Yun
>Priority: Minor
>
> When ExecuteProcess executed with specified time argument, I needed the 
> expression language like below.
> e.g.)
> Command: /home/hadoop/hive/bin/hive
> Command Arguments: -e;ALTER TABLE TAB1 ADD IF NOT EXISTS PARTITION 
> (LOG_DATE='${now():toNumber():plus(360):format('MMdd')}', 
> LOG_HOUR='${now():toNumber():plus(360):format('HH')}')
> Argument Delimiter: ;
> Thank you.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2844) The icon for the component is cut-off in the Cluster Summary window.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15657959#comment-15657959
 ] 

ASF GitHub Bot commented on NIFI-2844:
--

Github user moranr commented on the issue:

https://github.com/apache/nifi/pull/1104
  
@scottyaslan let's go ahead and get this in since the issue is fixed and 
the icons are now visible. I'll file a separate jira to completely update the 
styling.


> The icon for the component is cut-off in the Cluster  Summary 
> window.
> 
>
> Key: NIFI-2844
> URL: https://issues.apache.org/jira/browse/NIFI-2844
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core UI
>Affects Versions: 1.0.0
>Reporter: Andrew Lim
>Assignee: Scott Aslan
>Priority: Minor
> Attachments: NIFI-2844_connection.png, NIFI-2844_inputPort.png, 
> NIFI-2844_outputPort.png, NIFI-2844_processor.png, selected-component-info.png
>
>
> In a clustered environment, select "Summary" from the Global Menu.
> As an example, in the Processors tab, select "View processor details" icon 
> (the 3 cube cluster icon) for one of the processors.  The icon shown for the 
> processor is cut-off at the top.
> This also occurs for the following components:
> -Input Ports
> -Output Ports
> -Connections
> Screenshots attached.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #1104: [NIFI-2844] Update CSS styles for Cluster Summary Dialog i...

[moranr]

Github user moranr commented on the issue:

https://github.com/apache/nifi/pull/1104
  
@scottyaslan let's go ahead and get this in since the issue is fixed and 
the icons are now visible. I'll file a separate jira to completely update the 
styling.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (NIFI-2957) ZooKeeper migration toolkit

[Bryan Rosander (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Rosander resolved NIFI-2957.
--
Resolution: Fixed

> ZooKeeper migration toolkit
> ---
>
> Key: NIFI-2957
> URL: https://issues.apache.org/jira/browse/NIFI-2957
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Tools and Build
>Affects Versions: 1.0.0, 0.7.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
> Fix For: 1.2.0
>
>
> When upgrading from NiFi 0.x to 1.x, or when it is desired to move from the 
> embedded ZooKeeper to an external ZooKeeper, state from ZooKeeper needs to be 
> migrated.
> Initial considerations:
> * Username/password protection of nodes is not supported in NiFi 1.x.. Those 
> nodes that are configured that way in ZooKeeper need to be migrated to have 
> an open ACL.
> * The toolkit will support a mode to read data from a configurable root node 
> in a source ZooKeeper, and the data will be written to a file designated via 
> CLI.
> * The toolkit will support a mode to write data to a destination ZooKeeper
> * The toolkit will not allow data to be written to the same ZooKeeper from 
> which the source data was obtained.
> * The toolkit will not support reconnecting to ZooKeeper if it is 
> disconnected.  The user can rerun the tool.
> * The toolkit will support ZooKeepers configured with Kerberos.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2957) ZooKeeper migration toolkit

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15657907#comment-15657907
 ] 

ASF GitHub Bot commented on NIFI-2957:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1193


> ZooKeeper migration toolkit
> ---
>
> Key: NIFI-2957
> URL: https://issues.apache.org/jira/browse/NIFI-2957
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Tools and Build
>Affects Versions: 1.0.0, 0.7.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
> Fix For: 1.2.0
>
>
> When upgrading from NiFi 0.x to 1.x, or when it is desired to move from the 
> embedded ZooKeeper to an external ZooKeeper, state from ZooKeeper needs to be 
> migrated.
> Initial considerations:
> * Username/password protection of nodes is not supported in NiFi 1.x.. Those 
> nodes that are configured that way in ZooKeeper need to be migrated to have 
> an open ACL.
> * The toolkit will support a mode to read data from a configurable root node 
> in a source ZooKeeper, and the data will be written to a file designated via 
> CLI.
> * The toolkit will support a mode to write data to a destination ZooKeeper
> * The toolkit will not allow data to be written to the same ZooKeeper from 
> which the source data was obtained.
> * The toolkit will not support reconnecting to ZooKeeper if it is 
> disconnected.  The user can rerun the tool.
> * The toolkit will support ZooKeepers configured with Kerberos.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #1193: NIFI-2957 ZooKeeper Migration Toolkit

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1193


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2957) ZooKeeper migration toolkit

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15657902#comment-15657902
 ] 

ASF subversion and git services commented on NIFI-2957:
---

Commit ed6e03399f63d53aafdc2faadfa0878ff0b42188 in nifi's branch 
refs/heads/master from [~jtstorck]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=ed6e033 ]

NIFI-2957 ZooKeeper Migration Toolkit

Reads from and writes to a Zookeeper that is open or secured via user/password 
digest or SASL
Supports persisting of data obtained from Zookeeper to a file or standard out 
as JSON
Supports sending of Zookeeper data to Zookeeper from a file or standard out as 
JSON
Does not allow data obtained from a source Zookeeper to be written back to the 
same Zookeeper connect string and path

This closes #1193

Signed-off-by: Bryan Rosander 


> ZooKeeper migration toolkit
> ---
>
> Key: NIFI-2957
> URL: https://issues.apache.org/jira/browse/NIFI-2957
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Tools and Build
>Affects Versions: 1.0.0, 0.7.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
> Fix For: 1.2.0
>
>
> When upgrading from NiFi 0.x to 1.x, or when it is desired to move from the 
> embedded ZooKeeper to an external ZooKeeper, state from ZooKeeper needs to be 
> migrated.
> Initial considerations:
> * Username/password protection of nodes is not supported in NiFi 1.x.. Those 
> nodes that are configured that way in ZooKeeper need to be migrated to have 
> an open ACL.
> * The toolkit will support a mode to read data from a configurable root node 
> in a source ZooKeeper, and the data will be written to a file designated via 
> CLI.
> * The toolkit will support a mode to write data to a destination ZooKeeper
> * The toolkit will not allow data to be written to the same ZooKeeper from 
> which the source data was obtained.
> * The toolkit will not support reconnecting to ZooKeeper if it is 
> disconnected.  The user can rerun the tool.
> * The toolkit will support ZooKeepers configured with Kerberos.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi-minifi pull request #54: MINIFI-135 Removing 'Input Ports' and 'Output ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi-minifi/pull/54


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi pull request #53: MINIFI-134 Removing test scope in the root pom...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi-minifi/pull/53


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-2818) NIFI requires write access to NIFI_HOME/lib upon start

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt updated NIFI-2818:
--
Status: Patch Available  (was: Open)

> NIFI requires write access to NIFI_HOME/lib upon start
> --
>
> Key: NIFI-2818
> URL: https://issues.apache.org/jira/browse/NIFI-2818
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 0.7.0, 1.0.0
>Reporter: Andre
>Assignee: Joseph Witt
> Fix For: 1.1.0
>
>
> As part of NIFI-1500 we noted that NiFi requires what can be described as 
> excessive filesystem privileges to be executed.
> One of the issues identified is that NiFi requires write access to 
> NIFI_HOME/lib as illustrated by the following:
> {code}
> nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-nar-utils/src/main/java/org/apache/nifi/nar/NarUnpacker.java
>  for (Path narLibraryDir : narLibraryDirs) {
> File narDir = narLibraryDir.toFile();
> FileUtils.ensureDirectoryExistAndCanAccess(narDir);
> File[] dirFiles = narDir.listFiles(NAR_FILTER);
> if (dirFiles != null) {
> List fileList = Arrays.asList(dirFiles);
> narFiles.addAll(fileList);
> }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2996) Processor/Service validation takes exponentially longer to run as the graph grows

[Joseph Witt (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15657839#comment-15657839
 ] 

Joseph Witt commented on NIFI-2996:
---

[~mosermw][~mcgilman] is unclear what the resolution here is but want to flag 
this to get the fix version removed or understand when it will be resolved.  
trying to close down the 1.1.0 release

> Processor/Service validation takes exponentially longer to run as the graph 
> grows
> -
>
> Key: NIFI-2996
> URL: https://issues.apache.org/jira/browse/NIFI-2996
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0, 0.7.1
>Reporter: Michael Moser
>Assignee: Michael Moser
> Fix For: 1.1.0
>
>
> As you add processors that reference controller services to your NiFi, the 
> validation that occurs during normal UI usage increases dramatically.
> When running in a cluster, I have to increase the nifi.properties 
> nifi.cluster.node.read.timeout well beyond its 5 sec default timeout in order 
> for the UI to work.  Eventually, simple operations in the UI take close to a 
> minute to happen.
> As a test, I created an SSLContextService using certs created with the 
> amazingly useful nifi-toolkit.  I created a DistributedMapCacheClientService 
> that references this SSLContextService.  Then I created 108 
> FetchDistributedMapCache processors that reference the 
> DistributedMapCacheClient service. I used ${hostname(true)} for my 
> FetchDistributedMapCache processor's Cache Entry Identifier property.
> When NiFi is up with no UI connected, during a single "NiFi status" phase I 
> noticed that the SSLContextService was validated 108 times and the EL 
> hostname(true) was evaluated 216 times.  When I connect the UI and go through 
> a normal status refresh cycle, the SSLContextService was validated 432 times 
> and the EL hostname(true) was evaluated 864 times. These validations take a 
> full second on my slowest machine.
> In NiFi 0.x, the expensive REST API call is 
> /nifi-api/controller/controller-services/node.
> In NiFi 1.x, it appears to divide the work among REST API calls to 
> /nifi-api/process-groups/UUID and 
> /nifi-api/flow/process-groups/UUID/controller-services and 
> /nifi-api/flow/status
> Rather than attempting to fix StandardSSLContextService or the EL 
> HostnameEvaluator, I wonder if there was a more generic approach to helping 
> resolve this?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-3015) NiFi service starts from root user after installation

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt updated NIFI-3015:
--
Status: Patch Available  (was: Open)

removed fix version and flagged as patch submitted so it can get review 
traction.

once review/commit traction occurs we can put the fix version.  For the 1.1.0 
release in particular we're trying to close down to get the release out so this 
will likely need to happen in the next release.

> NiFi service starts from root user after installation
> -
>
> Key: NIFI-3015
> URL: https://issues.apache.org/jira/browse/NIFI-3015
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Configuration
>Affects Versions: 1.1.0
> Environment: Centos 7.2
>Reporter: Artem Yermakov
>Priority: Critical
>
> When install NiFi using command nifi.sh install, and then start NiFi by 
> command service nifi start, NiFi will start from user root.
> I suggest to run it from user nifi which is created during rpm installation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-3015) NiFi service starts from root user after installation

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt updated NIFI-3015:
--
Fix Version/s: (was: 1.1.0)

> NiFi service starts from root user after installation
> -
>
> Key: NIFI-3015
> URL: https://issues.apache.org/jira/browse/NIFI-3015
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Configuration
>Affects Versions: 1.1.0
> Environment: Centos 7.2
>Reporter: Artem Yermakov
>Priority: Critical
>
> When install NiFi using command nifi.sh install, and then start NiFi by 
> command service nifi start, NiFi will start from user root.
> I suggest to run it from user nifi which is created during rpm installation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2957) ZooKeeper migration toolkit

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15657785#comment-15657785
 ] 

ASF GitHub Bot commented on NIFI-2957:
--

Github user jtstorck commented on the issue:

https://github.com/apache/nifi/pull/1193
  
Also removed tests that specify a port for Zookeeper in favor of tests that 
check the connect string and path parsing from the command line.


> ZooKeeper migration toolkit
> ---
>
> Key: NIFI-2957
> URL: https://issues.apache.org/jira/browse/NIFI-2957
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Tools and Build
>Affects Versions: 1.0.0, 0.7.1
>Reporter: Jeff Storck
>Assignee: Jeff Storck
> Fix For: 1.2.0
>
>
> When upgrading from NiFi 0.x to 1.x, or when it is desired to move from the 
> embedded ZooKeeper to an external ZooKeeper, state from ZooKeeper needs to be 
> migrated.
> Initial considerations:
> * Username/password protection of nodes is not supported in NiFi 1.x.. Those 
> nodes that are configured that way in ZooKeeper need to be migrated to have 
> an open ACL.
> * The toolkit will support a mode to read data from a configurable root node 
> in a source ZooKeeper, and the data will be written to a file designated via 
> CLI.
> * The toolkit will support a mode to write data to a destination ZooKeeper
> * The toolkit will not allow data to be written to the same ZooKeeper from 
> which the source data was obtained.
> * The toolkit will not support reconnecting to ZooKeeper if it is 
> disconnected.  The user can rerun the tool.
> * The toolkit will support ZooKeepers configured with Kerberos.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #1193: NIFI-2957 ZooKeeper Migration Toolkit

[jtstorck]

Github user jtstorck commented on the issue:

https://github.com/apache/nifi/pull/1193
  
Also removed tests that specify a port for Zookeeper in favor of tests that 
check the connect string and path parsing from the command line.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-2912) Processor to generate a flow file from a string

[Andy LoPresto (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andy LoPresto updated NIFI-2912:

   Resolution: Fixed
Fix Version/s: (was: 1.2.0)
   1.1.0
   Status: Resolved  (was: Patch Available)

> Processor to generate a flow file from a string
> ---
>
> Key: NIFI-2912
> URL: https://issues.apache.org/jira/browse/NIFI-2912
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Alessio Palma
>Assignee: Pierre Villard
>Priority: Minor
> Fix For: 1.1.0
>
>
> Currently to build a flow file from a string we need ReplaceText and 
> GenerateFlowFile with a size of 0B. 
> Is here a way to have use only the GenerateFlowFile processor ? 
> This will be handy when you are using templates in workflows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2912) Processor to generate a flow file from a string

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15657734#comment-15657734
 ] 

ASF GitHub Bot commented on NIFI-2912:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1206


> Processor to generate a flow file from a string
> ---
>
> Key: NIFI-2912
> URL: https://issues.apache.org/jira/browse/NIFI-2912
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Alessio Palma
>Assignee: Pierre Villard
>Priority: Minor
> Fix For: 1.2.0
>
>
> Currently to build a flow file from a string we need ReplaceText and 
> GenerateFlowFile with a size of 0B. 
> Is here a way to have use only the GenerateFlowFile processor ? 
> This will be handy when you are using templates in workflows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2912) Processor to generate a flow file from a string

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15657733#comment-15657733
 ] 

ASF subversion and git services commented on NIFI-2912:
---

Commit 06f191ca102d306be1c895b0e8a925ee3abcdc5e in nifi's branch 
refs/heads/master from [~pvillard]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=06f191c ]

NIFI-2912 Allow custom text in GenerateFlowFile

This closes #1206.

Signed-off-by: Andy LoPresto 


> Processor to generate a flow file from a string
> ---
>
> Key: NIFI-2912
> URL: https://issues.apache.org/jira/browse/NIFI-2912
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Alessio Palma
>Assignee: Pierre Villard
>Priority: Minor
> Fix For: 1.2.0
>
>
> Currently to build a flow file from a string we need ReplaceText and 
> GenerateFlowFile with a size of 0B. 
> Is here a way to have use only the GenerateFlowFile processor ? 
> This will be handy when you are using templates in workflows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #1206: NIFI-2912 Allow custom text in GenerateFlowFile

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1206


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi-minifi pull request #:

[apiri]

Github user apiri commented on the pull request:


https://github.com/apache/nifi-minifi/commit/483a4f996197b33c3e4de594a33f51e4b573fc44#commitcomment-19789688
  
In minifi-docs/src/main/markdown/minifi-java-agent-quick-start.md:
In minifi-docs/src/main/markdown/minifi-java-agent-quick-start.md on line 
93:
You are correct on that one.  We should probably adjust to make that known. 
 Unfortunately we don't have that kind of granularity to exclude in that we are 
just reusing NiFi components.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2854) Enable repositories to support upgrades and rollback in well defined scenarios

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15657694#comment-15657694
 ] 

ASF GitHub Bot commented on NIFI-2854:
--

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1202#discussion_r87631992
  
--- Diff: 
nifi-commons/nifi-schema-utils/src/main/java/org/apache/nifi/repository/schema/Requirement.java
 ---
@@ -0,0 +1,22 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.repository.schema;
+
+public enum Requirement {
--- End diff --

I believe you are correct.


> Enable repositories to support upgrades and rollback in well defined scenarios
> --
>
> Key: NIFI-2854
> URL: https://issues.apache.org/jira/browse/NIFI-2854
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.1.0
>
>
> The flowfile, swapfile, provenance, and content repositories play a very 
> important roll in NiFi's ability to be safely upgraded and rolled back.  We 
> need to have well documented behaviors, designs, and version adherence so 
> that users can safely rely on these mechanisms.
> Once this is formalized and in place we should update our versioning guidance 
> to reflect this as well.
> The following would be true from NiFi 1.2.0 onward
> * No changes to how the repositories are persisted to disk can be made which 
> will break forward/backward compatibility and specifically this means that 
> things like the way each is serialized to disk cannot change.
> * If changes are made which impact forward or backward compatibility they 
> should be reserved for major releases only and should include a utility to 
> help users with pre-existing data convert from some older format to the newer 
> format.  It may not be feasible to have rollback on major releases.
> * The content repository should not be changed within a major release cycle 
> in any way that will harm forward or backward compatibility.
> * The flow file repository can change in that new fields can be added to 
> existing write ahead log record types but no fields can be removed nor can 
> any new types be added.  Once a field is considered required it must remain 
> required.  Changes may only be made across minor version changes - not 
> incremental.
> * Swap File storage should follow very similar rules to the flow file 
> repository.  Adding a schema to the swap file header may allow some variation 
> there but the variation should only be hints to optimize how they're 
> processed and not change their behavior otherwise. Changes are only permitted 
> during minor version releases.
> * Provenance repository changes are only permitted during minor version 
> releases.  These changes may include adding or removing fields from existing 
> event types.  If a field is considered required it must always be considered 
> required.  If a field is removed then it must not be a required field and 
> there must be a sensible default an older version could use if that value is 
> not found in new data once rolled back.  New event types may be added.  
> Fields or event types not known to older version, if seen after a rollback, 
> will simply be ignored.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2854) Enable repositories to support upgrades and rollback in well defined scenarios

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15657696#comment-15657696
 ] 

ASF GitHub Bot commented on NIFI-2854:
--

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1202#discussion_r87632081
  
--- Diff: 
nifi-commons/nifi-schema-utils/src/main/java/org/apache/nifi/repository/schema/RecordSchema.java
 ---
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.repository.schema;
+
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class RecordSchema {
+private static final String FIELD_NAME = "Field Name";
+private static final String FIELD_TYPE = "Field Type";
+private static final String REPETITION = "Repetition";
+private static final String SUBFIELDS = "SubFields";
+
+private static final String STRING_TYPE = "String";
+private static final String INT_TYPE = "Integer";
+private static final String LONG_TYPE = "Long";
+private static final String SUBFIELD_TYPE = "SubFieldList";
+
+private final List fields;
+
+public RecordSchema(final List fields) {
+this.fields = fields;
+}
+
+public RecordSchema(final RecordField... fields) {
+this(Arrays.asList(fields));
+}
+
+public List getFields() {
+return fields;
+}
+
+public RecordField getField(final String fieldName) {
+return fields.stream()
+.filter(field -> field.getFieldName().equals(fieldName))
+.findFirst()
+.orElse(null);
+}
+
+public void writeTo(final OutputStream out) throws IOException {
+try {
+final DataOutputStream dos = (out instanceof DataOutputStream) 
? (DataOutputStream) out : new DataOutputStream(out);
+
+dos.writeInt(fields.size());
+for (final RecordField field : fields) {
+writeField(field, dos);
+}
+} catch (final IOException ioe) {
+throw new IOException("Unable to write Record Schema to 
stream", ioe);
+}
+}
+
+private void writeField(final RecordField field, final 
DataOutputStream dos) throws IOException {
+dos.writeInt(4);// 4 fields.
--- End diff --

Thanks for pointing this out. Will re-word the comments.


> Enable repositories to support upgrades and rollback in well defined scenarios
> --
>
> Key: NIFI-2854
> URL: https://issues.apache.org/jira/browse/NIFI-2854
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.1.0
>
>
> The flowfile, swapfile, provenance, and content repositories play a very 
> important roll in NiFi's ability to be safely upgraded and rolled back.  We 
> need to have well documented behaviors, designs, and version adherence so 
> that users can safely rely on these mechanisms.
> Once this is formalized and in place we should update our versioning guidance 
> to reflect this as well.
> The following would be true from NiFi 1.2.0 onward
> * No changes to how the repositories are persisted to disk can be made which 
> will break forward/backward compatibility and specifically this means that 
> things like the way each is serialized to disk cannot change.
> * If changes are made which impact forward or backward compatibility they 
> should be