[jira] [Updated] (NIFI-10235) Provenance replay fails when repository encryption is enabled
[ https://issues.apache.org/jira/browse/NIFI-10235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-10235: Status: In Progress (was: Patch Available) > Provenance replay fails when repository encryption is enabled > - > > Key: NIFI-10235 > URL: https://issues.apache.org/jira/browse/NIFI-10235 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework, Security >Affects Versions: 1.16.3 > Environment: RHEL 8.5 >Reporter: Peter Kimberley >Assignee: David Handermann >Priority: Major > Labels: encryption, provenance, replay > Attachments: NiFi_Flow.json, error-base-install.log, error.log > > Time Spent: 1h > Remaining Estimate: 0h > > h3. Problem summary > When repository encryption is enabled, replaying a DROP provenance record > fails, with the following error appearing in the logs: > {quote}org.apache.nifi.processor.exception.FlowFileAccessException: Failed to > export > StandardFlowFileRecord[uuid=df985fc5-23da-4094-8783-2e0186bcb92d,claim=StandardContentClaim > [resourceClaim=StandardResourceClaim[id=1657864218374-23, container=default, > section=23], offset=379, > length=1048576],offset=0,name=b29633c4-324e-42fe-b3e8-1ea455fc3650,size=1048576] > to /opt/nifi/nifi-current/data/store/.b29633c4-324e-42fe-b3e8-1ea455fc3650 > due to java.io.EOFException: *Attempted to copy {color:#ff8b00}1048576{color} > bytes but only {color:#ff8b00}1048197{color} bytes were available* > {quote} > > I've observed that the difference between the sizes mentioned in the log is > {+}*always 379 bytes*{+}, regardless of the length of the input file. > > With repository encryption disabled, provenance replay works as expected. > h3. Configuration > # NiFi v1.16.3 running as a three-node cluster in Kubernetes. > # Each node has up to 8GB memory and 4 CPUs available to it. > # Testing has included both NFS and ephemeral (emptyDir) storage. > # The encryption key was generated by the following command, using the same > JDK version: > ## keytool -genseckey -alias key-1 -keyalg AES -keysize 256 -keystore > repository.p12 -storetype PKCS12 > h4. nifi.properties > {quote}nifi.repository.encryption.protocol.version=1 > nifi.repository.encryption.key.id=key-1 > nifi.repository.encryption.key.provider=KEYSTORE > nifi.repository.encryption.key.provider.keystore.location=conf/repository.p12 > nifi.repository.encryption.key.provider.keystore.password= > {quote} > h3. Processor group > GenerateFlowFile processor generating 1MB random files every second to a > PutFile processor. Have also tested with InvokeHTTP. > h3. Other comments > With repository encryption enabled, I am able to download files via the > provenance UI (suggesting that encryption/decryption works). The processor > group also performs all other actions as expected. > Not having the ability to replay provenance records is a blocker for our > deployment, which requires data to be encrypted at rest and in transit. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-10235) Provenance replay fails when repository encryption is enabled
[ https://issues.apache.org/jira/browse/NIFI-10235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-10235: Status: Patch Available (was: In Progress) > Provenance replay fails when repository encryption is enabled > - > > Key: NIFI-10235 > URL: https://issues.apache.org/jira/browse/NIFI-10235 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework, Security >Affects Versions: 1.16.3 > Environment: RHEL 8.5 >Reporter: Peter Kimberley >Assignee: David Handermann >Priority: Major > Labels: encryption, provenance, replay > Attachments: NiFi_Flow.json, error-base-install.log, error.log > > Time Spent: 10m > Remaining Estimate: 0h > > h3. Problem summary > When repository encryption is enabled, replaying a DROP provenance record > fails, with the following error appearing in the logs: > {quote}org.apache.nifi.processor.exception.FlowFileAccessException: Failed to > export > StandardFlowFileRecord[uuid=df985fc5-23da-4094-8783-2e0186bcb92d,claim=StandardContentClaim > [resourceClaim=StandardResourceClaim[id=1657864218374-23, container=default, > section=23], offset=379, > length=1048576],offset=0,name=b29633c4-324e-42fe-b3e8-1ea455fc3650,size=1048576] > to /opt/nifi/nifi-current/data/store/.b29633c4-324e-42fe-b3e8-1ea455fc3650 > due to java.io.EOFException: *Attempted to copy {color:#ff8b00}1048576{color} > bytes but only {color:#ff8b00}1048197{color} bytes were available* > {quote} > > I've observed that the difference between the sizes mentioned in the log is > {+}*always 379 bytes*{+}, regardless of the length of the input file. > > With repository encryption disabled, provenance replay works as expected. > h3. Configuration > # NiFi v1.16.3 running as a three-node cluster in Kubernetes. > # Each node has up to 8GB memory and 4 CPUs available to it. > # Testing has included both NFS and ephemeral (emptyDir) storage. > # The encryption key was generated by the following command, using the same > JDK version: > ## keytool -genseckey -alias key-1 -keyalg AES -keysize 256 -keystore > repository.p12 -storetype PKCS12 > h4. nifi.properties > {quote}nifi.repository.encryption.protocol.version=1 > nifi.repository.encryption.key.id=key-1 > nifi.repository.encryption.key.provider=KEYSTORE > nifi.repository.encryption.key.provider.keystore.location=conf/repository.p12 > nifi.repository.encryption.key.provider.keystore.password= > {quote} > h3. Processor group > GenerateFlowFile processor generating 1MB random files every second to a > PutFile processor. Have also tested with InvokeHTTP. > h3. Other comments > With repository encryption enabled, I am able to download files via the > provenance UI (suggesting that encryption/decryption works). The processor > group also performs all other actions as expected. > Not having the ability to replay provenance records is a blocker for our > deployment, which requires data to be encrypted at rest and in transit. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-10235) Provenance replay fails when repository encryption is enabled
[ https://issues.apache.org/jira/browse/NIFI-10235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Kimberley updated NIFI-10235: --- Attachment: NiFi_Flow.json > Provenance replay fails when repository encryption is enabled > - > > Key: NIFI-10235 > URL: https://issues.apache.org/jira/browse/NIFI-10235 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework, Security >Affects Versions: 1.16.3 > Environment: RHEL 8.5 >Reporter: Peter Kimberley >Priority: Major > Labels: encryption, provenance, replay > Attachments: NiFi_Flow.json, error-base-install.log, error.log > > > h3. Problem summary > When repository encryption is enabled, replaying a DROP provenance record > fails, with the following error appearing in the logs: > {quote}org.apache.nifi.processor.exception.FlowFileAccessException: Failed to > export > StandardFlowFileRecord[uuid=df985fc5-23da-4094-8783-2e0186bcb92d,claim=StandardContentClaim > [resourceClaim=StandardResourceClaim[id=1657864218374-23, container=default, > section=23], offset=379, > length=1048576],offset=0,name=b29633c4-324e-42fe-b3e8-1ea455fc3650,size=1048576] > to /opt/nifi/nifi-current/data/store/.b29633c4-324e-42fe-b3e8-1ea455fc3650 > due to java.io.EOFException: *Attempted to copy {color:#ff8b00}1048576{color} > bytes but only {color:#ff8b00}1048197{color} bytes were available* > {quote} > > I've observed that the difference between the sizes mentioned in the log is > {+}*always 379 bytes*{+}, regardless of the length of the input file. > > With repository encryption disabled, provenance replay works as expected. > h3. Configuration > # NiFi v1.16.3 running as a three-node cluster in Kubernetes. > # Each node has up to 8GB memory and 4 CPUs available to it. > # Testing has included both NFS and ephemeral (emptyDir) storage. > # The encryption key was generated by the following command, using the same > JDK version: > ## keytool -genseckey -alias key-1 -keyalg AES -keysize 256 -keystore > repository.p12 -storetype PKCS12 > h4. nifi.properties > {quote}nifi.repository.encryption.protocol.version=1 > nifi.repository.encryption.key.id=key-1 > nifi.repository.encryption.key.provider=KEYSTORE > nifi.repository.encryption.key.provider.keystore.location=conf/repository.p12 > nifi.repository.encryption.key.provider.keystore.password= > {quote} > h3. Processor group > GenerateFlowFile processor generating 1MB random files every second to a > PutFile processor. Have also tested with InvokeHTTP. > h3. Other comments > With repository encryption enabled, I am able to download files via the > provenance UI (suggesting that encryption/decryption works). The processor > group also performs all other actions as expected. > Not having the ability to replay provenance records is a blocker for our > deployment, which requires data to be encrypted at rest and in transit. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-10235) Provenance replay fails when repository encryption is enabled
[ https://issues.apache.org/jira/browse/NIFI-10235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Kimberley updated NIFI-10235: --- Environment: RHEL 8.5 (was: RHEL 8.5 / Kubernetes) > Provenance replay fails when repository encryption is enabled > - > > Key: NIFI-10235 > URL: https://issues.apache.org/jira/browse/NIFI-10235 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework, Security >Affects Versions: 1.16.3 > Environment: RHEL 8.5 >Reporter: Peter Kimberley >Priority: Major > Labels: encryption, provenance, replay > Attachments: error-base-install.log, error.log > > > h3. Problem summary > When repository encryption is enabled, replaying a DROP provenance record > fails, with the following error appearing in the logs: > {quote}org.apache.nifi.processor.exception.FlowFileAccessException: Failed to > export > StandardFlowFileRecord[uuid=df985fc5-23da-4094-8783-2e0186bcb92d,claim=StandardContentClaim > [resourceClaim=StandardResourceClaim[id=1657864218374-23, container=default, > section=23], offset=379, > length=1048576],offset=0,name=b29633c4-324e-42fe-b3e8-1ea455fc3650,size=1048576] > to /opt/nifi/nifi-current/data/store/.b29633c4-324e-42fe-b3e8-1ea455fc3650 > due to java.io.EOFException: *Attempted to copy {color:#ff8b00}1048576{color} > bytes but only {color:#ff8b00}1048197{color} bytes were available* > {quote} > > I've observed that the difference between the sizes mentioned in the log is > {+}*always 379 bytes*{+}, regardless of the length of the input file. > > With repository encryption disabled, provenance replay works as expected. > h3. Configuration > # NiFi v1.16.3 running as a three-node cluster in Kubernetes. > # Each node has up to 8GB memory and 4 CPUs available to it. > # Testing has included both NFS and ephemeral (emptyDir) storage. > # The encryption key was generated by the following command, using the same > JDK version: > ## keytool -genseckey -alias key-1 -keyalg AES -keysize 256 -keystore > repository.p12 -storetype PKCS12 > h4. nifi.properties > {quote}nifi.repository.encryption.protocol.version=1 > nifi.repository.encryption.key.id=key-1 > nifi.repository.encryption.key.provider=KEYSTORE > nifi.repository.encryption.key.provider.keystore.location=conf/repository.p12 > nifi.repository.encryption.key.provider.keystore.password= > {quote} > h3. Processor group > GenerateFlowFile processor generating 1MB random files every second to a > PutFile processor. Have also tested with InvokeHTTP. > h3. Other comments > With repository encryption enabled, I am able to download files via the > provenance UI (suggesting that encryption/decryption works). The processor > group also performs all other actions as expected. > Not having the ability to replay provenance records is a blocker for our > deployment, which requires data to be encrypted at rest and in transit. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-10235) Provenance replay fails when repository encryption is enabled
[ https://issues.apache.org/jira/browse/NIFI-10235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Kimberley updated NIFI-10235: --- Attachment: error-base-install.log > Provenance replay fails when repository encryption is enabled > - > > Key: NIFI-10235 > URL: https://issues.apache.org/jira/browse/NIFI-10235 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework, Security >Affects Versions: 1.16.3 > Environment: RHEL 8.5 >Reporter: Peter Kimberley >Priority: Major > Labels: encryption, provenance, replay > Attachments: error-base-install.log, error.log > > > h3. Problem summary > When repository encryption is enabled, replaying a DROP provenance record > fails, with the following error appearing in the logs: > {quote}org.apache.nifi.processor.exception.FlowFileAccessException: Failed to > export > StandardFlowFileRecord[uuid=df985fc5-23da-4094-8783-2e0186bcb92d,claim=StandardContentClaim > [resourceClaim=StandardResourceClaim[id=1657864218374-23, container=default, > section=23], offset=379, > length=1048576],offset=0,name=b29633c4-324e-42fe-b3e8-1ea455fc3650,size=1048576] > to /opt/nifi/nifi-current/data/store/.b29633c4-324e-42fe-b3e8-1ea455fc3650 > due to java.io.EOFException: *Attempted to copy {color:#ff8b00}1048576{color} > bytes but only {color:#ff8b00}1048197{color} bytes were available* > {quote} > > I've observed that the difference between the sizes mentioned in the log is > {+}*always 379 bytes*{+}, regardless of the length of the input file. > > With repository encryption disabled, provenance replay works as expected. > h3. Configuration > # NiFi v1.16.3 running as a three-node cluster in Kubernetes. > # Each node has up to 8GB memory and 4 CPUs available to it. > # Testing has included both NFS and ephemeral (emptyDir) storage. > # The encryption key was generated by the following command, using the same > JDK version: > ## keytool -genseckey -alias key-1 -keyalg AES -keysize 256 -keystore > repository.p12 -storetype PKCS12 > h4. nifi.properties > {quote}nifi.repository.encryption.protocol.version=1 > nifi.repository.encryption.key.id=key-1 > nifi.repository.encryption.key.provider=KEYSTORE > nifi.repository.encryption.key.provider.keystore.location=conf/repository.p12 > nifi.repository.encryption.key.provider.keystore.password= > {quote} > h3. Processor group > GenerateFlowFile processor generating 1MB random files every second to a > PutFile processor. Have also tested with InvokeHTTP. > h3. Other comments > With repository encryption enabled, I am able to download files via the > provenance UI (suggesting that encryption/decryption works). The processor > group also performs all other actions as expected. > Not having the ability to replay provenance records is a blocker for our > deployment, which requires data to be encrypted at rest and in transit. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-10235) Provenance replay fails when repository encryption is enabled
[ https://issues.apache.org/jira/browse/NIFI-10235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Kimberley updated NIFI-10235: --- Description: h3. Problem summary When repository encryption is enabled, replaying a DROP provenance record fails, with the following error appearing in the logs: {quote}org.apache.nifi.processor.exception.FlowFileAccessException: Failed to export StandardFlowFileRecord[uuid=df985fc5-23da-4094-8783-2e0186bcb92d,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1657864218374-23, container=default, section=23], offset=379, length=1048576],offset=0,name=b29633c4-324e-42fe-b3e8-1ea455fc3650,size=1048576] to /opt/nifi/nifi-current/data/store/.b29633c4-324e-42fe-b3e8-1ea455fc3650 due to java.io.EOFException: *Attempted to copy {color:#ff8b00}1048576{color} bytes but only {color:#ff8b00}1048197{color} bytes were available* {quote} I've observed that the difference between the sizes mentioned in the log is {+}*always 379 bytes*{+}, regardless of the length of the input file. With repository encryption disabled, provenance replay works as expected. h3. Configuration # NiFi v1.16.3 running as a three-node cluster in Kubernetes. # Each node has up to 8GB memory and 4 CPUs available to it. # Testing has included both NFS and ephemeral (emptyDir) storage. # The encryption key was generated by the following command, using the same JDK version: ## keytool -genseckey -alias key-1 -keyalg AES -keysize 256 -keystore repository.p12 -storetype PKCS12 h4. nifi.properties {quote}nifi.repository.encryption.protocol.version=1 nifi.repository.encryption.key.id=key-1 nifi.repository.encryption.key.provider=KEYSTORE nifi.repository.encryption.key.provider.keystore.location=conf/repository.p12 nifi.repository.encryption.key.provider.keystore.password= {quote} h3. Processor group GenerateFlowFile processor generating 1MB random files every second to a PutFile processor. Have also tested with InvokeHTTP. h3. Other comments With repository encryption enabled, I am able to download files via the provenance UI (suggesting that encryption/decryption works). The processor group also performs all other actions as expected. Not having the ability to replay provenance records is a blocker for our deployment, which requires data to be encrypted at rest and in transit. was: h3. Problem summary When repository encryption is enabled, replaying a DROP provenance record fails, with the following error appearing in the logs: {quote}org.apache.nifi.processor.exception.FlowFileAccessException: Failed to export StandardFlowFileRecord[uuid=df985fc5-23da-4094-8783-2e0186bcb92d,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1657864218374-23, container=default, section=23], offset=379, length=1048576],offset=0,name=b29633c4-324e-42fe-b3e8-1ea455fc3650,size=1048576] to /opt/nifi/nifi-current/data/store/.b29633c4-324e-42fe-b3e8-1ea455fc3650 due to java.io.EOFException: *Attempted to copy 1048576 bytes but only 1048197 bytes were available*{quote} The difference between the two size bytes is {+}*always 379*{+}, regardless of the length of the input file. With repository encryption disabled, provenance replay works as expected. h3. Configuration # NiFi v1.16.3 running as a three-node cluster in Kubernetes. # Each node has up to 8GB memory and 4 CPUs available to it. # Testing has included both NFS and ephemeral (emptyDir) storage. # The encryption key was generated by the following command, using the same JDK version: ## keytool -genseckey -alias key-1 -keyalg AES -keysize 256 -keystore repository.p12 -storetype PKCS12 h4. nifi.properties {quote}nifi.repository.encryption.protocol.version=1 nifi.repository.encryption.key.id=key-1 nifi.repository.encryption.key.provider=KEYSTORE nifi.repository.encryption.key.provider.keystore.location=conf/repository.p12 nifi.repository.encryption.key.provider.keystore.password={quote} h3. Processor group GenerateFlowFile processor generating 1MB random files every second to a PutFile processor. Have also tested with InvokeHTTP. h3. Other comments With repository encryption enabled, I am able to download files via the provenance UI (suggesting that encryption/decryption works). The processor group also performs all other actions as expected. Not having the ability to replay provenance records is a blocker for our deployment, which requires data to be encrypted at rest and in transit. > Provenance replay fails when repository encryption is enabled > - > > Key: NIFI-10235 > URL: https://issues.apache.org/jira/browse/NIFI-10235 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework, Security >Affects Versions: 1.16.3 > Environment: RHEL 8.5 / Kubernetes >Reporter: Peter K