[jira] [Updated] (NIFI-10235) Provenance replay fails when repository encryption is enabled
[
https://issues.apache.org/jira/browse/NIFI-10235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann updated NIFI-10235:
Status: In Progress (was: Patch Available)
> Provenance replay fails when repository encryption is enabled
> -
>
> Key: NIFI-10235
> URL: https://issues.apache.org/jira/browse/NIFI-10235
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework, Security
>Affects Versions: 1.16.3
> Environment: RHEL 8.5
>Reporter: Peter Kimberley
>Assignee: David Handermann
>Priority: Major
> Labels: encryption, provenance, replay
> Attachments: NiFi_Flow.json, error-base-install.log, error.log
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> h3. Problem summary
> When repository encryption is enabled, replaying a DROP provenance record
> fails, with the following error appearing in the logs:
> {quote}org.apache.nifi.processor.exception.FlowFileAccessException: Failed to
> export
> StandardFlowFileRecord[uuid=df985fc5-23da-4094-8783-2e0186bcb92d,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1657864218374-23, container=default,
> section=23], offset=379,
> length=1048576],offset=0,name=b29633c4-324e-42fe-b3e8-1ea455fc3650,size=1048576]
> to /opt/nifi/nifi-current/data/store/.b29633c4-324e-42fe-b3e8-1ea455fc3650
> due to java.io.EOFException: *Attempted to copy {color:#ff8b00}1048576{color}
> bytes but only {color:#ff8b00}1048197{color} bytes were available*
> {quote}
>
> I've observed that the difference between the sizes mentioned in the log is
> {+}*always 379 bytes*{+}, regardless of the length of the input file.
>
> With repository encryption disabled, provenance replay works as expected.
> h3. Configuration
> # NiFi v1.16.3 running as a three-node cluster in Kubernetes.
> # Each node has up to 8GB memory and 4 CPUs available to it.
> # Testing has included both NFS and ephemeral (emptyDir) storage.
> # The encryption key was generated by the following command, using the same
> JDK version:
> ## keytool -genseckey -alias key-1 -keyalg AES -keysize 256 -keystore
> repository.p12 -storetype PKCS12
> h4. nifi.properties
> {quote}nifi.repository.encryption.protocol.version=1
> nifi.repository.encryption.key.id=key-1
> nifi.repository.encryption.key.provider=KEYSTORE
> nifi.repository.encryption.key.provider.keystore.location=conf/repository.p12
> nifi.repository.encryption.key.provider.keystore.password=
> {quote}
> h3. Processor group
> GenerateFlowFile processor generating 1MB random files every second to a
> PutFile processor. Have also tested with InvokeHTTP.
> h3. Other comments
> With repository encryption enabled, I am able to download files via the
> provenance UI (suggesting that encryption/decryption works). The processor
> group also performs all other actions as expected.
> Not having the ability to replay provenance records is a blocker for our
> deployment, which requires data to be encrypted at rest and in transit.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-10235) Provenance replay fails when repository encryption is enabled
[
https://issues.apache.org/jira/browse/NIFI-10235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann updated NIFI-10235:
Status: Patch Available (was: In Progress)
> Provenance replay fails when repository encryption is enabled
> -
>
> Key: NIFI-10235
> URL: https://issues.apache.org/jira/browse/NIFI-10235
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework, Security
>Affects Versions: 1.16.3
> Environment: RHEL 8.5
>Reporter: Peter Kimberley
>Assignee: David Handermann
>Priority: Major
> Labels: encryption, provenance, replay
> Attachments: NiFi_Flow.json, error-base-install.log, error.log
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> h3. Problem summary
> When repository encryption is enabled, replaying a DROP provenance record
> fails, with the following error appearing in the logs:
> {quote}org.apache.nifi.processor.exception.FlowFileAccessException: Failed to
> export
> StandardFlowFileRecord[uuid=df985fc5-23da-4094-8783-2e0186bcb92d,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1657864218374-23, container=default,
> section=23], offset=379,
> length=1048576],offset=0,name=b29633c4-324e-42fe-b3e8-1ea455fc3650,size=1048576]
> to /opt/nifi/nifi-current/data/store/.b29633c4-324e-42fe-b3e8-1ea455fc3650
> due to java.io.EOFException: *Attempted to copy {color:#ff8b00}1048576{color}
> bytes but only {color:#ff8b00}1048197{color} bytes were available*
> {quote}
>
> I've observed that the difference between the sizes mentioned in the log is
> {+}*always 379 bytes*{+}, regardless of the length of the input file.
>
> With repository encryption disabled, provenance replay works as expected.
> h3. Configuration
> # NiFi v1.16.3 running as a three-node cluster in Kubernetes.
> # Each node has up to 8GB memory and 4 CPUs available to it.
> # Testing has included both NFS and ephemeral (emptyDir) storage.
> # The encryption key was generated by the following command, using the same
> JDK version:
> ## keytool -genseckey -alias key-1 -keyalg AES -keysize 256 -keystore
> repository.p12 -storetype PKCS12
> h4. nifi.properties
> {quote}nifi.repository.encryption.protocol.version=1
> nifi.repository.encryption.key.id=key-1
> nifi.repository.encryption.key.provider=KEYSTORE
> nifi.repository.encryption.key.provider.keystore.location=conf/repository.p12
> nifi.repository.encryption.key.provider.keystore.password=
> {quote}
> h3. Processor group
> GenerateFlowFile processor generating 1MB random files every second to a
> PutFile processor. Have also tested with InvokeHTTP.
> h3. Other comments
> With repository encryption enabled, I am able to download files via the
> provenance UI (suggesting that encryption/decryption works). The processor
> group also performs all other actions as expected.
> Not having the ability to replay provenance records is a blocker for our
> deployment, which requires data to be encrypted at rest and in transit.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-10235) Provenance replay fails when repository encryption is enabled
[
https://issues.apache.org/jira/browse/NIFI-10235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Kimberley updated NIFI-10235:
---
Attachment: NiFi_Flow.json
> Provenance replay fails when repository encryption is enabled
> -
>
> Key: NIFI-10235
> URL: https://issues.apache.org/jira/browse/NIFI-10235
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework, Security
>Affects Versions: 1.16.3
> Environment: RHEL 8.5
>Reporter: Peter Kimberley
>Priority: Major
> Labels: encryption, provenance, replay
> Attachments: NiFi_Flow.json, error-base-install.log, error.log
>
>
> h3. Problem summary
> When repository encryption is enabled, replaying a DROP provenance record
> fails, with the following error appearing in the logs:
> {quote}org.apache.nifi.processor.exception.FlowFileAccessException: Failed to
> export
> StandardFlowFileRecord[uuid=df985fc5-23da-4094-8783-2e0186bcb92d,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1657864218374-23, container=default,
> section=23], offset=379,
> length=1048576],offset=0,name=b29633c4-324e-42fe-b3e8-1ea455fc3650,size=1048576]
> to /opt/nifi/nifi-current/data/store/.b29633c4-324e-42fe-b3e8-1ea455fc3650
> due to java.io.EOFException: *Attempted to copy {color:#ff8b00}1048576{color}
> bytes but only {color:#ff8b00}1048197{color} bytes were available*
> {quote}
>
> I've observed that the difference between the sizes mentioned in the log is
> {+}*always 379 bytes*{+}, regardless of the length of the input file.
>
> With repository encryption disabled, provenance replay works as expected.
> h3. Configuration
> # NiFi v1.16.3 running as a three-node cluster in Kubernetes.
> # Each node has up to 8GB memory and 4 CPUs available to it.
> # Testing has included both NFS and ephemeral (emptyDir) storage.
> # The encryption key was generated by the following command, using the same
> JDK version:
> ## keytool -genseckey -alias key-1 -keyalg AES -keysize 256 -keystore
> repository.p12 -storetype PKCS12
> h4. nifi.properties
> {quote}nifi.repository.encryption.protocol.version=1
> nifi.repository.encryption.key.id=key-1
> nifi.repository.encryption.key.provider=KEYSTORE
> nifi.repository.encryption.key.provider.keystore.location=conf/repository.p12
> nifi.repository.encryption.key.provider.keystore.password=
> {quote}
> h3. Processor group
> GenerateFlowFile processor generating 1MB random files every second to a
> PutFile processor. Have also tested with InvokeHTTP.
> h3. Other comments
> With repository encryption enabled, I am able to download files via the
> provenance UI (suggesting that encryption/decryption works). The processor
> group also performs all other actions as expected.
> Not having the ability to replay provenance records is a blocker for our
> deployment, which requires data to be encrypted at rest and in transit.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-10235) Provenance replay fails when repository encryption is enabled
[
https://issues.apache.org/jira/browse/NIFI-10235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Kimberley updated NIFI-10235:
---
Environment: RHEL 8.5 (was: RHEL 8.5 / Kubernetes)
> Provenance replay fails when repository encryption is enabled
> -
>
> Key: NIFI-10235
> URL: https://issues.apache.org/jira/browse/NIFI-10235
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework, Security
>Affects Versions: 1.16.3
> Environment: RHEL 8.5
>Reporter: Peter Kimberley
>Priority: Major
> Labels: encryption, provenance, replay
> Attachments: error-base-install.log, error.log
>
>
> h3. Problem summary
> When repository encryption is enabled, replaying a DROP provenance record
> fails, with the following error appearing in the logs:
> {quote}org.apache.nifi.processor.exception.FlowFileAccessException: Failed to
> export
> StandardFlowFileRecord[uuid=df985fc5-23da-4094-8783-2e0186bcb92d,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1657864218374-23, container=default,
> section=23], offset=379,
> length=1048576],offset=0,name=b29633c4-324e-42fe-b3e8-1ea455fc3650,size=1048576]
> to /opt/nifi/nifi-current/data/store/.b29633c4-324e-42fe-b3e8-1ea455fc3650
> due to java.io.EOFException: *Attempted to copy {color:#ff8b00}1048576{color}
> bytes but only {color:#ff8b00}1048197{color} bytes were available*
> {quote}
>
> I've observed that the difference between the sizes mentioned in the log is
> {+}*always 379 bytes*{+}, regardless of the length of the input file.
>
> With repository encryption disabled, provenance replay works as expected.
> h3. Configuration
> # NiFi v1.16.3 running as a three-node cluster in Kubernetes.
> # Each node has up to 8GB memory and 4 CPUs available to it.
> # Testing has included both NFS and ephemeral (emptyDir) storage.
> # The encryption key was generated by the following command, using the same
> JDK version:
> ## keytool -genseckey -alias key-1 -keyalg AES -keysize 256 -keystore
> repository.p12 -storetype PKCS12
> h4. nifi.properties
> {quote}nifi.repository.encryption.protocol.version=1
> nifi.repository.encryption.key.id=key-1
> nifi.repository.encryption.key.provider=KEYSTORE
> nifi.repository.encryption.key.provider.keystore.location=conf/repository.p12
> nifi.repository.encryption.key.provider.keystore.password=
> {quote}
> h3. Processor group
> GenerateFlowFile processor generating 1MB random files every second to a
> PutFile processor. Have also tested with InvokeHTTP.
> h3. Other comments
> With repository encryption enabled, I am able to download files via the
> provenance UI (suggesting that encryption/decryption works). The processor
> group also performs all other actions as expected.
> Not having the ability to replay provenance records is a blocker for our
> deployment, which requires data to be encrypted at rest and in transit.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-10235) Provenance replay fails when repository encryption is enabled
[
https://issues.apache.org/jira/browse/NIFI-10235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Kimberley updated NIFI-10235:
---
Attachment: error-base-install.log
> Provenance replay fails when repository encryption is enabled
> -
>
> Key: NIFI-10235
> URL: https://issues.apache.org/jira/browse/NIFI-10235
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework, Security
>Affects Versions: 1.16.3
> Environment: RHEL 8.5
>Reporter: Peter Kimberley
>Priority: Major
> Labels: encryption, provenance, replay
> Attachments: error-base-install.log, error.log
>
>
> h3. Problem summary
> When repository encryption is enabled, replaying a DROP provenance record
> fails, with the following error appearing in the logs:
> {quote}org.apache.nifi.processor.exception.FlowFileAccessException: Failed to
> export
> StandardFlowFileRecord[uuid=df985fc5-23da-4094-8783-2e0186bcb92d,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1657864218374-23, container=default,
> section=23], offset=379,
> length=1048576],offset=0,name=b29633c4-324e-42fe-b3e8-1ea455fc3650,size=1048576]
> to /opt/nifi/nifi-current/data/store/.b29633c4-324e-42fe-b3e8-1ea455fc3650
> due to java.io.EOFException: *Attempted to copy {color:#ff8b00}1048576{color}
> bytes but only {color:#ff8b00}1048197{color} bytes were available*
> {quote}
>
> I've observed that the difference between the sizes mentioned in the log is
> {+}*always 379 bytes*{+}, regardless of the length of the input file.
>
> With repository encryption disabled, provenance replay works as expected.
> h3. Configuration
> # NiFi v1.16.3 running as a three-node cluster in Kubernetes.
> # Each node has up to 8GB memory and 4 CPUs available to it.
> # Testing has included both NFS and ephemeral (emptyDir) storage.
> # The encryption key was generated by the following command, using the same
> JDK version:
> ## keytool -genseckey -alias key-1 -keyalg AES -keysize 256 -keystore
> repository.p12 -storetype PKCS12
> h4. nifi.properties
> {quote}nifi.repository.encryption.protocol.version=1
> nifi.repository.encryption.key.id=key-1
> nifi.repository.encryption.key.provider=KEYSTORE
> nifi.repository.encryption.key.provider.keystore.location=conf/repository.p12
> nifi.repository.encryption.key.provider.keystore.password=
> {quote}
> h3. Processor group
> GenerateFlowFile processor generating 1MB random files every second to a
> PutFile processor. Have also tested with InvokeHTTP.
> h3. Other comments
> With repository encryption enabled, I am able to download files via the
> provenance UI (suggesting that encryption/decryption works). The processor
> group also performs all other actions as expected.
> Not having the ability to replay provenance records is a blocker for our
> deployment, which requires data to be encrypted at rest and in transit.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-10235) Provenance replay fails when repository encryption is enabled
[
https://issues.apache.org/jira/browse/NIFI-10235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Kimberley updated NIFI-10235:
---
Description:
h3. Problem summary
When repository encryption is enabled, replaying a DROP provenance record
fails, with the following error appearing in the logs:
{quote}org.apache.nifi.processor.exception.FlowFileAccessException: Failed to
export
StandardFlowFileRecord[uuid=df985fc5-23da-4094-8783-2e0186bcb92d,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1657864218374-23, container=default,
section=23], offset=379,
length=1048576],offset=0,name=b29633c4-324e-42fe-b3e8-1ea455fc3650,size=1048576]
to /opt/nifi/nifi-current/data/store/.b29633c4-324e-42fe-b3e8-1ea455fc3650 due
to java.io.EOFException: *Attempted to copy {color:#ff8b00}1048576{color} bytes
but only {color:#ff8b00}1048197{color} bytes were available*
{quote}
I've observed that the difference between the sizes mentioned in the log is
{+}*always 379 bytes*{+}, regardless of the length of the input file.
With repository encryption disabled, provenance replay works as expected.
h3. Configuration
# NiFi v1.16.3 running as a three-node cluster in Kubernetes.
# Each node has up to 8GB memory and 4 CPUs available to it.
# Testing has included both NFS and ephemeral (emptyDir) storage.
# The encryption key was generated by the following command, using the same
JDK version:
## keytool -genseckey -alias key-1 -keyalg AES -keysize 256 -keystore
repository.p12 -storetype PKCS12
h4. nifi.properties
{quote}nifi.repository.encryption.protocol.version=1
nifi.repository.encryption.key.id=key-1
nifi.repository.encryption.key.provider=KEYSTORE
nifi.repository.encryption.key.provider.keystore.location=conf/repository.p12
nifi.repository.encryption.key.provider.keystore.password=
{quote}
h3. Processor group
GenerateFlowFile processor generating 1MB random files every second to a
PutFile processor. Have also tested with InvokeHTTP.
h3. Other comments
With repository encryption enabled, I am able to download files via the
provenance UI (suggesting that encryption/decryption works). The processor
group also performs all other actions as expected.
Not having the ability to replay provenance records is a blocker for our
deployment, which requires data to be encrypted at rest and in transit.
was:
h3. Problem summary
When repository encryption is enabled, replaying a DROP provenance record
fails, with the following error appearing in the logs:
{quote}org.apache.nifi.processor.exception.FlowFileAccessException: Failed to
export
StandardFlowFileRecord[uuid=df985fc5-23da-4094-8783-2e0186bcb92d,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1657864218374-23, container=default,
section=23], offset=379,
length=1048576],offset=0,name=b29633c4-324e-42fe-b3e8-1ea455fc3650,size=1048576]
to /opt/nifi/nifi-current/data/store/.b29633c4-324e-42fe-b3e8-1ea455fc3650 due
to java.io.EOFException: *Attempted to copy 1048576 bytes but only 1048197
bytes were available*{quote}
The difference between the two size bytes is {+}*always 379*{+}, regardless of
the length of the input file.
With repository encryption disabled, provenance replay works as expected.
h3. Configuration
# NiFi v1.16.3 running as a three-node cluster in Kubernetes.
# Each node has up to 8GB memory and 4 CPUs available to it.
# Testing has included both NFS and ephemeral (emptyDir) storage.
# The encryption key was generated by the following command, using the same
JDK version:
## keytool -genseckey -alias key-1 -keyalg AES -keysize 256 -keystore
repository.p12 -storetype PKCS12
h4. nifi.properties
{quote}nifi.repository.encryption.protocol.version=1
nifi.repository.encryption.key.id=key-1
nifi.repository.encryption.key.provider=KEYSTORE
nifi.repository.encryption.key.provider.keystore.location=conf/repository.p12
nifi.repository.encryption.key.provider.keystore.password={quote}
h3. Processor group
GenerateFlowFile processor generating 1MB random files every second to a
PutFile processor. Have also tested with InvokeHTTP.
h3. Other comments
With repository encryption enabled, I am able to download files via the
provenance UI (suggesting that encryption/decryption works). The processor
group also performs all other actions as expected.
Not having the ability to replay provenance records is a blocker for our
deployment, which requires data to be encrypted at rest and in transit.
> Provenance replay fails when repository encryption is enabled
> -
>
> Key: NIFI-10235
> URL: https://issues.apache.org/jira/browse/NIFI-10235
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework, Security
>Affects Versions: 1.16.3
> Environment: RHEL 8.5 / Kubernetes
>Reporter: Peter
