[jira] [Updated] (NIFI-6113) GetHDFSFileInfo not working correctly with > 1 concurrent task
[ https://issues.apache.org/jira/browse/NIFI-6113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stan Antyufeev updated NIFI-6113: - Priority: Major (was: Minor) > GetHDFSFileInfo not working correctly with > 1 concurrent task > -- > > Key: NIFI-6113 > URL: https://issues.apache.org/jira/browse/NIFI-6113 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.8.0, 1.7.1, 1.9.0 >Reporter: Bryan Bende >Assignee: Stan Antyufeev >Priority: Major > Attachments: GetHDFSFileInfo.java > > Time Spent: 50m > Remaining Estimate: 0h > > The processor has a member variable that stores some information about what > directory to filter on, and the member variable is not protected for > concurrent access. If someone configures the processor with > 1 concurrent > tasks, than each execution of the processor can be updating the member > variable at the same time, leading to unexpected results. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8050) Custom Groovy writer breaks during upgrade
[
https://issues.apache.org/jira/browse/NIFI-8050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre Villard updated NIFI-8050:
-
Fix Version/s: (was: 1.10.0)
1.13.0
> Custom Groovy writer breaks during upgrade
> --
>
> Key: NIFI-8050
> URL: https://issues.apache.org/jira/browse/NIFI-8050
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: Pierre Villard
>Assignee: Matt Burgess
>Priority: Major
> Fix For: 1.13.0
>
>
> A couple of issues when upgrading NiFi and using a custom scripted writer
> with Groovy.
> The scripted writer was something like:
> {code:java}
> import ...
> class GroovyRecordSetWriter implements RecordSetWriter {
> ...
> @Override
> WriteResult write(Record r) throws IOException {
> ...
> }
> @Override
> String getMimeType() { ... }
> @Override
> WriteResult write(final RecordSet rs) throws IOException {
> ...
> }
> public void beginRecordSet() throws IOException { ... }
> @Override
> public WriteResult finishRecordSet() throws IOException { ... }
> @Override
> public void close() throws IOException {}
> @Override
> public void flush() throws IOException {}
> }
> class GroovyRecordSetWriterFactory extends AbstractControllerService
> implements RecordSetWriterFactory {
> @Override
> RecordSchema getSchema(Map variables, RecordSchema
> readSchema) throws SchemaNotFoundException, IOException {
>null
> }
> @Override
> RecordSetWriter createWriter(ComponentLog logger, RecordSchema schema,
> OutputStream out) throws SchemaNotFoundException, IOException {
>new GroovyRecordSetWriter(out)
> }
> }
> writer = new GroovyRecordSetWriterFactory()
> {code}
> With NIFI-6318 we changed a method in the interface RecordSetWriterFactory.
> When using the above code in NiFi 1.9.2, it works fine but after an upgrade
> on 1.11.4, this breaks. The Controller Service, when enabled, is throwing the
> below message:
> {quote}Can't have an abstract method in a non-abstract class. The class
> 'GroovyRecordSetWriterFactory' must be declared abstract or the method
> 'org.apache.nifi.serialization.RecordSetWriter
> createWriter(org.apache.nifi.logging.ComponentLog,
> org.apache.nifi.serialization.record.RecordSchema, java.io.OutputStream,
> java.util.Map)' must be implemented.
> {quote}
> However the controller service is successfully enabled and the processors
> referencing it can be started. When using the ConvertRecord processor with
> the problematic controller service, it will throw the below NPE:
> {code:java}
> 2020-11-26 15:46:13,876 ERROR [Timer-Driven Process Thread-25]
> o.a.n.processors.standard.ConvertRecord
> ConvertRecord[id=8b5456ae-71dc-3bd3-d0c0-df50d196fc00] Failed to process
> StandardFlowFileRecord[uuid=adebfcf6-b449-4d01-90a7-0463930aade0,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1606401933295-1, container=default,
> section=1], offset=80,
> length=296],offset=0,name=adebfcf6-b449-4d01-90a7-0463930aade0,size=296];
> will route to failure: java.lang.NullPointerException
> java.lang.NullPointerException: null at
> org.apache.nifi.processors.standard.AbstractRecordProcessor$1.process(AbstractRecordProcessor.java:151)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2986)
> at
> org.apache.nifi.processors.standard.AbstractRecordProcessor.onTrigger(AbstractRecordProcessor.java:122)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1173)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at
> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> The fix is quite simple, it's just required to add the proper implementation
> method in the Groovy script. Something like:
> {code:java}
> class GroovyRecordSetW
[GitHub] [nifi] pvillard31 commented on pull request #4837: NIFI-6113 Refactoring to remove instance level variable to make it stateless
pvillard31 commented on pull request #4837: URL: https://github.com/apache/nifi/pull/4837#issuecomment-784741212 @stanislav-antufeev - I added you as a contributor on JIRA and assigned the JIRA to you. You should have enough permissions going forward. Thanks for your contribution. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-6113) GetHDFSFileInfo not working correctly with > 1 concurrent task
[ https://issues.apache.org/jira/browse/NIFI-6113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-6113: - Status: Patch Available (was: Open) > GetHDFSFileInfo not working correctly with > 1 concurrent task > -- > > Key: NIFI-6113 > URL: https://issues.apache.org/jira/browse/NIFI-6113 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.9.0, 1.7.1, 1.8.0 >Reporter: Bryan Bende >Assignee: Stan Antyufeev >Priority: Minor > Attachments: GetHDFSFileInfo.java > > Time Spent: 40m > Remaining Estimate: 0h > > The processor has a member variable that stores some information about what > directory to filter on, and the member variable is not protected for > concurrent access. If someone configures the processor with > 1 concurrent > tasks, than each execution of the processor can be updating the member > variable at the same time, leading to unexpected results. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (NIFI-6113) GetHDFSFileInfo not working correctly with > 1 concurrent task
[ https://issues.apache.org/jira/browse/NIFI-6113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard reassigned NIFI-6113: Assignee: Stan Antyufeev > GetHDFSFileInfo not working correctly with > 1 concurrent task > -- > > Key: NIFI-6113 > URL: https://issues.apache.org/jira/browse/NIFI-6113 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.8.0, 1.7.1, 1.9.0 >Reporter: Bryan Bende >Assignee: Stan Antyufeev >Priority: Minor > Attachments: GetHDFSFileInfo.java > > Time Spent: 40m > Remaining Estimate: 0h > > The processor has a member variable that stores some information about what > directory to filter on, and the member variable is not protected for > concurrent access. If someone configures the processor with > 1 concurrent > tasks, than each execution of the processor can be updating the member > variable at the same time, leading to unexpected results. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] pvillard31 commented on pull request #2231: NIFI-4521 MS SQL CDC Processor
pvillard31 commented on pull request #2231: URL: https://github.com/apache/nifi/pull/2231#issuecomment-784739712 Yeah I know the feeling, I also have very old PRs that would be nice to have merged. Let me know how it goes with your investigation and we can include it for the next release. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-7246) JWT Generated by a node in the cluster is not honored by other nodes in the cluster.
[ https://issues.apache.org/jira/browse/NIFI-7246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289606#comment-17289606 ] David Handermann commented on NIFI-7246: [~shreyaskc] It looks like the current PR includes a number of changes that are not related to the purpose described. At minimum, it would be helpful to remove extraneous changes so that the purpose of the PR is clear. As others have described workarounds, it may also be worth reconsidering the approach. Introducing a static JWT raises some security concerns as it effectively provides a permanently valid credential for access. Another approach to consider might be leveraging the NiFi Sensitive Properties Key to derive a JWT signing key that could be verified on any node, since all nodes share the Sensitive Properties Key. This would require careful consideration and implementation, but it sounds like it would meet your requirements and provide a more secure approach. > JWT Generated by a node in the cluster is not honored by other nodes in the > cluster. > > > Key: NIFI-7246 > URL: https://issues.apache.org/jira/browse/NIFI-7246 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework, Security >Reporter: Shreyas KC >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > In an externally load balanced cluster without sticky session, it is not > possible to currently share the JWT generated by one node with the rest of > the nodes in the cluster. > Hence we need a mechanism where we can introduce static key in the > nifi.properties in its chosen by the cluster administrator. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-7774) Implement the GetKinesisStream processor using AWS SDK
[ https://issues.apache.org/jira/browse/NIFI-7774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289589#comment-17289589 ] Domenico Campagnolo commented on NIFI-7774: --- Hi [~rdskha...@gmail.com], Please find the attached code and the .nar file > Implement the GetKinesisStream processor using AWS SDK > -- > > Key: NIFI-7774 > URL: https://issues.apache.org/jira/browse/NIFI-7774 > Project: Apache NiFi > Issue Type: New Feature >Affects Versions: 1.11.4 >Reporter: Domenico Campagnolo >Priority: Major > Attachments: nifi-getkinesis-nar-1.11.4.nar, > nifi-getkinesis-processors.zip > > > Implement the GetKinesisStream processor to be consistent with the other AWS > processors. > As per AWS documentation > [https://docs.aws.amazon.com/streams/latest/dev/shared-throughput-consumers.html] > you can build a consumer using KCL (Kinesis client v1 or v2) or AWS SDK > Analyzing all the existent AWS processor I realized that they are implemented > using AWS SDK. > Actually I have already implemented it and also using it in production. > This ticket is to share my implementation with the Nifi community. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-7774) Implement the GetKinesisStream processor using AWS SDK
[ https://issues.apache.org/jira/browse/NIFI-7774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Domenico Campagnolo updated NIFI-7774: -- Attachment: nifi-getkinesis-nar-1.11.4.nar > Implement the GetKinesisStream processor using AWS SDK > -- > > Key: NIFI-7774 > URL: https://issues.apache.org/jira/browse/NIFI-7774 > Project: Apache NiFi > Issue Type: New Feature >Affects Versions: 1.11.4 >Reporter: Domenico Campagnolo >Priority: Major > Attachments: nifi-getkinesis-nar-1.11.4.nar, > nifi-getkinesis-processors.zip > > > Implement the GetKinesisStream processor to be consistent with the other AWS > processors. > As per AWS documentation > [https://docs.aws.amazon.com/streams/latest/dev/shared-throughput-consumers.html] > you can build a consumer using KCL (Kinesis client v1 or v2) or AWS SDK > Analyzing all the existent AWS processor I realized that they are implemented > using AWS SDK. > Actually I have already implemented it and also using it in production. > This ticket is to share my implementation with the Nifi community. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-7774) Implement the GetKinesisStream processor using AWS SDK
[ https://issues.apache.org/jira/browse/NIFI-7774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Domenico Campagnolo updated NIFI-7774: -- Attachment: nifi-getkinesis-processors.zip > Implement the GetKinesisStream processor using AWS SDK > -- > > Key: NIFI-7774 > URL: https://issues.apache.org/jira/browse/NIFI-7774 > Project: Apache NiFi > Issue Type: New Feature >Affects Versions: 1.11.4 >Reporter: Domenico Campagnolo >Priority: Major > Attachments: nifi-getkinesis-nar-1.11.4.nar, > nifi-getkinesis-processors.zip > > > Implement the GetKinesisStream processor to be consistent with the other AWS > processors. > As per AWS documentation > [https://docs.aws.amazon.com/streams/latest/dev/shared-throughput-consumers.html] > you can build a consumer using KCL (Kinesis client v1 or v2) or AWS SDK > Analyzing all the existent AWS processor I realized that they are implemented > using AWS SDK. > Actually I have already implemented it and also using it in production. > This ticket is to share my implementation with the Nifi community. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] stanislav-antufeev commented on pull request #4837: NIFI-6113 Refactoring to remove instance level variable to make it stateless
stanislav-antufeev commented on pull request #4837: URL: https://github.com/apache/nifi/pull/4837#issuecomment-784700878 @turcsanyip looks like i don't have enough pemissions on https://issues.apache.org to assign the ticket to me. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-7774) Implement the GetKinesisStream processor using AWS SDK
[ https://issues.apache.org/jira/browse/NIFI-7774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289573#comment-17289573 ] RIPU DAMAN commented on NIFI-7774: -- Can you please share the implementation, i do not see any nars or source code attached here? Thanks in advance! > Implement the GetKinesisStream processor using AWS SDK > -- > > Key: NIFI-7774 > URL: https://issues.apache.org/jira/browse/NIFI-7774 > Project: Apache NiFi > Issue Type: New Feature >Affects Versions: 1.11.4 >Reporter: Domenico Campagnolo >Priority: Major > > Implement the GetKinesisStream processor to be consistent with the other AWS > processors. > As per AWS documentation > [https://docs.aws.amazon.com/streams/latest/dev/shared-throughput-consumers.html] > you can build a consumer using KCL (Kinesis client v1 or v2) or AWS SDK > Analyzing all the existent AWS processor I realized that they are implemented > using AWS SDK. > Actually I have already implemented it and also using it in production. > This ticket is to share my implementation with the Nifi community. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-7246) JWT Generated by a node in the cluster is not honored by other nodes in the cluster.
[ https://issues.apache.org/jira/browse/NIFI-7246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289561#comment-17289561 ] Shreyas KC commented on NIFI-7246: -- I know, but i need approval to get this merged. On Fri, Oct 16, 2020, 10:29 PM Sebastian Valle (Jira) > JWT Generated by a node in the cluster is not honored by other nodes in the > cluster. > > > Key: NIFI-7246 > URL: https://issues.apache.org/jira/browse/NIFI-7246 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework, Security >Reporter: Shreyas KC >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > In an externally load balanced cluster without sticky session, it is not > possible to currently share the JWT generated by one node with the rest of > the nodes in the cluster. > Hence we need a mechanism where we can introduce static key in the > nifi.properties in its chosen by the cluster administrator. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8249) ExecuteSQLRecord "administratively yields" and rollls back session when fetch size < query partition size and an SQLException is thrown on subsequent next()
[ https://issues.apache.org/jira/browse/NIFI-8249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-8249: --- Status: Patch Available (was: In Progress) > ExecuteSQLRecord "administratively yields" and rollls back session when fetch > size < query partition size and an SQLException is thrown on subsequent next() > > > Key: NIFI-8249 > URL: https://issues.apache.org/jira/browse/NIFI-8249 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: David >Assignee: Matt Burgess >Priority: Major > Attachments: ExecuteSQLRecord.png, FlowFile to ESQLRec.png, > GenerateFlowFile.png, GenerateTableFetch.png, Screen Shot 2021-02-23 at > 3.00.47 PM.png, Screen Shot 2021-02-23 at 3.04.55 PM.png > > Time Spent: 10m > Remaining Estimate: 0h > > When ExecuteSQLRecord executes a query with a partition size > the fetch size > and one of the fetches throws an SQLException, the exception appears to get > "swallowed" and a more general "IOException: Could not obtain next record > from ResultSet: routing to failure" in addition to a "transfer relationship > not specified;Processor Administratively Yielded for 1 sec" errors are thrown. > The problematic query/flowfile gets put back into the incoming queue and will > continue to fail indefinitely until it is manually removed from the queue. > If the fetch size >= to the query partition size, the exact SQL error message > is thrown and the flowfile gets routed to the failure queue. > > Steps to reliably reproduce: > # Create an DB table with an ID column (for partitioning queries against) > and a VARCHAR column > # Insert a row where the VARCHAR column data is a numeric value (ie '123') > # Insert a row where the VARCHAR column data is a string value (ie 'abc') > # Create a GenerateTableFetch processor where the "columns to return" > includes TO_NUMBER() > # Create a ExecuteSQLRecord processor that accepts the "success" connection > from GenerateTableFetch > # Set ExecuteSQLRecord fetch size to 1 > # Run flow > What should happen: > * ExecuteSQLRecord should throw an java.sql.SQLSyntaxErrorException: > ORA-01722: invalid number > * Flowfile should get transferred to failure queue > What happens: > * Nifi throws one error and prints another, then puts the flowfile back into > the incoming queue > ** ExecuteSQLRecord throws "Unable to execute SQL select query ... > due to java.io.IOException: java.io.IOException: Could not obtain next record > from ResultSet" > ** Nifi prints errror "transfer relationship not specified; Processor > Administratively Yielded for 1 sec" > ** Flowfile is kept in current queue and penalized and re-processed again > after penalty duration expires > Thus, the problematic query will get executed indefinitely. > > Note: I believe the issue could be due to ResultSet .next() being called > first in the constructor to ResultSetRecordSetWithCallback > Subsequent .next() is called from resultSetWriter.write(recordSet) > [https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73|https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8249) ExecuteSQLRecord "administratively yields" and rollls back session when fetch size < query partition size and an SQLException is thrown on subsequent next()
[ https://issues.apache.org/jira/browse/NIFI-8249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-8249: --- Fix Version/s: 1.14.0 > ExecuteSQLRecord "administratively yields" and rollls back session when fetch > size < query partition size and an SQLException is thrown on subsequent next() > > > Key: NIFI-8249 > URL: https://issues.apache.org/jira/browse/NIFI-8249 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: David >Assignee: Matt Burgess >Priority: Major > Fix For: 1.14.0 > > Attachments: ExecuteSQLRecord.png, FlowFile to ESQLRec.png, > GenerateFlowFile.png, GenerateTableFetch.png, Screen Shot 2021-02-23 at > 3.00.47 PM.png, Screen Shot 2021-02-23 at 3.04.55 PM.png > > Time Spent: 10m > Remaining Estimate: 0h > > When ExecuteSQLRecord executes a query with a partition size > the fetch size > and one of the fetches throws an SQLException, the exception appears to get > "swallowed" and a more general "IOException: Could not obtain next record > from ResultSet: routing to failure" in addition to a "transfer relationship > not specified;Processor Administratively Yielded for 1 sec" errors are thrown. > The problematic query/flowfile gets put back into the incoming queue and will > continue to fail indefinitely until it is manually removed from the queue. > If the fetch size >= to the query partition size, the exact SQL error message > is thrown and the flowfile gets routed to the failure queue. > > Steps to reliably reproduce: > # Create an DB table with an ID column (for partitioning queries against) > and a VARCHAR column > # Insert a row where the VARCHAR column data is a numeric value (ie '123') > # Insert a row where the VARCHAR column data is a string value (ie 'abc') > # Create a GenerateTableFetch processor where the "columns to return" > includes TO_NUMBER() > # Create a ExecuteSQLRecord processor that accepts the "success" connection > from GenerateTableFetch > # Set ExecuteSQLRecord fetch size to 1 > # Run flow > What should happen: > * ExecuteSQLRecord should throw an java.sql.SQLSyntaxErrorException: > ORA-01722: invalid number > * Flowfile should get transferred to failure queue > What happens: > * Nifi throws one error and prints another, then puts the flowfile back into > the incoming queue > ** ExecuteSQLRecord throws "Unable to execute SQL select query ... > due to java.io.IOException: java.io.IOException: Could not obtain next record > from ResultSet" > ** Nifi prints errror "transfer relationship not specified; Processor > Administratively Yielded for 1 sec" > ** Flowfile is kept in current queue and penalized and re-processed again > after penalty duration expires > Thus, the problematic query will get executed indefinitely. > > Note: I believe the issue could be due to ResultSet .next() being called > first in the constructor to ResultSetRecordSetWithCallback > Subsequent .next() is called from resultSetWriter.write(recordSet) > [https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73|https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] mattyb149 opened a new pull request #4841: NIFI-8249: Fixed issue with error during multiple FlowFile results in ExecuteSQL processors
mattyb149 opened a new pull request #4841:
URL: https://github.com/apache/nifi/pull/4841
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
If successful ResultSets were written to FlowFiles, they didn't get removed
from the session if an error occurs in subsequent ResultSets.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [x] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically `main`)?
- [x] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [x] Have you written or updated unit tests to verify your changes?
- [x] Have you verified that the full build is successful on JDK 8?
- [ ] Have you verified that the full build is successful on JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for
build issues and submit an update to your PR as soon as possible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (NIFI-8249) ExecuteSQLRecord "administratively yields" and rollls back session when fetch size < query partition size and an SQLException is thrown on subsequent next()
[ https://issues.apache.org/jira/browse/NIFI-8249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289415#comment-17289415 ] Matt Burgess commented on NIFI-8249: Yep finally able to reproduce the issue with your 3-row data, looking into it now. Looks like we're removing the current flowfile from the session but not the ones that have been generated so far. > ExecuteSQLRecord "administratively yields" and rollls back session when fetch > size < query partition size and an SQLException is thrown on subsequent next() > > > Key: NIFI-8249 > URL: https://issues.apache.org/jira/browse/NIFI-8249 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.11.4 >Reporter: David >Assignee: Matt Burgess >Priority: Major > Attachments: ExecuteSQLRecord.png, FlowFile to ESQLRec.png, > GenerateFlowFile.png, GenerateTableFetch.png, Screen Shot 2021-02-23 at > 3.00.47 PM.png, Screen Shot 2021-02-23 at 3.04.55 PM.png > > > When ExecuteSQLRecord executes a query with a partition size > the fetch size > and one of the fetches throws an SQLException, the exception appears to get > "swallowed" and a more general "IOException: Could not obtain next record > from ResultSet: routing to failure" in addition to a "transfer relationship > not specified;Processor Administratively Yielded for 1 sec" errors are thrown. > The problematic query/flowfile gets put back into the incoming queue and will > continue to fail indefinitely until it is manually removed from the queue. > If the fetch size >= to the query partition size, the exact SQL error message > is thrown and the flowfile gets routed to the failure queue. > > Steps to reliably reproduce: > # Create an DB table with an ID column (for partitioning queries against) > and a VARCHAR column > # Insert a row where the VARCHAR column data is a numeric value (ie '123') > # Insert a row where the VARCHAR column data is a string value (ie 'abc') > # Create a GenerateTableFetch processor where the "columns to return" > includes TO_NUMBER() > # Create a ExecuteSQLRecord processor that accepts the "success" connection > from GenerateTableFetch > # Set ExecuteSQLRecord fetch size to 1 > # Run flow > What should happen: > * ExecuteSQLRecord should throw an java.sql.SQLSyntaxErrorException: > ORA-01722: invalid number > * Flowfile should get transferred to failure queue > What happens: > * Nifi throws one error and prints another, then puts the flowfile back into > the incoming queue > ** ExecuteSQLRecord throws "Unable to execute SQL select query ... > due to java.io.IOException: java.io.IOException: Could not obtain next record > from ResultSet" > ** Nifi prints errror "transfer relationship not specified; Processor > Administratively Yielded for 1 sec" > ** Flowfile is kept in current queue and penalized and re-processed again > after penalty duration expires > Thus, the problematic query will get executed indefinitely. > > Note: I believe the issue could be due to ResultSet .next() being called > first in the constructor to ResultSetRecordSetWithCallback > Subsequent .next() is called from resultSetWriter.write(recordSet) > [https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73|https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8249) ExecuteSQLRecord "administratively yields" and rollls back session when fetch size < query partition size and an SQLException is thrown on subsequent next()
[ https://issues.apache.org/jira/browse/NIFI-8249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-8249: --- Affects Version/s: (was: 1.11.4) > ExecuteSQLRecord "administratively yields" and rollls back session when fetch > size < query partition size and an SQLException is thrown on subsequent next() > > > Key: NIFI-8249 > URL: https://issues.apache.org/jira/browse/NIFI-8249 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: David >Assignee: Matt Burgess >Priority: Major > Attachments: ExecuteSQLRecord.png, FlowFile to ESQLRec.png, > GenerateFlowFile.png, GenerateTableFetch.png, Screen Shot 2021-02-23 at > 3.00.47 PM.png, Screen Shot 2021-02-23 at 3.04.55 PM.png > > > When ExecuteSQLRecord executes a query with a partition size > the fetch size > and one of the fetches throws an SQLException, the exception appears to get > "swallowed" and a more general "IOException: Could not obtain next record > from ResultSet: routing to failure" in addition to a "transfer relationship > not specified;Processor Administratively Yielded for 1 sec" errors are thrown. > The problematic query/flowfile gets put back into the incoming queue and will > continue to fail indefinitely until it is manually removed from the queue. > If the fetch size >= to the query partition size, the exact SQL error message > is thrown and the flowfile gets routed to the failure queue. > > Steps to reliably reproduce: > # Create an DB table with an ID column (for partitioning queries against) > and a VARCHAR column > # Insert a row where the VARCHAR column data is a numeric value (ie '123') > # Insert a row where the VARCHAR column data is a string value (ie 'abc') > # Create a GenerateTableFetch processor where the "columns to return" > includes TO_NUMBER() > # Create a ExecuteSQLRecord processor that accepts the "success" connection > from GenerateTableFetch > # Set ExecuteSQLRecord fetch size to 1 > # Run flow > What should happen: > * ExecuteSQLRecord should throw an java.sql.SQLSyntaxErrorException: > ORA-01722: invalid number > * Flowfile should get transferred to failure queue > What happens: > * Nifi throws one error and prints another, then puts the flowfile back into > the incoming queue > ** ExecuteSQLRecord throws "Unable to execute SQL select query ... > due to java.io.IOException: java.io.IOException: Could not obtain next record > from ResultSet" > ** Nifi prints errror "transfer relationship not specified; Processor > Administratively Yielded for 1 sec" > ** Flowfile is kept in current queue and penalized and re-processed again > after penalty duration expires > Thus, the problematic query will get executed indefinitely. > > Note: I believe the issue could be due to ResultSet .next() being called > first in the constructor to ResultSetRecordSetWithCallback > Subsequent .next() is called from resultSetWriter.write(recordSet) > [https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73|https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8119) ExecuteSQL does not properly free database resources
[
https://issues.apache.org/jira/browse/NIFI-8119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess updated NIFI-8119:
---
Summary: ExecuteSQL does not properly free database resources (was:
ExecuteSQL does not properly free database ressources)
> ExecuteSQL does not properly free database resources
>
>
> Key: NIFI-8119
> URL: https://issues.apache.org/jira/browse/NIFI-8119
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Christian Gumpert
>Priority: Minor
> Fix For: 1.14.0, 1.13.1
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> We are using Nifi to ingest data from a Teradata database into our S3-based
> data lake using a typical pattern of GenerateTableFetch and ExecuteSQL
> processors. Our Teradata database tables contain columns of type CLOB (which
> contains some JSON data).
> We have installed the Teradata JDBC driver from the Teradata Tools and
> Utilities package version 16.10.26.00 as described in this [Cloudera
> community
> article|https://community.cloudera.com/t5/Community-Articles/Using-Teradata-JDBC-connector-in-NiFi/ta-p/246783].
> After having configured a DBConnectionPool service with the Teradata
> connection parameters we are able to execute our flow. The GenerateTableFetch
> processors generates flowfiles containing SQL Queries which are then executed
> by the ExecuteSQL processor.
> After having processed the first 15 flowfiles the ExecuteSQL processor yields
> the following error:
> {noformat}
> 2020-12-17T12:53:17+01:00 L921000109090A nifi-app.log: 2020-12-17
> 12:53:11,578 ERROR [Timer-Driven Process Thread-2]
> o.a.nifi.processors.standard.ExecuteSQL
> ExecuteSQL[id=afa23b0f-2e57-1fb6-d047-13646de03ebf] Unable to execute SQL
> select query call devezv_replworkedec.get_edec_meldung(561, 562, 2, 0); for
> StandardFlowFileRecord[uuid=ff7219a7-14e9-404e-a57a-28121653fed8,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1607610786368-4986, container=repo0,
> section=890], offset=701888,
> length=1077672],offset=32266,name=ff7219a7-14e9-404e-a57a-28121653fed8,size=58]
> due to java.sql.SQLException: [Teradata Database] [TeraJDBC 16.10.00.07]
> [Error 3130] [SQLState HY000] GET_EDEC_MELDUNG:Response limit exceeded.;
> routing to failure: java.sql.SQLException: [Teradata Database] [TeraJDBC
> 16.10.00.07] [Error 3130] [SQLState HY000] GET_EDEC_MELDUNG:Response limit
> exceeded.
> java.sql.SQLException: [Teradata Database] [TeraJDBC 16.10.00.07] [Error
> 3130] [SQLState HY000] GET_EDEC_MELDUNG:Response limit exceeded.
> at
> com.teradata.jdbc.jdbc_4.util.ErrorFactory.makeDatabaseSQLException(ErrorFactory.java:309)
> at
> com.teradata.jdbc.jdbc_4.statemachine.ReceiveInitSubState.action(ReceiveInitSubState.java:103)
> at
> com.teradata.jdbc.jdbc_4.statemachine.StatementReceiveState.subStateMachine(StatementReceiveState.java:311)
> at
> com.teradata.jdbc.jdbc_4.statemachine.StatementReceiveState.action(StatementReceiveState.java:200)
> at
> com.teradata.jdbc.jdbc_4.statemachine.StatementController.runBody(StatementController.java:137)
> at
> com.teradata.jdbc.jdbc_4.statemachine.PreparedStatementController.run(PreparedStatementController.java:46)
> at
> com.teradata.jdbc.jdbc_4.TDStatement.executeStatement(TDStatement.java:389)
> at
> com.teradata.jdbc.jdbc_4.TDStatement.executeStatement(TDStatement.java:331)
> at
> com.teradata.jdbc.jdbc_4.TDPreparedStatement.doPrepExecute(TDPreparedStatement.java:177)
> at
> com.teradata.jdbc.jdbc_4.TDPreparedStatement.execute(TDPreparedStatement.java:2778)
> at
> org.apache.commons.dbcp2.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:94)
> at
> org.apache.commons.dbcp2.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:94)
> at
> org.apache.nifi.processors.standard.AbstractExecuteSQL.onTrigger(AbstractExecuteSQL.java:266)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurre
[GitHub] [nifi] patricker commented on pull request #2231: NIFI-4521 MS SQL CDC Processor
patricker commented on pull request #2231: URL: https://github.com/apache/nifi/pull/2231#issuecomment-784553630 @pvillard31 There is one open issue related to very large transaction id's not fitting in BigInt. I have not researched it very much, but am already feeling motivated to check it out if this open and active PR might get merged after 3.5 years... I would LOVE to get this merged. I'll take some time tomorrow to check out this one open issue tomorrow. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] turcsanyip commented on a change in pull request #4837: NIFI-6113 Refactoring to remove instance level variable to make it stateless
turcsanyip commented on a change in pull request #4837:
URL: https://github.com/apache/nifi/pull/4837#discussion_r581422845
##
File path:
nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/GetHDFSFileInfo.java
##
@@ -707,24 +642,24 @@ void finish(ProcessSession session) {
/*
* Keeps all request details in single object.
*/
-static class HDFSFileInfoRequest{
-enum Groupping {
+static class HDFSFileInfoRequest {
+protected enum Grouping {
Review comment:
As the enclosing class is `package private`, I don't think `protected`
really makes sense here. I believe the original was fine. Or do I miss
something?
##
File path:
nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/GetHDFSFileInfo.java
##
@@ -264,8 +263,6 @@ protected void init(final ProcessorInitializationContext
context) {
@Override
public void onPropertyModified(PropertyDescriptor descriptor, String
oldValue, String newValue) {
Review comment:
This method can be deleted now because it does not do anything just
calls the super.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Comment Edited] (NIFI-8249) ExecuteSQLRecord "administratively yields" and rollls back session when fetch size < query partition size and an SQLException is thrown on subsequent next()
[ https://issues.apache.org/jira/browse/NIFI-8249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289388#comment-17289388 ] David edited comment on NIFI-8249 at 2/23/21, 10:12 PM: I am also able to reproduce the same issue using a GenerateFlowFile -> ExecuteSQLRecord flow. ExecuteSQLRecord is using the same settings as previously mentioned (fetch = 1, max rows = 1) GenerateFlowFile custom text = "SELECT id, to_number(data) as converted_data FROM DAVID_NIFI_BUG_TEST WHERE id <= 3 AND id >= 1" Attached a screenshot for that flow as well was (Author: dfritter4): I am also able to reproduce the same issue using a GenerateFlowFile -> ExecuteSQLRecord flow GenerateFlowFile custom text = "SELECT id, to_number(data) as converted_data FROM DAVID_NIFI_BUG_TEST WHERE id <= 3 AND id >= 1" Attached a screenshot for that flow as well > ExecuteSQLRecord "administratively yields" and rollls back session when fetch > size < query partition size and an SQLException is thrown on subsequent next() > > > Key: NIFI-8249 > URL: https://issues.apache.org/jira/browse/NIFI-8249 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.11.4 >Reporter: David >Assignee: Matt Burgess >Priority: Major > Attachments: ExecuteSQLRecord.png, FlowFile to ESQLRec.png, > GenerateFlowFile.png, GenerateTableFetch.png, Screen Shot 2021-02-23 at > 3.00.47 PM.png, Screen Shot 2021-02-23 at 3.04.55 PM.png > > > When ExecuteSQLRecord executes a query with a partition size > the fetch size > and one of the fetches throws an SQLException, the exception appears to get > "swallowed" and a more general "IOException: Could not obtain next record > from ResultSet: routing to failure" in addition to a "transfer relationship > not specified;Processor Administratively Yielded for 1 sec" errors are thrown. > The problematic query/flowfile gets put back into the incoming queue and will > continue to fail indefinitely until it is manually removed from the queue. > If the fetch size >= to the query partition size, the exact SQL error message > is thrown and the flowfile gets routed to the failure queue. > > Steps to reliably reproduce: > # Create an DB table with an ID column (for partitioning queries against) > and a VARCHAR column > # Insert a row where the VARCHAR column data is a numeric value (ie '123') > # Insert a row where the VARCHAR column data is a string value (ie 'abc') > # Create a GenerateTableFetch processor where the "columns to return" > includes TO_NUMBER() > # Create a ExecuteSQLRecord processor that accepts the "success" connection > from GenerateTableFetch > # Set ExecuteSQLRecord fetch size to 1 > # Run flow > What should happen: > * ExecuteSQLRecord should throw an java.sql.SQLSyntaxErrorException: > ORA-01722: invalid number > * Flowfile should get transferred to failure queue > What happens: > * Nifi throws one error and prints another, then puts the flowfile back into > the incoming queue > ** ExecuteSQLRecord throws "Unable to execute SQL select query ... > due to java.io.IOException: java.io.IOException: Could not obtain next record > from ResultSet" > ** Nifi prints errror "transfer relationship not specified; Processor > Administratively Yielded for 1 sec" > ** Flowfile is kept in current queue and penalized and re-processed again > after penalty duration expires > Thus, the problematic query will get executed indefinitely. > > Note: I believe the issue could be due to ResultSet .next() being called > first in the constructor to ResultSetRecordSetWithCallback > Subsequent .next() is called from resultSetWriter.write(recordSet) > [https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73|https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8249) ExecuteSQLRecord "administratively yields" and rollls back session when fetch size < query partition size and an SQLException is thrown on subsequent next()
[ https://issues.apache.org/jira/browse/NIFI-8249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David updated NIFI-8249: Attachment: GenerateFlowFile.png > ExecuteSQLRecord "administratively yields" and rollls back session when fetch > size < query partition size and an SQLException is thrown on subsequent next() > > > Key: NIFI-8249 > URL: https://issues.apache.org/jira/browse/NIFI-8249 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.11.4 >Reporter: David >Assignee: Matt Burgess >Priority: Major > Attachments: ExecuteSQLRecord.png, FlowFile to ESQLRec.png, > GenerateFlowFile.png, GenerateTableFetch.png, Screen Shot 2021-02-23 at > 3.00.47 PM.png, Screen Shot 2021-02-23 at 3.04.55 PM.png > > > When ExecuteSQLRecord executes a query with a partition size > the fetch size > and one of the fetches throws an SQLException, the exception appears to get > "swallowed" and a more general "IOException: Could not obtain next record > from ResultSet: routing to failure" in addition to a "transfer relationship > not specified;Processor Administratively Yielded for 1 sec" errors are thrown. > The problematic query/flowfile gets put back into the incoming queue and will > continue to fail indefinitely until it is manually removed from the queue. > If the fetch size >= to the query partition size, the exact SQL error message > is thrown and the flowfile gets routed to the failure queue. > > Steps to reliably reproduce: > # Create an DB table with an ID column (for partitioning queries against) > and a VARCHAR column > # Insert a row where the VARCHAR column data is a numeric value (ie '123') > # Insert a row where the VARCHAR column data is a string value (ie 'abc') > # Create a GenerateTableFetch processor where the "columns to return" > includes TO_NUMBER() > # Create a ExecuteSQLRecord processor that accepts the "success" connection > from GenerateTableFetch > # Set ExecuteSQLRecord fetch size to 1 > # Run flow > What should happen: > * ExecuteSQLRecord should throw an java.sql.SQLSyntaxErrorException: > ORA-01722: invalid number > * Flowfile should get transferred to failure queue > What happens: > * Nifi throws one error and prints another, then puts the flowfile back into > the incoming queue > ** ExecuteSQLRecord throws "Unable to execute SQL select query ... > due to java.io.IOException: java.io.IOException: Could not obtain next record > from ResultSet" > ** Nifi prints errror "transfer relationship not specified; Processor > Administratively Yielded for 1 sec" > ** Flowfile is kept in current queue and penalized and re-processed again > after penalty duration expires > Thus, the problematic query will get executed indefinitely. > > Note: I believe the issue could be due to ResultSet .next() being called > first in the constructor to ResultSetRecordSetWithCallback > Subsequent .next() is called from resultSetWriter.write(recordSet) > [https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73|https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8249) ExecuteSQLRecord "administratively yields" and rollls back session when fetch size < query partition size and an SQLException is thrown on subsequent next()
[ https://issues.apache.org/jira/browse/NIFI-8249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289388#comment-17289388 ] David commented on NIFI-8249: - I am also able to reproduce the same issue using a GenerateFlowFile -> ExecuteSQLRecord flow GenerateFlowFile custom text = "SELECT id, to_number(data) as converted_data FROM DAVID_NIFI_BUG_TEST WHERE id <= 3 AND id >= 1" Attached a screenshot for that flow as well > ExecuteSQLRecord "administratively yields" and rollls back session when fetch > size < query partition size and an SQLException is thrown on subsequent next() > > > Key: NIFI-8249 > URL: https://issues.apache.org/jira/browse/NIFI-8249 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.11.4 >Reporter: David >Assignee: Matt Burgess >Priority: Major > Attachments: ExecuteSQLRecord.png, FlowFile to ESQLRec.png, > GenerateTableFetch.png, Screen Shot 2021-02-23 at 3.00.47 PM.png, Screen Shot > 2021-02-23 at 3.04.55 PM.png > > > When ExecuteSQLRecord executes a query with a partition size > the fetch size > and one of the fetches throws an SQLException, the exception appears to get > "swallowed" and a more general "IOException: Could not obtain next record > from ResultSet: routing to failure" in addition to a "transfer relationship > not specified;Processor Administratively Yielded for 1 sec" errors are thrown. > The problematic query/flowfile gets put back into the incoming queue and will > continue to fail indefinitely until it is manually removed from the queue. > If the fetch size >= to the query partition size, the exact SQL error message > is thrown and the flowfile gets routed to the failure queue. > > Steps to reliably reproduce: > # Create an DB table with an ID column (for partitioning queries against) > and a VARCHAR column > # Insert a row where the VARCHAR column data is a numeric value (ie '123') > # Insert a row where the VARCHAR column data is a string value (ie 'abc') > # Create a GenerateTableFetch processor where the "columns to return" > includes TO_NUMBER() > # Create a ExecuteSQLRecord processor that accepts the "success" connection > from GenerateTableFetch > # Set ExecuteSQLRecord fetch size to 1 > # Run flow > What should happen: > * ExecuteSQLRecord should throw an java.sql.SQLSyntaxErrorException: > ORA-01722: invalid number > * Flowfile should get transferred to failure queue > What happens: > * Nifi throws one error and prints another, then puts the flowfile back into > the incoming queue > ** ExecuteSQLRecord throws "Unable to execute SQL select query ... > due to java.io.IOException: java.io.IOException: Could not obtain next record > from ResultSet" > ** Nifi prints errror "transfer relationship not specified; Processor > Administratively Yielded for 1 sec" > ** Flowfile is kept in current queue and penalized and re-processed again > after penalty duration expires > Thus, the problematic query will get executed indefinitely. > > Note: I believe the issue could be due to ResultSet .next() being called > first in the constructor to ResultSetRecordSetWithCallback > Subsequent .next() is called from resultSetWriter.write(recordSet) > [https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73|https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8249) ExecuteSQLRecord "administratively yields" and rollls back session when fetch size < query partition size and an SQLException is thrown on subsequent next()
[ https://issues.apache.org/jira/browse/NIFI-8249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David updated NIFI-8249: Attachment: FlowFile to ESQLRec.png > ExecuteSQLRecord "administratively yields" and rollls back session when fetch > size < query partition size and an SQLException is thrown on subsequent next() > > > Key: NIFI-8249 > URL: https://issues.apache.org/jira/browse/NIFI-8249 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.11.4 >Reporter: David >Assignee: Matt Burgess >Priority: Major > Attachments: ExecuteSQLRecord.png, FlowFile to ESQLRec.png, > GenerateTableFetch.png, Screen Shot 2021-02-23 at 3.00.47 PM.png, Screen Shot > 2021-02-23 at 3.04.55 PM.png > > > When ExecuteSQLRecord executes a query with a partition size > the fetch size > and one of the fetches throws an SQLException, the exception appears to get > "swallowed" and a more general "IOException: Could not obtain next record > from ResultSet: routing to failure" in addition to a "transfer relationship > not specified;Processor Administratively Yielded for 1 sec" errors are thrown. > The problematic query/flowfile gets put back into the incoming queue and will > continue to fail indefinitely until it is manually removed from the queue. > If the fetch size >= to the query partition size, the exact SQL error message > is thrown and the flowfile gets routed to the failure queue. > > Steps to reliably reproduce: > # Create an DB table with an ID column (for partitioning queries against) > and a VARCHAR column > # Insert a row where the VARCHAR column data is a numeric value (ie '123') > # Insert a row where the VARCHAR column data is a string value (ie 'abc') > # Create a GenerateTableFetch processor where the "columns to return" > includes TO_NUMBER() > # Create a ExecuteSQLRecord processor that accepts the "success" connection > from GenerateTableFetch > # Set ExecuteSQLRecord fetch size to 1 > # Run flow > What should happen: > * ExecuteSQLRecord should throw an java.sql.SQLSyntaxErrorException: > ORA-01722: invalid number > * Flowfile should get transferred to failure queue > What happens: > * Nifi throws one error and prints another, then puts the flowfile back into > the incoming queue > ** ExecuteSQLRecord throws "Unable to execute SQL select query ... > due to java.io.IOException: java.io.IOException: Could not obtain next record > from ResultSet" > ** Nifi prints errror "transfer relationship not specified; Processor > Administratively Yielded for 1 sec" > ** Flowfile is kept in current queue and penalized and re-processed again > after penalty duration expires > Thus, the problematic query will get executed indefinitely. > > Note: I believe the issue could be due to ResultSet .next() being called > first in the constructor to ResultSetRecordSetWithCallback > Subsequent .next() is called from resultSetWriter.write(recordSet) > [https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73|https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8249) ExecuteSQLRecord "administratively yields" and rollls back session when fetch size < query partition size and an SQLException is thrown on subsequent next()
[ https://issues.apache.org/jira/browse/NIFI-8249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289377#comment-17289377 ] David commented on NIFI-8249: - One last thing: did you try adding a 3rd row to your test table? When I tried with just two rows I wasn't able to reproduce with fetch =1 max rows = 1. But when I added the 3rd row (so the problem row is ID =3, IDs 1 and 2 have a numeric value in the varchar) I see this issue. ||ID||DATA|| |1|'123'| |2|'456'| |3|'apple'| > ExecuteSQLRecord "administratively yields" and rollls back session when fetch > size < query partition size and an SQLException is thrown on subsequent next() > > > Key: NIFI-8249 > URL: https://issues.apache.org/jira/browse/NIFI-8249 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.11.4 >Reporter: David >Assignee: Matt Burgess >Priority: Major > Attachments: ExecuteSQLRecord.png, GenerateTableFetch.png, Screen > Shot 2021-02-23 at 3.00.47 PM.png, Screen Shot 2021-02-23 at 3.04.55 PM.png > > > When ExecuteSQLRecord executes a query with a partition size > the fetch size > and one of the fetches throws an SQLException, the exception appears to get > "swallowed" and a more general "IOException: Could not obtain next record > from ResultSet: routing to failure" in addition to a "transfer relationship > not specified;Processor Administratively Yielded for 1 sec" errors are thrown. > The problematic query/flowfile gets put back into the incoming queue and will > continue to fail indefinitely until it is manually removed from the queue. > If the fetch size >= to the query partition size, the exact SQL error message > is thrown and the flowfile gets routed to the failure queue. > > Steps to reliably reproduce: > # Create an DB table with an ID column (for partitioning queries against) > and a VARCHAR column > # Insert a row where the VARCHAR column data is a numeric value (ie '123') > # Insert a row where the VARCHAR column data is a string value (ie 'abc') > # Create a GenerateTableFetch processor where the "columns to return" > includes TO_NUMBER() > # Create a ExecuteSQLRecord processor that accepts the "success" connection > from GenerateTableFetch > # Set ExecuteSQLRecord fetch size to 1 > # Run flow > What should happen: > * ExecuteSQLRecord should throw an java.sql.SQLSyntaxErrorException: > ORA-01722: invalid number > * Flowfile should get transferred to failure queue > What happens: > * Nifi throws one error and prints another, then puts the flowfile back into > the incoming queue > ** ExecuteSQLRecord throws "Unable to execute SQL select query ... > due to java.io.IOException: java.io.IOException: Could not obtain next record > from ResultSet" > ** Nifi prints errror "transfer relationship not specified; Processor > Administratively Yielded for 1 sec" > ** Flowfile is kept in current queue and penalized and re-processed again > after penalty duration expires > Thus, the problematic query will get executed indefinitely. > > Note: I believe the issue could be due to ResultSet .next() being called > first in the constructor to ResultSetRecordSetWithCallback > Subsequent .next() is called from resultSetWriter.write(recordSet) > [https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73|https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8249) ExecuteSQLRecord "administratively yields" and rollls back session when fetch size < query partition size and an SQLException is thrown on subsequent next()
[ https://issues.apache.org/jira/browse/NIFI-8249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289376#comment-17289376 ] Matt Burgess commented on NIFI-8249: I'll give it one more shot with exactly the same table definitions and such, maybe I missed some setting you're using. > ExecuteSQLRecord "administratively yields" and rollls back session when fetch > size < query partition size and an SQLException is thrown on subsequent next() > > > Key: NIFI-8249 > URL: https://issues.apache.org/jira/browse/NIFI-8249 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.11.4 >Reporter: David >Assignee: Matt Burgess >Priority: Major > Attachments: ExecuteSQLRecord.png, GenerateTableFetch.png, Screen > Shot 2021-02-23 at 3.00.47 PM.png, Screen Shot 2021-02-23 at 3.04.55 PM.png > > > When ExecuteSQLRecord executes a query with a partition size > the fetch size > and one of the fetches throws an SQLException, the exception appears to get > "swallowed" and a more general "IOException: Could not obtain next record > from ResultSet: routing to failure" in addition to a "transfer relationship > not specified;Processor Administratively Yielded for 1 sec" errors are thrown. > The problematic query/flowfile gets put back into the incoming queue and will > continue to fail indefinitely until it is manually removed from the queue. > If the fetch size >= to the query partition size, the exact SQL error message > is thrown and the flowfile gets routed to the failure queue. > > Steps to reliably reproduce: > # Create an DB table with an ID column (for partitioning queries against) > and a VARCHAR column > # Insert a row where the VARCHAR column data is a numeric value (ie '123') > # Insert a row where the VARCHAR column data is a string value (ie 'abc') > # Create a GenerateTableFetch processor where the "columns to return" > includes TO_NUMBER() > # Create a ExecuteSQLRecord processor that accepts the "success" connection > from GenerateTableFetch > # Set ExecuteSQLRecord fetch size to 1 > # Run flow > What should happen: > * ExecuteSQLRecord should throw an java.sql.SQLSyntaxErrorException: > ORA-01722: invalid number > * Flowfile should get transferred to failure queue > What happens: > * Nifi throws one error and prints another, then puts the flowfile back into > the incoming queue > ** ExecuteSQLRecord throws "Unable to execute SQL select query ... > due to java.io.IOException: java.io.IOException: Could not obtain next record > from ResultSet" > ** Nifi prints errror "transfer relationship not specified; Processor > Administratively Yielded for 1 sec" > ** Flowfile is kept in current queue and penalized and re-processed again > after penalty duration expires > Thus, the problematic query will get executed indefinitely. > > Note: I believe the issue could be due to ResultSet .next() being called > first in the constructor to ResultSetRecordSetWithCallback > Subsequent .next() is called from resultSetWriter.write(recordSet) > [https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73|https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8249) ExecuteSQLRecord "administratively yields" and rollls back session when fetch size < query partition size and an SQLException is thrown on subsequent next()
[ https://issues.apache.org/jira/browse/NIFI-8249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289373#comment-17289373 ] Matt Burgess commented on NIFI-8249: The "transfer relationship not specified" comes from the framework (probably StandardProcessSession) and usually happens when an exception jumps the code out of a block before the flowfile has been transferred or removed. Correct handling should catch such exceptions and handle appropriately, either rolling back the session, removing the flowfile, or transferring the flowfile to a relationship (usually "failure" or "retry"). > ExecuteSQLRecord "administratively yields" and rollls back session when fetch > size < query partition size and an SQLException is thrown on subsequent next() > > > Key: NIFI-8249 > URL: https://issues.apache.org/jira/browse/NIFI-8249 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.11.4 >Reporter: David >Assignee: Matt Burgess >Priority: Major > Attachments: ExecuteSQLRecord.png, GenerateTableFetch.png, Screen > Shot 2021-02-23 at 3.00.47 PM.png, Screen Shot 2021-02-23 at 3.04.55 PM.png > > > When ExecuteSQLRecord executes a query with a partition size > the fetch size > and one of the fetches throws an SQLException, the exception appears to get > "swallowed" and a more general "IOException: Could not obtain next record > from ResultSet: routing to failure" in addition to a "transfer relationship > not specified;Processor Administratively Yielded for 1 sec" errors are thrown. > The problematic query/flowfile gets put back into the incoming queue and will > continue to fail indefinitely until it is manually removed from the queue. > If the fetch size >= to the query partition size, the exact SQL error message > is thrown and the flowfile gets routed to the failure queue. > > Steps to reliably reproduce: > # Create an DB table with an ID column (for partitioning queries against) > and a VARCHAR column > # Insert a row where the VARCHAR column data is a numeric value (ie '123') > # Insert a row where the VARCHAR column data is a string value (ie 'abc') > # Create a GenerateTableFetch processor where the "columns to return" > includes TO_NUMBER() > # Create a ExecuteSQLRecord processor that accepts the "success" connection > from GenerateTableFetch > # Set ExecuteSQLRecord fetch size to 1 > # Run flow > What should happen: > * ExecuteSQLRecord should throw an java.sql.SQLSyntaxErrorException: > ORA-01722: invalid number > * Flowfile should get transferred to failure queue > What happens: > * Nifi throws one error and prints another, then puts the flowfile back into > the incoming queue > ** ExecuteSQLRecord throws "Unable to execute SQL select query ... > due to java.io.IOException: java.io.IOException: Could not obtain next record > from ResultSet" > ** Nifi prints errror "transfer relationship not specified; Processor > Administratively Yielded for 1 sec" > ** Flowfile is kept in current queue and penalized and re-processed again > after penalty duration expires > Thus, the problematic query will get executed indefinitely. > > Note: I believe the issue could be due to ResultSet .next() being called > first in the constructor to ResultSetRecordSetWithCallback > Subsequent .next() is called from resultSetWriter.write(recordSet) > [https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73|https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8119) ExecuteSQL does not properly free database ressources
[
https://issues.apache.org/jira/browse/NIFI-8119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe Witt updated NIFI-8119:
---
Fix Version/s: 1.13.1
> ExecuteSQL does not properly free database ressources
> -
>
> Key: NIFI-8119
> URL: https://issues.apache.org/jira/browse/NIFI-8119
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Christian Gumpert
>Priority: Minor
> Fix For: 1.14.0, 1.13.1
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> We are using Nifi to ingest data from a Teradata database into our S3-based
> data lake using a typical pattern of GenerateTableFetch and ExecuteSQL
> processors. Our Teradata database tables contain columns of type CLOB (which
> contains some JSON data).
> We have installed the Teradata JDBC driver from the Teradata Tools and
> Utilities package version 16.10.26.00 as described in this [Cloudera
> community
> article|https://community.cloudera.com/t5/Community-Articles/Using-Teradata-JDBC-connector-in-NiFi/ta-p/246783].
> After having configured a DBConnectionPool service with the Teradata
> connection parameters we are able to execute our flow. The GenerateTableFetch
> processors generates flowfiles containing SQL Queries which are then executed
> by the ExecuteSQL processor.
> After having processed the first 15 flowfiles the ExecuteSQL processor yields
> the following error:
> {noformat}
> 2020-12-17T12:53:17+01:00 L921000109090A nifi-app.log: 2020-12-17
> 12:53:11,578 ERROR [Timer-Driven Process Thread-2]
> o.a.nifi.processors.standard.ExecuteSQL
> ExecuteSQL[id=afa23b0f-2e57-1fb6-d047-13646de03ebf] Unable to execute SQL
> select query call devezv_replworkedec.get_edec_meldung(561, 562, 2, 0); for
> StandardFlowFileRecord[uuid=ff7219a7-14e9-404e-a57a-28121653fed8,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1607610786368-4986, container=repo0,
> section=890], offset=701888,
> length=1077672],offset=32266,name=ff7219a7-14e9-404e-a57a-28121653fed8,size=58]
> due to java.sql.SQLException: [Teradata Database] [TeraJDBC 16.10.00.07]
> [Error 3130] [SQLState HY000] GET_EDEC_MELDUNG:Response limit exceeded.;
> routing to failure: java.sql.SQLException: [Teradata Database] [TeraJDBC
> 16.10.00.07] [Error 3130] [SQLState HY000] GET_EDEC_MELDUNG:Response limit
> exceeded.
> java.sql.SQLException: [Teradata Database] [TeraJDBC 16.10.00.07] [Error
> 3130] [SQLState HY000] GET_EDEC_MELDUNG:Response limit exceeded.
> at
> com.teradata.jdbc.jdbc_4.util.ErrorFactory.makeDatabaseSQLException(ErrorFactory.java:309)
> at
> com.teradata.jdbc.jdbc_4.statemachine.ReceiveInitSubState.action(ReceiveInitSubState.java:103)
> at
> com.teradata.jdbc.jdbc_4.statemachine.StatementReceiveState.subStateMachine(StatementReceiveState.java:311)
> at
> com.teradata.jdbc.jdbc_4.statemachine.StatementReceiveState.action(StatementReceiveState.java:200)
> at
> com.teradata.jdbc.jdbc_4.statemachine.StatementController.runBody(StatementController.java:137)
> at
> com.teradata.jdbc.jdbc_4.statemachine.PreparedStatementController.run(PreparedStatementController.java:46)
> at
> com.teradata.jdbc.jdbc_4.TDStatement.executeStatement(TDStatement.java:389)
> at
> com.teradata.jdbc.jdbc_4.TDStatement.executeStatement(TDStatement.java:331)
> at
> com.teradata.jdbc.jdbc_4.TDPreparedStatement.doPrepExecute(TDPreparedStatement.java:177)
> at
> com.teradata.jdbc.jdbc_4.TDPreparedStatement.execute(TDPreparedStatement.java:2778)
> at
> org.apache.commons.dbcp2.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:94)
> at
> org.apache.commons.dbcp2.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:94)
> at
> org.apache.nifi.processors.standard.AbstractExecuteSQL.onTrigger(AbstractExecuteSQL.java:266)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> ja
[jira] [Updated] (NIFI-8119) ExecuteSQL does not properly free database ressources
[
https://issues.apache.org/jira/browse/NIFI-8119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess updated NIFI-8119:
---
Fix Version/s: 1.14.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> ExecuteSQL does not properly free database ressources
> -
>
> Key: NIFI-8119
> URL: https://issues.apache.org/jira/browse/NIFI-8119
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Christian Gumpert
>Priority: Minor
> Fix For: 1.14.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> We are using Nifi to ingest data from a Teradata database into our S3-based
> data lake using a typical pattern of GenerateTableFetch and ExecuteSQL
> processors. Our Teradata database tables contain columns of type CLOB (which
> contains some JSON data).
> We have installed the Teradata JDBC driver from the Teradata Tools and
> Utilities package version 16.10.26.00 as described in this [Cloudera
> community
> article|https://community.cloudera.com/t5/Community-Articles/Using-Teradata-JDBC-connector-in-NiFi/ta-p/246783].
> After having configured a DBConnectionPool service with the Teradata
> connection parameters we are able to execute our flow. The GenerateTableFetch
> processors generates flowfiles containing SQL Queries which are then executed
> by the ExecuteSQL processor.
> After having processed the first 15 flowfiles the ExecuteSQL processor yields
> the following error:
> {noformat}
> 2020-12-17T12:53:17+01:00 L921000109090A nifi-app.log: 2020-12-17
> 12:53:11,578 ERROR [Timer-Driven Process Thread-2]
> o.a.nifi.processors.standard.ExecuteSQL
> ExecuteSQL[id=afa23b0f-2e57-1fb6-d047-13646de03ebf] Unable to execute SQL
> select query call devezv_replworkedec.get_edec_meldung(561, 562, 2, 0); for
> StandardFlowFileRecord[uuid=ff7219a7-14e9-404e-a57a-28121653fed8,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1607610786368-4986, container=repo0,
> section=890], offset=701888,
> length=1077672],offset=32266,name=ff7219a7-14e9-404e-a57a-28121653fed8,size=58]
> due to java.sql.SQLException: [Teradata Database] [TeraJDBC 16.10.00.07]
> [Error 3130] [SQLState HY000] GET_EDEC_MELDUNG:Response limit exceeded.;
> routing to failure: java.sql.SQLException: [Teradata Database] [TeraJDBC
> 16.10.00.07] [Error 3130] [SQLState HY000] GET_EDEC_MELDUNG:Response limit
> exceeded.
> java.sql.SQLException: [Teradata Database] [TeraJDBC 16.10.00.07] [Error
> 3130] [SQLState HY000] GET_EDEC_MELDUNG:Response limit exceeded.
> at
> com.teradata.jdbc.jdbc_4.util.ErrorFactory.makeDatabaseSQLException(ErrorFactory.java:309)
> at
> com.teradata.jdbc.jdbc_4.statemachine.ReceiveInitSubState.action(ReceiveInitSubState.java:103)
> at
> com.teradata.jdbc.jdbc_4.statemachine.StatementReceiveState.subStateMachine(StatementReceiveState.java:311)
> at
> com.teradata.jdbc.jdbc_4.statemachine.StatementReceiveState.action(StatementReceiveState.java:200)
> at
> com.teradata.jdbc.jdbc_4.statemachine.StatementController.runBody(StatementController.java:137)
> at
> com.teradata.jdbc.jdbc_4.statemachine.PreparedStatementController.run(PreparedStatementController.java:46)
> at
> com.teradata.jdbc.jdbc_4.TDStatement.executeStatement(TDStatement.java:389)
> at
> com.teradata.jdbc.jdbc_4.TDStatement.executeStatement(TDStatement.java:331)
> at
> com.teradata.jdbc.jdbc_4.TDPreparedStatement.doPrepExecute(TDPreparedStatement.java:177)
> at
> com.teradata.jdbc.jdbc_4.TDPreparedStatement.execute(TDPreparedStatement.java:2778)
> at
> org.apache.commons.dbcp2.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:94)
> at
> org.apache.commons.dbcp2.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:94)
> at
> org.apache.nifi.processors.standard.AbstractExecuteSQL.onTrigger(AbstractExecuteSQL.java:266)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$S
[jira] [Updated] (NIFI-8119) ExecuteSQL does not properly free database ressources
[
https://issues.apache.org/jira/browse/NIFI-8119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess updated NIFI-8119:
---
Affects Version/s: (was: 1.11.2)
Status: Patch Available (was: Open)
> ExecuteSQL does not properly free database ressources
> -
>
> Key: NIFI-8119
> URL: https://issues.apache.org/jira/browse/NIFI-8119
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Christian Gumpert
>Priority: Minor
> Time Spent: 40m
> Remaining Estimate: 0h
>
> We are using Nifi to ingest data from a Teradata database into our S3-based
> data lake using a typical pattern of GenerateTableFetch and ExecuteSQL
> processors. Our Teradata database tables contain columns of type CLOB (which
> contains some JSON data).
> We have installed the Teradata JDBC driver from the Teradata Tools and
> Utilities package version 16.10.26.00 as described in this [Cloudera
> community
> article|https://community.cloudera.com/t5/Community-Articles/Using-Teradata-JDBC-connector-in-NiFi/ta-p/246783].
> After having configured a DBConnectionPool service with the Teradata
> connection parameters we are able to execute our flow. The GenerateTableFetch
> processors generates flowfiles containing SQL Queries which are then executed
> by the ExecuteSQL processor.
> After having processed the first 15 flowfiles the ExecuteSQL processor yields
> the following error:
> {noformat}
> 2020-12-17T12:53:17+01:00 L921000109090A nifi-app.log: 2020-12-17
> 12:53:11,578 ERROR [Timer-Driven Process Thread-2]
> o.a.nifi.processors.standard.ExecuteSQL
> ExecuteSQL[id=afa23b0f-2e57-1fb6-d047-13646de03ebf] Unable to execute SQL
> select query call devezv_replworkedec.get_edec_meldung(561, 562, 2, 0); for
> StandardFlowFileRecord[uuid=ff7219a7-14e9-404e-a57a-28121653fed8,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1607610786368-4986, container=repo0,
> section=890], offset=701888,
> length=1077672],offset=32266,name=ff7219a7-14e9-404e-a57a-28121653fed8,size=58]
> due to java.sql.SQLException: [Teradata Database] [TeraJDBC 16.10.00.07]
> [Error 3130] [SQLState HY000] GET_EDEC_MELDUNG:Response limit exceeded.;
> routing to failure: java.sql.SQLException: [Teradata Database] [TeraJDBC
> 16.10.00.07] [Error 3130] [SQLState HY000] GET_EDEC_MELDUNG:Response limit
> exceeded.
> java.sql.SQLException: [Teradata Database] [TeraJDBC 16.10.00.07] [Error
> 3130] [SQLState HY000] GET_EDEC_MELDUNG:Response limit exceeded.
> at
> com.teradata.jdbc.jdbc_4.util.ErrorFactory.makeDatabaseSQLException(ErrorFactory.java:309)
> at
> com.teradata.jdbc.jdbc_4.statemachine.ReceiveInitSubState.action(ReceiveInitSubState.java:103)
> at
> com.teradata.jdbc.jdbc_4.statemachine.StatementReceiveState.subStateMachine(StatementReceiveState.java:311)
> at
> com.teradata.jdbc.jdbc_4.statemachine.StatementReceiveState.action(StatementReceiveState.java:200)
> at
> com.teradata.jdbc.jdbc_4.statemachine.StatementController.runBody(StatementController.java:137)
> at
> com.teradata.jdbc.jdbc_4.statemachine.PreparedStatementController.run(PreparedStatementController.java:46)
> at
> com.teradata.jdbc.jdbc_4.TDStatement.executeStatement(TDStatement.java:389)
> at
> com.teradata.jdbc.jdbc_4.TDStatement.executeStatement(TDStatement.java:331)
> at
> com.teradata.jdbc.jdbc_4.TDPreparedStatement.doPrepExecute(TDPreparedStatement.java:177)
> at
> com.teradata.jdbc.jdbc_4.TDPreparedStatement.execute(TDPreparedStatement.java:2778)
> at
> org.apache.commons.dbcp2.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:94)
> at
> org.apache.commons.dbcp2.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:94)
> at
> org.apache.nifi.processors.standard.AbstractExecuteSQL.onTrigger(AbstractExecuteSQL.java:266)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPool
[jira] [Commented] (NIFI-8119) ExecuteSQL does not properly free database ressources
[
https://issues.apache.org/jira/browse/NIFI-8119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289368#comment-17289368
]
ASF subversion and git services commented on NIFI-8119:
---
Commit f2555f27f18bceab7ca1b35905721ff668e8e4f5 in nifi's branch
refs/heads/main from Christian Gumpert
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=f2555f2 ]
NIFI-8119: properly free database resources when done with the processing of
BLOBs
Signed-off-by: Matthew Burgess
This closes #4791
> ExecuteSQL does not properly free database ressources
> -
>
> Key: NIFI-8119
> URL: https://issues.apache.org/jira/browse/NIFI-8119
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.11.2
>Reporter: Christian Gumpert
>Priority: Minor
> Time Spent: 40m
> Remaining Estimate: 0h
>
> We are using Nifi to ingest data from a Teradata database into our S3-based
> data lake using a typical pattern of GenerateTableFetch and ExecuteSQL
> processors. Our Teradata database tables contain columns of type CLOB (which
> contains some JSON data).
> We have installed the Teradata JDBC driver from the Teradata Tools and
> Utilities package version 16.10.26.00 as described in this [Cloudera
> community
> article|https://community.cloudera.com/t5/Community-Articles/Using-Teradata-JDBC-connector-in-NiFi/ta-p/246783].
> After having configured a DBConnectionPool service with the Teradata
> connection parameters we are able to execute our flow. The GenerateTableFetch
> processors generates flowfiles containing SQL Queries which are then executed
> by the ExecuteSQL processor.
> After having processed the first 15 flowfiles the ExecuteSQL processor yields
> the following error:
> {noformat}
> 2020-12-17T12:53:17+01:00 L921000109090A nifi-app.log: 2020-12-17
> 12:53:11,578 ERROR [Timer-Driven Process Thread-2]
> o.a.nifi.processors.standard.ExecuteSQL
> ExecuteSQL[id=afa23b0f-2e57-1fb6-d047-13646de03ebf] Unable to execute SQL
> select query call devezv_replworkedec.get_edec_meldung(561, 562, 2, 0); for
> StandardFlowFileRecord[uuid=ff7219a7-14e9-404e-a57a-28121653fed8,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1607610786368-4986, container=repo0,
> section=890], offset=701888,
> length=1077672],offset=32266,name=ff7219a7-14e9-404e-a57a-28121653fed8,size=58]
> due to java.sql.SQLException: [Teradata Database] [TeraJDBC 16.10.00.07]
> [Error 3130] [SQLState HY000] GET_EDEC_MELDUNG:Response limit exceeded.;
> routing to failure: java.sql.SQLException: [Teradata Database] [TeraJDBC
> 16.10.00.07] [Error 3130] [SQLState HY000] GET_EDEC_MELDUNG:Response limit
> exceeded.
> java.sql.SQLException: [Teradata Database] [TeraJDBC 16.10.00.07] [Error
> 3130] [SQLState HY000] GET_EDEC_MELDUNG:Response limit exceeded.
> at
> com.teradata.jdbc.jdbc_4.util.ErrorFactory.makeDatabaseSQLException(ErrorFactory.java:309)
> at
> com.teradata.jdbc.jdbc_4.statemachine.ReceiveInitSubState.action(ReceiveInitSubState.java:103)
> at
> com.teradata.jdbc.jdbc_4.statemachine.StatementReceiveState.subStateMachine(StatementReceiveState.java:311)
> at
> com.teradata.jdbc.jdbc_4.statemachine.StatementReceiveState.action(StatementReceiveState.java:200)
> at
> com.teradata.jdbc.jdbc_4.statemachine.StatementController.runBody(StatementController.java:137)
> at
> com.teradata.jdbc.jdbc_4.statemachine.PreparedStatementController.run(PreparedStatementController.java:46)
> at
> com.teradata.jdbc.jdbc_4.TDStatement.executeStatement(TDStatement.java:389)
> at
> com.teradata.jdbc.jdbc_4.TDStatement.executeStatement(TDStatement.java:331)
> at
> com.teradata.jdbc.jdbc_4.TDPreparedStatement.doPrepExecute(TDPreparedStatement.java:177)
> at
> com.teradata.jdbc.jdbc_4.TDPreparedStatement.execute(TDPreparedStatement.java:2778)
> at
> org.apache.commons.dbcp2.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:94)
> at
> org.apache.commons.dbcp2.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:94)
> at
> org.apache.nifi.processors.standard.AbstractExecuteSQL.onTrigger(AbstractExecuteSQL.java:266)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at
> java.util.concurrent.Executors$RunnableAdapter.ca
[GitHub] [nifi] mattyb149 closed pull request #4791: NIFI-8119: properly free database resources when done with the processing of BLOBs
mattyb149 closed pull request #4791: URL: https://github.com/apache/nifi/pull/4791 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] mattyb149 commented on pull request #4791: NIFI-8119: properly free database resources when done with the processing of BLOBs
mattyb149 commented on pull request #4791: URL: https://github.com/apache/nifi/pull/4791#issuecomment-784536060 +1 LGTM, I looked into unit tests but Derby makes BLOB stuff very painful. Since all InputStreams should be closed when processing is finished, I'm good with this as-is. Thanks for the improvement! Merging to main This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-8254) Timestamp and enums in Avro schema not working in PutDatabaseRecord on Oracle
[
https://issues.apache.org/jira/browse/NIFI-8254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Keller updated NIFI-8254:
--
Description:
eThere appears to be a regression in the PutDatabaseRecord processor with
1.13.0 (works in 1.12.1)
I have an Avro schema for a record which contained enum fields and timestamps.
After upgrading, a processor which had been working was failing on errors on
both of these field types. One stated it could not find the record type, the
other with the exception and stack trace below. The fields in question were
defined as below. The files which were erroring out were the exact same files,
and no changes had been made to the schema or the CSV reader controller between
the tests. I was also able to successfully move the flow.xml.gz file back to
the older version of NiFi and the PutDatabaseRecord processor was able to work
again to insert the database records.
{noformat}
{
"name": "PER_ORG",
"type": {
"type": "enum",
"name": "PerOrgFlag",
"symbols": [
"EMP",
"CWR"
]
}
},{
"name": "UPD_BT_DTM",
"type": {
"type": "long",
"logicalType": "timestamp-millis"
}
},
{noformat}
{noformat}
2021-02-23 11:44:26,496 ERROR [Timer-Driven Process Thread-3]
o.a.n.p.standard.PutDatabaseRecord
PutDatabaseRecord[id=93b478c8-0177-1000-1ca3-59b4189b1ecb] Failed to put
Records to database for
StandardFlowFileRecord[uuid=fd4e3be1-8f7b-4a7a-bf45-2e98627f732a,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1614107687354-5, container=default,
section=5], offset=682549,
length=6820],offset=0,name=DDODS_DVCMP_PS_JOB_2021-02-09_021816_2648.dat,size=6820].
Routing to failure.: java.sql.BatchUpdateException: ORA-00932: inconsistent
datatypes: expected NUMBER got TIMESTAMP
java.sql.BatchUpdateException: ORA-00932: inconsistent datatypes: expected
NUMBER got TIMESTAMP
at
oracle.jdbc.driver.OraclePreparedStatement.executeLargeBatch(OraclePreparedStatement.java:9711)
at
oracle.jdbc.driver.T4CPreparedStatement.executeLargeBatch(T4CPreparedStatement.java:1447)
at
oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedStatement.java:9487)
at
oracle.jdbc.driver.OracleStatementWrapper.executeBatch(OracleStatementWrapper.java:237)
at
org.apache.commons.dbcp2.DelegatingStatement.executeBatch(DelegatingStatement.java:242)
at
org.apache.commons.dbcp2.DelegatingStatement.executeBatch(DelegatingStatement.java:242)
at sun.reflect.GeneratedMethodAccessor583.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:254)
at
org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.access$100(StandardControllerServiceInvocationHandler.java:38)
at
org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler$ProxiedReturnObjectInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:240)
at com.sun.proxy.$Proxy360.executeBatch(Unknown Source)
at
org.apache.nifi.processors.standard.PutDatabaseRecord.executeDML(PutDatabaseRecord.java:751)
at
org.apache.nifi.processors.standard.PutDatabaseRecord.putToDatabase(PutDatabaseRecord.java:838)
at
org.apache.nifi.processors.standard.PutDatabaseRecord.onTrigger(PutDatabaseRecord.java:487)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1173)
at
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748
[jira] [Updated] (NIFI-8254) Timestamp and enums in Avro schema not working in PutDatabaseRecord on Oracle
[
https://issues.apache.org/jira/browse/NIFI-8254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Keller updated NIFI-8254:
--
Description:
There appears to be a regression in the PutDatabaseRecord processor with 1.13.0
(works in 1.12.1)
I have an Avro schema for a record which contained enum fields and timestamps.
After upgrading, a processor which had been working was failing on errors on
both of these field types. One stated it could not find the record type, the
other with the exception and stack trace below. The fields in question were
defined as below.
The files which were erroring out were the exact same files, and no changes had
been made to the schema or the CSV reader controller between the tests. I was
also able to successfully move the flow.xml.gz file back to the older version
of NiFi and the PutDatabaseRecord processor was able to work again to insert
the database records.
{noformat}
{
"name": "PER_ORG",
"type": {
"type": "enum",
"name": "PerOrgFlag",
"symbols": [
"EMP",
"CWR"
]
}
},{
"name": "UPD_BT_DTM",
"type": {
"type": "long",
"logicalType": "timestamp-millis"
}
},
{noformat}
{noformat}
2021-02-23 11:44:26,496 ERROR [Timer-Driven Process Thread-3]
o.a.n.p.standard.PutDatabaseRecord
PutDatabaseRecord[id=93b478c8-0177-1000-1ca3-59b4189b1ecb] Failed to put
Records to database for
StandardFlowFileRecord[uuid=fd4e3be1-8f7b-4a7a-bf45-2e98627f732a,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1614107687354-5, container=default,
section=5], offset=682549,
length=6820],offset=0,name=DDODS_DVCMP_PS_JOB_2021-02-09_021816_2648.dat,size=6820].
Routing to failure.: java.sql.BatchUpdateException: ORA-00932: inconsistent
datatypes: expected NUMBER got TIMESTAMP
java.sql.BatchUpdateException: ORA-00932: inconsistent datatypes: expected
NUMBER got TIMESTAMP
at
oracle.jdbc.driver.OraclePreparedStatement.executeLargeBatch(OraclePreparedStatement.java:9711)
at
oracle.jdbc.driver.T4CPreparedStatement.executeLargeBatch(T4CPreparedStatement.java:1447)
at
oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedStatement.java:9487)
at
oracle.jdbc.driver.OracleStatementWrapper.executeBatch(OracleStatementWrapper.java:237)
at
org.apache.commons.dbcp2.DelegatingStatement.executeBatch(DelegatingStatement.java:242)
at
org.apache.commons.dbcp2.DelegatingStatement.executeBatch(DelegatingStatement.java:242)
at sun.reflect.GeneratedMethodAccessor583.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:254)
at
org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.access$100(StandardControllerServiceInvocationHandler.java:38)
at
org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler$ProxiedReturnObjectInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:240)
at com.sun.proxy.$Proxy360.executeBatch(Unknown Source)
at
org.apache.nifi.processors.standard.PutDatabaseRecord.executeDML(PutDatabaseRecord.java:751)
at
org.apache.nifi.processors.standard.PutDatabaseRecord.putToDatabase(PutDatabaseRecord.java:838)
at
org.apache.nifi.processors.standard.PutDatabaseRecord.onTrigger(PutDatabaseRecord.java:487)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1173)
at
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748
[jira] [Commented] (NIFI-8249) ExecuteSQLRecord "administratively yields" and rollls back session when fetch size < query partition size and an SQLException is thrown on subsequent next()
[ https://issues.apache.org/jira/browse/NIFI-8249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289362#comment-17289362 ] David commented on NIFI-8249: - I've spent a long time browsing the Nifi code trying to find where the "transfer relationship not specified" error is being printed from. I think I remember seeing the message somewhere in a checkpoint() method for the process session? Could there be some sort of checkpointing setting for Nifi that could be in play? This is my first project working with Nifi, so I am not terribly familiar with all its components. Unfortunate that you're unable to reproduce, the test flow I sent the screenshot for is still running and throwing the error over and over. I'm not sure what other settings could be in play. Re: your previous comment: we did consider the high # of fetches to our source DB. I ran probably close to 100 permutations of these settings in our performance environment trying to determine the combination that produced the fastest end-to-end flow speed. I ran another performance test just now with 100k partition, 10k fetch and 10k max rows/flowfile and our end-to-end speed for extracting 18M records was ~21 minutes, vs ~14 minutes with 100k partition, 100 fetch, 100 max rows/flowfile. In our case the extra round-trips to the source database is worth it. We never see any backup in our downstream processors, the slowest piece always seems to be extracting the data from the source table. > ExecuteSQLRecord "administratively yields" and rollls back session when fetch > size < query partition size and an SQLException is thrown on subsequent next() > > > Key: NIFI-8249 > URL: https://issues.apache.org/jira/browse/NIFI-8249 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.11.4 >Reporter: David >Assignee: Matt Burgess >Priority: Major > Attachments: ExecuteSQLRecord.png, GenerateTableFetch.png, Screen > Shot 2021-02-23 at 3.00.47 PM.png, Screen Shot 2021-02-23 at 3.04.55 PM.png > > > When ExecuteSQLRecord executes a query with a partition size > the fetch size > and one of the fetches throws an SQLException, the exception appears to get > "swallowed" and a more general "IOException: Could not obtain next record > from ResultSet: routing to failure" in addition to a "transfer relationship > not specified;Processor Administratively Yielded for 1 sec" errors are thrown. > The problematic query/flowfile gets put back into the incoming queue and will > continue to fail indefinitely until it is manually removed from the queue. > If the fetch size >= to the query partition size, the exact SQL error message > is thrown and the flowfile gets routed to the failure queue. > > Steps to reliably reproduce: > # Create an DB table with an ID column (for partitioning queries against) > and a VARCHAR column > # Insert a row where the VARCHAR column data is a numeric value (ie '123') > # Insert a row where the VARCHAR column data is a string value (ie 'abc') > # Create a GenerateTableFetch processor where the "columns to return" > includes TO_NUMBER() > # Create a ExecuteSQLRecord processor that accepts the "success" connection > from GenerateTableFetch > # Set ExecuteSQLRecord fetch size to 1 > # Run flow > What should happen: > * ExecuteSQLRecord should throw an java.sql.SQLSyntaxErrorException: > ORA-01722: invalid number > * Flowfile should get transferred to failure queue > What happens: > * Nifi throws one error and prints another, then puts the flowfile back into > the incoming queue > ** ExecuteSQLRecord throws "Unable to execute SQL select query ... > due to java.io.IOException: java.io.IOException: Could not obtain next record > from ResultSet" > ** Nifi prints errror "transfer relationship not specified; Processor > Administratively Yielded for 1 sec" > ** Flowfile is kept in current queue and penalized and re-processed again > after penalty duration expires > Thus, the problematic query will get executed indefinitely. > > Note: I believe the issue could be due to ResultSet .next() being called > first in the constructor to ResultSetRecordSetWithCallback > Subsequent .next() is called from resultSetWriter.write(recordSet) > [https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73|https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sq
[jira] [Created] (NIFI-8254) Timestamp and enums in Avro schema not working in PutDatabaseRecord on Oracle
Jonathan Keller created NIFI-8254:
-
Summary: Timestamp and enums in Avro schema not working in
PutDatabaseRecord on Oracle
Key: NIFI-8254
URL: https://issues.apache.org/jira/browse/NIFI-8254
Project: Apache NiFi
Issue Type: Bug
Components: Core Framework
Affects Versions: 1.13.0
Reporter: Jonathan Keller
There appears to be a regression in the PutDatabaseRecord processor with 1.13.0
(works in 1.12.1)
I have an Avro schema for a record which contained enum fields and timestamps.
After upgrading, a processor which had been working was failing on errors on
both of these field types. One stated it could not find the record type, the
other with the exception and stack trace below. The fields in question were
defined as below. The files which were erroring out were the exact same files,
and no changes had been made to the schema or the CSV reader controller between
the tests. I was also able to successfully move the flow.xml.gz file back to
the older version of NiFi and the PutDatabaseRecord processor was able to work
again to insert the database records.
{noformat}
{
"name": "PER_ORG",
"type": {
"type": "enum",
"name": "PerOrgFlag",
"symbols": [
"EMP",
"CWR"
]
}
},{
"name": "UPD_BT_DTM",
"type": {
"type": "long",
"logicalType": "timestamp-millis"
}
},
{noformat}
{noformat}
2021-02-23 11:44:26,496 ERROR [Timer-Driven Process Thread-3]
o.a.n.p.standard.PutDatabaseRecord
PutDatabaseRecord[id=93b478c8-0177-1000-1ca3-59b4189b1ecb] Failed to put
Records to database for
StandardFlowFileRecord[uuid=fd4e3be1-8f7b-4a7a-bf45-2e98627f732a,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1614107687354-5, container=default,
section=5], offset=682549,
length=6820],offset=0,name=DDODS_DVCMP_PS_JOB_2021-02-09_021816_2648.dat,size=6820].
Routing to failure.: java.sql.BatchUpdateException: ORA-00932: inconsistent
datatypes: expected NUMBER got TIMESTAMP2021-02-23 11:44:26,496 ERROR
[Timer-Driven Process Thread-3] o.a.n.p.standard.PutDatabaseRecord
PutDatabaseRecord[id=93b478c8-0177-1000-1ca3-59b4189b1ecb] Failed to put
Records to database for
StandardFlowFileRecord[uuid=fd4e3be1-8f7b-4a7a-bf45-2e98627f732a,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1614107687354-5, container=default,
section=5], offset=682549,
length=6820],offset=0,name=DDODS_DVCMP_PS_JOB_2021-02-09_021816_2648.dat,size=6820].
Routing to failure.: java.sql.BatchUpdateException: ORA-00932: inconsistent
datatypes: expected NUMBER got TIMESTAMP
java.sql.BatchUpdateException: ORA-00932: inconsistent datatypes: expected
NUMBER got TIMESTAMP
at
oracle.jdbc.driver.OraclePreparedStatement.executeLargeBatch(OraclePreparedStatement.java:9711)
at
oracle.jdbc.driver.T4CPreparedStatement.executeLargeBatch(T4CPreparedStatement.java:1447)
at
oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedStatement.java:9487)
at
oracle.jdbc.driver.OracleStatementWrapper.executeBatch(OracleStatementWrapper.java:237)
at
org.apache.commons.dbcp2.DelegatingStatement.executeBatch(DelegatingStatement.java:242)
at
org.apache.commons.dbcp2.DelegatingStatement.executeBatch(DelegatingStatement.java:242)
at sun.reflect.GeneratedMethodAccessor583.invoke(Unknown Source) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:254)
at
org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.access$100(StandardControllerServiceInvocationHandler.java:38)
at
org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler$ProxiedReturnObjectInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:240)
at com.sun.proxy.$Proxy360.executeBatch(Unknown Source) at
org.apache.nifi.processors.standard.PutDatabaseRecord.executeDML(PutDatabaseRecord.java:751)
at
org.apache.nifi.processors.standard.PutDatabaseRecord.putToDatabase(PutDatabaseRecord.java:838)
at
org.apache.nifi.processors.standard.PutDatabaseRecord.onTrigger(PutDatabaseRecord.java:487)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1173)
at
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at org.apache.nifi.engine.FlowEngin
[jira] [Commented] (NIFI-8249) ExecuteSQLRecord "administratively yields" and rollls back session when fetch size < query partition size and an SQLException is thrown on subsequent next()
[ https://issues.apache.org/jira/browse/NIFI-8249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289349#comment-17289349 ] Matt Burgess commented on NIFI-8249: I set Max Rows Per FlowFile to 1 (Fetch Size 1), and couldn't reproduce with 1.11.4. I also noticed you'd set Column For Value Partitioning so I set that too but still couldn't reproduce. Each time it generates the ORA error and routes the flowfile to failure. > ExecuteSQLRecord "administratively yields" and rollls back session when fetch > size < query partition size and an SQLException is thrown on subsequent next() > > > Key: NIFI-8249 > URL: https://issues.apache.org/jira/browse/NIFI-8249 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.11.4 >Reporter: David >Assignee: Matt Burgess >Priority: Major > Attachments: ExecuteSQLRecord.png, GenerateTableFetch.png, Screen > Shot 2021-02-23 at 3.00.47 PM.png, Screen Shot 2021-02-23 at 3.04.55 PM.png > > > When ExecuteSQLRecord executes a query with a partition size > the fetch size > and one of the fetches throws an SQLException, the exception appears to get > "swallowed" and a more general "IOException: Could not obtain next record > from ResultSet: routing to failure" in addition to a "transfer relationship > not specified;Processor Administratively Yielded for 1 sec" errors are thrown. > The problematic query/flowfile gets put back into the incoming queue and will > continue to fail indefinitely until it is manually removed from the queue. > If the fetch size >= to the query partition size, the exact SQL error message > is thrown and the flowfile gets routed to the failure queue. > > Steps to reliably reproduce: > # Create an DB table with an ID column (for partitioning queries against) > and a VARCHAR column > # Insert a row where the VARCHAR column data is a numeric value (ie '123') > # Insert a row where the VARCHAR column data is a string value (ie 'abc') > # Create a GenerateTableFetch processor where the "columns to return" > includes TO_NUMBER() > # Create a ExecuteSQLRecord processor that accepts the "success" connection > from GenerateTableFetch > # Set ExecuteSQLRecord fetch size to 1 > # Run flow > What should happen: > * ExecuteSQLRecord should throw an java.sql.SQLSyntaxErrorException: > ORA-01722: invalid number > * Flowfile should get transferred to failure queue > What happens: > * Nifi throws one error and prints another, then puts the flowfile back into > the incoming queue > ** ExecuteSQLRecord throws "Unable to execute SQL select query ... > due to java.io.IOException: java.io.IOException: Could not obtain next record > from ResultSet" > ** Nifi prints errror "transfer relationship not specified; Processor > Administratively Yielded for 1 sec" > ** Flowfile is kept in current queue and penalized and re-processed again > after penalty duration expires > Thus, the problematic query will get executed indefinitely. > > Note: I believe the issue could be due to ResultSet .next() being called > first in the constructor to ResultSetRecordSetWithCallback > Subsequent .next() is called from resultSetWriter.write(recordSet) > [https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73|https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8249) ExecuteSQLRecord "administratively yields" and rollls back session when fetch size < query partition size and an SQLException is thrown on subsequent next()
[ https://issues.apache.org/jira/browse/NIFI-8249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289340#comment-17289340 ] Matt Burgess commented on NIFI-8249: Were the output flow files too large for downstream processing, or was the problem that all 100k rows were transferred downstream all at the same time? If the partition size is 100k and the fetch size is 100, you need 1000 fetches from the database for each partition, which could definitely slow things down. I'd only use something like that if there's some latency or timeout occurring while getting the rows from the DB. In general for this pattern I recommend the fetch size be set to the partition size where prudent, so all rows are returned in one fetch. Then if you need to break them up into smaller flowfiles (I'd be interested to hear about your flow) you could use SplitRecord. In any case I will try to reproduce with Max Rows set and identify any bugs that exist. Thanks for the discussion and reproduction path, it definitely helps! > ExecuteSQLRecord "administratively yields" and rollls back session when fetch > size < query partition size and an SQLException is thrown on subsequent next() > > > Key: NIFI-8249 > URL: https://issues.apache.org/jira/browse/NIFI-8249 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.11.4 >Reporter: David >Assignee: Matt Burgess >Priority: Major > Attachments: ExecuteSQLRecord.png, GenerateTableFetch.png, Screen > Shot 2021-02-23 at 3.00.47 PM.png, Screen Shot 2021-02-23 at 3.04.55 PM.png > > > When ExecuteSQLRecord executes a query with a partition size > the fetch size > and one of the fetches throws an SQLException, the exception appears to get > "swallowed" and a more general "IOException: Could not obtain next record > from ResultSet: routing to failure" in addition to a "transfer relationship > not specified;Processor Administratively Yielded for 1 sec" errors are thrown. > The problematic query/flowfile gets put back into the incoming queue and will > continue to fail indefinitely until it is manually removed from the queue. > If the fetch size >= to the query partition size, the exact SQL error message > is thrown and the flowfile gets routed to the failure queue. > > Steps to reliably reproduce: > # Create an DB table with an ID column (for partitioning queries against) > and a VARCHAR column > # Insert a row where the VARCHAR column data is a numeric value (ie '123') > # Insert a row where the VARCHAR column data is a string value (ie 'abc') > # Create a GenerateTableFetch processor where the "columns to return" > includes TO_NUMBER() > # Create a ExecuteSQLRecord processor that accepts the "success" connection > from GenerateTableFetch > # Set ExecuteSQLRecord fetch size to 1 > # Run flow > What should happen: > * ExecuteSQLRecord should throw an java.sql.SQLSyntaxErrorException: > ORA-01722: invalid number > * Flowfile should get transferred to failure queue > What happens: > * Nifi throws one error and prints another, then puts the flowfile back into > the incoming queue > ** ExecuteSQLRecord throws "Unable to execute SQL select query ... > due to java.io.IOException: java.io.IOException: Could not obtain next record > from ResultSet" > ** Nifi prints errror "transfer relationship not specified; Processor > Administratively Yielded for 1 sec" > ** Flowfile is kept in current queue and penalized and re-processed again > after penalty duration expires > Thus, the problematic query will get executed indefinitely. > > Note: I believe the issue could be due to ResultSet .next() being called > first in the constructor to ResultSetRecordSetWithCallback > Subsequent .next() is called from resultSetWriter.write(recordSet) > [https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73|https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (NIFI-8249) ExecuteSQLRecord "administratively yields" and rollls back session when fetch size < query partition size and an SQLException is thrown on subsequent next()
[ https://issues.apache.org/jira/browse/NIFI-8249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289336#comment-17289336 ] David edited comment on NIFI-8249 at 2/23/21, 9:15 PM: --- Our result sets are fairly large: we are extracting 500M records from a source table. When we encountered this scenario in production, our GenerateTableFetch partition size was 100k, max rows per flow file = 100, fetch size = 100. We fine tuned these parameters based on extensive testing in our performance environment. In our particular use case, each row contains a ~3Kb CLOB. Our "swallowed" SQL exception was actually a CLOB to CHAR buffer exception, one of the rows in the result set had a CLOB length > 4000, and we had a TO_CHAR() function in the select statement. was (Author: dfritter4): Our result sets are fairly large: we are extracting 500M records from a source table. When we encountered this scenario in production, our GenerateTableFetch partition size was 100k, max rows per flow file = 100, fetch size = 100. We fine tuned these parameters based on extensive testing in our performance environment. > ExecuteSQLRecord "administratively yields" and rollls back session when fetch > size < query partition size and an SQLException is thrown on subsequent next() > > > Key: NIFI-8249 > URL: https://issues.apache.org/jira/browse/NIFI-8249 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.11.4 >Reporter: David >Assignee: Matt Burgess >Priority: Major > Attachments: ExecuteSQLRecord.png, GenerateTableFetch.png, Screen > Shot 2021-02-23 at 3.00.47 PM.png, Screen Shot 2021-02-23 at 3.04.55 PM.png > > > When ExecuteSQLRecord executes a query with a partition size > the fetch size > and one of the fetches throws an SQLException, the exception appears to get > "swallowed" and a more general "IOException: Could not obtain next record > from ResultSet: routing to failure" in addition to a "transfer relationship > not specified;Processor Administratively Yielded for 1 sec" errors are thrown. > The problematic query/flowfile gets put back into the incoming queue and will > continue to fail indefinitely until it is manually removed from the queue. > If the fetch size >= to the query partition size, the exact SQL error message > is thrown and the flowfile gets routed to the failure queue. > > Steps to reliably reproduce: > # Create an DB table with an ID column (for partitioning queries against) > and a VARCHAR column > # Insert a row where the VARCHAR column data is a numeric value (ie '123') > # Insert a row where the VARCHAR column data is a string value (ie 'abc') > # Create a GenerateTableFetch processor where the "columns to return" > includes TO_NUMBER() > # Create a ExecuteSQLRecord processor that accepts the "success" connection > from GenerateTableFetch > # Set ExecuteSQLRecord fetch size to 1 > # Run flow > What should happen: > * ExecuteSQLRecord should throw an java.sql.SQLSyntaxErrorException: > ORA-01722: invalid number > * Flowfile should get transferred to failure queue > What happens: > * Nifi throws one error and prints another, then puts the flowfile back into > the incoming queue > ** ExecuteSQLRecord throws "Unable to execute SQL select query ... > due to java.io.IOException: java.io.IOException: Could not obtain next record > from ResultSet" > ** Nifi prints errror "transfer relationship not specified; Processor > Administratively Yielded for 1 sec" > ** Flowfile is kept in current queue and penalized and re-processed again > after penalty duration expires > Thus, the problematic query will get executed indefinitely. > > Note: I believe the issue could be due to ResultSet .next() being called > first in the constructor to ResultSetRecordSetWithCallback > Subsequent .next() is called from resultSetWriter.write(recordSet) > [https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73|https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (NIFI-8249) ExecuteSQLRecord "administratively yields" and rollls back session when fetch size < query partition size and an SQLException is thrown on subsequent next()
[ https://issues.apache.org/jira/browse/NIFI-8249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289336#comment-17289336 ] David edited comment on NIFI-8249 at 2/23/21, 9:15 PM: --- Our result sets are fairly large: we are extracting 500M records from a source table. When we encountered this scenario in production, our GenerateTableFetch partition size was 100k, max rows per flow file = 100, fetch size = 100. We fine tuned these parameters based on extensive testing in our performance environment. In our particular use case, each row contains a ~3Kb CLOB. Our "swallowed" SQL exception was actually a CLOB to CHAR "buffer to small" exception, one of the rows in the result set had a CLOB length > 4000, and we had a TO_CHAR() function in the select statement. was (Author: dfritter4): Our result sets are fairly large: we are extracting 500M records from a source table. When we encountered this scenario in production, our GenerateTableFetch partition size was 100k, max rows per flow file = 100, fetch size = 100. We fine tuned these parameters based on extensive testing in our performance environment. In our particular use case, each row contains a ~3Kb CLOB. Our "swallowed" SQL exception was actually a CLOB to CHAR buffer exception, one of the rows in the result set had a CLOB length > 4000, and we had a TO_CHAR() function in the select statement. > ExecuteSQLRecord "administratively yields" and rollls back session when fetch > size < query partition size and an SQLException is thrown on subsequent next() > > > Key: NIFI-8249 > URL: https://issues.apache.org/jira/browse/NIFI-8249 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.11.4 >Reporter: David >Assignee: Matt Burgess >Priority: Major > Attachments: ExecuteSQLRecord.png, GenerateTableFetch.png, Screen > Shot 2021-02-23 at 3.00.47 PM.png, Screen Shot 2021-02-23 at 3.04.55 PM.png > > > When ExecuteSQLRecord executes a query with a partition size > the fetch size > and one of the fetches throws an SQLException, the exception appears to get > "swallowed" and a more general "IOException: Could not obtain next record > from ResultSet: routing to failure" in addition to a "transfer relationship > not specified;Processor Administratively Yielded for 1 sec" errors are thrown. > The problematic query/flowfile gets put back into the incoming queue and will > continue to fail indefinitely until it is manually removed from the queue. > If the fetch size >= to the query partition size, the exact SQL error message > is thrown and the flowfile gets routed to the failure queue. > > Steps to reliably reproduce: > # Create an DB table with an ID column (for partitioning queries against) > and a VARCHAR column > # Insert a row where the VARCHAR column data is a numeric value (ie '123') > # Insert a row where the VARCHAR column data is a string value (ie 'abc') > # Create a GenerateTableFetch processor where the "columns to return" > includes TO_NUMBER() > # Create a ExecuteSQLRecord processor that accepts the "success" connection > from GenerateTableFetch > # Set ExecuteSQLRecord fetch size to 1 > # Run flow > What should happen: > * ExecuteSQLRecord should throw an java.sql.SQLSyntaxErrorException: > ORA-01722: invalid number > * Flowfile should get transferred to failure queue > What happens: > * Nifi throws one error and prints another, then puts the flowfile back into > the incoming queue > ** ExecuteSQLRecord throws "Unable to execute SQL select query ... > due to java.io.IOException: java.io.IOException: Could not obtain next record > from ResultSet" > ** Nifi prints errror "transfer relationship not specified; Processor > Administratively Yielded for 1 sec" > ** Flowfile is kept in current queue and penalized and re-processed again > after penalty duration expires > Thus, the problematic query will get executed indefinitely. > > Note: I believe the issue could be due to ResultSet .next() being called > first in the constructor to ResultSetRecordSetWithCallback > Subsequent .next() is called from resultSetWriter.write(recordSet) > [https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73|https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73] > -- This message was sent by Atlassian Jira
[jira] [Commented] (NIFI-8249) ExecuteSQLRecord "administratively yields" and rollls back session when fetch size < query partition size and an SQLException is thrown on subsequent next()
[ https://issues.apache.org/jira/browse/NIFI-8249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289336#comment-17289336 ] David commented on NIFI-8249: - Our result sets are fairly large: we are extracting 500M records from a source table. When we encountered this scenario in production, our GenerateTableFetch partition size was 100k, max rows per flow file = 100, fetch size = 100. We fine tuned these parameters based on extensive testing in our performance environment. > ExecuteSQLRecord "administratively yields" and rollls back session when fetch > size < query partition size and an SQLException is thrown on subsequent next() > > > Key: NIFI-8249 > URL: https://issues.apache.org/jira/browse/NIFI-8249 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.11.4 >Reporter: David >Assignee: Matt Burgess >Priority: Major > Attachments: ExecuteSQLRecord.png, GenerateTableFetch.png, Screen > Shot 2021-02-23 at 3.00.47 PM.png, Screen Shot 2021-02-23 at 3.04.55 PM.png > > > When ExecuteSQLRecord executes a query with a partition size > the fetch size > and one of the fetches throws an SQLException, the exception appears to get > "swallowed" and a more general "IOException: Could not obtain next record > from ResultSet: routing to failure" in addition to a "transfer relationship > not specified;Processor Administratively Yielded for 1 sec" errors are thrown. > The problematic query/flowfile gets put back into the incoming queue and will > continue to fail indefinitely until it is manually removed from the queue. > If the fetch size >= to the query partition size, the exact SQL error message > is thrown and the flowfile gets routed to the failure queue. > > Steps to reliably reproduce: > # Create an DB table with an ID column (for partitioning queries against) > and a VARCHAR column > # Insert a row where the VARCHAR column data is a numeric value (ie '123') > # Insert a row where the VARCHAR column data is a string value (ie 'abc') > # Create a GenerateTableFetch processor where the "columns to return" > includes TO_NUMBER() > # Create a ExecuteSQLRecord processor that accepts the "success" connection > from GenerateTableFetch > # Set ExecuteSQLRecord fetch size to 1 > # Run flow > What should happen: > * ExecuteSQLRecord should throw an java.sql.SQLSyntaxErrorException: > ORA-01722: invalid number > * Flowfile should get transferred to failure queue > What happens: > * Nifi throws one error and prints another, then puts the flowfile back into > the incoming queue > ** ExecuteSQLRecord throws "Unable to execute SQL select query ... > due to java.io.IOException: java.io.IOException: Could not obtain next record > from ResultSet" > ** Nifi prints errror "transfer relationship not specified; Processor > Administratively Yielded for 1 sec" > ** Flowfile is kept in current queue and penalized and re-processed again > after penalty duration expires > Thus, the problematic query will get executed indefinitely. > > Note: I believe the issue could be due to ResultSet .next() being called > first in the constructor to ResultSetRecordSetWithCallback > Subsequent .next() is called from resultSetWriter.write(recordSet) > [https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73|https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8249) ExecuteSQLRecord "administratively yields" and rollls back session when fetch size < query partition size and an SQLException is thrown on subsequent next()
[ https://issues.apache.org/jira/browse/NIFI-8249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289335#comment-17289335 ] Matt Burgess commented on NIFI-8249: Ah sorry I was updating the Jira while you were, missed your last comment. I will try with Max Rows Per FlowFile but that property is more for testing/debug purposes and its use is discouraged unless you're working with extremely large result sets. > ExecuteSQLRecord "administratively yields" and rollls back session when fetch > size < query partition size and an SQLException is thrown on subsequent next() > > > Key: NIFI-8249 > URL: https://issues.apache.org/jira/browse/NIFI-8249 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.11.4 >Reporter: David >Assignee: Matt Burgess >Priority: Major > Attachments: ExecuteSQLRecord.png, GenerateTableFetch.png, Screen > Shot 2021-02-23 at 3.00.47 PM.png, Screen Shot 2021-02-23 at 3.04.55 PM.png > > > When ExecuteSQLRecord executes a query with a partition size > the fetch size > and one of the fetches throws an SQLException, the exception appears to get > "swallowed" and a more general "IOException: Could not obtain next record > from ResultSet: routing to failure" in addition to a "transfer relationship > not specified;Processor Administratively Yielded for 1 sec" errors are thrown. > The problematic query/flowfile gets put back into the incoming queue and will > continue to fail indefinitely until it is manually removed from the queue. > If the fetch size >= to the query partition size, the exact SQL error message > is thrown and the flowfile gets routed to the failure queue. > > Steps to reliably reproduce: > # Create an DB table with an ID column (for partitioning queries against) > and a VARCHAR column > # Insert a row where the VARCHAR column data is a numeric value (ie '123') > # Insert a row where the VARCHAR column data is a string value (ie 'abc') > # Create a GenerateTableFetch processor where the "columns to return" > includes TO_NUMBER() > # Create a ExecuteSQLRecord processor that accepts the "success" connection > from GenerateTableFetch > # Set ExecuteSQLRecord fetch size to 1 > # Run flow > What should happen: > * ExecuteSQLRecord should throw an java.sql.SQLSyntaxErrorException: > ORA-01722: invalid number > * Flowfile should get transferred to failure queue > What happens: > * Nifi throws one error and prints another, then puts the flowfile back into > the incoming queue > ** ExecuteSQLRecord throws "Unable to execute SQL select query ... > due to java.io.IOException: java.io.IOException: Could not obtain next record > from ResultSet" > ** Nifi prints errror "transfer relationship not specified; Processor > Administratively Yielded for 1 sec" > ** Flowfile is kept in current queue and penalized and re-processed again > after penalty duration expires > Thus, the problematic query will get executed indefinitely. > > Note: I believe the issue could be due to ResultSet .next() being called > first in the constructor to ResultSetRecordSetWithCallback > Subsequent .next() is called from resultSetWriter.write(recordSet) > [https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73|https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8249) ExecuteSQLRecord "administratively yields" and rollls back session when fetch size < query partition size and an SQLException is thrown on subsequent next()
[ https://issues.apache.org/jira/browse/NIFI-8249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289334#comment-17289334 ] Matt Burgess commented on NIFI-8249: I just checked NiFi 1.11.4 (both Apache NiFi and a vendor version) and can't reproduce there either, all three versions I tried generated the expected exception but then routed the flowfile to failure. I tried with and without Max-Value Columns set using Oracle 11g. I also attached a debugger and followed the code where it fails (as you point out) but then a ProcessException is propagated to AbstractExecuteSQL which then routes the flowfile to failure: https://github.com/apache/nifi/blob/rel/nifi-1.11.4/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractExecuteSQL.java#L424 > ExecuteSQLRecord "administratively yields" and rollls back session when fetch > size < query partition size and an SQLException is thrown on subsequent next() > > > Key: NIFI-8249 > URL: https://issues.apache.org/jira/browse/NIFI-8249 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.11.4 >Reporter: David >Assignee: Matt Burgess >Priority: Major > Attachments: ExecuteSQLRecord.png, GenerateTableFetch.png, Screen > Shot 2021-02-23 at 3.00.47 PM.png, Screen Shot 2021-02-23 at 3.04.55 PM.png > > > When ExecuteSQLRecord executes a query with a partition size > the fetch size > and one of the fetches throws an SQLException, the exception appears to get > "swallowed" and a more general "IOException: Could not obtain next record > from ResultSet: routing to failure" in addition to a "transfer relationship > not specified;Processor Administratively Yielded for 1 sec" errors are thrown. > The problematic query/flowfile gets put back into the incoming queue and will > continue to fail indefinitely until it is manually removed from the queue. > If the fetch size >= to the query partition size, the exact SQL error message > is thrown and the flowfile gets routed to the failure queue. > > Steps to reliably reproduce: > # Create an DB table with an ID column (for partitioning queries against) > and a VARCHAR column > # Insert a row where the VARCHAR column data is a numeric value (ie '123') > # Insert a row where the VARCHAR column data is a string value (ie 'abc') > # Create a GenerateTableFetch processor where the "columns to return" > includes TO_NUMBER() > # Create a ExecuteSQLRecord processor that accepts the "success" connection > from GenerateTableFetch > # Set ExecuteSQLRecord fetch size to 1 > # Run flow > What should happen: > * ExecuteSQLRecord should throw an java.sql.SQLSyntaxErrorException: > ORA-01722: invalid number > * Flowfile should get transferred to failure queue > What happens: > * Nifi throws one error and prints another, then puts the flowfile back into > the incoming queue > ** ExecuteSQLRecord throws "Unable to execute SQL select query ... > due to java.io.IOException: java.io.IOException: Could not obtain next record > from ResultSet" > ** Nifi prints errror "transfer relationship not specified; Processor > Administratively Yielded for 1 sec" > ** Flowfile is kept in current queue and penalized and re-processed again > after penalty duration expires > Thus, the problematic query will get executed indefinitely. > > Note: I believe the issue could be due to ResultSet .next() being called > first in the constructor to ResultSetRecordSetWithCallback > Subsequent .next() is called from resultSetWriter.write(recordSet) > [https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73|https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8249) ExecuteSQLRecord "administratively yields" and rollls back session when fetch size < query partition size and an SQLException is thrown on subsequent next()
[ https://issues.apache.org/jira/browse/NIFI-8249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David updated NIFI-8249: Attachment: GenerateTableFetch.png > ExecuteSQLRecord "administratively yields" and rollls back session when fetch > size < query partition size and an SQLException is thrown on subsequent next() > > > Key: NIFI-8249 > URL: https://issues.apache.org/jira/browse/NIFI-8249 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.11.4 >Reporter: David >Assignee: Matt Burgess >Priority: Major > Attachments: ExecuteSQLRecord.png, GenerateTableFetch.png, Screen > Shot 2021-02-23 at 3.00.47 PM.png, Screen Shot 2021-02-23 at 3.04.55 PM.png > > > When ExecuteSQLRecord executes a query with a partition size > the fetch size > and one of the fetches throws an SQLException, the exception appears to get > "swallowed" and a more general "IOException: Could not obtain next record > from ResultSet: routing to failure" in addition to a "transfer relationship > not specified;Processor Administratively Yielded for 1 sec" errors are thrown. > The problematic query/flowfile gets put back into the incoming queue and will > continue to fail indefinitely until it is manually removed from the queue. > If the fetch size >= to the query partition size, the exact SQL error message > is thrown and the flowfile gets routed to the failure queue. > > Steps to reliably reproduce: > # Create an DB table with an ID column (for partitioning queries against) > and a VARCHAR column > # Insert a row where the VARCHAR column data is a numeric value (ie '123') > # Insert a row where the VARCHAR column data is a string value (ie 'abc') > # Create a GenerateTableFetch processor where the "columns to return" > includes TO_NUMBER() > # Create a ExecuteSQLRecord processor that accepts the "success" connection > from GenerateTableFetch > # Set ExecuteSQLRecord fetch size to 1 > # Run flow > What should happen: > * ExecuteSQLRecord should throw an java.sql.SQLSyntaxErrorException: > ORA-01722: invalid number > * Flowfile should get transferred to failure queue > What happens: > * Nifi throws one error and prints another, then puts the flowfile back into > the incoming queue > ** ExecuteSQLRecord throws "Unable to execute SQL select query ... > due to java.io.IOException: java.io.IOException: Could not obtain next record > from ResultSet" > ** Nifi prints errror "transfer relationship not specified; Processor > Administratively Yielded for 1 sec" > ** Flowfile is kept in current queue and penalized and re-processed again > after penalty duration expires > Thus, the problematic query will get executed indefinitely. > > Note: I believe the issue could be due to ResultSet .next() being called > first in the constructor to ResultSetRecordSetWithCallback > Subsequent .next() is called from resultSetWriter.write(recordSet) > [https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73|https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8249) ExecuteSQLRecord "administratively yields" and rollls back session when fetch size < query partition size and an SQLException is thrown on subsequent next()
[ https://issues.apache.org/jira/browse/NIFI-8249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David updated NIFI-8249: Attachment: ExecuteSQLRecord.png > ExecuteSQLRecord "administratively yields" and rollls back session when fetch > size < query partition size and an SQLException is thrown on subsequent next() > > > Key: NIFI-8249 > URL: https://issues.apache.org/jira/browse/NIFI-8249 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.11.4 >Reporter: David >Assignee: Matt Burgess >Priority: Major > Attachments: ExecuteSQLRecord.png, Screen Shot 2021-02-23 at 3.00.47 > PM.png, Screen Shot 2021-02-23 at 3.04.55 PM.png > > > When ExecuteSQLRecord executes a query with a partition size > the fetch size > and one of the fetches throws an SQLException, the exception appears to get > "swallowed" and a more general "IOException: Could not obtain next record > from ResultSet: routing to failure" in addition to a "transfer relationship > not specified;Processor Administratively Yielded for 1 sec" errors are thrown. > The problematic query/flowfile gets put back into the incoming queue and will > continue to fail indefinitely until it is manually removed from the queue. > If the fetch size >= to the query partition size, the exact SQL error message > is thrown and the flowfile gets routed to the failure queue. > > Steps to reliably reproduce: > # Create an DB table with an ID column (for partitioning queries against) > and a VARCHAR column > # Insert a row where the VARCHAR column data is a numeric value (ie '123') > # Insert a row where the VARCHAR column data is a string value (ie 'abc') > # Create a GenerateTableFetch processor where the "columns to return" > includes TO_NUMBER() > # Create a ExecuteSQLRecord processor that accepts the "success" connection > from GenerateTableFetch > # Set ExecuteSQLRecord fetch size to 1 > # Run flow > What should happen: > * ExecuteSQLRecord should throw an java.sql.SQLSyntaxErrorException: > ORA-01722: invalid number > * Flowfile should get transferred to failure queue > What happens: > * Nifi throws one error and prints another, then puts the flowfile back into > the incoming queue > ** ExecuteSQLRecord throws "Unable to execute SQL select query ... > due to java.io.IOException: java.io.IOException: Could not obtain next record > from ResultSet" > ** Nifi prints errror "transfer relationship not specified; Processor > Administratively Yielded for 1 sec" > ** Flowfile is kept in current queue and penalized and re-processed again > after penalty duration expires > Thus, the problematic query will get executed indefinitely. > > Note: I believe the issue could be due to ResultSet .next() being called > first in the constructor to ResultSetRecordSetWithCallback > Subsequent .next() is called from resultSetWriter.write(recordSet) > [https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73|https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8249) ExecuteSQLRecord "administratively yields" and rollls back session when fetch size < query partition size and an SQLException is thrown on subsequent next()
[ https://issues.apache.org/jira/browse/NIFI-8249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David updated NIFI-8249: Attachment: Screen Shot 2021-02-23 at 3.04.55 PM.png > ExecuteSQLRecord "administratively yields" and rollls back session when fetch > size < query partition size and an SQLException is thrown on subsequent next() > > > Key: NIFI-8249 > URL: https://issues.apache.org/jira/browse/NIFI-8249 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.11.4 >Reporter: David >Assignee: Matt Burgess >Priority: Major > Attachments: ExecuteSQLRecord.png, Screen Shot 2021-02-23 at 3.00.47 > PM.png, Screen Shot 2021-02-23 at 3.04.55 PM.png > > > When ExecuteSQLRecord executes a query with a partition size > the fetch size > and one of the fetches throws an SQLException, the exception appears to get > "swallowed" and a more general "IOException: Could not obtain next record > from ResultSet: routing to failure" in addition to a "transfer relationship > not specified;Processor Administratively Yielded for 1 sec" errors are thrown. > The problematic query/flowfile gets put back into the incoming queue and will > continue to fail indefinitely until it is manually removed from the queue. > If the fetch size >= to the query partition size, the exact SQL error message > is thrown and the flowfile gets routed to the failure queue. > > Steps to reliably reproduce: > # Create an DB table with an ID column (for partitioning queries against) > and a VARCHAR column > # Insert a row where the VARCHAR column data is a numeric value (ie '123') > # Insert a row where the VARCHAR column data is a string value (ie 'abc') > # Create a GenerateTableFetch processor where the "columns to return" > includes TO_NUMBER() > # Create a ExecuteSQLRecord processor that accepts the "success" connection > from GenerateTableFetch > # Set ExecuteSQLRecord fetch size to 1 > # Run flow > What should happen: > * ExecuteSQLRecord should throw an java.sql.SQLSyntaxErrorException: > ORA-01722: invalid number > * Flowfile should get transferred to failure queue > What happens: > * Nifi throws one error and prints another, then puts the flowfile back into > the incoming queue > ** ExecuteSQLRecord throws "Unable to execute SQL select query ... > due to java.io.IOException: java.io.IOException: Could not obtain next record > from ResultSet" > ** Nifi prints errror "transfer relationship not specified; Processor > Administratively Yielded for 1 sec" > ** Flowfile is kept in current queue and penalized and re-processed again > after penalty duration expires > Thus, the problematic query will get executed indefinitely. > > Note: I believe the issue could be due to ResultSet .next() being called > first in the constructor to ResultSetRecordSetWithCallback > Subsequent .next() is called from resultSetWriter.write(recordSet) > [https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73|https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (NIFI-8249) ExecuteSQLRecord "administratively yields" and rollls back session when fetch size < query partition size and an SQLException is thrown on subsequent next()
[ https://issues.apache.org/jira/browse/NIFI-8249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289332#comment-17289332 ] David edited comment on NIFI-8249 at 2/23/21, 9:02 PM: --- Apparently another key component to trigger this scenario: ExecuteSQLRecord must have "max rows per flow file" set to some value less than the # of rows in the ResultSet. In my scenario: i've set "max rows per flow file" = 1, fetch size = 1. Added a screenshot of my database table with values. was (Author: dfritter4): Apparently another key component to trigger this scenario: ExecuteSQLRecord must have "max rows per flow file" set to some value less than the # of rows in the ResultSet. In my scenario: i've set "max rows per flow file" = 1, fetch size = 1. Added a screenshot of my database table. > ExecuteSQLRecord "administratively yields" and rollls back session when fetch > size < query partition size and an SQLException is thrown on subsequent next() > > > Key: NIFI-8249 > URL: https://issues.apache.org/jira/browse/NIFI-8249 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.11.4 >Reporter: David >Assignee: Matt Burgess >Priority: Major > Attachments: Screen Shot 2021-02-23 at 3.00.47 PM.png > > > When ExecuteSQLRecord executes a query with a partition size > the fetch size > and one of the fetches throws an SQLException, the exception appears to get > "swallowed" and a more general "IOException: Could not obtain next record > from ResultSet: routing to failure" in addition to a "transfer relationship > not specified;Processor Administratively Yielded for 1 sec" errors are thrown. > The problematic query/flowfile gets put back into the incoming queue and will > continue to fail indefinitely until it is manually removed from the queue. > If the fetch size >= to the query partition size, the exact SQL error message > is thrown and the flowfile gets routed to the failure queue. > > Steps to reliably reproduce: > # Create an DB table with an ID column (for partitioning queries against) > and a VARCHAR column > # Insert a row where the VARCHAR column data is a numeric value (ie '123') > # Insert a row where the VARCHAR column data is a string value (ie 'abc') > # Create a GenerateTableFetch processor where the "columns to return" > includes TO_NUMBER() > # Create a ExecuteSQLRecord processor that accepts the "success" connection > from GenerateTableFetch > # Set ExecuteSQLRecord fetch size to 1 > # Run flow > What should happen: > * ExecuteSQLRecord should throw an java.sql.SQLSyntaxErrorException: > ORA-01722: invalid number > * Flowfile should get transferred to failure queue > What happens: > * Nifi throws one error and prints another, then puts the flowfile back into > the incoming queue > ** ExecuteSQLRecord throws "Unable to execute SQL select query ... > due to java.io.IOException: java.io.IOException: Could not obtain next record > from ResultSet" > ** Nifi prints errror "transfer relationship not specified; Processor > Administratively Yielded for 1 sec" > ** Flowfile is kept in current queue and penalized and re-processed again > after penalty duration expires > Thus, the problematic query will get executed indefinitely. > > Note: I believe the issue could be due to ResultSet .next() being called > first in the constructor to ResultSetRecordSetWithCallback > Subsequent .next() is called from resultSetWriter.write(recordSet) > [https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73|https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8249) ExecuteSQLRecord "administratively yields" and rollls back session when fetch size < query partition size and an SQLException is thrown on subsequent next()
[ https://issues.apache.org/jira/browse/NIFI-8249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David updated NIFI-8249: Attachment: Screen Shot 2021-02-23 at 3.00.47 PM.png > ExecuteSQLRecord "administratively yields" and rollls back session when fetch > size < query partition size and an SQLException is thrown on subsequent next() > > > Key: NIFI-8249 > URL: https://issues.apache.org/jira/browse/NIFI-8249 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.11.4 >Reporter: David >Assignee: Matt Burgess >Priority: Major > Attachments: Screen Shot 2021-02-23 at 3.00.47 PM.png > > > When ExecuteSQLRecord executes a query with a partition size > the fetch size > and one of the fetches throws an SQLException, the exception appears to get > "swallowed" and a more general "IOException: Could not obtain next record > from ResultSet: routing to failure" in addition to a "transfer relationship > not specified;Processor Administratively Yielded for 1 sec" errors are thrown. > The problematic query/flowfile gets put back into the incoming queue and will > continue to fail indefinitely until it is manually removed from the queue. > If the fetch size >= to the query partition size, the exact SQL error message > is thrown and the flowfile gets routed to the failure queue. > > Steps to reliably reproduce: > # Create an DB table with an ID column (for partitioning queries against) > and a VARCHAR column > # Insert a row where the VARCHAR column data is a numeric value (ie '123') > # Insert a row where the VARCHAR column data is a string value (ie 'abc') > # Create a GenerateTableFetch processor where the "columns to return" > includes TO_NUMBER() > # Create a ExecuteSQLRecord processor that accepts the "success" connection > from GenerateTableFetch > # Set ExecuteSQLRecord fetch size to 1 > # Run flow > What should happen: > * ExecuteSQLRecord should throw an java.sql.SQLSyntaxErrorException: > ORA-01722: invalid number > * Flowfile should get transferred to failure queue > What happens: > * Nifi throws one error and prints another, then puts the flowfile back into > the incoming queue > ** ExecuteSQLRecord throws "Unable to execute SQL select query ... > due to java.io.IOException: java.io.IOException: Could not obtain next record > from ResultSet" > ** Nifi prints errror "transfer relationship not specified; Processor > Administratively Yielded for 1 sec" > ** Flowfile is kept in current queue and penalized and re-processed again > after penalty duration expires > Thus, the problematic query will get executed indefinitely. > > Note: I believe the issue could be due to ResultSet .next() being called > first in the constructor to ResultSetRecordSetWithCallback > Subsequent .next() is called from resultSetWriter.write(recordSet) > [https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73|https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8249) ExecuteSQLRecord "administratively yields" and rollls back session when fetch size < query partition size and an SQLException is thrown on subsequent next()
[ https://issues.apache.org/jira/browse/NIFI-8249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289332#comment-17289332 ] David commented on NIFI-8249: - Apparently another key component to trigger this scenario: ExecuteSQLRecord must have "max rows per flow file" set to some value less than the # of rows in the ResultSet. In my scenario: i've set "max rows per flow file" = 1, fetch size = 1. Added a screenshot of my database table. > ExecuteSQLRecord "administratively yields" and rollls back session when fetch > size < query partition size and an SQLException is thrown on subsequent next() > > > Key: NIFI-8249 > URL: https://issues.apache.org/jira/browse/NIFI-8249 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.11.4 >Reporter: David >Assignee: Matt Burgess >Priority: Major > Attachments: Screen Shot 2021-02-23 at 3.00.47 PM.png > > > When ExecuteSQLRecord executes a query with a partition size > the fetch size > and one of the fetches throws an SQLException, the exception appears to get > "swallowed" and a more general "IOException: Could not obtain next record > from ResultSet: routing to failure" in addition to a "transfer relationship > not specified;Processor Administratively Yielded for 1 sec" errors are thrown. > The problematic query/flowfile gets put back into the incoming queue and will > continue to fail indefinitely until it is manually removed from the queue. > If the fetch size >= to the query partition size, the exact SQL error message > is thrown and the flowfile gets routed to the failure queue. > > Steps to reliably reproduce: > # Create an DB table with an ID column (for partitioning queries against) > and a VARCHAR column > # Insert a row where the VARCHAR column data is a numeric value (ie '123') > # Insert a row where the VARCHAR column data is a string value (ie 'abc') > # Create a GenerateTableFetch processor where the "columns to return" > includes TO_NUMBER() > # Create a ExecuteSQLRecord processor that accepts the "success" connection > from GenerateTableFetch > # Set ExecuteSQLRecord fetch size to 1 > # Run flow > What should happen: > * ExecuteSQLRecord should throw an java.sql.SQLSyntaxErrorException: > ORA-01722: invalid number > * Flowfile should get transferred to failure queue > What happens: > * Nifi throws one error and prints another, then puts the flowfile back into > the incoming queue > ** ExecuteSQLRecord throws "Unable to execute SQL select query ... > due to java.io.IOException: java.io.IOException: Could not obtain next record > from ResultSet" > ** Nifi prints errror "transfer relationship not specified; Processor > Administratively Yielded for 1 sec" > ** Flowfile is kept in current queue and penalized and re-processed again > after penalty duration expires > Thus, the problematic query will get executed indefinitely. > > Note: I believe the issue could be due to ResultSet .next() being called > first in the constructor to ResultSetRecordSetWithCallback > Subsequent .next() is called from resultSetWriter.write(recordSet) > [https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73|https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8249) ExecuteSQLRecord "administratively yields" and rollls back session when fetch size < query partition size and an SQLException is thrown on subsequent next()
[ https://issues.apache.org/jira/browse/NIFI-8249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289326#comment-17289326 ] David commented on NIFI-8249: - Just to be extra clear: the flowfile only gets put back if the SQLException is thrown on the second or later fetch / call to .next(). If the SQLException is thrown in the _first_ fetch, the flowfile is correctly routed to failure. I can provide more details on my setup if needed. I tested with multiple record writers (both Avro and Json) and I can reliably reproduce with either. > ExecuteSQLRecord "administratively yields" and rollls back session when fetch > size < query partition size and an SQLException is thrown on subsequent next() > > > Key: NIFI-8249 > URL: https://issues.apache.org/jira/browse/NIFI-8249 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.11.4 >Reporter: David >Assignee: Matt Burgess >Priority: Major > > When ExecuteSQLRecord executes a query with a partition size > the fetch size > and one of the fetches throws an SQLException, the exception appears to get > "swallowed" and a more general "IOException: Could not obtain next record > from ResultSet: routing to failure" in addition to a "transfer relationship > not specified;Processor Administratively Yielded for 1 sec" errors are thrown. > The problematic query/flowfile gets put back into the incoming queue and will > continue to fail indefinitely until it is manually removed from the queue. > If the fetch size >= to the query partition size, the exact SQL error message > is thrown and the flowfile gets routed to the failure queue. > > Steps to reliably reproduce: > # Create an DB table with an ID column (for partitioning queries against) > and a VARCHAR column > # Insert a row where the VARCHAR column data is a numeric value (ie '123') > # Insert a row where the VARCHAR column data is a string value (ie 'abc') > # Create a GenerateTableFetch processor where the "columns to return" > includes TO_NUMBER() > # Create a ExecuteSQLRecord processor that accepts the "success" connection > from GenerateTableFetch > # Set ExecuteSQLRecord fetch size to 1 > # Run flow > What should happen: > * ExecuteSQLRecord should throw an java.sql.SQLSyntaxErrorException: > ORA-01722: invalid number > * Flowfile should get transferred to failure queue > What happens: > * Nifi throws one error and prints another, then puts the flowfile back into > the incoming queue > ** ExecuteSQLRecord throws "Unable to execute SQL select query ... > due to java.io.IOException: java.io.IOException: Could not obtain next record > from ResultSet" > ** Nifi prints errror "transfer relationship not specified; Processor > Administratively Yielded for 1 sec" > ** Flowfile is kept in current queue and penalized and re-processed again > after penalty duration expires > Thus, the problematic query will get executed indefinitely. > > Note: I believe the issue could be due to ResultSet .next() being called > first in the constructor to ResultSetRecordSetWithCallback > Subsequent .next() is called from resultSetWriter.write(recordSet) > [https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73|https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8249) ExecuteSQLRecord "administratively yields" and rollls back session when fetch size < query partition size and an SQLException is thrown on subsequent next()
[ https://issues.apache.org/jira/browse/NIFI-8249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289322#comment-17289322 ] Matt Burgess commented on NIFI-8249: I can't reproduce this on the latest main branch (soon to be 1.13.1). I do get the ORA-01722 but then the FlowFile gets transferred to failure as expected. I can't see anything in the history between then and now that should affect the behavior, but I will check using 1.11.4 just in case. > ExecuteSQLRecord "administratively yields" and rollls back session when fetch > size < query partition size and an SQLException is thrown on subsequent next() > > > Key: NIFI-8249 > URL: https://issues.apache.org/jira/browse/NIFI-8249 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.11.4 >Reporter: David >Assignee: Matt Burgess >Priority: Major > > When ExecuteSQLRecord executes a query with a partition size > the fetch size > and one of the fetches throws an SQLException, the exception appears to get > "swallowed" and a more general "IOException: Could not obtain next record > from ResultSet: routing to failure" in addition to a "transfer relationship > not specified;Processor Administratively Yielded for 1 sec" errors are thrown. > The problematic query/flowfile gets put back into the incoming queue and will > continue to fail indefinitely until it is manually removed from the queue. > If the fetch size >= to the query partition size, the exact SQL error message > is thrown and the flowfile gets routed to the failure queue. > > Steps to reliably reproduce: > # Create an DB table with an ID column (for partitioning queries against) > and a VARCHAR column > # Insert a row where the VARCHAR column data is a numeric value (ie '123') > # Insert a row where the VARCHAR column data is a string value (ie 'abc') > # Create a GenerateTableFetch processor where the "columns to return" > includes TO_NUMBER() > # Create a ExecuteSQLRecord processor that accepts the "success" connection > from GenerateTableFetch > # Set ExecuteSQLRecord fetch size to 1 > # Run flow > What should happen: > * ExecuteSQLRecord should throw an java.sql.SQLSyntaxErrorException: > ORA-01722: invalid number > * Flowfile should get transferred to failure queue > What happens: > * Nifi throws one error and prints another, then puts the flowfile back into > the incoming queue > ** ExecuteSQLRecord throws "Unable to execute SQL select query ... > due to java.io.IOException: java.io.IOException: Could not obtain next record > from ResultSet" > ** Nifi prints errror "transfer relationship not specified; Processor > Administratively Yielded for 1 sec" > ** Flowfile is kept in current queue and penalized and re-processed again > after penalty duration expires > Thus, the problematic query will get executed indefinitely. > > Note: I believe the issue could be due to ResultSet .next() being called > first in the constructor to ResultSetRecordSetWithCallback > Subsequent .next() is called from resultSetWriter.write(recordSet) > [https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73|https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8253) GenerateTableFetch never commits the session
[ https://issues.apache.org/jira/browse/NIFI-8253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-8253: --- Priority: Blocker (was: Major) > GenerateTableFetch never commits the session > > > Key: NIFI-8253 > URL: https://issues.apache.org/jira/browse/NIFI-8253 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.13.0 >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Blocker > Fix For: 1.14.0, 1.13.1 > > Time Spent: 10m > Remaining Estimate: 0h > > In NIFI-8136, the session.commit() call was inadvertently removed from > GenerateTableFetch (rather than moved until after the session.setState() > call). This regression was not caught by unit tests because no assertion was > made that the session(s) had been committed. > The session.commit() call should be returned and the unit test(s) should > assert the session(s) have been committed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8253) GenerateTableFetch never commits the session
[ https://issues.apache.org/jira/browse/NIFI-8253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-8253: --- Status: Patch Available (was: In Progress) > GenerateTableFetch never commits the session > > > Key: NIFI-8253 > URL: https://issues.apache.org/jira/browse/NIFI-8253 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.13.0 >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Fix For: 1.14.0, 1.13.1 > > Time Spent: 10m > Remaining Estimate: 0h > > In NIFI-8136, the session.commit() call was inadvertently removed from > GenerateTableFetch (rather than moved until after the session.setState() > call). This regression was not caught by unit tests because no assertion was > made that the session(s) had been committed. > The session.commit() call should be returned and the unit test(s) should > assert the session(s) have been committed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] mattyb149 opened a new pull request #4840: NIFI-8253: Restore call to session.commit() in GenerateTableFetch
mattyb149 opened a new pull request #4840:
URL: https://github.com/apache/nifi/pull/4840
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
The session.commit() call was inadvertently removed from GenerateTableFetch,
this PR restores it and adds a test to ensure the session(s) have been
committed.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [x] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically `main`)?
- [x] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [x] Have you written or updated unit tests to verify your changes?
- [x] Have you verified that the full build is successful on JDK 8?
- [ ] Have you verified that the full build is successful on JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for
build issues and submit an update to your PR as soon as possible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (NIFI-8253) GenerateTableFetch never commits the session
Matt Burgess created NIFI-8253: -- Summary: GenerateTableFetch never commits the session Key: NIFI-8253 URL: https://issues.apache.org/jira/browse/NIFI-8253 Project: Apache NiFi Issue Type: Bug Components: Extensions Affects Versions: 1.13.0 Reporter: Matt Burgess Assignee: Matt Burgess Fix For: 1.14.0, 1.13.1 In NIFI-8136, the session.commit() call was inadvertently removed from GenerateTableFetch (rather than moved until after the session.setState() call). This regression was not caught by unit tests because no assertion was made that the session(s) had been committed. The session.commit() call should be returned and the unit test(s) should assert the session(s) have been committed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] turcsanyip commented on a change in pull request #4839: NIFI-8113 Adding persistent status history repository backed by embedded QuestDB
turcsanyip commented on a change in pull request #4839:
URL: https://github.com/apache/nifi/pull/4839#discussion_r581216003
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/status/history/storage/QuestDbComponentStatusStorage.java
##
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.controller.status.history.storage;
+
+import io.questdb.cairo.sql.Record;
+import org.apache.commons.lang3.time.DateUtils;
+import org.apache.commons.lang3.time.FastDateFormat;
+import org.apache.commons.math3.util.Pair;
+import org.apache.nifi.controller.status.history.ComponentDetailsStorage;
+import org.apache.nifi.controller.status.history.MetricDescriptor;
+import org.apache.nifi.controller.status.history.StandardStatusHistory;
+import org.apache.nifi.controller.status.history.StatusHistory;
+import org.apache.nifi.controller.status.history.StatusSnapshot;
+import org.apache.nifi.controller.status.history.questdb.QuestDbContext;
+import
org.apache.nifi.controller.status.history.questdb.QuestDbEntityReadingTemplate;
+import
org.apache.nifi.controller.status.history.questdb.QuestDbEntityWritingTemplate;
+import
org.apache.nifi.controller.status.history.questdb.QuestDbStatusSnapshotMapper;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Date;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.function.Function;
+
+/**
+ * Component specific implementation of the {@link ComponentStatusStorage}.
+ *
+ * @param Component status entry type.
+ */
+abstract class QuestDbComponentStatusStorage implements
ComponentStatusStorage {
Review comment:
It might be worth separating these `QuestDb***StatusStorage`
implementation classes to their own subpackage.
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/status/history/storage/StorageStatusReadingTemplate.java
##
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.controller.status.history.storage;
+
+import io.questdb.cairo.sql.Record;
+import io.questdb.cairo.sql.RecordCursor;
+import org.apache.nifi.controller.status.NodeStatus;
+import org.apache.nifi.controller.status.history.MetricDescriptor;
+import org.apache.nifi.controller.status.history.StandardMetricDescriptor;
+import
org.apache.nifi.controller.status.history.questdb.QuestDbReadingTemplate;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+public class StorageStatusReadingTemplate extends
QuestDbReadingTemplate,
Long>>> {
+private static final String STORAGE_FREE_DESCRIPTION = "The usable space
available for use by the underlying storage mechanism.";
+private static final String STORAGE_USED_DESCRIPTION = "The space in use
on the underlying storage mechanism";
+
+private static final String STORAGE_READING_QUERY =
+"SELECT * FROM storageStatus " +
+"WHERE capturedAt > to_timestamp('%s', '" +
StatusStorage.CAPTURE_DATE_FORMAT + "') " +
+"AND capturedAt < to_timestamp('%s', '" +
StatusStorage.CAPTURE_DATE_FORMAT + "') " +
+"ORDER BY capturedAt ASC";
+
+public StorageStatusReadingTemplate() {
[jira] [Updated] (NIFI-8249) ExecuteSQLRecord "administratively yields" and rollls back session when fetch size < query partition size and an SQLException is thrown on subsequent next()
[ https://issues.apache.org/jira/browse/NIFI-8249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David updated NIFI-8249: Description: When ExecuteSQLRecord executes a query with a partition size > the fetch size and one of the fetches throws an SQLException, the exception appears to get "swallowed" and a more general "IOException: Could not obtain next record from ResultSet: routing to failure" in addition to a "transfer relationship not specified;Processor Administratively Yielded for 1 sec" errors are thrown. The problematic query/flowfile gets put back into the incoming queue and will continue to fail indefinitely until it is manually removed from the queue. If the fetch size >= to the query partition size, the exact SQL error message is thrown and the flowfile gets routed to the failure queue. Steps to reliably reproduce: # Create an DB table with an ID column (for partitioning queries against) and a VARCHAR column # Insert a row where the VARCHAR column data is a numeric value (ie '123') # Insert a row where the VARCHAR column data is a string value (ie 'abc') # Create a GenerateTableFetch processor where the "columns to return" includes TO_NUMBER() # Create a ExecuteSQLRecord processor that accepts the "success" connection from GenerateTableFetch # Set ExecuteSQLRecord fetch size to 1 # Run flow What should happen: * ExecuteSQLRecord should throw an java.sql.SQLSyntaxErrorException: ORA-01722: invalid number * Flowfile should get transferred to failure queue What happens: * Nifi throws one error and prints another, then puts the flowfile back into the incoming queue ** ExecuteSQLRecord throws "Unable to execute SQL select query ... due to java.io.IOException: java.io.IOException: Could not obtain next record from ResultSet" ** Nifi prints errror "transfer relationship not specified; Processor Administratively Yielded for 1 sec" ** Flowfile is kept in current queue and penalized and re-processed again after penalty duration expires Thus, the problematic query will get executed indefinitely. Note: I believe the issue could be due to ResultSet .next() being called first in the constructor to ResultSetRecordSetWithCallback Subsequent .next() is called from resultSetWriter.write(recordSet) [https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73|https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73] was: When ExecuteSQLRecord executes a query with a partition size > the fetch size and one of the fetches throws an SQLException, the exception appears to get "swallowed" and a more general "IOException: Could not obtain next record from ResultSet: routing to failure" in addition to a "transfer relationship not specified;Processor Administratively Yielded for 1 sec" errors are thrown. The problematic query/flowfile gets put back into the incoming queue and will continue to fail indefinitely until it is manually removed from the queue. If the fetch size >= to the query partition size, the exact SQL error message is thrown and the flowfile gets routed to the failure queue. Steps to reliably reproduce: # Create an DB table with an ID column (for partitioning queries against) and a VARCHAR column # Insert a row where the VARCHAR column data is a numeric value (ie '123') # Insert a row where the VARCHAR column data is a string value (ie 'abc') # Create a GenerateTableFetch processor where the "columns to return" includes TO_NUMBER() # Create a ExecuteSQLRecord processor that accepts the "success" connection from GenerateTableFetch # Set ExecuteSQLRecord fetch size to 1 # Run flow What should happen: * ExecuteSQLRecord should throw an java.sql.SQLSyntaxErrorException: ORA-01722: invalid number * Flowfile should get transferred to failure queue What happens: * Nifi throws one error and prints another, then puts the flowfile back into the incoming queue ** ExecuteSQLRecord throws "Unable to execute SQL select query ... due to java.io.IOException: java.io.IOException: Could not obtain next record from ResultSet" ** Nifi prints errror "transfer relationship not specified; Processor Administratively Yielded for 1 sec" ** Flowfile is kept in current queue and penalized and re-processed again after penalty duration expires Thus, the problematic query will get executed indefinitely. Note: I believe the issue could be due to ResultSet .next() being called first in the constructor to ResultSetRecordSetWithCallback Subsequent .next() is called from resultSetWriter.write(recordSet) [https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa
[jira] [Updated] (NIFI-8249) ExecuteSQLRecord "administratively yields" and rollls back session when fetch size < query partition size and an SQLException is thrown on subsequent next()
[ https://issues.apache.org/jira/browse/NIFI-8249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David updated NIFI-8249: Description: When ExecuteSQLRecord executes a query with a partition size > the fetch size and one of the fetches throws an SQLException, the exception appears to get "swallowed" and a more general "IOException: Could not obtain next record from ResultSet: routing to failure" in addition to a "transfer relationship not specified;Processor Administratively Yielded for 1 sec" errors are thrown. The problematic query/flowfile gets put back into the incoming queue and will continue to fail indefinitely until it is manually removed from the queue. If the fetch size >= to the query partition size, the exact SQL error message is thrown and the flowfile gets routed to the failure queue. Steps to reliably reproduce: # Create an DB table with an ID column (for partitioning queries against) and a VARCHAR column # Insert a row where the VARCHAR column data is a numeric value (ie '123') # Insert a row where the VARCHAR column data is a string value (ie 'abc') # Create a GenerateTableFetch processor where the "columns to return" includes TO_NUMBER() # Create a ExecuteSQLRecord processor that accepts the "success" connection from GenerateTableFetch # Set ExecuteSQLRecord fetch size to 1 # Run flow What should happen: * ExecuteSQLRecord should throw an java.sql.SQLSyntaxErrorException: ORA-01722: invalid number * Flowfile should get transferred to failure queue What happens: * Nifi throws one error and prints another, then puts the flowfile back into the incoming queue ** ExecuteSQLRecord throws "Unable to execute SQL select query ... due to java.io.IOException: java.io.IOException: Could not obtain next record from ResultSet" ** Nifi prints errror "transfer relationship not specified; Processor Administratively Yielded for 1 sec" ** Flowfile is kept in current queue and penalized and re-processed again after penalty duration expires Thus, the problematic query will get executed indefinitely. Note: I believe the issue could be due to ResultSet .next() being called first in the constructor to ResultSetRecordSetWithCallback Subsequent .next() is called from resultSetWriter.write(recordSet) [https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L73|https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L71] was: When ExecuteSQLRecord executes a query with a partition size > the fetch size and one of the fetches throws an SQLException, the exception appears to get "swallowed" and a more general "IOException: Could not obtain next record from ResultSet: routing to failure" in addition to a "transfer relationship not specified;Processor Administratively Yielded for 1 sec" errors are thrown. The problematic query/flowfile gets put back into the incoming queue and will continue to fail indefinitely until it is manually removed from the queue. If the fetch size >= to the query partition size, the exact SQL error message is thrown and the flowfile gets routed to the failure queue. Steps to reliably reproduce: # Create an DB table with an ID column (for partitioning queries against) and a VARCHAR column # Insert a row where the VARCHAR column data is a numeric value (ie '123') # Insert a row where the VARCHAR column data is a string value (ie 'abc') # Create a GenerateTableFetch processor where the "columns to return" includes TO_NUMBER() # Create a ExecuteSQLRecord processor that accepts the "success" connection from GenerateTableFetch # Set ExecuteSQLRecord fetch size to 1 # Run flow What should happen: * ExecuteSQLRecord should throw an java.sql.SQLSyntaxErrorException: ORA-01722: invalid number * Flowfile should get transferred to failure queue What happens: * Nifi throws one error and prints another, then puts the flowfile back into the incoming queue ** ExecuteSQLRecord throws "Unable to execute SQL select query ... due to java.io.IOException: java.io.IOException: Could not obtain next record from ResultSet" ** Nifi prints errror "transfer relationship not specified; Processor Administratively Yielded for 1 sec" ** Flowfile is kept in current queue and penalized and re-processed again after penalty duration expires Thus, the problematic query will get executed indefinitely. Note: I believe the issue could be due to ResultSet .next() being called first in the constructor to ResultSetRecordSetWithCallback Subsequent .next() is called from resultSetWriter.write(recordSet) [https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa
[GitHub] [nifi] NissimShiman commented on pull request #3923: NIFI-6866 Created the RocksDBComponentStatusRepository to persist com…
NissimShiman commented on pull request #3923: URL: https://github.com/apache/nifi/pull/3923#issuecomment-784401727 @jfrazee I tested it on Red Hat Enterprise Linux 7 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-8249) ExecuteSQLRecord "administratively yields" and rollls back session when fetch size < query partition size and an SQLException is thrown on subsequent next()
[ https://issues.apache.org/jira/browse/NIFI-8249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David updated NIFI-8249: Description: When ExecuteSQLRecord executes a query with a partition size > the fetch size and one of the fetches throws an SQLException, the exception appears to get "swallowed" and a more general "IOException: Could not obtain next record from ResultSet: routing to failure" in addition to a "transfer relationship not specified;Processor Administratively Yielded for 1 sec" errors are thrown. The problematic query/flowfile gets put back into the incoming queue and will continue to fail indefinitely until it is manually removed from the queue. If the fetch size >= to the query partition size, the exact SQL error message is thrown and the flowfile gets routed to the failure queue. Steps to reliably reproduce: # Create an DB table with an ID column (for partitioning queries against) and a VARCHAR column # Insert a row where the VARCHAR column data is a numeric value (ie '123') # Insert a row where the VARCHAR column data is a string value (ie 'abc') # Create a GenerateTableFetch processor where the "columns to return" includes TO_NUMBER() # Create a ExecuteSQLRecord processor that accepts the "success" connection from GenerateTableFetch # Set ExecuteSQLRecord fetch size to 1 # Run flow What should happen: * ExecuteSQLRecord should throw an java.sql.SQLSyntaxErrorException: ORA-01722: invalid number * Flowfile should get transferred to failure queue What happens: * Nifi throws one error and prints another, then puts the flowfile back into the incoming queue ** ExecuteSQLRecord throws "Unable to execute SQL select query ... due to java.io.IOException: java.io.IOException: Could not obtain next record from ResultSet" ** Nifi prints errror "transfer relationship not specified; Processor Administratively Yielded for 1 sec" ** Flowfile is kept in current queue and penalized and re-processed again after penalty duration expires Thus, the problematic query will get executed indefinitely. Note: I believe the issue could be due to ResultSet .next() being called first in the constructor to ResultSetRecordSetWithCallback Subsequent .next() is called from resultSetWriter.write(recordSet) [https://github.com/apache/nifi/blob/aa741cc5967f62c3c38c2a47e712b7faa6fe19ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/sql/RecordSqlWriter.java#L71] was: When ExecuteSQLRecord executes a query with a partition size > the fetch size and one of the fetches throws an SQLException, the exception appears to get "swallowed" and a more general "IOException: Could not obtain next record from ResultSet: routing to failure" in addition to a "transfer relationship not specified;Processor Administratively Yielded for 1 sec" errors are thrown. The problematic query/flowfile gets put back into the incoming queue and will continue to fail indefinitely until it is manually removed from the queue. If the fetch size >= to the query partition size, the exact SQL error message is thrown and the flowfile gets routed to the failure queue. Steps to reliably reproduce: # Create an DB table with an ID column (for partitioning queries against) and a VARCHAR column # Insert a row where the VARCHAR column data is a numeric value (ie '123') # Insert a row where the VARCHAR column data is a string value (ie 'abc') # Create a GenerateTableFetch processor where the "columns to return" includes TO_NUMBER() # Create a ExecuteSQLRecord processor that accepts the "success" connection from GenerateTableFetch # Set ExecuteSQLRecord fetch size to 1 # Run flow What should happen: * ExecuteSQLRecord should throw an java.sql.SQLSyntaxErrorException: ORA-01722: invalid number * Flowfile should get transferred to failure queue What happens: * Nifi throws one error and prints another, then puts the flowfile back into the incoming queue ** ExecuteSQLRecord throws "Unable to execute SQL select query ... due to java.io.IOException: java.io.IOException: Could not obtain next record from ResultSet" ** Nifi prints errror "transfer relationship not specified; Processor Administratively Yielded for 1 sec" ** Flowfile is kept in current queue and penalized and re-processed again after penalty duration expires Thus, the problematic query will get executed indefinitely. > ExecuteSQLRecord "administratively yields" and rollls back session when fetch > size < query partition size and an SQLException is thrown on subsequent next() > > > Key: NIFI-8249 > URL: https://issues.apache.org/jira/browse/NIFI-8249 > Project: Apache NiFi > Issue Type: Bug >
[jira] [Updated] (NIFI-8249) ExecuteSQLRecord "administratively yields" and rollls back session when fetch size < query partition size and an SQLException is thrown on subsequent next()
[ https://issues.apache.org/jira/browse/NIFI-8249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David updated NIFI-8249: Description: When ExecuteSQLRecord executes a query with a partition size > the fetch size and one of the fetches throws an SQLException, the exception appears to get "swallowed" and a more general "IOException: Could not obtain next record from ResultSet: routing to failure" in addition to a "transfer relationship not specified;Processor Administratively Yielded for 1 sec" errors are thrown. The problematic query/flowfile gets put back into the incoming queue and will continue to fail indefinitely until it is manually removed from the queue. If the fetch size >= to the query partition size, the exact SQL error message is thrown and the flowfile gets routed to the failure queue. Steps to reliably reproduce: # Create an DB table with an ID column (for partitioning queries against) and a VARCHAR column # Insert a row where the VARCHAR column data is a numeric value (ie '123') # Insert a row where the VARCHAR column data is a string value (ie 'abc') # Create a GenerateTableFetch processor where the "columns to return" includes TO_NUMBER() # Create a ExecuteSQLRecord processor that accepts the "success" connection from GenerateTableFetch # Set ExecuteSQLRecord fetch size to 1 # Run flow What should happen: * ExecuteSQLRecord should throw an java.sql.SQLSyntaxErrorException: ORA-01722: invalid number * Flowfile should get transferred to failure queue What happens: * Nifi throws one error and prints another, then puts the flowfile back into the incoming queue ** ExecuteSQLRecord throws "Unable to execute SQL select query ... due to java.io.IOException: java.io.IOException: Could not obtain next record from ResultSet" ** Nifi prints errror "transfer relationship not specified; Processor Administratively Yielded for 1 sec" ** Flowfile is kept in current queue and penalized and re-processed again after penalty duration expires Thus, the problematic query will get executed indefinitely. was: When ExecuteSQLRecord executes a query with a partition size > the fetch size and one of the fetches throws an SQLException, the exception appears to get "swallowed" and a more general "IOException: Could not obtain next record from ResultSet: routing to failure" in addition to a "transfer relationship not specified;Processor Administratively Yielded for 1 sec" errors are thrown. The problematic query/flowfile gets put back into the incoming queue and will continue to fail indefinitely until it is manually removed from the queue. If the fetch size >= to the query partition size, the exact SQL error message is thrown and the flowfile gets routed to the failure queue. Steps to reliably reproduce: # Create an Oracle DB table with an ID column (for partitioning queries against) and a CLOB column # Insert a row with where the CLOB column data is larger than 4000 characters # Insert x rows (where x is larger than the below ExecuteSQLRecord's fetch size) where the CLOB column data is less than 4000 characters # Create a GenerateTableFetch processor where the "columns to return" includes TO_CHAR() # Create a ExecuteSQLRecord processor that accepts the "success" connection from GenerateTableFetch # Create some other processor for the success and failure relationships from ExecuteSQLRecord (this is just to make sure all transfer relationships are specified) # Set ExecuteSQLRecord fetch size to some value smaller than the partition size in GenerateTableFetch # Make sure SQLException is thrown in second or later calls to ResultSet.next() # Run flow What should happen: * ExecuteSQLRecord should throw an SQLException: ORA-22835: Buffer too small for CLOB to CHAR or BLOB to RAW conversion * Flowfile should get transferred to failure queue What happens: * Nifi throws one error and prints another, then puts the flowfile back into the incoming queue ** ExecuteSQLRecord throws "Unable to execute SQL select query ... due to java.io.IOException: java.io.IOException: Could not obtain next record from ResultSet" ** Nifi prints errror "transfer relationship not specified; Processor Administratively Yielded for 1 sec" ** Flowfile is kept in current queue and penalized and re-processed again after penalty duration expires Thus, the problematic query will get executed indefinitely. > ExecuteSQLRecord "administratively yields" and rollls back session when fetch > size < query partition size and an SQLException is thrown on subsequent next() > > > Key: NIFI-8249 > URL: https://issues.apache.org/jira/browse/NIFI-8249 > Project: Apache NiFi > Issue Type: Bug > Components: Core F
[jira] [Assigned] (NIFI-8249) ExecuteSQLRecord "administratively yields" and rollls back session when fetch size < query partition size and an SQLException is thrown on subsequent next()
[ https://issues.apache.org/jira/browse/NIFI-8249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess reassigned NIFI-8249: -- Assignee: Matt Burgess > ExecuteSQLRecord "administratively yields" and rollls back session when fetch > size < query partition size and an SQLException is thrown on subsequent next() > > > Key: NIFI-8249 > URL: https://issues.apache.org/jira/browse/NIFI-8249 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.11.4 >Reporter: David >Assignee: Matt Burgess >Priority: Major > > When ExecuteSQLRecord executes a query with a partition size > the fetch size > and one of the fetches throws an SQLException, the exception appears to get > "swallowed" and a more general "IOException: Could not obtain next record > from ResultSet: routing to failure" in addition to a "transfer relationship > not specified;Processor Administratively Yielded for 1 sec" errors are thrown. > The problematic query/flowfile gets put back into the incoming queue and will > continue to fail indefinitely until it is manually removed from the queue. > If the fetch size >= to the query partition size, the exact SQL error message > is thrown and the flowfile gets routed to the failure queue. > > Steps to reliably reproduce: > # Create an Oracle DB table with an ID column (for partitioning queries > against) and a CLOB column > # Insert a row with where the CLOB column data is larger than 4000 characters > # Insert x rows (where x is larger than the below ExecuteSQLRecord's fetch > size) where the CLOB column data is less than 4000 characters > # Create a GenerateTableFetch processor where the "columns to return" > includes TO_CHAR() > # Create a ExecuteSQLRecord processor that accepts the "success" connection > from GenerateTableFetch > # Create some other processor for the success and failure relationships from > ExecuteSQLRecord (this is just to make sure all transfer relationships are > specified) > # Set ExecuteSQLRecord fetch size to some value smaller than the partition > size in GenerateTableFetch > # Make sure SQLException is thrown in second or later calls to > ResultSet.next() > # Run flow > What should happen: > * ExecuteSQLRecord should throw an SQLException: ORA-22835: Buffer too small > for CLOB to CHAR or BLOB to RAW conversion > * Flowfile should get transferred to failure queue > What happens: > * Nifi throws one error and prints another, then puts the flowfile back into > the incoming queue > ** ExecuteSQLRecord throws "Unable to execute SQL select query ... > due to java.io.IOException: java.io.IOException: Could not obtain next record > from ResultSet" > ** Nifi prints errror "transfer relationship not specified; Processor > Administratively Yielded for 1 sec" > ** Flowfile is kept in current queue and penalized and re-processed again > after penalty duration expires > Thus, the problematic query will get executed indefinitely. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8249) ExecuteSQLRecord "administratively yields" and rollls back session when fetch size < query partition size and an SQLException is thrown on subsequent next()
[ https://issues.apache.org/jira/browse/NIFI-8249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David updated NIFI-8249: Summary: ExecuteSQLRecord "administratively yields" and rollls back session when fetch size < query partition size and an SQLException is thrown on subsequent next() (was: ExecuteSQLRecord "administratively yields" when fetch size < query partition size and an SQLException is thrown) > ExecuteSQLRecord "administratively yields" and rollls back session when fetch > size < query partition size and an SQLException is thrown on subsequent next() > > > Key: NIFI-8249 > URL: https://issues.apache.org/jira/browse/NIFI-8249 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.11.4 >Reporter: David >Priority: Major > > When ExecuteSQLRecord executes a query with a partition size > the fetch size > and one of the fetches throws an SQLException, the exception appears to get > "swallowed" and a more general "IOException: Could not obtain next record > from ResultSet: routing to failure" in addition to a "transfer relationship > not specified;Processor Administratively Yielded for 1 sec" errors are thrown. > The problematic query/flowfile gets put back into the incoming queue and will > continue to fail indefinitely until it is manually removed from the queue. > If the fetch size >= to the query partition size, the exact SQL error message > is thrown and the flowfile gets routed to the failure queue. > > Steps to reliably reproduce: > # Create an Oracle DB table with an ID column (for partitioning queries > against) and a CLOB column > # Insert a row with where the CLOB column data is larger than 4000 characters > # Insert x rows (where x is larger than the below ExecuteSQLRecord's fetch > size) where the CLOB column data is less than 4000 characters > # Create a GenerateTableFetch processor where the "columns to return" > includes TO_CHAR() > # Create a ExecuteSQLRecord processor that accepts the "success" connection > from GenerateTableFetch > # Create some other processor for the success and failure relationships from > ExecuteSQLRecord (this is just to make sure all transfer relationships are > specified) > # Set ExecuteSQLRecord fetch size to some value smaller than the partition > size in GenerateTableFetch > # Make sure SQLException is thrown in second or later calls to > ResultSet.next() > # Run flow > What should happen: > * ExecuteSQLRecord should throw an SQLException: ORA-22835: Buffer too small > for CLOB to CHAR or BLOB to RAW conversion > * Flowfile should get transferred to failure queue > What happens: > * Nifi throws one error and prints another, then puts the flowfile back into > the incoming queue > ** ExecuteSQLRecord throws "Unable to execute SQL select query ... > due to java.io.IOException: java.io.IOException: Could not obtain next record > from ResultSet" > ** Nifi prints errror "transfer relationship not specified; Processor > Administratively Yielded for 1 sec" > ** Flowfile is kept in current queue and penalized and re-processed again > after penalty duration expires > Thus, the problematic query will get executed indefinitely. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8205) Documentation improvement for the Wait processor
[ https://issues.apache.org/jira/browse/NIFI-8205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Witt updated NIFI-8205: --- Fix Version/s: 1.13.1 > Documentation improvement for the Wait processor > > > Key: NIFI-8205 > URL: https://issues.apache.org/jira/browse/NIFI-8205 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation & Website, Extensions >Reporter: Peter Gyori >Assignee: Peter Gyori >Priority: Minor > Labels: Wait/Notify > Fix For: 1.14.0, 1.13.1 > > Time Spent: 0.5h > Remaining Estimate: 0h > > The documentation for the Wait processor can be improved to clarify how the > Wait/Notify feature works. Also, an example about using Counters would be a > useful addition to the documentation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8205) Documentation improvement for the Wait processor
[ https://issues.apache.org/jira/browse/NIFI-8205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289211#comment-17289211 ] ASF subversion and git services commented on NIFI-8205: --- Commit d049bca7948227cdddcd59a50c2daf104e2883f8 in nifi's branch refs/heads/main from Peter Gyori [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=d049bca ] NIFI-8205: Documentation improvements for the Wait processor Signed-off-by: Pierre Villard This closes #4808. > Documentation improvement for the Wait processor > > > Key: NIFI-8205 > URL: https://issues.apache.org/jira/browse/NIFI-8205 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Peter Gyori >Assignee: Peter Gyori >Priority: Minor > Labels: Wait/Notify > Time Spent: 0.5h > Remaining Estimate: 0h > > The documentation for the Wait processor can be improved to clarify how the > Wait/Notify feature works. Also, an example about using Counters would be a > useful addition to the documentation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8205) Documentation improvement for the Wait processor
[ https://issues.apache.org/jira/browse/NIFI-8205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-8205: - Fix Version/s: 1.14.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Documentation improvement for the Wait processor > > > Key: NIFI-8205 > URL: https://issues.apache.org/jira/browse/NIFI-8205 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Peter Gyori >Assignee: Peter Gyori >Priority: Minor > Labels: Wait/Notify > Fix For: 1.14.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > The documentation for the Wait processor can be improved to clarify how the > Wait/Notify feature works. Also, an example about using Counters would be a > useful addition to the documentation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8205) Documentation improvement for the Wait processor
[ https://issues.apache.org/jira/browse/NIFI-8205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-8205: - Component/s: Documentation & Website > Documentation improvement for the Wait processor > > > Key: NIFI-8205 > URL: https://issues.apache.org/jira/browse/NIFI-8205 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation & Website, Extensions >Reporter: Peter Gyori >Assignee: Peter Gyori >Priority: Minor > Labels: Wait/Notify > Fix For: 1.14.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > The documentation for the Wait processor can be improved to clarify how the > Wait/Notify feature works. Also, an example about using Counters would be a > useful addition to the documentation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] asfgit closed pull request #4808: NIFI-8205: Documentation improvements for the Wait processor
asfgit closed pull request #4808: URL: https://github.com/apache/nifi/pull/4808 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-8249) ExecuteSQLRecord "administratively yields" when fetch size < query partition size and an SQLException is thrown
[ https://issues.apache.org/jira/browse/NIFI-8249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David updated NIFI-8249: Description: When ExecuteSQLRecord executes a query with a partition size > the fetch size and one of the fetches throws an SQLException, the exception appears to get "swallowed" and a more general "IOException: Could not obtain next record from ResultSet: routing to failure" in addition to a "transfer relationship not specified;Processor Administratively Yielded for 1 sec" errors are thrown. The problematic query/flowfile gets put back into the incoming queue and will continue to fail indefinitely until it is manually removed from the queue. If the fetch size >= to the query partition size, the exact SQL error message is thrown and the flowfile gets routed to the failure queue. Steps to reliably reproduce: # Create an Oracle DB table with an ID column (for partitioning queries against) and a CLOB column # Insert a row with where the CLOB column data is larger than 4000 characters # Insert x rows (where x is larger than the below ExecuteSQLRecord's fetch size) where the CLOB column data is less than 4000 characters # Create a GenerateTableFetch processor where the "columns to return" includes TO_CHAR() # Create a ExecuteSQLRecord processor that accepts the "success" connection from GenerateTableFetch # Create some other processor for the success and failure relationships from ExecuteSQLRecord (this is just to make sure all transfer relationships are specified) # Set ExecuteSQLRecord fetch size to some value smaller than the partition size in GenerateTableFetch # Make sure SQLException is thrown in second or later calls to ResultSet.next() # Run flow What should happen: * ExecuteSQLRecord should throw an SQLException: ORA-22835: Buffer too small for CLOB to CHAR or BLOB to RAW conversion * Flowfile should get transferred to failure queue What happens: * Nifi throws one error and prints another, then puts the flowfile back into the incoming queue ** ExecuteSQLRecord throws "Unable to execute SQL select query ... due to java.io.IOException: java.io.IOException: Could not obtain next record from ResultSet" ** Nifi prints errror "transfer relationship not specified; Processor Administratively Yielded for 1 sec" ** Flowfile is kept in current queue and penalized and re-processed again after penalty duration expires Thus, the problematic query will get executed indefinitely. was: When ExecuteSQLRecord executes a query with a partition size > the fetch size and one of the fetches throws an SQLException, the exception appears to get "swallowed" and a more general "IOException: Could not obtain next record from ResultSet: routing to failure" in addition to a "transfer relationship not specified;Processor Administratively Yielded for 1 sec" errors are thrown. The problematic query/flowfile gets put back into the incoming queue and will continue to fail indefinitely until it is manually removed from the queue. If the fetch size >= to the query partition size, the exact SQL error message is thrown and the flowfile gets routed to the failure queue. Steps to reliably reproduce: # Create an Oracle DB table with an ID column (for partitioning queries against) and a CLOB column # Insert a row with where the CLOB column data is larger than 4000 characters # Insert x rows (where x is larger than the below ExecuteSQLRecord's fetch size) where the CLOB column data is less than 4000 characters # Create a GenerateTableFetch processor where the "columns to return" includes TO_CHAR() # Create a ExecuteSQLRecord processor that accepts the "success" connection from GenerateTableFetch # Create some other processor for the success and failure relationships from ExecuteSQLRecord (this is just to make sure all transfer relationships are specified) # Set ExecuteSQLRecord fetch size to some value smaller than the partition size in GenerateTableFetch # Run flow What should happen: * ExecuteSQLRecord should throw an SQLException: ORA-22835: Buffer too small for CLOB to CHAR or BLOB to RAW conversion * Flowfile should get transferred to failure queue What happens: * Nifi throws one error and prints another, then puts the flowfile back into the incoming queue ** ExecuteSQLRecord throws "Unable to execute SQL select query ... due to java.io.IOException: java.io.IOException: Could not obtain next record from ResultSet" ** Nifi prints errror "transfer relationship not specified; Processor Administratively Yielded for 1 sec" ** Flowfile is kept in current queue and penalized and re-processed again after penalty duration expires Thus, the problematic query will get executed indefinitely. > ExecuteSQLRecord "administratively yields" when fetch size < query partition > size and an SQLException is thrown > ---
[jira] [Updated] (NIFI-8249) ExecuteSQLRecord "administratively yields" when fetch size < query partition size and an SQLException is thrown
[ https://issues.apache.org/jira/browse/NIFI-8249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David updated NIFI-8249: Description: When ExecuteSQLRecord executes a query with a partition size > the fetch size and one of the fetches throws an SQLException, the exception appears to get "swallowed" and a more general "IOException: Could not obtain next record from ResultSet: routing to failure" in addition to a "transfer relationship not specified;Processor Administratively Yielded for 1 sec" errors are thrown. The problematic query/flowfile gets put back into the incoming queue and will continue to fail indefinitely until it is manually removed from the queue. If the fetch size >= to the query partition size, the exact SQL error message is thrown and the flowfile gets routed to the failure queue. Steps to reliably reproduce: # Create an Oracle DB table with an ID column (for partitioning queries against) and a CLOB column # Insert a row with where the CLOB column data is larger than 4000 characters # Insert x rows (where x is larger than the below ExecuteSQLRecord's fetch size) where the CLOB column data is less than 4000 characters # Create a GenerateTableFetch processor where the "columns to return" includes TO_CHAR() # Create a ExecuteSQLRecord processor that accepts the "success" connection from GenerateTableFetch # Create some other processor for the success and failure relationships from ExecuteSQLRecord (this is just to make sure all transfer relationships are specified) # Set ExecuteSQLRecord fetch size to some value smaller than the partition size in GenerateTableFetch # Run flow What should happen: * ExecuteSQLRecord should throw an SQLException: ORA-22835: Buffer too small for CLOB to CHAR or BLOB to RAW conversion * Flowfile should get transferred to failure queue What happens: * Nifi throws one error and prints another, then puts the flowfile back into the incoming queue ** ExecuteSQLRecord throws "Unable to execute SQL select query ... due to java.io.IOException: java.io.IOException: Could not obtain next record from ResultSet" ** Nifi prints errror "transfer relationship not specified; Processor Administratively Yielded for 1 sec" ** Flowfile is kept in current queue and penalized and re-processed again after penalty duration expires Thus, the problematic query will get executed indefinitely. was: When ExecuteSQLRecord executes a query with a partition size > the fetch size and one of the fetches throws an SQLException, the exception appears to get "swallowed" and a more general "IOException: Could not obtain next record from ResultSet: routing to failure" in addition to a "transfer relationship not specified;Processor Administratively Yielded for 1 sec" errors are thrown. The problematic query/flowfile gets put back into the incoming queue and will continue to fail indefinitely until it is manually removed from the queue. If the fetch size >= to the query partition size, the exact SQL error message is thrown and the flowfile gets routed to the failure queue. Steps to reliably reproduce: # Create an Oracle DB table with an ID column (for partitioning queries against) and a CLOB column # Insert a row with where the CLOB column data is larger than 4000 characters # Insert x rows (where x is larger than the below ExecuteSQLRecord's fetch size) where the CLOB column data is less than 4000 characters # Create a GenerateTableFetch processor where the "columns to return" includes TO_CHAR() # Create a ExecuteSQLRecord processor that accepts the "success" connection from GenerateTableFetch # Create some other processor for the success and failure relationships from ExecuteSQLRecord (this is just to make sure all transfer relationships are specified) # Set ExecuteSQLRecord fetch size to some value smaller than the partition size in GenerateTableFetch # Run flow What should happen: * ExecuteSQLRecord should throw an SQLException: ORA-22835: Buffer too small for CLOB to CHAR or BLOB to RAW conversion * Flowfile should get transferred to failure queue What happens: * Nifi throws two errors and puts the flowfile back into the incoming queue ** ExecuteSQLRecord throws "Unable to execute SQL select query ... due to java.io.IOException: java.io.IOException: Could not obtain next record from ResultSet" ** Nifi throws "transfer relationship not specified; Processor Administratively Yielded for 1 sec" ** Flowfile is kept in current queue and penalized and re-processed again after penalty duration expires Thus, the problematic query will get executed indefinitely. > ExecuteSQLRecord "administratively yields" when fetch size < query partition > size and an SQLException is thrown > --- > > Key: NIFI-8249 > URL: https://issues
[GitHub] [nifi] thenatog commented on a change in pull request #4836: NIFI-7127 - Allow injection of SecureHasher into FingerprintFactory
thenatog commented on a change in pull request #4836:
URL: https://github.com/apache/nifi/pull/4836#discussion_r581189058
##
File path:
nifi-commons/nifi-security-utils/src/main/java/org/apache/nifi/security/util/crypto/SecureHasherFactory.java
##
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.security.util.crypto;
+
+import org.apache.nifi.security.util.KeyDerivationFunction;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.naming.ConfigurationException;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+public class SecureHasherFactory {
+private static final Logger LOGGER =
LoggerFactory.getLogger(SecureHasherFactory.class);
+
+private static Map>
registeredSecureHashers;
Review comment:
I did have it this way originally, but I thought it seemed a bit of a
roundabout way of getting to the KDF name so used it as a string instead.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] mattyb149 commented on a change in pull request #4577: NIFI-6752 Create ASN.1 RecordReader
mattyb149 commented on a change in pull request #4577: URL: https://github.com/apache/nifi/pull/4577#discussion_r581185499 ## File path: nifi-nar-bundles/nifi-asn1-bundle/nifi-asn1-services/src/test/resources/ASN1.xml ## @@ -0,0 +1,1360 @@ + Review comment: This needs an Apache header or to be added to the Rat exclusion list (I prefer the former) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (NIFI-8252) ForkRecord with Extract Mode produces null results
[ https://issues.apache.org/jira/browse/NIFI-8252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy resolved NIFI-8252. --- Resolution: Duplicate Duplicate issue: https://issues.apache.org/jira/browse/NIFI-6406 Also, not an issue apparently. Output schema needs to be set explicitly. > ForkRecord with Extract Mode produces null results > -- > > Key: NIFI-8252 > URL: https://issues.apache.org/jira/browse/NIFI-8252 > Project: Apache NiFi > Issue Type: Bug >Reporter: Tamas Palfy >Priority: Major > Attachments: fork_record_bug.xml > > > *ForkRecord* can extract sub-records from an array field of a record. > If the _Mode_ property is set to _Extract_ it should extract the sub-records > and write only those to the _fork_ relationship. > However the result shows records with the original schema with all the fields > set to _null_. > The issue is probably that the _RecordSetWriter_ is created with the original > schema. > Attached template. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (NIFI-8244) PutDatabaseRecord incorrect type resolution for partial columns
[
https://issues.apache.org/jira/browse/NIFI-8244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Burgess resolved NIFI-8244.
Fix Version/s: 1.13.1
1.14.0
Assignee: Matt Burgess
Resolution: Fixed
I tried the scenario from the description with the fix from NIFI-8237, so I
think it is fixed as well. Please feel free to reopen if that's not the case.
> PutDatabaseRecord incorrect type resolution for partial columns
> ---
>
> Key: NIFI-8244
> URL: https://issues.apache.org/jira/browse/NIFI-8244
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.13.0
>Reporter: Dorian Bugeja
>Assignee: Matt Burgess
>Priority: Major
> Fix For: 1.14.0, 1.13.1
>
> Attachments: image-2021-02-20-02-18-10-670.png,
> image-2021-02-20-02-18-51-952.png, image-2021-02-20-02-26-24-986.png
>
>
> Record schema type resolution for PutDatabaseRecord is being performed using
> fieldIndexes computed in generateInsert/Update/Delete. Problem is that when
> not all keys in the records are defined or the sequence of the keys in the
> record does not match the sequence of the column name of the table,
> includedColumns, which is being populated with the index of the currentRecord
> (JSON for example) rather than the index of the table schema, will result in
> incorrect type resolution. Meaning that in the following example, `count`
> will be converted to String, as the second column in table is a String
> (Performing UPDATE in PutDatabaseRecord with Update key set to id)
> Table definition =
> id INT,
> title VARCHAR,
> count INT
> Record Update
> {"id": 1, "count": 10}
> !image-2021-02-20-02-18-51-952.png!
> !image-2021-02-20-02-18-10-670.png!
> !image-2021-02-20-02-26-24-986.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (NIFI-8252) ForkRecord with Extract Mode produces null results
[ https://issues.apache.org/jira/browse/NIFI-8252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy updated NIFI-8252: -- Description: *ForkRecord* can extract sub-records from an array field of a record. If the _Mode_ property is set to _Extract_ it should extract the sub-records and write only those to the _fork_ relationship. However the result shows records with the original schema with all the fields set to _null_. The issue is probably that the _RecordSetWriter_ is created with the original schema. Attached template. was: *ForkRecord* can extract sub-records from an array field of a record. If the _Mode_ property is set to _Extract_ it should extract the sub-records and write only those to the _fork_ relationship. However the result shows records with the original schema with all the fields set to _null_. The issue is probably that the RecordSetWriter is created with the original schema. Attached template. > ForkRecord with Extract Mode produces null results > -- > > Key: NIFI-8252 > URL: https://issues.apache.org/jira/browse/NIFI-8252 > Project: Apache NiFi > Issue Type: Bug >Reporter: Tamas Palfy >Priority: Major > Attachments: fork_record_bug.xml > > > *ForkRecord* can extract sub-records from an array field of a record. > If the _Mode_ property is set to _Extract_ it should extract the sub-records > and write only those to the _fork_ relationship. > However the result shows records with the original schema with all the fields > set to _null_. > The issue is probably that the _RecordSetWriter_ is created with the original > schema. > Attached template. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8252) ForkRecord with Extract Mode produces null results
[ https://issues.apache.org/jira/browse/NIFI-8252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy updated NIFI-8252: -- Description: *ForkRecord* can extract sub-records from an array field of a record. If the _Mode_ property is set to _Extract_ it should extract the sub-records and write only those to the _fork_ relationship. However the result shows records with the original schema with all the fields set to _null_. The issue is probably that the RecordSetWriter is created with the original schema. Attached template. was: *ForkRecord* can extract sub-records from an array field of a record. If the _Mode_ property is set to _Extract_ it should extract the sub-records and write only those to the _fork_ relationship. However the result shows record with the original schema with all the fields set to _null_. The issue is probably that the RecordSetWriter is created with the original schema. Attached template. > ForkRecord with Extract Mode produces null results > -- > > Key: NIFI-8252 > URL: https://issues.apache.org/jira/browse/NIFI-8252 > Project: Apache NiFi > Issue Type: Bug >Reporter: Tamas Palfy >Priority: Major > Attachments: fork_record_bug.xml > > > *ForkRecord* can extract sub-records from an array field of a record. > If the _Mode_ property is set to _Extract_ it should extract the sub-records > and write only those to the _fork_ relationship. > However the result shows records with the original schema with all the fields > set to _null_. > The issue is probably that the RecordSetWriter is created with the original > schema. > Attached template. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (NIFI-8252) ForkRecord with Extract Mode produces null results
Tamas Palfy created NIFI-8252: - Summary: ForkRecord with Extract Mode produces null results Key: NIFI-8252 URL: https://issues.apache.org/jira/browse/NIFI-8252 Project: Apache NiFi Issue Type: Bug Reporter: Tamas Palfy Attachments: fork_record_bug.xml *ForkRecord* can extract sub-records from an array field of a records. If the _Mode_ property is set to _Extract_ it should extract the sub-records and write only those to the _fork_ relationship. However the result shows record with the original schema with all the fields set to _null_. The issue is probably that the RecordSetWriter is created with the original schema. Attached template. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8252) ForkRecord with Extract Mode produces null results
[ https://issues.apache.org/jira/browse/NIFI-8252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Palfy updated NIFI-8252: -- Description: *ForkRecord* can extract sub-records from an array field of a record. If the _Mode_ property is set to _Extract_ it should extract the sub-records and write only those to the _fork_ relationship. However the result shows record with the original schema with all the fields set to _null_. The issue is probably that the RecordSetWriter is created with the original schema. Attached template. was: *ForkRecord* can extract sub-records from an array field of a records. If the _Mode_ property is set to _Extract_ it should extract the sub-records and write only those to the _fork_ relationship. However the result shows record with the original schema with all the fields set to _null_. The issue is probably that the RecordSetWriter is created with the original schema. Attached template. > ForkRecord with Extract Mode produces null results > -- > > Key: NIFI-8252 > URL: https://issues.apache.org/jira/browse/NIFI-8252 > Project: Apache NiFi > Issue Type: Bug >Reporter: Tamas Palfy >Priority: Major > Attachments: fork_record_bug.xml > > > *ForkRecord* can extract sub-records from an array field of a record. > If the _Mode_ property is set to _Extract_ it should extract the sub-records > and write only those to the _fork_ relationship. > However the result shows record with the original schema with all the fields > set to _null_. > The issue is probably that the RecordSetWriter is created with the original > schema. > Attached template. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] simonbence commented on pull request #4821: NIFI-8113 Adding persistent status history repository backed by embed…
simonbence commented on pull request #4821: URL: https://github.com/apache/nifi/pull/4821#issuecomment-784309459 Please find the predecessor PR here: [PR 4839](https://github.com/apache/nifi/pull/4839) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] simonbence opened a new pull request #4839: NIFI-8113 Adding persistent status history repository backed by embedded QuestDB
simonbence opened a new pull request #4839: URL: https://github.com/apache/nifi/pull/4839 _Note: this review is a successor of [PR 4821](https://github.com/apache/nifi/pull/4821). In order to keep the changes readable I decided to abandon that one and start a fresh pull request._ This would be a proposal for having persistent status history for both component and node level metrics. I decided to approach the question using `QuestDB` which is an embeddable time series database written in Java. During my tests, the performance and the memory footprint are looked reasonable. As for code organisation of the new implementation I worked with three concepts: repository is the top level entity, provides the service for the clients (`FlowController`, etc.). In general, this is the same as before, with minor changes (the name has been changed to `StatusHistoryRepostiory` to server the intent better, also start and stop hooks were added). The storage classes are part of the repositories and are responsible for manage the details of a given type, like processor status, node status, etc. Finally the `WriterTemplates` and `ReaderTemplates` are merely helpers exist to deal with `QuestDB` API calls. Some remarks on design decisions: - The PR contains a small fix on ProcessGroupStatusDescriptor#calculateTaskMillis(): previously every subsequent level makes a nano->milli conversion, which accumulates, reducing the task time in children groups into 0. Now this should be fixed (The QuestDB tests are implicitly proves this as well) - Configuration contains the new parameters but they are commented out. These are relevant for the QuestDB implementation and if not used the implementation will go with default values. At this point I think, the VolatileComponentStatusRepository should be kept as default implementation - The implementation depends on the latest of the 4.X version of QuestDB. Currently QuestDB is at 5.0.5, but the 5.X line depends on Java 11. There are some non vital features in 5.X (all_tables request, dropping partition using WHERE closure), but these are not unavoidable for us. - QuestDB can be called using `PostrgeSQL` driver via `JDBC` if it is not embedded. Also it provides supports `InfluxDB` protocol. These are possibilities were not utilised: QuestDB's own API seems to be sufficient and robust for our purposes. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with **NIFI-** where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically `main`)? - [ ] Is your initial contribution a single, squashed commit? _Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not `squash` or use `--force` when pushing to allow for clean monitoring of changes._ ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via `mvn -Pcontrib-check clean install` at the root `nifi` folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] Have you verified that the full build is successful on JDK 8? - [ ] Have you verified that the full build is successful on JDK 11? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE` file, including the main `LICENSE` file under `nifi-assembly`? - [ ] If applicable, have you updated the `NOTICE` file, including the main `NOTICE` file found under `nifi-assembly`? - [ ] If adding new Properties, have you added `.displayName` in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check GitHub Actions CI for build issues and submit an update to your PR as soon as possible. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-7646) Improve performance of MergeContent / others that read content of many small FlowFiles
[ https://issues.apache.org/jira/browse/NIFI-7646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289153#comment-17289153 ] ASF subversion and git services commented on NIFI-7646: --- Commit 528fce2407d092d4ced1a58fcc14d0bc6e660b89 in nifi's branch refs/heads/main from Mark Payne [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=528fce2 ] NIFI-7646, NIFI-8222: Instead of having StandardProcessSession call ContentRepository.read(ContentClaim), introduced a new ContentRepository.read(ResourceClaim) and hold open the InputStream to the ResourceClaim. This can't be supported by EncryptedContentRepository, so introduced a method to allow using this or not. The benefit here is that when we have many FlowFiles read within a session, such as when using MergeContent/MergeRecord or a processor configured with a Run Duration, we can hold open a single InputStream instead of constantly opening FileInputStreams and seeking to the appropriate location. This is much faster. - Instead of entering a 'synchronized' block for every provenance event, serialize up to 1 MB worth of data, and then enter synchronized block to write that data out. This avoids large amounts of lock contention and context switches NIFI-7646: Removed TODO and unused Jackson dependency Signed-off-by: Matthew Burgess This closes #4818 > Improve performance of MergeContent / others that read content of many small > FlowFiles > -- > > Key: NIFI-7646 > URL: https://issues.apache.org/jira/browse/NIFI-7646 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.14.0, 1.13.1 > > Time Spent: 40m > Remaining Estimate: 0h > > When MergeContent merges together 1,000 FlowFiles, it must read the content > of each of those FlowFiles. This is done by calling > `ProcessSession.read(flowFile);` > Right now, the Process Session ends up calling > `ContentRepository.read(ContentClaim)` using the Content Claim from the given > FlowFile. As a result, the Content Repository creates a new FileInputStream > (1+ disk accesses). It then seeks to the appropriate location on disk (1 disk > access). The stream is then wrapped in a LimitingInputStream to prevent the > reader from going beyond the boundaries of the associated Content Claim. So > if the FlowFile is small, say 200 bytes, the result is that we perform 2+ > disk accesses to read those 200 bytes (even though 4K - 8K is a typical block > size and could be read in the same amount of time as those 200 bytes). > As a result, merging 1,000 FlowFiles can result in many disk accesses and a > huge degradation in performance. > At the same ProcessSession already has a notion of the currentReadClaimStream > and a currentReadClaim. We could get huge performance improvements by making > a couple of small changes in Content Repo & Process Session: > - In ContentRepository, introduce a new method: `InputStream > read(ResourceClaim resourceClaim) throws IOException`. This will allow the > Process Session to read the entire contents of the underlying Resource Claim > if necessary. This is safe since it doesn't provide raw access to any "user > code". The Process Session will protect the bounds properly. > - ProcessSession should use this new method to access the stream for an > entire ResourceClaim. It should then skip to the appropriate location, as > that will not have been done by the Content Repository. Then, the InputStream > should be wrapped in a BufferedInputStream. This will help for cases when a > LimitingInputStream restricts reads to only 200 bytes - in this case, the > disk access will still pull back 4-8K and that will live in the > BufferedInputStream. > - ProcessSession should change the currentReadClaim from a Content Claim to a > Resource Claim to allow for this to work. Additionally, the getInputStream() > method should relax the constraint "writeRecursionSet.isEmpty()" for reusing > the stream and instead use "!writeRecursionSet.contains(flowFile)", as this > will be important for MergeContent, since it will be writing to one FlowFile > while reading from another. > These changes will transparently (to the processors) provide a very > significant performance gain in cases where a Processor must read the content > of many small FlowFiles, if the FlowFiles all have the same Resource Claim > (which is the case more often than not). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-8222) When processing a lot of small FlowFiles, Provenance Repo spends most of its time in lock contention. That can be improved.
[ https://issues.apache.org/jira/browse/NIFI-8222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289154#comment-17289154 ] ASF subversion and git services commented on NIFI-8222: --- Commit 528fce2407d092d4ced1a58fcc14d0bc6e660b89 in nifi's branch refs/heads/main from Mark Payne [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=528fce2 ] NIFI-7646, NIFI-8222: Instead of having StandardProcessSession call ContentRepository.read(ContentClaim), introduced a new ContentRepository.read(ResourceClaim) and hold open the InputStream to the ResourceClaim. This can't be supported by EncryptedContentRepository, so introduced a method to allow using this or not. The benefit here is that when we have many FlowFiles read within a session, such as when using MergeContent/MergeRecord or a processor configured with a Run Duration, we can hold open a single InputStream instead of constantly opening FileInputStreams and seeking to the appropriate location. This is much faster. - Instead of entering a 'synchronized' block for every provenance event, serialize up to 1 MB worth of data, and then enter synchronized block to write that data out. This avoids large amounts of lock contention and context switches NIFI-7646: Removed TODO and unused Jackson dependency Signed-off-by: Matthew Burgess This closes #4818 > When processing a lot of small FlowFiles, Provenance Repo spends most of its > time in lock contention. That can be improved. > --- > > Key: NIFI-8222 > URL: https://issues.apache.org/jira/browse/NIFI-8222 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.14.0, 1.13.1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-7646) Improve performance of MergeContent / others that read content of many small FlowFiles
[ https://issues.apache.org/jira/browse/NIFI-7646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289155#comment-17289155 ] ASF subversion and git services commented on NIFI-7646: --- Commit 528fce2407d092d4ced1a58fcc14d0bc6e660b89 in nifi's branch refs/heads/main from Mark Payne [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=528fce2 ] NIFI-7646, NIFI-8222: Instead of having StandardProcessSession call ContentRepository.read(ContentClaim), introduced a new ContentRepository.read(ResourceClaim) and hold open the InputStream to the ResourceClaim. This can't be supported by EncryptedContentRepository, so introduced a method to allow using this or not. The benefit here is that when we have many FlowFiles read within a session, such as when using MergeContent/MergeRecord or a processor configured with a Run Duration, we can hold open a single InputStream instead of constantly opening FileInputStreams and seeking to the appropriate location. This is much faster. - Instead of entering a 'synchronized' block for every provenance event, serialize up to 1 MB worth of data, and then enter synchronized block to write that data out. This avoids large amounts of lock contention and context switches NIFI-7646: Removed TODO and unused Jackson dependency Signed-off-by: Matthew Burgess This closes #4818 > Improve performance of MergeContent / others that read content of many small > FlowFiles > -- > > Key: NIFI-7646 > URL: https://issues.apache.org/jira/browse/NIFI-7646 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.14.0, 1.13.1 > > Time Spent: 50m > Remaining Estimate: 0h > > When MergeContent merges together 1,000 FlowFiles, it must read the content > of each of those FlowFiles. This is done by calling > `ProcessSession.read(flowFile);` > Right now, the Process Session ends up calling > `ContentRepository.read(ContentClaim)` using the Content Claim from the given > FlowFile. As a result, the Content Repository creates a new FileInputStream > (1+ disk accesses). It then seeks to the appropriate location on disk (1 disk > access). The stream is then wrapped in a LimitingInputStream to prevent the > reader from going beyond the boundaries of the associated Content Claim. So > if the FlowFile is small, say 200 bytes, the result is that we perform 2+ > disk accesses to read those 200 bytes (even though 4K - 8K is a typical block > size and could be read in the same amount of time as those 200 bytes). > As a result, merging 1,000 FlowFiles can result in many disk accesses and a > huge degradation in performance. > At the same ProcessSession already has a notion of the currentReadClaimStream > and a currentReadClaim. We could get huge performance improvements by making > a couple of small changes in Content Repo & Process Session: > - In ContentRepository, introduce a new method: `InputStream > read(ResourceClaim resourceClaim) throws IOException`. This will allow the > Process Session to read the entire contents of the underlying Resource Claim > if necessary. This is safe since it doesn't provide raw access to any "user > code". The Process Session will protect the bounds properly. > - ProcessSession should use this new method to access the stream for an > entire ResourceClaim. It should then skip to the appropriate location, as > that will not have been done by the Content Repository. Then, the InputStream > should be wrapped in a BufferedInputStream. This will help for cases when a > LimitingInputStream restricts reads to only 200 bytes - in this case, the > disk access will still pull back 4-8K and that will live in the > BufferedInputStream. > - ProcessSession should change the currentReadClaim from a Content Claim to a > Resource Claim to allow for this to work. Additionally, the getInputStream() > method should relax the constraint "writeRecursionSet.isEmpty()" for reusing > the stream and instead use "!writeRecursionSet.contains(flowFile)", as this > will be important for MergeContent, since it will be writing to one FlowFile > while reading from another. > These changes will transparently (to the processors) provide a very > significant performance gain in cases where a Processor must read the content > of many small FlowFiles, if the FlowFiles all have the same Resource Claim > (which is the case more often than not). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-8222) When processing a lot of small FlowFiles, Provenance Repo spends most of its time in lock contention. That can be improved.
[ https://issues.apache.org/jira/browse/NIFI-8222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-8222: --- Resolution: Fixed Status: Resolved (was: Patch Available) > When processing a lot of small FlowFiles, Provenance Repo spends most of its > time in lock contention. That can be improved. > --- > > Key: NIFI-8222 > URL: https://issues.apache.org/jira/browse/NIFI-8222 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.14.0, 1.13.1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] mattyb149 closed pull request #4818: NIFI-7646, NIFI-8222: Reduced lock contention when updating Provenance Repository by buffering up to 1 MB of serialized records in memory; keep Inp
mattyb149 closed pull request #4818: URL: https://github.com/apache/nifi/pull/4818 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-7646) Improve performance of MergeContent / others that read content of many small FlowFiles
[ https://issues.apache.org/jira/browse/NIFI-7646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-7646: --- Resolution: Fixed Status: Resolved (was: Patch Available) > Improve performance of MergeContent / others that read content of many small > FlowFiles > -- > > Key: NIFI-7646 > URL: https://issues.apache.org/jira/browse/NIFI-7646 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.14.0, 1.13.1 > > Time Spent: 40m > Remaining Estimate: 0h > > When MergeContent merges together 1,000 FlowFiles, it must read the content > of each of those FlowFiles. This is done by calling > `ProcessSession.read(flowFile);` > Right now, the Process Session ends up calling > `ContentRepository.read(ContentClaim)` using the Content Claim from the given > FlowFile. As a result, the Content Repository creates a new FileInputStream > (1+ disk accesses). It then seeks to the appropriate location on disk (1 disk > access). The stream is then wrapped in a LimitingInputStream to prevent the > reader from going beyond the boundaries of the associated Content Claim. So > if the FlowFile is small, say 200 bytes, the result is that we perform 2+ > disk accesses to read those 200 bytes (even though 4K - 8K is a typical block > size and could be read in the same amount of time as those 200 bytes). > As a result, merging 1,000 FlowFiles can result in many disk accesses and a > huge degradation in performance. > At the same ProcessSession already has a notion of the currentReadClaimStream > and a currentReadClaim. We could get huge performance improvements by making > a couple of small changes in Content Repo & Process Session: > - In ContentRepository, introduce a new method: `InputStream > read(ResourceClaim resourceClaim) throws IOException`. This will allow the > Process Session to read the entire contents of the underlying Resource Claim > if necessary. This is safe since it doesn't provide raw access to any "user > code". The Process Session will protect the bounds properly. > - ProcessSession should use this new method to access the stream for an > entire ResourceClaim. It should then skip to the appropriate location, as > that will not have been done by the Content Repository. Then, the InputStream > should be wrapped in a BufferedInputStream. This will help for cases when a > LimitingInputStream restricts reads to only 200 bytes - in this case, the > disk access will still pull back 4-8K and that will live in the > BufferedInputStream. > - ProcessSession should change the currentReadClaim from a Content Claim to a > Resource Claim to allow for this to work. Additionally, the getInputStream() > method should relax the constraint "writeRecursionSet.isEmpty()" for reusing > the stream and instead use "!writeRecursionSet.contains(flowFile)", as this > will be important for MergeContent, since it will be writing to one FlowFile > while reading from another. > These changes will transparently (to the processors) provide a very > significant performance gain in cases where a Processor must read the content > of many small FlowFiles, if the FlowFiles all have the same Resource Claim > (which is the case more often than not). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [nifi] mattyb149 commented on pull request #4818: NIFI-7646, NIFI-8222: Reduced lock contention when updating Provenance Repository by buffering up to 1 MB of serialized records in memory; ke
mattyb149 commented on pull request #4818: URL: https://github.com/apache/nifi/pull/4818#issuecomment-784307135 +1 LGTM, ran full build with tests and a few flows using MergeContent and others with lots of small files. I removed the TODO comment and the unused Jackson dependency. Thanks for the improvement! Merging to main This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (MINIFICPP-1508) Remove the legacy Regex wrapper and use std::regex instead
[
https://issues.apache.org/jira/browse/MINIFICPP-1508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpad Boda updated MINIFICPP-1508:
--
Labels: MiNiFi-CPP-Hygiene (was: )
> Remove the legacy Regex wrapper and use std::regex instead
> --
>
> Key: MINIFICPP-1508
> URL: https://issues.apache.org/jira/browse/MINIFICPP-1508
> Project: Apache NiFi MiNiFi C++
> Issue Type: Improvement
>Reporter: Ferenc Gerlits
>Priority: Minor
> Labels: MiNiFi-CPP-Hygiene
>
> After MINIFICPP-1429, when we no longer support gcc 4.8, remove the
> {{utils::Regex}} class, and delete the files RegexUtils.h, RegexUtils.cpp and
> RegexUtilsTests.cpp. Instead, we should use {{std::regex}} directly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [nifi] tpalfy commented on a change in pull request #4577: NIFI-6752 Create ASN.1 RecordReader
tpalfy commented on a change in pull request #4577:
URL: https://github.com/apache/nifi/pull/4577#discussion_r581148769
##
File path:
nifi-nar-bundles/nifi-asn1-bundle/nifi-asn1-services/src/test/java/org/apache/nifi/jasn1/ExampleDataGenerator.java
##
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.jasn1;
+
+import com.beanit.asn1bean.ber.ReverseByteArrayOutputStream;
+import com.beanit.asn1bean.ber.types.BerBoolean;
+import com.beanit.asn1bean.ber.types.BerInteger;
+import com.beanit.asn1bean.ber.types.BerOctetString;
+import com.beanit.asn1bean.ber.types.string.BerUTF8String;
+import org.apache.nifi.jasn1.example.BasicTypeSet;
+import org.apache.nifi.jasn1.example.BasicTypes;
+import org.apache.nifi.jasn1.example.Composite;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/**
+ * Depends on generated test classes
+ */
+public class ExampleDataGenerator {
+
+private static final Logger LOG =
LoggerFactory.getLogger(ExampleDataGenerator.class);
+
+public static void main(String[] args) throws Exception {
+
+final File asnFile = new
File(ExampleDataGenerator.class.getResource("/example.asn").getFile());
+final File dir = new
File(asnFile.getParentFile().getParentFile().getParentFile(),
"src/test/resources/examples");
+
+generateBasicTypes(dir);
+
+generateComposite(dir);
+}
+
+private static void generateBasicTypes(File dir) throws IOException {
+final File file = new File(dir, "basic-types.dat");
+try (final ReverseByteArrayOutputStream rev = new
ReverseByteArrayOutputStream(1024);
+ final OutputStream out = new FileOutputStream(file)) {
+final BasicTypes basicTypes = new BasicTypes();
+basicTypes.setB(new BerBoolean(true));
+basicTypes.setI(new BerInteger(789));
+basicTypes.setOctStr(new BerOctetString(new byte[]{1, 2, 3, 4,
5}));
+basicTypes.setUtf8Str(new BerUTF8String("Some UTF-8 String.
こんにちは世界。"));
+final int encoded = basicTypes.encode(rev);
+out.write(rev.getArray(), 0, encoded);
+LOG.info("Generated {} bytes to {}", encoded, file);
+}
+}
+
+private static void generateComposite(File dir) throws IOException {
+final File file = new File(dir, "composite.dat");
Review comment:
The template uses `ForkRecord` for the composite data. It seems that
`ForkRecord` doesn't work. Not sure what exactly the issue is but I think it
might have something to do with the fact that no matter the settings it uses
the original schema when writing the extracted records. Not sure if the record
writer logic has changed at some point or this processor never worked properly,
at least with certain setup & input.
In any case, this doesn't seem to be related to the ASN.1 functionality.
I'll replace that processor with a different one in the template.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (MINIFICPP-1508) Remove the legacy Regex wrapper and use std::regex instead
Ferenc Gerlits created MINIFICPP-1508:
-
Summary: Remove the legacy Regex wrapper and use std::regex instead
Key: MINIFICPP-1508
URL: https://issues.apache.org/jira/browse/MINIFICPP-1508
Project: Apache NiFi MiNiFi C++
Issue Type: Improvement
Reporter: Ferenc Gerlits
After MINIFICPP-1429, when we no longer support gcc 4.8, remove the
{{utils::Regex}} class, and delete the files RegexUtils.h, RegexUtils.cpp and
RegexUtilsTests.cpp. Instead, we should use {{std::regex}} directly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [nifi] simonbence closed pull request #4821: NIFI-8113 Adding persistent status history repository backed by embed…
simonbence closed pull request #4821: URL: https://github.com/apache/nifi/pull/4821 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] simonbence commented on pull request #4821: NIFI-8113 Adding persistent status history repository backed by embed…
simonbence commented on pull request #4821: URL: https://github.com/apache/nifi/pull/4821#issuecomment-784292570 I do abandon this review. Based on @markap14 's great comments, I simplified the configuration which resulted a more clean codebase as well. This comes with some changes I reverted and in order to keep the review readable I decided to create a new. All comments in this review are answered or aimed here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] simonbence commented on a change in pull request #4821: NIFI-8113 Adding persistent status history repository backed by embed…
simonbence commented on a change in pull request #4821:
URL: https://github.com/apache/nifi/pull/4821#discussion_r581143770
##
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/status/history/EmbeddedQuestDbRolloverHandler.java
##
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.controller.status.history;
+
+import io.questdb.cairo.sql.Record;
+import io.questdb.cairo.sql.RecordCursor;
+import io.questdb.cairo.sql.RecordCursorFactory;
+import io.questdb.griffin.SqlCompiler;
+import io.questdb.griffin.SqlExecutionContext;
+import org.apache.commons.lang3.time.FastDateFormat;
+import org.apache.nifi.controller.status.history.questdb.QuestDbContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * QuestDB does not provide the possibility for deleting individual lines.
Instead there is the option to drop older
+ * partitions. In order to clean up older status information, the partitions
are outside of the scope of data we intend
+ * to keep will be deleted.
+ */
+public class EmbeddedQuestDbRolloverHandler implements Runnable {
+private static final Logger LOGGER =
LoggerFactory.getLogger(EmbeddedQuestDbRolloverHandler.class);
+
+// Drop keyword is intentionally not uppercase as the query parser only
recognizes it in this way
+private static final String DELETION_QUERY = "ALTER TABLE %s drop
PARTITION '%s'";
+// Distinct keyword is not recognized if the date mapping is not within an
inner query
+static final String SELECTION_QUERY = "SELECT DISTINCT * FROM (SELECT
(to_str(capturedAt, '-MM-dd')) AS partitionName FROM %s)";
+
+static final FastDateFormat DATE_FORMAT =
FastDateFormat.getInstance("-MM-dd");
+
+private final QuestDbContext dbContext;
+private final List tables = new ArrayList<>();
+private final int daysToKeepData;
+
+public EmbeddedQuestDbRolloverHandler(final Collection tables,
final int daysToKeepData, final QuestDbContext dbContext) {
+this.tables.addAll(tables);
+this.dbContext = dbContext;
+this.daysToKeepData = daysToKeepData;
+}
+
+@Override
+public void run() {
+LOGGER.debug("Starting rollover");
+tables.forEach(tableName -> rollOverTable(tableName));
+LOGGER.debug("Finishing rollover");
+}
+
+private void rollOverTable(final CharSequence tableName) {
+try {
+final Set partitions = getPartitions(tableName);
+final Set partitionToKeep = getPartitionsToKeep();
+
+for (final String partition : partitions) {
+if (!partitionToKeep.contains(partition)) {
+deletePartition(tableName, partition);
+}
+}
+} catch (final Exception e) {
+LOGGER.error("Could not rollover table " + tableName, e);
+}
+}
+
+private void deletePartition(final CharSequence tableName, final String
partition) {
+try (final SqlCompiler compiler = dbContext.getCompiler()) {
+compiler.compile(String.format(DELETION_QUERY, new
Object[]{tableName, partition}), dbContext.getSqlExecutionContext());
+} catch (final Exception e) {
+LOGGER.error("Dropping partition " + partition + " of table " +
tableName + " failed", e);
+}
+}
+
+private Set getPartitions(final CharSequence tableName) throws
Exception {
+final SqlExecutionContext executionContext =
dbContext.getSqlExecutionContext();
+final Set result = new HashSet<>();
+
+try (
+final SqlCompiler compiler = dbContext.getCompiler();
+final RecordCursorFactory recordCursorFactory =
compiler.compile(String.format(SELECTION_QUERY, new Object[]{tableName}),
executionContext).getRecordCursorFactory();
+final RecordCursor cursor =
recordCursorFactory.getCursor(executionContext);
+) {
+while (cursor.hasN
[GitHub] [nifi-minifi-cpp] arpadboda closed pull request #1014: MINIFICPP-1506 - Apply fix for high cpu usage
arpadboda closed pull request #1014: URL: https://github.com/apache/nifi-minifi-cpp/pull/1014 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-8237) PutDatabaseRecord can not insert into mysql table with TEXT column
[
https://issues.apache.org/jira/browse/NIFI-8237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289123#comment-17289123
]
ASF subversion and git services commented on NIFI-8237:
---
Commit 200c04c6d04ef56b45e8d85621e834c67f22cea4 in nifi's branch
refs/heads/main from Matt Burgess
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=200c04c ]
NIFI-8237: This closes #4835. Added missing SQL types to
getDataTypeFromSQLTypeValue(), added defensive code
Signed-off-by: Joe Witt
> PutDatabaseRecord can not insert into mysql table with TEXT column
> --
>
> Key: NIFI-8237
> URL: https://issues.apache.org/jira/browse/NIFI-8237
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Environment: Linux hb3-prod-gem-svc1-001 3.10.0-1127.19.1.el7.x86_64
> #1 SMP Tue Aug 25 17:23:54 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
> openjdk version "1.8.0_282"
> OpenJDK Runtime Environment (build 1.8.0_282-b08)
> OpenJDK 64-Bit Server VM (build 25.282-b08, mixed mode)
>Reporter: macdoor615
>Assignee: Matt Burgess
>Priority: Critical
> Fix For: 1.14.0, 1.13.1
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> 1. create table on mysql server
> {code:java}
> CREATE TABLE tmp_text (
> ver varchar(10) NULL,
> richtext text NULL
> ) ;{code}
> 2. insert data into table with PutDatabaseRecord processor , raise error
> {code:java}
> 2021-02-19 00:43:04,424 ERROR [Timer-Driven Process Thread-3]
> o.a.n.p.standard.PutDatabaseRecord
> PutDatabaseRecord[id=b54f68de-0177-1000-917a-119d1bd6f5bf] Failed to put
> Records to database for
> StandardFlowFileRecord[uuid=3ad1b100-5661-4988-a21b-795b24cc5222,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1613666502360-8, container=default,
> section=8], offset=87073,
> length=17],offset=0,name=3ad1b100-5661-4988-a21b-795b24cc5222,size=17].
> Routing to failure.: java.lang.NullPointerException
> java.lang.NullPointerException: null
> at
> org.apache.nifi.serialization.record.util.DataTypeUtils.convertType(DataTypeUtils.java:181)
> at
> org.apache.nifi.serialization.record.util.DataTypeUtils.convertType(DataTypeUtils.java:152)
> at
> org.apache.nifi.serialization.record.util.DataTypeUtils.convertType(DataTypeUtils.java:148)
> at
> org.apache.nifi.processors.standard.PutDatabaseRecord.executeDML(PutDatabaseRecord.java:707)
> at
> org.apache.nifi.processors.standard.PutDatabaseRecord.putToDatabase(PutDatabaseRecord.java:838)
> at
> org.apache.nifi.processors.standard.PutDatabaseRecord.onTrigger(PutDatabaseRecord.java:487)
> at
> org.pache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1173)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
>
> 3. downgrade to Nifi 1.12.1, no error, data inserted into table
> 4. change TEXT column to VARCHAR, no error
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (NIFI-8237) PutDatabaseRecord can not insert into mysql table with TEXT column
[
https://issues.apache.org/jira/browse/NIFI-8237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe Witt updated NIFI-8237:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
> PutDatabaseRecord can not insert into mysql table with TEXT column
> --
>
> Key: NIFI-8237
> URL: https://issues.apache.org/jira/browse/NIFI-8237
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Environment: Linux hb3-prod-gem-svc1-001 3.10.0-1127.19.1.el7.x86_64
> #1 SMP Tue Aug 25 17:23:54 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
> openjdk version "1.8.0_282"
> OpenJDK Runtime Environment (build 1.8.0_282-b08)
> OpenJDK 64-Bit Server VM (build 25.282-b08, mixed mode)
>Reporter: macdoor615
>Assignee: Matt Burgess
>Priority: Critical
> Fix For: 1.14.0, 1.13.1
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> 1. create table on mysql server
> {code:java}
> CREATE TABLE tmp_text (
> ver varchar(10) NULL,
> richtext text NULL
> ) ;{code}
> 2. insert data into table with PutDatabaseRecord processor , raise error
> {code:java}
> 2021-02-19 00:43:04,424 ERROR [Timer-Driven Process Thread-3]
> o.a.n.p.standard.PutDatabaseRecord
> PutDatabaseRecord[id=b54f68de-0177-1000-917a-119d1bd6f5bf] Failed to put
> Records to database for
> StandardFlowFileRecord[uuid=3ad1b100-5661-4988-a21b-795b24cc5222,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1613666502360-8, container=default,
> section=8], offset=87073,
> length=17],offset=0,name=3ad1b100-5661-4988-a21b-795b24cc5222,size=17].
> Routing to failure.: java.lang.NullPointerException
> java.lang.NullPointerException: null
> at
> org.apache.nifi.serialization.record.util.DataTypeUtils.convertType(DataTypeUtils.java:181)
> at
> org.apache.nifi.serialization.record.util.DataTypeUtils.convertType(DataTypeUtils.java:152)
> at
> org.apache.nifi.serialization.record.util.DataTypeUtils.convertType(DataTypeUtils.java:148)
> at
> org.apache.nifi.processors.standard.PutDatabaseRecord.executeDML(PutDatabaseRecord.java:707)
> at
> org.apache.nifi.processors.standard.PutDatabaseRecord.putToDatabase(PutDatabaseRecord.java:838)
> at
> org.apache.nifi.processors.standard.PutDatabaseRecord.onTrigger(PutDatabaseRecord.java:487)
> at
> org.pache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1173)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
>
> 3. downgrade to Nifi 1.12.1, no error, data inserted into table
> 4. change TEXT column to VARCHAR, no error
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [nifi] asfgit closed pull request #4835: NIFI-8237: Added missing SQL types to getDataTypeFromSQLTypeValue(), added defensive code
asfgit closed pull request #4835: URL: https://github.com/apache/nifi/pull/4835 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-8250) FTP Processors Get/List/Fetch/Put has issue with non-English characters
[ https://issues.apache.org/jira/browse/NIFI-8250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289122#comment-17289122 ] ASF subversion and git services commented on NIFI-8250: --- Commit 9101160cbfbc8bf39c2d781522ee6919fbd0c7ec in nifi's branch refs/heads/main from Pierre Villard [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=9101160 ] NIFI-8250 - This closes #4838. UTF8 encoding for FTP processors Signed-off-by: Joe Witt > FTP Processors Get/List/Fetch/Put has issue with non-English characters > > > Key: NIFI-8250 > URL: https://issues.apache.org/jira/browse/NIFI-8250 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.13.0 >Reporter: Jens M Kofoed >Assignee: Pierre Villard >Priority: Major > Fix For: 1.14.0, 1.13.1 > > Time Spent: 10m > Remaining Estimate: 0h > > Create a file with the following filename: امتحان.txt (google translate for > test in Arabic) > GetFTP and ListFTP will both show the filename as ??.txt > ListFTP does not have an attribute "Use UTF-8 Encoding" but the other FTP > processors has. > If using an UpdateAttribute process to set the filename as an input to > FetchFTP, it will still report that it can not find the file > Using a PutFTP the filename at the FTP server become امتØان.txt -- This message was sent by Atlassian Jira (v8.3.4#803005)
