[GitHub] [nifi] KuKuDeCheng opened a new pull request #4539: NIFI-7824:GenerateTableFetch Improvement

2020-09-20 Thread GitBox 


KuKuDeCheng opened a new pull request #4539:
URL: https://github.com/apache/nifi/pull/4539


   ‘Custom ORDER BY Column’ should have a higher priority than 'Maximum-value 
Columns' for generating query sqls when using 'order by'.
   
   Thank you for submitting a contribution to Apache NiFi.
   
   Please provide a short description of the PR here:
   
    Description of PR
   
   _Enables X functionality; fixes bug NIFI-._
   
   In order to streamline the review of the contribution we ask you
   to ensure the following steps have been taken:
   
   ### For all changes:
   - [√] Is there a JIRA ticket associated with this PR? Is it referenced 
in the commit message?
   
   - [√] Does your PR title start with **NIFI-** where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
   
   - [√] Has your PR been rebased against the latest commit within the target 
branch (typically `main`)?
   
   - [√] Is your initial contribution a single, squashed commit? _Additional 
commits in response to PR reviewer feedback should be made on this branch and 
pushed to allow change tracking. Do not `squash` or use `--force` when pushing 
to allow for clean monitoring of changes._
   
   ### For code changes:
   - [√] Have you ensured that the full suite of tests is executed via `mvn 
-Pcontrib-check clean install` at the root `nifi` folder?
   - [√] Have you written or updated unit tests to verify your changes?
   - [√] Have you verified that the full build is successful on JDK 8?
   - [ ] Have you verified that the full build is successful on JDK 11?
   - [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
   - [ ] If applicable, have you updated the `LICENSE` file, including the main 
`LICENSE` file under `nifi-assembly`?
   - [ ] If applicable, have you updated the `NOTICE` file, including the main 
`NOTICE` file found under `nifi-assembly`?
   - [ ] If adding new Properties, have you added `.displayName` in addition to 
.name (programmatic access) for each of the new properties?
   
   ### For documentation related changes:
   - [ ] Have you ensured that format looks appropriate for the output in which 
it is rendered?
   
   ### Note:
   Please ensure that once the PR is submitted, you check GitHub Actions CI for 
build issues and submit an update to your PR as soon as possible.
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (NIFI-7824) GenerateTableFetch Improvement

2020-09-20 Thread ZhangCheng (Jira) 
ZhangCheng created NIFI-7824:


 Summary: GenerateTableFetch Improvement
 Key: NIFI-7824
 URL: https://issues.apache.org/jira/browse/NIFI-7824
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Extensions
Reporter: ZhangCheng
Assignee: ZhangCheng


‘Custom ORDER BY Column’  should have a higher priority than 'Maximum-value 
Columns'  for genarating query sqls when using 'order by'.
For example: we usually use timestamp column as the 'Maximum-value Columns' , 
and Many values of timestamp are often the same. In this case, querying data 
often results in duplicate data because of 'order by timestamp '.So if we use 
timestamp as  'Maximum-value Columns' ,and use a unique column as ‘Custom ORDER 
BY Column’, and use 'order by unique column ', we would not query duplicate 
data.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (NIFI-7822) Set raw query string attribute in HandleHttpRequest processor

2020-09-20 Thread Otto Fowler (Jira) 


[ 
https://issues.apache.org/jira/browse/NIFI-7822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17199121#comment-17199121
 ] 

Otto Fowler commented on NIFI-7822:
---

I'm not saying we shouldn't do the raw sting option, just suggesting that we 
implement it such that we don't have multiple copies of things.


> Set raw query string attribute in HandleHttpRequest processor
> -
>
> Key: NIFI-7822
> URL: https://issues.apache.org/jira/browse/NIFI-7822
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.11.4
>Reporter: Yuriy Flyud
>Priority: Major
>
> HandleHttpRequest parses a query string and writes output to the following 
> attributes:
>  * http.query.string
>  * http.query.param.XXX
> The problem is that neither of these two options works as expected if I want 
> to use query parameters later in my flow.
> First option holds a DECODED query string, so if we are using some characters 
> like & or = in query parameter names or values - there is no way to resolve 
> this. E.g. query string 'text=abc%26notAProp%3D25' will be decoded to 
> 'text=abc¬AProp=25' which is a completely different query string with 
> additional parameter.
> Second option does not handle duplicating query parameters. So query like 
> 'name=John&name=Colin' will be resolved to a single attribute 'name', with 
> value 'Colin'.
>  
> A simple improvement would be to add an 'http.query.raw.string' at least to 
> give a possibility to parse this query string manually where needed.
> '



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Comment Edited] (NIFI-7822) Set raw query string attribute in HandleHttpRequest processor

2020-09-20 Thread Yuriy Flyud (Jira) 


[ 
https://issues.apache.org/jira/browse/NIFI-7822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17199084#comment-17199084
 ] 

Yuriy Flyud edited comment on NIFI-7822 at 9/20/20, 7:51 PM:
-

As a workaround I implemented a Groovy script that follows a HandleHttpRequest 
processor. It finds a response object by identifier from HttpContextMap, gets a 
request for it (Jetty specific implementation here), and calls a 
getQueryString() method.
{code:java}
final def IDENTIFIER_ATTR = 'http.context.identifier'
flowFile = session.get()
def contextIdentifier = flowFile.getAttribute(IDENTIFIER_ATTR)
def response = CTL.HttpContextMap.getResponse(contextIdentifier);
//context identifier might be expired already, or a different context map is 
set.
if(response == null){
   session.transfer(flowFile, REL_FAILURE);
   log.error("Failed to respond to HTTP request for $flowFile because FlowFile 
had an '$IDENTIFIER_ATTR' attribute of contextIdentifier but could not find an 
HTTP Response Object for this identifier")
   return
}
def request = response.getHttpChannel().getRequest()
flowFile = session.putAttribute(flowFile, 
'http.query.raw.string',request.getQueryString())
session.transfer(flowFile, REL_SUCCESS)
{code}
This might be useful for somebody who has the same need in an encoded query 
string until an improvement is implemented.


was (Author: yuflyud):
As a workaround I implemented a Groovy script that follows a HandleHttpRequest 
processor. It finds a response object by identifier from HttpContextMap, gets a 
request for it (Jetty specific implementation here), and calls a 
getQueryString() method.
{code:java}
final def IDENTIFIER_ATTR = 'http.context.identifier'
flowFile = session.get()
def contextIdentifier = flowFile.getAttribute(IDENTIFIER_ATTR)
def response = CTL.HttpContextMap.getResponse(contextIdentifier);
//context identifier might be expired already, or a different context map is 
set.
if(response == null){
   session.transfer(flowFile, REL_FAILURE);
   log.error("Failed to respond to HTTP request for $flowFile because FlowFile 
had an '$IDENTIFIER_ATTR' attribute of contextIdentifier but could not find an 
HTTP Response Object for this identifier")
   return
}
def request = response.getHttpChannel().getRequest()
flowFile = session.putAttribute(flowFile, 
'http.query.raw.string',request.getQueryString())
session.transfer(flowFile, REL_SUCCESS)
{code}
This might be useful for somebody who has the same need in an encoded query 
string.

> Set raw query string attribute in HandleHttpRequest processor
> -
>
> Key: NIFI-7822
> URL: https://issues.apache.org/jira/browse/NIFI-7822
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.11.4
>Reporter: Yuriy Flyud
>Priority: Major
>
> HandleHttpRequest parses a query string and writes output to the following 
> attributes:
>  * http.query.string
>  * http.query.param.XXX
> The problem is that neither of these two options works as expected if I want 
> to use query parameters later in my flow.
> First option holds a DECODED query string, so if we are using some characters 
> like & or = in query parameter names or values - there is no way to resolve 
> this. E.g. query string 'text=abc%26notAProp%3D25' will be decoded to 
> 'text=abc¬AProp=25' which is a completely different query string with 
> additional parameter.
> Second option does not handle duplicating query parameters. So query like 
> 'name=John&name=Colin' will be resolved to a single attribute 'name', with 
> value 'Colin'.
>  
> A simple improvement would be to add an 'http.query.raw.string' at least to 
> give a possibility to parse this query string manually where needed.
> '



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (NIFI-7822) Set raw query string attribute in HandleHttpRequest processor

2020-09-20 Thread Yuriy Flyud (Jira) 


[ 
https://issues.apache.org/jira/browse/NIFI-7822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17199084#comment-17199084
 ] 

Yuriy Flyud commented on NIFI-7822:
---

As a workaround I implemented a Groovy script that follows a HandleHttpRequest 
processor. It finds a response object by identifier from HttpContextMap, gets a 
request for it (Jetty specific implementation here), and calls a 
getQueryString() method.
{code:java}
final def IDENTIFIER_ATTR = 'http.context.identifier'
flowFile = session.get()
def contextIdentifier = flowFile.getAttribute(IDENTIFIER_ATTR)
def response = CTL.HttpContextMap.getResponse(contextIdentifier);
//context identifier might be expired already, or a different context map is 
set.
if(response == null){
   session.transfer(flowFile, REL_FAILURE);
   log.error("Failed to respond to HTTP request for $flowFile because FlowFile 
had an '$IDENTIFIER_ATTR' attribute of contextIdentifier but could not find an 
HTTP Response Object for this identifier")
   return
}
def request = response.getHttpChannel().getRequest()
flowFile = session.putAttribute(flowFile, 
'http.query.raw.string',request.getQueryString())
session.transfer(flowFile, REL_SUCCESS)
{code}
This might be useful for somebody who has the same need in an encoded query 
string.

> Set raw query string attribute in HandleHttpRequest processor
> -
>
> Key: NIFI-7822
> URL: https://issues.apache.org/jira/browse/NIFI-7822
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.11.4
>Reporter: Yuriy Flyud
>Priority: Major
>
> HandleHttpRequest parses a query string and writes output to the following 
> attributes:
>  * http.query.string
>  * http.query.param.XXX
> The problem is that neither of these two options works as expected if I want 
> to use query parameters later in my flow.
> First option holds a DECODED query string, so if we are using some characters 
> like & or = in query parameter names or values - there is no way to resolve 
> this. E.g. query string 'text=abc%26notAProp%3D25' will be decoded to 
> 'text=abc¬AProp=25' which is a completely different query string with 
> additional parameter.
> Second option does not handle duplicating query parameters. So query like 
> 'name=John&name=Colin' will be resolved to a single attribute 'name', with 
> value 'Colin'.
>  
> A simple improvement would be to add an 'http.query.raw.string' at least to 
> give a possibility to parse this query string manually where needed.
> '



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (NIFI-7818) PutHDFS results in EOF due to HDFS-15191

2020-09-20 Thread Pierre Villard (Jira) 


[ 
https://issues.apache.org/jira/browse/NIFI-7818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17199069#comment-17199069
 ] 

Pierre Villard commented on NIFI-7818:
--

Unfortunately, hadoop-client 3.2.2 nor 3.3.1 have been released yet. AFAICT a 
discussion started earlier in September to have a 3.2.2 release. I don't know 
when this will happen. An option is to downgrade to 3.2.0 where the issue does 
not seem to be existing. If you remain on NiFi 1.11.4 where the version 3.2.0 
is used, it should be fine.

> PutHDFS results in EOF due to HDFS-15191
> 
>
> Key: NIFI-7818
> URL: https://issues.apache.org/jira/browse/NIFI-7818
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Jason Iannone
>Priority: Major
>
> Nifi 1.11.4, 1.12.0, and 1.12.1-RC1 are using HDFS client libraries 3.2.1 
> which are impacted by https://issues.apache.org/jira/browse/HDFS-15191. This 
> Jira is to request upgrading Nifi client hadoop libraries to 3.2.2 or higher 
> where the issue has been fixed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (NIFI-5061) NiFi documentation incomplete/wrong for EL hierarchy.

2020-09-20 Thread Pierre Villard (Jira) 


 [ 
https://issues.apache.org/jira/browse/NIFI-5061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard resolved NIFI-5061.
--
Fix Version/s: 1.13.0
   Resolution: Fixed

> NiFi documentation incomplete/wrong for EL hierarchy.
> -
>
> Key: NIFI-5061
> URL: https://issues.apache.org/jira/browse/NIFI-5061
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation & Website
>Affects Versions: 1.5.0
> Environment: N/A
>Reporter: Matthew Clarke
>Assignee: Alasdair Brown
>Priority: Major
>  Labels: newbie
> Fix For: 1.13.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Within the NiFi Expression Language Guide under the "Structure of a NiFi 
> Expression" section, the hierarchy of how NiFi searches for a Attribute key 
> is wrong (still based off NiFi 0.x hierarchy) and incomplete:
> Current text:
> In this example, the value to be returned is the value of the "my attribute" 
> value, if it exists. If that attribute does not exist, the Expression 
> Language will then look for a System Environment Variable named "my 
> attribute." If unable to find this, it will look for a JVM System Property 
> named "my attribute." Finally, if none of these exists, the Expression 
> Language will return a null value.
> The current hierarchy in NiFi is as follows:
> 1. Search FlowFile for Attribute/key
> 2. Search Process Group for attribute/key
> 3. Search File Registry File for attribute/key
> 4. Search NiFi JVM Properties for attribute/key
> 5. Search System Environment Variables for attribute/key
> We should not only fix the above, but in doing so make the above hierarchy 
> more prominent in the documentation.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (NIFI-5061) NiFi documentation incomplete/wrong for EL hierarchy.

2020-09-20 Thread ASF subversion and git services (Jira) 


[ 
https://issues.apache.org/jira/browse/NIFI-5061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17199065#comment-17199065
 ] 

ASF subversion and git services commented on NIFI-5061:
---

Commit 953abba6d609f93f5b4e12772e25802007719901 in nifi's branch 
refs/heads/main from abrown
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=953abba ]

NIFI-5061-NiFi documentation incomplete/wrong for EL hierarchy.

Co-authored-by: Pierre Villard 
Signed-off-by: Pierre Villard 

This closes #4530.


> NiFi documentation incomplete/wrong for EL hierarchy.
> -
>
> Key: NIFI-5061
> URL: https://issues.apache.org/jira/browse/NIFI-5061
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation & Website
>Affects Versions: 1.5.0
> Environment: N/A
>Reporter: Matthew Clarke
>Assignee: Alasdair Brown
>Priority: Major
>  Labels: newbie
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Within the NiFi Expression Language Guide under the "Structure of a NiFi 
> Expression" section, the hierarchy of how NiFi searches for a Attribute key 
> is wrong (still based off NiFi 0.x hierarchy) and incomplete:
> Current text:
> In this example, the value to be returned is the value of the "my attribute" 
> value, if it exists. If that attribute does not exist, the Expression 
> Language will then look for a System Environment Variable named "my 
> attribute." If unable to find this, it will look for a JVM System Property 
> named "my attribute." Finally, if none of these exists, the Expression 
> Language will return a null value.
> The current hierarchy in NiFi is as follows:
> 1. Search FlowFile for Attribute/key
> 2. Search Process Group for attribute/key
> 3. Search File Registry File for attribute/key
> 4. Search NiFi JVM Properties for attribute/key
> 5. Search System Environment Variables for attribute/key
> We should not only fix the above, but in doing so make the above hierarchy 
> more prominent in the documentation.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [nifi] asfgit closed pull request #4530: NIFI-5061-NiFi documentation incomplete/wrong for EL hierarchy.

2020-09-20 Thread GitBox 


asfgit closed pull request #4530:
URL: https://github.com/apache/nifi/pull/4530


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (NIFI-7822) Set raw query string attribute in HandleHttpRequest processor

2020-09-20 Thread Otto Fowler (Jira) 


[ 
https://issues.apache.org/jira/browse/NIFI-7822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17199038#comment-17199038
 ] 

Otto Fowler commented on NIFI-7822:
---

I would think that we may want to consider having a query attribute strategy, 
were you could say RAW or Attributes or something maybe.  I wouldn't think we 
would want to have multiple copies of that.

As for the parameters, they are a map, so they must have unique names.  If you 
have something that has duplicated parameters, there is no way around this, 
unless the parameter names are made unique before putting in attributes, so if 
the processor says
" if we already have an attribute with this name, from this query, append a 
number to the name" or something.


> Set raw query string attribute in HandleHttpRequest processor
> -
>
> Key: NIFI-7822
> URL: https://issues.apache.org/jira/browse/NIFI-7822
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.11.4
>Reporter: Yuriy Flyud
>Priority: Major
>
> HandleHttpRequest parses a query string and writes output to the following 
> attributes:
>  * http.query.string
>  * http.query.param.XXX
> The problem is that neither of these two options works as expected if I want 
> to use query parameters later in my flow.
> First option holds a DECODED query string, so if we are using some characters 
> like & or = in query parameter names or values - there is no way to resolve 
> this. E.g. query string 'text=abc%26notAProp%3D25' will be decoded to 
> 'text=abc¬AProp=25' which is a completely different query string with 
> additional parameter.
> Second option does not handle duplicating query parameters. So query like 
> 'name=John&name=Colin' will be resolved to a single attribute 'name', with 
> value 'Colin'.
>  
> A simple improvement would be to add an 'http.query.raw.string' at least to 
> give a possibility to parse this query string manually where needed.
> '



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (NIFI-7731) Apache Nifi's CaptureChangeMySQL processor is not able to capture DDL events from AWS RDS MySQL DB, but the BEGIN, COMIT, INSERT, UPDATE AND DELETE are being captured

2020-09-20 Thread zeyk (Jira) 


 [ 
https://issues.apache.org/jira/browse/NIFI-7731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zeyk resolved NIFI-7731.

Resolution: Fixed

> Apache Nifi's CaptureChangeMySQL processor is not able to capture DDL events 
> from AWS RDS MySQL DB, but the BEGIN, COMIT, INSERT, UPDATE AND DELETE are 
> being captured
> --
>
> Key: NIFI-7731
> URL: https://issues.apache.org/jira/browse/NIFI-7731
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Tools and Build
> Environment: OS : Ubuntu EC2 image
>Reporter: zeyk
>Priority: Major
>
> Apache Nifi's CaptureChangeMySQL processor is not able to capture DDL events 
> from AWS RDS MySQL DB, but the BEGIN, COMMIT, INSERT, UPDATE AND DELETE are 
> being captured



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (NIFI-7785) CaptureChangeMySQL processor captures enum values as "INDEX of those values" from Mysql DB"

2020-09-20 Thread zeyk (Jira) 


 [ 
https://issues.apache.org/jira/browse/NIFI-7785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zeyk resolved NIFI-7785.

Resolution: Fixed

> CaptureChangeMySQL processor captures enum values as "INDEX of those values" 
> from Mysql DB"
> ---
>
> Key: NIFI-7785
> URL: https://issues.apache.org/jira/browse/NIFI-7785
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Tools and Build
>Affects Versions: 1.11.4
> Environment: Ubuntu EC2 instance with 8 GB ram
>Reporter: zeyk
>Priority: Major
>  Labels: features
> Attachments: Screenshot from 2020-09-15 05-29-48.png, Screenshot from 
> 2020-09-15 08-05-33.png, flow.xml.gz, sample_with_enum.xml, sqlcmdReturn.png
>
>
> CaptureChangeMySQL processor captures enum values as "INDEX of those values" 
> rather than the values specified.
> for example:
> A table has columns (id int, fruit enum ('apple','pears','orange'), price int)
> On doing an insert:
> insert into (1,'apple',45)
> insert into (2,'pears',56)
> I have used CaptureChangeMySql processor to capture the CDC changes, the 
> process does the capture but captures the enum column alone based on its 
> index like the sample below:
> for 1st insert:
>  
> {
>  "type":"insert",
>  "timestamp":1599004442000,
>  "binlog_filename":"mysql-bin-changelog.39",
>  "binlog_position":1537835,
>  "database":"sample",
>  "table_name":"sample",
>  "table_id":82,
>  "columns":[
>  {
>  "id":1,
>  "name":"id",
>  "column_type":-5,
>  "value":139
>  },
>  {
>  "id":2,
>  "name":"fruit",
>  "column_type":12,
>  "value":0
>  },
>  {
>  "id":3,
>  "name":"price",
>  "column_type":12,
>  "value":45
>  }
>  ]
> }
>  
> for 2nd insert:
>  
> {
>  "type":"insert",
>  "timestamp":1599004442000,
>  "binlog_filename":"mysql-bin-changelog.39",
>  "binlog_position":1537835,
>  "database":"sample",
>  "table_name":"sample",
>  "table_id":82,
>  "columns":[
>  {
>  "id":1,
>  "name":"id",
>  "column_type":-5,
>  "value":139
>  },
>  {
>  "id":2,
>  "name":"fruit",
>  "column_type":12,
>  "value":1
>  },
>  {
>  "id":3,
>  "name":"price",
>  "column_type":12,
>  "value":56
>  }
>  ]
> }
>  
>  
> So the above has 0 and 1 in place of apple and pears respectively.
>  
> Could you of you help me on this, if there are folks who have faced similar 
> kinda issue
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (NIFI-7820) How to connect to controller service DBCP connection pool and execute the sql using that connection via python

2020-09-20 Thread zeyk (Jira) 


 [ 
https://issues.apache.org/jira/browse/NIFI-7820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zeyk resolved NIFI-7820.

Resolution: Fixed

> How to connect to controller service DBCP connection pool and execute the sql 
> using that connection via python 
> ---
>
> Key: NIFI-7820
> URL: https://issues.apache.org/jira/browse/NIFI-7820
> Project: Apache NiFi
>  Issue Type: Task
>  Components: NiFi Stateless
>Reporter: zeyk
>Priority: Major
> Attachments: enumsWithPython.xml, execute_script.py, 
> image-2020-09-20-01-44-32-117.png, image-2020-09-20-01-45-19-739.png, 
> image-2020-09-20-01-58-03-615.png, image-2020-09-20-17-36-02-743.png, 
> image-2020-09-20-17-37-12-487.png, image-2020-09-20-17-37-49-409.png
>
>
> I have a python code updated with my own logic to replace the index of enums 
> with values, but in order to do so , i need to execute certain sql commands 
> to get some values and the best thing would be to get the dbcp connection 
> pool connection from nifi controller services and execute the commands , but 
> i dont know to how to implement that in python, since few modules such as 
> pymysql doesnt support in Nifi(Jython) ,Please find the code below:
> Any kind of help would be highly appreciated. Thanks in advance
>  
> {color:#89ddff}import{color}{color:#ee} json{color}
> {color:#89ddff}import{color}{color:#ee} re{color}
> {color:#89ddff}import{color}{color:#ee} sys{color}
> {color:#89ddff}import{color}{color:#ee} traceback{color}
> {color:#89ddff}from{color}{color:#ee} java.nio.charset 
> {color}{color:#89ddff}import{color}{color:#ee} StandardCharsets{color}
> {color:#89ddff}from{color}{color:#ee} org.apache.commons.io 
> {color}{color:#89ddff}import{color}{color:#ee} IOUtils{color}
> {color:#89ddff}from{color}{color:#ee} org.apache.nifi.processor.io 
> {color}{color:#89ddff}import{color}{color:#ee} StreamCallback{color}
> {color:#89ddff}from{color}{color:#ee} org.python.core.util 
> {color}{color:#89ddff}import{color}{color:#ee} StringUtil{color}
> {color:#c792ea}class{color}{color:#ee} 
> {color}{color:#ffcb6b}TransformCallback{color}{color:#89ddff}({color}{color:#c3e88d}StreamCallback{color}{color:#89ddff}):{color}
> {color:#ee}{color}{color:#c792ea}def{color}{color:#ee} 
> {color}{color:#82aaff}__init__{color}{color:#89ddff}({color}{color:#ff5370}self{color}{color:#89ddff}):{color}
> {color:#ee}{color}{color:#89ddff}pass{color}
> {color:#ee}{color}{color:#c792ea}def{color}{color:#ee} 
> {color}{color:#82aaff}process{color}{color:#89ddff}({color}{color:#ff5370}self{color}{color:#89ddff},{color}{color:#ee}
>  
> {color}{color:#ff5370}inputStream{color}{color:#89ddff},{color}{color:#ee}
>  {color}{color:#ff5370}outputStream{color}{color:#89ddff}):{color}
> {color:#ee}{color}{color:#89ddff}try{color}{color:#ee}:{color}
> {color:#89ddff}{color}{color:#546e7a}# Read input FlowFile 
> content{color}
> {color:#ee}input_text 
> {color}{color:#c792ea}={color}{color:#ee} 
> IOUtils.toString{color}{color:#89ddff}({color}{color:#ee}inputStream, 
> StandardCharsets.UTF_8{color}{color:#89ddff}){color}
> {color:#ee}input_obj 
> {color}{color:#c792ea}={color}{color:#ee} 
> json.loads{color}{color:#89ddff}({color}{color:#ee}input_text{color}{color:#89ddff}){color}
> {color:#ee} {color}
> {color:#ee}{color}
> {color:#ee}    {color}{color:#ee}table_name 
> {color}{color:#c792ea}={color}{color:#ee} 
> input_obj{color}{color:#89ddff}[{color}{color:#89ddff}'{color}{color:#c3e88d}table_name{color}{color:#89ddff}'{color}{color:#89ddff}]{color}
> {color:#ee}column_name 
> {color}{color:#c792ea}={color}{color:#ee}  
> {color}{color:#89ddff}"{color}{color:#f78c6c}SELECT{color}{color:#c3e88d} 
> column_name {color}{color:#f78c6c}FROM{color}{color:#c3e88d} 
> INFORMATION_SCHEMA.COLUMNS {color}{color:#f78c6c}WHERE{color}{color:#c3e88d} 
> table_name{color}{color:#c792ea}={color}{color:#89ddff}'{color}{color:#c3e88d}"+table_name+"'
>  {color}{color:#f78c6c}AND{color}{color:#c3e88d} 
> data_type{color}{color:#c792ea}={color}{color:#89ddff}'{color}{color:#c3e88d}enum'{color}{color:#89ddff}"{color}
> {color:#ee}enum_value_sql 
> {color}{color:#c792ea}={color}{color:#ee} 
> {color}{color:#89ddff}"{color}{color:#f78c6c}SELECT{color}{color:#c3e88d} 
> {color}{color:#82aaff}REPLACE{color}{color:#c3e88d}({color}{color:#82aaff}REPLACE{color}{color:#c3e88d}({color}{color:#82aaff}REPLACE{color}{color:#c3e88d}({color}{color:#82aaff}REPLACE{color}{color:#c3e88d}(column_type,{color}{color:#89ddff}'{color}{color:#c3e88d}enum',{c

[jira] [Created] (NIFI-7823) Retrieve groups from OIDC token using KeyClock

2020-09-20 Thread Raz Dobkies (Jira) 
Raz Dobkies created NIFI-7823:
-

 Summary: Retrieve groups from OIDC token using KeyClock
 Key: NIFI-7823
 URL: https://issues.apache.org/jira/browse/NIFI-7823
 Project: Apache NiFi
  Issue Type: Wish
  Components: Security
Reporter: Raz Dobkies


Nifi will can retrueve groups from a token if we will develop 
KeyCloakUserGroupProvider.

Keycloak does implement a lot of related functionality, including groups, 
users, and such. It would be great to have a first-class integration available.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (NIFI-7822) Set raw query string attribute in HandleHttpRequest processor

2020-09-20 Thread Yuriy Flyud (Jira) 
Yuriy Flyud created NIFI-7822:
-

 Summary: Set raw query string attribute in HandleHttpRequest 
processor
 Key: NIFI-7822
 URL: https://issues.apache.org/jira/browse/NIFI-7822
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Extensions
Affects Versions: 1.11.4
Reporter: Yuriy Flyud


HandleHttpRequest parses a query string and writes output to the following 
attributes:
 * http.query.string
 * http.query.param.XXX

The problem is that neither of these two options works as expected if I want to 
use query parameters later in my flow.

First option holds a DECODED query string, so if we are using some characters 
like & or = in query parameter names or values - there is no way to resolve 
this. E.g. query string 'text=abc%26notAProp%3D25' will be decoded to 
'text=abc¬AProp=25' which is a completely different query string with 
additional parameter.

Second option does not handle duplicating query parameters. So query like 
'name=John&name=Colin' will be resolved to a single attribute 'name', with 
value 'Colin'.

 

A simple improvement would be to add an 'http.query.raw.string' at least to 
give a possibility to parse this query string manually where needed.

'



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (NIFI-7821) Create a Cassandra implementation of DistributedMapCacheClient

2020-09-20 Thread Mike Thomsen (Jira) 
Mike Thomsen created NIFI-7821:
--

 Summary: Create a Cassandra implementation of 
DistributedMapCacheClient
 Key: NIFI-7821
 URL: https://issues.apache.org/jira/browse/NIFI-7821
 Project: Apache NiFi
  Issue Type: New Feature
Reporter: Mike Thomsen
Assignee: Mike Thomsen


Create a Cassandra-based version of DistributedMapCacheClient that includes 
support for TTLs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (NIFI-7820) How to connect to controller service DBCP connection pool and execute the sql using that connection via python

2020-09-20 Thread zeyk (Jira) 


[ 
https://issues.apache.org/jira/browse/NIFI-7820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17198973#comment-17198973
 ] 

zeyk commented on NIFI-7820:


Hi [~RobertoGarcia] ,

I have tried the above method with my own logic in it using 
ExecuteStreamCommand processor. The script throws up error stating "No module 
named mysql" error, but the module has been installed in local computed where 
nifi is installed, moreover the same script runs good from the terminal with 
hard coding the flowfile values. Please find the screenshots below

!image-2020-09-20-17-36-02-743.png!

 

!image-2020-09-20-17-37-49-409.png!

 

!image-2020-09-20-17-37-12-487.png!

 

Updated script:

 

#!/usr/bin/env python3

import re
import sys
import string
import mysql.connector
from mysql.connector import errorcode

config = {
 'user':'data',
 'password':'pass',
 'host':'host',
 'database':'ml'
}

try:
   cnx = mysql.connector.connect(**config)
except mysql.connector.Erro as err:
   if err.errno == errorcode.ER_ACCESS_DENIED_ERROR:
  print("Something is wrong with your user name or password")
   elif err.errno == errorcode.ER_BAD_DB_ERROR:
  print("Database does not exist")
   else:
  print(err)

cursor = cnx.cursor()
flowfile=sys.stdin.readline()
table_name = flowfile['table_name']
col_sql = "SELECT column_name FROM INFORMATION_SCHEMA.COLUMNS WHERE 
table_name='"+table_name+"' AND data_type='enum'"
cursor.execute(col_sql)
column_name = str(cursor.fetchone()[0])
col_query = (
 "SELECT 
REPLACE(REPLACE(REPLACE(REPLACE(column_type,'enum',''),')',''),'(',''),'\\'','')
 enums "
 "FROM INFORMATION_SCHEMA.COLUMNS "
 "WHERE table_name='"+table_name+"' AND column_name='"+column_name+"'")

cursor.execute(col_query)

enum_row = str(cursor.fetchone()[0])
cursor.close()
cnx.close()
enum_val = enum_row.split(',')

for col in flowfile['columns']:
    if col['name'] == str(column_name):
  col['value'] = enum_val[int(col['value']) - 1]


print(flowfile)

> How to connect to controller service DBCP connection pool and execute the sql 
> using that connection via python 
> ---
>
> Key: NIFI-7820
> URL: https://issues.apache.org/jira/browse/NIFI-7820
> Project: Apache NiFi
>  Issue Type: Task
>  Components: NiFi Stateless
>Reporter: zeyk
>Priority: Major
> Attachments: enumsWithPython.xml, execute_script.py, 
> image-2020-09-20-01-44-32-117.png, image-2020-09-20-01-45-19-739.png, 
> image-2020-09-20-01-58-03-615.png, image-2020-09-20-17-36-02-743.png, 
> image-2020-09-20-17-37-12-487.png, image-2020-09-20-17-37-49-409.png
>
>
> I have a python code updated with my own logic to replace the index of enums 
> with values, but in order to do so , i need to execute certain sql commands 
> to get some values and the best thing would be to get the dbcp connection 
> pool connection from nifi controller services and execute the commands , but 
> i dont know to how to implement that in python, since few modules such as 
> pymysql doesnt support in Nifi(Jython) ,Please find the code below:
> Any kind of help would be highly appreciated. Thanks in advance
>  
> {color:#89ddff}import{color}{color:#ee} json{color}
> {color:#89ddff}import{color}{color:#ee} re{color}
> {color:#89ddff}import{color}{color:#ee} sys{color}
> {color:#89ddff}import{color}{color:#ee} traceback{color}
> {color:#89ddff}from{color}{color:#ee} java.nio.charset 
> {color}{color:#89ddff}import{color}{color:#ee} StandardCharsets{color}
> {color:#89ddff}from{color}{color:#ee} org.apache.commons.io 
> {color}{color:#89ddff}import{color}{color:#ee} IOUtils{color}
> {color:#89ddff}from{color}{color:#ee} org.apache.nifi.processor.io 
> {color}{color:#89ddff}import{color}{color:#ee} StreamCallback{color}
> {color:#89ddff}from{color}{color:#ee} org.python.core.util 
> {color}{color:#89ddff}import{color}{color:#ee} StringUtil{color}
> {color:#c792ea}class{color}{color:#ee} 
> {color}{color:#ffcb6b}TransformCallback{color}{color:#89ddff}({color}{color:#c3e88d}StreamCallback{color}{color:#89ddff}):{color}
> {color:#ee}{color}{color:#c792ea}def{color}{color:#ee} 
> {color}{color:#82aaff}__init__{color}{color:#89ddff}({color}{color:#ff5370}self{color}{color:#89ddff}):{color}
> {color:#ee}{color}{color:#89ddff}pass{color}
> {color:#ee}{color}{color:#c792ea}def{color}{color:#ee} 
> {color}{color:#82aaff}process{color}{color:#89ddff}({color}{color:#ff5370}self{color}{color:#89ddff},{color}{color:#ee}
>  
> {color}{color:#ff5370}inputStream{color}{color:#89ddff},{color}{color:#ee}
>  {color}{color:#ff5370}outputStream{color}{color:#89ddff}):{color}
> {color:#ee}{color}{color:#89ddff}try{color}{color:#ee}:{colo

[jira] [Updated] (NIFI-7820) How to connect to controller service DBCP connection pool and execute the sql using that connection via python

2020-09-20 Thread zeyk (Jira) 


 [ 
https://issues.apache.org/jira/browse/NIFI-7820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zeyk updated NIFI-7820:
---
Attachment: image-2020-09-20-17-37-12-487.png

> How to connect to controller service DBCP connection pool and execute the sql 
> using that connection via python 
> ---
>
> Key: NIFI-7820
> URL: https://issues.apache.org/jira/browse/NIFI-7820
> Project: Apache NiFi
>  Issue Type: Task
>  Components: NiFi Stateless
>Reporter: zeyk
>Priority: Major
> Attachments: enumsWithPython.xml, execute_script.py, 
> image-2020-09-20-01-44-32-117.png, image-2020-09-20-01-45-19-739.png, 
> image-2020-09-20-01-58-03-615.png, image-2020-09-20-17-36-02-743.png, 
> image-2020-09-20-17-37-12-487.png, image-2020-09-20-17-37-49-409.png
>
>
> I have a python code updated with my own logic to replace the index of enums 
> with values, but in order to do so , i need to execute certain sql commands 
> to get some values and the best thing would be to get the dbcp connection 
> pool connection from nifi controller services and execute the commands , but 
> i dont know to how to implement that in python, since few modules such as 
> pymysql doesnt support in Nifi(Jython) ,Please find the code below:
> Any kind of help would be highly appreciated. Thanks in advance
>  
> {color:#89ddff}import{color}{color:#ee} json{color}
> {color:#89ddff}import{color}{color:#ee} re{color}
> {color:#89ddff}import{color}{color:#ee} sys{color}
> {color:#89ddff}import{color}{color:#ee} traceback{color}
> {color:#89ddff}from{color}{color:#ee} java.nio.charset 
> {color}{color:#89ddff}import{color}{color:#ee} StandardCharsets{color}
> {color:#89ddff}from{color}{color:#ee} org.apache.commons.io 
> {color}{color:#89ddff}import{color}{color:#ee} IOUtils{color}
> {color:#89ddff}from{color}{color:#ee} org.apache.nifi.processor.io 
> {color}{color:#89ddff}import{color}{color:#ee} StreamCallback{color}
> {color:#89ddff}from{color}{color:#ee} org.python.core.util 
> {color}{color:#89ddff}import{color}{color:#ee} StringUtil{color}
> {color:#c792ea}class{color}{color:#ee} 
> {color}{color:#ffcb6b}TransformCallback{color}{color:#89ddff}({color}{color:#c3e88d}StreamCallback{color}{color:#89ddff}):{color}
> {color:#ee}{color}{color:#c792ea}def{color}{color:#ee} 
> {color}{color:#82aaff}__init__{color}{color:#89ddff}({color}{color:#ff5370}self{color}{color:#89ddff}):{color}
> {color:#ee}{color}{color:#89ddff}pass{color}
> {color:#ee}{color}{color:#c792ea}def{color}{color:#ee} 
> {color}{color:#82aaff}process{color}{color:#89ddff}({color}{color:#ff5370}self{color}{color:#89ddff},{color}{color:#ee}
>  
> {color}{color:#ff5370}inputStream{color}{color:#89ddff},{color}{color:#ee}
>  {color}{color:#ff5370}outputStream{color}{color:#89ddff}):{color}
> {color:#ee}{color}{color:#89ddff}try{color}{color:#ee}:{color}
> {color:#89ddff}{color}{color:#546e7a}# Read input FlowFile 
> content{color}
> {color:#ee}input_text 
> {color}{color:#c792ea}={color}{color:#ee} 
> IOUtils.toString{color}{color:#89ddff}({color}{color:#ee}inputStream, 
> StandardCharsets.UTF_8{color}{color:#89ddff}){color}
> {color:#ee}input_obj 
> {color}{color:#c792ea}={color}{color:#ee} 
> json.loads{color}{color:#89ddff}({color}{color:#ee}input_text{color}{color:#89ddff}){color}
> {color:#ee} {color}
> {color:#ee}{color}
> {color:#ee}    {color}{color:#ee}table_name 
> {color}{color:#c792ea}={color}{color:#ee} 
> input_obj{color}{color:#89ddff}[{color}{color:#89ddff}'{color}{color:#c3e88d}table_name{color}{color:#89ddff}'{color}{color:#89ddff}]{color}
> {color:#ee}column_name 
> {color}{color:#c792ea}={color}{color:#ee}  
> {color}{color:#89ddff}"{color}{color:#f78c6c}SELECT{color}{color:#c3e88d} 
> column_name {color}{color:#f78c6c}FROM{color}{color:#c3e88d} 
> INFORMATION_SCHEMA.COLUMNS {color}{color:#f78c6c}WHERE{color}{color:#c3e88d} 
> table_name{color}{color:#c792ea}={color}{color:#89ddff}'{color}{color:#c3e88d}"+table_name+"'
>  {color}{color:#f78c6c}AND{color}{color:#c3e88d} 
> data_type{color}{color:#c792ea}={color}{color:#89ddff}'{color}{color:#c3e88d}enum'{color}{color:#89ddff}"{color}
> {color:#ee}enum_value_sql 
> {color}{color:#c792ea}={color}{color:#ee} 
> {color}{color:#89ddff}"{color}{color:#f78c6c}SELECT{color}{color:#c3e88d} 
> {color}{color:#82aaff}REPLACE{color}{color:#c3e88d}({color}{color:#82aaff}REPLACE{color}{color:#c3e88d}({color}{color:#82aaff}REPLACE{color}{color:#c3e88d}({color}{color:#82aaff}REPLACE{color}{color:#c3e88d}(column_type,{color}{color:#89ddff}'{col

[jira] [Updated] (NIFI-7820) How to connect to controller service DBCP connection pool and execute the sql using that connection via python

2020-09-20 Thread zeyk (Jira) 


 [ 
https://issues.apache.org/jira/browse/NIFI-7820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zeyk updated NIFI-7820:
---
Attachment: image-2020-09-20-17-37-49-409.png

> How to connect to controller service DBCP connection pool and execute the sql 
> using that connection via python 
> ---
>
> Key: NIFI-7820
> URL: https://issues.apache.org/jira/browse/NIFI-7820
> Project: Apache NiFi
>  Issue Type: Task
>  Components: NiFi Stateless
>Reporter: zeyk
>Priority: Major
> Attachments: enumsWithPython.xml, execute_script.py, 
> image-2020-09-20-01-44-32-117.png, image-2020-09-20-01-45-19-739.png, 
> image-2020-09-20-01-58-03-615.png, image-2020-09-20-17-36-02-743.png, 
> image-2020-09-20-17-37-12-487.png, image-2020-09-20-17-37-49-409.png
>
>
> I have a python code updated with my own logic to replace the index of enums 
> with values, but in order to do so , i need to execute certain sql commands 
> to get some values and the best thing would be to get the dbcp connection 
> pool connection from nifi controller services and execute the commands , but 
> i dont know to how to implement that in python, since few modules such as 
> pymysql doesnt support in Nifi(Jython) ,Please find the code below:
> Any kind of help would be highly appreciated. Thanks in advance
>  
> {color:#89ddff}import{color}{color:#ee} json{color}
> {color:#89ddff}import{color}{color:#ee} re{color}
> {color:#89ddff}import{color}{color:#ee} sys{color}
> {color:#89ddff}import{color}{color:#ee} traceback{color}
> {color:#89ddff}from{color}{color:#ee} java.nio.charset 
> {color}{color:#89ddff}import{color}{color:#ee} StandardCharsets{color}
> {color:#89ddff}from{color}{color:#ee} org.apache.commons.io 
> {color}{color:#89ddff}import{color}{color:#ee} IOUtils{color}
> {color:#89ddff}from{color}{color:#ee} org.apache.nifi.processor.io 
> {color}{color:#89ddff}import{color}{color:#ee} StreamCallback{color}
> {color:#89ddff}from{color}{color:#ee} org.python.core.util 
> {color}{color:#89ddff}import{color}{color:#ee} StringUtil{color}
> {color:#c792ea}class{color}{color:#ee} 
> {color}{color:#ffcb6b}TransformCallback{color}{color:#89ddff}({color}{color:#c3e88d}StreamCallback{color}{color:#89ddff}):{color}
> {color:#ee}{color}{color:#c792ea}def{color}{color:#ee} 
> {color}{color:#82aaff}__init__{color}{color:#89ddff}({color}{color:#ff5370}self{color}{color:#89ddff}):{color}
> {color:#ee}{color}{color:#89ddff}pass{color}
> {color:#ee}{color}{color:#c792ea}def{color}{color:#ee} 
> {color}{color:#82aaff}process{color}{color:#89ddff}({color}{color:#ff5370}self{color}{color:#89ddff},{color}{color:#ee}
>  
> {color}{color:#ff5370}inputStream{color}{color:#89ddff},{color}{color:#ee}
>  {color}{color:#ff5370}outputStream{color}{color:#89ddff}):{color}
> {color:#ee}{color}{color:#89ddff}try{color}{color:#ee}:{color}
> {color:#89ddff}{color}{color:#546e7a}# Read input FlowFile 
> content{color}
> {color:#ee}input_text 
> {color}{color:#c792ea}={color}{color:#ee} 
> IOUtils.toString{color}{color:#89ddff}({color}{color:#ee}inputStream, 
> StandardCharsets.UTF_8{color}{color:#89ddff}){color}
> {color:#ee}input_obj 
> {color}{color:#c792ea}={color}{color:#ee} 
> json.loads{color}{color:#89ddff}({color}{color:#ee}input_text{color}{color:#89ddff}){color}
> {color:#ee} {color}
> {color:#ee}{color}
> {color:#ee}    {color}{color:#ee}table_name 
> {color}{color:#c792ea}={color}{color:#ee} 
> input_obj{color}{color:#89ddff}[{color}{color:#89ddff}'{color}{color:#c3e88d}table_name{color}{color:#89ddff}'{color}{color:#89ddff}]{color}
> {color:#ee}column_name 
> {color}{color:#c792ea}={color}{color:#ee}  
> {color}{color:#89ddff}"{color}{color:#f78c6c}SELECT{color}{color:#c3e88d} 
> column_name {color}{color:#f78c6c}FROM{color}{color:#c3e88d} 
> INFORMATION_SCHEMA.COLUMNS {color}{color:#f78c6c}WHERE{color}{color:#c3e88d} 
> table_name{color}{color:#c792ea}={color}{color:#89ddff}'{color}{color:#c3e88d}"+table_name+"'
>  {color}{color:#f78c6c}AND{color}{color:#c3e88d} 
> data_type{color}{color:#c792ea}={color}{color:#89ddff}'{color}{color:#c3e88d}enum'{color}{color:#89ddff}"{color}
> {color:#ee}enum_value_sql 
> {color}{color:#c792ea}={color}{color:#ee} 
> {color}{color:#89ddff}"{color}{color:#f78c6c}SELECT{color}{color:#c3e88d} 
> {color}{color:#82aaff}REPLACE{color}{color:#c3e88d}({color}{color:#82aaff}REPLACE{color}{color:#c3e88d}({color}{color:#82aaff}REPLACE{color}{color:#c3e88d}({color}{color:#82aaff}REPLACE{color}{color:#c3e88d}(column_type,{color}{color:#89ddff}'{col

[jira] [Updated] (NIFI-7820) How to connect to controller service DBCP connection pool and execute the sql using that connection via python

2020-09-20 Thread zeyk (Jira) 


 [ 
https://issues.apache.org/jira/browse/NIFI-7820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zeyk updated NIFI-7820:
---
Attachment: image-2020-09-20-17-36-02-743.png

> How to connect to controller service DBCP connection pool and execute the sql 
> using that connection via python 
> ---
>
> Key: NIFI-7820
> URL: https://issues.apache.org/jira/browse/NIFI-7820
> Project: Apache NiFi
>  Issue Type: Task
>  Components: NiFi Stateless
>Reporter: zeyk
>Priority: Major
> Attachments: enumsWithPython.xml, execute_script.py, 
> image-2020-09-20-01-44-32-117.png, image-2020-09-20-01-45-19-739.png, 
> image-2020-09-20-01-58-03-615.png, image-2020-09-20-17-36-02-743.png
>
>
> I have a python code updated with my own logic to replace the index of enums 
> with values, but in order to do so , i need to execute certain sql commands 
> to get some values and the best thing would be to get the dbcp connection 
> pool connection from nifi controller services and execute the commands , but 
> i dont know to how to implement that in python, since few modules such as 
> pymysql doesnt support in Nifi(Jython) ,Please find the code below:
> Any kind of help would be highly appreciated. Thanks in advance
>  
> {color:#89ddff}import{color}{color:#ee} json{color}
> {color:#89ddff}import{color}{color:#ee} re{color}
> {color:#89ddff}import{color}{color:#ee} sys{color}
> {color:#89ddff}import{color}{color:#ee} traceback{color}
> {color:#89ddff}from{color}{color:#ee} java.nio.charset 
> {color}{color:#89ddff}import{color}{color:#ee} StandardCharsets{color}
> {color:#89ddff}from{color}{color:#ee} org.apache.commons.io 
> {color}{color:#89ddff}import{color}{color:#ee} IOUtils{color}
> {color:#89ddff}from{color}{color:#ee} org.apache.nifi.processor.io 
> {color}{color:#89ddff}import{color}{color:#ee} StreamCallback{color}
> {color:#89ddff}from{color}{color:#ee} org.python.core.util 
> {color}{color:#89ddff}import{color}{color:#ee} StringUtil{color}
> {color:#c792ea}class{color}{color:#ee} 
> {color}{color:#ffcb6b}TransformCallback{color}{color:#89ddff}({color}{color:#c3e88d}StreamCallback{color}{color:#89ddff}):{color}
> {color:#ee}{color}{color:#c792ea}def{color}{color:#ee} 
> {color}{color:#82aaff}__init__{color}{color:#89ddff}({color}{color:#ff5370}self{color}{color:#89ddff}):{color}
> {color:#ee}{color}{color:#89ddff}pass{color}
> {color:#ee}{color}{color:#c792ea}def{color}{color:#ee} 
> {color}{color:#82aaff}process{color}{color:#89ddff}({color}{color:#ff5370}self{color}{color:#89ddff},{color}{color:#ee}
>  
> {color}{color:#ff5370}inputStream{color}{color:#89ddff},{color}{color:#ee}
>  {color}{color:#ff5370}outputStream{color}{color:#89ddff}):{color}
> {color:#ee}{color}{color:#89ddff}try{color}{color:#ee}:{color}
> {color:#89ddff}{color}{color:#546e7a}# Read input FlowFile 
> content{color}
> {color:#ee}input_text 
> {color}{color:#c792ea}={color}{color:#ee} 
> IOUtils.toString{color}{color:#89ddff}({color}{color:#ee}inputStream, 
> StandardCharsets.UTF_8{color}{color:#89ddff}){color}
> {color:#ee}input_obj 
> {color}{color:#c792ea}={color}{color:#ee} 
> json.loads{color}{color:#89ddff}({color}{color:#ee}input_text{color}{color:#89ddff}){color}
> {color:#ee} {color}
> {color:#ee}{color}
> {color:#ee}    {color}{color:#ee}table_name 
> {color}{color:#c792ea}={color}{color:#ee} 
> input_obj{color}{color:#89ddff}[{color}{color:#89ddff}'{color}{color:#c3e88d}table_name{color}{color:#89ddff}'{color}{color:#89ddff}]{color}
> {color:#ee}column_name 
> {color}{color:#c792ea}={color}{color:#ee}  
> {color}{color:#89ddff}"{color}{color:#f78c6c}SELECT{color}{color:#c3e88d} 
> column_name {color}{color:#f78c6c}FROM{color}{color:#c3e88d} 
> INFORMATION_SCHEMA.COLUMNS {color}{color:#f78c6c}WHERE{color}{color:#c3e88d} 
> table_name{color}{color:#c792ea}={color}{color:#89ddff}'{color}{color:#c3e88d}"+table_name+"'
>  {color}{color:#f78c6c}AND{color}{color:#c3e88d} 
> data_type{color}{color:#c792ea}={color}{color:#89ddff}'{color}{color:#c3e88d}enum'{color}{color:#89ddff}"{color}
> {color:#ee}enum_value_sql 
> {color}{color:#c792ea}={color}{color:#ee} 
> {color}{color:#89ddff}"{color}{color:#f78c6c}SELECT{color}{color:#c3e88d} 
> {color}{color:#82aaff}REPLACE{color}{color:#c3e88d}({color}{color:#82aaff}REPLACE{color}{color:#c3e88d}({color}{color:#82aaff}REPLACE{color}{color:#c3e88d}({color}{color:#82aaff}REPLACE{color}{color:#c3e88d}(column_type,{color}{color:#89ddff}'{color}{color:#c3e88d}enum',{color}{color:#89ddff}'{color}{color:#c3e88d}'),{