[jira] [Commented] (NIFI-5723) PutDatabaseRecord strips underbar character from column names
[
https://issues.apache.org/jira/browse/NIFI-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16660445#comment-16660445
]
Uwe Geercken commented on NIFI-5723:
Another comment. The column names are forced to uppercase. If in other
processors, a case sensitive compare is done - e.g. schema versus table column
names, then this would not work. So I would either remove the to uppercase here
or make it configurable in the processors properties.
Uwe
> PutDatabaseRecord strips underbar character from column names
> -
>
> Key: NIFI-5723
> URL: https://issues.apache.org/jira/browse/NIFI-5723
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core UI
>Affects Versions: 1.7.1
> Environment: Red Hat Enterprise Linux Server release 7.5 (Maipo).
> Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz 16 cores. 32 Gb RAM. OpenJDK
> Runtime Environment (build 1.8.0_181-b13)
>Reporter: Uwe Geercken
>Priority: Major
>
> In *PutDatabaseRecord.java* there is following code:
> private static String normalizeColumnName(final String colName, final boolean
> translateColumnNames) {
> return colName == null ? null : (translateColumnNames ?
> colName.toUpperCase().replace("_", "") : colName);
> }
> If translateColumnNames is set to true, then the underbar character is
> stripped from the column names.
> And it is also here:
> final ColumnDescription desc =
> tableSchema.getColumns().get(normalizeColumnName(fieldName,
> settings.translateFieldNames));
> if (desc == null && !settings.ignoreUnmappedFields) {
> throw new SQLDataException("Cannot map field '" +
> fieldName + "' to any column in the database");
> }
> I do not see a reason why this is done. An underbar is a common delimiter in
> database column names.
> Specifically, I have data comming from an Oracle database which is all
> uppercase and with the underbar character as the delimiter in the column
> names and due to the current code, I can not update my MySQL database using
> the same column names.
> Trying to solve the issue in other ways does not really work as well.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (NIFI-5723) PutDatabaseRecord stripps underbar character from column names
Uwe Geercken created NIFI-5723:
--
Summary: PutDatabaseRecord stripps underbar character from column
names
Key: NIFI-5723
URL: https://issues.apache.org/jira/browse/NIFI-5723
Project: Apache NiFi
Issue Type: Bug
Components: Core UI
Affects Versions: 1.7.1
Environment: Red Hat Enterprise Linux Server release 7.5 (Maipo).
Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz 16 cores. 32 Gb RAM. OpenJDK Runtime
Environment (build 1.8.0_181-b13)
Reporter: Uwe Geercken
In *PutDatabaseRecord.java* there is following code:
private static String normalizeColumnName(final String colName, final boolean
translateColumnNames) {
return colName == null ? null : (translateColumnNames ?
colName.toUpperCase().replace("_", "") : colName);
}
If translateColumnNames is set to true, then the underbar character is stripped
from the column names.
And it is also here:
final ColumnDescription desc =
tableSchema.getColumns().get(normalizeColumnName(fieldName,
settings.translateFieldNames));
if (desc == null && !settings.ignoreUnmappedFields) {
throw new SQLDataException("Cannot map field '" + fieldName
+ "' to any column in the database");
}
I do not see a reason why this is done. An underbar is a common delimiter in
database column names.
Specifically, I have data comming from an Oracle database which is all
uppercase and with the underbar character as the delimiter in the column names
and due to the current code, I can not update my MySQL database using the same
column names.
Trying to solve the issue in other ways does not really work as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (NIFI-4541) Variables can not be used for connection string in RedisConnectionPoolService
Uwe Geercken created NIFI-4541:
--
Summary: Variables can not be used for connection string in
RedisConnectionPoolService
Key: NIFI-4541
URL: https://issues.apache.org/jira/browse/NIFI-4541
Project: Apache NiFi
Issue Type: Bug
Components: Core Framework
Affects Versions: 1.4.0
Environment: Fedora 26, Nifi 1.4.0
Reporter: Uwe Geercken
Priority: Minor
The controller service RedisConnectionPoolService is not accpting variables for
the connection string field.
Steps:
1) Create a variable on the Nifi Flow level named "redis.server" with the value
of "localhost:6379"
2) Create a new controller service on the Nifi Flow level:
RedisConnectionPoolService
3) As connection string enter "${redis.server}
Result:
When the configuration dialog is closed an error is displayed: 'Connection
String' is invalid because Connection String cannot be blank
When the value "localhost:6379" is entered directly in the connection string
field of the service controller, it does work correctly.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (NIFI-3988) SplitRecord processor missing fragment attributes
Uwe Geercken created NIFI-3988: -- Summary: SplitRecord processor missing fragment attributes Key: NIFI-3988 URL: https://issues.apache.org/jira/browse/NIFI-3988 Project: Apache NiFi Issue Type: Bug Components: Core Framework Affects Versions: 1.2.0 Environment: Fedora 25 Reporter: Uwe Geercken Priority: Minor There are several SplitXXX processors available. They all add some attributes for the relevant fragment to the flowfile. Such as e.g. fragment.index, fragment.count and fragment.identifier. The SplitRecord processor does not add these attributes. For a consistent user experience, and partly for compatibility with MergeContent and other processors please add the relevant attributes to SplitRecord. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3610) PutSql truncates data after 80 characters
[ https://issues.apache.org/jira/browse/NIFI-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15930632#comment-15930632 ] Uwe Geercken commented on NIFI-3610: I am using MySQL, Matt. I will check the driver on monday when I am back in the office. Strang - I have never heard of this problem or faced it... > PutSql truncates data after 80 characters > - > > Key: NIFI-3610 > URL: https://issues.apache.org/jira/browse/NIFI-3610 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.1.1 > Environment: Fedora 25 >Reporter: Uwe Geercken > > I have a ConvertJsonToSQL processor with following attribute: > sql.args.4.value > Type: flights, Station: hel, Month offset: -1, Environment: dev hello uwe > what are you doing here > the sql.args.4.type is: -1 (LONGVARCHAR) > This processor is connected to a PutSQL processor. The same attribute looks > like this: > sql.args.4.value > Type: flights, Station: hel, Month offset: -1, Environment: dev hello uwe > what a > The data is truncated after 80 characters and arrives like this in the > database. I have checked the database - it allows up to 255 characters. > The insert query from PutSQL - just for the completeness - is: > INSERT INTO process_monitoring (process_name, process_name_monitoring, > last_runtime_automatic, parameters) VALUES (?, ?, ?, ?). And the record does > get inserted - but column "parameters" is truncated to 80 characters. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3607) TailFile does not notice new files in folder
[ https://issues.apache.org/jira/browse/NIFI-3607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15929004#comment-15929004 ] Uwe Geercken commented on NIFI-3607: Sorry for being short worded... I create small and simple json files from shell scripts in a central folder. They are used for monitoring some processes. They contain just a few bytes of data. I have configured the processor for multiple files, given a reg expression (monitoring_.*.json), and provided the base directory. I start the processor and there are a few json files in the folder and the TailFile picks them up correctly. But then over time more json files are created. These new files don't get picked up unless I stop and start the TailFile processor. But when I do, they get correctly picked up. > TailFile does not notice new files in folder > > > Key: NIFI-3607 > URL: https://issues.apache.org/jira/browse/NIFI-3607 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.1.1 > Environment: Fedora 25 >Reporter: Uwe Geercken >Priority: Critical > > I have configured the TailFile processor to read multiple (small) files from > a folder. > This works, but if more files are created in the folder, the processor does > not pick them up, even though the "Lookup frequency" is set (10 minutes). > If I stop and start the processor, it correctly picks up the new files. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3608) SplitText should be configurable if header row is put in all splits
[ https://issues.apache.org/jira/browse/NIFI-3608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15928993#comment-15928993 ] Uwe Geercken commented on NIFI-3608: Unfortunately not. If there is a header row, that is fine, I just don't think it should be populated to all flow files. It's an overhead in the following processors. I would recommend to at least make it configurable. > SplitText should be configurable if header row is put in all splits > --- > > Key: NIFI-3608 > URL: https://issues.apache.org/jira/browse/NIFI-3608 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.1.1 > Environment: Fedora 25 >Reporter: Uwe Geercken > > The documentation for the SplitText processor states: > "The number of lines that should be considered part of the header; the header > lines will be duplicated to all split files" > There should be an option to turn off this behavior and to not output the > header row to each split file. > When processing the splits further then the header row has to be handled in > each processor that follows which is unnecessary overhead. > As an alternative one could think of assigning the details of the header row > to the flow file attributes -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (NIFI-3610) PutSql truncates data after 80 characters
Uwe Geercken created NIFI-3610: -- Summary: PutSql truncates data after 80 characters Key: NIFI-3610 URL: https://issues.apache.org/jira/browse/NIFI-3610 Project: Apache NiFi Issue Type: Bug Components: Core UI Affects Versions: 1.1.1 Environment: Fedora 25 Reporter: Uwe Geercken I have a ConvertJsonToSQL processor with following attribute: sql.args.4.value Type: flights, Station: hel, Month offset: -1, Environment: dev hello uwe what are you doing here the sql.args.4.type is: -1 (LONGVARCHAR) This processor is connected to a PutSQL processor. The same attribute looks like this: sql.args.4.value Type: flights, Station: hel, Month offset: -1, Environment: dev hello uwe what a The data is truncated after 80 characters and arrives like this in the database. I have checked the database - it allows up to 255 characters. The insert query from PutSQL - just for the completeness - is: INSERT INTO process_monitoring (process_name, process_name_monitoring, last_runtime_automatic, parameters) VALUES (?, ?, ?, ?). And the record does get inserted - but column "parameters" is truncated to 80 characters. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (NIFI-3555) UpdateAttribute on filename adds extra space
[
https://issues.apache.org/jira/browse/NIFI-3555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15927121#comment-15927121
]
Uwe Geercken commented on NIFI-3555:
Yes that's right. I copied it from the wrong place. The issue is still valid
though.
> UpdateAttribute on filename adds extra space
>
>
> Key: NIFI-3555
> URL: https://issues.apache.org/jira/browse/NIFI-3555
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core UI
>Affects Versions: 1.1.1
> Environment: Linux, Fedora 25, Nifi 1.1.1
>Reporter: Uwe Geercken
>Priority: Minor
>
> I have used the GetFile and then SplitText processors to split a csv file
> into rows.
> Here is the original filename:
> allCountries_100.txt
> Then I have created an Expression in the UpdateAttribute processor:
> ${filename:substringBefore('.'):append('_fragment_'):append(${fragment.index}):append('_details.csv')}
>
> It creates a new filename by using the original filename plus the fragment
> number from the SplitText processor and adds a string at the end.
> In the output of the PutFile processor all files have an extra space at the
> end behing the ".csv".
> here is an example:
> allCountries_100_fragment_35.csv
> please note the extra space at the end.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (NIFI-3608) SplitText should be configurable if header row is put in all splits
Uwe Geercken created NIFI-3608: -- Summary: SplitText should be configurable if header row is put in all splits Key: NIFI-3608 URL: https://issues.apache.org/jira/browse/NIFI-3608 Project: Apache NiFi Issue Type: Improvement Components: Core UI Affects Versions: 1.1.1 Environment: Fedora 25 Reporter: Uwe Geercken The documentation for the SplitText processor states: "The number of lines that should be considered part of the header; the header lines will be duplicated to all split files" There should be an option to turn off this behavior and to not output the header row to each split file. When processing the splits further then the header row has to be handled in each processor that follows which is unnecessary overhead. As an alternative one could think of assigning the details of the header row to the flow file attributes -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (NIFI-3607) TailFile does not notice new files in folder
Uwe Geercken created NIFI-3607: -- Summary: TailFile does not notice new files in folder Key: NIFI-3607 URL: https://issues.apache.org/jira/browse/NIFI-3607 Project: Apache NiFi Issue Type: Bug Components: Core UI Affects Versions: 1.1.1 Environment: Fedora 25 Reporter: Uwe Geercken Priority: Critical I have configured the TailFile processor to read multiple (small) files from a folder. This works, but if more files are created in the folder, the processor does not pick them up, even though the "Lookup frequency" is set (10 minutes). If I stop and start the processor, it correctly picks up the new files. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (NIFI-3555) UpdateAttribute on filename adds extra space
Uwe Geercken created NIFI-3555:
--
Summary: UpdateAttribute on filename adds extra space
Key: NIFI-3555
URL: https://issues.apache.org/jira/browse/NIFI-3555
Project: Apache NiFi
Issue Type: Bug
Components: Core UI
Affects Versions: 1.1.1
Environment: Linux, Fedora 25, Nifi 1.1.1
Reporter: Uwe Geercken
Priority: Minor
I have used the GetFile and then SplitText processors to split a csv file into
rows.
Here is the original filename:
allCountries_100.txt
Then I have created an Expression in the UpdateAttribute processor:
${filename:substringBefore('.'):append('_fragment_'):append(${fragment.index}):append('_details.csv')}
It creates a new filename by using the original filename plus the fragment
number from the SplitText processor and adds a string at the end.
In the output of the PutFile processor all files have an extra space at the end
behing the ".csv".
here is an example:
allCountries_100_fragment_35.csv
please note the extra space at the end.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (NIFI-3533) Enhance QueryDatabaseTable processor to output different formats
Uwe Geercken created NIFI-3533: -- Summary: Enhance QueryDatabaseTable processor to output different formats Key: NIFI-3533 URL: https://issues.apache.org/jira/browse/NIFI-3533 Project: Apache NiFi Issue Type: Improvement Components: Core Framework Affects Versions: 1.1.1 Reporter: Uwe Geercken Priority: Minor The QueryDatabaseTable processor outputs in AVRO format. Enhance the processor to allow the user to select the desired output format. Such as JSON, CSV, etc. This can be achieved using other processors, but converting to AVRO and then to e.g. CSV it double work and slows down the process. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (NIFI-3532) Add a one-click filter to show provenance events of the last x minutes
Uwe Geercken created NIFI-3532: -- Summary: Add a one-click filter to show provenance events of the last x minutes Key: NIFI-3532 URL: https://issues.apache.org/jira/browse/NIFI-3532 Project: Apache NiFi Issue Type: Improvement Components: Core UI Affects Versions: 1.1.1 Reporter: Uwe Geercken Priority: Minor The provenance screen potentially lists a lot of data. To define a search takes multiple clicks and input by the user. Add a sort of one-click filter to e.g. show the events of the last 5 minutes, 10 minutes, etc. Alternatively one could think of defining a search and then giving the user the possibility to save the search and persist it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
