[GitHub] nifi issue #2117: NIFI-4334: Added Apache Commons Net to scripting NAR
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/2117 +1, merging, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #2117: NIFI-4334: Added Apache Commons Net to scripting NA...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/2117 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-4334) Add Apache Commons Net to scripting NAR
[ https://issues.apache.org/jira/browse/NIFI-4334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16148564#comment-16148564 ] ASF GitHub Bot commented on NIFI-4334: -- Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/2117 +1, merging, thanks! > Add Apache Commons Net to scripting NAR > --- > > Key: NIFI-4334 > URL: https://issues.apache.org/jira/browse/NIFI-4334 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess > > The scripting NAR, which includes scripting processors, reporting tasks, and > controller services, is often used to prototype custom connections to > external entities when built-in NiFi support is not yet present. The Apache > Commons Net library has a number of clients and utilities that would make > such connections easier. At only half an MB in size, it would be nice to add > this as a dependency to the scripting NAR such that any scripted solution > could leverage these classes. One example could be to prototype an FTPS > client to use until NIFI-2278 is implemented. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NIFI-4334) Add Apache Commons Net to scripting NAR
[ https://issues.apache.org/jira/browse/NIFI-4334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16148563#comment-16148563 ] ASF subversion and git services commented on NIFI-4334: --- Commit 20a6374bf788827fc12c4636093d905312552d92 in nifi's branch refs/heads/master from [~ca9mbu] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=20a6374 ] NIFI-4334: Added Apache Commons Net to scripting NAR Signed-off-by: Pierre Villard This closes #2117. > Add Apache Commons Net to scripting NAR > --- > > Key: NIFI-4334 > URL: https://issues.apache.org/jira/browse/NIFI-4334 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess > > The scripting NAR, which includes scripting processors, reporting tasks, and > controller services, is often used to prototype custom connections to > external entities when built-in NiFi support is not yet present. The Apache > Commons Net library has a number of clients and utilities that would make > such connections easier. At only half an MB in size, it would be nice to add > this as a dependency to the scripting NAR such that any scripted solution > could leverage these classes. One example could be to prototype an FTPS > client to use until NIFI-2278 is implemented. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NIFI-4334) Add Apache Commons Net to scripting NAR
[ https://issues.apache.org/jira/browse/NIFI-4334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16148565#comment-16148565 ] ASF GitHub Bot commented on NIFI-4334: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/2117 > Add Apache Commons Net to scripting NAR > --- > > Key: NIFI-4334 > URL: https://issues.apache.org/jira/browse/NIFI-4334 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess > > The scripting NAR, which includes scripting processors, reporting tasks, and > controller services, is often used to prototype custom connections to > external entities when built-in NiFi support is not yet present. The Apache > Commons Net library has a number of clients and utilities that would make > such connections easier. At only half an MB in size, it would be nice to add > this as a dependency to the scripting NAR such that any scripted solution > could leverage these classes. One example could be to prototype an FTPS > client to use until NIFI-2278 is implemented. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (NIFI-4334) Add Apache Commons Net to scripting NAR
[ https://issues.apache.org/jira/browse/NIFI-4334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-4334: - Resolution: Fixed Fix Version/s: 1.4.0 Status: Resolved (was: Patch Available) > Add Apache Commons Net to scripting NAR > --- > > Key: NIFI-4334 > URL: https://issues.apache.org/jira/browse/NIFI-4334 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess > Fix For: 1.4.0 > > > The scripting NAR, which includes scripting processors, reporting tasks, and > controller services, is often used to prototype custom connections to > external entities when built-in NiFi support is not yet present. The Apache > Commons Net library has a number of clients and utilities that would make > such connections easier. At only half an MB in size, it would be nice to add > this as a dependency to the scripting NAR such that any scripted solution > could leverage these classes. One example could be to prototype an FTPS > client to use until NIFI-2278 is implemented. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (NIFI-4336) Support setting Hadoop configuration values from NiFi properties for HDFS processors
Takanobu Asanuma created NIFI-4336: -- Summary: Support setting Hadoop configuration values from NiFi properties for HDFS processors Key: NIFI-4336 URL: https://issues.apache.org/jira/browse/NIFI-4336 Project: Apache NiFi Issue Type: New Feature Reporter: Takanobu Asanuma Assignee: Takanobu Asanuma If users want to set hadoop configurations for HDFS processors, the configuration xml files need to be set in the NiFi nodes and users have to set the file paths to the processors. It would be useful if users also could set each hadoop configuration value from NiFi properties. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (NIFI-4337) Support Kerberos password for HDFS processors
Takanobu Asanuma created NIFI-4337: -- Summary: Support Kerberos password for HDFS processors Key: NIFI-4337 URL: https://issues.apache.org/jira/browse/NIFI-4337 Project: Apache NiFi Issue Type: New Feature Reporter: Takanobu Asanuma Assignee: Takanobu Asanuma HDFS processors only support keytab login for Kerberos authentication. Let's support the case of Kerberos password. (Please note users can't use keytab and password together.) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (NIFI-4338) Support SSL Context Service in HDFS processors
Takanobu Asanuma created NIFI-4338: -- Summary: Support SSL Context Service in HDFS processors Key: NIFI-4338 URL: https://issues.apache.org/jira/browse/NIFI-4338 Project: Apache NiFi Issue Type: Improvement Reporter: Takanobu Asanuma Assignee: Takanobu Asanuma HDFS has a RESTful API called WebHDFS and SSL can be configured on it. (It is called SWebHDFS.) HDFS processors can use WebHDFS but not support SSL Context Service. Let's support SSL Context Service for SWebHDFS. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (NIFI-4337) Support Kerberos password for HDFS processors
[ https://issues.apache.org/jira/browse/NIFI-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated NIFI-4337: --- Issue Type: Improvement (was: New Feature) > Support Kerberos password for HDFS processors > - > > Key: NIFI-4337 > URL: https://issues.apache.org/jira/browse/NIFI-4337 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma > > HDFS processors only support keytab login for Kerberos authentication. Let's > support the case of Kerberos password. (Please note users can't use keytab > and password together.) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] nifi issue #1985: NIFI-2529: added support for SSLContextService protocols i...
Github user m-hogue commented on the issue: https://github.com/apache/nifi/pull/1985 @trkurc @alopresto : I've made changes here in line with #1986 and forced the use of `RestrictedSSLContextService` here as well. This is ready for review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-2529) Update HandleHttpRequest to honor SSLContextService Protocols
[ https://issues.apache.org/jira/browse/NIFI-2529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16149009#comment-16149009 ] ASF GitHub Bot commented on NIFI-2529: -- Github user m-hogue commented on the issue: https://github.com/apache/nifi/pull/1985 @trkurc @alopresto : I've made changes here in line with #1986 and forced the use of `RestrictedSSLContextService` here as well. This is ready for review. > Update HandleHttpRequest to honor SSLContextService Protocols > - > > Key: NIFI-2529 > URL: https://issues.apache.org/jira/browse/NIFI-2529 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.0.0, 0.8.0, 0.7.1 >Reporter: Joe Skora >Assignee: Michael Hogue > > Update HandleHttpRequest to honor SSLContextService Protocols as [NIFI-1688] > did for PostHTTP. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NIFI-4335) Update Listen* components to refer to RestrictedSSLContextService
[
https://issues.apache.org/jira/browse/NIFI-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16149018#comment-16149018
]
Michael Hogue commented on NIFI-4335:
-
NOTE: I've changed {{HandleHTTPRequest}} to refer to the
RestrictedSSLContextService in PR 1985: https://github.com/apache/nifi/pull/1985
> Update Listen* components to refer to RestrictedSSLContextService
> -
>
> Key: NIFI-4335
> URL: https://issues.apache.org/jira/browse/NIFI-4335
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework, Extensions
>Affects Versions: 1.3.0
>Reporter: Andy LoPresto
>Assignee: Michael Hogue
> Labels: jetty, security, tls
>
> [~m-hogue] added a {{RestrictedSSLContextService}} in NIFI-2528 due to
> discussions regarding the baseline TLS protocol versions that are supported
> as of NiFi 1.2.0. The {{ListenHTTP}} processor was updated to require this
> new interface, but other {{Listen*}} processors and services need to be
> updated as well.
> From discussion on PR 1986:
> {quote}
> Also, as @joewitt noted earlier, we should change the available interface for
> other "listener" processors. Here's a preliminary list I put together, but I
> would like confirmation from another member:
> * `HandleHTTPRequest`
> * `ListenBeats`
> *
> `DistributedCacheServer`/`DistributedSetCacheServer`/`DistributedMapCacheServer`
> * `ListenSMTP`
> * `ListenGRPC`
> * `ListenLumberjack` (*Deprecated*)
> * `ListenRELP`
> * `ListenSyslog`
> * `ListenTCP`/`ListenTCPRecord`
> Also:
> * `AbstractSiteToSiteReportingTask`
> * `org.apache.nifi.processors.slack.TestServer`
> * `WebSocketService`/`JettyWebSocketService`
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (NIFI-4333) Dockerize TLS Toolkit
[ https://issues.apache.org/jira/browse/NIFI-4333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aldrin Piri reassigned NIFI-4333: - Assignee: Aldrin Piri > Dockerize TLS Toolkit > - > > Key: NIFI-4333 > URL: https://issues.apache.org/jira/browse/NIFI-4333 > Project: Apache NiFi > Issue Type: Task > Components: Docker >Reporter: Aldrin Piri >Assignee: Aldrin Piri > > It would be helpful to have a TLS Toolkit image to work in conjunction with > NiFI and would help in orchestration of clustered instances as well as > provisioning security items. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NIFI-3248) GetSolr can miss recently updated documents
[
https://issues.apache.org/jira/browse/NIFI-3248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16149066#comment-16149066
]
Johannes Peter commented on NIFI-3248:
--
[~ijokarumawak] Sure. I will start within next week.
> GetSolr can miss recently updated documents
> ---
>
> Key: NIFI-3248
> URL: https://issues.apache.org/jira/browse/NIFI-3248
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.0.0, 0.5.0, 0.6.0, 0.5.1, 0.7.0, 0.6.1, 1.1.0, 0.7.1,
> 1.0.1
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
> Attachments: nifi-flow.png, query-result-with-curly-bracket.png,
> query-result-with-square-bracket.png
>
>
> GetSolr holds the last query timestamp so that it only fetches documents
> those have been added or updated since the last query.
> However, GetSolr misses some of those updated documents, and once the
> documents date field value becomes older than last query timestamp, the
> document won't be able to be queried by GetSolr any more.
> This JIRA is for tracking the process of investigating this behavior, and
> discussion on them.
> Here are things that can be a cause of this behavior:
> |#|Short description|Should we address it?|
> |1|Timestamp range filter, curly or square bracket?|No|
> |2|Timezone difference between update and query|Additional docs might be
> helpful|
> |3|Lag comes from NearRealTIme nature of Solr|Should be documented at least,
> add 'commit lag-time'?|
> h2. 1. Timestamp range filter, curly or square bracket?
> At the first glance, using curly and square bracket in mix looked strange
> ([source
> code|https://github.com/apache/nifi/blob/support/nifi-0.5.x/nifi-nar-bundles/nifi-solr-bundle/nifi-solr-processors/src/main/java/org/apache/nifi/processors/solr/GetSolr.java#L202]).
> But these difference has a meaning.
> The square bracket on the range query is inclusive and the curly bracket is
> exclusive. If we use inclusive on both sides and a document has a time stamp
> exactly on the boundary then it could be returned in two consecutive
> executions, and we only want it in one.
> This is intentional, and it should be as it is.
> h2. 2. Timezone difference between update and query
> Solr treats date fields as [UTC
> representation|https://cwiki.apache.org/confluence/display/solr/Working+with+Dates|].
> If date field String value of an updated document represents time without
> timezone, and NiFi is running on an environment using timezone other than
> UTC, GetSolr can't perform date range query as users expect.
> Let's say NiFi is running with JST(UTC+9). A process added a document to Solr
> at 15:00 JST. But the date field doesn't have timezone. So, Solr indexed it
> as 15:00 UTC. Then GetSolr performs range query at 15:10 JST, targeting any
> documents updated from 15:00 to 15:10 JST. GetSolr formatted dates using UTC,
> i.e. 6:00 to 6:10 UTC. The updated document won't be matched with the date
> range filter.
> To avoid this, updated documents must have proper timezone in date field
> string representation.
> If one uses NiFi expression language to set current timestamp to that date
> field, following NiFi expression can be used:
> {code}
> ${now():format("-MM-dd'T'HH:mm:ss.SSSZ")}
> {code}
> It will produce a result like:
> {code}
> 2016-12-27T15:30:04.895+0900
> {code}
> Then it will be indexed in Solr with UTC and will be queried by GetSolr as
> expected.
> h2. 3. Lag comes from NearRealTIme nature of Solr
> Solr provides Near Real Time search capability, that means, the recently
> updated documents can be queried in Near Real Time, but it's not real time.
> This latency can be controlled by either on client side which requests the
> update operation by specifying "commitWithin" parameter, or on the Solr
> server side, "autoCommit" and "autoSoftCommit" in
> [solrconfig.xml|https://cwiki.apache.org/confluence/display/solr/UpdateHandlers+in+SolrConfig#UpdateHandlersinSolrConfig-Commits].
> Since commit and updating index can be costly, it's recommended to set this
> interval long enough up to the maximum tolerable latency.
> However, this can be problematic with GetSolr. For instance, as shown in the
> simple NiFi flow below, GetSolr can miss updated documents:
> {code}
> t1: GetSolr queried
> t2: GenerateFlowFile set date = t2
> t3: PutSolrContentStream stored new doc
> t4: GetSolr queried again, from t1 to t4, but the new doc hasn't been indexed
> t5: Solr completed index
> t6: GetSolr queried again, from t4 to t6, the doc didn't match query
> {code}
> This behavior should be at least documented.
> Plus, it would be helpful to add a new configuration property to GetSolr, to
> specify commit lag-time so that GetSolr aims older timestamp range to query
> documents.
> {code}
> //
[jira] [Comment Edited] (NIFI-3869) add http/2 support to HTTP listening processors
[
https://issues.apache.org/jira/browse/NIFI-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16135852#comment-16135852
]
Michael Hogue edited comment on NIFI-3869 at 8/31/17 3:45 PM:
--
After changes to {{ListenHTTP}}, i wrote up a really quick performance test to
see if there were any performance gains/losses for non-secure communications. I
published the code and results on github:
https://github.com/m-hogue/nifi-http2
http2 is faster across the board in my testing for messages ranging in size
between [10B, 100MB].
was (Author: m-hogue):
After changes to {{ListenHTTP}}, i wrote up a really quick performance test to
see if there were any performance gains/losses. I published the code and
results on github:
https://github.com/m-hogue/nifi-http2
http2 is faster across the board in my testing for messages ranging in size
between [10B, 100MB].
> add http/2 support to HTTP listening processors
> ---
>
> Key: NIFI-3869
> URL: https://issues.apache.org/jira/browse/NIFI-3869
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: Tony Kurc
>Assignee: Michael Hogue
>Priority: Minor
>
> investigate whether some of the HTTP processors could use HTTP/2
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (NIFI-4339) Possible NPE in ExtractEmailHeaders if no recipients exist
Kevin Doran created NIFI-4339:
-
Summary: Possible NPE in ExtractEmailHeaders if no recipients exist
Key: NIFI-4339
URL: https://issues.apache.org/jira/browse/NIFI-4339
Project: Apache NiFi
Issue Type: Bug
Components: Extensions
Reporter: Kevin Doran
Assignee: Kevin Doran
Priority: Minor
Fix For: 1.4.0
A user uncovered a *NullPointerException* in the ExtractEmailHeaders processor
and [asked about it on
StackOverflow|https://stackoverflow.com/questions/45927852/nifi-email-consumeimap-filter-by-subject-from-to-and-date].
Here is the summary: If the TO, CC, and BCC are not present in the email
message, a call to {{javax.mail.internet.MimeMessage getAllRecipients()}} will
return {{null}}.
While this is a rare case, the reporter has a use case in which an email
account is sending automated emails to the same address as the source, and
whatever email program is being used appears to not set any address fields in
this case:
{quote}It is an auto email from client to their own inbox and does not have
"to", "cc"and "bcc"{quote}
The {{javax.mail.Message getAllRecipients()}} method, which is called from
{{javax.mail.internet.MimeMessage getAllRecipients()}}, will return null in
this case, as is documented and apparent from the source code:
{code:java}
/**
* Get all the recipient addresses for the message.
* The default implementation extracts the TO, CC, and BCC
* recipients using the getRecipients method.
*
* This method returns null if none of the recipient
* headers are present in this message. It may Return an empty array
* if any recipient header is present, but contains no addresses.
*
* @return array of Address objects
* @exception MessagingException
*/
public Address[] getAllRecipients() throws MessagingException {
Address[] to = getRecipients(RecipientType.TO);
Address[] cc = getRecipients(RecipientType.CC);
Address[] bcc = getRecipients(RecipientType.BCC);
if (cc == null && bcc == null)
return to; // a common case
// ...
{code}
Here is a stack trace for the NPE:
{noformat}
2017-08-30 16:36:05,930 ERROR [Timer-Driven Process Thread-8]
o.a.n.p.email.ExtractEmailHeaders ExtractEmailHeaders [...] Could not parse the
flowfile StandardFlowFileRecord [...] as an email, treating as failure:
java.lang.NullPointerException: null
at java.lang.reflect.Array.getLength(Native Method)
at
org.apache.nifi.processors.email.ExtractEmailHeaders$1.process(ExtractEmailHeaders.java:171)
at
org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2177)
at
org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2147)
at
org.apache.nifi.processors.email.ExtractEmailHeaders.onTrigger(ExtractEmailHeaders.java:146)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1118)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
{noformat}
The NPE occurs as ExtractEmailHeaders:L171:
{code:java}
if (Array.getLength(originalMessage.getAllRecipients()) > 0) {
{code}
{{Array.getLength(Array array)}} expects a non-null argument, but
{{originalMessage.getAllRecipients()}} is returning {{null}}.
One proposed solution is to check if {{originalMessage.getAllRecipients()}} is
{{null}}, and if so, treat it as an empty array. That is, the condition would
be:
{code:java}
Address[] allRecipients = originalMessage.getAllRecipients();
if (allRecipients != null && Array.getLength(allRecipients) > 0) {
{code}
A test case for ExtractEmailHeaders that reproduces this NPE should also be
added.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (NIFI-4150) NiFi starting failure and key file already existing
[
https://issues.apache.org/jira/browse/NIFI-4150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16149169#comment-16149169
]
Pierre Villard commented on NIFI-4150:
--
You worked on a similar JIRA recently, could you have a look [~alopresto]?
> NiFi starting failure and key file already existing
> ---
>
> Key: NIFI-4150
> URL: https://issues.apache.org/jira/browse/NIFI-4150
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.2.0, 1.3.0
>Reporter: Pierre Villard
>
> In some cases, if NiFi cannot start (in my case: debug port was already in
> use), the key file is still created and not deleted. Then it won't be
> possible to restart NiFi unless the key file is manually deleted.
> Logs from bootstrap:
> {code}
> 2017-07-04 13:57:40,460 INFO [main] o.a.n.b.NotificationServiceManager
> Successfully loaded the following 0 services: []
> 2017-07-04 13:57:40,466 INFO [main] org.apache.nifi.bootstrap.RunNiFi
> Registered no Notification Services for Notification Type NIFI_STARTED
> 2017-07-04 13:57:40,466 INFO [main] org.apache.nifi.bootstrap.RunNiFi
> Registered no Notification Services for Notification Type NIFI_STOPPED
> 2017-07-04 13:57:40,466 INFO [main] org.apache.nifi.bootstrap.RunNiFi
> Registered no Notification Services for Notification Type NIFI_DIED
> 2017-07-04 13:57:40,508 INFO [main] org.apache.nifi.bootstrap.Command
> Starting Apache NiFi...
> 2017-07-04 13:57:40,509 INFO [main] org.apache.nifi.bootstrap.Command Working
> Directory: /usr/hdf/current/nifi
> 2017-07-04 13:57:40,510 INFO [main] org.apache.nifi.bootstrap.Command
> Command: /usr/jdk64/jdk1.8.0_112/bin/java -classpath
> /usr/hdf/current/nifi/conf:/usr/hdf/current/nifi/lib/nifi-runtime-1.2.0.3.0.0.0-453.jar:/usr/hdf/current/nifi/lib/javax.servlet-api-3.1.0.jar:/usr/hdf/current/nifi/lib/jcl-over-slf4j-1.7.25.jar:/usr/hdf/current/nifi/lib/jetty-schemas-3.1.jar:/usr/hdf/current/nifi/lib/jul-to-slf4j-1.7.25.jar:/usr/hdf/current/nifi/lib/log4j-over-slf4j-1.7.25.jar:/usr/hdf/current/nifi/lib/logback-classic-1.2.3.jar:/usr/hdf/current/nifi/lib/logback-core-1.2.3.jar:/usr/hdf/current/nifi/lib/nifi-api-1.2.0.3.0.0.0-453.jar:/usr/hdf/current/nifi/lib/nifi-nar-utils-1.2.0.3.0.0.0-453.jar:/usr/hdf/current/nifi/lib/nifi-properties-1.2.0.3.0.0.0-453.jar:/usr/hdf/current/nifi/lib/slf4j-api-1.7.25.jar:/usr/hdf/current/nifi/lib/nifi-framework-api-1.2.0.3.0.0.0-453.jar
> -Dorg.apache.jasper.compiler.disablejsr199=true
> -Djava.security.auth.login.config=/usr/hdf/current/nifi/conf/nifi_jaas.conf
> -Xmx512m -Xms512m
> -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000
> -Dambari.application.id=nifi
> -Dambari.metrics.collector.url=http://pvillard-1:6188/ws/v1/timeline/metrics
> -Dsun.net.http.allowRestrictedHeaders=true -Djava.net.preferIPv4Stack=true
> -Djava.awt.headless=true -XX:+UseG1GC
> -Djava.protocol.handler.pkgs=sun.net.www.protocol
> -Dnifi.properties.file.path=/usr/hdf/current/nifi/conf/nifi.properties
> -Dnifi.bootstrap.listen.port=33910 -Dapp=NiFi
> -Dorg.apache.nifi.bootstrap.config.log.dir=/var/log/nifi org.apache.nifi.NiFi
> -K /usr/hdf/current/nifi/conf/sensitive.key
> 2017-07-04 13:57:40,532 INFO [main] org.apache.nifi.bootstrap.Command
> Launched Apache NiFi with Process ID 30029
> 2017-07-04 13:57:40,647 ERROR [NiFi logging handler] org.apache.nifi.StdErr
> ERROR: transport error 202: bind failed: Address already in use
> 2017-07-04 13:57:40,648 ERROR [NiFi logging handler] org.apache.nifi.StdErr
> ERROR: JDWP Transport dt_socket failed to initialize, TRANSPORT_INIT(510)
> 2017-07-04 13:57:40,648 ERROR [NiFi logging handler] org.apache.nifi.StdErr
> JDWP exit error AGENT_ERROR_TRANSPORT_INIT(197): No transports initialized
> [debugInit.c:750]
> 2017-07-04 13:57:41,536 INFO [main] org.apache.nifi.bootstrap.RunNiFi NiFi
> never started. Will not restart NiFi
> 2017-07-04 14:03:26,828 INFO [main] o.a.n.b.NotificationServiceManager
> Successfully loaded the following 0 services: []
> 2017-07-04 14:03:26,834 INFO [main] org.apache.nifi.bootstrap.RunNiFi
> Registered no Notification Services for Notification Type NIFI_STARTED
> 2017-07-04 14:03:26,835 INFO [main] org.apache.nifi.bootstrap.RunNiFi
> Registered no Notification Services for Notification Type NIFI_STOPPED
> 2017-07-04 14:03:26,835 INFO [main] org.apache.nifi.bootstrap.RunNiFi
> Registered no Notification Services for Notification Type NIFI_DIED
> 2017-07-04 14:03:26,838 INFO [main] org.apache.nifi.bootstrap.Command Apache
> NiFi is not currently running
> 2017-07-04 14:03:46,063 INFO [main] o.a.n.b.NotificationServiceManager
> Successfully loaded the following 0 services: []
> 2017-07-04 14:03:46,070 INFO [main] org.apache.nifi.bootstrap.RunNiFi
> Registered no Notification Services fo
[jira] [Commented] (NIFI-4150) NiFi starting failure and key file already existing
[
https://issues.apache.org/jira/browse/NIFI-4150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16149188#comment-16149188
]
Andy LoPresto commented on NIFI-4150:
-
Thanks [~pvillard]. I will investigate.
> NiFi starting failure and key file already existing
> ---
>
> Key: NIFI-4150
> URL: https://issues.apache.org/jira/browse/NIFI-4150
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.2.0, 1.3.0
>Reporter: Pierre Villard
>Assignee: Andy LoPresto
>
> In some cases, if NiFi cannot start (in my case: debug port was already in
> use), the key file is still created and not deleted. Then it won't be
> possible to restart NiFi unless the key file is manually deleted.
> Logs from bootstrap:
> {code}
> 2017-07-04 13:57:40,460 INFO [main] o.a.n.b.NotificationServiceManager
> Successfully loaded the following 0 services: []
> 2017-07-04 13:57:40,466 INFO [main] org.apache.nifi.bootstrap.RunNiFi
> Registered no Notification Services for Notification Type NIFI_STARTED
> 2017-07-04 13:57:40,466 INFO [main] org.apache.nifi.bootstrap.RunNiFi
> Registered no Notification Services for Notification Type NIFI_STOPPED
> 2017-07-04 13:57:40,466 INFO [main] org.apache.nifi.bootstrap.RunNiFi
> Registered no Notification Services for Notification Type NIFI_DIED
> 2017-07-04 13:57:40,508 INFO [main] org.apache.nifi.bootstrap.Command
> Starting Apache NiFi...
> 2017-07-04 13:57:40,509 INFO [main] org.apache.nifi.bootstrap.Command Working
> Directory: /usr/hdf/current/nifi
> 2017-07-04 13:57:40,510 INFO [main] org.apache.nifi.bootstrap.Command
> Command: /usr/jdk64/jdk1.8.0_112/bin/java -classpath
> /usr/hdf/current/nifi/conf:/usr/hdf/current/nifi/lib/nifi-runtime-1.2.0.3.0.0.0-453.jar:/usr/hdf/current/nifi/lib/javax.servlet-api-3.1.0.jar:/usr/hdf/current/nifi/lib/jcl-over-slf4j-1.7.25.jar:/usr/hdf/current/nifi/lib/jetty-schemas-3.1.jar:/usr/hdf/current/nifi/lib/jul-to-slf4j-1.7.25.jar:/usr/hdf/current/nifi/lib/log4j-over-slf4j-1.7.25.jar:/usr/hdf/current/nifi/lib/logback-classic-1.2.3.jar:/usr/hdf/current/nifi/lib/logback-core-1.2.3.jar:/usr/hdf/current/nifi/lib/nifi-api-1.2.0.3.0.0.0-453.jar:/usr/hdf/current/nifi/lib/nifi-nar-utils-1.2.0.3.0.0.0-453.jar:/usr/hdf/current/nifi/lib/nifi-properties-1.2.0.3.0.0.0-453.jar:/usr/hdf/current/nifi/lib/slf4j-api-1.7.25.jar:/usr/hdf/current/nifi/lib/nifi-framework-api-1.2.0.3.0.0.0-453.jar
> -Dorg.apache.jasper.compiler.disablejsr199=true
> -Djava.security.auth.login.config=/usr/hdf/current/nifi/conf/nifi_jaas.conf
> -Xmx512m -Xms512m
> -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000
> -Dambari.application.id=nifi
> -Dambari.metrics.collector.url=http://pvillard-1:6188/ws/v1/timeline/metrics
> -Dsun.net.http.allowRestrictedHeaders=true -Djava.net.preferIPv4Stack=true
> -Djava.awt.headless=true -XX:+UseG1GC
> -Djava.protocol.handler.pkgs=sun.net.www.protocol
> -Dnifi.properties.file.path=/usr/hdf/current/nifi/conf/nifi.properties
> -Dnifi.bootstrap.listen.port=33910 -Dapp=NiFi
> -Dorg.apache.nifi.bootstrap.config.log.dir=/var/log/nifi org.apache.nifi.NiFi
> -K /usr/hdf/current/nifi/conf/sensitive.key
> 2017-07-04 13:57:40,532 INFO [main] org.apache.nifi.bootstrap.Command
> Launched Apache NiFi with Process ID 30029
> 2017-07-04 13:57:40,647 ERROR [NiFi logging handler] org.apache.nifi.StdErr
> ERROR: transport error 202: bind failed: Address already in use
> 2017-07-04 13:57:40,648 ERROR [NiFi logging handler] org.apache.nifi.StdErr
> ERROR: JDWP Transport dt_socket failed to initialize, TRANSPORT_INIT(510)
> 2017-07-04 13:57:40,648 ERROR [NiFi logging handler] org.apache.nifi.StdErr
> JDWP exit error AGENT_ERROR_TRANSPORT_INIT(197): No transports initialized
> [debugInit.c:750]
> 2017-07-04 13:57:41,536 INFO [main] org.apache.nifi.bootstrap.RunNiFi NiFi
> never started. Will not restart NiFi
> 2017-07-04 14:03:26,828 INFO [main] o.a.n.b.NotificationServiceManager
> Successfully loaded the following 0 services: []
> 2017-07-04 14:03:26,834 INFO [main] org.apache.nifi.bootstrap.RunNiFi
> Registered no Notification Services for Notification Type NIFI_STARTED
> 2017-07-04 14:03:26,835 INFO [main] org.apache.nifi.bootstrap.RunNiFi
> Registered no Notification Services for Notification Type NIFI_STOPPED
> 2017-07-04 14:03:26,835 INFO [main] org.apache.nifi.bootstrap.RunNiFi
> Registered no Notification Services for Notification Type NIFI_DIED
> 2017-07-04 14:03:26,838 INFO [main] org.apache.nifi.bootstrap.Command Apache
> NiFi is not currently running
> 2017-07-04 14:03:46,063 INFO [main] o.a.n.b.NotificationServiceManager
> Successfully loaded the following 0 services: []
> 2017-07-04 14:03:46,070 INFO [main] org.apache.nifi.bootstrap.RunNiFi
> Registered no Notification Services f
[jira] [Assigned] (NIFI-4150) NiFi starting failure and key file already existing
[
https://issues.apache.org/jira/browse/NIFI-4150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andy LoPresto reassigned NIFI-4150:
---
Assignee: Andy LoPresto
> NiFi starting failure and key file already existing
> ---
>
> Key: NIFI-4150
> URL: https://issues.apache.org/jira/browse/NIFI-4150
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.2.0, 1.3.0
>Reporter: Pierre Villard
>Assignee: Andy LoPresto
>
> In some cases, if NiFi cannot start (in my case: debug port was already in
> use), the key file is still created and not deleted. Then it won't be
> possible to restart NiFi unless the key file is manually deleted.
> Logs from bootstrap:
> {code}
> 2017-07-04 13:57:40,460 INFO [main] o.a.n.b.NotificationServiceManager
> Successfully loaded the following 0 services: []
> 2017-07-04 13:57:40,466 INFO [main] org.apache.nifi.bootstrap.RunNiFi
> Registered no Notification Services for Notification Type NIFI_STARTED
> 2017-07-04 13:57:40,466 INFO [main] org.apache.nifi.bootstrap.RunNiFi
> Registered no Notification Services for Notification Type NIFI_STOPPED
> 2017-07-04 13:57:40,466 INFO [main] org.apache.nifi.bootstrap.RunNiFi
> Registered no Notification Services for Notification Type NIFI_DIED
> 2017-07-04 13:57:40,508 INFO [main] org.apache.nifi.bootstrap.Command
> Starting Apache NiFi...
> 2017-07-04 13:57:40,509 INFO [main] org.apache.nifi.bootstrap.Command Working
> Directory: /usr/hdf/current/nifi
> 2017-07-04 13:57:40,510 INFO [main] org.apache.nifi.bootstrap.Command
> Command: /usr/jdk64/jdk1.8.0_112/bin/java -classpath
> /usr/hdf/current/nifi/conf:/usr/hdf/current/nifi/lib/nifi-runtime-1.2.0.3.0.0.0-453.jar:/usr/hdf/current/nifi/lib/javax.servlet-api-3.1.0.jar:/usr/hdf/current/nifi/lib/jcl-over-slf4j-1.7.25.jar:/usr/hdf/current/nifi/lib/jetty-schemas-3.1.jar:/usr/hdf/current/nifi/lib/jul-to-slf4j-1.7.25.jar:/usr/hdf/current/nifi/lib/log4j-over-slf4j-1.7.25.jar:/usr/hdf/current/nifi/lib/logback-classic-1.2.3.jar:/usr/hdf/current/nifi/lib/logback-core-1.2.3.jar:/usr/hdf/current/nifi/lib/nifi-api-1.2.0.3.0.0.0-453.jar:/usr/hdf/current/nifi/lib/nifi-nar-utils-1.2.0.3.0.0.0-453.jar:/usr/hdf/current/nifi/lib/nifi-properties-1.2.0.3.0.0.0-453.jar:/usr/hdf/current/nifi/lib/slf4j-api-1.7.25.jar:/usr/hdf/current/nifi/lib/nifi-framework-api-1.2.0.3.0.0.0-453.jar
> -Dorg.apache.jasper.compiler.disablejsr199=true
> -Djava.security.auth.login.config=/usr/hdf/current/nifi/conf/nifi_jaas.conf
> -Xmx512m -Xms512m
> -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000
> -Dambari.application.id=nifi
> -Dambari.metrics.collector.url=http://pvillard-1:6188/ws/v1/timeline/metrics
> -Dsun.net.http.allowRestrictedHeaders=true -Djava.net.preferIPv4Stack=true
> -Djava.awt.headless=true -XX:+UseG1GC
> -Djava.protocol.handler.pkgs=sun.net.www.protocol
> -Dnifi.properties.file.path=/usr/hdf/current/nifi/conf/nifi.properties
> -Dnifi.bootstrap.listen.port=33910 -Dapp=NiFi
> -Dorg.apache.nifi.bootstrap.config.log.dir=/var/log/nifi org.apache.nifi.NiFi
> -K /usr/hdf/current/nifi/conf/sensitive.key
> 2017-07-04 13:57:40,532 INFO [main] org.apache.nifi.bootstrap.Command
> Launched Apache NiFi with Process ID 30029
> 2017-07-04 13:57:40,647 ERROR [NiFi logging handler] org.apache.nifi.StdErr
> ERROR: transport error 202: bind failed: Address already in use
> 2017-07-04 13:57:40,648 ERROR [NiFi logging handler] org.apache.nifi.StdErr
> ERROR: JDWP Transport dt_socket failed to initialize, TRANSPORT_INIT(510)
> 2017-07-04 13:57:40,648 ERROR [NiFi logging handler] org.apache.nifi.StdErr
> JDWP exit error AGENT_ERROR_TRANSPORT_INIT(197): No transports initialized
> [debugInit.c:750]
> 2017-07-04 13:57:41,536 INFO [main] org.apache.nifi.bootstrap.RunNiFi NiFi
> never started. Will not restart NiFi
> 2017-07-04 14:03:26,828 INFO [main] o.a.n.b.NotificationServiceManager
> Successfully loaded the following 0 services: []
> 2017-07-04 14:03:26,834 INFO [main] org.apache.nifi.bootstrap.RunNiFi
> Registered no Notification Services for Notification Type NIFI_STARTED
> 2017-07-04 14:03:26,835 INFO [main] org.apache.nifi.bootstrap.RunNiFi
> Registered no Notification Services for Notification Type NIFI_STOPPED
> 2017-07-04 14:03:26,835 INFO [main] org.apache.nifi.bootstrap.RunNiFi
> Registered no Notification Services for Notification Type NIFI_DIED
> 2017-07-04 14:03:26,838 INFO [main] org.apache.nifi.bootstrap.Command Apache
> NiFi is not currently running
> 2017-07-04 14:03:46,063 INFO [main] o.a.n.b.NotificationServiceManager
> Successfully loaded the following 0 services: []
> 2017-07-04 14:03:46,070 INFO [main] org.apache.nifi.bootstrap.RunNiFi
> Registered no Notification Services for Notification Type NIFI_STARTED
> 2017-07-04 14:03:46,070 INFO
[jira] [Created] (NIFI-4340) RecordPathResult may throw NPE
Pierre Villard created NIFI-4340:
Summary: RecordPathResult may throw NPE
Key: NIFI-4340
URL: https://issues.apache.org/jira/browse/NIFI-4340
Project: Apache NiFi
Issue Type: Bug
Components: Core Framework
Affects Versions: 1.3.0
Reporter: Pierre Villard
When using UpdateRecord and indicating a path existing in the schema but
pointing to a null field, it'll throw a NullPointerException.
Basically, NPE is thrown at:
{noformat}
result.getSelectedFields().forEach(...)
{noformat}
or
{noformat}
recordPath.evaluate(record).getSelectedFields().iterator().hasNext()
{noformat}
Stack trace:
{noformat}
2017-08-31 18:21:24,202 ERROR [Timer-Driven Process Thread-5]
o.a.n.processors.standard.UpdateRecord
UpdateRecord[id=39158233-015e-1000-c81d-5cbd8f5cddf1] Failed to process
StandardFlowFileRecord[uuid=f1807e4b-3374-4d9d-bfcf-b44ef7b6910b,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1504194471839-1, container=default,
section=1], offset=63430,
length=5291],offset=0,name=340191697112484,size=5291]:
java.lang.NullPointerException
java.lang.NullPointerException: null
at
org.apache.nifi.record.path.paths.ChildFieldPath.getChild(ChildFieldPath.java:50)
at
org.apache.nifi.record.path.paths.ChildFieldPath.lambda$evaluate$0(ChildFieldPath.java:67)
at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at
java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
at java.util.stream.IntPipeline$4$1.accept(IntPipeline.java:250)
at
java.util.stream.Streams$RangeIntSpliterator.forEachRemaining(Streams.java:110)
at java.util.Spliterator$OfInt.forEachRemaining(Spliterator.java:693)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at
java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
at
java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at
java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
at
java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:270)
at
java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at
java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
at java.util.stream.IntPipeline$4$1.accept(IntPipeline.java:250)
at
java.util.stream.Streams$RangeIntSpliterator.forEachRemaining(Streams.java:110)
at java.util.Spliterator$OfInt.forEachRemaining(Spliterator.java:693)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at
java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
at
java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at
java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
at
java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:270)
at
java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at
java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
at java.util.stream.IntPipeline$4$1.accept(IntPipeline.java:250)
at
java.util.stream.Streams$RangeIntSpliterator.forEachRemaining(Streams.java:110)
at java.util.Spliterator$OfInt.forEachRemaining(Spliterator.java:693)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at
java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
at
java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at
java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
at
java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:270)
at
java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at
java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
at java.util.stream.IntPipeline$4$1.accept(IntPipeline.java:250)
[jira] [Assigned] (NIFI-4340) RecordPathResult may throw NPE
[
https://issues.apache.org/jira/browse/NIFI-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne reassigned NIFI-4340:
Assignee: Mark Payne
> RecordPathResult may throw NPE
> --
>
> Key: NIFI-4340
> URL: https://issues.apache.org/jira/browse/NIFI-4340
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.3.0
>Reporter: Pierre Villard
>Assignee: Mark Payne
>
> When using UpdateRecord and indicating a path existing in the schema but
> pointing to a null field, it'll throw a NullPointerException.
> Basically, NPE is thrown at:
> {noformat}
> result.getSelectedFields().forEach(...)
> {noformat}
> or
> {noformat}
> recordPath.evaluate(record).getSelectedFields().iterator().hasNext()
> {noformat}
> Stack trace:
> {noformat}
> 2017-08-31 18:21:24,202 ERROR [Timer-Driven Process Thread-5]
> o.a.n.processors.standard.UpdateRecord
> UpdateRecord[id=39158233-015e-1000-c81d-5cbd8f5cddf1] Failed to process
> StandardFlowFileRecord[uuid=f1807e4b-3374-4d9d-bfcf-b44ef7b6910b,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1504194471839-1, container=default,
> section=1], offset=63430,
> length=5291],offset=0,name=340191697112484,size=5291]:
> java.lang.NullPointerException
> java.lang.NullPointerException: null
> at
> org.apache.nifi.record.path.paths.ChildFieldPath.getChild(ChildFieldPath.java:50)
> at
> org.apache.nifi.record.path.paths.ChildFieldPath.lambda$evaluate$0(ChildFieldPath.java:67)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
> at java.util.stream.IntPipeline$4$1.accept(IntPipeline.java:250)
> at
> java.util.stream.Streams$RangeIntSpliterator.forEachRemaining(Streams.java:110)
> at java.util.Spliterator$OfInt.forEachRemaining(Spliterator.java:693)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
> at
> java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:270)
> at
> java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
> at java.util.stream.IntPipeline$4$1.accept(IntPipeline.java:250)
> at
> java.util.stream.Streams$RangeIntSpliterator.forEachRemaining(Streams.java:110)
> at java.util.Spliterator$OfInt.forEachRemaining(Spliterator.java:693)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
> at
> java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:270)
> at
> java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
> at java.util.stream.IntPipeline$4$1.accept(IntPipeline.java:250)
> at
> java.util.stream.Streams$RangeIntSpliterator.forEachRemaining(Streams.java:110)
> at java.util.Spliterator$OfInt.forEachRemaining(Spliterator.java:693)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
> at
> java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:270)
> at
> java.util.stream.Referen
[jira] [Commented] (NIFI-4340) RecordPathResult may throw NPE
[
https://issues.apache.org/jira/browse/NIFI-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16149254#comment-16149254
]
Mark Payne commented on NIFI-4340:
--
This same issue is likely to affect DescendantFieldPath and possibly ParentPath
as well.
> RecordPathResult may throw NPE
> --
>
> Key: NIFI-4340
> URL: https://issues.apache.org/jira/browse/NIFI-4340
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.3.0
>Reporter: Pierre Villard
>Assignee: Mark Payne
>
> When using UpdateRecord and indicating a path existing in the schema but
> pointing to a null field, it'll throw a NullPointerException.
> Basically, NPE is thrown at:
> {noformat}
> result.getSelectedFields().forEach(...)
> {noformat}
> or
> {noformat}
> recordPath.evaluate(record).getSelectedFields().iterator().hasNext()
> {noformat}
> Stack trace:
> {noformat}
> 2017-08-31 18:21:24,202 ERROR [Timer-Driven Process Thread-5]
> o.a.n.processors.standard.UpdateRecord
> UpdateRecord[id=39158233-015e-1000-c81d-5cbd8f5cddf1] Failed to process
> StandardFlowFileRecord[uuid=f1807e4b-3374-4d9d-bfcf-b44ef7b6910b,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1504194471839-1, container=default,
> section=1], offset=63430,
> length=5291],offset=0,name=340191697112484,size=5291]:
> java.lang.NullPointerException
> java.lang.NullPointerException: null
> at
> org.apache.nifi.record.path.paths.ChildFieldPath.getChild(ChildFieldPath.java:50)
> at
> org.apache.nifi.record.path.paths.ChildFieldPath.lambda$evaluate$0(ChildFieldPath.java:67)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
> at java.util.stream.IntPipeline$4$1.accept(IntPipeline.java:250)
> at
> java.util.stream.Streams$RangeIntSpliterator.forEachRemaining(Streams.java:110)
> at java.util.Spliterator$OfInt.forEachRemaining(Spliterator.java:693)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
> at
> java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:270)
> at
> java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
> at java.util.stream.IntPipeline$4$1.accept(IntPipeline.java:250)
> at
> java.util.stream.Streams$RangeIntSpliterator.forEachRemaining(Streams.java:110)
> at java.util.Spliterator$OfInt.forEachRemaining(Spliterator.java:693)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
> at
> java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:270)
> at
> java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
> at java.util.stream.IntPipeline$4$1.accept(IntPipeline.java:250)
> at
> java.util.stream.Streams$RangeIntSpliterator.forEachRemaining(Streams.java:110)
> at java.util.Spliterator$OfInt.forEachRemaining(Spliterator.java:693)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
> at
[jira] [Created] (NIFIREG-12) Add Search Capabilities
Bryan Bende created NIFIREG-12: -- Summary: Add Search Capabilities Key: NIFIREG-12 URL: https://issues.apache.org/jira/browse/NIFIREG-12 Project: NiFi Registry Issue Type: Improvement Reporter: Bryan Bende Assignee: Bryan Bende Priority: Minor We should be able to perform basic search operations against items in the registry, with support for paging and sorting. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] nifi pull request #2118: NIFI-4339: Fix NPE in ExtractEmailHeaders
GitHub user kevdoran opened a pull request: https://github.com/apache/nifi/pull/2118 NIFI-4339: Fix NPE in ExtractEmailHeaders Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kevdoran/nifi NIFI-4339 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2118.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2118 commit ec431048fb80d1150559850af79f6d6ef47e74a8 Author: Kevin Doran Date: 2017-08-31T15:58:16Z NIFI-4339: Fix NPE in ExtractEmailHeaders --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-4339) Possible NPE in ExtractEmailHeaders if no recipients exist
[
https://issues.apache.org/jira/browse/NIFI-4339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16149343#comment-16149343
]
ASF GitHub Bot commented on NIFI-4339:
--
GitHub user kevdoran opened a pull request:
https://github.com/apache/nifi/pull/2118
NIFI-4339: Fix NPE in ExtractEmailHeaders
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with NIFI- where is the JIRA number
you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically master)?
- [ ] Is your initial contribution a single, squashed commit?
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file, including the main
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in
which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kevdoran/nifi NIFI-4339
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/2118.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2118
commit ec431048fb80d1150559850af79f6d6ef47e74a8
Author: Kevin Doran
Date: 2017-08-31T15:58:16Z
NIFI-4339: Fix NPE in ExtractEmailHeaders
> Possible NPE in ExtractEmailHeaders if no recipients exist
> --
>
> Key: NIFI-4339
> URL: https://issues.apache.org/jira/browse/NIFI-4339
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Kevin Doran
>Assignee: Kevin Doran
>Priority: Minor
> Fix For: 1.4.0
>
>
> A user uncovered a *NullPointerException* in the ExtractEmailHeaders
> processor and [asked about it on
> StackOverflow|https://stackoverflow.com/questions/45927852/nifi-email-consumeimap-filter-by-subject-from-to-and-date].
> Here is the summary: If the TO, CC, and BCC are not present in the email
> message, a call to {{javax.mail.internet.MimeMessage getAllRecipients()}}
> will return {{null}}.
> While this is a rare case, the reporter has a use case in which an email
> account is sending automated emails to the same address as the source, and
> whatever email program is being used appears to not set any address fields in
> this case:
> {quote}It is an auto email from client to their own inbox and does not have
> "to", "cc"and "bcc"{quote}
> The {{javax.mail.Message getAllRecipients()}} method, which is called from
> {{javax.mail.internet.MimeMessage getAllRecipients()}}, will return null in
> this case, as is documented and apparent from the source code:
> {code:java}
> /**
> * Get all the recipient addresses for the message.
> * The default implementation extracts the TO, CC, and BCC
> * recipients using the getRecipients method.
> *
> * This method returns null if none of the recipient
> * headers are present in this message. It may Return an empty array
> * if any recipient header is present, but contains no addresses.
> *
> * @return array of Address objects
> * @exception MessagingException
> */
> public Address[] getAllRecipients() throws MessagingException {
> Address[] to = getRecipients(RecipientType.TO);
> Address[] cc = getRecipients(RecipientType.CC);
> Address[] bcc = getRecipients(RecipientType.BCC);
> if (cc == null && bcc == null)
> re
[GitHub] nifi issue #2111: NIFI-4326 - ExtractEmailHeaders.java unhandled NullPointer...
Github user kevdoran commented on the issue: https://github.com/apache/nifi/pull/2111 Whoops, didn't see this (or the corresponding JIRA) until after I opened a ticket and PR for the same issue ([NIFI-4339](https://issues.apache.org/jira/browse/NIFI-4339), [PR 2118](https://github.com/apache/nifi/pull/2118)) after it showed up on [StackOverflow](https://stackoverflow.com/questions/45927852/nifi-email-consumeimap-filter-by-subject-from-to-and-date/45928680#comment78898203_45928680). Looks like I accidentally duplicated your effort @btwood, my apologies. I was able to come up with a test case for the issue, see my branch if interested. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-4326) ExtractEmailHeaders.java unhandled Exceptions
[ https://issues.apache.org/jira/browse/NIFI-4326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16149392#comment-16149392 ] ASF GitHub Bot commented on NIFI-4326: -- Github user kevdoran commented on the issue: https://github.com/apache/nifi/pull/2111 Whoops, didn't see this (or the corresponding JIRA) until after I opened a ticket and PR for the same issue ([NIFI-4339](https://issues.apache.org/jira/browse/NIFI-4339), [PR 2118](https://github.com/apache/nifi/pull/2118)) after it showed up on [StackOverflow](https://stackoverflow.com/questions/45927852/nifi-email-consumeimap-filter-by-subject-from-to-and-date/45928680#comment78898203_45928680). Looks like I accidentally duplicated your effort @btwood, my apologies. I was able to come up with a test case for the issue, see my branch if interested. > ExtractEmailHeaders.java unhandled Exceptions > - > > Key: NIFI-4326 > URL: https://issues.apache.org/jira/browse/NIFI-4326 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.3.0 > Environment: jdk 1.8.0_121-b13 >Reporter: Benjamin Wood >Priority: Minor > Original Estimate: 0.5h > Remaining Estimate: 0.5h > > The ExtractEmailHeaders processor throws a NullPointerException if there is > no TO, CC, and BCC recipients. > If there are no recipients "originalMessage.getAllRecipients()" returns NULL, > and not a 0 length array. > If an address is empty (<> or " ") then getRecipients() will throw an "Empty > Address" AddressException > It's possible this is only an issue with Oracle Java. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] nifi issue #2111: NIFI-4326 - ExtractEmailHeaders.java unhandled NullPointer...
Github user btwood commented on the issue: https://github.com/apache/nifi/pull/2111 @kevdoran No worries. I've actually been working on this for a while. I have a solution for both the NullPointerException, and also a problem I was having in my own production environment with strict email addresses. I've all but committed the code at this point. I'll be making a commit very soon, and you can see if my fork solves your problem. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-4326) ExtractEmailHeaders.java unhandled Exceptions
[ https://issues.apache.org/jira/browse/NIFI-4326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16149402#comment-16149402 ] ASF GitHub Bot commented on NIFI-4326: -- Github user btwood commented on the issue: https://github.com/apache/nifi/pull/2111 @kevdoran No worries. I've actually been working on this for a while. I have a solution for both the NullPointerException, and also a problem I was having in my own production environment with strict email addresses. I've all but committed the code at this point. I'll be making a commit very soon, and you can see if my fork solves your problem. > ExtractEmailHeaders.java unhandled Exceptions > - > > Key: NIFI-4326 > URL: https://issues.apache.org/jira/browse/NIFI-4326 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.3.0 > Environment: jdk 1.8.0_121-b13 >Reporter: Benjamin Wood >Priority: Minor > Original Estimate: 0.5h > Remaining Estimate: 0.5h > > The ExtractEmailHeaders processor throws a NullPointerException if there is > no TO, CC, and BCC recipients. > If there are no recipients "originalMessage.getAllRecipients()" returns NULL, > and not a 0 length array. > If an address is empty (<> or " ") then getRecipients() will throw an "Empty > Address" AddressException > It's possible this is only an issue with Oracle Java. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] nifi pull request #2114: NIFI-4320 - Escape quotes for JSON output in QueryC...
Github user pvillard31 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2114#discussion_r136420279
--- Diff:
nifi-nar-bundles/nifi-cassandra-bundle/nifi-cassandra-processors/src/main/java/org/apache/nifi/processors/cassandra/QueryCassandra.java
---
@@ -492,6 +492,8 @@ protected static String getJsonElement(Object value) {
SimpleDateFormat dateFormat = new SimpleDateFormat("-MM-dd
HH:mm:ssZ");
dateFormat.setTimeZone(TimeZone.getTimeZone("UTC"));
return "\"" + dateFormat.format((Date) value) + "\"";
+} else if (value instanceof String) {
+return "\"" + ((String) value).replaceAll("\"", "\"") +
"\"";
--- End diff --
Of course, thanks! I updated the PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (NIFI-4320) Processor QueryCassandra doens't escape double-quote when output format is JSON
[
https://issues.apache.org/jira/browse/NIFI-4320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16149438#comment-16149438
]
ASF GitHub Bot commented on NIFI-4320:
--
Github user pvillard31 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2114#discussion_r136420279
--- Diff:
nifi-nar-bundles/nifi-cassandra-bundle/nifi-cassandra-processors/src/main/java/org/apache/nifi/processors/cassandra/QueryCassandra.java
---
@@ -492,6 +492,8 @@ protected static String getJsonElement(Object value) {
SimpleDateFormat dateFormat = new SimpleDateFormat("-MM-dd
HH:mm:ssZ");
dateFormat.setTimeZone(TimeZone.getTimeZone("UTC"));
return "\"" + dateFormat.format((Date) value) + "\"";
+} else if (value instanceof String) {
+return "\"" + ((String) value).replaceAll("\"", "\"") +
"\"";
--- End diff --
Of course, thanks! I updated the PR.
> Processor QueryCassandra doens't escape double-quote when output format is
> JSON
> ---
>
> Key: NIFI-4320
> URL: https://issues.apache.org/jira/browse/NIFI-4320
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.3.0
> Environment: Ubuntu 14.05/Java 8 Oracle/Cassandra 3.4
>Reporter: Jean-Louis
>Assignee: Pierre Villard
> Labels: cassandra, json
>
> When the output format is JSON, if a content contains double-quote ("),
> they're not escape! So, the produced JSON malformed
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (NIFI-4326) ExtractEmailHeaders.java unhandled Exceptions
[ https://issues.apache.org/jira/browse/NIFI-4326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16149457#comment-16149457 ] ASF GitHub Bot commented on NIFI-4326: -- Github user btwood commented on the issue: https://github.com/apache/nifi/pull/2111 @kevdoran The patch is in. I'll see if I can get it committed to mainline. Let me know if this solves your issue. I noticed you made some changes to a few other files in your PR. So I'm not sure if this operates the same way. I didn't make any changes to the jUnit tests, and I'm not sure I need to. I guess I could add "non-strict" addresses in the CC field or something... > ExtractEmailHeaders.java unhandled Exceptions > - > > Key: NIFI-4326 > URL: https://issues.apache.org/jira/browse/NIFI-4326 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.3.0 > Environment: jdk 1.8.0_121-b13 >Reporter: Benjamin Wood >Priority: Minor > Original Estimate: 0.5h > Remaining Estimate: 0.5h > > The ExtractEmailHeaders processor throws a NullPointerException if there is > no TO, CC, and BCC recipients. > If there are no recipients "originalMessage.getAllRecipients()" returns NULL, > and not a 0 length array. > If an address is empty (<> or " ") then getRecipients() will throw an "Empty > Address" AddressException > It's possible this is only an issue with Oracle Java. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] nifi issue #2111: NIFI-4326 - ExtractEmailHeaders.java unhandled NullPointer...
Github user btwood commented on the issue: https://github.com/apache/nifi/pull/2111 @kevdoran The patch is in. I'll see if I can get it committed to mainline. Let me know if this solves your issue. I noticed you made some changes to a few other files in your PR. So I'm not sure if this operates the same way. I didn't make any changes to the jUnit tests, and I'm not sure I need to. I guess I could add "non-strict" addresses in the CC field or something... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (NIFI-4326) ExtractEmailHeaders.java unhandled Exceptions
[ https://issues.apache.org/jira/browse/NIFI-4326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Wood updated NIFI-4326: Fix Version/s: 1.4.0 > ExtractEmailHeaders.java unhandled Exceptions > - > > Key: NIFI-4326 > URL: https://issues.apache.org/jira/browse/NIFI-4326 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.3.0 > Environment: jdk 1.8.0_121-b13 >Reporter: Benjamin Wood >Priority: Minor > Fix For: 1.4.0 > > Original Estimate: 0.5h > Remaining Estimate: 0.5h > > The ExtractEmailHeaders processor throws a NullPointerException if there is > no TO, CC, and BCC recipients. > If there are no recipients "originalMessage.getAllRecipients()" returns NULL, > and not a 0 length array. > If an address is empty (<> or " ") then getRecipients() will throw an "Empty > Address" AddressException > It's possible this is only an issue with Oracle Java. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] nifi issue #2111: NIFI-4326 - ExtractEmailHeaders.java unhandled NullPointer...
Github user kevdoran commented on the issue: https://github.com/apache/nifi/pull/2111 @btwood Thanks! I think they are close enough that I still consider them duplicates -- or more precisely, my patch is a subset of the issues yours addresses, as you have also added handling for non-strict addresses. Your changes should fix the case reported on StackOverflow, which is what my PR was intended to solve, but as I don't have access to that data the OP on SO will have to verify. I've encouraged that in a comment there. I think the best thing to do is for me to close my current PR and instead submit a patch to your branch that adds my unit test case. Would you be open to that? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-4326) ExtractEmailHeaders.java unhandled Exceptions
[ https://issues.apache.org/jira/browse/NIFI-4326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16149470#comment-16149470 ] ASF GitHub Bot commented on NIFI-4326: -- Github user kevdoran commented on the issue: https://github.com/apache/nifi/pull/2111 @btwood Thanks! I think they are close enough that I still consider them duplicates -- or more precisely, my patch is a subset of the issues yours addresses, as you have also added handling for non-strict addresses. Your changes should fix the case reported on StackOverflow, which is what my PR was intended to solve, but as I don't have access to that data the OP on SO will have to verify. I've encouraged that in a comment there. I think the best thing to do is for me to close my current PR and instead submit a patch to your branch that adds my unit test case. Would you be open to that? > ExtractEmailHeaders.java unhandled Exceptions > - > > Key: NIFI-4326 > URL: https://issues.apache.org/jira/browse/NIFI-4326 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.3.0 > Environment: jdk 1.8.0_121-b13 >Reporter: Benjamin Wood >Priority: Minor > Fix For: 1.4.0 > > Original Estimate: 0.5h > Remaining Estimate: 0.5h > > The ExtractEmailHeaders processor throws a NullPointerException if there is > no TO, CC, and BCC recipients. > If there are no recipients "originalMessage.getAllRecipients()" returns NULL, > and not a 0 length array. > If an address is empty (<> or " ") then getRecipients() will throw an "Empty > Address" AddressException > It's possible this is only an issue with Oracle Java. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] nifi issue #2111: NIFI-4326 - ExtractEmailHeaders.java unhandled NullPointer...
Github user btwood commented on the issue: https://github.com/apache/nifi/pull/2111 Yes. We can definitely do that. Once you submit to my brand I'll review/accept the changes. Side note, how can I make myself the "Assignee" in JIRA? I can't seem to make any updates that I'm doing the work, or that the work is complete (or almost complete). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-4326) ExtractEmailHeaders.java unhandled Exceptions
[ https://issues.apache.org/jira/browse/NIFI-4326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16149478#comment-16149478 ] ASF GitHub Bot commented on NIFI-4326: -- Github user btwood commented on the issue: https://github.com/apache/nifi/pull/2111 Yes. We can definitely do that. Once you submit to my brand I'll review/accept the changes. Side note, how can I make myself the "Assignee" in JIRA? I can't seem to make any updates that I'm doing the work, or that the work is complete (or almost complete). > ExtractEmailHeaders.java unhandled Exceptions > - > > Key: NIFI-4326 > URL: https://issues.apache.org/jira/browse/NIFI-4326 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.3.0 > Environment: jdk 1.8.0_121-b13 >Reporter: Benjamin Wood >Priority: Minor > Fix For: 1.4.0 > > Original Estimate: 0.5h > Remaining Estimate: 0.5h > > The ExtractEmailHeaders processor throws a NullPointerException if there is > no TO, CC, and BCC recipients. > If there are no recipients "originalMessage.getAllRecipients()" returns NULL, > and not a 0 length array. > If an address is empty (<> or " ") then getRecipients() will throw an "Empty > Address" AddressException > It's possible this is only an issue with Oracle Java. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] nifi pull request #2111: NIFI-4326 - ExtractEmailHeaders.java unhandled Null...
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2111#discussion_r136448958
--- Diff:
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ExtractEmailAttachments.java
---
@@ -131,11 +132,15 @@ public void process(final InputStream rawIn) throws
IOException {
MimeMessageParser parser = new
MimeMessageParser(originalMessage).parse();
// RFC-2822 determines that a message must have a
"From:" header
// if a message lacks the field, it is flagged as
invalid
-Address[] from = originalMessage.getFrom();
+if
(InternetAddress.parseHeader(originalMessage.getHeader("From", ","), false) ==
null) {
+if
(InternetAddress.parseHeader(originalMessage.getHeader("Sender", ","), false)
== null) {
--- End diff --
Do you think falling back to "Sender" if "From" is not present for RFC
validation is desired behavior? According to RFC-2882, and the RFCs that update
it ([RFC-5322](https://tools.ietf.org/html/rfc5322),
[RFC-6854](https://tools.ietf.org/html/rfc6854)), the "Sender" field should
only be present if "From" is also present.
That said, @btwood, have you encountered email servers that use "Sender"
without "From"? If so, it might be worth considering adjusting the
implementation here to reflect what is likely to be encountered in the wild vs.
strict RFC specification.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #2111: NIFI-4326 - ExtractEmailHeaders.java unhandled Null...
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2111#discussion_r136450708
--- Diff:
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ExtractEmailHeaders.java
---
@@ -168,21 +173,40 @@ public void process(final InputStream rawIn) throws
IOException {
}
}
}
-if
(Array.getLength(originalMessage.getAllRecipients()) > 0) {
-for (int toCount = 0; toCount <
ArrayUtils.getLength(originalMessage.getRecipients(Message.RecipientType.TO));
toCount++) {
-attributes.put(EMAIL_HEADER_TO + "." +
toCount,
originalMessage.getRecipients(Message.RecipientType.TO)[toCount].toString());
+
+// Get Non-Strict Recipient Addresses
+InternetAddress[] recipients;
+if
(originalMessage.getHeader(Message.RecipientType.TO.toString(), ",") != null) {
+recipients =
InternetAddress.parseHeader(originalMessage.getHeader(Message.RecipientType.TO.toString(),
","), false);
+for (int toCount = 0; toCount <
ArrayUtils.getLength(recipients); toCount++) {
+attributes.put(EMAIL_HEADER_TO + "." +
toCount, recipients[toCount].toString());
}
-for (int toCount = 0; toCount <
ArrayUtils.getLength(originalMessage.getRecipients(Message.RecipientType.BCC));
toCount++) {
-attributes.put(EMAIL_HEADER_BCC + "." +
toCount,
originalMessage.getRecipients(Message.RecipientType.BCC)[toCount].toString());
+}
+if
(originalMessage.getHeader(Message.RecipientType.BCC.toString(), ",") != null) {
+recipients =
InternetAddress.parseHeader(originalMessage.getHeader(Message.RecipientType.BCC.toString(),
","), false);
+for (int toCount = 0; toCount <
ArrayUtils.getLength(recipients); toCount++) {
+attributes.put(EMAIL_HEADER_BCC + "." +
toCount, recipients[toCount].toString());
}
-for (int toCount = 0; toCount <
ArrayUtils.getLength(originalMessage.getRecipients(Message.RecipientType.CC));
toCount++) {
-attributes.put(EMAIL_HEADER_CC + "." +
toCount,
originalMessage.getRecipients(Message.RecipientType.CC)[toCount].toString());
+}
+if
(originalMessage.getHeader(Message.RecipientType.CC.toString(), ",") != null) {
+recipients =
InternetAddress.parseHeader(originalMessage.getHeader(Message.RecipientType.CC.toString(),
","), false);
+for (int toCount = 0; toCount <
ArrayUtils.getLength(recipients); toCount++) {
+attributes.put(EMAIL_HEADER_CC + "." +
toCount, recipients[toCount].toString());
}
}
-// Incredibly enough RFC-2822 specified From as a
"mailbox-list" so an array I returned by getFrom
-for (int toCount = 0; toCount <
ArrayUtils.getLength(originalMessage.getFrom()); toCount++) {
-attributes.put(EMAIL_HEADER_FROM + "." + toCount,
originalMessage.getFrom()[toCount].toString());
+
+// Get Non-Strict Sender Addresses
+InternetAddress[] sender = null;
+if (originalMessage.getHeader("From",",") != null) {
+sender =
(InternetAddress[])ArrayUtils.addAll(sender,
InternetAddress.parseHeader(originalMessage.getHeader("From", ","), false));
+}
+if (originalMessage.getHeader("Sender",",") != null) {
+sender =
(InternetAddress[])ArrayUtils.addAll(sender,
InternetAddress.parseHeader(originalMessage.getHeader("Sender", ","), false));
--- End diff --
I don't think we should combine the values of the Sender and From headers
here, for two reasons:
1. It breaks the documentation for the processor that states
email.headers.from will contain the values in From
2. If From is a mailbox-list, Sender might be a single mailbox in the From
mailbox-list, so the result here could be a duplicate email address.
I think this could break expectations from folks already using the
ExtractEmailHeaders processor. If this functionality is needed, a new header
field (ie, email.headers.sender) should be added. However, in that case, this
functionality could already be achieved using the Additional Headers capability
offered by ExtractEmai
[jira] [Commented] (NIFI-4326) ExtractEmailHeaders.java unhandled Exceptions
[
https://issues.apache.org/jira/browse/NIFI-4326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16149631#comment-16149631
]
ASF GitHub Bot commented on NIFI-4326:
--
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2111#discussion_r136450708
--- Diff:
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ExtractEmailHeaders.java
---
@@ -168,21 +173,40 @@ public void process(final InputStream rawIn) throws
IOException {
}
}
}
-if
(Array.getLength(originalMessage.getAllRecipients()) > 0) {
-for (int toCount = 0; toCount <
ArrayUtils.getLength(originalMessage.getRecipients(Message.RecipientType.TO));
toCount++) {
-attributes.put(EMAIL_HEADER_TO + "." +
toCount,
originalMessage.getRecipients(Message.RecipientType.TO)[toCount].toString());
+
+// Get Non-Strict Recipient Addresses
+InternetAddress[] recipients;
+if
(originalMessage.getHeader(Message.RecipientType.TO.toString(), ",") != null) {
+recipients =
InternetAddress.parseHeader(originalMessage.getHeader(Message.RecipientType.TO.toString(),
","), false);
+for (int toCount = 0; toCount <
ArrayUtils.getLength(recipients); toCount++) {
+attributes.put(EMAIL_HEADER_TO + "." +
toCount, recipients[toCount].toString());
}
-for (int toCount = 0; toCount <
ArrayUtils.getLength(originalMessage.getRecipients(Message.RecipientType.BCC));
toCount++) {
-attributes.put(EMAIL_HEADER_BCC + "." +
toCount,
originalMessage.getRecipients(Message.RecipientType.BCC)[toCount].toString());
+}
+if
(originalMessage.getHeader(Message.RecipientType.BCC.toString(), ",") != null) {
+recipients =
InternetAddress.parseHeader(originalMessage.getHeader(Message.RecipientType.BCC.toString(),
","), false);
+for (int toCount = 0; toCount <
ArrayUtils.getLength(recipients); toCount++) {
+attributes.put(EMAIL_HEADER_BCC + "." +
toCount, recipients[toCount].toString());
}
-for (int toCount = 0; toCount <
ArrayUtils.getLength(originalMessage.getRecipients(Message.RecipientType.CC));
toCount++) {
-attributes.put(EMAIL_HEADER_CC + "." +
toCount,
originalMessage.getRecipients(Message.RecipientType.CC)[toCount].toString());
+}
+if
(originalMessage.getHeader(Message.RecipientType.CC.toString(), ",") != null) {
+recipients =
InternetAddress.parseHeader(originalMessage.getHeader(Message.RecipientType.CC.toString(),
","), false);
+for (int toCount = 0; toCount <
ArrayUtils.getLength(recipients); toCount++) {
+attributes.put(EMAIL_HEADER_CC + "." +
toCount, recipients[toCount].toString());
}
}
-// Incredibly enough RFC-2822 specified From as a
"mailbox-list" so an array I returned by getFrom
-for (int toCount = 0; toCount <
ArrayUtils.getLength(originalMessage.getFrom()); toCount++) {
-attributes.put(EMAIL_HEADER_FROM + "." + toCount,
originalMessage.getFrom()[toCount].toString());
+
+// Get Non-Strict Sender Addresses
+InternetAddress[] sender = null;
+if (originalMessage.getHeader("From",",") != null) {
+sender =
(InternetAddress[])ArrayUtils.addAll(sender,
InternetAddress.parseHeader(originalMessage.getHeader("From", ","), false));
+}
+if (originalMessage.getHeader("Sender",",") != null) {
+sender =
(InternetAddress[])ArrayUtils.addAll(sender,
InternetAddress.parseHeader(originalMessage.getHeader("Sender", ","), false));
--- End diff --
I don't think we should combine the values of the Sender and From headers
here, for two reasons:
1. It breaks the documentation for the processor that states
email.headers.from will contain the values in From
2. If From is a mailbox-list, Sender might be a single mailbox in the From
mailbox-list, so the result here could be a duplicate email address.
I think this could break expectations from folks already using the
Extract
[jira] [Commented] (NIFI-4326) ExtractEmailHeaders.java unhandled Exceptions
[
https://issues.apache.org/jira/browse/NIFI-4326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16149630#comment-16149630
]
ASF GitHub Bot commented on NIFI-4326:
--
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2111#discussion_r136448958
--- Diff:
nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ExtractEmailAttachments.java
---
@@ -131,11 +132,15 @@ public void process(final InputStream rawIn) throws
IOException {
MimeMessageParser parser = new
MimeMessageParser(originalMessage).parse();
// RFC-2822 determines that a message must have a
"From:" header
// if a message lacks the field, it is flagged as
invalid
-Address[] from = originalMessage.getFrom();
+if
(InternetAddress.parseHeader(originalMessage.getHeader("From", ","), false) ==
null) {
+if
(InternetAddress.parseHeader(originalMessage.getHeader("Sender", ","), false)
== null) {
--- End diff --
Do you think falling back to "Sender" if "From" is not present for RFC
validation is desired behavior? According to RFC-2882, and the RFCs that update
it ([RFC-5322](https://tools.ietf.org/html/rfc5322),
[RFC-6854](https://tools.ietf.org/html/rfc6854)), the "Sender" field should
only be present if "From" is also present.
That said, @btwood, have you encountered email servers that use "Sender"
without "From"? If so, it might be worth considering adjusting the
implementation here to reflect what is likely to be encountered in the wild vs.
strict RFC specification.
> ExtractEmailHeaders.java unhandled Exceptions
> -
>
> Key: NIFI-4326
> URL: https://issues.apache.org/jira/browse/NIFI-4326
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.3.0
> Environment: jdk 1.8.0_121-b13
>Reporter: Benjamin Wood
>Priority: Minor
> Fix For: 1.4.0
>
> Original Estimate: 0.5h
> Remaining Estimate: 0.5h
>
> The ExtractEmailHeaders processor throws a NullPointerException if there is
> no TO, CC, and BCC recipients.
> If there are no recipients "originalMessage.getAllRecipients()" returns NULL,
> and not a 0 length array.
> If an address is empty (<> or " ") then getRecipients() will throw an "Empty
> Address" AddressException
> It's possible this is only an issue with Oracle Java.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] nifi issue #2111: NIFI-4326 - ExtractEmailHeaders.java unhandled NullPointer...
Github user kevdoran commented on the issue: https://github.com/apache/nifi/pull/2111 Oh, and to get the permissions for assigning yourself tickets on the NiFi JIRA, [subscribe to the dev mailing list](https://nifi.apache.org/mailing_lists.html) and send the dev list an email requesting that. Someone with sufficient permissions will modify your account. Pretty good NiFi discussions take place on that list. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-4326) ExtractEmailHeaders.java unhandled Exceptions
[ https://issues.apache.org/jira/browse/NIFI-4326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16149678#comment-16149678 ] ASF GitHub Bot commented on NIFI-4326: -- Github user kevdoran commented on the issue: https://github.com/apache/nifi/pull/2111 Oh, and to get the permissions for assigning yourself tickets on the NiFi JIRA, [subscribe to the dev mailing list](https://nifi.apache.org/mailing_lists.html) and send the dev list an email requesting that. Someone with sufficient permissions will modify your account. Pretty good NiFi discussions take place on that list. > ExtractEmailHeaders.java unhandled Exceptions > - > > Key: NIFI-4326 > URL: https://issues.apache.org/jira/browse/NIFI-4326 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.3.0 > Environment: jdk 1.8.0_121-b13 >Reporter: Benjamin Wood >Priority: Minor > Fix For: 1.4.0 > > Original Estimate: 0.5h > Remaining Estimate: 0.5h > > The ExtractEmailHeaders processor throws a NullPointerException if there is > no TO, CC, and BCC recipients. > If there are no recipients "originalMessage.getAllRecipients()" returns NULL, > and not a 0 length array. > If an address is empty (<> or " ") then getRecipients() will throw an "Empty > Address" AddressException > It's possible this is only an issue with Oracle Java. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (NIFI-3248) GetSolr can miss recently updated documents
[
https://issues.apache.org/jira/browse/NIFI-3248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Koji Kawamura reassigned NIFI-3248:
---
Assignee: Johannes Peter (was: Koji Kawamura)
[~jope] Thank you very much! I added you as a contributor. Please let me know
if you find any challenge.
> GetSolr can miss recently updated documents
> ---
>
> Key: NIFI-3248
> URL: https://issues.apache.org/jira/browse/NIFI-3248
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.0.0, 0.5.0, 0.6.0, 0.5.1, 0.7.0, 0.6.1, 1.1.0, 0.7.1,
> 1.0.1
>Reporter: Koji Kawamura
>Assignee: Johannes Peter
> Attachments: nifi-flow.png, query-result-with-curly-bracket.png,
> query-result-with-square-bracket.png
>
>
> GetSolr holds the last query timestamp so that it only fetches documents
> those have been added or updated since the last query.
> However, GetSolr misses some of those updated documents, and once the
> documents date field value becomes older than last query timestamp, the
> document won't be able to be queried by GetSolr any more.
> This JIRA is for tracking the process of investigating this behavior, and
> discussion on them.
> Here are things that can be a cause of this behavior:
> |#|Short description|Should we address it?|
> |1|Timestamp range filter, curly or square bracket?|No|
> |2|Timezone difference between update and query|Additional docs might be
> helpful|
> |3|Lag comes from NearRealTIme nature of Solr|Should be documented at least,
> add 'commit lag-time'?|
> h2. 1. Timestamp range filter, curly or square bracket?
> At the first glance, using curly and square bracket in mix looked strange
> ([source
> code|https://github.com/apache/nifi/blob/support/nifi-0.5.x/nifi-nar-bundles/nifi-solr-bundle/nifi-solr-processors/src/main/java/org/apache/nifi/processors/solr/GetSolr.java#L202]).
> But these difference has a meaning.
> The square bracket on the range query is inclusive and the curly bracket is
> exclusive. If we use inclusive on both sides and a document has a time stamp
> exactly on the boundary then it could be returned in two consecutive
> executions, and we only want it in one.
> This is intentional, and it should be as it is.
> h2. 2. Timezone difference between update and query
> Solr treats date fields as [UTC
> representation|https://cwiki.apache.org/confluence/display/solr/Working+with+Dates|].
> If date field String value of an updated document represents time without
> timezone, and NiFi is running on an environment using timezone other than
> UTC, GetSolr can't perform date range query as users expect.
> Let's say NiFi is running with JST(UTC+9). A process added a document to Solr
> at 15:00 JST. But the date field doesn't have timezone. So, Solr indexed it
> as 15:00 UTC. Then GetSolr performs range query at 15:10 JST, targeting any
> documents updated from 15:00 to 15:10 JST. GetSolr formatted dates using UTC,
> i.e. 6:00 to 6:10 UTC. The updated document won't be matched with the date
> range filter.
> To avoid this, updated documents must have proper timezone in date field
> string representation.
> If one uses NiFi expression language to set current timestamp to that date
> field, following NiFi expression can be used:
> {code}
> ${now():format("-MM-dd'T'HH:mm:ss.SSSZ")}
> {code}
> It will produce a result like:
> {code}
> 2016-12-27T15:30:04.895+0900
> {code}
> Then it will be indexed in Solr with UTC and will be queried by GetSolr as
> expected.
> h2. 3. Lag comes from NearRealTIme nature of Solr
> Solr provides Near Real Time search capability, that means, the recently
> updated documents can be queried in Near Real Time, but it's not real time.
> This latency can be controlled by either on client side which requests the
> update operation by specifying "commitWithin" parameter, or on the Solr
> server side, "autoCommit" and "autoSoftCommit" in
> [solrconfig.xml|https://cwiki.apache.org/confluence/display/solr/UpdateHandlers+in+SolrConfig#UpdateHandlersinSolrConfig-Commits].
> Since commit and updating index can be costly, it's recommended to set this
> interval long enough up to the maximum tolerable latency.
> However, this can be problematic with GetSolr. For instance, as shown in the
> simple NiFi flow below, GetSolr can miss updated documents:
> {code}
> t1: GetSolr queried
> t2: GenerateFlowFile set date = t2
> t3: PutSolrContentStream stored new doc
> t4: GetSolr queried again, from t1 to t4, but the new doc hasn't been indexed
> t5: Solr completed index
> t6: GetSolr queried again, from t4 to t6, the doc didn't match query
> {code}
> This behavior should be at least documented.
> Plus, it would be helpful to add a new configuration property to GetSolr, to
> specify commit lag-time so that GetSolr aims older
[jira] [Commented] (NIFI-4092) ClassCastException Warning during cluster sync
[ https://issues.apache.org/jira/browse/NIFI-4092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16150042#comment-16150042 ] Randy Bovay commented on NIFI-4092: --- [~msclarke] May want to see this. > ClassCastException Warning during cluster sync > -- > > Key: NIFI-4092 > URL: https://issues.apache.org/jira/browse/NIFI-4092 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.3.0 >Reporter: Joseph Gresock > > This is the strack trace I receive, though I'm not sure it affects anything, > since the cluster is eventually able to connect. > 2017-06-20 13:46:44,680 WARN [Reconnect ip-172-31-55-36.ec2.internal:8443] > o.a.n.c.c.node.NodeClusterCoordinator Problem encountered issuing > reconnection request to node ip-172-31-55-36.ec2.internal:8443 > java.io.IOException: > org.apache.nifi.controller.serialization.FlowSerializationException: > java.lang.ClassCastException: > org.apache.nifi.web.api.dto.TemplateDTO$JaxbAccessorM_getDescription_setDescription_java_lang_String > cannot be cast to com.sun.xml.internal.bind.v2.runtime.reflect.Accessor > at > org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.save(StandardXMLFlowConfigurationDAO.java:143) > at > org.apache.nifi.controller.StandardFlowService.createDataFlowFromController(StandardFlowService.java:607) > at > org.apache.nifi.controller.StandardFlowService.createDataFlowFromController(StandardFlowService.java:100) > at > org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator$2.run(NodeClusterCoordinator.java:706) > at java.lang.Thread.run(Thread.java:748) > Caused by: > org.apache.nifi.controller.serialization.FlowSerializationException: > java.lang.ClassCastException: > org.apache.nifi.web.api.dto.TemplateDTO$JaxbAccessorM_getDescription_setDescription_java_lang_String > cannot be cast to com.sun.xml.internal.bind.v2.runtime.reflect.Accessor > at > org.apache.nifi.controller.serialization.StandardFlowSerializer.addTemplate(StandardFlowSerializer.java:546) > at > org.apache.nifi.controller.serialization.StandardFlowSerializer.addProcessGroup(StandardFlowSerializer.java:203) > at > org.apache.nifi.controller.serialization.StandardFlowSerializer.addProcessGroup(StandardFlowSerializer.java:187) > at > org.apache.nifi.controller.serialization.StandardFlowSerializer.addProcessGroup(StandardFlowSerializer.java:187) > at > org.apache.nifi.controller.serialization.StandardFlowSerializer.serialize(StandardFlowSerializer.java:97) > at > org.apache.nifi.controller.FlowController.serialize(FlowController.java:1544) > at > org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.save(StandardXMLFlowConfigurationDAO.java:141) > ... 4 common frames omitted > Caused by: java.lang.ClassCastException: > org.apache.nifi.web.api.dto.TemplateDTO$JaxbAccessorM_getDescription_setDescription_java_lang_String > cannot be cast to com.sun.xml.internal.bind.v2.runtime.reflect.Accessor > at > com.sun.xml.internal.bind.v2.runtime.reflect.opt.OptimizedAccessorFactory.instanciate(OptimizedAccessorFactory.java:190) > at > com.sun.xml.internal.bind.v2.runtime.reflect.opt.OptimizedAccessorFactory.get(OptimizedAccessorFactory.java:129) > at > com.sun.xml.internal.bind.v2.runtime.reflect.Accessor$GetterSetterReflection.optimize(Accessor.java:388) > at > com.sun.xml.internal.bind.v2.runtime.property.SingleElementLeafProperty.(SingleElementLeafProperty.java:77) > at sun.reflect.GeneratedConstructorAccessor435.newInstance(Unknown > Source) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > com.sun.xml.internal.bind.v2.runtime.property.PropertyFactory.create(PropertyFactory.java:113) > at > com.sun.xml.internal.bind.v2.runtime.ClassBeanInfoImpl.(ClassBeanInfoImpl.java:166) > at > com.sun.xml.internal.bind.v2.runtime.JAXBContextImpl.getOrCreate(JAXBContextImpl.java:488) > at > com.sun.xml.internal.bind.v2.runtime.JAXBContextImpl.(JAXBContextImpl.java:305) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (NIFI-4092) ClassCastException Warning during cluster sync
[ https://issues.apache.org/jira/browse/NIFI-4092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16150041#comment-16150041 ] Randy Bovay commented on NIFI-4092: --- I'm having the same issue now, and it's declustering my cluster often. Makes the UI unusable, and if we make changes forces us to have to rebuild the node that goes out of sync. > ClassCastException Warning during cluster sync > -- > > Key: NIFI-4092 > URL: https://issues.apache.org/jira/browse/NIFI-4092 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.3.0 >Reporter: Joseph Gresock > > This is the strack trace I receive, though I'm not sure it affects anything, > since the cluster is eventually able to connect. > 2017-06-20 13:46:44,680 WARN [Reconnect ip-172-31-55-36.ec2.internal:8443] > o.a.n.c.c.node.NodeClusterCoordinator Problem encountered issuing > reconnection request to node ip-172-31-55-36.ec2.internal:8443 > java.io.IOException: > org.apache.nifi.controller.serialization.FlowSerializationException: > java.lang.ClassCastException: > org.apache.nifi.web.api.dto.TemplateDTO$JaxbAccessorM_getDescription_setDescription_java_lang_String > cannot be cast to com.sun.xml.internal.bind.v2.runtime.reflect.Accessor > at > org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.save(StandardXMLFlowConfigurationDAO.java:143) > at > org.apache.nifi.controller.StandardFlowService.createDataFlowFromController(StandardFlowService.java:607) > at > org.apache.nifi.controller.StandardFlowService.createDataFlowFromController(StandardFlowService.java:100) > at > org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator$2.run(NodeClusterCoordinator.java:706) > at java.lang.Thread.run(Thread.java:748) > Caused by: > org.apache.nifi.controller.serialization.FlowSerializationException: > java.lang.ClassCastException: > org.apache.nifi.web.api.dto.TemplateDTO$JaxbAccessorM_getDescription_setDescription_java_lang_String > cannot be cast to com.sun.xml.internal.bind.v2.runtime.reflect.Accessor > at > org.apache.nifi.controller.serialization.StandardFlowSerializer.addTemplate(StandardFlowSerializer.java:546) > at > org.apache.nifi.controller.serialization.StandardFlowSerializer.addProcessGroup(StandardFlowSerializer.java:203) > at > org.apache.nifi.controller.serialization.StandardFlowSerializer.addProcessGroup(StandardFlowSerializer.java:187) > at > org.apache.nifi.controller.serialization.StandardFlowSerializer.addProcessGroup(StandardFlowSerializer.java:187) > at > org.apache.nifi.controller.serialization.StandardFlowSerializer.serialize(StandardFlowSerializer.java:97) > at > org.apache.nifi.controller.FlowController.serialize(FlowController.java:1544) > at > org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.save(StandardXMLFlowConfigurationDAO.java:141) > ... 4 common frames omitted > Caused by: java.lang.ClassCastException: > org.apache.nifi.web.api.dto.TemplateDTO$JaxbAccessorM_getDescription_setDescription_java_lang_String > cannot be cast to com.sun.xml.internal.bind.v2.runtime.reflect.Accessor > at > com.sun.xml.internal.bind.v2.runtime.reflect.opt.OptimizedAccessorFactory.instanciate(OptimizedAccessorFactory.java:190) > at > com.sun.xml.internal.bind.v2.runtime.reflect.opt.OptimizedAccessorFactory.get(OptimizedAccessorFactory.java:129) > at > com.sun.xml.internal.bind.v2.runtime.reflect.Accessor$GetterSetterReflection.optimize(Accessor.java:388) > at > com.sun.xml.internal.bind.v2.runtime.property.SingleElementLeafProperty.(SingleElementLeafProperty.java:77) > at sun.reflect.GeneratedConstructorAccessor435.newInstance(Unknown > Source) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > com.sun.xml.internal.bind.v2.runtime.property.PropertyFactory.create(PropertyFactory.java:113) > at > com.sun.xml.internal.bind.v2.runtime.ClassBeanInfoImpl.(ClassBeanInfoImpl.java:166) > at > com.sun.xml.internal.bind.v2.runtime.JAXBContextImpl.getOrCreate(JAXBContextImpl.java:488) > at > com.sun.xml.internal.bind.v2.runtime.JAXBContextImpl.(JAXBContextImpl.java:305) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
