[jira] [Commented] (NIFI-1594) Add option to bulk using Index or Update to PutElasticsearch
[
https://issues.apache.org/jira/browse/NIFI-1594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15181420#comment-15181420
]
ASF GitHub Bot commented on NIFI-1594:
--
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/255#discussion_r55112614
--- Diff:
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearch.java
---
@@ -99,6 +99,14 @@
AttributeExpression.ResultType.STRING, true))
.build();
+public static final PropertyDescriptor INDEX_OP = new
PropertyDescriptor.Builder()
+.name("Index Operation")
+.description("The type of the operation used to index (index,
update, upsert)")
--- End diff --
The description mentions three modes but only two are allowed. Should they
be "insert" and "update/upsert", or is "index" the preferred terminology for
adding a new document?
> Add option to bulk using Index or Update to PutElasticsearch
>
>
> Key: NIFI-1594
> URL: https://issues.apache.org/jira/browse/NIFI-1594
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: João Henrique Ferreira de Freitas
>Priority: Minor
>
> I have a use case where two flowfiles needs to be write using
> PutElasticsearch. Both will write to the same document but in different
> properties.
> The proposal is to let the user choice if an Update operation or Index is
> needed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-1563) Remove storage of components' stats and bulletins from NCM
[ https://issues.apache.org/jira/browse/NIFI-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15180589#comment-15180589 ] ASF subversion and git services commented on NIFI-1563: --- Commit 8f187274afdbfa08538c3b867aad664716bc8a44 in nifi's branch refs/heads/NIFI-1563 from [~mcgilman] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=8f18727 ] NIFI-1563: - Updating UI to support restructured status history DTO. > Remove storage of components' stats and bulletins from NCM > -- > > Key: NIFI-1563 > URL: https://issues.apache.org/jira/browse/NIFI-1563 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.0.0 > > Original Estimate: 168h > Remaining Estimate: 168h > > Currently, each node in a cluster sends a period heartbeat that contains the > stats for all components in the node. This happens every 5 seconds by > default. This results in quite a lot of chatter between the NCM and nodes. It > made sense to take this approach when the clustering concept was designed > because there were no Process Groups, and we had no notion of merging > responses from nodes for a web request. > However, now we should replicate the request to all nodes and then merge the > responses on demand, rather than storing this information on the NCM. This > requires far less bandwidth because we only need to pull the stats for a > particular Process Group and only on demand instead of every 5 seconds. > Additionally, this is laying groundwork for the Zero-Master clustering that > we want to have in place for 1.0.0. > In order to remove the stats from the NCM, we will also need to remove the > bulletins and stats history. These requests will all need to be federated and > the responses merged on-demand. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
nifi git commit: NIFI-1563: - Updating UI to support restructured status history DTO.
Repository: nifi
Updated Branches:
refs/heads/NIFI-1563 ad32cb824 -> 8f187274a
NIFI-1563:
- Updating UI to support restructured status history DTO.
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/8f187274
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/8f187274
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/8f187274
Branch: refs/heads/NIFI-1563
Commit: 8f187274afdbfa08538c3b867aad664716bc8a44
Parents: ad32cb8
Author: Matt Gilman
Authored: Fri Mar 4 16:42:29 2016 -0500
Committer: Matt Gilman
Committed: Fri Mar 4 16:42:29 2016 -0500
--
.../api/dto/status/NodeStatusSnapshotsDTO.java | 85
.../web/api/dto/status/StatusHistoryDTO.java| 22 +-
.../cluster/manager/impl/WebClusterManager.java | 21 +-
.../src/main/webapp/js/nf/canvas/nf-actions.js | 28 +-
.../src/main/webapp/js/nf/nf-status-history.js | 409 ++-
.../webapp/js/nf/summary/nf-summary-table.js| 40 +-
6 files changed, 239 insertions(+), 366 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/8f187274/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/status/NodeStatusSnapshotsDTO.java
--
diff --git
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/status/NodeStatusSnapshotsDTO.java
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/status/NodeStatusSnapshotsDTO.java
new file mode 100644
index 000..f70da4a
--- /dev/null
+++
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/status/NodeStatusSnapshotsDTO.java
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.web.api.dto.status;
+
+import com.wordnik.swagger.annotations.ApiModelProperty;
+
+import javax.xml.bind.annotation.XmlType;
+import java.util.List;
+
+/**
+ * DTO for serializing the status history of a single component across the
cluster.
+ */
+@XmlType(name = "nodeStatusSnapshots")
+public class NodeStatusSnapshotsDTO {
+
+private String nodeId;
+private String address;
+private Integer apiPort;
+private List statusSnapshots;
+
+/**
+ * @return node's host/IP address
+ */
+@ApiModelProperty(
+value = "The node's host/ip address."
+)
+public String getAddress() {
+return address;
+}
+
+public void setAddress(String address) {
+this.address = address;
+}
+
+/**
+ * @return node ID
+ */
+@ApiModelProperty(
+value = "The id of the node."
+)
+public String getNodeId() {
+return nodeId;
+}
+
+public void setNodeId(String nodeId) {
+this.nodeId = nodeId;
+}
+
+/**
+ * @return port the node is listening for API requests
+ */
+@ApiModelProperty(
+value = "The port the node is listening for API requests."
+)
+public Integer getApiPort() {
+return apiPort;
+}
+
+public void setApiPort(Integer port) {
+this.apiPort = port;
+}
+
+@ApiModelProperty("A list of StatusSnapshotDTO objects that provide the
actual metric values for the component for this node.")
+public List getStatusSnapshots() {
+return statusSnapshots;
+}
+
+public void setStatusSnapshots(List statusSnapshots) {
+this.statusSnapshots = statusSnapshots;
+}
+}
http://git-wip-us.apache.org/repos/asf/nifi/blob/8f187274/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/status/StatusHistoryDTO.java
--
diff --git
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/status/StatusHistoryDTO.java
b/nifi-nar-bundles/nifi-framework-bundle/nifi-
[jira] [Commented] (NIFI-1594) Add option to bulk using Index or Update to PutElasticsearch
[ https://issues.apache.org/jira/browse/NIFI-1594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15180555#comment-15180555 ] Matt Burgess commented on NIFI-1594: Added link to related case for ES improvements > Add option to bulk using Index or Update to PutElasticsearch > > > Key: NIFI-1594 > URL: https://issues.apache.org/jira/browse/NIFI-1594 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: João Henrique Ferreira de Freitas >Priority: Minor > > I have a use case where two flowfiles needs to be write using > PutElasticsearch. Both will write to the same document but in different > properties. > The proposal is to let the user choice if an Update operation or Index is > needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-1587) ListHDFS processor does not properly clear state
[ https://issues.apache.org/jira/browse/NIFI-1587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-1587: - Fix Version/s: 0.6.0 > ListHDFS processor does not properly clear state > > > Key: NIFI-1587 > URL: https://issues.apache.org/jira/browse/NIFI-1587 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 0.4.1 > Environment: latest 0.5.1 build >Reporter: Matthew Clarke > Fix For: 0.6.0 > > Attachments: NIFI-1587.patch > > > The ListHDFS processor successfully retains state in zookeeper; however, > state is not properly cleared. You can right click, select "view state" , > and then "clear state"; however, regardless of what the state window shows > state is still retained in zookeeper. If a new file that was not previously > seen before is listed by the ListHDFS processor, all previous state is > repopulated in the view state window. Files will not be picked up again > unless the ListHDFS processor is replaced by a new one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-1587) ListHDFS processor does not properly clear state
[ https://issues.apache.org/jira/browse/NIFI-1587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-1587: - Attachment: NIFI-1587.patch > ListHDFS processor does not properly clear state > > > Key: NIFI-1587 > URL: https://issues.apache.org/jira/browse/NIFI-1587 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 0.4.1 > Environment: latest 0.5.1 build >Reporter: Matthew Clarke > Fix For: 0.6.0 > > Attachments: NIFI-1587.patch > > > The ListHDFS processor successfully retains state in zookeeper; however, > state is not properly cleared. You can right click, select "view state" , > and then "clear state"; however, regardless of what the state window shows > state is still retained in zookeeper. If a new file that was not previously > seen before is listed by the ListHDFS processor, all previous state is > repopulated in the view state window. Files will not be picked up again > unless the ListHDFS processor is replaced by a new one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1594) Add option to bulk using Index or Update to PutElasticsearch
[ https://issues.apache.org/jira/browse/NIFI-1594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15180537#comment-15180537 ] ASF GitHub Bot commented on NIFI-1594: -- GitHub user joaohf opened a pull request: https://github.com/apache/nifi/pull/255 NIFI-1594: Add option to bulk using Index or Update. Signed-off-by: João Henrique Ferreira de Freitas You can merge this pull request into a Git repository by running: $ git pull https://github.com/joaohf/nifi NIFI-1594 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/255.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #255 commit 84597d253c61ed4f23c3078607da5f05145187c0 Author: João Henrique Ferreira de Freitas Date: 2016-02-24T17:44:34Z NIFI-1594: Add option to bulk using Index or Update. Signed-off-by: João Henrique Ferreira de Freitas > Add option to bulk using Index or Update to PutElasticsearch > > > Key: NIFI-1594 > URL: https://issues.apache.org/jira/browse/NIFI-1594 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: João Henrique Ferreira de Freitas >Priority: Minor > > I have a use case where two flowfiles needs to be write using > PutElasticsearch. Both will write to the same document but in different > properties. > The proposal is to let the user choice if an Update operation or Index is > needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1420) Splunk Processors
[ https://issues.apache.org/jira/browse/NIFI-1420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15180536#comment-15180536 ] ASF GitHub Bot commented on NIFI-1420: -- Github user bbende commented on the pull request: https://github.com/apache/nifi/pull/233#issuecomment-192466857 @JPercivall Pushed up another commit that addresses the additional comments from today. Part of this change I decided to go the route that @trixpan suggested and change ListenSplunkForwarder to ListenTCP, and as a result moved it to the standard bundle. This will open it up to a lot more use cases and it wasn't really Splunk specific. As a result I decided to take out the mime.type attribute since it is writing bytes to FlowFiles and may not really be text/plain all the time. Let me know if anything else needs updating or was left out. > Splunk Processors > - > > Key: NIFI-1420 > URL: https://issues.apache.org/jira/browse/NIFI-1420 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Bryan Bende >Assignee: Bryan Bende >Priority: Minor > Fix For: 0.6.0 > > > To continue improving NiFi's ability to collect logs, a good integration > point would be to have a processor that could listen for data from a Splunk > forwarder (https://docs.splunk.com/Splexicon:Universalforwarder). Being able > to push log messages to Splunk would also be useful. > Splunk provides an SDK that may be helpful: > https://github.com/splunk/splunk-sdk-java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-1587) ListHDFS processor does not properly clear state
[ https://issues.apache.org/jira/browse/NIFI-1587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-1587: - Description: The ListHDFS processor successfully retains state in zookeeper; however, state is not properly cleared. You can right click, select "view state" , and then "clear state"; however, regardless of what the state window shows state is still retained in zookeeper. If a new file that was not previously seen before is listed by the ListHDFS processor, all previous state is repopulated in the view state window. Files will not be picked up again unless the ListHDFS processor is replaced by a new one. (was: The ListHDFS processor successfully retains state in zookeeper; however, state is not properly cleared. You can right click, select "view state" , and then "clear state"; however, regardless of what the state window shows state is still retained in zookeeper. If a new file that was not previously seen before is listed by the ListHDFS processor, all previous state is repopulated in the view state window. Files will not be picked up again unless the listHDFS processor is replaced by a new one.) > ListHDFS processor does not properly clear state > > > Key: NIFI-1587 > URL: https://issues.apache.org/jira/browse/NIFI-1587 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 0.4.1 > Environment: latest 0.5.1 build >Reporter: Matthew Clarke > > The ListHDFS processor successfully retains state in zookeeper; however, > state is not properly cleared. You can right click, select "view state" , > and then "clear state"; however, regardless of what the state window shows > state is still retained in zookeeper. If a new file that was not previously > seen before is listed by the ListHDFS processor, all previous state is > repopulated in the view state window. Files will not be picked up again > unless the ListHDFS processor is replaced by a new one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-1587) ListHDFS processor does not properly clear state
[ https://issues.apache.org/jira/browse/NIFI-1587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-1587: - Summary: ListHDFS processor does not properly clear state (was: listHDFS processor does not properly clear state) > ListHDFS processor does not properly clear state > > > Key: NIFI-1587 > URL: https://issues.apache.org/jira/browse/NIFI-1587 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 0.4.1 > Environment: latest 0.5.1 build >Reporter: Matthew Clarke > > The ListHDFS processor successfully retains state in zookeeper; however, > state is not properly cleared. You can right click, select "view state" , > and then "clear state"; however, regardless of what the state window shows > state is still retained in zookeeper. If a new file that was not previously > seen before is listed by the ListHDFS processor, all previous state is > repopulated in the view state window. Files will not be picked up again > unless the listHDFS processor is replaced by a new one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-1594) Add option to bulk using Index or Update to PutElasticsearch
João Henrique Ferreira de Freitas created NIFI-1594: --- Summary: Add option to bulk using Index or Update to PutElasticsearch Key: NIFI-1594 URL: https://issues.apache.org/jira/browse/NIFI-1594 Project: Apache NiFi Issue Type: Improvement Components: Extensions Reporter: João Henrique Ferreira de Freitas Priority: Minor I have a use case where two flowfiles needs to be write using PutElasticsearch. Both will write to the same document but in different properties. The proposal is to let the user choice if an Update operation or Index is needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-1584) FetchHDFS should support completion Strategy
[ https://issues.apache.org/jira/browse/NIFI-1584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Payne updated NIFI-1584: - Issue Type: Improvement (was: Bug) > FetchHDFS should support completion Strategy > > > Key: NIFI-1584 > URL: https://issues.apache.org/jira/browse/NIFI-1584 > Project: Apache NiFi > Issue Type: Improvement > Components: Configuration, Core Framework >Affects Versions: 0.4.1 >Reporter: Matthew Clarke >Priority: Minor > > FetchHDFS should allow users to specifies what to do with the original > file in the HDFS directory once it has been pulled into NiFi. If the > Completion Strategy fails, a warning will be logged but the data will still > be transferred. > Using GetHDFS to delete source files is not a good option within a Clustered > NiFi. > Completion Strategy: > None Leave the file as-is > Move File Move the file to the directory specified by the Destination Directory> property > Delete FileDeletes the original file from the remote system -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-1593) New Processor to make use of a template engine
Uwe Geercken created NIFI-1593: -- Summary: New Processor to make use of a template engine Key: NIFI-1593 URL: https://issues.apache.org/jira/browse/NIFI-1593 Project: Apache NiFi Issue Type: New Feature Components: Core UI Affects Versions: 0.4.1 Environment: all Reporter: Uwe Geercken Priority: Minor Add a new processor that allows to use the functionality of a template engine such as Apache Velocity. The user would provide either the template code or reference a template file. In the process the flowfile content/attributes would be merged with the template and thus transforming the content. A use case would be: Get a Json file >> EvaluateJsonPath >> transform the attributes of the Json file to a different format using the given template code or template file (using the template engine) >> conversion into any other text based format such as xml, html, text. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-840) Create ListS3 processor
[
https://issues.apache.org/jira/browse/NIFI-840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15180369#comment-15180369
]
Ravikumar Visweswara commented on NIFI-840:
---
In ListS3, It will be very helpful to allow expressions on prefix. In a shared
bucket, usually files may be uploaded with a date directory or a date prefix.
Code:
String prefix =
context.getProperty(PREFIX).evaluateAttributeExpressions().getValue();
Example use:
daily-feed-${now():toNumber():minus(8640):format('MMdd')}
> Create ListS3 processor
> ---
>
> Key: NIFI-840
> URL: https://issues.apache.org/jira/browse/NIFI-840
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: Aldrin Piri
>
> A processor is needed that can provide an S3 listing to use in conjunction
> with FetchS3Object. This is to provide a similar user experience as with the
> HDFS processors that perform List/Get.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-1582) New processor to update attributes with state
[ https://issues.apache.org/jira/browse/NIFI-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15180343#comment-15180343 ] Uwe Geercken commented on NIFI-1582: Over or below threshold is a good example. But then maybe another indicator would be the deviation from a standard value. Example: Maybe you know that a certain value sometimes goes over the threshold and then returns to a normal value and that is fine for you. But maybe you want to know when a certain value (or maybe an average) deviates by a certain amount or percentage from the regular value. > New processor to update attributes with state > - > > Key: NIFI-1582 > URL: https://issues.apache.org/jira/browse/NIFI-1582 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Joseph Percivall > > This idea was sparked by a thread on the user list and should allow basic > data science: > I expect that in the future I’ll need something a little more sophisticated > but for now my problem is very simple: > I want to be able to trigger an alert (only once) when an attribute in an > incoming stream, for instance, goes over a predefined threshold. The > Processor should then trigger (only once again) another trigger when the > signal goes back to normal (below threshold). Basically a RouteByAttribute > but with memory. > Thanks > Claudio > > Hello Claudio, > Your use-case actually could leverage a couple of recently added features to > create a really cool open-source processor. The two key features that were > added are State Management and the ability to reference processor specific > variables in expression language. You can take a look at RouteText to see > both in action. > By utilizing both you can create a processor that is configured with multiple > Expression language expressions. There would be dynamic properties which > would accept expression language and then store the evaluated value via state > management. Then there would be a routing property (that supports expression > language) that could simply add an attribute to the flowfile with the > evaluated value which would allow it to be used by flowing processors for > routing. > This would allow you to do your use-case where you store the value for the > incoming stream and route differently once you go over a threshold. It could > even allow more complex use-cases. One instance, I believe, would be possible > is to have a running average and standard deviation and route data to > different locations based on it's standard deviation. > You can think of this like an UpdateAttribute with the ability to store and > calculate variables using expression language. > Joe -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-1592) Add keyboard shortcuts to Web Interface
Uwe Geercken created NIFI-1592: -- Summary: Add keyboard shortcuts to Web Interface Key: NIFI-1592 URL: https://issues.apache.org/jira/browse/NIFI-1592 Project: Apache NiFi Issue Type: Improvement Components: Core UI Affects Versions: 0.4.1 Environment: all Reporter: Uwe Geercken Priority: Minor A regular action to do is e.g. to add a processor to the flow by dragging it on the canvas. Please add a configurable keyboard shortcut that does the same. So the user would simply press a define key or combination and the relevant action is executed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-1591) Show errors in processor config dialog
Uwe Geercken created NIFI-1591: -- Summary: Show errors in processor config dialog Key: NIFI-1591 URL: https://issues.apache.org/jira/browse/NIFI-1591 Project: Apache NiFi Issue Type: Improvement Components: Core UI Affects Versions: 0.4.1 Environment: all Reporter: Uwe Geercken Priority: Minor When there is an error with a certain processor, then it is shown in the upper left hand corner. Hovering with the mouse over it will reveal the error text. But once you open the config dialog there is no way to see the error message again - other than closing the dialog and hovering over the icon again. This is not user friendly. The improvement request is to add functionality to read the error message also when the config dialog is open. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-1590) No error message if nifi starts with port conflict
[ https://issues.apache.org/jira/browse/NIFI-1590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Randy Gelhausen updated NIFI-1590: -- Description: I have another webapp running on port 8080. On starting NiFi with default configs (web port 8080), I don't see any unusual behavior except that "nifi.sh run" immediately exits: HW11401:nifi-0.5.1.1.1.2.0-32 randy$ ./bin/nifi.sh run Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_65.jdk/Contents/Home/ NiFi home: /Users/randy/nifi-0.5.1.1.1.2.0-32 Bootstrap Config File: /Users/randy/nifi-0.5.1.1.1.2.0-32/conf/bootstrap.conf 2016-03-04 13:21:04,345 INFO [main] org.apache.nifi.bootstrap.Command Starting Apache NiFi... 2016-03-04 13:21:04,346 INFO [main] org.apache.nifi.bootstrap.Command Working Directory: /Users/randy/nifi-0.5.1.1.1.2.0-32 2016-03-04 13:21:04,346 INFO [main] org.apache.nifi.bootstrap.Command Command: /Library/Java/JavaVirtualMachines/jdk1.8.0_65.jdk/Contents/Home/bin/java -classpath /Users/randy/nifi-0.5.1.1.1.2.0-32/./conf:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/jcl-over-slf4j-1.7.12.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/jul-to-slf4j-1.7.12.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/log4j-over-slf4j-1.7.12.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/logback-classic-1.1.3.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/logback-core-1.1.3.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/nifi-api-0.5.1.1.1.2.0-32.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/nifi-documentation-0.5.1.1.1.2.0-32.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/nifi-nar-utils-0.5.1.1.1.2.0-32.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/nifi-properties-0.5.1.1.1.2.0-32.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/nifi-runtime-0.5.1.1.1.2.0-32.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/slf4j-api-1.7.12.jar -Dorg.apache.jasper.compiler.disablejsr199=true -Xmx512m -Xms512m -Dsun.net.http.allowRestrictedHeaders=true -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Djava.protocol.handler.pkgs=sun.net.www.protocol -Dnifi.properties.file.path=/Users/randy/nifi-0.5.1.1.1.2.0-32/./conf/nifi.properties -Dnifi.bootstrap.listen.port=54630 -Dapp=NiFi org.apache.nifi.NiFi HW11401:nifi-0.5.1.1.1.2.0-32 randy$ ps aux | grep nifi randy 98845 0.0 0.0 2432772672 s001 S+1:21PM 0:00.00 grep nifi was: I have another webapp running on port 8080. On starting NiFi, I don't see any unusual behavior except that "nifi.sh run" immediately exits: HW11401:nifi-0.5.1.1.1.2.0-32 randy$ ./bin/nifi.sh run Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_65.jdk/Contents/Home/ NiFi home: /Users/randy/nifi-0.5.1.1.1.2.0-32 Bootstrap Config File: /Users/randy/nifi-0.5.1.1.1.2.0-32/conf/bootstrap.conf 2016-03-04 13:21:04,345 INFO [main] org.apache.nifi.bootstrap.Command Starting Apache NiFi... 2016-03-04 13:21:04,346 INFO [main] org.apache.nifi.bootstrap.Command Working Directory: /Users/randy/nifi-0.5.1.1.1.2.0-32 2016-03-04 13:21:04,346 INFO [main] org.apache.nifi.bootstrap.Command Command: /Library/Java/JavaVirtualMachines/jdk1.8.0_65.jdk/Contents/Home/bin/java -classpath /Users/randy/nifi-0.5.1.1.1.2.0-32/./conf:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/jcl-over-slf4j-1.7.12.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/jul-to-slf4j-1.7.12.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/log4j-over-slf4j-1.7.12.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/logback-classic-1.1.3.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/logback-core-1.1.3.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/nifi-api-0.5.1.1.1.2.0-32.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/nifi-documentation-0.5.1.1.1.2.0-32.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/nifi-nar-utils-0.5.1.1.1.2.0-32.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/nifi-properties-0.5.1.1.1.2.0-32.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/nifi-runtime-0.5.1.1.1.2.0-32.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/slf4j-api-1.7.12.jar -Dorg.apache.jasper.compiler.disablejsr199=true -Xmx512m -Xms512m -Dsun.net.http.allowRestrictedHeaders=true -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Djava.protocol.handler.pkgs=sun.net.www.protocol -Dnifi.properties.file.path=/Users/randy/nifi-0.5.1.1.1.2.0-32/./conf/nifi.properties -Dnifi.bootstrap.listen.port=54630 -Dapp=NiFi org.apache.nifi.NiFi HW11401:nifi-0.5.1.1.1.2.0-32 randy$ ps aux | grep nifi randy 98845 0.0 0.0 2432772672 s001 S+1:21PM 0:00.00 grep nifi > No error message if nifi starts with port conflict > -- > > Key: NIFI-1590 > URL: https://issues.apache.org/jira/browse/NIFI-1590 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Randy Gelhausen > > I have another webapp running on port 8080. > On starting NiFi with default configs (web port 8080), I don't see any > unusual be
[jira] [Created] (NIFI-1590) No error message if nifi starts with port conflict
Randy Gelhausen created NIFI-1590: - Summary: No error message if nifi starts with port conflict Key: NIFI-1590 URL: https://issues.apache.org/jira/browse/NIFI-1590 Project: Apache NiFi Issue Type: Bug Components: Core Framework Reporter: Randy Gelhausen I have another webapp running on port 8080. On starting NiFi, I don't see any unusual behavior except that "nifi.sh run" immediately exits: HW11401:nifi-0.5.1.1.1.2.0-32 randy$ ./bin/nifi.sh run Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_65.jdk/Contents/Home/ NiFi home: /Users/randy/nifi-0.5.1.1.1.2.0-32 Bootstrap Config File: /Users/randy/nifi-0.5.1.1.1.2.0-32/conf/bootstrap.conf 2016-03-04 13:21:04,345 INFO [main] org.apache.nifi.bootstrap.Command Starting Apache NiFi... 2016-03-04 13:21:04,346 INFO [main] org.apache.nifi.bootstrap.Command Working Directory: /Users/randy/nifi-0.5.1.1.1.2.0-32 2016-03-04 13:21:04,346 INFO [main] org.apache.nifi.bootstrap.Command Command: /Library/Java/JavaVirtualMachines/jdk1.8.0_65.jdk/Contents/Home/bin/java -classpath /Users/randy/nifi-0.5.1.1.1.2.0-32/./conf:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/jcl-over-slf4j-1.7.12.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/jul-to-slf4j-1.7.12.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/log4j-over-slf4j-1.7.12.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/logback-classic-1.1.3.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/logback-core-1.1.3.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/nifi-api-0.5.1.1.1.2.0-32.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/nifi-documentation-0.5.1.1.1.2.0-32.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/nifi-nar-utils-0.5.1.1.1.2.0-32.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/nifi-properties-0.5.1.1.1.2.0-32.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/nifi-runtime-0.5.1.1.1.2.0-32.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/slf4j-api-1.7.12.jar -Dorg.apache.jasper.compiler.disablejsr199=true -Xmx512m -Xms512m -Dsun.net.http.allowRestrictedHeaders=true -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Djava.protocol.handler.pkgs=sun.net.www.protocol -Dnifi.properties.file.path=/Users/randy/nifi-0.5.1.1.1.2.0-32/./conf/nifi.properties -Dnifi.bootstrap.listen.port=54630 -Dapp=NiFi org.apache.nifi.NiFi HW11401:nifi-0.5.1.1.1.2.0-32 randy$ ps aux | grep nifi randy 98845 0.0 0.0 2432772672 s001 S+1:21PM 0:00.00 grep nifi -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1420) Splunk Processors
[ https://issues.apache.org/jira/browse/NIFI-1420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15180025#comment-15180025 ] ASF GitHub Bot commented on NIFI-1420: -- Github user JPercivall commented on the pull request: https://github.com/apache/nifi/pull/233#issuecomment-192325813 Should have made this comment on the first commit but can the LogGenerator be put into a util package on the same path? It makes it more readable to only have the test classes in package "org.apache.nifi.processors.splunk". > Splunk Processors > - > > Key: NIFI-1420 > URL: https://issues.apache.org/jira/browse/NIFI-1420 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Bryan Bende >Assignee: Bryan Bende >Priority: Minor > Fix For: 0.6.0 > > > To continue improving NiFi's ability to collect logs, a good integration > point would be to have a processor that could listen for data from a Splunk > forwarder (https://docs.splunk.com/Splexicon:Universalforwarder). Being able > to push log messages to Splunk would also be useful. > Splunk provides an SDK that may be helpful: > https://github.com/splunk/splunk-sdk-java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1579) Improve ListenSyslog Performance
[ https://issues.apache.org/jira/browse/NIFI-1579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15179933#comment-15179933 ] ASF subversion and git services commented on NIFI-1579: --- Commit 19e53962ca45fd00a46efd671b674d388ff10053 in nifi's branch refs/heads/master from [~bbende] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=19e5396 ] NIFI-1579 Performance improvements for ListenSyslog which include removing an unnecessary yield and exposing a configurable size for the internal queue used by the processor, changing ListenSyslog to use a 20ms poll and use a long poll when batching, also including same improvements for ListenRELP > Improve ListenSyslog Performance > > > Key: NIFI-1579 > URL: https://issues.apache.org/jira/browse/NIFI-1579 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Bryan Bende >Assignee: Bryan Bende > Fix For: 0.6.0 > > Attachments: NIFI-1579-ListenSyslog_Performance-Part2.patch, > NIFI-1579-ListenSyslog_Performance.patch > > > While testing ListenSyslog at various data rates it was observed that a > significant amount of packets were dropped when using UDP. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
nifi git commit: NIFI-1579 Performance improvements for ListenSyslog which include removing an unnecessary yield and exposing a configurable size for the internal queue used by the processor, changing
Repository: nifi
Updated Branches:
refs/heads/master 6776060ac -> 19e53962c
NIFI-1579 Performance improvements for ListenSyslog which include removing an
unnecessary yield and exposing a configurable size for the internal queue used
by the processor, changing ListenSyslog to use a 20ms poll and use a long poll
when batching, also including same improvements for ListenRELP
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/19e53962
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/19e53962
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/19e53962
Branch: refs/heads/master
Commit: 19e53962ca45fd00a46efd671b674d388ff10053
Parents: 6776060
Author: Bryan Bende
Authored: Tue Mar 1 12:54:52 2016 -0500
Committer: Bryan Bende
Committed: Fri Mar 4 09:17:45 2016 -0500
--
.../listen/AbstractListenEventProcessor.java| 25 ++
.../nifi/processors/standard/ListenRELP.java| 6 ++--
.../nifi/processors/standard/ListenSyslog.java | 34 +++-
.../processors/standard/TestListenSyslog.java | 5 +--
4 files changed, 52 insertions(+), 18 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/19e53962/nifi-commons/nifi-processor-utilities/src/main/java/org/apache/nifi/processor/util/listen/AbstractListenEventProcessor.java
--
diff --git
a/nifi-commons/nifi-processor-utilities/src/main/java/org/apache/nifi/processor/util/listen/AbstractListenEventProcessor.java
b/nifi-commons/nifi-processor-utilities/src/main/java/org/apache/nifi/processor/util/listen/AbstractListenEventProcessor.java
index 029f6db..d56255d 100644
---
a/nifi-commons/nifi-processor-utilities/src/main/java/org/apache/nifi/processor/util/listen/AbstractListenEventProcessor.java
+++
b/nifi-commons/nifi-processor-utilities/src/main/java/org/apache/nifi/processor/util/listen/AbstractListenEventProcessor.java
@@ -85,6 +85,15 @@ public abstract class AbstractListenEventProcessor extends Abst
.defaultValue("1 MB")
.required(true)
.build();
+public static final PropertyDescriptor MAX_MESSAGE_QUEUE_SIZE = new
PropertyDescriptor.Builder()
+.name("Max Size of Message Queue")
+.description("The maximum size of the internal queue used to
buffer messages being transferred from the underlying channel to the processor.
" +
+"Setting this value higher allows more messages to be
buffered in memory during surges of incoming messages, but increases the total
" +
+"memory used by the processor.")
+.addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
+.defaultValue("1")
+.required(true)
+.build();
// Putting these properties here so sub-classes don't have to redefine
them, but they are
// not added to the properties by default since not all processors may
need them
@@ -119,7 +128,7 @@ public abstract class AbstractListenEventProcessor extends Abst
.description("Messages received successfully will be sent out this
relationship.")
.build();
-public static final int POLL_TIMEOUT_MS = 100;
+public static final int POLL_TIMEOUT_MS = 20;
private Set relationships;
private List descriptors;
@@ -127,7 +136,7 @@ public abstract class AbstractListenEventProcessor extends Abst
protected volatile int port;
protected volatile Charset charset;
protected volatile ChannelDispatcher dispatcher;
-protected volatile BlockingQueue events = new LinkedBlockingQueue<>(10);
+protected volatile BlockingQueue events;
protected volatile BlockingQueue errorEvents = new
LinkedBlockingQueue<>();
@Override
@@ -135,6 +144,7 @@ public abstract class AbstractListenEventProcessor extends Abst
final List descriptors = new ArrayList<>();
descriptors.add(PORT);
descriptors.add(RECV_BUFFER_SIZE);
+descriptors.add(MAX_MESSAGE_QUEUE_SIZE);
descriptors.add(MAX_SOCKET_BUFFER_SIZE);
descriptors.add(CHARSET);
descriptors.addAll(getAdditionalProperties());
@@ -178,6 +188,7 @@ public abstract class AbstractListenEventProcessor extends Abst
public void onScheduled(final ProcessContext context) throws IOException {
charset = Charset.forName(context.getProperty(CHARSET).getValue());
port = context.getProperty(PORT).asInteger();
+events = new
LinkedBlockingQueue<>(context.getProperty(MAX_MESSAGE_QUEUE_SIZE).asInteger());
final int maxChannelBufferSize =
context.getProperty(MAX_SOCKET_BUFFER_SIZE).asDataSize(DataUnit.B).intValue();
@@ -230,7 +241,7 @@ public abstract class AbstractListenEventProcessor extends Abst
*
[jira] [Commented] (NIFI-1579) Improve ListenSyslog Performance
[ https://issues.apache.org/jira/browse/NIFI-1579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15179919#comment-15179919 ] Joseph Witt commented on NIFI-1579: --- Very nice Bryan. Quite significant difference. +1 > Improve ListenSyslog Performance > > > Key: NIFI-1579 > URL: https://issues.apache.org/jira/browse/NIFI-1579 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Bryan Bende >Assignee: Bryan Bende > Fix For: 0.6.0 > > Attachments: NIFI-1579-ListenSyslog_Performance-Part2.patch, > NIFI-1579-ListenSyslog_Performance.patch > > > While testing ListenSyslog at various data rates it was observed that a > significant amount of packets were dropped when using UDP. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-1589) replaceText can't match [
[ https://issues.apache.org/jira/browse/NIFI-1589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Djee updated NIFI-1589: --- Description: When trying to match "[", the opening bracket causes an error if not escaped (Not a Java Regular Expression). But, "\[" will not match "[" but "\[". I am not sure if this a bug or working as intended, but there should be a way to regex the opening bracket character. was: When trying to match "[", the opening bracket causes an error if not escaped (Not a Java Regular Expression). But, "\[" will not match "[" but "\\[". I am not sure if this a bug or working as intended, but there should be a way to regex the opening bracket character. > replaceText can't match [ > - > > Key: NIFI-1589 > URL: https://issues.apache.org/jira/browse/NIFI-1589 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Djee >Priority: Minor > > When trying to match "[", the opening bracket causes an error if not escaped > (Not a Java Regular Expression). But, "\[" will not match "[" but > "\[". > I am not sure if this a bug or working as intended, but there should be a way > to regex the opening bracket character. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-1589) replaceText can't match [
[ https://issues.apache.org/jira/browse/NIFI-1589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Djee updated NIFI-1589: --- Description: When trying to match "[", the opening bracket causes an error if not escaped (Not a Java Regular Expression). But, "\[" will not match "[" but "\\[". I am not sure if this a bug or working as intended, but there should be a way to regex the opening bracket character. was: When trying to match "[", the opening bracket causes an error if not escaped (Not a Java Regular Expression). But, "\\[" will not match "[" but "\\[". I am not sure if this a bug or working as intended, but there should be a way to regex the opening bracket character. > replaceText can't match [ > - > > Key: NIFI-1589 > URL: https://issues.apache.org/jira/browse/NIFI-1589 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Djee >Priority: Minor > > When trying to match "[", the opening bracket causes an error if not escaped > (Not a Java Regular Expression). But, "\[" will not match "[" but "\\[". > I am not sure if this a bug or working as intended, but there should be a way > to regex the opening bracket character. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-1589) replaceText can't match [
[ https://issues.apache.org/jira/browse/NIFI-1589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Djee updated NIFI-1589: --- Description: When trying to match "[", the opening bracket causes an error if not escaped (Not a Java Regular Expression). But, "\\[" will not match "[" but "\\[". I am not sure if this a bug or working as intended, but there should be a way to regex the opening bracket character. was: When trying to match "[", the opening bracket causes an error if not escaped (Not a Java Regular Expression). But, "\[" will not match "[" but "\[". I am not sure if this a bug or working as intended, but there should be a way to regex the opening bracket character. > replaceText can't match [ > - > > Key: NIFI-1589 > URL: https://issues.apache.org/jira/browse/NIFI-1589 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Djee >Priority: Minor > > When trying to match "[", the opening bracket causes an error if not escaped > (Not a Java Regular Expression). But, "\\[" will not match "[" but "\\[". > I am not sure if this a bug or working as intended, but there should be a way > to regex the opening bracket character. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-1589) replaceText can't match [
Djee created NIFI-1589: -- Summary: replaceText can't match [ Key: NIFI-1589 URL: https://issues.apache.org/jira/browse/NIFI-1589 Project: Apache NiFi Issue Type: Bug Components: Core Framework Reporter: Djee Priority: Minor When trying to match "[", the opening bracket causes an error if not escaped (Not a Java Regular Expression). But, "\[" will not match "[" but "\[". I am not sure if this a bug or working as intended, but there should be a way to regex the opening bracket character. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
