[jira] [Created] (MINIFI-322) Create MergeContent processor
Randy Gelhausen created MINIFI-322: -- Summary: Create MergeContent processor Key: MINIFI-322 URL: https://issues.apache.org/jira/browse/MINIFI-322 Project: Apache NiFi MiNiFi Issue Type: Bug Components: C++, Extensions Reporter: Randy Gelhausen http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.2.0/org.apache.nifi.processors.standard.MergeContent/index.html A primary use case for MiNiFi is flow management of high volume data at the edge of networks. Today, the S2S protocol provides compression for multiple flowfiles but this does not provide sufficient control over prioritization, rates, and sizes. The addition of the MergeContent processor would provide significantly more end-user customization of how and when MiNiFi ships data to NiFi and other systems. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (MINIFI-266) MiNiFi-cpp should log to $MINIFI_HOME/logs/minifi-app.log
Randy Gelhausen created MINIFI-266: -- Summary: MiNiFi-cpp should log to $MINIFI_HOME/logs/minifi-app.log Key: MINIFI-266 URL: https://issues.apache.org/jira/browse/MINIFI-266 Project: Apache NiFi MiNiFi Issue Type: Bug Components: C++ Reporter: Randy Gelhausen MiNiFi-java logs to $MINIFI_HOME/logs/minifi-app.log, which is very similar to NiFi logging to $NIFI_HOME/logs/nifi-app.log MiNiFi-cpp logs to $MINIFI_HOME/minifi-app.log.txt Since one of the goals for MiNiFi implementations is to be as interchangeable as possible, MiNiFi-cpp's default log configuration should be changed to write to $MINIFI_HOME/logs/minifi-app.log -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (MINIFI-260) Create a "Getting Started" Guide for MiNiFi
[ https://issues.apache.org/jira/browse/MINIFI-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15962189#comment-15962189 ] Randy Gelhausen commented on MINIFI-260: A patch is available here: https://github.com/apache/nifi-site/pull/17 > Create a "Getting Started" Guide for MiNiFi > --- > > Key: MINIFI-260 > URL: https://issues.apache.org/jira/browse/MINIFI-260 > Project: Apache NiFi MiNiFi > Issue Type: Bug >Reporter: Randy Gelhausen >Assignee: Randy Gelhausen > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (MINIFI-260) Create a "Getting Started" Guide for MiNiFi
Randy Gelhausen created MINIFI-260: -- Summary: Create a "Getting Started" Guide for MiNiFi Key: MINIFI-260 URL: https://issues.apache.org/jira/browse/MINIFI-260 Project: Apache NiFi MiNiFi Issue Type: Bug Reporter: Randy Gelhausen Assignee: Randy Gelhausen -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (MINIFI-166) Append command and switches to FlowFiles generated by ExecuteProcess
[ https://issues.apache.org/jira/browse/MINIFI-166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Randy Gelhausen resolved MINIFI-166. Resolution: Duplicate This is a duplicate of MINIFI-161 > Append command and switches to FlowFiles generated by ExecuteProcess > > > Key: MINIFI-166 > URL: https://issues.apache.org/jira/browse/MINIFI-166 > Project: Apache NiFi MiNiFi > Issue Type: Improvement > Components: C++, Core Framework >Reporter: Randy Gelhausen > > As a flow designer, I want to use ExecuteProcess (in minifi-cpp) to run > commands (ps, netstat, ping, traceroute, etc.) and ship outputs to a NiFi > hub. I want to pass each command output as a distinct data source. > As operator of the NiFi hub, I want to allow app teams to send output from > arbitrary commands without creating separate input ports for each command. > I need to route command output based on the command and arguments used to > generate the output, thus I need "command" and "arguments" attributes on the > FlowFiles I receive. > ExecuteProcess should append "command" and "arguments" as FlowFile attributes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MINIFI-166) Append command and switches to FlowFiles generated by ExecuteProcess
Randy Gelhausen created MINIFI-166: -- Summary: Append command and switches to FlowFiles generated by ExecuteProcess Key: MINIFI-166 URL: https://issues.apache.org/jira/browse/MINIFI-166 Project: Apache NiFi MiNiFi Issue Type: Improvement Components: C++, Core Framework Reporter: Randy Gelhausen As a flow designer, I want to use ExecuteProcess (in minifi-cpp) to run commands (ps, netstat, ping, traceroute, etc.) and ship outputs to a NiFi hub. I want to pass each command output as a distinct data source. As operator of the NiFi hub, I want to allow app teams to send output from arbitrary commands without creating separate input ports for each command. I need to route command output based on the command and arguments used to generate the output, thus I need "command" and "arguments" attributes on the FlowFiles I receive. ExecuteProcess should append "command" and "arguments" as FlowFile attributes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MINIFI-159) Create AppendHostInfo processor
Randy Gelhausen created MINIFI-159: -- Summary: Create AppendHostInfo processor Key: MINIFI-159 URL: https://issues.apache.org/jira/browse/MINIFI-159 Project: Apache NiFi MiNiFi Issue Type: Bug Components: C++, Extensions Reporter: Randy Gelhausen Downstream consumers of data produced by nifi-minifi agents often need to route FlowFiles based on hostname or IP of the machine running the agent. nifi-minifi-cpp needs an AppendHostInfo processor which reads its' hostname and IPv4 address for a specified network interface. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MINIFI-96) MiNiFi-cpp GetFile "recurse subdirectories" only recurses one directory deep
[ https://issues.apache.org/jira/browse/MINIFI-96?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Randy Gelhausen resolved MINIFI-96. --- Resolution: Cannot Reproduce > MiNiFi-cpp GetFile "recurse subdirectories" only recurses one directory deep > > > Key: MINIFI-96 > URL: https://issues.apache.org/jira/browse/MINIFI-96 > Project: Apache NiFi MiNiFi > Issue Type: Bug > Components: C++, Core Framework >Reporter: Randy Gelhausen > > I am generating files with the following naming schems: > /host-data/metrics/$METRIC/$HOSTNAME/$MINUTE/$UNIXTIME > $METRIC is "ps", "nmon", "netstat", etc. > Using the following GetFile yml config: > Processors: > - name: GetNMon > class: org.apache.nifi.processors.standard.GetFile > max concurrent tasks: 1 > scheduling strategy: TIMER_DRIVEN > scheduling period: 0 sec > penalization period: 30 sec > yield period: 1 sec > run duration nanos: 0 > auto-terminated relationships list: [] > Properties: > Batch Size: > File Filter: > Ignore Hidden Files: > Input Directory: /host-data/metrics/nmon/HOSTNAME > Keep Source File: > Maximum File Age: > Maximum File Size: > Minimum File Age: 10 sec > Minimum File Size: > Path Filter: > Polling Interval: > Recurse Subdirectories: > In minifi-app.log.txt I see repeated entries for the $MINUTE folders, but no > entries for the $UNIXTIME files underneath those $MINUTE folders. > I suspect the default "false" Keep Subdirectories value is causing MiNiFi to > repeatedly send the $MINUTE folders and never progress into the files > underneath. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MINIFI-95) MiNiFi-cpp GetFile processor sends directories as files
[ https://issues.apache.org/jira/browse/MINIFI-95?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Randy Gelhausen resolved MINIFI-95. --- Resolution: Cannot Reproduce > MiNiFi-cpp GetFile processor sends directories as files > --- > > Key: MINIFI-95 > URL: https://issues.apache.org/jira/browse/MINIFI-95 > Project: Apache NiFi MiNiFi > Issue Type: Bug > Components: C++, Core Framework >Reporter: Randy Gelhausen > Labels: c++, cpp, minifi-cpp > > When recurse subdirectories is true, MiNiFi-cpp sends a flowfile for each > subdirectory, but does not send flowfiles for files in subdirectories. > Processors: > - name: GetNMon > class: org.apache.nifi.processors.standard.GetFile > max concurrent tasks: 1 > scheduling strategy: TIMER_DRIVEN > scheduling period: 0 sec > penalization period: 30 sec > yield period: 1 sec > run duration nanos: 0 > auto-terminated relationships list: [] > Properties: > Batch Size: > File Filter: > Ignore Hidden Files: > Input Directory: /host-data/metrics/nmon/HOSTNAME > Keep Source File: false > Maximum File Age: > Maximum File Size: > Minimum File Age: 10 sec > Minimum File Size: > Path Filter: > Polling Interval: > Recurse Subdirectories: true -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MINIFI-119) Append source.hostname attribute to FlowFiles sent via Site2Site
Randy Gelhausen created MINIFI-119: -- Summary: Append source.hostname attribute to FlowFiles sent via Site2Site Key: MINIFI-119 URL: https://issues.apache.org/jira/browse/MINIFI-119 Project: Apache NiFi MiNiFi Issue Type: Bug Components: C++, Core Framework Reporter: Randy Gelhausen NiFi data consumers commonly need to know the original source (hostname) of flowfiles. NiFi collectors may create a unique input port per Site2Site client, but this in unmanageable with more than a trivial number of clients. Clients can manually inject source metadata as part of their flow definitions, but this is a burden that can be automated given that hostname is standard metadata. The Site2Site protocol implementation should automatically inject local hostname into FlowFiles. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MINIFI-114) nifi-minifi-cpp default "State Location" value "Local" causes error
Randy Gelhausen created MINIFI-114: -- Summary: nifi-minifi-cpp default "State Location" value "Local" causes error Key: MINIFI-114 URL: https://issues.apache.org/jira/browse/MINIFI-114 Project: Apache NiFi MiNiFi Issue Type: Bug Components: C++, Core Framework Reporter: Randy Gelhausen Example config: - name: TailFile class: org.apache.nifi.processors.standard.TailFile max concurrent tasks: 1 scheduling strategy: TIMER_DRIVEN scheduling period: 0 sec penalization period: 30 sec yield period: 1 sec run duration nanos: 0 auto-terminated relationships list: [] Properties: File Location: Local File to Tail: /var/log/anaconda/syslog Initial Start Position: Beginning of File Rolling Filename Pattern: [2016-10-03 17:05:50.273] [minifi log] [error] store state file failed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MINIFI-113) nifi-minifi-cpp throws "Invalid File Filter regex" error for GetFile's default "File Filter" property
Randy Gelhausen created MINIFI-113: -- Summary: nifi-minifi-cpp throws "Invalid File Filter regex" error for GetFile's default "File Filter" property Key: MINIFI-113 URL: https://issues.apache.org/jira/browse/MINIFI-113 Project: Apache NiFi MiNiFi Issue Type: Bug Components: C++, Core Framework Reporter: Randy Gelhausen I used the 0.0.1 minifi converter toolkit for a flow containing a GetFile processor with default "Filter Filter" properties ([^\.].*). It generated the following config: - name: GetFile class: org.apache.nifi.processors.standard.GetFile max concurrent tasks: 1 scheduling strategy: TIMER_DRIVEN scheduling period: 1 sec penalization period: 30 sec yield period: 1 sec run duration nanos: 0 auto-terminated relationships list: [] Properties: Batch Size: '10' File Filter: '[^\.].*' Ignore Hidden Files: 'true' Input Directory: /host-data/metrics Keep Source File: 'false' Maximum File Age: Maximum File Size: Minimum File Age: 10 sec Minimum File Size: 0 B Path Filter: Polling Interval: 0 sec Recurse Subdirectories: 'true' However, at runtime, minifi gives an invalid regex error: [2016-10-03 16:31:11.318] [minifi log] [error] Invalid File Filter regex: regex_error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MINIFI-112) Support UpdateAttribute for nifi-minifi-cpp
Randy Gelhausen created MINIFI-112: -- Summary: Support UpdateAttribute for nifi-minifi-cpp Key: MINIFI-112 URL: https://issues.apache.org/jira/browse/MINIFI-112 Project: Apache NiFi MiNiFi Issue Type: Bug Components: C++, Core Framework Reporter: Randy Gelhausen nifi-minifi-cpp agents can generate multiple "streams" of flowfiles. For instance, to monitor a host, nifi-minifi-cpp runs nmon, ps, netstat, and gathers logfiles from applications. But, for a given flowfile, any downstream NiFi collectors wont have visibility into the originating hostname, nor metadata about which "stream" (ExecuteProcess(nmon), ExecuteProcess(ps), TailFile(app1), TailFile(app2)) generated it. One solution is to use a separate InputPort for each stream. This works, but burdens both the team working on agent flows, and the team managing the collector- they have to be in concert. A simpler (better?) approach is to allow agent teams to tag flowfiles with differentiating metadata via use of UpdateAttribute. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MINIFI-111) Addition of ExecuteProcess processor breaks build on master
Randy Gelhausen created MINIFI-111: -- Summary: Addition of ExecuteProcess processor breaks build on master Key: MINIFI-111 URL: https://issues.apache.org/jira/browse/MINIFI-111 Project: Apache NiFi MiNiFi Issue Type: Bug Components: Build, C++ Reporter: Randy Gelhausen Included here are environment details and the errors when running 'make' from project root: [root@d0a671fc9da1 nifi-minifi-cpp]# gcc --version gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-4) Copyright (C) 2015 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. [root@d0a671fc9da1 nifi-minifi-cpp]# g++ --version g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-4) Copyright (C) 2015 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. [root@d0a671fc9da1 nifi-minifi-cpp]# yum install -y boost-devel libxml2-devel Loaded plugins: fastestmirror, ovl Loading mirror speeds from cached hostfile * base: mirrors.umflint.edu * extras: mirror.trouble-free.net * updates: mirrors.centos.webair.com Package boost-devel-1.53.0-25.el7.x86_64 already installed and latest version Package libxml2-devel-2.9.1-6.el7_2.3.x86_64 already installed and latest version Nothing to do [root@d0a671fc9da1 nifi-minifi-cpp]# make mkdir -p ./build mkdir -p ./target make -C thirdparty/uuid make[1]: Entering directory `/nifi-minifi-cpp/thirdparty/uuid' gcc -g -Wall -rdynamic -O0 -ggdb -g3 -fexceptions -Wno-write-strings -fvisibility=hidden -fPIC -I.. ./tst_uuid.c libuuid.a -o ./tst_uuid make[1]: Leaving directory `/nifi-minifi-cpp/thirdparty/uuid' g++ -Os -fexceptions -fpermissive -Wno-write-strings -std=c++11 -fPIC -Wall -g -Wno-unused-private-field -I./inc -I./src -I./test -I./thirdparty -I/usr/include/libxml2 -I./thirdparty/yaml-cpp-yaml-cpp-0.5.3/include -o build/ExecuteProcess.o -c src/ExecuteProcess.cpp src/ExecuteProcess.cpp: In member function 'virtual void ExecuteProcess::onTrigger(ProcessContext*, ProcessSession*)': src/ExecuteProcess.cpp:101:2: error: 'strcpy' is not a member of 'std' std::strcpy(cstr, _fullCommand.c_str()); ^ src/ExecuteProcess.cpp:101:2: note: suggested alternative: In file included from ./inc/TimeUtil.h:25:0, from src/ExecuteProcess.cpp:20: /usr/include/string.h:125:14: note: 'strcpy' extern char *strcpy (char *__restrict __dest, const char *__restrict __src) ^ src/ExecuteProcess.cpp:102:12: error: 'strtok' is not a member of 'std' char *p = std::strtok (cstr, " "); ^ src/ExecuteProcess.cpp:102:12: note: suggested alternative: In file included from ./inc/TimeUtil.h:25:0, from src/ExecuteProcess.cpp:20: /usr/include/string.h:344:14: note: 'strtok' extern char *strtok (char *__restrict __s, const char *__restrict __delim) ^ src/ExecuteProcess.cpp:108:7: error: 'strtok' is not a member of 'std' p = std::strtok(NULL, " "); ^ src/ExecuteProcess.cpp:108:7: note: suggested alternative: In file included from ./inc/TimeUtil.h:25:0, from src/ExecuteProcess.cpp:20: /usr/include/string.h:344:14: note: 'strtok' extern char *strtok (char *__restrict __s, const char *__restrict __delim) ^ src/ExecuteProcess.cpp:198:49: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] if (numRead == (sizeof(buffer) - totalRead)) ^ At global scope: -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MINIFI-95) MiNiFi-cpp GetFile processor sends directories as files
[ https://issues.apache.org/jira/browse/MINIFI-95?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15445073#comment-15445073 ] Randy Gelhausen commented on MINIFI-95: --- Just to confirm, the fs does recognize directories vs files: [root@2ef5d2331f09 a]# stat test.txt File: 'test.txt' Size: 4 Blocks: 8 IO Block: 4096 regular file Device: fd05h/64773dInode: 167794696 Links: 1 Access: (0644/-rw-r--r--) Uid: (0/root) Gid: (0/root) Access: 2016-08-29 06:59:51.914447813 + Modify: 2016-08-29 06:57:25.699454601 + Change: 2016-08-29 07:01:41.452442727 + Birth: - [root@2ef5d2331f09 a]# cd .. [root@2ef5d2331f09 test]# stat a/ File: 'a/' Size: 21 Blocks: 0 IO Block: 4096 directory Device: fd05h/64773dInode: 301997110 Links: 2 Access: (0755/drwxr-xr-x) Uid: (0/root) Gid: (0/root) Access: 2016-08-29 07:12:53.337411533 + Modify: 2016-08-29 07:01:41.452442727 + Change: 2016-08-29 07:01:41.452442727 + Birth: - > MiNiFi-cpp GetFile processor sends directories as files > --- > > Key: MINIFI-95 > URL: https://issues.apache.org/jira/browse/MINIFI-95 > Project: Apache NiFi MiNiFi > Issue Type: Bug > Components: C++, Core Framework >Reporter: Randy Gelhausen > Labels: c++, cpp, minifi-cpp > > When recurse subdirectories is true, MiNiFi-cpp sends a flowfile for each > subdirectory, but does not send flowfiles for files in subdirectories. > Processors: > - name: GetNMon > class: org.apache.nifi.processors.standard.GetFile > max concurrent tasks: 1 > scheduling strategy: TIMER_DRIVEN > scheduling period: 0 sec > penalization period: 30 sec > yield period: 1 sec > run duration nanos: 0 > auto-terminated relationships list: [] > Properties: > Batch Size: > File Filter: > Ignore Hidden Files: > Input Directory: /host-data/metrics/nmon/HOSTNAME > Keep Source File: false > Maximum File Age: > Maximum File Size: > Minimum File Age: 10 sec > Minimum File Size: > Path Filter: > Polling Interval: > Recurse Subdirectories: true -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MINIFI-95) MiNiFi-cpp GetFile processor sends directories as files
[ https://issues.apache.org/jira/browse/MINIFI-95?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Randy Gelhausen updated MINIFI-95: -- Description: When recurse subdirectories is true, MiNiFi-cpp sends a flowfile for each subdirectory, but does not send flowfiles for files in subdirectories. Processors: - name: GetNMon class: org.apache.nifi.processors.standard.GetFile max concurrent tasks: 1 scheduling strategy: TIMER_DRIVEN scheduling period: 0 sec penalization period: 30 sec yield period: 1 sec run duration nanos: 0 auto-terminated relationships list: [] Properties: Batch Size: File Filter: Ignore Hidden Files: Input Directory: /host-data/metrics/nmon/HOSTNAME Keep Source File: false Maximum File Age: Maximum File Size: Minimum File Age: 10 sec Minimum File Size: Path Filter: Polling Interval: Recurse Subdirectories: true was: When recurse subdirectories is true (in below example, it is blank, which should default to true), MiNiFi-cpp sends a flowfile for each file in each subdirectory, but _also_ sends a flowfile for each subdirectory folder. Feel free to close if this is expected behavior, but I assumed GetFile would send only files, not directories. Processors: - name: GetNMon class: org.apache.nifi.processors.standard.GetFile max concurrent tasks: 1 scheduling strategy: TIMER_DRIVEN scheduling period: 0 sec penalization period: 30 sec yield period: 1 sec run duration nanos: 0 auto-terminated relationships list: [] Properties: Batch Size: File Filter: Ignore Hidden Files: Input Directory: /host-data/metrics/nmon/HOSTNAME Keep Source File: Maximum File Age: Maximum File Size: Minimum File Age: 10 sec Minimum File Size: Path Filter: Polling Interval: Recurse Subdirectories: > MiNiFi-cpp GetFile processor sends directories as files > --- > > Key: MINIFI-95 > URL: https://issues.apache.org/jira/browse/MINIFI-95 > Project: Apache NiFi MiNiFi > Issue Type: Bug > Components: C++, Core Framework >Reporter: Randy Gelhausen > Labels: c++, cpp, minifi-cpp > > When recurse subdirectories is true, MiNiFi-cpp sends a flowfile for each > subdirectory, but does not send flowfiles for files in subdirectories. > Processors: > - name: GetNMon > class: org.apache.nifi.processors.standard.GetFile > max concurrent tasks: 1 > scheduling strategy: TIMER_DRIVEN > scheduling period: 0 sec > penalization period: 30 sec > yield period: 1 sec > run duration nanos: 0 > auto-terminated relationships list: [] > Properties: > Batch Size: > File Filter: > Ignore Hidden Files: > Input Directory: /host-data/metrics/nmon/HOSTNAME > Keep Source File: false > Maximum File Age: > Maximum File Size: > Minimum File Age: 10 sec > Minimum File Size: > Path Filter: > Polling Interval: > Recurse Subdirectories: true -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MINIFI-95) MiNiFi-cpp GetFile processor sends directories as files
[ https://issues.apache.org/jira/browse/MINIFI-95?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15445055#comment-15445055 ] Randy Gelhausen commented on MINIFI-95: --- I added logging statements to GetFile::performListing. It seems that subdirectories are not recognized as directories, but instead possibly as type "DT_UNKNOWN", and assumed to be regular files. This is on CentOS 7: cat /etc/redhat-release -> CentOS Linux release 7.2.1511 (Core) I created /test/a/test.txt and ran a GetFile processor with recursion true on /test: [2016-08-29 07:01:45.451] [minifi log] [info] MiNiFi started [2016-08-29 07:01:45.451] [minifi log] [info] Checking: /test/. [2016-08-29 07:01:45.451] [minifi log] [info] /test/.: entry->d_type: 4 [2016-08-29 07:01:45.451] [minifi log] [info] entry->d_type & DT_DIR == true [2016-08-29 07:01:45.451] [minifi log] [info] Checking: /test/.. [2016-08-29 07:01:45.451] [minifi log] [info] /test/..: entry->d_type: 4 [2016-08-29 07:01:45.451] [minifi log] [info] entry->d_type & DT_DIR == true [2016-08-29 07:01:45.451] [minifi log] [info] Checking: /test/a [2016-08-29 07:01:45.451] [minifi log] [info] /test/a: entry->d_type: 0 Since d_type is not DT_DIR, minifi is never attempting to list files in /test/a > MiNiFi-cpp GetFile processor sends directories as files > --- > > Key: MINIFI-95 > URL: https://issues.apache.org/jira/browse/MINIFI-95 > Project: Apache NiFi MiNiFi > Issue Type: Bug > Components: C++, Core Framework >Reporter: Randy Gelhausen > Labels: c++, cpp, minifi-cpp > > When recurse subdirectories is true (in below example, it is blank, which > should default to true), MiNiFi-cpp sends a flowfile for each file in each > subdirectory, but _also_ sends a flowfile for each subdirectory folder. > Feel free to close if this is expected behavior, but I assumed GetFile would > send only files, not directories. > Processors: > - name: GetNMon > class: org.apache.nifi.processors.standard.GetFile > max concurrent tasks: 1 > scheduling strategy: TIMER_DRIVEN > scheduling period: 0 sec > penalization period: 30 sec > yield period: 1 sec > run duration nanos: 0 > auto-terminated relationships list: [] > Properties: > Batch Size: > File Filter: > Ignore Hidden Files: > Input Directory: /host-data/metrics/nmon/HOSTNAME > Keep Source File: > Maximum File Age: > Maximum File Size: > Minimum File Age: 10 sec > Minimum File Size: > Path Filter: > Polling Interval: > Recurse Subdirectories: -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MINIFI-96) MiNiFi-cpp GetFile "recurse subdirectories" only recurses one directory deep
Randy Gelhausen created MINIFI-96: - Summary: MiNiFi-cpp GetFile "recurse subdirectories" only recurses one directory deep Key: MINIFI-96 URL: https://issues.apache.org/jira/browse/MINIFI-96 Project: Apache NiFi MiNiFi Issue Type: Bug Components: C++, Core Framework Reporter: Randy Gelhausen I am generating files with the following naming schems: /host-data/metrics/$METRIC/$HOSTNAME/$MINUTE/$UNIXTIME $METRIC is "ps", "nmon", "netstat", etc. Using the following GetFile yml config: Processors: - name: GetNMon class: org.apache.nifi.processors.standard.GetFile max concurrent tasks: 1 scheduling strategy: TIMER_DRIVEN scheduling period: 0 sec penalization period: 30 sec yield period: 1 sec run duration nanos: 0 auto-terminated relationships list: [] Properties: Batch Size: File Filter: Ignore Hidden Files: Input Directory: /host-data/metrics/nmon/HOSTNAME Keep Source File: Maximum File Age: Maximum File Size: Minimum File Age: 10 sec Minimum File Size: Path Filter: Polling Interval: Recurse Subdirectories: In minifi-app.log.txt I see repeated entries for the $MINUTE folders, but no entries for the $UNIXTIME files underneath those $MINUTE folders. I suspect the default "false" Keep Subdirectories value is causing MiNiFi to repeatedly send the $MINUTE folders and never progress into the files underneath. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MINIFI-95) MiNiFi-cpp GetFile processor sends directories as files
Randy Gelhausen created MINIFI-95: - Summary: MiNiFi-cpp GetFile processor sends directories as files Key: MINIFI-95 URL: https://issues.apache.org/jira/browse/MINIFI-95 Project: Apache NiFi MiNiFi Issue Type: Bug Components: C++, Core Framework Reporter: Randy Gelhausen When recurse subdirectories is true (in below example, it is blank, which should default to true), MiNiFi-cpp sends a flowfile for each file in each subdirectory, but _also_ sends a flowfile for each subdirectory folder. Feel free to close if this is expected behavior, but I assumed GetFile would send only files, not directories. Processors: - name: GetNMon class: org.apache.nifi.processors.standard.GetFile max concurrent tasks: 1 scheduling strategy: TIMER_DRIVEN scheduling period: 0 sec penalization period: 30 sec yield period: 1 sec run duration nanos: 0 auto-terminated relationships list: [] Properties: Batch Size: File Filter: Ignore Hidden Files: Input Directory: /host-data/metrics/nmon/HOSTNAME Keep Source File: Maximum File Age: Maximum File Size: Minimum File Age: 10 sec Minimum File Size: Path Filter: Polling Interval: Recurse Subdirectories: -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MINIFI-92) Create ExecuteProcess processor for NiFi-MiNiFi-cpp
[ https://issues.apache.org/jira/browse/MINIFI-92?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Randy Gelhausen updated MINIFI-92: -- Labels: C++ cpp nifi-minifi-cpp (was: cpp nifi-minifi-cpp) > Create ExecuteProcess processor for NiFi-MiNiFi-cpp > --- > > Key: MINIFI-92 > URL: https://issues.apache.org/jira/browse/MINIFI-92 > Project: Apache NiFi MiNiFi > Issue Type: Bug > Components: Core Framework >Reporter: Randy Gelhausen > Labels: C++, cpp, nifi-minifi-cpp > > Instrumenting application servers/devices with (Mi)NiFi agents gives ops > teams the ability to add arbitrary system check and metrics. > NiFi proper's ExecuteProcess lets you use the wide variety of existing Linux > tools and data gatherers, but MiNiFi-cpp lacks this today. Thus > servers/containers need to be configured during build time (for example: > https://github.com/randerzander/TechOps/blob/master/containers/base/scripts/nmon.sh), > which is tedious and error prone compared to being able to have the MiNiFi > agent invoke processes at will. > ExecuteProcess in MiNiFi-cpp would significantly improve the value of the > agent for server/container instrumentation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MINIFI-93) MiNiFi-cpp does not honor "yield period"
[ https://issues.apache.org/jira/browse/MINIFI-93?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Randy Gelhausen updated MINIFI-93: -- Labels: C++ (was: ) > MiNiFi-cpp does not honor "yield period" > > > Key: MINIFI-93 > URL: https://issues.apache.org/jira/browse/MINIFI-93 > Project: Apache NiFi MiNiFi > Issue Type: Bug > Components: Core Framework >Reporter: Randy Gelhausen > Labels: C++ > > If the minifi-cpp agent cannot communicate with the specified input port, it > continuously tries to reconnect, ignoring the specified yield period. See > example below. > Example-- > Remote Processing Groups: > - name: http://monitor.dev:8080/nifi > url: http://monitor.dev:8080/nifi > comment: '' > timeout: 30 sec > yield period: 10 sec > Input Ports: > - id: f25b1117-9c24-4230-a52c-b3c8d9e4d587 > name: MiNiFiInput > comment: '' > max concurrent tasks: 1 > use compression: false > Properties: # Deviates from spec and will later be removed when this is > autonegotiated > Port: 1026 > Host Name: monitor.dev > minifi logs continuing forever: > [2016-08-28 18:00:59.178] [minifi log] [error] socket connect failed to > monitor.dev 1026 > [2016-08-28 18:00:59.178] [minifi log] [error] Site2Site peer socket open > failed > [2016-08-28 18:00:59.179] [minifi log] [error] socket connect failed to > monitor.dev 1026 > [2016-08-28 18:00:59.180] [minifi log] [error] Site2Site peer socket open > failed > [2016-08-28 18:00:59.182] [minifi log] [error] socket connect failed to > monitor.dev 1026 > [2016-08-28 18:00:59.182] [minifi log] [error] Site2Site peer socket open > failed > [2016-08-28 18:00:59.184] [minifi log] [error] socket connect failed to > monitor.dev 1026 > [2016-08-28 18:00:59.184] [minifi log] [error] Site2Site peer socket open > failed > [2016-08-28 18:00:59.188] [minifi log] [error] socket connect failed to > monitor.dev 1026 > [2016-08-28 18:00:59.188] [minifi log] [error] Site2Site peer socket open > failed > [2016-08-28 18:00:59.190] [minifi log] [error] socket connect failed to > monitor.dev 1026 > [2016-08-28 18:00:59.190] [minifi log] [error] Site2Site peer socket open > failed > [2016-08-28 18:00:59.192] [minifi log] [error] socket connect failed to > monitor.dev 1026 > [2016-08-28 18:00:59.192] [minifi log] [error] Site2Site peer socket open > failed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MINIFI-92) Create ExecuteProcess processor for NiFi-MiNiFi-cpp
[ https://issues.apache.org/jira/browse/MINIFI-92?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Randy Gelhausen updated MINIFI-92: -- Labels: cpp nifi-minifi-cpp (was: ) > Create ExecuteProcess processor for NiFi-MiNiFi-cpp > --- > > Key: MINIFI-92 > URL: https://issues.apache.org/jira/browse/MINIFI-92 > Project: Apache NiFi MiNiFi > Issue Type: Bug > Components: Core Framework >Reporter: Randy Gelhausen > Labels: cpp, nifi-minifi-cpp > > Instrumenting application servers/devices with (Mi)NiFi agents gives ops > teams the ability to add arbitrary system check and metrics. > NiFi proper's ExecuteProcess lets you use the wide variety of existing Linux > tools and data gatherers, but MiNiFi-cpp lacks this today. Thus > servers/containers need to be configured during build time (for example: > https://github.com/randerzander/TechOps/blob/master/containers/base/scripts/nmon.sh), > which is tedious and error prone compared to being able to have the MiNiFi > agent invoke processes at will. > ExecuteProcess in MiNiFi-cpp would significantly improve the value of the > agent for server/container instrumentation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MINIFI-93) MiNiFi-cpp does not honor "yield period"
Randy Gelhausen created MINIFI-93: - Summary: MiNiFi-cpp does not honor "yield period" Key: MINIFI-93 URL: https://issues.apache.org/jira/browse/MINIFI-93 Project: Apache NiFi MiNiFi Issue Type: Bug Components: Core Framework Reporter: Randy Gelhausen If the minifi-cpp agent cannot communicate with the specified input port, it continuously tries to reconnect, ignoring the specified yield period. See example below. Example-- Remote Processing Groups: - name: http://monitor.dev:8080/nifi url: http://monitor.dev:8080/nifi comment: '' timeout: 30 sec yield period: 10 sec Input Ports: - id: f25b1117-9c24-4230-a52c-b3c8d9e4d587 name: MiNiFiInput comment: '' max concurrent tasks: 1 use compression: false Properties: # Deviates from spec and will later be removed when this is autonegotiated Port: 1026 Host Name: monitor.dev minifi logs continuing forever: [2016-08-28 18:00:59.178] [minifi log] [error] socket connect failed to monitor.dev 1026 [2016-08-28 18:00:59.178] [minifi log] [error] Site2Site peer socket open failed [2016-08-28 18:00:59.179] [minifi log] [error] socket connect failed to monitor.dev 1026 [2016-08-28 18:00:59.180] [minifi log] [error] Site2Site peer socket open failed [2016-08-28 18:00:59.182] [minifi log] [error] socket connect failed to monitor.dev 1026 [2016-08-28 18:00:59.182] [minifi log] [error] Site2Site peer socket open failed [2016-08-28 18:00:59.184] [minifi log] [error] socket connect failed to monitor.dev 1026 [2016-08-28 18:00:59.184] [minifi log] [error] Site2Site peer socket open failed [2016-08-28 18:00:59.188] [minifi log] [error] socket connect failed to monitor.dev 1026 [2016-08-28 18:00:59.188] [minifi log] [error] Site2Site peer socket open failed [2016-08-28 18:00:59.190] [minifi log] [error] socket connect failed to monitor.dev 1026 [2016-08-28 18:00:59.190] [minifi log] [error] Site2Site peer socket open failed [2016-08-28 18:00:59.192] [minifi log] [error] socket connect failed to monitor.dev 1026 [2016-08-28 18:00:59.192] [minifi log] [error] Site2Site peer socket open failed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MINIFI-92) Create ExecuteProcess processor for NiFi-MiNiFi-cpp
[ https://issues.apache.org/jira/browse/MINIFI-92?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Randy Gelhausen updated MINIFI-92: -- Description: Instrumenting application servers/devices with (Mi)NiFi agents gives ops teams the ability to add arbitrary system check and metrics. NiFi proper's ExecuteProcess lets you use the wide variety of existing Linux tools and data gatherers, but MiNiFi-cpp lacks this today. Thus servers/containers need to be configured during build time (for example: https://github.com/randerzander/TechOps/blob/master/containers/base/scripts/nmon.sh), which is tedious and error prone compared to being able to have the MiNiFi agent invoke processes at will. ExecuteProcess in MiNiFi-cpp would significantly improve the value of the agent for server/container instrumentation. was: Instrumenting application servers/devices with (Mi)NiFi agents gives ops teams the ability to add arbitrary system check and metrics. NiFi proper's ExecuteProcess lets you use the wide variety of existing Linux tools and data gatherers, but MiNiFi-cpp lacks this today. Thus servers/containers need to be configured during build time, which is tedious and error prone compared to being able to have the MiNiFi agent invoke processes at will. ExecuteProcess in MiNiFi-cpp would significantly improve the value of the agent for server/container instrumentation. > Create ExecuteProcess processor for NiFi-MiNiFi-cpp > --- > > Key: MINIFI-92 > URL: https://issues.apache.org/jira/browse/MINIFI-92 > Project: Apache NiFi MiNiFi > Issue Type: Bug > Components: Core Framework >Reporter: Randy Gelhausen > > Instrumenting application servers/devices with (Mi)NiFi agents gives ops > teams the ability to add arbitrary system check and metrics. > NiFi proper's ExecuteProcess lets you use the wide variety of existing Linux > tools and data gatherers, but MiNiFi-cpp lacks this today. Thus > servers/containers need to be configured during build time (for example: > https://github.com/randerzander/TechOps/blob/master/containers/base/scripts/nmon.sh), > which is tedious and error prone compared to being able to have the MiNiFi > agent invoke processes at will. > ExecuteProcess in MiNiFi-cpp would significantly improve the value of the > agent for server/container instrumentation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MINIFI-92) Create ExecuteProcess processor for NiFi-MiNiFi-cpp
Randy Gelhausen created MINIFI-92: - Summary: Create ExecuteProcess processor for NiFi-MiNiFi-cpp Key: MINIFI-92 URL: https://issues.apache.org/jira/browse/MINIFI-92 Project: Apache NiFi MiNiFi Issue Type: Bug Components: Core Framework Reporter: Randy Gelhausen Instrumenting application servers/devices with (Mi)NiFi agents gives ops teams the ability to add arbitrary system check and metrics. NiFi proper's ExecuteProcess lets you use the wide variety of existing Linux tools and data gatherers, but MiNiFi-cpp lacks this today. Thus servers/containers need to be configured during build time, which is tedious and error prone compared to being able to have the MiNiFi agent invoke processes at will. ExecuteProcess in MiNiFi-cpp would significantly improve the value of the agent for server/container instrumentation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MINIFI-84) nifi-minifi-cpp flow.yml startup errors
[ https://issues.apache.org/jira/browse/MINIFI-84?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15443760#comment-15443760 ] Randy Gelhausen commented on MINIFI-84: --- Thanks, [~aldrin], the fix works for me! > nifi-minifi-cpp flow.yml startup errors > --- > > Key: MINIFI-84 > URL: https://issues.apache.org/jira/browse/MINIFI-84 > Project: Apache NiFi MiNiFi > Issue Type: Bug > Components: C++, Processing Configuration >Reporter: Randy Gelhausen >Assignee: Aldrin Piri > Fix For: cpp-0.0.1 > > Attachments: MiNiFi-Flow.xml, flow.yml > > > Using a recent NiFi 1.0 build, I created a template with GetFile and TailFile > gathering data and pushing it to a Remote Process Group. > After exporting the template, I used nifi-minifi's minifi-toolkit to convert > the XML into a yaml flow definition. > On attempting to start nifi-minifi-cpp with the flow.yml generated by > minifi-toolkit, I get 'YAML::TypedBadConversion' errors. > Attached are the template & flow.yml. Below are the mini-toolkit converter > and minifi logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MINIFI-84) nifi-minifi-cpp flow.yml startup errors
[ https://issues.apache.org/jira/browse/MINIFI-84?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Randy Gelhausen updated MINIFI-84: -- Description: Using a recent NiFi 1.0 build, I created a template with GetFile and TailFile gathering data and pushing it to a Remote Process Group. After exporting the template, I used nifi-minifi's minifi-toolkit to convert the XML into a yaml flow definition. On attempting to start nifi-minifi-cpp with the flow.yml generated by minifi-toolkit, I get 'YAML::TypedBadConversion' errors. Attached are the template & flow.yml. Below are the mini-toolkit converter and minifi logs. was: Using a recent NiFi 1.0 build, I created a template with GetFile and TailFile gathering data and pushing it to a Remote Process Group. After exporting the template, I used nifi-minifi's minifi-toolkit to convert the XML into a yaml flow definition. On attempting to start nifi-minifi-cpp with the flow.yml generated by minifi-toolkit, I get 'YAML::TypedBadConversion' errors. Attached are the template, flow.yml, and minifi logs. > nifi-minifi-cpp flow.yml startup errors > --- > > Key: MINIFI-84 > URL: https://issues.apache.org/jira/browse/MINIFI-84 > Project: Apache NiFi MiNiFi > Issue Type: Bug > Components: C++ >Reporter: Randy Gelhausen > Attachments: MiNiFi-Flow.xml, flow.yml > > > Using a recent NiFi 1.0 build, I created a template with GetFile and TailFile > gathering data and pushing it to a Remote Process Group. > After exporting the template, I used nifi-minifi's minifi-toolkit to convert > the XML into a yaml flow definition. > On attempting to start nifi-minifi-cpp with the flow.yml generated by > minifi-toolkit, I get 'YAML::TypedBadConversion' errors. > Attached are the template & flow.yml. Below are the mini-toolkit converter > and minifi logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MINIFI-84) nifi-minifi-cpp flow.yml startup errors
[ https://issues.apache.org/jira/browse/MINIFI-84?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Randy Gelhausen updated MINIFI-84: -- Attachment: flow.yml > nifi-minifi-cpp flow.yml startup errors > --- > > Key: MINIFI-84 > URL: https://issues.apache.org/jira/browse/MINIFI-84 > Project: Apache NiFi MiNiFi > Issue Type: Bug > Components: C++ >Reporter: Randy Gelhausen > Attachments: MiNiFi-Flow.xml, flow.yml > > > Using a recent NiFi 1.0 build, I created a template with GetFile and TailFile > gathering data and pushing it to a Remote Process Group. > After exporting the template, I used nifi-minifi's minifi-toolkit to convert > the XML into a yaml flow definition. > On attempting to start nifi-minifi-cpp with the flow.yml generated by > minifi-toolkit, I get 'YAML::TypedBadConversion' errors. > Attached are the template & flow.yml. Below are the mini-toolkit converter > and minifi logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MINIFI-84) nifi-minifi-cpp flow.yml startup errors
[ https://issues.apache.org/jira/browse/MINIFI-84?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15421637#comment-15421637 ] Randy Gelhausen commented on MINIFI-84: --- ./target/minifi [2016-08-15 16:20:11.201] [console] [info] MiNiFi started [2016-08-15 16:20:11.201] [console] [info] Performing parsing of specified config.yml [2016-08-15 16:20:11.204] [console] [info] Added 2 processor configs. [2016-08-15 16:20:11.205] [console] [info] NiFi Server Name localhost [2016-08-15 16:20:11.205] [console] [info] NiFi Server Port: [9000] [2016-08-15 16:20:11.205] [console] [info] NiFi server report interval: [0] ms [2016-08-15 16:20:11.205] [console] [info] FlowController NiFi XML file ./conf/flow.yml [2016-08-15 16:20:11.205] [console] [info] FlowController created [2016-08-15 16:20:11.205] [console] [info] Load Flow Controller from file ./conf/flow.yml [2016-08-15 16:20:11.205] [console] [info] Detected a YAML configuration file for processing. [2016-08-15 16:20:11.208] [console] [info] ProcessGroup MiNiFi-Flow created [2016-08-15 16:20:11.208] [console] [info] Processor GetFile created UUID 13569f19-a28c-4f08-b9ec-f6c82ca72265 [2016-08-15 16:20:11.208] [console] [info] Processor GetFile supported property name Batch Size [2016-08-15 16:20:11.208] [console] [info] Processor GetFile supported property name Ignore Hidden Files [2016-08-15 16:20:11.208] [console] [info] Processor GetFile supported property name Input Directory [2016-08-15 16:20:11.208] [console] [info] Processor GetFile supported property name Keep Source File [2016-08-15 16:20:11.208] [console] [info] Processor GetFile supported property name Maximum File Age [2016-08-15 16:20:11.208] [console] [info] Processor GetFile supported property name Maximum File Size [2016-08-15 16:20:11.208] [console] [info] Processor GetFile supported property name Minimum File Age [2016-08-15 16:20:11.208] [console] [info] Processor GetFile supported property name Minimum File Size [2016-08-15 16:20:11.208] [console] [info] Processor GetFile supported property name Polling Interval [2016-08-15 16:20:11.208] [console] [info] Processor GetFile supported property name Recurse Subdirectories [2016-08-15 16:20:11.208] [console] [info] Processor GetFile supported relationship name success terminate called after throwing an instance of 'YAML::TypedBadConversion' what(): yaml-cpp: error at line 56, column 5: bad conversion Aborted > nifi-minifi-cpp flow.yml startup errors > --- > > Key: MINIFI-84 > URL: https://issues.apache.org/jira/browse/MINIFI-84 > Project: Apache NiFi MiNiFi > Issue Type: Bug > Components: C++ >Reporter: Randy Gelhausen > Attachments: MiNiFi-Flow.xml > > > Using a recent NiFi 1.0 build, I created a template with GetFile and TailFile > gathering data and pushing it to a Remote Process Group. > After exporting the template, I used nifi-minifi's minifi-toolkit to convert > the XML into a yaml flow definition. > On attempting to start nifi-minifi-cpp with the flow.yml generated by > minifi-toolkit, I get 'YAML::TypedBadConversion' errors. > Attached are the template, flow.yml, and minifi logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MINIFI-84) nifi-minifi-cpp flow.yml startup errors
[ https://issues.apache.org/jira/browse/MINIFI-84?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Randy Gelhausen updated MINIFI-84: -- Attachment: MiNiFi-Flow.xml > nifi-minifi-cpp flow.yml startup errors > --- > > Key: MINIFI-84 > URL: https://issues.apache.org/jira/browse/MINIFI-84 > Project: Apache NiFi MiNiFi > Issue Type: Bug > Components: C++ >Reporter: Randy Gelhausen > Attachments: MiNiFi-Flow.xml > > > Using a recent NiFi 1.0 build, I created a template with GetFile and TailFile > gathering data and pushing it to a Remote Process Group. > After exporting the template, I used nifi-minifi's minifi-toolkit to convert > the XML into a yaml flow definition. > On attempting to start nifi-minifi-cpp with the flow.yml generated by > minifi-toolkit, I get 'YAML::TypedBadConversion' errors. > Attached are the template, flow.yml, and minifi logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MINIFI-84) nifi-minifi-cpp flow.yml startup errors
[ https://issues.apache.org/jira/browse/MINIFI-84?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Randy Gelhausen updated MINIFI-84: -- HW11401:nifi-minifi randy$ cd minifi-toolkit/minifi-toolkit-assembly/target/minifi-0.1.0-SNAPSHOT-bin/minifi-toolkit-0.1.0-SNAPSHOT/ HW11401:minifi-toolkit-0.1.0-SNAPSHOT randy$ ls LICENSE NOTICE README bin lib HW11401:minifi-toolkit-0.1.0-SNAPSHOT randy$ ./bin/config.sh transform ~/Downloads/MiNiFi-Flow.xml flow.yml Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_65.jdk/Contents/Home/ MiNiFi Toolkit home: /Users/randy/projects/nifi-minifi/minifi-toolkit/minifi-toolkit-assembly/target/minifi-0.1.0-SNAPSHOT-bin/minifi-toolkit-0.1.0-SNAPSHOT No validation errors found in template. > nifi-minifi-cpp flow.yml startup errors > --- > > Key: MINIFI-84 > URL: https://issues.apache.org/jira/browse/MINIFI-84 > Project: Apache NiFi MiNiFi > Issue Type: Bug > Components: C++ >Reporter: Randy Gelhausen > Attachments: MiNiFi-Flow.xml > > > Using a recent NiFi 1.0 build, I created a template with GetFile and TailFile > gathering data and pushing it to a Remote Process Group. > After exporting the template, I used nifi-minifi's minifi-toolkit to convert > the XML into a yaml flow definition. > On attempting to start nifi-minifi-cpp with the flow.yml generated by > minifi-toolkit, I get 'YAML::TypedBadConversion' errors. > Attached are the template, flow.yml, and minifi logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MINIFI-84) nifi-minifi-cpp flow.yml startup errors
Randy Gelhausen created MINIFI-84: - Summary: nifi-minifi-cpp flow.yml startup errors Key: MINIFI-84 URL: https://issues.apache.org/jira/browse/MINIFI-84 Project: Apache NiFi MiNiFi Issue Type: Bug Components: C++ Reporter: Randy Gelhausen Using a recent NiFi 1.0 build, I created a template with GetFile and TailFile gathering data and pushing it to a Remote Process Group. After exporting the template, I used nifi-minifi's minifi-toolkit to convert the XML into a yaml flow definition. On attempting to start nifi-minifi-cpp with the flow.yml generated by minifi-toolkit, I get 'YAML::TypedBadConversion' errors. Attached are the template, flow.yml, and minifi logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-2055) Processor bulletins are displayed oldest first, which makes debugging difficult
[ https://issues.apache.org/jira/browse/NIFI-2055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Randy Gelhausen updated NIFI-2055: -- Component/s: Core UI > Processor bulletins are displayed oldest first, which makes debugging > difficult > --- > > Key: NIFI-2055 > URL: https://issues.apache.org/jira/browse/NIFI-2055 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Reporter: Randy Gelhausen > Attachments: Screen Shot 2016-06-18 at 5.29.30 PM.png > > > When debugging a flow, it's common to change processor configuration items > iteratively until everything is set correctly/as desired. > By default, bulletins expire after 5 minutes, which is helpful. However, > bulletins are displayed (when mousing over the processor) by oldest timestamp > first. This means it's difficult to see errors/warnings generated by my > newest configurations unless I wait 5 minutes for all the old ones to expire. > I propose changing the bulletin display order to most-recent bulletins first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-2055) Processor bulletins are displayed oldest first, which makes debugging difficult
[ https://issues.apache.org/jira/browse/NIFI-2055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Randy Gelhausen updated NIFI-2055: -- Attachment: Screen Shot 2016-06-18 at 5.29.30 PM.png > Processor bulletins are displayed oldest first, which makes debugging > difficult > --- > > Key: NIFI-2055 > URL: https://issues.apache.org/jira/browse/NIFI-2055 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Reporter: Randy Gelhausen > Attachments: Screen Shot 2016-06-18 at 5.29.30 PM.png > > > When debugging a flow, it's common to change processor configuration items > iteratively until everything is set correctly/as desired. > By default, bulletins expire after 5 minutes, which is helpful. However, > bulletins are displayed (when mousing over the processor) by oldest timestamp > first. This means it's difficult to see errors/warnings generated by my > newest configurations unless I wait 5 minutes for all the old ones to expire. > I propose changing the bulletin display order to most-recent bulletins first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-2055) Processor bulletins are displayed oldest first, which makes debugging difficult
Randy Gelhausen created NIFI-2055: - Summary: Processor bulletins are displayed oldest first, which makes debugging difficult Key: NIFI-2055 URL: https://issues.apache.org/jira/browse/NIFI-2055 Project: Apache NiFi Issue Type: Bug Reporter: Randy Gelhausen When debugging a flow, it's common to change processor configuration items iteratively until everything is set correctly/as desired. By default, bulletins expire after 5 minutes, which is helpful. However, bulletins are displayed (when mousing over the processor) by oldest timestamp first. This means it's difficult to see errors/warnings generated by my newest configurations unless I wait 5 minutes for all the old ones to expire. I propose changing the bulletin display order to most-recent bulletins first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-1803) Exclude spark classes from nifi-spark-receiver
Randy Gelhausen created NIFI-1803: - Summary: Exclude spark classes from nifi-spark-receiver Key: NIFI-1803 URL: https://issues.apache.org/jira/browse/NIFI-1803 Project: Apache NiFi Issue Type: Bug Components: Extensions Reporter: Randy Gelhausen The nifi-spark-receiver artifacts includes spark-streaming as a dependency. However, when deploying Spark apps, the Spark framework provides all these classes. Including them in a third party artifact leads to class conflicts and debugging hell. The nifi-spark-receiver pom needs to mark spark-streaming as scope provided. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (NIFI-1775) Add time-based merge capability to MergeContent processor
[ https://issues.apache.org/jira/browse/NIFI-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Randy Gelhausen resolved NIFI-1775. --- Resolution: Invalid > Add time-based merge capability to MergeContent processor > - > > Key: NIFI-1775 > URL: https://issues.apache.org/jira/browse/NIFI-1775 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Randy Gelhausen > > MergeContent collects a certain size of data or number of events prior to > emitting a merged flowfile. This works well for rapid ingestion, as usually > downstream consumers see merged FlowFiles with acceptable latency. > However, some ingest rates are spiky, or infrequent, greatly increasing the > latency a downstream consumer sees. Adding a time-based merge window would be > very useful. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1775) Add time-based merge capability to MergeContent processor
[ https://issues.apache.org/jira/browse/NIFI-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15242356#comment-15242356 ] Randy Gelhausen commented on NIFI-1775: --- This works! Thanks! > Add time-based merge capability to MergeContent processor > - > > Key: NIFI-1775 > URL: https://issues.apache.org/jira/browse/NIFI-1775 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Randy Gelhausen > > MergeContent collects a certain size of data or number of events prior to > emitting a merged flowfile. This works well for rapid ingestion, as usually > downstream consumers see merged FlowFiles with acceptable latency. > However, some ingest rates are spiky, or infrequent, greatly increasing the > latency a downstream consumer sees. Adding a time-based merge window would be > very useful. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-1775) Add time-based merge capability to MergeContent processor
Randy Gelhausen created NIFI-1775: - Summary: Add time-based merge capability to MergeContent processor Key: NIFI-1775 URL: https://issues.apache.org/jira/browse/NIFI-1775 Project: Apache NiFi Issue Type: Bug Components: Extensions Reporter: Randy Gelhausen MergeContent collects a certain size of data or number of events prior to emitting a merged flowfile. This works well for rapid ingestion, as usually downstream consumers see merged FlowFiles with acceptable latency. However, some ingest rates are spiky, or infrequent, greatly increasing the latency a downstream consumer sees. Adding a time-based merge window would be very useful. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-1750) Document ScanAttribute's dictionary refresh logic
Randy Gelhausen created NIFI-1750: - Summary: Document ScanAttribute's dictionary refresh logic Key: NIFI-1750 URL: https://issues.apache.org/jira/browse/NIFI-1750 Project: Apache NiFi Issue Type: Bug Components: Extensions Reporter: Randy Gelhausen See https://community.hortonworks.com/questions/26858/how-would-i-do-a-stream-join-of-one-data-source-to.html ScanAttribute's docs don't mention anything about how or when dictionaries are periodically refreshed- http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.ScanAttribute/index.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-1681) NiFi Docs Video HowTo links broken
[ https://issues.apache.org/jira/browse/NIFI-1681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Randy Gelhausen updated NIFI-1681: -- Attachment: Screen Shot 2016-03-23 at 7.01.03 PM.png > NiFi Docs Video HowTo links broken > -- > > Key: NIFI-1681 > URL: https://issues.apache.org/jira/browse/NIFI-1681 > Project: Apache NiFi > Issue Type: Bug > Components: Documentation & Website >Reporter: Randy Gelhausen > Attachments: Screen Shot 2016-03-23 at 7.01.03 PM.png > > > Clicking on any of the videos at http://nifi.apache.org/videos.html results > in a JavaScript error: > foundation.js:1371 Uncaught TypeError: modal.on(...).trigger is not a function -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-1681) NiFi Docs Video HowTo links broken
Randy Gelhausen created NIFI-1681: - Summary: NiFi Docs Video HowTo links broken Key: NIFI-1681 URL: https://issues.apache.org/jira/browse/NIFI-1681 Project: Apache NiFi Issue Type: Bug Components: Documentation & Website Reporter: Randy Gelhausen Clicking on any of the videos at http://nifi.apache.org/videos.html results in a JavaScript error: foundation.js:1371 Uncaught TypeError: modal.on(...).trigger is not a function -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-1635) Expound on suggested disk setup and size for the content, flowfile, and provenance repos
Randy Gelhausen created NIFI-1635: - Summary: Expound on suggested disk setup and size for the content, flowfile, and provenance repos Key: NIFI-1635 URL: https://issues.apache.org/jira/browse/NIFI-1635 Project: Apache NiFi Issue Type: Bug Components: Documentation & Website Reporter: Randy Gelhausen http://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#system-requirements -- would really benefit by including a link to a subsection with basically the content at: https://community.hortonworks.com/questions/22517/nifi-repository-typical-disk-usage-ratios-among-th.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1488) Add Kerberos Support to HBase processors
[ https://issues.apache.org/jira/browse/NIFI-1488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15191342#comment-15191342 ] Randy Gelhausen commented on NIFI-1488: --- Can this be re-opened? The "Kerberos principal" validation rules reject "-" in principal names, but that character should be allowed. > Add Kerberos Support to HBase processors > > > Key: NIFI-1488 > URL: https://issues.apache.org/jira/browse/NIFI-1488 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 0.4.0, 0.4.1 >Reporter: Bryan Bende >Assignee: Ricky Saltzer > Fix For: 0.6.0 > > Attachments: > 0001-NIFI-1488-Adjusting-unused-imports-and-adding-licens.patch > > > Our current HBase integration does not support communicating with a > Kerberized HBase install. We should support this just like we do for the HDFS > processors. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-1590) No error message if nifi starts with port conflict
[ https://issues.apache.org/jira/browse/NIFI-1590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Randy Gelhausen updated NIFI-1590: -- Description: I have another webapp running on port 8080. On starting NiFi with default configs (web port 8080), I don't see any unusual behavior except that "nifi.sh run" immediately exits: HW11401:nifi-0.5.1.1.1.2.0-32 randy$ ./bin/nifi.sh run Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_65.jdk/Contents/Home/ NiFi home: /Users/randy/nifi-0.5.1.1.1.2.0-32 Bootstrap Config File: /Users/randy/nifi-0.5.1.1.1.2.0-32/conf/bootstrap.conf 2016-03-04 13:21:04,345 INFO [main] org.apache.nifi.bootstrap.Command Starting Apache NiFi... 2016-03-04 13:21:04,346 INFO [main] org.apache.nifi.bootstrap.Command Working Directory: /Users/randy/nifi-0.5.1.1.1.2.0-32 2016-03-04 13:21:04,346 INFO [main] org.apache.nifi.bootstrap.Command Command: /Library/Java/JavaVirtualMachines/jdk1.8.0_65.jdk/Contents/Home/bin/java -classpath /Users/randy/nifi-0.5.1.1.1.2.0-32/./conf:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/jcl-over-slf4j-1.7.12.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/jul-to-slf4j-1.7.12.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/log4j-over-slf4j-1.7.12.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/logback-classic-1.1.3.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/logback-core-1.1.3.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/nifi-api-0.5.1.1.1.2.0-32.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/nifi-documentation-0.5.1.1.1.2.0-32.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/nifi-nar-utils-0.5.1.1.1.2.0-32.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/nifi-properties-0.5.1.1.1.2.0-32.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/nifi-runtime-0.5.1.1.1.2.0-32.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/slf4j-api-1.7.12.jar -Dorg.apache.jasper.compiler.disablejsr199=true -Xmx512m -Xms512m -Dsun.net.http.allowRestrictedHeaders=true -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Djava.protocol.handler.pkgs=sun.net.www.protocol -Dnifi.properties.file.path=/Users/randy/nifi-0.5.1.1.1.2.0-32/./conf/nifi.properties -Dnifi.bootstrap.listen.port=54630 -Dapp=NiFi org.apache.nifi.NiFi HW11401:nifi-0.5.1.1.1.2.0-32 randy$ ps aux | grep nifi randy 98845 0.0 0.0 2432772672 s001 S+1:21PM 0:00.00 grep nifi was: I have another webapp running on port 8080. On starting NiFi, I don't see any unusual behavior except that "nifi.sh run" immediately exits: HW11401:nifi-0.5.1.1.1.2.0-32 randy$ ./bin/nifi.sh run Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_65.jdk/Contents/Home/ NiFi home: /Users/randy/nifi-0.5.1.1.1.2.0-32 Bootstrap Config File: /Users/randy/nifi-0.5.1.1.1.2.0-32/conf/bootstrap.conf 2016-03-04 13:21:04,345 INFO [main] org.apache.nifi.bootstrap.Command Starting Apache NiFi... 2016-03-04 13:21:04,346 INFO [main] org.apache.nifi.bootstrap.Command Working Directory: /Users/randy/nifi-0.5.1.1.1.2.0-32 2016-03-04 13:21:04,346 INFO [main] org.apache.nifi.bootstrap.Command Command: /Library/Java/JavaVirtualMachines/jdk1.8.0_65.jdk/Contents/Home/bin/java -classpath /Users/randy/nifi-0.5.1.1.1.2.0-32/./conf:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/jcl-over-slf4j-1.7.12.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/jul-to-slf4j-1.7.12.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/log4j-over-slf4j-1.7.12.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/logback-classic-1.1.3.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/logback-core-1.1.3.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/nifi-api-0.5.1.1.1.2.0-32.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/nifi-documentation-0.5.1.1.1.2.0-32.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/nifi-nar-utils-0.5.1.1.1.2.0-32.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/nifi-properties-0.5.1.1.1.2.0-32.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/nifi-runtime-0.5.1.1.1.2.0-32.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/slf4j-api-1.7.12.jar -Dorg.apache.jasper.compiler.disablejsr199=true -Xmx512m -Xms512m -Dsun.net.http.allowRestrictedHeaders=true -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Djava.protocol.handler.pkgs=sun.net.www.protocol -Dnifi.properties.file.path=/Users/randy/nifi-0.5.1.1.1.2.0-32/./conf/nifi.properties -Dnifi.bootstrap.listen.port=54630 -Dapp=NiFi org.apache.nifi.NiFi HW11401:nifi-0.5.1.1.1.2.0-32 randy$ ps aux | grep nifi randy 98845 0.0 0.0 2432772672 s001 S+1:21PM 0:00.00 grep nifi > No error message if nifi starts with port conflict > -- > > Key: NIFI-1590 > URL: https://issues.apache.org/jira/browse/NIFI-1590 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Randy Gelhausen > > I have another webapp running on port 8080. > On starting NiFi with default configs (web port 8080), I don't see any > unusual
[jira] [Created] (NIFI-1590) No error message if nifi starts with port conflict
Randy Gelhausen created NIFI-1590: - Summary: No error message if nifi starts with port conflict Key: NIFI-1590 URL: https://issues.apache.org/jira/browse/NIFI-1590 Project: Apache NiFi Issue Type: Bug Components: Core Framework Reporter: Randy Gelhausen I have another webapp running on port 8080. On starting NiFi, I don't see any unusual behavior except that "nifi.sh run" immediately exits: HW11401:nifi-0.5.1.1.1.2.0-32 randy$ ./bin/nifi.sh run Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_65.jdk/Contents/Home/ NiFi home: /Users/randy/nifi-0.5.1.1.1.2.0-32 Bootstrap Config File: /Users/randy/nifi-0.5.1.1.1.2.0-32/conf/bootstrap.conf 2016-03-04 13:21:04,345 INFO [main] org.apache.nifi.bootstrap.Command Starting Apache NiFi... 2016-03-04 13:21:04,346 INFO [main] org.apache.nifi.bootstrap.Command Working Directory: /Users/randy/nifi-0.5.1.1.1.2.0-32 2016-03-04 13:21:04,346 INFO [main] org.apache.nifi.bootstrap.Command Command: /Library/Java/JavaVirtualMachines/jdk1.8.0_65.jdk/Contents/Home/bin/java -classpath /Users/randy/nifi-0.5.1.1.1.2.0-32/./conf:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/jcl-over-slf4j-1.7.12.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/jul-to-slf4j-1.7.12.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/log4j-over-slf4j-1.7.12.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/logback-classic-1.1.3.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/logback-core-1.1.3.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/nifi-api-0.5.1.1.1.2.0-32.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/nifi-documentation-0.5.1.1.1.2.0-32.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/nifi-nar-utils-0.5.1.1.1.2.0-32.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/nifi-properties-0.5.1.1.1.2.0-32.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/nifi-runtime-0.5.1.1.1.2.0-32.jar:/Users/randy/nifi-0.5.1.1.1.2.0-32/./lib/slf4j-api-1.7.12.jar -Dorg.apache.jasper.compiler.disablejsr199=true -Xmx512m -Xms512m -Dsun.net.http.allowRestrictedHeaders=true -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Djava.protocol.handler.pkgs=sun.net.www.protocol -Dnifi.properties.file.path=/Users/randy/nifi-0.5.1.1.1.2.0-32/./conf/nifi.properties -Dnifi.bootstrap.listen.port=54630 -Dapp=NiFi org.apache.nifi.NiFi HW11401:nifi-0.5.1.1.1.2.0-32 randy$ ps aux | grep nifi randy 98845 0.0 0.0 2432772672 s001 S+1:21PM 0:00.00 grep nifi -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1575) Add ability to query DB table using "last value" from column
[ https://issues.apache.org/jira/browse/NIFI-1575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15172089#comment-15172089 ] Randy Gelhausen commented on NIFI-1575: --- I see use cases where the query needs to run not based exclusively on monotonically increasing values, but on some context provided by an external trigger. Thus a very useful (and hopefully not too complex) additional feature would be to allow the user to pull the "last value" from DistributedMapCache. This way external flows/activities may act as a more customizable and fine-grained control mechanism for the QueryDBTable processor. > Add ability to query DB table using "last value" from column > > > Key: NIFI-1575 > URL: https://issues.apache.org/jira/browse/NIFI-1575 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess > > It would be a useful feature for a processor to be able to query from a > database table for only those records that have are added/available since the > last time the query was executed. For example, if the processor could keep > the max value of a timestamp column for the last result set returned, the > next time the query was issued, it could use that timestamp value and column > to only return records whose timestamps were greater than "the last time". > This shouldn't be limited to timestamps of course; it would be useful for any > strictly-increasing value (like primary key ID) but would be up to the user > to select the table and column. The processor would simply keep the state for > the specified column's max value. > Proposed is a "QueryDBTable" processor which would have a properties to > specify the table name and the column in the table for which to keep state > information about the last maximum value retrieved. Subsequent queries of the > table use this value to filter for rows whose value for the specified column > are greater than the "last maximum value". Upon each successful query, the > "last maximum value" is updated. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1508) Site-to-Site documentation needs to better articulate the purpose and value
[ https://issues.apache.org/jira/browse/NIFI-1508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15151194#comment-15151194 ] Randy Gelhausen commented on NIFI-1508: --- I still think a diagram in the site-to-site docs would be very helpful. The specific context of the ask is in a discussion about the NiFi->Spark Streaming site2site integration, it wasn't clear that using a single nifi node's address would automatically cause a Spark Streaming job to connect to and pull data from the rest of the NiFi nodes in the cluster. Should this JIRA be re-opened or do we need another JIRA? > Site-to-Site documentation needs to better articulate the purpose and value > --- > > Key: NIFI-1508 > URL: https://issues.apache.org/jira/browse/NIFI-1508 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation & Website >Reporter: Joseph Witt >Priority: Minor > Fix For: 0.5.1 > > > https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#site-to-site > These docs are a good start. But they would benefit from some images which > help users understand setup more and config pointers for admins who have to > set it up. The admin guide refers to site-to-site but there is no > explanation of what it is. Perhaps it deserves its own document - not sure. > A related issue is that the first sentence about site-to-site mentions > instance to instance which can imply a node-to-node thing which obviously > doesn't sound very scalable. Need to rewrite that to better articulate the > value proposition and inherent scalability and load balanced and auto-failed > over behavior of site-to-site. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-1205) Allow GetFile's "File Filter" property to support expression language
Randy Gelhausen created NIFI-1205: - Summary: Allow GetFile's "File Filter" property to support expression language Key: NIFI-1205 URL: https://issues.apache.org/jira/browse/NIFI-1205 Project: Apache NiFi Issue Type: Improvement Components: Extensions Reporter: Randy Gelhausen Priority: Minor GetFile is useful as a scheduled processor, but currently assumes foreknowledge of which files the user desires to match. There may be external signals, e.g. a process crash, that users desire to trigger GetFile to pickup and do work on artifacts specific to the trigger. Adding expression language support for the File Filter property would allow incoming flowfiles to point the processor at different files of interest. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-1201) Allow ExecuteSQL to run queries with that use a variable timestamp or sequence id
[
https://issues.apache.org/jira/browse/NIFI-1201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Randy Gelhausen updated NIFI-1201:
--
Summary: Allow ExecuteSQL to run queries with that use a variable timestamp
or sequence id (was: Allow ExecuteSQL to run queries with that use a variable
timestamp or sequenced)
> Allow ExecuteSQL to run queries with that use a variable timestamp or
> sequence id
> -
>
> Key: NIFI-1201
> URL: https://issues.apache.org/jira/browse/NIFI-1201
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Reporter: Randy Gelhausen
>
> Users are employing ExecuteSQL as a means to schedule periodic queries
> against remote databases. Other tools that do this type of task include the
> ability to maintain and automatically increment a sequence or timestamp used
> in query predicates.
> For example:
> select * from src_table where created_at > "2015-11-19 12:00:00"
> Then a minute later:
> select * from src_table where created_at > "2015-11-19 12:01:00"
> Or:
> insert into my_table values (${prev_id}+1, ${now()})
> Today users can implement the same logic with a series of processors, but
> much work could be saved by allowing ExecuteSQL to maintain these bits of
> state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (NIFI-1201) Allow ExecuteSQL to run queries with that use a variable timestamp or sequenced
Randy Gelhausen created NIFI-1201:
-
Summary: Allow ExecuteSQL to run queries with that use a variable
timestamp or sequenced
Key: NIFI-1201
URL: https://issues.apache.org/jira/browse/NIFI-1201
Project: Apache NiFi
Issue Type: Bug
Components: Extensions
Reporter: Randy Gelhausen
Users are employing ExecuteSQL as a means to schedule periodic queries against
remote databases. Other tools that do this type of task include the ability to
maintain and automatically increment a sequence or timestamp used in query
predicates.
For example:
select * from src_table where created_at > "2015-11-19 12:00:00"
Then a minute later:
select * from src_table where created_at > "2015-11-19 12:01:00"
Or:
insert into my_table values (${prev_id}+1, ${now()})
Today users can implement the same logic with a series of processors, but much
work could be saved by allowing ExecuteSQL to maintain these bits of state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-1205) Allow GetFile's "File Filter" property to support expression language
[ https://issues.apache.org/jira/browse/NIFI-1205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15015094#comment-15015094 ] Randy Gelhausen commented on NIFI-1205: --- agreed. closing. > Allow GetFile's "File Filter" property to support expression language > - > > Key: NIFI-1205 > URL: https://issues.apache.org/jira/browse/NIFI-1205 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Randy Gelhausen >Priority: Minor > > GetFile is useful as a scheduled processor, but currently assumes > foreknowledge of which files the user desires to match. > There may be external signals, e.g. a process crash, that users desire to > trigger GetFile to pickup and do work on artifacts specific to the trigger. > Adding expression language support for the File Filter property would allow > incoming flowfiles to point the processor at different files of interest. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (NIFI-1205) Allow GetFile's "File Filter" property to support expression language
[ https://issues.apache.org/jira/browse/NIFI-1205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Randy Gelhausen resolved NIFI-1205. --- Resolution: Duplicate > Allow GetFile's "File Filter" property to support expression language > - > > Key: NIFI-1205 > URL: https://issues.apache.org/jira/browse/NIFI-1205 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Randy Gelhausen >Priority: Minor > > GetFile is useful as a scheduled processor, but currently assumes > foreknowledge of which files the user desires to match. > There may be external signals, e.g. a process crash, that users desire to > trigger GetFile to pickup and do work on artifacts specific to the trigger. > Adding expression language support for the File Filter property would allow > incoming flowfiles to point the processor at different files of interest. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1094) Searching Provenance Repo by component name results in js error
[ https://issues.apache.org/jira/browse/NIFI-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15015123#comment-15015123 ] Randy Gelhausen commented on NIFI-1094: --- A build of .4 from last night no longer gives me this issue. Closing. > Searching Provenance Repo by component name results in js error > --- > > Key: NIFI-1094 > URL: https://issues.apache.org/jira/browse/NIFI-1094 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Reporter: Randy Gelhausen >Assignee: Matt Gilman > > Reproduction Steps: > 1. Open the Provenance Repository UI > 2. Type some component name > 3. Open js debug console (Command+Shift+J on macs) > 4. Hit enter > See the following error in the console: > "Uncaught TypeError: Cannot read property 'search' of undefined" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (NIFI-1094) Searching Provenance Repo by component name results in js error
[ https://issues.apache.org/jira/browse/NIFI-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Randy Gelhausen resolved NIFI-1094. --- Resolution: Cannot Reproduce > Searching Provenance Repo by component name results in js error > --- > > Key: NIFI-1094 > URL: https://issues.apache.org/jira/browse/NIFI-1094 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Reporter: Randy Gelhausen >Assignee: Matt Gilman > > Reproduction Steps: > 1. Open the Provenance Repository UI > 2. Type some component name > 3. Open js debug console (Command+Shift+J on macs) > 4. Hit enter > See the following error in the console: > "Uncaught TypeError: Cannot read property 'search' of undefined" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-1094) Searching Provenance Repo by component name results in js error
Randy Gelhausen created NIFI-1094: - Summary: Searching Provenance Repo by component name results in js error Key: NIFI-1094 URL: https://issues.apache.org/jira/browse/NIFI-1094 Project: Apache NiFi Issue Type: Bug Components: Core UI Reporter: Randy Gelhausen Reproduction Steps: 1. Open the Provenance Repository UI 2. Type some component name 3. Open js debug console (Command+Shift+J on macs) 4. Hit enter See the following error in the console: "Uncaught TypeError: Cannot read property 'search' of undefined" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-1093) ConvertJSONToSQL incorrectly detects required columns
Randy Gelhausen created NIFI-1093: - Summary: ConvertJSONToSQL incorrectly detects required columns Key: NIFI-1093 URL: https://issues.apache.org/jira/browse/NIFI-1093 Project: Apache NiFi Issue Type: Bug Components: Extensions Reporter: Randy Gelhausen create table device_pings(id varchar not null, ts varchar not null, bssid varchar, ssid varchar, noise integer, signal integer, constraint pk primary key (id, ts)) With the example DDL above, neither SSID nor BSSID are required fields, yet ConvertJSONToSQL throws an exception if the input JSON lacks those fields: 2015-11-01 17:47:10,373 ERROR [Timer-Driven Process Thread-6] o.a.n.p.standard.ConvertJSONToSQL ConvertJSONToSQL[id=a336eb2b-fc63-4118-b098-c0ded1dd5520] Failed to convert StandardFlowFileRecord[uuid=5d2c05f0-982e-4feb-94b1-d9946be730d4,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1446416358991-2, container=default, section=2], offset=203306, length=614],offset=0,name=1446418019645796000,size=614] to a SQL INSERT statement due to org.apache.nifi.processor.exception.ProcessException: JSON does not have a value for the Required column 'BSSID'; routing to failure: org.apache.nifi.processor.exception.ProcessException: JSON does not have a value for the Required column 'BSSID' 2015-11-01 17:47:10,381 ERROR [Timer-Driven Process Thread-6] o.a.n.p.standard.ConvertJSONToSQL ConvertJSONToSQL[id=a336eb2b-fc63-4118-b098-c0ded1dd5520] Failed to convert StandardFlowFileRecord[uuid=727b8c3c-66c5-4d6a-8cdc-602da8b80132,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1446416358991-2, container=default, section=2], offset=203920, length=674],offset=0,name=1446418019645796000,size=674] to a SQL INSERT statement due to org.apache.nifi.processor.exception.ProcessException: JSON does not have a value for the Required column 'SSID'; routing to failure: org.apache.nifi.processor.exception.ProcessException: JSON does not have a value for the Required column 'SSID' The processor has "Unmatched Field Behavior" property. Should there be an additional property "Unmatched Column Behavior" property that allows SQL statements to be generated from the set of columns actually available in the JSON? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1057) ConvertJSONToSQL fails with OOM
[ https://issues.apache.org/jira/browse/NIFI-1057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14984586#comment-14984586 ] Randy Gelhausen commented on NIFI-1057: --- This seems to have fixed it! Thanks! > ConvertJSONToSQL fails with OOM > --- > > Key: NIFI-1057 > URL: https://issues.apache.org/jira/browse/NIFI-1057 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Randy Gelhausen > Attachments: nifi-app.log.gz > > > Running NiFi built from master this afternoon, using Java 1.7.0_79 on OSX. > The flow pulls JSON from a web endpoint, splits into about 200 JSON objects > which are passed to ConvertJSONToSQL and eventually to PutSQL. Based on > nifi-app.log, ConvertJSONToSQL is causing an OOM. > This same setup works fine on Centos 6.6 with Java 1.8.0_40 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1078) ${now()} returns string dates instead of a number of milliseconds
[
https://issues.apache.org/jira/browse/NIFI-1078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977656#comment-14977656
]
Randy Gelhausen commented on NIFI-1078:
---
{now():toNumber()} worked:
ts:1446003503325
I'm not sure if this is a doc bug or a mistake in the now() function.
> ${now()} returns string dates instead of a number of milliseconds
> -
>
> Key: NIFI-1078
> URL: https://issues.apache.org/jira/browse/NIFI-1078
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 0.4.0
>Reporter: Randy Gelhausen
>
> Per the docs: The now function returns the current date and time as the
> number of milliseconds since Midnight GMT on January 1, 1970.
> However, when using ${now()} in expression language, the result is: "Tue Oct
> 27 22:12:21 EDT 2015"
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (NIFI-1078) ${now()} returns string dates instead of a number of milliseconds
Randy Gelhausen created NIFI-1078:
-
Summary: ${now()} returns string dates instead of a number of
milliseconds
Key: NIFI-1078
URL: https://issues.apache.org/jira/browse/NIFI-1078
Project: Apache NiFi
Issue Type: Bug
Components: Core Framework
Affects Versions: 0.4.0
Reporter: Randy Gelhausen
Per the docs: The now function returns the current date and time as the number
of milliseconds since Midnight GMT on January 1, 1970.
However, when using ${now()} in expression language, the result is: "Tue Oct 27
22:12:21 EDT 2015"
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (NIFI-1057) ConvertJSONToSQL fails with OOM
Randy Gelhausen created NIFI-1057: - Summary: ConvertJSONToSQL fails with OOM Key: NIFI-1057 URL: https://issues.apache.org/jira/browse/NIFI-1057 Project: Apache NiFi Issue Type: Bug Components: Extensions Reporter: Randy Gelhausen Attachments: nifi-app.log.gz Running NiFi built from master this afternoon, using Java 1.7.0_79 on OSX. The flow pulls JSON from a web endpoint, splits into about 200 JSON objects which are passed to ConvertJSONToSQL and eventually to PutSQL. Based on nifi-app.log, ConvertJSONToSQL is causing an OOM. This same setup works fine on Centos 6.6 with Java 1.8.0_40 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-1061) DBConnectionPool service unable to terminate
Randy Gelhausen created NIFI-1061: - Summary: DBConnectionPool service unable to terminate Key: NIFI-1061 URL: https://issues.apache.org/jira/browse/NIFI-1061 Project: Apache NiFi Issue Type: Bug Components: Core Framework Reporter: Randy Gelhausen Fix For: 0.4.0 I had a defined flow with an enabled DBConnectionPool using the phoenix-client.jar driver. After stopping the NiFi process and restarting it, the connection pool is unable to reconnect. Attempting to disable the pool never completes the operation. See attached screenshot & nifi-app.log file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-1061) DBConnectionPool service unable to terminate
[ https://issues.apache.org/jira/browse/NIFI-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Randy Gelhausen updated NIFI-1061: -- Attachment: nifi-app.log.gz Screen Shot 2015-10-23 at 7.17.31 PM.png > DBConnectionPool service unable to terminate > > > Key: NIFI-1061 > URL: https://issues.apache.org/jira/browse/NIFI-1061 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Randy Gelhausen > Fix For: 0.4.0 > > Attachments: Screen Shot 2015-10-23 at 7.17.31 PM.png, nifi-app.log.gz > > > I had a defined flow with an enabled DBConnectionPool using the > phoenix-client.jar driver. After stopping the NiFi process and restarting it, > the connection pool is unable to reconnect. Attempting to disable the pool > never completes the operation. See attached screenshot & nifi-app.log file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-1049) ExecuteSQL generates NPE on any null valued column
Randy Gelhausen created NIFI-1049: - Summary: ExecuteSQL generates NPE on any null valued column Key: NIFI-1049 URL: https://issues.apache.org/jira/browse/NIFI-1049 Project: Apache NiFi Issue Type: Bug Components: Extensions Affects Versions: 0.3.0 Reporter: Randy Gelhausen I have a table defined as: create table test (id integer not null, val1 integer, val2 integer, constraint my_pk primary key (id)); I have two rows in the table: 0, null, 1 1, 1, 1 When ExecuteSQL runs 'select * from test' I get an NPE: 2015-10-20 22:37:19,887 ERROR [Timer-Driven Process Thread-5] o.a.nifi.processors.standard.ExecuteSQL ExecuteSQL[id=540ec717-f092-4c45-b9f5-1f60a15c15be] ExecuteSQL[id=540ec717-f092-4c45-b9f5-1f60a15c15be] failed to process session due to org.apache.avro.file.DataFileWriter$AppendWriteException: java.lang.NullPointerException: null of int in field VAL1 of any.data.TEST: org.apache.avro.file.DataFileWriter$AppendWriteException: java.lang.NullPointerException: null of int in field VAL1 of any.data.TEST 2015-10-20 22:37:19,887 WARN [Timer-Driven Process Thread-5] o.a.nifi.processors.standard.ExecuteSQL ExecuteSQL[id=540ec717-f092-4c45-b9f5-1f60a15c15be] Processor Administratively Yielded for 1 sec due to processing failure 2015-10-20 22:37:19,888 WARN [Timer-Driven Process Thread-5] o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding ExecuteSQL[id=540ec717-f092-4c45-b9f5-1f60a15c15be] due to uncaught Exception: org.apache.avro.file.DataFileWriter$AppendWriteException: java.lang.NullPointerException: null of int in field VAL1 of any.data.TEST 2015-10-20 22:37:19,889 WARN [Timer-Driven Process Thread-5] o.a.n.c.t.ContinuallyRunProcessorTask org.apache.avro.file.DataFileWriter$AppendWriteException: java.lang.NullPointerException: null of int in field VAL1 of any.data.TEST at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:296) ~[na:na] at org.apache.nifi.processors.standard.util.JdbcCommon.convertToAvroStream(JdbcCommon.java:87) ~[na:na] at org.apache.nifi.processors.standard.ExecuteSQL$1.process(ExecuteSQL.java:149) ~[na:na] at org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:1937) ~[nifi-framework-core-0.3.1-SNAPSHOT.jar:0.3.1-SNAPSHOT] at org.apache.nifi.processors.standard.ExecuteSQL.onTrigger(ExecuteSQL.java:143) ~[na:na] at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) ~[nifi-api-0.3.1-SNAPSHOT.jar:0.3.1-SNAPSHOT] at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1077) ~[nifi-framework-core-0.3.1-SNAPSHOT.jar:0.3.1-SNAPSHOT] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:127) [nifi-framework-core-0.3.1-SNAPSHOT.jar:0.3.1-SNAPSHOT] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:49) [nifi-framework-core-0.3.1-SNAPSHOT.jar:0.3.1-SNAPSHOT] at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:119) [nifi-framework-core-0.3.1-SNAPSHOT.jar:0.3.1-SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_79] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) [na:1.7.0_79] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_79] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79] Caused by: java.lang.NullPointerException: null of int in field VAL1 of any.data.TEST at org.apache.avro.generic.GenericDatumWriter.npe(GenericDatumWriter.java:93) ~[na:na] at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:87) ~[na:na] at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:58) ~[na:na] at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:290) ~[na:na] ... 16 common frames omitted Caused by: java.lang.NullPointerException: null at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:78) ~[na:na] at org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:114) ~[na:na] at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:104) ~[na:na] at
[jira] [Commented] (NIFI-994) Processor to tail files
[ https://issues.apache.org/jira/browse/NIFI-994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905759#comment-14905759 ] Randy Gelhausen commented on NIFI-994: -- A good test case is: Start tailing processor pointing at /var/log/app.log Stop tailing processor Allow logrotate to rotate /var/log/app.log Start tailing processor Expected Result: Emit line 1 Emit line 2 Emit line 3 etc. Essentially the processor needs to resume emitting from where it left off in the sequence of log-lines. It should handle recognizing where it left off, read any available rotation archives (app.log.1, app.log.2, etc.) in order, and then catch back up emitting from the live app.log file. > Processor to tail files > --- > > Key: NIFI-994 > URL: https://issues.apache.org/jira/browse/NIFI-994 > Project: Apache NiFi > Issue Type: New Feature >Affects Versions: 0.4.0 >Reporter: Joseph Percivall > > It's a very common data ingest situation to want to input text into the > system by "tailing" a file, most commonly log files. Currently we don't have > an easy way to do this. > A simple processor to tail a file would benefit many users. There would need > to be an option to not just tail a file but pick up where the processor left > off if it is interrupted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-991) Add "upsert" verb support for ConvertJSONToSQLProcessor
Randy Gelhausen created NIFI-991: Summary: Add "upsert" verb support for ConvertJSONToSQLProcessor Key: NIFI-991 URL: https://issues.apache.org/jira/browse/NIFI-991 Project: Apache NiFi Issue Type: Improvement Components: Extensions Affects Versions: 0.3.0 Reporter: Randy Gelhausen Apache Phoenix supports only the "upsert" SQL verb. To support Nifi->Phoenix flows, UPSERT_TYPE should be added to the ConvertJSONToSQL standard processor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-958) Support appending to files for PutHDFS and PutFile
Randy Gelhausen created NIFI-958: Summary: Support appending to files for PutHDFS and PutFile Key: NIFI-958 URL: https://issues.apache.org/jira/browse/NIFI-958 Project: Apache NiFi Issue Type: Improvement Components: Extensions Reporter: Randy Gelhausen -- This message was sent by Atlassian JIRA (v6.3.4#6332)
