Re: [PR] MINIFICPP-2309 Add JSON flow config examples [nifi-minifi-cpp]
szaszm closed pull request #1740: MINIFICPP-2309 Add JSON flow config examples URL: https://github.com/apache/nifi-minifi-cpp/pull/1740 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINIFICPP-2309 Add JSON flow config examples [nifi-minifi-cpp]
lordgamez commented on code in PR #1740: URL: https://github.com/apache/nifi-minifi-cpp/pull/1740#discussion_r1562167008 ## examples/querydbtable_config.nifi.schema.json: ## @@ -0,0 +1,245 @@ +{ +"encodingVersion": { +"majorVersion": 2, +"minorVersion": 0 +}, +"maxTimerDrivenThreadCount": 1, +"maxEventDrivenThreadCount": 1, +"parameterContexts": [], +"rootGroup": { +"identifier": "1c2f34d2-ee57-448a-b150-8140f0f7643c", +"instanceIdentifier": "9af45830-320e-437e-8204-dcd3c2933aa6", +"name": "root", +"position": { +"x": 0.0, +"y": 0.0 +}, +"processGroups": [], +"remoteProcessGroups": [], +"processors": [ +{ +"identifier": "0f469727-71d2-420f-aca5-d51599cb7507", +"instanceIdentifier": "9e22e432-9a97-4cd9-a7a4-d1ffeca5fbfe", +"name": "Query users table", +"comments": "", +"position": { +"x": 90.22606801867788, +"y": 3.5556140585132995 +}, +"type": "org.apache.nifi.minifi.processors.QueryDatabaseTable", +"bundle": { +"group": "org.apache.nifi.minifi", +"artifact": "minifi-sql", +"version": "0.15.0" +}, +"properties": { +"Table Name": "users", +"Where Clause": "id = 1", +"DB Controller Service": "ODBCService", +"Columns to Return": "id,name", +"Output Format": "JSON", +"Max Rows Per Flow File": "0" +}, +"propertyDescriptors": { +"Table Name": { +"name": "Table Name", +"identifiesControllerService": false, +"sensitive": false +}, +"Where Clause": { +"name": "Where Clause", +"identifiesControllerService": false, +"sensitive": false +}, +"DB Controller Service": { +"name": "DB Controller Service", +"identifiesControllerService": false, +"sensitive": false +}, +"Columns to Return": { +"name": "Columns to Return", +"identifiesControllerService": false, +"sensitive": false +}, +"Maximum-value Columns": { +"name": "Maximum-value Columns", +"identifiesControllerService": false, +"sensitive": false +}, +"Output Format": { +"name": "Output Format", +"identifiesControllerService": false, +"sensitive": false +}, +"Max Rows Per Flow File": { +"name": "Max Rows Per Flow File", +"identifiesControllerService": false, +"sensitive": false +} +}, +"style": {}, +"schedulingPeriod": "1 sec", +"schedulingStrategy": "TIMER_DRIVEN", +"executionNode": "ALL", +"penaltyDuration": "3 ms", +"yieldDuration": "1000 ms", +"bulletinLevel": "WARN", +"runDurationMillis": 0, +"concurrentlySchedulableTaskCount": 1, +"autoTerminatedRelationships": [], +"componentType": "PROCESSOR", +"groupIdentifier": "1c2f34d2-ee57-448a-b150-8140f0f7643c" +}, +{ +"identifier": "ca0439aa-2454-40f8-90fb-72b792d71bc1", +"instanceIdentifier": "bb8a7729-d7d4-4b82-9cd0-4e5c28a5d52e", +"name": "Put files to /tmp/output", +"comments": "", +"position": { +"x": 382.2260680186779, +"y": 12.5556140585133 +}, +"type": "org.apache.nifi.minifi.processors.PutFile", +"bundle": { +"group": "org.apache.nifi.minifi", +"artifact": "minifi-standard-processors", +"version": "0.15.0" +}, +"properties": { +"Create Missing Directories": "true", +"Maximum File Count": "-1", +"Directory": "/tmp/output", +"Conflict Resolution Strategy": "fail" +}, +"propertyDescriptor
Re: [PR] MINIFICPP-2309 Add JSON flow config examples [nifi-minifi-cpp]
szaszm commented on code in PR #1740: URL: https://github.com/apache/nifi-minifi-cpp/pull/1740#discussion_r1561318465 ## examples/querydbtable_config.nifi.schema.json: ## @@ -0,0 +1,245 @@ +{ +"encodingVersion": { +"majorVersion": 2, +"minorVersion": 0 +}, +"maxTimerDrivenThreadCount": 1, +"maxEventDrivenThreadCount": 1, +"parameterContexts": [], +"rootGroup": { +"identifier": "1c2f34d2-ee57-448a-b150-8140f0f7643c", +"instanceIdentifier": "9af45830-320e-437e-8204-dcd3c2933aa6", +"name": "root", +"position": { +"x": 0.0, +"y": 0.0 +}, +"processGroups": [], +"remoteProcessGroups": [], +"processors": [ +{ +"identifier": "0f469727-71d2-420f-aca5-d51599cb7507", +"instanceIdentifier": "9e22e432-9a97-4cd9-a7a4-d1ffeca5fbfe", +"name": "Query users table", +"comments": "", +"position": { +"x": 90.22606801867788, +"y": 3.5556140585132995 +}, +"type": "org.apache.nifi.minifi.processors.QueryDatabaseTable", +"bundle": { +"group": "org.apache.nifi.minifi", +"artifact": "minifi-sql", +"version": "0.15.0" +}, +"properties": { +"Table Name": "users", +"Where Clause": "id = 1", +"DB Controller Service": "ODBCService", +"Columns to Return": "id,name", +"Output Format": "JSON", +"Max Rows Per Flow File": "0" +}, +"propertyDescriptors": { +"Table Name": { +"name": "Table Name", +"identifiesControllerService": false, +"sensitive": false +}, +"Where Clause": { +"name": "Where Clause", +"identifiesControllerService": false, +"sensitive": false +}, +"DB Controller Service": { +"name": "DB Controller Service", +"identifiesControllerService": false, +"sensitive": false +}, +"Columns to Return": { +"name": "Columns to Return", +"identifiesControllerService": false, +"sensitive": false +}, +"Maximum-value Columns": { +"name": "Maximum-value Columns", +"identifiesControllerService": false, +"sensitive": false +}, +"Output Format": { +"name": "Output Format", +"identifiesControllerService": false, +"sensitive": false +}, +"Max Rows Per Flow File": { +"name": "Max Rows Per Flow File", +"identifiesControllerService": false, +"sensitive": false +} +}, +"style": {}, +"schedulingPeriod": "1 sec", +"schedulingStrategy": "TIMER_DRIVEN", +"executionNode": "ALL", +"penaltyDuration": "3 ms", +"yieldDuration": "1000 ms", +"bulletinLevel": "WARN", +"runDurationMillis": 0, +"concurrentlySchedulableTaskCount": 1, +"autoTerminatedRelationships": [], +"componentType": "PROCESSOR", +"groupIdentifier": "1c2f34d2-ee57-448a-b150-8140f0f7643c" +}, +{ +"identifier": "ca0439aa-2454-40f8-90fb-72b792d71bc1", +"instanceIdentifier": "bb8a7729-d7d4-4b82-9cd0-4e5c28a5d52e", +"name": "Put files to /tmp/output", +"comments": "", +"position": { +"x": 382.2260680186779, +"y": 12.5556140585133 +}, +"type": "org.apache.nifi.minifi.processors.PutFile", +"bundle": { +"group": "org.apache.nifi.minifi", +"artifact": "minifi-standard-processors", +"version": "0.15.0" +}, +"properties": { +"Create Missing Directories": "true", +"Maximum File Count": "-1", +"Directory": "/tmp/output", +"Conflict Resolution Strategy": "fail" +}, +"propertyDescriptors":
Re: [PR] MINIFICPP-2309 Add JSON flow config examples [nifi-minifi-cpp]
lordgamez commented on code in PR #1740: URL: https://github.com/apache/nifi-minifi-cpp/pull/1740#discussion_r1559688807 ## examples/getfile_putfile_config.json: ## @@ -0,0 +1,50 @@ +{ +"rootGroup": { +"name": "MiNiFi Flow", +"processors": [ +{ +"name": "GetFile", +"identifier": "b01793b6-5275-4ee3-9fc0-d9e39ce3162f", +"type": "org.apache.nifi.processors.standard.GetFile", +"schedulingStrategy": "TIMER_DRIVEN", +"schedulingPeriod": "2 sec", +"properties": { +"Input Directory": "/tmp/input" Review Comment: Good catch, fixed in 20feddbeb43868d968be1a932939a2539c653eb8 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINIFICPP-2309 Add JSON flow config examples [nifi-minifi-cpp]
adamdebreceni commented on code in PR #1740: URL: https://github.com/apache/nifi-minifi-cpp/pull/1740#discussion_r1559636533 ## examples/getfile_putfile_config.json: ## @@ -0,0 +1,50 @@ +{ +"rootGroup": { +"name": "MiNiFi Flow", +"processors": [ +{ +"name": "GetFile", +"identifier": "b01793b6-5275-4ee3-9fc0-d9e39ce3162f", +"type": "org.apache.nifi.processors.standard.GetFile", +"schedulingStrategy": "TIMER_DRIVEN", +"schedulingPeriod": "2 sec", +"properties": { +"Input Directory": "/tmp/input" Review Comment: this seems to be different from its yaml counterpart, multiple properties seem to be missing for both processors -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINIFICPP-2309 Add JSON flow config examples [nifi-minifi-cpp]
lordgamez commented on code in PR #1740: URL: https://github.com/apache/nifi-minifi-cpp/pull/1740#discussion_r1559138925 ## examples/splittext_puttcp_config.json: ## @@ -0,0 +1,86 @@ +{ +"rootGroup": { +"name": "MiNiFi Flow", +"processors": [ +{ +"name": "Get files from /tmp/input", +"identifier": "a8fd96f8-6f3e-4ab8-9cc7-7dcb3304d60b", +"type": "org.apache.nifi.processors.standard.GetFile", +"schedulingStrategy": "TIMER_DRIVEN", +"schedulingPeriod": "2 sec", +"penaltyDuration": "30 sec", +"yieldDuration": "1 sec", +"properties": { +"Input Directory": "/tmp/input" +}, +"autoTerminatedRelationships": [] +}, +{ +"name": "Split text by lines skipping the first 3 header lines", +"identifier": "40b45f52-ae1d-443c-9465-7e070ce3bea0", +"type": "org.apache.nifi.processors.standard.SplitText", +"schedulingStrategy": "EVENT_DRIVEN", +"schedulingPeriod": "1 sec", Review Comment: Updated in dcab8ffd0883e3fc881e85f6a30fcbdf0cb7e9f2 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINIFICPP-2309 Add JSON flow config examples [nifi-minifi-cpp]
szaszm commented on code in PR #1740: URL: https://github.com/apache/nifi-minifi-cpp/pull/1740#discussion_r1555894471 ## examples/splittext_puttcp_config.json: ## @@ -0,0 +1,86 @@ +{ +"rootGroup": { +"name": "MiNiFi Flow", +"processors": [ +{ +"name": "Get files from /tmp/input", +"identifier": "a8fd96f8-6f3e-4ab8-9cc7-7dcb3304d60b", +"type": "org.apache.nifi.processors.standard.GetFile", +"schedulingStrategy": "TIMER_DRIVEN", +"schedulingPeriod": "2 sec", +"penaltyDuration": "30 sec", +"yieldDuration": "1 sec", +"properties": { +"Input Directory": "/tmp/input" +}, +"autoTerminatedRelationships": [] +}, +{ +"name": "Split text by lines skipping the first 3 header lines", +"identifier": "40b45f52-ae1d-443c-9465-7e070ce3bea0", +"type": "org.apache.nifi.processors.standard.SplitText", +"schedulingStrategy": "EVENT_DRIVEN", +"schedulingPeriod": "1 sec", Review Comment: I think we should remove the scheduling period setting from EVENT_DRIVEN processors. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINIFICPP-2309 Add JSON flow config examples [nifi-minifi-cpp]
lordgamez commented on code in PR #1740: URL: https://github.com/apache/nifi-minifi-cpp/pull/1740#discussion_r1539352073 ## examples/tailfile_config.json: ## @@ -0,0 +1,51 @@ +{ +"rootGroup": { +"name": "MiNiFi Flow", +"processors": [ +{ +"name": "Log tailfile attributes", +"identifier": "41141af1-3c43-4fa4-8236-2b207b0fe9d8", +"type": "org.apache.nifi.minifi.processors.LogAttribute", +"schedulingStrategy": "TIMER_DRIVEN", +"schedulingPeriod": "1000 ms", Review Comment: I think it should be event driven as we recommend using that in this case, there are other occurrences of timer driven cases. Updated in d67362d317cc2369536a9eede32709da58541444 ## examples/README.md: ## @@ -37,122 +43,158 @@ The following examples show simple flow configurations for several common use ca - [Upload Blob to Azure Storage](#upload-blob-to-azure-storage) - [Put Object in AWS S3 Bucket](#put-object-in-aws-s3-bucket) - [List and Fetch Content from AWS S3 Bucket](#list-and-fetch-content-from-aws-s3-bucket) + - [Merge, compress and upload files to Google Cloud Storage](#merge-compress-and-upload-files-to-google-cloud-storage) - [SQL Operations](#sql-operations) - [Query Database Table](#query-database-table) -- [ExecuteScript](#ExecuteScript) - - [Reverse flowfile's content with lua](#reverse-content-with-lua) - - [Reverse flowfile's content with python](#reverse-content-with-python) +- [ExecuteScript](#executescript) + - [Reverse Content with Scripts](#reverse-content-with-scripts) +- [Grafana Loki](#grafana-loki) + - [Send logs to Grafana Loki](#send-logs-to-grafana-loki) +- [MQTT Operations](#mqtt-operations) + - [Publish Message to MQTT Broker](#publish-message-to-mqtt-broker) +- [Network Operations](#network-operations) + - [Send Data to Remote Host through TCP](#send-data-to-remote-host-through-tcp) ## Filesystem Operations ### Getting File Data and Putting It in an Output Directory -Using the [getfile_putfile_config.yml](getfile_putfile_config.yml) flow configuration MiNiFi gets all files of minimum 1MB size from the `/tmp/getfile_dir` directory and puts them in the `/tmp/out_dir` output directory. +Using the [getfile_putfile_config.yml](getfile_putfile_config.yml)/[getfile_putfile_config.json](getfile_putfile_config.json) flow configuration MiNiFi gets all files of minimum 1MB size from the `/tmp/getfile_dir` directory and puts them in the `/tmp/out_dir` output directory. -The flow: GetFile ➔ success ➔ PutFile +The flow: [GetFile](../PROCESSORS.md#getfile) ➔ [PutFile](../PROCESSORS.md#putfile) ### Tailing a Single File -Using the [tailfile_config.yml](tailfile_config.yml) flow configuration MiNiFi tails a single file `/tmp/test_file.log` and creates flowfiles from every single line, then logs the flowfile attributes. +Using the [tailfile_config.yml](tailfile_config.yml)/[tailfile_config.json](tailfile_config.json) flow configuration MiNiFi tails a single file `/tmp/test_file.log` and creates flowfiles from every single line, then logs the flowfile attributes. -The flow: TailFile ➔ success ➔ LogAttribute +The flow: [TailFile](../PROCESSORS.md#tailfile) ➔ [LogAttribute](../PROCESSORS.md#logattribute) ## Windows Specific Processors ### Consuming Windows Event Logs -Using the [cwel_config.yml](cwel_config.yml) flow configuration MiNiFi queries all Windows system events and puts them to the `C:\temp\` directory in flattened JSON format. +Using the [cwel_config.yml](cwel_config.yml)/[cwel_config.json](cwel_config.json) flow configuration MiNiFi queries all Windows system events and puts them to the `C:\temp\` directory in flattened JSON format. -The flow: ConsumeWindowsEventLog ➔ PutFile +The flow: [ConsumeWindowsEventLog](../PROCESSORS.md#consumewindowseventlog) ➔ [PutFile](../PROCESSORS.md#putfile) ### Reading System Performance Data -Using the [pdh_config.yml](pdh_config.yml) flow configuration MiNiFi reads CPU and Disk performance data through Windows' Performance Data Helper (PDH) component and puts the data to the `C:\temp\` directory in a compact JSON format. +Using the [pdh_config.yml](pdh_config.yml)/[pdh_config.json](pdh_config.json) flow configuration MiNiFi reads CPU and Disk performance data through Windows' Performance Data Helper (PDH) component and puts the data to the `C:\temp\` directory in a compact JSON format. -The flow: PerformanceDataMonitor ➔ PutFile +The flow: [PerformanceDataMonitor](../PROCESSORS.md#performancedatamonitor) ➔ [PutFile](../PROCESSORS.md#putfile) ## Linux Specific Processors ### Consume Systemd-Journald System Journal Messages -Using the [consumejournald_config.yml](consumejournald_config.yml) flow configuration MiNiFi reads systemd-journald journal messages and logs them on `info` level. +Using the [consumejournald_config.yml](consumejourn
Re: [PR] MINIFICPP-2309 Add JSON flow config examples [nifi-minifi-cpp]
fgerlits commented on code in PR #1740: URL: https://github.com/apache/nifi-minifi-cpp/pull/1740#discussion_r1535688898 ## examples/README.md: ## @@ -37,122 +43,158 @@ The following examples show simple flow configurations for several common use ca - [Upload Blob to Azure Storage](#upload-blob-to-azure-storage) - [Put Object in AWS S3 Bucket](#put-object-in-aws-s3-bucket) - [List and Fetch Content from AWS S3 Bucket](#list-and-fetch-content-from-aws-s3-bucket) + - [Merge, compress and upload files to Google Cloud Storage](#merge-compress-and-upload-files-to-google-cloud-storage) - [SQL Operations](#sql-operations) - [Query Database Table](#query-database-table) -- [ExecuteScript](#ExecuteScript) - - [Reverse flowfile's content with lua](#reverse-content-with-lua) - - [Reverse flowfile's content with python](#reverse-content-with-python) +- [ExecuteScript](#executescript) + - [Reverse Content with Scripts](#reverse-content-with-scripts) +- [Grafana Loki](#grafana-loki) + - [Send logs to Grafana Loki](#send-logs-to-grafana-loki) +- [MQTT Operations](#mqtt-operations) + - [Publish Message to MQTT Broker](#publish-message-to-mqtt-broker) +- [Network Operations](#network-operations) + - [Send Data to Remote Host through TCP](#send-data-to-remote-host-through-tcp) ## Filesystem Operations ### Getting File Data and Putting It in an Output Directory -Using the [getfile_putfile_config.yml](getfile_putfile_config.yml) flow configuration MiNiFi gets all files of minimum 1MB size from the `/tmp/getfile_dir` directory and puts them in the `/tmp/out_dir` output directory. +Using the [getfile_putfile_config.yml](getfile_putfile_config.yml)/[getfile_putfile_config.json](getfile_putfile_config.json) flow configuration MiNiFi gets all files of minimum 1MB size from the `/tmp/getfile_dir` directory and puts them in the `/tmp/out_dir` output directory. -The flow: GetFile ➔ success ➔ PutFile +The flow: [GetFile](../PROCESSORS.md#getfile) ➔ [PutFile](../PROCESSORS.md#putfile) ### Tailing a Single File -Using the [tailfile_config.yml](tailfile_config.yml) flow configuration MiNiFi tails a single file `/tmp/test_file.log` and creates flowfiles from every single line, then logs the flowfile attributes. +Using the [tailfile_config.yml](tailfile_config.yml)/[tailfile_config.json](tailfile_config.json) flow configuration MiNiFi tails a single file `/tmp/test_file.log` and creates flowfiles from every single line, then logs the flowfile attributes. -The flow: TailFile ➔ success ➔ LogAttribute +The flow: [TailFile](../PROCESSORS.md#tailfile) ➔ [LogAttribute](../PROCESSORS.md#logattribute) ## Windows Specific Processors ### Consuming Windows Event Logs -Using the [cwel_config.yml](cwel_config.yml) flow configuration MiNiFi queries all Windows system events and puts them to the `C:\temp\` directory in flattened JSON format. +Using the [cwel_config.yml](cwel_config.yml)/[cwel_config.json](cwel_config.json) flow configuration MiNiFi queries all Windows system events and puts them to the `C:\temp\` directory in flattened JSON format. -The flow: ConsumeWindowsEventLog ➔ PutFile +The flow: [ConsumeWindowsEventLog](../PROCESSORS.md#consumewindowseventlog) ➔ [PutFile](../PROCESSORS.md#putfile) ### Reading System Performance Data -Using the [pdh_config.yml](pdh_config.yml) flow configuration MiNiFi reads CPU and Disk performance data through Windows' Performance Data Helper (PDH) component and puts the data to the `C:\temp\` directory in a compact JSON format. +Using the [pdh_config.yml](pdh_config.yml)/[pdh_config.json](pdh_config.json) flow configuration MiNiFi reads CPU and Disk performance data through Windows' Performance Data Helper (PDH) component and puts the data to the `C:\temp\` directory in a compact JSON format. -The flow: PerformanceDataMonitor ➔ PutFile +The flow: [PerformanceDataMonitor](../PROCESSORS.md#performancedatamonitor) ➔ [PutFile](../PROCESSORS.md#putfile) ## Linux Specific Processors ### Consume Systemd-Journald System Journal Messages -Using the [consumejournald_config.yml](consumejournald_config.yml) flow configuration MiNiFi reads systemd-journald journal messages and logs them on `info` level. +Using the [consumejournald_config.yml](consumejournald_config.yml)/[consumejournald_config.json](consumejournald_config.json)/[consumejournald_config.nifi.schema.json](consumejournald_config.nifi.schema.json) flow configuration MiNiFi reads systemd-journald journal messages and logs them on `info` level. -The flow: ConsumeJournald ➔ LogAttribute +The flow: [ConsumeJournald](../PROCESSORS.md#consumejournald) ➔ [LogAttribute](../PROCESSORS.md#logattribute) ## HTTP Operations ### HTTP POST Invocation -Using the [http_post_config.yml](http_post_config.yml) flow configuration MiNiFi transfers flowfile data received from the GetFile processor by invoking an HTTP endpoint with POST method. +Using the [h