martinzink commented on code in PR #2186:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2186#discussion_r3317730408
##########
minifi_rust/extensions/minifi_rs_playground/features/steps/steps.py:
##########
@@ -0,0 +1,45 @@
+from behave import then, when, given
+import humanfriendly
+
+from minifi_behave.steps import checking_steps # noqa: F401
+from minifi_behave.steps import configuration_steps # noqa: F401
+from minifi_behave.steps import core_steps # noqa: F401
+from minifi_behave.steps import flow_building_steps # noqa: F401
+from minifi_behave.core.helpers import wait_for_condition
+from minifi_behave.core.minifi_test_context import MinifiTestContext
+
+
+@when("the MiNiFi instance is started without assertions")
Review Comment:
We should probably move these into the framework later, but I didnt want to
touch anything outside of minifi_rust dir
##########
cmake/MiNiFiOptions.cmake:
##########
@@ -116,6 +116,7 @@ add_minifi_option(ENABLE_EXECUTE_PROCESS "Enable
ExecuteProcess processor" OFF)
add_minifi_option(ENABLE_CONTROLLER "Enables the build of MiNiFi controller
binary." ON)
add_minifi_option(ENABLE_LLAMACPP "Enables llama.cpp support." ON)
add_minifi_option(ENABLE_OPC "Instructs the build system to enable the OPC
extension" ON)
+add_minifi_option(MINIFI_RUST "Enables the build of rust based extension." ON)
Review Comment:
It might be better to keep it OFF since it requires rust compiler. (also
just we probably need to handle this in bootstrap)
##########
minifi_rust/extensions/minifi_rs_playground/src/controller_services/lorem_ipsum_controller_service/properties.rs:
##########
@@ -0,0 +1,13 @@
+use minifi_native::{Property, StandardPropertyValidator};
+
+pub(crate) const LENGTH: Property = Property {
Review Comment:
We should discuss what kind of style we want to follow. Single file per
processor or splitting the properties, relationships, processor definitions
into the mods folder
##########
minifi_rust/extensions/minifi_rs_playground/src/processors/put_file.rs:
##########
@@ -0,0 +1,239 @@
+use crate::processors::put_file::relationships::{FAILURE, SUCCESS};
+use minifi_native::macros::{ComponentIdentifier, DefaultMetrics,
NoAdvancedProcessorFeatures};
+use minifi_native::{
+ FlowFileTransform, GetAttribute, GetControllerService, GetProperty,
InputStream, Logger,
+ MinifiError, Schedule, TransformedFlowFile, trace, warn,
+};
+use std::path::{Path, PathBuf};
+use strum_macros::{Display, EnumString, IntoStaticStr, VariantNames};
+use walkdir::WalkDir;
+
+mod properties;
+mod relationships;
+#[cfg(unix)]
+mod unix_only_properties;
+
+#[derive(Debug, Clone, Copy, PartialEq, Display, EnumString, VariantNames,
IntoStaticStr)]
+#[strum(serialize_all = "camelCase", const_into_str)]
+enum ConflictResolutionStrategy {
+ Fail,
+ Replace,
+ Ignore,
+}
+
+#[cfg(unix)]
+#[derive(Debug)]
+struct PutFileUnixPermissions {
+ file_permissions: Option<std::fs::Permissions>,
+ directory_permissions: Option<std::fs::Permissions>,
+}
+
+#[cfg(unix)]
+impl PutFileUnixPermissions {
+ fn set_directory_permissions(&self, path: &Path) -> std::io::Result<()> {
+ if let Some(permissions) = self.directory_permissions.as_ref().map(|p|
p.clone()) {
+ return std::fs::set_permissions(path, permissions);
+ }
+ Ok(())
+ }
+
+ fn set_file_permissions(&self, file: &Path) -> std::io::Result<()> {
+ if let Some(permissions) = self.file_permissions.as_ref().map(|p|
p.clone()) {
+ return std::fs::set_permissions(file, permissions);
+ }
+ Ok(())
+ }
+}
+
+#[cfg(windows)]
+#[derive(Debug)]
+struct PutFileUnixPermissions {}
+
+#[cfg(windows)]
+impl PutFileUnixPermissions {
+ fn set_directory_permissions(&self, _path: &Path) -> std::io::Result<()> {
+ Ok(())
+ }
+
+ fn set_file_permissions(&self, _file: &Path) -> std::io::Result<()> {
+ Ok(())
+ }
+}
+
+#[derive(Debug, ComponentIdentifier, DefaultMetrics,
NoAdvancedProcessorFeatures)]
+pub(crate) struct PutFileRs {
+ conflict_resolution_strategy: ConflictResolutionStrategy,
+ try_make_dirs: bool,
+ maximum_file_count: Option<u64>,
+ unix_permissions: PutFileUnixPermissions,
+}
+
+impl PutFileRs {
+ pub(crate) fn directory_is_full(&self, p0: &Path) -> bool {
+ if let Some(max_file_count) = self.maximum_file_count
+ && let Some(parent) = p0.parent()
+ {
+ parent.exists()
+ && WalkDir::new(parent)
+ .into_iter()
+ .filter_map(Result::ok)
+ .filter(|e| e.file_type().is_file())
+ .count()
+ >= max_file_count as usize
+ } else {
+ false
+ }
+ }
+
+ fn get_destination_path<Ctx>(context: &Ctx) -> Result<PathBuf, MinifiError>
+ where
+ Ctx: GetProperty + GetAttribute,
+ {
+ let directory = context
+ .get_property(&properties::DIRECTORY)?
+ .expect("required property");
+
+ let file_name = context
+ .get_attribute("filename")?
+ .unwrap_or("foo.txt".to_string()); // TODO fallback to UUID
Review Comment:
The api didnt support getting flowfile IDs until recently so I've hardcoded
something here, this should be fixed and fallback to flowfile uuid in a
followup PR
##########
minifi_rust/extensions/minifi_rs_playground/src/processors/get_file.rs:
##########
@@ -0,0 +1,304 @@
+use crate::processors::get_file::output_attributes::{
Review Comment:
This is the reimplementation of the already existing standard GetFile
processor
##########
minifi_rust/extensions/minifi_rs_playground/Cargo.toml:
##########
@@ -0,0 +1,20 @@
+[package]
+name = "minifi_rs_playground"
Review Comment:
This extension contains reimplentations of already existing processors and
test only processors. These are not intended for production usage, therefore
they probably don't require that much scrutiny in the review process.
If for some reason we ever want to ship something from this extension than
that processor will have to be extracted and rereviewed.
##########
minifi_rust/minifi_rs_behave/linux_build.sh:
##########
@@ -0,0 +1,94 @@
+#!/usr/bin/env bash
Review Comment:
This shell script was written partly using llm
##########
minifi_rust/minifi_native_macros/src/lib.rs:
##########
@@ -0,0 +1,48 @@
+use proc_macro::TokenStream;
+use quote::quote;
+use syn::{DeriveInput, parse_macro_input};
+
+#[proc_macro_derive(ComponentIdentifier)]
+pub fn derive_component_identifier(input: TokenStream) -> TokenStream {
+ let input = parse_macro_input!(input as DeriveInput);
+ let name = &input.ident;
+ let name_str = name.to_string();
+
+ let expanded = quote! {
+ impl ::minifi_native::ComponentIdentifier for #name {
+ const CLASS_NAME: &'static str = concat!(module_path!(), "::",
#name_str);
+ const GROUP_NAME: &'static str = env!("CARGO_PKG_NAME");
+ const VERSION: &'static str = env!("CARGO_PKG_VERSION");
+ }
+ };
+
+ TokenStream::from(expanded)
+}
+
+#[proc_macro_derive(DefaultMetrics)]
+pub fn derive_default_metrics(input: TokenStream) -> TokenStream {
+ let input = parse_macro_input!(input as DeriveInput);
+ let name = &input.ident;
+ let expanded = quote! {
+ impl ::minifi_native::CalculateMetrics for #name {
+ fn calculate_metrics(&self) -> Vec<(String, f64)> {
+ vec![]
+ }
+ }
+ };
+
+ TokenStream::from(expanded)
+}
+
+#[proc_macro_derive(NoAdvancedProcessorFeatures)]
+pub fn derive_no_advanced_processor_features(input: TokenStream) ->
TokenStream {
Review Comment:
The API was simplified so there is only a single advanced prcessor feature
(get_trigger_when_empty) so we should probably want to rename this
##########
minifi_rust/minifi_rs_behave/Cargo.toml:
##########
@@ -0,0 +1,12 @@
+[package]
+name = "minifi_rs_behave"
+version = "0.1.0"
+edition = "2024"
+
+[dependencies]
+glob = "0.3.3"
+minifi_rs_playground = { path = "../extensions/minifi_rs_playground" }
+minifi_native_sys = { path = "../minifi_native_sys" }
Review Comment:
We should probably remove this dependancy
##########
minifi_rust/extensions/minifi_rs_playground/src/processors/generate_flow_file.rs:
##########
@@ -0,0 +1,170 @@
+use minifi_native::macros::{ComponentIdentifier, NoAdvancedProcessorFeatures};
Review Comment:
This is the reimplementation of the already existing standard
GenerateFlowFile processor
##########
minifi_rust/minifi_rs_behave/Cargo.toml:
##########
@@ -0,0 +1,12 @@
+[package]
+name = "minifi_rs_behave"
+version = "0.1.0"
+edition = "2024"
+
+[dependencies]
+glob = "0.3.3"
+minifi_rs_playground = { path = "../extensions/minifi_rs_playground" }
+minifi_native_sys = { path = "../minifi_native_sys" }
Review Comment:
we currently list minifi_rs_playground as a dependancy which is technically
untrue (see unused_crate_dependencies = "allow"), but this allows a
linux/windows user to run cargo behave and have all the required artifacts be
built before behave tests.
(this also means we are building everything twice in the CI (once bare metal
due to dependency and once in docker due to CI using alpine))
##########
minifi_rust/extensions/minifi_rs_playground/src/processors/count_actual_logging.rs:
##########
@@ -0,0 +1,57 @@
+use minifi_native::macros::{ComponentIdentifier, DefaultMetrics,
NoAdvancedProcessorFeatures};
Review Comment:
This is used to test lazy logging
##########
minifi_rust/extensions/minifi_rs_playground/src/controller_services/mod.rs:
##########
@@ -0,0 +1,2 @@
+pub(crate) mod dummy_controller_service;
+pub(crate) mod lorem_ipsum_controller_service;
Review Comment:
test controller services, (multiple so we can test setting and quering the
wrong one)
##########
minifi_rust/extensions/minifi_rs_playground/src/processors/asciify_german.rs:
##########
@@ -0,0 +1,67 @@
+use crate::processors::asciify_german::relationships::FAILURE;
Review Comment:
This is used to test streaming flow file transforms, with changing flowfiles
sizes
##########
minifi_rust/extensions/minifi_rs_playground/minifi_rs_playground.md:
##########
@@ -0,0 +1,260 @@
+<!--
Review Comment:
Autogenerated via generate_docs.sh
##########
minifi_rust/extensions/minifi_rs_playground/src/processors/log_attribute.rs:
##########
@@ -0,0 +1,160 @@
+use crate::processors::log_attribute::properties::{FLOW_FILES_TO_LOG,
LOG_LEVEL, LOG_PAYLOAD};
Review Comment:
This is the reimplementation of the already existing standard LogAttribute
processor
##########
minifi_rust/generate_docs/generate_docs.dockerfile:
##########
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+ARG BASE_IMAGE="apacheminificpp:behave"
+
+FROM ${BASE_IMAGE} AS builder
+LABEL maintainer="Martin Zink <[email protected]>"
+
+COPY ./target/release/libminifi_rs_playground.so
/opt/minifi/minifi-current/extensions
Review Comment:
Its hardcoded currently we should probably want something more clever
##########
minifi_rust/minifi_rs_behave/debian.dockerfile:
##########
@@ -0,0 +1,26 @@
+FROM rust:slim-bullseye AS chef
+RUN apt-get update && apt-get install -y clang lld pkg-config curl tar
+RUN cargo install cargo-chef
+WORKDIR /app
+
+FROM chef AS planner
+COPY . .
+RUN cargo chef prepare --recipe-path recipe.json
+
+FROM chef AS builder
+ARG MINIFI_SDK_PATH
+ENV MINIFI_SDK_PATH=${MINIFI_SDK_PATH}
+
+COPY --from=planner /app/recipe.json recipe.json
+
+# Conditionally copy the local SDK files from the target directory
+COPY target/.docker_sd[k] /app/target/.docker_sdk/
+COPY target/.docker_sdk.zi[p] /app/target/
+
+RUN cargo chef cook --release --recipe-path recipe.json
+COPY . .
+RUN cargo build --release
Review Comment:
this simply uses docker pattern matching so it wont fail if the target is
missing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
