[jira] [Commented] (NIFI-2020) Enhance JoltTransformJSON processor to support custom transforms
[ https://issues.apache.org/jira/browse/NIFI-2020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15369374#comment-15369374 ] ASF GitHub Bot commented on NIFI-2020: -- Github user YolandaMDavis commented on the issue: https://github.com/apache/nifi/pull/564 @mattyb149 just pushed commits (no rebase so that you can easily see changes). I wanted to get your input on the stream/lambdas implementation I have in ClassLoaderUtils...not too happy with it :(. Unfortunately lambda's don't handle checked exceptions too well, so I'm forced right now to wrap in a RuntimeException when attempting to extract the URL from a URI (it throws MalformedURLException potentially) which is no bueno imo. Any thoughts on that or alternatives? > Enhance JoltTransformJSON processor to support custom transforms > > > Key: NIFI-2020 > URL: https://issues.apache.org/jira/browse/NIFI-2020 > Project: Apache NiFi > Issue Type: New Feature > Components: Core Framework >Affects Versions: 1.0.0 >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis > Fix For: 1.0.0 > > > Jolt supports additional custom transforms via fully-qualified Java > classnames. Would like to provide the ability to support custom > transformation (via drop in jars) for the Jolt Transform processor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] nifi pull request #483: NIFI-1899 - Introduce ExtractEmailAttachments and Ex...
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/483#discussion_r70169919 --- Diff: nifi-nar-bundles/nifi-email-bundle/nifi-email-processors/src/main/java/org/apache/nifi/processors/email/ExtractEmailAttachments.java --- @@ -0,0 +1,201 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.email; + +import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Properties; +import java.util.Set; +import java.util.Date; + +import javax.activation.DataSource; +import javax.mail.Address; +import javax.mail.MessagingException; +import javax.mail.Session; +import javax.mail.internet.MimeMessage; + +import org.apache.commons.io.IOUtils; +import org.apache.commons.lang3.StringUtils; +import org.apache.commons.mail.util.MimeMessageParser; + + +import org.apache.nifi.annotation.behavior.EventDriven; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.InputRequirement.Requirement; +import org.apache.nifi.annotation.behavior.SideEffectFree; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.flowfile.attributes.CoreAttributes; +import org.apache.nifi.logging.ComponentLog; +import org.apache.nifi.processor.AbstractProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.ProcessorInitializationContext; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.io.InputStreamCallback; +import org.apache.nifi.processor.io.OutputStreamCallback; +import org.apache.nifi.stream.io.BufferedInputStream; + + + + +@EventDriven +@SideEffectFree +@Tags({"split", "email"}) +@InputRequirement(Requirement.INPUT_REQUIRED) +@CapabilityDescription("Extract attachments from a mime formatted email file, splitting them into individual flowfiles.") +@WritesAttributes({ +@WritesAttribute(attribute = "filename ", description = "The filename of the attachment"), +@WritesAttribute(attribute = "email.attachment.parent.filename ", description = "The filename of the parent FlowFile"), +@WritesAttribute(attribute = "email.attachment.parent.uuid", description = "The UUID of the original FlowFile."), +@WritesAttribute(attribute = "mime.type", description = "The mime type of the attachment.")}) + --- End diff -- This annotation gives the Data Flow Manager to the option to batch session commits together in order to increase throughput (the slider in the "schedule" tab). By adding it you aren't making it less safe, you're merely saying "this processor is safe to roll back multiple sessions if necessary in order to give it higher throughput overall". Since this processor merely extracts information within NiFi and doesn't touch another system, it is safe to support batching. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-2020) Enhance JoltTransformJSON processor to support custom transforms
[ https://issues.apache.org/jira/browse/NIFI-2020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15369207#comment-15369207 ] ASF GitHub Bot commented on NIFI-2020: -- Github user YolandaMDavis commented on a diff in the pull request: https://github.com/apache/nifi/pull/564#discussion_r70169819 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/JoltTransformJSON.java --- @@ -95,6 +99,24 @@ .required(false) .build(); +public static final PropertyDescriptor CUSTOM_CLASS = new PropertyDescriptor.Builder() +.name("jolt-custom-class") +.displayName("Custom Transformation Class Name") +.description("Fully Qualified Class Name for Custom Transformation Module Directory should be specified") --- End diff -- I just took that second part out and will elaborate in the additionalDetails page and a bit on the custom module directory description. > Enhance JoltTransformJSON processor to support custom transforms > > > Key: NIFI-2020 > URL: https://issues.apache.org/jira/browse/NIFI-2020 > Project: Apache NiFi > Issue Type: New Feature > Components: Core Framework >Affects Versions: 1.0.0 >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis > Fix For: 1.0.0 > > > Jolt supports additional custom transforms via fully-qualified Java > classnames. Would like to provide the ability to support custom > transformation (via drop in jars) for the Jolt Transform processor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] nifi pull request #564: NIFI-2020 - Enhance JoltTransformJSON processor to s...
Github user YolandaMDavis commented on a diff in the pull request: https://github.com/apache/nifi/pull/564#discussion_r70169819 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/JoltTransformJSON.java --- @@ -95,6 +99,24 @@ .required(false) .build(); +public static final PropertyDescriptor CUSTOM_CLASS = new PropertyDescriptor.Builder() +.name("jolt-custom-class") +.displayName("Custom Transformation Class Name") +.description("Fully Qualified Class Name for Custom Transformation Module Directory should be specified") --- End diff -- I just took that second part out and will elaborate in the additionalDetails page and a bit on the custom module directory description. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #564: NIFI-2020 - Enhance JoltTransformJSON processor to s...
Github user YolandaMDavis commented on a diff in the pull request: https://github.com/apache/nifi/pull/564#discussion_r70169616 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-jolt-transform-json-ui/src/main/java/org/apache/nifi/web/standard/api/transformjson/TransformJSONResource.java --- @@ -84,6 +84,36 @@ public Response executeSpec(JoltSpecificationDTO specificationDTO) { } +protected Transform getTransformation(JoltSpecificationDTO specificationDTO) throws Exception{ + +Object specJson = getSpecificationJsonObject(specificationDTO.getSpecification()); +String transformName = specificationDTO.getTransform(); +String modules = specificationDTO.getModules(); + +ClassLoader customClassLoader = null; +Transform transform ; +if(modules != null && !modules.isEmpty()){ +customClassLoader = ClassLoaderUtils.getCustomClassLoader(specificationDTO.getModules(),this.getClass().getClassLoader(), getJarFilenameFilter()); +} + +if(transformName.equals("jolt-transform-custom")) { +transform = TransformFactory.getCustomTransform(customClassLoader,specificationDTO.getCustomClass(), specJson); --- End diff -- agreed this code should be improved in general, which I'll take care of. The case I should be covering here is the fact that customClassLoader could be used with a custom transform or other transforms (such as a chain which can contain custom transformations). The code assumes that if you selected custom you should have a module set which isn't good to do. I'll update and get it out here. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (NIFI-2020) Enhance JoltTransformJSON processor to support custom transforms
[ https://issues.apache.org/jira/browse/NIFI-2020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15369201#comment-15369201 ] ASF GitHub Bot commented on NIFI-2020: -- Github user YolandaMDavis commented on a diff in the pull request: https://github.com/apache/nifi/pull/564#discussion_r70169616 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-jolt-transform-json-ui/src/main/java/org/apache/nifi/web/standard/api/transformjson/TransformJSONResource.java --- @@ -84,6 +84,36 @@ public Response executeSpec(JoltSpecificationDTO specificationDTO) { } +protected Transform getTransformation(JoltSpecificationDTO specificationDTO) throws Exception{ + +Object specJson = getSpecificationJsonObject(specificationDTO.getSpecification()); +String transformName = specificationDTO.getTransform(); +String modules = specificationDTO.getModules(); + +ClassLoader customClassLoader = null; +Transform transform ; +if(modules != null && !modules.isEmpty()){ +customClassLoader = ClassLoaderUtils.getCustomClassLoader(specificationDTO.getModules(),this.getClass().getClassLoader(), getJarFilenameFilter()); +} + +if(transformName.equals("jolt-transform-custom")) { +transform = TransformFactory.getCustomTransform(customClassLoader,specificationDTO.getCustomClass(), specJson); --- End diff -- agreed this code should be improved in general, which I'll take care of. The case I should be covering here is the fact that customClassLoader could be used with a custom transform or other transforms (such as a chain which can contain custom transformations). The code assumes that if you selected custom you should have a module set which isn't good to do. I'll update and get it out here. > Enhance JoltTransformJSON processor to support custom transforms > > > Key: NIFI-2020 > URL: https://issues.apache.org/jira/browse/NIFI-2020 > Project: Apache NiFi > Issue Type: New Feature > Components: Core Framework >Affects Versions: 1.0.0 >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis > Fix For: 1.0.0 > > > Jolt supports additional custom transforms via fully-qualified Java > classnames. Would like to provide the ability to support custom > transformation (via drop in jars) for the Jolt Transform processor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2020) Enhance JoltTransformJSON processor to support custom transforms
[ https://issues.apache.org/jira/browse/NIFI-2020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15369186#comment-15369186 ] ASF GitHub Bot commented on NIFI-2020: -- Github user YolandaMDavis commented on a diff in the pull request: https://github.com/apache/nifi/pull/564#discussion_r70169425 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/JoltTransformJSON.java --- @@ -95,6 +99,24 @@ .required(false) .build(); +public static final PropertyDescriptor CUSTOM_CLASS = new PropertyDescriptor.Builder() +.name("jolt-custom-class") +.displayName("Custom Transformation Class Name") +.description("Fully Qualified Class Name for Custom Transformation Module Directory should be specified") +.required(false) +.expressionLanguageSupported(false) +.addValidator(StandardValidators.NON_EMPTY_VALIDATOR) +.build(); + +public static final PropertyDescriptor MODULES = new PropertyDescriptor.Builder() +.name("jolt-custom-modules") +.displayName("Custom Module Directory") +.description("Comma-separated list of paths to files and/or directories which contain modules containing custom transformations.") +.required(false) +.expressionLanguageSupported(false) --- End diff -- @mattyb149 please see note above > Enhance JoltTransformJSON processor to support custom transforms > > > Key: NIFI-2020 > URL: https://issues.apache.org/jira/browse/NIFI-2020 > Project: Apache NiFi > Issue Type: New Feature > Components: Core Framework >Affects Versions: 1.0.0 >Reporter: Yolanda M. Davis >Assignee: Yolanda M. Davis > Fix For: 1.0.0 > > > Jolt supports additional custom transforms via fully-qualified Java > classnames. Would like to provide the ability to support custom > transformation (via drop in jars) for the Jolt Transform processor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2179) ExecuteSQL and PutSQL can't assignt controller service
[ https://issues.apache.org/jira/browse/NIFI-2179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15369176#comment-15369176 ] Joseph Percivall commented on NIFI-2179: Thank you [~akell] for following up on this issue and closing it. If you need any help resolving custom code problems, please feel free to send a ticket to "d...@nifi.apache.org". > ExecuteSQL and PutSQL can't assignt controller service > -- > > Key: NIFI-2179 > URL: https://issues.apache.org/jira/browse/NIFI-2179 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 0.6.0, 0.6.1 > Environment: Windows 7 , Linux Red Hat Enterprise >Reporter: Alexander Kell >Priority: Critical > Labels: ControllerService, SQL > Attachments: Wrong_controllerservicetype.png > > > ExecuteSQL Processor is broken, it is not possible to assignt a neccesary > controllerservice connection. > I created a working SQL Controller , it is not possible to assignt this > controller to created ExecuteSQL Processor , even if i create one > controllerservice out of property page , it is not possible for the processor > to use this service. > It has probably nothing to do with changes in processor code ... it has > probably spomething to do with controller service ... i just tested it with > 0.5.0 and it works ... it is broken since 0.6.0 > Maybe it has something to do with NIFI-1800: Tie Controller Services to > Process Groups. > or > NIFI-1994: Fixed issue with Controller Service Fully Qualified Class -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] nifi pull request #624: NIFI-1688 PostHTTP processor now uses protocol speci...
GitHub user alopresto opened a pull request: https://github.com/apache/nifi/pull/624 NIFI-1688 PostHTTP processor now uses protocol specified by SSLContextService You can merge this pull request into a Git repository by running: $ git pull https://github.com/alopresto/nifi NIFI-1688 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/624.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #624 commit 4222d354c14d939f429fb6176cb3b9499b4861ad Author: Andy LoPrestoDate: 2016-06-21T22:31:56Z NIFI-1688 Added test skeleton. commit dcfb41e48738d101e4676b60300ffa33c8c68c07 Author: Andy LoPresto Date: 2016-06-21T22:32:54Z NIFI-1688 Cleaned up unnecessary imports. commit 8ef7bfd9af60b78db05f8620a5ea0a8b0dc9f44b Author: Andy LoPresto Date: 2016-06-23T03:52:37Z NIFI-1688 Implemented integration test that generates key pair, inserts into and persists keystore, starts embedded HTTPS Jetty server, connects, and verifies response. Currently only TLSv1.2 connections are successful because of the overlap of cipher suites. Will manually insert cipher suites into server for TLSv1 and TLSv1.1 support. commit de1c9d1680033cc38a0c3b330da44866ae91e07d Author: Andy LoPresto Date: 2016-06-23T15:13:08Z NIFI-1688 Added debug information for supported cipher suites. commit a7ec6814878974033a944f9635348372f79b4832 Author: Andy LoPresto Date: 2016-06-24T01:01:28Z NIFI-1688 Added test Groovlet for handling POST requests. Modified construction of SSLSocketFactory to avoid hardcoding supported protocol. Added integration tests (2 of 4 pass -- TLSv1.2 is supported on my machine but TLSv1 and TLSv1.1 are not). commit 4b3955155a0cd8631594cc38065266f7a66d8058 Author: Andy LoPresto Date: 2016-06-24T03:43:04Z NIFI-1688 Resolved issue in PostHTTP -- now uses SSLContextService's protocol setting. Tests pass (require cleanup). Previously, dynamically-generated keystores with only RSA certificates were not acceptable for TLSv1 or TLSv1.1 connections which required DSA/DSS cipher suites for some reason. commit 66472854b7c01ec7a38d15507860a1320c1ac062 Author: Andy LoPresto Date: 2016-06-24T03:48:16Z NIFI-1688 Tests pass (cleanup still required). commit f0fd38e4d3a0cd76459a343930e623c1cd667b10 Author: Andy LoPresto Date: 2016-06-29T22:41:32Z NIFI-1688 Tests pass (cleanup still required). commit 850fc8183820a95570df5187c0856060ca10f112 Author: Andy LoPresto Date: 2016-07-09T06:13:35Z NIFI-1688 Finished integration tests for PostHTTP processor. commit c741043ee901e92858fcd08297bb8758139da6d1 Author: Andy LoPresto Date: 2016-07-09T06:29:32Z NIFI-1688 Removed legacy comments. Added ASF license to Groovlet handlers for test. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Resolved] (NIFI-2179) ExecuteSQL and PutSQL can't assignt controller service
[ https://issues.apache.org/jira/browse/NIFI-2179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Kell resolved NIFI-2179. -- Resolution: Not A Problem Fix Version/s: (was: 1.0.0) It had something to do with a custom nar org.apache.nifi.nar.ExtensionManager Attempt was made to load org.apache.nifi.dbcp.DBCPConnectionPool from org.apache.nifi.nar.NarClassLoader[.\work\nar\extensions\nifi-dbcp-service-nar-0.6.1.nar-unpacked] but that class name is already loaded/registered from org.apache.nifi.nar.NarClassLoader[.\work\nar\extensions\nifi-hsdg-nar-1.0.nar-unpacked]. This may cause unpredictable behavior. Order of NARs is not guaranteed. > ExecuteSQL and PutSQL can't assignt controller service > -- > > Key: NIFI-2179 > URL: https://issues.apache.org/jira/browse/NIFI-2179 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 0.6.0, 0.6.1 > Environment: Windows 7 , Linux Red Hat Enterprise >Reporter: Alexander Kell >Priority: Critical > Labels: ControllerService, SQL > Attachments: Wrong_controllerservicetype.png > > > ExecuteSQL Processor is broken, it is not possible to assignt a neccesary > controllerservice connection. > I created a working SQL Controller , it is not possible to assignt this > controller to created ExecuteSQL Processor , even if i create one > controllerservice out of property page , it is not possible for the processor > to use this service. > It has probably nothing to do with changes in processor code ... it has > probably spomething to do with controller service ... i just tested it with > 0.5.0 and it works ... it is broken since 0.6.0 > Maybe it has something to do with NIFI-1800: Tie Controller Services to > Process Groups. > or > NIFI-1994: Fixed issue with Controller Service Fully Qualified Class -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2179) ExecuteSQL and PutSQL can't assignt controller service
[ https://issues.apache.org/jira/browse/NIFI-2179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368959#comment-15368959 ] Alexander Kell commented on NIFI-2179: -- I just rechecked on my home computer withour custome nar , it works with 0.61 and 0.6.0 ... so it has probably something to do with our custom processor nar ... i just figured our what went wrong ... org.apache.nifi.nar.ExtensionManager Attempt was made to load org.apache.nifi.dbcp.DBCPConnectionPool from org.apache.nifi.nar.NarClassLoader[.\work\nar\extensions\nifi-dbcp-service-nar-0.6.1.nar-unpacked] but that class name is already loaded/registered from org.apache.nifi.nar.NarClassLoader[.\work\nar\extensions\nifi-hsdg-nar-1.0.nar-unpacked]. This may cause unpredictable behavior. Order of NARs is not guaranteed. > ExecuteSQL and PutSQL can't assignt controller service > -- > > Key: NIFI-2179 > URL: https://issues.apache.org/jira/browse/NIFI-2179 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 0.6.0, 0.6.1 > Environment: Windows 7 , Linux Red Hat Enterprise >Reporter: Alexander Kell >Priority: Critical > Labels: ControllerService, SQL > Fix For: 1.0.0 > > Attachments: Wrong_controllerservicetype.png > > > ExecuteSQL Processor is broken, it is not possible to assignt a neccesary > controllerservice connection. > I created a working SQL Controller , it is not possible to assignt this > controller to created ExecuteSQL Processor , even if i create one > controllerservice out of property page , it is not possible for the processor > to use this service. > It has probably nothing to do with changes in processor code ... it has > probably spomething to do with controller service ... i just tested it with > 0.5.0 and it works ... it is broken since 0.6.0 > Maybe it has something to do with NIFI-1800: Tie Controller Services to > Process Groups. > or > NIFI-1994: Fixed issue with Controller Service Fully Qualified Class -- This message was sent by Atlassian JIRA (v6.3.4#6332)