[jira] [Updated] (MINIFI-500) How to send the data from MiNiFi to NiFi
[ https://issues.apache.org/jira/browse/MINIFI-500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Witt updated MINIFI-500: --- Fix Version/s: (was: 1.0.0) > How to send the data from MiNiFi to NiFi > > > Key: MINIFI-500 > URL: https://issues.apache.org/jira/browse/MINIFI-500 > Project: Apache NiFi MiNiFi > Issue Type: Wish > Components: Data Transmission >Affects Versions: 0.4.0 > Environment: Ubuntu >Reporter: Bikarm Rout >Priority: Major > Labels: test > Attachments: MiniFi_To_Nifi.odt, template.zip > > > Hi , > > I am trying to send the data from MiNiFi to NiFi . I followed the below steps > mentioned in the below URL to send the data from MinIFi to Nifi but it' not > working for me . > > [https://nifi.apache.org/minifi/getting-started.html] > > NB: I downloaded the ubuntu libraries to send the data from MiNiFi to NiFi. > > Kindly advise. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MINIFI-422) Incorporate MiNiFi Java functionality as a specialized assembly of NiFI
[ https://issues.apache.org/jira/browse/MINIFI-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16302186#comment-16302186 ] Joseph Witt commented on MINIFI-422: big +1 from me and happy to work with you on this. > Incorporate MiNiFi Java functionality as a specialized assembly of NiFI > > > Key: MINIFI-422 > URL: https://issues.apache.org/jira/browse/MINIFI-422 > Project: Apache NiFi MiNiFi > Issue Type: Task >Reporter: Aldrin Piri > > At its core the Java implementation of MiNiFi has largely been a core body of > NiFi core libraries in a separate assembly with some additional extension > points, namely those of configuration (via YAML) and configuration change > listeners. > Due to working with some of the internals of NiFi that are not exactly meant > for external consumption, there has been a certain impedance with each > successive release to make use of the latest and greatest. > This ticket is to investigate and consider the incorporation of MiNiFi Java > into the NiFi code base in a manner as highlighted above, extending/adapting > the core libraries, providing some additional extension points, and then > generating a custom assembly. > The idea is that in lieu of duplicating bits of code and providing > workarounds around some of the internal APIs we can have a more streamlined > build and keep these items in lockstep with the core NiFi libraries being > more aware of changes that MiNiFi is inherently dependent upon. > To our users, there should be little perceptible change. The core means of > interaction should remain while providing a similar footprint. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (MINIFI-421) The minifi-toolkit for converting templates to config.yml is misaligned for identifiers for s2s
Joseph Witt created MINIFI-421: -- Summary: The minifi-toolkit for converting templates to config.yml is misaligned for identifiers for s2s Key: MINIFI-421 URL: https://issues.apache.org/jira/browse/MINIFI-421 Project: Apache NiFi MiNiFi Issue Type: Bug Affects Versions: 0.3.0 Reporter: Joseph Witt noticed during RC3 validation. If I create a simple flow to send to an RPG port and create a template then do the config transform from the toolkit it uses the original identifier instead of the new 'target identifier' so s2s requires manual editing to work until the transformer is fixed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (MINIFI-415) Bundle version number should not be compared as a simple String
[
https://issues.apache.org/jira/browse/MINIFI-415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16288611#comment-16288611
]
Joseph Witt commented on MINIFI-415:
I think we need to take a consistent approach between NiFi and MiniFi in
regards to how we automatically map to a given bundle. Have you evaluated that?
> Bundle version number should not be compared as a simple String
> ---
>
> Key: MINIFI-415
> URL: https://issues.apache.org/jira/browse/MINIFI-415
> Project: Apache NiFi MiNiFi
> Issue Type: Bug
> Components: Agent Configuration/Installation
>Affects Versions: 0.3.0
>Reporter: Koji Kawamura
>Priority: Minor
>
> MINIFI-408 added support for picking the latest bundle version automatically
> if there are multiple versions for the same Nar. However, it compares version
> as simple Strings and may not be able to pick the latest one semantically.
> https://github.com/apache/nifi-minifi/pull/99/files#diff-c7d8398db8540d6e85f1a9207438ebddR138
> Following code shows the problematic inputs and possible solution. Current
> implementation picks "1.0.9", and using Version class, it can pick "1.0.12".
> {code}
> class Test {
> public static void main(String[] args) throws Exception {
> Set componentToEnrichBundleVersions = new HashSet<>();
> componentToEnrichBundleVersions.add("1.0.0");
> componentToEnrichBundleVersions.add("1.0.5");
> componentToEnrichBundleVersions.add("1.0.9");
> componentToEnrichBundleVersions.add("1.0.11-SNAPSHOT");
> componentToEnrichBundleVersions.add("1.0.12");
> // Current implementation
> final String bundleVersion =
> componentToEnrichBundleVersions.stream().sorted()
> .reduce((version, otherVersion) -> otherVersion).orElse(null);
> // Suggestion
> final Version latestVersion =
> componentToEnrichBundleVersions.stream().map(Version::fromString).sorted()
> .reduce((version, otherVersion) -> otherVersion).orElse(null);
> System.out.println(bundleVersion);
> System.out.println(latestVersion.toVersionString());
> }
> }
> class Version implements Comparable {
> private static Pattern P =
> Pattern.compile("^([\\d]+)\\.([\\d]+)\\.([\\d]+)([^\\d]*)$");
> final int major;
> final int minor;
> final int patch;
> final String opt;
> public static Version fromString(String s) {
> final Matcher matcher = P.matcher(s);
> if (matcher.matches()) {
> return new Version(
> Integer.parseInt(matcher.group(1)),
> Integer.parseInt(matcher.group(2)),
> Integer.parseInt(matcher.group(3)),
> matcher.group(4));
> }
> throw new IllegalArgumentException("Unknown version pattern " + s);
> }
> private Version(int major, int minor, int patch, String opt) {
> this.major = major;
> this.minor = minor;
> this.patch = patch;
> this.opt = opt;
> }
> @Override
> public int compareTo(@NotNull Version o) {
> final int ma = new Integer(major).compareTo(o.major);
> if (ma != 0) {
> return ma;
> }
> final int mi = new Integer(minor).compareTo(o.minor);
> if (mi != 0) {
> return mi;
> }
> final int pa = new Integer(patch).compareTo(o.patch);
> if (pa != 0) {
> return pa;
> }
> final String o1 = opt != null ? opt : "";
> final String o2 = o.opt != null ? o.opt : "";
> final int op = o1.compareTo(o2);
> return op;
> }
> public String toVersionString() {
> return String.format("%d.%d.%d%s", major, minor, patch, opt);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (MINIFI-353) warning building docker - groupadd: GID '50' already exists
[ https://issues.apache.org/jira/browse/MINIFI-353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Witt updated MINIFI-353: --- Fix Version/s: 0.3.0 > warning building docker - groupadd: GID '50' already exists > --- > > Key: MINIFI-353 > URL: https://issues.apache.org/jira/browse/MINIFI-353 > Project: Apache NiFi MiNiFi > Issue Type: Bug > Components: Docker >Affects Versions: 0.1.0 >Reporter: Jim Zucker >Assignee: Aldrin Piri > Fix For: 0.3.0 > > > Removing intermediate container fcbf6b637cf8 > Step 9/14 : RUN groupadd -g $GID minifi || groupmod -n minifi `getent group > $GID | cut -d: -f1` > ---> Running in 46c966e2c6df > * groupadd: GID '50' already exists * > ---> 1f9485499053 > Removing intermediate container 46c966e2c6df > Step 10/14 : RUN useradd --shell /bin/b -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (MINIFI-348) Schema validation error parsing Flow Configuration
[
https://issues.apache.org/jira/browse/MINIFI-348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Witt updated MINIFI-348:
---
Fix Version/s: 0.3.0
> Schema validation error parsing Flow Configuration
> --
>
> Key: MINIFI-348
> URL: https://issues.apache.org/jira/browse/MINIFI-348
> Project: Apache NiFi MiNiFi
> Issue Type: Bug
> Components: Processing Configuration
>Affects Versions: 0.2.0
> Environment: Rasbian Jessie (current version)
>Reporter: Nathan Green
>Assignee: Aldrin Piri
>Priority: Minor
> Fix For: 0.3.0
>
> Attachments: EdgeCollect.xml, config.yml, flow.xml, minifi-app.log,
> minifi-bootstrap.log
>
>
> Using Minifi 0.2.0 Java for a project using Raspberry Pi's.
> Have created a simple flow in NIFI, saved template and exported then
> converted to yaml using the Minifi Toolkit 0.2.0. Validation of the template
> passes.
> When executing Minifi, get the following warning in the log:
> 2017-07-10 09:45:34,671 WARN [main] o.a.n.c.StandardFlowSynchronizer Schema
> validation error parsing Flow Configuration at line 17, col 27:
> cvc-complex-type.2.4.a: Invalid content was found starting with element
> 'maxConcurrentTasks'. One of '{bundle}' is expected.
> 2017-07-10 09:45:34,692 WARN [main] o.a.n.c.StandardFlowSynchronizer Schema
> validation error parsing Flow Configuration at line 79, col 27:
> cvc-complex-type.2.4.a: Invalid content was found starting with element
> 'maxConcurrentTasks'. One of '{bundle}' is expected.
> Files attached: EdgeCollect.xml - exported template from Nifi
> config.yml - the yaml config generated by toolkit
> (some minor manual edits)
> flow.xml - Minifi created flow.xml file
> *.log - log files from minifi server
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (MINIFI-408) Improve handling of bundle versioning in NARs with other NARs as dependencies
[ https://issues.apache.org/jira/browse/MINIFI-408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Witt updated MINIFI-408: --- Fix Version/s: 0.3.0 > Improve handling of bundle versioning in NARs with other NARs as dependencies > - > > Key: MINIFI-408 > URL: https://issues.apache.org/jira/browse/MINIFI-408 > Project: Apache NiFi MiNiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 0.2.0 >Reporter: Aldrin Piri >Assignee: Aldrin Piri >Priority: Blocker > Fix For: 0.3.0 > > > Parts of this have cropped up in a few tickets affecting CNFEs and inability > to load processors. This is typically exacerbated by the inclusion of NiFi > processors which also depend on NiFi Standard Services API NAR (typically for > SSL Context Service). This can lead to problems when MiNiFi is started as > the transformed flow is effectively a <1.2 version which does not have the > bundling information established. Accordingly, in the case above where I > bring a separate processor NAR with its associated versioned, services NAR, > there are two instances which leads to a ghost implementation. For NiFI, > this is resolved in the UI to select the desired version but we have no such > facility in MiNiFi. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (MINIFI-255) Upgrade to NiFi 1.2.0 dependencies when released
[ https://issues.apache.org/jira/browse/MINIFI-255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16003975#comment-16003975 ] Joseph Witt commented on MINIFI-255: all the updates were pulled in from a squashed version of aldrin's PR https://github.com/apache/nifi-minifi/commit/b556187701f38c9d1f8232913f2ba9ae64db3956 > Upgrade to NiFi 1.2.0 dependencies when released > > > Key: MINIFI-255 > URL: https://issues.apache.org/jira/browse/MINIFI-255 > Project: Apache NiFi MiNiFi > Issue Type: Task > Components: Core Framework >Reporter: Aldrin Piri >Assignee: Aldrin Piri >Priority: Blocker > Fix For: 0.2.0 > > > At minimum, to solve the build issues around the casing change of jBCrypt but > will also provide the Site to Site configuration for attaching to an > interface for sending. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (MINIFI-308) Update logback.conf to reflect new logback version requirements
[ https://issues.apache.org/jira/browse/MINIFI-308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Witt resolved MINIFI-308. Resolution: Fixed resolved in commit for parent ticket > Update logback.conf to reflect new logback version requirements > --- > > Key: MINIFI-308 > URL: https://issues.apache.org/jira/browse/MINIFI-308 > Project: Apache NiFi MiNiFi > Issue Type: Sub-task > Components: Build, Core Framework >Reporter: Joseph Witt >Assignee: Joseph Witt > Fix For: 0.2.0 > > > Need the commit to reflect the changes we did in nifi just like > https://github.com/apache/nifi/pull/1678/commits/2ec695caf625b8c1c6d3844f0d90e7cc6cd36be4#diff-1fc20c14727b9d39d726a70e5842f9f4 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (MINIFI-255) Upgrade to NiFi 1.2.0 dependencies when released
[ https://issues.apache.org/jira/browse/MINIFI-255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16003969#comment-16003969 ] Joseph Witt commented on MINIFI-255: [~aldrin] i've gone ahead and fixed the logback.conf and updated the jetty, logback, and log4j deps just as we did in nifi. This solves the errors/info that spews when starting minifi. Otherwise, full clean build w/contrib check is solid. Ran a test flow from the minifi cpp config and it worked perfectly. +1 will merge to master and close. Now, having said all this there is clearly a tremendous amount of work required to align with the latest nifi releases. Makes me wonder if we're swimming upstream a bit on the current approach to minifi java development. Perhaps we should consider alternatives like having minifi -java just be a stripped down assembly in nifi and plugging in more optimized implementations of things like the flowcontroller/etc.. At least worth a good discussion. > Upgrade to NiFi 1.2.0 dependencies when released > > > Key: MINIFI-255 > URL: https://issues.apache.org/jira/browse/MINIFI-255 > Project: Apache NiFi MiNiFi > Issue Type: Task > Components: Core Framework >Reporter: Aldrin Piri >Assignee: Aldrin Piri >Priority: Blocker > Fix For: 0.2.0 > > > At minimum, to solve the build issues around the casing change of jBCrypt but > will also provide the Site to Site configuration for attaching to an > interface for sending. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (MINIFI-308) Update logback.conf to reflect new logback version requirements
Joseph Witt created MINIFI-308: -- Summary: Update logback.conf to reflect new logback version requirements Key: MINIFI-308 URL: https://issues.apache.org/jira/browse/MINIFI-308 Project: Apache NiFi MiNiFi Issue Type: Sub-task Components: Build, Core Framework Reporter: Joseph Witt Assignee: Joseph Witt Fix For: 0.2.0 Need the commit to reflect the changes we did in nifi just like https://github.com/apache/nifi/pull/1678/commits/2ec695caf625b8c1c6d3844f0d90e7cc6cd36be4#diff-1fc20c14727b9d39d726a70e5842f9f4 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (MINIFI-275) Configuration without IDs for components causes exceptions
[ https://issues.apache.org/jira/browse/MINIFI-275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15983017#comment-15983017 ] Joseph Witt commented on MINIFI-275: awesome [~kdoran]! Welcome to the Apache NiFi community! > Configuration without IDs for components causes exceptions > -- > > Key: MINIFI-275 > URL: https://issues.apache.org/jira/browse/MINIFI-275 > Project: Apache NiFi MiNiFi > Issue Type: Bug > Components: C++, Processing Configuration >Reporter: Aldrin Piri >Priority: Blocker > Fix For: cpp-0.2.0 > > > One of the changes to how components are handled in C++ introduced a defect > into the original construct over the version 1 schema of the YAML. > The absence of this ID causes a YAML exception. > We should provide handling to support configurations how they were created > originally, possibly providing a default/generated ID where one isn't > specified, and start laying the foundation for versioned schemas as provided > in our Java implementation. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (MINIFI-244) Create ArchiveLens processor
[ https://issues.apache.org/jira/browse/MINIFI-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15936636#comment-15936636 ] Joseph Witt commented on MINIFI-244: Ah ok yeah now I'm tracking on where you're heading. - i'm good with the point about consistency being good but not required between processors in minifi/nifi. And actually i can see why in minifi this processor could be the difference between being able to do a certain flow or not within a given resource constraint whereas in nifi it would be more useful than required. - the notion of this in general does make sense too. It is for when we just want to learn/understand/extract something buried inside a given archive to help us decide we want to route something a certain way for followon action (transformation/send/drop/etc..). And this could indeed be far faster for the cases it targets. - the downside of this is that it really struggles against the notion of highly cohesive processors. It would require a set of components which do lens type operations on a certain set of archive types which would duplicate other similar ones which expect the data already split out. We could possibly have a ControllerService instead which defines the lens and understands a given archive type and then have the base processors built to utilize that lens instead of just going after the raw object. Not sure but I think there is a concept there to play with. Anyway - this is pretty cool/innovative thinking. Glad you brought it up > Create ArchiveLens processor > > > Key: MINIFI-244 > URL: https://issues.apache.org/jira/browse/MINIFI-244 > Project: Apache NiFi MiNiFi > Issue Type: Task > Components: C++, Extensions >Reporter: Andrew Christianson >Assignee: Andrew Christianson >Priority: Minor > > Create an ArchiveLens processor. A concise, though informal, definition of a > lens is as follows: > "Essentially, they represent the act of “peering into” or “focusing in on” > some particular piece/path of a complex data object such that you can more > precisely target particular operations without losing the context or > structure of the overall data you’re working with." > https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq > Why an ArchiveLens in MiNiFi? Simply put, it will enable us to "focus in on" > an entry in the archive, perform processing *in-context* of that entry, then > re-focus on the overall archive. This allows for transformation or other > processing of an entry in the archive without losing the overall context of > the archive. > Initial format support is tar, due to its simplicity and ubiquity. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (MINIFI-244) Create ArchiveLens processor
[ https://issues.apache.org/jira/browse/MINIFI-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15936544#comment-15936544 ] Joseph Witt commented on MINIFI-244: [~achristianson] could you describe how this might be used in a flow? I think perhaps it isn't clear what is meant to be the purpose of the processor but it might just be the name that is throwing me off. If the notion is to extract data or metadata from an archive formatted data object (such as tar, zip, 7z, rar, etc..) then we'd want to call the processor ExtractArchive and it would perhaps have a mode indicating whether it is content or metadata that is of interest, some way to query what the definition/range of the lens is, etc.. The lens notion is interesting of course but it doesnt' seem like that is the part that would be interesting to an end user so much as it is interesting to the developer of the processor. If i'm way off base on understanding can you please help reel me back in with an end to end example of how it might be used in a flow? > Create ArchiveLens processor > > > Key: MINIFI-244 > URL: https://issues.apache.org/jira/browse/MINIFI-244 > Project: Apache NiFi MiNiFi > Issue Type: Task > Components: C++, Extensions >Reporter: Andrew Christianson >Assignee: Andrew Christianson >Priority: Minor > > Create an ArchiveLens processor. A concise, though informal, definition of a > lens is as follows: > "Essentially, they represent the act of “peering into” or “focusing in on” > some particular piece/path of a complex data object such that you can more > precisely target particular operations without losing the context or > structure of the overall data you’re working with." > https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq > Why an ArchiveLens in MiNiFi? Simply put, it will enable us to "focus in on" > an entry in the archive, perform processing *in-context* of that entry, then > re-focus on the overall archive. This allows for transformation or other > processing of an entry in the archive without losing the overall context of > the archive. > Initial format support is tar, due to its simplicity and ubiquity. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (MINIFI-54) ConfigSchema duplicate validation only works when processors is not null
[ https://issues.apache.org/jira/browse/MINIFI-54?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15367905#comment-15367905 ] Joseph Witt commented on MINIFI-54: --- [~JPercivall] [~bryanrosan...@gmail.com] this is closed/fixed but has no fix version. Please verify/fix. Thanks > ConfigSchema duplicate validation only works when processors is not null > > > Key: MINIFI-54 > URL: https://issues.apache.org/jira/browse/MINIFI-54 > Project: Apache NiFi MiNiFi > Issue Type: Bug >Reporter: Bryan Rosander > > ConfigSchema has logic to detect duplicate processor, connection, and remote > processing group names. This is to help MiNiFi ensure it can generate a > valid flow. > Unfortunately, the check is short circuited if processors are null, > regardless of which duplicates it is checking for, causing for possibly > missing validation errors. > https://github.com/apache/nifi-minifi/blob/2d1e43e73b90df55abc71845160b6422d7a6f03f/minifi-commons/minifi-commons-schema/src/main/java/org/apache/nifi/minifi/commons/schema/ConfigSchema.java#L122 > Should be changed to a null check on the names parameter -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2160) Enabled ControllerServices disabled on restart
[ https://issues.apache.org/jira/browse/NIFI-2160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15362719#comment-15362719 ] Joseph Witt commented on NIFI-2160: --- Brandon wrote "the controller service I'm observing the issue with is not one of them. However, it does depend on 2 other controller services." > Enabled ControllerServices disabled on restart > -- > > Key: NIFI-2160 > URL: https://issues.apache.org/jira/browse/NIFI-2160 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Brandon DeVries >Assignee: Oleg Zhurakousky >Priority: Critical > Fix For: 1.0.0, 0.7.0 > > > As a result of the fix for NIFI-2032, *previously enabled ControllerServices > become disabled after a restart* if they are not referenced by another > component. However, we use a custom domain specific langauge that can > reference a controller service from a query defined as a custom processor's > property. This means that we use a number of controller service that are > only used in this way (i.e. are never directly referred to by another > component). Upon restart, these are now disabled causing issues with our > flows. > I have not yet stepped through the new enableControllerServices() \[1\] > method to figure out exactly where the issue is coming from, but I wanted to > get the ticket out there and on the radar, as this breaks backwards > compatibility on a feature we heavily rely on. > \[1\] > https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/service/StandardControllerServiceProvider.java#L301-336 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-2160) Enabled ControllerServices disabled on restart
[ https://issues.apache.org/jira/browse/NIFI-2160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Witt updated NIFI-2160: -- Fix Version/s: 1.0.0 > Enabled ControllerServices disabled on restart > -- > > Key: NIFI-2160 > URL: https://issues.apache.org/jira/browse/NIFI-2160 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Brandon DeVries >Assignee: Oleg Zhurakousky >Priority: Critical > Fix For: 1.0.0, 0.7.0 > > > As a result of the fix for NIFI-2032, *previously enabled ControllerServices > become disabled after a restart* if they are not referenced by another > component. However, we use a custom domain specific langauge that can > reference a controller service from a query defined as a custom processor's > property. This means that we use a number of controller service that are > only used in this way (i.e. are never directly referred to by another > component). Upon restart, these are now disabled causing issues with our > flows. > I have not yet stepped through the new enableControllerServices() \[1\] > method to figure out exactly where the issue is coming from, but I wanted to > get the ticket out there and on the radar, as this breaks backwards > compatibility on a feature we heavily rely on. > \[1\] > https://github.com/apache/nifi/blob/0.x/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/service/StandardControllerServiceProvider.java#L301-336 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-2137) Improve Mailing List Alignment for NiFi community
Joseph Witt created NIFI-2137: - Summary: Improve Mailing List Alignment for NiFi community Key: NIFI-2137 URL: https://issues.apache.org/jira/browse/NIFI-2137 Project: Apache NiFi Issue Type: Task Components: Documentation & Website Reporter: Joseph Witt Assignee: Joseph Witt 1) Request INFRA to create issues@nifi.a.o 2) Request INFRA to send all Github notifications to issues@nifi.a.o 3) Request INFRA to send all JIRA notifications to issues@nifi.a.o 4) Document on webpage for mailing lists the purpose of each mailing list (dev, commits, issues, users) 5) Document on webpage for mailing lists a link to https://lists.apache.org/list.html?nifi.apache.org -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2116) Translate Documentation Into Spanish
[ https://issues.apache.org/jira/browse/NIFI-2116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15349950#comment-15349950 ] Joseph Witt commented on NIFI-2116: --- No problem being new. However, we might have a hard time finding someone to help review :-) So a good document to start with is this one https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide It will help explain various things to consider when prepping a PR and such. A good thing to start with is to include the JIRA in your commit message. So for instance instead of 'Initial Draft for Spanish readme.md' you want to do 'NIFI-2116 Initial Draft for Spanish readme.md' I think we'll want to name this file README_es_ES.md and start to incorporate i18n procedures for looking up resources. https://docs.oracle.com/javase/tutorial/i18n/intro/steps.html This particular file, readme.md, is not something that the app itself will load. I am not sure if github supports i18n such that it would load the readme based on your browsers locale. Thanks for starting this. > Translate Documentation Into Spanish > > > Key: NIFI-2116 > URL: https://issues.apache.org/jira/browse/NIFI-2116 > Project: Apache NiFi > Issue Type: Task > Components: Documentation & Website >Reporter: Chagara >Assignee: Chagara >Priority: Minor > > Create documentation in Spanish to expand NIFI reach. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-2120) source release produces an extraneous file
Joseph Witt created NIFI-2120: - Summary: source release produces an extraneous file Key: NIFI-2120 URL: https://issues.apache.org/jira/browse/NIFI-2120 Project: Apache NiFi Issue Type: Bug Components: Tools and Build Reporter: Joseph Witt Priority: Trivial The source release now includes 'appveyor.yml' at the top of the source directory. This file should not end up in the source release ideally. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-2119) Secure clustering returning bad request response
Joseph Witt created NIFI-2119: - Summary: Secure clustering returning bad request response Key: NIFI-2119 URL: https://issues.apache.org/jira/browse/NIFI-2119 Project: Apache NiFi Issue Type: Bug Components: Core Framework Reporter: Joseph Witt Cannot get a secured cluster working that worked well on 0.6.0. After upgrading now seeing the following line. It either means I upgraded incorrectly, or we're missing critical migration guidance, or we have introduced a new bug. 2016-06-25 14:19:12,017 INFO [NiFi Web Server-23] o.a.n.w.a.c.IllegalArgumentExceptionMapper java.lang.IllegalArgumentException: User account already created CN=box1.testing.org, OU=NIFI, O=Apache-NiFi, L=Here, ST=There, C=EVERYWHERE. Returning Bad Request response. Speaking with [~mcgilman] about this he looked into it and says "the socket used for cluster communications is configured with an sslContext that has client auth set to none... which seems to be why the we're not getting the NCM DN during connection i think the issue is this part of this commit https://github.com/apache/nifi/commit/7b5583f3a8c8e3f62e2985059a3466a5bb36f4e8#diff-a14f46a45c394fbd82a2b99730e04bcbR68; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-2118) Correct license/notice issues
Joseph Witt created NIFI-2118: - Summary: Correct license/notice issues Key: NIFI-2118 URL: https://issues.apache.org/jira/browse/NIFI-2118 Project: Apache NiFi Issue Type: Bug Components: Tools and Build Reporter: Joseph Witt Fix For: 0.7.0 *The LICENSE file has this entry "This product bundles 'Jolt' which is available under an Apache License" We should not have apache licensed items in our license file. We only need to carry forward NOTICE information of such dependencies and those would go in our NOTICE if applicable. * The nifi-assembly/NOTICE file has an "MIT License" section. This should be removed. MIT licensed items should be reflected in the LICENSE file and Luaj already appears to be there. * The nifi-assembly/NOTICE file references "Python Software Foundation" section. Python license is category A. Such dependency belongs in LICENSE file and not NOTICE. Only if that dependency has a NOTICE with legally required items should there be a reference to it in our NOTICE as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2115) Enhanced About Box Version Information
[ https://issues.apache.org/jira/browse/NIFI-2115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15349865#comment-15349865 ] Joseph Witt commented on NIFI-2115: --- Well, we could also consider as an alternative keeping 'about' pretty generic and then adding a cluster topology type page which lists all this information for each node. I am totally with you that this is useful. As for the branching/commit info, the *only* thing we officially release is the source release (the -src.zip). The convenience binaries are not the official release. So with that in mind we could figure out a way to have this occur during the source release process where by this i mean have it set some maven properties. And those could be used when the unknown thing happens as you outline below. > Enhanced About Box Version Information > -- > > Key: NIFI-2115 > URL: https://issues.apache.org/jira/browse/NIFI-2115 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI, Tools and Build >Affects Versions: 1.0.0 >Reporter: James Wing >Assignee: James Wing >Priority: Minor > > The UI's About dialog and underlying API provide the version of NiFi, like > "0.7.0-SNAPSHOT". For many bug reports and troubleshooting requests, this is > not very precise, especially around rapidly changing code or > platform-dependent behavior. It would help if NiFi captured and displayed > additional information: > * NiFi build commit hash > * NiFi build branch > * NiFi build date/time > * Java Version > * Java Vendor (Oracle, OpenJDK) > * OS > Then a simple copy/paste from the about box would provide very specific > information about a particular NiFi installation and which code it was > derived from. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2115) Enhanced About Box Version Information
[ https://issues.apache.org/jira/browse/NIFI-2115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15349651#comment-15349651 ] Joseph Witt commented on NIFI-2115: --- [~jameswing] I think this is a great idea. Couple of questions/comments to think about. * We need to take into account what should be rendered when what we're looking at is a cluster of nodes. They could be heterogenous in terms of Java version, OS version, NiFi version, build version, and such. I think we would need to factor those things in and what we want to render when there are differences. * How would this work as far as gathering build information for the source release? Those maven properties would need to be populated with their values (not variables) when we make the source release I assume so that the convenience binaries produced would have the proper info I think. Also, please consider removing the fix version for now until this approaches being merge-able. It makes closing down on releases like 1.0 easier. [~rmoran] Could you help with providing a sketch of what this could look like? > Enhanced About Box Version Information > -- > > Key: NIFI-2115 > URL: https://issues.apache.org/jira/browse/NIFI-2115 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI, Tools and Build >Affects Versions: 1.0.0 >Reporter: James Wing >Assignee: James Wing >Priority: Minor > Fix For: 1.0.0 > > > The UI's About dialog and underlying API provide the version of NiFi, like > "0.7.0-SNAPSHOT". For many bug reports and troubleshooting requests, this is > not very precise, especially around rapidly changing code or > platform-dependent behavior. It would help if NiFi captured and displayed > additional information: > * NiFi build commit hash > * NiFi build branch > * NiFi build date/time > * Java Version > * Java Vendor (Oracle, OpenJDK) > * OS > Then a simple copy/paste from the about box would provide very specific > information about a particular NiFi installation and which code it was > derived from. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2116) Translate Documentation Into Spanish
[ https://issues.apache.org/jira/browse/NIFI-2116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15349647#comment-15349647 ] Joseph Witt commented on NIFI-2116: --- Hello Chagara. I added you to the contributors role so you should be able to assign yourself now. It is awesome you are willing to dive into this. We'll need to work in support for internationalization so that the proper language set is chosen. But we can work our way there incrementally. thanks! > Translate Documentation Into Spanish > > > Key: NIFI-2116 > URL: https://issues.apache.org/jira/browse/NIFI-2116 > Project: Apache NiFi > Issue Type: Task > Components: Documentation & Website >Reporter: Chagara >Assignee: Chagara >Priority: Minor > > Create documentation in Spanish to expand NIFI reach. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-2116) Translate Documentation Into Spanish
[ https://issues.apache.org/jira/browse/NIFI-2116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Witt updated NIFI-2116: -- Assignee: Chagara > Translate Documentation Into Spanish > > > Key: NIFI-2116 > URL: https://issues.apache.org/jira/browse/NIFI-2116 > Project: Apache NiFi > Issue Type: Task > Components: Documentation & Website >Reporter: Chagara >Assignee: Chagara >Priority: Minor > > Create documentation in Spanish to expand NIFI reach. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-2107) Other panels like 'content-viewer' still using old style
Joseph Witt created NIFI-2107: - Summary: Other panels like 'content-viewer' still using old style Key: NIFI-2107 URL: https://issues.apache.org/jira/browse/NIFI-2107 Project: Apache NiFi Issue Type: Sub-task Reporter: Joseph Witt Fix For: 1.0.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-2106) Clicking refresh in 'Status History' window causes window to grow
Joseph Witt created NIFI-2106: - Summary: Clicking refresh in 'Status History' window causes window to grow Key: NIFI-2106 URL: https://issues.apache.org/jira/browse/NIFI-2106 Project: Apache NiFi Issue Type: Sub-task Reporter: Joseph Witt Fix For: 1.0.0 Attachments: Screen Shot 2016-06-23 at 2.19.57 PM.png -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-2106) Clicking refresh in 'Status History' window causes window to grow
[ https://issues.apache.org/jira/browse/NIFI-2106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Witt updated NIFI-2106: -- Attachment: Screen Shot 2016-06-23 at 2.19.57 PM.png > Clicking refresh in 'Status History' window causes window to grow > - > > Key: NIFI-2106 > URL: https://issues.apache.org/jira/browse/NIFI-2106 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Joseph Witt > Fix For: 1.0.0 > > Attachments: Screen Shot 2016-06-23 at 2.19.57 PM.png > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-2105) Flow status graphs when resizing window get wacky
Joseph Witt created NIFI-2105: - Summary: Flow status graphs when resizing window get wacky Key: NIFI-2105 URL: https://issues.apache.org/jira/browse/NIFI-2105 Project: Apache NiFi Issue Type: Sub-task Reporter: Joseph Witt Attachments: Screen Shot 2016-06-23 at 2.11.44 PM.png When you first bring up the status history it looks great. If you make the window smaller it quickly becomes messed up. If you resize window larger again it looks ok again. If you make it smaller again it looks bad again. If you change tabs and come back then it is fine again. So seems like some missing lifecycle call? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-2104) Text in Processor dialogue gets cutoff
Joseph Witt created NIFI-2104: - Summary: Text in Processor dialogue gets cutoff Key: NIFI-2104 URL: https://issues.apache.org/jira/browse/NIFI-2104 Project: Apache NiFi Issue Type: Sub-task Reporter: Joseph Witt Fix For: 1.0.0 Attachments: Screen Shot 2016-06-23 at 2.08.03 PM.png see attached image. RElationship description gets cutoff. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-2104) Text in Processor dialogue gets cutoff
[ https://issues.apache.org/jira/browse/NIFI-2104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Witt updated NIFI-2104: -- Attachment: Screen Shot 2016-06-23 at 2.08.03 PM.png > Text in Processor dialogue gets cutoff > -- > > Key: NIFI-2104 > URL: https://issues.apache.org/jira/browse/NIFI-2104 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Joseph Witt > Fix For: 1.0.0 > > Attachments: Screen Shot 2016-06-23 at 2.08.03 PM.png > > > see attached image. RElationship description gets cutoff. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-2103) Flow diagram gets cutoff
[ https://issues.apache.org/jira/browse/NIFI-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Witt updated NIFI-2103: -- Attachment: Screen Shot 2016-06-23 at 2.01.27 PM.png > Flow diagram gets cutoff > > > Key: NIFI-2103 > URL: https://issues.apache.org/jira/browse/NIFI-2103 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Affects Versions: 1.0.0 >Reporter: Joseph Witt > Fix For: 1.0.0 > > Attachments: Screen Shot 2016-06-23 at 2.01.27 PM.png > > > See attached image. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-2103) Flow diagram gets cutoff
Joseph Witt created NIFI-2103: - Summary: Flow diagram gets cutoff Key: NIFI-2103 URL: https://issues.apache.org/jira/browse/NIFI-2103 Project: Apache NiFi Issue Type: Sub-task Components: Core UI Affects Versions: 1.0.0 Reporter: Joseph Witt Fix For: 1.0.0 See attached image. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2063) Install Script Relative Path Mismatch from Init Dir
[
https://issues.apache.org/jira/browse/NIFI-2063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15341861#comment-15341861
]
Joseph Witt commented on NIFI-2063:
---
Ryan H via nifi.apache.org (sent by ryan.andrew.hendrick...@gmail.com)
10:21 AM (0 minutes ago)
to dev
I'm trying to get the 0.7.0 NiFi to start on boot on linux/centos 7.
During all this, I've noticed 0.6.1 doesn't quite work either, left some
notes at the bottom about that.
*For 0.7.0:*
*I followed the modified install commands for the nifi.sh script:*
I untar'd it in:
#/opt/nifi/current -> nifi-0.7.0-SNAPSHOT
I followed these steps:
##Edited the nifi.sh script for the SCRIPT_DIR issue.
#/opt/nifi/current/bin/nifi.sh install
*#chkconfig nifi on <--- Turn on for boot 2345 run levels*
#service nifi start
NiFi is now started.
I reboot the box.
*NiFi does not start.* There's no logs in /var/log/messages or
/opt/nifi/current/logs indicating why. (This script should probably log
someplace)
*Why?*
The current script has a command that starts as:
#cd ${NIFI_HOME} && sudo -u ${run_as} &
The sudo part is omitted if there is no ${run_as} user defined. This works
for starting the service by hand. However, if this script is set to start
on boot with a ${run_as} user, in this case using chkconfig, it will
silently fail when starting on boot because of the "sudo" part. Not sure
why "sudo" isn't well liked in CentOS 7 in a service script.
*How we fixed it:*
Fixed by structuring the command like this the nifi.sh script like this:
Old Command:
## RUN_NIFI_CMD="cd "\""${NIFI_HOME}"\"" && ${sudo_cmd_prefix}
"\""${JAVA}"\"" -cp "\""${BOOTSTRAP_CLASSPATH}"\"" -Xms12m -Xmx24m
${BOOTSTRAP_DIR_PARAMS} org.apache.nifi.bootstrap.RunNiFi"
Put it into the if's:
#if [ "$1" = "start" ]; then
#RUN_NIFI_CMD="su -c "\""cd "\""${NIFI_HOME}"\"" && "\""${JAVA}"\""
-cp "\""${BOOTSTRAP_CLASSPATH}"\"" -Xms12m -Xmx24m ${BOOTSTRAP_DIR_PARAMS}
org.apache.nifi.bootstrap.RunNiFi $@ &"\"" ${run_as}"
#(eval $RUN_NIFI_CMD)
#else
#RUN_NIFI_CMD="su -c "\""cd "\""${NIFI_HOME}"\"" && "\""${JAVA}"\""
-cp "\""${BOOTSTRAP_CLASSPATH}"\"" -Xms12m -Xmx24m ${BOOTSTRAP_DIR_PARAMS}
org.apache.nifi.bootstrap.RunNiFi $@"\"" ${run_as}"
#(eval $RUN_NIFI_CMD)
#fi
It now starts on boot.
*Logging in this file:*
* I created a /var/log/nifi dir as root
* I started piping echo statements to "tee -a /var/log/nifi/init.log"
** *Example: #echo "Attempting to start NiFi" | tee -a
/var/log/nifi/init.log
*For 0.6.1:*
*I followed the standard install commands for the nifi.sh script.*
I untar'd it in:
#/opt/nifi/current -> nifi-0.7.0-SNAPSHOT
I followed these steps:
#/opt/nifi/current/bin/nifi.sh install
*#chkconfig nifi on <--- Turn on for boot 2345 run levels*
#service nifi start
NiFi is now started.
I reboot the box.
*NiFi does not start.* There's no logs in /var/log/messages or
/opt/nifi/current/logs indicating why. (This script should probably log
someplace)
*Why?*
The current script has a command that starts as:
#cd ${NIFI_HOME} && sudo -u ${run_as} &
The sudo part is omitted if there is no ${run_as} user defined. This works
for starting the service by hand. However, if this script is set to start
on boot with a ${run_as} user, in this case using chkconfig, it will
silently fail when starting on boot because of the "sudo" part. Not sure
why "sudo" isn't well liked in CentOS 7 in a service script.
*How we fixed it:*
Fixed by structuring the command like this:
#su -c "cd ${NIFI_HOME} && &" ${run_as}
This works when starting on boot if you have a ${run_as} user defined,
though not sure of the behavior if there is no ${run_as} user defined or if
the ${run_as} user is root.
--
Thanks,
Ryan
> Install Script Relative Path Mismatch from Init Dir
> ---
>
> Key: NIFI-2063
> URL: https://issues.apache.org/jira/browse/NIFI-2063
> Project: Apache NiFi
> Issue Type: Bug
> Components: Configuration
>Affects Versions: 1.0.0, 0.7.0
> Environment: Linux/Unix with SysVInit
>Reporter: James Wing
> Fix For: 0.7.0
>
>
> Ryan Hendrickson noticed that installing and running NiFi as a service in
> 0.7.0-SNAPSHOT is not working right. Running commands like the following:
> {code}
> $ sudo /opt/nifi/nifi-0.7.0-SNAPSHOT/bin/nifi.sh install
> Service nifi installed
> $ sudo service nifi start
> /etc/init.d/nifi: line 28: /etc/init.d/nifi-env.sh: No such file or directory
> {code}
> Results in errors loading nifi-env.sh
> *Used-To-Be*
> As I understand it, {{nifi.sh install}} used to:
> 1. copy bin/nifi.sh to /etc/init.d/nifi
> 2. Run sed to change NIFI_HOME initializations to the path to nifi.sh, where
> install
[jira] [Commented] (NIFI-1815) Tesseract OCR Processor
[ https://issues.apache.org/jira/browse/NIFI-1815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15339897#comment-15339897 ] Joseph Witt commented on NIFI-1815: --- BSD is fine. The (L)GPL ones are the problem. [~jeremy.dyer] could you find out if those three are critical? IF not we could exclude them. > Tesseract OCR Processor > --- > > Key: NIFI-1815 > URL: https://issues.apache.org/jira/browse/NIFI-1815 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Jeremy Dyer >Assignee: Jeremy Dyer > Attachments: 0006-changes-to-the-OCR-processor.patch, > nifi_1815_1.x_patch.zip > > > This ticket is a follow-up to NIFI-1718 minus the use of the Tika library > Expose OCR capabilities through a new processor which uses the Tesseract > library. Use of this processor would require that Tesseract be installed on > the NiFi host. Since the processor will have a system dependency care must be > taken to ensure that the overall NiFi cluster continues to function properly > in the absence of the Tesseract system dependency even though the OCR > processor itself will be unable to perform its duties. In the event that the > system dependencies are not detected the processor should display a > validation warning rather than failing or preventing the NiFi instance from > booting properly. > Properties expose to configure Tesseract > tesseractPath - Path to tesseract installation folder, if not on system path. > language - Language ID (e.g. "eng"); language dictionary to be used. > pageSegMode - Tesseract page segmentation mode, defaults to 1. > minFileSizeToOcr - Minimum file size to submit file to OCR, defaults to 0. > maxFileSizeToOcr - Maximum file size to submit file to OCR, defaults to > Integer.MAX_VALUE. > timeout - Maximum time (in seconds) to wait for the OCR process termination; > defaults to 120. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1769) Add support for SSE-KMS and S3 Signature Version 4 Authentication AWS
[ https://issues.apache.org/jira/browse/NIFI-1769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15336484#comment-15336484 ] Joseph Witt commented on NIFI-1769: --- Anyone with sufficient expertise/time available to help work this seemingly cool contrib forward? > Add support for SSE-KMS and S3 Signature Version 4 Authentication AWS > - > > Key: NIFI-1769 > URL: https://issues.apache.org/jira/browse/NIFI-1769 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 0.5.1 >Reporter: Michiel Moonen >Priority: Minor > Labels: newbie, patch, security > > Currently there is no support for SSE-KMS S3 Signature Version 4 > Authentication. This is necessary for enhanced security features -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1834) Create PutTCP Processor
[ https://issues.apache.org/jira/browse/NIFI-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15336471#comment-15336471 ] Joseph Witt commented on NIFI-1834: --- you want 1.0.0 on here for fix version too right [~JPercivall] > Create PutTCP Processor > --- > > Key: NIFI-1834 > URL: https://issues.apache.org/jira/browse/NIFI-1834 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Matt Brown >Priority: Minor > Fix For: 0.7.0 > > Attachments: 0001-PutTCP-Processor-created.patch > > > Create a PutTCP Processor to send FlowFile content over a TCP connection to a > TCP server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1152) Mock Framework allows processor to route to Relationships that the Processor does not support
[ https://issues.apache.org/jira/browse/NIFI-1152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15336469#comment-15336469 ] Joseph Witt commented on NIFI-1152: --- i just added [~puspendu.baner...@gmail.com] to the contributors list on Apache NiFi JIRA. Should be able to assign things for himself now. > Mock Framework allows processor to route to Relationships that the Processor > does not support > - > > Key: NIFI-1152 > URL: https://issues.apache.org/jira/browse/NIFI-1152 > Project: Apache NiFi > Issue Type: Bug > Components: Tools and Build >Reporter: Mark Payne > Labels: beginner, newbie > Fix For: 1.0.0 > > Attachments: > 0001-Fix-for-NIFI-1838-NIFI-1152-Code-modification-for-ty.patch > > > If a processor calls ProcessSession.transfer(flowFile, > NON_EXISTENT_RELATIONSHIP) the NiFi framework will throw a > FlowFileHandlingException. However, the Mock Framework simply allows it and > does not throw any sort of Exception. This needs to be addressed so that the > Mock framework functions the same way as the normal NiFi framework. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-329) Provide processor(s) to interact with IRC
[ https://issues.apache.org/jira/browse/NIFI-329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15335966#comment-15335966 ] Joseph Witt commented on NIFI-329: -- just added you to jira contributors for nifi so hopefully now you're able to take such things on yourself. I've assigned it to you in the meantime. Thanks! > Provide processor(s) to interact with IRC > - > > Key: NIFI-329 > URL: https://issues.apache.org/jira/browse/NIFI-329 > Project: Apache NiFi > Issue Type: Sub-task > Components: Examples >Reporter: Joseph Witt >Assignee: Andre >Priority: Minor > Labels: beginner > > - Processor(s) to interact with IRC (sending/receiving) > One approach: A single processor which both sends and receives data from a > given IRC channel. The user can configure the IRC host, username, password, > channel, etc... The connection then is held open and the processor will > produce an output flow file for every message received in the IRC channel > which will have as attributes the message content, sender, time, etc.. That > same processor can also read flow files from its queue which contain message > text in an attribute. In this manner the processor can support bidirectional > interaction with IRC. > Would then also be interesting to make it really easy for a user to generate > a message via the UI as well as easily to consume a message via the UI. > These could be very generic processors/widgets for creation/consumption and > good for these sorts of cases. > There are active IRC channels which are great for demonstration of relatively > active datastreams. Weather updates, Wikipedia updates, etc... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-329) Provide processor(s) to interact with IRC
[ https://issues.apache.org/jira/browse/NIFI-329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Witt updated NIFI-329: - Assignee: Andre > Provide processor(s) to interact with IRC > - > > Key: NIFI-329 > URL: https://issues.apache.org/jira/browse/NIFI-329 > Project: Apache NiFi > Issue Type: Sub-task > Components: Examples >Reporter: Joseph Witt >Assignee: Andre >Priority: Minor > Labels: beginner > > - Processor(s) to interact with IRC (sending/receiving) > One approach: A single processor which both sends and receives data from a > given IRC channel. The user can configure the IRC host, username, password, > channel, etc... The connection then is held open and the processor will > produce an output flow file for every message received in the IRC channel > which will have as attributes the message content, sender, time, etc.. That > same processor can also read flow files from its queue which contain message > text in an attribute. In this manner the processor can support bidirectional > interaction with IRC. > Would then also be interesting to make it really easy for a user to generate > a message via the UI as well as easily to consume a message via the UI. > These could be very generic processors/widgets for creation/consumption and > good for these sorts of cases. > There are active IRC channels which are great for demonstration of relatively > active datastreams. Weather updates, Wikipedia updates, etc... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2023) CompressContent Processor should not always log decompression failures as an ERROR
[ https://issues.apache.org/jira/browse/NIFI-2023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15330531#comment-15330531 ] Joseph Witt commented on NIFI-2023: --- I feel like it is probably correct for compresscontent to throw an error when it is given data it cannot decompress. A common pattern is to put an IdentifyMimeType processor ahead of this processor so that it can determine the mime type then use decompress with mime.type configured on compress content. You can also put a route on attribute processor in between to make routing decisions and if it is not a known/supported compression type it will skip it. Does this make sense ? > CompressContent Processor should not always log decompression failures as an > ERROR > -- > > Key: NIFI-2023 > URL: https://issues.apache.org/jira/browse/NIFI-2023 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework, Core UI >Affects Versions: 0.6.1 >Reporter: Christopher McDermott >Priority: Minor > > Sometimes you don't you don't know if file compression is used or what kind > of compression is used. In these cases it is normal I think to chain > together a bunch CompressContent processors via the the failed output to > attempt decompression. If a stage fails in the chain it is not desirable to > emit and ERROR bulletin. > Perhaps a configuration parameter could be added to control the level at > which the failure is logged. See NIFI-1724 for a similar enhancement request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1795) Improve documentation regarding PropertyDescriptor name & displayName
[
https://issues.apache.org/jira/browse/NIFI-1795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15328075#comment-15328075
]
Joseph Witt commented on NIFI-1795:
---
could we remove 0.7 from this and just address in 1.0?
> Improve documentation regarding PropertyDescriptor name & displayName
> -
>
> Key: NIFI-1795
> URL: https://issues.apache.org/jira/browse/NIFI-1795
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Documentation & Website
>Affects Versions: 0.6.1
>Reporter: Andy LoPresto
>Priority: Minor
> Fix For: 1.0.0, 0.7.0
>
>
> As evidenced in numerous pull requests, developers are often unaware of the
> behavior of {{PropertyDescriptor}} {{name}} and {{displayName}} attributes.
> Originally, {{name}} existed and served as both the UI display value for the
> processor properties configuration dialog and as the unique identifier when
> writing/reading the properties from file storage. However, a change to the
> {{name}} value after the flow had been persisted could cause issues
> retrieving the value from file storage. Thus, {{displayName}} was introduced
> as the UI display value to allow for backward-compatible updates. The
> documentation does not sufficiently explain this information and can better
> encourage developers to provide both {{name}} and {{displayName}} attributes
> in order to be proactive on flow compatibility and to improve the foundation
> for internationalization and localization of the UI.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (NIFI-2004) NiFi should not allow creation of templates with the same name
[ https://issues.apache.org/jira/browse/NIFI-2004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Witt updated NIFI-2004: -- Fix Version/s: (was: 0.7.0) > NiFi should not allow creation of templates with the same name > --- > > Key: NIFI-2004 > URL: https://issues.apache.org/jira/browse/NIFI-2004 > Project: Apache NiFi > Issue Type: Bug >Reporter: Oleg Zhurakousky > Fix For: 1.0.0 > > > NiFi allows creation of multiple templates with the same name regardless > wether such templates are created from the same or different flows -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2004) NiFi should not allow creation of templates with the same name
[ https://issues.apache.org/jira/browse/NIFI-2004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15328069#comment-15328069 ] Joseph Witt commented on NIFI-2004: --- only an issue for 1.0 release > NiFi should not allow creation of templates with the same name > --- > > Key: NIFI-2004 > URL: https://issues.apache.org/jira/browse/NIFI-2004 > Project: Apache NiFi > Issue Type: Bug >Reporter: Oleg Zhurakousky > Fix For: 1.0.0 > > > NiFi allows creation of multiple templates with the same name regardless > wether such templates are created from the same or different flows -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-1789) TestPutUDP testUnkownHostname fails because all hostnames resolve
[
https://issues.apache.org/jira/browse/NIFI-1789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Witt updated NIFI-1789:
--
Fix Version/s: (was: 0.7.0)
(was: 1.0.0)
> TestPutUDP testUnkownHostname fails because all hostnames resolve
> -
>
> Key: NIFI-1789
> URL: https://issues.apache.org/jira/browse/NIFI-1789
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 0.6.1
>Reporter: Andy LoPresto
>
> On some systems, arbitrary hostnames (e.g. {{sgfgfdgs}}) resolves to a valid
> IP address depending on the DNS resolution scheme. Therefore, the test, which
> assumes that the host does not resolve, fails.
> {code}
> hw12203:/Users/alopresto (master) alopresto
> 0s @ 20:10:32 $ nslookup asdfsdfsdf
> Server: 192.168.1.1
> Address: 192.168.1.1#53
> Non-authoritative answer:
> Name: asdfsdfsdf
> Address: 198.105.254.228
> Name: asdfsdfsdf
> Address: 198.105.244.228
> hw12203:/Users/alopresto (master) alopresto
> 39s @ 20:12:24 $ dig asdfasdf
> ; <<>> DiG 9.8.3-P1 <<>> asdfasdf
> ;; global options: +cmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 31295
> ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0
> ;; QUESTION SECTION:
> ;asdfasdf.IN A
> ;; ANSWER SECTION:
> asdfasdf. 10 IN A 198.105.254.228
> asdfasdf. 10 IN A 198.105.244.228
> ;; Query time: 81 msec
> ;; SERVER: 192.168.1.1#53(192.168.1.1)
> ;; WHEN: Tue Apr 19 20:12:30 2016
> ;; MSG SIZE rcvd: 58
> hw12203:/Users/alopresto (master) alopresto
> 0s @ 20:12:30 $
> {code}
> Reported on Mac OS X 10.11.4
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (NIFI-1644) If unable to write to Content Repository, Process Session should automatically roll itself back
[
https://issues.apache.org/jira/browse/NIFI-1644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Witt updated NIFI-1644:
--
Fix Version/s: 1.0.0
> If unable to write to Content Repository, Process Session should
> automatically roll itself back
> ---
>
> Key: NIFI-1644
> URL: https://issues.apache.org/jira/browse/NIFI-1644
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: Mark Payne
> Fix For: 1.0.0, 0.7.0
>
>
> If we write to the Content Repository and get an IOException (for example,
> out of disk space), the ProcessSession catches this and then removes the
> temporary content claim and then throws a FlowFileAccessException. However,
> the entire session really should be rolled back, because the FlowFIle no
> longer has a valid Content Claim. An example of this is in the
> StandardProcessSession.write method:
> {code}
> } catch (final FlowFileAccessException ffae) {
> resetWriteClaims(); // need to reset write claim before we can
> remove the claim
> destroyContent(newClaim);
> throw ffae;
> }
> {code}
> Processors that then catch Throwable or the general Exception and route to
> failure pass along an invalid FlowFile. We end up seeing the following in the
> logs:
> {code}
> 2016-03-17 04:21:04,742 WARN [Timer-Driven Process Thread-17]
> o.a.n.c.r.WriteAheadFlowFileRepository Repository Record
> StandardRepositoryRecord[UpdateType=CONTENTMISSING,Record=StandardFlowFileRecord[uuid=01efcc28-e28f-45ab-9373-cba8933a010c,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1458188464723-45209, container=pub1,
> section=153], offset=0,
> length=1017],offset=0,name=26304456091229115,size=1017]] is marked to be
> aborted; it will be persisted in the FlowFileRepository as a DELETE record
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (NIFI-1957) Modal dialogs do not handle browser window resize
[ https://issues.apache.org/jira/browse/NIFI-1957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Witt updated NIFI-1957: -- Fix Version/s: (was: 0.7.0) > Modal dialogs do not handle browser window resize > - > > Key: NIFI-1957 > URL: https://issues.apache.org/jira/browse/NIFI-1957 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Affects Versions: 0.6.1 >Reporter: Andy LoPresto >Priority: Critical > Labels: dialog, ui > Fix For: 1.0.0 > > Attachments: Screen Shot 2016-06-02 at 10.31.31 AM.png, Screen Shot > 2016-06-02 at 10.34.12 AM.png, Screen Shot 2016-06-02 at 10.42.52 AM.png, > Screen Shot 2016-06-02 at 10.47.37 AM.png, Screen Shot 2016-06-02 at 10.47.41 > AM.png, Screen Shot 2016-06-02 at 10.47.45 AM.png, Screen Shot 2016-06-02 at > 10.48.04 AM.png, Screen Shot 2016-06-02 at 10.48.09 AM.png, Screen Shot > 2016-06-02 at 10.48.22 AM.png, Screen Shot 2016-06-02 at 10.48.34 AM.png, > Screen Shot 2016-06-02 at 10.48.48 AM.png > > > When the processor configuration dialog is open and the browser window is > resized, the dialog does not move or resize to fit in the new visible canvas. > In master (1.0) branch, the processor component is also truncated after > resizing the window, and refreshing the UI does not fix this issue. See > screenshots attached. > This is reported on Mac OS X 10.11.4 using Google Chrome Version > 50.0.2661.102 (64-bit) and Safari Version 9.1 (11601.5.17.1). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1935) Added ConvertDynamicJsonToAvro processor
[ https://issues.apache.org/jira/browse/NIFI-1935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15328041#comment-15328041 ] Joseph Witt commented on NIFI-1935: --- Processors are expected to behave well against whichever branch they are applied to. For the scenario you describe it sounds like further evaluation needs to occur to understand the difference in behavior being seen. Can you share more specifics about the behavior you see on the master (1.x) line versus the 0.x line? > Added ConvertDynamicJsonToAvro processor > > > Key: NIFI-1935 > URL: https://issues.apache.org/jira/browse/NIFI-1935 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Affects Versions: 1.0.0, 0.7.0 >Reporter: Daniel Cave >Assignee: Alex Halldin >Priority: Minor > Fix For: 1.0.0, 0.7.0 > > Attachments: > 0001-NIFI-1935-Added-ConvertDynamicJSONToAvro.java.-Added.patch > > > ConvertJsonToAvro required a predefined Avro schema to convert JSON and > required the presence of all field on the incoming JSON. > ConvertDynamicJsonToAvro functions similarly, however it now accepts the JSON > and schema as incoming flowfiles and creates the Avro dynamically. > This processor requires the InferAvroSchema processor in its upstream flow so > that it can use the original and schema flowfiles as input. These two > flowfiles will have the unique attribute inferredAvroId set on them by > InferAvroSchema so that they can be properly matched in > ConvertDynamicJsonToAvro. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1935) Added ConvertDynamicJsonToAvro processor
[ https://issues.apache.org/jira/browse/NIFI-1935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15328019#comment-15328019 ] Joseph Witt commented on NIFI-1935: --- How the release lines will be managed is described here https://cwiki.apache.org/confluence/display/NIFI/Git+Branching+and+Release+Line+Management > Added ConvertDynamicJsonToAvro processor > > > Key: NIFI-1935 > URL: https://issues.apache.org/jira/browse/NIFI-1935 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Affects Versions: 1.0.0, 0.7.0 >Reporter: Daniel Cave >Assignee: Alex Halldin >Priority: Minor > Fix For: 1.0.0, 0.7.0 > > Attachments: > 0001-NIFI-1935-Added-ConvertDynamicJSONToAvro.java.-Added.patch > > > ConvertJsonToAvro required a predefined Avro schema to convert JSON and > required the presence of all field on the incoming JSON. > ConvertDynamicJsonToAvro functions similarly, however it now accepts the JSON > and schema as incoming flowfiles and creates the Avro dynamically. > This processor requires the InferAvroSchema processor in its upstream flow so > that it can use the original and schema flowfiles as input. These two > flowfiles will have the unique attribute inferredAvroId set on them by > InferAvroSchema so that they can be properly matched in > ConvertDynamicJsonToAvro. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-1620) Allow empty Content-Type in InvokeHTTP processor
[
https://issues.apache.org/jira/browse/NIFI-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Witt updated NIFI-1620:
--
Fix Version/s: 1.0.0
> Allow empty Content-Type in InvokeHTTP processor
>
>
> Key: NIFI-1620
> URL: https://issues.apache.org/jira/browse/NIFI-1620
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 0.5.1
>Reporter: Pierre Villard
>Assignee: Pierre Villard
> Fix For: 1.0.0, 0.7.0
>
>
> External API could expect a POST request without a Content-Type property or
> at least with this property empty:
> *Error example:*
> {quote}
> You provided a non-empty HTTP "Content-Type" header ("application/json").
> This API function requires that the header be missing or empty.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (NIFI-1862) User Guide corrections/improvements
[ https://issues.apache.org/jira/browse/NIFI-1862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Witt updated NIFI-1862: -- Fix Version/s: 0.7.0 1.0.0 > User Guide corrections/improvements > --- > > Key: NIFI-1862 > URL: https://issues.apache.org/jira/browse/NIFI-1862 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation & Website >Affects Versions: 0.6.0 >Reporter: Andrew Lim >Priority: Minor > Fix For: 1.0.0, 0.7.0 > > > Creating this ticket to capture multiple edits to the User Guide > documentation for correcting errors (spelling/grammatical) and improving > readability. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (NIFI-1886) Extend HashAttribute to provide multiple algorithms
[ https://issues.apache.org/jira/browse/NIFI-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Witt resolved NIFI-1886. --- Resolution: Duplicate > Extend HashAttribute to provide multiple algorithms > --- > > Key: NIFI-1886 > URL: https://issues.apache.org/jira/browse/NIFI-1886 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 0.6.1 >Reporter: Aldrin Piri > > Currently HashAttribute only provides MD5 as the algorithm for performing the > hashing. It would be helpful to allow selection of other algorithms like > HashContent. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (NIFI-1886) Extend HashAttribute to provide multiple algorithms
[ https://issues.apache.org/jira/browse/NIFI-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Witt reopened NIFI-1886: --- > Extend HashAttribute to provide multiple algorithms > --- > > Key: NIFI-1886 > URL: https://issues.apache.org/jira/browse/NIFI-1886 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 0.6.1 >Reporter: Aldrin Piri > > Currently HashAttribute only provides MD5 as the algorithm for performing the > hashing. It would be helpful to allow selection of other algorithms like > HashContent. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-1892) nifi-mock module does not compile
[ https://issues.apache.org/jira/browse/NIFI-1892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Witt updated NIFI-1892: -- Fix Version/s: 1.0.0 > nifi-mock module does not compile > - > > Key: NIFI-1892 > URL: https://issues.apache.org/jira/browse/NIFI-1892 > Project: Apache NiFi > Issue Type: Bug > Components: Tools and Build >Affects Versions: 1.0.0 >Reporter: Mark Payne >Priority: Blocker > Fix For: 1.0.0 > > > The StandardProcessorTestRunner class of nifi-mock does not compile, as lines > 327 and 337 both have the following compilation error: > Cannot refer to the non-final local variable attributeName defined in an > enclosing scope -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-1943) PutFile should have ReadsAttribute annotation for "filename"
[ https://issues.apache.org/jira/browse/NIFI-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Witt updated NIFI-1943: -- Fix Version/s: 1.0.0 > PutFile should have ReadsAttribute annotation for "filename" > > > Key: NIFI-1943 > URL: https://issues.apache.org/jira/browse/NIFI-1943 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Joseph Percivall >Assignee: Joseph Percivall >Priority: Minor > Fix For: 1.0.0, 0.7.0 > > > The PutFile processor writes the flowfile to disk using the value contained > in the "filename" attribute. There should be documentation telling the user > this. > All that needs to be done is to add a "ReadsAttribute" annotation for this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-1937) GetHTTP should support configurable cookie policy
[ https://issues.apache.org/jira/browse/NIFI-1937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Witt updated NIFI-1937: -- Fix Version/s: 1.0.0 > GetHTTP should support configurable cookie policy > - > > Key: NIFI-1937 > URL: https://issues.apache.org/jira/browse/NIFI-1937 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 0.6.1 >Reporter: Michael Moser >Assignee: Michael Moser >Priority: Minor > Fix For: 1.0.0, 0.7.0 > > > After changes to GetHTTP in NIFI-1714, I found a corporate web site where > GetHTTP fails to download content. GetHTTP could successfully download > content from this site before NIFI-1714 was implemented. So that change > effectively broke access to this site. > I propose we add a new property to GetHTTP that allows the NiFi user to > choose the HTTPClient (Apache HTTPComponents) cookie policy. The property > would be called Redirect Cookie Policy which would be "When a HTTP server > responds to a request with a redirect, this is the cookie specification used > to copy cookies to the following request" -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-1880) AsciiDoc warnings in "NiFi In-Depth" documentation
[
https://issues.apache.org/jira/browse/NIFI-1880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Witt updated NIFI-1880:
--
Fix Version/s: (was: 0.7.0)
(was: 1.0.0)
> AsciiDoc warnings in "NiFi In-Depth" documentation
> --
>
> Key: NIFI-1880
> URL: https://issues.apache.org/jira/browse/NIFI-1880
> Project: Apache NiFi
> Issue Type: Bug
> Components: Documentation & Website
>Affects Versions: 0.6.1
>Reporter: Andy LoPresto
>Priority: Minor
> Labels: beginner, documentation
>
> When building the project, there are asciidoc errors and warnings in
> {{nifi-in-depth.adoc}}.
> {code}
> [INFO] --- asciidoctor-maven-plugin:1.5.2:process-asciidoc (output-html) @
> nifi-docs ---
> [INFO] Rendered
> /Users/alopresto/Workspace/nifi/nifi-docs/target/asciidoc/administration-guide.adoc
> [INFO] Rendered
> /Users/alopresto/Workspace/nifi/nifi-docs/target/asciidoc/developer-guide.adoc
> [INFO] Rendered
> /Users/alopresto/Workspace/nifi/nifi-docs/target/asciidoc/expression-language-guide.adoc
> [INFO] Rendered
> /Users/alopresto/Workspace/nifi/nifi-docs/target/asciidoc/getting-started.adoc
> asciidoctor: WARNING: nifi-in-depth.adoc: line 48: section title out of
> sequence: expected level 3, got level 4
> asciidoctor: WARNING: nifi-in-depth.adoc: line 57: section title out of
> sequence: expected level 3, got level 4
> asciidoctor: WARNING: nifi-in-depth.adoc: line 70: section title out of
> sequence: expected level 3, got level 4
> asciidoctor: WARNING: nifi-in-depth.adoc: line 89: section title out of
> sequence: expected level 3, got level 4
> asciidoctor: WARNING: nifi-in-depth.adoc: line 100: section title out of
> sequence: expected level 3, got level 4
> asciidoctor: WARNING: nifi-in-depth.adoc: line 104: section title out of
> sequence: expected level 3, got level 4
> asciidoctor: WARNING: nifi-in-depth.adoc: line 120: section title out of
> sequence: expected level 2, got level 4
> asciidoctor: WARNING: nifi-in-depth.adoc: line 147: section title out of
> sequence: expected level 3, got level 4
> asciidoctor: WARNING: nifi-in-depth.adoc: line 154: section title out of
> sequence: expected level 3, got level 4
> asciidoctor: WARNING: nifi-in-depth.adoc: line 170: section title out of
> sequence: expected level 3, got level 4
> asciidoctor: WARNING: nifi-in-depth.adoc: line 186: section title out of
> sequence: expected level 3, got level 4
> asciidoctor: WARNING: nifi-in-depth.adoc: line 193: section title out of
> sequence: expected level 3, got level 4
> asciidoctor: WARNING: nifi-in-depth.adoc: line 201: section title out of
> sequence: expected level 3, got level 4
> asciidoctor: ERROR: nifi-in-depth.adoc: line 206: only book doctypes can
> contain level 0 sections
> asciidoctor: ERROR: nifi-in-depth.adoc: line 206: only book doctypes can
> contain level 0 sections
> asciidoctor: ERROR: nifi-in-depth.adoc: line 206: only book doctypes can
> contain level 0 sections
> asciidoctor: ERROR: nifi-in-depth.adoc: line 206: only book doctypes can
> contain level 0 sections
> [INFO] Rendered
> /Users/alopresto/Workspace/nifi/nifi-docs/target/asciidoc/nifi-in-depth.adoc
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-1795) Improve documentation regarding PropertyDescriptor name & displayName
[
https://issues.apache.org/jira/browse/NIFI-1795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317833#comment-15317833
]
Joseph Witt commented on NIFI-1795:
---
[~alopresto] Can you advise what is next for this one? The PR appears closed.
> Improve documentation regarding PropertyDescriptor name & displayName
> -
>
> Key: NIFI-1795
> URL: https://issues.apache.org/jira/browse/NIFI-1795
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Documentation & Website
>Affects Versions: 0.6.1
>Reporter: Andy LoPresto
>Priority: Minor
> Fix For: 1.0.0, 0.7.0
>
>
> As evidenced in numerous pull requests, developers are often unaware of the
> behavior of {{PropertyDescriptor}} {{name}} and {{displayName}} attributes.
> Originally, {{name}} existed and served as both the UI display value for the
> processor properties configuration dialog and as the unique identifier when
> writing/reading the properties from file storage. However, a change to the
> {{name}} value after the flow had been persisted could cause issues
> retrieving the value from file storage. Thus, {{displayName}} was introduced
> as the UI display value to allow for backward-compatible updates. The
> documentation does not sufficiently explain this information and can better
> encourage developers to provide both {{name}} and {{displayName}} attributes
> in order to be proactive on flow compatibility and to improve the foundation
> for internationalization and localization of the UI.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (NIFI-1835) Support text/xml in the content view ui
[ https://issues.apache.org/jira/browse/NIFI-1835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Witt updated NIFI-1835: -- Fix Version/s: (was: 0.7.0) > Support text/xml in the content view ui > --- > > Key: NIFI-1835 > URL: https://issues.apache.org/jira/browse/NIFI-1835 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI >Reporter: Devin Fisher >Priority: Minor > > Currently, the content view supports application/xml mime type. This support > is nice when dealing with XML (especially the formatted view). But this > content viewer does not seem to support the text/xml mime type even though it > is the same as application/xml. > I'm using a work around of using UpdateAttribute to change the mime.type > attribute to appliaction/xml. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-1768) Add SSL Support for Solr Processors
[ https://issues.apache.org/jira/browse/NIFI-1768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Witt updated NIFI-1768: -- Fix Version/s: (was: 0.7.0) > Add SSL Support for Solr Processors > --- > > Key: NIFI-1768 > URL: https://issues.apache.org/jira/browse/NIFI-1768 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Bryan Bende >Assignee: Bryan Bende >Priority: Minor > > Currently the Solr processors do not support communicating with a Solr > instance that is secured with SSL. > We should be able to add the SSLContextService to the processor and pass an > SSLContext to the underlying HttpClient used by the SolrClient in the > SolrProcessor base class. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (NIFI-1828) PropertyDescriptor should use display name when doing validation
[ https://issues.apache.org/jira/browse/NIFI-1828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Witt resolved NIFI-1828. --- Resolution: Duplicate > PropertyDescriptor should use display name when doing validation > > > Key: NIFI-1828 > URL: https://issues.apache.org/jira/browse/NIFI-1828 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 0.6.1 >Reporter: Bryan Bende >Priority: Minor > > When PropertyDescriptor calls the validators for the given property, it > passes getName() as the subject, which means the regular name will be used in > the validation message in the UI. This should be the display name for cases > when the regular name is not a user friendly name. > One place where this occurs: > https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/components/PropertyDescriptor.java#L147 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-1731) CapturingLogger mishandles formatted messages
[
https://issues.apache.org/jira/browse/NIFI-1731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Witt updated NIFI-1731:
--
Fix Version/s: (was: 0.7.0)
> CapturingLogger mishandles formatted messages
> -
>
> Key: NIFI-1731
> URL: https://issues.apache.org/jira/browse/NIFI-1731
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Oleg Zhurakousky
>Assignee: Oleg Zhurakousky
>Priority: Minor
>
> Here is the example of such mishandling
> {code}
> public void warn(String format, Object arg) {
> warnMessages.add(new LogMessage(null, format, null, arg));
> logger.warn(format, arg);
> }
> {code}
> where it really has to be
> {code}
> public void warn(String format, Object arg) {
> Object[] args = new Object[] { arg };
> warnMessages.add(new LogMessage(null,
> MessageFormatter.arrayFormat(format, args).getMessage(), null, args));
> logger.warn(format, arg);
> }
> {code}
> Basically instead of _message_ we are currently passing _format_ into the
> 'message' field of LogMessage constructor which results in something like
> "{}" being the value of the message
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (NIFI-1694) EncryptContent processor should accept keyring file or individual key file for PGP encryption/decryption
[
https://issues.apache.org/jira/browse/NIFI-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Witt updated NIFI-1694:
--
Fix Version/s: (was: 0.7.0)
> EncryptContent processor should accept keyring file or individual key file
> for PGP encryption/decryption
>
>
> Key: NIFI-1694
> URL: https://issues.apache.org/jira/browse/NIFI-1694
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 0.6.0, 0.5.1
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Minor
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> As reported on the mailing lists [1], prior to {{0.5.1}}, users could provide
> the path to an individual PGP public key file (i.e.
> {{exportedPublicKey.asc}}) as the {{Public Keyring File}} property in the
> {{EncryptContent}} processor. With [NIFI-1324], the handling of the keyring
> files became more strict. While this follows the explicit naming of the
> property, some users may have individual keys rather than the keyring file.
> Add logic to support individual key handling for public and secret keys in
> {{EncryptContent}} processor via {{OpenPGPKeyBasedEncryptor}}.
> [1]
> http://mail-archives.apache.org/mod_mbox/nifi-dev/201603.mbox/%3ccaf3pksz8cdis5pvjnnm_qwoctl6pohr2gw-udnt0h2nfse3...@mail.gmail.com%3e
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-1660) Enhance the expression language with jsonPath function
[
https://issues.apache.org/jira/browse/NIFI-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317827#comment-15317827
]
Joseph Witt commented on NIFI-1660:
---
[~ch...@mcdermott.net] You think you'll have a chance to tackle this? We're
closing down for an 0.7.0 release and want to have this in if able.
> Enhance the expression language with jsonPath function
> --
>
> Key: NIFI-1660
> URL: https://issues.apache.org/jira/browse/NIFI-1660
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 0.5.1
>Reporter: Christopher McDermott
>Assignee: Joseph Witt
>Priority: Minor
> Fix For: 1.0.0, 0.7.0
>
>
> jsonPath would evaluate a JSON path provided, as an argument, against the
> subject.
> Example
> {quote}
> $\{kafka.key:jsonPath('$.foo.bar')\}
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (NIFI-1692) Upon shutdown instances of NarClassLoader are not closed
[ https://issues.apache.org/jira/browse/NIFI-1692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Witt updated NIFI-1692: -- Fix Version/s: (was: 0.7.0) > Upon shutdown instances of NarClassLoader are not closed > > > Key: NIFI-1692 > URL: https://issues.apache.org/jira/browse/NIFI-1692 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 0.6.0 >Reporter: Oleg Zhurakousky >Assignee: Oleg Zhurakousky >Priority: Minor > > Given that NarClassLoader is an instance of UrlClassLoader it must be > explicitly closed via close() method to allow for closing of any open > files/resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1628) Add support for configuring SSL for Tibco as a provider in new JMS module
[ https://issues.apache.org/jira/browse/NIFI-1628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317825#comment-15317825 ] Joseph Witt commented on NIFI-1628: --- [~JPercivall] [~ozhurakousky] is this ready to merge? Looks like it from the pr thread. Can we close this one out? Also should it include 1.0.0? > Add support for configuring SSL for Tibco as a provider in new JMS module > - > > Key: NIFI-1628 > URL: https://issues.apache.org/jira/browse/NIFI-1628 > Project: Apache NiFi > Issue Type: New Feature >Reporter: Oleg Zhurakousky >Assignee: Oleg Zhurakousky >Priority: Minor > Fix For: 0.7.0 > > > See discussion here https://github.com/apache/nifi/pull/222 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1594) Add option to bulk using Index or Update to PutElasticsearch
[ https://issues.apache.org/jira/browse/NIFI-1594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317823#comment-15317823 ] Joseph Witt commented on NIFI-1594: --- [~mattyb149] you happen to have time to review this one? Also should it also be tied to 1.0.0? > Add option to bulk using Index or Update to PutElasticsearch > > > Key: NIFI-1594 > URL: https://issues.apache.org/jira/browse/NIFI-1594 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: João Henrique Ferreira de Freitas >Priority: Minor > Fix For: 0.7.0 > > > I have a use case where two flowfiles needs to be write using > PutElasticsearch. Both will write to the same document but in different > properties. > The proposal is to let the user choice if an Update operation or Index is > needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-32) Swap File format should include summary at end for faster NiFi system startup
[ https://issues.apache.org/jira/browse/NIFI-32?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Witt updated NIFI-32: Fix Version/s: (was: 0.7.0) > Swap File format should include summary at end for faster NiFi system startup > - > > Key: NIFI-32 > URL: https://issues.apache.org/jira/browse/NIFI-32 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Minor > > When NiFi is restarted with many hundreds or thousands of Swap Files, it can > take a very long time to startup. > If when we write out the Swap Files, we append a summary of the Swap File > (queue identifier, number of FlowFiles, size of FlowFiles, and ContentClaim > info/count map), we can then read far less data for restart. This means we > would serialize the summary to a ByteArrayOutputStream, then get the length, > and then write out the summary, followed by length. On restore, we read just > the last 8 bytes, and that allows us to generate the offset into the file to > start reading the summary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-1958) FormatUtils does not handle time units larger than "day"
[
https://issues.apache.org/jira/browse/NIFI-1958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Witt updated NIFI-1958:
--
Fix Version/s: (was: 0.7.0)
(was: 1.0.0)
> FormatUtils does not handle time units larger than "day"
>
>
> Key: NIFI-1958
> URL: https://issues.apache.org/jira/browse/NIFI-1958
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 0.6.1
>Reporter: Andy LoPresto
> Labels: beginner, time, unit
>
> {{FormatUtils}}, the class used to handle unit conversions from user-provided
> configuration values, does not accept values using "week", "month", "year",
> etc. These units should be added.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-1866) Exception thrown by StandardProcessSession.exportTo implies issues reading from repository
[
https://issues.apache.org/jira/browse/NIFI-1866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317812#comment-15317812
]
Joseph Witt commented on NIFI-1866:
---
[~markap14] can you review and if good merge and close?
> Exception thrown by StandardProcessSession.exportTo implies issues reading
> from repository
> --
>
> Key: NIFI-1866
> URL: https://issues.apache.org/jira/browse/NIFI-1866
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 0.7.0
>
>
> When calling ProcessSession.exportTo, if unable to write to the OutputStream
> provided, we end up with a FlowFileAccessException, rather than IOException.
> This implies that we were unable to access the ContentRepository, but this is
> not accurate. We need to ensure that we throw the appropriate Exception here
> so that Processors that cause ProcessException are able to handle the
> Exception properly
> 23:23:03,338 ERROR Timer-Driven Process Thread-3
> standard.HandleHttpResponse:306 -
> org.apache.nifi.processor.exception.FlowFileAccessException: Failed to export
> StandardFlowFileRecord[uuid=366b3598-a1f7-446e-83b6-9d0404532691,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1462828983178-2, container=default,
> section=2], offset=49798,
> length=5120],offset=0,name=1095188501442536,size=5120] to
> HttpOutput@1eb25c5d{OPEN} due to org.eclipse.jetty.io.EofException
> at
> org.apache.nifi.controller.repository.StandardProcessSession.exportTo(StandardProcessSession.java:2322)
> at
> org.apache.nifi.processors.standard.HandleHttpResponse.onTrigger(HandleHttpResponse.java:166)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-1789) TestPutUDP testUnkownHostname fails because all hostnames resolve
[
https://issues.apache.org/jira/browse/NIFI-1789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317810#comment-15317810
]
Joseph Witt commented on NIFI-1789:
---
[~alopresto] can this be closed?
> TestPutUDP testUnkownHostname fails because all hostnames resolve
> -
>
> Key: NIFI-1789
> URL: https://issues.apache.org/jira/browse/NIFI-1789
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 0.6.1
>Reporter: Andy LoPresto
> Fix For: 1.0.0, 0.7.0
>
>
> On some systems, arbitrary hostnames (e.g. {{sgfgfdgs}}) resolves to a valid
> IP address depending on the DNS resolution scheme. Therefore, the test, which
> assumes that the host does not resolve, fails.
> {code}
> hw12203:/Users/alopresto (master) alopresto
> 0s @ 20:10:32 $ nslookup asdfsdfsdf
> Server: 192.168.1.1
> Address: 192.168.1.1#53
> Non-authoritative answer:
> Name: asdfsdfsdf
> Address: 198.105.254.228
> Name: asdfsdfsdf
> Address: 198.105.244.228
> hw12203:/Users/alopresto (master) alopresto
> 39s @ 20:12:24 $ dig asdfasdf
> ; <<>> DiG 9.8.3-P1 <<>> asdfasdf
> ;; global options: +cmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 31295
> ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0
> ;; QUESTION SECTION:
> ;asdfasdf.IN A
> ;; ANSWER SECTION:
> asdfasdf. 10 IN A 198.105.254.228
> asdfasdf. 10 IN A 198.105.244.228
> ;; Query time: 81 msec
> ;; SERVER: 192.168.1.1#53(192.168.1.1)
> ;; WHEN: Tue Apr 19 20:12:30 2016
> ;; MSG SIZE rcvd: 58
> hw12203:/Users/alopresto (master) alopresto
> 0s @ 20:12:30 $
> {code}
> Reported on Mac OS X 10.11.4
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-1730) org/apache/nifi/util/ReflectionUtils.java (and possibly other classes) exist in two different modules under the same FQN
[
https://issues.apache.org/jira/browse/NIFI-1730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317809#comment-15317809
]
Joseph Witt commented on NIFI-1730:
---
i recommend fixing it for 1.0.0 and deprecate them in 0.x line.
> org/apache/nifi/util/ReflectionUtils.java (and possibly other classes) exist
> in two different modules under the same FQN
>
>
> Key: NIFI-1730
> URL: https://issues.apache.org/jira/browse/NIFI-1730
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Oleg Zhurakousky
>Assignee: Oleg Zhurakousky
> Fix For: 1.0.0
>
>
> _org/apache/nifi/util/ReflectionUtils.java_ specifically exists in _framework
> core_ and _mock_ under the same FQN (see below)
> *
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/util/ReflectionUtils.java
> *
> https://github.com/apache/nifi/blob/master/nifi-mock/src/main/java/org/apache/nifi/util/ReflectionUtils.java
> This means that if the two versions are not identical one can start seeing
> something along the lines of this:
> {code}
> java.lang.NoSuchMethodError:
> org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(Ljava/lang/Class;Ljava/lang/Class;Ljava/lang/Object;[Ljava/lang/Object;)V
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-1722) Intermittent deadlocks in Kafka API when when stopping GetKafka
[
https://issues.apache.org/jira/browse/NIFI-1722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317808#comment-15317808
]
Joseph Witt commented on NIFI-1722:
---
is this being worked or should it be moved off 0.7.0?
> Intermittent deadlocks in Kafka API when when stopping GetKafka
> ---
>
> Key: NIFI-1722
> URL: https://issues.apache.org/jira/browse/NIFI-1722
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Oleg Zhurakousky
>Assignee: Oleg Zhurakousky
> Fix For: 0.7.0
>
>
> It appears that Kafka gets in the state of deadlock when
> _ConsumerConnector.commitOffsets(..)_ is executed during stop call in
> GetKafka. Looking at the thread dump it may be related to the fact that we
> are shutting down consumer when onTrigger is still executing. But It also
> appears that onTrigger is in the deadlock state as well.
> Below are the relevant thread dump segments
> {code}
> "StandardProcessScheduler Thread-7" Id=115 BLOCKED on
> java.lang.Object@2baae51
> at
> kafka.consumer.ZookeeperConsumerConnector.commitOffsets(ZookeeperConsumerConnector.scala:333)
> at
> kafka.consumer.ZookeeperConsumerConnector.commitOffsets(ZookeeperConsumerConnector.scala:324)
> "ConsumerFetcherThread-74993895-50e9-3962-90e3-97af1fed7294_daves-nifi-cluster-2-1459431214410-d3b5143c-0-1001"
> Id=25120 WAITING on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@23dc03a2
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:350)
> at
> kafka.consumer.PartitionTopicInfo.enqueue(PartitionTopicInfo.scala:60)
> at
> kafka.consumer.ConsumerFetcherThread.processPartitionData(ConsumerFetcherThread.scala:53)
> at
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1$$anonfun$apply$3.apply(AbstractFetcherThread.scala:142)
> at
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1$$anonfun$apply$3.apply(AbstractFetcherThread.scala:126)
> at scala.Option.foreach(Option.scala:236)
> at
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1.apply(AbstractFetcherThread.scala:126)
> at
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2$$anonfun$apply$mcV$sp$1.apply(AbstractFetcherThread.scala:123)
> at scala.collection.immutable.Map$Map2.foreach(Map.scala:130)
> at
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2.apply$mcV$sp(AbstractFetcherThread.scala:123)
> at
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2.apply(AbstractFetcherThread.scala:123)
> at
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$2.apply(AbstractFetcherThread.scala:123)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:298)
> at
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:122)
> at
> kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
> Number of Locked Synchronizers: 1
> - java.util.concurrent.locks.ReentrantLock$NonfairSync@6b99259f
> "ConsumerFetcherThread-33156eec-156c-4b32-9598-d5fc3ca460ce_daves-nifi-cluster-2-1459519432175-5beddff4-0-1001"
> Id=35266 RUNNABLE (in native code)
> at sun.nio.ch.Net.poll(Native Method)
> at sun.nio.ch.SocketChannelImpl.poll(SocketChannelImpl.java:954)
> - waiting on java.lang.Object@3e91ba2b
> at
> sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:204)
> - waiting on java.lang.Object@494070b0
> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
> - waiting on sun.nio.ch.SocketAdaptor$SocketInputStream@3ade0aef
> at
> java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
> - waiting on java.lang.Object@7d49c744
> at kafka.utils.CoreUtils$.read(CoreUtils.scala:192)
> at
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
> at kafka.network.Receive$class.readCompletely(Transmission.scala:56)
> at
> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
> at kafka.network.BlockingChannel.receive(BlockingChannel.scala:131)
> at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:81)
> at
>
[jira] [Updated] (NIFI-1730) org/apache/nifi/util/ReflectionUtils.java (and possibly other classes) exist in two different modules under the same FQN
[
https://issues.apache.org/jira/browse/NIFI-1730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Witt updated NIFI-1730:
--
Fix Version/s: (was: 0.7.0)
1.0.0
> org/apache/nifi/util/ReflectionUtils.java (and possibly other classes) exist
> in two different modules under the same FQN
>
>
> Key: NIFI-1730
> URL: https://issues.apache.org/jira/browse/NIFI-1730
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Oleg Zhurakousky
>Assignee: Oleg Zhurakousky
> Fix For: 1.0.0
>
>
> _org/apache/nifi/util/ReflectionUtils.java_ specifically exists in _framework
> core_ and _mock_ under the same FQN (see below)
> *
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/util/ReflectionUtils.java
> *
> https://github.com/apache/nifi/blob/master/nifi-mock/src/main/java/org/apache/nifi/util/ReflectionUtils.java
> This means that if the two versions are not identical one can start seeing
> something along the lines of this:
> {code}
> java.lang.NoSuchMethodError:
> org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(Ljava/lang/Class;Ljava/lang/Class;Ljava/lang/Object;[Ljava/lang/Object;)V
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (NIFI-1674) PutKafka should pull in multiple FlowFiles to meet Batch Size if not using delimiter
[ https://issues.apache.org/jira/browse/NIFI-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Witt resolved NIFI-1674. --- Resolution: Invalid Fix Version/s: (was: 0.7.0) this is no longer relevant as the kafka processors have been reimplemented and taken in a new direction. > PutKafka should pull in multiple FlowFiles to meet Batch Size if not using > delimiter > > > Key: NIFI-1674 > URL: https://issues.apache.org/jira/browse/NIFI-1674 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Mark Payne > > If a Batch Size is specified in PutKafka, but there is no delimiter used, > then we should pull in up to BATCH_SIZE FlowFiles and send them together as a > single batch instead of sending each FlowFile individually -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1118) Enable SplitText processor to limit line length and filter header lines
[ https://issues.apache.org/jira/browse/NIFI-1118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317792#comment-15317792 ] Joseph Witt commented on NIFI-1118: --- Hello - is this one done? Looks like it was merged. If so please close [~mosermw]. > Enable SplitText processor to limit line length and filter header lines > --- > > Key: NIFI-1118 > URL: https://issues.apache.org/jira/browse/NIFI-1118 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Mark Bean >Assignee: Mark Bean > Fix For: 0.7.0 > > > Include the following functionality to the SplitText processor: > 1) Maximum size limit of the split file(s) > A new split file will be created if the next line to be added to the current > split file exceeds a user-defined maximum file size > 2) Header line marker > User-defined character(s) can be used to identify the header line(s) of the > data file rather than a predetermined number of lines > These changes are additions, not a replacement of any property or behavior. > In the case of header line marker, the existing property "Header Line Count" > must be zero for the new property and behavior to be used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-53) Framework should not trigger components to run if input queue has only penalized FlowFiles
[ https://issues.apache.org/jira/browse/NIFI-53?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Witt updated NIFI-53: Fix Version/s: (was: 0.7.0) (was: 1.0.0) > Framework should not trigger components to run if input queue has only > penalized FlowFiles > -- > > Key: NIFI-53 > URL: https://issues.apache.org/jira/browse/NIFI-53 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Joseph Witt >Assignee: Oleg Zhurakousky > > Framework should avoid calling on trigger in the event all items in the input > queue are penalized (this is for those cases when checking 'do you have data > to process') -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-53) Framework should not trigger components to run if input queue has only penalized FlowFiles
[ https://issues.apache.org/jira/browse/NIFI-53?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317790#comment-15317790 ] Joseph Witt commented on NIFI-53: - It is quite reasonable that when pulling data from a queue that this queue could be empty when doing so. That is all this means. And since we allow processors to pull zero or more things in a given pass then another thread could be given a thread and by then there could be no data left. Not at all an issue in my view. > Framework should not trigger components to run if input queue has only > penalized FlowFiles > -- > > Key: NIFI-53 > URL: https://issues.apache.org/jira/browse/NIFI-53 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: Joseph Witt >Assignee: Oleg Zhurakousky > > Framework should avoid calling on trigger in the event all items in the input > queue are penalized (this is for those cases when checking 'do you have data > to process') -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-1696) Event driven tasks appear not to auto restart when data in queues but no new data
[ https://issues.apache.org/jira/browse/NIFI-1696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Witt updated NIFI-1696: -- Fix Version/s: (was: 0.7.0) > Event driven tasks appear not to auto restart when data in queues but no new > data > - > > Key: NIFI-1696 > URL: https://issues.apache.org/jira/browse/NIFI-1696 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 0.6.0 >Reporter: Joseph Witt >Priority: Critical > > The scenario: > - 3 node cluster > - node 'NodeA' is primary and running a processor 'ProcP1' set to event > driven scheduling > - 'ProcP1' wasn't working due to a config issue but there were a few events > on its input queues. > - restarted node 'NodeA' after fixing config issue. > - No new events had come in but those events were sitting there. > Result: The 'ProcP1' appears to be in scheduled state but does not get > scheduled to run - perhaps because no new event triggered it? Seems like > perhaps we're not checking if there are already events at restart? Stopping > and starting 'ProcP1' then resulted in things going back to normal. > It is not obvious to me how to recreate this in a repeatable way. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1052) Improve flow behavior on refactored processor names/packages
[ https://issues.apache.org/jira/browse/NIFI-1052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15314882#comment-15314882 ] Joseph Witt commented on NIFI-1052: --- just chatted with mark on this. Question was what to do given we don't know whether a processor's properties are sensitive. So we'll have to treat all of them as sensitive and not make them visible. But at least nifi will start up and this thing would be invalid and once they fix the classpath all will be good again > Improve flow behavior on refactored processor names/packages > > > Key: NIFI-1052 > URL: https://issues.apache.org/jira/browse/NIFI-1052 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: sumanth chinthagunta >Assignee: Joseph Witt > Fix For: 0.7.0 > > > My case is simple and easy to reproduce: > 1. Create a flow with custom processor (e.g. abc.MyProcesser ) > 2. Make sure flow is working. Then stop the server. > 3. Replace nar in lib with new nar after refactoring processor's package ( > e.g., xzy.MyProcesser) > 4. When you try to start nifi , it fails with error in the log saying class > not found ( obviously coz deployed flow is still looking for old class) > My expectation is system should at least start and only throw error when > user try to start a flow that that had reference to class(processor) which > is no more available in the newly deployed nar. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (NIFI-1970) Memorialize Apache NiFi SWAG logo
[ https://issues.apache.org/jira/browse/NIFI-1970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Witt updated NIFI-1970: -- Attachment: logo-nifi-drop.ai logo-nifi.ai > Memorialize Apache NiFi SWAG logo > - > > Key: NIFI-1970 > URL: https://issues.apache.org/jira/browse/NIFI-1970 > Project: Apache NiFi > Issue Type: Task >Reporter: Joseph Witt > Attachments: logo-nifi-drop.ai, logo-nifi.ai > > > Rob Moran made this image. It was used to order apache nifi tshirts and > stickers which I'm documenting on the wiki. Using this to hold the image > file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (NIFI-1970) Memorialize Apache NiFi SWAG logo
Joseph Witt created NIFI-1970: - Summary: Memorialize Apache NiFi SWAG logo Key: NIFI-1970 URL: https://issues.apache.org/jira/browse/NIFI-1970 Project: Apache NiFi Issue Type: Task Reporter: Joseph Witt Rob Moran made this image. It was used to order apache nifi tshirts and stickers which I'm documenting on the wiki. Using this to hold the image file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-915) Properties are not trimming whitespace
[ https://issues.apache.org/jira/browse/NIFI-915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15310373#comment-15310373 ] Joseph Witt commented on NIFI-915: -- [~msclarke] Yes but we need to be careful about where we do auto trimming. I think doing so for things in nifi.properties should be fine. Doing it on processor properties is another matter. > Properties are not trimming whitespace > -- > > Key: NIFI-915 > URL: https://issues.apache.org/jira/browse/NIFI-915 > Project: Apache NiFi > Issue Type: Bug > Components: Configuration >Affects Versions: 0.3.0 >Reporter: Edgardo Vega >Priority: Minor > > Automatically trim trailing white space for properties in nifi.properties. > Ran into this when setting nifi.sensitive.props.key -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1754) Rollback log messages should include the flowfile filename and UUID to assist in flow management.
[ https://issues.apache.org/jira/browse/NIFI-1754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15308125#comment-15308125 ] Joseph Witt commented on NIFI-1754: --- i do too - i looked last night and thought it was good to go. Sorry I did not comment. Nationals game was on! [~jskora] we should def have a conversation about what we should be thinking for next phase(s) on what/how to continually make this better in terms of helping dev and ops have maximum communication about issues so they can be understood and resolved quickly. The whole lifecycle of development and operations is something we should constantly fight to improve. > Rollback log messages should include the flowfile filename and UUID to assist > in flow management. > - > > Key: NIFI-1754 > URL: https://issues.apache.org/jira/browse/NIFI-1754 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.0.0, 0.7.0 >Reporter: Joe Skora >Assignee: Joe Skora > > When a processor calls rollback on a flowfile, the filename and UUID of the > flowfile should be included in log entry to make it easier for the offending > files to be found and fixed or removed from the flow by the dataflow managers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1265) Precompiling message-page.jsp causes runtime error in CentOS 6.7 and Fedora 23
[
https://issues.apache.org/jira/browse/NIFI-1265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303359#comment-15303359
]
Joseph Witt commented on NIFI-1265:
---
[~mcgilman] This looks a lot like what is mentioned here
http://stackoverflow.com/questions/37462150/two-java-exceptions-but-new-to-java-on-linux-using-the-nifi-by-apache/37473899#37473899
Also the previous comment suggests perhaps jetty 9.2.-latest has this patched
as well.
> Precompiling message-page.jsp causes runtime error in CentOS 6.7 and Fedora 23
> --
>
> Key: NIFI-1265
> URL: https://issues.apache.org/jira/browse/NIFI-1265
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core UI
>Reporter: Matt Gilman
>Priority: Minor
> Fix For: 0.7.0
>
> Attachments: jetty.447816.patch
>
>
> During start up it appears that when message-page.jsp is precompiled and run
> in CentOS 6.7 or Fedora 23 the following method in ServletHolder (Jetty)
> fails with:
> {noformat}
> 2015-12-07 10:01:26,557 WARN [main] org.apache.nifi.web.server.JettyServer
> Failed to start web server... shutting down.
> java.lang.IllegalArgumentException: Comparison method violates its general
> contract!
> at java.util.ComparableTimSort.mergeHi(ComparableTimSort.java:866)
> ~[na:1.8.0_65]
> at java.util.ComparableTimSort.mergeAt(ComparableTimSort.java:483)
> ~[na:1.8.0_65]
> at
> java.util.ComparableTimSort.mergeForceCollapse(ComparableTimSort.java:422)
> ~[na:1.8.0_65]
> at java.util.ComparableTimSort.sort(ComparableTimSort.java:222)
> ~[na:1.8.0_65]
> at java.util.Arrays.sort(Arrays.java:1246) ~[na:1.8.0_65]
> at
> org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:865)
> ~[jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
> at
> org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:298)
> ~[jetty-servlet-9.2.11.v20150529.jar:9.2.11.v20150529]
> at
> org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1349)
> ~[jetty-webapp-9.2.11.v20150529.jar:9.2.11.v20150529]
> {noformat}
> {code}
> @Override
> public int compareTo(ServletHolder sh)
> {
> if (sh==this)
> return 0;
> if (sh._initOrder<_initOrder)
> return 1;
> if (sh._initOrder>_initOrder)
> return -1;
> int c=(_className!=null &&
> sh._className!=null)?_className.compareTo(sh._className):0;
> if (c==0)
> c=_name.compareTo(sh._name);
> return c;
> }
> {code}
> After a quick glance it may be that this method is not transitive when
> falling back to comparing by className and name. From the Java API
> {noformat}
> The implementor must also ensure that the relation is transitive:
> ((compare(x, y)>0) && (compare(y, z)>0)) implies compare(x, z)>0.
> {noformat}
> This could fail with the following scenario:
> {noformat}
> x._initOrder = -1
> x._className = null
> x._name = Login
> y._initOrder = -1
> y._className = org.apache.nifi.web.jsp.WEB_002dINF.pages.message_002dpage_jsp
> y._name = org.apache.nifi.web.jsp.WEB_002dINF.pages.message_002dpage_jsp
> z._initOrder = -1
> z._className = org.apache.nifi.web.servlet.DownloadSvg
> z._name = DownloadSvg
> {noformat}
> Unfortunately, I am unable to replicate the exception. However, my
> observation from debugging show that when JSPs are precompiled the _className
> and _name are the same (like y above). When a Servlet is referenced in the
> web.xml it's name is overridden (like z above). And when a JSP is referenced
> in the web.xml the _className is null and the _name is set to the name from
> the web.xml (like x above).
> In the scenario outlined: X < Y and Y < Z but Z < X. When running locally I
> did not see the exception mentioned above but the resulting sort order was
> not correct as it ultimately depended on which elements were compared to each
> other.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (NIFI-1922) Add support for WASB / Azure Blob Storage with HDFS processors
Joseph Witt created NIFI-1922: - Summary: Add support for WASB / Azure Blob Storage with HDFS processors Key: NIFI-1922 URL: https://issues.apache.org/jira/browse/NIFI-1922 Project: Apache NiFi Issue Type: New Feature Reporter: Joseph Witt https://hadoop.apache.org/docs/stable2/hadoop-azure/index.html Received two mailing list questions on this in the past couple weeks. We should look at what things we'd need to do in the processors to make this possible. Adding the dependency is one, anything else? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1909) Add ability to handle schemaless records to ConvertAvroToJson
[ https://issues.apache.org/jira/browse/NIFI-1909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294004#comment-15294004 ] Joseph Witt commented on NIFI-1909: --- [~rpersaud] Thanks for submitting the idea. How would you envision this working? Would you like to see it based on some processor property which specifies/provides a given schema definition or would you like to see it based on some schema registry/lookup mechanism? > Add ability to handle schemaless records to ConvertAvroToJson > - > > Key: NIFI-1909 > URL: https://issues.apache.org/jira/browse/NIFI-1909 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Ryan Persaud > > In the interest of saving space, Avro messages are not always accompanied by > their schema. Currently, ConvertAvroToJson does not handle such schemaless > records. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-742) discuss github-pr and patch-on-jira tradeoff in contrib guide
[ https://issues.apache.org/jira/browse/NIFI-742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15293869#comment-15293869 ] Joseph Witt commented on NIFI-742: -- i agree this is working pretty well for now but would welcome other views. What really turned the corner for us was when Aldrin got Travis-CI integrated (and the upcoming appveyor piece). We're doing pretty well but our ability to stay on top of PRs in Github and such does need some focus. I am hopeful the upcoming meetup/hackathon will help put a dent in that which obviously isnt a fix but still.. > discuss github-pr and patch-on-jira tradeoff in contrib guide > - > > Key: NIFI-742 > URL: https://issues.apache.org/jira/browse/NIFI-742 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation & Website >Reporter: Sean Busbey > > once NIFI-577 goes in, add a note in the contrib guide about the tradeoff for > github pr vs patch on jira. > * review ease > * response traffic > * precommit patch test > (this will likely change substantially if the github support is live in the > release of the patch tester we start with) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1799) Auto adjust flow layout
[
https://issues.apache.org/jira/browse/NIFI-1799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15285901#comment-15285901
]
Joseph Witt commented on NIFI-1799:
---
my view is that this is fair game as 1.x only. It is a feature in 1.x to aid
migration from 0.x so seems ok not to port back.
> Auto adjust flow layout
> ---
>
> Key: NIFI-1799
> URL: https://issues.apache.org/jira/browse/NIFI-1799
> Project: Apache NiFi
> Issue Type: Sub-task
> Components: Core Framework, Core UI
>Reporter: Matt Gilman
>Assignee: Jeff Storck
> Fix For: 1.0.0
>
>
> Updates to the canvas to reflect the modernized look and feel and component
> level authorization will increase the size of each component. We should auto
> adjust the layouts to the components will not overlap. Below are the current
> size of each component and the new size:
> {code}
> Processor 310x100 350x130
> Process Group 365x142 380x172
> Remote Process Group 365x140 380x176
> Connection (largest) 188x53 200x55
> Port (root) 160x56 240x75
> Port 160x40 240x50
> Funnel 61x61 48x48
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-1754) Rollback log messages should include the flowfile filename and UUID to assist in flow management.
[
https://issues.apache.org/jira/browse/NIFI-1754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15285269#comment-15285269
]
Joseph Witt commented on NIFI-1754:
---
{quote}
The original requirement was on an operations need, without being able to
identify flow files causing the Administrative Yield they couldn't track
whether the root causes were system issues, framework issues, processor issues,
or flow file problems.
{quote}
They are only issues of 'what happens inside processor executed code' and those
issues could be as you list (bad system state, bad framework thing, bad
processor code). But we cannot know the difference. The question is 'what
information can and should we make available through the REST API and UI when
this occurs'. I don't believe this has been fully thought through yet. The
relationship between a flowfile and a session that happened to get rolled back
is not clear. We can only really say "hey this flowfile was part of a session
recently that had to get rolled back and here the stacktrace that prompted the
rollback". I agree that information is useful to a developer and that is why I
am in favor of making that information available through the logs. I think we
can do more than current API/UI enabled information of "hey a rollback
happened" but I don't think that is changing attributes on a flow file. More
thought and user experience discussion needs to occur there.
{quote}
1. Move the rollback count value from the FF attribute to a FlowFileRecord
property.
{quote}
In my opinion this should be avoided at this point. We do not have a fully
thought through user experience for this scenario. I would like to see much
more discussion around that user experience and how that might work. For
example we should track process session statistics tied to a given
component/processor and in that we could include information about recent
rollback which could correlate to flowfiles. Just ideas...but point is we need
to think about the user experience and then think about code/api implications.
{quote}
2. Move the unacknowledged FF logging from AbstractProcessor into the
ProcessSession where the counting already is being done.
{quote}
Yes I agree that is a good next step. I'd keep all of the session logging
within the session class itself and not alter the existing API at all. This
allows us to make available more helpful debug data while we evaluate other
options for offering a better 'developer' centric user experience to expose
these things if that makes sense. We need to think about things like
authorization as well because this could include some rather sensitive
information.
{quote}
3. Eliminate the configuration entries by moving the logging to DEBUG level to
allow it to be controlled by logging configuration if necessary and always
counting rollbacks on the FlowFileRecord.
{quote}
I believe the item in #2 will require no changes to properties so yes I'd say
remove those. I agree just logback.xml changes would be needed for cases where
people want those details in the logs. For counting rollbacks my response from
#1 applies.
{quote}
4. Rename the FlowDebugger processor to DebugFlow and incorporate as a separate
ticket and PR.
{quote}
Yes please. I did not honestly dig into the details of DebugFlow yet but very
much think it is a cool concept and something I had been wanting to do/have for
a long time. Happy to review that one as well.
Thanks for the discussion. I absolutely want to respect what has clearly been
a lot of thought and effort.
> Rollback log messages should include the flowfile filename and UUID to assist
> in flow management.
> -
>
> Key: NIFI-1754
> URL: https://issues.apache.org/jira/browse/NIFI-1754
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.0.0, 0.7.0
>Reporter: Joe Skora
>Assignee: Joe Skora
>
> When a processor calls rollback on a flowfile, the filename and UUID of the
> flowfile should be included in log entry to make it easier for the offending
> files to be found and fixed or removed from the flow by the dataflow managers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-1754) Rollback log messages should include the flowfile filename and UUID to assist in flow management.
[ https://issues.apache.org/jira/browse/NIFI-1754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15285237#comment-15285237 ] Joseph Witt commented on NIFI-1754: --- The only things that came to mind are like what you already have covered already in the log string generated. Good to only log a few as well. > Rollback log messages should include the flowfile filename and UUID to assist > in flow management. > - > > Key: NIFI-1754 > URL: https://issues.apache.org/jira/browse/NIFI-1754 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.0.0, 0.7.0 >Reporter: Joe Skora >Assignee: Joe Skora > > When a processor calls rollback on a flowfile, the filename and UUID of the > flowfile should be included in log entry to make it easier for the offending > files to be found and fixed or removed from the flow by the dataflow managers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1754) Rollback log messages should include the flowfile filename and UUID to assist in flow management.
[
https://issues.apache.org/jira/browse/NIFI-1754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15283953#comment-15283953
]
Joseph Witt commented on NIFI-1754:
---
In response to feedback about the new api method getUnacknowledgedFlowFiles, on
the Github PR [~jskora] wrote:
{quote}
The root problem is tracking "Administratively Yielding" messages back to a
flow file, so it lists the unacknowledged flow files since committed and
transferred files were already processed. Looking at where the
"Administratively Yielding" messages come from in EventDrivenSchedulingAgent
and ContinuallyRunProcessTask, the framework doesn't have visibility into the
active processor that threw the uncaught exception. This implementation
populates the log message without taxing the framework or requiring support in
the processors, but it can still be overridden for customization by individual
processors.
Providing flow files objects instead of a log message would expose internals
between framework layers and complicate api and implementation module
dependencies in the build. The upper level framework delegates flow file
handling further down, so this approach similarly delegates the log message
creation with minimal risk and impact on the framework.
{quote}
Let's address this first statement as it appears to be close to a requirement.
It says
{quote}
The root problem is tracking "Administratively Yielding" messages back to a
flow file, so it lists the unacknowledged flow files since committed and
transferred files were already processed.
{quote}
The concept of a rollback exists to return the state of the flow back to where
it was before a given session was created or last committed with the notable
exception of any provenance events which are deemed to need to be persisted
because they represent things which happened and cannot be rolled back and
therefore must be tracked (such as giving data to another system). The primary
case where rollback can occur is when the framework catches an unhandled
exception from the processor which clearly indicates a programming error with
that processor or some faulty system state which caused it (like an out of
memory exception). Rollback could also occur due to a failure on commit or due
to the programmer intentionally calling rollback. In the event of rollback due
to unhandled exceptions then the framework will also cause an 'administrative
yield' which is simply there to protect the framework from what is very
obviously a programming error in that particular processor. This is really the
case we're trying to solve here i believe so let's restate the requirement...
_As a developer of a processor in a NiFi flow I need detailed information about
the state of the processor at the time some unexpected exception occurred. The
state of the processor includes the detailed stacktrace and what flowfile(s) it
might have been operating on at that time. As an administrator of a NiFi flow
I also need to be able to gather this information so I can provide it to the
developer so they can fix the processor._
An important thing to keep in mind here is we are only talking about how to get
developers better information so they can fix their processor.
Next statement
{quote}
Looking at where the "Administratively Yielding" messages come from in
EventDrivenSchedulingAgent and ContinuallyRunProcessTask, the framework doesn't
have visibility into the active processor that threw the uncaught exception.
{quote}
It does have the ProcessorNode and ProcessContext objects so it knows precisely
which component the uncaught exception came from. What it does not have today
is the details of which flow files were involved in the process session at the
time of the uncaught exception. I'll comment on that when we get to the part
about leaking information across layers.
Next statement
{quote}
This implementation populates the log message without taxing the framework or
requiring support in the processors, but it can still be overridden for
customization by individual processors.
{quote}
It appears the log message is generated by the ProcessSession implementation.
With that in mind I don't understand how processors can override its behavior.
Next statement
{quote}
Providing flow files objects instead of a log message would expose internals
between framework layers and complicate api and implementation module
dependencies in the build.
{quote}
I agree this is not ideal. I'd prefer we avoid adding the String method as-is
in the patch and I'd also like to withdraw my suggestion to do this flow file
thing too. Instead what I recommend is that this patch simply focus in on the
simplest possible step we can take to honor the requirement as stated above and
to the original spirit of the ticket.
To that end I propose the change be as simple as in the event rollback is
called on the standard
[jira] [Commented] (NIFI-1856) ExecuteStreamCommand Needs to Consume Standard Error
[
https://issues.apache.org/jira/browse/NIFI-1856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15283782#comment-15283782
]
Joseph Witt commented on NIFI-1856:
---
Karthik - i've listed you in jira as a contributor to apache nifi and assigned
teh JIRA to you. Thanks!
> ExecuteStreamCommand Needs to Consume Standard Error
>
>
> Key: NIFI-1856
> URL: https://issues.apache.org/jira/browse/NIFI-1856
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Alan Jackoway
>Assignee: Karthik Narayanan
>
> I was using ExecuteStreamProcess to run certain hdfs commands that are tricky
> to write in nifi but easy in bash (e.g. {{hadoop fs -rm -r
> /data/*/2014/05/05}})
> However, my larger commands kept hanging even though when I run them from the
> command line they finish quickly.
> Based on
> http://www.javaworld.com/article/2071275/core-java/when-runtime-exec---won-t.html
> I believe that ExecuteStreamCommand and possibly other processors need to
> consume the standard error stream to prevent the processes from blocking when
> standard error gets filled.
> To reproduce. Create this as ~/write.py
> {code:python}
> import sys
> count = int(sys.argv[1])
> for x in range(count):
> sys.stderr.write("ERROR %d\n" % x)
> sys.stdout.write("OUTPUT %d\n" % x)
> {code}
> Create a flow that goes
> # GenerateFlowFile - 5 minutes schedule 0 bytes size
> # ExecuteStreamCommand - Command arguments /Users/alanj/write.py;100 Command
> Path python
> # PutFile - /tmp/write/
> routing output stream of ExecuteStreamCommand to PutFile
> When you turn everything on, you get 100 lines (not 200) of just the standard
> output in /tmp/write.
> Next, change the command arguments to /Users/alanj/write.py;10 and turn
> everything on again. The command will hang.
> I believe that whenever you execute a process the way ExecuteStreamCommand is
> doing, you need to consume the standard error stream to keep it from
> blocking. This may also affect things like ExecuteProcess and ExecuteScript
> as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (NIFI-1856) ExecuteStreamCommand Needs to Consume Standard Error
[
https://issues.apache.org/jira/browse/NIFI-1856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Witt updated NIFI-1856:
--
Assignee: Karthik Narayanan
> ExecuteStreamCommand Needs to Consume Standard Error
>
>
> Key: NIFI-1856
> URL: https://issues.apache.org/jira/browse/NIFI-1856
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Alan Jackoway
>Assignee: Karthik Narayanan
>
> I was using ExecuteStreamProcess to run certain hdfs commands that are tricky
> to write in nifi but easy in bash (e.g. {{hadoop fs -rm -r
> /data/*/2014/05/05}})
> However, my larger commands kept hanging even though when I run them from the
> command line they finish quickly.
> Based on
> http://www.javaworld.com/article/2071275/core-java/when-runtime-exec---won-t.html
> I believe that ExecuteStreamCommand and possibly other processors need to
> consume the standard error stream to prevent the processes from blocking when
> standard error gets filled.
> To reproduce. Create this as ~/write.py
> {code:python}
> import sys
> count = int(sys.argv[1])
> for x in range(count):
> sys.stderr.write("ERROR %d\n" % x)
> sys.stdout.write("OUTPUT %d\n" % x)
> {code}
> Create a flow that goes
> # GenerateFlowFile - 5 minutes schedule 0 bytes size
> # ExecuteStreamCommand - Command arguments /Users/alanj/write.py;100 Command
> Path python
> # PutFile - /tmp/write/
> routing output stream of ExecuteStreamCommand to PutFile
> When you turn everything on, you get 100 lines (not 200) of just the standard
> output in /tmp/write.
> Next, change the command arguments to /Users/alanj/write.py;10 and turn
> everything on again. The command will hang.
> I believe that whenever you execute a process the way ExecuteStreamCommand is
> doing, you need to consume the standard error stream to keep it from
> blocking. This may also affect things like ExecuteProcess and ExecuteScript
> as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-1754) Rollback log messages should include the flowfile filename and UUID to assist in flow management.
[ https://issues.apache.org/jira/browse/NIFI-1754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15283091#comment-15283091 ] Joseph Witt commented on NIFI-1754: --- [~jskora] I commented on the pira itself with a few more specific things but general thoughts: # I think this should be simplified to just the basic original intent. The intent of the JIRA, as i understood it, was to provide additional details about impacted flow files in the case of a session rollback and specifically in the 'log message' of a rollback. So that is why I suggested there be a toString or similar method which was presuming it would be a private/internally called thing by the 'rollback' method to ensure that a nicely logged output occurred. That should not require API changes. # The additional/new nifi properties should be avoided unless it became clear at some point user configuration of something like this was necessary. # Avoid changing flow file attributes. They do not get persisted when a flow is restarted nor do they appear to get reset once the condition causing them to rollback is cleared and thus it gets committed. So it could be misleading down on later in the flow. Instead, another mechanism of storing this information for internal or user experience purposes should be pursued such as the FlowFileRecord itself which does not get exposed to the public API portions - is just an internal thing which could be wired up with queue management/visualization/REST responses. But I'd recommend deferring this part until later and if it was clear that it was worth it. With already done queue management work and with your first initial idea further changes here may be not necessary. Rollback is intended to 'get back to the state we were in with the flowfile before we called this clearly broken processor code'. Therefore we don't want to be changing the nature of the flow file in such a way that could be used to alter the flow itself - as in - these attributes changing should not then be used to route the flow files elsewhere and so on. These are not persistent, outside the tracking of provenance, etc.. so not a good track. But your original intent/idea I am totally behind which is to help provide more tooling - initially in the form of logs - to understand flowfiles impacted by a rollback. I see that as valuable personally because that can at times help inform one trying to figure out what the bug was in the processor. Because again rollback should only ever occur due to a bug in the processor calling the session commit OR due to a failure of the framework itself while trying to commit. # Consider applying to 1.x line only but if needing to be in both there would likely need to be a PR for each to accommodate the differences. # FlowDebugger is a cool concept. It should be named 'DebugFlow' to follow the VerbSubject naming construct and it should not be in this PR/JIRA. It should be its own. And it can be applied to both 1.x and 0.x. > Rollback log messages should include the flowfile filename and UUID to assist > in flow management. > - > > Key: NIFI-1754 > URL: https://issues.apache.org/jira/browse/NIFI-1754 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.0.0, 0.7.0 >Reporter: Joe Skora >Assignee: Joe Skora > > When a processor calls rollback on a flowfile, the filename and UUID of the > flowfile should be included in log entry to make it easier for the offending > files to be found and fixed or removed from the flow by the dataflow managers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1872) ZooKeeper State Provider unit tests failing
[ https://issues.apache.org/jira/browse/NIFI-1872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15282992#comment-15282992 ] Joseph Witt commented on NIFI-1872: --- +1 for pr441. Pushed to master. > ZooKeeper State Provider unit tests failing > --- > > Key: NIFI-1872 > URL: https://issues.apache.org/jira/browse/NIFI-1872 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.0.0, 0.7.0 > > > There appears to be an issue with the unit tests for the ZooKeeper State > Provider. The client intermittently receives a ConnectionLossException when > attempting to fetch the state from ZK. We need to properly address the unit > test, but i propose that we should immediately mark the unit test as ignored > until we have a chance to properly fix the test, as it is currently > preventing master from building. We end up with the following stack trace: > Tests run: 171, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 153.028 sec > <<< FAILURE! - in TestSuite > testStateTooLargeExceptionThrownOnReplace on > testStateTooLargeExceptionThrownOnReplace(org.apache.nifi.controller.state.providers.zookeeper.TestZooKeeperStateProvider)(org.apache.nifi.controller.state.providers.zookeeper.TestZooKeeperStateProvider) > Time elapsed: 5.841 sec <<< FAILURE! > java.io.IOException: Failed to set cluster-wide state in ZooKeeper for > component with ID 1---- > at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1270) > at > org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:318) > at > org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:283) > at > org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:228) > at > org.apache.nifi.controller.state.providers.zookeeper.TestZooKeeperStateProvider.testStateTooLargeExceptionThrownOnReplace(TestZooKeeperStateProvider.java:163) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1872) ZooKeeper State Provider unit tests failing
[ https://issues.apache.org/jira/browse/NIFI-1872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15282967#comment-15282967 ] Joseph Witt commented on NIFI-1872: --- +1 for pr440. Pushed to master. Checking 0.x now > ZooKeeper State Provider unit tests failing > --- > > Key: NIFI-1872 > URL: https://issues.apache.org/jira/browse/NIFI-1872 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne >Assignee: Mark Payne > Fix For: 1.0.0, 0.7.0 > > > There appears to be an issue with the unit tests for the ZooKeeper State > Provider. The client intermittently receives a ConnectionLossException when > attempting to fetch the state from ZK. We need to properly address the unit > test, but i propose that we should immediately mark the unit test as ignored > until we have a chance to properly fix the test, as it is currently > preventing master from building. We end up with the following stack trace: > Tests run: 171, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 153.028 sec > <<< FAILURE! - in TestSuite > testStateTooLargeExceptionThrownOnReplace on > testStateTooLargeExceptionThrownOnReplace(org.apache.nifi.controller.state.providers.zookeeper.TestZooKeeperStateProvider)(org.apache.nifi.controller.state.providers.zookeeper.TestZooKeeperStateProvider) > Time elapsed: 5.841 sec <<< FAILURE! > java.io.IOException: Failed to set cluster-wide state in ZooKeeper for > component with ID 1---- > at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1270) > at > org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:318) > at > org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:283) > at > org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:228) > at > org.apache.nifi.controller.state.providers.zookeeper.TestZooKeeperStateProvider.testStateTooLargeExceptionThrownOnReplace(TestZooKeeperStateProvider.java:163) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1872) ZooKeeper State Provider unit tests failing
[ https://issues.apache.org/jira/browse/NIFI-1872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15282789#comment-15282789 ] Joseph Witt commented on NIFI-1872: --- and also this Tests run: 171, Failures: 1, Errors: 0, Skipped: 2, Time elapsed: 204.917 sec <<< FAILURE! - in TestSuite testStateTooLargeExceptionThrownOnSetState on testStateTooLargeExceptionThrownOnSetState(org.apache.nifi.controller.state.providers.zookeeper.TestZooKeeperStateProvider)(org.apache.nifi.controller.state.providers.zookeeper.TestZooKeeperStateProvider) Time elapsed: 6.199 sec <<< FAILURE! java.io.IOException: Unable to remove state for component with ID '1---- with exception code CONNECTIONLOSS at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472) at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1500) at org.apache.zookeeper.ZKUtil.listSubTreeBFS(ZKUtil.java:114) at org.apache.zookeeper.ZKUtil.deleteRecursive(ZKUtil.java:49) at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.onComponentRemoved(ZooKeeperStateProvider.java:201) at org.apache.nifi.controller.state.providers.zookeeper.TestZooKeeperStateProvider.clear(TestZooKeeperStateProvider.java:102) > ZooKeeper State Provider unit tests failing > --- > > Key: NIFI-1872 > URL: https://issues.apache.org/jira/browse/NIFI-1872 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Reporter: Mark Payne > Fix For: 0.7.0 > > > There appears to be an issue with the unit tests for the ZooKeeper State > Provider. The client intermittently receives a ConnectionLossException when > attempting to fetch the state from ZK. We need to properly address the unit > test, but i propose that we should immediately mark the unit test as ignored > until we have a chance to properly fix the test, as it is currently > preventing master from building. We end up with the following stack trace: > Tests run: 171, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 153.028 sec > <<< FAILURE! - in TestSuite > testStateTooLargeExceptionThrownOnReplace on > testStateTooLargeExceptionThrownOnReplace(org.apache.nifi.controller.state.providers.zookeeper.TestZooKeeperStateProvider)(org.apache.nifi.controller.state.providers.zookeeper.TestZooKeeperStateProvider) > Time elapsed: 5.841 sec <<< FAILURE! > java.io.IOException: Failed to set cluster-wide state in ZooKeeper for > component with ID 1---- > at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1270) > at > org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:318) > at > org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:283) > at > org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:228) > at > org.apache.nifi.controller.state.providers.zookeeper.TestZooKeeperStateProvider.testStateTooLargeExceptionThrownOnReplace(TestZooKeeperStateProvider.java:163) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-1280) Create FilterCSVColumns Processor
[ https://issues.apache.org/jira/browse/NIFI-1280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15280556#comment-15280556 ] Joseph Witt commented on NIFI-1280: --- [~Toivo Adams] Really awesome idea and contribution! > Create FilterCSVColumns Processor > - > > Key: NIFI-1280 > URL: https://issues.apache.org/jira/browse/NIFI-1280 > Project: Apache NiFi > Issue Type: Task > Components: Extensions >Reporter: Mark Payne >Assignee: Toivo Adams > > We should have a Processor that allows users to easily filter out specific > columns from CSV data. For instance, a user would configure two different > properties: "Columns of Interest" (a comma-separated list of column indexes) > and "Filtering Strategy" (Keep Only These Columns, Remove Only These Columns). > We can do this today with ReplaceText, but it is far more difficult than it > would be with this Processor, as the user has to use Regular Expressions, > etc. with ReplaceText. > Eventually a Custom UI could even be built that allows a user to upload a > Sample CSV and choose which columns from there, similar to the way that Excel > works when importing CSV by dragging and selecting the desired columns? That > would certainly be a larger undertaking and would not need to be done for an > initial implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
