[jira] [Commented] (NIFI-13328) WindowsEventLogReader should parse RenderingInfo
[
https://issues.apache.org/jira/browse/NIFI-13328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17851869#comment-17851869
]
Sean Hunter commented on NIFI-13328:
I think I'd be alright with closing mine if we could get this JIRA knocked out.
Now I'm wondering how MiNiFi C++ handles this...
> WindowsEventLogReader should parse RenderingInfo
>
>
> Key: NIFI-13328
> URL: https://issues.apache.org/jira/browse/NIFI-13328
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
>Affects Versions: 1.24.0
> Environment: Docker
>Reporter: Stephen Jeffrey Hindmarch
>Priority: Major
>
> If windows events are extracted from the windows event collector they will
> include a "RenderingInfo" tag. However, this tag is not expected by the
> WindowsEventLogReader and will throw an error and pass the flow file into the
> failure relationship if the event contains the tag. This tag should be
> supported as it is a legitimate part of the Windows Event XML schema.
> See
> [https://learn.microsoft.com/en-us/windows/win32/wes/eventschema-renderingtype-complextype]
> and
> [https://learn.microsoft.com/en-us/windows/win32/wec/windows-event-collector]
> . In this particular use case, events are being collected from field
> technicians' laptops to perform a cybersecurity audit after they have
> plugging their laptops into customer networks.
> When these events are processed through a WindowsEventLogReader, the reader
> throws the following error.
> {noformat}
> ConvertRecord[id=7b99392f-2b54-139e-8791-349e930904cd] Failed to process
> FlowFile[filename=ffca2ea2-edd5-4ad1-8380-2bc4c8dae1ac]; will route to
> failure: org.apache.nifi.processor.exception.ProcessException: Could not
> parse incoming data
> - Caused by: org.apache.nifi.serialization.MalformedRecordException: Error
> reading records to determine the FlowFile's RecordSchema
> - Caused by: javax.xml.stream.XMLStreamException: Expecting tag but
> found unknown/invalid tag RenderingInfo{noformat}
> An example of the event record might be
> {noformat}
> https://schemas.microsoft.com/win/2004/08/events/event";>
>
> Guid="{555908d1-a6d7-4695-8e1e-26931d2012f4}" EventSourceName="Service
> Control Manager"/>
> 7036
> 0
> 4
> 0
> 0
> 0x8080
>
> 34153
>
>
> System
> WIN-O05CNUCF16M.hdf.local
>
>
>
> Smart Card Device Enumeration Service
> param2
>
> 5300630044006500760069006300650045006E0075006D002F003400
>
>
> This is a message
>
> {noformat}
> Removing the tag allows the event to be processed as normal.
> One possible workaround is to use a ReplaceText processor to remove the tag
> before reading, but this then involves either discarding the tag contents, or
> using an enrichment fork to find some other way of processing it. Another
> workaround is to use the XMLReader service, but this is a generic parser and
> has a its own problems.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-11784) XML to JSON conversion loses data (ConvertRecord processor - XMLReader and JSONRecordSetWriter)
[ https://issues.apache.org/jira/browse/NIFI-11784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17741172#comment-17741172 ] Sean Hunter commented on NIFI-11784: That workaround does appear to work. Thank you! I might also point out that it would probably be worth investigating if this is what's causing the WindowsEventLogReader to fail these logs as well. I started using the XMLReader because the WindowsEventLogReader was just dumping them into the failure queue. > XML to JSON conversion loses data (ConvertRecord processor - XMLReader and > JSONRecordSetWriter) > --- > > Key: NIFI-11784 > URL: https://issues.apache.org/jira/browse/NIFI-11784 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.22.0 > Environment: Tested on Windows with ConsumeWindowsEventLog, > particularly obvious (100% failure rate) on ForwardedEvents channel in a > WEF/WEC setup >Reporter: Sean Hunter >Priority: Major > Attachments: 735fcbc4-13ee-4b9e-bee0-91eb9084cb7e.xml, > image-2023-07-06-14-42-46-700.png, image-2023-07-06-14-45-15-939.png, > image-2023-07-06-15-04-04-992.png, image-2023-07-06-15-04-53-194.png > > > Screenshot of event going into ConvertRecord processor in XML with valid data > in SubjectUserName field (one example of data that's lost): > !image-2023-07-06-14-42-46-700.png! > Screenshot of the same flowfile once processed, showing that SubjectUserName > field has lost information: > !image-2023-07-06-14-45-15-939.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (NIFI-11784) XML to JSON conversion loses data (ConvertRecord processor - XMLReader and JSONRecordSetWriter)
[ https://issues.apache.org/jira/browse/NIFI-11784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17740784#comment-17740784 ] Sean Hunter edited comment on NIFI-11784 at 7/6/23 8:10 PM: [^735fcbc4-13ee-4b9e-bee0-91eb9084cb7e.xml] A download of the XML flowfile from Nifi, but sanitized. ETA: This is distinct from the one provided in screenshots, but will reproduce the issue nonetheless. was (Author: JIRAUSER299395): [^735fcbc4-13ee-4b9e-bee0-91eb9084cb7e.xml] A download of the XML flowfile from Nifi, but sanitized. > XML to JSON conversion loses data (ConvertRecord processor - XMLReader and > JSONRecordSetWriter) > --- > > Key: NIFI-11784 > URL: https://issues.apache.org/jira/browse/NIFI-11784 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.22.0 > Environment: Tested on Windows with ConsumeWindowsEventLog, > particularly obvious (100% failure rate) on ForwardedEvents channel in a > WEF/WEC setup >Reporter: Sean Hunter >Priority: Major > Attachments: 735fcbc4-13ee-4b9e-bee0-91eb9084cb7e.xml, > image-2023-07-06-14-42-46-700.png, image-2023-07-06-14-45-15-939.png, > image-2023-07-06-15-04-04-992.png, image-2023-07-06-15-04-53-194.png > > > Screenshot of event going into ConvertRecord processor in XML with valid data > in SubjectUserName field (one example of data that's lost): > !image-2023-07-06-14-42-46-700.png! > Screenshot of the same flowfile once processed, showing that SubjectUserName > field has lost information: > !image-2023-07-06-14-45-15-939.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-11784) XML to JSON conversion loses data (ConvertRecord processor - XMLReader and JSONRecordSetWriter)
[ https://issues.apache.org/jira/browse/NIFI-11784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17740784#comment-17740784 ] Sean Hunter commented on NIFI-11784: [^735fcbc4-13ee-4b9e-bee0-91eb9084cb7e.xml] A download of the XML flowfile from Nifi, but sanitized. > XML to JSON conversion loses data (ConvertRecord processor - XMLReader and > JSONRecordSetWriter) > --- > > Key: NIFI-11784 > URL: https://issues.apache.org/jira/browse/NIFI-11784 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.22.0 > Environment: Tested on Windows with ConsumeWindowsEventLog, > particularly obvious (100% failure rate) on ForwardedEvents channel in a > WEF/WEC setup >Reporter: Sean Hunter >Priority: Major > Attachments: 735fcbc4-13ee-4b9e-bee0-91eb9084cb7e.xml, > image-2023-07-06-14-42-46-700.png, image-2023-07-06-14-45-15-939.png, > image-2023-07-06-15-04-04-992.png, image-2023-07-06-15-04-53-194.png > > > Screenshot of event going into ConvertRecord processor in XML with valid data > in SubjectUserName field (one example of data that's lost): > !image-2023-07-06-14-42-46-700.png! > Screenshot of the same flowfile once processed, showing that SubjectUserName > field has lost information: > !image-2023-07-06-14-45-15-939.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11784) XML to JSON conversion loses data (ConvertRecord processor - XMLReader and JSONRecordSetWriter)
[ https://issues.apache.org/jira/browse/NIFI-11784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Hunter updated NIFI-11784: --- Attachment: 735fcbc4-13ee-4b9e-bee0-91eb9084cb7e.xml > XML to JSON conversion loses data (ConvertRecord processor - XMLReader and > JSONRecordSetWriter) > --- > > Key: NIFI-11784 > URL: https://issues.apache.org/jira/browse/NIFI-11784 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.22.0 > Environment: Tested on Windows with ConsumeWindowsEventLog, > particularly obvious (100% failure rate) on ForwardedEvents channel in a > WEF/WEC setup >Reporter: Sean Hunter >Priority: Major > Attachments: 735fcbc4-13ee-4b9e-bee0-91eb9084cb7e.xml, > image-2023-07-06-14-42-46-700.png, image-2023-07-06-14-45-15-939.png, > image-2023-07-06-15-04-04-992.png, image-2023-07-06-15-04-53-194.png > > > Screenshot of event going into ConvertRecord processor in XML with valid data > in SubjectUserName field (one example of data that's lost): > !image-2023-07-06-14-42-46-700.png! > Screenshot of the same flowfile once processed, showing that SubjectUserName > field has lost information: > !image-2023-07-06-14-45-15-939.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-11784) XML to JSON conversion loses data (ConvertRecord processor - XMLReader and JSONRecordSetWriter)
[ https://issues.apache.org/jira/browse/NIFI-11784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17740782#comment-17740782 ] Sean Hunter commented on NIFI-11784: XML Reader config: !image-2023-07-06-15-04-04-992.png! JSON writer config (IIRC, only change from default is writing JSON per-line instead of per-flowfile): !image-2023-07-06-15-04-53-194.png! Do feel free to reach out if you need any help or additional context! > XML to JSON conversion loses data (ConvertRecord processor - XMLReader and > JSONRecordSetWriter) > --- > > Key: NIFI-11784 > URL: https://issues.apache.org/jira/browse/NIFI-11784 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.22.0 > Environment: Tested on Windows with ConsumeWindowsEventLog, > particularly obvious (100% failure rate) on ForwardedEvents channel in a > WEF/WEC setup >Reporter: Sean Hunter >Priority: Major > Attachments: image-2023-07-06-14-42-46-700.png, > image-2023-07-06-14-45-15-939.png, image-2023-07-06-15-04-04-992.png, > image-2023-07-06-15-04-53-194.png > > > Screenshot of event going into ConvertRecord processor in XML with valid data > in SubjectUserName field (one example of data that's lost): > !image-2023-07-06-14-42-46-700.png! > Screenshot of the same flowfile once processed, showing that SubjectUserName > field has lost information: > !image-2023-07-06-14-45-15-939.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11784) XML to JSON conversion loses data (ConvertRecord processor - XMLReader and JSONRecordSetWriter)
[ https://issues.apache.org/jira/browse/NIFI-11784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Hunter updated NIFI-11784: --- Attachment: image-2023-07-06-15-04-53-194.png > XML to JSON conversion loses data (ConvertRecord processor - XMLReader and > JSONRecordSetWriter) > --- > > Key: NIFI-11784 > URL: https://issues.apache.org/jira/browse/NIFI-11784 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.22.0 > Environment: Tested on Windows with ConsumeWindowsEventLog, > particularly obvious (100% failure rate) on ForwardedEvents channel in a > WEF/WEC setup >Reporter: Sean Hunter >Priority: Major > Attachments: image-2023-07-06-14-42-46-700.png, > image-2023-07-06-14-45-15-939.png, image-2023-07-06-15-04-04-992.png, > image-2023-07-06-15-04-53-194.png > > > Screenshot of event going into ConvertRecord processor in XML with valid data > in SubjectUserName field (one example of data that's lost): > !image-2023-07-06-14-42-46-700.png! > Screenshot of the same flowfile once processed, showing that SubjectUserName > field has lost information: > !image-2023-07-06-14-45-15-939.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-11784) XML to JSON conversion loses data (ConvertRecord processor - XMLReader and JSONRecordSetWriter)
[ https://issues.apache.org/jira/browse/NIFI-11784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Hunter updated NIFI-11784: --- Attachment: image-2023-07-06-15-04-04-992.png > XML to JSON conversion loses data (ConvertRecord processor - XMLReader and > JSONRecordSetWriter) > --- > > Key: NIFI-11784 > URL: https://issues.apache.org/jira/browse/NIFI-11784 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.22.0 > Environment: Tested on Windows with ConsumeWindowsEventLog, > particularly obvious (100% failure rate) on ForwardedEvents channel in a > WEF/WEC setup >Reporter: Sean Hunter >Priority: Major > Attachments: image-2023-07-06-14-42-46-700.png, > image-2023-07-06-14-45-15-939.png, image-2023-07-06-15-04-04-992.png, > image-2023-07-06-15-04-53-194.png > > > Screenshot of event going into ConvertRecord processor in XML with valid data > in SubjectUserName field (one example of data that's lost): > !image-2023-07-06-14-42-46-700.png! > Screenshot of the same flowfile once processed, showing that SubjectUserName > field has lost information: > !image-2023-07-06-14-45-15-939.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-11784) XML to JSON conversion loses data (ConvertRecord processor - XMLReader and JSONRecordSetWriter)
Sean Hunter created NIFI-11784: -- Summary: XML to JSON conversion loses data (ConvertRecord processor - XMLReader and JSONRecordSetWriter) Key: NIFI-11784 URL: https://issues.apache.org/jira/browse/NIFI-11784 Project: Apache NiFi Issue Type: Bug Components: Extensions Affects Versions: 1.22.0 Environment: Tested on Windows with ConsumeWindowsEventLog, particularly obvious (100% failure rate) on ForwardedEvents channel in a WEF/WEC setup Reporter: Sean Hunter Attachments: image-2023-07-06-14-42-46-700.png, image-2023-07-06-14-45-15-939.png Screenshot of event going into ConvertRecord processor in XML with valid data in SubjectUserName field (one example of data that's lost): !image-2023-07-06-14-42-46-700.png! Screenshot of the same flowfile once processed, showing that SubjectUserName field has lost information: !image-2023-07-06-14-45-15-939.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-11434) Error with UnpackContent
[
https://issues.apache.org/jira/browse/NIFI-11434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17740762#comment-17740762
]
Sean Hunter commented on NIFI-11434:
It looks like I'm having a similar issue still in 1.22.0, for what it's worth.
> Error with UnpackContent
>
>
> Key: NIFI-11434
> URL: https://issues.apache.org/jira/browse/NIFI-11434
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.18.0
> Environment: OS Ubuntu 20.04.5, running in a K8S cluster but as
> standalone node, limited to 8gb, free on cpu usage.
>Reporter: Rafael Fracasso Gomes
>Priority: Blocker
> Attachments: 58174fd0-750e-4e45-89bf-c43d9ab01b98.zip
>
>
> I posted in NiFi discussion group about a error that occurred in
> UnpackContent processor, I will paste it here:
> {panel:title=Error with UnpackContent}
> I'm trying to unzip some files with NiFi and getting this errors:
> {code:java}
> UnpackContent[id=5c38660b-0187-1000-52a2-0b71f8e868c0] Unable to unpack
> FlowFile[filename=58174fd0-750e-4e45-89bf-c43d9ab01b98]; routing to failure:
> org.apache.nifi.processor.exception.ProcessException: IOException thrown from
> UnpackContent[id=5c38660b-0187-1000-52a2-0b71f8e868c0]:
> org.apache.commons.compress.archivers.zip.UnsupportedZipFeatureException:
> Unsupported feature data descriptor used in entry
> files/20230228_131431_84_24c8_3B_udm2_clip.tif - Caused by:
> org.apache.commons.compress.archivers.zip.UnsupportedZipFeatureException:
> Unsupported feature data descriptor used in entry
> files/20230228_131431_84_24c8_3B_udm2_clip.tif{code}
> I found some workaround in the link below but it doesn't suits me because i
> need to keep my attributes in the flowfiles:
> [https://community.cloudera.com/t5/Support-Questions/Unzip-file-containing-log-gz-files-with-nifi/m-p/304042]
> And found some help in stackoverflow about this problem but it doesn't fit on
> NiFi processor: (
> [https://stackoverflow.com/questions/15738312/how-to-fix-org-apache-commons-compress-archivers-zip-unsupportedzipfeatureexcept]
> )
> I attached a sample file that I downloaded from nifi content.{panel}
>
> David Handermann replyed me:
> {panel}
> As indicated by the error message, the Zip file in question includes a
> specialized data descriptor feature that is not supported in the default
> configuration of the UnpackContent Processor.
>
> The UnpackContent Processor relies on Apache Commons Compress for extraction,
> which has optional support for handling this feature. The feature is disabled
> by default, which is the reason for the error.
>
> Apache Commons Compress issue COMPRESS-555 [1] provides some additional
> background on why this feature is disabled by default, but it sounds like
> something that could be evaluated for adjustment in Apache NiFi.
> [1] [https://issues.apache.org/jira/projects/COMPRESS/issues/COMPRESS-555]
> {panel}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
