[
https://issues.apache.org/jira/browse/NIFI-4944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16393821#comment-16393821
]
ASF GitHub Bot commented on NIFI-4944:
--------------------------------------
Github user mcgilman commented on the issue:
https://github.com/apache/nifi/pull/2519
@moonkev I think there a couple different scenario's here. This is
admittedly pretty complicated, but my understanding is as follows:
- In a case like PutHDFS, snappy compression is supported through the
native hadoop library. In this case, the native snappy library is not loaded.
Issues seen here are do to multiple classloaders attempting to load "hadoop"
native library.
- In a case like CompressContent, snappy support is supplied by the native
snappy library loaded indirectly through snappy-java. In more recent version,
they effectively bypass the native loading issues by copying the native library
to a temp directory and loading it with a UUID appended to the name. This
allows the native library to multiple times (since the name is different each
time).
- In a case like PutHiveStreaming, snappy support is also supplied through
snappy-java. The difference is that its an old version of snappy-java that does
not perform the UUID trick. Force upgrading in this case is tricky since it is
a transitive dependency (PutHiveStreaming -> Hive -> avro -> snappy). The
legacy snappy-java addresses the native library issue by directly injecting the
class that loads the native library into the root classloader. The ultimate
issue here was that multiple processors were attempting to do this loading at
the same time (due to the loading happening in onTrigger).
@mattyb149 Please chime in here if any of my understanding isn't correct.
Hope this helps
> PutHiveStreaming multiple instances with Snappy fail intermittently
> -------------------------------------------------------------------
>
> Key: NIFI-4944
> URL: https://issues.apache.org/jira/browse/NIFI-4944
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Reporter: Matt Burgess
> Assignee: Matt Burgess
> Priority: Major
>
> When data coming into PutHiveStreaming is compressed with Snappy, then
> multiple instances of PutHiveStreaming in a flow can cause a failure, the log
> often shows the following:
> {{org.apache.nifi.processors.hive.PutHiveStreaming$$Lambda$510/1467586448@68a5884d
> failed to process due to org.xerial.snappy.SnappyError:
> [FAILED_TO_LOAD_NATIVE_LIBRARY] null; rolling back session: {}}}
> This is due to a race condition in Snappy 1.0.5 (the version used by the Hive
> NAR) where two classloaders can try to define the native loader class, thus
> the second one would fail, giving the error above.
> The proposed solution is to guarantee that Snappy is loaded before this
> situation is encountered (i.e. before the InstanceClassLoaders are created).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
