Hi Eric,
We also noticed an issue with PutAzureBlobStorage when upgrading to NiFi
1.12.0, and I believe it has to do with changes made for
https://issues.apache.org/jira/browse/NIFI-6913, where the default behavior
changed to check to see if a container exists before writing a file, and
creating the container if it doesn't exist. The SAS tokens we are using
within our flows do not have permission to list containers, so this check
is failing, and therefore the overall Put operation is failing. We are
seeing a different error, but I suspect it is related since this in your
stack trace:
at
com.microsoft.azure.storage.blob.CloudBlobContainer.createIfNotExists(CloudBlobContainer.java:354)
at
com.microsoft.azure.storage.blob.CloudBlobContainer.createIfNotExists(CloudBlobContainer.java:301)
which indicates that yours is also trying to create the container and
failing due to this new check. Do you have permission to list and create
containers?
I reported this issue in https://issues.apache.org/jira/browse/NIFI-7794.
I plan to submit a fix for it by adding a property to the processor to
determine if a container should be created if it doesn't exist, but if
someone else beats me to it, that'd be great.
Paul
On Wed, Sep 16, 2020 at 7:10 PM Joey Frazee <joey.fra...@icloud.com> wrote:
> Eric, can you share any details about your config (e.g., what do you have
> in the Blob property)? I tried the following scenarios in an upgrade to
> 1.12.0 and main and they seem to work.
>
> Pre-existing object: A/B/test.json
>
> New object: A/B/${filename}.json
>
> New object with new pseudo-dirs:
> ${random():mod(10):plus(1)}/${random():mod(10):plus(1)}/${filename}.json
>
> -joey
>
> On Sep 16, 2020, 11:02 AM -0700, Eric Secules <esecu...@gmail.com>, wrote:
>
> Hello everyone,
>
>
> I was able to see why this is an issue. It's an issue that the blob is
> stored several layers deep at "my-container/A/B/my_test_blob.json"
>
>
> -Eric
>
>
> On Wed, Sep 16, 2020 at 10:49 AM Eric Secules <esecu...@gmail.com> wrote:
>
>> Hello everyone,
>>
>> I tried upgrading to 1.12.0 and right away noticed that
>> PutAzureBlobStorage is failing due to the following error. I don't think
>> it's an issue with access because I can use a ListAzureBlobStorage on the
>> same container and I haven't changed the permissions of the container. And
>> I didn't change any parameters during the upgrade and it was working on
>> 1.11.4.
>>
>> I am writing the blob to a container path that already exists:
>> "my-container/A/B/my_test_blob.json:"
>>
>> 2020-09-16 00:59:17,283 ERROR [Timer-Driven Process Thread-6]
>> o.a.n.p.a.storage.PutAzureBlobStorage
>> PutAzureBlobStorage[id=15404d54-bc14-350c-7847-521b765dd57f] Failed to put
>> Azure blob my_test_blob.json: com.microsoft.azure.storage.StorageException:
>> The requested URI does not represent any resource on the server.
>> com.microsoft.azure.storage.StorageException: The requested URI does not
>> represent any resource on the server.
>> at
>> com.microsoft.azure.storage.StorageException.translateException(StorageException.java:87)
>> at
>> com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:220)
>> at
>> com.microsoft.azure.storage.blob.CloudBlobContainer.exists(CloudBlobContainer.java:744)
>> at
>> com.microsoft.azure.storage.blob.CloudBlobContainer.createIfNotExists(CloudBlobContainer.java:354)
>> at
>> com.microsoft.azure.storage.blob.CloudBlobContainer.createIfNotExists(CloudBlobContainer.java:301)
>> at
>> org.apache.nifi.processors.azure.storage.PutAzureBlobStorage.onTrigger(PutAzureBlobStorage.java:100)
>> at
>> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>> at
>> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1174)
>> at
>> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
>> at
>> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
>> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> at
>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>> Caused by: java.lang.NullPointerException: null
>> at
>> com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:202)
>> ... 16 common frames omitted
>>
>> Thanks,
>> Eric
>>
>
