Re: Possible Regression in PutAzureBlobStorage 1.12.0
Hello,
I noticed I maybe was not configuring the processor correctly but nothing
went wrong until now.
I had the following configured:
*Container Name*: "base_container/A/B"
*Blob*: ${filename}
I changed it to this and it works:
*Container Name*: "base_container"
*Blob*: "/A/B/"${filename}
Thanks,
Eric
On Wed, Sep 16, 2020 at 1:12 PM Paul Kelly wrote:
> Hi Eric,
>
> We also noticed an issue with PutAzureBlobStorage when upgrading to NiFi
> 1.12.0, and I believe it has to do with changes made for
> https://issues.apache.org/jira/browse/NIFI-6913, where the default
> behavior changed to check to see if a container exists before writing a
> file, and creating the container if it doesn't exist. The SAS tokens we
> are using within our flows do not have permission to list containers, so
> this check is failing, and therefore the overall Put operation is failing.
> We are seeing a different error, but I suspect it is related since this in
> your stack trace:
>
> at
> com.microsoft.azure.storage.blob.CloudBlobContainer.createIfNotExists(CloudBlobContainer.java:354)
> at
> com.microsoft.azure.storage.blob.CloudBlobContainer.createIfNotExists(CloudBlobContainer.java:301)
>
> which indicates that yours is also trying to create the container and
> failing due to this new check. Do you have permission to list and create
> containers?
>
> I reported this issue in https://issues.apache.org/jira/browse/NIFI-7794.
> I plan to submit a fix for it by adding a property to the processor to
> determine if a container should be created if it doesn't exist, but if
> someone else beats me to it, that'd be great.
>
> Paul
>
> On Wed, Sep 16, 2020 at 7:10 PM Joey Frazee
> wrote:
>
>> Eric, can you share any details about your config (e.g., what do you have
>> in the Blob property)? I tried the following scenarios in an upgrade to
>> 1.12.0 and main and they seem to work.
>>
>> Pre-existing object: A/B/test.json
>>
>> New object: A/B/${filename}.json
>>
>> New object with new pseudo-dirs:
>> ${random():mod(10):plus(1)}/${random():mod(10):plus(1)}/${filename}.json
>>
>> -joey
>>
>> On Sep 16, 2020, 11:02 AM -0700, Eric Secules ,
>> wrote:
>>
>> Hello everyone,
>>
>>
>> I was able to see why this is an issue. It's an issue that the blob is
>> stored several layers deep at "my-container/A/B/my_test_blob.json"
>>
>>
>> -Eric
>>
>>
>> On Wed, Sep 16, 2020 at 10:49 AM Eric Secules wrote:
>>
>>> Hello everyone,
>>>
>>> I tried upgrading to 1.12.0 and right away noticed that
>>> PutAzureBlobStorage is failing due to the following error. I don't think
>>> it's an issue with access because I can use a ListAzureBlobStorage on the
>>> same container and I haven't changed the permissions of the container. And
>>> I didn't change any parameters during the upgrade and it was working on
>>> 1.11.4.
>>>
>>> I am writing the blob to a container path that already exists:
>>> "my-container/A/B/my_test_blob.json:"
>>>
>>> 2020-09-16 00:59:17,283 ERROR [Timer-Driven Process Thread-6]
>>> o.a.n.p.a.storage.PutAzureBlobStorage
>>> PutAzureBlobStorage[id=15404d54-bc14-350c-7847-521b765dd57f] Failed to put
>>> Azure blob my_test_blob.json: com.microsoft.azure.storage.StorageException:
>>> The requested URI does not represent any resource on the server.
>>> com.microsoft.azure.storage.StorageException: The requested URI does not
>>> represent any resource on the server.
>>> at
>>> com.microsoft.azure.storage.StorageException.translateException(StorageException.java:87)
>>> at
>>> com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:220)
>>> at
>>> com.microsoft.azure.storage.blob.CloudBlobContainer.exists(CloudBlobContainer.java:744)
>>> at
>>> com.microsoft.azure.storage.blob.CloudBlobContainer.createIfNotExists(CloudBlobContainer.java:354)
>>> at
>>> com.microsoft.azure.storage.blob.CloudBlobContainer.createIfNotExists(CloudBlobContainer.java:301)
>>> at
>>> org.apache.nifi.processors.azure.storage.PutAzureBlobStorage.onTrigger(PutAzureBlobStorage.java:100)
>>> at
>>> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>>> at
>>> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1174)
>>> at
>>> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
>>> at
>>> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
>>> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
>>> at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>> at
>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFuture
Re: Possible Regression in PutAzureBlobStorage 1.12.0
Hi Eric,
We also noticed an issue with PutAzureBlobStorage when upgrading to NiFi
1.12.0, and I believe it has to do with changes made for
https://issues.apache.org/jira/browse/NIFI-6913, where the default behavior
changed to check to see if a container exists before writing a file, and
creating the container if it doesn't exist. The SAS tokens we are using
within our flows do not have permission to list containers, so this check
is failing, and therefore the overall Put operation is failing. We are
seeing a different error, but I suspect it is related since this in your
stack trace:
at
com.microsoft.azure.storage.blob.CloudBlobContainer.createIfNotExists(CloudBlobContainer.java:354)
at
com.microsoft.azure.storage.blob.CloudBlobContainer.createIfNotExists(CloudBlobContainer.java:301)
which indicates that yours is also trying to create the container and
failing due to this new check. Do you have permission to list and create
containers?
I reported this issue in https://issues.apache.org/jira/browse/NIFI-7794.
I plan to submit a fix for it by adding a property to the processor to
determine if a container should be created if it doesn't exist, but if
someone else beats me to it, that'd be great.
Paul
On Wed, Sep 16, 2020 at 7:10 PM Joey Frazee wrote:
> Eric, can you share any details about your config (e.g., what do you have
> in the Blob property)? I tried the following scenarios in an upgrade to
> 1.12.0 and main and they seem to work.
>
> Pre-existing object: A/B/test.json
>
> New object: A/B/${filename}.json
>
> New object with new pseudo-dirs:
> ${random():mod(10):plus(1)}/${random():mod(10):plus(1)}/${filename}.json
>
> -joey
>
> On Sep 16, 2020, 11:02 AM -0700, Eric Secules , wrote:
>
> Hello everyone,
>
>
> I was able to see why this is an issue. It's an issue that the blob is
> stored several layers deep at "my-container/A/B/my_test_blob.json"
>
>
> -Eric
>
>
> On Wed, Sep 16, 2020 at 10:49 AM Eric Secules wrote:
>
>> Hello everyone,
>>
>> I tried upgrading to 1.12.0 and right away noticed that
>> PutAzureBlobStorage is failing due to the following error. I don't think
>> it's an issue with access because I can use a ListAzureBlobStorage on the
>> same container and I haven't changed the permissions of the container. And
>> I didn't change any parameters during the upgrade and it was working on
>> 1.11.4.
>>
>> I am writing the blob to a container path that already exists:
>> "my-container/A/B/my_test_blob.json:"
>>
>> 2020-09-16 00:59:17,283 ERROR [Timer-Driven Process Thread-6]
>> o.a.n.p.a.storage.PutAzureBlobStorage
>> PutAzureBlobStorage[id=15404d54-bc14-350c-7847-521b765dd57f] Failed to put
>> Azure blob my_test_blob.json: com.microsoft.azure.storage.StorageException:
>> The requested URI does not represent any resource on the server.
>> com.microsoft.azure.storage.StorageException: The requested URI does not
>> represent any resource on the server.
>> at
>> com.microsoft.azure.storage.StorageException.translateException(StorageException.java:87)
>> at
>> com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:220)
>> at
>> com.microsoft.azure.storage.blob.CloudBlobContainer.exists(CloudBlobContainer.java:744)
>> at
>> com.microsoft.azure.storage.blob.CloudBlobContainer.createIfNotExists(CloudBlobContainer.java:354)
>> at
>> com.microsoft.azure.storage.blob.CloudBlobContainer.createIfNotExists(CloudBlobContainer.java:301)
>> at
>> org.apache.nifi.processors.azure.storage.PutAzureBlobStorage.onTrigger(PutAzureBlobStorage.java:100)
>> at
>> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>> at
>> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1174)
>> at
>> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
>> at
>> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
>> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> at
>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>> Caused by: java.lang.NullPointerException: null
>> at
>> com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:202)
>> ... 16 common frames omitted
Re: Possible Regression in PutAzureBlobStorage 1.12.0
Eric, can you share any details about your config (e.g., what do you have in
the Blob property)? I tried the following scenarios in an upgrade to 1.12.0 and
main and they seem to work.
Pre-existing object: A/B/test.json
New object: A/B/${filename}.json
New object with new pseudo-dirs:
${random():mod(10):plus(1)}/${random():mod(10):plus(1)}/${filename}.json
-joey
On Sep 16, 2020, 11:02 AM -0700, Eric Secules , wrote:
> Hello everyone,
>
> I was able to see why this is an issue. It's an issue that the blob is stored
> several layers deep at "my-container/A/B/my_test_blob.json"
>
> -Eric
>
On Wed, Sep 16, 2020 at 10:49 AM Eric Secules wrote:
> Hello everyone,
>
> I tried upgrading to 1.12.0 and right away noticed that PutAzureBlobStorage
> is failing due to the following error. I don't think it's an issue with
> access because I can use a ListAzureBlobStorage on the same container and I
> haven't changed the permissions of the container. And I didn't change any
> parameters during the upgrade and it was working on 1.11.4.
>
> I am writing the blob to a container path that already exists:
> "my-container/A/B/my_test_blob.json:"
>
> 2020-09-16 00:59:17,283 ERROR [Timer-Driven Process Thread-6]
> o.a.n.p.a.storage.PutAzureBlobStorage
> PutAzureBlobStorage[id=15404d54-bc14-350c-7847-521b765dd57f] Failed to put
> Azure blob my_test_blob.json: com.microsoft.azure.storage.StorageException:
> The requested URI does not represent any resource on the server.
> com.microsoft.azure.storage.StorageException: The requested URI does not
> represent any resource on the server.
> at
> com.microsoft.azure.storage.StorageException.translateException(StorageException.java:87)
> at
> com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:220)
> at
> com.microsoft.azure.storage.blob.CloudBlobContainer.exists(CloudBlobContainer.java:744)
> at
> com.microsoft.azure.storage.blob.CloudBlobContainer.createIfNotExists(CloudBlobContainer.java:354)
> at
> com.microsoft.azure.storage.blob.CloudBlobContainer.createIfNotExists(CloudBlobContainer.java:301)
> at
> org.apache.nifi.processors.azure.storage.PutAzureBlobStorage.onTrigger(PutAzureBlobStorage.java:100)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1174)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NullPointerException: null
> at
> com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:202)
> ... 16 common frames omitted
>
> Thanks,
> Eric
Re: Possible Regression in PutAzureBlobStorage 1.12.0
Hello everyone, I was able to see why this is an issue. It's an issue that the blob is stored several layers deep at "my-container/A/B/my_test_blob.json" -Eric On Wed, Sep 16, 2020 at 10:49 AM Eric Secules wrote: > Hello everyone, > > I tried upgrading to 1.12.0 and right away noticed that > PutAzureBlobStorage is failing due to the following error. I don't think > it's an issue with access because I can use a ListAzureBlobStorage on the > same container and I haven't changed the permissions of the container. And > I didn't change any parameters during the upgrade and it was working on > 1.11.4. > > I am writing the blob to a container path that already exists: > "my-container/A/B/my_test_blob.json:" > > 2020-09-16 00:59:17,283 ERROR [Timer-Driven Process Thread-6] > o.a.n.p.a.storage.PutAzureBlobStorage > PutAzureBlobStorage[id=15404d54-bc14-350c-7847-521b765dd57f] Failed to put > Azure blob my_test_blob.json: com.microsoft.azure.storage.StorageException: > The requested URI does not represent any resource on the server. > com.microsoft.azure.storage.StorageException: The requested URI does not > represent any resource on the server. > at > com.microsoft.azure.storage.StorageException.translateException(StorageException.java:87) > at > com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:220) > at > com.microsoft.azure.storage.blob.CloudBlobContainer.exists(CloudBlobContainer.java:744) > at > com.microsoft.azure.storage.blob.CloudBlobContainer.createIfNotExists(CloudBlobContainer.java:354) > at > com.microsoft.azure.storage.blob.CloudBlobContainer.createIfNotExists(CloudBlobContainer.java:301) > at > org.apache.nifi.processors.azure.storage.PutAzureBlobStorage.onTrigger(PutAzureBlobStorage.java:100) > at > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1174) > at > org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213) > at > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) > at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.NullPointerException: null > at > com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:202) > ... 16 common frames omitted > > Thanks, > Eric >
Possible Regression in PutAzureBlobStorage 1.12.0
Hello everyone, I tried upgrading to 1.12.0 and right away noticed that PutAzureBlobStorage is failing due to the following error. I don't think it's an issue with access because I can use a ListAzureBlobStorage on the same container and I haven't changed the permissions of the container. And I didn't change any parameters during the upgrade and it was working on 1.11.4. I am writing the blob to a container path that already exists: "my-container/A/B/my_test_blob.json:" 2020-09-16 00:59:17,283 ERROR [Timer-Driven Process Thread-6] o.a.n.p.a.storage.PutAzureBlobStorage PutAzureBlobStorage[id=15404d54-bc14-350c-7847-521b765dd57f] Failed to put Azure blob my_test_blob.json: com.microsoft.azure.storage.StorageException: The requested URI does not represent any resource on the server. com.microsoft.azure.storage.StorageException: The requested URI does not represent any resource on the server. at com.microsoft.azure.storage.StorageException.translateException(StorageException.java:87) at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:220) at com.microsoft.azure.storage.blob.CloudBlobContainer.exists(CloudBlobContainer.java:744) at com.microsoft.azure.storage.blob.CloudBlobContainer.createIfNotExists(CloudBlobContainer.java:354) at com.microsoft.azure.storage.blob.CloudBlobContainer.createIfNotExists(CloudBlobContainer.java:301) at org.apache.nifi.processors.azure.storage.PutAzureBlobStorage.onTrigger(PutAzureBlobStorage.java:100) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1174) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.NullPointerException: null at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:202) ... 16 common frames omitted Thanks, Eric
