I won't have access to an IDE for the most of today, or an active azure account so I won't be able to debug myself in short order, but I'm suspecting the same thing joe alluded to at this point - this being an issue with the client API, possibly complicated by with the way NiFi allows access to the underlying bytes to the file.
On Tue, Sep 6, 2016 at 6:21 PM, Kumiko Yada <kumiko.y...@ds-iq.com> wrote: > Lorenzo was not clear how we tested. I wrote the sample java program > using the Azure SDK, then uploaded the 50 MB file and it’s worked without > error. Nifi custom processor used the same SDK code; however, it’s > failing when the custom processor is tried to uploaded 50 MB file. > > > > Thanks > > Kumiko > > > > *From:* Tony Kurc [mailto:trk...@gmail.com] > *Sent:* Tuesday, September 6, 2016 3:02 PM > *To:* Kumiko Yada <kumiko.y...@ds-iq.com> > *Cc:* users@nifi.apache.org; Joe Witt <joe.w...@gmail.com>; #Operations > Automation and Tools <#opsto...@ds-iq.com>; Kevin Verhoeven < > kevin.verhoe...@ds-iq.com> > > *Subject:* RE: Issue writing file (~50mb) to azure data lake with Nifi > > > > I apologize if I'm missing something, as I'm trying to read the code on my > phone, but it looks like the test script is using a different api call to > perform the upload - did you already test using the same call in your > script? > > > > On Sep 6, 2016 5:48 PM, "Kumiko Yada" <kumiko.y...@ds-iq.com> wrote: > > Here is the code: https://github.com/kyada1/ADL_UploadFile. > > > > I removed the following values for the security reason. > > > > *final static *String *ADLS_ACCOUNT_NAME *= *""*; > *final static *String *RESOURCE_GROUP_NAME *= *""*; > *final static *String *LOCATION *= *""*; > *final static *String *TENANT_ID *= *""*; > *final static *String *SUBSCRIPTION_ID *= *""*; > *final static *String *CLIENT_ID *= *""*; > *final static *String *CLIENT_SECRET *= *""*; */* > > > > Thanks > > Kumiko > > > > *From:* Tony Kurc [mailto:trk...@gmail.com] > *Sent:* Tuesday, September 6, 2016 2:41 PM > *To:* Kumiko Yada <kumiko.y...@ds-iq.com> > *Cc:* Joe Witt <joe.w...@gmail.com>; users@nifi.apache.org; #Operations > Automation and Tools <#opsto...@ds-iq.com>; Kevin Verhoeven < > kevin.verhoe...@ds-iq.com> > *Subject:* RE: Issue writing file (~50mb) to azure data lake with Nifi > > > > I was referring to this: "Then we wrote a java sdk script to upload this > same file without Nifi into the data lake and it worked successfully." > > Is that code somewhere? > > > > On Sep 6, 2016 5:38 PM, "Kumiko Yada" <kumiko.y...@ds-iq.com> wrote: > > I didn’t add any test code. This custom controller and processor is > working for a small size file. > > > > Thanks > > Kumiko > > > > *From:* Tony Kurc [mailto:trk...@gmail.com] > *Sent:* Tuesday, September 6, 2016 2:32 PM > *To:* users@nifi.apache.org > *Cc:* Joe Witt <joe.w...@gmail.com>; #Operations Automation and Tools <# > opsto...@ds-iq.com>; Kevin Verhoeven <kevin.verhoe...@ds-iq.com> > *Subject:* RE: Issue writing file (~50mb) to azure data lake with Nifi > > > > I didn't see the test script that worked in the source code - did I miss > it, or is it not in the tree? > > > > On Sep 6, 2016 3:17 PM, "Kumiko Yada" <kumiko.y...@ds-iq.com> wrote: > > Joe, > > > > Here is the log (there was no callstack related to this error) and code, > https://github.com/kyada1/dl_sdkworkaround/tree/master/ > nifi-azure-dlstore-bundle. > > > > 2016-09-06 12:06:50,508 INFO [NiFi Web Server-19] > c.s.j.s.i.application.WebApplicationImpl > Initiating Jersey application, version 'Jersey: 1.19 02/11/2015 03:25 AM' > > 2016-09-06 12:07:00,991 INFO [StandardProcessScheduler Thread-1] > o.a.n.c.s.TimerDrivenSchedulingAgent > Scheduled PutFileAzureDLStore[id=00dd95dc-0157-1000-8ab1-2de88c159b55] to > run with 1 threads > > 2016-09-06 12:07:01,545 INFO [Flow Service Tasks Thread-1] > o.a.nifi.controller.StandardFlowService Saved flow controller > org.apache.nifi.controller.FlowController@28f414a1 // Another save > pending = false > > 2016-09-06 12:07:01,904 INFO [pool-27-thread-1] > c.m.aad.adal4j.AuthenticationAuthority > [Correlation ID: 564fb5ec-643b-43e6-ab68-59f259a4843a] Instance discovery > was successful > > 2016-09-06 12:08:05,988 ERROR [Timer-Driven Process Thread-1] > n.a.d.processors.PutFileAzureDLStore PutFileAzureDLStore[id= > 00dd95dc-0157-1000-8ab1-2de88c159b55] File was not created: > /kumiko/test/20160906120701022.txt > com.microsoft.azure.management.datalake.store.models.AdlsErrorException: > Invalid status code 404 > > 2016-09-06 12:08:12,541 INFO [Provenance Maintenance Thread-1] > o.a.n.p.PersistentProvenanceRepository > Created new Provenance Event Writers for events starting with ID 618 > > 2016-09-06 12:08:12,838 INFO [Provenance Repository Rollover Thread-1] > o.a.n.p.PersistentProvenanceRepository Successfully merged 16 journal > files (1 records) into single Provenance Log File > .\provenance_repository\617.prov in 305 milliseconds > > 2016-09-06 12:08:12,838 INFO [Provenance Repository Rollover Thread-1] > o.a.n.p.PersistentProvenanceRepository Successfully Rolled over > Provenance Event file containing 1 records > > 2016-09-06 12:08:21,148 INFO [NiFi Web Server-28] > o.a.n.controller.StandardProcessorNode > Stopping processor: class nifi.azure.dlstore.processors. > PutFileAzureDLStore > > > > Thanks > > Kumiko > > > > *From:* Joe Witt [mailto:joe.w...@gmail.com] > *Sent:* Friday, September 2, 2016 2:56 PM > *To:* users@nifi.apache.org > *Cc:* #Operations Automation and Tools <#opsto...@ds-iq.com>; Kevin > Verhoeven <kevin.verhoe...@ds-iq.com> > *Subject:* Re: Issue writing file (~50mb) to azure data lake with Nifi > > > > Lorenzo > > Without seeing the code and logs it would be very difficult to help. > nifi has no trouble by design writing large files (GBs) to many things > including hdfs so the issue is probably in how this client library > interacts with the data stream. > > > > On Sep 2, 2016 4:19 PM, "Lorenzo Peder" <lorenzo.pe...@ds-iq.com> wrote: > > Hi All, > > > > We’ve run into an issue uploading a larger file (~50Mb) into an Azure Data > Lake using a custom processor in nifi 0.7-1.0. This custom processor has > worked consistently for smaller files, but once encountered with this > larger file, it spits http error 404 (file not found). Eventually a minor > portion of the file wrote to the data lake. > > We used fiddler to capture network traffic between Nifi and the Azure data > lake while the processor was running and captured http error 204 (no > contents). > > Then we wrote a java sdk script to upload this same file without Nifi into > the data lake and it worked successfully. > > These findings lead us to believe that this issue is occurring within > Nifi, if someone could please point us in the right direction in resolving > this issue it would be greatly appreciated. > > > > Thank you, > > > > Lorenzo Peder > > Operations Analyst, Campaign Operations & Services > > > > 425.974.1363 : Office > > 425.260.5027 : Mobile > > www.ds-iq.com > > > > Dynamic Shopper Intelligence > > > > This e-mail may contain confidential or privileged information. > > If you are not the intended recipient, please notify the sender > immediately and then delete this message. > > > > > >
