Hi list, I'm a NiFi newbie and I'm trying to figure out the best way to use it as a batch ingestion system for satellite imagery as raster files.The files are pushed on the FS by an external system and then they must be processed and published through WMS protocols.I tried to draft the flow using the NiFi processors available, summarizing I used: - GetFile and PutFile processors to watch for incoming files to process - UpdateAttributes to manage the location of the incoming files and the intermediate processing products- ExecuteStreamProcess to call the gdalwarp and gdaladdo command line utilities to do the geospatial processing needed (http://www.gdal.org/) The issue I found with my use case is the fact that what represent flowfiles are big raster files (1 to 6GB) and I would like to minimize the number of copies of that resource. I used the GetFile processor to watch a FileSystem folder for incoming files.Once a new file is found, it is imported in the NiFi internal content repository so I can't reference it with its absolute FS path anymore since it is transformed into a flowfile (Did I misunderstand something here?)So I have to copy it again somewhere else on the FS to process it, the geospatial processing utilities I have to use require the abs path of the files to process. There could be maybe a solution which better address the design of this flow, for example I can watch not for the real file but for a txt file which describe its FS abs path. Anyway I was wondering if it is possible to configure the GetFile processors to use as flowfile payload only the absolute path of the file found, but I guess that in that case I have to write my own GetFile processor. (the same could be valid also for other Getxxx processors)
Anyone has some hints to suggest me? Am I on the right path? I would be sure that I am not overlooking at some NiFi concept/feature that can allows me to better manage this Use case. I hope to have been clear enough, any shared shared would be extremely appreciated! Best regards,Damiano -- == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Damiano Giampaoli Software Engineer GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) Italy phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 333 8128928 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- AVVERTENZE AI SENSI DEL D.Lgs. 196/2003 Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003. The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.
