Hi Natalia,
You can set the "directory" in the Prefix property of ListS3. In this case
no UpdateAttribute is needed, just connect ListS3 -> FetchS3Object.
Regards,
Peter Turcsanyi
On Mon, May 16, 2022 at 4:24 PM Fill, Natalia <natalia.f...@lgim.com> wrote:
> Hi Nifi Team,
>
> Following from tests below I need to retrieve files from specific
> directories on S3. The files are placed into dir/subdir/.
> I tried various expression to specify 'filename' variable that feeds to
> FetchS3Object and still get error that the key doesn't exist.
> I am not sure if FetchS3Object doesn't like my directories pattern or the
> file name. If I don't specify any pattern and just fetch what was listed,
> it works fine.
> The flow goes as follows: ListS3 (lists all the files as expected) -
> UpdateAttribute (updates filename with the pattern to use) - FetchS3Object
> - LogAttribute on success and on failure.
>
> Some of my tries for filename pattern below:
> *( dir/subdir/test_put_\d*_\d*\.txt).*
> *( test_put_\d*_\d*\.txt).*
> dir/subdir /test_put_\d*_\d*\.txt
> dir/subdir /test_put_${now():format('yyyy-MM-dd')}_*
> /dir/subdir /test_put_${now():format('yyyy-MM-dd')}_*
>
> List of files on s3 server is like below:
> 2022-05-16 13:36:03 14 test_put_20220516_013601.txt
> 2022-05-16 13:36:04 14 test_put_20220516_013602.txt
>
> Can anyone suggest how do I retrieve files in dir/subdir that start with a
> particular string (e.g. test_put_) contain today's date and end with
> sequence of numbers.
>
> Thanks
> Natalia
>
> .....
> -----Original Message-----
> From: Fill, Natalia
> Sent: 16 May 2022 13:15
> To: dev@nifi.apache.org
> Subject: RE: [EXTERNAL] Re: Nifi putting object to S3 directory
>
> Thanks Peter, removing leading slash worked and now Nifi puts file into
> specified subdirectory.
>
> This is a simple test flow to test read/write to s3: GenerateFlowFile - on
> success - UpdateAttribute (filename to meaningful name) - on success -
> PutS3Object - on success and on failure LogAttribute
>
> Regards
> Natalia
>
> -----Original Message-----
> From: Peter Turcsanyi <turcsa...@apache.org>
> Sent: 16 May 2022 12:41
> To: dev@nifi.apache.org
> Subject: [EXTERNAL] Re: Nifi putting object to S3 directory
>
> CAUTION: This email originated from outside of the organisation. Do not
> click links or open attachments unless you recognise the sender and know
> the content is safe.
>
>
> Hi Natalia,
>
> Is the FlowFile transferred to the success or the failure relationship
> from PutS3Object?
>
> In general, you need to use dirtag/subdirtag /${filename}, so no leading
> slash.
> Leading slash results in a directory called '/' on S3.
>
> Regards,
> Peter Turcsanyi
>
> On Mon, May 16, 2022 at 12:46 PM Fill, Natalia <natalia.f...@lgim.com>
> wrote:
>
> > Hi Nifi Team,
> >
> > I am working on a solution to put file into S3 server. I used
> > PutS3Object and managed to put some files into a bucket, but it failed
> > to put files into specific directories on S3. No errors, but I don't
> > see the files in the bucket. I used the following format for object key:
> > /dirtag/subdirtag /${filename}. The tags do exist on S3.
> >
> > Does anyone have any suggestions how to put S3 object into specific
> > directory and not just on top level in a bucket.
> >
> > Thanks
> > Natalia
> > This e-mail (and any attachments) may contain privileged and/or
> > confidential information which may be protected by copyright or other
> > intellectual property rights. If you are not the intended recipient
> > please do not disclose, copy, distribute, disseminate or take any
> > action in reliance on it. If you have received this e-mail in error
> > please reply to the sender and then immediately delete it (including,
> any attachments).
> > Should you wish to communicate with us by e-mail we cannot guarantee
> > the security of any data outside our own computer systems or that any
> > e-mail will be virus free.
> >
> > Any information contained in this e-mail may be subject to applicable
> > terms and conditions and must not be construed as giving investment
> > advice within or outside the United Kingdom or the Republic of Ireland.
> >
> > Telephone conversations and calls via other telecommunication
> > facilities may be recorded, including to comply with our legal and/or
> > regulatory requirements and/or to monitor the quality of our service.
> > For information about how we use your personal data, including your
> > legal rights, please refer to our privacy policy at:
> >
> https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.legalandgeneral.com%2Finstitutional%2Fprivacy-policy%2F&data=05%7C01%7CNatalia.Fill%40lgim.com%7C6ac22dcdbd82472c4c5508da37310939%7Cd246baabcc004ed2bc4ef8a46cbc590d%7C0%7C0%7C637882980987854174%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=fRb3k0FqLCtu6JlV1B77Sq1PCeQ7rOEUC8%2B5KtxkEN4%3D&reserved=0
> .
> >
> > Legal & General Investment Management Limited (Company number
> > 02091894), LGIM Real Assets (Operator) Limited (Company number
> > 05522016), LGIM International Limited (Company number 07716001), Legal
> > & General (Unit Trust Managers) Limited (Company number 01009418), GO
> > ETF Solutions LLP (Company number OC329482) and LGIM Corporate
> > Director Limited (Company number 07105051) are each authorised and
> > regulated by the Financial Conduct Authority. All are registered in
> > England & Wales with a registered office at One Coleman Street, London,
> EC2R 5AA.
> >
> > Legal and General Assurance (Pensions Management) Limited (Company
> > number
> > 01006112) is authorised by the Prudential Regulation Authority and
> > regulated by the Financial Conduct Authority and the Prudential
> > Regulation Authority. It is registered in England & Wales with a
> > registered office at One Coleman Street, London, EC2R 5AA.
> >
> > Legal & General Property Limited (Registration number 02091897) is
> > authorised and regulated by the Financial Conduct Authority for
> > insurance mediation activities. It is registered in England & Wales
> > with a registered office at One Coleman Street, London, EC2R 5AA.
> >
> > LGIM Managers (Europe) Limited is authorised and regulated by the
> > Central Bank of Ireland (Reference No C173733). It is registered in
> > the Republic of Ireland (Number 609677) with its principal business
> > address at 33/34 Sir John Rogerson's Quay, Dublin 2, D02 XK09.
> >
> > The ultimate parent company is Legal & General Group PLC (Company
> > number
> > 01417162) which is registered in England & Wales and has a registered
> > office at One Coleman Street, London, EC2R 5AA.
> >
> > ______________________________________________________________________
> > __
> > **** This email has come from the internet and has been scanned for
> > all viruses and potentially offensive content by Messagelabs on behalf
> > of Legal & General ****
>
>
> ________________________________________________________________________
> *** This email has come from the internet and has been scanned for all
> viruses and potentially offensive content by Messagelabs on behalf of Legal
> & General. Please report unwanted spam email to s...@landg.com and any
> malicious content to s...@landg.com ***
>
> Please consider the environment before printing this email.
> This e-mail (and any attachments) may contain privileged and/or
> confidential information which may be protected by copyright or other
> intellectual property rights. If you are not the intended recipient please
> do not disclose, copy, distribute, disseminate or take any action in
> reliance on it. If you have received this e-mail in error please reply to
> the sender and then immediately delete it (including, any attachments).
> Should you wish to communicate with us by e-mail we cannot guarantee the
> security of any data outside our own computer systems or that any e-mail
> will be virus free.
>
> Any information contained in this e-mail may be subject to applicable
> terms and conditions and must not be construed as giving investment advice
> within or outside the United Kingdom or the Republic of Ireland.
>
> Telephone conversations and calls via other telecommunication facilities
> may be recorded, including to comply with our legal and/or regulatory
> requirements and/or to monitor the quality of our service. For information
> about how we use your personal data, including your legal rights, please
> refer to our privacy policy at:
> www.legalandgeneral.com/institutional/privacy-policy/.
>
> Legal & General Investment Management Limited (Company number 02091894),
> LGIM Real Assets (Operator) Limited (Company number 05522016), LGIM
> International Limited (Company number 07716001), Legal & General (Unit
> Trust Managers) Limited (Company number 01009418), GO ETF Solutions LLP
> (Company number OC329482) and LGIM Corporate Director Limited (Company
> number 07105051) are each authorised and regulated by the Financial Conduct
> Authority. All are registered in England & Wales with a registered office
> at One Coleman Street, London, EC2R 5AA.
>
> Legal and General Assurance (Pensions Management) Limited (Company number
> 01006112) is authorised by the Prudential Regulation Authority and
> regulated by the Financial Conduct Authority and the Prudential Regulation
> Authority. It is registered in England & Wales with a registered office at
> One Coleman Street, London, EC2R 5AA.
>
> Legal & General Property Limited (Registration number 02091897) is
> authorised and regulated by the Financial Conduct Authority for insurance
> mediation activities. It is registered in England & Wales with a registered
> office at One Coleman Street, London, EC2R 5AA.
>
> LGIM Managers (Europe) Limited is authorised and regulated by the Central
> Bank of Ireland (Reference No C173733). It is registered in the Republic of
> Ireland (Number 609677) with its principal business address at 33/34 Sir
> John Rogerson's Quay, Dublin 2, D02 XK09.
>
> The ultimate parent company is Legal & General Group PLC (Company number
> 01417162) which is registered in England & Wales and has a registered
> office at One Coleman Street, London, EC2R 5AA.
>
> ________________________________________________________________________
> **** This email has come from the internet and has been scanned for all
> viruses and potentially offensive content by Messagelabs on behalf of Legal
> & General ****
>
