Hello Pierre, We have data in a schema and then multiple tables. so we have few doubts on that
1. Which processer is to be best used for bulk load 2. Which processor to be used to Insert update record from one center to another in incremental form. 3. Which process is to be used for update deletion format. We are having data in 3 different locations (Delhi, Bangalore and Lucknow) and from all locations data has to come to Delhi location. All inserts and update are to be done from Delhi. We are using nifi of version 1.12.1 and DB is in postgres version 12.12. Nifi configerations has been attached in onedrive. https://hpe-my.sharepoint.com/:f:/p/meghna_arora/EmHrg__NKRpKsmPAv_BDG8YBhDYTSz4Dn23MLGxjmgpREg?e=Y4gGF6 Password: Hpe@!23 Right now we test and it is giving some error "ERROR: column "created_on" is of type timestamp without time zone but expression is of type character varying" Can you please look into it. Regards Meghna Arora [https://attachments.office.net/owa/meghna.arora%40hpe.com/service.svc/s/GetAttachmentThumbnail?id=AAMkADIyNDgzODc5LTU2NjctNDVmNS04ZmViLTQ5NTZlYzc5ODNhOABGAAAAAAC8Ib0GF8fSQKF5%2Fq7H9aFLBwD3XB7kcFaLS6dFCvhJoQB1AAAAAAEPAACkZDZm4whhRrbLHA0d403dAAAAAOslAAABEgAQAO6RTZxOc4hCnN8NvW%2FFw38%3D&thumbnailType=2&token=eyJhbGciOiJSUzI1NiIsImtpZCI6IjMwODE3OUNFNUY0QjUyRTc4QjJEQjg5NjZCQUY0RUNDMzcyN0FFRUUiLCJ0eXAiOiJKV1QiLCJ4NXQiOiJNSUY1emw5TFV1ZUxMYmlXYTY5T3pEY25ydTQifQ.eyJvcmlnaW4iOiJodHRwczovL291dGxvb2sub2ZmaWNlLmNvbSIsInVjIjoiODU3YTllOWZmOTVmNGNiMzhlOTU2MGE2OGFkN2E2YTgiLCJzaWduaW5fc3RhdGUiOiJbXCJrbXNpXCJdIiwidmVyIjoiRXhjaGFuZ2UuQ2FsbGJhY2suVjEiLCJhcHBjdHhzZW5kZXIiOiJPd2FEb3dubG9hZEAxMDViMjA2MS1iNjY5LTRiMzEtOTJhYy0yNGQzMDRkMTk1ZGMiLCJpc3NyaW5nIjoiV1ciLCJhcHBjdHgiOiJ7XCJtc2V4Y2hwcm90XCI6XCJvd2FcIixcInB1aWRcIjpcIjExNTM4MDExMTY2NTAxMzkwNzNcIixcInNjb3BlXCI6XCJPd2FEb3dubG9hZFwiLFwib2lkXCI6XCIzYTNlNWMyMS02MjkyLTQ4ZGYtYjliZS1jOTcyOTUzNjZmNDlcIixcInByaW1hcnlzaWRcIjpcIlMtMS01LTIxLTQxNjM0NTY4MDctNDA3MTQwNTc3NS0yMjI5OTk0MTctODA0OTA0MVwifSIsIm5iZiI6MTYxMzY0NDU3OSwiZXhwIjoxNjEzNjQ1MTc5LCJpc3MiOiIwMDAwMDAwMi0wMDAwLTBmZjEtY2UwMC0wMDAwMDAwMDAwMDBAMTA1YjIwNjEtYjY2OS00YjMxLTkyYWMtMjRkMzA0ZDE5NWRjIiwiYXVkIjoiMDAwMDAwMDItMDAwMC0wZmYxLWNlMDAtMDAwMDAwMDAwMDAwL2F0dGFjaG1lbnRzLm9mZmljZS5uZXRAMTA1YjIwNjEtYjY2OS00YjMxLTkyYWMtMjRkMzA0ZDE5NWRjIiwiaGFwcCI6Im93YSJ9.UsbBPbOidWFW0XPytopAMq4nNSi6MqEyUJrxYAQpWnWWNF8cWJhZ-3vKRiJZKntOIfUcxYLpoDHGKUGE1k7xMUBeCHu0B-Z1nFzmtacNQ2MQaMIAidCHinuPA7X3GKDLQvxx0y7s3XyWBKF_4ERbWiQjc5DOMafmnVv7sEMAatb_6Xb9XVmYmb2fSADE3T_Ovbl_punlWSh-M-Jka2DdQXlGnRIv5_4VePmKBVZjdtPQ7Q1oBmtUjT98F7ZjOEM2FsxxkJ6tTIrwem2yEWGITS_OfcQM8SAdl6i7rMiu-XJuxZGRnmZ2EIPa0T5rY0-8Kdd2RBfLaaNA_TG1lCGWbw&X-OWA-CANARY=zhVPKf0AjkyDT3niYio4MYBFiVv509gYi5xUCf57jDnG_CYlZwgQBlr8sULLakIgD4xpgFtP10I.&owa=outlook.office.com&scriptVer=20210208002.03&animation=true] From: Pierre Villard <pierre.villard...@gmail.com> Sent: Tuesday, February 16, 2021 5:43 PM To: Arora, Meghna <meghna.ar...@hpe.com> Cc: dev@nifi.apache.org; Poddar, Rakesh Kumar <rakesh-kumar.pod...@hpe.com>; Bhattacharjee, Koushik <koushik.bhattachar...@hpe.com>; Kiri, Aarti <aarti.k...@hpe.com> Subject: Re: Connectivity issue (Nifi to postgres) Hi, As said before, we need more information. We don't see the images you're adding to your messages and I don't recommend attaching zip files to messages here. Can please add your logs/images/etc on something like Gist by Github [1]. Also, please describe the use case, the processors you are using, the configuration of the processors, what is your flow, NiFi version you have, how many nodes, etc. The more info you give, the more likely someone would be willing to spend some of their time to help you. Thanks, Pierre [1] https://gist.github.com/ Le mar. 16 févr. 2021 à 16:04, Arora, Meghna <meghna.ar...@hpe.com<mailto:meghna.ar...@hpe.com>> a écrit : Hello Team, Any update on this. We have to process 3TB records and an incremental value of 1 lakh records from source to destination. we are getting following error. LOGS: 2021-02-16 15:48:48,496 ERROR [Cleanup Archive for default] o.a.n.c.repository.FileSystemRepository java.lang.OutOfMemoryError: GC overhead limit exceeded 2021-02-16 15:49:55,755 ERROR [Framework Task Thread Thread-1] o.a.n.c.s.StandardProcessScheduler Failed to run Framework Task Expire FlowFiles due to java.lang.OutOfMemoryError: GC overhead limit exceeded As an option I am trying to explore Query Record to solve problem. Is it correct? Regards Meghna ________________________________ From: Arora, Meghna Sent: Thursday, February 4, 2021 6:22 PM To: dev@nifi.apache.org<mailto:dev@nifi.apache.org> <dev@nifi.apache.org<mailto:dev@nifi.apache.org>>; pierre.villard...@gmail.com<mailto:pierre.villard...@gmail.com> <pierre.villard...@gmail.com<mailto:pierre.villard...@gmail.com>> Cc: Poddar, Rakesh Kumar <rakesh-kumar.pod...@hpe.com<mailto:rakesh-kumar.pod...@hpe.com>>; Bhattacharjee, Koushik <koushik.bhattachar...@hpe.com<mailto:koushik.bhattachar...@hpe.com>> Subject: RE: Connectivity issue (Nifi to postgres) Hello Pierre, Logs has been attached for same. Regards Meghna Arora [https://attachments.office.net/owa/meghna.arora%40hpe.com/service.svc/s/GetAttachmentThumbnail?id=AAMkADIyNDgzODc5LTU2NjctNDVmNS04ZmViLTQ5NTZlYzc5ODNhOABGAAAAAAC8Ib0GF8fSQKF5%2Fq7H9aFLBwD3XB7kcFaLS6dFCvhJoQB1AAAAAAEPAACkZDZm4whhRrbLHA0d403dAAAAAOslAAABEgAQAO6RTZxOc4hCnN8NvW%2FFw38%3D&thumbnailType=2&token=eyJhbGciOiJSUzI1NiIsImtpZCI6IjMwODE3OUNFNUY0QjUyRTc4QjJEQjg5NjZCQUY0RUNDMzcyN0FFRUUiLCJ0eXAiOiJKV1QiLCJ4NXQiOiJNSUY1emw5TFV1ZUxMYmlXYTY5T3pEY25ydTQifQ.eyJvcmlnaW4iOiJodHRwczovL291dGxvb2sub2ZmaWNlLmNvbSIsInVjIjoiODU3YTllOWZmOTVmNGNiMzhlOTU2MGE2OGFkN2E2YTgiLCJzaWduaW5fc3RhdGUiOiJbXCJrbXNpXCJdIiwidmVyIjoiRXhjaGFuZ2UuQ2FsbGJhY2suVjEiLCJhcHBjdHhzZW5kZXIiOiJPd2FEb3dubG9hZEAxMDViMjA2MS1iNjY5LTRiMzEtOTJhYy0yNGQzMDRkMTk1ZGMiLCJpc3NyaW5nIjoiV1ciLCJhcHBjdHgiOiJ7XCJtc2V4Y2hwcm90XCI6XCJvd2FcIixcInB1aWRcIjpcIjExNTM4MDExMTY2NTAxMzkwNzNcIixcInNjb3BlXCI6XCJPd2FEb3dubG9hZFwiLFwib2lkXCI6XCIzYTNlNWMyMS02MjkyLTQ4ZGYtYjliZS1jOTcyOTUzNjZmNDlcIixcInByaW1hcnlzaWRcIjpcIlMtMS01LTIxLTQxNjM0NTY4MDctNDA3MTQwNTc3NS0yMjI5OTk0MTctODA0OTA0MVwifSIsIm5iZiI6MTYxMzY0NDU3OSwiZXhwIjoxNjEzNjQ1MTc5LCJpc3MiOiIwMDAwMDAwMi0wMDAwLTBmZjEtY2UwMC0wMDAwMDAwMDAwMDBAMTA1YjIwNjEtYjY2OS00YjMxLTkyYWMtMjRkMzA0ZDE5NWRjIiwiYXVkIjoiMDAwMDAwMDItMDAwMC0wZmYxLWNlMDAtMDAwMDAwMDAwMDAwL2F0dGFjaG1lbnRzLm9mZmljZS5uZXRAMTA1YjIwNjEtYjY2OS00YjMxLTkyYWMtMjRkMzA0ZDE5NWRjIiwiaGFwcCI6Im93YSJ9.UsbBPbOidWFW0XPytopAMq4nNSi6MqEyUJrxYAQpWnWWNF8cWJhZ-3vKRiJZKntOIfUcxYLpoDHGKUGE1k7xMUBeCHu0B-Z1nFzmtacNQ2MQaMIAidCHinuPA7X3GKDLQvxx0y7s3XyWBKF_4ERbWiQjc5DOMafmnVv7sEMAatb_6Xb9XVmYmb2fSADE3T_Ovbl_punlWSh-M-Jka2DdQXlGnRIv5_4VePmKBVZjdtPQ7Q1oBmtUjT98F7ZjOEM2FsxxkJ6tTIrwem2yEWGITS_OfcQM8SAdl6i7rMiu-XJuxZGRnmZ2EIPa0T5rY0-8Kdd2RBfLaaNA_TG1lCGWbw&X-OWA-CANARY=zhVPKf0AjkyDT3niYio4MYBFiVv509gYi5xUCf57jDnG_CYlZwgQBlr8sULLakIgD4xpgFtP10I.&owa=outlook.office.com&scriptVer=20210208002.03&animation=true] ________________________________ From: Pierre Villard <pierre.villard...@gmail.com<mailto:pierre.villard...@gmail.com>> Sent: 29 January 2021 23:52 To: dev@nifi.apache.org<mailto:dev@nifi.apache.org> <dev@nifi.apache.org<mailto:dev@nifi.apache.org>> Cc: iss...@nifi.apache.org<mailto:iss...@nifi.apache.org> <iss...@nifi.apache.org<mailto:iss...@nifi.apache.org>>; us...@nifi.apache.org<mailto:us...@nifi.apache.org> <us...@nifi.apache.org<mailto:us...@nifi.apache.org>>; Bhattacharjee, Koushik <koushik.bhattachar...@hpe.com<mailto:koushik.bhattachar...@hpe.com>>; Poddar, Rakesh Kumar <rakesh-kumar.pod...@hpe.com<mailto:rakesh-kumar.pod...@hpe.com>> Subject: Re: Connectivity issue (Nifi to postgres) Hi, We'll need more information like the error message, stack trace, versions you're using for NiFi / Registry / Postgres, configuration, etc. Pierre Le ven. 29 janv. 2021 à 16:48, Arora, Meghna <meghna.ar...@hpe.com<mailto:meghna.ar...@hpe.com>> a écrit : Hello Team, We are facing some issue in NIFI connection. JDBC Driver of postgres is not connecting. Please let me know how to resolve this issue. Regards Meghna
