RE: ConverJsonToSql issue
Hi Toivo, Thanks for your reply. Yes, error message contain database name and table name(database.table) But names of the fields are same and I tried with option “Translate Field Names” also but facing same issue. GetFile-->Split-->InferAvroSchema-->convertCsvtoAvro-->ConvertAvroToJson-->ConvertJsonToSQL-->PutSQL-->LogAttribute Please suggest me. Thanks, Sravani From: Toivo Adams [mailto:toivo.ad...@gmail.com] Sent: Friday, January 20, 2017 7:23 PM To: users@nifi.apache.org Subject: Re: ConverJsonToSql issue Hi Sravani, Weird, error message should also contain database table name. It seems none of the database table column names match with JSON field names. If names are different, you can use "Translate Field Names" property to specify mappings. Regards, Toivo 2017-01-20 12:04 GMT+02:00 Gadiputi, Sravani <sravani.gadip...@capgemini.com<mailto:sravani.gadip...@capgemini.com>>: Hi Team, I am facing the issue with ConverJsonToSql processor in Nifi. I am trying to read the csv file and load the data in to Teradata table. My flow is look like below: GetFile-->Split-->convertCsvtoAvro-->ConvertAvroToJson--->ConvertJsonToSQL-->PutSQL-->LogAttribute When it comes to convertJsonToSql, it is not able to map the JSON fields with table field. Error: Failed to convert Slandered Flowfile record and None of the fields in the JSON map to the columns defined by the table. Any ideas/suggestions to resolve this? Thanks in Advance!! Regards, Sravani This message contains information that may be privileged or confidential and is the property of the Capgemini Group. It is intended only for the person to whom it is addressed. If you are not the intended recipient, you are not authorized to read, print, retain, copy, disseminate, distribute, or use this message or any part thereof. If you receive this message in error, please notify the sender immediately and delete all copies of this message.
ConverJsonToSql issue
Hi Team, I am facing the issue with ConverJsonToSql processor in Nifi. I am trying to read the csv file and load the data in to Teradata table. My flow is look like below: GetFile-->Split-->convertCsvtoAvro-->ConvertAvroToJson--->ConvertJsonToSQL-->PutSQL-->LogAttribute When it comes to convertJsonToSql, it is not able to map the JSON fields with table field. Error: Failed to convert Slandered Flowfile record and None of the fields in the JSON map to the columns defined by the table. Any ideas/suggestions to resolve this? Thanks in Advance!! Regards, Sravani This message contains information that may be privileged or confidential and is the property of the Capgemini Group. It is intended only for the person to whom it is addressed. If you are not the intended recipient, you are not authorized to read, print, retain, copy, disseminate, distribute, or use this message or any part thereof. If you receive this message in error, please notify the sender immediately and delete all copies of this message.
RE: Nifi- PutEmail processor issue
Thanks Joe and Oleg for your inputs. The issue is mainly sending the email through PutEmail. Irrespective of the flow success or failure, the mail should trigger. In this case, when putemail receives intimation to trigger mail to the corresponding recipient, at that time it is failing due to connection timed out. I think it's because of network availability or network traffic. It's purely about putemail processor not about transferring the file.I Provided correct host,username..etc details also. We can partially overcome this issue with the help of Re-try concept. Whenever putemail encountered the error, it will keep on try to connect for number of times which is set by us using expression language. In-between putemail should connect to host and send the email. What I am looking is there any other way in Putemail process itself to overcome this issue. Thanks, Sravani -Original Message- From: Joe Witt [mailto:joe.w...@gmail.com] Sent: Tuesday, November 15, 2016 6:54 PM To: users@nifi.apache.org Cc: jtsw...@gmail.com Subject: Re: Nifi- PutEmail processor issue Sravani, The flow you describe makes sense. So now lets focus on the PutEmail processor. Can you please clarify if when it fails to connect the flowfile is transferred to the 'failure' relationship or the 'success' relationship? If it flows to the 'failure' relationship then as Conrad points out you can just simply have failure loop back to that processor so it will continue to retry until it actually succeeds in transferring. If it flows to the 'success' relationship even though the email send fails then we need to look into it further and the stack trace oleg requests will be helpful. Thanks Joe On Tue, Nov 15, 2016 at 7:47 AM, Oleg Zhurakousky <ozhurakou...@hortonworks.com> wrote: > Sravani > > Would you be able to provide a full stack trace of the connection exception. > Also, while I assume you are providing the correct connection > properties (i.e., host,port etc) I would still recommend to check the > they are correct, but in any event the full stack trace would > definitely help and you cn find it the the wifi app logs. > > Cheers > Oleg > > On Nov 15, 2016, at 4:07 AM, Gadiputi, Sravani > <sravani.gadip...@capgemini.com> wrote: > > Thank you for your reply. > > My requirement is , I just try to send/copy the 3 different files from > source to destination through Nifi, and these jobs runs weekly once. > So I wanted to know which file is successfully moved through email. > In this process, I have configured putemail for each flow. There are > hardly > 3 notifications only. > Though files have been moved to destination, we could not receive the > notifications properly and giving the below error. > > Please suggest. > > Thanks, > Sravani > > > From: Jeff [mailto:jtsw...@gmail.com] > Sent: Tuesday, November 15, 2016 1:25 PM > To: users@nifi.apache.org > Subject: Re: Nifi- PutEmail processor issue > > Hello Sravani, > > Could it be possible that the SMTP server you're using is denying > connections due to the volume of emails your flow might be sending? > How many emails are sent per flow file, and how many emails do you > estimate are sent per minute? > > If this is the case, you can modify your flow to aggregate flowfiles > with a processor like MergeContent so that you can send emails that > resemble a digest, rather than a separate email for each flowfile that > moves through your flow. > > On Mon, Nov 14, 2016 at 11:59 PM Gadiputi, Sravani > <sravani.gadip...@capgemini.com> wrote: > > > Hi, > > I have used PutEmail processor in my project to send email > notification for successful/failure copying of a files. > Each file flow having corresponding PutEmail to send email > notification to respective recipients. > > Here the issue is, sometimes email notification will send to > respective recipients successfully for successful/failure job. > But sometimes for any one specific job email notification will not be > send to recipients though job is successful, due to below error. > > Error: > > Could not connect to SMTP host > Java.net.ConnectException: Connection timed out > > Could you please suggest me how we can overcome this error. > > > Thanks, > Sravani > > This message contains information that may be privileged or > confidential and is the property of the Capgemini Group. It is > intended only for the person to whom it is addressed. If you are not > the intended recipient, you are not authorized to read, print, retain, > copy, disseminate, distribute, or use this message or any part > thereof. If you receive this message in error, please notify the sender > immediately and delete all copies of this message. >
RE: Nifi- PutEmail processor issue
Thank you for your reply. My requirement is , I just try to send/copy the 3 different files from source to destination through Nifi, and these jobs runs weekly once. So I wanted to know which file is successfully moved through email. In this process, I have configured putemail for each flow. There are hardly 3 notifications only. Though files have been moved to destination, we could not receive the notifications properly and giving the below error. Please suggest. Thanks, Sravani From: Jeff [mailto:jtsw...@gmail.com] Sent: Tuesday, November 15, 2016 1:25 PM To: users@nifi.apache.org Subject: Re: Nifi- PutEmail processor issue Hello Sravani, Could it be possible that the SMTP server you're using is denying connections due to the volume of emails your flow might be sending? How many emails are sent per flow file, and how many emails do you estimate are sent per minute? If this is the case, you can modify your flow to aggregate flowfiles with a processor like MergeContent so that you can send emails that resemble a digest, rather than a separate email for each flowfile that moves through your flow. On Mon, Nov 14, 2016 at 11:59 PM Gadiputi, Sravani <sravani.gadip...@capgemini.com<mailto:sravani.gadip...@capgemini.com>> wrote: Hi, I have used PutEmail processor in my project to send email notification for successful/failure copying of a files. Each file flow having corresponding PutEmail to send email notification to respective recipients. Here the issue is, sometimes email notification will send to respective recipients successfully for successful/failure job. But sometimes for any one specific job email notification will not be send to recipients though job is successful, due to below error. Error: Could not connect to SMTP host Java.net.ConnectException: Connection timed out Could you please suggest me how we can overcome this error. Thanks, Sravani This message contains information that may be privileged or confidential and is the property of the Capgemini Group. It is intended only for the person to whom it is addressed. If you are not the intended recipient, you are not authorized to read, print, retain, copy, disseminate, distribute, or use this message or any part thereof. If you receive this message in error, please notify the sender immediately and delete all copies of this message.
Nifi- PutEmail processor issue
Hi, I have used PutEmail processor in my project to send email notification for successful/failure copying of a files. Each file flow having corresponding PutEmail to send email notification to respective recipients. Here the issue is, sometimes email notification will send to respective recipients successfully for successful/failure job. But sometimes for any one specific job email notification will not be send to recipients though job is successful, due to below error. Error: Could not connect to SMTP host Java.net.ConnectException: Connection timed out Could you please suggest me how we can overcome this error. Thanks, Sravani This message contains information that may be privileged or confidential and is the property of the Capgemini Group. It is intended only for the person to whom it is addressed. If you are not the intended recipient, you are not authorized to read, print, retain, copy, disseminate, distribute, or use this message or any part thereof. If you receive this message in error, please notify the sender immediately and delete all copies of this message.
RE: Passing the values to the processors Dynamically
Thanks for the reply Bryan.
I can use UpdateAttribute processor to set the attribute value like
username=myuser but how username value will set to myuser.
If I am trying in wrong way Could you please explain with some example.
Looking forward for your valuable inputs/suggestions.
Thanks in advance!
Regards,
Sravani
From: Bryan Bende [mailto:bbe...@gmail.com]
Sent: Tuesday, July 19, 2016 10:02 PM
To: users@nifi.apache.org
Subject: Re: Passing the values to the processors Dynamically
Hello,
The properties for hostname, username, and remote path, all support expression
language.
Somewhere before FetchSFTP if you set an attribute on a flow file like username
= myuser then in the username property of FetchSFTP you could set the value to
${username} to reference it.
Hope that helps.
-Bryan
On Tue, Jul 19, 2016 at 3:19 AM, Gadiputi, Sravani
<sravani.gadip...@capgemini.com<mailto:sravani.gadip...@capgemini.com>> wrote:
Hi Team,
I have a requirement where I have to move the files from Teradata to S3.
Here I am using FetchSFTP processor to fetch the file and putS3 processor to
store the file into S3 , and I have configured FetchSFTP processor with
properties like hostname, username,
Password.
Now I wanted to pass the Host name,username and Remote path values dynamically.
How can we achieve this.
Please suggest me, It would be great help.
Thanks,
Sravani
This message contains information that may be privileged or confidential and is
the property of the Capgemini Group. It is intended only for the person to whom
it is addressed. If you are not the intended recipient, you are not authorized
to read, print, retain, copy, disseminate, distribute, or use this message or
any part thereof. If you receive this message in error, please notify the
sender immediately and delete all copies of this message.
Passing the values to the processors Dynamically
Hi Team, I have a requirement where I have to move the files from Teradata to S3. Here I am using FetchSFTP processor to fetch the file and putS3 processor to store the file into S3 , and I have configured FetchSFTP processor with properties like hostname, username, Password. Now I wanted to pass the Host name,username and Remote path values dynamically. How can we achieve this. Please suggest me, It would be great help. Thanks, Sravani This message contains information that may be privileged or confidential and is the property of the Capgemini Group. It is intended only for the person to whom it is addressed. If you are not the intended recipient, you are not authorized to read, print, retain, copy, disseminate, distribute, or use this message or any part thereof. If you receive this message in error, please notify the sender immediately and delete all copies of this message.
RE: Shell script execution through Nifi
Hi Bryan, Thank you for solution. Here I am adding few more to my question. 1)Can we capture any output from the execution of the shell script in flowfile. 2)is there any process to validate logging of remote machine through Nifi. 3) We have to do remote login using ssh to the remote server but it should be passwordless. How can we achieve? After connecting to server, how to execute shell script in remote server through NIFI? Please give me suggestions/inputs on the above points. Thanks a lot for your assistance. Looking forward for your reply. Regards, Sravani From: Bryan Bende [mailto:bbe...@gmail.com] Sent: Monday, July 11, 2016 6:25 PM To: users@nifi.apache.org Subject: Re: Shell script execution through Nifi Hello, Can you use ExecuteProcess to call a local shell script that SSH's to the remote machine and executes whatever you need there? -Bryan On Mon, Jul 11, 2016 at 8:49 AM, Gadiputi, Sravani <sravani.gadip...@capgemini.com<mailto:sravani.gadip...@capgemini.com>> wrote: Hi Team, Need your assistance/inputs for below requirement. I wanted to Execute shell script/spark jobs in remote machines through Nifi .How can we achieve this? Could you please suggest any solution/work abounds, that would be great help. Thanks in advance!! Regards, Sravani This message contains information that may be privileged or confidential and is the property of the Capgemini Group. It is intended only for the person to whom it is addressed. If you are not the intended recipient, you are not authorized to read, print, retain, copy, disseminate, distribute, or use this message or any part thereof. If you receive this message in error, please notify the sender immediately and delete all copies of this message.
Shell script execution through Nifi
Hi Team, Need your assistance/inputs for below requirement. I wanted to Execute shell script/spark jobs in remote machines through Nifi .How can we achieve this? Could you please suggest any solution/work abounds, that would be great help. Thanks in advance!! Regards, Sravani This message contains information that may be privileged or confidential and is the property of the Capgemini Group. It is intended only for the person to whom it is addressed. If you are not the intended recipient, you are not authorized to read, print, retain, copy, disseminate, distribute, or use this message or any part thereof. If you receive this message in error, please notify the sender immediately and delete all copies of this message.
RE: ExecuteProcess usecases
Hi, Any information regarding below questions. Thanks, Sravani From: Gadiputi, Sravani [mailto:sravani.gadip...@capgemini.com] Sent: Friday, July 08, 2016 2:05 PM To: users@nifi.apache.org Subject: ExecuteProcess usecases WARNING: Kindly be aware the Sender Address on this mail may be forged. It appears to be from capgemini.com but the message has been received from a server outside Capgemini Group perimeter. User discretion is necessary before performing actions mentioned in this mail. Hi, I need more information on ExecuteProcess,below are the questions. 1) What is the scope of the ExecuteProcess processor in Nifi? 2) Can we use ExecuteProcess to execute shell script in remote server. For example I wanted to delete the files in hadoop system file path using Nifi. so I have taken ExecuteProcess. Filename:test_del.sh Path:/tmp/test_del.sh Content of the .sh file is: "hadoop fs -rm /usr/dev/test_files/*" Note: Nifi is installed in same instance. In the property section of ExecuteProcess: Command: bash Command Arguments: :/tmp/test_del.sh So I can achieve execution of .sh file in this way, lets say in the local machine. Here my requirement is None of those scripts are in the server where nifi is running. And, we cannot install nifi in any other place. So how to do it for executing the shell script in remote machine . Is it possible with only ExecuteProcess or we can also use other processors to achieve this? Please help me in this.. Thanks in advance!! Regards, Sravani This message contains information that may be privileged or confidential and is the property of the Capgemini Group. It is intended only for the person to whom it is addressed. If you are not the intended recipient, you are not authorized to read, print, retain, copy, disseminate, distribute, or use this message or any part thereof. If you receive this message in error, please notify the sender immediately and delete all copies of this message.
ExecuteProcess usecases
Hi, I need more information on ExecuteProcess,below are the questions. 1) What is the scope of the ExecuteProcess processor in Nifi? 2) Can we use ExecuteProcess to execute shell script in remote server. For example I wanted to delete the files in hadoop system file path using Nifi. so I have taken ExecuteProcess. Filename:test_del.sh Path:/tmp/test_del.sh Content of the .sh file is: "hadoop fs -rm /usr/dev/test_files/*" Note: Nifi is installed in same instance. In the property section of ExecuteProcess: Command: bash Command Arguments: :/tmp/test_del.sh So I can achieve execution of .sh file in this way, lets say in the local machine. Here my requirement is None of those scripts are in the server where nifi is running. And, we cannot install nifi in any other place. So how to do it for executing the shell script in remote machine . Is it possible with only ExecuteProcess or we can also use other processors to achieve this? Please help me in this.. Thanks in advance!! Regards, Sravani This message contains information that may be privileged or confidential and is the property of the Capgemini Group. It is intended only for the person to whom it is addressed. If you are not the intended recipient, you are not authorized to read, print, retain, copy, disseminate, distribute, or use this message or any part thereof. If you receive this message in error, please notify the sender immediately and delete all copies of this message.
RE: CRON driven scheduling in NiFi
Thanks Lee, now it is working fine as per our requirement(0 30 20 ? * SAT *). Thanks, Sravani -Original Message- From: Lee Laim (leelaim) [mailto:leel...@micron.com] Sent: Monday, July 04, 2016 1:29 PM To: users@nifi.apache.org Subject: Re: CRON driven scheduling in NiFi Sravani, I believe the cron scheduling is based on the Quartz scheduler, which differs from the UNIX crontab. A question mark represents no specific value for either dayofweek or dayofmonth 0 30 20 ? * SAT * should work for every Saturday 830pm (Saturday regardless of dayofmonth). Thanks, Lee leel...@micron.com<mailto:leel...@micron.com> On Jul 3, 2016, at 11:50 PM, Gadiputi, Sravani <sravani.gadip...@capgemini.com<mailto:sravani.gadip...@capgemini.com>> wrote: Hi Team, As per my project requirement I have to schedule the process for every Saturday 8.30pm. So I have used the CRON driven type scheduling in processor in order to repeat the process for every Saturday. But its is throwing error while setup the value for “day of the week”. As per NiFi documentation below is the instruction should fallow. 1. Seconds 2. Minutes 3. Hours 4. Day of Month 5. Month 6. Day of Week 7. Year As per instructions, I have given details. Example I have tried for scheduling the process for every Wednesday 3:00pm and per every 5 sec. So I have given in CRON driven schedule “*/5 * 3 * 6 5 ?” It is working fine for “*/5 * 3 * 6 ?”—up to day of the month. When I try to particular day of the week it is not working, I mean while configuring the values itself its throwing configuration error. Error: ”scheduling ‘*/5 * 3 * 6 4 ?’ is not a valid CRON expression”. I stuck with this and it is more impacting on my development. Please help me in this. Thanks in advance!!! Regards, Sravani This message contains information that may be privileged or confidential and is the property of the Capgemini Group. It is intended only for the person to whom it is addressed. If you are not the intended recipient, you are not authorized to read, print, retain, copy, disseminate, distribute, or use this message or any part thereof. If you receive this message in error, please notify the sender immediately and delete all copies of this message.
