Re: Urgent help on S3 CSV file reader DataStream Job

2020-12-16 Thread Wei Zhong
Hi Deep, You can try to change the `FileProcessingMode.PROCESS_ONCE` to `FileProcessingMode.PROCESS_CONTINUOUSLY`. Best, Wei > 在 2020年12月15日,20:18,DEEP NARAYAN Singh 写道: > > Hi Wei, > Could you please suggest , how to fix this below issues. > > Thanks & Regards, > Deep > > On Mon, 14 Dec, 2

Re: Urgent help on S3 CSV file reader DataStream Job

2020-12-08 Thread Wei Zhong
Hi Deep, It seems that the TypeInformation array in your code has 2 elements, but we only need one here. This approach treats the entire csv file as a Row which has only a one column, so there should be only one `BasicTypeInfo.STRING_TYPE_INFO` in the array. And if you use the TextInputFormat i

Re: Urgent help on S3 CSV file reader DataStream Job

2020-12-07 Thread Wei Zhong
Hi Deep, Could you show your current code snippet? I have tried the Csv file data on my local machine and it works fine, so I guess what might be wrong elsewhere. Best, Wei > 在 2020年12月8日,03:20,DEEP NARAYAN Singh 写道: > > Hi Wei and Till, > Thanks for the quick reply. > > @Wei, I tried with

Re: Urgent help on S3 CSV file reader DataStream Job

2020-12-07 Thread Till Rohrmann
Hi Deep, Could you use the TextInputFormat which reads a file line by line? That way you can do the JSON parsing as part of a mapper which consumes the file lines. Cheers, Till On Mon, Dec 7, 2020 at 1:05 PM Wei Zhong wrote: > Hi Deep, > > (redirecting this to user mailing list as this is not

Re: Urgent help on S3 CSV file reader DataStream Job

2020-12-07 Thread Wei Zhong
Hi Deep, (redirecting this to user mailing list as this is not a dev question) You can try to set the line delimiter and field delimiter of the RowCsvInputFormat to a non-printing character (assume there is no non-printing characters in the csv files). It will read all the content of a csv file