Re: Tab delimited csv import and empty columns

[German Schiavon Matteo]

Hey,

I understand that your empty values in your CSV are "" , if so, try this
option:

*.option("emptyValue", "\"\"")*

Hope it helps

On Thu, 30 Jul 2020 at 08:49, Stephen Coy 
wrote:

> Hi there,
>
> I’m trying to import a tab delimited file with:
>
> Dataset catalogData = sparkSession
>   .read()
>   .option("sep", "\t")
>   .option("header", "true")
>   .csv(args[0])
>   .cache();
>
>
> This works great, except for the fact that any column that is empty is
> given the value null, when I need these values to be literal empty strings.
>
> Is there any option combination that will achieve this?
>
> Thanks,
>
> Steve C
>
>
> 
> This email contains confidential information of and is the copyright of
> Infomedia. It must not be forwarded, amended or disclosed without consent
> of the sender. If you received this message by mistake, please advise the
> sender and delete all copies. Security of transmission on the internet
> cannot be guaranteed, could be infected, intercepted, or corrupted and you
> should ensure you have suitable antivirus protection in place. By sending
> us your or any third party personal details, you consent to (or confirm you
> have obtained consent from such third parties) to Infomedia’s privacy
> policy. http://www.infomedia.com.au/privacy-policy/
>

Tab delimited csv import and empty columns

[Stephen Coy]

Hi there,

I’m trying to import a tab delimited file with:

Dataset catalogData = sparkSession
  .read()
  .option("sep", "\t")
  .option("header", "true")
  .csv(args[0])
  .cache();

This works great, except for the fact that any column that is empty is given 
the value null, when I need these values to be literal empty strings.

Is there any option combination that will achieve this?

Thanks,

Steve C


[http://downloads.ifmsystems.com/data/marketing/images/signatures/driving-force-newsletter.jpg]

This email contains confidential information of and is the copyright of 
Infomedia. It must not be forwarded, amended or disclosed without consent of 
the sender. If you received this message by mistake, please advise the sender 
and delete all copies. Security of transmission on the internet cannot be 
guaranteed, could be infected, intercepted, or corrupted and you should ensure 
you have suitable antivirus protection in place. By sending us your or any 
third party personal details, you consent to (or confirm you have obtained 
consent from such third parties) to Infomedia’s privacy policy. 
http://www.infomedia.com.au/privacy-policy/