RE: Using Avro file format with SparkSQL

2022-02-14 Thread Morven Huang
’. Regards, Morven Huang On 2022/02/10 03:25:28 "Karanika, Anna" wrote: > Hello, > > I have been trying to use spark SQL’s operations that are related to the Avro > file format, > e.g., stored as, save, load, in a Java class but they keep failing with the

RE: Re: Using Avro file format with SparkSQL

2022-02-14 Thread Morven Huang
rom a Maven repository. > > This means we never have to build fat or uber jars. > > It does mean that the Apache Ivy configuration has to be set up correctly > though. > > Cheers, > > Steve C > > > On 15 Feb 2022, at 5:58 pm, Morven Huang wrote: > > >

Re: Cast int to string not possible?

2022-02-16 Thread Morven Huang
Hi Rico, you have any code snippet? I have no problem casting int to string. > 2022年2月17日 上午12:26,Rico Bergmann 写道: > > Hi! > > I am reading a partitioned dataFrame into spark using automatic type > inference for the partition columns. For one partition column the data > contains an integer,

Re: [Spark] spark client for Hadoop 2.x

2022-04-06 Thread Morven Huang
I remember that ./dev/make-distribution.sh in spark source allows people to specify Hadoop version. > 2022年4月6日 下午4:31,Amin Borjian 写道: > > From Spark version 3.1.0 onwards, the clients provided for Spark are built > with Hadoop 3 and placed in maven Repository. Unfortunately we use Hadoop >