Re: [SPARK-SQL] How to return GenericInternalRow from spark udf

The UDF should return the result value you want, not a whole Row. In Scala it figures out the schema of the UDF's result from the signature. On Thu, Aug 6, 2020 at 7:56 AM Amit Joshi wrote: > > Hi, > > I have a spark udf written in scala that takes couuple of columns and apply > some logic and

[SPARK-SQL] How to return GenericInternalRow from spark udf

Hi, I have a spark udf written in scala that takes couuple of columns and apply some logic and output InternalRow. There is spark schema of StructType also present. But when I try to return the InternalRow from UDF there is exception

join doesn't work

I've using streamline pulsar connector, each dataset receives the data properly but cannot make join to be working Dataset datasetPolicyWithWtm = datasetPolicy.withWatermark("__publishTime", "5 minutes").as("pol"); Dataset datasetPhoneWithWtm = datasetPhone.withWatermark("__publishTime", "5

Re: S3 read/write from PySpark

Hi, Thanks for your help. Problem solved, but I thought I should add something in case this problem is encountered by others. Both responses are correct; BasicAWSCredentialsProvider is gone, but simply making the substitution leads to the traceback just below. java.lang.NoSuchMethodError: 'void

Re: S3 read/write from PySpark

Hi Daniel, It looks like …BasicAWSCredentialsProvider has become org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider. However, the way that the username and password are provided appears to have changed so you will probably need to look in to that. Cheers, Steve C On 6 Aug 2020, at 11:15

Re: S3 read/write from PySpark

Hey, I think *BasicAWSCredentialsProvider *is no longer supported by hadoop. I couldn't find it the master branch but I could in 2.8 branch. Maybe that's why with Hadoop 2.7 works. I use *TemporaryAWSCredentialsProvider.* Hope it helps On Thu, 6 Aug 2020 at 03:16, Daniel Stojanov wrote: >