Hi, I've described the problem in Stack Overflow with a lot of detailing, can you kindly check and help if possible?
https://stackoverflow.com/q/61643910/5536733 I'd be absolutely fine if someone solves it using Spark SQL APIs rather than plain spark SQL query. Thanks, Aakash.