Grouping and counting occurences of specific column rows

Hi all, Sorry for posting this twice, I need to know how to group by several column attributes (e.g.,List groupByAttributes) a dataset (dataset) and then count the occurrences of associated grouped rows, how do i achieve that ? I tried through the following code: > Dataset groupedRows =

Streaming write to orc problem

Hello all, I’m just trying to build a pipeline reading data from a streaming source and write to orc file. But I don’t see any file that is written to the file system nor any exceptions Here is an example val df = spark.readStream.format(“...") .option( “Topic", "Some

[Spark Core]: Unexpectedly exiting executor while gracefully decommissioning

Hello all, we are running into some issues while attempting graceful decommissioning of executors. We are running spark-thriftserver (3.2.0) on Kubernetes (GKE 1.20.15-gke.2500). We enabled: - spark.decommission.enabled - spark.storage.decommission.rddBlocks.enabled -