Re: Executor heartbeats on Kubernetes

Hi Kristopher I believe that heartbeat between driver and executors works traditionally on k8s. spark.kubernetes.executor.missingPodDetectDelta works on executor lifecycle. When a registered executor’s POD is missing from the Kubernetes API server’s polled list of PODs then this delta time is tak

Executor heartbeats on Kubernetes

Due to settings like, "spark.kubernetes.executor.missingPodDetectDelta" I've begun to wonder about heartbeats on Kubernetes. Do executors still conduct the traditional heartbeat to the driver when run on Kubernetes? Thanks, Kris --

Re: Efficiently updating running sums only on new data

You can tag the last entry with each key using the same window you're using for your rolling sum. Something like this: "LEAD(1) OVER your_window IS NULL as last_record". Then, you just UNION ALL the last entry of each key(which you tagged) with the new data and run the same query over the windowed