Re: Spark Issue with Istio in Distributed Mode

Was able to resolve the idle connections being terminated issue using EnvoyFilter On Sat, 3 Sept 2022 at 18:14, Ilan Filonenko wrote: > Must be set in envoy (maybe could passthrough via istio) > > https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/core/v3/protocol.proto#envoy-v3-api-field

RE: [EXTERNAL] Re: Dynamic shuffle partitions in a single job

AFAIK there is no way to set it at each Exchange. If you are using Spark 3.2+ AQE Performance Tuning - Spark 3.3.0 Documentation (apache.org) is enabled by default, which can automatically changes post shuf

Long running task in spark

Hello Users, My 2 tasks are running forever. One of them gave a java heap space error. I have 10 Joins , all tables are big. I understand this is data skewness. Apart from changes at code level , any property which can be used in Spark Config? I am using Spark2 hence AQE can not be used. Thank

[SPARK STRUCTURED STREAMING] : Rocks DB uses off-heap usage

Hi Team, We are trying to shift from HDFS State Manager to Rocks DB State Manager, but while doing POC we realised it is using much more off-heap space than expected. Because of this, the executors get killed with : *out of** physical memory exception.* Could you please help in understanding, wh

Re: Pipelined execution in Spark (???)

Hi, for some tasks as repartitionbyrange, it is indeed quite annoying sometimes to wait for the maps to complete before reduce starts. @Sean Owen do you have any comments? Regards, Gourav Sengupta On Thu, Sep 8, 2022 at 12:10 AM Russell Jurney wrote: > I could be wrong , but… just start it.