Re: Spark Issue with Istio in Distributed Mode

2022-09-11 Thread Deepak Sharma
Was able to resolve the idle connections being terminated issue using EnvoyFilter On Sat, 3 Sept 2022 at 18:14, Ilan Filonenko wrote: > Must be set in envoy (maybe could passthrough via istio) > > https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/core/v3/protocol.proto#envoy-v3-api-field

RE: [EXTERNAL] Re: Dynamic shuffle partitions in a single job

2022-09-11 Thread Kapil Kumar Singh
AFAIK there is no way to set it at each Exchange. If you are using Spark 3.2+ AQE Performance Tuning - Spark 3.3.0 Documentation (apache.org) is enabled by default, which can automatically changes post shuf

Long running task in spark

2022-09-11 Thread rajat kumar
Hello Users, My 2 tasks are running forever. One of them gave a java heap space error. I have 10 Joins , all tables are big. I understand this is data skewness. Apart from changes at code level , any property which can be used in Spark Config? I am using Spark2 hence AQE can not be used. Thank

[SPARK STRUCTURED STREAMING] : Rocks DB uses off-heap usage

2022-09-11 Thread akshit marwah
Hi Team, We are trying to shift from HDFS State Manager to Rocks DB State Manager, but while doing POC we realised it is using much more off-heap space than expected. Because of this, the executors get killed with : *out of** physical memory exception.* Could you please help in understanding, wh

Re: Pipelined execution in Spark (???)

2022-09-11 Thread Gourav Sengupta
Hi, for some tasks as repartitionbyrange, it is indeed quite annoying sometimes to wait for the maps to complete before reduce starts. @Sean Owen do you have any comments? Regards, Gourav Sengupta On Thu, Sep 8, 2022 at 12:10 AM Russell Jurney wrote: > I could be wrong , but… just start it.