Re: Connection reset by peer : failed to remove cache rdd

On 2021/08/30 13:32:19, Jacek Laskowski wrote: > Hi, > > No idea what might be going on here, but I'd not worry much about it and > simply monitor disk usage as some broadcast blocks might have left over. > > Do you know when in your application lifecycle it happens? Spark SQL or >

Who can reply me about SPARK-36584

Hi All I recently raised an issue to the community about Spark's dynamic scaling. When there is a broadcast and dynamic scaling is enabled at the same time, the driver will send a BlockUpdate message when the broadcast is completed. ExecutorMonitor#OnBlockUpdate will receive this event and

question regarding spark streaming continuous processing

Hi, Hi all, I have a couple questions regarding continuous processing: 1. What is the plan for continuous processing moving forward? Will this eventually be released as a production feature as it seems it is still experimental? 2. In microbatch streaming, there is a StreamingQueryListener

[Spark JDBC] Spark jdbc on SQL server failed with filter and disable pushDownPredicate not work

Hi guys, I recently met with an error with 'JDBC_PUSHDOWN_PREDICATE' option not work. The background is val url = "jdbc:sqlserver://XX" val properties = new Properties val df = spark.read.jdbc(url, "movies", properties) df.filter("rated == true").show() I am using this code to read from

Spark usage help

Hi: I found that the following methods are used when setting parameters to create a sparksession access hive table 1) hive.execution.engine:spark spark = SparkSession.builder() .appName("get data from hive") .config("hive.execution.engine", "spark") .enableHiveSupport() .getOrCreate()