Re: Lock issue with SQLConf.getConf

2021-09-11 Thread Kohki Nishio 
Awesome, thanks!

On Sat, Sep 11, 2021 at 6:34 AM Sean Owen  wrote:

> Looks like this was improved in
> https://issues.apache.org/jira/browse/SPARK-35701 for 3.2.0
>
> On Fri, Sep 10, 2021 at 10:21 PM Kohki Nishio  wrote:
>
>> Hello,
>> I'm running spark in local mode and seeing multiple threads showing like
>> below, anybody knows why it's not using a concurrent hash map  ?
>>
>> ---
>> "request-handler-dispatcher-19785" #107532 prio=5 os_prio=0
>> tid=0x7fbd78036000 nid=0x4ebf runnable [0x7fc6e83af000]
>>java.lang.Thread.State: RUNNABLE
>> at java.util.Collections$SynchronizedMap.get(Collections.java:2586)
>> - locked <0x7fc901c7d9f8> (a java.util.Collections$SynchronizedMap)
>> at org.apache.spark.sql.internal.SQLConf.getConf(SQLConf.scala:3750)
>> at
>> org.apache.spark.sql.internal.SQLConf.constraintPropagationEnabled(SQLConf.scala:3346)
>>
>> "Executor task launch worker for task 23.0 in stage 279783.0 (TID
>> 449746)" #107929 daemon prio=5 os_prio=0 tid=0x7fbe2c005000 nid=0x55f6
>> waiting for monitor entry [0x7fc6e4037000]
>>java.lang.Thread.State: BLOCKED (on object monitor)
>> at java.util.Collections$SynchronizedMap.get(Collections.java:2586)
>> - waiting to lock <0x7fc901c7d9f8> (a
>> java.util.Collections$SynchronizedMap)
>> at org.apache.spark.sql.internal.SQLConf.getConf(SQLConf.scala:3750)
>> at
>> org.apache.spark.sql.internal.SQLConf.methodSplitThreshold(SQLConf.scala:3330)
>>
>> "Executor task launch worker for task 16.0 in stage 279883.0 (TID
>> 449728)" #107690 daemon prio=5 os_prio=0 tid=0x7fbc60056800 nid=0x53c6
>> waiting for monitor entry [0x7fc65a3a8000]
>>java.lang.Thread.State: BLOCKED (on object monitor)
>> at java.util.Collections$SynchronizedMap.get(Collections.java:2586)
>> - waiting to lock <0x7fc901c7d9f8> (a
>> java.util.Collections$SynchronizedMap)
>> at org.apache.spark.sql.internal.SQLConf.getConf(SQLConf.scala:3750)
>> at
>> org.apache.spark.sql.internal.SQLConf.planChangeLogLevel(SQLConf.scala:3160)
>> at
>> org.apache.spark.sql.catalyst.rules.PlanChangeLogger.(RuleExecutor.scala:49)
>>
>> ---
>>
>>
>> --
>> Kohki Nishio
>>
>

-- 
Kohki Nishio

Re: Lock issue with SQLConf.getConf

2021-09-11 Thread Sean Owen 
Looks like this was improved in
https://issues.apache.org/jira/browse/SPARK-35701 for 3.2.0

On Fri, Sep 10, 2021 at 10:21 PM Kohki Nishio  wrote:

> Hello,
> I'm running spark in local mode and seeing multiple threads showing like
> below, anybody knows why it's not using a concurrent hash map  ?
>
> ---
> "request-handler-dispatcher-19785" #107532 prio=5 os_prio=0
> tid=0x7fbd78036000 nid=0x4ebf runnable [0x7fc6e83af000]
>java.lang.Thread.State: RUNNABLE
> at java.util.Collections$SynchronizedMap.get(Collections.java:2586)
> - locked <0x7fc901c7d9f8> (a java.util.Collections$SynchronizedMap)
> at org.apache.spark.sql.internal.SQLConf.getConf(SQLConf.scala:3750)
> at
> org.apache.spark.sql.internal.SQLConf.constraintPropagationEnabled(SQLConf.scala:3346)
>
> "Executor task launch worker for task 23.0 in stage 279783.0 (TID 449746)"
> #107929 daemon prio=5 os_prio=0 tid=0x7fbe2c005000 nid=0x55f6 waiting
> for monitor entry [0x7fc6e4037000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at java.util.Collections$SynchronizedMap.get(Collections.java:2586)
> - waiting to lock <0x7fc901c7d9f8> (a
> java.util.Collections$SynchronizedMap)
> at org.apache.spark.sql.internal.SQLConf.getConf(SQLConf.scala:3750)
> at
> org.apache.spark.sql.internal.SQLConf.methodSplitThreshold(SQLConf.scala:3330)
>
> "Executor task launch worker for task 16.0 in stage 279883.0 (TID 449728)"
> #107690 daemon prio=5 os_prio=0 tid=0x7fbc60056800 nid=0x53c6 waiting
> for monitor entry [0x7fc65a3a8000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at java.util.Collections$SynchronizedMap.get(Collections.java:2586)
> - waiting to lock <0x7fc901c7d9f8> (a
> java.util.Collections$SynchronizedMap)
> at org.apache.spark.sql.internal.SQLConf.getConf(SQLConf.scala:3750)
> at
> org.apache.spark.sql.internal.SQLConf.planChangeLogLevel(SQLConf.scala:3160)
> at
> org.apache.spark.sql.catalyst.rules.PlanChangeLogger.(RuleExecutor.scala:49)
>
> ---
>
>
> --
> Kohki Nishio
>

Lock issue with SQLConf.getConf

2021-09-10 Thread Kohki Nishio 
Hello,
I'm running spark in local mode and seeing multiple threads showing like
below, anybody knows why it's not using a concurrent hash map  ?

---
"request-handler-dispatcher-19785" #107532 prio=5 os_prio=0
tid=0x7fbd78036000 nid=0x4ebf runnable [0x7fc6e83af000]
   java.lang.Thread.State: RUNNABLE
at java.util.Collections$SynchronizedMap.get(Collections.java:2586)
- locked <0x7fc901c7d9f8> (a java.util.Collections$SynchronizedMap)
at org.apache.spark.sql.internal.SQLConf.getConf(SQLConf.scala:3750)
at
org.apache.spark.sql.internal.SQLConf.constraintPropagationEnabled(SQLConf.scala:3346)

"Executor task launch worker for task 23.0 in stage 279783.0 (TID 449746)"
#107929 daemon prio=5 os_prio=0 tid=0x7fbe2c005000 nid=0x55f6 waiting
for monitor entry [0x7fc6e4037000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at java.util.Collections$SynchronizedMap.get(Collections.java:2586)
- waiting to lock <0x7fc901c7d9f8> (a
java.util.Collections$SynchronizedMap)
at org.apache.spark.sql.internal.SQLConf.getConf(SQLConf.scala:3750)
at
org.apache.spark.sql.internal.SQLConf.methodSplitThreshold(SQLConf.scala:3330)

"Executor task launch worker for task 16.0 in stage 279883.0 (TID 449728)"
#107690 daemon prio=5 os_prio=0 tid=0x7fbc60056800 nid=0x53c6 waiting
for monitor entry [0x7fc65a3a8000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at java.util.Collections$SynchronizedMap.get(Collections.java:2586)
- waiting to lock <0x7fc901c7d9f8> (a
java.util.Collections$SynchronizedMap)
at org.apache.spark.sql.internal.SQLConf.getConf(SQLConf.scala:3750)
at
org.apache.spark.sql.internal.SQLConf.planChangeLogLevel(SQLConf.scala:3160)
at
org.apache.spark.sql.catalyst.rules.PlanChangeLogger.(RuleExecutor.scala:49)

---


-- 
Kohki Nishio