[jira] [Issue Comment Deleted] (SPARK-29322) History server is stuck reading incomplete event log file compressed with zstd
[
https://issues.apache.org/jira/browse/SPARK-29322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated SPARK-29322:
-
Comment: was deleted
(was: For event log, we seem to still use "com.github.luben:zstd-jni" unless
specified manually.
https://github.com/apache/spark/blob/3b1674cb1f244598463e879477d89632b0817578/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala#L74-L79
https://github.com/apache/spark/blob/3b1674cb1f244598463e879477d89632b0817578/core/src/main/scala/org/apache/spark/io/CompressionCodec.scala#L54-L67
https://github.com/apache/spark/blob/3b1674cb1f244598463e879477d89632b0817578/core/src/main/scala/org/apache/spark/io/CompressionCodec.scala#L200-L236)
> History server is stuck reading incomplete event log file compressed with zstd
> --
>
> Key: SPARK-29322
> URL: https://issues.apache.org/jira/browse/SPARK-29322
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 3.0.0
>Reporter: Jungtaek Lim
>Priority: Major
> Attachments: history-server-1.jstack, history-server-2.jstack,
> history-server-3.jstack, history-server-4.jstack
>
>
> While working on SPARK-28869, I've discovered the issue that reading
> inprogress event log file on zstd compression could lead the thread being
> stuck. I just experimented with Spark History Server and observed same issue.
> I'll attach the jstack files.
> This is very easy to reproduce: setting configuration as below
> - spark.eventLog.enabled=true
> - spark.eventLog.compress=true
> - spark.eventLog.compression.codec=zstd
> and start Spark application. While the application is running, load the
> application in SHS webpage. It may succeed to replay the event log, but high
> likely it will be stuck and loading page will be also stuck.
> Only listing the thread stack trace being stuck across jstack files:
> {code}
> 2019-10-02 11:32:36
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.191-b12 mixed mode):
> ...
> "qtp2072313080-30" #30 daemon prio=5 os_prio=31 tid=0x7ff5b90e7800
> nid=0x9703 runnable [0x7f22]
>java.lang.Thread.State: RUNNABLE
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:255)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.read(RawLocalFileSystem.java:156)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5f97c60> (a
> org.apache.hadoop.fs.BufferedFSInputStream)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at
> org.apache.hadoop.fs.FSInputChecker.readFully(FSInputChecker.java:436)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.readChunk(ChecksumFileSystem.java:257)
> at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:276)
> at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:228)
> at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:196)
> - locked <0x0007b5f97b58> (a
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5f97af8> (a java.io.BufferedInputStream)
> at
> com.github.luben.zstd.ZstdInputStream.readInternal(ZstdInputStream.java:129)
> at com.github.luben.zstd.ZstdInputStream.read(ZstdInputStream.java:107)
> - locked <0x0007b5f97ac0> (a com.github.luben.zstd.ZstdInputStream)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5cd3bd0> (a java.io.BufferedInputStream)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
> - locked <0x0007b5f94a00> (a java.io.InputStreamReader)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:161)
> at java.io.BufferedReader.readLine(BufferedReader.java:324)
> - locked <0x0007b5f94a00> (a java.io.InputStreamReader)
> at java.io.BufferedReader.readLine(BufferedReader.java:389)
> at
> scala.io.BufferedSource$BufferedLineIterator.hasNext(BufferedSource.scala:74)
>
[jira] [Commented] (SPARK-29322) History server is stuck reading incomplete event log file compressed with zstd
[
https://issues.apache.org/jira/browse/SPARK-29322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942508#comment-16942508
]
Jungtaek Lim commented on SPARK-29322:
--
For event log, we seem to still use "com.github.luben:zstd-jni" unless
specified manually.
https://github.com/apache/spark/blob/3b1674cb1f244598463e879477d89632b0817578/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala#L74-L79
https://github.com/apache/spark/blob/3b1674cb1f244598463e879477d89632b0817578/core/src/main/scala/org/apache/spark/io/CompressionCodec.scala#L54-L67
https://github.com/apache/spark/blob/3b1674cb1f244598463e879477d89632b0817578/core/src/main/scala/org/apache/spark/io/CompressionCodec.scala#L200-L236
> History server is stuck reading incomplete event log file compressed with zstd
> --
>
> Key: SPARK-29322
> URL: https://issues.apache.org/jira/browse/SPARK-29322
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 3.0.0
>Reporter: Jungtaek Lim
>Priority: Major
> Attachments: history-server-1.jstack, history-server-2.jstack,
> history-server-3.jstack, history-server-4.jstack
>
>
> While working on SPARK-28869, I've discovered the issue that reading
> inprogress event log file on zstd compression could lead the thread being
> stuck. I just experimented with Spark History Server and observed same issue.
> I'll attach the jstack files.
> This is very easy to reproduce: setting configuration as below
> - spark.eventLog.enabled=true
> - spark.eventLog.compress=true
> - spark.eventLog.compression.codec=zstd
> and start Spark application. While the application is running, load the
> application in SHS webpage. It may succeed to replay the event log, but high
> likely it will be stuck and loading page will be also stuck.
> Only listing the thread stack trace being stuck across jstack files:
> {code}
> 2019-10-02 11:32:36
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.191-b12 mixed mode):
> ...
> "qtp2072313080-30" #30 daemon prio=5 os_prio=31 tid=0x7ff5b90e7800
> nid=0x9703 runnable [0x7f22]
>java.lang.Thread.State: RUNNABLE
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:255)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.read(RawLocalFileSystem.java:156)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5f97c60> (a
> org.apache.hadoop.fs.BufferedFSInputStream)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at
> org.apache.hadoop.fs.FSInputChecker.readFully(FSInputChecker.java:436)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.readChunk(ChecksumFileSystem.java:257)
> at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:276)
> at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:228)
> at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:196)
> - locked <0x0007b5f97b58> (a
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5f97af8> (a java.io.BufferedInputStream)
> at
> com.github.luben.zstd.ZstdInputStream.readInternal(ZstdInputStream.java:129)
> at com.github.luben.zstd.ZstdInputStream.read(ZstdInputStream.java:107)
> - locked <0x0007b5f97ac0> (a com.github.luben.zstd.ZstdInputStream)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5cd3bd0> (a java.io.BufferedInputStream)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
> - locked <0x0007b5f94a00> (a java.io.InputStreamReader)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:161)
> at java.io.BufferedReader.readLine(BufferedReader.java:324)
> - locked <0x0007b5f94a00> (a java.io.InputStreamReader)
> at java.io.BufferedReader.readLine(BufferedReader.java:389)
> at
>
[jira] [Comment Edited] (SPARK-29322) History server is stuck reading incomplete event log file compressed with zstd
[
https://issues.apache.org/jira/browse/SPARK-29322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942502#comment-16942502
]
Dongjoon Hyun edited comment on SPARK-29322 at 10/2/19 5:43 AM:
-AFAIK, we are using Hadoop ZStandardCodec which is added by HADOOP-13578 at
2.9.0 and 3.0.0.- My bad. I checked the code path. You're right. I was confused
with Parquet in SQL module.
was (Author: dongjoon):
-AFAIK, we are using Hadoop ZStandardCodec which is added by HADOOP-13578 at
2.9.0 and 3.0.0.- My bad. I checked the code path. You're right.
> History server is stuck reading incomplete event log file compressed with zstd
> --
>
> Key: SPARK-29322
> URL: https://issues.apache.org/jira/browse/SPARK-29322
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 3.0.0
>Reporter: Jungtaek Lim
>Priority: Major
> Attachments: history-server-1.jstack, history-server-2.jstack,
> history-server-3.jstack, history-server-4.jstack
>
>
> While working on SPARK-28869, I've discovered the issue that reading
> inprogress event log file on zstd compression could lead the thread being
> stuck. I just experimented with Spark History Server and observed same issue.
> I'll attach the jstack files.
> This is very easy to reproduce: setting configuration as below
> - spark.eventLog.enabled=true
> - spark.eventLog.compress=true
> - spark.eventLog.compression.codec=zstd
> and start Spark application. While the application is running, load the
> application in SHS webpage. It may succeed to replay the event log, but high
> likely it will be stuck and loading page will be also stuck.
> Only listing the thread stack trace being stuck across jstack files:
> {code}
> 2019-10-02 11:32:36
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.191-b12 mixed mode):
> ...
> "qtp2072313080-30" #30 daemon prio=5 os_prio=31 tid=0x7ff5b90e7800
> nid=0x9703 runnable [0x7f22]
>java.lang.Thread.State: RUNNABLE
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:255)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.read(RawLocalFileSystem.java:156)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5f97c60> (a
> org.apache.hadoop.fs.BufferedFSInputStream)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at
> org.apache.hadoop.fs.FSInputChecker.readFully(FSInputChecker.java:436)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.readChunk(ChecksumFileSystem.java:257)
> at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:276)
> at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:228)
> at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:196)
> - locked <0x0007b5f97b58> (a
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5f97af8> (a java.io.BufferedInputStream)
> at
> com.github.luben.zstd.ZstdInputStream.readInternal(ZstdInputStream.java:129)
> at com.github.luben.zstd.ZstdInputStream.read(ZstdInputStream.java:107)
> - locked <0x0007b5f97ac0> (a com.github.luben.zstd.ZstdInputStream)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5cd3bd0> (a java.io.BufferedInputStream)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
> - locked <0x0007b5f94a00> (a java.io.InputStreamReader)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:161)
> at java.io.BufferedReader.readLine(BufferedReader.java:324)
> - locked <0x0007b5f94a00> (a java.io.InputStreamReader)
> at java.io.BufferedReader.readLine(BufferedReader.java:389)
> at
> scala.io.BufferedSource$BufferedLineIterator.hasNext(BufferedSource.scala:74)
> at scala.collection.Iterator$$anon$20.hasNext(Iterator.scala:884)
> at
[jira] [Comment Edited] (SPARK-29322) History server is stuck reading incomplete event log file compressed with zstd
[
https://issues.apache.org/jira/browse/SPARK-29322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942502#comment-16942502
]
Dongjoon Hyun edited comment on SPARK-29322 at 10/2/19 5:42 AM:
-AFAIK, we are using Hadoop ZStandardCodec which is added by HADOOP-13578 at
2.9.0 and 3.0.0.- My bad. I checked the code path. You're right.
was (Author: dongjoon):
~AFAIK, we are using Hadoop ZStandardCodec which is added by HADOOP-13578 at
2.9.0 and 3.0.0.~ My bad. I checked the code path. You're right.
> History server is stuck reading incomplete event log file compressed with zstd
> --
>
> Key: SPARK-29322
> URL: https://issues.apache.org/jira/browse/SPARK-29322
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 3.0.0
>Reporter: Jungtaek Lim
>Priority: Major
> Attachments: history-server-1.jstack, history-server-2.jstack,
> history-server-3.jstack, history-server-4.jstack
>
>
> While working on SPARK-28869, I've discovered the issue that reading
> inprogress event log file on zstd compression could lead the thread being
> stuck. I just experimented with Spark History Server and observed same issue.
> I'll attach the jstack files.
> This is very easy to reproduce: setting configuration as below
> - spark.eventLog.enabled=true
> - spark.eventLog.compress=true
> - spark.eventLog.compression.codec=zstd
> and start Spark application. While the application is running, load the
> application in SHS webpage. It may succeed to replay the event log, but high
> likely it will be stuck and loading page will be also stuck.
> Only listing the thread stack trace being stuck across jstack files:
> {code}
> 2019-10-02 11:32:36
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.191-b12 mixed mode):
> ...
> "qtp2072313080-30" #30 daemon prio=5 os_prio=31 tid=0x7ff5b90e7800
> nid=0x9703 runnable [0x7f22]
>java.lang.Thread.State: RUNNABLE
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:255)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.read(RawLocalFileSystem.java:156)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5f97c60> (a
> org.apache.hadoop.fs.BufferedFSInputStream)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at
> org.apache.hadoop.fs.FSInputChecker.readFully(FSInputChecker.java:436)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.readChunk(ChecksumFileSystem.java:257)
> at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:276)
> at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:228)
> at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:196)
> - locked <0x0007b5f97b58> (a
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5f97af8> (a java.io.BufferedInputStream)
> at
> com.github.luben.zstd.ZstdInputStream.readInternal(ZstdInputStream.java:129)
> at com.github.luben.zstd.ZstdInputStream.read(ZstdInputStream.java:107)
> - locked <0x0007b5f97ac0> (a com.github.luben.zstd.ZstdInputStream)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5cd3bd0> (a java.io.BufferedInputStream)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
> - locked <0x0007b5f94a00> (a java.io.InputStreamReader)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:161)
> at java.io.BufferedReader.readLine(BufferedReader.java:324)
> - locked <0x0007b5f94a00> (a java.io.InputStreamReader)
> at java.io.BufferedReader.readLine(BufferedReader.java:389)
> at
> scala.io.BufferedSource$BufferedLineIterator.hasNext(BufferedSource.scala:74)
> at scala.collection.Iterator$$anon$20.hasNext(Iterator.scala:884)
> at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:511)
> at
>
[jira] [Comment Edited] (SPARK-29322) History server is stuck reading incomplete event log file compressed with zstd
[
https://issues.apache.org/jira/browse/SPARK-29322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942502#comment-16942502
]
Dongjoon Hyun edited comment on SPARK-29322 at 10/2/19 5:42 AM:
~AFAIK, we are using Hadoop ZStandardCodec which is added by HADOOP-13578 at
2.9.0 and 3.0.0.~ My bad. I checked the code path. You're right.
was (Author: dongjoon):
AFAIK, we are using Hadoop ZStandardCodec which is added by HADOOP-13578 at
2.9.0 and 3.0.0.
> History server is stuck reading incomplete event log file compressed with zstd
> --
>
> Key: SPARK-29322
> URL: https://issues.apache.org/jira/browse/SPARK-29322
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 3.0.0
>Reporter: Jungtaek Lim
>Priority: Major
> Attachments: history-server-1.jstack, history-server-2.jstack,
> history-server-3.jstack, history-server-4.jstack
>
>
> While working on SPARK-28869, I've discovered the issue that reading
> inprogress event log file on zstd compression could lead the thread being
> stuck. I just experimented with Spark History Server and observed same issue.
> I'll attach the jstack files.
> This is very easy to reproduce: setting configuration as below
> - spark.eventLog.enabled=true
> - spark.eventLog.compress=true
> - spark.eventLog.compression.codec=zstd
> and start Spark application. While the application is running, load the
> application in SHS webpage. It may succeed to replay the event log, but high
> likely it will be stuck and loading page will be also stuck.
> Only listing the thread stack trace being stuck across jstack files:
> {code}
> 2019-10-02 11:32:36
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.191-b12 mixed mode):
> ...
> "qtp2072313080-30" #30 daemon prio=5 os_prio=31 tid=0x7ff5b90e7800
> nid=0x9703 runnable [0x7f22]
>java.lang.Thread.State: RUNNABLE
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:255)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.read(RawLocalFileSystem.java:156)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5f97c60> (a
> org.apache.hadoop.fs.BufferedFSInputStream)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at
> org.apache.hadoop.fs.FSInputChecker.readFully(FSInputChecker.java:436)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.readChunk(ChecksumFileSystem.java:257)
> at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:276)
> at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:228)
> at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:196)
> - locked <0x0007b5f97b58> (a
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5f97af8> (a java.io.BufferedInputStream)
> at
> com.github.luben.zstd.ZstdInputStream.readInternal(ZstdInputStream.java:129)
> at com.github.luben.zstd.ZstdInputStream.read(ZstdInputStream.java:107)
> - locked <0x0007b5f97ac0> (a com.github.luben.zstd.ZstdInputStream)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5cd3bd0> (a java.io.BufferedInputStream)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
> - locked <0x0007b5f94a00> (a java.io.InputStreamReader)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:161)
> at java.io.BufferedReader.readLine(BufferedReader.java:324)
> - locked <0x0007b5f94a00> (a java.io.InputStreamReader)
> at java.io.BufferedReader.readLine(BufferedReader.java:389)
> at
> scala.io.BufferedSource$BufferedLineIterator.hasNext(BufferedSource.scala:74)
> at scala.collection.Iterator$$anon$20.hasNext(Iterator.scala:884)
> at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:511)
> at
>
[jira] [Updated] (SPARK-29325) approxQuantile() results are incorrect and vary significantly for small changes in relativeError
[
https://issues.apache.org/jira/browse/SPARK-29325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Verbus updated SPARK-29325:
-
Description:
The [approxQuantile()
method|https://github.com/apache/spark/blob/3b1674cb1f244598463e879477d89632b0817578/sql/core/src/main/scala/org/apache/spark/sql/DataFrameStatFunctions.scala#L40]
returns sometimes incorrect results that are sensitively dependent upon the
choice of the relativeError.
Below is an example in the latest Spark version (2.4.4). You can see the result
varies significantly for modest changes in the specified relativeError
parameter. The result varies much more than would be expected based upon the
relativeError parameter.
{code:java}
Welcome to
__
/ __/__ ___ _/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.4
/_/
Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_212)
Type in expressions to have them evaluated.
Type :help for more information.
scala> val df = spark.read.format("csv").option("header",
"true").option("inferSchema",
"true").load("./20191001_example_data_approx_quantile_bug")
df: org.apache.spark.sql.DataFrame = [value: double]
scala> df.stat.approxQuantile("value", Array(0.9), 0)
res0: Array[Double] = Array(0.5929591082174609)
scala> df.stat.approxQuantile("value", Array(0.9), 0.001)
res1: Array[Double] = Array(0.67621027121925)
scala> df.stat.approxQuantile("value", Array(0.9), 0.002)
res2: Array[Double] = Array(0.5926195654486178)
scala> df.stat.approxQuantile("value", Array(0.9), 0.003)
res3: Array[Double] = Array(0.5924693999048418)
scala> df.stat.approxQuantile("value", Array(0.9), 0.004)
res4: Array[Double] = Array(0.67621027121925)
scala> df.stat.approxQuantile("value", Array(0.9), 0.005)
res5: Array[Double] = Array(0.5923925937051544)
{code}
I attached a zip file containing the data used for the above example
demonstrating the bug.
Also, the following demonstrates that there is data for intermediate quantile
values between the 0.5926195654486178 and 0.67621027121925 values observed
above.
{code:java}
scala> df.stat.approxQuantile("value", Array(0.9), 0.0)
res10: Array[Double] = Array(0.5929591082174609)
scala> df.stat.approxQuantile("value", Array(0.91), 0.0)
res11: Array[Double] = Array(0.5966354540849995)
scala> df.stat.approxQuantile("value", Array(0.92), 0.0)
res12: Array[Double] = Array(0.6015676591185595)
scala> df.stat.approxQuantile("value", Array(0.93), 0.0)
res13: Array[Double] = Array(0.6029240823799614)
scala> df.stat.approxQuantile("value", Array(0.94), 0.0)
res14: Array[Double] = Array(0.611764547134)
scala> df.stat.approxQuantile("value", Array(0.95), 0.0)
res15: Array[Double] = Array(0.6185162204274052)
scala> df.stat.approxQuantile("value", Array(0.96), 0.0)
res16: Array[Double] = Array(0.625983000807062)
scala> df.stat.approxQuantile("value", Array(0.97), 0.0)
res17: Array[Double] = Array(0.6306892943412258)
scala> df.stat.approxQuantile("value", Array(0.98), 0.0)
res18: Array[Double] = Array(0.6365567375994333)
scala> df.stat.approxQuantile("value", Array(0.99), 0.0)
res19: Array[Double] = Array(0.6554479197566019)
{code}
was:
The [approxQuantile()
method|https://github.com/apache/spark/blob/3b1674cb1f244598463e879477d89632b0817578/sql/core/src/main/scala/org/apache/spark/sql/DataFrameStatFunctions.scala#L40]
returns sometimes incorrect results that are sensitively dependent upon the
choice of the relativeError.
Below is an example in the latest Spark version (2.4.4). You can see the result
varies significantly for modest changes in the specified relativeError
parameter. The result varies much more than would be expected based upon the
relativeError parameter.
{code:java}
Welcome to
__
/ __/__ ___ _/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.4
/_/
Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_212)
Type in expressions to have them evaluated.
Type :help for more information.
scala> val df = spark.read.format("csv").option("header",
"true").option("inferSchema",
"true").load("./20191001_example_data_approx_quantile_bug")
df: org.apache.spark.sql.DataFrame = [value: double]
scala> df.stat.approxQuantile("value", Array(0.9), 0)
res0: Array[Double] = Array(0.5929591082174609)
scala> df.stat.approxQuantile("value", Array(0.9), 0.001)
res1: Array[Double] = Array(0.67621027121925)
scala> df.stat.approxQuantile("value", Array(0.9), 0.002)
res2: Array[Double] = Array(0.5926195654486178)
scala> df.stat.approxQuantile("value", Array(0.9), 0.003)
res3: Array[Double] = Array(0.5924693999048418)
scala> df.stat.approxQuantile("value", Array(0.9), 0.004)
res4: Array[Double] = Array(0.67621027121925)
scala> df.stat.approxQuantile("value", Array(0.9), 0.005)
res5:
[jira] [Commented] (SPARK-29322) History server is stuck reading incomplete event log file compressed with zstd
[
https://issues.apache.org/jira/browse/SPARK-29322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942502#comment-16942502
]
Dongjoon Hyun commented on SPARK-29322:
---
AFAIK, we are using Hadoop ZStandardCodec which is added by HADOOP-13578 at
2.9.0 and 3.0.0.
> History server is stuck reading incomplete event log file compressed with zstd
> --
>
> Key: SPARK-29322
> URL: https://issues.apache.org/jira/browse/SPARK-29322
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 3.0.0
>Reporter: Jungtaek Lim
>Priority: Major
> Attachments: history-server-1.jstack, history-server-2.jstack,
> history-server-3.jstack, history-server-4.jstack
>
>
> While working on SPARK-28869, I've discovered the issue that reading
> inprogress event log file on zstd compression could lead the thread being
> stuck. I just experimented with Spark History Server and observed same issue.
> I'll attach the jstack files.
> This is very easy to reproduce: setting configuration as below
> - spark.eventLog.enabled=true
> - spark.eventLog.compress=true
> - spark.eventLog.compression.codec=zstd
> and start Spark application. While the application is running, load the
> application in SHS webpage. It may succeed to replay the event log, but high
> likely it will be stuck and loading page will be also stuck.
> Only listing the thread stack trace being stuck across jstack files:
> {code}
> 2019-10-02 11:32:36
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.191-b12 mixed mode):
> ...
> "qtp2072313080-30" #30 daemon prio=5 os_prio=31 tid=0x7ff5b90e7800
> nid=0x9703 runnable [0x7f22]
>java.lang.Thread.State: RUNNABLE
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:255)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.read(RawLocalFileSystem.java:156)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5f97c60> (a
> org.apache.hadoop.fs.BufferedFSInputStream)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at
> org.apache.hadoop.fs.FSInputChecker.readFully(FSInputChecker.java:436)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.readChunk(ChecksumFileSystem.java:257)
> at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:276)
> at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:228)
> at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:196)
> - locked <0x0007b5f97b58> (a
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5f97af8> (a java.io.BufferedInputStream)
> at
> com.github.luben.zstd.ZstdInputStream.readInternal(ZstdInputStream.java:129)
> at com.github.luben.zstd.ZstdInputStream.read(ZstdInputStream.java:107)
> - locked <0x0007b5f97ac0> (a com.github.luben.zstd.ZstdInputStream)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5cd3bd0> (a java.io.BufferedInputStream)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
> - locked <0x0007b5f94a00> (a java.io.InputStreamReader)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:161)
> at java.io.BufferedReader.readLine(BufferedReader.java:324)
> - locked <0x0007b5f94a00> (a java.io.InputStreamReader)
> at java.io.BufferedReader.readLine(BufferedReader.java:389)
> at
> scala.io.BufferedSource$BufferedLineIterator.hasNext(BufferedSource.scala:74)
> at scala.collection.Iterator$$anon$20.hasNext(Iterator.scala:884)
> at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:511)
> at
> org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:80)
> at
> org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:58)
> at
> org.apache.spark.deploy.history.FsHistoryProvider.$anonfun$rebuildAppStore$5(FsHistoryProvider.scala:976)
[jira] [Updated] (SPARK-29325) approxQuantile() results are incorrect and vary significantly for small changes in relativeError
[
https://issues.apache.org/jira/browse/SPARK-29325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Verbus updated SPARK-29325:
-
Description:
The [approxQuantile()
method|https://github.com/apache/spark/blob/3b1674cb1f244598463e879477d89632b0817578/sql/core/src/main/scala/org/apache/spark/sql/DataFrameStatFunctions.scala#L40]
returns sometimes incorrect results that are sensitively dependent upon the
choice of the relativeError.
Below is an example in the latest Spark version (2.4.4). You can see the result
varies significantly for modest changes in the specified relativeError
parameter. The result varies much more than would be expected based upon the
relativeError parameter.
{code:java}
Welcome to
__
/ __/__ ___ _/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.4
/_/
Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_212)
Type in expressions to have them evaluated.
Type :help for more information.
scala> val df = spark.read.format("csv").option("header",
"true").option("inferSchema",
"true").load("./20191001_example_data_approx_quantile_bug")
df: org.apache.spark.sql.DataFrame = [value: double]
scala> df.stat.approxQuantile("value", Array(0.9), 0)
res0: Array[Double] = Array(0.5929591082174609)
scala> df.stat.approxQuantile("value", Array(0.9), 0.001)
res1: Array[Double] = Array(0.67621027121925)
scala> df.stat.approxQuantile("value", Array(0.9), 0.002)
res2: Array[Double] = Array(0.5926195654486178)
scala> df.stat.approxQuantile("value", Array(0.9), 0.003)
res3: Array[Double] = Array(0.5924693999048418)
scala> df.stat.approxQuantile("value", Array(0.9), 0.004)
res4: Array[Double] = Array(0.67621027121925)
scala> df.stat.approxQuantile("value", Array(0.9), 0.005)
res5: Array[Double] = Array(0.5923925937051544)
{code}
I attached a zip file containing the data used for the above example
demonstrating the bug.
Also, the following demonstrates that there is data for intermediate quantile
values between the 0.5926195654486178 and 0.67621027121925 values observed
above.
{code:java}
scala> df.stat.approxQuantile("value", Array(0.9), 0.0) res10: Array[Double] =
Array(0.5929591082174609)
scala> df.stat.approxQuantile("value", Array(0.91), 0.0) res11: Array[Double] =
Array(0.5966354540849995)
scala> df.stat.approxQuantile("value", Array(0.92), 0.0) res12: Array[Double] =
Array(0.6015676591185595)
scala> df.stat.approxQuantile("value", Array(0.93), 0.0) res13: Array[Double] =
Array(0.6029240823799614)
scala> df.stat.approxQuantile("value", Array(0.94), 0.0) res14: Array[Double] =
Array(0.611764547134)
scala> df.stat.approxQuantile("value", Array(0.95), 0.0) res15: Array[Double] =
Array(0.6185162204274052)
scala> df.stat.approxQuantile("value", Array(0.96), 0.0) res16: Array[Double] =
Array(0.625983000807062)
scala> df.stat.approxQuantile("value", Array(0.97), 0.0) res17: Array[Double] =
Array(0.6306892943412258)
scala> df.stat.approxQuantile("value", Array(0.98), 0.0) res18: Array[Double] =
Array(0.6365567375994333)
scala> df.stat.approxQuantile("value", Array(0.99), 0.0) res19: Array[Double] =
Array(0.6554479197566019)
{code}
was:
The [approxQuantile()
method|https://github.com/apache/spark/blob/3b1674cb1f244598463e879477d89632b0817578/sql/core/src/main/scala/org/apache/spark/sql/DataFrameStatFunctions.scala#L40]
returns sometimes incorrect results that are sensitively dependent upon the
choice of the relativeError.
Below is an example in the latest Spark version (2.4.4). You can see the result
varies significantly for modest changes in the specified relativeError
parameter. The result varies much more than would be expected based upon the
relativeError parameter.
{code:java}
Welcome to
__
/ __/__ ___ _/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.4
/_/
Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_212)
Type in expressions to have them evaluated.
Type :help for more information.
scala> val df = spark.read.format("csv").option("header",
"true").option("inferSchema",
"true").load("./20191001_example_data_approx_quantile_bug")
df: org.apache.spark.sql.DataFrame = [value: double]
scala> df.stat.approxQuantile("value", Array(0.9), 0)
res0: Array[Double] = Array(0.5929591082174609)
scala> df.stat.approxQuantile("value", Array(0.9), 0.001)
res1: Array[Double] = Array(0.67621027121925)
scala> df.stat.approxQuantile("value", Array(0.9), 0.002)
res2: Array[Double] = Array(0.5926195654486178)
scala> df.stat.approxQuantile("value", Array(0.9), 0.003)
res3: Array[Double] = Array(0.5924693999048418)
scala> df.stat.approxQuantile("value", Array(0.9), 0.004)
res4: Array[Double] = Array(0.67621027121925)
scala> df.stat.approxQuantile("value", Array(0.9), 0.005)
res5:
[jira] [Updated] (SPARK-29325) approxQuantile() results are incorrect and vary significantly for small changes in relativeError
[
https://issues.apache.org/jira/browse/SPARK-29325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Verbus updated SPARK-29325:
-
Description:
The [approxQuantile()
method|https://github.com/apache/spark/blob/3b1674cb1f244598463e879477d89632b0817578/sql/core/src/main/scala/org/apache/spark/sql/DataFrameStatFunctions.scala#L40]
returns sometimes incorrect results that are sensitively dependent upon the
choice of the relativeError.
Below is an example in the latest Spark version (2.4.4). You can see the result
varies significantly for modest changes in the specified relativeError
parameter. The result varies much more than would be expected based upon the
relativeError parameter.
{code:java}
Welcome to
__
/ __/__ ___ _/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.4
/_/
Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_212)
Type in expressions to have them evaluated.
Type :help for more information.
scala> val df = spark.read.format("csv").option("header",
"true").option("inferSchema",
"true").load("./20191001_example_data_approx_quantile_bug")
df: org.apache.spark.sql.DataFrame = [value: double]
scala> df.stat.approxQuantile("value", Array(0.9), 0)
res0: Array[Double] = Array(0.5929591082174609)
scala> df.stat.approxQuantile("value", Array(0.9), 0.001)
res1: Array[Double] = Array(0.67621027121925)
scala> df.stat.approxQuantile("value", Array(0.9), 0.002)
res2: Array[Double] = Array(0.5926195654486178)
scala> df.stat.approxQuantile("value", Array(0.9), 0.003)
res3: Array[Double] = Array(0.5924693999048418)
scala> df.stat.approxQuantile("value", Array(0.9), 0.004)
res4: Array[Double] = Array(0.67621027121925)
scala> df.stat.approxQuantile("value", Array(0.9), 0.005)
res5: Array[Double] = Array(0.5923925937051544)
{code}
I attached a zip file containing the data used for the above example
demonstrating the bug.
Also, the following demonstrates that there is data for intermediate quantile
values between the `0.5926195654486178` and `0.67621027121925` values observed
above.
{code:java}
scala> df.stat.approxQuantile("value", Array(0.9), 0.0) res10: Array[Double] =
Array(0.5929591082174609)
scala> df.stat.approxQuantile("value", Array(0.91), 0.0) res11: Array[Double] =
Array(0.5966354540849995)
scala> df.stat.approxQuantile("value", Array(0.92), 0.0) res12: Array[Double] =
Array(0.6015676591185595)
scala> df.stat.approxQuantile("value", Array(0.93), 0.0) res13: Array[Double] =
Array(0.6029240823799614)
scala> df.stat.approxQuantile("value", Array(0.94), 0.0) res14: Array[Double] =
Array(0.611764547134)
scala> df.stat.approxQuantile("value", Array(0.95), 0.0) res15: Array[Double] =
Array(0.6185162204274052)
scala> df.stat.approxQuantile("value", Array(0.96), 0.0) res16: Array[Double] =
Array(0.625983000807062)
scala> df.stat.approxQuantile("value", Array(0.97), 0.0) res17: Array[Double] =
Array(0.6306892943412258)
scala> df.stat.approxQuantile("value", Array(0.98), 0.0) res18: Array[Double] =
Array(0.6365567375994333)
scala> df.stat.approxQuantile("value", Array(0.99), 0.0) res19: Array[Double] =
Array(0.6554479197566019)
{code}
was:
The [approxQuantile()
method|https://github.com/apache/spark/blob/3b1674cb1f244598463e879477d89632b0817578/sql/core/src/main/scala/org/apache/spark/sql/DataFrameStatFunctions.scala#L40]
returns sometimes incorrect results that are sensitively dependent upon the
choice of the relativeError.
Below is an example in the latest Spark version (2.4.4). You can see the result
varies significantly for modest changes in the specified relativeError
parameter. The result varies much more than would be expected based upon the
relativeError parameter.
{code:java}
Welcome to
__
/ __/__ ___ _/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.4
/_/
Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_212)
Type in expressions to have them evaluated.
Type :help for more information.
scala> val df = spark.read.format("csv").option("header",
"true").option("inferSchema",
"true").load("./20191001_example_data_approx_quantile_bug")
df: org.apache.spark.sql.DataFrame = [value: double]
scala> df.stat.approxQuantile("value", Array(0.9), 0)
res0: Array[Double] = Array(0.5929591082174609)
scala> df.stat.approxQuantile("value", Array(0.9), 0.001)
res1: Array[Double] = Array(0.67621027121925)
scala> df.stat.approxQuantile("value", Array(0.9), 0.002)
res2: Array[Double] = Array(0.5926195654486178)
scala> df.stat.approxQuantile("value", Array(0.9), 0.003)
res3: Array[Double] = Array(0.5924693999048418)
scala> df.stat.approxQuantile("value", Array(0.9), 0.004)
res4: Array[Double] = Array(0.67621027121925)
scala> df.stat.approxQuantile("value", Array(0.9), 0.005)
res5:
[jira] [Updated] (SPARK-29322) History server is stuck reading incomplete event log file compressed with zstd
[
https://issues.apache.org/jira/browse/SPARK-29322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated SPARK-29322:
-
Description:
While working on SPARK-28869, I've discovered the issue that reading inprogress
event log file on zstd compression could lead the thread being stuck. I just
experimented with Spark History Server and observed same issue. I'll attach the
jstack files.
This is very easy to reproduce: setting configuration as below
- spark.eventLog.enabled=true
- spark.eventLog.compress=true
- spark.eventLog.compression.codec=zstd
and start Spark application. While the application is running, load the
application in SHS webpage. It may succeed to replay the event log, but high
likely it will be stuck and loading page will be also stuck.
Only listing the thread stack trace being stuck across jstack files:
{code}
2019-10-02 11:32:36
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.191-b12 mixed mode):
...
"qtp2072313080-30" #30 daemon prio=5 os_prio=31 tid=0x7ff5b90e7800
nid=0x9703 runnable [0x7f22]
java.lang.Thread.State: RUNNABLE
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:255)
at
org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.read(RawLocalFileSystem.java:156)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
- locked <0x0007b5f97c60> (a
org.apache.hadoop.fs.BufferedFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.fs.FSInputChecker.readFully(FSInputChecker.java:436)
at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.readChunk(ChecksumFileSystem.java:257)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:276)
at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:228)
at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:196)
- locked <0x0007b5f97b58> (a
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker)
at java.io.DataInputStream.read(DataInputStream.java:149)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
- locked <0x0007b5f97af8> (a java.io.BufferedInputStream)
at
com.github.luben.zstd.ZstdInputStream.readInternal(ZstdInputStream.java:129)
at com.github.luben.zstd.ZstdInputStream.read(ZstdInputStream.java:107)
- locked <0x0007b5f97ac0> (a com.github.luben.zstd.ZstdInputStream)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
- locked <0x0007b5cd3bd0> (a java.io.BufferedInputStream)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
- locked <0x0007b5f94a00> (a java.io.InputStreamReader)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:161)
at java.io.BufferedReader.readLine(BufferedReader.java:324)
- locked <0x0007b5f94a00> (a java.io.InputStreamReader)
at java.io.BufferedReader.readLine(BufferedReader.java:389)
at
scala.io.BufferedSource$BufferedLineIterator.hasNext(BufferedSource.scala:74)
at scala.collection.Iterator$$anon$20.hasNext(Iterator.scala:884)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:511)
at
org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:80)
at
org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:58)
at
org.apache.spark.deploy.history.FsHistoryProvider.$anonfun$rebuildAppStore$5(FsHistoryProvider.scala:976)
at
org.apache.spark.deploy.history.FsHistoryProvider.$anonfun$rebuildAppStore$5$adapted(FsHistoryProvider.scala:975)
at
org.apache.spark.deploy.history.FsHistoryProvider$$Lambda$662/1267867461.apply(Unknown
Source)
at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2567)
at
org.apache.spark.deploy.history.FsHistoryProvider.rebuildAppStore(FsHistoryProvider.scala:975)
at
org.apache.spark.deploy.history.FsHistoryProvider.createInMemoryStore(FsHistoryProvider.scala:1093)
at
org.apache.spark.deploy.history.FsHistoryProvider.getAppUI(FsHistoryProvider.scala:346)
at
org.apache.spark.deploy.history.HistoryServer.getAppUI(HistoryServer.scala:188)
at
[jira] [Updated] (SPARK-29325) approxQuantile() results are incorrect and vary significantly for small changes in relativeError
[
https://issues.apache.org/jira/browse/SPARK-29325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Verbus updated SPARK-29325:
-
Description:
The [approxQuantile()
method|https://github.com/apache/spark/blob/3b1674cb1f244598463e879477d89632b0817578/sql/core/src/main/scala/org/apache/spark/sql/DataFrameStatFunctions.scala#L40]
returns sometimes incorrect results that are sensitively dependent upon the
choice of the relativeError.
Below is an example in the latest Spark version (2.4.4). You can see the result
varies significantly for modest changes in the specified relativeError
parameter. The result varies much more than would be expected based upon the
relativeError parameter.
{code:java}
Welcome to
__
/ __/__ ___ _/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.4
/_/
Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_212)
Type in expressions to have them evaluated.
Type :help for more information.
scala> val df = spark.read.format("csv").option("header",
"true").option("inferSchema",
"true").load("./20191001_example_data_approx_quantile_bug")
df: org.apache.spark.sql.DataFrame = [value: double]
scala> df.stat.approxQuantile("value", Array(0.9), 0)
res0: Array[Double] = Array(0.5929591082174609)
scala> df.stat.approxQuantile("value", Array(0.9), 0.001)
res1: Array[Double] = Array(0.67621027121925)
scala> df.stat.approxQuantile("value", Array(0.9), 0.002)
res2: Array[Double] = Array(0.5926195654486178)
scala> df.stat.approxQuantile("value", Array(0.9), 0.003)
res3: Array[Double] = Array(0.5924693999048418)
scala> df.stat.approxQuantile("value", Array(0.9), 0.004)
res4: Array[Double] = Array(0.67621027121925)
scala> df.stat.approxQuantile("value", Array(0.9), 0.005)
res5: Array[Double] = Array(0.5923925937051544)
{code}
I attached a zip file containing the data used for the above example
demonstrating the bug.
was:
The [approxQuantile()
method|https://github.com/apache/spark/blob/3b1674cb1f244598463e879477d89632b0817578/sql/core/src/main/scala/org/apache/spark/sql/DataFrameStatFunctions.scala#L40]
returns sometimes incorrect results that are sensitively dependent upon the
choice of the relativeError.
Below is an example in the latest Spark version (2.4.4). You can see the result
varies significantly for modest changes in the specified relativeError
parameter. The result varies much more than the magnitude of the relativeError
parameter.
{code:java}
Welcome to
__
/ __/__ ___ _/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.4
/_/
Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_212)
Type in expressions to have them evaluated.
Type :help for more information.
scala> val df = spark.read.format("csv").option("header",
"true").option("inferSchema",
"true").load("./20191001_example_data_approx_quantile_bug")
df: org.apache.spark.sql.DataFrame = [value: double]
scala> df.stat.approxQuantile("value", Array(0.9), 0)
res0: Array[Double] = Array(0.5929591082174609)
scala> df.stat.approxQuantile("value", Array(0.9), 0.001)
res1: Array[Double] = Array(0.67621027121925)
scala> df.stat.approxQuantile("value", Array(0.9), 0.002)
res2: Array[Double] = Array(0.5926195654486178)
scala> df.stat.approxQuantile("value", Array(0.9), 0.003)
res3: Array[Double] = Array(0.5924693999048418)
scala> df.stat.approxQuantile("value", Array(0.9), 0.004)
res4: Array[Double] = Array(0.67621027121925)
scala> df.stat.approxQuantile("value", Array(0.9), 0.005)
res5: Array[Double] = Array(0.5923925937051544)
{code}
I attached a zip file containing the data used for the above example
demonstrating the bug.
> approxQuantile() results are incorrect and vary significantly for small
> changes in relativeError
>
>
> Key: SPARK-29325
> URL: https://issues.apache.org/jira/browse/SPARK-29325
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.3.2, 2.4.4
> Environment: I was using OSX 10.14.6.
> I was using Scala 2.11.12 and Spark 2.4.4.
> I also verified the bug exists for Scala 2.11.8 and Spark 2.3.2.
>Reporter: James Verbus
>Priority: Major
> Labels: correctness
> Attachments: 20191001_example_data_approx_quantile_bug.zip
>
>
> The [approxQuantile()
> method|https://github.com/apache/spark/blob/3b1674cb1f244598463e879477d89632b0817578/sql/core/src/main/scala/org/apache/spark/sql/DataFrameStatFunctions.scala#L40]
> returns sometimes incorrect results that are sensitively dependent upon the
> choice of the relativeError.
> Below is an example in the latest Spark version (2.4.4).
[jira] [Commented] (SPARK-29322) History server is stuck reading incomplete event log file compressed with zstd
[
https://issues.apache.org/jira/browse/SPARK-29322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942493#comment-16942493
]
Jungtaek Lim commented on SPARK-29322:
--
{quote}
Since this is ZSTD, you are using `hadoop-3.2` profile on the master branch
with Hadoop 3.2.0, right?
{quote}
Spark 2.x also has support for zstd, as well as I have been working without
enabling Hadoop-3.2 for SPARK-28869 so if I'm not missing anything, I don't
think it's related to `hadoop-3.2` profile.
{quote}
please update the reproducible procedure into the JIRA description.
{quote}
Thanks for guiding! I'll update the description accordingly.
> History server is stuck reading incomplete event log file compressed with zstd
> --
>
> Key: SPARK-29322
> URL: https://issues.apache.org/jira/browse/SPARK-29322
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 3.0.0
>Reporter: Jungtaek Lim
>Priority: Major
> Attachments: history-server-1.jstack, history-server-2.jstack,
> history-server-3.jstack, history-server-4.jstack
>
>
> While working on SPARK-28869, I've discovered the issue that reading
> inprogress event log file on zstd compression could lead the thread being
> stuck. I just experimented with Spark History Server and observed same issue.
> I'll attach the jstack files.
> Only listing the thread stack trace being stuck across jstack files:
> {code}
> 2019-10-02 11:32:36
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.191-b12 mixed mode):
> ...
> "qtp2072313080-30" #30 daemon prio=5 os_prio=31 tid=0x7ff5b90e7800
> nid=0x9703 runnable [0x7f22]
>java.lang.Thread.State: RUNNABLE
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:255)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.read(RawLocalFileSystem.java:156)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5f97c60> (a
> org.apache.hadoop.fs.BufferedFSInputStream)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at
> org.apache.hadoop.fs.FSInputChecker.readFully(FSInputChecker.java:436)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.readChunk(ChecksumFileSystem.java:257)
> at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:276)
> at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:228)
> at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:196)
> - locked <0x0007b5f97b58> (a
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5f97af8> (a java.io.BufferedInputStream)
> at
> com.github.luben.zstd.ZstdInputStream.readInternal(ZstdInputStream.java:129)
> at com.github.luben.zstd.ZstdInputStream.read(ZstdInputStream.java:107)
> - locked <0x0007b5f97ac0> (a com.github.luben.zstd.ZstdInputStream)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5cd3bd0> (a java.io.BufferedInputStream)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
> - locked <0x0007b5f94a00> (a java.io.InputStreamReader)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:161)
> at java.io.BufferedReader.readLine(BufferedReader.java:324)
> - locked <0x0007b5f94a00> (a java.io.InputStreamReader)
> at java.io.BufferedReader.readLine(BufferedReader.java:389)
> at
> scala.io.BufferedSource$BufferedLineIterator.hasNext(BufferedSource.scala:74)
> at scala.collection.Iterator$$anon$20.hasNext(Iterator.scala:884)
> at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:511)
> at
> org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:80)
> at
> org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:58)
> at
> org.apache.spark.deploy.history.FsHistoryProvider.$anonfun$rebuildAppStore$5(FsHistoryProvider.scala:976)
> at
>
[jira] [Updated] (SPARK-29325) approxQuantile() results are incorrect and vary significantly for small changes in relativeError
[
https://issues.apache.org/jira/browse/SPARK-29325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Verbus updated SPARK-29325:
-
Description:
The [approxQuantile()
method|https://github.com/apache/spark/blob/3b1674cb1f244598463e879477d89632b0817578/sql/core/src/main/scala/org/apache/spark/sql/DataFrameStatFunctions.scala#L40]
returns sometimes incorrect results that are sensitively dependent upon the
choice of the relativeError.
Below is an example in the latest Spark version (2.4.4). You can see the result
varies significantly for modest changes in the specified relativeError
parameter. The result varies much more than the magnitude of the relativeError
parameter.
{code:java}
Welcome to
__
/ __/__ ___ _/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.4
/_/
Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_212)
Type in expressions to have them evaluated.
Type :help for more information.
scala> val df = spark.read.format("csv").option("header",
"true").option("inferSchema",
"true").load("./20191001_example_data_approx_quantile_bug")
df: org.apache.spark.sql.DataFrame = [value: double]
scala> df.stat.approxQuantile("value", Array(0.9), 0)
res0: Array[Double] = Array(0.5929591082174609)
scala> df.stat.approxQuantile("value", Array(0.9), 0.001)
res1: Array[Double] = Array(0.67621027121925)
scala> df.stat.approxQuantile("value", Array(0.9), 0.002)
res2: Array[Double] = Array(0.5926195654486178)
scala> df.stat.approxQuantile("value", Array(0.9), 0.003)
res3: Array[Double] = Array(0.5924693999048418)
scala> df.stat.approxQuantile("value", Array(0.9), 0.004)
res4: Array[Double] = Array(0.67621027121925)
scala> df.stat.approxQuantile("value", Array(0.9), 0.005)
res5: Array[Double] = Array(0.5923925937051544)
{code}
I attached a zip file containing the data used for the above example
demonstrating the bug.
was:
The [approxQuantile()
method|https://github.com/apache/spark/blob/3b1674cb1f244598463e879477d89632b0817578/sql/core/src/main/scala/org/apache/spark/sql/DataFrameStatFunctions.scala#L40]
returns sometimes incorrect results that are sensitively dependent upon the
choice of the relativeError.
Below is an example in the latest Spark version (2.4.4). You can see the result
varies significantly for modest changes in the specified relativeError
parameter. The result varies much more than the magnitude of the relativeError
parameter.
{code:java}
Welcome to
__
/ __/__ ___ _/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.4
/_/
Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_212)
Type in expressions to have them evaluated.
Type :help for more information.
scala> val df = spark.read.format("csv").option("header",
"true").option("inferSchema",
"true").load("./20191001_example_data_approx_quantile_bug")
df: org.apache.spark.sql.DataFrame = [value: double]
scala> df.stat.approxQuantile("value", Array(0.9), 0)
res0: Array[Double] = Array(0.5929591082174609)
scala> df.stat.approxQuantile("value", Array(0.9), 0.001)
res1: Array[Double] = Array(0.67621027121925)
scala> df.stat.approxQuantile("value", Array(0.9), 0.002)
res2: Array[Double] = Array(0.5926195654486178)
scala> df.stat.approxQuantile("value", Array(0.9), 0.003)
res3: Array[Double] = Array(0.5924693999048418)
scala> df.stat.approxQuantile("value", Array(0.9), 0.004)
res4: Array[Double] = Array(0.67621027121925)
scala> df.stat.approxQuantile("value", Array(0.9), 0.005)
res5: Array[Double] = Array(0.5923925937051544)
{code}
> approxQuantile() results are incorrect and vary significantly for small
> changes in relativeError
>
>
> Key: SPARK-29325
> URL: https://issues.apache.org/jira/browse/SPARK-29325
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.3.2, 2.4.4
> Environment: I was using OSX 10.14.6.
> I was using Scala 2.11.12 and Spark 2.4.4.
> I also verified the bug exists for Scala 2.11.8 and Spark 2.3.2.
>Reporter: James Verbus
>Priority: Major
> Labels: correctness
> Attachments: 20191001_example_data_approx_quantile_bug.zip
>
>
> The [approxQuantile()
> method|https://github.com/apache/spark/blob/3b1674cb1f244598463e879477d89632b0817578/sql/core/src/main/scala/org/apache/spark/sql/DataFrameStatFunctions.scala#L40]
> returns sometimes incorrect results that are sensitively dependent upon the
> choice of the relativeError.
> Below is an example in the latest Spark version (2.4.4). You can see the
> result varies significantly for modest changes in the specified relativeError
>
[jira] [Updated] (SPARK-29325) approxQuantile() results are incorrect and vary significantly for small changes in relativeError
[
https://issues.apache.org/jira/browse/SPARK-29325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Verbus updated SPARK-29325:
-
Attachment: 20191001_example_data_approx_quantile_bug.zip
> approxQuantile() results are incorrect and vary significantly for small
> changes in relativeError
>
>
> Key: SPARK-29325
> URL: https://issues.apache.org/jira/browse/SPARK-29325
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.3.2, 2.4.4
> Environment: I was using OSX 10.14.6.
> I was using Scala 2.11.12 and Spark 2.4.4.
> I also verified the bug exists for Scala 2.11.8 and Spark 2.3.2.
>Reporter: James Verbus
>Priority: Major
> Labels: correctness
> Attachments: 20191001_example_data_approx_quantile_bug.zip
>
>
> The [approxQuantile()
> method|https://github.com/apache/spark/blob/3b1674cb1f244598463e879477d89632b0817578/sql/core/src/main/scala/org/apache/spark/sql/DataFrameStatFunctions.scala#L40]
> returns sometimes incorrect results that are sensitively dependent upon the
> choice of the relativeError.
> Below is an example in the latest Spark version (2.4.4). You can see the
> result varies significantly for modest changes in the specified relativeError
> parameter. The result varies much more than the magnitude of the
> relativeError parameter.
>
> {code:java}
> Welcome to
> __
> / __/__ ___ _/ /__
> _\ \/ _ \/ _ `/ __/ '_/
> /___/ .__/\_,_/_/ /_/\_\ version 2.4.4
> /_/
>
> Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java
> 1.8.0_212)
> Type in expressions to have them evaluated.
> Type :help for more information.
> scala> val df = spark.read.format("csv").option("header",
> "true").option("inferSchema",
> "true").load("./20191001_example_data_approx_quantile_bug")
> df: org.apache.spark.sql.DataFrame = [value: double]
> scala> df.stat.approxQuantile("value", Array(0.9), 0)
> res0: Array[Double] = Array(0.5929591082174609)
> scala> df.stat.approxQuantile("value", Array(0.9), 0.001)
> res1: Array[Double] = Array(0.67621027121925)
> scala> df.stat.approxQuantile("value", Array(0.9), 0.002)
> res2: Array[Double] = Array(0.5926195654486178)
> scala> df.stat.approxQuantile("value", Array(0.9), 0.003)
> res3: Array[Double] = Array(0.5924693999048418)
> scala> df.stat.approxQuantile("value", Array(0.9), 0.004)
> res4: Array[Double] = Array(0.67621027121925)
> scala> df.stat.approxQuantile("value", Array(0.9), 0.005)
> res5: Array[Double] = Array(0.5923925937051544)
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-29325) approxQuantile() results are incorrect and vary significantly for small changes in relativeError
James Verbus created SPARK-29325:
Summary: approxQuantile() results are incorrect and vary
significantly for small changes in relativeError
Key: SPARK-29325
URL: https://issues.apache.org/jira/browse/SPARK-29325
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.4.4, 2.3.2
Environment: I was using OSX 10.14.6.
I was using Scala 2.11.12 and Spark 2.4.4.
I also verified the bug exists for Scala 2.11.8 and Spark 2.3.2.
Reporter: James Verbus
Attachments: 20191001_example_data_approx_quantile_bug.zip
The [approxQuantile()
method|https://github.com/apache/spark/blob/3b1674cb1f244598463e879477d89632b0817578/sql/core/src/main/scala/org/apache/spark/sql/DataFrameStatFunctions.scala#L40]
returns sometimes incorrect results that are sensitively dependent upon the
choice of the relativeError.
Below is an example in the latest Spark version (2.4.4). You can see the result
varies significantly for modest changes in the specified relativeError
parameter. The result varies much more than the magnitude of the relativeError
parameter.
{code:java}
Welcome to
__
/ __/__ ___ _/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.4
/_/
Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_212)
Type in expressions to have them evaluated.
Type :help for more information.
scala> val df = spark.read.format("csv").option("header",
"true").option("inferSchema",
"true").load("./20191001_example_data_approx_quantile_bug")
df: org.apache.spark.sql.DataFrame = [value: double]
scala> df.stat.approxQuantile("value", Array(0.9), 0)
res0: Array[Double] = Array(0.5929591082174609)
scala> df.stat.approxQuantile("value", Array(0.9), 0.001)
res1: Array[Double] = Array(0.67621027121925)
scala> df.stat.approxQuantile("value", Array(0.9), 0.002)
res2: Array[Double] = Array(0.5926195654486178)
scala> df.stat.approxQuantile("value", Array(0.9), 0.003)
res3: Array[Double] = Array(0.5924693999048418)
scala> df.stat.approxQuantile("value", Array(0.9), 0.004)
res4: Array[Double] = Array(0.67621027121925)
scala> df.stat.approxQuantile("value", Array(0.9), 0.005)
res5: Array[Double] = Array(0.5923925937051544)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-29322) History server is stuck reading incomplete event log file compressed with zstd
[
https://issues.apache.org/jira/browse/SPARK-29322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942489#comment-16942489
]
Dongjoon Hyun commented on SPARK-29322:
---
Also, cc [~dbtsai]
> History server is stuck reading incomplete event log file compressed with zstd
> --
>
> Key: SPARK-29322
> URL: https://issues.apache.org/jira/browse/SPARK-29322
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 3.0.0
>Reporter: Jungtaek Lim
>Priority: Major
> Attachments: history-server-1.jstack, history-server-2.jstack,
> history-server-3.jstack, history-server-4.jstack
>
>
> While working on SPARK-28869, I've discovered the issue that reading
> inprogress event log file on zstd compression could lead the thread being
> stuck. I just experimented with Spark History Server and observed same issue.
> I'll attach the jstack files.
> Only listing the thread stack trace being stuck across jstack files:
> {code}
> 2019-10-02 11:32:36
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.191-b12 mixed mode):
> ...
> "qtp2072313080-30" #30 daemon prio=5 os_prio=31 tid=0x7ff5b90e7800
> nid=0x9703 runnable [0x7f22]
>java.lang.Thread.State: RUNNABLE
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:255)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.read(RawLocalFileSystem.java:156)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5f97c60> (a
> org.apache.hadoop.fs.BufferedFSInputStream)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at
> org.apache.hadoop.fs.FSInputChecker.readFully(FSInputChecker.java:436)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.readChunk(ChecksumFileSystem.java:257)
> at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:276)
> at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:228)
> at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:196)
> - locked <0x0007b5f97b58> (a
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5f97af8> (a java.io.BufferedInputStream)
> at
> com.github.luben.zstd.ZstdInputStream.readInternal(ZstdInputStream.java:129)
> at com.github.luben.zstd.ZstdInputStream.read(ZstdInputStream.java:107)
> - locked <0x0007b5f97ac0> (a com.github.luben.zstd.ZstdInputStream)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5cd3bd0> (a java.io.BufferedInputStream)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
> - locked <0x0007b5f94a00> (a java.io.InputStreamReader)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:161)
> at java.io.BufferedReader.readLine(BufferedReader.java:324)
> - locked <0x0007b5f94a00> (a java.io.InputStreamReader)
> at java.io.BufferedReader.readLine(BufferedReader.java:389)
> at
> scala.io.BufferedSource$BufferedLineIterator.hasNext(BufferedSource.scala:74)
> at scala.collection.Iterator$$anon$20.hasNext(Iterator.scala:884)
> at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:511)
> at
> org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:80)
> at
> org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:58)
> at
> org.apache.spark.deploy.history.FsHistoryProvider.$anonfun$rebuildAppStore$5(FsHistoryProvider.scala:976)
> at
> org.apache.spark.deploy.history.FsHistoryProvider.$anonfun$rebuildAppStore$5$adapted(FsHistoryProvider.scala:975)
> at
> org.apache.spark.deploy.history.FsHistoryProvider$$Lambda$662/1267867461.apply(Unknown
> Source)
> at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2567)
> at
> org.apache.spark.deploy.history.FsHistoryProvider.rebuildAppStore(FsHistoryProvider.scala:975)
> at
>
[jira] [Commented] (SPARK-29322) History server is stuck reading incomplete event log file compressed with zstd
[
https://issues.apache.org/jira/browse/SPARK-29322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942488#comment-16942488
]
Dongjoon Hyun commented on SPARK-29322:
---
Since this is ZSTD, you are using `hadoop-3.2` profile on the master branch
with Hadoop 3.2.0, right? In addition to that, please update the reproducible
procedure into the JIRA description. The content on the mailing list would be a
good candidate.
> History server is stuck reading incomplete event log file compressed with zstd
> --
>
> Key: SPARK-29322
> URL: https://issues.apache.org/jira/browse/SPARK-29322
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 3.0.0
>Reporter: Jungtaek Lim
>Priority: Major
> Attachments: history-server-1.jstack, history-server-2.jstack,
> history-server-3.jstack, history-server-4.jstack
>
>
> While working on SPARK-28869, I've discovered the issue that reading
> inprogress event log file on zstd compression could lead the thread being
> stuck. I just experimented with Spark History Server and observed same issue.
> I'll attach the jstack files.
> Only listing the thread stack trace being stuck across jstack files:
> {code}
> 2019-10-02 11:32:36
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.191-b12 mixed mode):
> ...
> "qtp2072313080-30" #30 daemon prio=5 os_prio=31 tid=0x7ff5b90e7800
> nid=0x9703 runnable [0x7f22]
>java.lang.Thread.State: RUNNABLE
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:255)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.read(RawLocalFileSystem.java:156)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5f97c60> (a
> org.apache.hadoop.fs.BufferedFSInputStream)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at
> org.apache.hadoop.fs.FSInputChecker.readFully(FSInputChecker.java:436)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.readChunk(ChecksumFileSystem.java:257)
> at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:276)
> at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:228)
> at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:196)
> - locked <0x0007b5f97b58> (a
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5f97af8> (a java.io.BufferedInputStream)
> at
> com.github.luben.zstd.ZstdInputStream.readInternal(ZstdInputStream.java:129)
> at com.github.luben.zstd.ZstdInputStream.read(ZstdInputStream.java:107)
> - locked <0x0007b5f97ac0> (a com.github.luben.zstd.ZstdInputStream)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5cd3bd0> (a java.io.BufferedInputStream)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
> - locked <0x0007b5f94a00> (a java.io.InputStreamReader)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:161)
> at java.io.BufferedReader.readLine(BufferedReader.java:324)
> - locked <0x0007b5f94a00> (a java.io.InputStreamReader)
> at java.io.BufferedReader.readLine(BufferedReader.java:389)
> at
> scala.io.BufferedSource$BufferedLineIterator.hasNext(BufferedSource.scala:74)
> at scala.collection.Iterator$$anon$20.hasNext(Iterator.scala:884)
> at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:511)
> at
> org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:80)
> at
> org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:58)
> at
> org.apache.spark.deploy.history.FsHistoryProvider.$anonfun$rebuildAppStore$5(FsHistoryProvider.scala:976)
> at
> org.apache.spark.deploy.history.FsHistoryProvider.$anonfun$rebuildAppStore$5$adapted(FsHistoryProvider.scala:975)
> at
> org.apache.spark.deploy.history.FsHistoryProvider$$Lambda$662/1267867461.apply(Unknown
>
[jira] [Created] (SPARK-29324) saveAsTable with overwrite mode results in metadata loss
Karuppayya created SPARK-29324:
--
Summary: saveAsTable with overwrite mode results in metadata loss
Key: SPARK-29324
URL: https://issues.apache.org/jira/browse/SPARK-29324
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.0.0
Reporter: Karuppayya
{code:java}
scala> spark.range(1).write.option("path",
"file:///tmp/tbl").format("orc").saveAsTable("tbl")
scala> spark.sql("desc extended tbl").collect.foreach(println)
[id,bigint,null]
[,,]
[# Detailed Table Information,,]
[Database,default,]
[Table,tbl,]
[Owner,karuppayyar,]
[Created Time,Wed Oct 02 09:29:06 IST 2019,]
[Last Access,UNKNOWN,]
[Created By,Spark 3.0.0-SNAPSHOT,]
[Type,EXTERNAL,]
[Provider,orc,]
[Location,file:/tmp/tbl_loc,]
[Serde Library,org.apache.hadoop.hive.ql.io.orc.OrcSerde,]
[InputFormat,org.apache.hadoop.hive.ql.io.orc.OrcInputFormat,]
[OutputFormat,org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat,]
{code}
{code:java}
// Overwriting table
scala> spark.range(100).write.mode("overwrite").saveAsTable("tbl")
scala> spark.sql("desc extended tbl").collect.foreach(println)
[id,bigint,null]
[,,]
[# Detailed Table Information,,]
[Database,default,]
[Table,tbl,]
[Owner,karuppayyar,]
[Created Time,Wed Oct 02 09:30:36 IST 2019,]
[Last Access,UNKNOWN,]
[Created By,Spark 3.0.0-SNAPSHOT,]
[Type,MANAGED,]
[Provider,parquet,]
[Location,file:/tmp/tbl,]
[Serde Library,org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe,]
[InputFormat,org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat,]
[OutputFormat,org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat,]
{code}
The first code block creates an EXTERNAL table in Orc format
The second code block overwrites it with more data
After the overwrite,
1. The external table became a managed table.
2. The fileformat has changed from Orc to parquet(default fileformat).
And other information(like owner, TBLPROPERTIES) are also overwritten.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-29322) History server is stuck reading incomplete event log file compressed with zstd
[
https://issues.apache.org/jira/browse/SPARK-29322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942483#comment-16942483
]
Dongjoon Hyun edited comment on SPARK-29322 at 10/2/19 4:53 AM:
Can we file and link an upstream issue Hadoop or ZSTD here? BTW, thanks!
was (Author: dongjoon):
Can we file and link an upstream issue Hadoop or ZSTD here?
> History server is stuck reading incomplete event log file compressed with zstd
> --
>
> Key: SPARK-29322
> URL: https://issues.apache.org/jira/browse/SPARK-29322
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 3.0.0
>Reporter: Jungtaek Lim
>Priority: Major
> Attachments: history-server-1.jstack, history-server-2.jstack,
> history-server-3.jstack, history-server-4.jstack
>
>
> While working on SPARK-28869, I've discovered the issue that reading
> inprogress event log file on zstd compression could lead the thread being
> stuck. I just experimented with Spark History Server and observed same issue.
> I'll attach the jstack files.
> Only listing the thread stack trace being stuck across jstack files:
> {code}
> 2019-10-02 11:32:36
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.191-b12 mixed mode):
> ...
> "qtp2072313080-30" #30 daemon prio=5 os_prio=31 tid=0x7ff5b90e7800
> nid=0x9703 runnable [0x7f22]
>java.lang.Thread.State: RUNNABLE
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:255)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.read(RawLocalFileSystem.java:156)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5f97c60> (a
> org.apache.hadoop.fs.BufferedFSInputStream)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at
> org.apache.hadoop.fs.FSInputChecker.readFully(FSInputChecker.java:436)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.readChunk(ChecksumFileSystem.java:257)
> at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:276)
> at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:228)
> at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:196)
> - locked <0x0007b5f97b58> (a
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5f97af8> (a java.io.BufferedInputStream)
> at
> com.github.luben.zstd.ZstdInputStream.readInternal(ZstdInputStream.java:129)
> at com.github.luben.zstd.ZstdInputStream.read(ZstdInputStream.java:107)
> - locked <0x0007b5f97ac0> (a com.github.luben.zstd.ZstdInputStream)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5cd3bd0> (a java.io.BufferedInputStream)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
> - locked <0x0007b5f94a00> (a java.io.InputStreamReader)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:161)
> at java.io.BufferedReader.readLine(BufferedReader.java:324)
> - locked <0x0007b5f94a00> (a java.io.InputStreamReader)
> at java.io.BufferedReader.readLine(BufferedReader.java:389)
> at
> scala.io.BufferedSource$BufferedLineIterator.hasNext(BufferedSource.scala:74)
> at scala.collection.Iterator$$anon$20.hasNext(Iterator.scala:884)
> at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:511)
> at
> org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:80)
> at
> org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:58)
> at
> org.apache.spark.deploy.history.FsHistoryProvider.$anonfun$rebuildAppStore$5(FsHistoryProvider.scala:976)
> at
> org.apache.spark.deploy.history.FsHistoryProvider.$anonfun$rebuildAppStore$5$adapted(FsHistoryProvider.scala:975)
> at
> org.apache.spark.deploy.history.FsHistoryProvider$$Lambda$662/1267867461.apply(Unknown
> Source)
> at
[jira] [Commented] (SPARK-29322) History server is stuck reading incomplete event log file compressed with zstd
[
https://issues.apache.org/jira/browse/SPARK-29322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942483#comment-16942483
]
Dongjoon Hyun commented on SPARK-29322:
---
Can we file and link an upstream issue Hadoop or ZSTD here?
> History server is stuck reading incomplete event log file compressed with zstd
> --
>
> Key: SPARK-29322
> URL: https://issues.apache.org/jira/browse/SPARK-29322
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 3.0.0
>Reporter: Jungtaek Lim
>Priority: Major
> Attachments: history-server-1.jstack, history-server-2.jstack,
> history-server-3.jstack, history-server-4.jstack
>
>
> While working on SPARK-28869, I've discovered the issue that reading
> inprogress event log file on zstd compression could lead the thread being
> stuck. I just experimented with Spark History Server and observed same issue.
> I'll attach the jstack files.
> Only listing the thread stack trace being stuck across jstack files:
> {code}
> 2019-10-02 11:32:36
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.191-b12 mixed mode):
> ...
> "qtp2072313080-30" #30 daemon prio=5 os_prio=31 tid=0x7ff5b90e7800
> nid=0x9703 runnable [0x7f22]
>java.lang.Thread.State: RUNNABLE
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:255)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.read(RawLocalFileSystem.java:156)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5f97c60> (a
> org.apache.hadoop.fs.BufferedFSInputStream)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at
> org.apache.hadoop.fs.FSInputChecker.readFully(FSInputChecker.java:436)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.readChunk(ChecksumFileSystem.java:257)
> at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:276)
> at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:228)
> at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:196)
> - locked <0x0007b5f97b58> (a
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5f97af8> (a java.io.BufferedInputStream)
> at
> com.github.luben.zstd.ZstdInputStream.readInternal(ZstdInputStream.java:129)
> at com.github.luben.zstd.ZstdInputStream.read(ZstdInputStream.java:107)
> - locked <0x0007b5f97ac0> (a com.github.luben.zstd.ZstdInputStream)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5cd3bd0> (a java.io.BufferedInputStream)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
> - locked <0x0007b5f94a00> (a java.io.InputStreamReader)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:161)
> at java.io.BufferedReader.readLine(BufferedReader.java:324)
> - locked <0x0007b5f94a00> (a java.io.InputStreamReader)
> at java.io.BufferedReader.readLine(BufferedReader.java:389)
> at
> scala.io.BufferedSource$BufferedLineIterator.hasNext(BufferedSource.scala:74)
> at scala.collection.Iterator$$anon$20.hasNext(Iterator.scala:884)
> at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:511)
> at
> org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:80)
> at
> org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:58)
> at
> org.apache.spark.deploy.history.FsHistoryProvider.$anonfun$rebuildAppStore$5(FsHistoryProvider.scala:976)
> at
> org.apache.spark.deploy.history.FsHistoryProvider.$anonfun$rebuildAppStore$5$adapted(FsHistoryProvider.scala:975)
> at
> org.apache.spark.deploy.history.FsHistoryProvider$$Lambda$662/1267867461.apply(Unknown
> Source)
> at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2567)
> at
> org.apache.spark.deploy.history.FsHistoryProvider.rebuildAppStore(FsHistoryProvider.scala:975)
>
[jira] [Comment Edited] (SPARK-29322) History server is stuck reading incomplete event log file compressed with zstd
[
https://issues.apache.org/jira/browse/SPARK-29322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942456#comment-16942456
]
Jungtaek Lim edited comment on SPARK-29322 at 10/2/19 4:17 AM:
---
Just initiated discussion on dev. mailing list to see which approach is
preferred to deal with this.
https://lists.apache.org/thread.html/24f2c371a5859f2a31f305ef42c8aaed1b1c6add2239d6cf97f57203@%3Cdev.spark.apache.org%3E
was (Author: kabhwan):
Just initiated discussion on dev. mailing list to see which approach is
preferred to deal with this.
> History server is stuck reading incomplete event log file compressed with zstd
> --
>
> Key: SPARK-29322
> URL: https://issues.apache.org/jira/browse/SPARK-29322
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 3.0.0
>Reporter: Jungtaek Lim
>Priority: Major
> Attachments: history-server-1.jstack, history-server-2.jstack,
> history-server-3.jstack, history-server-4.jstack
>
>
> While working on SPARK-28869, I've discovered the issue that reading
> inprogress event log file on zstd compression could lead the thread being
> stuck. I just experimented with Spark History Server and observed same issue.
> I'll attach the jstack files.
> Only listing the thread stack trace being stuck across jstack files:
> {code}
> 2019-10-02 11:32:36
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.191-b12 mixed mode):
> ...
> "qtp2072313080-30" #30 daemon prio=5 os_prio=31 tid=0x7ff5b90e7800
> nid=0x9703 runnable [0x7f22]
>java.lang.Thread.State: RUNNABLE
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:255)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.read(RawLocalFileSystem.java:156)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5f97c60> (a
> org.apache.hadoop.fs.BufferedFSInputStream)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at
> org.apache.hadoop.fs.FSInputChecker.readFully(FSInputChecker.java:436)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.readChunk(ChecksumFileSystem.java:257)
> at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:276)
> at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:228)
> at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:196)
> - locked <0x0007b5f97b58> (a
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5f97af8> (a java.io.BufferedInputStream)
> at
> com.github.luben.zstd.ZstdInputStream.readInternal(ZstdInputStream.java:129)
> at com.github.luben.zstd.ZstdInputStream.read(ZstdInputStream.java:107)
> - locked <0x0007b5f97ac0> (a com.github.luben.zstd.ZstdInputStream)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5cd3bd0> (a java.io.BufferedInputStream)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
> - locked <0x0007b5f94a00> (a java.io.InputStreamReader)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:161)
> at java.io.BufferedReader.readLine(BufferedReader.java:324)
> - locked <0x0007b5f94a00> (a java.io.InputStreamReader)
> at java.io.BufferedReader.readLine(BufferedReader.java:389)
> at
> scala.io.BufferedSource$BufferedLineIterator.hasNext(BufferedSource.scala:74)
> at scala.collection.Iterator$$anon$20.hasNext(Iterator.scala:884)
> at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:511)
> at
> org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:80)
> at
> org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:58)
> at
> org.apache.spark.deploy.history.FsHistoryProvider.$anonfun$rebuildAppStore$5(FsHistoryProvider.scala:976)
> at
>
[jira] [Commented] (SPARK-29323) Add tooltip for The Executors Tab's column names in the Spark history server Page
[ https://issues.apache.org/jira/browse/SPARK-29323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942471#comment-16942471 ] liucht-inspur commented on SPARK-29323: --- I will submit the PR later,thanks. > Add tooltip for The Executors Tab's column names in the Spark history server > Page > - > > Key: SPARK-29323 > URL: https://issues.apache.org/jira/browse/SPARK-29323 > Project: Spark > Issue Type: New Feature > Components: Web UI >Affects Versions: 2.4.4 >Reporter: liucht-inspur >Priority: Major > Fix For: 2.4.4 > > > the spark Executors of history Tab page, the Summary part shows the line in > the list of title, but format is irregular. > Some column names have tooltip, such as Storage Memory, Task Time(GC Time), > Input, Shuffle Read,Shuffle Write and Blacklisted, but there are still some > list names that have not tooltip. They are RDD Blocks, Disk Used,Cores, > Activity Tasks, Failed Tasks , Complete Tasks and Total Tasks. oddly, > Executors section below,All the column names Contains the column names above > have tooltip . -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-29323) Add tooltip for The Executors Tab's column names in the Spark history server Page
liucht-inspur created SPARK-29323: - Summary: Add tooltip for The Executors Tab's column names in the Spark history server Page Key: SPARK-29323 URL: https://issues.apache.org/jira/browse/SPARK-29323 Project: Spark Issue Type: New Feature Components: Web UI Affects Versions: 2.4.4 Reporter: liucht-inspur Fix For: 2.4.4 the spark Executors of history Tab page, the Summary part shows the line in the list of title, but format is irregular. Some column names have tooltip, such as Storage Memory, Task Time(GC Time), Input, Shuffle Read,Shuffle Write and Blacklisted, but there are still some list names that have not tooltip. They are RDD Blocks, Disk Used,Cores, Activity Tasks, Failed Tasks , Complete Tasks and Total Tasks. oddly, Executors section below,All the column names Contains the column names above have tooltip . -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-29313) Failure on writing to `noop` datasource in benchmarks
[
https://issues.apache.org/jira/browse/SPARK-29313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun reassigned SPARK-29313:
-
Assignee: Maxim Gekk
> Failure on writing to `noop` datasource in benchmarks
> -
>
> Key: SPARK-29313
> URL: https://issues.apache.org/jira/browse/SPARK-29313
> Project: Spark
> Issue Type: Test
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Maxim Gekk
>Assignee: Maxim Gekk
>Priority: Minor
>
> This failure happens on overwriting results of ExtractBenchmark via command:
> {code}
> SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/test:runMain
> org.apache.spark.sql.execution.benchmark.ExtractBenchmark"
> {code}
> with the exception:
> {code}
> [info] Running benchmark: Invoke extract for timestamp
> [info] Running case: cast to timestamp
> [error] Exception in thread "main" org.apache.spark.sql.AnalysisException:
> TableProvider implementation noop cannot be written with ErrorIfExists mode,
> please use Append or Overwrite modes instead.;
> [error] at
> org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:284)
> [error] at
> org.apache.spark.sql.execution.benchmark.ExtractBenchmark$.$anonfun$doBenchmark$1(ExtractBenchmark.scala:46)
> [error] at
> org.apache.spark.sql.catalyst.plans.SQLHelper.withSQLConf(SQLHelper.scala:47)
> [error] at
> org.apache.spark.sql.catalyst.plans.SQLHelper.withSQLConf$(SQLHelper.scala:31)
> [error] at
> org.apache.spark.sql.execution.benchmark.ExtractBenchmark$.withSQLConf(ExtractBenchmark.scala:37)
> [error] at
> org.apache.spark.sql.execution.benchmark.ExtractBenchmark$.doBenchmark(ExtractBenchmark.scala:47)
> [error] at
> org.apache.spark.sql.execution.benchmark.ExtractBenchmark$.$anonfun$run$1(ExtractBenchmark.scala:57)
> [error] at
> org.apache.spark.benchmark.Benchmark.$anonfun$addCase$1(Benchmark.scala:77)
> [error] at
> org.apache.spark.benchmark.Benchmark.$anonfun$addCase$1$adapted(Benchmark.scala:75)
> [error] at
> org.apache.spark.benchmark.Benchmark.measure(Benchmark.scala:139)
> [error] at
> org.apache.spark.benchmark.Benchmark.$anonfun$run$1(Benchmark.scala:106)
> [error] at
> scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
> [error] at
> scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
> [error] at
> scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
> [error] at
> scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
> [error] at
> scala.collection.TraversableLike.map(TraversableLike.scala:238)
> [error] at
> scala.collection.TraversableLike.map$(TraversableLike.scala:231)
> [error] at
> scala.collection.AbstractTraversable.map(Traversable.scala:108)
> [error] at org.apache.spark.benchmark.Benchmark.run(Benchmark.scala:104)
> [error] at
> org.apache.spark.sql.execution.benchmark.ExtractBenchmark$.$anonfun$runBenchmarkSuite$2(ExtractBenchmark.scala:101)
> [error] at
> org.apache.spark.sql.execution.benchmark.ExtractBenchmark$.$anonfun$runBenchmarkSuite$2$adapted(ExtractBenchmark.scala:95)
> [error] at scala.collection.immutable.List.foreach(List.scala:392)
> [error] at
> org.apache.spark.sql.execution.benchmark.ExtractBenchmark$.$anonfun$runBenchmarkSuite$1(ExtractBenchmark.scala:95)
> [error] at
> org.apache.spark.sql.execution.benchmark.ExtractBenchmark$.$anonfun$runBenchmarkSuite$1$adapted(ExtractBenchmark.scala:94)
> [error] at scala.collection.immutable.List.foreach(List.scala:392)
> [error] at
> org.apache.spark.sql.execution.benchmark.ExtractBenchmark$.runBenchmarkSuite(ExtractBenchmark.scala:94)
> [error] at
> org.apache.spark.benchmark.BenchmarkBase.main(BenchmarkBase.scala:57)
> [error] at
> org.apache.spark.sql.execution.benchmark.ExtractBenchmark.main(ExtractBenchmark.scala)
> java.lang.RuntimeException: Nonzero exit code returned from runner: 1
> at scala.sys.package$.error(package.scala:27)
> [trace] Stack trace suppressed: run last sql/test:runMain for the full output.
> [error] (sql/test:runMain) Nonzero exit code returned from runner: 1
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-29313) Failure on writing to `noop` datasource in benchmarks
[
https://issues.apache.org/jira/browse/SPARK-29313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun resolved SPARK-29313.
---
Fix Version/s: 3.0.0
Resolution: Fixed
Issue resolved by pull request 25988
[https://github.com/apache/spark/pull/25988]
> Failure on writing to `noop` datasource in benchmarks
> -
>
> Key: SPARK-29313
> URL: https://issues.apache.org/jira/browse/SPARK-29313
> Project: Spark
> Issue Type: Test
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Maxim Gekk
>Assignee: Maxim Gekk
>Priority: Minor
> Fix For: 3.0.0
>
>
> This failure happens on overwriting results of ExtractBenchmark via command:
> {code}
> SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/test:runMain
> org.apache.spark.sql.execution.benchmark.ExtractBenchmark"
> {code}
> with the exception:
> {code}
> [info] Running benchmark: Invoke extract for timestamp
> [info] Running case: cast to timestamp
> [error] Exception in thread "main" org.apache.spark.sql.AnalysisException:
> TableProvider implementation noop cannot be written with ErrorIfExists mode,
> please use Append or Overwrite modes instead.;
> [error] at
> org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:284)
> [error] at
> org.apache.spark.sql.execution.benchmark.ExtractBenchmark$.$anonfun$doBenchmark$1(ExtractBenchmark.scala:46)
> [error] at
> org.apache.spark.sql.catalyst.plans.SQLHelper.withSQLConf(SQLHelper.scala:47)
> [error] at
> org.apache.spark.sql.catalyst.plans.SQLHelper.withSQLConf$(SQLHelper.scala:31)
> [error] at
> org.apache.spark.sql.execution.benchmark.ExtractBenchmark$.withSQLConf(ExtractBenchmark.scala:37)
> [error] at
> org.apache.spark.sql.execution.benchmark.ExtractBenchmark$.doBenchmark(ExtractBenchmark.scala:47)
> [error] at
> org.apache.spark.sql.execution.benchmark.ExtractBenchmark$.$anonfun$run$1(ExtractBenchmark.scala:57)
> [error] at
> org.apache.spark.benchmark.Benchmark.$anonfun$addCase$1(Benchmark.scala:77)
> [error] at
> org.apache.spark.benchmark.Benchmark.$anonfun$addCase$1$adapted(Benchmark.scala:75)
> [error] at
> org.apache.spark.benchmark.Benchmark.measure(Benchmark.scala:139)
> [error] at
> org.apache.spark.benchmark.Benchmark.$anonfun$run$1(Benchmark.scala:106)
> [error] at
> scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
> [error] at
> scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
> [error] at
> scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
> [error] at
> scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
> [error] at
> scala.collection.TraversableLike.map(TraversableLike.scala:238)
> [error] at
> scala.collection.TraversableLike.map$(TraversableLike.scala:231)
> [error] at
> scala.collection.AbstractTraversable.map(Traversable.scala:108)
> [error] at org.apache.spark.benchmark.Benchmark.run(Benchmark.scala:104)
> [error] at
> org.apache.spark.sql.execution.benchmark.ExtractBenchmark$.$anonfun$runBenchmarkSuite$2(ExtractBenchmark.scala:101)
> [error] at
> org.apache.spark.sql.execution.benchmark.ExtractBenchmark$.$anonfun$runBenchmarkSuite$2$adapted(ExtractBenchmark.scala:95)
> [error] at scala.collection.immutable.List.foreach(List.scala:392)
> [error] at
> org.apache.spark.sql.execution.benchmark.ExtractBenchmark$.$anonfun$runBenchmarkSuite$1(ExtractBenchmark.scala:95)
> [error] at
> org.apache.spark.sql.execution.benchmark.ExtractBenchmark$.$anonfun$runBenchmarkSuite$1$adapted(ExtractBenchmark.scala:94)
> [error] at scala.collection.immutable.List.foreach(List.scala:392)
> [error] at
> org.apache.spark.sql.execution.benchmark.ExtractBenchmark$.runBenchmarkSuite(ExtractBenchmark.scala:94)
> [error] at
> org.apache.spark.benchmark.BenchmarkBase.main(BenchmarkBase.scala:57)
> [error] at
> org.apache.spark.sql.execution.benchmark.ExtractBenchmark.main(ExtractBenchmark.scala)
> java.lang.RuntimeException: Nonzero exit code returned from runner: 1
> at scala.sys.package$.error(package.scala:27)
> [trace] Stack trace suppressed: run last sql/test:runMain for the full output.
> [error] (sql/test:runMain) Nonzero exit code returned from runner: 1
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-29310) TestMemoryManager should implement getExecutionMemoryUsageForTask()
[
https://issues.apache.org/jira/browse/SPARK-29310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenchen Fan resolved SPARK-29310.
-
Fix Version/s: 3.0.0
Resolution: Fixed
Issue resolved by pull request 25985
[https://github.com/apache/spark/pull/25985]
> TestMemoryManager should implement getExecutionMemoryUsageForTask()
> ---
>
> Key: SPARK-29310
> URL: https://issues.apache.org/jira/browse/SPARK-29310
> Project: Spark
> Issue Type: Improvement
> Components: Tests
>Affects Versions: 3.0.0
>Reporter: Josh Rosen
>Assignee: Josh Rosen
>Priority: Major
> Fix For: 3.0.0
>
>
> Spark uses a {{TestMemoryManager}} class to mock out memory manager
> functionality in tests, allowing test authors to exercise control over
> certain behaviors (e.g. to simulate OOMs).
> Our tests have memory-leak detection to ensure that all allocated memory is
> cleaned up at the end of each test case; this helps to guard against bugs
> that could cause production memory leaks.
> Unfortunately, however, this leak-detection logic is broken for tests which
> use {{TestMemoryManager}} because it does not override the
> {{getExecutionMemoryUsageForTask()}} method which the leak-detection
> mechanism needs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Issue Comment Deleted] (SPARK-29322) History server is stuck reading incomplete event log file compressed with zstd
[
https://issues.apache.org/jira/browse/SPARK-29322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated SPARK-29322:
-
Comment: was deleted
(was: I'll work on PR to propose removing zstd from supported compressions for
event log. We may want to apply another approach: we can discuss further in the
PR.)
> History server is stuck reading incomplete event log file compressed with zstd
> --
>
> Key: SPARK-29322
> URL: https://issues.apache.org/jira/browse/SPARK-29322
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 3.0.0
>Reporter: Jungtaek Lim
>Priority: Major
> Attachments: history-server-1.jstack, history-server-2.jstack,
> history-server-3.jstack, history-server-4.jstack
>
>
> While working on SPARK-28869, I've discovered the issue that reading
> inprogress event log file on zstd compression could lead the thread being
> stuck. I just experimented with Spark History Server and observed same issue.
> I'll attach the jstack files.
> Only listing the thread stack trace being stuck across jstack files:
> {code}
> 2019-10-02 11:32:36
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.191-b12 mixed mode):
> ...
> "qtp2072313080-30" #30 daemon prio=5 os_prio=31 tid=0x7ff5b90e7800
> nid=0x9703 runnable [0x7f22]
>java.lang.Thread.State: RUNNABLE
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:255)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.read(RawLocalFileSystem.java:156)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5f97c60> (a
> org.apache.hadoop.fs.BufferedFSInputStream)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at
> org.apache.hadoop.fs.FSInputChecker.readFully(FSInputChecker.java:436)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.readChunk(ChecksumFileSystem.java:257)
> at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:276)
> at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:228)
> at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:196)
> - locked <0x0007b5f97b58> (a
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5f97af8> (a java.io.BufferedInputStream)
> at
> com.github.luben.zstd.ZstdInputStream.readInternal(ZstdInputStream.java:129)
> at com.github.luben.zstd.ZstdInputStream.read(ZstdInputStream.java:107)
> - locked <0x0007b5f97ac0> (a com.github.luben.zstd.ZstdInputStream)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5cd3bd0> (a java.io.BufferedInputStream)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
> - locked <0x0007b5f94a00> (a java.io.InputStreamReader)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:161)
> at java.io.BufferedReader.readLine(BufferedReader.java:324)
> - locked <0x0007b5f94a00> (a java.io.InputStreamReader)
> at java.io.BufferedReader.readLine(BufferedReader.java:389)
> at
> scala.io.BufferedSource$BufferedLineIterator.hasNext(BufferedSource.scala:74)
> at scala.collection.Iterator$$anon$20.hasNext(Iterator.scala:884)
> at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:511)
> at
> org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:80)
> at
> org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:58)
> at
> org.apache.spark.deploy.history.FsHistoryProvider.$anonfun$rebuildAppStore$5(FsHistoryProvider.scala:976)
> at
> org.apache.spark.deploy.history.FsHistoryProvider.$anonfun$rebuildAppStore$5$adapted(FsHistoryProvider.scala:975)
> at
> org.apache.spark.deploy.history.FsHistoryProvider$$Lambda$662/1267867461.apply(Unknown
> Source)
> at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2567)
> at
>
[jira] [Commented] (SPARK-29322) History server is stuck reading incomplete event log file compressed with zstd
[
https://issues.apache.org/jira/browse/SPARK-29322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942456#comment-16942456
]
Jungtaek Lim commented on SPARK-29322:
--
Just initiated discussion on dev. mailing list to see which approach is
preferred to deal with this.
> History server is stuck reading incomplete event log file compressed with zstd
> --
>
> Key: SPARK-29322
> URL: https://issues.apache.org/jira/browse/SPARK-29322
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 3.0.0
>Reporter: Jungtaek Lim
>Priority: Major
> Attachments: history-server-1.jstack, history-server-2.jstack,
> history-server-3.jstack, history-server-4.jstack
>
>
> While working on SPARK-28869, I've discovered the issue that reading
> inprogress event log file on zstd compression could lead the thread being
> stuck. I just experimented with Spark History Server and observed same issue.
> I'll attach the jstack files.
> Only listing the thread stack trace being stuck across jstack files:
> {code}
> 2019-10-02 11:32:36
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.191-b12 mixed mode):
> ...
> "qtp2072313080-30" #30 daemon prio=5 os_prio=31 tid=0x7ff5b90e7800
> nid=0x9703 runnable [0x7f22]
>java.lang.Thread.State: RUNNABLE
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:255)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.read(RawLocalFileSystem.java:156)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5f97c60> (a
> org.apache.hadoop.fs.BufferedFSInputStream)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at
> org.apache.hadoop.fs.FSInputChecker.readFully(FSInputChecker.java:436)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.readChunk(ChecksumFileSystem.java:257)
> at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:276)
> at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:228)
> at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:196)
> - locked <0x0007b5f97b58> (a
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5f97af8> (a java.io.BufferedInputStream)
> at
> com.github.luben.zstd.ZstdInputStream.readInternal(ZstdInputStream.java:129)
> at com.github.luben.zstd.ZstdInputStream.read(ZstdInputStream.java:107)
> - locked <0x0007b5f97ac0> (a com.github.luben.zstd.ZstdInputStream)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5cd3bd0> (a java.io.BufferedInputStream)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
> - locked <0x0007b5f94a00> (a java.io.InputStreamReader)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:161)
> at java.io.BufferedReader.readLine(BufferedReader.java:324)
> - locked <0x0007b5f94a00> (a java.io.InputStreamReader)
> at java.io.BufferedReader.readLine(BufferedReader.java:389)
> at
> scala.io.BufferedSource$BufferedLineIterator.hasNext(BufferedSource.scala:74)
> at scala.collection.Iterator$$anon$20.hasNext(Iterator.scala:884)
> at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:511)
> at
> org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:80)
> at
> org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:58)
> at
> org.apache.spark.deploy.history.FsHistoryProvider.$anonfun$rebuildAppStore$5(FsHistoryProvider.scala:976)
> at
> org.apache.spark.deploy.history.FsHistoryProvider.$anonfun$rebuildAppStore$5$adapted(FsHistoryProvider.scala:975)
> at
> org.apache.spark.deploy.history.FsHistoryProvider$$Lambda$662/1267867461.apply(Unknown
> Source)
> at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2567)
> at
>
[jira] [Commented] (SPARK-29322) History server is stuck reading incomplete event log file compressed with zstd
[
https://issues.apache.org/jira/browse/SPARK-29322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942446#comment-16942446
]
Jungtaek Lim commented on SPARK-29322:
--
FYI, thread being stuck was finished immediately when I stopped the application.
> History server is stuck reading incomplete event log file compressed with zstd
> --
>
> Key: SPARK-29322
> URL: https://issues.apache.org/jira/browse/SPARK-29322
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 3.0.0
>Reporter: Jungtaek Lim
>Priority: Major
> Attachments: history-server-1.jstack, history-server-2.jstack,
> history-server-3.jstack, history-server-4.jstack
>
>
> While working on SPARK-28869, I've discovered the issue that reading
> inprogress event log file on zstd compression could lead the thread being
> stuck. I just experimented with Spark History Server and observed same issue.
> I'll attach the jstack files.
> Only listing the thread stack trace being stuck across jstack files:
> {code}
> 2019-10-02 11:32:36
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.191-b12 mixed mode):
> ...
> "qtp2072313080-30" #30 daemon prio=5 os_prio=31 tid=0x7ff5b90e7800
> nid=0x9703 runnable [0x7f22]
>java.lang.Thread.State: RUNNABLE
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:255)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.read(RawLocalFileSystem.java:156)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5f97c60> (a
> org.apache.hadoop.fs.BufferedFSInputStream)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at
> org.apache.hadoop.fs.FSInputChecker.readFully(FSInputChecker.java:436)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.readChunk(ChecksumFileSystem.java:257)
> at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:276)
> at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:228)
> at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:196)
> - locked <0x0007b5f97b58> (a
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5f97af8> (a java.io.BufferedInputStream)
> at
> com.github.luben.zstd.ZstdInputStream.readInternal(ZstdInputStream.java:129)
> at com.github.luben.zstd.ZstdInputStream.read(ZstdInputStream.java:107)
> - locked <0x0007b5f97ac0> (a com.github.luben.zstd.ZstdInputStream)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5cd3bd0> (a java.io.BufferedInputStream)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
> - locked <0x0007b5f94a00> (a java.io.InputStreamReader)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:161)
> at java.io.BufferedReader.readLine(BufferedReader.java:324)
> - locked <0x0007b5f94a00> (a java.io.InputStreamReader)
> at java.io.BufferedReader.readLine(BufferedReader.java:389)
> at
> scala.io.BufferedSource$BufferedLineIterator.hasNext(BufferedSource.scala:74)
> at scala.collection.Iterator$$anon$20.hasNext(Iterator.scala:884)
> at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:511)
> at
> org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:80)
> at
> org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:58)
> at
> org.apache.spark.deploy.history.FsHistoryProvider.$anonfun$rebuildAppStore$5(FsHistoryProvider.scala:976)
> at
> org.apache.spark.deploy.history.FsHistoryProvider.$anonfun$rebuildAppStore$5$adapted(FsHistoryProvider.scala:975)
> at
> org.apache.spark.deploy.history.FsHistoryProvider$$Lambda$662/1267867461.apply(Unknown
> Source)
> at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2567)
> at
>
[jira] [Commented] (SPARK-29322) History server is stuck reading incomplete event log file compressed with zstd
[
https://issues.apache.org/jira/browse/SPARK-29322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942445#comment-16942445
]
Jungtaek Lim commented on SPARK-29322:
--
I'll work on PR to propose removing zstd from supported compressions for event
log. We may want to apply another approach: we can discuss further in the PR.
> History server is stuck reading incomplete event log file compressed with zstd
> --
>
> Key: SPARK-29322
> URL: https://issues.apache.org/jira/browse/SPARK-29322
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 3.0.0
>Reporter: Jungtaek Lim
>Priority: Major
> Attachments: history-server-1.jstack, history-server-2.jstack,
> history-server-3.jstack, history-server-4.jstack
>
>
> While working on SPARK-28869, I've discovered the issue that reading
> inprogress event log file on zstd compression could lead the thread being
> stuck. I just experimented with Spark History Server and observed same issue.
> I'll attach the jstack files.
> Only listing the thread stack trace being stuck across jstack files:
> {code}
> 2019-10-02 11:32:36
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.191-b12 mixed mode):
> ...
> "qtp2072313080-30" #30 daemon prio=5 os_prio=31 tid=0x7ff5b90e7800
> nid=0x9703 runnable [0x7f22]
>java.lang.Thread.State: RUNNABLE
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:255)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.read(RawLocalFileSystem.java:156)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5f97c60> (a
> org.apache.hadoop.fs.BufferedFSInputStream)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at
> org.apache.hadoop.fs.FSInputChecker.readFully(FSInputChecker.java:436)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.readChunk(ChecksumFileSystem.java:257)
> at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:276)
> at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:228)
> at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:196)
> - locked <0x0007b5f97b58> (a
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5f97af8> (a java.io.BufferedInputStream)
> at
> com.github.luben.zstd.ZstdInputStream.readInternal(ZstdInputStream.java:129)
> at com.github.luben.zstd.ZstdInputStream.read(ZstdInputStream.java:107)
> - locked <0x0007b5f97ac0> (a com.github.luben.zstd.ZstdInputStream)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5cd3bd0> (a java.io.BufferedInputStream)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
> - locked <0x0007b5f94a00> (a java.io.InputStreamReader)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:161)
> at java.io.BufferedReader.readLine(BufferedReader.java:324)
> - locked <0x0007b5f94a00> (a java.io.InputStreamReader)
> at java.io.BufferedReader.readLine(BufferedReader.java:389)
> at
> scala.io.BufferedSource$BufferedLineIterator.hasNext(BufferedSource.scala:74)
> at scala.collection.Iterator$$anon$20.hasNext(Iterator.scala:884)
> at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:511)
> at
> org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:80)
> at
> org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:58)
> at
> org.apache.spark.deploy.history.FsHistoryProvider.$anonfun$rebuildAppStore$5(FsHistoryProvider.scala:976)
> at
> org.apache.spark.deploy.history.FsHistoryProvider.$anonfun$rebuildAppStore$5$adapted(FsHistoryProvider.scala:975)
> at
> org.apache.spark.deploy.history.FsHistoryProvider$$Lambda$662/1267867461.apply(Unknown
> Source)
> at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2567)
> at
>
[jira] [Updated] (SPARK-29322) History server is stuck reading incomplete event log file compressed with zstd
[
https://issues.apache.org/jira/browse/SPARK-29322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated SPARK-29322:
-
Attachment: history-server-4.jstack
history-server-3.jstack
history-server-2.jstack
history-server-1.jstack
> History server is stuck reading incomplete event log file compressed with zstd
> --
>
> Key: SPARK-29322
> URL: https://issues.apache.org/jira/browse/SPARK-29322
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 3.0.0
>Reporter: Jungtaek Lim
>Priority: Major
> Attachments: history-server-1.jstack, history-server-2.jstack,
> history-server-3.jstack, history-server-4.jstack
>
>
> While working on SPARK-28869, I've discovered the issue that reading
> inprogress event log file on zstd compression could lead the thread being
> stuck. I just experimented with Spark History Server and observed same issue.
> I'll attach the jstack files.
> Only listing the thread stack trace being stuck across jstack files:
> {code}
> 2019-10-02 11:32:36
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.191-b12 mixed mode):
> ...
> "qtp2072313080-30" #30 daemon prio=5 os_prio=31 tid=0x7ff5b90e7800
> nid=0x9703 runnable [0x7f22]
>java.lang.Thread.State: RUNNABLE
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:255)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.read(RawLocalFileSystem.java:156)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5f97c60> (a
> org.apache.hadoop.fs.BufferedFSInputStream)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at
> org.apache.hadoop.fs.FSInputChecker.readFully(FSInputChecker.java:436)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.readChunk(ChecksumFileSystem.java:257)
> at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:276)
> at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:228)
> at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:196)
> - locked <0x0007b5f97b58> (a
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5f97af8> (a java.io.BufferedInputStream)
> at
> com.github.luben.zstd.ZstdInputStream.readInternal(ZstdInputStream.java:129)
> at com.github.luben.zstd.ZstdInputStream.read(ZstdInputStream.java:107)
> - locked <0x0007b5f97ac0> (a com.github.luben.zstd.ZstdInputStream)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x0007b5cd3bd0> (a java.io.BufferedInputStream)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
> - locked <0x0007b5f94a00> (a java.io.InputStreamReader)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:161)
> at java.io.BufferedReader.readLine(BufferedReader.java:324)
> - locked <0x0007b5f94a00> (a java.io.InputStreamReader)
> at java.io.BufferedReader.readLine(BufferedReader.java:389)
> at
> scala.io.BufferedSource$BufferedLineIterator.hasNext(BufferedSource.scala:74)
> at scala.collection.Iterator$$anon$20.hasNext(Iterator.scala:884)
> at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:511)
> at
> org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:80)
> at
> org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:58)
> at
> org.apache.spark.deploy.history.FsHistoryProvider.$anonfun$rebuildAppStore$5(FsHistoryProvider.scala:976)
> at
> org.apache.spark.deploy.history.FsHistoryProvider.$anonfun$rebuildAppStore$5$adapted(FsHistoryProvider.scala:975)
> at
> org.apache.spark.deploy.history.FsHistoryProvider$$Lambda$662/1267867461.apply(Unknown
> Source)
> at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2567)
> at
>
[jira] [Created] (SPARK-29322) History server is stuck reading incomplete event log file compressed with zstd
Jungtaek Lim created SPARK-29322:
Summary: History server is stuck reading incomplete event log file
compressed with zstd
Key: SPARK-29322
URL: https://issues.apache.org/jira/browse/SPARK-29322
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 3.0.0
Reporter: Jungtaek Lim
While working on SPARK-28869, I've discovered the issue that reading inprogress
event log file on zstd compression could lead the thread being stuck. I just
experimented with Spark History Server and observed same issue. I'll attach the
jstack files.
Only listing the thread stack trace being stuck across jstack files:
{code}
2019-10-02 11:32:36
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.191-b12 mixed mode):
...
"qtp2072313080-30" #30 daemon prio=5 os_prio=31 tid=0x7ff5b90e7800
nid=0x9703 runnable [0x7f22]
java.lang.Thread.State: RUNNABLE
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:255)
at
org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.read(RawLocalFileSystem.java:156)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
- locked <0x0007b5f97c60> (a
org.apache.hadoop.fs.BufferedFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.fs.FSInputChecker.readFully(FSInputChecker.java:436)
at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.readChunk(ChecksumFileSystem.java:257)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:276)
at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:228)
at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:196)
- locked <0x0007b5f97b58> (a
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker)
at java.io.DataInputStream.read(DataInputStream.java:149)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
- locked <0x0007b5f97af8> (a java.io.BufferedInputStream)
at
com.github.luben.zstd.ZstdInputStream.readInternal(ZstdInputStream.java:129)
at com.github.luben.zstd.ZstdInputStream.read(ZstdInputStream.java:107)
- locked <0x0007b5f97ac0> (a com.github.luben.zstd.ZstdInputStream)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
- locked <0x0007b5cd3bd0> (a java.io.BufferedInputStream)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
- locked <0x0007b5f94a00> (a java.io.InputStreamReader)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:161)
at java.io.BufferedReader.readLine(BufferedReader.java:324)
- locked <0x0007b5f94a00> (a java.io.InputStreamReader)
at java.io.BufferedReader.readLine(BufferedReader.java:389)
at
scala.io.BufferedSource$BufferedLineIterator.hasNext(BufferedSource.scala:74)
at scala.collection.Iterator$$anon$20.hasNext(Iterator.scala:884)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:511)
at
org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:80)
at
org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:58)
at
org.apache.spark.deploy.history.FsHistoryProvider.$anonfun$rebuildAppStore$5(FsHistoryProvider.scala:976)
at
org.apache.spark.deploy.history.FsHistoryProvider.$anonfun$rebuildAppStore$5$adapted(FsHistoryProvider.scala:975)
at
org.apache.spark.deploy.history.FsHistoryProvider$$Lambda$662/1267867461.apply(Unknown
Source)
at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2567)
at
org.apache.spark.deploy.history.FsHistoryProvider.rebuildAppStore(FsHistoryProvider.scala:975)
at
org.apache.spark.deploy.history.FsHistoryProvider.createInMemoryStore(FsHistoryProvider.scala:1093)
at
org.apache.spark.deploy.history.FsHistoryProvider.getAppUI(FsHistoryProvider.scala:346)
at
org.apache.spark.deploy.history.HistoryServer.getAppUI(HistoryServer.scala:188)
at
org.apache.spark.deploy.history.ApplicationCache.$anonfun$loadApplicationEntry$2(ApplicationCache.scala:163)
at
[jira] [Assigned] (SPARK-29311) Return seconds with fraction from `date_part`/`extract`
[
https://issues.apache.org/jira/browse/SPARK-29311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-29311:
Assignee: Maxim Gekk
> Return seconds with fraction from `date_part`/`extract`
> ---
>
> Key: SPARK-29311
> URL: https://issues.apache.org/jira/browse/SPARK-29311
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Maxim Gekk
>Assignee: Maxim Gekk
>Priority: Minor
>
> The `date_part()` and `extract` should return seconds with fractional part
> for the `SECOND` field as PostgreSQL does:
> {code}
> # SELECT date_part('SECONDS', timestamp'2019-10-01 00:00:01.01');
> date_part
> ---
> 1.01
> (1 row)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-29311) Return seconds with fraction from `date_part`/`extract`
[
https://issues.apache.org/jira/browse/SPARK-29311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-29311.
--
Fix Version/s: 3.0.0
Resolution: Fixed
Issue resolved by pull request 25986
[https://github.com/apache/spark/pull/25986]
> Return seconds with fraction from `date_part`/`extract`
> ---
>
> Key: SPARK-29311
> URL: https://issues.apache.org/jira/browse/SPARK-29311
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.0.0
>Reporter: Maxim Gekk
>Assignee: Maxim Gekk
>Priority: Minor
> Fix For: 3.0.0
>
>
> The `date_part()` and `extract` should return seconds with fractional part
> for the `SECOND` field as PostgreSQL does:
> {code}
> # SELECT date_part('SECONDS', timestamp'2019-10-01 00:00:01.01');
> date_part
> ---
> 1.01
> (1 row)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28759) Upgrade scala-maven-plugin to 4.2.0
[ https://issues.apache.org/jira/browse/SPARK-28759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942408#comment-16942408 ] Hyukjin Kwon commented on SPARK-28759: -- BTW, given author's comment (https://github.com/davidB/scala-maven-plugin/issues/229#issuecomment-365601096), incremental compile could be disabled from JDK 9+. If it's not easy to fix, maybe we should just explicitly drop the incremental compile as an option. > Upgrade scala-maven-plugin to 4.2.0 > --- > > Key: SPARK-28759 > URL: https://issues.apache.org/jira/browse/SPARK-28759 > Project: Spark > Issue Type: Sub-task > Components: Build >Affects Versions: 3.0.0 >Reporter: Dongjoon Hyun >Assignee: Hyukjin Kwon >Priority: Major > Fix For: 3.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28759) Upgrade scala-maven-plugin to 4.2.0
[
https://issues.apache.org/jira/browse/SPARK-28759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942407#comment-16942407
]
Hyukjin Kwon commented on SPARK-28759:
--
I didn't notice too but I know this plugin had a radical change about
incremental compile (and {{zinc}}), which broke the build on Windows (and fixed
by https://github.com/apache/spark/pull/25633)
I am sure, if something's wrong, it will be between:
https://github.com/davidB/scala-maven-plugin/commits/4.0.0
https://github.com/davidB/scala-maven-plugin/commits/3.4.6
in particular the changes below:
https://github.com/davidB/scala-maven-plugin/commit/ba7ab932530ce9a3090e4bbbcc778bc85f56c894
https://github.com/davidB/scala-maven-plugin/commit/b93f1fcf8c775b3fa9aa94ac8bc41aef23963672
https://github.com/davidB/scala-maven-plugin/commit/4eb8b817c92c988b258c10a19fa83f669ca77b63
https://github.com/davidB/scala-maven-plugin/commit/0f01f2fe37aaadbefb02fb713d469dbd10acf8b1
> Upgrade scala-maven-plugin to 4.2.0
> ---
>
> Key: SPARK-28759
> URL: https://issues.apache.org/jira/browse/SPARK-28759
> Project: Spark
> Issue Type: Sub-task
> Components: Build
>Affects Versions: 3.0.0
>Reporter: Dongjoon Hyun
>Assignee: Hyukjin Kwon
>Priority: Major
> Fix For: 3.0.0
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-29318) orc library is incorrect in dev/deps/spark-deps-hadoop-3.2
[
https://issues.apache.org/jira/browse/SPARK-29318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942384#comment-16942384
]
Dongjoon Hyun commented on SPARK-29318:
---
Closed this one and SPARK-29308 moves under SPARK-23710 .
> orc library is incorrect in dev/deps/spark-deps-hadoop-3.2
> ---
>
> Key: SPARK-29318
> URL: https://issues.apache.org/jira/browse/SPARK-29318
> Project: Spark
> Issue Type: Sub-task
> Components: Build
>Affects Versions: 3.0.0
>Reporter: Dongjoon Hyun
>Priority: Major
>
> {code}
> orc-core-1.5.6-nohive.jar
> orc-mapreduce-1.5.6-nohive.jar
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-29308) dev/deps/spark-deps-hadoop-3.2 orc jar is incorrect
[ https://issues.apache.org/jira/browse/SPARK-29308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29308: -- Parent: SPARK-23710 Issue Type: Sub-task (was: Task) > dev/deps/spark-deps-hadoop-3.2 orc jar is incorrect > > > Key: SPARK-29308 > URL: https://issues.apache.org/jira/browse/SPARK-29308 > Project: Spark > Issue Type: Sub-task > Components: Build >Affects Versions: 3.0.0 >Reporter: angerszhu >Priority: Major > > In hadoop 3.2, orc.classfier is empty. > https://github.com/apache/spark/blob/d841b33ba3a9b0504597dbccd4b0d11fa810abf3/pom.xml#L2924 > here is incorrect. > https://github.com/apache/spark/blob/101839054276bfd52fdc29a98ffbf8e5c0383426/dev/deps/spark-deps-hadoop-3.2#L181-L182 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Closed] (SPARK-29318) orc library is incorrect in dev/deps/spark-deps-hadoop-3.2
[
https://issues.apache.org/jira/browse/SPARK-29318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun closed SPARK-29318.
-
> orc library is incorrect in dev/deps/spark-deps-hadoop-3.2
> ---
>
> Key: SPARK-29318
> URL: https://issues.apache.org/jira/browse/SPARK-29318
> Project: Spark
> Issue Type: Sub-task
> Components: Build
>Affects Versions: 3.0.0
>Reporter: Dongjoon Hyun
>Priority: Major
>
> {code}
> orc-core-1.5.6-nohive.jar
> orc-mapreduce-1.5.6-nohive.jar
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-29318) orc library is incorrect in dev/deps/spark-deps-hadoop-3.2
[
https://issues.apache.org/jira/browse/SPARK-29318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun resolved SPARK-29318.
---
Resolution: Duplicate
> orc library is incorrect in dev/deps/spark-deps-hadoop-3.2
> ---
>
> Key: SPARK-29318
> URL: https://issues.apache.org/jira/browse/SPARK-29318
> Project: Spark
> Issue Type: Sub-task
> Components: Build
>Affects Versions: 3.0.0
>Reporter: Dongjoon Hyun
>Priority: Major
>
> {code}
> orc-core-1.5.6-nohive.jar
> orc-mapreduce-1.5.6-nohive.jar
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-29318) orc library is incorrect in dev/deps/spark-deps-hadoop-3.2
[
https://issues.apache.org/jira/browse/SPARK-29318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942383#comment-16942383
]
Dongjoon Hyun commented on SPARK-29318:
---
Oops~
> orc library is incorrect in dev/deps/spark-deps-hadoop-3.2
> ---
>
> Key: SPARK-29318
> URL: https://issues.apache.org/jira/browse/SPARK-29318
> Project: Spark
> Issue Type: Sub-task
> Components: Build
>Affects Versions: 3.0.0
>Reporter: Dongjoon Hyun
>Priority: Major
>
> {code}
> orc-core-1.5.6-nohive.jar
> orc-mapreduce-1.5.6-nohive.jar
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-28759) Upgrade scala-maven-plugin to 4.2.0
[ https://issues.apache.org/jira/browse/SPARK-28759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942381#comment-16942381 ] Dongjoon Hyun edited comment on SPARK-28759 at 10/1/19 11:46 PM: - I didn't notice(realize) that. Where did you hit that, [~tgraves]? was (Author: dongjoon): I didn't notice that. Where did you hit that, [~tgraves]? > Upgrade scala-maven-plugin to 4.2.0 > --- > > Key: SPARK-28759 > URL: https://issues.apache.org/jira/browse/SPARK-28759 > Project: Spark > Issue Type: Sub-task > Components: Build >Affects Versions: 3.0.0 >Reporter: Dongjoon Hyun >Assignee: Hyukjin Kwon >Priority: Major > Fix For: 3.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28759) Upgrade scala-maven-plugin to 4.2.0
[ https://issues.apache.org/jira/browse/SPARK-28759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942381#comment-16942381 ] Dongjoon Hyun commented on SPARK-28759: --- I didn't notice that. Where did you hit that, [~tgraves]? > Upgrade scala-maven-plugin to 4.2.0 > --- > > Key: SPARK-28759 > URL: https://issues.apache.org/jira/browse/SPARK-28759 > Project: Spark > Issue Type: Sub-task > Components: Build >Affects Versions: 3.0.0 >Reporter: Dongjoon Hyun >Assignee: Hyukjin Kwon >Priority: Major > Fix For: 3.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-29097) Spark driver memory exceeded the storage memory
[ https://issues.apache.org/jira/browse/SPARK-29097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942339#comment-16942339 ] George Papa commented on SPARK-29097: - I think that SPARK-29055 resolved the increasing of the Storage Memory, so now I don't observe the high values of the driver memory. But, I don't know how the fixing SPARK-29055 secures the used memory < total memory in all the times. > Spark driver memory exceeded the storage memory > --- > > Key: SPARK-29097 > URL: https://issues.apache.org/jira/browse/SPARK-29097 > Project: Spark > Issue Type: Bug > Components: Block Manager, Spark Core, Web UI >Affects Versions: 2.3.3, 2.4.4 >Reporter: George Papa >Priority: Minor > > In Spark UI, the driver used storage memory (2.1GB) exceeds the total > available memory (1.5GB). When I use the same application with Spark 2.1.1 I > don't have this behavior. > > !https://i.stack.imgur.com/4gFYk.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-29097) Spark driver memory exceeded the storage memory
[ https://issues.apache.org/jira/browse/SPARK-29097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942332#comment-16942332 ] Jungtaek Lim commented on SPARK-29097: -- Could you check SPARK-29055 also resolves this? If then please close this as "Duplicated". > Spark driver memory exceeded the storage memory > --- > > Key: SPARK-29097 > URL: https://issues.apache.org/jira/browse/SPARK-29097 > Project: Spark > Issue Type: Bug > Components: Block Manager, Spark Core, Web UI >Affects Versions: 2.3.3, 2.4.4 >Reporter: George Papa >Priority: Minor > > In Spark UI, the driver used storage memory (2.1GB) exceeds the total > available memory (1.5GB). When I use the same application with Spark 2.1.1 I > don't have this behavior. > > !https://i.stack.imgur.com/4gFYk.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-28759) Upgrade scala-maven-plugin to 4.2.0
[ https://issues.apache.org/jira/browse/SPARK-28759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942308#comment-16942308 ] Thomas Graves edited comment on SPARK-28759 at 10/1/19 9:05 PM: [~dongjoon] [~hyukjin.kwon] Did either of you noticed that is seems incremental re-compiles seems to have been broken (rebuilding same thing again is taking forever because its recompiling all files again). I was rolling back to see what caused it and it seems like its this change. Wondering if you are seeing the same thing? was (Author: tgraves): [~dongjoon] [~hyukjin.kwon] Did either of you noticed that is seems incremental re-compiles seems to have been broken. I was rolling back to see what caused it and it seems like its this change. Wondering if you are seeing the same thing? > Upgrade scala-maven-plugin to 4.2.0 > --- > > Key: SPARK-28759 > URL: https://issues.apache.org/jira/browse/SPARK-28759 > Project: Spark > Issue Type: Sub-task > Components: Build >Affects Versions: 3.0.0 >Reporter: Dongjoon Hyun >Assignee: Hyukjin Kwon >Priority: Major > Fix For: 3.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28759) Upgrade scala-maven-plugin to 4.2.0
[ https://issues.apache.org/jira/browse/SPARK-28759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942308#comment-16942308 ] Thomas Graves commented on SPARK-28759: --- [~dongjoon] [~hyukjin.kwon] Did either of you noticed that is seems incremental re-compiles seems to have been broken. I was rolling back to see what caused it and it seems like its this change. Wondering if you are seeing the same thing? > Upgrade scala-maven-plugin to 4.2.0 > --- > > Key: SPARK-28759 > URL: https://issues.apache.org/jira/browse/SPARK-28759 > Project: Spark > Issue Type: Sub-task > Components: Build >Affects Versions: 3.0.0 >Reporter: Dongjoon Hyun >Assignee: Hyukjin Kwon >Priority: Major > Fix For: 3.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-29097) Spark driver memory exceeded the storage memory
[ https://issues.apache.org/jira/browse/SPARK-29097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] George Papa updated SPARK-29097: Affects Version/s: (was: 2.4.3) 2.3.3 > Spark driver memory exceeded the storage memory > --- > > Key: SPARK-29097 > URL: https://issues.apache.org/jira/browse/SPARK-29097 > Project: Spark > Issue Type: Bug > Components: Block Manager, Spark Core, Web UI >Affects Versions: 2.3.3, 2.4.4 >Reporter: George Papa >Priority: Minor > > In Spark UI, the driver used storage memory (2.1GB) exceeds the total > available memory (1.5GB). When I use the same application with Spark 2.1.1 I > don't have this behavior. > > !https://i.stack.imgur.com/4gFYk.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-29097) Spark driver memory exceeded the storage memory
[ https://issues.apache.org/jira/browse/SPARK-29097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] George Papa updated SPARK-29097: Component/s: Web UI > Spark driver memory exceeded the storage memory > --- > > Key: SPARK-29097 > URL: https://issues.apache.org/jira/browse/SPARK-29097 > Project: Spark > Issue Type: Bug > Components: Block Manager, Spark Core, Web UI >Affects Versions: 2.4.3, 2.4.4 >Reporter: George Papa >Priority: Minor > > In Spark UI, the driver used storage memory (2.1GB) exceeds the total > available memory (1.5GB). When I use the same application with Spark 2.1.1 I > don't have this behavior. > > !https://i.stack.imgur.com/4gFYk.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-29097) Spark driver memory exceeded the storage memory
[ https://issues.apache.org/jira/browse/SPARK-29097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] George Papa updated SPARK-29097: Component/s: Spark Core > Spark driver memory exceeded the storage memory > --- > > Key: SPARK-29097 > URL: https://issues.apache.org/jira/browse/SPARK-29097 > Project: Spark > Issue Type: Bug > Components: Spark Core, Web UI >Affects Versions: 2.4.3, 2.4.4 >Reporter: George Papa >Priority: Minor > > In Spark UI, the driver used storage memory (2.1GB) exceeds the total > available memory (1.5GB). When I use the same application with Spark 2.1.1 I > don't have this behavior. > > !https://i.stack.imgur.com/4gFYk.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-29097) Spark driver memory exceeded the storage memory
[ https://issues.apache.org/jira/browse/SPARK-29097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] George Papa updated SPARK-29097: Component/s: (was: Web UI) Block Manager > Spark driver memory exceeded the storage memory > --- > > Key: SPARK-29097 > URL: https://issues.apache.org/jira/browse/SPARK-29097 > Project: Spark > Issue Type: Bug > Components: Block Manager, Spark Core >Affects Versions: 2.4.3, 2.4.4 >Reporter: George Papa >Priority: Minor > > In Spark UI, the driver used storage memory (2.1GB) exceeds the total > available memory (1.5GB). When I use the same application with Spark 2.1.1 I > don't have this behavior. > > !https://i.stack.imgur.com/4gFYk.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-29321) Possible memory leak in Spark
[
https://issues.apache.org/jira/browse/SPARK-29321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942301#comment-16942301
]
George Papa commented on SPARK-29321:
-
[~kabhwan] I also update the description based on last comments. Now it's ready
for investigation :)
> Possible memory leak in Spark
> -
>
> Key: SPARK-29321
> URL: https://issues.apache.org/jira/browse/SPARK-29321
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.3.3
>Reporter: George Papa
>Priority: Major
>
> This issue is a clone of the (SPARK-29055). After Spark version 2.3.3,
> I{color:#172b4d} observe that the JVM memory is increasing slightly overtime.
> This behavior also affects the application performance because when I run my
> real application in testing environment, after a while the persisted
> dataframes stop fitting into the executors memory and I have spill to
> disk.{color}
> {color:#172b4d}JVM memory usage (based on htop command){color}
> ||Time||RES||SHR||MEM%||
> |1min|{color:#de350b}1349{color}|32724|1.5|
> |3min|{color:#de350b}1936{color}|32724|2.2|
> |5min|{color:#de350b}2506{color}|32724|2.6|
> |7min|{color:#de350b}2564{color}|32724|2.7|
> |9min|{color:#de350b}2584{color}|32724|2.7|
> |11min|{color:#de350b}2585{color}|32724|2.7|
> |13min|{color:#de350b}2592{color}|32724|2.7|
> |15min|{color:#de350b}2591{color}|32724|2.7|
> |17min|{color:#de350b}2591{color}|32724|2.7|
> |30min|{color:#de350b}2600{color}|32724|2.7|
> |1h|{color:#de350b}2618{color}|32724|2.7|
>
> *HOW TO REPRODUCE THIS BEHAVIOR:*
> Reproduce the above behavior, by running the snippet code (I prefer to run
> without any sleeping delay) and track the JVM memory with top or htop command.
> {code:java}
> import time
> import os
> from pyspark.sql import SparkSession
> from pyspark.sql import functions as F
> from pyspark.sql import types as T
> target_dir = "..."
> spark=SparkSession.builder.appName("DataframeCount").getOrCreate()
> while True:
> for f in os.listdir(target_dir):
> df = spark.read.load(target_dir + f, format="csv")
> print("Number of records: {0}".format(df.count()))
> time.sleep(15){code}
>
> *TESTED CASES WITH THE SAME BEHAVIOUR:*
> * I tested with default settings (spark-defaults.conf)
> * Add spark.cleaner.periodicGC.interval 1min (or less)
> * {{Turn spark.cleaner.referenceTracking.blocking}}=false
> * Run the application in cluster mode
> * Increase/decrease the resources of the executors and driver
> * I tested with extraJavaOptions in driver and executor -XX:+UseG1GC
> -XX:InitiatingHeapOccupancyPercent=35 -XX:ConcGCThreads=12
> * It is also tested with the Spark 2.4.4 (latest) and had the same behavior.
>
> *DEPENDENCIES*
> * Operation system: Ubuntu 16.04.3 LTS
> * Java: jdk1.8.0_131 (tested also with jdk1.8.0_221)
> * Python: Python 2.7.12
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Issue Comment Deleted] (SPARK-29321) Possible memory leak in Spark
[
https://issues.apache.org/jira/browse/SPARK-29321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
George Papa updated SPARK-29321:
Comment: was deleted
(was: I run the code in the snippet (I test it without any sleeping time, in
order to see the results faster) and I have recorded the JVM memory usage for
approximately 1 hour between Spark 2.4.4 and your branch with your patch.
Spark JVM memory with Spark 2.4.4:
||Time||RES||SHR||MEM%||
|1min|{color:#de350b}1349{color}|32724|1.5|
|3min|{color:#de350b}1936{color}|32724|2.2|
|5min|{color:#de350b}2506{color}|32724|2.6|
|7min|{color:#de350b}2564{color}|32724|2.7|
|9min|{color:#de350b}2584{color}|32724|2.7|
|11min|{color:#de350b}2585{color}|32724|2.7|
|13min|{color:#de350b}2592{color}|32724|2.7|
|15min|{color:#de350b}2591{color}|32724|2.7|
|17min|{color:#de350b}2591{color}|32724|2.7|
|30min|{color:#de350b}2600{color}|32724|2.7|
|1h|{color:#de350b}2618{color}|32724|2.7|
Spark JVM memory with Spark patch([GitHub Pull Request
#25973|https://github.com/apache/spark/pull/25973])
||Time||RES||SHR||MEM%||
|1min|{color:#de350b}1134{color}|25380|1.4|
|3min|{color:#de350b}1520{color}|25380|1.6|
|5min|{color:#de350b}1570{color}|25380|1.6|
|7min|{color:#de350b}1598{color}|25380|1.7|
|9min|{color:#de350b}1613{color}|25380|1.7|
|11min|{color:#de350b}1616{color}|25380|1.7|
|15min|{color:#de350b}1620{color}|25380|1.7|
|17min|{color:#de350b}1625{color}|25380|1.7|
|30min|{color:#de350b}1629{color}|25380|1.7|
|1h|{color:#de350b}1660{color}|25380|1.7|
As you can see the RES memory is slightly increasing in both cases overtime.
Also, when I tested with a real streaming application in a testing env after
hours, the persisted dataframes overflows the memory and spill to disk.
*NOTE:* You can easily reproduce the above behavior, by running the snippet
code (I prefer to run without any sleeping delay) and track the JVM memory with
top or htop command.
)
> Possible memory leak in Spark
> -
>
> Key: SPARK-29321
> URL: https://issues.apache.org/jira/browse/SPARK-29321
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.3.3
>Reporter: George Papa
>Priority: Major
>
> This issue is a clone of the (SPARK-29055). After Spark version 2.3.3,
> I{color:#172b4d} observe that the JVM memory is increasing slightly overtime.
> This behavior also affects the application performance because when I run my
> real application in testing environment, after a while the persisted
> dataframes stop fitting into the executors memory and I have spill to
> disk.{color}
> {color:#172b4d}JVM memory usage (based on htop command){color}
> ||Time||RES||SHR||MEM%||
> |1min|{color:#de350b}1349{color}|32724|1.5|
> |3min|{color:#de350b}1936{color}|32724|2.2|
> |5min|{color:#de350b}2506{color}|32724|2.6|
> |7min|{color:#de350b}2564{color}|32724|2.7|
> |9min|{color:#de350b}2584{color}|32724|2.7|
> |11min|{color:#de350b}2585{color}|32724|2.7|
> |13min|{color:#de350b}2592{color}|32724|2.7|
> |15min|{color:#de350b}2591{color}|32724|2.7|
> |17min|{color:#de350b}2591{color}|32724|2.7|
> |30min|{color:#de350b}2600{color}|32724|2.7|
> |1h|{color:#de350b}2618{color}|32724|2.7|
>
> *HOW TO REPRODUCE THIS BEHAVIOR:*
> Reproduce the above behavior, by running the snippet code (I prefer to run
> without any sleeping delay) and track the JVM memory with top or htop command.
> {code:java}
> import time
> import os
> from pyspark.sql import SparkSession
> from pyspark.sql import functions as F
> from pyspark.sql import types as T
> target_dir = "..."
> spark=SparkSession.builder.appName("DataframeCount").getOrCreate()
> while True:
> for f in os.listdir(target_dir):
> df = spark.read.load(target_dir + f, format="csv")
> print("Number of records: {0}".format(df.count()))
> time.sleep(15){code}
>
> *TESTED CASES WITH THE SAME BEHAVIOUR:*
> * I tested with default settings (spark-defaults.conf)
> * Add spark.cleaner.periodicGC.interval 1min (or less)
> * {{Turn spark.cleaner.referenceTracking.blocking}}=false
> * Run the application in cluster mode
> * Increase/decrease the resources of the executors and driver
> * I tested with extraJavaOptions in driver and executor -XX:+UseG1GC
> -XX:InitiatingHeapOccupancyPercent=35 -XX:ConcGCThreads=12
> * It is also tested with the Spark 2.4.4 (latest) and had the same behavior.
>
> *DEPENDENCIES*
> * Operation system: Ubuntu 16.04.3 LTS
> * Java: jdk1.8.0_131 (tested also with jdk1.8.0_221)
> * Python: Python 2.7.12
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-29318) orc library is incorrect in dev/deps/spark-deps-hadoop-3.2
[
https://issues.apache.org/jira/browse/SPARK-29318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942298#comment-16942298
]
L. C. Hsieh commented on SPARK-29318:
-
Seems duplicate to SPARK-29308.
> orc library is incorrect in dev/deps/spark-deps-hadoop-3.2
> ---
>
> Key: SPARK-29318
> URL: https://issues.apache.org/jira/browse/SPARK-29318
> Project: Spark
> Issue Type: Sub-task
> Components: Build
>Affects Versions: 3.0.0
>Reporter: Dongjoon Hyun
>Priority: Major
>
> {code}
> orc-core-1.5.6-nohive.jar
> orc-mapreduce-1.5.6-nohive.jar
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-29321) Possible memory leak in Spark
[
https://issues.apache.org/jira/browse/SPARK-29321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
George Papa updated SPARK-29321:
Description:
This issue is a clone of the (SPARK-29055). After Spark version 2.3.3,
I{color:#172b4d} observe that the JVM memory is increasing slightly overtime.
This behavior also affects the application performance because when I run my
real application in testing environment, after a while the persisted dataframes
stop fitting into the executors memory and I have spill to disk.{color}
{color:#172b4d}JVM memory usage (based on htop command){color}
||Time||RES||SHR||MEM%||
|1min|{color:#de350b}1349{color}|32724|1.5|
|3min|{color:#de350b}1936{color}|32724|2.2|
|5min|{color:#de350b}2506{color}|32724|2.6|
|7min|{color:#de350b}2564{color}|32724|2.7|
|9min|{color:#de350b}2584{color}|32724|2.7|
|11min|{color:#de350b}2585{color}|32724|2.7|
|13min|{color:#de350b}2592{color}|32724|2.7|
|15min|{color:#de350b}2591{color}|32724|2.7|
|17min|{color:#de350b}2591{color}|32724|2.7|
|30min|{color:#de350b}2600{color}|32724|2.7|
|1h|{color:#de350b}2618{color}|32724|2.7|
*HOW TO REPRODUCE THIS BEHAVIOR:*
Reproduce the above behavior, by running the snippet code (I prefer to run
without any sleeping delay) and track the JVM memory with top or htop command.
{code:java}
import time
import os
from pyspark.sql import SparkSession
from pyspark.sql import functions as F
from pyspark.sql import types as T
target_dir = "..."
spark=SparkSession.builder.appName("DataframeCount").getOrCreate()
while True:
for f in os.listdir(target_dir):
df = spark.read.load(target_dir + f, format="csv")
print("Number of records: {0}".format(df.count()))
time.sleep(15){code}
*TESTED CASES WITH THE SAME BEHAVIOUR:*
* I tested with default settings (spark-defaults.conf)
* Add spark.cleaner.periodicGC.interval 1min (or less)
* {{Turn spark.cleaner.referenceTracking.blocking}}=false
* Run the application in cluster mode
* Increase/decrease the resources of the executors and driver
* I tested with extraJavaOptions in driver and executor -XX:+UseG1GC
-XX:InitiatingHeapOccupancyPercent=35 -XX:ConcGCThreads=12
* It is also tested with the Spark 2.4.4 (latest) and had the same behavior.
*DEPENDENCIES*
* Operation system: Ubuntu 16.04.3 LTS
* Java: jdk1.8.0_131 (tested also with jdk1.8.0_221)
* Python: Python 2.7.12
was:
This issue is a clone of the (SPARK-29055). After Spark version 2.3.3,
I{color:#172b4d} observe that the JVM memory is increasing slightly overtime.
This behavior also affects the application performance because when I run my
real application in testing environment, after a while the persisted dataframes
stop fitting into the executors memory and I have spill to disk.{color}
*HOW TO REPRODUCE THIS BEHAVIOR:*
Reproduce the above behavior, by running the snippet code (I prefer to run
without any sleeping delay) and track the JVM memory with top or htop command.
{code:java}
import time
import os
from pyspark.sql import SparkSession
from pyspark.sql import functions as F
from pyspark.sql import types as T
target_dir = "..."
spark=SparkSession.builder.appName("DataframeCount").getOrCreate()
while True:
for f in os.listdir(target_dir):
df = spark.read.load(target_dir + f, format="csv")
print("Number of records: {0}".format(df.count()))
time.sleep(15){code}
*TESTED CASES WITH THE SAME BEHAVIOUR:*
* I tested with default settings (spark-defaults.conf)
* Add spark.cleaner.periodicGC.interval 1min (or less)
* {{Turn spark.cleaner.referenceTracking.blocking}}=false
* Run the application in cluster mode
* Increase/decrease the resources of the executors and driver
* I tested with extraJavaOptions in driver and executor -XX:+UseG1GC
-XX:InitiatingHeapOccupancyPercent=35 -XX:ConcGCThreads=12
* It is also tested with the Spark 2.4.4 (latest) and had the same behavior.
*DEPENDENCIES*
* Operation system: Ubuntu 16.04.3 LTS
* Java: jdk1.8.0_131 (tested also with jdk1.8.0_221)
* Python: Python 2.7.12
> Possible memory leak in Spark
> -
>
> Key: SPARK-29321
> URL: https://issues.apache.org/jira/browse/SPARK-29321
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.3.3
>Reporter: George Papa
>Priority: Major
>
> This issue is a clone of the (SPARK-29055). After Spark version 2.3.3,
> I{color:#172b4d} observe that the JVM memory is increasing slightly overtime.
> This behavior also affects the application performance because when I run my
> real application in testing environment, after a while the persisted
> dataframes stop fitting into the executors memory and I have spill to
> disk.{color}
> {color:#172b4d}JVM memory usage (based on htop command){color}
> ||Time||RES||SHR||MEM%||
>
[jira] [Updated] (SPARK-29321) Possible memory leak in Spark
[
https://issues.apache.org/jira/browse/SPARK-29321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
George Papa updated SPARK-29321:
Description:
This issue is a clone of the (SPARK-29055). After Spark version 2.3.3,
I{color:#172b4d} observe that the JVM memory is increasing slightly overtime.
This behavior also affects the application performance because when I run my
real application in testing environment, after a while the persisted dataframes
stop fitting into the executors memory and I have spill to disk.{color}
*HOW TO REPRODUCE THIS BEHAVIOR:*
Reproduce the above behavior, by running the snippet code (I prefer to run
without any sleeping delay) and track the JVM memory with top or htop command.
{code:java}
import time
import os
from pyspark.sql import SparkSession
from pyspark.sql import functions as F
from pyspark.sql import types as T
target_dir = "..."
spark=SparkSession.builder.appName("DataframeCount").getOrCreate()
while True:
for f in os.listdir(target_dir):
df = spark.read.load(target_dir + f, format="csv")
print("Number of records: {0}".format(df.count()))
time.sleep(15){code}
*TESTED CASES WITH THE SAME BEHAVIOUR:*
* I tested with default settings (spark-defaults.conf)
* Add spark.cleaner.periodicGC.interval 1min (or less)
* {{Turn spark.cleaner.referenceTracking.blocking}}=false
* Run the application in cluster mode
* Increase/decrease the resources of the executors and driver
* I tested with extraJavaOptions in driver and executor -XX:+UseG1GC
-XX:InitiatingHeapOccupancyPercent=35 -XX:ConcGCThreads=12
* It is also tested with the Spark 2.4.4 (latest) and had the same behavior.
*DEPENDENCIES*
* Operation system: Ubuntu 16.04.3 LTS
* Java: jdk1.8.0_131 (tested also with jdk1.8.0_221)
* Python: Python 2.7.12
was:
This issue is a clone of the (SPARK-29055). After Spark version 2.3.3,
I{color:#172b4d} observe that the JVM memory is increasing slightly overtime.
This behavior also affects the application performance because the persisted
dataframes don't fit in the executors memory and I have spill to disk.{color}
*HOW TO REPRODUCE THIS BEHAVIOR:*
Reproduce the above behavior, by running the snippet code (I prefer to run
without any sleeping delay) and track the JVM memory with top or htop command
{code:java}
import time
import os
from pyspark.sql import SparkSession
from pyspark.sql import functions as F
from pyspark.sql import types as T
target_dir = "..."
spark=SparkSession.builder.appName("DataframeCount").getOrCreate()
while True:
for f in os.listdir(target_dir):
df = spark.read.load(target_dir + f, format="csv")
print("Number of records: {0}".format(df.count()))
time.sleep(15){code}
*TESTED CASES WITH THE SAME BEHAVIOUR:*
* I tested with default settings (spark-defaults.conf)
* Add spark.cleaner.periodicGC.interval 1min (or less)
* {{Turn spark.cleaner.referenceTracking.blocking}}=false
* Run the application in cluster mode
* Increase/decrease the resources of the executors and driver
* I tested with extraJavaOptions in driver and executor -XX:+UseG1GC
-XX:InitiatingHeapOccupancyPercent=35 -XX:ConcGCThreads=12
*DEPENDENCIES*
* Operation system: Ubuntu 16.04.3 LTS
* Java: jdk1.8.0_131 (tested also with jdk1.8.0_221)
* Python: Python 2.7.12
> Possible memory leak in Spark
> -
>
> Key: SPARK-29321
> URL: https://issues.apache.org/jira/browse/SPARK-29321
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.3.3
>Reporter: George Papa
>Priority: Major
>
> This issue is a clone of the (SPARK-29055). After Spark version 2.3.3,
> I{color:#172b4d} observe that the JVM memory is increasing slightly overtime.
> This behavior also affects the application performance because when I run my
> real application in testing environment, after a while the persisted
> dataframes stop fitting into the executors memory and I have spill to
> disk.{color}
> *HOW TO REPRODUCE THIS BEHAVIOR:*
> Reproduce the above behavior, by running the snippet code (I prefer to run
> without any sleeping delay) and track the JVM memory with top or htop command.
> {code:java}
> import time
> import os
> from pyspark.sql import SparkSession
> from pyspark.sql import functions as F
> from pyspark.sql import types as T
> target_dir = "..."
> spark=SparkSession.builder.appName("DataframeCount").getOrCreate()
> while True:
> for f in os.listdir(target_dir):
> df = spark.read.load(target_dir + f, format="csv")
> print("Number of records: {0}".format(df.count()))
> time.sleep(15){code}
>
> *TESTED CASES WITH THE SAME BEHAVIOUR:*
> * I tested with default settings (spark-defaults.conf)
> * Add spark.cleaner.periodicGC.interval 1min (or less)
> * {{Turn
[jira] [Updated] (SPARK-29321) Possible memory leak in Spark
[
https://issues.apache.org/jira/browse/SPARK-29321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
George Papa updated SPARK-29321:
Description:
This issue is a clone of the (SPARK-29055). After Spark version 2.3.3,
I{color:#172b4d} observe that the JVM memory is increasing slightly overtime.
This behavior also affects the application performance because the persisted
dataframes don't fit in the executors memory and I have spill to disk.{color}
*HOW TO REPRODUCE THIS BEHAVIOR:*
Reproduce the above behavior, by running the snippet code (I prefer to run
without any sleeping delay) and track the JVM memory with top or htop command
{code:java}
import time
import os
from pyspark.sql import SparkSession
from pyspark.sql import functions as F
from pyspark.sql import types as T
target_dir = "..."
spark=SparkSession.builder.appName("DataframeCount").getOrCreate()
while True:
for f in os.listdir(target_dir):
df = spark.read.load(target_dir + f, format="csv")
print("Number of records: {0}".format(df.count()))
time.sleep(15){code}
*TESTED CASES WITH THE SAME BEHAVIOUR:*
* I tested with default settings (spark-defaults.conf)
* Add spark.cleaner.periodicGC.interval 1min (or less)
* {{Turn spark.cleaner.referenceTracking.blocking}}=false
* Run the application in cluster mode
* Increase/decrease the resources of the executors and driver
* I tested with extraJavaOptions in driver and executor -XX:+UseG1GC
-XX:InitiatingHeapOccupancyPercent=35 -XX:ConcGCThreads=12
*DEPENDENCIES*
* Operation system: Ubuntu 16.04.3 LTS
* Java: jdk1.8.0_131 (tested also with jdk1.8.0_221)
* Python: Python 2.7.12
was:
This issue is a clone of the (SPARK-29055). After Spark version 2.3.3,
I{color:#ff}{color:#172b4d} observe that the JVM memory is increasing
slightly overtime. This behavior also affects the application performance
because the persisted dataframes don't fit in the executors memory and I have
spill to disk.{color}
{color}
In more detail, the driver memory and executors memory have the same used
memory storage and after each iteration the storage memory is increasing. You
can reproduce this behavior by running the following snippet code. The
following example, is very simple, without any dataframe persistence, but the
memory consumption is not stable as it was in former Spark versions
(Specifically until Spark 2.3.2).
Also, I tested with Spark streaming and structured streaming API and I had the
same behavior. I tested with an existing application which reads from Kafka
source and do some aggregations, persist dataframes and then unpersist them.
The persist and unpersist it works correct, I see the dataframes in the storage
tab in Spark UI and after the unpersist, all dataframe have removed. But, after
the unpersist the executors memory is not zero, BUT has the same value with the
driver memory. This behavior also affects the application performance because
the memory of the executors is increasing as the driver increasing and after a
while the persisted dataframes are not fit in the executors memory and I have
spill to disk.
Another error which I had after a long running, was
{color:#ff}java.lang.OutOfMemoryError: GC overhead limit exceeded, but I
don't know if its relevant with the above behavior or not.{color}
*HOW TO REPRODUCE THIS BEHAVIOR:*
Create a very simple application(streaming count_file.py) in order to reproduce
this behavior. This application reads CSV files from a directory and count the
rows.
{code:java}
import time
import os
from pyspark.sql import SparkSession
from pyspark.sql import functions as F
from pyspark.sql import types as T
target_dir = "..."
spark=SparkSession.builder.appName("DataframeCount").getOrCreate()
while True:
for f in os.listdir(target_dir):
df = spark.read.load(target_dir + f, format="csv")
print("Number of records: {0}".format(df.count()))
time.sleep(15){code}
Submit code:
{code:java}
spark-submit
--master spark://xxx.xxx.xx.xxx
--deploy-mode client
--executor-memory 4g
--executor-cores 3
streaming count_file.py
{code}
*TESTED CASES WITH THE SAME BEHAVIOUR:*
* I tested with default settings (spark-defaults.conf)
* Add spark.cleaner.periodicGC.interval 1min (or less)
* {{Turn spark.cleaner.referenceTracking.blocking}}=false
* Run the application in cluster mode
* Increase/decrease the resources of the executors and driver
* I tested with extraJavaOptions in driver and executor -XX:+UseG1GC
-XX:InitiatingHeapOccupancyPercent=35 -XX:ConcGCThreads=12
*DEPENDENCIES*
* Operation system: Ubuntu 16.04.3 LTS
* Java: jdk1.8.0_131 (tested also with jdk1.8.0_221)
* Python: Python 2.7.12
*NOTE:* In Spark 2.1.1 the driver memory consumption (Storage Memory tab) was
extremely low and after the run of ContextCleaner and BlockManager the memory
was decreasing.
> Possible memory
[jira] [Updated] (SPARK-29321) Possible memory leak in Spark
[
https://issues.apache.org/jira/browse/SPARK-29321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
George Papa updated SPARK-29321:
Description:
This issue is a clone of the (SPARK-29055). After Spark version 2.3.3,
I{color:#ff}{color:#172b4d} observe that the JVM memory is increasing
slightly overtime. This behavior also affects the application performance
because the persisted dataframes don't fit in the executors memory and I have
spill to disk.{color}
{color}
In more detail, the driver memory and executors memory have the same used
memory storage and after each iteration the storage memory is increasing. You
can reproduce this behavior by running the following snippet code. The
following example, is very simple, without any dataframe persistence, but the
memory consumption is not stable as it was in former Spark versions
(Specifically until Spark 2.3.2).
Also, I tested with Spark streaming and structured streaming API and I had the
same behavior. I tested with an existing application which reads from Kafka
source and do some aggregations, persist dataframes and then unpersist them.
The persist and unpersist it works correct, I see the dataframes in the storage
tab in Spark UI and after the unpersist, all dataframe have removed. But, after
the unpersist the executors memory is not zero, BUT has the same value with the
driver memory. This behavior also affects the application performance because
the memory of the executors is increasing as the driver increasing and after a
while the persisted dataframes are not fit in the executors memory and I have
spill to disk.
Another error which I had after a long running, was
{color:#ff}java.lang.OutOfMemoryError: GC overhead limit exceeded, but I
don't know if its relevant with the above behavior or not.{color}
*HOW TO REPRODUCE THIS BEHAVIOR:*
Create a very simple application(streaming count_file.py) in order to reproduce
this behavior. This application reads CSV files from a directory and count the
rows.
{code:java}
import time
import os
from pyspark.sql import SparkSession
from pyspark.sql import functions as F
from pyspark.sql import types as T
target_dir = "..."
spark=SparkSession.builder.appName("DataframeCount").getOrCreate()
while True:
for f in os.listdir(target_dir):
df = spark.read.load(target_dir + f, format="csv")
print("Number of records: {0}".format(df.count()))
time.sleep(15){code}
Submit code:
{code:java}
spark-submit
--master spark://xxx.xxx.xx.xxx
--deploy-mode client
--executor-memory 4g
--executor-cores 3
streaming count_file.py
{code}
*TESTED CASES WITH THE SAME BEHAVIOUR:*
* I tested with default settings (spark-defaults.conf)
* Add spark.cleaner.periodicGC.interval 1min (or less)
* {{Turn spark.cleaner.referenceTracking.blocking}}=false
* Run the application in cluster mode
* Increase/decrease the resources of the executors and driver
* I tested with extraJavaOptions in driver and executor -XX:+UseG1GC
-XX:InitiatingHeapOccupancyPercent=35 -XX:ConcGCThreads=12
*DEPENDENCIES*
* Operation system: Ubuntu 16.04.3 LTS
* Java: jdk1.8.0_131 (tested also with jdk1.8.0_221)
* Python: Python 2.7.12
*NOTE:* In Spark 2.1.1 the driver memory consumption (Storage Memory tab) was
extremely low and after the run of ContextCleaner and BlockManager the memory
was decreasing.
was:
I used Spark 2.1.1 and I upgraded into new versions. After Spark version 2.3.3,
I observed from Spark UI that the driver memory is{color:#ff} increasing
continuously.{color}
In more detail, the driver memory and executors memory have the same used
memory storage and after each iteration the storage memory is increasing. You
can reproduce this behavior by running the following snippet code. The
following example, is very simple, without any dataframe persistence, but the
memory consumption is not stable as it was in former Spark versions
(Specifically until Spark 2.3.2).
Also, I tested with Spark streaming and structured streaming API and I had the
same behavior. I tested with an existing application which reads from Kafka
source and do some aggregations, persist dataframes and then unpersist them.
The persist and unpersist it works correct, I see the dataframes in the storage
tab in Spark UI and after the unpersist, all dataframe have removed. But, after
the unpersist the executors memory is not zero, BUT has the same value with the
driver memory. This behavior also affects the application performance because
the memory of the executors is increasing as the driver increasing and after a
while the persisted dataframes are not fit in the executors memory and I have
spill to disk.
Another error which I had after a long running, was
{color:#ff}java.lang.OutOfMemoryError: GC overhead limit exceeded, but I
don't know if its relevant with the above behavior or not.{color}
*HOW TO
[jira] [Commented] (SPARK-29244) ArrayIndexOutOfBoundsException on TaskCompletionListener during releasing of memory blocks
[
https://issues.apache.org/jira/browse/SPARK-29244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942295#comment-16942295
]
Viacheslav Tradunsky commented on SPARK-29244:
--
Thank you!
> ArrayIndexOutOfBoundsException on TaskCompletionListener during releasing of
> memory blocks
> --
>
> Key: SPARK-29244
> URL: https://issues.apache.org/jira/browse/SPARK-29244
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.4.0
> Environment: Release label:emr-5.20.0
> Hadoop distribution:Amazon 2.8.5
> Applications:Livy 0.5.0, Spark 2.4.0
>Reporter: Viacheslav Tradunsky
>Assignee: L. C. Hsieh
>Priority: Major
> Fix For: 2.4.5, 3.0.0
>
> Attachments: executor_oom.txt
>
>
> At the end of task completion an exception happened:
> {code:java}
> 19/09/25 09:03:58 ERROR TaskContextImpl: Error in
> TaskCompletionListener19/09/25 09:03:58 ERROR TaskContextImpl: Error in
> TaskCompletionListenerjava.lang.ArrayIndexOutOfBoundsException: -3 at
> org.apache.spark.memory.TaskMemoryManager.freePage(TaskMemoryManager.java:333)
> at org.apache.spark.memory.MemoryConsumer.freePage(MemoryConsumer.java:130)
> at org.apache.spark.memory.MemoryConsumer.freeArray(MemoryConsumer.java:108)
> at org.apache.spark.unsafe.map.BytesToBytesMap.free(BytesToBytesMap.java:803)
> at
> org.apache.spark.sql.execution.UnsafeFixedWidthAggregationMap.free(UnsafeFixedWidthAggregationMap.java:225)
> at
> org.apache.spark.sql.execution.UnsafeFixedWidthAggregationMap.lambda$new$0(UnsafeFixedWidthAggregationMap.java:111)
> at
> org.apache.spark.TaskContextImpl$$anonfun$markTaskCompleted$1.apply(TaskContextImpl.scala:117)
> at
> org.apache.spark.TaskContextImpl$$anonfun$markTaskCompleted$1.apply(TaskContextImpl.scala:117)
> at
> org.apache.spark.TaskContextImpl$$anonfun$invokeListeners$1.apply(TaskContextImpl.scala:130)
> at
> org.apache.spark.TaskContextImpl$$anonfun$invokeListeners$1.apply(TaskContextImpl.scala:128)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at
> org.apache.spark.TaskContextImpl.invokeListeners(TaskContextImpl.scala:128)
> at
> org.apache.spark.TaskContextImpl.markTaskCompleted(TaskContextImpl.scala:116)
> at org.apache.spark.scheduler.Task.run(Task.scala:131) at
> org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402)
> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360) at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408) at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
>
> Important to note, that before this one, there was OOM of allocating some
> pages. It looks like everything related to each other, but on OOM the whole
> flow goes abnormally, so no resources are fried correctly.
> {code:java}
> java.lang.NullPointerExceptionjava.lang.NullPointerException at
> org.apache.spark.util.collection.unsafe.sort.UnsafeInMemorySorter.getMemoryUsage(UnsafeInMemorySorter.java:208)
> at
> org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.getMemoryUsage(UnsafeExternalSorter.java:249)
> at
> org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.updatePeakMemoryUsed(UnsafeExternalSorter.java:253)
> at
> org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.freeMemory(UnsafeExternalSorter.java:296)
> at
> org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.cleanupResources(UnsafeExternalSorter.java:328)
> at
> org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.lambda$new$0(UnsafeExternalSorter.java:178)
> at
> org.apache.spark.TaskContextImpl$$anonfun$markTaskCompleted$1.apply(TaskContextImpl.scala:117)
> at
> org.apache.spark.TaskContextImpl$$anonfun$markTaskCompleted$1.apply(TaskContextImpl.scala:117)
> at
> org.apache.spark.TaskContextImpl$$anonfun$invokeListeners$1.apply(TaskContextImpl.scala:130)
> at
> org.apache.spark.TaskContextImpl$$anonfun$invokeListeners$1.apply(TaskContextImpl.scala:128)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at
> org.apache.spark.TaskContextImpl.invokeListeners(TaskContextImpl.scala:128)
> at
> org.apache.spark.TaskContextImpl.markTaskCompleted(TaskContextImpl.scala:116)
> at org.apache.spark.scheduler.Task.run(Task.scala:131) at
>
[jira] [Commented] (SPARK-29321) Possible memory leak in Spark
[
https://issues.apache.org/jira/browse/SPARK-29321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942293#comment-16942293
]
George Papa commented on SPARK-29321:
-
I run the code in the snippet (I test it without any sleeping time, in order to
see the results faster) and I have recorded the JVM memory usage for
approximately 1 hour between Spark 2.4.4 and your branch with your patch.
Spark JVM memory with Spark 2.4.4:
||Time||RES||SHR||MEM%||
|1min|{color:#de350b}1349{color}|32724|1.5|
|3min|{color:#de350b}1936{color}|32724|2.2|
|5min|{color:#de350b}2506{color}|32724|2.6|
|7min|{color:#de350b}2564{color}|32724|2.7|
|9min|{color:#de350b}2584{color}|32724|2.7|
|11min|{color:#de350b}2585{color}|32724|2.7|
|13min|{color:#de350b}2592{color}|32724|2.7|
|15min|{color:#de350b}2591{color}|32724|2.7|
|17min|{color:#de350b}2591{color}|32724|2.7|
|30min|{color:#de350b}2600{color}|32724|2.7|
|1h|{color:#de350b}2618{color}|32724|2.7|
Spark JVM memory with Spark patch([GitHub Pull Request
#25973|https://github.com/apache/spark/pull/25973])
||Time||RES||SHR||MEM%||
|1min|{color:#de350b}1134{color}|25380|1.4|
|3min|{color:#de350b}1520{color}|25380|1.6|
|5min|{color:#de350b}1570{color}|25380|1.6|
|7min|{color:#de350b}1598{color}|25380|1.7|
|9min|{color:#de350b}1613{color}|25380|1.7|
|11min|{color:#de350b}1616{color}|25380|1.7|
|15min|{color:#de350b}1620{color}|25380|1.7|
|17min|{color:#de350b}1625{color}|25380|1.7|
|30min|{color:#de350b}1629{color}|25380|1.7|
|1h|{color:#de350b}1660{color}|25380|1.7|
As you can see the RES memory is slightly increasing in both cases overtime.
Also, when I tested with a real streaming application in a testing env after
hours, the persisted dataframes overflows the memory and spill to disk.
*NOTE:* You can easily reproduce the above behavior, by running the snippet
code (I prefer to run without any sleeping delay) and track the JVM memory with
top or htop command.
> Possible memory leak in Spark
> -
>
> Key: SPARK-29321
> URL: https://issues.apache.org/jira/browse/SPARK-29321
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.3.3
>Reporter: George Papa
>Priority: Major
>
> I used Spark 2.1.1 and I upgraded into new versions. After Spark version
> 2.3.3, I observed from Spark UI that the driver memory is{color:#ff}
> increasing continuously.{color}
> In more detail, the driver memory and executors memory have the same used
> memory storage and after each iteration the storage memory is increasing. You
> can reproduce this behavior by running the following snippet code. The
> following example, is very simple, without any dataframe persistence, but the
> memory consumption is not stable as it was in former Spark versions
> (Specifically until Spark 2.3.2).
> Also, I tested with Spark streaming and structured streaming API and I had
> the same behavior. I tested with an existing application which reads from
> Kafka source and do some aggregations, persist dataframes and then unpersist
> them. The persist and unpersist it works correct, I see the dataframes in the
> storage tab in Spark UI and after the unpersist, all dataframe have removed.
> But, after the unpersist the executors memory is not zero, BUT has the same
> value with the driver memory. This behavior also affects the application
> performance because the memory of the executors is increasing as the driver
> increasing and after a while the persisted dataframes are not fit in the
> executors memory and I have spill to disk.
> Another error which I had after a long running, was
> {color:#ff}java.lang.OutOfMemoryError: GC overhead limit exceeded, but I
> don't know if its relevant with the above behavior or not.{color}
>
> *HOW TO REPRODUCE THIS BEHAVIOR:*
> Create a very simple application(streaming count_file.py) in order to
> reproduce this behavior. This application reads CSV files from a directory,
> count the rows and then remove the processed files.
> {code:java}
> import time
> import os
> from pyspark.sql import SparkSession
> from pyspark.sql import functions as F
> from pyspark.sql import types as T
> target_dir = "..."
> spark=SparkSession.builder.appName("DataframeCount").getOrCreate()
> while True:
> for f in os.listdir(target_dir):
> df = spark.read.load(target_dir + f, format="csv")
> print("Number of records: {0}".format(df.count()))
> time.sleep(15){code}
> Submit code:
> {code:java}
> spark-submit
> --master spark://xxx.xxx.xx.xxx
> --deploy-mode client
> --executor-memory 4g
> --executor-cores 3
> streaming count_file.py
> {code}
>
> *TESTED CASES WITH THE SAME BEHAVIOUR:*
> * I tested with default settings (spark-defaults.conf)
> * Add spark.cleaner.periodicGC.interval 1min (or less)
> * {{Turn
[jira] [Comment Edited] (SPARK-13446) Spark need to support reading data from Hive 2.0.0 metastore
[
https://issues.apache.org/jira/browse/SPARK-13446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942275#comment-16942275
]
JP Bordenave edited comment on SPARK-13446 at 10/1/19 8:17 PM:
---
1) ok i am back, after some internet issue, i make restore all hive 1.2.1
jars from spark 2.4.4
2) hive v2.3.3 get conflict mysql schema 2.3.0 with spark 2.4.4 because it use
1.2.1 schema
3) i make cp hive-site.xml into spark/conf and i disable schema check, it is
working fine under spark-shell
{{}}
{\{ hive.metastore.schema.verification}}
{\{ false}}
{{}}
{{But i doenst understand why spark 2.4.4 use old hive schema 1.2.1 ?(not
realy clear for me) ??? and why i must disable it for new hive version 2.3.3?}}
{noformat}
^
scala> spark.sql("show databases").show
++
|databaseName|
++
| default|
++
scala> spark.sql("show tables").show
19/10/01 21:44:07 WARN metastore.ObjectStore: Failed to get database
global_temp, returning NoSuchObjectException
++-+---+
|database|tableName|isTemporary|
++-+---+
| default| employee| false|
++-+---+
scala> spark.sql("select * from employee").show
+---+---+-+
| id| name| dept|
+---+---+-+
| 1| Allen| IT|
| 2|Mag|Sales|
| 14| Pierre| xXx|
| 1| Allen| IT|
| 3|Rob|Sales|
| 4| Dana| IT|
| 7| Pierre| xXx|
| 11| Pierre| xXx|
| 10| Pierre| xXx|
| 12| Pierre| xXx|
| 13| Pierre| xXx|
+---+---+-+
{noformat}
{{}}
{{}}
{{}}
was (Author: jpbordi):
1) ok i am back, after some internet issue, i make restore all hive 1.2.1
jars from spark 2.4.4
2) hive v2.3.3 get conflict mysql schema 2.3.0 with spark 2.4.4 because it use
1.2.1 schema
3) i make cp hive-site.xml into spark/conf and i disable schema check, it is
working fine under spark-shell
{{}}
{\{ hive.metastore.schema.verification}}
{\{ false}}
{{}}
{{But i doenst understand why spark 2.4.4 use old hive schema 1.2.1 ?(not
realy clear for me) ??? and why i must disable it ?}}
{noformat}
^
scala> spark.sql("show databases").show
++
|databaseName|
++
| default|
++
scala> spark.sql("show tables").show
19/10/01 21:44:07 WARN metastore.ObjectStore: Failed to get database
global_temp, returning NoSuchObjectException
++-+---+
|database|tableName|isTemporary|
++-+---+
| default| employee| false|
++-+---+
scala> spark.sql("select * from employee").show
+---+---+-+
| id| name| dept|
+---+---+-+
| 1| Allen| IT|
| 2|Mag|Sales|
| 14| Pierre| xXx|
| 1| Allen| IT|
| 3|Rob|Sales|
| 4| Dana| IT|
| 7| Pierre| xXx|
| 11| Pierre| xXx|
| 10| Pierre| xXx|
| 12| Pierre| xXx|
| 13| Pierre| xXx|
+---+---+-+
{noformat}
{{}}
{{}}
{{}}
> Spark need to support reading data from Hive 2.0.0 metastore
>
>
> Key: SPARK-13446
> URL: https://issues.apache.org/jira/browse/SPARK-13446
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 1.6.0
>Reporter: Lifeng Wang
>Assignee: Xiao Li
>Priority: Major
> Fix For: 2.2.0
>
>
> Spark provided HIveContext class to read data from hive metastore directly.
> While it only supports hive 1.2.1 version and older. Since hive 2.0.0 has
> released, it's better to upgrade to support Hive 2.0.0.
> {noformat}
> 16/02/23 02:35:02 INFO metastore: Trying to connect to metastore with URI
> thrift://hsw-node13:9083
> 16/02/23 02:35:02 INFO metastore: Opened a connection to metastore, current
> connections: 1
> 16/02/23 02:35:02 INFO metastore: Connected to metastore.
> Exception in thread "main" java.lang.NoSuchFieldError: HIVE_STATS_JDBC_TIMEOUT
> at
> org.apache.spark.sql.hive.HiveContext.configure(HiveContext.scala:473)
> at
> org.apache.spark.sql.hive.HiveContext.metadataHive$lzycompute(HiveContext.scala:192)
> at
> org.apache.spark.sql.hive.HiveContext.metadataHive(HiveContext.scala:185)
> at
> org.apache.spark.sql.hive.HiveContext$$anon$1.(HiveContext.scala:422)
> at
> org.apache.spark.sql.hive.HiveContext.catalog$lzycompute(HiveContext.scala:422)
> at
> org.apache.spark.sql.hive.HiveContext.catalog(HiveContext.scala:421)
> at org.apache.spark.sql.hive.HiveContext.catalog(HiveContext.scala:72)
> at
[jira] [Comment Edited] (SPARK-13446) Spark need to support reading data from Hive 2.0.0 metastore
[
https://issues.apache.org/jira/browse/SPARK-13446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942275#comment-16942275
]
JP Bordenave edited comment on SPARK-13446 at 10/1/19 8:14 PM:
---
1) ok i am back, after some internet issue, i make restore all hive 1.2.1
jars from spark 2.4.4
2) hive v2.3.3 get conflict mysql schema 2.3.0 with spark 2.4.4 because it use
1.2.1 schema
3) i make cp hive-site.xml into spark/conf and i disable schema check, it is
working fine under spark-shell
{{}}
{\{ hive.metastore.schema.verification}}
{\{ false}}
{{}}
{{But i doenst understand why spark 2.4.4 use old hive schema 1.2.1 ?(not
realy clear for me) ??? and why i must disable it ?}}
{noformat}
^
scala> spark.sql("show databases").show
++
|databaseName|
++
| default|
++
scala> spark.sql("show tables").show
19/10/01 21:44:07 WARN metastore.ObjectStore: Failed to get database
global_temp, returning NoSuchObjectException
++-+---+
|database|tableName|isTemporary|
++-+---+
| default| employee| false|
++-+---+
scala> spark.sql("select * from employee").show
+---+---+-+
| id| name| dept|
+---+---+-+
| 1| Allen| IT|
| 2|Mag|Sales|
| 14| Pierre| xXx|
| 1| Allen| IT|
| 3|Rob|Sales|
| 4| Dana| IT|
| 7| Pierre| xXx|
| 11| Pierre| xXx|
| 10| Pierre| xXx|
| 12| Pierre| xXx|
| 13| Pierre| xXx|
+---+---+-+
{noformat}
{{}}
{{}}
{{}}
was (Author: jpbordi):
1) ok i am back, after some internet issue, i make restore all hive 1.2.1
jars from spark 2.4.4
2) hive v2.3.3 get conflict mysql schema 2.3.0 with spark 2.4.4 because it use
1.2.1 schema
3) i make cp hive-site.xml into spark/conf and i disable schema check, it is
working fine under spark-shell
{{}}
{\{ hive.metastore.schema.verification}}
{\{ false}}
{{}}
{{But i doenst understand why spark 2.4.4 use old hive schema 1.2.1 ?(not
realy clear for me) ??? and why i must disable it}}
{noformat}
^
scala> spark.sql("show databases").show
++
|databaseName|
++
| default|
++
scala> spark.sql("show tables").show
19/10/01 21:44:07 WARN metastore.ObjectStore: Failed to get database
global_temp, returning NoSuchObjectException
++-+---+
|database|tableName|isTemporary|
++-+---+
| default| employee| false|
++-+---+
scala> spark.sql("select * from employee").show
+---+---+-+
| id| name| dept|
+---+---+-+
| 1| Allen| IT|
| 2|Mag|Sales|
| 14| Pierre| xXx|
| 1| Allen| IT|
| 3|Rob|Sales|
| 4| Dana| IT|
| 7| Pierre| xXx|
| 11| Pierre| xXx|
| 10| Pierre| xXx|
| 12| Pierre| xXx|
| 13| Pierre| xXx|
+---+---+-+
{noformat}
{{}}
{{}}
{{}}
> Spark need to support reading data from Hive 2.0.0 metastore
>
>
> Key: SPARK-13446
> URL: https://issues.apache.org/jira/browse/SPARK-13446
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 1.6.0
>Reporter: Lifeng Wang
>Assignee: Xiao Li
>Priority: Major
> Fix For: 2.2.0
>
>
> Spark provided HIveContext class to read data from hive metastore directly.
> While it only supports hive 1.2.1 version and older. Since hive 2.0.0 has
> released, it's better to upgrade to support Hive 2.0.0.
> {noformat}
> 16/02/23 02:35:02 INFO metastore: Trying to connect to metastore with URI
> thrift://hsw-node13:9083
> 16/02/23 02:35:02 INFO metastore: Opened a connection to metastore, current
> connections: 1
> 16/02/23 02:35:02 INFO metastore: Connected to metastore.
> Exception in thread "main" java.lang.NoSuchFieldError: HIVE_STATS_JDBC_TIMEOUT
> at
> org.apache.spark.sql.hive.HiveContext.configure(HiveContext.scala:473)
> at
> org.apache.spark.sql.hive.HiveContext.metadataHive$lzycompute(HiveContext.scala:192)
> at
> org.apache.spark.sql.hive.HiveContext.metadataHive(HiveContext.scala:185)
> at
> org.apache.spark.sql.hive.HiveContext$$anon$1.(HiveContext.scala:422)
> at
> org.apache.spark.sql.hive.HiveContext.catalog$lzycompute(HiveContext.scala:422)
> at
> org.apache.spark.sql.hive.HiveContext.catalog(HiveContext.scala:421)
> at org.apache.spark.sql.hive.HiveContext.catalog(HiveContext.scala:72)
> at org.apache.spark.sql.SQLContext.table(SQLContext.scala:739)
> at
[jira] [Comment Edited] (SPARK-13446) Spark need to support reading data from Hive 2.0.0 metastore
[
https://issues.apache.org/jira/browse/SPARK-13446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942275#comment-16942275
]
JP Bordenave edited comment on SPARK-13446 at 10/1/19 8:14 PM:
---
1) ok i am back, after some internet issue, i make restore all hive 1.2.1
jars from spark 2.4.4
2) hive v2.3.3 get conflict mysql schema 2.3.0 with spark 2.4.4 because it use
1.2.1 schema
3) i make cp hive-site.xml into spark/conf and i disable schema check, it is
working fine under spark-shell
{{}}
{\{ hive.metastore.schema.verification}}
{\{ false}}
{{}}
{{But i doenst understand why spark 2.4.4 use old hive schema 1.2.1 ?(not
realy clear for me) ??? and why i must disable it}}
{noformat}
^
scala> spark.sql("show databases").show
++
|databaseName|
++
| default|
++
scala> spark.sql("show tables").show
19/10/01 21:44:07 WARN metastore.ObjectStore: Failed to get database
global_temp, returning NoSuchObjectException
++-+---+
|database|tableName|isTemporary|
++-+---+
| default| employee| false|
++-+---+
scala> spark.sql("select * from employee").show
+---+---+-+
| id| name| dept|
+---+---+-+
| 1| Allen| IT|
| 2|Mag|Sales|
| 14| Pierre| xXx|
| 1| Allen| IT|
| 3|Rob|Sales|
| 4| Dana| IT|
| 7| Pierre| xXx|
| 11| Pierre| xXx|
| 10| Pierre| xXx|
| 12| Pierre| xXx|
| 13| Pierre| xXx|
+---+---+-+
{noformat}
{{}}
{{}}
{{}}
was (Author: jpbordi):
1) ok i am back, after some internet issue, i make restore all hive 1.2.1
jars from spark 2.4.4
2) hive v2.3.3 get conflict mysql schema 2.3.0 with spark 2.4.4 because it use
1.2.1 schema
3) i make cp hive-site.xml into spark/conf and i disable schema check, it is
working fine under spark-shell
{{}}
{\{ hive.metastore.schema.verification}}
{\{ false}}
{{}}
{{But i doenst understand why spark 2.4.4 use old hive schema 1.2.1 (not realy
clear for me)}}
{{}}
{noformat}
^
scala> spark.sql("show databases").show
++
|databaseName|
++
| default|
++
scala> spark.sql("show tables").show
19/10/01 21:44:07 WARN metastore.ObjectStore: Failed to get database
global_temp, returning NoSuchObjectException
++-+---+
|database|tableName|isTemporary|
++-+---+
| default| employee| false|
++-+---+
scala> spark.sql("select * from employee").show
+---+---+-+
| id| name| dept|
+---+---+-+
| 1| Allen| IT|
| 2|Mag|Sales|
| 14| Pierre| xXx|
| 1| Allen| IT|
| 3|Rob|Sales|
| 4| Dana| IT|
| 7| Pierre| xXx|
| 11| Pierre| xXx|
| 10| Pierre| xXx|
| 12| Pierre| xXx|
| 13| Pierre| xXx|
+---+---+-+
{noformat}
{{}}
{{}}
{{}}
> Spark need to support reading data from Hive 2.0.0 metastore
>
>
> Key: SPARK-13446
> URL: https://issues.apache.org/jira/browse/SPARK-13446
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 1.6.0
>Reporter: Lifeng Wang
>Assignee: Xiao Li
>Priority: Major
> Fix For: 2.2.0
>
>
> Spark provided HIveContext class to read data from hive metastore directly.
> While it only supports hive 1.2.1 version and older. Since hive 2.0.0 has
> released, it's better to upgrade to support Hive 2.0.0.
> {noformat}
> 16/02/23 02:35:02 INFO metastore: Trying to connect to metastore with URI
> thrift://hsw-node13:9083
> 16/02/23 02:35:02 INFO metastore: Opened a connection to metastore, current
> connections: 1
> 16/02/23 02:35:02 INFO metastore: Connected to metastore.
> Exception in thread "main" java.lang.NoSuchFieldError: HIVE_STATS_JDBC_TIMEOUT
> at
> org.apache.spark.sql.hive.HiveContext.configure(HiveContext.scala:473)
> at
> org.apache.spark.sql.hive.HiveContext.metadataHive$lzycompute(HiveContext.scala:192)
> at
> org.apache.spark.sql.hive.HiveContext.metadataHive(HiveContext.scala:185)
> at
> org.apache.spark.sql.hive.HiveContext$$anon$1.(HiveContext.scala:422)
> at
> org.apache.spark.sql.hive.HiveContext.catalog$lzycompute(HiveContext.scala:422)
> at
> org.apache.spark.sql.hive.HiveContext.catalog(HiveContext.scala:421)
> at org.apache.spark.sql.hive.HiveContext.catalog(HiveContext.scala:72)
> at org.apache.spark.sql.SQLContext.table(SQLContext.scala:739)
> at
[jira] [Commented] (SPARK-29321) Possible memory leak in Spark
[
https://issues.apache.org/jira/browse/SPARK-29321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942282#comment-16942282
]
Jungtaek Lim commented on SPARK-29321:
--
I'm sorry I meant your last comment not real last comment -
https://issues.apache.org/jira/browse/SPARK-29055?focusedCommentId=16941736=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16941736
I can't copy the format (table, etc) so it would be ideal if you could try edit
the comment and copy in text mode and paste here in text mode.
> Possible memory leak in Spark
> -
>
> Key: SPARK-29321
> URL: https://issues.apache.org/jira/browse/SPARK-29321
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.3.3
>Reporter: George Papa
>Priority: Major
>
> I used Spark 2.1.1 and I upgraded into new versions. After Spark version
> 2.3.3, I observed from Spark UI that the driver memory is{color:#ff}
> increasing continuously.{color}
> In more detail, the driver memory and executors memory have the same used
> memory storage and after each iteration the storage memory is increasing. You
> can reproduce this behavior by running the following snippet code. The
> following example, is very simple, without any dataframe persistence, but the
> memory consumption is not stable as it was in former Spark versions
> (Specifically until Spark 2.3.2).
> Also, I tested with Spark streaming and structured streaming API and I had
> the same behavior. I tested with an existing application which reads from
> Kafka source and do some aggregations, persist dataframes and then unpersist
> them. The persist and unpersist it works correct, I see the dataframes in the
> storage tab in Spark UI and after the unpersist, all dataframe have removed.
> But, after the unpersist the executors memory is not zero, BUT has the same
> value with the driver memory. This behavior also affects the application
> performance because the memory of the executors is increasing as the driver
> increasing and after a while the persisted dataframes are not fit in the
> executors memory and I have spill to disk.
> Another error which I had after a long running, was
> {color:#ff}java.lang.OutOfMemoryError: GC overhead limit exceeded, but I
> don't know if its relevant with the above behavior or not.{color}
>
> *HOW TO REPRODUCE THIS BEHAVIOR:*
> Create a very simple application(streaming count_file.py) in order to
> reproduce this behavior. This application reads CSV files from a directory,
> count the rows and then remove the processed files.
> {code:java}
> import time
> import os
> from pyspark.sql import SparkSession
> from pyspark.sql import functions as F
> from pyspark.sql import types as T
> target_dir = "..."
> spark=SparkSession.builder.appName("DataframeCount").getOrCreate()
> while True:
> for f in os.listdir(target_dir):
> df = spark.read.load(target_dir + f, format="csv")
> print("Number of records: {0}".format(df.count()))
> time.sleep(15){code}
> Submit code:
> {code:java}
> spark-submit
> --master spark://xxx.xxx.xx.xxx
> --deploy-mode client
> --executor-memory 4g
> --executor-cores 3
> streaming count_file.py
> {code}
>
> *TESTED CASES WITH THE SAME BEHAVIOUR:*
> * I tested with default settings (spark-defaults.conf)
> * Add spark.cleaner.periodicGC.interval 1min (or less)
> * {{Turn spark.cleaner.referenceTracking.blocking}}=false
> * Run the application in cluster mode
> * Increase/decrease the resources of the executors and driver
> * I tested with extraJavaOptions in driver and executor -XX:+UseG1GC
> -XX:InitiatingHeapOccupancyPercent=35 -XX:ConcGCThreads=12
>
> *DEPENDENCIES*
> * Operation system: Ubuntu 16.04.3 LTS
> * Java: jdk1.8.0_131 (tested also with jdk1.8.0_221)
> * Python: Python 2.7.12
>
> *NOTE:* In Spark 2.1.1 the driver memory consumption (Storage Memory tab) was
> extremely low and after the run of ContextCleaner and BlockManager the memory
> was decreasing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-29321) Possible memory leak in Spark
[
https://issues.apache.org/jira/browse/SPARK-29321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942280#comment-16942280
]
George Papa commented on SPARK-29321:
-
[~kabhwan] will you update the description based on the latest comments of the
SPARK-29055 ?
> Possible memory leak in Spark
> -
>
> Key: SPARK-29321
> URL: https://issues.apache.org/jira/browse/SPARK-29321
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.3.3
>Reporter: George Papa
>Priority: Major
>
> I used Spark 2.1.1 and I upgraded into new versions. After Spark version
> 2.3.3, I observed from Spark UI that the driver memory is{color:#ff}
> increasing continuously.{color}
> In more detail, the driver memory and executors memory have the same used
> memory storage and after each iteration the storage memory is increasing. You
> can reproduce this behavior by running the following snippet code. The
> following example, is very simple, without any dataframe persistence, but the
> memory consumption is not stable as it was in former Spark versions
> (Specifically until Spark 2.3.2).
> Also, I tested with Spark streaming and structured streaming API and I had
> the same behavior. I tested with an existing application which reads from
> Kafka source and do some aggregations, persist dataframes and then unpersist
> them. The persist and unpersist it works correct, I see the dataframes in the
> storage tab in Spark UI and after the unpersist, all dataframe have removed.
> But, after the unpersist the executors memory is not zero, BUT has the same
> value with the driver memory. This behavior also affects the application
> performance because the memory of the executors is increasing as the driver
> increasing and after a while the persisted dataframes are not fit in the
> executors memory and I have spill to disk.
> Another error which I had after a long running, was
> {color:#ff}java.lang.OutOfMemoryError: GC overhead limit exceeded, but I
> don't know if its relevant with the above behavior or not.{color}
>
> *HOW TO REPRODUCE THIS BEHAVIOR:*
> Create a very simple application(streaming count_file.py) in order to
> reproduce this behavior. This application reads CSV files from a directory,
> count the rows and then remove the processed files.
> {code:java}
> import time
> import os
> from pyspark.sql import SparkSession
> from pyspark.sql import functions as F
> from pyspark.sql import types as T
> target_dir = "..."
> spark=SparkSession.builder.appName("DataframeCount").getOrCreate()
> while True:
> for f in os.listdir(target_dir):
> df = spark.read.load(target_dir + f, format="csv")
> print("Number of records: {0}".format(df.count()))
> time.sleep(15){code}
> Submit code:
> {code:java}
> spark-submit
> --master spark://xxx.xxx.xx.xxx
> --deploy-mode client
> --executor-memory 4g
> --executor-cores 3
> streaming count_file.py
> {code}
>
> *TESTED CASES WITH THE SAME BEHAVIOUR:*
> * I tested with default settings (spark-defaults.conf)
> * Add spark.cleaner.periodicGC.interval 1min (or less)
> * {{Turn spark.cleaner.referenceTracking.blocking}}=false
> * Run the application in cluster mode
> * Increase/decrease the resources of the executors and driver
> * I tested with extraJavaOptions in driver and executor -XX:+UseG1GC
> -XX:InitiatingHeapOccupancyPercent=35 -XX:ConcGCThreads=12
>
> *DEPENDENCIES*
> * Operation system: Ubuntu 16.04.3 LTS
> * Java: jdk1.8.0_131 (tested also with jdk1.8.0_221)
> * Python: Python 2.7.12
>
> *NOTE:* In Spark 2.1.1 the driver memory consumption (Storage Memory tab) was
> extremely low and after the run of ContextCleaner and BlockManager the memory
> was decreasing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-29321) Possible memory leak in Spark
[
https://issues.apache.org/jira/browse/SPARK-29321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcelo Masiero Vanzin reassigned SPARK-29321:
--
Assignee: (was: Jungtaek Lim)
> Possible memory leak in Spark
> -
>
> Key: SPARK-29321
> URL: https://issues.apache.org/jira/browse/SPARK-29321
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.3.3
>Reporter: George Papa
>Priority: Major
>
> I used Spark 2.1.1 and I upgraded into new versions. After Spark version
> 2.3.3, I observed from Spark UI that the driver memory is{color:#ff}
> increasing continuously.{color}
> In more detail, the driver memory and executors memory have the same used
> memory storage and after each iteration the storage memory is increasing. You
> can reproduce this behavior by running the following snippet code. The
> following example, is very simple, without any dataframe persistence, but the
> memory consumption is not stable as it was in former Spark versions
> (Specifically until Spark 2.3.2).
> Also, I tested with Spark streaming and structured streaming API and I had
> the same behavior. I tested with an existing application which reads from
> Kafka source and do some aggregations, persist dataframes and then unpersist
> them. The persist and unpersist it works correct, I see the dataframes in the
> storage tab in Spark UI and after the unpersist, all dataframe have removed.
> But, after the unpersist the executors memory is not zero, BUT has the same
> value with the driver memory. This behavior also affects the application
> performance because the memory of the executors is increasing as the driver
> increasing and after a while the persisted dataframes are not fit in the
> executors memory and I have spill to disk.
> Another error which I had after a long running, was
> {color:#ff}java.lang.OutOfMemoryError: GC overhead limit exceeded, but I
> don't know if its relevant with the above behavior or not.{color}
>
> *HOW TO REPRODUCE THIS BEHAVIOR:*
> Create a very simple application(streaming count_file.py) in order to
> reproduce this behavior. This application reads CSV files from a directory,
> count the rows and then remove the processed files.
> {code:java}
> import time
> import os
> from pyspark.sql import SparkSession
> from pyspark.sql import functions as F
> from pyspark.sql import types as T
> target_dir = "..."
> spark=SparkSession.builder.appName("DataframeCount").getOrCreate()
> while True:
> for f in os.listdir(target_dir):
> df = spark.read.load(target_dir + f, format="csv")
> print("Number of records: {0}".format(df.count()))
> time.sleep(15){code}
> Submit code:
> {code:java}
> spark-submit
> --master spark://xxx.xxx.xx.xxx
> --deploy-mode client
> --executor-memory 4g
> --executor-cores 3
> streaming count_file.py
> {code}
>
> *TESTED CASES WITH THE SAME BEHAVIOUR:*
> * I tested with default settings (spark-defaults.conf)
> * Add spark.cleaner.periodicGC.interval 1min (or less)
> * {{Turn spark.cleaner.referenceTracking.blocking}}=false
> * Run the application in cluster mode
> * Increase/decrease the resources of the executors and driver
> * I tested with extraJavaOptions in driver and executor -XX:+UseG1GC
> -XX:InitiatingHeapOccupancyPercent=35 -XX:ConcGCThreads=12
>
> *DEPENDENCIES*
> * Operation system: Ubuntu 16.04.3 LTS
> * Java: jdk1.8.0_131 (tested also with jdk1.8.0_221)
> * Python: Python 2.7.12
>
> *NOTE:* In Spark 2.1.1 the driver memory consumption (Storage Memory tab) was
> extremely low and after the run of ContextCleaner and BlockManager the memory
> was decreasing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-29321) Possible memory leak in Spark
[
https://issues.apache.org/jira/browse/SPARK-29321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated SPARK-29321:
-
Component/s: (was: Block Manager)
> Possible memory leak in Spark
> -
>
> Key: SPARK-29321
> URL: https://issues.apache.org/jira/browse/SPARK-29321
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.3.3
>Reporter: George Papa
>Assignee: Jungtaek Lim
>Priority: Major
>
> I used Spark 2.1.1 and I upgraded into new versions. After Spark version
> 2.3.3, I observed from Spark UI that the driver memory is{color:#ff}
> increasing continuously.{color}
> In more detail, the driver memory and executors memory have the same used
> memory storage and after each iteration the storage memory is increasing. You
> can reproduce this behavior by running the following snippet code. The
> following example, is very simple, without any dataframe persistence, but the
> memory consumption is not stable as it was in former Spark versions
> (Specifically until Spark 2.3.2).
> Also, I tested with Spark streaming and structured streaming API and I had
> the same behavior. I tested with an existing application which reads from
> Kafka source and do some aggregations, persist dataframes and then unpersist
> them. The persist and unpersist it works correct, I see the dataframes in the
> storage tab in Spark UI and after the unpersist, all dataframe have removed.
> But, after the unpersist the executors memory is not zero, BUT has the same
> value with the driver memory. This behavior also affects the application
> performance because the memory of the executors is increasing as the driver
> increasing and after a while the persisted dataframes are not fit in the
> executors memory and I have spill to disk.
> Another error which I had after a long running, was
> {color:#ff}java.lang.OutOfMemoryError: GC overhead limit exceeded, but I
> don't know if its relevant with the above behavior or not.{color}
>
> *HOW TO REPRODUCE THIS BEHAVIOR:*
> Create a very simple application(streaming count_file.py) in order to
> reproduce this behavior. This application reads CSV files from a directory,
> count the rows and then remove the processed files.
> {code:java}
> import time
> import os
> from pyspark.sql import SparkSession
> from pyspark.sql import functions as F
> from pyspark.sql import types as T
> target_dir = "..."
> spark=SparkSession.builder.appName("DataframeCount").getOrCreate()
> while True:
> for f in os.listdir(target_dir):
> df = spark.read.load(target_dir + f, format="csv")
> print("Number of records: {0}".format(df.count()))
> time.sleep(15){code}
> Submit code:
> {code:java}
> spark-submit
> --master spark://xxx.xxx.xx.xxx
> --deploy-mode client
> --executor-memory 4g
> --executor-cores 3
> streaming count_file.py
> {code}
>
> *TESTED CASES WITH THE SAME BEHAVIOUR:*
> * I tested with default settings (spark-defaults.conf)
> * Add spark.cleaner.periodicGC.interval 1min (or less)
> * {{Turn spark.cleaner.referenceTracking.blocking}}=false
> * Run the application in cluster mode
> * Increase/decrease the resources of the executors and driver
> * I tested with extraJavaOptions in driver and executor -XX:+UseG1GC
> -XX:InitiatingHeapOccupancyPercent=35 -XX:ConcGCThreads=12
>
> *DEPENDENCIES*
> * Operation system: Ubuntu 16.04.3 LTS
> * Java: jdk1.8.0_131 (tested also with jdk1.8.0_221)
> * Python: Python 2.7.12
>
> *NOTE:* In Spark 2.1.1 the driver memory consumption (Storage Memory tab) was
> extremely low and after the run of ContextCleaner and BlockManager the memory
> was decreasing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-29321) CLONE - Possible memory leak in Spark
[
https://issues.apache.org/jira/browse/SPARK-29321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942279#comment-16942279
]
Jungtaek Lim commented on SPARK-29321:
--
[~Geopap] Appreciate if you could migrate the last comment on SPARK-29055.
I would like to unassign myself but I can't do it. Appreciate if any committer
could adjust this.
> CLONE - Possible memory leak in Spark
> -
>
> Key: SPARK-29321
> URL: https://issues.apache.org/jira/browse/SPARK-29321
> Project: Spark
> Issue Type: Bug
> Components: Block Manager, Spark Core
>Affects Versions: 2.3.3
>Reporter: George Papa
>Assignee: Jungtaek Lim
>Priority: Major
>
> I used Spark 2.1.1 and I upgraded into new versions. After Spark version
> 2.3.3, I observed from Spark UI that the driver memory is{color:#ff}
> increasing continuously.{color}
> In more detail, the driver memory and executors memory have the same used
> memory storage and after each iteration the storage memory is increasing. You
> can reproduce this behavior by running the following snippet code. The
> following example, is very simple, without any dataframe persistence, but the
> memory consumption is not stable as it was in former Spark versions
> (Specifically until Spark 2.3.2).
> Also, I tested with Spark streaming and structured streaming API and I had
> the same behavior. I tested with an existing application which reads from
> Kafka source and do some aggregations, persist dataframes and then unpersist
> them. The persist and unpersist it works correct, I see the dataframes in the
> storage tab in Spark UI and after the unpersist, all dataframe have removed.
> But, after the unpersist the executors memory is not zero, BUT has the same
> value with the driver memory. This behavior also affects the application
> performance because the memory of the executors is increasing as the driver
> increasing and after a while the persisted dataframes are not fit in the
> executors memory and I have spill to disk.
> Another error which I had after a long running, was
> {color:#ff}java.lang.OutOfMemoryError: GC overhead limit exceeded, but I
> don't know if its relevant with the above behavior or not.{color}
>
> *HOW TO REPRODUCE THIS BEHAVIOR:*
> Create a very simple application(streaming count_file.py) in order to
> reproduce this behavior. This application reads CSV files from a directory,
> count the rows and then remove the processed files.
> {code:java}
> import time
> import os
> from pyspark.sql import SparkSession
> from pyspark.sql import functions as F
> from pyspark.sql import types as T
> target_dir = "..."
> spark=SparkSession.builder.appName("DataframeCount").getOrCreate()
> while True:
> for f in os.listdir(target_dir):
> df = spark.read.load(target_dir + f, format="csv")
> print("Number of records: {0}".format(df.count()))
> time.sleep(15){code}
> Submit code:
> {code:java}
> spark-submit
> --master spark://xxx.xxx.xx.xxx
> --deploy-mode client
> --executor-memory 4g
> --executor-cores 3
> streaming count_file.py
> {code}
>
> *TESTED CASES WITH THE SAME BEHAVIOUR:*
> * I tested with default settings (spark-defaults.conf)
> * Add spark.cleaner.periodicGC.interval 1min (or less)
> * {{Turn spark.cleaner.referenceTracking.blocking}}=false
> * Run the application in cluster mode
> * Increase/decrease the resources of the executors and driver
> * I tested with extraJavaOptions in driver and executor -XX:+UseG1GC
> -XX:InitiatingHeapOccupancyPercent=35 -XX:ConcGCThreads=12
>
> *DEPENDENCIES*
> * Operation system: Ubuntu 16.04.3 LTS
> * Java: jdk1.8.0_131 (tested also with jdk1.8.0_221)
> * Python: Python 2.7.12
>
> *NOTE:* In Spark 2.1.1 the driver memory consumption (Storage Memory tab) was
> extremely low and after the run of ContextCleaner and BlockManager the memory
> was decreasing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-29321) Possible memory leak in Spark
[
https://issues.apache.org/jira/browse/SPARK-29321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated SPARK-29321:
-
Summary: Possible memory leak in Spark (was: CLONE - Possible memory leak
in Spark)
> Possible memory leak in Spark
> -
>
> Key: SPARK-29321
> URL: https://issues.apache.org/jira/browse/SPARK-29321
> Project: Spark
> Issue Type: Bug
> Components: Block Manager, Spark Core
>Affects Versions: 2.3.3
>Reporter: George Papa
>Assignee: Jungtaek Lim
>Priority: Major
>
> I used Spark 2.1.1 and I upgraded into new versions. After Spark version
> 2.3.3, I observed from Spark UI that the driver memory is{color:#ff}
> increasing continuously.{color}
> In more detail, the driver memory and executors memory have the same used
> memory storage and after each iteration the storage memory is increasing. You
> can reproduce this behavior by running the following snippet code. The
> following example, is very simple, without any dataframe persistence, but the
> memory consumption is not stable as it was in former Spark versions
> (Specifically until Spark 2.3.2).
> Also, I tested with Spark streaming and structured streaming API and I had
> the same behavior. I tested with an existing application which reads from
> Kafka source and do some aggregations, persist dataframes and then unpersist
> them. The persist and unpersist it works correct, I see the dataframes in the
> storage tab in Spark UI and after the unpersist, all dataframe have removed.
> But, after the unpersist the executors memory is not zero, BUT has the same
> value with the driver memory. This behavior also affects the application
> performance because the memory of the executors is increasing as the driver
> increasing and after a while the persisted dataframes are not fit in the
> executors memory and I have spill to disk.
> Another error which I had after a long running, was
> {color:#ff}java.lang.OutOfMemoryError: GC overhead limit exceeded, but I
> don't know if its relevant with the above behavior or not.{color}
>
> *HOW TO REPRODUCE THIS BEHAVIOR:*
> Create a very simple application(streaming count_file.py) in order to
> reproduce this behavior. This application reads CSV files from a directory,
> count the rows and then remove the processed files.
> {code:java}
> import time
> import os
> from pyspark.sql import SparkSession
> from pyspark.sql import functions as F
> from pyspark.sql import types as T
> target_dir = "..."
> spark=SparkSession.builder.appName("DataframeCount").getOrCreate()
> while True:
> for f in os.listdir(target_dir):
> df = spark.read.load(target_dir + f, format="csv")
> print("Number of records: {0}".format(df.count()))
> time.sleep(15){code}
> Submit code:
> {code:java}
> spark-submit
> --master spark://xxx.xxx.xx.xxx
> --deploy-mode client
> --executor-memory 4g
> --executor-cores 3
> streaming count_file.py
> {code}
>
> *TESTED CASES WITH THE SAME BEHAVIOUR:*
> * I tested with default settings (spark-defaults.conf)
> * Add spark.cleaner.periodicGC.interval 1min (or less)
> * {{Turn spark.cleaner.referenceTracking.blocking}}=false
> * Run the application in cluster mode
> * Increase/decrease the resources of the executors and driver
> * I tested with extraJavaOptions in driver and executor -XX:+UseG1GC
> -XX:InitiatingHeapOccupancyPercent=35 -XX:ConcGCThreads=12
>
> *DEPENDENCIES*
> * Operation system: Ubuntu 16.04.3 LTS
> * Java: jdk1.8.0_131 (tested also with jdk1.8.0_221)
> * Python: Python 2.7.12
>
> *NOTE:* In Spark 2.1.1 the driver memory consumption (Storage Memory tab) was
> extremely low and after the run of ContextCleaner and BlockManager the memory
> was decreasing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-13446) Spark need to support reading data from Hive 2.0.0 metastore
[
https://issues.apache.org/jira/browse/SPARK-13446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942275#comment-16942275
]
JP Bordenave edited comment on SPARK-13446 at 10/1/19 8:03 PM:
---
1) ok i am back, after some internet issue, i make restore all hive 1.2.1
jars from spark 2.4.4
2) hive v2.3.3 get conflict mysql schema 2.3.0 with spark 2.4.4 because it use
1.2.1 schema
3) i make cp hive-site.xml into spark/conf and i disable schema check, it is
working fine under spark-shell
{{}}
{\{ hive.metastore.schema.verification}}
{\{ false}}
{{}}
{{But i doenst understand why spark 2.4.4 use old hive schema 1.2.1 (not realy
clear for me)}}
{{}}
{noformat}
^
scala> spark.sql("show databases").show
++
|databaseName|
++
| default|
++
scala> spark.sql("show tables").show
19/10/01 21:44:07 WARN metastore.ObjectStore: Failed to get database
global_temp, returning NoSuchObjectException
++-+---+
|database|tableName|isTemporary|
++-+---+
| default| employee| false|
++-+---+
scala> spark.sql("select * from employee").show
+---+---+-+
| id| name| dept|
+---+---+-+
| 1| Allen| IT|
| 2|Mag|Sales|
| 14| Pierre| xXx|
| 1| Allen| IT|
| 3|Rob|Sales|
| 4| Dana| IT|
| 7| Pierre| xXx|
| 11| Pierre| xXx|
| 10| Pierre| xXx|
| 12| Pierre| xXx|
| 13| Pierre| xXx|
+---+---+-+
{noformat}
{{}}
{{}}
{{}}
was (Author: jpbordi):
1) ok i am back, after some internet issue, i make restore all hive 1.2.1
jars from spark 2.4.4
2) hive v2.3.3 get conflict mysql schema 2.3.0 with spark 2.4.4 because it use
1.2.1 schema
3) i make cp hive-site.xml into spark/conf and i disable schema check, it is
working fine under spark-shell
{{}}
{\{ hive.metastore.schema.verification}}
{\{ false}}
{{}}
{{But i doenst understand why spark 2.4.4 use old hive schema 1.2.1 (not realy
clear for me)}}
{{}}
{noformat}
++
|databaseName|
++
| default|
++
scala> select * from employee;
:24: error: not found: value select
select * from employee;
^
:24: error: not found: value from
select * from employee;
^scala> spark.sql("show databases").show
++
|databaseName|
++
| default|
++
scala> spark.sql("show tables").show
19/10/01 21:44:07 WARN metastore.ObjectStore: Failed to get database
global_temp, returning NoSuchObjectException
++-+---+
|database|tableName|isTemporary|
++-+---+
| default| employee| false|
++-+---+
scala> spark.sql("select * from employee").show
+---+---+-+
| id| name| dept|
+---+---+-+
| 1| Allen| IT|
| 2|Mag|Sales|
| 14| Pierre| xXx|
| 1| Allen| IT|
| 3|Rob|Sales|
| 4| Dana| IT|
| 7| Pierre| xXx|
| 11| Pierre| xXx|
| 10| Pierre| xXx|
| 12| Pierre| xXx|
| 13| Pierre| xXx|
+---+---+-+
{noformat}
{{}}
{{}}
{{}}
> Spark need to support reading data from Hive 2.0.0 metastore
>
>
> Key: SPARK-13446
> URL: https://issues.apache.org/jira/browse/SPARK-13446
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 1.6.0
>Reporter: Lifeng Wang
>Assignee: Xiao Li
>Priority: Major
> Fix For: 2.2.0
>
>
> Spark provided HIveContext class to read data from hive metastore directly.
> While it only supports hive 1.2.1 version and older. Since hive 2.0.0 has
> released, it's better to upgrade to support Hive 2.0.0.
> {noformat}
> 16/02/23 02:35:02 INFO metastore: Trying to connect to metastore with URI
> thrift://hsw-node13:9083
> 16/02/23 02:35:02 INFO metastore: Opened a connection to metastore, current
> connections: 1
> 16/02/23 02:35:02 INFO metastore: Connected to metastore.
> Exception in thread "main" java.lang.NoSuchFieldError: HIVE_STATS_JDBC_TIMEOUT
> at
> org.apache.spark.sql.hive.HiveContext.configure(HiveContext.scala:473)
> at
> org.apache.spark.sql.hive.HiveContext.metadataHive$lzycompute(HiveContext.scala:192)
> at
> org.apache.spark.sql.hive.HiveContext.metadataHive(HiveContext.scala:185)
> at
> org.apache.spark.sql.hive.HiveContext$$anon$1.(HiveContext.scala:422)
> at
> org.apache.spark.sql.hive.HiveContext.catalog$lzycompute(HiveContext.scala:422)
> at
>
[jira] [Updated] (SPARK-29321) CLONE - Possible memory leak in Spark
[
https://issues.apache.org/jira/browse/SPARK-29321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
George Papa updated SPARK-29321:
Summary: CLONE - Possible memory leak in Spark (was: Possible memory leak
in Spark)
> CLONE - Possible memory leak in Spark
> -
>
> Key: SPARK-29321
> URL: https://issues.apache.org/jira/browse/SPARK-29321
> Project: Spark
> Issue Type: Bug
> Components: Block Manager, Spark Core
>Affects Versions: 2.3.3
>Reporter: George Papa
>Assignee: Jungtaek Lim
>Priority: Major
>
> I used Spark 2.1.1 and I upgraded into new versions. After Spark version
> 2.3.3, I observed from Spark UI that the driver memory is{color:#ff}
> increasing continuously.{color}
> In more detail, the driver memory and executors memory have the same used
> memory storage and after each iteration the storage memory is increasing. You
> can reproduce this behavior by running the following snippet code. The
> following example, is very simple, without any dataframe persistence, but the
> memory consumption is not stable as it was in former Spark versions
> (Specifically until Spark 2.3.2).
> Also, I tested with Spark streaming and structured streaming API and I had
> the same behavior. I tested with an existing application which reads from
> Kafka source and do some aggregations, persist dataframes and then unpersist
> them. The persist and unpersist it works correct, I see the dataframes in the
> storage tab in Spark UI and after the unpersist, all dataframe have removed.
> But, after the unpersist the executors memory is not zero, BUT has the same
> value with the driver memory. This behavior also affects the application
> performance because the memory of the executors is increasing as the driver
> increasing and after a while the persisted dataframes are not fit in the
> executors memory and I have spill to disk.
> Another error which I had after a long running, was
> {color:#ff}java.lang.OutOfMemoryError: GC overhead limit exceeded, but I
> don't know if its relevant with the above behavior or not.{color}
>
> *HOW TO REPRODUCE THIS BEHAVIOR:*
> Create a very simple application(streaming count_file.py) in order to
> reproduce this behavior. This application reads CSV files from a directory,
> count the rows and then remove the processed files.
> {code:java}
> import time
> import os
> from pyspark.sql import SparkSession
> from pyspark.sql import functions as F
> from pyspark.sql import types as T
> target_dir = "..."
> spark=SparkSession.builder.appName("DataframeCount").getOrCreate()
> while True:
> for f in os.listdir(target_dir):
> df = spark.read.load(target_dir + f, format="csv")
> print("Number of records: {0}".format(df.count()))
> time.sleep(15){code}
> Submit code:
> {code:java}
> spark-submit
> --master spark://xxx.xxx.xx.xxx
> --deploy-mode client
> --executor-memory 4g
> --executor-cores 3
> streaming count_file.py
> {code}
>
> *TESTED CASES WITH THE SAME BEHAVIOUR:*
> * I tested with default settings (spark-defaults.conf)
> * Add spark.cleaner.periodicGC.interval 1min (or less)
> * {{Turn spark.cleaner.referenceTracking.blocking}}=false
> * Run the application in cluster mode
> * Increase/decrease the resources of the executors and driver
> * I tested with extraJavaOptions in driver and executor -XX:+UseG1GC
> -XX:InitiatingHeapOccupancyPercent=35 -XX:ConcGCThreads=12
>
> *DEPENDENCIES*
> * Operation system: Ubuntu 16.04.3 LTS
> * Java: jdk1.8.0_131 (tested also with jdk1.8.0_221)
> * Python: Python 2.7.12
>
> *NOTE:* In Spark 2.1.1 the driver memory consumption (Storage Memory tab) was
> extremely low and after the run of ContextCleaner and BlockManager the memory
> was decreasing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-29321) Possible memory leak in Spark
[
https://issues.apache.org/jira/browse/SPARK-29321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated SPARK-29321:
-
Summary: Possible memory leak in Spark (was: CLONE - Possible memory leak
in Spark)
> Possible memory leak in Spark
> -
>
> Key: SPARK-29321
> URL: https://issues.apache.org/jira/browse/SPARK-29321
> Project: Spark
> Issue Type: Bug
> Components: Block Manager, Spark Core
>Affects Versions: 2.3.3
>Reporter: George Papa
>Assignee: Jungtaek Lim
>Priority: Major
>
> I used Spark 2.1.1 and I upgraded into new versions. After Spark version
> 2.3.3, I observed from Spark UI that the driver memory is{color:#ff}
> increasing continuously.{color}
> In more detail, the driver memory and executors memory have the same used
> memory storage and after each iteration the storage memory is increasing. You
> can reproduce this behavior by running the following snippet code. The
> following example, is very simple, without any dataframe persistence, but the
> memory consumption is not stable as it was in former Spark versions
> (Specifically until Spark 2.3.2).
> Also, I tested with Spark streaming and structured streaming API and I had
> the same behavior. I tested with an existing application which reads from
> Kafka source and do some aggregations, persist dataframes and then unpersist
> them. The persist and unpersist it works correct, I see the dataframes in the
> storage tab in Spark UI and after the unpersist, all dataframe have removed.
> But, after the unpersist the executors memory is not zero, BUT has the same
> value with the driver memory. This behavior also affects the application
> performance because the memory of the executors is increasing as the driver
> increasing and after a while the persisted dataframes are not fit in the
> executors memory and I have spill to disk.
> Another error which I had after a long running, was
> {color:#ff}java.lang.OutOfMemoryError: GC overhead limit exceeded, but I
> don't know if its relevant with the above behavior or not.{color}
>
> *HOW TO REPRODUCE THIS BEHAVIOR:*
> Create a very simple application(streaming count_file.py) in order to
> reproduce this behavior. This application reads CSV files from a directory,
> count the rows and then remove the processed files.
> {code:java}
> import time
> import os
> from pyspark.sql import SparkSession
> from pyspark.sql import functions as F
> from pyspark.sql import types as T
> target_dir = "..."
> spark=SparkSession.builder.appName("DataframeCount").getOrCreate()
> while True:
> for f in os.listdir(target_dir):
> df = spark.read.load(target_dir + f, format="csv")
> print("Number of records: {0}".format(df.count()))
> time.sleep(15){code}
> Submit code:
> {code:java}
> spark-submit
> --master spark://xxx.xxx.xx.xxx
> --deploy-mode client
> --executor-memory 4g
> --executor-cores 3
> streaming count_file.py
> {code}
>
> *TESTED CASES WITH THE SAME BEHAVIOUR:*
> * I tested with default settings (spark-defaults.conf)
> * Add spark.cleaner.periodicGC.interval 1min (or less)
> * {{Turn spark.cleaner.referenceTracking.blocking}}=false
> * Run the application in cluster mode
> * Increase/decrease the resources of the executors and driver
> * I tested with extraJavaOptions in driver and executor -XX:+UseG1GC
> -XX:InitiatingHeapOccupancyPercent=35 -XX:ConcGCThreads=12
>
> *DEPENDENCIES*
> * Operation system: Ubuntu 16.04.3 LTS
> * Java: jdk1.8.0_131 (tested also with jdk1.8.0_221)
> * Python: Python 2.7.12
>
> *NOTE:* In Spark 2.1.1 the driver memory consumption (Storage Memory tab) was
> extremely low and after the run of ContextCleaner and BlockManager the memory
> was decreasing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-29321) CLONE - Possible memory leak in Spark
[
https://issues.apache.org/jira/browse/SPARK-29321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated SPARK-29321:
-
Fix Version/s: (was: 2.4.5)
(was: 3.0.0)
> CLONE - Possible memory leak in Spark
> -
>
> Key: SPARK-29321
> URL: https://issues.apache.org/jira/browse/SPARK-29321
> Project: Spark
> Issue Type: Bug
> Components: Block Manager, Spark Core
>Affects Versions: 2.3.3
>Reporter: George Papa
>Assignee: Jungtaek Lim
>Priority: Major
>
> I used Spark 2.1.1 and I upgraded into new versions. After Spark version
> 2.3.3, I observed from Spark UI that the driver memory is{color:#ff}
> increasing continuously.{color}
> In more detail, the driver memory and executors memory have the same used
> memory storage and after each iteration the storage memory is increasing. You
> can reproduce this behavior by running the following snippet code. The
> following example, is very simple, without any dataframe persistence, but the
> memory consumption is not stable as it was in former Spark versions
> (Specifically until Spark 2.3.2).
> Also, I tested with Spark streaming and structured streaming API and I had
> the same behavior. I tested with an existing application which reads from
> Kafka source and do some aggregations, persist dataframes and then unpersist
> them. The persist and unpersist it works correct, I see the dataframes in the
> storage tab in Spark UI and after the unpersist, all dataframe have removed.
> But, after the unpersist the executors memory is not zero, BUT has the same
> value with the driver memory. This behavior also affects the application
> performance because the memory of the executors is increasing as the driver
> increasing and after a while the persisted dataframes are not fit in the
> executors memory and I have spill to disk.
> Another error which I had after a long running, was
> {color:#ff}java.lang.OutOfMemoryError: GC overhead limit exceeded, but I
> don't know if its relevant with the above behavior or not.{color}
>
> *HOW TO REPRODUCE THIS BEHAVIOR:*
> Create a very simple application(streaming count_file.py) in order to
> reproduce this behavior. This application reads CSV files from a directory,
> count the rows and then remove the processed files.
> {code:java}
> import time
> import os
> from pyspark.sql import SparkSession
> from pyspark.sql import functions as F
> from pyspark.sql import types as T
> target_dir = "..."
> spark=SparkSession.builder.appName("DataframeCount").getOrCreate()
> while True:
> for f in os.listdir(target_dir):
> df = spark.read.load(target_dir + f, format="csv")
> print("Number of records: {0}".format(df.count()))
> time.sleep(15){code}
> Submit code:
> {code:java}
> spark-submit
> --master spark://xxx.xxx.xx.xxx
> --deploy-mode client
> --executor-memory 4g
> --executor-cores 3
> streaming count_file.py
> {code}
>
> *TESTED CASES WITH THE SAME BEHAVIOUR:*
> * I tested with default settings (spark-defaults.conf)
> * Add spark.cleaner.periodicGC.interval 1min (or less)
> * {{Turn spark.cleaner.referenceTracking.blocking}}=false
> * Run the application in cluster mode
> * Increase/decrease the resources of the executors and driver
> * I tested with extraJavaOptions in driver and executor -XX:+UseG1GC
> -XX:InitiatingHeapOccupancyPercent=35 -XX:ConcGCThreads=12
>
> *DEPENDENCIES*
> * Operation system: Ubuntu 16.04.3 LTS
> * Java: jdk1.8.0_131 (tested also with jdk1.8.0_221)
> * Python: Python 2.7.12
>
> *NOTE:* In Spark 2.1.1 the driver memory consumption (Storage Memory tab) was
> extremely low and after the run of ContextCleaner and BlockManager the memory
> was decreasing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-29055) Spark UI storage memory increasing overtime
[
https://issues.apache.org/jira/browse/SPARK-29055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
George Papa updated SPARK-29055:
Summary: Spark UI storage memory increasing overtime (was: Possible memory
leak in Spark)
> Spark UI storage memory increasing overtime
> ---
>
> Key: SPARK-29055
> URL: https://issues.apache.org/jira/browse/SPARK-29055
> Project: Spark
> Issue Type: Bug
> Components: Block Manager, Spark Core
>Affects Versions: 2.3.3
>Reporter: George Papa
>Assignee: Jungtaek Lim
>Priority: Major
> Fix For: 2.4.5, 3.0.0
>
> Attachments: test_csvs.zip
>
>
> I used Spark 2.1.1 and I upgraded into new versions. After Spark version
> 2.3.3, I observed from Spark UI that the driver memory is{color:#ff}
> increasing continuously.{color}
> In more detail, the driver memory and executors memory have the same used
> memory storage and after each iteration the storage memory is increasing. You
> can reproduce this behavior by running the following snippet code. The
> following example, is very simple, without any dataframe persistence, but the
> memory consumption is not stable as it was in former Spark versions
> (Specifically until Spark 2.3.2).
> Also, I tested with Spark streaming and structured streaming API and I had
> the same behavior. I tested with an existing application which reads from
> Kafka source and do some aggregations, persist dataframes and then unpersist
> them. The persist and unpersist it works correct, I see the dataframes in the
> storage tab in Spark UI and after the unpersist, all dataframe have removed.
> But, after the unpersist the executors memory is not zero, BUT has the same
> value with the driver memory. This behavior also affects the application
> performance because the memory of the executors is increasing as the driver
> increasing and after a while the persisted dataframes are not fit in the
> executors memory and I have spill to disk.
> Another error which I had after a long running, was
> {color:#ff}java.lang.OutOfMemoryError: GC overhead limit exceeded, but I
> don't know if its relevant with the above behavior or not.{color}
>
> *HOW TO REPRODUCE THIS BEHAVIOR:*
> Create a very simple application(streaming count_file.py) in order to
> reproduce this behavior. This application reads CSV files from a directory,
> count the rows and then remove the processed files.
> {code:java}
> import time
> import os
> from pyspark.sql import SparkSession
> from pyspark.sql import functions as F
> from pyspark.sql import types as T
> target_dir = "..."
> spark=SparkSession.builder.appName("DataframeCount").getOrCreate()
> while True:
> for f in os.listdir(target_dir):
> df = spark.read.load(target_dir + f, format="csv")
> print("Number of records: {0}".format(df.count()))
> time.sleep(15){code}
> Submit code:
> {code:java}
> spark-submit
> --master spark://xxx.xxx.xx.xxx
> --deploy-mode client
> --executor-memory 4g
> --executor-cores 3
> streaming count_file.py
> {code}
>
> *TESTED CASES WITH THE SAME BEHAVIOUR:*
> * I tested with default settings (spark-defaults.conf)
> * Add spark.cleaner.periodicGC.interval 1min (or less)
> * {{Turn spark.cleaner.referenceTracking.blocking}}=false
> * Run the application in cluster mode
> * Increase/decrease the resources of the executors and driver
> * I tested with extraJavaOptions in driver and executor -XX:+UseG1GC
> -XX:InitiatingHeapOccupancyPercent=35 -XX:ConcGCThreads=12
>
> *DEPENDENCIES*
> * Operation system: Ubuntu 16.04.3 LTS
> * Java: jdk1.8.0_131 (tested also with jdk1.8.0_221)
> * Python: Python 2.7.12
>
> *NOTE:* In Spark 2.1.1 the driver memory consumption (Storage Memory tab) was
> extremely low and after the run of ContextCleaner and BlockManager the memory
> was decreasing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-29321) CLONE - Possible memory leak in Spark
George Papa created SPARK-29321:
---
Summary: CLONE - Possible memory leak in Spark
Key: SPARK-29321
URL: https://issues.apache.org/jira/browse/SPARK-29321
Project: Spark
Issue Type: Bug
Components: Block Manager, Spark Core
Affects Versions: 2.3.3
Reporter: George Papa
Assignee: Jungtaek Lim
Fix For: 2.4.5, 3.0.0
I used Spark 2.1.1 and I upgraded into new versions. After Spark version 2.3.3,
I observed from Spark UI that the driver memory is{color:#ff} increasing
continuously.{color}
In more detail, the driver memory and executors memory have the same used
memory storage and after each iteration the storage memory is increasing. You
can reproduce this behavior by running the following snippet code. The
following example, is very simple, without any dataframe persistence, but the
memory consumption is not stable as it was in former Spark versions
(Specifically until Spark 2.3.2).
Also, I tested with Spark streaming and structured streaming API and I had the
same behavior. I tested with an existing application which reads from Kafka
source and do some aggregations, persist dataframes and then unpersist them.
The persist and unpersist it works correct, I see the dataframes in the storage
tab in Spark UI and after the unpersist, all dataframe have removed. But, after
the unpersist the executors memory is not zero, BUT has the same value with the
driver memory. This behavior also affects the application performance because
the memory of the executors is increasing as the driver increasing and after a
while the persisted dataframes are not fit in the executors memory and I have
spill to disk.
Another error which I had after a long running, was
{color:#ff}java.lang.OutOfMemoryError: GC overhead limit exceeded, but I
don't know if its relevant with the above behavior or not.{color}
*HOW TO REPRODUCE THIS BEHAVIOR:*
Create a very simple application(streaming count_file.py) in order to reproduce
this behavior. This application reads CSV files from a directory, count the
rows and then remove the processed files.
{code:java}
import time
import os
from pyspark.sql import SparkSession
from pyspark.sql import functions as F
from pyspark.sql import types as T
target_dir = "..."
spark=SparkSession.builder.appName("DataframeCount").getOrCreate()
while True:
for f in os.listdir(target_dir):
df = spark.read.load(target_dir + f, format="csv")
print("Number of records: {0}".format(df.count()))
time.sleep(15){code}
Submit code:
{code:java}
spark-submit
--master spark://xxx.xxx.xx.xxx
--deploy-mode client
--executor-memory 4g
--executor-cores 3
streaming count_file.py
{code}
*TESTED CASES WITH THE SAME BEHAVIOUR:*
* I tested with default settings (spark-defaults.conf)
* Add spark.cleaner.periodicGC.interval 1min (or less)
* {{Turn spark.cleaner.referenceTracking.blocking}}=false
* Run the application in cluster mode
* Increase/decrease the resources of the executors and driver
* I tested with extraJavaOptions in driver and executor -XX:+UseG1GC
-XX:InitiatingHeapOccupancyPercent=35 -XX:ConcGCThreads=12
*DEPENDENCIES*
* Operation system: Ubuntu 16.04.3 LTS
* Java: jdk1.8.0_131 (tested also with jdk1.8.0_221)
* Python: Python 2.7.12
*NOTE:* In Spark 2.1.1 the driver memory consumption (Storage Memory tab) was
extremely low and after the run of ContextCleaner and BlockManager the memory
was decreasing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-13446) Spark need to support reading data from Hive 2.0.0 metastore
[
https://issues.apache.org/jira/browse/SPARK-13446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942275#comment-16942275
]
JP Bordenave edited comment on SPARK-13446 at 10/1/19 7:57 PM:
---
1) ok i am back, after some internet issue, i make restore all hive 1.2.1
jars from spark 2.4.4
2) hive v2.3.3 get conflict mysql schema 2.3.0 with spark 2.4.4 because it use
1.2.1 schema
3) i make cp hive-site.xml into spark/conf and i disable schema check, it is
working fine under spark-shell
{{}}
{\{ hive.metastore.schema.verification}}
{\{ false}}
{{}}
{{But i doenst understand why spark 2.4.4 use old hive schema 1.2.1 (not realy
clear for me)}}
{{}}
{noformat}
++
|databaseName|
++
| default|
++
scala> select * from employee;
:24: error: not found: value select
select * from employee;
^
:24: error: not found: value from
select * from employee;
^scala> spark.sql("show databases").show
++
|databaseName|
++
| default|
++
scala> spark.sql("show tables").show
19/10/01 21:44:07 WARN metastore.ObjectStore: Failed to get database
global_temp, returning NoSuchObjectException
++-+---+
|database|tableName|isTemporary|
++-+---+
| default| employee| false|
++-+---+
scala> spark.sql("select * from employee").show
+---+---+-+
| id| name| dept|
+---+---+-+
| 1| Allen| IT|
| 2|Mag|Sales|
| 14| Pierre| xXx|
| 1| Allen| IT|
| 3|Rob|Sales|
| 4| Dana| IT|
| 7| Pierre| xXx|
| 11| Pierre| xXx|
| 10| Pierre| xXx|
| 12| Pierre| xXx|
| 13| Pierre| xXx|
+---+---+-+
{noformat}
{{}}
{{}}
{{}}
was (Author: jpbordi):
1) ok i am back, after some internet issue, i make restore all hive 1.2.1
jars from spark 2.4.4
2) hive v2.3.3 get conflict mysql schema 2.3.0 with spark 2.4.4 because it use
1.2.1 schema
3) i make cp hive-site.xml into spark/conf and i disable schema check, it is
working fine
{{}}
{{hive.metastore.schema.verification}}
{{false}}
{{}}
{{But i doenst understand why spark 2.4.4 use old hive schema 1.2.1 (not realy
clear for me)}}
{{}}
{noformat}
++
|databaseName|
++
| default|
++
scala> select * from employee;
:24: error: not found: value select
select * from employee;
^
:24: error: not found: value from
select * from employee;
^scala> spark.sql("show databases").show
++
|databaseName|
++
| default|
++
scala> spark.sql("show tables").show
19/10/01 21:44:07 WARN metastore.ObjectStore: Failed to get database
global_temp, returning NoSuchObjectException
++-+---+
|database|tableName|isTemporary|
++-+---+
| default| employee| false|
++-+---+
scala> spark.sql("select * from employee").show
+---+---+-+
| id| name| dept|
+---+---+-+
| 1| Allen| IT|
| 2|Mag|Sales|
| 14| Pierre| xXx|
| 1| Allen| IT|
| 3|Rob|Sales|
| 4| Dana| IT|
| 6|Jean-Pierre|Bordenave|
| 7| Pierre| xXx|
| 11| Pierre| xXx|
| 10| Pierre| xXx|
| 12| Pierre| xXx|
| 13| Pierre| xXx|
+---+---+-+
{noformat}
{{}}
{{}}
{{}}
> Spark need to support reading data from Hive 2.0.0 metastore
>
>
> Key: SPARK-13446
> URL: https://issues.apache.org/jira/browse/SPARK-13446
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 1.6.0
>Reporter: Lifeng Wang
>Assignee: Xiao Li
>Priority: Major
> Fix For: 2.2.0
>
>
> Spark provided HIveContext class to read data from hive metastore directly.
> While it only supports hive 1.2.1 version and older. Since hive 2.0.0 has
> released, it's better to upgrade to support Hive 2.0.0.
> {noformat}
> 16/02/23 02:35:02 INFO metastore: Trying to connect to metastore with URI
> thrift://hsw-node13:9083
> 16/02/23 02:35:02 INFO metastore: Opened a connection to metastore, current
> connections: 1
> 16/02/23 02:35:02 INFO metastore: Connected to metastore.
> Exception in thread "main" java.lang.NoSuchFieldError: HIVE_STATS_JDBC_TIMEOUT
> at
> org.apache.spark.sql.hive.HiveContext.configure(HiveContext.scala:473)
> at
> org.apache.spark.sql.hive.HiveContext.metadataHive$lzycompute(HiveContext.scala:192)
> at
>
[jira] [Commented] (SPARK-13446) Spark need to support reading data from Hive 2.0.0 metastore
[
https://issues.apache.org/jira/browse/SPARK-13446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942275#comment-16942275
]
JP Bordenave commented on SPARK-13446:
--
1) ok i am back, after some internet issue, i make restore all hive 1.2.1
jars from spark 2.4.4
2) hive v2.3.3 get conflict mysql schema 2.3.0 with spark 2.4.4 because it use
1.2.1 schema
3) i make cp hive-site.xml into spark/conf and i disable schema check, it is
working fine
{{}}
{{hive.metastore.schema.verification}}
{{false}}
{{}}
{{But i doenst understand why spark 2.4.4 use old hive schema 1.2.1 (not realy
clear for me)}}
{{}}
{noformat}
++
|databaseName|
++
| default|
++
scala> select * from employee;
:24: error: not found: value select
select * from employee;
^
:24: error: not found: value from
select * from employee;
^scala> spark.sql("show databases").show
++
|databaseName|
++
| default|
++
scala> spark.sql("show tables").show
19/10/01 21:44:07 WARN metastore.ObjectStore: Failed to get database
global_temp, returning NoSuchObjectException
++-+---+
|database|tableName|isTemporary|
++-+---+
| default| employee| false|
++-+---+
scala> spark.sql("select * from employee").show
+---+---+-+
| id| name| dept|
+---+---+-+
| 1| Allen| IT|
| 2|Mag|Sales|
| 14| Pierre| xXx|
| 1| Allen| IT|
| 3|Rob|Sales|
| 4| Dana| IT|
| 6|Jean-Pierre|Bordenave|
| 7| Pierre| xXx|
| 11| Pierre| xXx|
| 10| Pierre| xXx|
| 12| Pierre| xXx|
| 13| Pierre| xXx|
+---+---+-+
{noformat}
{{}}
{{}}
{{}}
> Spark need to support reading data from Hive 2.0.0 metastore
>
>
> Key: SPARK-13446
> URL: https://issues.apache.org/jira/browse/SPARK-13446
> Project: Spark
> Issue Type: Improvement
> Components: SQL
>Affects Versions: 1.6.0
>Reporter: Lifeng Wang
>Assignee: Xiao Li
>Priority: Major
> Fix For: 2.2.0
>
>
> Spark provided HIveContext class to read data from hive metastore directly.
> While it only supports hive 1.2.1 version and older. Since hive 2.0.0 has
> released, it's better to upgrade to support Hive 2.0.0.
> {noformat}
> 16/02/23 02:35:02 INFO metastore: Trying to connect to metastore with URI
> thrift://hsw-node13:9083
> 16/02/23 02:35:02 INFO metastore: Opened a connection to metastore, current
> connections: 1
> 16/02/23 02:35:02 INFO metastore: Connected to metastore.
> Exception in thread "main" java.lang.NoSuchFieldError: HIVE_STATS_JDBC_TIMEOUT
> at
> org.apache.spark.sql.hive.HiveContext.configure(HiveContext.scala:473)
> at
> org.apache.spark.sql.hive.HiveContext.metadataHive$lzycompute(HiveContext.scala:192)
> at
> org.apache.spark.sql.hive.HiveContext.metadataHive(HiveContext.scala:185)
> at
> org.apache.spark.sql.hive.HiveContext$$anon$1.(HiveContext.scala:422)
> at
> org.apache.spark.sql.hive.HiveContext.catalog$lzycompute(HiveContext.scala:422)
> at
> org.apache.spark.sql.hive.HiveContext.catalog(HiveContext.scala:421)
> at org.apache.spark.sql.hive.HiveContext.catalog(HiveContext.scala:72)
> at org.apache.spark.sql.SQLContext.table(SQLContext.scala:739)
> at org.apache.spark.sql.SQLContext.table(SQLContext.scala:735)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-29055) Possible memory leak in Spark
[
https://issues.apache.org/jira/browse/SPARK-29055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
George Papa updated SPARK-29055:
Summary: Possible memory leak in Spark (was: Memory leak in Spark)
> Possible memory leak in Spark
> -
>
> Key: SPARK-29055
> URL: https://issues.apache.org/jira/browse/SPARK-29055
> Project: Spark
> Issue Type: Bug
> Components: Block Manager, Spark Core
>Affects Versions: 2.3.3
>Reporter: George Papa
>Assignee: Jungtaek Lim
>Priority: Major
> Fix For: 2.4.5, 3.0.0
>
> Attachments: test_csvs.zip
>
>
> I used Spark 2.1.1 and I upgraded into new versions. After Spark version
> 2.3.3, I observed from Spark UI that the driver memory is{color:#ff}
> increasing continuously.{color}
> In more detail, the driver memory and executors memory have the same used
> memory storage and after each iteration the storage memory is increasing. You
> can reproduce this behavior by running the following snippet code. The
> following example, is very simple, without any dataframe persistence, but the
> memory consumption is not stable as it was in former Spark versions
> (Specifically until Spark 2.3.2).
> Also, I tested with Spark streaming and structured streaming API and I had
> the same behavior. I tested with an existing application which reads from
> Kafka source and do some aggregations, persist dataframes and then unpersist
> them. The persist and unpersist it works correct, I see the dataframes in the
> storage tab in Spark UI and after the unpersist, all dataframe have removed.
> But, after the unpersist the executors memory is not zero, BUT has the same
> value with the driver memory. This behavior also affects the application
> performance because the memory of the executors is increasing as the driver
> increasing and after a while the persisted dataframes are not fit in the
> executors memory and I have spill to disk.
> Another error which I had after a long running, was
> {color:#ff}java.lang.OutOfMemoryError: GC overhead limit exceeded, but I
> don't know if its relevant with the above behavior or not.{color}
>
> *HOW TO REPRODUCE THIS BEHAVIOR:*
> Create a very simple application(streaming count_file.py) in order to
> reproduce this behavior. This application reads CSV files from a directory,
> count the rows and then remove the processed files.
> {code:java}
> import time
> import os
> from pyspark.sql import SparkSession
> from pyspark.sql import functions as F
> from pyspark.sql import types as T
> target_dir = "..."
> spark=SparkSession.builder.appName("DataframeCount").getOrCreate()
> while True:
> for f in os.listdir(target_dir):
> df = spark.read.load(target_dir + f, format="csv")
> print("Number of records: {0}".format(df.count()))
> time.sleep(15){code}
> Submit code:
> {code:java}
> spark-submit
> --master spark://xxx.xxx.xx.xxx
> --deploy-mode client
> --executor-memory 4g
> --executor-cores 3
> streaming count_file.py
> {code}
>
> *TESTED CASES WITH THE SAME BEHAVIOUR:*
> * I tested with default settings (spark-defaults.conf)
> * Add spark.cleaner.periodicGC.interval 1min (or less)
> * {{Turn spark.cleaner.referenceTracking.blocking}}=false
> * Run the application in cluster mode
> * Increase/decrease the resources of the executors and driver
> * I tested with extraJavaOptions in driver and executor -XX:+UseG1GC
> -XX:InitiatingHeapOccupancyPercent=35 -XX:ConcGCThreads=12
>
> *DEPENDENCIES*
> * Operation system: Ubuntu 16.04.3 LTS
> * Java: jdk1.8.0_131 (tested also with jdk1.8.0_221)
> * Python: Python 2.7.12
>
> *NOTE:* In Spark 2.1.1 the driver memory consumption (Storage Memory tab) was
> extremely low and after the run of ContextCleaner and BlockManager the memory
> was decreasing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-29055) Memory leak in Spark
[
https://issues.apache.org/jira/browse/SPARK-29055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942267#comment-16942267
]
Jungtaek Lim commented on SPARK-29055:
--
Looks like I can update the title and description, but then the information of
"possible memory leak" is lost.
I could also clone the issue but it changes the reporter to me (as I don't have
privilege to modify reporter) which is not desired.
[~Geopap] [~vanzin] [~viirya] Could anyone clone this issue? After cloning I'll
update this issue accordingly.
> Memory leak in Spark
>
>
> Key: SPARK-29055
> URL: https://issues.apache.org/jira/browse/SPARK-29055
> Project: Spark
> Issue Type: Bug
> Components: Block Manager, Spark Core
>Affects Versions: 2.3.3
>Reporter: George Papa
>Assignee: Jungtaek Lim
>Priority: Major
> Fix For: 2.4.5, 3.0.0
>
> Attachments: test_csvs.zip
>
>
> I used Spark 2.1.1 and I upgraded into new versions. After Spark version
> 2.3.3, I observed from Spark UI that the driver memory is{color:#ff}
> increasing continuously.{color}
> In more detail, the driver memory and executors memory have the same used
> memory storage and after each iteration the storage memory is increasing. You
> can reproduce this behavior by running the following snippet code. The
> following example, is very simple, without any dataframe persistence, but the
> memory consumption is not stable as it was in former Spark versions
> (Specifically until Spark 2.3.2).
> Also, I tested with Spark streaming and structured streaming API and I had
> the same behavior. I tested with an existing application which reads from
> Kafka source and do some aggregations, persist dataframes and then unpersist
> them. The persist and unpersist it works correct, I see the dataframes in the
> storage tab in Spark UI and after the unpersist, all dataframe have removed.
> But, after the unpersist the executors memory is not zero, BUT has the same
> value with the driver memory. This behavior also affects the application
> performance because the memory of the executors is increasing as the driver
> increasing and after a while the persisted dataframes are not fit in the
> executors memory and I have spill to disk.
> Another error which I had after a long running, was
> {color:#ff}java.lang.OutOfMemoryError: GC overhead limit exceeded, but I
> don't know if its relevant with the above behavior or not.{color}
>
> *HOW TO REPRODUCE THIS BEHAVIOR:*
> Create a very simple application(streaming count_file.py) in order to
> reproduce this behavior. This application reads CSV files from a directory,
> count the rows and then remove the processed files.
> {code:java}
> import time
> import os
> from pyspark.sql import SparkSession
> from pyspark.sql import functions as F
> from pyspark.sql import types as T
> target_dir = "..."
> spark=SparkSession.builder.appName("DataframeCount").getOrCreate()
> while True:
> for f in os.listdir(target_dir):
> df = spark.read.load(target_dir + f, format="csv")
> print("Number of records: {0}".format(df.count()))
> time.sleep(15){code}
> Submit code:
> {code:java}
> spark-submit
> --master spark://xxx.xxx.xx.xxx
> --deploy-mode client
> --executor-memory 4g
> --executor-cores 3
> streaming count_file.py
> {code}
>
> *TESTED CASES WITH THE SAME BEHAVIOUR:*
> * I tested with default settings (spark-defaults.conf)
> * Add spark.cleaner.periodicGC.interval 1min (or less)
> * {{Turn spark.cleaner.referenceTracking.blocking}}=false
> * Run the application in cluster mode
> * Increase/decrease the resources of the executors and driver
> * I tested with extraJavaOptions in driver and executor -XX:+UseG1GC
> -XX:InitiatingHeapOccupancyPercent=35 -XX:ConcGCThreads=12
>
> *DEPENDENCIES*
> * Operation system: Ubuntu 16.04.3 LTS
> * Java: jdk1.8.0_131 (tested also with jdk1.8.0_221)
> * Python: Python 2.7.12
>
> *NOTE:* In Spark 2.1.1 the driver memory consumption (Storage Memory tab) was
> extremely low and after the run of ContextCleaner and BlockManager the memory
> was decreasing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-29244) ArrayIndexOutOfBoundsException on TaskCompletionListener during releasing of memory blocks
[
https://issues.apache.org/jira/browse/SPARK-29244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
L. C. Hsieh reassigned SPARK-29244:
---
Assignee: L. C. Hsieh
> ArrayIndexOutOfBoundsException on TaskCompletionListener during releasing of
> memory blocks
> --
>
> Key: SPARK-29244
> URL: https://issues.apache.org/jira/browse/SPARK-29244
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.4.0
> Environment: Release label:emr-5.20.0
> Hadoop distribution:Amazon 2.8.5
> Applications:Livy 0.5.0, Spark 2.4.0
>Reporter: Viacheslav Tradunsky
>Assignee: L. C. Hsieh
>Priority: Major
> Attachments: executor_oom.txt
>
>
> At the end of task completion an exception happened:
> {code:java}
> 19/09/25 09:03:58 ERROR TaskContextImpl: Error in
> TaskCompletionListener19/09/25 09:03:58 ERROR TaskContextImpl: Error in
> TaskCompletionListenerjava.lang.ArrayIndexOutOfBoundsException: -3 at
> org.apache.spark.memory.TaskMemoryManager.freePage(TaskMemoryManager.java:333)
> at org.apache.spark.memory.MemoryConsumer.freePage(MemoryConsumer.java:130)
> at org.apache.spark.memory.MemoryConsumer.freeArray(MemoryConsumer.java:108)
> at org.apache.spark.unsafe.map.BytesToBytesMap.free(BytesToBytesMap.java:803)
> at
> org.apache.spark.sql.execution.UnsafeFixedWidthAggregationMap.free(UnsafeFixedWidthAggregationMap.java:225)
> at
> org.apache.spark.sql.execution.UnsafeFixedWidthAggregationMap.lambda$new$0(UnsafeFixedWidthAggregationMap.java:111)
> at
> org.apache.spark.TaskContextImpl$$anonfun$markTaskCompleted$1.apply(TaskContextImpl.scala:117)
> at
> org.apache.spark.TaskContextImpl$$anonfun$markTaskCompleted$1.apply(TaskContextImpl.scala:117)
> at
> org.apache.spark.TaskContextImpl$$anonfun$invokeListeners$1.apply(TaskContextImpl.scala:130)
> at
> org.apache.spark.TaskContextImpl$$anonfun$invokeListeners$1.apply(TaskContextImpl.scala:128)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at
> org.apache.spark.TaskContextImpl.invokeListeners(TaskContextImpl.scala:128)
> at
> org.apache.spark.TaskContextImpl.markTaskCompleted(TaskContextImpl.scala:116)
> at org.apache.spark.scheduler.Task.run(Task.scala:131) at
> org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402)
> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360) at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408) at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
>
> Important to note, that before this one, there was OOM of allocating some
> pages. It looks like everything related to each other, but on OOM the whole
> flow goes abnormally, so no resources are fried correctly.
> {code:java}
> java.lang.NullPointerExceptionjava.lang.NullPointerException at
> org.apache.spark.util.collection.unsafe.sort.UnsafeInMemorySorter.getMemoryUsage(UnsafeInMemorySorter.java:208)
> at
> org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.getMemoryUsage(UnsafeExternalSorter.java:249)
> at
> org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.updatePeakMemoryUsed(UnsafeExternalSorter.java:253)
> at
> org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.freeMemory(UnsafeExternalSorter.java:296)
> at
> org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.cleanupResources(UnsafeExternalSorter.java:328)
> at
> org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.lambda$new$0(UnsafeExternalSorter.java:178)
> at
> org.apache.spark.TaskContextImpl$$anonfun$markTaskCompleted$1.apply(TaskContextImpl.scala:117)
> at
> org.apache.spark.TaskContextImpl$$anonfun$markTaskCompleted$1.apply(TaskContextImpl.scala:117)
> at
> org.apache.spark.TaskContextImpl$$anonfun$invokeListeners$1.apply(TaskContextImpl.scala:130)
> at
> org.apache.spark.TaskContextImpl$$anonfun$invokeListeners$1.apply(TaskContextImpl.scala:128)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at
> org.apache.spark.TaskContextImpl.invokeListeners(TaskContextImpl.scala:128)
> at
> org.apache.spark.TaskContextImpl.markTaskCompleted(TaskContextImpl.scala:116)
> at org.apache.spark.scheduler.Task.run(Task.scala:131) at
> org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402)
> at
[jira] [Resolved] (SPARK-29244) ArrayIndexOutOfBoundsException on TaskCompletionListener during releasing of memory blocks
[
https://issues.apache.org/jira/browse/SPARK-29244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
L. C. Hsieh resolved SPARK-29244.
-
Fix Version/s: 3.0.0
2.4.5
Resolution: Fixed
Issue resolved by pull request 25953
[https://github.com/apache/spark/pull/25953]
> ArrayIndexOutOfBoundsException on TaskCompletionListener during releasing of
> memory blocks
> --
>
> Key: SPARK-29244
> URL: https://issues.apache.org/jira/browse/SPARK-29244
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.4.0
> Environment: Release label:emr-5.20.0
> Hadoop distribution:Amazon 2.8.5
> Applications:Livy 0.5.0, Spark 2.4.0
>Reporter: Viacheslav Tradunsky
>Assignee: L. C. Hsieh
>Priority: Major
> Fix For: 2.4.5, 3.0.0
>
> Attachments: executor_oom.txt
>
>
> At the end of task completion an exception happened:
> {code:java}
> 19/09/25 09:03:58 ERROR TaskContextImpl: Error in
> TaskCompletionListener19/09/25 09:03:58 ERROR TaskContextImpl: Error in
> TaskCompletionListenerjava.lang.ArrayIndexOutOfBoundsException: -3 at
> org.apache.spark.memory.TaskMemoryManager.freePage(TaskMemoryManager.java:333)
> at org.apache.spark.memory.MemoryConsumer.freePage(MemoryConsumer.java:130)
> at org.apache.spark.memory.MemoryConsumer.freeArray(MemoryConsumer.java:108)
> at org.apache.spark.unsafe.map.BytesToBytesMap.free(BytesToBytesMap.java:803)
> at
> org.apache.spark.sql.execution.UnsafeFixedWidthAggregationMap.free(UnsafeFixedWidthAggregationMap.java:225)
> at
> org.apache.spark.sql.execution.UnsafeFixedWidthAggregationMap.lambda$new$0(UnsafeFixedWidthAggregationMap.java:111)
> at
> org.apache.spark.TaskContextImpl$$anonfun$markTaskCompleted$1.apply(TaskContextImpl.scala:117)
> at
> org.apache.spark.TaskContextImpl$$anonfun$markTaskCompleted$1.apply(TaskContextImpl.scala:117)
> at
> org.apache.spark.TaskContextImpl$$anonfun$invokeListeners$1.apply(TaskContextImpl.scala:130)
> at
> org.apache.spark.TaskContextImpl$$anonfun$invokeListeners$1.apply(TaskContextImpl.scala:128)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at
> org.apache.spark.TaskContextImpl.invokeListeners(TaskContextImpl.scala:128)
> at
> org.apache.spark.TaskContextImpl.markTaskCompleted(TaskContextImpl.scala:116)
> at org.apache.spark.scheduler.Task.run(Task.scala:131) at
> org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402)
> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360) at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408) at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
>
> Important to note, that before this one, there was OOM of allocating some
> pages. It looks like everything related to each other, but on OOM the whole
> flow goes abnormally, so no resources are fried correctly.
> {code:java}
> java.lang.NullPointerExceptionjava.lang.NullPointerException at
> org.apache.spark.util.collection.unsafe.sort.UnsafeInMemorySorter.getMemoryUsage(UnsafeInMemorySorter.java:208)
> at
> org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.getMemoryUsage(UnsafeExternalSorter.java:249)
> at
> org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.updatePeakMemoryUsed(UnsafeExternalSorter.java:253)
> at
> org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.freeMemory(UnsafeExternalSorter.java:296)
> at
> org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.cleanupResources(UnsafeExternalSorter.java:328)
> at
> org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.lambda$new$0(UnsafeExternalSorter.java:178)
> at
> org.apache.spark.TaskContextImpl$$anonfun$markTaskCompleted$1.apply(TaskContextImpl.scala:117)
> at
> org.apache.spark.TaskContextImpl$$anonfun$markTaskCompleted$1.apply(TaskContextImpl.scala:117)
> at
> org.apache.spark.TaskContextImpl$$anonfun$invokeListeners$1.apply(TaskContextImpl.scala:130)
> at
> org.apache.spark.TaskContextImpl$$anonfun$invokeListeners$1.apply(TaskContextImpl.scala:128)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at
> org.apache.spark.TaskContextImpl.invokeListeners(TaskContextImpl.scala:128)
> at
> org.apache.spark.TaskContextImpl.markTaskCompleted(TaskContextImpl.scala:116)
> at
[jira] [Commented] (SPARK-29055) Memory leak in Spark
[
https://issues.apache.org/jira/browse/SPARK-29055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942231#comment-16942231
]
L. C. Hsieh commented on SPARK-29055:
-
Should we update the title of this ticket? It is not real memory leak in Spark,
right?
> Memory leak in Spark
>
>
> Key: SPARK-29055
> URL: https://issues.apache.org/jira/browse/SPARK-29055
> Project: Spark
> Issue Type: Bug
> Components: Block Manager, Spark Core
>Affects Versions: 2.3.3
>Reporter: George Papa
>Assignee: Jungtaek Lim
>Priority: Major
> Fix For: 2.4.5, 3.0.0
>
> Attachments: test_csvs.zip
>
>
> I used Spark 2.1.1 and I upgraded into new versions. After Spark version
> 2.3.3, I observed from Spark UI that the driver memory is{color:#ff}
> increasing continuously.{color}
> In more detail, the driver memory and executors memory have the same used
> memory storage and after each iteration the storage memory is increasing. You
> can reproduce this behavior by running the following snippet code. The
> following example, is very simple, without any dataframe persistence, but the
> memory consumption is not stable as it was in former Spark versions
> (Specifically until Spark 2.3.2).
> Also, I tested with Spark streaming and structured streaming API and I had
> the same behavior. I tested with an existing application which reads from
> Kafka source and do some aggregations, persist dataframes and then unpersist
> them. The persist and unpersist it works correct, I see the dataframes in the
> storage tab in Spark UI and after the unpersist, all dataframe have removed.
> But, after the unpersist the executors memory is not zero, BUT has the same
> value with the driver memory. This behavior also affects the application
> performance because the memory of the executors is increasing as the driver
> increasing and after a while the persisted dataframes are not fit in the
> executors memory and I have spill to disk.
> Another error which I had after a long running, was
> {color:#ff}java.lang.OutOfMemoryError: GC overhead limit exceeded, but I
> don't know if its relevant with the above behavior or not.{color}
>
> *HOW TO REPRODUCE THIS BEHAVIOR:*
> Create a very simple application(streaming count_file.py) in order to
> reproduce this behavior. This application reads CSV files from a directory,
> count the rows and then remove the processed files.
> {code:java}
> import time
> import os
> from pyspark.sql import SparkSession
> from pyspark.sql import functions as F
> from pyspark.sql import types as T
> target_dir = "..."
> spark=SparkSession.builder.appName("DataframeCount").getOrCreate()
> while True:
> for f in os.listdir(target_dir):
> df = spark.read.load(target_dir + f, format="csv")
> print("Number of records: {0}".format(df.count()))
> time.sleep(15){code}
> Submit code:
> {code:java}
> spark-submit
> --master spark://xxx.xxx.xx.xxx
> --deploy-mode client
> --executor-memory 4g
> --executor-cores 3
> streaming count_file.py
> {code}
>
> *TESTED CASES WITH THE SAME BEHAVIOUR:*
> * I tested with default settings (spark-defaults.conf)
> * Add spark.cleaner.periodicGC.interval 1min (or less)
> * {{Turn spark.cleaner.referenceTracking.blocking}}=false
> * Run the application in cluster mode
> * Increase/decrease the resources of the executors and driver
> * I tested with extraJavaOptions in driver and executor -XX:+UseG1GC
> -XX:InitiatingHeapOccupancyPercent=35 -XX:ConcGCThreads=12
>
> *DEPENDENCIES*
> * Operation system: Ubuntu 16.04.3 LTS
> * Java: jdk1.8.0_131 (tested also with jdk1.8.0_221)
> * Python: Python 2.7.12
>
> *NOTE:* In Spark 2.1.1 the driver memory consumption (Storage Memory tab) was
> extremely low and after the run of ContextCleaner and BlockManager the memory
> was decreasing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-29055) Memory leak in Spark
[
https://issues.apache.org/jira/browse/SPARK-29055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942224#comment-16942224
]
Jim Kleckner commented on SPARK-29055:
--
[~vanzin] It seems from the previous comment that it is not resolved by PR
#25973 as the memory continues to grow.
[~Geopap] I recorded an internal tech talk about how to connect/grab heap
dump/analyze with jvisualvm if you are interested here:
[https://www.youtube.com/channel/UCA81uPFG3aqo2X1YgZRAoEg]
> Memory leak in Spark
>
>
> Key: SPARK-29055
> URL: https://issues.apache.org/jira/browse/SPARK-29055
> Project: Spark
> Issue Type: Bug
> Components: Block Manager, Spark Core
>Affects Versions: 2.3.3
>Reporter: George Papa
>Assignee: Jungtaek Lim
>Priority: Major
> Fix For: 2.4.5, 3.0.0
>
> Attachments: test_csvs.zip
>
>
> I used Spark 2.1.1 and I upgraded into new versions. After Spark version
> 2.3.3, I observed from Spark UI that the driver memory is{color:#ff}
> increasing continuously.{color}
> In more detail, the driver memory and executors memory have the same used
> memory storage and after each iteration the storage memory is increasing. You
> can reproduce this behavior by running the following snippet code. The
> following example, is very simple, without any dataframe persistence, but the
> memory consumption is not stable as it was in former Spark versions
> (Specifically until Spark 2.3.2).
> Also, I tested with Spark streaming and structured streaming API and I had
> the same behavior. I tested with an existing application which reads from
> Kafka source and do some aggregations, persist dataframes and then unpersist
> them. The persist and unpersist it works correct, I see the dataframes in the
> storage tab in Spark UI and after the unpersist, all dataframe have removed.
> But, after the unpersist the executors memory is not zero, BUT has the same
> value with the driver memory. This behavior also affects the application
> performance because the memory of the executors is increasing as the driver
> increasing and after a while the persisted dataframes are not fit in the
> executors memory and I have spill to disk.
> Another error which I had after a long running, was
> {color:#ff}java.lang.OutOfMemoryError: GC overhead limit exceeded, but I
> don't know if its relevant with the above behavior or not.{color}
>
> *HOW TO REPRODUCE THIS BEHAVIOR:*
> Create a very simple application(streaming count_file.py) in order to
> reproduce this behavior. This application reads CSV files from a directory,
> count the rows and then remove the processed files.
> {code:java}
> import time
> import os
> from pyspark.sql import SparkSession
> from pyspark.sql import functions as F
> from pyspark.sql import types as T
> target_dir = "..."
> spark=SparkSession.builder.appName("DataframeCount").getOrCreate()
> while True:
> for f in os.listdir(target_dir):
> df = spark.read.load(target_dir + f, format="csv")
> print("Number of records: {0}".format(df.count()))
> time.sleep(15){code}
> Submit code:
> {code:java}
> spark-submit
> --master spark://xxx.xxx.xx.xxx
> --deploy-mode client
> --executor-memory 4g
> --executor-cores 3
> streaming count_file.py
> {code}
>
> *TESTED CASES WITH THE SAME BEHAVIOUR:*
> * I tested with default settings (spark-defaults.conf)
> * Add spark.cleaner.periodicGC.interval 1min (or less)
> * {{Turn spark.cleaner.referenceTracking.blocking}}=false
> * Run the application in cluster mode
> * Increase/decrease the resources of the executors and driver
> * I tested with extraJavaOptions in driver and executor -XX:+UseG1GC
> -XX:InitiatingHeapOccupancyPercent=35 -XX:ConcGCThreads=12
>
> *DEPENDENCIES*
> * Operation system: Ubuntu 16.04.3 LTS
> * Java: jdk1.8.0_131 (tested also with jdk1.8.0_221)
> * Python: Python 2.7.12
>
> *NOTE:* In Spark 2.1.1 the driver memory consumption (Storage Memory tab) was
> extremely low and after the run of ContextCleaner and BlockManager the memory
> was decreasing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-29055) Memory leak in Spark
[
https://issues.apache.org/jira/browse/SPARK-29055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcelo Masiero Vanzin resolved SPARK-29055.
Fix Version/s: 3.0.0
2.4.5
Assignee: Jungtaek Lim
Resolution: Fixed
> Memory leak in Spark
>
>
> Key: SPARK-29055
> URL: https://issues.apache.org/jira/browse/SPARK-29055
> Project: Spark
> Issue Type: Bug
> Components: Block Manager, Spark Core
>Affects Versions: 2.3.3
>Reporter: George Papa
>Assignee: Jungtaek Lim
>Priority: Major
> Fix For: 2.4.5, 3.0.0
>
> Attachments: test_csvs.zip
>
>
> I used Spark 2.1.1 and I upgraded into new versions. After Spark version
> 2.3.3, I observed from Spark UI that the driver memory is{color:#ff}
> increasing continuously.{color}
> In more detail, the driver memory and executors memory have the same used
> memory storage and after each iteration the storage memory is increasing. You
> can reproduce this behavior by running the following snippet code. The
> following example, is very simple, without any dataframe persistence, but the
> memory consumption is not stable as it was in former Spark versions
> (Specifically until Spark 2.3.2).
> Also, I tested with Spark streaming and structured streaming API and I had
> the same behavior. I tested with an existing application which reads from
> Kafka source and do some aggregations, persist dataframes and then unpersist
> them. The persist and unpersist it works correct, I see the dataframes in the
> storage tab in Spark UI and after the unpersist, all dataframe have removed.
> But, after the unpersist the executors memory is not zero, BUT has the same
> value with the driver memory. This behavior also affects the application
> performance because the memory of the executors is increasing as the driver
> increasing and after a while the persisted dataframes are not fit in the
> executors memory and I have spill to disk.
> Another error which I had after a long running, was
> {color:#ff}java.lang.OutOfMemoryError: GC overhead limit exceeded, but I
> don't know if its relevant with the above behavior or not.{color}
>
> *HOW TO REPRODUCE THIS BEHAVIOR:*
> Create a very simple application(streaming count_file.py) in order to
> reproduce this behavior. This application reads CSV files from a directory,
> count the rows and then remove the processed files.
> {code:java}
> import time
> import os
> from pyspark.sql import SparkSession
> from pyspark.sql import functions as F
> from pyspark.sql import types as T
> target_dir = "..."
> spark=SparkSession.builder.appName("DataframeCount").getOrCreate()
> while True:
> for f in os.listdir(target_dir):
> df = spark.read.load(target_dir + f, format="csv")
> print("Number of records: {0}".format(df.count()))
> time.sleep(15){code}
> Submit code:
> {code:java}
> spark-submit
> --master spark://xxx.xxx.xx.xxx
> --deploy-mode client
> --executor-memory 4g
> --executor-cores 3
> streaming count_file.py
> {code}
>
> *TESTED CASES WITH THE SAME BEHAVIOUR:*
> * I tested with default settings (spark-defaults.conf)
> * Add spark.cleaner.periodicGC.interval 1min (or less)
> * {{Turn spark.cleaner.referenceTracking.blocking}}=false
> * Run the application in cluster mode
> * Increase/decrease the resources of the executors and driver
> * I tested with extraJavaOptions in driver and executor -XX:+UseG1GC
> -XX:InitiatingHeapOccupancyPercent=35 -XX:ConcGCThreads=12
>
> *DEPENDENCIES*
> * Operation system: Ubuntu 16.04.3 LTS
> * Java: jdk1.8.0_131 (tested also with jdk1.8.0_221)
> * Python: Python 2.7.12
>
> *NOTE:* In Spark 2.1.1 the driver memory consumption (Storage Memory tab) was
> extremely low and after the run of ContextCleaner and BlockManager the memory
> was decreasing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-20427) Issue with Spark interpreting Oracle datatype NUMBER
[
https://issues.apache.org/jira/browse/SPARK-20427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942090#comment-16942090
]
Paul Wu edited comment on SPARK-20427 at 10/1/19 4:13 PM:
--
Some one asked me this problem months ago and I found a solution for him , but
I forgot the solution when another one in my team asked me again yesterday. I
had to spend several hours on this since her query was quite complex. For a
record and my own reference, I would like to put the solution here (inspired
by [~sobusiak] and [~yumwang] ): Add the customSchema option after the read()
that specifies all potential trouble makers as Double types. It can probably
resolve most cases in real applications. Surely, this is supposed that one
does not particularly concern about the exact significant digits in his/her
applications.
{code:java}
.read()
.option("customSchema", "col1 Double, col2 Double") //where col1, col2... are
columns that could cause the trouble.
{code}
Also, some may think their issues come from the .write() operation, but the
issues are in fact from the .read() operation. The col1, col2...column names
are not necessarily from the original tables. They could be the calculated
fields for output in the queries. One could mistakenly bark at a wrong place
to try to fix the issues.
was (Author: zwu@gmail.com):
Some one asked me this problem months ago and I found a solution for him , but
I forgot the solution when another one in my team asked me again yesterday. I
had to spend several hours on this since her query was quite complex. For a
record and my own reference, I would like to put the solution here (inspired
by [~sobusiak] and [~yumwang] ): Add the customSchema option after the read()
that specifies all potential trouble makers as Double types. It can probably
resolve most cases in real applications. Surely, this is supposed that one
does not particularly concern about the exact significant digits in his/her
applications.
{code:java}
.read()
.option("customSchema", "col1 Double, col2 Double") //where col1, col2... are
columns that could cause the trouble.
{code}
> Issue with Spark interpreting Oracle datatype NUMBER
>
>
> Key: SPARK-20427
> URL: https://issues.apache.org/jira/browse/SPARK-20427
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.1.0
>Reporter: Alexander Andrushenko
>Assignee: Yuming Wang
>Priority: Major
> Fix For: 2.3.0
>
>
> In Oracle exists data type NUMBER. When defining a filed in a table of type
> NUMBER the field has two components, precision and scale.
> For example, NUMBER(p,s) has precision p and scale s.
> Precision can range from 1 to 38.
> Scale can range from -84 to 127.
> When reading such a filed Spark can create numbers with precision exceeding
> 38. In our case it has created fields with precision 44,
> calculated as sum of the precision (in our case 34 digits) and the scale (10):
> "...java.lang.IllegalArgumentException: requirement failed: Decimal precision
> 44 exceeds max precision 38...".
> The result was, that a data frame was read from a table on one schema but
> could not be inserted in the identical table on other schema.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-29320) Compare `sql/core` module in JDK8/11 (Part 1)
[ https://issues.apache.org/jira/browse/SPARK-29320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29320: -- Parent: SPARK-29194 Issue Type: Sub-task (was: Task) > Compare `sql/core` module in JDK8/11 (Part 1) > - > > Key: SPARK-29320 > URL: https://issues.apache.org/jira/browse/SPARK-29320 > Project: Spark > Issue Type: Sub-task > Components: Tests >Affects Versions: 3.0.0 >Reporter: Dongjoon Hyun >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-29320) Compare `sql/core` module in JDK8/11 (Part 1)
Dongjoon Hyun created SPARK-29320: - Summary: Compare `sql/core` module in JDK8/11 (Part 1) Key: SPARK-29320 URL: https://issues.apache.org/jira/browse/SPARK-29320 Project: Spark Issue Type: Task Components: Tests Affects Versions: 3.0.0 Reporter: Dongjoon Hyun -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-29319) Memory and disk usage not accurate when blocks are evicted and re-loaded in memory
Marcelo Masiero Vanzin created SPARK-29319:
--
Summary: Memory and disk usage not accurate when blocks are
evicted and re-loaded in memory
Key: SPARK-29319
URL: https://issues.apache.org/jira/browse/SPARK-29319
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 2.4.4, 3.0.0
Reporter: Marcelo Masiero Vanzin
I found this while running more targeted tests for the underlying behavior of
this code, triggered by SPARK-27468. I ran this code:
{code}
import java.util.Arrays
import org.apache.spark.rdd._
import org.apache.spark.storage._
def newCachedRDD(level: StorageLevel): RDD[Array[Long]] = {
val rdd = sc.parallelize(1 to 64, 64).map { i =>
val a = new Array[Long](1024 * 1024)
Arrays.fill(a, i)
a
}
rdd.persist(level)
rdd.count()
rdd
}
val r1 = newCachedRDD(level = StorageLevel.MEMORY_AND_DISK)
val r2 = newCachedRDD(level = StorageLevel.MEMORY_ONLY)
{code}
With {{./bin/spark-shell --master 'local-cluster[1,1,1024]'}}.
After it runs, you end up with the expected values: r1 has everything cached,
only using disk, because all its memory blocks were evicted by r2; r2 has as
many blocks as the memory can hold.
The problem shows up when you start playing with those RDDs again.
Calling {{r1.count()}} will cause all of r2's blocks to be evicted, since r1's
blocks are loaded back in memory. But no block update is sent to the driver
about that load, so the driver does not know that the blocks are now in memory.
The UI will show that r1 is using 0 bytes of memory, and r2 disappears from the
storage page (this last one as expected).
Calling {{r2.count()}} after that will cause r1's blocks to be evicted again.
This will send updates to the driver, which will now double-count the disk
usage. So if you keep doing this back and forth, r1's disk usage will keep
growing in the UI, when in fact it doesn't change at all.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-29288) Spark SQL add jar can't support HTTP path.
[ https://issues.apache.org/jira/browse/SPARK-29288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942095#comment-16942095 ] Dongjoon Hyun commented on SPARK-29288: --- What is your example, [~angerszhuuu]? > Spark SQL add jar can't support HTTP path. > --- > > Key: SPARK-29288 > URL: https://issues.apache.org/jira/browse/SPARK-29288 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 2.4.0, 3.0.0 >Reporter: angerszhu >Priority: Major > > SparkSQL > `ADD JAR` can't support url with http, livy schema , do we need to support it? > cc [~sro...@scient.com] > [~hyukjin.kwon][~dongjoon][~jerryshao][~juliuszsompolski] > Hive 2.3 support it, do we need to support it? > I can work on this. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27475) parquet-hadoop-bundle is incorrect in dev/deps/spark-deps-hadoop-3.2
[ https://issues.apache.org/jira/browse/SPARK-27475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942093#comment-16942093 ] Dongjoon Hyun commented on SPARK-27475: --- SPARK-29318 is created for `orc`. > parquet-hadoop-bundle is incorrect in dev/deps/spark-deps-hadoop-3.2 > > > Key: SPARK-27475 > URL: https://issues.apache.org/jira/browse/SPARK-27475 > Project: Spark > Issue Type: Sub-task > Components: Build >Affects Versions: 3.0.0 >Reporter: Yuming Wang >Priority: Major > > parquet-hadoop-bundle-1.6.0.jar should be parquet-hadoop-bundle-1.8.1.jar. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-29318) orc library is incorrect in dev/deps/spark-deps-hadoop-3.2
[
https://issues.apache.org/jira/browse/SPARK-29318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942092#comment-16942092
]
Dongjoon Hyun commented on SPARK-29318:
---
cc [~yumwang]
> orc library is incorrect in dev/deps/spark-deps-hadoop-3.2
> ---
>
> Key: SPARK-29318
> URL: https://issues.apache.org/jira/browse/SPARK-29318
> Project: Spark
> Issue Type: Sub-task
> Components: Build
>Affects Versions: 3.0.0
>Reporter: Dongjoon Hyun
>Priority: Major
>
> {code}
> orc-core-1.5.6-nohive.jar
> orc-mapreduce-1.5.6-nohive.jar
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-29318) orc library is incorrect in dev/deps/spark-deps-hadoop-3.2
Dongjoon Hyun created SPARK-29318:
-
Summary: orc library is incorrect in
dev/deps/spark-deps-hadoop-3.2
Key: SPARK-29318
URL: https://issues.apache.org/jira/browse/SPARK-29318
Project: Spark
Issue Type: Bug
Components: Build
Affects Versions: 3.0.0
Reporter: Dongjoon Hyun
{code}
orc-core-1.5.6-nohive.jar
orc-mapreduce-1.5.6-nohive.jar
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-29318) orc library is incorrect in dev/deps/spark-deps-hadoop-3.2
[
https://issues.apache.org/jira/browse/SPARK-29318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-29318:
--
Parent: SPARK-23710
Issue Type: Sub-task (was: Bug)
> orc library is incorrect in dev/deps/spark-deps-hadoop-3.2
> ---
>
> Key: SPARK-29318
> URL: https://issues.apache.org/jira/browse/SPARK-29318
> Project: Spark
> Issue Type: Sub-task
> Components: Build
>Affects Versions: 3.0.0
>Reporter: Dongjoon Hyun
>Priority: Major
>
> {code}
> orc-core-1.5.6-nohive.jar
> orc-mapreduce-1.5.6-nohive.jar
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-20427) Issue with Spark interpreting Oracle datatype NUMBER
[
https://issues.apache.org/jira/browse/SPARK-20427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942090#comment-16942090
]
Paul Wu edited comment on SPARK-20427 at 10/1/19 3:49 PM:
--
Some one asked me this problem months ago and I found a solution for him , but
I forgot the solution when another one in my team asked me again yesterday. I
had to spend several hours on this since her query was quite complex. For a
record and my own reference, I would like to put the solution here (inspired
by [~sobusiak] and [~yumwang] ): Add the customSchema option after the read()
that specifies all potential trouble makers as Double types. It can probably
resolve most cases in real applications. Surely, this is supposed that one
does not particularly concern about the exact significant digits in his/her
applications.
{code:java}
.read()
.option("customSchema", "col1 Double, col2 Double") //where col1, col2... are
columns that could cause the trouble.
{code}
was (Author: zwu@gmail.com):
Some one asked me this problem months ago and I found a solution for him , but
I forgot the solution when another one in my team asked me again yesterday. I
had to spend several hours on this since her query was quite complex. For a
record and my own reference, I would like to put the solution here (inspired
by [~sobusiak] and [~yumwang] ): Add the customSchema option after the read()
that specifies all potential trouble makers as Double types. It can probably
resolve most cases in real applications. Surely, this is supposed one does not
particularly concern about the exact significant digits in his/her
applications.
{code:java}
.read()
.option("customSchema", "col1 Double, col2 Double") //where col1, col2... are
columns that could cause the trouble.
{code}
> Issue with Spark interpreting Oracle datatype NUMBER
>
>
> Key: SPARK-20427
> URL: https://issues.apache.org/jira/browse/SPARK-20427
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.1.0
>Reporter: Alexander Andrushenko
>Assignee: Yuming Wang
>Priority: Major
> Fix For: 2.3.0
>
>
> In Oracle exists data type NUMBER. When defining a filed in a table of type
> NUMBER the field has two components, precision and scale.
> For example, NUMBER(p,s) has precision p and scale s.
> Precision can range from 1 to 38.
> Scale can range from -84 to 127.
> When reading such a filed Spark can create numbers with precision exceeding
> 38. In our case it has created fields with precision 44,
> calculated as sum of the precision (in our case 34 digits) and the scale (10):
> "...java.lang.IllegalArgumentException: requirement failed: Decimal precision
> 44 exceeds max precision 38...".
> The result was, that a data frame was read from a table on one schema but
> could not be inserted in the identical table on other schema.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-27475) parquet-hadoop-bundle is incorrect in dev/deps/spark-deps-hadoop-3.2
[ https://issues.apache.org/jira/browse/SPARK-27475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27475: -- Summary: parquet-hadoop-bundle is incorrect in dev/deps/spark-deps-hadoop-3.2 (was: dev/deps/spark-deps-hadoop-3.2 is incorrect) > parquet-hadoop-bundle is incorrect in dev/deps/spark-deps-hadoop-3.2 > > > Key: SPARK-27475 > URL: https://issues.apache.org/jira/browse/SPARK-27475 > Project: Spark > Issue Type: Sub-task > Components: Build >Affects Versions: 3.0.0 >Reporter: Yuming Wang >Priority: Major > > parquet-hadoop-bundle-1.6.0.jar should be parquet-hadoop-bundle-1.8.1.jar. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20427) Issue with Spark interpreting Oracle datatype NUMBER
[
https://issues.apache.org/jira/browse/SPARK-20427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942090#comment-16942090
]
Paul Wu commented on SPARK-20427:
-
Some one asked me this problem months ago and I found a solution for him , but
I forgot the solution when another one in my team asked me again yesterday. I
had to spend several hours on this since her query was quite complex. For a
record and my own reference, I would like to put the solution here (inspired
by [~sobusiak] and [~yumwang] ): Add the customSchema option after the read()
that specifies all potential trouble makers as Double types. It can probably
resolve most cases in real applications. Surely, this is supposed one does not
particularly concern about the exact significant digits in his/her
applications.
{code:java}
.read()
.option("customSchema", "col1 Double, col2 Double") //where col1, col2... are
columns that could cause the trouble.
{code}
> Issue with Spark interpreting Oracle datatype NUMBER
>
>
> Key: SPARK-20427
> URL: https://issues.apache.org/jira/browse/SPARK-20427
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.1.0
>Reporter: Alexander Andrushenko
>Assignee: Yuming Wang
>Priority: Major
> Fix For: 2.3.0
>
>
> In Oracle exists data type NUMBER. When defining a filed in a table of type
> NUMBER the field has two components, precision and scale.
> For example, NUMBER(p,s) has precision p and scale s.
> Precision can range from 1 to 38.
> Scale can range from -84 to 127.
> When reading such a filed Spark can create numbers with precision exceeding
> 38. In our case it has created fields with precision 44,
> calculated as sum of the precision (in our case 34 digits) and the scale (10):
> "...java.lang.IllegalArgumentException: requirement failed: Decimal precision
> 44 exceeds max precision 38...".
> The result was, that a data frame was read from a table on one schema but
> could not be inserted in the identical table on other schema.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-27475) dev/deps/spark-deps-hadoop-3.2 is incorrect
[ https://issues.apache.org/jira/browse/SPARK-27475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942088#comment-16942088 ] Dongjoon Hyun commented on SPARK-27475: --- That's right, but this issue didn't aim that. Let me have one for `orc`. > dev/deps/spark-deps-hadoop-3.2 is incorrect > --- > > Key: SPARK-27475 > URL: https://issues.apache.org/jira/browse/SPARK-27475 > Project: Spark > Issue Type: Sub-task > Components: Build >Affects Versions: 3.0.0 >Reporter: Yuming Wang >Priority: Major > > parquet-hadoop-bundle-1.6.0.jar should be parquet-hadoop-bundle-1.8.1.jar. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-29022) SparkSQLCLI can not use 'ADD JAR' 's jar as Serde class
[
https://issues.apache.org/jira/browse/SPARK-29022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen updated SPARK-29022:
-
Summary: SparkSQLCLI can not use 'ADD JAR' 's jar as Serde class (was:
SparkSQLCLI can not use 'ADD JAR' 's jar as Serder class)
> SparkSQLCLI can not use 'ADD JAR' 's jar as Serde class
> ---
>
> Key: SPARK-29022
> URL: https://issues.apache.org/jira/browse/SPARK-29022
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.0.2, 2.1.3, 2.2.3, 2.3.4, 2.4.4, 3.0.0
>Reporter: angerszhu
>Assignee: angerszhu
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: image-2019-09-17-13-54-50-896.png
>
>
> Spark SQL CLI can't use class in jars add by SQL 'ADD JAR'
> {code:java}
> spark-sql> add jar
> /root/.m2/repository/org/apache/hive/hcatalog/hive-hcatalog-core/2.3.6/hive-hcatalog-core-2.3.6.jar;
> ADD JAR
> /root/.m2/repository/org/apache/hive/hcatalog/hive-hcatalog-core/2.3.6/hive-hcatalog-core-2.3.6.jar
> spark-sql> CREATE TABLE addJar(key string) ROW FORMAT SERDE
> 'org.apache.hive.hcatalog.data.JsonSerDe';
> spark-sql> select * from addJar;
> 19/09/07 03:06:54 ERROR SparkSQLDriver: Failed in [select * from addJar]
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
> org.apache.hive.hcatalog.data.JsonSerDe
> at
> org.apache.hadoop.hive.ql.plan.TableDesc.getDeserializerClass(TableDesc.java:79)
> at
> org.apache.spark.sql.hive.execution.HiveTableScanExec.addColumnMetadataToConf(HiveTableScanExec.scala:123)
> at
> org.apache.spark.sql.hive.execution.HiveTableScanExec.hadoopConf$lzycompute(HiveTableScanExec.scala:101)
> at
> org.apache.spark.sql.hive.execution.HiveTableScanExec.hadoopConf(HiveTableScanExec.scala:98)
> at
> org.apache.spark.sql.hive.execution.HiveTableScanExec.hadoopReader$lzycompute(HiveTableScanExec.scala:110)
> at
> org.apache.spark.sql.hive.execution.HiveTableScanExec.hadoopReader(HiveTableScanExec.scala:105)
> at
> org.apache.spark.sql.hive.execution.HiveTableScanExec.$anonfun$doExecute$1(HiveTableScanExec.scala:188)
> at org.apache.spark.util.Utils$.withDummyCallSite(Utils.scala:2488)
> at
> org.apache.spark.sql.hive.execution.HiveTableScanExec.doExecute(HiveTableScanExec.scala:188)
> at
> org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:189)
> at
> org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:227)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
> at
> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:224)
> at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:185)
> at
> org.apache.spark.sql.execution.SparkPlan.getByteArrayRdd(SparkPlan.scala:329)
> at
> org.apache.spark.sql.execution.SparkPlan.executeCollect(SparkPlan.scala:378)
> at
> org.apache.spark.sql.execution.SparkPlan.executeCollectPublic(SparkPlan.scala:408)
> at
> org.apache.spark.sql.execution.HiveResult$.hiveResultString(HiveResult.scala:52)
> at
> org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.$anonfun$run$1(SparkSQLDriver.scala:65)
> at
> org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$4(SQLExecution.scala:100)
> at
> org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:160)
> at
> org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:87)
> at
> org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:65)
> at
> org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:367)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
> at
> org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:272)
> at
> org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
> at
> org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:920)
> at
> org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:179)
> at
[jira] [Resolved] (SPARK-29022) SparkSQLCLI can not use 'ADD JAR' 's jar as Serder class
[
https://issues.apache.org/jira/browse/SPARK-29022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen resolved SPARK-29022.
--
Fix Version/s: 3.0.0
Resolution: Fixed
Issue resolved by pull request 25729
[https://github.com/apache/spark/pull/25729]
> SparkSQLCLI can not use 'ADD JAR' 's jar as Serder class
>
>
> Key: SPARK-29022
> URL: https://issues.apache.org/jira/browse/SPARK-29022
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.0.2, 2.1.3, 2.2.3, 2.3.4, 2.4.4, 3.0.0
>Reporter: angerszhu
>Assignee: angerszhu
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: image-2019-09-17-13-54-50-896.png
>
>
> Spark SQL CLI can't use class in jars add by SQL 'ADD JAR'
> {code:java}
> spark-sql> add jar
> /root/.m2/repository/org/apache/hive/hcatalog/hive-hcatalog-core/2.3.6/hive-hcatalog-core-2.3.6.jar;
> ADD JAR
> /root/.m2/repository/org/apache/hive/hcatalog/hive-hcatalog-core/2.3.6/hive-hcatalog-core-2.3.6.jar
> spark-sql> CREATE TABLE addJar(key string) ROW FORMAT SERDE
> 'org.apache.hive.hcatalog.data.JsonSerDe';
> spark-sql> select * from addJar;
> 19/09/07 03:06:54 ERROR SparkSQLDriver: Failed in [select * from addJar]
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
> org.apache.hive.hcatalog.data.JsonSerDe
> at
> org.apache.hadoop.hive.ql.plan.TableDesc.getDeserializerClass(TableDesc.java:79)
> at
> org.apache.spark.sql.hive.execution.HiveTableScanExec.addColumnMetadataToConf(HiveTableScanExec.scala:123)
> at
> org.apache.spark.sql.hive.execution.HiveTableScanExec.hadoopConf$lzycompute(HiveTableScanExec.scala:101)
> at
> org.apache.spark.sql.hive.execution.HiveTableScanExec.hadoopConf(HiveTableScanExec.scala:98)
> at
> org.apache.spark.sql.hive.execution.HiveTableScanExec.hadoopReader$lzycompute(HiveTableScanExec.scala:110)
> at
> org.apache.spark.sql.hive.execution.HiveTableScanExec.hadoopReader(HiveTableScanExec.scala:105)
> at
> org.apache.spark.sql.hive.execution.HiveTableScanExec.$anonfun$doExecute$1(HiveTableScanExec.scala:188)
> at org.apache.spark.util.Utils$.withDummyCallSite(Utils.scala:2488)
> at
> org.apache.spark.sql.hive.execution.HiveTableScanExec.doExecute(HiveTableScanExec.scala:188)
> at
> org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:189)
> at
> org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:227)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
> at
> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:224)
> at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:185)
> at
> org.apache.spark.sql.execution.SparkPlan.getByteArrayRdd(SparkPlan.scala:329)
> at
> org.apache.spark.sql.execution.SparkPlan.executeCollect(SparkPlan.scala:378)
> at
> org.apache.spark.sql.execution.SparkPlan.executeCollectPublic(SparkPlan.scala:408)
> at
> org.apache.spark.sql.execution.HiveResult$.hiveResultString(HiveResult.scala:52)
> at
> org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.$anonfun$run$1(SparkSQLDriver.scala:65)
> at
> org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$4(SQLExecution.scala:100)
> at
> org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:160)
> at
> org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:87)
> at
> org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:65)
> at
> org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:367)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
> at
> org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:272)
> at
> org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
> at
> org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:920)
> at
> org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:179)
> at
[jira] [Assigned] (SPARK-29022) SparkSQLCLI can not use 'ADD JAR' 's jar as Serder class
[
https://issues.apache.org/jira/browse/SPARK-29022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen reassigned SPARK-29022:
Assignee: angerszhu
> SparkSQLCLI can not use 'ADD JAR' 's jar as Serder class
>
>
> Key: SPARK-29022
> URL: https://issues.apache.org/jira/browse/SPARK-29022
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.0.2, 2.1.3, 2.2.3, 2.3.4, 2.4.4, 3.0.0
>Reporter: angerszhu
>Assignee: angerszhu
>Priority: Major
> Attachments: image-2019-09-17-13-54-50-896.png
>
>
> Spark SQL CLI can't use class in jars add by SQL 'ADD JAR'
> {code:java}
> spark-sql> add jar
> /root/.m2/repository/org/apache/hive/hcatalog/hive-hcatalog-core/2.3.6/hive-hcatalog-core-2.3.6.jar;
> ADD JAR
> /root/.m2/repository/org/apache/hive/hcatalog/hive-hcatalog-core/2.3.6/hive-hcatalog-core-2.3.6.jar
> spark-sql> CREATE TABLE addJar(key string) ROW FORMAT SERDE
> 'org.apache.hive.hcatalog.data.JsonSerDe';
> spark-sql> select * from addJar;
> 19/09/07 03:06:54 ERROR SparkSQLDriver: Failed in [select * from addJar]
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
> org.apache.hive.hcatalog.data.JsonSerDe
> at
> org.apache.hadoop.hive.ql.plan.TableDesc.getDeserializerClass(TableDesc.java:79)
> at
> org.apache.spark.sql.hive.execution.HiveTableScanExec.addColumnMetadataToConf(HiveTableScanExec.scala:123)
> at
> org.apache.spark.sql.hive.execution.HiveTableScanExec.hadoopConf$lzycompute(HiveTableScanExec.scala:101)
> at
> org.apache.spark.sql.hive.execution.HiveTableScanExec.hadoopConf(HiveTableScanExec.scala:98)
> at
> org.apache.spark.sql.hive.execution.HiveTableScanExec.hadoopReader$lzycompute(HiveTableScanExec.scala:110)
> at
> org.apache.spark.sql.hive.execution.HiveTableScanExec.hadoopReader(HiveTableScanExec.scala:105)
> at
> org.apache.spark.sql.hive.execution.HiveTableScanExec.$anonfun$doExecute$1(HiveTableScanExec.scala:188)
> at org.apache.spark.util.Utils$.withDummyCallSite(Utils.scala:2488)
> at
> org.apache.spark.sql.hive.execution.HiveTableScanExec.doExecute(HiveTableScanExec.scala:188)
> at
> org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:189)
> at
> org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:227)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
> at
> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:224)
> at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:185)
> at
> org.apache.spark.sql.execution.SparkPlan.getByteArrayRdd(SparkPlan.scala:329)
> at
> org.apache.spark.sql.execution.SparkPlan.executeCollect(SparkPlan.scala:378)
> at
> org.apache.spark.sql.execution.SparkPlan.executeCollectPublic(SparkPlan.scala:408)
> at
> org.apache.spark.sql.execution.HiveResult$.hiveResultString(HiveResult.scala:52)
> at
> org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.$anonfun$run$1(SparkSQLDriver.scala:65)
> at
> org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$4(SQLExecution.scala:100)
> at
> org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:160)
> at
> org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:87)
> at
> org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:65)
> at
> org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:367)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
> at
> org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:272)
> at
> org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
> at
> org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:920)
> at
> org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:179)
> at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:202)
> at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:89)
> at
>
