[
https://issues.apache.org/jira/browse/SPARK-37254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramakrishna chilaka updated SPARK-37254:
----------------------------------------
Description:
We are trying to use Spark thrift server as a distributed sql query engine, the
queries work when the resident memory occupied by Spark thrift server
identified through HTOP is comparatively less than the driver memory. The same
queries result in 100% cpu usage when the resident memory occupied by spark
thrift server is greater than the configured driver memory and keeps running at
100% cpu usage. I am using incremental collect as false, as i need faster
responses for exploratory queries. I am trying to understand the following
points
* Why isn't spark thrift server releasing back the memory, when there are no
queries.
* What is causing spark thrift server to go into 100% cpu usage on all the
cores, when spark thrift server's memory is greater than the driver memory (by
10% usually) and why are queries just stuck.
was:
We are trying to use Spark thrift server as a distributed sql query engine, the
queries work when the resident memory occupied by Spark thrift server
identified through HTOP is comparatively less than the driver memory. The same
queries result in 100% cpu usage when the resident memory occupied by spark
thrift server is greater than the configured driver memory. I am using
incremental collect as false, as i need faster responses for exploratory
queries. I am trying to understand the following points
* Why isn't spark thrift server releasing back the memory, when there are no
queries.
* What is causing spark thrift server to go into 100% cpu usage on all the
cores and why are queries just stuck.
> 100% CPU usage on Spark Thrift Server.
> --------------------------------------
>
> Key: SPARK-37254
> URL: https://issues.apache.org/jira/browse/SPARK-37254
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 3.1.2
> Reporter: ramakrishna chilaka
> Priority: Major
>
> We are trying to use Spark thrift server as a distributed sql query engine,
> the queries work when the resident memory occupied by Spark thrift server
> identified through HTOP is comparatively less than the driver memory. The
> same queries result in 100% cpu usage when the resident memory occupied by
> spark thrift server is greater than the configured driver memory and keeps
> running at 100% cpu usage. I am using incremental collect as false, as i need
> faster responses for exploratory queries. I am trying to understand the
> following points
> * Why isn't spark thrift server releasing back the memory, when there are no
> queries.
> * What is causing spark thrift server to go into 100% cpu usage on all the
> cores, when spark thrift server's memory is greater than the driver memory
> (by 10% usually) and why are queries just stuck.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
