[jira] [Reopened] (SPARK-23130) Spark Thrift does not clean-up temporary files (/tmp/*_resources and /tmp/hive/*.pipeout)
[
https://issues.apache.org/jira/browse/SPARK-23130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Roberts reopened SPARK-23130:
--
> Spark Thrift does not clean-up temporary files (/tmp/*_resources and
> /tmp/hive/*.pipeout)
> -
>
> Key: SPARK-23130
> URL: https://issues.apache.org/jira/browse/SPARK-23130
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.6.3, 2.1.0, 2.2.0
> Environment: * Hadoop distributions: HDP 2.5 - 2.6.3.0
> * OS: Seen on SLES12, RHEL 7.3 & RHEL 7.4
>Reporter: Sean Roberts
>Priority: Major
> Labels: thrift
>
> Spark Thrift is not cleaning up /tmp for files & directories named like:
> /tmp/hive/*.pipeout
> /tmp/*_resources
> There are such a large number that /tmp quickly runs out of inodes *causing
> the partition to be unusable and many services to crash*. This is even true
> when the only jobs submitted are routine service checks.
> Used `strace` to show that Spark Thrift is responsible:
> {code:java}
> strace.out.118864:04:53:49
> open("/tmp/hive/55ad7fc1-f79a-4ad8-8e02-26bbeaa86bbc7288010135864174970.pipeout",
> O_RDWR|O_CREAT|O_EXCL, 0666) = 134
> strace.out.118864:04:53:49
> mkdir("/tmp/b6dfbf9e-2f7c-4c25-95a1-73c44318ecf4_resources", 0777) = 0
> {code}
> *Those files were left behind, even days later.*
>
> Example files:
> {code:java}
> # stat
> /tmp/hive/55ad7fc1-f79a-4ad8-8e02-26bbeaa86bbc7288010135864174970.pipeout
> File:
> ‘/tmp/hive/55ad7fc1-f79a-4ad8-8e02-26bbeaa86bbc7288010135864174970.pipeout’
> Size: 0 Blocks: 0 IO Block: 4096 regular empty file
> Device: fe09h/65033d Inode: 678 Links: 1
> Access: (0644/-rw-r--r--) Uid: ( 1000/hive) Gid: ( 1002/ hadoop)
> Access: 2017-12-19 04:53:49.126777260 -0600
> Modify: 2017-12-19 04:53:49.126777260 -0600
> Change: 2017-12-19 04:53:49.126777260 -0600
> Birth: -
> # stat /tmp/b6dfbf9e-2f7c-4c25-95a1-73c44318ecf4_resources
> File: ‘/tmp/b6dfbf9e-2f7c-4c25-95a1-73c44318ecf4_resources’
> Size: 4096 Blocks: 8 IO Block: 4096 directory
> Device: fe09h/65033d Inode: 668 Links: 2
> Access: (0700/drwx--) Uid: ( 1000/hive) Gid: ( 1002/ hadoop)
> Access: 2017-12-19 04:57:38.458937635 -0600
> Modify: 2017-12-19 04:53:49.062777216 -0600
> Change: 2017-12-19 04:53:49.066777218 -0600
> Birth: -
> {code}
> Showing the large number:
> {code:java}
> # find /tmp/ -name '*_resources' | wc -l
> 68340
> # find /tmp/hive -name "*.pipeout" | wc -l
> 51837
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23130) Spark Thrift does not clean-up temporary files (/tmp/*_resources and /tmp/hive/*.pipeout)
[
https://issues.apache.org/jira/browse/SPARK-23130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16329464#comment-16329464
]
Sean Roberts commented on SPARK-23130:
--
Marco - Which JIRA resolves the pipeout issue or incorporates the pipeout fixes
into Spark Thrift?
> Spark Thrift does not clean-up temporary files (/tmp/*_resources and
> /tmp/hive/*.pipeout)
> -
>
> Key: SPARK-23130
> URL: https://issues.apache.org/jira/browse/SPARK-23130
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.6.3, 2.1.0, 2.2.0
> Environment: * Hadoop distributions: HDP 2.5 - 2.6.3.0
> * OS: Seen on SLES12, RHEL 7.3 & RHEL 7.4
>Reporter: Sean Roberts
>Priority: Major
> Labels: thrift
>
> Spark Thrift is not cleaning up /tmp for files & directories named like:
> /tmp/hive/*.pipeout
> /tmp/*_resources
> There are such a large number that /tmp quickly runs out of inodes *causing
> the partition to be unusable and many services to crash*. This is even true
> when the only jobs submitted are routine service checks.
> Used `strace` to show that Spark Thrift is responsible:
> {code:java}
> strace.out.118864:04:53:49
> open("/tmp/hive/55ad7fc1-f79a-4ad8-8e02-26bbeaa86bbc7288010135864174970.pipeout",
> O_RDWR|O_CREAT|O_EXCL, 0666) = 134
> strace.out.118864:04:53:49
> mkdir("/tmp/b6dfbf9e-2f7c-4c25-95a1-73c44318ecf4_resources", 0777) = 0
> {code}
> *Those files were left behind, even days later.*
>
> Example files:
> {code:java}
> # stat
> /tmp/hive/55ad7fc1-f79a-4ad8-8e02-26bbeaa86bbc7288010135864174970.pipeout
> File:
> ‘/tmp/hive/55ad7fc1-f79a-4ad8-8e02-26bbeaa86bbc7288010135864174970.pipeout’
> Size: 0 Blocks: 0 IO Block: 4096 regular empty file
> Device: fe09h/65033d Inode: 678 Links: 1
> Access: (0644/-rw-r--r--) Uid: ( 1000/hive) Gid: ( 1002/ hadoop)
> Access: 2017-12-19 04:53:49.126777260 -0600
> Modify: 2017-12-19 04:53:49.126777260 -0600
> Change: 2017-12-19 04:53:49.126777260 -0600
> Birth: -
> # stat /tmp/b6dfbf9e-2f7c-4c25-95a1-73c44318ecf4_resources
> File: ‘/tmp/b6dfbf9e-2f7c-4c25-95a1-73c44318ecf4_resources’
> Size: 4096 Blocks: 8 IO Block: 4096 directory
> Device: fe09h/65033d Inode: 668 Links: 2
> Access: (0700/drwx--) Uid: ( 1000/hive) Gid: ( 1002/ hadoop)
> Access: 2017-12-19 04:57:38.458937635 -0600
> Modify: 2017-12-19 04:53:49.062777216 -0600
> Change: 2017-12-19 04:53:49.066777218 -0600
> Birth: -
> {code}
> Showing the large number:
> {code:java}
> # find /tmp/ -name '*_resources' | wc -l
> 68340
> # find /tmp/hive -name "*.pipeout" | wc -l
> 51837
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-23130) Spark Thrift does not clean-up temporary files (/tmp/*_resources and /tmp/hive/*.pipeout)
[
https://issues.apache.org/jira/browse/SPARK-23130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Roberts updated SPARK-23130:
-
Labels: thrift (was: )
> Spark Thrift does not clean-up temporary files (/tmp/*_resources and
> /tmp/hive/*.pipeout)
> -
>
> Key: SPARK-23130
> URL: https://issues.apache.org/jira/browse/SPARK-23130
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.6.3, 2.1.0, 2.2.0
> Environment: * Hadoop distributions: HDP 2.5 - 2.6.3.0
> * OS: Seen on SLES12, RHEL 7.3 & RHEL 7.4
>Reporter: Sean Roberts
>Priority: Major
> Labels: thrift
>
> Spark Thrift is not cleaning up /tmp for files & directories named like:
> /tmp/hive/*.pipeout
> /tmp/*_resources
> There are such a large number that /tmp quickly runs out of inodes *causing
> the partition to be unusable and many services to crash*. This is even true
> when the only jobs submitted are routine service checks.
> Used `strace` to show that Spark Thrift is responsible:
> {code:java}
> strace.out.118864:04:53:49
> open("/tmp/hive/55ad7fc1-f79a-4ad8-8e02-26bbeaa86bbc7288010135864174970.pipeout",
> O_RDWR|O_CREAT|O_EXCL, 0666) = 134
> strace.out.118864:04:53:49
> mkdir("/tmp/b6dfbf9e-2f7c-4c25-95a1-73c44318ecf4_resources", 0777) = 0
> {code}
> *Those files were left behind, even days later.*
>
> Example files:
> {code:java}
> # stat
> /tmp/hive/55ad7fc1-f79a-4ad8-8e02-26bbeaa86bbc7288010135864174970.pipeout
> File:
> ‘/tmp/hive/55ad7fc1-f79a-4ad8-8e02-26bbeaa86bbc7288010135864174970.pipeout’
> Size: 0 Blocks: 0 IO Block: 4096 regular empty file
> Device: fe09h/65033d Inode: 678 Links: 1
> Access: (0644/-rw-r--r--) Uid: ( 1000/hive) Gid: ( 1002/ hadoop)
> Access: 2017-12-19 04:53:49.126777260 -0600
> Modify: 2017-12-19 04:53:49.126777260 -0600
> Change: 2017-12-19 04:53:49.126777260 -0600
> Birth: -
> # stat /tmp/b6dfbf9e-2f7c-4c25-95a1-73c44318ecf4_resources
> File: ‘/tmp/b6dfbf9e-2f7c-4c25-95a1-73c44318ecf4_resources’
> Size: 4096 Blocks: 8 IO Block: 4096 directory
> Device: fe09h/65033d Inode: 668 Links: 2
> Access: (0700/drwx--) Uid: ( 1000/hive) Gid: ( 1002/ hadoop)
> Access: 2017-12-19 04:57:38.458937635 -0600
> Modify: 2017-12-19 04:53:49.062777216 -0600
> Change: 2017-12-19 04:53:49.066777218 -0600
> Birth: -
> {code}
> Showing the large number:
> {code:java}
> # find /tmp/ -name '*_resources' | wc -l
> 68340
> # find /tmp/hive -name "*.pipeout" | wc -l
> 51837
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23130) Spark Thrift does not clean-up temporary files (/tmp/*_resources and /tmp/hive/*.pipeout)
[
https://issues.apache.org/jira/browse/SPARK-23130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328568#comment-16328568
]
Sean Roberts commented on SPARK-23130:
--
* SPARK-15401: Similar report for the "_resources" files
* HIVE-6091: Possibly a fix for the "pipeout" files
> Spark Thrift does not clean-up temporary files (/tmp/*_resources and
> /tmp/hive/*.pipeout)
> -
>
> Key: SPARK-23130
> URL: https://issues.apache.org/jira/browse/SPARK-23130
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.6.3, 2.1.0, 2.2.0
> Environment: * Hadoop distributions: HDP 2.5 - 2.6.3.0
> * OS: Seen on SLES12, RHEL 7.3 & RHEL 7.4
>Reporter: Sean Roberts
>Priority: Major
>
> Spark Thrift is not cleaning up /tmp for files & directories named like:
> /tmp/hive/*.pipeout
> /tmp/*_resources
> There are such a large number that /tmp quickly runs out of inodes *causing
> the partition to be unusable and many services to crash*. This is even true
> when the only jobs submitted are routine service checks.
> Used `strace` to show that Spark Thrift is responsible:
> {code:java}
> strace.out.118864:04:53:49
> open("/tmp/hive/55ad7fc1-f79a-4ad8-8e02-26bbeaa86bbc7288010135864174970.pipeout",
> O_RDWR|O_CREAT|O_EXCL, 0666) = 134
> strace.out.118864:04:53:49
> mkdir("/tmp/b6dfbf9e-2f7c-4c25-95a1-73c44318ecf4_resources", 0777) = 0
> {code}
> *Those files were left behind, even days later.*
> **
>
> Example files:
> {code:java}
> # stat
> /tmp/hive/55ad7fc1-f79a-4ad8-8e02-26bbeaa86bbc7288010135864174970.pipeout
> File:
> ‘/tmp/hive/55ad7fc1-f79a-4ad8-8e02-26bbeaa86bbc7288010135864174970.pipeout’
> Size: 0 Blocks: 0 IO Block: 4096 regular empty file
> Device: fe09h/65033d Inode: 678 Links: 1
> Access: (0644/-rw-r--r--) Uid: ( 1000/hive) Gid: ( 1002/ hadoop)
> Access: 2017-12-19 04:53:49.126777260 -0600
> Modify: 2017-12-19 04:53:49.126777260 -0600
> Change: 2017-12-19 04:53:49.126777260 -0600
> Birth: -
> # stat /tmp/b6dfbf9e-2f7c-4c25-95a1-73c44318ecf4_resources
> File: ‘/tmp/b6dfbf9e-2f7c-4c25-95a1-73c44318ecf4_resources’
> Size: 4096 Blocks: 8 IO Block: 4096 directory
> Device: fe09h/65033d Inode: 668 Links: 2
> Access: (0700/drwx--) Uid: ( 1000/hive) Gid: ( 1002/ hadoop)
> Access: 2017-12-19 04:57:38.458937635 -0600
> Modify: 2017-12-19 04:53:49.062777216 -0600
> Change: 2017-12-19 04:53:49.066777218 -0600
> Birth: -
> {code}
> Showing the large number:
> {code:java}
> # find /tmp/ -name '*_resources' | wc -l
> 68340
> # find /tmp/hive -name "*.pipeout" | wc -l
> 51837
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-23130) Spark Thrift does not clean-up temporary files (/tmp/*_resources and /tmp/hive/*.pipeout)
[
https://issues.apache.org/jira/browse/SPARK-23130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Roberts updated SPARK-23130:
-
Description:
Spark Thrift is not cleaning up /tmp for files & directories named like:
/tmp/hive/*.pipeout
/tmp/*_resources
There are such a large number that /tmp quickly runs out of inodes *causing the
partition to be unusable and many services to crash*. This is even true when
the only jobs submitted are routine service checks.
Used `strace` to show that Spark Thrift is responsible:
{code:java}
strace.out.118864:04:53:49
open("/tmp/hive/55ad7fc1-f79a-4ad8-8e02-26bbeaa86bbc7288010135864174970.pipeout",
O_RDWR|O_CREAT|O_EXCL, 0666) = 134
strace.out.118864:04:53:49
mkdir("/tmp/b6dfbf9e-2f7c-4c25-95a1-73c44318ecf4_resources", 0777) = 0
{code}
*Those files were left behind, even days later.*
Example files:
{code:java}
# stat /tmp/hive/55ad7fc1-f79a-4ad8-8e02-26bbeaa86bbc7288010135864174970.pipeout
File:
‘/tmp/hive/55ad7fc1-f79a-4ad8-8e02-26bbeaa86bbc7288010135864174970.pipeout’
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: fe09h/65033dInode: 678 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 1000/hive) Gid: ( 1002/ hadoop)
Access: 2017-12-19 04:53:49.126777260 -0600
Modify: 2017-12-19 04:53:49.126777260 -0600
Change: 2017-12-19 04:53:49.126777260 -0600
Birth: -
# stat /tmp/b6dfbf9e-2f7c-4c25-95a1-73c44318ecf4_resources
File: ‘/tmp/b6dfbf9e-2f7c-4c25-95a1-73c44318ecf4_resources’
Size: 4096Blocks: 8 IO Block: 4096 directory
Device: fe09h/65033dInode: 668 Links: 2
Access: (0700/drwx--) Uid: ( 1000/hive) Gid: ( 1002/ hadoop)
Access: 2017-12-19 04:57:38.458937635 -0600
Modify: 2017-12-19 04:53:49.062777216 -0600
Change: 2017-12-19 04:53:49.066777218 -0600
Birth: -
{code}
Showing the large number:
{code:java}
# find /tmp/ -name '*_resources' | wc -l
68340
# find /tmp/hive -name "*.pipeout" | wc -l
51837
{code}
was:
Spark Thrift is not cleaning up /tmp for files & directories named like:
/tmp/hive/*.pipeout
/tmp/*_resources
There are such a large number that /tmp quickly runs out of inodes *causing the
partition to be unusable and many services to crash*. This is even true when
the only jobs submitted are routine service checks.
Used `strace` to show that Spark Thrift is responsible:
{code:java}
strace.out.118864:04:53:49
open("/tmp/hive/55ad7fc1-f79a-4ad8-8e02-26bbeaa86bbc7288010135864174970.pipeout",
O_RDWR|O_CREAT|O_EXCL, 0666) = 134
strace.out.118864:04:53:49
mkdir("/tmp/b6dfbf9e-2f7c-4c25-95a1-73c44318ecf4_resources", 0777) = 0
{code}
*Those files were left behind, even days later.*
**
Example files:
{code:java}
# stat /tmp/hive/55ad7fc1-f79a-4ad8-8e02-26bbeaa86bbc7288010135864174970.pipeout
File:
‘/tmp/hive/55ad7fc1-f79a-4ad8-8e02-26bbeaa86bbc7288010135864174970.pipeout’
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: fe09h/65033dInode: 678 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 1000/hive) Gid: ( 1002/ hadoop)
Access: 2017-12-19 04:53:49.126777260 -0600
Modify: 2017-12-19 04:53:49.126777260 -0600
Change: 2017-12-19 04:53:49.126777260 -0600
Birth: -
# stat /tmp/b6dfbf9e-2f7c-4c25-95a1-73c44318ecf4_resources
File: ‘/tmp/b6dfbf9e-2f7c-4c25-95a1-73c44318ecf4_resources’
Size: 4096Blocks: 8 IO Block: 4096 directory
Device: fe09h/65033dInode: 668 Links: 2
Access: (0700/drwx--) Uid: ( 1000/hive) Gid: ( 1002/ hadoop)
Access: 2017-12-19 04:57:38.458937635 -0600
Modify: 2017-12-19 04:53:49.062777216 -0600
Change: 2017-12-19 04:53:49.066777218 -0600
Birth: -
{code}
Showing the large number:
{code:java}
# find /tmp/ -name '*_resources' | wc -l
68340
# find /tmp/hive -name "*.pipeout" | wc -l
51837
{code}
> Spark Thrift does not clean-up temporary files (/tmp/*_resources and
> /tmp/hive/*.pipeout)
> -
>
> Key: SPARK-23130
> URL: https://issues.apache.org/jira/browse/SPARK-23130
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.6.3, 2.1.0, 2.2.0
> Environment: * Hadoop distributions: HDP 2.5 - 2.6.3.0
> * OS: Seen on SLES12, RHEL 7.3 & RHEL 7.4
>Reporter: Sean Roberts
>Priority: Major
>
> Spark Thrift is not cleaning up /tmp for files & directories named like:
> /tmp/hive/*.pipeout
> /tmp/*_resources
> There are such a large number that /tmp quickly runs out of inodes *causing
> the partition to be unusable and many services to crash*. This is even true
> when the only jobs submitted are routine service checks.
> Used `strace` to show that Spark Thrift is responsible:
> {code:java}
> strace.out.118864:04:53:49
>
[jira] [Updated] (SPARK-23130) Spark Thrift does not clean-up temporary files (/tmp/*_resources and /tmp/hive/*.pipeout)
[
https://issues.apache.org/jira/browse/SPARK-23130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Roberts updated SPARK-23130:
-
Environment:
* Spark versions: 1.6.3, 2.1.0, 2.2.0
* Hadoop distributions: HDP 2.5 - 2.6.3.0
* OS: Seen on SLES12, RHEL 7.3 & RHEL 7.4
was:
* OS: Seen on SLES12, RHEL 7.3 & RHEL 7.4
* Spark versions: 1.6.3, 2.1.0, 2.2.0
* Hadoop distributions: HDP 2.5 - 2.6.3.0
> Spark Thrift does not clean-up temporary files (/tmp/*_resources and
> /tmp/hive/*.pipeout)
> -
>
> Key: SPARK-23130
> URL: https://issues.apache.org/jira/browse/SPARK-23130
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.6.3, 2.1.0, 2.2.0
> Environment: * Spark versions: 1.6.3, 2.1.0, 2.2.0
> * Hadoop distributions: HDP 2.5 - 2.6.3.0
> * OS: Seen on SLES12, RHEL 7.3 & RHEL 7.4
>Reporter: Sean Roberts
>Priority: Major
>
> Spark Thrift is not cleaning up /tmp for files & directories named like:
> /tmp/hive/*.pipeout
> /tmp/*_resources
> There are such a large number that /tmp quickly runs out of inodes *causing
> the partition to be unusable and many services to crash*. This is even true
> when the only jobs submitted are routine service checks.
> Used `strace` to show that Spark Thrift is responsible:
> {code:java}
> strace.out.118864:04:53:49
> open("/tmp/hive/55ad7fc1-f79a-4ad8-8e02-26bbeaa86bbc7288010135864174970.pipeout",
> O_RDWR|O_CREAT|O_EXCL, 0666) = 134
> strace.out.118864:04:53:49
> mkdir("/tmp/b6dfbf9e-2f7c-4c25-95a1-73c44318ecf4_resources", 0777) = 0
> {code}
> *Those files were left behind, even days later.*
> **
>
> Example files:
> {code:java}
> # stat
> /tmp/hive/55ad7fc1-f79a-4ad8-8e02-26bbeaa86bbc7288010135864174970.pipeout
> File:
> ‘/tmp/hive/55ad7fc1-f79a-4ad8-8e02-26bbeaa86bbc7288010135864174970.pipeout’
> Size: 0 Blocks: 0 IO Block: 4096 regular empty file
> Device: fe09h/65033d Inode: 678 Links: 1
> Access: (0644/-rw-r--r--) Uid: ( 1000/hive) Gid: ( 1002/ hadoop)
> Access: 2017-12-19 04:53:49.126777260 -0600
> Modify: 2017-12-19 04:53:49.126777260 -0600
> Change: 2017-12-19 04:53:49.126777260 -0600
> Birth: -
> # stat /tmp/b6dfbf9e-2f7c-4c25-95a1-73c44318ecf4_resources
> File: ‘/tmp/b6dfbf9e-2f7c-4c25-95a1-73c44318ecf4_resources’
> Size: 4096 Blocks: 8 IO Block: 4096 directory
> Device: fe09h/65033d Inode: 668 Links: 2
> Access: (0700/drwx--) Uid: ( 1000/hive) Gid: ( 1002/ hadoop)
> Access: 2017-12-19 04:57:38.458937635 -0600
> Modify: 2017-12-19 04:53:49.062777216 -0600
> Change: 2017-12-19 04:53:49.066777218 -0600
> Birth: -
> {code}
> Showing the large number:
> {code:java}
> # find /tmp/ -name '*_resources' | wc -l
> 68340
> # find /tmp/hive -name "*.pipeout" | wc -l
> 51837
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-23130) Spark Thrift does not clean-up temporary files (/tmp/*_resources and /tmp/hive/*.pipeout)
[
https://issues.apache.org/jira/browse/SPARK-23130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Roberts updated SPARK-23130:
-
Environment:
* Hadoop distributions: HDP 2.5 - 2.6.3.0
* OS: Seen on SLES12, RHEL 7.3 & RHEL 7.4
was:
* Spark versions: 1.6.3, 2.1.0, 2.2.0
* Hadoop distributions: HDP 2.5 - 2.6.3.0
* OS: Seen on SLES12, RHEL 7.3 & RHEL 7.4
> Spark Thrift does not clean-up temporary files (/tmp/*_resources and
> /tmp/hive/*.pipeout)
> -
>
> Key: SPARK-23130
> URL: https://issues.apache.org/jira/browse/SPARK-23130
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.6.3, 2.1.0, 2.2.0
> Environment: * Hadoop distributions: HDP 2.5 - 2.6.3.0
> * OS: Seen on SLES12, RHEL 7.3 & RHEL 7.4
>Reporter: Sean Roberts
>Priority: Major
>
> Spark Thrift is not cleaning up /tmp for files & directories named like:
> /tmp/hive/*.pipeout
> /tmp/*_resources
> There are such a large number that /tmp quickly runs out of inodes *causing
> the partition to be unusable and many services to crash*. This is even true
> when the only jobs submitted are routine service checks.
> Used `strace` to show that Spark Thrift is responsible:
> {code:java}
> strace.out.118864:04:53:49
> open("/tmp/hive/55ad7fc1-f79a-4ad8-8e02-26bbeaa86bbc7288010135864174970.pipeout",
> O_RDWR|O_CREAT|O_EXCL, 0666) = 134
> strace.out.118864:04:53:49
> mkdir("/tmp/b6dfbf9e-2f7c-4c25-95a1-73c44318ecf4_resources", 0777) = 0
> {code}
> *Those files were left behind, even days later.*
> **
>
> Example files:
> {code:java}
> # stat
> /tmp/hive/55ad7fc1-f79a-4ad8-8e02-26bbeaa86bbc7288010135864174970.pipeout
> File:
> ‘/tmp/hive/55ad7fc1-f79a-4ad8-8e02-26bbeaa86bbc7288010135864174970.pipeout’
> Size: 0 Blocks: 0 IO Block: 4096 regular empty file
> Device: fe09h/65033d Inode: 678 Links: 1
> Access: (0644/-rw-r--r--) Uid: ( 1000/hive) Gid: ( 1002/ hadoop)
> Access: 2017-12-19 04:53:49.126777260 -0600
> Modify: 2017-12-19 04:53:49.126777260 -0600
> Change: 2017-12-19 04:53:49.126777260 -0600
> Birth: -
> # stat /tmp/b6dfbf9e-2f7c-4c25-95a1-73c44318ecf4_resources
> File: ‘/tmp/b6dfbf9e-2f7c-4c25-95a1-73c44318ecf4_resources’
> Size: 4096 Blocks: 8 IO Block: 4096 directory
> Device: fe09h/65033d Inode: 668 Links: 2
> Access: (0700/drwx--) Uid: ( 1000/hive) Gid: ( 1002/ hadoop)
> Access: 2017-12-19 04:57:38.458937635 -0600
> Modify: 2017-12-19 04:53:49.062777216 -0600
> Change: 2017-12-19 04:53:49.066777218 -0600
> Birth: -
> {code}
> Showing the large number:
> {code:java}
> # find /tmp/ -name '*_resources' | wc -l
> 68340
> # find /tmp/hive -name "*.pipeout" | wc -l
> 51837
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-23130) Spark Thrift does not clean-up temporary files (/tmp/*_resources and /tmp/hive/*.pipeout)
Sean Roberts created SPARK-23130:
Summary: Spark Thrift does not clean-up temporary files
(/tmp/*_resources and /tmp/hive/*.pipeout)
Key: SPARK-23130
URL: https://issues.apache.org/jira/browse/SPARK-23130
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.2.0, 2.1.0, 1.6.3
Environment: * OS: Seen on SLES12, RHEL 7.3 & RHEL 7.4
* Spark versions: 1.6.3, 2.1.0, 2.2.0
* Hadoop distributions: HDP 2.5 - 2.6.3.0
Reporter: Sean Roberts
Spark Thrift is not cleaning up /tmp for files & directories named like:
/tmp/hive/*.pipeout
/tmp/*_resources
There are such a large number that /tmp quickly runs out of inodes *causing the
partition to be unusable and many services to crash*. This is even true when
the only jobs submitted are routine service checks.
Used `strace` to show that Spark Thrift is responsible:
{code:java}
strace.out.118864:04:53:49
open("/tmp/hive/55ad7fc1-f79a-4ad8-8e02-26bbeaa86bbc7288010135864174970.pipeout",
O_RDWR|O_CREAT|O_EXCL, 0666) = 134
strace.out.118864:04:53:49
mkdir("/tmp/b6dfbf9e-2f7c-4c25-95a1-73c44318ecf4_resources", 0777) = 0
{code}
*Those files were left behind, even days later.*
**
Example files:
{code:java}
# stat /tmp/hive/55ad7fc1-f79a-4ad8-8e02-26bbeaa86bbc7288010135864174970.pipeout
File:
‘/tmp/hive/55ad7fc1-f79a-4ad8-8e02-26bbeaa86bbc7288010135864174970.pipeout’
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: fe09h/65033dInode: 678 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 1000/hive) Gid: ( 1002/ hadoop)
Access: 2017-12-19 04:53:49.126777260 -0600
Modify: 2017-12-19 04:53:49.126777260 -0600
Change: 2017-12-19 04:53:49.126777260 -0600
Birth: -
# stat /tmp/b6dfbf9e-2f7c-4c25-95a1-73c44318ecf4_resources
File: ‘/tmp/b6dfbf9e-2f7c-4c25-95a1-73c44318ecf4_resources’
Size: 4096Blocks: 8 IO Block: 4096 directory
Device: fe09h/65033dInode: 668 Links: 2
Access: (0700/drwx--) Uid: ( 1000/hive) Gid: ( 1002/ hadoop)
Access: 2017-12-19 04:57:38.458937635 -0600
Modify: 2017-12-19 04:53:49.062777216 -0600
Change: 2017-12-19 04:53:49.066777218 -0600
Birth: -
{code}
Showing the large number:
{code:java}
# find /tmp/ -name '*_resources' | wc -l
68340
# find /tmp/hive -name "*.pipeout" | wc -l
51837
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-23089) "Unable to create operation log session directory" when parent directory not present
[
https://issues.apache.org/jira/browse/SPARK-23089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Roberts updated SPARK-23089:
-
Environment:
/usr/hdp/2.6.3.0-235/spark2/jars/spark-hive-thriftserver_2.11-2.2.0.2.6.3.0-235.jar
$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.4 (Maipo)
$ ps aux|grep ^hive.*spark.*thrift
hive 1468503 0.9 0.5 13319628 1411676 ?Sl Jan15 10:18
/usr/java/default/bin/java -Dhdp.version=2.6.3.0-235 -cp
/usr/hdp/current/spark2-thriftserver/conf/:/usr/hdp/current/spark2-thriftserver/jars/*:/usr/hdp/current/hadoop-client/conf/
-Xmx2048m org.apache.spark.deploy.SparkSubmit --properties-file
/usr/hdp/current/spark2-thriftserver/conf/spark-thrift-sparkconf.conf --class
org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 --name Thrift
JDBC/ODBC Server spark-internal
> "Unable to create operation log session directory" when parent directory not
> present
>
>
> Key: SPARK-23089
> URL: https://issues.apache.org/jira/browse/SPARK-23089
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.2.0
> Environment:
> /usr/hdp/2.6.3.0-235/spark2/jars/spark-hive-thriftserver_2.11-2.2.0.2.6.3.0-235.jar
> $ cat /etc/redhat-release
> Red Hat Enterprise Linux Server release 7.4 (Maipo)
> $ ps aux|grep ^hive.*spark.*thrift
> hive 1468503 0.9 0.5 13319628 1411676 ?Sl Jan15 10:18
> /usr/java/default/bin/java -Dhdp.version=2.6.3.0-235 -cp
> /usr/hdp/current/spark2-thriftserver/conf/:/usr/hdp/current/spark2-thriftserver/jars/*:/usr/hdp/current/hadoop-client/conf/
> -Xmx2048m org.apache.spark.deploy.SparkSubmit --properties-file
> /usr/hdp/current/spark2-thriftserver/conf/spark-thrift-sparkconf.conf --class
> org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 --name Thrift
> JDBC/ODBC Server spark-internal
>Reporter: Sean Roberts
>Priority: Minor
>
> When creating a session directory, Thrift should create the parent directory
> _(i.e. /tmp/hive/operation_logs)_ if it is not present.
> It's common for operators to clean-up old and empty directories in /tmp, or
> to have tools (systemd-tmpfiles or tmpwatch) that do it automatically.
> This was fixed in HIVE-12262 but not in Spark Thrift as seen by this:
> {code}18/01/15 14:22:49 WARN HiveSessionImpl: Unable to create operation log
> session directory:
> /tmp/hive/operation_logs/683a6318-adc4-42c4-b665-11dad14d7ec7{code}
> Resolved by manually creating /tmp/hive/operation_logs/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-23089) "Unable to create operation log session directory" when parent directory not present
[
https://issues.apache.org/jira/browse/SPARK-23089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Roberts updated SPARK-23089:
-
Description:
When creating a session directory, Thrift should create the parent directory
_(i.e. /tmp/hive/operation_logs)_ if it is not present.
It's common for operators to clean-up old and empty directories in /tmp, or to
have tools (systemd-tmpfiles or tmpwatch) that do it automatically.
This was fixed in HIVE-12262 but not in Spark Thrift as seen by this:
{code}18/01/15 14:22:49 WARN HiveSessionImpl: Unable to create operation log
session directory:
/tmp/hive/operation_logs/683a6318-adc4-42c4-b665-11dad14d7ec7{code}
Resolved by manually creating /tmp/hive/operation_logs/
We are experiencing this with Spark Thrift provided by:
/usr/hdp/2.6.3.0-235/spark2/jars/spark-hive-thriftserver_2.11-2.2.0.2.6.3.0-235.jar
> "Unable to create operation log session directory" when parent directory not
> present
>
>
> Key: SPARK-23089
> URL: https://issues.apache.org/jira/browse/SPARK-23089
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.2.0
>Reporter: Sean Roberts
>Priority: Minor
>
> When creating a session directory, Thrift should create the parent directory
> _(i.e. /tmp/hive/operation_logs)_ if it is not present.
> It's common for operators to clean-up old and empty directories in /tmp, or
> to have tools (systemd-tmpfiles or tmpwatch) that do it automatically.
> This was fixed in HIVE-12262 but not in Spark Thrift as seen by this:
> {code}18/01/15 14:22:49 WARN HiveSessionImpl: Unable to create operation log
> session directory:
> /tmp/hive/operation_logs/683a6318-adc4-42c4-b665-11dad14d7ec7{code}
> Resolved by manually creating /tmp/hive/operation_logs/
> We are experiencing this with Spark Thrift provided by:
> /usr/hdp/2.6.3.0-235/spark2/jars/spark-hive-thriftserver_2.11-2.2.0.2.6.3.0-235.jar
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-23089) "Unable to create operation log session directory" when parent directory not present
[
https://issues.apache.org/jira/browse/SPARK-23089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Roberts updated SPARK-23089:
-
Environment: (was: When creating a session directory, Thrift should
create the parent directory _(i.e. /tmp/hive/operation_logs)_ if it is not
present.
It's common for operators to clean-up old and empty directories in /tmp, or to
have tools (systemd-tmpfiles or tmpwatch) that do it automatically.
This was fixed in HIVE-12262 but not in Spark Thrift as seen by this:
{code}18/01/15 14:22:49 WARN HiveSessionImpl: Unable to create operation log
session directory:
/tmp/hive/operation_logs/683a6318-adc4-42c4-b665-11dad14d7ec7{code}
Resolved by manually creating /tmp/hive/operation_logs/
We are experiencing this with Spark Thrift provided by:
/usr/hdp/2.6.3.0-235/spark2/jars/spark-hive-thriftserver_2.11-2.2.0.2.6.3.0-235.jar)
> "Unable to create operation log session directory" when parent directory not
> present
>
>
> Key: SPARK-23089
> URL: https://issues.apache.org/jira/browse/SPARK-23089
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.2.0
>Reporter: Sean Roberts
>Priority: Minor
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-23089) "Unable to create operation log session directory" when parent directory not present
[
https://issues.apache.org/jira/browse/SPARK-23089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Roberts updated SPARK-23089:
-
Description:
When creating a session directory, Thrift should create the parent directory
_(i.e. /tmp/hive/operation_logs)_ if it is not present.
It's common for operators to clean-up old and empty directories in /tmp, or to
have tools (systemd-tmpfiles or tmpwatch) that do it automatically.
This was fixed in HIVE-12262 but not in Spark Thrift as seen by this:
{code}18/01/15 14:22:49 WARN HiveSessionImpl: Unable to create operation log
session directory:
/tmp/hive/operation_logs/683a6318-adc4-42c4-b665-11dad14d7ec7{code}
Resolved by manually creating /tmp/hive/operation_logs/
was:
When creating a session directory, Thrift should create the parent directory
_(i.e. /tmp/hive/operation_logs)_ if it is not present.
It's common for operators to clean-up old and empty directories in /tmp, or to
have tools (systemd-tmpfiles or tmpwatch) that do it automatically.
This was fixed in HIVE-12262 but not in Spark Thrift as seen by this:
{code}18/01/15 14:22:49 WARN HiveSessionImpl: Unable to create operation log
session directory:
/tmp/hive/operation_logs/683a6318-adc4-42c4-b665-11dad14d7ec7{code}
Resolved by manually creating /tmp/hive/operation_logs/
We are experiencing this with Spark Thrift provided by:
/usr/hdp/2.6.3.0-235/spark2/jars/spark-hive-thriftserver_2.11-2.2.0.2.6.3.0-235.jar
> "Unable to create operation log session directory" when parent directory not
> present
>
>
> Key: SPARK-23089
> URL: https://issues.apache.org/jira/browse/SPARK-23089
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.2.0
>Reporter: Sean Roberts
>Priority: Minor
>
> When creating a session directory, Thrift should create the parent directory
> _(i.e. /tmp/hive/operation_logs)_ if it is not present.
> It's common for operators to clean-up old and empty directories in /tmp, or
> to have tools (systemd-tmpfiles or tmpwatch) that do it automatically.
> This was fixed in HIVE-12262 but not in Spark Thrift as seen by this:
> {code}18/01/15 14:22:49 WARN HiveSessionImpl: Unable to create operation log
> session directory:
> /tmp/hive/operation_logs/683a6318-adc4-42c4-b665-11dad14d7ec7{code}
> Resolved by manually creating /tmp/hive/operation_logs/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-23089) "Unable to create operation log session directory" when parent directory not present
Sean Roberts created SPARK-23089:
Summary: "Unable to create operation log session directory" when
parent directory not present
Key: SPARK-23089
URL: https://issues.apache.org/jira/browse/SPARK-23089
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.2.0
Environment: When creating a session directory, Thrift should create
the parent directory _(i.e. /tmp/hive/operation_logs)_ if it is not present.
It's common for operators to clean-up old and empty directories in /tmp, or to
have tools (systemd-tmpfiles or tmpwatch) that do it automatically.
This was fixed in HIVE-12262 but not in Spark Thrift as seen by this:
{code}18/01/15 14:22:49 WARN HiveSessionImpl: Unable to create operation log
session directory:
/tmp/hive/operation_logs/683a6318-adc4-42c4-b665-11dad14d7ec7{code}
Resolved by manually creating /tmp/hive/operation_logs/
We are experiencing this with Spark Thrift provided by:
/usr/hdp/2.6.3.0-235/spark2/jars/spark-hive-thriftserver_2.11-2.2.0.2.6.3.0-235.jar
Reporter: Sean Roberts
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
