[GitHub] spark pull request #18328: [SPARK-21121][SQL] Support changing storage level...
Github user dosoft commented on a diff in the pull request:
https://github.com/apache/spark/pull/18328#discussion_r129031454
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala ---
@@ -106,6 +105,11 @@ class CacheManager extends Logging {
}
}
+ def cacheQuery(query: Dataset[_], tableName: Option[String] = None):
Unit = writeLock {
+cacheQuery(query, tableName, StorageLevel.fromString(
+ query.sparkSession.sessionState.conf.cacheStorageLevel))
--- End diff --
Could you explain your question in a bit more details?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #18328: [SPARK-21121][SQL] Support changing storage level...
GitHub user dosoft opened a pull request: https://github.com/apache/spark/pull/18328 [SPARK-21121][SQL] Support changing storage level via the spark.sql.inMemoryColumnarStorage.level variable ## What changes were proposed in this pull request? As described in title ## How was this patch tested? You can merge this pull request into a Git repository by running: $ git pull https://github.com/dosoft/spark SPARK-21121 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/18328.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #18328 commit f6dfaabb1f62005500b496f79bcb57e2a3b0d15e Author: Oleg Danilov Date: 2017-06-15T08:01:16Z [SPARK-21121][SQL] Support changing storage level via the spark.sql.inMemoryColumnarStorage.level variable --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16924: [SPARK-19531] Send UPDATE_LENGTH for Spark History servi...
Github user dosoft commented on the issue: https://github.com/apache/spark/pull/16924 @cloud-fan spark thriftserver as long-running application? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16924: [SPARK-19531] Send UPDATE_LENGTH for Spark History servi...
Github user dosoft commented on the issue: https://github.com/apache/spark/pull/16924 Just an example of the use case. This issue affects all long-life apps. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16924: [SPARK-19531] Send UPDATE_LENGTH for Spark History servi...
Github user dosoft commented on the issue: https://github.com/apache/spark/pull/16924 This issue has been reported by our customer as following: "Subsequently connecting to spark thriftserver via beeline and running any MR job, it doesnt get reflected in spark history server UI even after the job completion. If we stop and start spark history server, then this job info gets displayed in UI. Shouldnt this UI get auto refreshed with info on completed jobs?" --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16924: [SPARK-19531] Send UPDATE_LENGTH for Spark Histor...
GitHub user dosoft opened a pull request: https://github.com/apache/spark/pull/16924 [SPARK-19531] Send UPDATE_LENGTH for Spark History service ## What changes were proposed in this pull request? During writing to the .inprogress file (stored on the HDFS) Hadoop doesn't update file length until close and therefor Spark's history server can't detect any changes. We have to send UPDATE_LENGTH manually. You can merge this pull request into a Git repository by running: $ git pull https://github.com/dosoft/spark SPARK-19531 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/16924.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #16924 commit f87c5155832435c9dc17053521d61ae0ce06f8d8 Author: Oleg Danilov Date: 2017-02-01T13:06:22Z [SPARK-19531] Send UPDATE_LENGTH for Spark History service --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16924: [SPARK-19531] Send UPDATE_LENGTH for Spark Histor...
Github user dosoft commented on a diff in the pull request:
https://github.com/apache/spark/pull/16924#discussion_r101161834
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala ---
@@ -137,7 +138,13 @@ private[spark] class EventLoggingListener(
// scalastyle:on println
if (flushLogger) {
writer.foreach(_.flush())
- hadoopDataStream.foreach(_.hflush())
+ hadoopDataStream.foreach(ds => {
--- End diff --
seems like hflush() is not required there
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16924: [SPARK-19531] Send UPDATE_LENGTH for Spark Histor...
Github user dosoft commented on a diff in the pull request:
https://github.com/apache/spark/pull/16924#discussion_r101487342
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala ---
@@ -137,7 +138,13 @@ private[spark] class EventLoggingListener(
// scalastyle:on println
if (flushLogger) {
writer.foreach(_.flush())
- hadoopDataStream.foreach(_.hflush())
+ hadoopDataStream.foreach(ds => {
--- End diff --
hsync() is even stronger than hflush(), since under the cover both methods
use the same flushOrSync(), but hsync performs an additional tasks like
flushing OS buffers (fsync).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14990][SQL] nvl, coalesce, array with p...
GitHub user dosoft opened a pull request: https://github.com/apache/spark/pull/12768 [SPARK-14990][SQL] nvl, coalesce, array with parameter of type 'array' ## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) ## How was this patch tested? (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) You can merge this pull request into a Git repository by running: $ git pull https://github.com/dosoft/spark SPARK-14990 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/12768.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #12768 commit 5674efc19a74b1fb0f8f6a6058b09c9fb6cd763d Author: Oleg Danilov Date: 2016-04-28T22:00:26Z [SPARK-14990][SQL] nvl, coalesce, array with parameter of type 'array' --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14261][SQL] Memory leak in Spark Thrift...
GitHub user dosoft opened a pull request: https://github.com/apache/spark/pull/12932 [SPARK-14261][SQL] Memory leak in Spark Thrift Server Fixed memory leak (HiveConf in the CommandProcessorFactory) You can merge this pull request into a Git repository by running: $ git pull https://github.com/dosoft/spark SPARK-14261 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/12932.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #12932 commit 559db12bf0b708d95d5066d4c41220ab493c70c9 Author: Oleg Danilov Date: 2016-05-05T13:22:34Z [SPARK-14261][SQL] Memory leak in Spark Thrift Server --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2905 Fixed path sbin => bin
GitHub user dosoft opened a pull request: https://github.com/apache/spark/pull/1835 SPARK-2905 Fixed path sbin => bin You can merge this pull request into a Git repository by running: $ git pull https://github.com/dosoft/spark SPARK-2905 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/1835.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1835 commit 4df423cd3c970b69d9bf8e89f79edbc1ccab6d3c Author: Oleg Danilov Date: 2014-08-07T12:58:33Z SPARK-2905 Fixed path sbin => bin --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
