Repository: spark
Updated Branches:
refs/heads/master 0c7b45290 - 9497b12d4
[SPARK-3006] Failed to execute spark-shell in Windows OS
Modified the order of the options and arguments in spark-shell.cmd
Author: Masayoshi TSUZUKI tsudu...@oss.nttdata.co.jp
Closes #1918 from
Repository: spark
Updated Branches:
refs/heads/branch-1.0 9e783a270 - 6c6409e8b
SPARK-3009: Reverted readObject method in ApplicationInfo so that Applic...
...ationInfo is initialized properly after deserialization
Author: Jacek Lewandowski lewandowski.ja...@gmail.com
Closes #1922 from
Repository: spark
Updated Branches:
refs/heads/branch-1.1 c39a3f337 - f5d9176fb
SPARK-3009: Reverted readObject method in ApplicationInfo so that Applic...
...ationInfo is initialized properly after deserialization
Author: Jacek Lewandowski lewandowski.ja...@gmail.com
Closes #1947 from
Repository: spark
Updated Branches:
refs/heads/branch-1.1 e08333463 - 25cabd7ee
[SPARK-2718] [yarn] Handle quotes and other characters in user args.
Due to the way Yarn runs things through bash, normal quoting doesn't
work as expected. This change applies the necessary voodoo to the user
args
exception message that distinguishes between these cases will prevent similar
headaches that I have gone through.
Author: Andrew Or andrewo...@gmail.com
Closes #2067 from andrewor14/python-exception and squashes the following
commits:
742f823 [Andrew Or] Further clarify warning messages
.
Author: Andrew Or andrewo...@gmail.com
Closes #1810 from andrewor14/service-name and squashes the following commits:
8c459ed [Andrew Or] Use a common variable for driver/executor actor system names
3a92843 [Andrew Or] Change actor name to sparkDriver and sparkExecutor
921363e [Andrew Or] Merge
.
Author: Andrew Or andrewo...@gmail.com
Closes #1810 from andrewor14/service-name and squashes the following commits:
8c459ed [Andrew Or] Use a common variable for driver/executor actor system names
3a92843 [Andrew Or] Change actor name to sparkDriver and sparkExecutor
921363e [Andrew Or] Merge branch
Repository: spark
Updated Branches:
refs/heads/master b92d823ad - d8298c46b
[SPARK-3170][CORE][BUG]:RDD info loss in StorageTab and ExecutorTab
compeleted stage only need to remove its own partitions that are no longer
cached. However, StorageTab may lost some rdds which are cached actually.
Repository: spark
Updated Branches:
refs/heads/branch-1.1 1d468df33 - 8f8e2a4ee
[SPARK-3170][CORE][BUG]:RDD info loss in StorageTab and ExecutorTab
compeleted stage only need to remove its own partitions that are no longer
cached. However, StorageTab may lost some rdds which are cached
Repository: spark
Updated Branches:
refs/heads/master 96df92906 - 92af2314f
SPARK-3082. yarn.Client.logClusterResourceDetails throws NPE if requeste...
...d queue doesn't exist
Author: Sandy Ryza sa...@cloudera.com
Closes #1984 from sryza/sandy-spark-3082 and squashes the following commits:
Repository: spark
Updated Branches:
refs/heads/branch-1.1 0b9718add - f4cbf5e3d
SPARK-3082. yarn.Client.logClusterResourceDetails throws NPE if requeste...
...d queue doesn't exist
Author: Sandy Ryza sa...@cloudera.com
Closes #1984 from sryza/sandy-spark-3082 and squashes the following
Repository: spark
Updated Branches:
refs/heads/master 32ec0a8cd - 378b2315b
[SPARK-3061] Fix Maven build under Windows
The Maven build was failing on Windows because it tried to call the unix
`unzip` utility to extract the Py4J files into core's build directory. I've
fixed this issue by
Closes #2211 from andrewor14/windows-shell-jars and squashes the following
commits:
262c6a2 [Andrew Or] Oops... Add the new code to the correct place
0d5a0c1 [Andrew Or] Format jar path only for adding to shell classpath
42bd626 [Andrew Or] Remove unnecessary code
0049f1b [Andrew Or] Remove
Repository: spark
Updated Branches:
refs/heads/branch-1.0 d47581638 - 8dd7690e2
[SPARK-3216] [SPARK-3232] Spark-shell is broken in branch-1.0 / Backport
SPARK-3006 into branch-1.0
Author: Kousuke Saruta saru...@oss.nttdata.co.jp
Author: Andrew Or andrewo...@gmail.com
Closes #2136 from
Repository: spark
Updated Branches:
refs/heads/master e5d376801 - ccc69e26e
[SPARK-2845] Add timestamps to block manager events.
These are not used by the UI but are useful when analysing the
logs from a spark job.
Author: Marcelo Vanzin van...@cloudera.com
Closes #654 from
Repository: spark
Updated Branches:
refs/heads/master ccc69e26e - f2b5b619a
[SPARK-3388] Expose aplication ID in ApplicationStart event, use it in history
server.
This change exposes the application ID generated by the Spark Master, Mesos or
Yarn
via the SparkListenerApplicationStart event.
Repository: spark
Updated Branches:
refs/heads/master f2b5b619a - 2784822e4
[Minor] Fix outdated Spark version
This is causing the event logs to include a file called SPARK_VERSION_1.0.0,
which is not accurate.
Author: Andrew Or andrewo...@gmail.com
Author: andrewor14 andrewo...@gmail.com
Repository: spark
Updated Branches:
refs/heads/master a52240792 - e08ea7393
[SPARK-3303][core] fix SparkContextSchedulerCreationSuite test error
run test with the master branch with this command when mesos native lib is set
sbt/sbt -Phive test-only
Repository: spark
Updated Branches:
refs/heads/master e08ea7393 - 4bba10c41
[SPARK-3233] Executor never stop its SparnEnv, BlockManager, ConnectionManager
etc.
Author: Kousuke Saruta saru...@oss.nttdata.co.jp
Closes #2138 from sarutak/SPARK-3233 and squashes the following commits:
c0205b7
Repository: spark
Updated Branches:
refs/heads/master 2b7ab814f - 092e2f152
SPARK-2425 Don't kill a still-running Application because of some misbehaving
Executors
Introduces a LOADING - RUNNING ApplicationState transition and prevents Master
from removing an Application with RUNNING
Repository: spark
Updated Branches:
refs/heads/branch-1.1 7a236dcf8 - e884805ce
SPARK-2425 Don't kill a still-running Application because of some misbehaving
Executors
Introduces a LOADING - RUNNING ApplicationState transition and prevents Master
from removing an Application with RUNNING
Repository: spark
Updated Branches:
refs/heads/master f0f1ba09b - 26862337c
[SPARK-3193]output errer info when Process exit code is not zero in test suite
https://issues.apache.org/jira/browse/SPARK-3193
I noticed that sometimes pr tests failed due to the Process exitcode != 0,refer
to
Repository: spark
Updated Branches:
refs/heads/branch-1.1 24262684e - e5f77ae97
[SPARK-3345] Do correct parameters for ShuffleFileGroup
In the method `newFileGroup` of class `FileShuffleBlockManager`, the parameters
for creating new `ShuffleFileGroup` object is in wrong order.
Because in
...@gmail.com
Closes #2211 from andrewor14/windows-shell-jars and squashes the following
commits:
262c6a2 [Andrew Or] Oops... Add the new code to the correct place
0d5a0c1 [Andrew Or] Format jar path only for adding to shell classpath
42bd626 [Andrew Or] Remove unnecessary code
0049f1b [Andrew Or] Remove
Repository: spark
Updated Branches:
refs/heads/master e4f4886d7 - 558962a83
[SPARK-3411] Improve load-balancing of concurrently-submitted drivers across
workers
If the waiting driver array is too big, the drivers in it will be dispatched to
the first worker we get(if it has enough
Repository: spark
Updated Branches:
refs/heads/master 6324eb7b5 - ce59725b8
[SPARK-3429] Don't include the empty string as a defaultAclUser
Changes logging from
```
14/09/05 02:01:08 INFO SecurityManager: Changing view acls to: aash,
14/09/05 02:01:08 INFO SecurityManager: Changing modify
Repository: spark
Updated Branches:
refs/heads/branch-1.1 2ffc7980c - 4245404e8
[SPARK-3429] Don't include the empty string as a defaultAclUser
Changes logging from
```
14/09/05 02:01:08 INFO SecurityManager: Changing view acls to: aash,
14/09/05 02:01:08 INFO SecurityManager: Changing
Repository: spark
Updated Branches:
refs/heads/master f858f4668 - 33c7a738a
SPARK-2482: Resolve sbt warnings during build
At the same time, import the `scala.language.postfixOps` and `
org.scalatest.time.SpanSugar._` cause `scala.language.postfixOps` doesn't work
Author: witgo wi...@qq.com
Repository: spark
Updated Branches:
refs/heads/branch-1.1 4245404e8 - e69deb818
[SPARK-3465] fix task metrics aggregation in local mode
Before overwrite t.taskMetrics, take a deepcopy of it.
Author: Davies Liu davies@gmail.com
Closes #2338 from davies/fix_metric and squashes the
Repository: spark
Updated Branches:
refs/heads/branch-1.1 e69deb818 - f17b7957a
Revert [Spark-3490] Disable SparkUI for tests
This reverts commit 2ffc7980c6818eec05e32141c52e335bc71daed9.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit:
Repository: spark
Updated Branches:
refs/heads/master e59fac1f9 - 37d925280
[SPARK-2714] DAGScheduler logs jobid when runJob finishes
DAGScheduler logs jobid when runJob finishes
Author: yantangzhai tyz0...@163.com
Closes #1617 from YanTangZhai/SPARK-2714 and squashes the following commits:
Repository: spark
Updated Branches:
refs/heads/branch-1.1 856156b40 - 937de93e8
[SPARK-3490] Disable SparkUI for tests (backport into 1.1)
Original PR: #2363
Author: Andrew Or andrewo...@gmail.com
Closes #2415 from andrewor14/disable-ui-for-tests-1.1 and squashes the
following commits
Repository: spark
Updated Branches:
refs/heads/branch-1.1 85e7c52ba - 0690410e4
Docs: move HA subsections to a deeper indentation level
Makes the table of contents read better
Author: Andrew Ash and...@andrewash.com
Closes #2402 from ash211/docs/better-indentation and squashes the following
Repository: spark
Updated Branches:
refs/heads/master 7fc3bb7c8 - cbc065039
[SPARK-3571] Spark standalone cluster mode doesn't work.
I think, this issue is caused by #1106
Author: Kousuke Saruta saru...@oss.nttdata.co.jp
Closes #2436 from sarutak/SPARK-3571 and squashes the following
Repository: spark
Updated Branches:
refs/heads/master cbc065039 - 6688a266f
[SPARK-3564][WebUI] Display App ID on HistoryPage
Author: Kousuke Saruta saru...@oss.nttdata.co.jp
Closes #2424 from sarutak/display-appid-on-webui and squashes the following
commits:
417fe90 [Kousuke Saruta] Added
Repository: spark
Updated Branches:
refs/heads/branch-1.1 0690410e4 - 3f1f9744b
[SPARK-3564][WebUI] Display App ID on HistoryPage
Author: Kousuke Saruta saru...@oss.nttdata.co.jp
Closes #2424 from sarutak/display-appid-on-webui and squashes the following
commits:
417fe90 [Kousuke Saruta]
Repository: spark
Updated Branches:
refs/heads/master 6688a266f - 1147973f1
[SPARK-3567] appId field in SparkDeploySchedulerBackend should be volatile
Author: Kousuke Saruta saru...@oss.nttdata.co.jp
Closes #2428 from sarutak/appid-volatile-modification and squashes the
following commits:
Repository: spark
Updated Branches:
refs/heads/master 471e6a3a4 - b3ed37e5b
[SPARK-3560] Fixed setting spark.jars system property in yarn-cluster mode
Author: Victsm victor@gmail.com
Author: Min Shen ms...@linkedin.com
Closes #2449 from Victsm/SPARK-3560 and squashes the following
Repository: spark
Updated Branches:
refs/heads/branch-1.1 2b286926b - 832dff64d
[SPARK-3560] Fixed setting spark.jars system property in yarn-cluster mode
Author: Victsm victor@gmail.com
Author: Min Shen ms...@linkedin.com
Closes #2449 from Victsm/SPARK-3560 and squashes the following
Repository: spark
Updated Branches:
refs/heads/master b3ed37e5b - 9306297d1
[Minor Hot Fix] Move a line in SparkSubmit to the right place
This was introduced in #2449
Author: Andrew Or andrewo...@gmail.com
Closes #2452 from andrewor14/standalone-hot-fix and squashes the following
commits
Repository: spark
Updated Branches:
refs/heads/master 99b06b6fd - 8af237061
[Docs] Fix outdated docs for standalone cluster
This is now supported!
Author: andrewor14 andrewo...@gmail.com
Author: Andrew Or andrewo...@gmail.com
Closes #2461 from andrewor14/document-standalone-cluster
Repository: spark
Updated Branches:
refs/heads/branch-1.1 1687d6ba9 - fd8835323
[Docs] Fix outdated docs for standalone cluster
This is now supported!
Author: andrewor14 andrewo...@gmail.com
Author: Andrew Or andrewo...@gmail.com
Closes #2461 from andrewor14/document-standalone-cluster
,
this property is simply not propagated to the worker nodes.
`SPARK_EXECUTOR_MEMORY` is picked up from `SparkContext`, but in cluster mode
the driver runs on one of the worker machines, where this environment variable
may not be set.
Author: Andrew Or andrewo...@gmail.com
Closes #2500 from andrewor14
,
this property is simply not propagated to the worker nodes.
`SPARK_EXECUTOR_MEMORY` is picked up from `SparkContext`, but in cluster mode
the driver runs on one of the worker machines, where this environment variable
may not be set.
Author: Andrew Or andrewo...@gmail.com
Closes #2500 from andrewor14
andrewor14, mubarak
Author: Mubarak Seyed mubarak.se...@gmail.com
Author: Tathagata Das tathagata.das1...@gmail.com
Closes #2464 from tdas/streaming-callsite and squashes the following commits:
dc54c71 [Tathagata Das] Made changes based on PR comments.
390b45d [Tathagata Das] Fixed minor bugs
a look andrewor14, mubarak
Author: Mubarak Seyed mubarak.se...@gmail.com
Author: Tathagata Das tathagata.das1...@gmail.com
Closes #2464 from tdas/streaming-callsite and squashes the following commits:
dc54c71 [Tathagata Das] Made changes based on PR comments.
390b45d [Tathagata Das] Fixed minor bugs
Repository: spark
Updated Branches:
refs/heads/master 30461c6ac - 8da10bf14
[SPARK-3476] Remove outdated memory checks in Yarn
See description in [JIRA](https://issues.apache.org/jira/browse/SPARK-3476).
Author: Andrew Or andrewo...@gmail.com
Closes #2528 from andrewor14/yarn-memory-checks
Repository: spark
Updated Branches:
refs/heads/branch-1.1 d6ed5abf1 - d9d94e0b0
Docs : use --total-executor-cores rather than --cores after spark-shell
Author: CrazyJvm crazy...@gmail.com
Closes #2540 from CrazyJvm/standalone-core and squashes the following commits:
66d9fc6 [CrazyJvm] use
Repository: spark
Updated Branches:
refs/heads/branch-1.1 a7d2df42b - c8c3b49f2
[SPARK-3755][Core] Do not bind port 1 - 1024 to server in spark
Non-root user use port 1- 1024 to start jetty server will get the exception
java.net.SocketException: Permission denied, so not use these ports
Repository: spark
Updated Branches:
refs/heads/master 127e97bee - 8081ce8bd
[SPARK-3755][Core] avoid trying privileged port when request a non-privileged
port
pwendell, ```tryPort``` is not compatible with old code in last PR, this is to
fix it.
And after discuss with srowen renamed the
Repository: spark
Updated Branches:
refs/heads/branch-1.1 68693519e - 16789f62a
[SPARK-3755][Core] avoid trying privileged port when request a non-privileged
port
pwendell, ```tryPort``` is not compatible with old code in last PR, this is to
fix it.
And after discuss with srowen renamed the
Repository: spark
Updated Branches:
refs/heads/master 8081ce8bd - 42d5077fd
[DEPLOY] SPARK-3759: Return the exit code of the driver process
SparkSubmitDriverBootstrapper.scala now returns the exit code of the driver
process, instead of always returning 0.
Author: Eric Eijkelenboom
Repository: spark
Updated Branches:
refs/heads/branch-1.1 16789f62a - 699af62dc
[DEPLOY] SPARK-3759: Return the exit code of the driver process
SparkSubmitDriverBootstrapper.scala now returns the exit code of the driver
process, instead of always returning 0.
Author: Eric Eijkelenboom
Repository: spark
Updated Branches:
refs/heads/branch-1.1 5d991db8a - d5af9e160
[SPARK-3696]Do not override the user-difined conf_dir
https://issues.apache.org/jira/browse/SPARK-3696
We see if SPARK_CONF_DIR is already defined before assignment.
Author: WangTaoTheTonic
Repository: spark
Updated Branches:
refs/heads/master 6a1d48f4f - a8c52d534
[SPARK-3535][Mesos] Fix resource handling.
Author: Brenden Matthews bren...@diddyinc.com
Closes #2401 from brndnmtthws/master and squashes the following commits:
4abaa5d [Brenden Matthews] [SPARK-3535][Mesos] Fix
Repository: spark
Updated Branches:
refs/heads/branch-1.1 d5af9e160 - 6f1509784
[SPARK-3535][Mesos] Fix resource handling.
Author: Brenden Matthews bren...@diddyinc.com
Closes #2401 from brndnmtthws/master and squashes the following commits:
4abaa5d [Brenden Matthews] [SPARK-3535][Mesos]
Repository: spark
Updated Branches:
refs/heads/branch-1.1 6f1509784 - f13025669
[SPARK-3775] Not suitable error message in spark-shell.cmd
Modified some sentence of error message in bin\*.cmd.
Author: Masayoshi TSUZUKI tsudu...@oss.nttdata.co.jp
Closes #2640 from tsudukim/feature/SPARK-3775
Repository: spark
Updated Branches:
refs/heads/master 358d7ffd0 - e5566e05b
[SPARK-3774] typo comment in bin/utils.sh
Modified the comment of bin/utils.sh.
Author: Masayoshi TSUZUKI tsudu...@oss.nttdata.co.jp
Closes #2639 from tsudukim/feature/SPARK-3774 and squashes the following
commits:
Repository: spark
Updated Branches:
refs/heads/branch-1.1 f13025669 - e4ddedee6
[SPARK-3774] typo comment in bin/utils.sh
Modified the comment of bin/utils.sh.
Author: Masayoshi TSUZUKI tsudu...@oss.nttdata.co.jp
Closes #2639 from tsudukim/feature/SPARK-3774 and squashes the following
Repository: spark
Updated Branches:
refs/heads/master e5566e05b - 30abef154
[SPARK-3606] [yarn] Correctly configure AmIpFilter for Yarn HA.
The existing code only considered one of the RMs when running in
Yarn HA mode, so it was possible to get errors if the active RM
was not registered in
Repository: spark
Updated Branches:
refs/heads/master 30abef154 - 1eb8389cb
[SPARK-3763] The example of building with sbt should be sbt assembly instead
of sbt compile
In building-spark.md, there are some examples for making assembled package with
maven but the example for building with sbt
Repository: spark
Updated Branches:
refs/heads/master 1eb8389cb - 79e45c932
[SPARK-3377] [SPARK-3610] Metrics can be accidentally aggregated / History
server log name should not be based on user input
This PR is another solution for #2250
I'm using codahale base MetricsSystem of Spark with
Repository: spark
Updated Branches:
refs/heads/master cf1d32e3e - 32fad4233
[SPARK-3597][Mesos] Implement `killTask`.
The MesosSchedulerBackend did not previously implement `killTask`,
resulting in an exception.
Author: Brenden Matthews bren...@diddyinc.com
Closes #2453 from
Repository: spark
Updated Branches:
refs/heads/branch-1.1 e4ddedee6 - d9cf4d08a
[SPARK-3597][Mesos] Implement `killTask`.
The MesosSchedulerBackend did not previously implement `killTask`,
resulting in an exception.
Author: Brenden Matthews bren...@diddyinc.com
Closes #2453 from
Repository: spark
Updated Branches:
refs/heads/master 32fad4233 - a7c73130f
SPARK-1656: Fix potential resource leaks
JIRA: https://issues.apache.org/jira/browse/SPARK-1656
Author: zsxwing zsxw...@gmail.com
Closes #577 from zsxwing/SPARK-1656 and squashes the following commits:
c431095
Repository: spark
Updated Branches:
refs/heads/branch-1.1 d9cf4d08a - c068d9084
SPARK-1656: Fix potential resource leaks
JIRA: https://issues.apache.org/jira/browse/SPARK-1656
Author: zsxwing zsxw...@gmail.com
Closes #577 from zsxwing/SPARK-1656 and squashes the following commits:
c431095
Repository: spark
Updated Branches:
refs/heads/branch-1.1 82ab4a796 - 3a7875d9d
[SPARK-3808] PySpark fails to start in Windows
Modified syntax error of *.cmd script.
Author: Masayoshi TSUZUKI tsudu...@oss.nttdata.co.jp
Closes #2669 from tsudukim/feature/SPARK-3808 and squashes the following
Closes #2688 from andrewor14/cache-log-message and squashes the following
commits:
28e33d6 [Andrew Or] Shy away from unrolling
5638c49 [Andrew Or] Grammar
39a0c28 [Andrew Or] Log more detail when unrolling a block fails
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http
Repository: spark
Updated Branches:
refs/heads/master 553737c6e - 446063eca
[SPARK-3777] Display Executor ID for Tasks in Stage page
Now the Stage page only displays Executor(host) for tasks. However, there may
be more than one Executors running in the same host. Currently, when some task
Repository: spark
Updated Branches:
refs/heads/branch-1.1 e8afb733a - a1f833f75
[SPARK-3829] Make Spark logo image on the header of HistoryPage as a link to
HistoryPage's page #1
There is a Spark logo on the header of HistoryPage.
We can have too many HistoryPages if we run 20+ applications.
Repository: spark
Updated Branches:
refs/heads/branch-1.1 a44af7302 - 09d6a81ad
[SPARK-3844][UI] Truncate appName in WebUI if it is too long
Truncate appName in WebUI if it is too long.
Author: Xiangrui Meng m...@databricks.com
Closes #2707 from mengxr/truncate-app-name and squashes the
Repository: spark
Updated Branches:
refs/heads/master b9df8af62 - 86b392942
[SPARK-3844][UI] Truncate appName in WebUI if it is too long
Truncate appName in WebUI if it is too long.
Author: Xiangrui Meng m...@databricks.com
Closes #2707 from mengxr/truncate-app-name and squashes the
Repository: spark
Updated Branches:
refs/heads/master 86b392942 - 13cab5ba4
add spark.driver.memory to config docs
It took me a minute to track this down, so I thought it could be useful to have
it in the docs.
I'm unsure if 512mb is the default for spark.driver.memory? Also - there could
Repository: spark
Updated Branches:
refs/heads/master 4d26aca77 - 186b497c9
[SPARK-3921] Fix CoarseGrainedExecutorBackend's arguments for Standalone mode
The goal of this patch is to fix the swapped arguments in standalone mode,
which was caused by
Repository: spark
Updated Branches:
refs/heads/master 9b6de6fbc - 7ced88b0d
[SPARK-3946] gitignore in /python includes wrong directory
Modified to ignore not the docs/ directory, but only the docs/_build/ which is
the output directory of sphinx build.
Author: Masayoshi TSUZUKI
Repository: spark
Updated Branches:
refs/heads/master 7ced88b0d - 24b818b97
[SPARK-3944][Core] Using Option[String] where value of String can be null
Author: shitis ssaxena@gmail.com
Closes #2795 from Shiti/master and squashes the following commits:
46897d7 [shitis] Using Option Wrapper
Repository: spark
Updated Branches:
refs/heads/master 56096dbaa - 7b4f39f64
[SPARK-3869] ./bin/spark-class miss Java version with _JAVA_OPTIONS set
When _JAVA_OPTIONS environment variable is set, a command java -version
outputs a message like Picked up _JAVA_OPTIONS: -Dfile.encoding=UTF-8.
Repository: spark
Updated Branches:
refs/heads/master 7b4f39f64 - 66af8e250
[SPARK-3943] Some scripts bin\*.cmd pollutes environment variables in Windows
Modified not to pollute environment variables.
Just moved the main logic into `XXX2.cmd` from `XXX.cmd`, and call `XXX2.cmd`
with cmd
Repository: spark
Updated Branches:
refs/heads/master 18ab6bd70 - 293a0b5db
[SPARK-2098] All Spark processes should support spark-defaults.conf, config file
This is another implementation about #1256
cc andrewor14 vanzin
Author: GuoQiang Li wi...@qq.com
Closes #2379 from witgo/SPARK-2098
Repository: spark
Updated Branches:
refs/heads/master 2fe0ba956 - 7f7b50ed9
[SPARK-3923] Increase Akka heartbeat pause above heartbeat interval
Something about the 2.3.4 upgrade seems to have made the issue manifest where
all the services disconnect from each other after exactly 1000 seconds
Repository: spark
Updated Branches:
refs/heads/master 7f7b50ed9 - be2ec4a91
[SQL]typo in HiveFromSpark
Author: Kun Li jacky.li...@gmail.com
Closes #2809 from jackylk/patch-1 and squashes the following commits:
46c926b [Kun Li] typo in HiveFromSpark
Project:
Repository: spark
Updated Branches:
refs/heads/master be2ec4a91 - 642b246be
[SPARK-3941][CORE] _remainingmem should not increase twice when updateBlockInfo
In BlockManagermasterActor, _remainingMem would increase memSize for twice when
updateBlockInfo if new storageLevel is invalid and old
Repository: spark
Updated Branches:
refs/heads/branch-1.1 61e590354 - 2c4117083
[SPARK-3890][Docs]remove redundant spark.executor.memory in doc
Introduced in
https://github.com/pwendell/spark/commit/f7e79bc42c1635686c3af01eef147dae92de2529,
I'm not sure why we need two spark.executor.memory
Repository: spark
Updated Branches:
refs/heads/master 56fd34af5 - dedace83f
[SPARK-3067] JobProgressPage could not show Fair Scheduler Pools section
sometimes
JobProgressPage could not show Fair Scheduler Pools section sometimes.
SparkContext starts webui and then postEnvironmentUpdate.
Repository: spark
Updated Branches:
refs/heads/branch-1.1 2c4117083 - 35875e9ec
[SPARK-3067] JobProgressPage could not show Fair Scheduler Pools section
sometimes
JobProgressPage could not show Fair Scheduler Pools section sometimes.
SparkContext starts webui and then postEnvironmentUpdate.
Repository: spark
Updated Branches:
refs/heads/branch-1.1 35875e9ec - 0d958f163
[SPARK-3606] [yarn] Correctly configure AmIpFilter for Yarn HA (1.1 vers...
...ion).
This is a backport of SPARK-3606 to branch-1.1. Some of the code had to be
duplicated since branch-1.1 doesn't have the cleanup
Repository: spark
Updated Branches:
refs/heads/master e678b9f02 - c35186206
[SPARK-3935][Core] log the number of records that has been written
There is a unused variable(count) in saveAsHadoopDataset in
PairRDDFunctions.scala. The initial idea of this variable seems to count the
number of
|
|---|--||
| Read | spark.shuffle.spill.compress | spark.shuffle.compress |
| Write | spark.shuffle.spill.compress | spark.shuffle.compress |
Thanks to andrewor14 for debugging this with me!
Author: Josh Rosen joshro...@databricks.com
Closes #2890 from
Repository: spark
Updated Branches:
refs/heads/master 813effc70 - 137d94235
[SPARK-3877][YARN] Throw an exception when application is not successful so
that the exit code wil be set to 1
When an yarn application fails (yarn-cluster mode), the exit code of
spark-submit is still 0. It's hard
Repository: spark
Updated Branches:
refs/heads/branch-1.1 457ef5955 - eb62094e8
[SPARK-3877][YARN] Throw an exception when application is not successful so
that the exit code wil be set to 1 (for branch-1.1)
This is a patch to fix SPARK-3877 in branch-1.1. See also #2732
Author: zsxwing
Repository: spark
Updated Branches:
refs/heads/master 6a40a7684 - 7aacb7bfa
[SPARK-2713] Executors of same application in same host should only download
files jars once
If Spark lunched multiple executors in one host for one application, every
executor would download it dependent files and
Repository: spark
Updated Branches:
refs/heads/master 7aacb7bfa - 30ea2868e
[SPARK-4076] Parameter expansion in spark-config is wrong
In sbin/spark-config.sh, parameter expansion is used to extract source root as
follows.
this=${BASH_SOURCE-$0}
I think, the parameter expansion should
Repository: spark
Updated Branches:
refs/heads/branch-1.1 926f8ca53 - 386fc46bc
[SPARK-4076] Parameter expansion in spark-config is wrong
In sbin/spark-config.sh, parameter expansion is used to extract source root as
follows.
this=${BASH_SOURCE-$0}
I think, the parameter expansion
Repository: spark
Updated Branches:
refs/heads/master 30ea2868e - 098f83c7c
[SPARK-4075] [Deploy] Jar url validation is not enough for Jar file
In deploy.ClientArguments.isValidJarUrl, the url is checked as follows.
def isValidJarUrl(s: String): Boolean = s.matches((.+):(.+)jar)
So, it
Repository: spark
Updated Branches:
refs/heads/branch-1.1 386fc46bc - 80dde80a6
[SPARK-4075] [Deploy] Jar url validation is not enough for Jar file
In deploy.ClientArguments.isValidJarUrl, the url is checked as follows.
def isValidJarUrl(s: String): Boolean = s.matches((.+):(.+)jar)
So,
necessary
when the executor is lost or killed because we receive a disassociation event
for each of these actor systems.
This is blocking #2840.
Author: Andrew Or andrewo...@gmail.com
Closes #2863 from andrewor14/executor-actor-system and squashes the following
commits:
44ce2e0 [Andrew
Repository: spark
Updated Branches:
refs/heads/branch-1.1 80dde80a6 - 59297e951
[SPARK-4006] In long running contexts, we encountered the situation of d...
...ouble registe...
...r without a remove in between. The cause for that is unknown, and assumed a
temp network issue.
However, since
Repository: spark
Updated Branches:
refs/heads/master bf589fc71 - 677852c3f
Just fixing comment that shows usage
Author: anant asthana anant.a...@gmail.com
Closes #2948 from anantasty/patch-1 and squashes the following commits:
d8fea0b [anant asthana] Just fixing comment that shows usage
Repository: spark
Updated Branches:
refs/heads/master f4e8c289d - 6377adaf3
[SPARK-3970] Remove duplicate removal of local dirs
The shutdown hook of `DiskBlockManager` would remove localDirs. So do not need
to register them with `Utils.registerShutdownDeleteDir`. It causes duplicate
removal
Repository: spark
Updated Branches:
refs/heads/master 7768a800d - 44d8b45a3
[SPARK-4110] Wrong comments about default settings in spark-daemon.sh
In spark-daemon.sh, thare are following comments.
# SPARK_CONF_DIR Alternate conf dir. Default is ${SPARK_PREFIX}/conf.
#
1 - 100 of 1431 matches
Mail list logo
