Github user vijoshi commented on the issue:
https://github.com/apache/spark/pull/17731
Thanks, I will close this PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user vijoshi closed the pull request at:
https://github.com/apache/spark/pull/17731
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user vijoshi commented on the issue:
https://github.com/apache/spark/pull/17731
Thanks, I tried this out - looks like doing a `rm(".sparkRsession",
envir=SparkR:::.sparkREnv)` is a way to prevent the infinite loop situation. If
I need to setup an acti
Github user vijoshi commented on the issue:
https://github.com/apache/spark/pull/17731
"I understand these 2 cases, can you explain how your change connect to
these two?"
Say, I do this:
```
delayAssign(delayedAssign(".sparkRsession&quo
Github user vijoshi commented on the issue:
https://github.com/apache/spark/pull/17731
@felixcheung yes. We need to support these two types of possibilities:
```
#do not call sparkR.session() - followed by implicit reference to
sparkSession
a <- createDataFrame(i
Github user vijoshi commented on the issue:
https://github.com/apache/spark/pull/17731
@felixcheung
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
GitHub user vijoshi opened a pull request:
https://github.com/apache/spark/pull/17731
[SPARK-20440][SparkR] Allow SparkR session and context to have delayed
bindings
## What changes were proposed in this pull request?
Allow SparkR to ignore the "promise already
Github user vijoshi commented on a diff in the pull request:
https://github.com/apache/spark/pull/16454#discussion_r94903274
--- Diff: python/pyspark/sql/session.py ---
@@ -214,8 +214,14 @@ def __init__(self, sparkContext, jsparkSession=None):
self._wrapped
Github user vijoshi commented on the issue:
https://github.com/apache/spark/pull/16119
thanks @viirya. will wait for your PR to go through.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user vijoshi commented on the issue:
https://github.com/apache/spark/pull/16119
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user vijoshi commented on the issue:
https://github.com/apache/spark/pull/16119
@cloud-fan sure, i'll push an update soon
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user vijoshi commented on a diff in the pull request:
https://github.com/apache/spark/pull/16119#discussion_r93596548
--- Diff: python/pyspark/sql/tests.py ---
@@ -1851,6 +1851,71 @@ def test_hivecontext(self):
self.assertIn("default", out.dec
Github user vijoshi commented on a diff in the pull request:
https://github.com/apache/spark/pull/16119#discussion_r93594980
--- Diff: python/pyspark/sql/context.py ---
@@ -72,8 +72,13 @@ def __init__(self, sparkContext, sparkSession=None,
jsqlContext=None):
self._sc
Github user vijoshi commented on a diff in the pull request:
https://github.com/apache/spark/pull/16119#discussion_r93594802
--- Diff: python/pyspark/sql/context.py ---
@@ -72,8 +72,13 @@ def __init__(self, sparkContext, sparkSession=None,
jsqlContext=None):
self._sc
Github user vijoshi commented on a diff in the pull request:
https://github.com/apache/spark/pull/16119#discussion_r92333556
--- Diff: python/pyspark/sql/context.py ---
@@ -72,8 +72,13 @@ def __init__(self, sparkContext, sparkSession=None,
jsqlContext=None):
self._sc
Github user vijoshi commented on the issue:
https://github.com/apache/spark/pull/16119
@holdenk reminder
also pinging @davies
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user vijoshi commented on the issue:
https://github.com/apache/spark/pull/16119
@holdenk test case added
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user vijoshi commented on the issue:
https://github.com/apache/spark/pull/16119
ping @holdenk @srowen
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes
GitHub user vijoshi opened a pull request:
https://github.com/apache/spark/pull/16119
reuse SparkSession if sparkContext specified same as active one
## What changes were proposed in this pull request?
Change is for SQLContext to reuse the active SparkSession during
Github user vijoshi closed the pull request at:
https://github.com/apache/spark/pull/15991
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user vijoshi commented on the issue:
https://github.com/apache/spark/pull/15991
@tgravescs would like to have this in 2.0 along with the other improvement
that got accepted for backport (spark-18010). so would you consider allowing
this into 2.0 as well?
---
If your project
Github user vijoshi commented on the issue:
https://github.com/apache/spark/pull/15991
@tgravesc - are you ok to merge this backport to branch-2.0 ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user vijoshi closed the pull request at:
https://github.com/apache/spark/pull/15655
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user vijoshi commented on a diff in the pull request:
https://github.com/apache/spark/pull/15991#discussion_r89349091
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/HistoryPage.scala ---
@@ -33,21 +33,40 @@ private[history] class HistoryPage(parent
Github user vijoshi commented on the issue:
https://github.com/apache/spark/pull/15991
@tgravescs - Looking to get this change early with 2.0.x. This merges
cleanly - are you ok to let this into branch-2.0?
---
If your project is set up for it, you can reply to this email and have
GitHub user vijoshi opened a pull request:
https://github.com/apache/spark/pull/15991
[SPARK-17843][WEB UI] Indicate event logs pending for processing on hâ¦
## What changes were proposed in this pull request?
Backport PR #15410 to branch-2.0
## How was this patch
Github user vijoshi commented on the issue:
https://github.com/apache/spark/pull/15855
Thanks @vanzin ! Closed the PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user vijoshi closed the pull request at:
https://github.com/apache/spark/pull/15855
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user vijoshi commented on the issue:
https://github.com/apache/spark/pull/15855
@vanzin - ping
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user vijoshi commented on the issue:
https://github.com/apache/spark/pull/15655
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user vijoshi commented on the issue:
https://github.com/apache/spark/pull/15742
@vanzin @mariobriggs Opened https://github.com/apache/spark/pull/15855 for
merging to branch-2.0
---
If your project is set up for it, you can reply to this email and have your
reply appear
GitHub user vijoshi opened a pull request:
https://github.com/apache/spark/pull/15855
[SPARK-16808][Core] History Server main page does not honor
APPLICATION_WEB_PROXY_BASE
## What changes were proposed in this pull request?
Backport SPARK-16808 (https://github.com/apache
Github user vijoshi commented on the issue:
https://github.com/apache/spark/pull/15655
@srowen - rebased on branch 2.0. thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user vijoshi commented on the issue:
https://github.com/apache/spark/pull/15410
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user vijoshi commented on a diff in the pull request:
https://github.com/apache/spark/pull/15742#discussion_r87055661
--- Diff:
core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala ---
@@ -258,23 +260,92 @@ class HistoryServerSuite extends
Github user vijoshi commented on the issue:
https://github.com/apache/spark/pull/15410
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user vijoshi commented on the issue:
https://github.com/apache/spark/pull/15410
@tgravescs thanks, have updated the fix.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user vijoshi commented on a diff in the pull request:
https://github.com/apache/spark/pull/15742#discussion_r86711698
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/HistoryServer.scala ---
@@ -143,6 +143,12 @@ class HistoryServer(
appCache.stop
Github user vijoshi commented on the issue:
https://github.com/apache/spark/pull/15742
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user vijoshi commented on the issue:
https://github.com/apache/spark/pull/15742
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user vijoshi commented on the issue:
https://github.com/apache/spark/pull/15742
@vanzin @ajbozarth besides your review comments, I noticed that the test
case was getting blocked in the `after { }` where the history server is
stopped. Tracked it to Jetty waiting for 30 secs
Github user vijoshi commented on a diff in the pull request:
https://github.com/apache/spark/pull/15742#discussion_r86662740
--- Diff:
core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala ---
@@ -275,6 +277,79 @@ class HistoryServerSuite extends
Github user vijoshi commented on a diff in the pull request:
https://github.com/apache/spark/pull/15742#discussion_r86662723
--- Diff:
core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala ---
@@ -275,6 +277,79 @@ class HistoryServerSuite extends
Github user vijoshi commented on the issue:
https://github.com/apache/spark/pull/15742
@ajbozarth updated the code. Guess, there is agreement to keep `uiRoot` in
`webui.js` to make it available to any more templates / javascript that need
it. I removed the `undefined` check
Github user vijoshi commented on the issue:
https://github.com/apache/spark/pull/15410
@tgravescs @andrewor14 - any outstanding concerns with this change, please
let me know? Can this merge?
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user vijoshi commented on the issue:
https://github.com/apache/spark/pull/15742
@ajbozarth @srowen - for your review if possible.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
GitHub user vijoshi opened a pull request:
https://github.com/apache/spark/pull/15742
initial commit
## What changes were proposed in this pull request?
Application links generated on the history server UI no longer (regression
from 1.6) contain the configured
Github user vijoshi commented on the issue:
https://github.com/apache/spark/pull/15556
@vanzin @srowen - thanks, i've opened pull request #15655 on branch-2.0
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
GitHub user vijoshi opened a pull request:
https://github.com/apache/spark/pull/15655
[SPARK-18010][CORE] Reduce work performed for building up the application
list for the History Server app list UI page
## What changes were proposed in this pull request?
backport
Github user vijoshi commented on the issue:
https://github.com/apache/spark/pull/15556
@vanzin @srowen @ajbozarth - can this go into branch-2.0 as well? is there
a deciding criteria? i can open a pull request.
---
If your project is set up for it, you can reply to this email
Github user vijoshi commented on a diff in the pull request:
https://github.com/apache/spark/pull/15556#discussion_r84849414
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala ---
@@ -554,10 +559,15 @@ private[history] class FsHistoryProvider
Github user vijoshi commented on a diff in the pull request:
https://github.com/apache/spark/pull/15556#discussion_r84774049
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala ---
@@ -554,10 +559,15 @@ private[history] class FsHistoryProvider
Github user vijoshi commented on a diff in the pull request:
https://github.com/apache/spark/pull/15556#discussion_r84689281
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala ---
@@ -557,7 +560,8 @@ private[history] class FsHistoryProvider(conf
Github user vijoshi commented on the issue:
https://github.com/apache/spark/pull/15410
@ajbozarth @tgravescs @andrewor14 - updated the implementation to always
display "Last Updated Date-Time" when we have atleast one completed scan of the
event log directory. I ha
Github user vijoshi commented on a diff in the pull request:
https://github.com/apache/spark/pull/15556#discussion_r84482418
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala ---
@@ -557,7 +560,8 @@ private[history] class FsHistoryProvider(conf
Github user vijoshi commented on the issue:
https://github.com/apache/spark/pull/15410
@ajbozarth yeah sounds useful. but note that for the very first load, we
would have no 'Last Updated' value to display since that gets set only after
the log scan+replay cycle completes at least
Github user vijoshi commented on a diff in the pull request:
https://github.com/apache/spark/pull/15556#discussion_r84430841
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala ---
@@ -43,38 +43,56 @@ private[spark] class ReplayListenerBus extends
Github user vijoshi commented on a diff in the pull request:
https://github.com/apache/spark/pull/15556#discussion_r84413566
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala ---
@@ -43,38 +43,56 @@ private[spark] class ReplayListenerBus extends
Github user vijoshi commented on the issue:
https://github.com/apache/spark/pull/15410
@tgravescs Yes, the description for the pull request now has screen shots
for the "Last Updated: XXX" case as well. I added description for the "Last
Updated" message on the ap
Github user vijoshi commented on the issue:
https://github.com/apache/spark/pull/15556
@tgravescs I have tried it with a 150MB file and speedup is very noticeable
(refer the spark issue for results I see). I have noticed the JSON
deserialization is particularly slow, while simply
Github user vijoshi commented on a diff in the pull request:
https://github.com/apache/spark/pull/15556#discussion_r84261940
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala ---
@@ -43,19 +43,25 @@ private[spark] class ReplayListenerBus extends
Github user vijoshi commented on the issue:
https://github.com/apache/spark/pull/15410
This needs to remain open - clicked the wrong button!!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user vijoshi closed the pull request at:
https://github.com/apache/spark/pull/15410
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
GitHub user vijoshi reopened a pull request:
https://github.com/apache/spark/pull/15410
[SPARK-17843][Web UI] Indicate event logs pending for processing on history
server UI
## What changes were proposed in this pull request?
History Server UI's application listing
Github user vijoshi commented on the issue:
https://github.com/apache/spark/pull/15410
@tgravescs @ajbozarth @andrewor14 can this be merged now? can we get it
into the 2.0 branch - should i simply open a pull request on 2.0?
---
If your project is set up for it, you can reply
Github user vijoshi commented on a diff in the pull request:
https://github.com/apache/spark/pull/15556#discussion_r84162567
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala ---
@@ -43,19 +43,25 @@ private[spark] class ReplayListenerBus extends
Github user vijoshi commented on a diff in the pull request:
https://github.com/apache/spark/pull/15556#discussion_r84118567
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala ---
@@ -43,19 +43,25 @@ private[spark] class ReplayListenerBus extends
GitHub user vijoshi opened a pull request:
https://github.com/apache/spark/pull/15556
[SPARK-18010][Core] Reduce work performed for building up the application
list for the History Server app list UI page
## What changes were proposed in this pull request?
allow
Github user vijoshi commented on the issue:
https://github.com/apache/spark/pull/15410
@ajbozarth @tgravescs @andrewor14 - updated pull request to show a Last
Updated message on the app list page when there are no pending logs
being processed. I would have liked to have
Github user vijoshi commented on the issue:
https://github.com/apache/spark/pull/15410
*Display applications that are pending, instead of a count*
Though agree that this would be better UX-wise, but we would only have the
filenames (`logInfos`) to go by to build this list
Github user vijoshi commented on the issue:
https://github.com/apache/spark/pull/15410
1. display "Last updated " with a generic message, possibly
omitting the count of logs
Ok, while looking at this I notice that the scan date-time the
FsHistoryProvider
Github user vijoshi commented on the issue:
https://github.com/apache/spark/pull/15410
@tgravescs - you're right - for newer logs that are generated, there could
be a window of time (10 secs or whatever the user configures) where the new
logs are not picked up for replay and the UI
Github user vijoshi commented on a diff in the pull request:
https://github.com/apache/spark/pull/15410#discussion_r82789161
--- Diff:
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala ---
@@ -316,24 +320,41 @@ private[history] class FsHistoryProvider
GitHub user vijoshi opened a pull request:
https://github.com/apache/spark/pull/15410
initial commit
## What changes were proposed in this pull request?
History Server UI's application listing to display information on currently
under process event logs so a user knows
74 matches
Mail list logo
