[GitHub] spark issue #22963: [SPARK-25962][BUILD][PYTHON] Specify minimum versions fo...

2018-11-09 Thread rekhajoshm
Github user rekhajoshm commented on the issue: https://github.com/apache/spark/pull/22963 > pydocstyle tests passed w/o issue btw: > https://amplab.cs.berkeley.edu/jenkins/job/ubuntuSparkPRB/134/consoleFull > > this is on ubuntu w/python 3.5, flake8 3.6.0, pydo

[GitHub] spark issue #22425: [SPARK-23367][Build] Include python document style check...

2018-10-19 Thread rekhajoshm
Github user rekhajoshm commented on the issue: https://github.com/apache/spark/pull/22425 Thank you @srowen @holdenk @HyukjinKwon for your review. As per discussion, in this PR, we are enabling docstyle change but not updating pydocs. Hence tox.ini has doc error codes

[GitHub] spark issue #22425: [SPARK-23367][Build] Include python document style check...

2018-09-26 Thread rekhajoshm
Github user rekhajoshm commented on the issue: https://github.com/apache/spark/pull/22425 Sorry, I was waiting from Feb2018 to Jul2018 on the previous PR #20556 , and hence there is a bit of disconnect from this issue and delay in providing context in which this PR was created

[GitHub] spark pull request #22425: [SPARK-23367][Build] Include python document styl...

2018-09-26 Thread rekhajoshm
Github user rekhajoshm commented on a diff in the pull request: https://github.com/apache/spark/pull/22425#discussion_r220757773 --- Diff: dev/tox.ini --- @@ -14,6 +14,8 @@ # limitations under the License. [pycodestyle] -ignore=E226,E241,E305,E402,E722,E731,E741

[GitHub] spark pull request #22425: [SPARK-23367][Build] Include python document styl...

2018-09-26 Thread rekhajoshm
Github user rekhajoshm commented on a diff in the pull request: https://github.com/apache/spark/pull/22425#discussion_r220756119 --- Diff: dev/tox.ini --- @@ -14,6 +14,8 @@ # limitations under the License. [pycodestyle] -ignore=E226,E241,E305,E402,E722,E731,E741

[GitHub] spark issue #20556: [SPARK-23367][Build] Include python document style check...

2018-09-14 Thread rekhajoshm
Github user rekhajoshm commented on the issue: https://github.com/apache/spark/pull/20556 PR #22425 Closing this.Thanks --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark pull request #20556: [SPARK-23367][Build] Include python document styl...

2018-09-14 Thread rekhajoshm
Github user rekhajoshm closed the pull request at: https://github.com/apache/spark/pull/20556 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22425: [SPARK-23367][Build] Include python document styl...

2018-09-14 Thread rekhajoshm
GitHub user rekhajoshm opened a pull request: https://github.com/apache/spark/pull/22425 [SPARK-23367][Build] Include python document style checking ## What changes were proposed in this pull request? Includes python document style checking. - Use sphinx like check, run

[GitHub] spark issue #20556: [SPARK-23367][Build] Include python document style check...

2018-09-14 Thread rekhajoshm
Github user rekhajoshm commented on the issue: https://github.com/apache/spark/pull/20556 Yes @holdenk , seems like I missed the last notification on it. Please let me look into it this weekend. Thanks

[GitHub] spark issue #22231: [SPARK-25238][PYTHON] lint-python: Upgrade pycodestyle t...

2018-08-27 Thread rekhajoshm
Github user rekhajoshm commented on the issue: https://github.com/apache/spark/pull/22231 thanks for the fyi @HyukjinKwon , ack. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21684: [SPARK-24470][Core] RestSubmissionClient to be robust ag...

2018-07-06 Thread rekhajoshm
Github user rekhajoshm commented on the issue: https://github.com/apache/spark/pull/21684 ``` Discovery starting. Discovery completed in 3 seconds, 75 milliseconds. Run starting. Expected test count is: 16 StandaloneRestSubmitSuite: - construct submit request

[GitHub] spark pull request #21684: [SPARK-24470][Core] RestSubmissionClient to be ro...

2018-07-06 Thread rekhajoshm
Github user rekhajoshm commented on a diff in the pull request: https://github.com/apache/spark/pull/21684#discussion_r200784938 --- Diff: core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionClient.scala --- @@ -233,30 +233,43 @@ private[spark] class

[GitHub] spark pull request #21684: [SPARK-24470][Core] RestSubmissionClient to be ro...

2018-07-06 Thread rekhajoshm
Github user rekhajoshm commented on a diff in the pull request: https://github.com/apache/spark/pull/21684#discussion_r200741018 --- Diff: core/src/test/scala/org/apache/spark/deploy/rest/StandaloneRestSubmitSuite.scala --- @@ -366,7 +366,8 @@ class StandaloneRestSubmitSuite

[GitHub] spark pull request #21684: [SPARK-24470][Core] RestSubmissionClient to be ro...

2018-07-06 Thread rekhajoshm
Github user rekhajoshm commented on a diff in the pull request: https://github.com/apache/spark/pull/21684#discussion_r200737560 --- Diff: core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionClient.scala --- @@ -233,30 +233,41 @@ private[spark] class

[GitHub] spark pull request #21684: [SPARK-24470][Core] RestSubmissionClient to be ro...

2018-07-06 Thread rekhajoshm
Github user rekhajoshm commented on a diff in the pull request: https://github.com/apache/spark/pull/21684#discussion_r200735457 --- Diff: core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionClient.scala --- @@ -233,30 +233,41 @@ private[spark] class

[GitHub] spark issue #21684: [SPARK-24470][Core] RestSubmissionClient to be robust ag...

2018-07-06 Thread rekhajoshm
Github user rekhajoshm commented on the issue: https://github.com/apache/spark/pull/21684 retest please.the failure unrelated to this PR. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark issue #21684: [SPARK-24470][Core] RestSubmissionClient to be robust ag...

2018-07-05 Thread rekhajoshm
Github user rekhajoshm commented on the issue: https://github.com/apache/spark/pull/21684 ``` Discovery starting. Discovery completed in 5 seconds, 946 milliseconds. Run starting. Expected test count is: 16 StandaloneRestSubmitSuite: - construct submit request

[GitHub] spark pull request #21684: [SPARK-24470][Core] RestSubmissionClient to be ro...

2018-06-30 Thread rekhajoshm
GitHub user rekhajoshm opened a pull request: https://github.com/apache/spark/pull/21684 [SPARK-24470][Core] RestSubmissionClient to be robust against 404 & non json responses ## What changes were proposed in this pull request? Added check for 404, to avoid json par

[GitHub] spark pull request #21660: [SPARK-24683][K8S] Fix k8s no resource

2018-06-30 Thread rekhajoshm
Github user rekhajoshm commented on a diff in the pull request: https://github.com/apache/spark/pull/21660#discussion_r199335279 --- Diff: resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala --- @@ -21,17

[GitHub] spark pull request #21683: [SPARK-24507][Documentation] Update streaming gui...

2018-06-30 Thread rekhajoshm
GitHub user rekhajoshm opened a pull request: https://github.com/apache/spark/pull/21683 [SPARK-24507][Documentation] Update streaming guide ## What changes were proposed in this pull request? Updated streaming guide for direct stream and link to integration guide

[GitHub] spark pull request #21604: [SPARK-24614][PySpark] : Fix for SyntaxWarning on...

2018-06-21 Thread rekhajoshm
GitHub user rekhajoshm opened a pull request: https://github.com/apache/spark/pull/21604 [SPARK-24614][PySpark] : Fix for SyntaxWarning on tests.py ## What changes were proposed in this pull request? Fix for SyntaxWarning on tests.py ## How was this patch tested

[GitHub] spark pull request #19404: [SPARK-21760] [Streaming] Fix for Structured stre...

2018-04-25 Thread rekhajoshm
Github user rekhajoshm closed the pull request at: https://github.com/apache/spark/pull/19404 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #19404: [SPARK-21760] [Streaming] Fix for Structured streaming t...

2018-04-25 Thread rekhajoshm
Github user rekhajoshm commented on the issue: https://github.com/apache/spark/pull/19404 Thanks for the good inputs.Closing this PR. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20501: [SPARK-22430][R][Docs] Unknown tag warnings when buildin...

2018-03-04 Thread rekhajoshm
Github user rekhajoshm commented on the issue: https://github.com/apache/spark/pull/20501 done @felixcheung thanks --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #20501: [SPARK-22430][Docs] Unknown tag warnings when bui...

2018-03-04 Thread rekhajoshm
GitHub user rekhajoshm reopened a pull request: https://github.com/apache/spark/pull/20501 [SPARK-22430][Docs] Unknown tag warnings when building R docs with Roxygen 6.0.1 ## What changes were proposed in this pull request? Removed @export tag to get rid of unknown tag warnings

[GitHub] spark pull request #20556: [SPARK-23367][Build] Include python document styl...

2018-02-26 Thread rekhajoshm
Github user rekhajoshm commented on a diff in the pull request: https://github.com/apache/spark/pull/20556#discussion_r170792111 --- Diff: dev/lint-python --- @@ -82,6 +87,34 @@ else rm "$PYCODESTYLE_REPORT_PATH" fi +# Check python document style,

[GitHub] spark pull request #20556: [SPARK-23367][Build] Include python document styl...

2018-02-26 Thread rekhajoshm
Github user rekhajoshm commented on a diff in the pull request: https://github.com/apache/spark/pull/20556#discussion_r170792037 --- Diff: dev/tox.ini --- @@ -17,3 +17,5 @@ ignore=E402,E731,E241,W503,E226,E722,E741,E305 max-line-length=100 exclude=cloudpickle.py

[GitHub] spark pull request #20556: [SPARK-23367][Build] Include python document styl...

2018-02-26 Thread rekhajoshm
Github user rekhajoshm commented on a diff in the pull request: https://github.com/apache/spark/pull/20556#discussion_r170792051 --- Diff: dev/lint-python --- @@ -82,6 +87,34 @@ else rm "$PYCODESTYLE_REPORT_PATH" fi +# Check python document style,

[GitHub] spark pull request #20556: [SPARK-23367][Build] Include python document styl...

2018-02-26 Thread rekhajoshm
Github user rekhajoshm commented on a diff in the pull request: https://github.com/apache/spark/pull/20556#discussion_r170792029 --- Diff: dev/lint-python --- @@ -21,10 +21,15 @@ SCRIPT_DIR="$( cd "$( dirname "$0" )" && pwd )"

[GitHub] spark pull request #20556: [SPARK-23367][Build] Include python document styl...

2018-02-26 Thread rekhajoshm
Github user rekhajoshm commented on a diff in the pull request: https://github.com/apache/spark/pull/20556#discussion_r170792015 --- Diff: dev/lint-python --- @@ -21,10 +21,15 @@ SCRIPT_DIR="$( cd "$( dirname "$0" )" && pwd )"

[GitHub] spark issue #20556: [SPARK-23367][Build] Include python document style check...

2018-02-13 Thread rekhajoshm
Github user rekhajoshm commented on the issue: https://github.com/apache/spark/pull/20556 build error unrelated to this PR. retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark pull request #20501: [SPARK-22430][Docs] Unknown tag warnings when bui...

2018-02-08 Thread rekhajoshm
Github user rekhajoshm closed the pull request at: https://github.com/apache/spark/pull/20501 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #20501: [SPARK-22430][Docs] Unknown tag warnings when building R...

2018-02-08 Thread rekhajoshm
Github user rekhajoshm commented on the issue: https://github.com/apache/spark/pull/20501 Ack. thanks for the update @felixcheung @srowen Closing this. --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark pull request #20378: [SPARK-11222][Build][Python] Python document styl...

2018-02-08 Thread rekhajoshm
Github user rekhajoshm closed the pull request at: https://github.com/apache/spark/pull/20378 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #20378: [SPARK-11222][Build][Python] Python document style check...

2018-02-08 Thread rekhajoshm
Github user rekhajoshm commented on the issue: https://github.com/apache/spark/pull/20378 @HyukjinKwon @holdenk @ueshin @viirya @icexelloss @felixcheung @BryanCutler and @MrBago - This was one of the possible approach that I was running by you. I have proposed another approach

[GitHub] spark issue #20378: [SPARK-11222][Build][Python] Python document style check...

2018-02-08 Thread rekhajoshm
Github user rekhajoshm commented on the issue: https://github.com/apache/spark/pull/20378 @HyukjinKwon Identifying docstyle failures does not help much as it is not straightforward to exclude in this version

[GitHub] spark pull request #20378: [SPARK-11222][Build][Python] Python document styl...

2018-02-08 Thread rekhajoshm
Github user rekhajoshm commented on a diff in the pull request: https://github.com/apache/spark/pull/20378#discussion_r167148657 --- Diff: dev/lint-python --- @@ -83,6 +84,53 @@ else rm "$PEP8_REPORT_PATH" fi + Python Document St

[GitHub] spark pull request #20556: [SPARK-23367][Build] Include python document styl...

2018-02-08 Thread rekhajoshm
GitHub user rekhajoshm opened a pull request: https://github.com/apache/spark/pull/20556 [SPARK-23367][Build] Include python document style checking ## What changes were proposed in this pull request? Include python document style checking. This PR includes the pydocstyle

[GitHub] spark issue #20378: [SPARK-11222][Build][Python] Python document style check...

2018-02-06 Thread rekhajoshm
Github user rekhajoshm commented on the issue: https://github.com/apache/spark/pull/20378 @HyukjinKwon I will check it over weekend.thanks --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark pull request #20501: [SPARK-22430][Docs] Unknown tag warnings when bui...

2018-02-03 Thread rekhajoshm
GitHub user rekhajoshm opened a pull request: https://github.com/apache/spark/pull/20501 [SPARK-22430][Docs] Unknown tag warnings when building R docs with Roxygen 6.0.1 ## What changes were proposed in this pull request? Removed @export tag to get rid of unknown tag warnings

[GitHub] spark pull request #20500: [SPARK-14023][MLlib] Make exceptions consistent r...

2018-02-03 Thread rekhajoshm
GitHub user rekhajoshm opened a pull request: https://github.com/apache/spark/pull/20500 [SPARK-14023][MLlib] Make exceptions consistent regarding fields and columns ## What changes were proposed in this pull request? Make exceptions consistent regarding fields and columns

[GitHub] spark pull request #20378: [SPARK-11222][Build][Python] Python document styl...

2018-01-30 Thread rekhajoshm
Github user rekhajoshm commented on a diff in the pull request: https://github.com/apache/spark/pull/20378#discussion_r164926603 --- Diff: dev/lint-python --- @@ -83,6 +84,53 @@ else rm "$PEP8_REPORT_PATH" fi + Python Document St

[GitHub] spark issue #20432: [SPARK-23174][BUILD][PYTHON][FOLLOWUP] Add pycodestyle*....

2018-01-29 Thread rekhajoshm
Github user rekhajoshm commented on the issue: https://github.com/apache/spark/pull/20432 👍 LGTM @ueshin --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark issue #20378: [SPARK-11222][Build][Python] Python document style check...

2018-01-29 Thread rekhajoshm
Github user rekhajoshm commented on the issue: https://github.com/apache/spark/pull/20378 Thanks @HyukjinKwon for your update. @HyukjinKwon @holdenk @ueshin @viirya @icexelloss @felixcheung @BryanCutler and @MrBago - While you are thinking on it, below is my analysis

[GitHub] spark issue #20378: [SPARK-11222][Build][Python] Python document style check...

2018-01-28 Thread rekhajoshm
Github user rekhajoshm commented on the issue: https://github.com/apache/spark/pull/20378 Thanks @HyukjinKwon for review. This checks only for python files, as $PATHS_TO_CHECK is passed, and it takes only python files(find . -name "*.py").Also the single fi

[GitHub] spark pull request #20418: [SPARK-790][MESOS] Implement reregistered callbac...

2018-01-28 Thread rekhajoshm
GitHub user rekhajoshm opened a pull request: https://github.com/apache/spark/pull/20418 [SPARK-790][MESOS] Implement reregistered callback for MesosScheduler ## What changes were proposed in this pull request? Implement reregistered callback for MesosScheduler

[GitHub] spark issue #20347: [SPARK-20129][Core] JavaSparkContext should use SparkCon...

2018-01-26 Thread rekhajoshm
Github user rekhajoshm commented on the issue: https://github.com/apache/spark/pull/20347 Thank you @srowen I admire you for doing what you do over all the jira/PR's I have studied, and followed up. If its ok, will keep this PR open for few days, and close if jira is getting

[GitHub] spark pull request #20378: [SPARK-11222][Build][Python] Python document styl...

2018-01-23 Thread rekhajoshm
GitHub user rekhajoshm opened a pull request: https://github.com/apache/spark/pull/20378 [SPARK-11222][Build][Python] Python document style checker added ## What changes were proposed in this pull request? Using pydocstyle for python document style checker https://github.com

[GitHub] spark issue #20338: [SPARK-23174][BUILD][PYTHON] python code style checker u...

2018-01-23 Thread rekhajoshm
Github user rekhajoshm commented on the issue: https://github.com/apache/spark/pull/20338 @HyukjinKwon @ueshin was bit hesitant to change variable/references as pep8 is very clear within python circles as python style checker.however after pondering, have done the needful. please

[GitHub] spark pull request #20347: [SPARK-20129][Core] JavaSparkContext should use S...

2018-01-21 Thread rekhajoshm
GitHub user rekhajoshm opened a pull request: https://github.com/apache/spark/pull/20347 [SPARK-20129][Core] JavaSparkContext should use SparkContext.getOrCreate ## What changes were proposed in this pull request? Using SparkContext getOrCreate() instead of recreating new sc

[GitHub] spark pull request #20338: [SPARK-23174][BUILD][PYTHON] python code style ch...

2018-01-21 Thread rekhajoshm
Github user rekhajoshm commented on a diff in the pull request: https://github.com/apache/spark/pull/20338#discussion_r162838933 --- Diff: dev/tox.ini --- @@ -13,7 +13,7 @@ # See the License for the specific language governing permissions and # limitations under

[GitHub] spark pull request #20338: [SPARK-11222][BUILD][PYTHON] python code style ch...

2018-01-21 Thread rekhajoshm
Github user rekhajoshm commented on a diff in the pull request: https://github.com/apache/spark/pull/20338#discussion_r162837153 --- Diff: dev/tox.ini --- @@ -13,7 +13,7 @@ # See the License for the specific language governing permissions and # limitations under

[GitHub] spark issue #20338: [SPARK-11222][BUILD][PYTHON] python code style checker u...

2018-01-21 Thread rekhajoshm
Github user rekhajoshm commented on the issue: https://github.com/apache/spark/pull/20338 Thanks @HyukjinKwon for your review.updated, please verify. --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark pull request #20338: [SPARK-11222][BUILD][PYTHON] python code style ch...

2018-01-21 Thread rekhajoshm
GitHub user rekhajoshm opened a pull request: https://github.com/apache/spark/pull/20338 [SPARK-11222][BUILD][PYTHON] python code style checker update ## What changes were proposed in this pull request? Referencing latest python code style checking from PyPi/pycodestyle

[GitHub] spark pull request #20337: [SPARK-11630] [core] ClosureCleaner moved from wa...

2018-01-21 Thread rekhajoshm
GitHub user rekhajoshm opened a pull request: https://github.com/apache/spark/pull/20337 [SPARK-11630] [core] ClosureCleaner moved from warning to debug ## What changes were proposed in this pull request? ClosureCleaner moved from warning to debug ## How was this patch

[GitHub] spark issue #20337: [SPARK-11630] [core] ClosureCleaner moved from warning t...

2018-01-21 Thread rekhajoshm
Github user rekhajoshm commented on the issue: https://github.com/apache/spark/pull/20337 thanks @srowen done. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #19407: [SPARK-21667][Streaming] ConsoleSink should not f...

2017-11-10 Thread rekhajoshm
Github user rekhajoshm commented on a diff in the pull request: https://github.com/apache/spark/pull/19407#discussion_r150330668 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamWriter.scala --- @@ -267,11 +267,12 @@ final class DataStreamWriter[T

[GitHub] spark pull request #19418: [SPARK-19984][SQL] Fix for ERROR codegen.CodeGene...

2017-10-04 Thread rekhajoshm
Github user rekhajoshm closed the pull request at: https://github.com/apache/spark/pull/19418 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #19418: [SPARK-19984][SQL] Fix for ERROR codegen.CodeGenerator: ...

2017-10-04 Thread rekhajoshm
Github user rekhajoshm commented on the issue: https://github.com/apache/spark/pull/19418 @viirya You are correct, i am on latest master, and i did not get it yet. This PR was to have a defensive condition. As, if this happens only under certain unique data/flow, this is the one

[GitHub] spark pull request #19418: [SPARK-19984][SQL] Fix for ERROR codegen.CodeGene...

2017-10-04 Thread rekhajoshm
Github user rekhajoshm commented on a diff in the pull request: https://github.com/apache/spark/pull/19418#discussion_r142848020 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala --- @@ -697,7 +697,12 @@ class

[GitHub] spark issue #19422: [SPARK-22193][SQL] Minor typo fix

2017-10-03 Thread rekhajoshm
Github user rekhajoshm commented on the issue: https://github.com/apache/spark/pull/19422 @srowen got it, just the pull you mean, didn't think of it then. thanks --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark pull request #19422: [SPARK-22193][SQL] Minor typo fix

2017-10-03 Thread rekhajoshm
GitHub user rekhajoshm opened a pull request: https://github.com/apache/spark/pull/19422 [SPARK-22193][SQL] Minor typo fix ## What changes were proposed in this pull request? [SPARK-22193][SQL] Minor typo fix ## How was this patch tested? existing tests You

[GitHub] spark pull request #19418: [SPARK-19984][SQL] Fix for ERROR codegen.CodeGene...

2017-10-03 Thread rekhajoshm
Github user rekhajoshm commented on a diff in the pull request: https://github.com/apache/spark/pull/19418#discussion_r142475030 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala --- @@ -697,7 +697,12 @@ class

[GitHub] spark pull request #19418: [SPARK-19984][SQL] Fix for ERROR codegen.CodeGene...

2017-10-02 Thread rekhajoshm
GitHub user rekhajoshm opened a pull request: https://github.com/apache/spark/pull/19418 [SPARK-19984][SQL] Fix for ERROR codegen.CodeGenerator: failed to compile: org.codehaus.commons.compiler.CompileException: for generated java file ## What changes were proposed in this pull

[GitHub] spark pull request #19407: [SPARK-21667][Streaming] ConsoleSink should not f...

2017-10-01 Thread rekhajoshm
Github user rekhajoshm commented on a diff in the pull request: https://github.com/apache/spark/pull/19407#discussion_r142023140 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamWriter.scala --- @@ -269,7 +269,7 @@ final class DataStreamWriter[T] private

[GitHub] spark pull request #19407: [SPARK-21667][Streaming] ConsoleSink should not f...

2017-10-01 Thread rekhajoshm
GitHub user rekhajoshm opened a pull request: https://github.com/apache/spark/pull/19407 [SPARK-21667][Streaming] ConsoleSink should not fail streaming query with checkpointLocation option ## What changes were proposed in this pull request? Fix to allow recovery on console

[GitHub] spark issue #19404: [SPARK-21760] [Streaming] Fix for Structured streaming t...

2017-09-30 Thread rekhajoshm
Github user rekhajoshm commented on the issue: https://github.com/apache/spark/pull/19404 Seems to be apache spark git/jenkins issue.Please retest after a while.thanks --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark pull request #19404: [SPARK-21760] [Streaming] Fix for Structured stre...

2017-09-30 Thread rekhajoshm
GitHub user rekhajoshm opened a pull request: https://github.com/apache/spark/pull/19404 [SPARK-21760] [Streaming] Fix for Structured streaming terminates with Exception ## What changes were proposed in this pull request? Updated serialize call for flush and FSDataOutputStream

[GitHub] spark pull request: [SPARK-14372] [SQL] : Dataset.randomSplit() ne...

2016-04-07 Thread rekhajoshm
Github user rekhajoshm commented on the pull request: https://github.com/apache/spark/pull/12184#issuecomment-207131155 thanks @srowen .updated. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-14372] [SQL] : Dataset.randomSplit() ne...

2016-04-07 Thread rekhajoshm
Github user rekhajoshm commented on a diff in the pull request: https://github.com/apache/spark/pull/12184#discussion_r58959234 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala --- @@ -1492,6 +1491,8 @@ class Dataset[T] private[sql]( * @param weights

[GitHub] spark pull request: [SPARK-14372] [SQL] : Dataset.randomSplit() ne...

2016-04-06 Thread rekhajoshm
Github user rekhajoshm commented on the pull request: https://github.com/apache/spark/pull/12184#issuecomment-206624502 @marmbrus thanks for your review. added test.thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark pull request: [SPARK-14410] [SQL] : SessionCatalog needs to ...

2016-04-06 Thread rekhajoshm
Github user rekhajoshm closed the pull request at: https://github.com/apache/spark/pull/12183 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request: [SPARK-14410] [SQL] : SessionCatalog needs to ...

2016-04-06 Thread rekhajoshm
Github user rekhajoshm commented on the pull request: https://github.com/apache/spark/pull/12183#issuecomment-206171895 ok thanks @andrewor14 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-14410] [SQL] : SessionCatalog needs to ...

2016-04-05 Thread rekhajoshm
Github user rekhajoshm commented on the pull request: https://github.com/apache/spark/pull/12183#issuecomment-206017437 @yhuai - will do, one question is., as database calls had ignoreIfExists/ignoreIfNotExists flag, now with adding existence check this boolean loses meaning.Do we

[GitHub] spark pull request: [SPARK-14372] [SQL] : Dataset.randomSplit() ne...

2016-04-05 Thread rekhajoshm
GitHub user rekhajoshm opened a pull request: https://github.com/apache/spark/pull/12184 [SPARK-14372] [SQL] : Dataset.randomSplit() needs a Java version ## What changes were proposed in this pull request? 1.Added method randomSplitAsList() in Dataset for java

[GitHub] spark pull request: [SPARK-14410] [SQL] : SessionCatalog needs to ...

2016-04-05 Thread rekhajoshm
GitHub user rekhajoshm opened a pull request: https://github.com/apache/spark/pull/12183 [SPARK-14410] [SQL] : SessionCatalog needs to check database/table/function existence ## What changes were proposed in this pull request? 1.Check for existence of Function in catalog

[GitHub] spark pull request: [SPARK-14006] [SparkR] R style changes

2016-03-20 Thread rekhajoshm
Github user rekhajoshm closed the pull request at: https://github.com/apache/spark/pull/11827 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request: [SPARK-14006] [SparkR] R style changes

2016-03-19 Thread rekhajoshm
GitHub user rekhajoshm opened a pull request: https://github.com/apache/spark/pull/11827 [SPARK-14006] [SparkR] R style changes ## What changes were proposed in this pull request? https://issues.apache.org/jira/browse/SPARK-14006 Most of them were fixed in latest build

[GitHub] spark pull request: [SPARK-14006] [SparkR] R style changes

2016-03-18 Thread rekhajoshm
Github user rekhajoshm commented on the pull request: https://github.com/apache/spark/pull/11827#issuecomment-198517461 I did this in earlier pull, but while merging saw them fixed in latest build. --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark pull request: [SPARK-13973] [PySpark]: `ipython notebook` is...

2016-03-18 Thread rekhajoshm
GitHub user rekhajoshm opened a pull request: https://github.com/apache/spark/pull/11829 [SPARK-13973] [PySpark]: `ipython notebook` is going away ## What changes were proposed in this pull request? https://issues.apache.org/jira/browse/SPARK-13973 ## How was this patch

[GitHub] spark pull request: [SPARK-14006] [SparkR] R style changes

2016-03-18 Thread rekhajoshm
Github user rekhajoshm commented on the pull request: https://github.com/apache/spark/pull/11837#issuecomment-198579660 seems like some issue with branch-1.6 merge.. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark pull request: [SPARK-14006] [SparkR] R style changes

2016-03-18 Thread rekhajoshm
GitHub user rekhajoshm opened a pull request: https://github.com/apache/spark/pull/11837 [SPARK-14006] [SparkR] R style changes ## What changes were proposed in this pull request? https://issues.apache.org/jira/browse/SPARK-14006 You can merge this pull request

[GitHub] spark pull request: [SPARK-14006] [SparkR] R style changes

2016-03-18 Thread rekhajoshm
Github user rekhajoshm closed the pull request at: https://github.com/apache/spark/pull/11837 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request: [SPARK-14006] [SparkR] R style changes

2016-03-18 Thread rekhajoshm
Github user rekhajoshm commented on the pull request: https://github.com/apache/spark/pull/11837#issuecomment-198624802 yes, was to fix it on branch-1.6, some issue with this branch.closing this pr. --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark pull request: [SPARK-11531] [ML] : SparseVector error Msg

2016-01-05 Thread rekhajoshm
Github user rekhajoshm commented on the pull request: https://github.com/apache/spark/pull/9525#issuecomment-169196705 Thanks @jkbradley might have missed it or thought it was under discussion.updated.thanks --- If your project is set up for it, you can reply to this email and have

[GitHub] spark pull request: [SPARK-11968] [MLlib] : MatrixFactorizationMod...

2015-12-13 Thread rekhajoshm
Github user rekhajoshm closed the pull request at: https://github.com/apache/spark/pull/9980 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request: [SPARK-11968] [MLlib] : MatrixFactorizationMod...

2015-11-30 Thread rekhajoshm
Github user rekhajoshm commented on the pull request: https://github.com/apache/spark/pull/9980#issuecomment-160803099 I concur @mengxr . Tried YourKit, and VisualVm Profiling.This does not fix the concern based on my runs with MovieLensALS and RecommendationExample. I do run

[GitHub] spark pull request: [SPARK-11968] [MLlib] : MatrixFactorizationMod...

2015-11-25 Thread rekhajoshm
GitHub user rekhajoshm opened a pull request: https://github.com/apache/spark/pull/9980 [SPARK-11968] [MLlib] : MatrixFactorizationModel recommendAll for GC times Fix for ALS recommend all methods for GC times You can merge this pull request into a Git repository by running

[GitHub] spark pull request: [SPARK-11968] [MLlib] : MatrixFactorizationMod...

2015-11-25 Thread rekhajoshm
Github user rekhajoshm commented on the pull request: https://github.com/apache/spark/pull/9980#issuecomment-159772331 Thanks @mengxr Any alternative suggestion for improving upon objects needed on recommendAll functionality? I did multiple profiling/heap dump by running

[GitHub] spark pull request: [SPARK-11531] [ML] : SparseVector error Msg

2015-11-06 Thread rekhajoshm
GitHub user rekhajoshm opened a pull request: https://github.com/apache/spark/pull/9525 [SPARK-11531] [ML] : SparseVector error Msg PySpark SparseVector should have "Found duplicate indices" error message You can merge this pull request into a Git repository by running:

[GitHub] spark pull request: [Spark-11478] [ML] ML StringIndexer return inc...

2015-11-04 Thread rekhajoshm
Github user rekhajoshm commented on the pull request: https://github.com/apache/spark/pull/9440#issuecomment-153831974 Thanks @yanboliang for your comments.My findings were similar to yours, and that nullable is the cause, driven by attr.toStructField().This was a few secs quick look

[GitHub] spark pull request: [SPARK-10317] [Core] Compatibility between his...

2015-10-02 Thread rekhajoshm
Github user rekhajoshm commented on the pull request: https://github.com/apache/spark/pull/8758#issuecomment-145134027 done.please have a look. thanks @vanzin @srowen ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark pull request: [SPARK-10317] [Core] Compatibility between his...

2015-10-02 Thread rekhajoshm
Github user rekhajoshm commented on the pull request: https://github.com/apache/spark/pull/8758#issuecomment-144941702 Unrelated errors. ``` [error] SERVER ERROR: Service Temporarily Unavailable url=http://maven.twttr.com/org/apache/hadoop/hadoop-yarn-server/2.2.0/hadoop-yarn

[GitHub] spark pull request: [SPARK-10317] [Core] Compatibility between his...

2015-10-01 Thread rekhajoshm
Github user rekhajoshm commented on the pull request: https://github.com/apache/spark/pull/8758#issuecomment-144863677 makes sense.good discussion @vanzin @srowen .updated.please review.thanks! --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark pull request: [SPARK-10317] [Core] Compatibility between his...

2015-10-01 Thread rekhajoshm
Github user rekhajoshm commented on the pull request: https://github.com/apache/spark/pull/8758#issuecomment-144914254 updated.kind of know @vanzin needs parseOtherArgs() to be named differently :-) please let me know. thanks @vanzin @srowen ! --- If your project is set up

[GitHub] spark pull request: [SPARK-10317] [Core] Compatibility between his...

2015-10-01 Thread rekhajoshm
Github user rekhajoshm commented on the pull request: https://github.com/apache/spark/pull/8758#issuecomment-144922099 unrelated error. ``` ERROR: Error fetching remote repo 'origin' hudson.plugins.git.GitException: Failed to fetch from https://github.com/apache/spark.git

[GitHub] spark pull request: [SPARK-10317] [Core] Compatibility between his...

2015-09-30 Thread rekhajoshm
Github user rekhajoshm commented on the pull request: https://github.com/apache/spark/pull/8758#issuecomment-144604656 Thanks for your comments @vanzin ! However that does not seem an accurate assessment.Before this patch, the script accepted only first parameter and set

[GitHub] spark pull request: [SPARK-10317] [Core] Compatibility between his...

2015-09-29 Thread rekhajoshm
Github user rekhajoshm commented on the pull request: https://github.com/apache/spark/pull/8758#issuecomment-144286924 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-10317] [Core] Compatibility between his...

2015-09-29 Thread rekhajoshm
Github user rekhajoshm commented on the pull request: https://github.com/apache/spark/pull/8758#issuecomment-144286887 unrelated error.ERROR: Publisher 'Publish JUnit test result report' failed: No test report files were found. Configuration error? Build succeeded locally

[GitHub] spark pull request: [SPARK-10317] [Core] Compatibility between his...

2015-09-29 Thread rekhajoshm
Github user rekhajoshm commented on the pull request: https://github.com/apache/spark/pull/8758#issuecomment-144287146 Thanks @srowen .I was earlier doing counters logic(below) to validate only if first param in HSArguments and it seemed not as clean/intentful as the script. Hence

  1   2   >