[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16795 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16795 **[Test build #72473 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72473/testReport)** for PR 16795 at commit

[GitHub] spark issue #16376: [SPARK-18967][SCHEDULER] compute locality levels even if...

2017-02-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16376 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16376: [SPARK-18967][SCHEDULER] compute locality levels even if...

2017-02-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16376 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72477/ Test PASSed. ---

[GitHub] spark issue #16376: [SPARK-18967][SCHEDULER] compute locality levels even if...

2017-02-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16376 **[Test build #72477 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72477/testReport)** for PR 16376 at commit

[GitHub] spark issue #11760: [SPARK-13931] Resolve stage hanging up problem in a part...

2017-02-06 Thread kayousterhout
Github user kayousterhout commented on the issue: https://github.com/apache/spark/pull/11760 Great thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so,

[GitHub] spark issue #16797: [SPARK-19455][SQL] Add option for case-insensitive Parqu...

2017-02-06 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/16797 how about we add a new SQL command to refresh the table schema in metastore by inferring schema with data files? This is a compatibility issue and we should have provided a way for users to

[GitHub] spark issue #16797: [SPARK-19455][SQL] Add option for case-insensitive Parqu...

2017-02-06 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/16797 > I'll double check, but I don't think spark.sql.hive.manageFilesourcePartitions=false would solve this issue since we're still deriving the file relation's dataSchema parameter from the schema of

[GitHub] spark issue #16620: [SPARK-19263] DAGScheduler should avoid sending conflict...

2017-02-06 Thread kayousterhout
Github user kayousterhout commented on the issue: https://github.com/apache/spark/pull/16620 @mridulm yeah once I saw this it seemed like something that's probably been a lurking issue for a bunch of jobs!! Will be great to get this fixed -- thanks for finding it @jinxing64! ---

[GitHub] spark issue #16171: [SPARK-18739][ML][PYSPARK] Classification and regression...

2017-02-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16171 **[Test build #72482 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72482/testReport)** for PR 16171 at commit

[GitHub] spark issue #16495: SPARK-16920: Add a stress test for evaluateEachIteration...

2017-02-06 Thread jkbradley
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/16495 @mhmoudr Will you be able to update this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request #16737: [SPARK-19397] [SQL] Make option names of LIBSVM a...

2017-02-06 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/16737#discussion_r99732168 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/text/TextSuite.scala --- @@ -125,6 +124,25 @@ class TextSuite extends

[GitHub] spark issue #16814: [SPARK-19467][ML][PYTHON]Remove cyclic imports from pysp...

2017-02-06 Thread zero323
Github user zero323 commented on the issue: https://github.com/apache/spark/pull/16814 @jkbradley, @srowen Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark pull request #16737: [SPARK-19397] [SQL] Make option names of LIBSVM a...

2017-02-06 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/16737#discussion_r99732105 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/text/TextSuite.scala --- @@ -125,6 +124,25 @@ class TextSuite extends

[GitHub] spark issue #16747: SPARK-16636 Add CalendarIntervalType to documentation

2017-02-06 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/16747 ^ I want to be very sure if we are not going to expose this or not. Could any SQL committer guy or PMC confirm this? > `CalendarIntervalType` only for compatibility with similar type in

[GitHub] spark pull request #16814: [SPARK-19467][ML][PYTHON]Remove cyclic imports fr...

2017-02-06 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/16814 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #16787: [WIP][SPARK-19448][SQL]optimize some duplication functio...

2017-02-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16787 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16787: [WIP][SPARK-19448][SQL]optimize some duplication functio...

2017-02-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16787 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72478/ Test FAILed. ---

[GitHub] spark issue #16814: [SPARK-19467][ML][PYTHON]Remove cyclic imports from pysp...

2017-02-06 Thread jkbradley
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/16814 Btw, do you have a need to backport this to previous releases? Or is master sufficient? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark issue #16787: [WIP][SPARK-19448][SQL]optimize some duplication functio...

2017-02-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16787 **[Test build #72478 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72478/testReport)** for PR 16787 at commit

[GitHub] spark issue #16814: [SPARK-19467][ML][PYTHON]Remove cyclic imports from pysp...

2017-02-06 Thread jkbradley
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/16814 LGTM Merging with master Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16744: [SPARK-19405][STREAMING] Support for cross-account Kines...

2017-02-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16744 **[Test build #72481 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72481/testReport)** for PR 16744 at commit

[GitHub] spark issue #16747: SPARK-16636 Add CalendarIntervalType to documentation

2017-02-06 Thread terma
Github user terma commented on the issue: https://github.com/apache/spark/pull/16747 @srowen As I understood ```CalendarIntervalType``` only for compatibility with similar type in ```Hive```. So probably better to mark it as internal and close jira? --- If your project is set up

[GitHub] spark issue #15009: [SPARK-17443][SPARK-11035] Stop Spark Application if lau...

2017-02-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15009 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15009: [SPARK-17443][SPARK-11035] Stop Spark Application if lau...

2017-02-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15009 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72475/ Test PASSed. ---

[GitHub] spark pull request #16783: [SPARK-19441] [SQL] Remove IN type coercion from ...

2017-02-06 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/16783 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #15009: [SPARK-17443][SPARK-11035] Stop Spark Application if lau...

2017-02-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15009 **[Test build #72475 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72475/testReport)** for PR 15009 at commit

[GitHub] spark pull request #16737: [SPARK-19397] [SQL] Make option names of LIBSVM a...

2017-02-06 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/16737#discussion_r99730718 --- Diff: mllib/src/main/scala/org/apache/spark/ml/source/libsvm/LibSVMOptions.scala --- @@ -0,0 +1,51 @@ +/* + * Licensed to the Apache Software

[GitHub] spark issue #16783: [SPARK-19441] [SQL] Remove IN type coercion from Promote...

2017-02-06 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/16783 LGTM, merging to master! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #16744: [SPARK-19405][STREAMING] Support for cross-account Kines...

2017-02-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16744 **[Test build #72480 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72480/testReport)** for PR 16744 at commit

[GitHub] spark issue #16744: [SPARK-19405][STREAMING] Support for cross-account Kines...

2017-02-06 Thread budde
Github user budde commented on the issue: https://github.com/apache/spark/pull/16744 PR has been amended to reflect feedback. Thanks for taking a look, @brkyvz. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #11760: [SPARK-13931] Resolve stage hanging up problem in a part...

2017-02-06 Thread GavinGavinNo1
Github user GavinGavinNo1 commented on the issue: https://github.com/apache/spark/pull/11760 @kayousterhout Sorry that I tried before but internet connection is poor in my company and then I forget. I'll work on it back home today or tomorrow. Thank you for caring for this PR.

[GitHub] spark issue #16783: [SPARK-19441] [SQL] Remove IN type coercion from Promote...

2017-02-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16783 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16783: [SPARK-19441] [SQL] Remove IN type coercion from Promote...

2017-02-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16783 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72474/ Test PASSed. ---

[GitHub] spark issue #16783: [SPARK-19441] [SQL] Remove IN type coercion from Promote...

2017-02-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16783 **[Test build #72474 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72474/testReport)** for PR 16783 at commit

[GitHub] spark pull request #16824: [SPARK-18069][PYTHON] Make PySpark doctests for S...

2017-02-06 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/16824#discussion_r99729087 --- Diff: python/pyspark/sql/session.py --- @@ -51,8 +51,10 @@ def toDF(self, schema=None, sampleRatio=None): :param samplingRatio: the

[GitHub] spark pull request #16824: [SPARK-18069][PYTHON] Make PySpark doctests for S...

2017-02-06 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/16824#discussion_r99728911 --- Diff: python/pyspark/sql/dataframe.py --- @@ -553,10 +594,10 @@ def repartition(self, numPartitions, *cols): +---+-+

[GitHub] spark issue #16737: [SPARK-19397] [SQL] Make option names of LIBSVM and TEXT...

2017-02-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16737 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72476/ Test PASSed. ---

[GitHub] spark issue #16737: [SPARK-19397] [SQL] Make option names of LIBSVM and TEXT...

2017-02-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16737 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16737: [SPARK-19397] [SQL] Make option names of LIBSVM and TEXT...

2017-02-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16737 **[Test build #72476 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72476/testReport)** for PR 16737 at commit

[GitHub] spark pull request #16824: [SPARK-18069][PYTHON] Make PySpark doctests for S...

2017-02-06 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/16824#discussion_r99728558 --- Diff: python/pyspark/sql/dataframe.py --- @@ -296,8 +312,10 @@ def show(self, n=20, truncate=True): If set to a number greater than

[GitHub] spark pull request #16824: [SPARK-18069][PYTHON] Make PySpark doctests for S...

2017-02-06 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/16824#discussion_r99728503 --- Diff: python/pyspark/sql/dataframe.py --- @@ -247,9 +261,11 @@ def explain(self, extended=False): :param extended: boolean,

[GitHub] spark issue #16620: [SPARK-19263] DAGScheduler should avoid sending conflict...

2017-02-06 Thread mridulm
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/16620 @kayousterhout That sounds more clear, and I can see this being a problem (and probably explains some hung jobs I had seen a while earlier), thanks ! --- If your project is set up for it, you can

[GitHub] spark pull request #16824: [SPARK-18069][PYTHON] Make PySpark doctests for S...

2017-02-06 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/16824#discussion_r99727122 --- Diff: python/pyspark/sql/dataframe.py --- @@ -233,11 +245,13 @@ def schema(self): def printSchema(self): """Prints out the

[GitHub] spark issue #16715: [Spark-18080][ML] Python API & Examples for Locality Sen...

2017-02-06 Thread yanboliang
Github user yanboliang commented on the issue: https://github.com/apache/spark/pull/16715 @Yunni I'm on travel at Spark Summit East these days, and will review after the summit. Thanks for your patience. --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark pull request #16824: [SPARK-18069][PYTHON] Make PySpark doctests for S...

2017-02-06 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/16824#discussion_r99727033 --- Diff: python/pyspark/sql/dataframe.py --- @@ -218,8 +228,10 @@ def writeStream(self): def schema(self): """Returns the schema

[GitHub] spark issue #16825: [SPARK-19481][REPL][maven]Avoid to leak SparkContext in ...

2017-02-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16825 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72472/ Test PASSed. ---

[GitHub] spark issue #16825: [SPARK-19481][REPL][maven]Avoid to leak SparkContext in ...

2017-02-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16825 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16825: [SPARK-19481][REPL][maven]Avoid to leak SparkContext in ...

2017-02-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16825 **[Test build #72472 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72472/testReport)** for PR 16825 at commit

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16795 **[Test build #72479 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72479/testReport)** for PR 16795 at commit

[GitHub] spark pull request #15505: [SPARK-18890][CORE] Move task serialization from ...

2017-02-06 Thread kayousterhout
Github user kayousterhout commented on a diff in the pull request: https://github.com/apache/spark/pull/15505#discussion_r99724943 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskDescription.scala --- @@ -66,7 +100,8 @@ private[spark] object TaskDescription {

[GitHub] spark pull request #15505: [SPARK-18890][CORE] Move task serialization from ...

2017-02-06 Thread kayousterhout
Github user kayousterhout commented on a diff in the pull request: https://github.com/apache/spark/pull/15505#discussion_r99725005 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala --- @@ -243,27 +245,42 @@ class

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-06 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/16795 Retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #16787: [WIP][SPARK-19448][SQL]optimize some duplication functio...

2017-02-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16787 **[Test build #72478 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72478/testReport)** for PR 16787 at commit

[GitHub] spark issue #16797: [SPARK-19455][SQL] Add option for case-insensitive Parqu...

2017-02-06 Thread budde
Github user budde commented on the issue: https://github.com/apache/spark/pull/16797 > Should we roll these behaviors into one flag? e.g. ```spark.sql.hive.mixedCaseSchemaSupport``` That sounds reasonable to me. The only thing I wonder about is if there's any use case where

[GitHub] spark pull request #16744: [SPARK-19405][STREAMING] Support for cross-accoun...

2017-02-06 Thread budde
Github user budde commented on a diff in the pull request: https://github.com/apache/spark/pull/16744#discussion_r99718950 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala --- @@ -35,10 +36,65 @@ import

[GitHub] spark issue #16741: [SPARK-19402][DOCS] Support LaTex inline formula correct...

2017-02-06 Thread jkbradley
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/16741 Thanks for these many cleanups! It's a shame to lose links. Do you think we should use fully qualified names rather than abandoning the links? --- If your project is set up for it, you can

[GitHub] spark pull request #16744: [SPARK-19405][STREAMING] Support for cross-accoun...

2017-02-06 Thread budde
Github user budde commented on a diff in the pull request: https://github.com/apache/spark/pull/16744#discussion_r99718545 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/examples/streaming/KinesisExampleUtils.scala --- @@ -0,0 +1,22 @@ +/* + * Licensed to

[GitHub] spark issue #16376: [SPARK-18967][SCHEDULER] compute locality levels even if...

2017-02-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16376 **[Test build #72477 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72477/testReport)** for PR 16376 at commit

[GitHub] spark issue #11760: [SPARK-13931] Resolve stage hanging up problem in a part...

2017-02-06 Thread kayousterhout
Github user kayousterhout commented on the issue: https://github.com/apache/spark/pull/11760 @GavinGavinNo1 if you don't have time to work on this PR, can you close it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark issue #15326: [SPARK-17759] [CORE] Avoid adding duplicate schedulables

2017-02-06 Thread kayousterhout
Github user kayousterhout commented on the issue: https://github.com/apache/spark/pull/15326 @erenavsarogullari what's the status of this PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #15604: [SPARK-18066] [CORE] [TESTS] Add Pool usage policies tes...

2017-02-06 Thread kayousterhout
Github user kayousterhout commented on the issue: https://github.com/apache/spark/pull/15604 @erenavsarogullari what's the status with this PR? I know you have at least one other fair scheduler PR outstanding, and it makes reviews easier to minimize the number of outstanding things

[GitHub] spark issue #16376: [SPARK-18967][SCHEDULER] compute locality levels even if...

2017-02-06 Thread kayousterhout
Github user kayousterhout commented on the issue: https://github.com/apache/spark/pull/16376 @squito I just noticed this hasn't been merged. Is this good to go pending tests passing again? --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark issue #16376: [SPARK-18967][SCHEDULER] compute locality levels even if...

2017-02-06 Thread kayousterhout
Github user kayousterhout commented on the issue: https://github.com/apache/spark/pull/16376 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #16813: [SPARK-19466][CORE][SCHEDULER] Improve Fair Sched...

2017-02-06 Thread kayousterhout
Github user kayousterhout commented on a diff in the pull request: https://github.com/apache/spark/pull/16813#discussion_r99716361 --- Diff: core/src/main/scala/org/apache/spark/scheduler/SchedulableBuilder.scala --- @@ -69,19 +72,29 @@ private[spark] class

[GitHub] spark pull request #16813: [SPARK-19466][CORE][SCHEDULER] Improve Fair Sched...

2017-02-06 Thread kayousterhout
Github user kayousterhout commented on a diff in the pull request: https://github.com/apache/spark/pull/16813#discussion_r99717574 --- Diff: core/src/main/scala/org/apache/spark/scheduler/SchedulableBuilder.scala --- @@ -69,19 +72,29 @@ private[spark] class

[GitHub] spark pull request #16813: [SPARK-19466][CORE][SCHEDULER] Improve Fair Sched...

2017-02-06 Thread kayousterhout
Github user kayousterhout commented on a diff in the pull request: https://github.com/apache/spark/pull/16813#discussion_r99717754 --- Diff: core/src/test/scala/org/apache/spark/scheduler/PoolSuite.scala --- @@ -201,6 +202,49 @@ class PoolSuite extends SparkFunSuite with

[GitHub] spark pull request #16813: [SPARK-19466][CORE][SCHEDULER] Improve Fair Sched...

2017-02-06 Thread kayousterhout
Github user kayousterhout commented on a diff in the pull request: https://github.com/apache/spark/pull/16813#discussion_r99717324 --- Diff: core/src/main/scala/org/apache/spark/scheduler/SchedulableBuilder.scala --- @@ -69,19 +72,29 @@ private[spark] class

[GitHub] spark pull request #16813: [SPARK-19466][CORE][SCHEDULER] Improve Fair Sched...

2017-02-06 Thread kayousterhout
Github user kayousterhout commented on a diff in the pull request: https://github.com/apache/spark/pull/16813#discussion_r99712707 --- Diff: core/src/test/scala/org/apache/spark/scheduler/PoolSuite.scala --- @@ -28,19 +29,19 @@ import org.apache.spark.scheduler.SchedulingMode._

[GitHub] spark issue #16740: [SPARK-19400][ML] Allow GLM to handle intercept only mod...

2017-02-06 Thread sethah
Github user sethah commented on the issue: https://github.com/apache/spark/pull/16740 Regarding the tests - I don't think the tests should change _depending on_ the implementation. I don't think it's valid to say that we don't need to test this thoroughly because we know that it's

[GitHub] spark pull request #16744: [SPARK-19405][STREAMING] Support for cross-accoun...

2017-02-06 Thread brkyvz
Github user brkyvz commented on a diff in the pull request: https://github.com/apache/spark/pull/16744#discussion_r99713106 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala --- @@ -35,10 +36,65 @@ import

[GitHub] spark issue #16737: [SPARK-19397] [SQL] Make option names of LIBSVM and TEXT...

2017-02-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16737 **[Test build #72476 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72476/testReport)** for PR 16737 at commit

[GitHub] spark pull request #16824: [SPARK-18069][PYTHON] Make PySpark doctests for S...

2017-02-06 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/16824#discussion_r99711908 --- Diff: python/pyspark/sql/dataframe.py --- @@ -1191,6 +1287,12 @@ def dropna(self, how='any', thresh=None, subset=None): This

[GitHub] spark pull request #16744: [SPARK-19405][STREAMING] Support for cross-accoun...

2017-02-06 Thread brkyvz
Github user brkyvz commented on a diff in the pull request: https://github.com/apache/spark/pull/16744#discussion_r99711844 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisTestUtils.scala --- @@ -205,6 +205,8 @@ private[kinesis] object

[GitHub] spark issue #16826: Fork SparkSession with option to inherit a copy of the S...

2017-02-06 Thread kunalkhamar
Github user kunalkhamar commented on the issue: https://github.com/apache/spark/pull/16826 Hey @zsxwing Can you take another look and let me know if anything needs changes --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark issue #16826: Fork SparkSession with option to inherit a copy of the S...

2017-02-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16826 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark issue #16740: [SPARK-19400][ML] Allow GLM to handle intercept only mod...

2017-02-06 Thread imatiach-msft
Github user imatiach-msft commented on the issue: https://github.com/apache/spark/pull/16740 @actuaryzhang the changes look good to me. I had some nit-picks which you marked as won't fix, and I'm ok with that. Thank you for fixing this issue! Maybe a committer can review this -

[GitHub] spark pull request #16826: Fork SparkSession with option to inherit a copy o...

2017-02-06 Thread kunalkhamar
GitHub user kunalkhamar opened a pull request: https://github.com/apache/spark/pull/16826 Fork SparkSession with option to inherit a copy of the SessionState. ## What changes were proposed in this pull request? Forking a newSession() from SparkSession currently makes a new

[GitHub] spark issue #16783: [SPARK-19441] [SQL] Remove IN type coercion from Promote...

2017-02-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16783 **[Test build #72474 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72474/testReport)** for PR 16783 at commit

[GitHub] spark issue #15009: [SPARK-17443][SPARK-11035] Stop Spark Application if lau...

2017-02-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15009 **[Test build #72475 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72475/testReport)** for PR 15009 at commit

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16795 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16795 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72463/ Test FAILed. ---

[GitHub] spark issue #16762: [SPARK-19419] [SPARK-19420] Fix the cross join detection

2017-02-06 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/16762 cc @srinathshankar @sameeragarwal @hvanhovell @cloud-fan --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request #16620: [SPARK-19263] DAGScheduler should avoid sending c...

2017-02-06 Thread kayousterhout
Github user kayousterhout commented on a diff in the pull request: https://github.com/apache/spark/pull/16620#discussion_r99660324 --- Diff: core/src/test/scala/org/apache/spark/scheduler/SchedulerIntegrationSuite.scala --- @@ -391,17 +391,18 @@ private[spark] abstract class

[GitHub] spark pull request #16620: [SPARK-19263] DAGScheduler should avoid sending c...

2017-02-06 Thread kayousterhout
Github user kayousterhout commented on a diff in the pull request: https://github.com/apache/spark/pull/16620#discussion_r99659930 --- Diff: core/src/test/scala/org/apache/spark/scheduler/SchedulerIntegrationSuite.scala --- @@ -519,6 +520,18 @@ class TestTaskScheduler(sc:

[GitHub] spark pull request #16620: [SPARK-19263] DAGScheduler should avoid sending c...

2017-02-06 Thread kayousterhout
Github user kayousterhout commented on a diff in the pull request: https://github.com/apache/spark/pull/16620#discussion_r99661979 --- Diff: core/src/test/scala/org/apache/spark/scheduler/SchedulerIntegrationSuite.scala --- @@ -648,4 +661,70 @@ class

[GitHub] spark issue #16722: [SPARK-9478][ML][MLlib] Add sample weights to decision t...

2017-02-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16722 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72471/ Test PASSed. ---

[GitHub] spark pull request #16620: [SPARK-19263] DAGScheduler should avoid sending c...

2017-02-06 Thread kayousterhout
Github user kayousterhout commented on a diff in the pull request: https://github.com/apache/spark/pull/16620#discussion_r99660627 --- Diff: core/src/test/scala/org/apache/spark/scheduler/SchedulerIntegrationSuite.scala --- @@ -648,4 +661,70 @@ class

[GitHub] spark issue #16783: [SPARK-19441] [SQL] Remove IN type coercion from Promote...

2017-02-06 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/16783 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #16783: [SPARK-19441] [SQL] Remove IN type coercion from Promote...

2017-02-06 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/16783 cc @cloud-fan --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so,

[GitHub] spark issue #16722: [SPARK-9478][ML][MLlib] Add sample weights to decision t...

2017-02-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16722 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16722: [SPARK-9478][ML][MLlib] Add sample weights to decision t...

2017-02-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16722 **[Test build #72471 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72471/testReport)** for PR 16722 at commit

[GitHub] spark pull request #16744: [SPARK-19405][STREAMING] Support for cross-accoun...

2017-02-06 Thread brkyvz
Github user brkyvz commented on a diff in the pull request: https://github.com/apache/spark/pull/16744#discussion_r99709988 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/examples/streaming/KinesisExampleUtils.scala --- @@ -0,0 +1,22 @@ +/* + * Licensed to

[GitHub] spark issue #16744: [SPARK-19405][STREAMING] Support for cross-account Kines...

2017-02-06 Thread brkyvz
Github user brkyvz commented on the issue: https://github.com/apache/spark/pull/16744 Hi @budde, taking a look at this now. Sorry for the wait --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16795 **[Test build #72463 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72463/testReport)** for PR 16795 at commit

[GitHub] spark issue #16797: [SPARK-19455][SQL] Add option for case-insensitive Parqu...

2017-02-06 Thread ericl
Github user ericl commented on the issue: https://github.com/apache/spark/pull/16797 > I'll double check, but I don't think spark.sql.hive.manageFilesourcePartitions=false would solve this issue since we're still deriving the file relation's dataSchema parameter from the schema of

[GitHub] spark issue #16825: [SPARK-19481][REPL][maven]Avoid to leak SparkContext in ...

2017-02-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16825 **[Test build #72472 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72472/testReport)** for PR 16825 at commit

[GitHub] spark issue #16795: [SPARK-19409][BUILD][test-maven] Fix ParquetAvroCompatib...

2017-02-06 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16795 **[Test build #72473 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72473/testReport)** for PR 16795 at commit

[GitHub] spark issue #16740: [SPARK-19400][ML] Allow GLM to handle intercept only mod...

2017-02-06 Thread actuaryzhang
Github user actuaryzhang commented on the issue: https://github.com/apache/spark/pull/16740 @sethah @imatiach-msft Could you take another look and let me know if there are any additional changes needed on this PR? Thanks! --- If your project is set up for it, you can reply to this

[GitHub] spark pull request #16825: Avoid leak SparkContext in Signaling.cancelOnInte...

2017-02-06 Thread zsxwing
GitHub user zsxwing opened a pull request: https://github.com/apache/spark/pull/16825 Avoid leak SparkContext in Signaling.cancelOnInterrupt ## What changes were proposed in this pull request? `Signaling.cancelOnInterrupt` leaks a SparkContext per call and it makes

<    1   2   3   4   5   6   >