[GitHub] spark issue #17618: [SPARK-20291][SQL][BACKPORT] NaNvl(FloatType, NullType) ...
Github user dbtsai commented on the issue: https://github.com/apache/spark/pull/17618 @viirya Added. Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17617: [SPARK-20244][Core] Handle incorrect bytesRead metrics w...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17617 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17616: [SPARK-20304][SQL] AssertNotNull should not include path...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17616 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17618: [SPARK-20291][SQL][BACKPORT] NaNvl(FloatType, NullType) ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17618 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17601: [MINOR][SQL] Fix the @since tag when backporting SPARK-1...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17601 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17618: [SPARK-20291][SQL][BACKPORT] NaNvl(FloatType, NullType) ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17618 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75732/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17618: [SPARK-20291][SQL][BACKPORT] NaNvl(FloatType, NullType) ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17618 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75731/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17616: [SPARK-20304][SQL] AssertNotNull should not include path...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17616 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75728/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17617: [SPARK-20244][Core] Handle incorrect bytesRead metrics w...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17617 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75729/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17601: [MINOR][SQL] Fix the @since tag when backporting SPARK-1...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17601 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75730/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17524: [SPARK-19235] [SQL] [TEST] [FOLLOW-UP] Enable Test Cases...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17524 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75727/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17615: [SPARK-20303][SQL] Rename createTempFunction to register...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17615 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75725/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17613: [SPARK-20301][FLAKY-TEST][DO NOT MERGE] Fix Hadoop Shell...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17613 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17524: [SPARK-19235] [SQL] [TEST] [FOLLOW-UP] Enable Test Cases...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17524 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17613: [SPARK-20301][FLAKY-TEST][DO NOT MERGE] Fix Hadoop Shell...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17613 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75726/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17615: [SPARK-20303][SQL] Rename createTempFunction to register...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17615 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17618: [SPARK-20291][SQL][BACKPORT] NaNvl(FloatType, NullType) ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17618 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17613: [SPARK-20301][FLAKY-TEST] Fix Hadoop Shell.runCommand fl...
Github user brkyvz commented on the issue: https://github.com/apache/spark/pull/17613 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17613: [SPARK-20301][FLAKY-TEST] Fix Hadoop Shell.runCommand fl...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17613 **[Test build #75733 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75733/testReport)** for PR 17613 at commit [`4d6e3cb`](https://github.com/apache/spark/commit/4d6e3cb957e5c08a0ba2b62d7a4445cc218f5e83). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17618: [SPARK-20291][SQL][BACKPORT] NaNvl(FloatType, NullType) ...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17618 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17618: [SPARK-20291][SQL][BACKPORT] NaNvl(FloatType, NullType) ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17618 **[Test build #75734 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75734/consoleFull)** for PR 17618 at commit [`a50c2e7`](https://github.com/apache/spark/commit/a50c2e7ab027cf64bc7a261f02851b04e2525dc5). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17606: [SPARK-20291][SQL] NaNvl(FloatType, NullType) sho...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17606#discussion_r111078930
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala
---
@@ -571,6 +571,7 @@ object TypeCoercion {
NaNvl(l, Cast(r, DoubleType))
case NaNvl(l, r) if l.dataType == FloatType && r.dataType ==
DoubleType =>
NaNvl(Cast(l, DoubleType), r)
+ case NaNvl(l, r) if r.dataType == NullType => NaNvl(l, Cast(r,
l.dataType))
--- End diff --
Because `FunctionArgumentConversion` is executed before
`ImplicitTypeCasts`. When there is no danger of loss of information, the cast
can be implicit for better usability. We can add the extra configuration flag
for users to stop implicit casting.
If we do not upcast `NaNvl(FloatType, DoubleType)` to `NaNvl(DoubleType,
DoubleType)`, what is the output data type?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17615: [SPARK-20303][SQL] Rename createTempFunction to register...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17615 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17615: [SPARK-20303][SQL] Rename createTempFunction to register...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17615 **[Test build #75735 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75735/testReport)** for PR 17615 at commit [`e876af1`](https://github.com/apache/spark/commit/e876af1882a53fcd5569594e9ea486dba66850b4). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17524: [SPARK-19235] [SQL] [TEST] [FOLLOW-UP] Enable Test Cases...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17524 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17581: [SPARK-20248][ SQL]Spark SQL add limit parameter ...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17581#discussion_r111080132
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -359,6 +359,17 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val THRIFTSERVER_RESULT_LIMIT =
+buildConf("spark.sql.thriftserver.retainedResults")
+ .internal()
+ .doc("The maximum number of rows that can be returned by Thrift
Server when running " +
+"a query without a Limit clause. The query could produce
non-deterministic results " +
+"when the query result is not sorted. This parameter is omitted
when a Limit " +
+"clause is part of the query or the input is not a positive
integer.")
+ .intConf
+ .checkValue(limit => limit >= 0, "The number of sql results is not
less than 0.")
--- End diff --
I think we can remove this line now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17619: [SPARK-19755][Mesos] Blacklist is always active f...
GitHub user timout opened a pull request: https://github.com/apache/spark/pull/17619 [SPARK-19755][Mesos] Blacklist is always active for MesosCoarseGrainedSchedulerBackend. ## What changes were proposed in this pull request? MesosCoarseGrainedSchedulerBackend ignored spark.blacklist.enabled configuration property and used hardcoded MAX_SLAVE_FAILURES = 2. The purpose of that fix is to remove that hard-coded behaviour. BlacklistTracker is resposible for blacklist functionality. ## How was this patch tested? Unit tests, Manual testing. This patch is a clean up. That functionality is tested by BlacklistTracker tests. Author: tabaku...@pulsepoint.com You can merge this pull request into a Git repository by running: $ git pull https://github.com/timout/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/17619.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #17619 commit 70ed95ad8c73c1c1ff46dcf191b26f96c51ea09e Author: antiout Date: 2017-04-12T06:17:30Z Removed hardcoded blacklist functionality, must be controled by BlacklistTracker commit 078634e63aaeacc1b2361a16e1999f0213284ecf Author: antiout Date: 2017-04-12T06:17:34Z Merge remote-tracking branch 'upstream/master' commit df2f319a518e1a533dae04d5d6bfa019a8b6845c Author: antiout Date: 2017-04-12T06:35:28Z Removed hardcoded blacklist functionality, must be controled by BlacklistTracker --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17581: [SPARK-20248][ SQL]Spark SQL add limit parameter ...
Github user shaolinliu commented on a diff in the pull request:
https://github.com/apache/spark/pull/17581#discussion_r111080385
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -359,6 +359,16 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val THRIFTSERVER_RESULT_LIMIT =
+buildConf("spark.sql.thriftserver.retainedResults")
--- End diff --
In hive, we use parameter "hive.fetch.task.conversion=minimal" to take
result from mr job's output(from disk), in this mode hive will not collect the
result to the memory, avoiding the hive process crash. And I can not think of a
good name.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17616: [SPARK-20304][SQL] AssertNotNull should not include path...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17616 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17616: [SPARK-20304][SQL] AssertNotNull should not include path...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17616 **[Test build #75736 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75736/testReport)** for PR 17616 at commit [`bc30530`](https://github.com/apache/spark/commit/bc30530e9c08e33e234b5d16099d1fbdb33fbef0). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17619: [SPARK-19755][Mesos] Blacklist is always active for Meso...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17619 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17546: [SPARK-20233] [SQL] Apply star-join filter heuris...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/17546#discussion_r111081203
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/CostBasedJoinReorder.scala
---
@@ -54,8 +54,6 @@ case class CostBasedJoinReorder(conf: SQLConf) extends
Rule[LogicalPlan] with Pr
private def reorder(plan: LogicalPlan, output: Seq[Attribute]):
LogicalPlan = {
val (items, conditions) = extractInnerJoins(plan)
-// TODO: Compute the set of star-joins and use them in the join
enumeration
-// algorithm to prune un-optimal plan choices.
--- End diff --
What the meaning of "remove the call from `ReorderJoin`"?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17506: [SPARK-20189][DStream] Fix spark kinesis testcase...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/17506#discussion_r111082480
--- Diff:
external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisInputDStream.scala
---
@@ -267,7 +267,7 @@ object KinesisInputDStream {
getRequiredParam(checkpointAppName, "checkpointAppName"),
checkpointInterval.getOrElse(ssc.graph.batchDuration),
storageLevel.getOrElse(DEFAULT_STORAGE_LEVEL),
-handler,
+ssc.sc.clean(handler),
--- End diff --
Is this related?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17506: [SPARK-20189][DStream] Fix spark kinesis testcase...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/17506#discussion_r111082438
--- Diff:
external/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/KinesisStreamSuite.scala
---
@@ -233,11 +241,15 @@ abstract class KinesisStreamTests(aggregateTestData:
Boolean) extends KinesisFun
val localTestUtils = new KPLBasedKinesisTestUtils(1)
localTestUtils.createStream()
try {
- val awsCredentials = KinesisTestUtils.getAWSCredentials()
- val stream = KinesisUtils.createStream(ssc, localAppName,
localTestUtils.streamName,
-localTestUtils.endpointUrl, localTestUtils.regionName,
InitialPositionInStream.LATEST,
-Seconds(10), StorageLevel.MEMORY_ONLY,
-awsCredentials.getAWSAccessKeyId, awsCredentials.getAWSSecretKey)
+ val stream = KinesisInputDStream.builder.streamingContext(ssc)
--- End diff --
Just for my understanding, it's no longer necessary to pass in the
credentials explicitly?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17611: [SPARK-20298][SparkR][MINOR] fixed spelling mistake "cha...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17611 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17611: [SPARK-20298][SparkR][MINOR] fixed spelling mistake "cha...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17611 **[Test build #75737 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75737/testReport)** for PR 17611 at commit [`58b2fa8`](https://github.com/apache/spark/commit/58b2fa882feef2c2a303c86616ff4c921ba558b0). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17617: [SPARK-20244][Core] Handle incorrect bytesRead metrics w...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/17617 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17617: [SPARK-20244][Core] Handle incorrect bytesRead metrics w...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17617 **[Test build #75738 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75738/testReport)** for PR 17617 at commit [`d6f3c42`](https://github.com/apache/spark/commit/d6f3c42c74ab38b0b6becc80a80b5aeda4459c40). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17477: [SPARK-18692][BUILD][DOCS] Test Java 8 unidoc build on J...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17477 **[Test build #75739 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75739/testReport)** for PR 17477 at commit [`aefae0f`](https://github.com/apache/spark/commit/aefae0fbdc1a5323cc11336dd82e48a97ac8b971). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17477: [SPARK-18692][BUILD][DOCS] Test Java 8 unidoc bui...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/17477#discussion_r111086455
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -704,12 +704,12 @@ private[spark] object TaskSchedulerImpl {
* Used to balance containers across hosts.
*
* Accepts a map of hosts to resource offers for that host, and returns
a prioritized list of
- * resource offers representing the order in which the offers should be
used. The resource
+ * resource offers representing the order in which the offers should be
used. The resource
* offers are ordered such that we'll allocate one container on each
host before allocating a
* second container on any host, and so on, in order to reduce the
damage if a host fails.
*
- * For example, given , , ,
returns
- * [o1, o5, o4, 02, o6, o3]
+ * For example, given {@literal }, {@literal } and
+ * {@literal }, returns {@literal [o1, o5, o4, o2, o6,
o3]}.
--- End diff --
It seems we can't use `@code` here if there are codes such as `` (it
seems `< A...>` case looks fine. I ran some tests with the comments below:
```
* For example, given {@code < h1, [o1, o2, o3] >}, {@code < h2, [o4]>} and
{@code },
* returns {@code [o1, o5, o4, o2, o6, o3]}.
*
* For example, given
*
* {@code },
*
* {@code },
*
* returns {@code [o1, o5, o4, o2, o6, o3]}.
```
**Scaladoc**
**Javadoc**
If we use `@literal`, it seems fine.
**Scaladoc**
**Javadoc**
This seems not exposed in the API documentation anyway.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17477: [SPARK-18692][BUILD][DOCS] Test Java 8 unidoc bui...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/17477#discussion_r111086738
--- Diff:
resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala
---
@@ -296,7 +296,7 @@ trait MesosSchedulerUtils extends Logging {
/**
* Parses the attributes constraints provided to spark and build a
matching data struct:
- * Map[, Set[values-to-match]]
+ * {@literal Map[, Set[values-to-match]}
--- End diff --
Same instance with
https://github.com/apache/spark/pull/17477/files#r111086455.
- `@code`
- `@literal`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17477: [SPARK-18692][BUILD][DOCS] Test Java 8 unidoc bui...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/17477#discussion_r111086919
--- Diff:
sql/hive-thriftserver/src/main/java/org/apache/hive/service/auth/HttpAuthUtils.java
---
@@ -89,7 +89,7 @@ public static String getKerberosServiceTicket(String
principal, String host,
* @param clientUserName Client User name.
* @return An unsigned cookie token generated from input parameters.
* The final cookie generated is of the following format :
- * cu=&rn=&s=
+ * {@code cu=&rn=&s=}
--- End diff --
This is java code. So, `@code` should be fine. This also seems not exposed
to the documentation anyway.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17477: [SPARK-18692][BUILD][DOCS] Test Java 8 unidoc bui...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/17477#discussion_r111087424
--- Diff: core/src/main/scala/org/apache/spark/rpc/RpcEndpoint.scala ---
@@ -35,7 +35,7 @@ private[spark] trait RpcEnvFactory {
*
* The life-cycle of an endpoint is:
*
- * constructor -> onStart -> receive* -> onStop
+ * {@code constructor -> onStart -> receive* -> onStop}
--- End diff --
After this, it produces the documentation as below (manually tested)
**Scaladoc**
**Javadoc**
This also seems not exposed to API documentation anyway.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17564: [MINOR][DOCS] Fix spacings in Structured Streaming Progr...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/17564 Merged to master/2.1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17609: [SPARK-20296][TRIVIAL][DOCS] Count distinct error messag...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17609 **[Test build #3660 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3660/testReport)** for PR 17609 at commit [`a4d34c5`](https://github.com/apache/spark/commit/a4d34c5bcfe53ef05c56f8ce6838bbcda30c9f7e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17024: [SPARK-19525][CORE] Compressing checkpoints.
Github user aramesh117 commented on the issue: https://github.com/apache/spark/pull/17024 @mridulm Waiting for @tdas and @zsxwing has taken more than a month now. Is there any other way we can reach them or is there anyone else that can take a look at this merge request? This is a critical change that is needed for Conviva's use case if we are to upgrade to later versions of Spark. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17620: [SPARK-20305][Spark Core]Master may keep in the s...
GitHub user lvdongr opened a pull request: https://github.com/apache/spark/pull/17620 [SPARK-20305][Spark Core]Master may keep in the state of "COMPELETING… ## What changes were proposed in this pull request? Master may keep in the state of "COMPELETING_RECOVERY",then all the application registered cannot get resources, when the leader master change. This happend when a exception was thrown during the Master trying to recovery(completeRecovery method in the master.scala ). Then the leader will always in COMPLETING_RECOVERY state ,for the leader can only change to alive from state of RecoveryState.RECOVERING. ## How was this patch tested? manual tests Please review http://spark.apache.org/contributing.html before opening a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/lvdongr/spark SPARK20305 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/17620.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #17620 commit 44b9415dd1c6ac854a9debddd67c9dcb00e8df69 Author: lvdongr Date: 2017-04-12T07:34:03Z [SPARK-20305][Spark Core]Master may keep in the state of "COMPELETING_RECOVERY",then all the application registered cannot get resources, when the leader master change. has been successfully created. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17564: [MINOR][DOCS] Fix spacings in Structured Streamin...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/17564 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17620: [SPARK-20305][Spark Core]Master may keep in the state of...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17620 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17602: [MINOR][DOCS] JSON APIs related documentation fixes
Github user srowen commented on the issue: https://github.com/apache/spark/pull/17602 Merged to master --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17575: [SPARK-20265][MLlib] Improve Prefix'span pre-processing ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17575 **[Test build #3661 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3661/testReport)** for PR 17575 at commit [`d799d46`](https://github.com/apache/spark/commit/d799d460e215c017b4385e8ecbbca8b92128096a). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17602: [MINOR][DOCS] JSON APIs related documentation fix...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/17602 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17611: [SPARK-20298][SparkR][MINOR] fixed spelling mistake "cha...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17611 **[Test build #75737 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75737/testReport)** for PR 17611 at commit [`58b2fa8`](https://github.com/apache/spark/commit/58b2fa882feef2c2a303c86616ff4c921ba558b0). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17611: [SPARK-20298][SparkR][MINOR] fixed spelling mistake "cha...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17611 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17611: [SPARK-20298][SparkR][MINOR] fixed spelling mistake "cha...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17611 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75737/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17602: [MINOR][DOCS] JSON APIs related documentation fixes
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/17602 Thank you @srowen. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17611: [SPARK-20298][SparkR][MINOR] fixed spelling mistake "cha...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/17611 Merged to master --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17611: [SPARK-20298][SparkR][MINOR] fixed spelling mista...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/17611 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17614: [SPARK-20302][SQL] Short circuit cast when from and to t...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17614 Merging in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17614: [SPARK-20302][SQL] Short circuit cast when from a...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/17614 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17436: [SPARK-20101][SQL] Use OffHeapColumnVector when "...
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/17436#discussion_r111094256 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java --- @@ -90,6 +91,8 @@ */ private ColumnarBatch columnarBatch; + private MemoryMode memoryMode = null; --- End diff -- why do we need to record the memory mode? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17436: [SPARK-20101][SQL] Use OffHeapColumnVector when "...
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/17436#discussion_r111095838
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala
---
@@ -351,11 +351,12 @@ class ParquetFileFormat
if (pushed.isDefined) {
ParquetInputFormat.setFilterPredicate(hadoopAttemptContext.getConfiguration,
pushed.get)
}
+ val taskContext = Option(TaskContext.get())
val parquetReader = if (enableVectorizedReader) {
val vectorizedReader = new VectorizedParquetRecordReader()
vectorizedReader.initialize(split, hadoopAttemptContext)
logDebug(s"Appending $partitionSchema ${file.partitionValues}")
-vectorizedReader.initBatch(partitionSchema, file.partitionValues)
+vectorizedReader.initBatch(partitionSchema, file.partitionValues,
taskContext.isDefined)
--- End diff --
When `taskContext.isDefined` is false, we force to use on-heap.
This is because we cannot register `iter.close` at
[here](https://github.com/apache/spark/pull/17436/files#diff-ee26d4c4be21e92e92a02e9f16dbc285R380)
to release off-heap.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17546: [SPARK-20233] [SQL] Apply star-join filter heuris...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/17546#discussion_r111098128
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/CostBasedJoinReorder.scala
---
@@ -150,12 +148,15 @@ object JoinReorderDP extends PredicateHelper with
Logging {
case (item, id) => Set(id) -> JoinPlan(Set(id), item, Set(), Cost(0,
0))
}.toMap)
+// Build filters from the join graph to be used by the search
algorithm.
+val filters = JoinReorderDPFilters.buildJoinGraphInfo(conf, items,
conditions, itemIndex)
--- End diff --
why call it filters? should we name it `planInfo`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17546: [SPARK-20233] [SQL] Apply star-join filter heuris...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/17546#discussion_r111098412
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/CostBasedJoinReorder.scala
---
@@ -327,3 +349,109 @@ object JoinReorderDP extends PredicateHelper with
Logging {
case class Cost(card: BigInt, size: BigInt) {
def +(other: Cost): Cost = Cost(this.card + other.card, this.size +
other.size)
}
+
+/**
+ * Implements optional filters to reduce the search space for join
enumeration.
+ *
+ * 1) Star-join filters: Plan star-joins together since they are assumed
+ *to have an optimal execution based on their RI relationship.
+ * 2) Cartesian products: Defer their planning later in the graph to avoid
--- End diff --
We already have this logic in the dp join reorder algorithm.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17546: [SPARK-20233] [SQL] Apply star-join filter heuris...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/17546#discussion_r111098712
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/CostBasedJoinReorder.scala
---
@@ -327,3 +349,109 @@ object JoinReorderDP extends PredicateHelper with
Logging {
case class Cost(card: BigInt, size: BigInt) {
def +(other: Cost): Cost = Cost(this.card + other.card, this.size +
other.size)
}
+
+/**
+ * Implements optional filters to reduce the search space for join
enumeration.
+ *
+ * 1) Star-join filters: Plan star-joins together since they are assumed
+ *to have an optimal execution based on their RI relationship.
+ * 2) Cartesian products: Defer their planning later in the graph to avoid
+ *large intermediate results (expanding joins, in general).
+ * 3) Composite inners: Don't generate "bushy tree" plans to avoid
materializing
+ * intermediate results.
--- End diff --
I don't get it, doesn't left-deep tree materialize intermediate results?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17436: [SPARK-20101][SQL] Use OffHeapColumnVector when "...
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/17436#discussion_r111098912
--- Diff:
core/src/main/scala/org/apache/spark/memory/UnifiedMemoryManager.scala ---
@@ -210,7 +210,7 @@ object UnifiedMemoryManager {
private def getMaxMemory(conf: SparkConf): Long = {
val systemMemory = conf.getLong("spark.testing.memory",
Runtime.getRuntime.maxMemory)
val reservedMemory = conf.getLong("spark.testing.reservedMemory",
- if (conf.contains("spark.testing")) 0 else
RESERVED_SYSTEM_MEMORY_BYTES)
+ if (conf.contains("spark.testing") || true) 0 else
RESERVED_SYSTEM_MEMORY_BYTES)
--- End diff --
Sorry, it is my mistake.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17436: [SPARK-20101][SQL] Use OffHeapColumnVector when "spark.m...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17436 **[Test build #75740 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75740/testReport)** for PR 17436 at commit [`71aef5f`](https://github.com/apache/spark/commit/71aef5f0ea598a85dc103d31d2edf88fce68df1f). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17436: [SPARK-20101][SQL] Use OffHeapColumnVector when "...
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/17436#discussion_r00556
--- Diff:
core/src/test/scala/org/apache/spark/memory/StaticMemoryManagerSuite.scala ---
@@ -48,7 +48,10 @@ class StaticMemoryManagerSuite extends
MemoryManagerSuite {
conf.clone
.set("spark.memory.fraction", "1")
.set("spark.testing.memory", maxOnHeapExecutionMemory.toString)
-.set("spark.memory.offHeap.size",
maxOffHeapExecutionMemory.toString),
+.set("spark.memory.offHeap.size",
+ if (maxOffHeapExecutionMemory != 0L) {
maxOffHeapExecutionMemory.toString } else {
+conf.get("spark.memory.offHeap.size",
maxOffHeapExecutionMemory.toString)
--- End diff --
If `maxOffHeapExecutionMemory` is not explicitly specified like [this
code](https://github.com/apache/spark/blob/master/core/src/test/scala/org/apache/spark/memory/MemoryManagerSuite.scala#L162)),
we want to use the value in `"spark.memory.offHeap.size"` when
`spark.memory.offHeap.enabled` is `true`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17575: [SPARK-20265][MLlib] Improve Prefix'span pre-processing ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17575 **[Test build #3661 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3661/testReport)** for PR 17575 at commit [`d799d46`](https://github.com/apache/spark/commit/d799d460e215c017b4385e8ecbbca8b92128096a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17436: [SPARK-20101][SQL] Use OffHeapColumnVector when "...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/17436#discussion_r01490
--- Diff:
core/src/test/scala/org/apache/spark/memory/StaticMemoryManagerSuite.scala ---
@@ -48,7 +48,10 @@ class StaticMemoryManagerSuite extends
MemoryManagerSuite {
conf.clone
.set("spark.memory.fraction", "1")
.set("spark.testing.memory", maxOnHeapExecutionMemory.toString)
-.set("spark.memory.offHeap.size",
maxOffHeapExecutionMemory.toString),
+.set("spark.memory.offHeap.size",
+ if (maxOffHeapExecutionMemory != 0L) {
maxOffHeapExecutionMemory.toString } else {
+conf.get("spark.memory.offHeap.size",
maxOffHeapExecutionMemory.toString)
--- End diff --
can we just change the default value of `maxOffHeapExecutionMemory` from 0
to something like 1000?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17613: [SPARK-20301][FLAKY-TEST] Fix Hadoop Shell.runCommand fl...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17613 **[Test build #75733 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75733/testReport)** for PR 17613 at commit [`4d6e3cb`](https://github.com/apache/spark/commit/4d6e3cb957e5c08a0ba2b62d7a4445cc218f5e83). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17618: [SPARK-20291][SQL][BACKPORT] NaNvl(FloatType, NullType) ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17618 **[Test build #75734 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75734/consoleFull)** for PR 17618 at commit [`a50c2e7`](https://github.com/apache/spark/commit/a50c2e7ab027cf64bc7a261f02851b04e2525dc5). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17613: [SPARK-20301][FLAKY-TEST] Fix Hadoop Shell.runCommand fl...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17613 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17613: [SPARK-20301][FLAKY-TEST] Fix Hadoop Shell.runCommand fl...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17613 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75733/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17618: [SPARK-20291][SQL][BACKPORT] NaNvl(FloatType, NullType) ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17618 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75734/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17618: [SPARK-20291][SQL][BACKPORT] NaNvl(FloatType, NullType) ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17618 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17024: [SPARK-19525][CORE] Compressing checkpoints.
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/17024 @aramesh117 Unfortunately, since this heavily affects streaming, I cannot sign off on it without someone more familiar with spark streaming reviews it as well. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17149: [SPARK-19257][SQL]location for table/partition/da...
Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/17149#discussion_r04548 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala --- @@ -285,7 +285,7 @@ private[spark] class HiveExternalCatalog(conf: SparkConf, hadoopConf: Configurat // compatible format, which means the data source is file-based and must have a `path`. require(table.storage.locationUri.isDefined, "External file-based data source table must have a `path` entry in storage properties.") -Some(new Path(table.location).toUri.toString) --- End diff -- there's actually some discussion [about defining this properly](https://issues.apache.org/jira/browse/HADOOP-14217), related to the problem of "colons in object store paths". That's not going to help directly, but a sign of something which has historically been underspecified --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17615: [SPARK-20303][SQL] Rename createTempFunction to register...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17615 **[Test build #75735 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75735/testReport)** for PR 17615 at commit [`e876af1`](https://github.com/apache/spark/commit/e876af1882a53fcd5569594e9ea486dba66850b4). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17615: [SPARK-20303][SQL] Rename createTempFunction to register...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17615 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75735/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17615: [SPARK-20303][SQL] Rename createTempFunction to register...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17615 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17616: [SPARK-20304][SQL] AssertNotNull should not include path...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17616 **[Test build #75736 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75736/testReport)** for PR 17616 at commit [`bc30530`](https://github.com/apache/spark/commit/bc30530e9c08e33e234b5d16099d1fbdb33fbef0). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17616: [SPARK-20304][SQL] AssertNotNull should not include path...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17616 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17616: [SPARK-20304][SQL] AssertNotNull should not include path...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17616 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75736/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17621: [SPARK-6227][MLLIB][PYSPARK] Implement PySpark wr...
GitHub user MLnick opened a pull request: https://github.com/apache/spark/pull/17621 [SPARK-6227][MLLIB][PYSPARK] Implement PySpark wrappers for SVD and PCA (v2) Add PCA and SVD to PySpark's wrappers for `RowMatrix` and `IndexedRowMatrix` (SVD only). Based on #7963, updated. ## How was this patch tested? New doc tests and unit tests. Ran all examples locally. You can merge this pull request into a Git repository by running: $ git pull https://github.com/MLnick/spark SPARK-6227-pyspark-svd-pca Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/17621.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #17621 commit 6248d0e85bdb69ac64fd664049b38819cb021daf Author: MechCoder Date: 2015-08-05T13:08:36Z [SPARK-6227] [MLlib] [PySpark] Implement PySpark wrappers for SVD commit 921d5b6ea7d74032dcf1b9e7c41db22a090de547 Author: MechCoder Date: 2015-08-05T17:55:13Z Add PCA Wrappers commit 5558fa9bffca7d81b0f35152512cd3692455ec99 Author: MechCoder Date: 2015-08-05T18:18:06Z Added docs commit 50ed70011d47521a4a20e415afd87736f357876e Author: MechCoder Date: 2015-08-05T19:01:08Z Add support for multiply and computeSVD in IRM commit 59f53d51cda6f9744fd2eed3cd633f1fecf11063 Author: MechCoder Date: 2015-08-05T20:56:56Z Added tests commit 70a871d189d92c41e477e6f426e12a9fbb549ce7 Author: MechCoder Date: 2015-08-10T07:45:46Z minor changes to doc commit 0bc6a3caf25c5c5a3cfcc661432890aff2293584 Author: MechCoder Date: 2016-05-27T01:27:09Z Add check for DenseMatrix commit 31db3a3a1f44e4db5fcf90d83600d7020577ae03 Author: Nick Pentreath Date: 2017-04-11T11:27:27Z Merge remote-tracking branch 'apache-github/master' into SPARK-6227-pyspark-svd-pca commit 3ba24114c0281502c93b82e2d781bd1cdc8c5aa1 Author: Nick Pentreath Date: 2017-04-12T09:36:52Z Update examples, add Python examples, doc and since tags --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17621: [SPARK-6227][MLLIB][PYSPARK] Implement PySpark wrappers ...
Github user MLnick commented on the issue: https://github.com/apache/spark/pull/17621 Based on @MechCoder's work in #7963. cc @holdenk @dusenberrymw --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #7963: [SPARK-6227] [MLlib] [PySpark] Implement PySpark wrappers...
Github user MLnick commented on the issue: https://github.com/apache/spark/pull/7963 Note I revived this at #17621 based on @MechCoder's work. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17621: [SPARK-6227][MLLIB][PYSPARK] Implement PySpark wrappers ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17621 **[Test build #75741 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75741/testReport)** for PR 17621 at commit [`3ba2411`](https://github.com/apache/spark/commit/3ba24114c0281502c93b82e2d781bd1cdc8c5aa1). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17621: [SPARK-6227][MLLIB][PYSPARK] Implement PySpark wrappers ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17621 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75741/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17621: [SPARK-6227][MLLIB][PYSPARK] Implement PySpark wrappers ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17621 **[Test build #75741 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75741/testReport)** for PR 17621 at commit [`3ba2411`](https://github.com/apache/spark/commit/3ba24114c0281502c93b82e2d781bd1cdc8c5aa1). * This patch **fails Python style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17621: [SPARK-6227][MLLIB][PYSPARK] Implement PySpark wrappers ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17621 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17620: [SPARK-20305][Spark Core]Master may keep in the s...
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/17620#discussion_r11204
--- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala
---
@@ -539,7 +539,7 @@ private[deploy] class Master(
private def completeRecovery() {
// Ensure "only-once" recovery semantics using a short synchronization
period.
-if (state != RecoveryState.RECOVERING) { return }
+if (state != RecoveryState.RECOVERING && state !=
RecoveryState.COMPLETING_RECOVERY) { return }
--- End diff --
@lvdongr would you please explain what will be happened after this change?
With your change, which code will call this `completeRecovery`? I see there're
4 places which will call this `completeRecovery`.
Also if we have exception during recovery, should we fail fast?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17621: [SPARK-6227][MLLIB][PYSPARK] Implement PySpark wrappers ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17621 **[Test build #75742 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75742/testReport)** for PR 17621 at commit [`3ea88e2`](https://github.com/apache/spark/commit/3ea88e27aa2d91adda7457afc64b8b5839e6396c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17621: [SPARK-6227][MLLIB][PYSPARK] Implement PySpark wrappers ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17621 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17621: [SPARK-6227][MLLIB][PYSPARK] Implement PySpark wrappers ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17621 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75742/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17621: [SPARK-6227][MLLIB][PYSPARK] Implement PySpark wrappers ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17621 **[Test build #75742 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75742/testReport)** for PR 17621 at commit [`3ea88e2`](https://github.com/apache/spark/commit/3ea88e27aa2d91adda7457afc64b8b5839e6396c). * This patch **fails Python style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17621: [SPARK-6227][MLLIB][PYSPARK] Implement PySpark wrappers ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17621 **[Test build #75743 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75743/testReport)** for PR 17621 at commit [`0118f2c`](https://github.com/apache/spark/commit/0118f2c0839a45ea68a65a639efaf9fc2eab4883). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17609: [SPARK-20296][TRIVIAL][DOCS] Count distinct error messag...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17609 **[Test build #3660 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3660/testReport)** for PR 17609 at commit [`a4d34c5`](https://github.com/apache/spark/commit/a4d34c5bcfe53ef05c56f8ce6838bbcda30c9f7e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17609: [SPARK-20296][TRIVIAL][DOCS] Count distinct error messag...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/17609 Merged to master --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17609: [SPARK-20296][TRIVIAL][DOCS] Count distinct error...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/17609 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
