[GitHub] spark issue #15100: [SPARK-17317][SparkR] Add SparkR vignette to branch 2.0
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15100 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15100: [SPARK-17317][SparkR] Add SparkR vignette to branch 2.0
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15100 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65425/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15100: [SPARK-17317][SparkR] Add SparkR vignette to branch 2.0
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15100 **[Test build #65425 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65425/consoleFull)** for PR 15100 at commit [`878c9e0`](https://github.com/apache/spark/commit/878c9e0507ed5128b9218f13de05bf5892bc3b79). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14444: [SPARK-16839] [SQL] redundant aliases after clean...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/1#discussion_r78899346
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala
---
@@ -174,59 +174,32 @@ case class CreateMap(children: Seq[Expression])
extends Expression {
/**
* Returns a Row containing the evaluation of all children expressions.
*/
-@ExpressionDescription(
- usage = "_FUNC_(col1, col2, col3, ...) - Creates a struct with the given
field values.")
-case class CreateStruct(children: Seq[Expression]) extends Expression {
-
- override def foldable: Boolean = children.forall(_.foldable)
-
- override lazy val dataType: StructType = {
-val fields = children.zipWithIndex.map { case (child, idx) =>
- child match {
-case ne: NamedExpression =>
- StructField(ne.name, ne.dataType, ne.nullable, ne.metadata)
-case _ =>
- StructField(s"col${idx + 1}", child.dataType, child.nullable,
Metadata.empty)
+object CreateStruct extends (Seq[Expression] => CreateNamedStruct) {
+ def apply(children: Seq[Expression]) : CreateNamedStruct = {
+val namedStructArgs = {
+ val attNames = for {
+(child, idx) <- children.zipWithIndex
+ } yield {
+child match {
+ case ne: NamedExpression => ne.name
+ case _ => s"col${idx + 1}"
--- End diff --
I checked with hive, hive always use `col1`, `col2` as the field name for
`CreateStrcut`, let's follow it and simplify the logic here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15004: [SPARK-17440] [SPARK-17441] Fixed Multiple Bugs in ALTER...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15004 Thank you! : ) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15004: [SPARK-17440] [SPARK-17441] Fixed Multiple Bugs i...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/15004#discussion_r78898853
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala ---
@@ -383,7 +385,14 @@ case class AlterTableRenamePartitionCommand(
extends RunnableCommand {
override def run(sparkSession: SparkSession): Seq[Row] = {
-sparkSession.sessionState.catalog.renamePartitions(
+val catalog = sparkSession.sessionState.catalog
+val table = catalog.getTableMetadata(tableName)
+if (DDLUtils.isDatasourceTable(table)) {
+ throw new AnalysisException(
+"ALTER TABLE RENAME PARTITION is not allowed for tables defined
using the datasource API")
--- End diff --
Sure, will do it. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15054: [SPARK-17502] [SQL] Fix Multiple Bugs in DDL Statements ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15054 **[Test build #65426 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65426/consoleFull)** for PR 15054 at commit [`a01f6b3`](https://github.com/apache/spark/commit/a01f6b39f2073390ebf2445a1eca1c3925d24841). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15004: [SPARK-17440] [SPARK-17441] Fixed Multiple Bugs i...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/15004 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14659: [SPARK-16757] Set up Spark caller context to HDFS
Github user Sherry302 commented on a diff in the pull request:
https://github.com/apache/spark/pull/14659#discussion_r78898595
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2418,6 +2418,21 @@ private[spark] object Utils extends Logging {
sparkJars.map(_.split(",")).map(_.filter(_.nonEmpty)).toSeq.flatten
}
}
+
+ def setCallerContext(context: String): Boolean = {
+var succeed = false
+try {
+ val Builder =
Utils.classForName("org.apache.hadoop.ipc.CallerContext$Builder")
+ val builderInst =
Builder.getConstructor(classOf[String]).newInstance(context)
+ val ret = Builder.getMethod("build").invoke(builderInst)
--- End diff --
Yes. hdfsContext is more readable.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15004: [SPARK-17440] [SPARK-17441] Fixed Multiple Bugs in ALTER...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/15004 thanks, merging to master! you can address my last comment in your other PRs --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14659: [SPARK-16757] Set up Spark caller context to HDFS
Github user Sherry302 commented on a diff in the pull request:
https://github.com/apache/spark/pull/14659#discussion_r78898257
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2418,6 +2418,21 @@ private[spark] object Utils extends Logging {
sparkJars.map(_.split(",")).map(_.filter(_.nonEmpty)).toSeq.flatten
}
}
+
+ def setCallerContext(context: String): Boolean = {
+var succeed = false
+try {
+ val Builder =
Utils.classForName("org.apache.hadoop.ipc.CallerContext$Builder")
+ val builderInst =
Builder.getConstructor(classOf[String]).newInstance(context)
+ val ret = Builder.getMethod("build").invoke(builderInst)
+ val callerContext =
Utils.classForName("org.apache.hadoop.ipc.CallerContext")
--- End diff --
If make `val callerContext =
Utils.classForName("org.apache.hadoop.ipc.CallerContext")` out of `try` block,
Spark will throw exception when it runs on hadoop before 2.8.0. Also, moving
that line to the first of `try` block does not make any difference since
`Utils.classForName("org.apache.hadoop.ipc.CallerContext$Builder")` also needs
to check if `org.apache.hadoop.ipc.CallerContext` exists. I am not sure if I
got your point, could you please give more info about it? Thanks a lot.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15004: [SPARK-17440] [SPARK-17441] Fixed Multiple Bugs i...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/15004#discussion_r78897919
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala ---
@@ -383,7 +385,14 @@ case class AlterTableRenamePartitionCommand(
extends RunnableCommand {
override def run(sparkSession: SparkSession): Seq[Row] = {
-sparkSession.sessionState.catalog.renamePartitions(
+val catalog = sparkSession.sessionState.catalog
+val table = catalog.getTableMetadata(tableName)
+if (DDLUtils.isDatasourceTable(table)) {
+ throw new AnalysisException(
+"ALTER TABLE RENAME PARTITION is not allowed for tables defined
using the datasource API")
--- End diff --
we can just say `data source table`, to be consistent with other places.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15107: [SPARK-17551][SQL] complete the NULL ordering support in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15107 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65421/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15107: [SPARK-17551][SQL] complete the NULL ordering support in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15107 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15107: [SPARK-17551][SQL] complete the NULL ordering support in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15107 **[Test build #65421 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65421/consoleFull)** for PR 15107 at commit [`69a9d8a`](https://github.com/apache/spark/commit/69a9d8aacb3832f3e346a7b9dbb13dc522e4dc68). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14659: [SPARK-16757] Set up Spark caller context to HDFS
Github user Sherry302 commented on a diff in the pull request:
https://github.com/apache/spark/pull/14659#discussion_r78897068
--- Diff: core/src/main/scala/org/apache/spark/scheduler/Task.scala ---
@@ -79,6 +82,13 @@ private[spark] abstract class Task[T](
metrics)
TaskContext.setTaskContext(context)
taskThread = Thread.currentThread()
+
+val callerContext =
+
s"Spark_AppId_${appId.getOrElse("")}_AppAttemptId_${appAttemptId.getOrElse("None")}"
+
+
s"_JobId_${jobId.getOrElse("0")}_StageID_${stageId}_stageAttemptId_${stageAttemptId}"
+
+s"_taskID_${taskAttemptId}_attemptNumber_${attemptNumber}"
+Utils.setCallerContext(callerContext)
--- End diff --
Yes. Good catch!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #11105: [SPARK-12469][CORE] Data Property accumulators for Spark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/11105 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65424/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #11105: [SPARK-12469][CORE] Data Property accumulators for Spark
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11105 **[Test build #65424 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65424/consoleFull)** for PR 11105 at commit [`04814a3`](https://github.com/apache/spark/commit/04814a3b37f6b1d41331026d1a3ff63c4c66de1c). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #11105: [SPARK-12469][CORE] Data Property accumulators for Spark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/11105 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15100: [SPARK-17317][SparkR] Add SparkR vignette to branch 2.0
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15100 **[Test build #65425 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65425/consoleFull)** for PR 15100 at commit [`878c9e0`](https://github.com/apache/spark/commit/878c9e0507ed5128b9218f13de05bf5892bc3b79). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14659: [SPARK-16757] Set up Spark caller context to HDFS
Github user Sherry302 commented on a diff in the pull request:
https://github.com/apache/spark/pull/14659#discussion_r78896928
--- Diff: core/src/main/scala/org/apache/spark/scheduler/Task.scala ---
@@ -79,6 +82,13 @@ private[spark] abstract class Task[T](
metrics)
TaskContext.setTaskContext(context)
taskThread = Thread.currentThread()
+
+val callerContext =
+
s"Spark_AppId_${appId.getOrElse("")}_AppAttemptId_${appAttemptId.getOrElse("None")}"
+
+
s"_JobId_${jobId.getOrElse("0")}_StageID_${stageId}_stageAttemptId_${stageAttemptId}"
+
+s"_taskID_${taskAttemptId}_attemptNumber_${attemptNumber}"
--- End diff --
I have updated the PR to make the string shorter.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14659: [SPARK-16757] Set up Spark caller context to HDFS
Github user Sherry302 commented on a diff in the pull request:
https://github.com/apache/spark/pull/14659#discussion_r78896972
--- Diff: core/src/main/scala/org/apache/spark/scheduler/Task.scala ---
@@ -79,6 +82,13 @@ private[spark] abstract class Task[T](
metrics)
TaskContext.setTaskContext(context)
taskThread = Thread.currentThread()
+
+val callerContext =
+
s"Spark_AppId_${appId.getOrElse("")}_AppAttemptId_${appAttemptId.getOrElse("None")}"
+
--- End diff --
Yes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14659: [SPARK-16757] Set up Spark caller context to HDFS
Github user Sherry302 commented on a diff in the pull request:
https://github.com/apache/spark/pull/14659#discussion_r78896863
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala ---
@@ -184,6 +184,9 @@ private[spark] class ApplicationMaster(
try {
val appAttemptId = client.getAttemptId()
+ var context =
s"Spark_AppName_${System.getProperty("spark.app.name")}" +
--- End diff --
A CallerContext class has been added.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15099: [SPARK-17541][SQL] fix some DDL bugs about table managem...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15099 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65420/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15099: [SPARK-17541][SQL] fix some DDL bugs about table managem...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15099 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15099: [SPARK-17541][SQL] fix some DDL bugs about table managem...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15099 **[Test build #65420 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65420/consoleFull)** for PR 15099 at commit [`4ab777e`](https://github.com/apache/spark/commit/4ab777ecf8e0ae51835dc7e83cf299d245f151ea). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14659: [SPARK-16757] Set up Spark caller context to HDFS
Github user Sherry302 commented on a diff in the pull request:
https://github.com/apache/spark/pull/14659#discussion_r78896816
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2418,6 +2418,21 @@ private[spark] object Utils extends Logging {
sparkJars.map(_.split(",")).map(_.filter(_.nonEmpty)).toSeq.flatten
}
}
+
+ def setCallerContext(context: String): Boolean = {
+var succeed = false
+try {
+ val Builder =
Utils.classForName("org.apache.hadoop.ipc.CallerContext$Builder")
+ val builderInst =
Builder.getConstructor(classOf[String]).newInstance(context)
+ val ret = Builder.getMethod("build").invoke(builderInst)
+ val callerContext =
Utils.classForName("org.apache.hadoop.ipc.CallerContext")
+ callerContext.getMethod("setCurrent", callerContext).invoke(null,
ret)
+ succeed = true
+} catch {
+ case NonFatal(e) => logDebug(s"$e", e)
--- End diff --
I have updated this to "case NonFatal(e) => logInfo("Fail to set Spark
caller context", e)"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14659: [SPARK-16757] Set up Spark caller context to HDFS
Github user Sherry302 commented on a diff in the pull request:
https://github.com/apache/spark/pull/14659#discussion_r78896707
--- Diff: core/src/main/scala/org/apache/spark/scheduler/Task.scala ---
@@ -54,7 +54,10 @@ private[spark] abstract class Task[T](
val partitionId: Int,
// The default value is only used in tests.
val metrics: TaskMetrics = TaskMetrics.registered,
-@transient var localProperties: Properties = new Properties) extends
Serializable {
+@transient var localProperties: Properties = new Properties,
+val jobId: Option[Int] = None,
--- End diff --
Done.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14659: [SPARK-16757] Set up Spark caller context to HDFS
Github user Sherry302 commented on a diff in the pull request: https://github.com/apache/spark/pull/14659#discussion_r78896692 --- Diff: core/src/main/scala/org/apache/spark/scheduler/ShuffleMapTask.scala --- @@ -51,8 +51,12 @@ private[spark] class ShuffleMapTask( partition: Partition, @transient private var locs: Seq[TaskLocation], metrics: TaskMetrics, -localProperties: Properties) - extends Task[MapStatus](stageId, stageAttemptId, partition.index, metrics, localProperties) +localProperties: Properties, --- End diff -- Done. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14659: [SPARK-16757] Set up Spark caller context to HDFS
Github user Sherry302 commented on a diff in the pull request: https://github.com/apache/spark/pull/14659#discussion_r78896681 --- Diff: core/src/main/scala/org/apache/spark/scheduler/ResultTask.scala --- @@ -51,8 +51,12 @@ private[spark] class ResultTask[T, U]( locs: Seq[TaskLocation], val outputId: Int, localProperties: Properties, -metrics: TaskMetrics) - extends Task[U](stageId, stageAttemptId, partition.index, metrics, localProperties) +metrics: TaskMetrics, +jobId: Option[Int] = None, --- End diff -- Done. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14659: [SPARK-16757] Set up Spark caller context to HDFS
Github user Sherry302 commented on the issue: https://github.com/apache/spark/pull/14659 Hi, @tgravescs Thank you very much for the review. I have updated the PR based on your every comment, including adding a CallerContext class, updating java doc, and made the caller context string shorter, etc. Manual Tests against some Spark applications in Yarn client mode and Yarn cluster mode, and spark caller contexts are written into HDFS `hdfs-audit.log` successfully. The following is the screenshot of the audit log (SparkKMeans in yarn client mode): https://cloud.githubusercontent.com/assets/8546874/18539563/1eb16748-7acd-11e6-840a-0e8bfabf5954.png";> This is the caller context which was written into `hdfs-audit.log` by `Yarn Client`: ``` 2016-09-14 22:28:59,341 INFO FSNamesystem.audit: allowed=true ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/lr_big.txt dst=nullperm=null proto=rpc callerContext=SPARK_AppName_SparkKMeans_AppID_application_1473908768790_0007 ``` The callerContext above is `SPARK_AppName_***_AppID_***` These are the caller contexts which were written into `hdfs-audit.log` by `Task`: ``` 2016-09-14 22:29:06,525 INFO FSNamesystem.audit: allowed=true ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=opensrc=/lr_big.txt dst=nullperm=null proto=rpc callerContext=SPARK_AppID_application_1473908768790_0007_JobID_0_StageID_0_0_TaskId_1_0 2016-09-14 22:29:06,526 INFO FSNamesystem.audit: allowed=true ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=opensrc=/lr_big.txt dst=nullperm=null proto=rpc callerContext=SPARK_AppID_application_1473908768790_0007_JobID_0_StageID_0_0_TaskId_0_0 2016-09-14 22:29:06,526 INFO FSNamesystem.audit: allowed=true ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=opensrc=/lr_big.txt dst=nullperm=null proto=rpc callerContext=SPARK_AppID_application_1473908768790_0007_JobID_0_StageID_0_0_TaskId_2_0 ``` The callContext above is `SPARK_AppID_***_JobID_***_StageID_***_(StageAttemptID)_TaskId_***_(TaskAttemptNumber)`. The static strings `jobAttemptID`, `stageAttemptID`, and `attemptNumber` of tasks have been deleted. (For `jobAttemptID`, please refer the following records produced by SparkKMeans ran in Yarn cluster mode) The records below were written into `hdfs-audit.log` when SparkKMeans ran in Yarn cluster mode: ``` 2016-09-14 22:25:30,100 INFO FSNamesystem.audit: allowed=true ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=mkdirs src=/private/tmp/hadoop-wyang/nm-local-dir/usercache/wyang/appcache/application_1473908768790_0006/container_1473908768790_0006_01_01/spark-warehouse dst=nullperm=wyang:supergroup:rwxr-xr-x proto=rpc callerContext=SPARK_AppName_org.apache.spark.examples.SparkKMeans_AppID_application_1473908768790_0006_1 2016-09-14 22:25:33,635 INFO FSNamesystem.audit: allowed=true ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=opensrc=/lr_big.txt dst=nullperm=null proto=rpc callerContext=SPARK_AppID_application_1473908768790_0006_1_JobID_0_StageID_0_0_TaskId_0_0 2016-09-14 22:25:33,635 INFO FSNamesystem.audit: allowed=true ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=opensrc=/lr_big.txt dst=nullperm=null proto=rpc callerContext=SPARK_AppID_application_1473908768790_0006_1_JobID_0_StageID_0_0_TaskId_2_0 2016-09-14 22:25:33,635 INFO FSNamesystem.audit: allowed=true ugi=wyang (auth:SIMPLE) ip=/127.0.0.1 cmd=opensrc=/lr_big.txt dst=nullperm=null proto=rpc callerContext=SPARK_AppID_application_1473908768790_0006_1_JobID_0_StageID_0_0_TaskId_1_0 ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15108: [SPARK-17543] [Missing log4j config file for tests in co...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15108 **[Test build #65423 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65423/consoleFull)** for PR 15108 at commit [`f476755`](https://github.com/apache/spark/commit/f476755bf7653bca008d3e32375424c769ddb835). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #11105: [SPARK-12469][CORE] Data Property accumulators for Spark
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11105 **[Test build #65424 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65424/consoleFull)** for PR 11105 at commit [`04814a3`](https://github.com/apache/spark/commit/04814a3b37f6b1d41331026d1a3ff63c4c66de1c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #11105: [SPARK-12469][CORE] Data Property accumulators for Spark
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/11105 Seems like a possible MiMa missfire, jenkin retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15108: [SPARK-17543] [Missing log4j config file for test...
GitHub user jagadeesanas2 opened a pull request: https://github.com/apache/spark/pull/15108 [SPARK-17543] [Missing log4j config file for tests in common/network-⦠## What changes were proposed in this pull request? The Maven module `common/network-shuffle` does not have a log4j configuration file for its test cases. So, added `log4j.properties` in the directory `src/test/resources`. â¦shuffle] You can merge this pull request into a Git repository by running: $ git pull https://github.com/ibmsoe/spark SPARK-17543 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/15108.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #15108 commit f476755bf7653bca008d3e32375424c769ddb835 Author: Jagadeesan Date: 2016-09-15T05:56:01Z [SPARK-17543] [Missing log4j config file for tests in common/network-shuffle] --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15004: [SPARK-17440] [SPARK-17441] Fixed Multiple Bugs in ALTER...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15004 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65419/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15004: [SPARK-17440] [SPARK-17441] Fixed Multiple Bugs in ALTER...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15004 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15004: [SPARK-17440] [SPARK-17441] Fixed Multiple Bugs in ALTER...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15004 **[Test build #65419 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65419/consoleFull)** for PR 15004 at commit [`91c3a28`](https://github.com/apache/spark/commit/91c3a2815346c48190342bca474c514b67d6e6ad). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15096: [SPARK-17537] [SQL] Reading parquet schema from driver d...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/15096 Is this a duplicate of https://github.com/apache/spark/pull/14660? I think the PR tries to deal with the exact same case. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14659: [SPARK-16757] Set up Spark caller context to HDFS
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14659 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14659: [SPARK-16757] Set up Spark caller context to HDFS
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14659 **[Test build #65422 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65422/consoleFull)** for PR 14659 at commit [`4fc8c79`](https://github.com/apache/spark/commit/4fc8c799894fe265f96250e567377c1bf2702617). * This patch **fails build dependency tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14659: [SPARK-16757] Set up Spark caller context to HDFS
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14659 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65422/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15082: [SPARK-17528][SQL] MutableProjection should not cache co...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15082 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65418/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14659: [SPARK-16757] Set up Spark caller context to HDFS
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14659 **[Test build #65422 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65422/consoleFull)** for PR 14659 at commit [`4fc8c79`](https://github.com/apache/spark/commit/4fc8c799894fe265f96250e567377c1bf2702617). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15082: [SPARK-17528][SQL] MutableProjection should not cache co...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15082 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15082: [SPARK-17528][SQL] MutableProjection should not cache co...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15082 **[Test build #65418 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65418/consoleFull)** for PR 15082 at commit [`da6a285`](https://github.com/apache/spark/commit/da6a2859c8fe0c90b4aa14a68c2234e6ede36fce). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14444: [SPARK-16839] [SQL] redundant aliases after clean...
Github user eyalfa commented on a diff in the pull request:
https://github.com/apache/spark/pull/1#discussion_r78891139
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala
---
@@ -174,59 +174,32 @@ case class CreateMap(children: Seq[Expression])
extends Expression {
/**
* Returns a Row containing the evaluation of all children expressions.
*/
-@ExpressionDescription(
- usage = "_FUNC_(col1, col2, col3, ...) - Creates a struct with the given
field values.")
-case class CreateStruct(children: Seq[Expression]) extends Expression {
-
- override def foldable: Boolean = children.forall(_.foldable)
-
- override lazy val dataType: StructType = {
-val fields = children.zipWithIndex.map { case (child, idx) =>
- child match {
-case ne: NamedExpression =>
- StructField(ne.name, ne.dataType, ne.nullable, ne.metadata)
-case _ =>
- StructField(s"col${idx + 1}", child.dataType, child.nullable,
Metadata.empty)
+object CreateStruct extends (Seq[Expression] => CreateNamedStruct) {
+ def apply(children: Seq[Expression]) : CreateNamedStruct = {
+val namedStructArgs = {
+ val attNames = for {
+(child, idx) <- children.zipWithIndex
+ } yield {
+child match {
+ case ne: NamedExpression => ne.name
+ case _ => s"col${idx + 1}"
--- End diff --
the column names are used to infer the schema, these are the names used to
access the sruct's fields.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15104: [SPARK-17547] Ensure temp shuffle data file is cleaned u...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15104 cc @zsxwing --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15074: [SPARK-17520] Implement a better __eq__ for SparseMatrix
Github user gglanzani commented on the issue: https://github.com/apache/spark/pull/15074 @thunderdb thanks! In that case I'll update the DenseMatrix equality as well, as I've used it as an inspiration! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15107: [SPARK-17551][SQL] complete the NULL ordering support in...
Github user xwu0226 commented on the issue: https://github.com/apache/spark/pull/15107 @hvanhovell --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15107: [SPARK-17551][SQL] complete the NULL ordering support in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15107 **[Test build #65421 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65421/consoleFull)** for PR 15107 at commit [`69a9d8a`](https://github.com/apache/spark/commit/69a9d8aacb3832f3e346a7b9dbb13dc522e4dc68). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15107: [SPARK-17551][SQL] complete the NULL ordering sup...
GitHub user xwu0226 opened a pull request: https://github.com/apache/spark/pull/15107 [SPARK-17551][SQL] complete the NULL ordering support in DataFrame APIs ## What changes were proposed in this pull request? This PR is to complete the feature - NULLS FIRST | LAST in sorting for DataFrame/Dataset APIs. ## How was this patch tested? New DataFrame test case are added. You can merge this pull request into a Git repository by running: $ git pull https://github.com/xwu0226/spark null_order Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/15107.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #15107 commit 69a9d8aacb3832f3e346a7b9dbb13dc522e4dc68 Author: Xin Wu Date: 2016-09-15T04:12:43Z complete the NULL ordering support in DataFrame APIs --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15097: [SPARK-17540][SparkR][Spark Core] fix SparkR array serde...
Github user WeichenXu123 commented on the issue: https://github.com/apache/spark/pull/15097 @shivaram Oh...I found this way still has problems, Array[Nothing] in scala, after compiling with type erasing, at last it turned into type `Ljava.lang.object`, but primitive type Array such as Array[Int] after compile turned into `[I` so the two type still not compatible... Now I am thinking about another way, when R-side we pass an array with zero length, scala/java side regard it as `null` value, so it can be assigned to any type parameter. A typical scenario of this problem is, we want to pass an array type parameter from R-side, but sometime we only want to pass `null` array to indicate we want to use the default parameter value. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15047: [SPARK-17495] [SQL] Add Hash capability semantically equ...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/15047 Can we move this into catalyst.expressions in sql/catalyst? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15099: [SPARK-17541][SQL] fix some DDL bugs about table managem...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15099 **[Test build #65420 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65420/consoleFull)** for PR 15099 at commit [`4ab777e`](https://github.com/apache/spark/commit/4ab777ecf8e0ae51835dc7e83cf299d245f151ea). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15004: [SPARK-17440] [SPARK-17441] Fixed Multiple Bugs in ALTER...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15004 **[Test build #65419 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65419/consoleFull)** for PR 15004 at commit [`91c3a28`](https://github.com/apache/spark/commit/91c3a2815346c48190342bca474c514b67d6e6ad). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15004: [SPARK-17440] [SPARK-17441] Fixed Multiple Bugs in ALTER...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15004 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65417/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15004: [SPARK-17440] [SPARK-17441] Fixed Multiple Bugs in ALTER...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15004 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15004: [SPARK-17440] [SPARK-17441] Fixed Multiple Bugs in ALTER...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15004 **[Test build #65417 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65417/consoleFull)** for PR 15004 at commit [`159f101`](https://github.com/apache/spark/commit/159f10106db12c8d47bb8b8ec26118dec286a4e0). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15082: [SPARK-17528][SQL] MutableProjection should not cache co...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15082 **[Test build #65418 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65418/consoleFull)** for PR 15082 at commit [`da6a285`](https://github.com/apache/spark/commit/da6a2859c8fe0c90b4aa14a68c2234e6ede36fce). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15001: [SPARK-17438][WebUI] Show Application.executorLimit in t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15001 **[Test build #3266 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3266/consoleFull)** for PR 15001 at commit [`b41aa04`](https://github.com/apache/spark/commit/b41aa04f941e39ff941be3bbdd2860cd207ff6ea). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14834: [SPARK-17163][ML] Unified LogisticRegression interface
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14834 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14834: [SPARK-17163][ML] Unified LogisticRegression interface
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14834 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65416/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14834: [SPARK-17163][ML] Unified LogisticRegression interface
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14834 **[Test build #65416 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65416/consoleFull)** for PR 14834 at commit [`292f481`](https://github.com/apache/spark/commit/292f48153262875f481aa4713ae0521ecca9c78f). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15073: [SPARK-17518] [SQL] Block Users to Specify the Internal ...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15073 cc @cloud-fan --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15054: [SPARK-17502] [SQL] Fix Multiple Bugs in DDL Statements ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15054 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65413/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15054: [SPARK-17502] [SQL] Fix Multiple Bugs in DDL Statements ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15054 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15013: [SPARK-17451] [CORE] CoarseGrainedExecutorBackend should...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15013 **[Test build #3265 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3265/consoleFull)** for PR 15013 at commit [`71fa2e3`](https://github.com/apache/spark/commit/71fa2e32bba298920f974c652b24983306068b09). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15054: [SPARK-17502] [SQL] Fix Multiple Bugs in DDL Statements ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15054 **[Test build #65413 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65413/consoleFull)** for PR 15054 at commit [`7c1a8e5`](https://github.com/apache/spark/commit/7c1a8e53dd1e3b36bf0f20d4ba37a6f2b71cbb2f). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15004: [SPARK-17440] [SPARK-17441] Fixed Multiple Bugs in ALTER...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15004 **[Test build #65417 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65417/consoleFull)** for PR 15004 at commit [`159f101`](https://github.com/apache/spark/commit/159f10106db12c8d47bb8b8ec26118dec286a4e0). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15085: [SPARK-17484] Prevent invalid block locations from being...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15085 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65410/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15085: [SPARK-17484] Prevent invalid block locations from being...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15085 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15085: [SPARK-17484] Prevent invalid block locations from being...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15085 **[Test build #65410 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65410/consoleFull)** for PR 15085 at commit [`47f9636`](https://github.com/apache/spark/commit/47f9636b3f98cf03cee907b7d9b41364701f2a98). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15106: [SPARK-16439] [SQL] bring back the separator in SQL UI
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15106 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15106: [SPARK-16439] [SQL] bring back the separator in SQL UI
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15106 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65412/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15106: [SPARK-16439] [SQL] bring back the separator in SQL UI
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15106 **[Test build #65412 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65412/consoleFull)** for PR 15106 at commit [`11a989e`](https://github.com/apache/spark/commit/11a989eacf26ebf0ae3710fa23ef3a63dc618b51). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15043: [SPARK-17491] Close serialization stream to fix wrong an...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15043 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65407/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15043: [SPARK-17491] Close serialization stream to fix wrong an...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15043 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15043: [SPARK-17491] Close serialization stream to fix wrong an...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15043 **[Test build #65407 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65407/consoleFull)** for PR 15043 at commit [`daf447b`](https://github.com/apache/spark/commit/daf447bdd612289f014d199007ed84049d79a009). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15006: [SPARK-17364][SQL] Antlr lexer wrongly treats full quali...
Github user dilipbiswal commented on the issue: https://github.com/apache/spark/pull/15006 Thank you @clockfly @hvanhovell @gatorsmile --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15102 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65405/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15102 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15102: [SPARK-17346][SQL] Add Kafka source for Structured Strea...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15102 **[Test build #65405 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65405/consoleFull)** for PR 15102 at commit [`d3c0754`](https://github.com/apache/spark/commit/d3c0754c9b5c9aa2c29b0d3ca0b28ddae041b054). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15001: [SPARK-17438][WebUI] Show Application.executorLimit in t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15001 **[Test build #3266 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3266/consoleFull)** for PR 15001 at commit [`b41aa04`](https://github.com/apache/spark/commit/b41aa04f941e39ff941be3bbdd2860cd207ff6ea). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #11105: [SPARK-12469][CORE] Data Property accumulators for Spark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/11105 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65415/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #11105: [SPARK-12469][CORE] Data Property accumulators for Spark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/11105 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #11105: [SPARK-12469][CORE] Data Property accumulators for Spark
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11105 **[Test build #65415 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65415/consoleFull)** for PR 11105 at commit [`04814a3`](https://github.com/apache/spark/commit/04814a3b37f6b1d41331026d1a3ff63c4c66de1c). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14834: [SPARK-17163][ML] Unified LogisticRegression interface
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14834 **[Test build #65416 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65416/consoleFull)** for PR 14834 at commit [`292f481`](https://github.com/apache/spark/commit/292f48153262875f481aa4713ae0521ecca9c78f). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14834: [SPARK-17163][ML] Unified LogisticRegression interface
Github user sethah commented on the issue: https://github.com/apache/spark/pull/14834 jenkins test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15001: [SPARK-17438][WebUI] Show Application.executorLimit in t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15001 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65408/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15001: [SPARK-17438][WebUI] Show Application.executorLimit in t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15001 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15001: [SPARK-17438][WebUI] Show Application.executorLimit in t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15001 **[Test build #65408 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65408/consoleFull)** for PR 15001 at commit [`b41aa04`](https://github.com/apache/spark/commit/b41aa04f941e39ff941be3bbdd2860cd207ff6ea). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15065: [SPARK-17463][Core]Add memory barriers for accumulators
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/15065 I'm going to close this one for now since I found there was a huge regression for `MiscBenchmark."filter & aggregate without group"`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15065: [SPARK-17463][Core]Add memory barriers for accumu...
Github user zsxwing closed the pull request at: https://github.com/apache/spark/pull/15065 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #11105: [SPARK-12469][CORE] Data Property accumulators for Spark
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/11105 **[Test build #65415 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65415/consoleFull)** for PR 11105 at commit [`04814a3`](https://github.com/apache/spark/commit/04814a3b37f6b1d41331026d1a3ff63c4c66de1c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15105: [SPARK-17548] [MLlib] Word2VecModel.findSynonyms no long...
Github user willb commented on the issue: https://github.com/apache/spark/pull/15105 Thanks, @hhbyyh. This is what I get for running `sbt "testOnly ..."` I'll push a fix. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14834: [SPARK-17163][ML] Unified LogisticRegression interface
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14834 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14834: [SPARK-17163][ML] Unified LogisticRegression interface
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14834 **[Test build #65414 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65414/consoleFull)** for PR 14834 at commit [`292f481`](https://github.com/apache/spark/commit/292f48153262875f481aa4713ae0521ecca9c78f). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14834: [SPARK-17163][ML] Unified LogisticRegression interface
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14834 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65414/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15101: [SPARK-17114][SQL] Fix aggregates grouped by literals wi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15101 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65406/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
