[GitHub] spark pull request: [SPARK-9223] [PySpark] [MLlib] Support model s...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7587#issuecomment-123582701 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9199][CORE] Update Tachyon dependency f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7577#issuecomment-123582713 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9223] [PySpark] [MLlib] Support model s...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7587#issuecomment-123582688 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9199][CORE] Update Tachyon dependency f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7577#issuecomment-123582693 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8176] [SPARK-8197] [SQL] function to_da...
Github user adrian-wang commented on a diff in the pull request:
https://github.com/apache/spark/pull/6988#discussion_r35186005
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala
---
@@ -258,3 +258,70 @@ case class DateFormatClass(left: Expression, right:
Expression) extends BinaryEx
})
}
}
+
+/**
+ * Returns the date part of a timestamp string.
+ */
+case class ToDate(child: Expression) extends UnaryExpression with
ImplicitCastInputTypes {
+
+ override def inputTypes: Seq[AbstractDataType] = Seq(DateType)
--- End diff --
int and long could not be cast into datetype.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9199][CORE] Update Tachyon dependency f...
Github user haoyuan commented on the pull request: https://github.com/apache/spark/pull/7577#issuecomment-123582588 Jenkins, ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8176] [SPARK-8197] [SQL] function to_da...
Github user adrian-wang commented on a diff in the pull request:
https://github.com/apache/spark/pull/6988#discussion_r35185971
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DateExpressionsSuite.scala
---
@@ -246,4 +247,45 @@ class DateExpressionsSuite extends SparkFunSuite with
ExpressionEvalHelper {
}
}
+ test("datetime function current_date") {
+val d0 = DateTimeUtils.millisToDays(System.currentTimeMillis())
+val cd = CurrentDate().eval(EmptyRow).asInstanceOf[Int]
+val d1 = DateTimeUtils.millisToDays(System.currentTimeMillis())
+assert(d0 <= cd && cd <= d1 && d1 - d0 <= 1)
--- End diff --
why? it should be.
anyway, this is just a movement.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8176] [SPARK-8197] [SQL] function to_da...
Github user adrian-wang commented on a diff in the pull request:
https://github.com/apache/spark/pull/6988#discussion_r35185944
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala
---
@@ -258,3 +258,70 @@ case class DateFormatClass(left: Expression, right:
Expression) extends BinaryEx
})
}
}
+
+/**
+ * Returns the date part of a timestamp string.
+ */
+case class ToDate(child: Expression) extends UnaryExpression with
ImplicitCastInputTypes {
+
+ override def inputTypes: Seq[AbstractDataType] = Seq(DateType)
+
+ override def dataType: DataType = DateType
+
+ override def eval(input: InternalRow): Any = {
+child.eval(input).asInstanceOf[Int]
+ }
+
+ override def genCode(ctx: CodeGenContext, ev: GeneratedExpressionCode):
String = {
+nullSafeCodeGen(ctx, ev, (time) => {
+ time
+})
+ }
+}
+
+/**
+ * Returns date truncated to the unit specified by the format.
+ */
+case class Trunc(date: Expression, format: Expression)
+ extends BinaryExpression with ImplicitCastInputTypes {
+ override def left: Expression = date
+ override def right: Expression = format
+
+ override def inputTypes: Seq[AbstractDataType] = Seq(DateType,
StringType)
+ override def dataType: DataType = DateType
+
+ override def nullSafeEval(d: Any, fmt: Any): Any = {
+val minItem =
DateTimeUtils.getFmt(fmt.asInstanceOf[UTF8String].toUpperCase.toString)
+if (minItem == -1) {
+ // unknown format
+ null
+} else {
+ val days = d.asInstanceOf[Int]
+ val year = DateTimeUtils.getYear(days)
+ if (minItem == Calendar.YEAR) {
+days - DateTimeUtils.getDayInYear(days) + 1
+ } else {
+// trunc to MONTH
+days - DateTimeUtils.getDayOfMonth(days) + 1
+ }
+}
+ }
+
+ override def genCode(ctx: CodeGenContext, ev: GeneratedExpressionCode):
String = {
+nullSafeCodeGen(ctx, ev, (dateVal, fmt) => {
+ val dtu = DateTimeUtils.getClass.getName.stripSuffix("$")
--- End diff --
use the full name is ok.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8176] [SPARK-8197] [SQL] function to_da...
Github user adrian-wang commented on a diff in the pull request:
https://github.com/apache/spark/pull/6988#discussion_r35185903
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -2262,6 +2246,22 @@ object functions {
//
/**
+ * Returns the current date.
+ *
+ * @group datetime_funcs
+ * @since 1.5.0
+ */
+ def current_date(): Column = CurrentDate()
+
+ /**
+ * Returns the current timestamp.
+ *
+ * @group datetime_funcs
+ * @since 1.5.0
+ */
+ def current_timestamp(): Column = CurrentTimestamp()
--- End diff --
This is not like the case of e/pi, there were some considerations in the
previous pr for this.
Anyway, this is just a movement.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4366] [SQL] [Follow-up] Fix SqlParser c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7588#issuecomment-123581989 [Test build #38047 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38047/console) for PR 7588 at commit [`25f5f36`](https://github.com/apache/spark/commit/25f5f3665cf7fd1641600e57e0821558fa4b2746). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4366] [SQL] [Follow-up] Fix SqlParser c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7588#issuecomment-123581924 [Test build #38047 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38047/consoleFull) for PR 7588 at commit [`25f5f36`](https://github.com/apache/spark/commit/25f5f3665cf7fd1641600e57e0821558fa4b2746). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4366] [SQL] [Follow-up] Fix SqlParser c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7588#issuecomment-123581994 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8231][SQL] Add array_contains
Github user chenghao-intel commented on the pull request: https://github.com/apache/spark/pull/7580#issuecomment-123581993 @EntilZha can you also add the python DF API support? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8231][SQL] Add array_contains
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/7580#discussion_r35185849
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -2081,6 +2081,22 @@ object functions {
*/
def size(column: Column): Column = Size(column.expr)
+ /**
+ * Returns true if the array contain the value
+ * @group collection_funcs
+ * @since 1.5.0
+ */
+ def array_contains(columnName: String, value: Any): Column =
--- End diff --
Let's remove this function, as we are now only trying to add `column`
version of the function for DF.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9223] [PySpark] [MLlib] Support model s...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7587#issuecomment-123581518 [Test build #38046 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38046/console) for PR 7587 at commit [`a8e70d9`](https://github.com/apache/spark/commit/a8e70d999a2f05aeb8a85d9af5181ee39a890db5). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9223] [PySpark] [MLlib] Support model s...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7587#issuecomment-123581526 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8231][SQL] Add array_contains
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/7580#discussion_r35185806
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala ---
@@ -298,4 +298,32 @@ class DataFrameFunctionsSuite extends QueryTest {
Seq(Row(2), Row(0), Row(3))
)
}
+
+ test("array contains function") {
+val df = Seq(
+ (Seq[Int](1, 2), "x"),
+ (Seq[Int](), "y"),
+ (null, "z")
+).toDF("a", "b")
+checkAnswer(
+ df.select(array_contains("a", 1)),
+ Seq(Row(true), Row(false), Row(false))
+)
+checkAnswer(
+ df.selectExpr("array_contains(a, 1)"),
+ Seq(Row(true), Row(false), Row(false))
+)
+//checkAnswer(
--- End diff --
remove the unused code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4366] [SQL] [Follow-up] Fix SqlParser c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7588#issuecomment-123581379 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4366] [SQL] [Follow-up] Fix SqlParser c...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7588#issuecomment-123581336 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4366] [SQL] [Follow-up] Fix SqlParser c...
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/7588#discussion_r35185783
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala ---
@@ -271,8 +271,7 @@ class SqlParser extends AbstractSparkSQLParser with
DataTypeParser {
lexical.normalizeKeyword(udfName) match {
case "sum" => SumDistinct(exprs.head)
case "count" => CountDistinct(exprs)
-case name => UnresolvedFunction(name, exprs, isDistinct = true)
-case _ => throw new AnalysisException(s"function $udfName does not
support DISTINCT")
+case _ => UnresolvedFunction(udfName, exprs, isDistinct = true)
--- End diff --
Since we can register a UDAF and use DISTINCT with it, we can just create
an `UnresolvedFunction` at here. If it is actually a built-in aggregate
function and it does not support DISTINCT, we will throw an error later.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9223] [PySpark] [MLlib] Support model s...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7587#issuecomment-123580994 [Test build #38046 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38046/consoleFull) for PR 7587 at commit [`a8e70d9`](https://github.com/apache/spark/commit/a8e70d999a2f05aeb8a85d9af5181ee39a890db5). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8231][SQL] Add array_contains
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/7580#discussion_r35185755
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
---
@@ -35,3 +37,57 @@ case class Size(child: Expression) extends
UnaryExpression with ExpectsInputType
nullSafeCodeGen(ctx, ev, c => s"${ev.primitive} = ($c).size();")
}
}
+
+case class ArrayContains(left: Expression, right: Expression) extends
BinaryExpression {
+ override def dataType: DataType = BooleanType
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+if (!left.dataType.isInstanceOf[ArrayType]) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of first input must be an array, not
${left.dataType.simpleString}")
+} else if (left.dataType.asInstanceOf[ArrayType].elementType !=
right.dataType) {
+ TypeCheckResult.TypeCheckFailure(
+s"type of value must match array type " +
+
s"${left.dataType.asInstanceOf[ArrayType].elementType.simpleString}, not "+
+ s"${right.dataType.simpleString}")
+} else {
+ TypeCheckResult.TypeCheckSuccess
+}
+ }
+
+ override def eval(input: InternalRow): Boolean = {
--- End diff --
Nit: override the `nullSafeEval` instead, and we can save some code in null
value checking.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4366] [SQL] [Follow-up] Fix SqlParser c...
GitHub user yhuai opened a pull request: https://github.com/apache/spark/pull/7588 [SPARK-4366] [SQL] [Follow-up] Fix SqlParser compiling warning. You can merge this pull request into a Git repository by running: $ git pull https://github.com/yhuai/spark SPARK-4366-update1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/7588.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #7588 commit 25f5f3665cf7fd1641600e57e0821558fa4b2746 Author: Yin Huai Date: 2015-07-22T06:48:27Z Fix SqlParser Warning. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9223] [PySpark] [MLlib] Support model s...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7587#issuecomment-123579966 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9238][SQL]two extra useless entries for...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7582#issuecomment-123580082 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9238][SQL]two extra useless entries for...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7582#issuecomment-123579960 [Test build #38026 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38026/console) for PR 7582 at commit [`8bddd01`](https://github.com/apache/spark/commit/8bddd0143cff2f24e17a5b3ed53103f6fd59e4fb). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8176] [SPARK-8197] [SQL] function to_da...
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/6988#discussion_r35185572
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala
---
@@ -258,3 +258,70 @@ case class DateFormatClass(left: Expression, right:
Expression) extends BinaryEx
})
}
}
+
+/**
+ * Returns the date part of a timestamp string.
+ */
+case class ToDate(child: Expression) extends UnaryExpression with
ImplicitCastInputTypes {
+
+ override def inputTypes: Seq[AbstractDataType] = Seq(DateType)
+
+ override def dataType: DataType = DateType
+
+ override def eval(input: InternalRow): Any = {
--- End diff --
ok, sounds reasonable to me.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4233] [SPARK-4367] [SPARK-3947] [SPARK-...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7458#issuecomment-123579945 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9223] [PySpark] [MLlib] Support model s...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7587#issuecomment-123579989 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4233] [SPARK-4367] [SPARK-3947] [SPARK-...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7458#issuecomment-123579811 [Test build #38030 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38030/console) for PR 7458 at commit [`7865f5e`](https://github.com/apache/spark/commit/7865f5ef072d1877b73276f28e84afa98c247a82). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class UnresolvedFunction(` * `case class Average(child: Expression) extends AlgebraicAggregate ` * `case class Count(child: Expression) extends AlgebraicAggregate ` * `case class First(child: Expression) extends AlgebraicAggregate ` * `case class Last(child: Expression) extends AlgebraicAggregate ` * `case class Max(child: Expression) extends AlgebraicAggregate ` * `case class Min(child: Expression) extends AlgebraicAggregate ` * `case class Sum(child: Expression) extends AlgebraicAggregate ` * `abstract class AlgebraicAggregate extends AggregateFunction2 with Serializable ` * ` implicit class RichAttribute(a: AttributeReference) ` * `trait AggregateExpression1 extends AggregateExpression ` * `trait PartialAggregate1 extends AggregateExpression1 ` * `case class Min(child: Expression) extends UnaryExpression with PartialAggregate1 ` * `case class MinFunction(expr: Expression, base: AggregateExpression1) extends AggregateFunction1 ` * `case class Max(child: Expression) extends UnaryExpression with PartialAggregate1 ` * `case class MaxFunction(expr: Expression, base: AggregateExpression1) extends AggregateFunction1 ` * `case class Count(child: Expression) extends UnaryExpression with PartialAggregate1 ` * `case class CountFunction(expr: Expression, base: AggregateExpression1) extends AggregateFunction1 ` * `case class CountDistinct(expressions: Seq[Expression]) extends PartialAggregate1 ` * `case class CollectHashSet(expressions: Seq[Expression]) extends AggregateExpression1 ` * `case class CombineSetsAndCount(inputSet: Expression) extends AggregateExpression1 ` * `case class Average(child: Expression) extends UnaryExpression with PartialAggregate1 ` * `case class AverageFunction(expr: Expression, base: AggregateExpression1)` * `case class Sum(child: Expression) extends UnaryExpression with PartialAggregate1 ` * `case class SumFunction(expr: Expression, base: AggregateExpression1) extends AggregateFunction1 ` * `case class CombineSum(child: Expression) extends AggregateExpression1 ` * `case class CombineSumFunction(expr: Expression, base: AggregateExpression1)` * `case class SumDistinct(child: Expression) extends UnaryExpression with PartialAggregate1 ` * `case class SumDistinctFunction(expr: Expression, base: AggregateExpression1)` * `case class CombineSetsAndSum(inputSet: Expression, base: Expression) extends AggregateExpression1 ` * `case class First(child: Expression) extends UnaryExpression with PartialAggregate1 ` * `case class FirstFunction(expr: Expression, base: AggregateExpression1) extends AggregateFunction1 ` * `case class Last(child: Expression) extends UnaryExpression with PartialAggregate1 ` * `case class LastFunction(expr: Expression, base: AggregateExpression1) extends AggregateFunction1 ` * `case class Aggregate2Sort(` * `case class FinalAndCompleteAggregate2Sort(` * `class GroupingIterator(` * `class PartialSortAggregationIterator(` * `class PartialMergeSortAggregationIterator(` * `class FinalSortAggregationIterator(` * `class FinalAndCompleteSortAggregationIterator(` * `abstract class UserDefinedAggregateFunction extends Serializable ` * `case class ScalaUDAF(` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9223] [PySpark] [MLlib] Support model s...
GitHub user MechCoder opened a pull request: https://github.com/apache/spark/pull/7587 [SPARK-9223] [PySpark] [MLlib] Support model save/load in LDA Since save / load has been merged in LDA, it takes no time to write the wrappers in Python as well. You can merge this pull request into a Git repository by running: $ git pull https://github.com/MechCoder/spark python_lda_save_load Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/7587.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #7587 commit a8e70d999a2f05aeb8a85d9af5181ee39a890db5 Author: MechCoder Date: 2015-07-22T06:44:14Z [SPARK-9223] [PySpark] Support model save/load in LDA --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8176] [SPARK-8197] [SQL] function to_da...
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/6988#discussion_r35185504
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala
---
@@ -258,3 +258,70 @@ case class DateFormatClass(left: Expression, right:
Expression) extends BinaryEx
})
}
}
+
+/**
+ * Returns the date part of a timestamp string.
+ */
+case class ToDate(child: Expression) extends UnaryExpression with
ImplicitCastInputTypes {
+
+ override def inputTypes: Seq[AbstractDataType] = Seq(DateType)
--- End diff --
I mean we don't accept the `IntegerType`, `LongType`, do we? Hence we use
the `TypeCollection`, and we'd better inherits from the `ExpectInputTypes`,
instead of the `ImplicitCastInputTypes`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8405][Doc] Add how to view logs on Web ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7463#issuecomment-123579283 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8405][Doc] Add how to view logs on Web ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7463#issuecomment-123579179 [Test build #38044 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38044/console) for PR 7463 at commit [`74df3a1`](https://github.com/apache/spark/commit/74df3a120e7452be3c1ef00920326b7b7cd8cc0e). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7584#issuecomment-123578411 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7584#issuecomment-123578856 [Test build #38045 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38045/console) for PR 7584 at commit [`de3e0db`](https://github.com/apache/spark/commit/de3e0dbfa52e3676580f0f730d02d961dded063f). * This patch **fails to build**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class UnresolvedFunction(` * `case class Average(child: Expression) extends AlgebraicAggregate ` * `case class Count(child: Expression) extends AlgebraicAggregate ` * `case class First(child: Expression) extends AlgebraicAggregate ` * `case class Last(child: Expression) extends AlgebraicAggregate ` * `case class Max(child: Expression) extends AlgebraicAggregate ` * `case class Min(child: Expression) extends AlgebraicAggregate ` * `case class Sum(child: Expression) extends AlgebraicAggregate ` * `abstract class AlgebraicAggregate extends AggregateFunction2 with Serializable ` * ` implicit class RichAttribute(a: AttributeReference) ` * `trait AggregateExpression1 extends AggregateExpression ` * `trait PartialAggregate1 extends AggregateExpression1 ` * `case class Min(child: Expression) extends UnaryExpression with PartialAggregate1 ` * `case class MinFunction(expr: Expression, base: AggregateExpression1) extends AggregateFunction1 ` * `case class Max(child: Expression) extends UnaryExpression with PartialAggregate1 ` * `case class MaxFunction(expr: Expression, base: AggregateExpression1) extends AggregateFunction1 ` * `case class Count(child: Expression) extends UnaryExpression with PartialAggregate1 ` * `case class CountFunction(expr: Expression, base: AggregateExpression1) extends AggregateFunction1 ` * `case class CountDistinct(expressions: Seq[Expression]) extends PartialAggregate1 ` * `case class CollectHashSet(expressions: Seq[Expression]) extends AggregateExpression1 ` * `case class CombineSetsAndCount(inputSet: Expression) extends AggregateExpression1 ` * `case class Average(child: Expression) extends UnaryExpression with PartialAggregate1 ` * `case class AverageFunction(expr: Expression, base: AggregateExpression1)` * `case class Sum(child: Expression) extends UnaryExpression with PartialAggregate1 ` * `case class SumFunction(expr: Expression, base: AggregateExpression1) extends AggregateFunction1 ` * `case class CombineSum(child: Expression) extends AggregateExpression1 ` * `case class CombineSumFunction(expr: Expression, base: AggregateExpression1)` * `case class SumDistinct(child: Expression) extends UnaryExpression with PartialAggregate1 ` * `case class SumDistinctFunction(expr: Expression, base: AggregateExpression1)` * `case class CombineSetsAndSum(inputSet: Expression, base: Expression) extends AggregateExpression1 ` * `case class First(child: Expression) extends UnaryExpression with PartialAggregate1 ` * `case class FirstFunction(expr: Expression, base: AggregateExpression1) extends AggregateFunction1 ` * `case class Last(child: Expression) extends UnaryExpression with PartialAggregate1 ` * `case class LastFunction(expr: Expression, base: AggregateExpression1) extends AggregateFunction1 ` * `case class Aggregate2Sort(` * `case class FinalAndCompleteAggregate2Sort(` * `class GroupingIterator(` * `class PartialSortAggregationIterator(` * `class PartialMergeSortAggregationIterator(` * `class FinalSortAggregationIterator(` * `class FinalAndCompleteSortAggregationIterator(` * `abstract class UserDefinedAggregateFunction extends Serializable ` * `case class ScalaUDAF(` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7584#issuecomment-123578795 [Test build #38045 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38045/consoleFull) for PR 7584 at commit [`de3e0db`](https://github.com/apache/spark/commit/de3e0dbfa52e3676580f0f730d02d961dded063f). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7584#issuecomment-123578864 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4233] [SPARK-4367] [SPARK-3947] [SPARK-...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7458#issuecomment-123578831 [Test build #1152 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1152/console) for PR 7458 at commit [`7865f5e`](https://github.com/apache/spark/commit/7865f5ef072d1877b73276f28e84afa98c247a82). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class UnresolvedFunction(` * `case class Average(child: Expression) extends AlgebraicAggregate ` * `case class Count(child: Expression) extends AlgebraicAggregate ` * `case class First(child: Expression) extends AlgebraicAggregate ` * `case class Last(child: Expression) extends AlgebraicAggregate ` * `case class Max(child: Expression) extends AlgebraicAggregate ` * `case class Min(child: Expression) extends AlgebraicAggregate ` * `case class Sum(child: Expression) extends AlgebraicAggregate ` * `abstract class AlgebraicAggregate extends AggregateFunction2 with Serializable ` * ` implicit class RichAttribute(a: AttributeReference) ` * `trait AggregateExpression1 extends AggregateExpression ` * `trait PartialAggregate1 extends AggregateExpression1 ` * `case class Min(child: Expression) extends UnaryExpression with PartialAggregate1 ` * `case class MinFunction(expr: Expression, base: AggregateExpression1) extends AggregateFunction1 ` * `case class Max(child: Expression) extends UnaryExpression with PartialAggregate1 ` * `case class MaxFunction(expr: Expression, base: AggregateExpression1) extends AggregateFunction1 ` * `case class Count(child: Expression) extends UnaryExpression with PartialAggregate1 ` * `case class CountFunction(expr: Expression, base: AggregateExpression1) extends AggregateFunction1 ` * `case class CountDistinct(expressions: Seq[Expression]) extends PartialAggregate1 ` * `case class CollectHashSet(expressions: Seq[Expression]) extends AggregateExpression1 ` * `case class CombineSetsAndCount(inputSet: Expression) extends AggregateExpression1 ` * `case class Average(child: Expression) extends UnaryExpression with PartialAggregate1 ` * `case class AverageFunction(expr: Expression, base: AggregateExpression1)` * `case class Sum(child: Expression) extends UnaryExpression with PartialAggregate1 ` * `case class SumFunction(expr: Expression, base: AggregateExpression1) extends AggregateFunction1 ` * `case class CombineSum(child: Expression) extends AggregateExpression1 ` * `case class CombineSumFunction(expr: Expression, base: AggregateExpression1)` * `case class SumDistinct(child: Expression) extends UnaryExpression with PartialAggregate1 ` * `case class SumDistinctFunction(expr: Expression, base: AggregateExpression1)` * `case class CombineSetsAndSum(inputSet: Expression, base: Expression) extends AggregateExpression1 ` * `case class First(child: Expression) extends UnaryExpression with PartialAggregate1 ` * `case class FirstFunction(expr: Expression, base: AggregateExpression1) extends AggregateFunction1 ` * `case class Last(child: Expression) extends UnaryExpression with PartialAggregate1 ` * `case class LastFunction(expr: Expression, base: AggregateExpression1) extends AggregateFunction1 ` * `case class FormatString(children: Expression*) extends Expression with ImplicitCastInputTypes ` * `case class Aggregate2Sort(` * `case class FinalAndCompleteAggregate2Sort(` * `class GroupingIterator(` * `class PartialSortAggregationIterator(` * `class PartialMergeSortAggregationIterator(` * `class FinalSortAggregationIterator(` * `class FinalAndCompleteSortAggregationIterator(` * `abstract class UserDefinedAggregateFunction extends Serializable ` * `case class ScalaUDAF(` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7584#issuecomment-123578250 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8176] [SPARK-8197] [SQL] function to_da...
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/6988#discussion_r35185346
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -2262,6 +2246,22 @@ object functions {
//
/**
+ * Returns the current date.
+ *
+ * @group datetime_funcs
+ * @since 1.5.0
+ */
+ def current_date(): Column = CurrentDate()
--- End diff --
Probably we don't want to this as the DF API.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8176] [SPARK-8197] [SQL] function to_da...
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/6988#discussion_r35185354
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -2262,6 +2246,22 @@ object functions {
//
/**
+ * Returns the current date.
+ *
+ * @group datetime_funcs
+ * @since 1.5.0
+ */
+ def current_date(): Column = CurrentDate()
+
+ /**
+ * Returns the current timestamp.
+ *
+ * @group datetime_funcs
+ * @since 1.5.0
+ */
+ def current_timestamp(): Column = CurrentTimestamp()
--- End diff --
Probably we don't want to this as the DF API.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7584#issuecomment-123578208 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7584#issuecomment-123578076 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7584#issuecomment-123578065 [Test build #51 has started](https://amplab.cs.berkeley.edu/jenkins/job/SlowSparkPullRequestBuilder/51/consoleFull) for PR 7584 at commit [`de3e0db`](https://github.com/apache/spark/commit/de3e0dbfa52e3676580f0f730d02d961dded063f). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7584#issuecomment-123578074 [Test build #51 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SlowSparkPullRequestBuilder/51/console) for PR 7584 at commit [`de3e0db`](https://github.com/apache/spark/commit/de3e0dbfa52e3676580f0f730d02d961dded063f). * This patch **fails to build**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class UnresolvedFunction(` * `case class Average(child: Expression) extends AlgebraicAggregate ` * `case class Count(child: Expression) extends AlgebraicAggregate ` * `case class First(child: Expression) extends AlgebraicAggregate ` * `case class Last(child: Expression) extends AlgebraicAggregate ` * `case class Max(child: Expression) extends AlgebraicAggregate ` * `case class Min(child: Expression) extends AlgebraicAggregate ` * `case class Sum(child: Expression) extends AlgebraicAggregate ` * `abstract class AlgebraicAggregate extends AggregateFunction2 with Serializable ` * ` implicit class RichAttribute(a: AttributeReference) ` * `trait AggregateExpression1 extends AggregateExpression ` * `trait PartialAggregate1 extends AggregateExpression1 ` * `case class Min(child: Expression) extends UnaryExpression with PartialAggregate1 ` * `case class MinFunction(expr: Expression, base: AggregateExpression1) extends AggregateFunction1 ` * `case class Max(child: Expression) extends UnaryExpression with PartialAggregate1 ` * `case class MaxFunction(expr: Expression, base: AggregateExpression1) extends AggregateFunction1 ` * `case class Count(child: Expression) extends UnaryExpression with PartialAggregate1 ` * `case class CountFunction(expr: Expression, base: AggregateExpression1) extends AggregateFunction1 ` * `case class CountDistinct(expressions: Seq[Expression]) extends PartialAggregate1 ` * `case class CollectHashSet(expressions: Seq[Expression]) extends AggregateExpression1 ` * `case class CombineSetsAndCount(inputSet: Expression) extends AggregateExpression1 ` * `case class Average(child: Expression) extends UnaryExpression with PartialAggregate1 ` * `case class AverageFunction(expr: Expression, base: AggregateExpression1)` * `case class Sum(child: Expression) extends UnaryExpression with PartialAggregate1 ` * `case class SumFunction(expr: Expression, base: AggregateExpression1) extends AggregateFunction1 ` * `case class CombineSum(child: Expression) extends AggregateExpression1 ` * `case class CombineSumFunction(expr: Expression, base: AggregateExpression1)` * `case class SumDistinct(child: Expression) extends UnaryExpression with PartialAggregate1 ` * `case class SumDistinctFunction(expr: Expression, base: AggregateExpression1)` * `case class CombineSetsAndSum(inputSet: Expression, base: Expression) extends AggregateExpression1 ` * `case class First(child: Expression) extends UnaryExpression with PartialAggregate1 ` * `case class FirstFunction(expr: Expression, base: AggregateExpression1) extends AggregateFunction1 ` * `case class Last(child: Expression) extends UnaryExpression with PartialAggregate1 ` * `case class LastFunction(expr: Expression, base: AggregateExpression1) extends AggregateFunction1 ` * `case class Aggregate2Sort(` * `case class FinalAndCompleteAggregate2Sort(` * `class GroupingIterator(` * `class PartialSortAggregationIterator(` * `class PartialMergeSortAggregationIterator(` * `class FinalSortAggregationIterator(` * `class FinalAndCompleteSortAggregationIterator(` * `abstract class UserDefinedAggregateFunction extends Serializable ` * `case class ScalaUDAF(` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7584#issuecomment-123578009 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...
Github user yu-iskw commented on the pull request: https://github.com/apache/spark/pull/7584#issuecomment-123578035 @shivaram sure. I'll add those new issues on JIRA. Thank you for your continuous support! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8176] [SPARK-8197] [SQL] function to_da...
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/6988#discussion_r35185304
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DateExpressionsSuite.scala
---
@@ -246,4 +247,45 @@ class DateExpressionsSuite extends SparkFunSuite with
ExpressionEvalHelper {
}
}
+ test("datetime function current_date") {
+val d0 = DateTimeUtils.millisToDays(System.currentTimeMillis())
+val cd = CurrentDate().eval(EmptyRow).asInstanceOf[Int]
+val d1 = DateTimeUtils.millisToDays(System.currentTimeMillis())
+assert(d0 <= cd && cd <= d1 && d1 - d0 <= 1)
--- End diff --
Can we remove the ` && d1 - d0 <= 1`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7584#issuecomment-123578029 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9024] Unsafe HashJoin/HashOuterJoin/Has...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7480#issuecomment-123577946 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9024] Unsafe HashJoin/HashOuterJoin/Has...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7480#issuecomment-123577927 [Test build #38028 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38028/console) for PR 7480 at commit [`6294b1e`](https://github.com/apache/spark/commit/6294b1e3de357c94646c323eba2d4bde80971c45). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/7584#issuecomment-123577833 @yu-iskw The lint output is great ! Getting pretty close to zero warnings. BTW could you create a new JIRA for fixing `Closing curly-braces should always be on their own line` and `local variable assigned but may not be used` ? I'll review this PR tomorrow --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8176] [SPARK-8197] [SQL] function to_da...
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/6988#discussion_r35185223
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala
---
@@ -258,3 +258,70 @@ case class DateFormatClass(left: Expression, right:
Expression) extends BinaryEx
})
}
}
+
+/**
+ * Returns the date part of a timestamp string.
+ */
+case class ToDate(child: Expression) extends UnaryExpression with
ImplicitCastInputTypes {
+
+ override def inputTypes: Seq[AbstractDataType] = Seq(DateType)
+
+ override def dataType: DataType = DateType
+
+ override def eval(input: InternalRow): Any = {
+child.eval(input).asInstanceOf[Int]
+ }
+
+ override def genCode(ctx: CodeGenContext, ev: GeneratedExpressionCode):
String = {
+nullSafeCodeGen(ctx, ev, (time) => {
+ time
+})
+ }
+}
+
+/**
+ * Returns date truncated to the unit specified by the format.
+ */
+case class Trunc(date: Expression, format: Expression)
+ extends BinaryExpression with ImplicitCastInputTypes {
+ override def left: Expression = date
+ override def right: Expression = format
+
+ override def inputTypes: Seq[AbstractDataType] = Seq(DateType,
StringType)
+ override def dataType: DataType = DateType
+
+ override def nullSafeEval(d: Any, fmt: Any): Any = {
+val minItem =
DateTimeUtils.getFmt(fmt.asInstanceOf[UTF8String].toUpperCase.toString)
+if (minItem == -1) {
+ // unknown format
+ null
+} else {
+ val days = d.asInstanceOf[Int]
+ val year = DateTimeUtils.getYear(days)
+ if (minItem == Calendar.YEAR) {
+days - DateTimeUtils.getDayInYear(days) + 1
+ } else {
+// trunc to MONTH
+days - DateTimeUtils.getDayOfMonth(days) + 1
+ }
+}
+ }
+
+ override def genCode(ctx: CodeGenContext, ev: GeneratedExpressionCode):
String = {
+nullSafeCodeGen(ctx, ev, (dateVal, fmt) => {
+ val dtu = DateTimeUtils.getClass.getName.stripSuffix("$")
--- End diff --
Is `DateTimeUtils` imported by default in the codegen class?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8176] [SPARK-8197] [SQL] function to_da...
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/6988#discussion_r35185201
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala
---
@@ -258,3 +258,70 @@ case class DateFormatClass(left: Expression, right:
Expression) extends BinaryEx
})
}
}
+
+/**
+ * Returns the date part of a timestamp string.
+ */
+case class ToDate(child: Expression) extends UnaryExpression with
ImplicitCastInputTypes {
+
+ override def inputTypes: Seq[AbstractDataType] = Seq(DateType)
+
+ override def dataType: DataType = DateType
+
+ override def eval(input: InternalRow): Any = {
+child.eval(input).asInstanceOf[Int]
+ }
+
+ override def genCode(ctx: CodeGenContext, ev: GeneratedExpressionCode):
String = {
+nullSafeCodeGen(ctx, ev, (time) => {
+ time
+})
+ }
+}
+
+/**
+ * Returns date truncated to the unit specified by the format.
+ */
+case class Trunc(date: Expression, format: Expression)
+ extends BinaryExpression with ImplicitCastInputTypes {
+ override def left: Expression = date
+ override def right: Expression = format
+
+ override def inputTypes: Seq[AbstractDataType] = Seq(DateType,
StringType)
+ override def dataType: DataType = DateType
+
+ override def nullSafeEval(d: Any, fmt: Any): Any = {
+val minItem =
DateTimeUtils.getFmt(fmt.asInstanceOf[UTF8String].toUpperCase.toString)
+if (minItem == -1) {
+ // unknown format
+ null
+} else {
+ val days = d.asInstanceOf[Int]
+ val year = DateTimeUtils.getYear(days)
+ if (minItem == Calendar.YEAR) {
+days - DateTimeUtils.getDayInYear(days) + 1
+ } else {
+// trunc to MONTH
+days - DateTimeUtils.getDayOfMonth(days) + 1
+ }
+}
+ }
+
+ override def genCode(ctx: CodeGenContext, ev: GeneratedExpressionCode):
String = {
+nullSafeCodeGen(ctx, ev, (dateVal, fmt) => {
+ val dtu = DateTimeUtils.getClass.getName.stripSuffix("$")
+ val form = ctx.freshName("form")
+ s"""
+$form = $dtu.getFmt($fmt);
--- End diff --
Forgot to add the type for `$form`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...
Github user yu-iskw commented on the pull request: https://github.com/apache/spark/pull/7584#issuecomment-123577620 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7584#issuecomment-123577509 [Test build #38043 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38043/console) for PR 7584 at commit [`de3e0db`](https://github.com/apache/spark/commit/de3e0dbfa52e3676580f0f730d02d961dded063f). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7584#issuecomment-123577497 [Test build #38043 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38043/consoleFull) for PR 7584 at commit [`de3e0db`](https://github.com/apache/spark/commit/de3e0dbfa52e3676580f0f730d02d961dded063f). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7584#issuecomment-123577513 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8405][Doc] Add how to view logs on Web ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7463#issuecomment-123577339 [Test build #38044 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38044/consoleFull) for PR 7463 at commit [`74df3a1`](https://github.com/apache/spark/commit/74df3a120e7452be3c1ef00920326b7b7cd8cc0e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8176] [SPARK-8197] [SQL] function to_da...
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/6988#discussion_r35185035
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
---
@@ -573,4 +573,10 @@ object DateTimeUtils {
dayInYear - 334
}
}
+
--- End diff --
Add a scaladoc, particularly for the value of `-1`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...
Github user yu-iskw commented on the pull request: https://github.com/apache/spark/pull/7584#issuecomment-123577298 @shivaram This is the new result of lintr after modifying them you pointed out and rebasing with the master branch. https://gist.github.com/yu-iskw/8bb1a937ac08d0389182 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7584#issuecomment-123576913 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8405][Doc] Add how to view logs on Web ...
Github user carsonwang commented on the pull request: https://github.com/apache/spark/pull/7463#issuecomment-123576915 Sounds great, @tgravescs . The text was updated as you suggested. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7584#issuecomment-123576895 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8176] [SPARK-8197] [SQL] function to_da...
Github user adrian-wang commented on a diff in the pull request:
https://github.com/apache/spark/pull/6988#discussion_r35184929
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala
---
@@ -258,3 +258,70 @@ case class DateFormatClass(left: Expression, right:
Expression) extends BinaryEx
})
}
}
+
+/**
+ * Returns the date part of a timestamp string.
+ */
+case class ToDate(child: Expression) extends UnaryExpression with
ImplicitCastInputTypes {
+
+ override def inputTypes: Seq[AbstractDataType] = Seq(DateType)
+
+ override def dataType: DataType = DateType
+
+ override def eval(input: InternalRow): Any = {
--- End diff --
I'll delete `asInstanceOf`, so no need for nullSafeEval
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8405][Doc] Add how to view logs on Web ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7463#issuecomment-123576898 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8405][Doc] Add how to view logs on Web ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7463#issuecomment-123576912 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...
Github user yu-iskw commented on a diff in the pull request:
https://github.com/apache/spark/pull/7584#discussion_r35184909
--- Diff: R/pkg/inst/tests/test_sparkSQL.R ---
@@ -664,10 +664,10 @@ test_that("column binary mathfunctions", {
expect_equal(collect(select(df, atan2(df$a, df$b)))[2, "ATAN2(a, b)"],
atan2(2, 6))
expect_equal(collect(select(df, atan2(df$a, df$b)))[3, "ATAN2(a, b)"],
atan2(3, 7))
expect_equal(collect(select(df, atan2(df$a, df$b)))[4, "ATAN2(a, b)"],
atan2(4, 8))
- expect_equal(collect(select(df, hypot(df$a, df$b)))[1, "HYPOT(a, b)"],
sqrt(1^2 + 5^2))
- expect_equal(collect(select(df, hypot(df$a, df$b)))[2, "HYPOT(a, b)"],
sqrt(2^2 + 6^2))
- expect_equal(collect(select(df, hypot(df$a, df$b)))[3, "HYPOT(a, b)"],
sqrt(3^2 + 7^2))
- expect_equal(collect(select(df, hypot(df$a, df$b)))[4, "HYPOT(a, b)"],
sqrt(4^2 + 8^2))
+ expect_equal(collect(select(df, hypot(df$a, df$b)))[1, "HYPOT(a, b)"],
sqrt(1 ^ 2 + 5 ^ 2))
--- End diff --
Anyway, I will suggest any configuration to exclude infix tokens at
`infix_space_linter` to the lintr community later. It is a little bit annoying
to add the comment.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...
Github user yu-iskw commented on the pull request: https://github.com/apache/spark/pull/7584#issuecomment-123576750 I have rebased this PR with the master branch. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8176] [SPARK-8197] [SQL] function to_da...
Github user adrian-wang commented on a diff in the pull request:
https://github.com/apache/spark/pull/6988#discussion_r35184895
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala
---
@@ -258,3 +258,70 @@ case class DateFormatClass(left: Expression, right:
Expression) extends BinaryEx
})
}
}
+
+/**
+ * Returns the date part of a timestamp string.
+ */
+case class ToDate(child: Expression) extends UnaryExpression with
ImplicitCastInputTypes {
+
+ override def inputTypes: Seq[AbstractDataType] = Seq(DateType)
--- End diff --
TimestampType, StringType should all be convert into DateType.
Actually, I can add a rule in optimizer to delete this node after we have
done implicit cast.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Spark-8092][ml] Allow OneVsRest Classifier fe...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6631#issuecomment-123576709 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Spark-8092][ml] Allow OneVsRest Classifier fe...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6631#issuecomment-123576702 [Test build #38041 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38041/console) for PR 6631 at commit [`b7024b1`](https://github.com/apache/spark/commit/b7024b15bce598e173b0bc17a79aa55918b0f87c). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7584#issuecomment-123576331 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7584#issuecomment-123576346 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4233] [SPARK-4367] [SPARK-3947] [SPARK-...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/7458 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Spark-8092][ml] Allow OneVsRest Classifier fe...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6631#issuecomment-123576117 [Test build #38041 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38041/consoleFull) for PR 6631 at commit [`b7024b1`](https://github.com/apache/spark/commit/b7024b15bce598e173b0bc17a79aa55918b0f87c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4233] [SPARK-4367] [SPARK-3947] [SPARK-...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/7458#issuecomment-123575979 I'm going to merge this epic change! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4233] [SPARK-4367] [SPARK-3947] [SPARK-...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7458#issuecomment-123575788 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4233] [SPARK-4367] [SPARK-3947] [SPARK-...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7458#issuecomment-123575730 [Test build #38029 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38029/console) for PR 7458 at commit [`f1d5901`](https://github.com/apache/spark/commit/f1d5901d44fe47586449b9e7b2ea4cac9be5aa40). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class UnresolvedFunction(` * `case class Average(child: Expression) extends AlgebraicAggregate ` * `case class Count(child: Expression) extends AlgebraicAggregate ` * `case class First(child: Expression) extends AlgebraicAggregate ` * `case class Last(child: Expression) extends AlgebraicAggregate ` * `case class Max(child: Expression) extends AlgebraicAggregate ` * `case class Min(child: Expression) extends AlgebraicAggregate ` * `case class Sum(child: Expression) extends AlgebraicAggregate ` * `abstract class AlgebraicAggregate extends AggregateFunction2 with Serializable ` * ` implicit class RichAttribute(a: AttributeReference) ` * `trait AggregateExpression1 extends AggregateExpression ` * `trait PartialAggregate1 extends AggregateExpression1 ` * `case class Min(child: Expression) extends UnaryExpression with PartialAggregate1 ` * `case class MinFunction(expr: Expression, base: AggregateExpression1) extends AggregateFunction1 ` * `case class Max(child: Expression) extends UnaryExpression with PartialAggregate1 ` * `case class MaxFunction(expr: Expression, base: AggregateExpression1) extends AggregateFunction1 ` * `case class Count(child: Expression) extends UnaryExpression with PartialAggregate1 ` * `case class CountFunction(expr: Expression, base: AggregateExpression1) extends AggregateFunction1 ` * `case class CountDistinct(expressions: Seq[Expression]) extends PartialAggregate1 ` * `case class CollectHashSet(expressions: Seq[Expression]) extends AggregateExpression1 ` * `case class CombineSetsAndCount(inputSet: Expression) extends AggregateExpression1 ` * `case class Average(child: Expression) extends UnaryExpression with PartialAggregate1 ` * `case class AverageFunction(expr: Expression, base: AggregateExpression1)` * `case class Sum(child: Expression) extends UnaryExpression with PartialAggregate1 ` * `case class SumFunction(expr: Expression, base: AggregateExpression1) extends AggregateFunction1 ` * `case class CombineSum(child: Expression) extends AggregateExpression1 ` * `case class CombineSumFunction(expr: Expression, base: AggregateExpression1)` * `case class SumDistinct(child: Expression) extends UnaryExpression with PartialAggregate1 ` * `case class SumDistinctFunction(expr: Expression, base: AggregateExpression1)` * `case class CombineSetsAndSum(inputSet: Expression, base: Expression) extends AggregateExpression1 ` * `case class First(child: Expression) extends UnaryExpression with PartialAggregate1 ` * `case class FirstFunction(expr: Expression, base: AggregateExpression1) extends AggregateFunction1 ` * `case class Last(child: Expression) extends UnaryExpression with PartialAggregate1 ` * `case class LastFunction(expr: Expression, base: AggregateExpression1) extends AggregateFunction1 ` * `case class Aggregate2Sort(` * `case class FinalAndCompleteAggregate2Sort(` * `class GroupingIterator(` * `class PartialSortAggregationIterator(` * `class PartialMergeSortAggregationIterator(` * `class FinalSortAggregationIterator(` * `class FinalAndCompleteSortAggregationIterator(` * `abstract class UserDefinedAggregateFunction extends Serializable ` * `case class ScalaUDAF(` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Spark-8092][ml] Allow OneVsRest Classifier fe...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6631#issuecomment-123575130 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Spark-8092][ml] Allow OneVsRest Classifier fe...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6631#issuecomment-123575100 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...
Github user yu-iskw commented on a diff in the pull request:
https://github.com/apache/spark/pull/7584#discussion_r35184538
--- Diff: R/pkg/inst/tests/test_sparkSQL.R ---
@@ -664,10 +664,10 @@ test_that("column binary mathfunctions", {
expect_equal(collect(select(df, atan2(df$a, df$b)))[2, "ATAN2(a, b)"],
atan2(2, 6))
expect_equal(collect(select(df, atan2(df$a, df$b)))[3, "ATAN2(a, b)"],
atan2(3, 7))
expect_equal(collect(select(df, atan2(df$a, df$b)))[4, "ATAN2(a, b)"],
atan2(4, 8))
- expect_equal(collect(select(df, hypot(df$a, df$b)))[1, "HYPOT(a, b)"],
sqrt(1^2 + 5^2))
- expect_equal(collect(select(df, hypot(df$a, df$b)))[2, "HYPOT(a, b)"],
sqrt(2^2 + 6^2))
- expect_equal(collect(select(df, hypot(df$a, df$b)))[3, "HYPOT(a, b)"],
sqrt(3^2 + 7^2))
- expect_equal(collect(select(df, hypot(df$a, df$b)))[4, "HYPOT(a, b)"],
sqrt(4^2 + 8^2))
+ expect_equal(collect(select(df, hypot(df$a, df$b)))[1, "HYPOT(a, b)"],
sqrt(1 ^ 2 + 5 ^ 2))
--- End diff --
I understand.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9244] Increase some memory defaults
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7586#issuecomment-123574889 [Test build #38039 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38039/consoleFull) for PR 7586 at commit [`ce0038a`](https://github.com/apache/spark/commit/ce0038acc0527495b94a19534ebd5c2b83a9ee19). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9053][SparkR] Fix spaces around parens,...
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/7584#discussion_r35184460
--- Diff: R/pkg/inst/tests/test_sparkSQL.R ---
@@ -664,10 +664,10 @@ test_that("column binary mathfunctions", {
expect_equal(collect(select(df, atan2(df$a, df$b)))[2, "ATAN2(a, b)"],
atan2(2, 6))
expect_equal(collect(select(df, atan2(df$a, df$b)))[3, "ATAN2(a, b)"],
atan2(3, 7))
expect_equal(collect(select(df, atan2(df$a, df$b)))[4, "ATAN2(a, b)"],
atan2(4, 8))
- expect_equal(collect(select(df, hypot(df$a, df$b)))[1, "HYPOT(a, b)"],
sqrt(1^2 + 5^2))
- expect_equal(collect(select(df, hypot(df$a, df$b)))[2, "HYPOT(a, b)"],
sqrt(2^2 + 6^2))
- expect_equal(collect(select(df, hypot(df$a, df$b)))[3, "HYPOT(a, b)"],
sqrt(3^2 + 7^2))
- expect_equal(collect(select(df, hypot(df$a, df$b)))[4, "HYPOT(a, b)"],
sqrt(4^2 + 8^2))
+ expect_equal(collect(select(df, hypot(df$a, df$b)))[1, "HYPOT(a, b)"],
sqrt(1 ^ 2 + 5 ^ 2))
--- End diff --
I meant adding `# nolint start` and `# nolint end` as described in
https://github.com/jimhester/lintr#project-configuration
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9144] Remove DAGScheduler.runLocallyWit...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7585#issuecomment-123574302 [Test build #38040 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38040/consoleFull) for PR 7585 at commit [`1d9739a`](https://github.com/apache/spark/commit/1d9739ab232ff04b85cff8cec348d97b829965e3). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8176] [SPARK-8197] [SQL] function to_da...
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/6988#discussion_r35184384
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala
---
@@ -258,3 +258,70 @@ case class DateFormatClass(left: Expression, right:
Expression) extends BinaryEx
})
}
}
+
+/**
+ * Returns the date part of a timestamp string.
+ */
+case class ToDate(child: Expression) extends UnaryExpression with
ImplicitCastInputTypes {
+
+ override def inputTypes: Seq[AbstractDataType] = Seq(DateType)
+
+ override def dataType: DataType = DateType
+
+ override def eval(input: InternalRow): Any = {
+child.eval(input).asInstanceOf[Int]
+ }
+
+ override def genCode(ctx: CodeGenContext, ev: GeneratedExpressionCode):
String = {
--- End diff --
override the `nullSafeCodeGen` instead.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9244] Increase some memory defaults
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7586#issuecomment-123573781 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8176] [SPARK-8197] [SQL] function to_da...
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/6988#discussion_r35184304
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala
---
@@ -258,3 +258,70 @@ case class DateFormatClass(left: Expression, right:
Expression) extends BinaryEx
})
}
}
+
+/**
+ * Returns the date part of a timestamp string.
+ */
+case class ToDate(child: Expression) extends UnaryExpression with
ImplicitCastInputTypes {
+
+ override def inputTypes: Seq[AbstractDataType] = Seq(DateType)
+
+ override def dataType: DataType = DateType
+
+ override def eval(input: InternalRow): Any = {
--- End diff --
`nullSafeEval`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9144] Remove DAGScheduler.runLocallyWit...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7585#issuecomment-123573752 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9144] Remove DAGScheduler.runLocallyWit...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7585#issuecomment-123573808 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9244] Increase some memory defaults
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7586#issuecomment-123573739 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9244] Increase some memory defaults
GitHub user mateiz opened a pull request: https://github.com/apache/spark/pull/7586 [SPARK-9244] Increase some memory defaults There are a few memory limits that people hit often and that we could make higher, especially now that memory sizes have grown. - spark.akka.frameSize: This defaults at 10 but is often hit for map output statuses in large shuffles. This memory is not fully allocated up-front, so we can just make this larger and still not affect jobs that never sent a status that large. We increase it to 128. - spark.executor.memory: Defaults at 512m, which is really small. We increase it to 1g. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mateiz/spark configs Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/7586.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #7586 commit ce0038acc0527495b94a19534ebd5c2b83a9ee19 Author: Matei Zaharia Date: 2015-07-22T06:13:54Z [SPARK-9244] Increase some memory defaults There are a few memory limits that people hit often and that we could make higher, especially now that memory sizes have grown. - spark.akka.frameSize: This defaults at 10 but is often hit for map output statuses in large shuffles. This memory is not fully allocated up-front, so we can just make this larger and still not affect jobs that never sent a status that large. We increase it to 128. - spark.executor.memory: Defaults at 512m, which is really small. We increase it to 1g. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8176] [SPARK-8197] [SQL] function to_da...
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/6988#discussion_r35184263
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeFunctions.scala
---
@@ -258,3 +258,70 @@ case class DateFormatClass(left: Expression, right:
Expression) extends BinaryEx
})
}
}
+
+/**
+ * Returns the date part of a timestamp string.
+ */
+case class ToDate(child: Expression) extends UnaryExpression with
ImplicitCastInputTypes {
+
+ override def inputTypes: Seq[AbstractDataType] = Seq(DateType)
--- End diff --
Probably only accept the `DateType`, `TimestampType` and the `StringType`,
should be `Seq(TypeCollection(DateType, TimestampType, StringType))`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9144] Remove DAGScheduler.runLocallyWit...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/7484#issuecomment-123573536 I submitted https://github.com/apache/spark/pull/7585 to bring this up to date. We can merge that one. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9144] Remove DAGScheduler.runLocallyWit...
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/7585 [SPARK-9144] Remove DAGScheduler.runLocallyWithinThread and spark.localExecution.enabled Spark has an option called spark.localExecution.enabled; according to the docs: ``` Enables Spark to run certain jobs, such as first() or take() on the driver, without sending tasks to the cluster. This can make certain jobs execute very quickly, but may require shipping a whole partition of data to the driver. ``` This feature ends up adding quite a bit of complexity to DAGScheduler, especially in the runLocallyWithinThread method, but as far as I know nobody uses this feature (I searched the mailing list and haven't seen any recent mentions of the configuration nor stacktraces including the runLocally method). As a step towards scheduler complexity reduction, I propose that we remove this feature and all code related to it for Spark 1.5. This pull request simply brings #7484 up to date. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rxin/spark remove-local-exec Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/7585.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #7585 commit ffa8c9b6091559fb6f69d89d000eaa1b17dedef8 Author: Josh Rosen Date: 2015-07-17T22:10:39Z Remove documentation for configuration commit 8975d965215d02e0ed7b963325ece4a4645bb870 Author: Josh Rosen Date: 2015-07-17T22:16:21Z Remove local execution tests. commit b0835dc09dc6d7df1856fa96667ae56749ef8ce9 Author: Josh Rosen Date: 2015-07-17T22:21:07Z Remove local execution code in DAGScheduler commit eec39fa1d5d8f9f5bdfaf7b4907f2fe20b635c8f Author: Josh Rosen Date: 2015-07-17T22:53:48Z Remove allowLocal(); deprecate user-facing uses of it. commit 1d9739ab232ff04b85cff8cec348d97b829965e3 Author: Reynold Xin Date: 2015-07-22T06:12:35Z Merge pull request #7484 from JoshRosen/remove-localexecution [SPARK-9144] Remove DAGScheduler.runLocallyWithinThread and spark.localExecution.enabled Conflicts: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9144] Remove DAGScheduler.runLocallyWit...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/7484#issuecomment-123573395 I think the main time when this can help a lot is when you are connecting to a busy cluster, and in that case, it can take a while to get something scheduled. If the cluster is idle, it takes just a few ms to launch a task, and as a result users won't be able to tell the difference at all. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5155] [PySpark] [Streaming] Mqtt stream...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/4229#issuecomment-123573243 @prabeesh I urge you to apply the necessary changes soon if we have to make it Spark 1.5. The merge window is closing very soon, and it will be pretty sad if we cant make it in after coming this close. @zsxwing Please take a look as soon as he updates. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8935][SQL] Implement code generation fo...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7365#issuecomment-123573084 [Test build #1153 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1153/consoleFull) for PR 7365 at commit [`eaece18`](https://github.com/apache/spark/commit/eaece189f577d9ff9afe111dba7473a58a55baec). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL][minor] remove literal in agg group expre...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7583#issuecomment-123573113 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
