[GitHub] spark pull request #14156: [SPARK-16499][ML][MLLib] improve ApplyInPlace fun...
GitHub user WeichenXu123 opened a pull request: https://github.com/apache/spark/pull/14156 [SPARK-16499][ML][MLLib] improve ApplyInPlace function in ANN code ## What changes were proposed in this pull request? I re-code the following fuction using breeze's matrix operating function. def apply(x: BDM[Double], y: BDM[Double], func: Double => Double): Unit ## How was this patch tested? Existing test. You can merge this pull request into a Git repository by running: $ git pull https://github.com/WeichenXu123/spark improve_ann_apply_in_place_fun Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/14156.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #14156 commit c7b2059c5799404c3a3e99615e2ad7dc32989fda Author: WeichenXu Date: 2016-07-11T11:57:19Z improve_ann_apply_in_place_fun --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14140: [SPARK-16426][MLlib] Fix bug that caused NaNs in ...
Github user neggert commented on a diff in the pull request:
https://github.com/apache/spark/pull/14140#discussion_r70450795
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/regression/IsotonicRegression.scala
---
@@ -408,8 +409,12 @@ class IsotonicRegression private (private var
isotonic: Boolean) extends Seriali
*/
private def parallelPoolAdjacentViolators(
input: RDD[(Double, Double, Double)]): Array[(Double, Double,
Double)] = {
-val parallelStepResult = input
- .sortBy(x => (x._2, x._1))
+val keyedInput = input
--- End diff --
`repartitionAndSortWithinPartitions` requires the partition key and the
sortBy key to be the same. We want to partition by feature, then sort by
feature *and* label. So it would still require a second step to sort the
partitions, although they'd be mostly sorted already. Maybe there could be a
speed-up by using an insertion or Timsort on the mostly-sorted partition...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14149: [SPARK-16435][YARN][MINOR] Add warning log if ini...
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/14149#discussion_r70450332
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2342,6 +2342,12 @@ private[spark] object Utils extends Logging {
* Return the initial number of executors for dynamic allocation.
*/
def getDynamicAllocationInitialExecutors(conf: SparkConf): Int = {
+if (conf.get(DYN_ALLOCATION_INITIAL_EXECUTORS) <
conf.get(DYN_ALLOCATION_MIN_EXECUTORS)) {
+ logWarning(s"${DYN_ALLOCATION_INITIAL_EXECUTORS.key} less than " +
+s"${DYN_ALLOCATION_MIN_EXECUTORS.key} is invalid, will use " +
+ s"${DYN_ALLOCATION_MIN_EXECUTORS} instead.")
--- End diff --
executor instances is what gets set if user puts in --num-executors which I
see easily happening now and if min is already set and > then they are
different and we will use min so we should warn here also.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14155: [SPARK-16498][SQL][WIP] move hive hack for data s...
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/14155#discussion_r70449078
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/ParseDriver.scala
---
@@ -34,7 +34,7 @@ import org.apache.spark.sql.types.DataType
abstract class AbstractSqlParser extends ParserInterface with Logging {
/** Creates/Resolves DataType for a given SQL string. */
- def parseDataType(sqlText: String): DataType = parse(sqlText) { parser =>
+ override def parseDataType(sqlText: String): DataType = parse(sqlText) {
parser =>
// TODO add this to the parser interface.
--- End diff --
Remove TODO :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14154: [SPARK-16497][SQL] Don't throw an exception if drop non-...
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/14154 This is by design; we throw exceptions for all invalid commands. Why should we follow Hive here? See for more context: https://github.com/apache/spark/pull/13307 & https://github.com/apache/spark/pull/13302 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14155: [SPARK-16498][SQL][WIP] move hive hack for data source t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14155 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62170/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14155: [SPARK-16498][SQL][WIP] move hive hack for data source t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14155 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14155: [SPARK-16498][SQL][WIP] move hive hack for data source t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14155 **[Test build #62170 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62170/consoleFull)** for PR 14155 at commit [`d519968`](https://github.com/apache/spark/commit/d5199681d03d397292c7866a826e7d2eb9fc). * This patch **fails to build**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class CreateDataSourceTableCommand(table: CatalogTable, ifNotExists: Boolean)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14150: [SPARK-16494] [ML] Upgrade breeze version to 0.12
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14150 **[Test build #62171 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62171/consoleFull)** for PR 14150 at commit [`3127733`](https://github.com/apache/spark/commit/3127733403e99b8aa7131a4fe9420714b4a0ae55). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14148: [SPARK-16482] [SQL] Describe Table Command for Tables Re...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/14148 @rxin The failed test case is interesting! `REFRESH TABLE` command does not refresh the metadata stored in the external catalog. When the tables are data source tables, it is a bug? Please let me know if this is by design. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14155: [SPARK-16498][SQL][WIP] move hive hack for data source t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14155 **[Test build #62170 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62170/consoleFull)** for PR 14155 at commit [`d519968`](https://github.com/apache/spark/commit/d5199681d03d397292c7866a826e7d2eb9fc). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14149: [SPARK-16435][YARN][MINOR] Add warning log if ini...
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/14149#discussion_r70444399
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2342,6 +2342,12 @@ private[spark] object Utils extends Logging {
* Return the initial number of executors for dynamic allocation.
*/
def getDynamicAllocationInitialExecutors(conf: SparkConf): Int = {
+if (conf.get(DYN_ALLOCATION_INITIAL_EXECUTORS) <
conf.get(DYN_ALLOCATION_MIN_EXECUTORS)) {
+ logWarning(s"${DYN_ALLOCATION_INITIAL_EXECUTORS.key} less than " +
+s"${DYN_ALLOCATION_MIN_EXECUTORS.key} is invalid, will use " +
+ s"${DYN_ALLOCATION_MIN_EXECUTORS} instead.")
--- End diff --
also note that the comment above isn't exactly correct because if the
number of executor instances is > then the min it would use that number instead
of the min executors.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14155: [SPARK-16498][SQL][WIP] move hive hack for data s...
GitHub user cloud-fan opened a pull request: https://github.com/apache/spark/pull/14155 [SPARK-16498][SQL][WIP] move hive hack for data source table into HiveExternalCatalog ## What changes were proposed in this pull request? Spark SQL doesn't have its own meta store yet, and use hive's currently. However, hive's meta store has some limitations(e.g. columns can't be too many, not case-preserving, bad decimal type support, etc.), so we have some hacks to successfully store data source table metadata into hive meta store, i.e. put all the information in table properties. This PR moves these hacks to `HiveExternalCatalog`, tries to isolate hive specific logic in one place. It's based on https://github.com/apache/spark/pull/14071 ## How was this patch tested? existing tests. You can merge this pull request into a Git repository by running: $ git pull https://github.com/cloud-fan/spark catalog-table Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/14155.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #14155 commit 4d65609ae71b2e30cea7b39e1b5a1a9ecfdd2de4 Author: Wenchen Fan Date: 2016-07-06T17:08:43Z make CatalogTable more general and less hive specific commit b218bb72c8c81192e11db2baed9321578c61cb5e Author: Wenchen Fan Date: 2016-07-08T03:07:13Z address comments commit d5199681d03d397292c7866a826e7d2eb9fc Author: Wenchen Fan Date: 2016-07-12T14:01:32Z move hive hack for data source table into HiveExternalCatalog --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14154: [SPARK-16497][SQL] Don't throw an exception if drop non-...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14154 **[Test build #62169 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62169/consoleFull)** for PR 14154 at commit [`7b64de2`](https://github.com/apache/spark/commit/7b64de2ee800815db4e82b94e520831c25726bdd). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14149: [SPARK-16435][YARN][MINOR] Add warning log if ini...
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/14149#discussion_r70444156
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2342,6 +2342,12 @@ private[spark] object Utils extends Logging {
* Return the initial number of executors for dynamic allocation.
*/
def getDynamicAllocationInitialExecutors(conf: SparkConf): Int = {
+if (conf.get(DYN_ALLOCATION_INITIAL_EXECUTORS) <
conf.get(DYN_ALLOCATION_MIN_EXECUTORS)) {
+ logWarning(s"${DYN_ALLOCATION_INITIAL_EXECUTORS.key} less than " +
+s"${DYN_ALLOCATION_MIN_EXECUTORS.key} is invalid, will use " +
+ s"${DYN_ALLOCATION_MIN_EXECUTORS} instead.")
--- End diff --
so this doesn't cover is the user specified the executor instances < the
minimum seems like we should also put out a warning for that also.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14142: [SPARK-16439] Fix number formatting in SQL UI
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14142 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14142: [SPARK-16439] Fix number formatting in SQL UI
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14142 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62163/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14142: [SPARK-16439] Fix number formatting in SQL UI
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14142 **[Test build #62163 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62163/consoleFull)** for PR 14142 at commit [`fef15ce`](https://github.com/apache/spark/commit/fef15cef2fb90c5dc06332723a14958bb584ed5c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14149: [SPARK-16435][YARN][MINOR] Add warning log if ini...
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/14149#discussion_r70442221
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -2342,6 +2342,12 @@ private[spark] object Utils extends Logging {
* Return the initial number of executors for dynamic allocation.
*/
def getDynamicAllocationInitialExecutors(conf: SparkConf): Int = {
+if (conf.get(DYN_ALLOCATION_INITIAL_EXECUTORS) <
conf.get(DYN_ALLOCATION_MIN_EXECUTORS)) {
--- End diff --
this isn't going to warn if executor instances was specified and is less
then the minimum. We could simply move this check til after the max and see if
the number that is going to be returned is < min executors.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14154: [SPARK-16497][SQL] Don't throw an exception if drop non-...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14154 **[Test build #62168 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62168/consoleFull)** for PR 14154 at commit [`6eb9e8e`](https://github.com/apache/spark/commit/6eb9e8e916cea391aff0d07a85781e77121ce9b9). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14154: [SPARK-16497][SQL] Don't throw an exception if dr...
GitHub user lianhuiwang opened a pull request: https://github.com/apache/spark/pull/14154 [SPARK-16497][SQL] Don't throw an exception if drop non-existent TABLE/VIEW/Function/Partitions ## What changes were proposed in this pull request? from https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.exec.drop.ignorenonexistent, Hive use 'hive.exec.drop.ignorenonexistent'(default=true) to do not report an error if DROP TABLE/VIEW/PARTITION/INDEX/TEMPORARY FUNCTION specifies a non-existent table/view. So SparkSQL also should support it. ## How was this patch tested? add unit tests. You can merge this pull request into a Git repository by running: $ git pull https://github.com/lianhuiwang/spark drop-ignorenonexist Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/14154.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #14154 commit 25bd4b8fad66784aec7235ddfea5c4dee1450d11 Author: Lianhui Wang Date: 2016-07-11T14:43:37Z init commit commit 6eb9e8e916cea391aff0d07a85781e77121ce9b9 Author: Lianhui Wang Date: 2016-07-12T13:39:29Z update --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14036: [SPARK-16323] [SQL] Add IntegerDivide to avoid un...
Github user techaddict commented on a diff in the pull request: https://github.com/apache/spark/pull/14036#discussion_r70437720 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala --- @@ -237,6 +229,9 @@ case class Divide(left: Expression, right: Expression) } } + // Used by doGenCode + protected def divide(eval1: ExprCode, eval2: ExprCode, javaType: String): String --- End diff -- I did it on purpose. we can't call `$div` on `byte's` and plus if I try to call `value = value1 / value2;` for decimals, I get ```Binary numeric promotion not possible on types "org.apache.spark.sql.types.Decimal" and "org.apache.spark.sql.types.Decimal"```. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14132: [SPARK-16475][SQL][WIP] Broadcast Hint for SQL Queries
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14132 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14132: [SPARK-16475][SQL][WIP] Broadcast Hint for SQL Queries
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14132 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62162/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14132: [SPARK-16475][SQL][WIP] Broadcast Hint for SQL Queries
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14132 **[Test build #62162 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62162/consoleFull)** for PR 14132 at commit [`edec2e4`](https://github.com/apache/spark/commit/edec2e4ff0648c73f2efc743edca80a705565406). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class Hint(name: String, parameters: Seq[String], child: LogicalPlan) extends UnaryNode ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14151: [SPARK-16496][SQL] Add wholetext as data source for SQL.
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14151 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62161/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14151: [SPARK-16496][SQL] Add wholetext as data source for SQL.
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14151 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14151: [SPARK-16496][SQL] Add wholetext as data source for SQL.
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14151 **[Test build #62161 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62161/consoleFull)** for PR 14151 at commit [`dafe981`](https://github.com/apache/spark/commit/dafe981176dac55a623c24dcd8f439ecdf27d72a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13990: [SPARK-16287][SQL] Implement str_to_map SQL funct...
Github user techaddict commented on a diff in the pull request:
https://github.com/apache/spark/pull/13990#discussion_r70434309
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala
---
@@ -393,3 +394,84 @@ case class CreateNamedStructUnsafe(children:
Seq[Expression]) extends Expression
override def prettyName: String = "named_struct_unsafe"
}
+
+/**
+ * Creates a map after splitting the input text into key/value pairs using
delimeters
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(text[, pairDelim, keyValueDelim]) - Creates a map after
splitting the text " +
+"into key/value pairs using delimiters. " +
+"Default delimiters are ',' for pairDelim and ':' for keyValueDelim.",
+ extended = """ > SELECT _FUNC_('a:1,b:2,c:3',',',':');\n
map("a":"1","b":"2","c":"3") """)
+case class StringToMap(text: Expression, pairDelim: Expression,
keyValueDelim: Expression)
+ extends TernaryExpression {
+
+ def this(child: Expression, pairDelim: Expression) = {
+this(child, pairDelim, Literal(":"))
+ }
+
+ def this(child: Expression) = {
+this(child, Literal(","), Literal(":"))
+ }
+
+ override def children: Seq[Expression] = Seq(text, pairDelim,
keyValueDelim)
+
+ override def dataType: DataType = MapType(StringType, StringType,
valueContainsNull = false)
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+if (children.map(_.dataType).forall(_ == StringType)) {
+ TypeCheckResult.TypeCheckSuccess
+} else {
+ TypeCheckResult.TypeCheckFailure(s"String To Map's all arguments
should be string literal.")
--- End diff --
only text should be foldable ? or all three ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14152: [SPARK-16395] [STREAMING] Fail if too many CheckpointWri...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14152 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62164/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14152: [SPARK-16395] [STREAMING] Fail if too many CheckpointWri...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14152 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14152: [SPARK-16395] [STREAMING] Fail if too many CheckpointWri...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14152 **[Test build #62164 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62164/consoleFull)** for PR 14152 at commit [`a6fce66`](https://github.com/apache/spark/commit/a6fce665b4de00d37b13d12ace4eebad364c1a3d). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14149: [SPARK-16435][YARN][MINOR] Add warning log if initialExe...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/14149 @tgravescs and @rdblue , please help to review, thanks a lot. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14150: [SPARK-16494] [ML] Upgrade breeze version to 0.12
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14150 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62167/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14150: [SPARK-16494] [ML] Upgrade breeze version to 0.12
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14150 **[Test build #62167 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62167/consoleFull)** for PR 14150 at commit [`d2d567f`](https://github.com/apache/spark/commit/d2d567fbb650dab9a2646f56a0e99ba24e979432). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14150: [SPARK-16494] [ML] Upgrade breeze version to 0.12
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14150 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14153: [SPARK-16440] [MLLIB] Undeleted broadcast variables in W...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14153 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62165/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14153: [SPARK-16440] [MLLIB] Undeleted broadcast variables in W...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14153 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14153: [SPARK-16440] [MLLIB] Undeleted broadcast variables in W...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14153 **[Test build #62165 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62165/consoleFull)** for PR 14153 at commit [`0a933d8`](https://github.com/apache/spark/commit/0a933d8f3687a36664b294350192a26ee3e7d9cd). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14150: [SPARK-16494] [ML] Upgrade breeze version to 0.12
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14150 **[Test build #62167 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62167/consoleFull)** for PR 14150 at commit [`d2d567f`](https://github.com/apache/spark/commit/d2d567fbb650dab9a2646f56a0e99ba24e979432). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13990: [SPARK-16287][SQL] Implement str_to_map SQL funct...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/13990#discussion_r70432140
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala
---
@@ -393,3 +394,84 @@ case class CreateNamedStructUnsafe(children:
Seq[Expression]) extends Expression
override def prettyName: String = "named_struct_unsafe"
}
+
+/**
+ * Creates a map after splitting the input text into key/value pairs using
delimeters
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(text[, pairDelim, keyValueDelim]) - Creates a map after
splitting the text " +
+"into key/value pairs using delimiters. " +
+"Default delimiters are ',' for pairDelim and ':' for keyValueDelim.",
+ extended = """ > SELECT _FUNC_('a:1,b:2,c:3',',',':');\n
map("a":"1","b":"2","c":"3") """)
+case class StringToMap(text: Expression, pairDelim: Expression,
keyValueDelim: Expression)
+ extends TernaryExpression {
+
+ def this(child: Expression, pairDelim: Expression) = {
+this(child, pairDelim, Literal(":"))
+ }
+
+ def this(child: Expression) = {
+this(child, Literal(","), Literal(":"))
+ }
+
+ override def children: Seq[Expression] = Seq(text, pairDelim,
keyValueDelim)
+
+ override def dataType: DataType = MapType(StringType, StringType,
valueContainsNull = false)
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+if (children.map(_.dataType).forall(_ == StringType)) {
+ TypeCheckResult.TypeCheckSuccess
+} else {
+ TypeCheckResult.TypeCheckFailure(s"String To Map's all arguments
should be string literal.")
+}
+ }
+
+ override def nullSafeEval(str: Any, delim1: Any, delim2: Any): Any = {
+val array = str.asInstanceOf[UTF8String]
+ .split(delim1.asInstanceOf[UTF8String], -1)
+ .map { kv =>
+val arr = kv.split(delim2.asInstanceOf[UTF8String], 2)
+if(arr.length < 2) {
+ Array(arr(0), null)
+} else {
+ arr
+}
+ }
+ArrayBasedMapData(array.map(_(0)), array.map(_(1)))
+ }
+
+ override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
--- End diff --
for this expression I don't think the codegen version will be much faster.
Let's extends `CodegenFallback` first, to make the PR smaller.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14150: [SPARK-16494] [ML] Upgrade breeze version to 0.12
Github user yanboliang commented on the issue: https://github.com/apache/spark/pull/14150 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13990: [SPARK-16287][SQL] Implement str_to_map SQL funct...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/13990#discussion_r70431713
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala
---
@@ -393,3 +394,84 @@ case class CreateNamedStructUnsafe(children:
Seq[Expression]) extends Expression
override def prettyName: String = "named_struct_unsafe"
}
+
+/**
+ * Creates a map after splitting the input text into key/value pairs using
delimeters
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(text[, pairDelim, keyValueDelim]) - Creates a map after
splitting the text " +
+"into key/value pairs using delimiters. " +
+"Default delimiters are ',' for pairDelim and ':' for keyValueDelim.",
+ extended = """ > SELECT _FUNC_('a:1,b:2,c:3',',',':');\n
map("a":"1","b":"2","c":"3") """)
+case class StringToMap(text: Expression, pairDelim: Expression,
keyValueDelim: Expression)
+ extends TernaryExpression {
+
+ def this(child: Expression, pairDelim: Expression) = {
+this(child, pairDelim, Literal(":"))
+ }
+
+ def this(child: Expression) = {
+this(child, Literal(","), Literal(":"))
+ }
+
+ override def children: Seq[Expression] = Seq(text, pairDelim,
keyValueDelim)
+
+ override def dataType: DataType = MapType(StringType, StringType,
valueContainsNull = false)
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+if (children.map(_.dataType).forall(_ == StringType)) {
+ TypeCheckResult.TypeCheckSuccess
+} else {
+ TypeCheckResult.TypeCheckFailure(s"String To Map's all arguments
should be string literal.")
--- End diff --
BTW the error message is wrong, the delimiters should be literal but the
`text` should not.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13990: [SPARK-16287][SQL] Implement str_to_map SQL funct...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/13990#discussion_r70431411
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala
---
@@ -393,3 +394,84 @@ case class CreateNamedStructUnsafe(children:
Seq[Expression]) extends Expression
override def prettyName: String = "named_struct_unsafe"
}
+
+/**
+ * Creates a map after splitting the input text into key/value pairs using
delimeters
+ */
+@ExpressionDescription(
+ usage = "_FUNC_(text[, pairDelim, keyValueDelim]) - Creates a map after
splitting the text " +
+"into key/value pairs using delimiters. " +
+"Default delimiters are ',' for pairDelim and ':' for keyValueDelim.",
+ extended = """ > SELECT _FUNC_('a:1,b:2,c:3',',',':');\n
map("a":"1","b":"2","c":"3") """)
+case class StringToMap(text: Expression, pairDelim: Expression,
keyValueDelim: Expression)
+ extends TernaryExpression {
+
+ def this(child: Expression, pairDelim: Expression) = {
+this(child, pairDelim, Literal(":"))
+ }
+
+ def this(child: Expression) = {
+this(child, Literal(","), Literal(":"))
+ }
+
+ override def children: Seq[Expression] = Seq(text, pairDelim,
keyValueDelim)
+
+ override def dataType: DataType = MapType(StringType, StringType,
valueContainsNull = false)
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+if (children.map(_.dataType).forall(_ == StringType)) {
+ TypeCheckResult.TypeCheckSuccess
+} else {
+ TypeCheckResult.TypeCheckFailure(s"String To Map's all arguments
should be string literal.")
--- End diff --
we should also check the `foldable`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13873: [SPARK-16167][SQL] RowEncoder should preserve array/map ...
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/13873 ping @cloud-fan --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13990: [SPARK-16287][SQL] Implement str_to_map SQL function
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13990 **[Test build #62166 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62166/consoleFull)** for PR 13990 at commit [`1592ffc`](https://github.com/apache/spark/commit/1592ffca6506d4c1dcb099a8dc233d6184ac7d83). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14140: [SPARK-16426][MLlib] Fix bug that caused NaNs in ...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/14140#discussion_r70429952
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/regression/IsotonicRegression.scala
---
@@ -408,8 +409,12 @@ class IsotonicRegression private (private var
isotonic: Boolean) extends Seriali
*/
private def parallelPoolAdjacentViolators(
input: RDD[(Double, Double, Double)]): Array[(Double, Double,
Double)] = {
-val parallelStepResult = input
- .sortBy(x => (x._2, x._1))
+val keyedInput = input
--- End diff --
I think there may be shorter ways to write this with `groupBy`, but, this
and other approaches like that have the big drawback of reading lots of data
into memory. Here you have to sort the whole partition in memory (!).
How about `repartitionAndSortWithinPartitions`? oddly specific method, but,
likely just what you need here, to both partition according to some criteria
but then end up with sorted partitions. It's more scalable.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13704: [SPARK-15985][SQL] Eliminate redundant cast from an arra...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13704 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13704: [SPARK-15985][SQL] Eliminate redundant cast from an arra...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13704 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62159/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14150: [SPARK-16494] [ML] Upgrade breeze version to 0.12
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14150 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62160/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14150: [SPARK-16494] [ML] Upgrade breeze version to 0.12
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14150 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14150: [SPARK-16494] [ML] Upgrade breeze version to 0.12
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14150 **[Test build #62160 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62160/consoleFull)** for PR 14150 at commit [`d2d567f`](https://github.com/apache/spark/commit/d2d567fbb650dab9a2646f56a0e99ba24e979432). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13704: [SPARK-15985][SQL] Eliminate redundant cast from an arra...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13704 **[Test build #62159 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62159/consoleFull)** for PR 13704 at commit [`e6a5772`](https://github.com/apache/spark/commit/e6a5772f4a957cef353d493763cec84321d365fb). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14151: [SPARK-16496][SQL] Add wholetext as data source for SQL.
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14151 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14151: [SPARK-16496][SQL] Add wholetext as data source for SQL.
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14151 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62158/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14151: [SPARK-16496][SQL] Add wholetext as data source for SQL.
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14151 **[Test build #62158 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62158/consoleFull)** for PR 14151 at commit [`bd2936d`](https://github.com/apache/spark/commit/bd2936dfbecd9f65b2e3ccea088ebf13fa1acc5d). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class HadoopFileWholeTextReader(file: PartitionedFile, conf: Configuration) extends Iterator[Text] ` * `class WholeTextFileFormat extends TextFileFormat ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13701: [SPARK-15639][SQL] Try to push down filter at RowGroups ...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/13701 ping @liancheng @yhuai again. Can you take a look? Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14132: [SPARK-16475][SQL][WIP] Broadcast Hint for SQL Qu...
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/14132#discussion_r70426742
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -86,6 +86,7 @@ class Analyzer(
WindowsSubstitution,
EliminateUnions),
Batch("Resolution", fixedPoint,
+ ResolveHint ::
--- End diff --
Move this to substitution? Seems more appropriate.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14132: [SPARK-16475][SQL][WIP] Broadcast Hint for SQL Qu...
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/14132#discussion_r70426735
--- Diff:
sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 ---
@@ -945,8 +955,12 @@ SIMPLE_COMMENT
: '--' ~[\r\n]* '\r'? '\n'? -> channel(HIDDEN)
;
+BRACKETED_EMPTY_COMMENT
--- End diff --
My advice would be to add the HINT_PREFIX rule ('/*+')
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14132: [SPARK-16475][SQL][WIP] Broadcast Hint for SQL Qu...
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/14132#discussion_r70426683
--- Diff:
sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 ---
@@ -347,6 +347,15 @@ querySpecification
windows?)
;
+hint
+: '/*+' hintStatement '*/'
+;
+
+hintStatement
+: hintName=identifier identifierList
+| hintName=identifier '(' parameter1=identifier parameter2=identifier
')'
--- End diff --
I think you can also do this (is easier in the AST builder): `|
hintName=identifier '(' parameters+=identifier parameters+=identifier ')'`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14153: [SPARK-16440] [MLLIB] Undeleted broadcast variables in W...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14153 **[Test build #62165 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62165/consoleFull)** for PR 14153 at commit [`0a933d8`](https://github.com/apache/spark/commit/0a933d8f3687a36664b294350192a26ee3e7d9cd). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14132: [SPARK-16475][SQL][WIP] Broadcast Hint for SQL Qu...
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/14132#discussion_r70425722
--- Diff:
sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 ---
@@ -347,6 +347,15 @@ querySpecification
windows?)
;
+hint
+: '/*+' hintStatement '*/'
+;
+
+hintStatement
+: hintName=identifier identifierList
+| hintName=identifier '(' parameter1=identifier parameter2=identifier
')'
--- End diff --
What are we trying to support here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14153: [SPARK-16440] [MLLIB] Undeleted broadcast variabl...
GitHub user srowen opened a pull request: https://github.com/apache/spark/pull/14153 [SPARK-16440] [MLLIB] Undeleted broadcast variables in Word2Vec causing OoM for long runs ## What changes were proposed in this pull request? Unpersist broadcasted vars in Word2Vec.fit for more timely / reliable resource cleanup ## How was this patch tested? Jenkins tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/srowen/spark SPARK-16440 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/14153.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #14153 commit 0a933d8f3687a36664b294350192a26ee3e7d9cd Author: Sean Owen Date: 2016-07-12T12:08:05Z Unpersist broadcasted vars in Word2Vec.fit for more timely / reliable resource cleanup --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14122: [SPARK-16470][ML][Optimizer] Check linear regress...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/14122 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14122: [SPARK-16470][ML][Optimizer] Check linear regression tra...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/14122 Merged to master, and 2.0 on the grounds that it's just an added informative warning --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14152: [SPARK-16395] [STREAMING] Fail if too many CheckpointWri...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14152 **[Test build #62164 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62164/consoleFull)** for PR 14152 at commit [`a6fce66`](https://github.com/apache/spark/commit/a6fce665b4de00d37b13d12ace4eebad364c1a3d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14152: [SPARK-16395] [STREAMING] Fail if too many CheckpointWri...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/14152 CC @tdas --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14152: [SPARK-16395] [STREAMING] Fail if too many Checkp...
GitHub user srowen opened a pull request: https://github.com/apache/spark/pull/14152 [SPARK-16395] [STREAMING] Fail if too many CheckpointWriteHandlers are queued up in the fixed thread pool ## What changes were proposed in this pull request? Begin failing if checkpoint writes will likely keep up with storage's ability to write them, to fail fast instead of slowly filling memory ## How was this patch tested? Jenkins tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/srowen/spark SPARK-16395 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/14152.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #14152 commit a6fce665b4de00d37b13d12ace4eebad364c1a3d Author: Sean Owen Date: 2016-07-12T12:00:14Z Begin failing if checkpoint writes will likely keep up with storage's ability to write them, to fail fast instead of slowly filling memory --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #11293: [SPARK-13080] [SQL] Implement new Catalog API usi...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/11293#discussion_r70422619
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala
---
@@ -127,33 +166,30 @@ abstract class Catalog {
* @param name name of the function
* @param className fully qualified class name, e.g.
"org.apache.spark.util.MyFunc"
*/
-case class Function(
- name: String,
- className: String
-)
+case class CatalogFunction(name: String, className: String)
/**
* Storage format, used to describe how a partition or a table is stored.
*/
-case class StorageFormat(
- locationUri: String,
- inputFormat: String,
- outputFormat: String,
- serde: String,
- serdeProperties: Map[String, String]
-)
+case class CatalogStorageFormat(
+locationUri: Option[String],
+inputFormat: Option[String],
+outputFormat: Option[String],
+serde: Option[String],
+serdeProperties: Map[String, String])
/**
* A column in a table.
*/
-case class Column(
- name: String,
- dataType: String,
- nullable: Boolean,
- comment: String
-)
+case class CatalogColumn(
+name: String,
+// This may be null when used to create views. TODO: make this
type-safe; this is left
+// as a string due to issues in converting Hive varchars to and from
SparkSQL strings.
--- End diff --
What is the hive varchars issue exactly?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13494: [SPARK-15752] [SQL] Optimize metadata only query that ha...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13494 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13494: [SPARK-15752] [SQL] Optimize metadata only query that ha...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13494 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62156/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14142: [SPARK-16439] Fix number formatting in SQL UI
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14142 **[Test build #62163 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62163/consoleFull)** for PR 14142 at commit [`fef15ce`](https://github.com/apache/spark/commit/fef15cef2fb90c5dc06332723a14958bb584ed5c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13494: [SPARK-15752] [SQL] Optimize metadata only query that ha...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13494 **[Test build #62156 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62156/consoleFull)** for PR 13494 at commit [`030776a`](https://github.com/apache/spark/commit/030776ae49484c4e5db7f775344e5e40dff27e9a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14104: [SPARK-16438] Add Asynchronous Actions documentation
Github user srowen commented on the issue: https://github.com/apache/spark/pull/14104 @phalodi no need to ping. We don't merge things immediately, especially when non-essential. It gives more time for review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14142: [SPARK-16439] Fix number formatting in SQL UI
Github user srowen commented on the issue: https://github.com/apache/spark/pull/14142 Jenkins test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14142: [SPARK-16439] Fix number formatting in SQL UI
Github user srowen commented on the issue: https://github.com/apache/spark/pull/14142 Jenkins add to whitelist --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13704: [SPARK-15985][SQL] Eliminate redundant cast from an arra...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13704 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13704: [SPARK-15985][SQL] Eliminate redundant cast from an arra...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13704 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62155/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13704: [SPARK-15985][SQL] Eliminate redundant cast from an arra...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13704 **[Test build #62155 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62155/consoleFull)** for PR 13704 at commit [`466ba88`](https://github.com/apache/spark/commit/466ba882d1630a6ade3a94c381be20551ec19111). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14132: [SPARK-16475][SQL][WIP] Broadcast Hint for SQL Queries
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14132 **[Test build #62162 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62162/consoleFull)** for PR 14132 at commit [`edec2e4`](https://github.com/apache/spark/commit/edec2e4ff0648c73f2efc743edca80a705565406). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14151: [SPARK-16496][SQL] Add wholetext as data source for SQL.
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14151 **[Test build #62161 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62161/consoleFull)** for PR 14151 at commit [`dafe981`](https://github.com/apache/spark/commit/dafe981176dac55a623c24dcd8f439ecdf27d72a). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13704: [SPARK-15985][SQL] Eliminate redundant cast from ...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/13704#discussion_r70418445
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/SimplifyCastsSuite.scala
---
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.dsl.expressions._
+import org.apache.spark.sql.catalyst.dsl.plans._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.PlanTest
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.rules.RuleExecutor
+import org.apache.spark.sql.types._
+
+class SimplifyCastsSuite extends PlanTest {
+
+ object Optimize extends RuleExecutor[LogicalPlan] {
+val batches = Batch("SimplifyCasts", FixedPoint(50), SimplifyCasts) ::
Nil
+ }
+
+ test("non-nullable to non-nullable array cast") {
+val input = LocalRelation('a.array(ArrayType(IntegerType)))
--- End diff --
for map, we have
```
def map(keyType: DataType, valueType: DataType): AttributeReference =
map(MapType(keyType, valueType))
def map(mapType: MapType): AttributeReference =
AttributeReference(s, mapType, nullable = true)()
```
we can also add similar one to array
```
def array(arrayType: ArrayType): AttributeReference
```
Then here we can create an array type attribute with expected nullability,
and avoid using literal in optimizer test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14150: [SPARK-16494] [ML] Upgrade breeze version to 0.12
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14150 **[Test build #62160 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62160/consoleFull)** for PR 14150 at commit [`d2d567f`](https://github.com/apache/spark/commit/d2d567fbb650dab9a2646f56a0e99ba24e979432). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14151: [SPARK-16496][SQL] Add wholetext as data source for SQL.
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14151 **[Test build #62158 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62158/consoleFull)** for PR 14151 at commit [`bd2936d`](https://github.com/apache/spark/commit/bd2936dfbecd9f65b2e3ccea088ebf13fa1acc5d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13704: [SPARK-15985][SQL] Eliminate redundant cast from an arra...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13704 **[Test build #62159 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62159/consoleFull)** for PR 13704 at commit [`e6a5772`](https://github.com/apache/spark/commit/e6a5772f4a957cef353d493763cec84321d365fb). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14151: [SPARK-16496][SQL] Add wholetext as data source f...
GitHub user ScrapCodes opened a pull request: https://github.com/apache/spark/pull/14151 [SPARK-16496][SQL] Add wholetext as data source for SQL. ## What changes were proposed in this pull request? In multiple text analysis, problems it is not often desirable for the rows to be split by "\n". There exists a wholeText reader for RDD API, and this JIRA just adds the same support for Dataset API. ## How was this patch tested? Added two new tests. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ScrapCodes/spark SPARK-16496/wholetext Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/14151.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #14151 commit bd2936dfbecd9f65b2e3ccea088ebf13fa1acc5d Author: Prashant Sharma Date: 2016-07-08T08:48:37Z [SPARK-16496][SQL] Add wholetext as data source for SQL. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13704: [SPARK-15985][SQL] Eliminate redundant cast from an arra...
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/13704 Thanks again for your kindly reviews. I addresses both comments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14150: [SPARK-16494] [ML] Upgrade breeze version to 0.12
Github user yanboliang commented on the issue: https://github.com/apache/spark/pull/14150 @hvanhovell Yes, I updated the dependency manifests manually. I will change to use the scripts you provided. Thanks for your kindly remind! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14014: [SPARK-16344][SQL] Decoding Parquet array of stru...
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14014#discussion_r70415596
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala
---
@@ -260,7 +260,7 @@ private[parquet] class ParquetSchemaConverter(
{
// For legacy 2-level list types with primitive element type, e.g.:
//
- //// List (nullable list, non-null elements)
+ //// ARRAY (nullable list, non-null elements)
--- End diff --
I'm using concrete SQL types here for better readability.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14149: [SPARK-16435][YARN][MINOR] Add warning log if initialExe...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14149 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62153/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14149: [SPARK-16435][YARN][MINOR] Add warning log if initialExe...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14149 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14149: [SPARK-16435][YARN][MINOR] Add warning log if initialExe...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14149 **[Test build #62153 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62153/consoleFull)** for PR 14149 at commit [`f08eabd`](https://github.com/apache/spark/commit/f08eabdc9c20f29dd3a007784a4b893172bdc2db). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14150: [SPARK-16494] [ML] Upgrade breeze version to 0.12
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/14150 Hmmm it is also pulling in shapeless... did you manually update the manifests? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13704: [SPARK-15985][SQL] Eliminate redundant cast from an arra...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/13704 LGTM, one last comment: https://github.com/apache/spark/pull/13704/files#r70381491 BTW can you also update the PR description? e.g. we don't need to show the difference of generated code as there is no codegen related changes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14150: [SPARK-16494] [ML] Upgrade breeze version to 0.12
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14150 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62157/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14150: [SPARK-16494] [ML] Upgrade breeze version to 0.12
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14150 **[Test build #62157 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62157/consoleFull)** for PR 14150 at commit [`ba01c7e`](https://github.com/apache/spark/commit/ba01c7e1d369fecda8a80f4b77eb4ee663b73c07). * This patch **fails build dependency tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14150: [SPARK-16494] [ML] Upgrade breeze version to 0.12
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/14150 NVM... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14150: [SPARK-16494] [ML] Upgrade breeze version to 0.12
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/14150 @yanboliang you need to update the dependency manifests. Run: `./dev/test-dependencies.sh --replace-manifest` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14150: [SPARK-16494] [ML] Upgrade breeze version to 0.12
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14150 **[Test build #62157 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62157/consoleFull)** for PR 14150 at commit [`ba01c7e`](https://github.com/apache/spark/commit/ba01c7e1d369fecda8a80f4b77eb4ee663b73c07). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14150: [SPARK-16494] [ML] Upgrade breeze version to 0.12
Github user yanboliang commented on the issue: https://github.com/apache/spark/pull/14150 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
