[GitHub] spark pull request: [SPARK-14388][SQL][WIP] Implement CREATE TABLE
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/12271#discussion_r59111479
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
---
@@ -320,8 +320,6 @@ class DDLSuite extends QueryTest with SharedSQLContext
with BeforeAndAfterEach {
assertUnsupported("ALTER TABLE dbx.tab1 NOT STORED AS DIRECTORIES")
}
- // TODO: ADD a testcase for Drop Database in Restric when we can create
tables in SQLContext
--- End diff --
this TODO comment is duplicated in line 197
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/11242#discussion_r59111476
--- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
@@ -62,8 +62,14 @@ class UnionRDD[T: ClassTag](
var rdds: Seq[RDD[T]])
extends RDD[T](sc, Nil) { // Nil since we implement getDependencies
+ // visible for testing
+ private[spark] val isPartitionEvalParallel: Boolean =
+rdds.length > conf.getInt("spark.rdd.parallelListingThreshold", 10)
+
override def getPartitions: Array[Partition] = {
-val array = new Array[Partition](rdds.map(_.partitions.length).sum)
+val parRDDs = if (isPartitionEvalParallel) rdds.par else rdds
--- End diff --
just create a shared pool that can be used here.
no user code wouldn't necessarily slow down the operation. just imagine
somebody running the following:
```scala
(1 to 1000).par.foreach(_ => Thread.sleep(1))
```
suddenly nobody can use union anymore.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14388][SQL][WIP] Implement CREATE TABLE
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/12271#discussion_r59111469
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala ---
@@ -176,29 +176,6 @@ case class DescribeDatabase(
}
/**
- * A command that renames a table/view.
- *
- * The syntax of this command is:
- * {{{
- *ALTER TABLE table1 RENAME TO table2;
- *ALTER VIEW view1 RENAME TO view2;
- * }}}
- */
-case class AlterTableRename(
--- End diff --
moved to `tables.scala`. I just moved this one command for now to avoid
inflating the diff too much.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14388][SQL][WIP] Implement CREATE TABLE
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12271#issuecomment-207730021 **[Test build #55425 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55425/consoleFull)** for PR 12271 at commit [`5e0fe03`](https://github.com/apache/spark/commit/5e0fe03bfa655c6de854bc8adaa73186a17a0b0c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14388][SQL][WIP] Implement CREATE TABLE
GitHub user andrewor14 opened a pull request: https://github.com/apache/spark/pull/12271 [SPARK-14388][SQL][WIP] Implement CREATE TABLE ## What changes were proposed in this pull request? This patch implements the `CREATE TABLE` command using the `SessionCatalog`. Previously we handled only `CTAS` and `CREATE TABLE ... USING`. This requires us to refactor `CatalogTable` to accept various fields (e.g. bucket and skew columns) and pass them to Hive. ## How was this patch tested? Tests will come in a future commit. You can merge this pull request into a Git repository by running: $ git pull https://github.com/andrewor14/spark create-table-ddl Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/12271.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #12271 commit 014c38e28e8f4545f926ef60ccb2ee4acae07b59 Author: Andrew Or Date: 2016-04-08T21:32:39Z Parse various parts of the CREATE TABLE command We need to reconcile the differences between what's added here in SparkSqlParser and HiveSqlParser. That will come in the next commit. This currently still fails tests, obviously because create table is not implemented yet! commit 15bb3b6c76e61d708538bee5d797981689ab6a8f Author: Andrew Or Date: 2016-04-01T21:20:37Z Refactor CatalogTable column semantics Before: CatalogTable has schema, partitionColumns and sortColumns. There are no constraints between the 3. However, Hive will complain if schema and partitionColumns overlap. After: CatalogTable has schema, partitionColumnNames, sortColumnNames, bucketColumnNames and skewColumnNames. All the columns must be a subset of schema. This means splitting up schema into (schema, partitionCols) before passing it to Hive. This allows us to store the columns more uniformly. Otherwise partition columns would be the odd one out. This commit also fixes "alter table bucketing", which was incorrectly using partition columns as bucket columns. commit b6b4d293c2efeb537110ef56fa9ffdcad90c9bb0 Author: Andrew Or Date: 2016-04-09T00:53:18Z Implement CREATE TABLE in Hive parser This involves reverting part of the changes in an earlier commit, where we tried to implement the parsing logic in the general SQL parser and introduced a bunch of case classes that we won't end up using. As of this commit the actual CREATE TABLE logic is not there yet. It will come in a future commit. commit 5e0fe03bfa655c6de854bc8adaa73186a17a0b0c Author: Andrew Or Date: 2016-04-09T06:52:20Z Implement it --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14482][SQL] Change default Parquet code...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/12256 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14451][SQL] Move encoder definition int...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12231#discussion_r59111429
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/typedaggregators.scala
---
@@ -27,28 +27,20 @@ import org.apache.spark.sql.expressions.Aggregator
-class TypedSum[IN, OUT : Numeric](f: IN => OUT) extends Aggregator[IN,
OUT, OUT] {
- val numeric = implicitly[Numeric[OUT]]
- override def zero: OUT = numeric.zero
- override def reduce(b: OUT, a: IN): OUT = numeric.plus(b, f(a))
- override def merge(b1: OUT, b2: OUT): OUT = numeric.plus(b1, b2)
- override def finish(reduction: OUT): OUT = reduction
-
- // TODO(ekl) java api support once this is exposed in scala
-}
-
-
class TypedSumDouble[IN](f: IN => Double) extends Aggregator[IN, Double,
Double] {
override def zero: Double = 0.0
override def reduce(b: Double, a: IN): Double = b + f(a)
override def merge(b1: Double, b2: Double): Double = b1 + b2
override def finish(reduction: Double): Double = reduction
+ override def bufferEncoder: Encoder[Double] = ExpressionEncoder[Double]()
--- End diff --
this is internal, so it is not that bad to use the internal api.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14451][SQL] Move encoder definition int...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/12231#issuecomment-207729726 Merging in master. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14482][SQL] Change default Parquet code...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/12256#issuecomment-207729416 Merging in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14480][SQL] Simplify CSV parsing proces...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12268#issuecomment-207729065 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14480][SQL] Simplify CSV parsing proces...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12268#issuecomment-207729067 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55420/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14480][SQL] Simplify CSV parsing proces...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12268#issuecomment-207729042 **[Test build #55420 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55420/consoleFull)** for PR 12268 at commit [`b5a9669`](https://github.com/apache/spark/commit/b5a966962845011d6d56bb45d704d83eb5e06e38). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14372] [SQL] : Dataset.randomSplit() ne...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/12184#issuecomment-207725605 Looks pretty good to me. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14362] [SPARK-14406] [SQL] DDL Native S...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12146#issuecomment-207724257 **[Test build #55424 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55424/consoleFull)** for PR 12146 at commit [`f280ec4`](https://github.com/apache/spark/commit/f280ec4e539e6cae5ef687ae3916711beca08f6a). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14419] [SQL] Improve HashedRelation for...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12190#issuecomment-207716800 **[Test build #55423 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55423/consoleFull)** for PR 12190 at commit [`32c2165`](https://github.com/apache/spark/commit/32c2165e455fc1e1dd864b60a3f1b0b16991d50b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14419] [SQL] Improve HashedRelation for...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12190#issuecomment-207715638 **[Test build #55422 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55422/consoleFull)** for PR 12190 at commit [`5656379`](https://github.com/apache/spark/commit/5656379df2c37c17f108a990d5f98a705d7774ce). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14419] [SQL] Improve HashedRelation for...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12190#issuecomment-207715642 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55422/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14419] [SQL] Improve HashedRelation for...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12190#issuecomment-207715640 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14419] [SQL] Improve HashedRelation for...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12190#issuecomment-207715602 **[Test build #55422 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55422/consoleFull)** for PR 12190 at commit [`5656379`](https://github.com/apache/spark/commit/5656379df2c37c17f108a990d5f98a705d7774ce). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14419] [SQL] Improve HashedRelation for...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12190#issuecomment-207714622 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14419] [SQL] Improve HashedRelation for...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12190#issuecomment-207714624 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55421/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14419] [SQL] Improve HashedRelation for...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12190#issuecomment-207714621 **[Test build #55421 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55421/consoleFull)** for PR 12190 at commit [`028eec5`](https://github.com/apache/spark/commit/028eec539960b92a0b0188a8ba3d8f149230e5a5). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14480][SQL] Simplify CSV parsing proces...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12268#issuecomment-207714578 **[Test build #55420 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55420/consoleFull)** for PR 12268 at commit [`b5a9669`](https://github.com/apache/spark/commit/b5a966962845011d6d56bb45d704d83eb5e06e38). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14419] [SQL] Improve HashedRelation for...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12190#issuecomment-207714577 **[Test build #55421 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55421/consoleFull)** for PR 12190 at commit [`028eec5`](https://github.com/apache/spark/commit/028eec539960b92a0b0188a8ba3d8f149230e5a5). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14480][SQL] Simplify CSV parsing proces...
Github user HyukjinKwon commented on the pull request: https://github.com/apache/spark/pull/12268#issuecomment-207714445 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14481] [SQL] Issue Exceptions for All U...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12255#issuecomment-207709555 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14481] [SQL] Issue Exceptions for All U...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12255#issuecomment-207709559 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55417/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14481] [SQL] Issue Exceptions for All U...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12255#issuecomment-207709229 **[Test build #55417 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55417/consoleFull)** for PR 12255 at commit [`2440869`](https://github.com/apache/spark/commit/244086916ff95aa2b324ea19a044f773ef696431). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14362] [SPARK-14406] [SQL] DDL Native S...
Github user gatorsmile commented on the pull request: https://github.com/apache/spark/pull/12146#issuecomment-207709045 Actually, when we drop the external table, we do delete the data. If we do not want to delete the data, we need to change the following value from `true` to `false`, which is the @param `deleteData` (i.e., deletes the underlying data along with metadata) https://github.com/apache/spark/blob/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala#L345 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14462][ML][MLLIB] add the mllib-local b...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12241#issuecomment-207707531 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55414/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14462][ML][MLLIB] add the mllib-local b...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12241#issuecomment-207707525 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14462][ML][MLLIB] add the mllib-local b...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12241#issuecomment-207707201 **[Test build #55414 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55414/consoleFull)** for PR 12241 at commit [`4d6e394`](https://github.com/apache/spark/commit/4d6e394ae36044b97389d47d2b9dfcdbbfefde52). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14462][ML][MLLIB] add the mllib-local b...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12241#issuecomment-207706848 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55413/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14462][ML][MLLIB] add the mllib-local b...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12241#issuecomment-207706847 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14504] Enable Oracle docker tests
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12270#issuecomment-207706828 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14504] Enable Oracle docker tests
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12270#issuecomment-207706829 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55419/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14504] Enable Oracle docker tests
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12270#issuecomment-207706825 **[Test build #55419 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55419/consoleFull)** for PR 12270 at commit [`9f65ab3`](https://github.com/apache/spark/commit/9f65ab332370371abc32f81c3b240f9f93e00f0c). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13419] Update SubquerySuite to use chec...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12269#issuecomment-207706579 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55416/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14462][ML][MLLIB] add the mllib-local b...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12241#issuecomment-207706526 **[Test build #55413 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55413/consoleFull)** for PR 12241 at commit [`eba1e4d`](https://github.com/apache/spark/commit/eba1e4d1624f0be3c5d463b92fecf2b8a8134000). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13419] Update SubquerySuite to use chec...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12269#issuecomment-207706575 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13419] Update SubquerySuite to use chec...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12269#issuecomment-207706428 **[Test build #55416 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55416/consoleFull)** for PR 12269 at commit [`d0652c9`](https://github.com/apache/spark/commit/d0652c93560700ed86b7bf827a0f97aab3886176). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14480][SQL] Simplify CSV parsing proces...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12268#issuecomment-207706409 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55415/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14480][SQL] Simplify CSV parsing proces...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12268#issuecomment-207706408 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14480][SQL] Simplify CSV parsing proces...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12268#issuecomment-207706375 **[Test build #55415 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55415/consoleFull)** for PR 12268 at commit [`b5a9669`](https://github.com/apache/spark/commit/b5a966962845011d6d56bb45d704d83eb5e06e38). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14504] Enable Oracle docker tests
Github user lresende commented on the pull request: https://github.com/apache/spark/pull/12270#issuecomment-207706296 This PR is dependent on PR #9893 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14504] Enable Oracle docker tests
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12270#issuecomment-207706281 **[Test build #55419 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55419/consoleFull)** for PR 12270 at commit [`9f65ab3`](https://github.com/apache/spark/commit/9f65ab332370371abc32f81c3b240f9f93e00f0c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14504] Enable Oracle docker tests
GitHub user lresende opened a pull request: https://github.com/apache/spark/pull/12270 [SPARK-14504] Enable Oracle docker tests ## What changes were proposed in this pull request? Enable Oracle docker tests ## How was this patch tested? Existing tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/lresende/spark oracle Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/12270.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #12270 commit 1e3fcfee232e19f18e2155b85639207ba176ad8a Author: Luciano Resende Date: 2015-11-22T16:09:09Z [SPARK-10521][SQL] Utilize Docker for test DB2 JDBS Dialect support commit 97957529acb5ed7c3fd104ef9a622026e5069f4f Author: Luciano Resende Date: 2015-12-04T01:16:53Z [SPARK-10521] Update valid DB2 data types in test commit a7bade9b188a86007d351c18335ca0215f6be4ac Author: Luciano Resende Date: 2015-12-08T01:30:15Z [SPARK-10521] Add required parameters to start DB2 Docker Image commit 95e15bf52591aec297a90fe4b83bf6ddd3823d25 Author: Luciano Resende Date: 2015-12-16T22:58:00Z [SPARK-10521] Add DB2 dependencies to pom.xml The pom.xml was also updated with instructions on how to install the drivers on a local repository or directly into your own local cached maven. commit 844efd2eb7b7793c0007b420375e501670ddc0e0 Author: Luciano Resende Date: 2016-01-18T00:13:31Z [SPARK-10521] Change password for db2inst user Follow same password pattern as other db dialects to avoid warning about password containing parts of the username commit b5993f1cd4aad055c339f250e15f82ee023ee47f Author: Luciano Resende Date: 2016-01-18T00:19:19Z [SPARK-10521] Enable optional startup process configuration commit 8d9ee233ee6e9fb77e9d4f4a0884da623b753d41 Author: Luciano Resende Date: 2016-01-18T04:08:46Z [SPARK-10521] Update DB2 docker image used for test DB2 requires a DB to be created via DB2 tools before a JDBC connectiion can be established commit a8837bbbd7ceefe83ed8d34bd02a3b024266cfe2 Author: Luciano Resende Date: 2016-01-28T01:19:07Z [SPARK-10521] Fix scala style issues commit 2452aa3e9db15822b0bd06f102f93634cbfb3516 Author: Luciano Resende Date: 2016-01-28T01:19:44Z [SPARK-10521] Add DB2 repository commit ac3a2eeddc99cc6fe370ecfdf00743adb0b2fd42 Author: Luciano Resende Date: 2016-03-06T05:14:34Z [SPARK-10521] Update OracleIntegrationSuite with startup process name commit 70ded1e6a271a1dfc2b2ceb5a9f4b8340915819a Author: Luciano Resende Date: 2016-03-17T19:00:37Z [SPARK-10521] Refactor DB2 docker based tests to new project location commit b9bf63518c249f45f6cbf9c6e06871ade2b5b355 Author: Luciano Resende Date: 2016-03-23T21:59:05Z [SPARK-10521] Utilize ipcMode to avoid shared memory issues with DB2 Upgrade docker-client to latest version that has support for passing ipcMode configuration to docker which will fix DB2 shared memory issues. commit 9c872862afd1ada303dbbffdc09692b35a999b64 Author: Luciano Resende Date: 2016-03-25T00:23:16Z [SPARK-10521] Ignore test due to unsupported kernel version in build machine The AMPLab jenkins slaves are not using the recommended level of Linux kernel and this is causing issues trying to use shared memory thus failing to start DB2 server commit 31e1d8b5e99a477dcc2e63fa2e192650c2d3680d Author: Luciano Resende Date: 2016-04-09T01:23:58Z [SPARK-10521] Fix scala style issue commit 9f65ab332370371abc32f81c3b240f9f93e00f0c Author: Luciano Resende Date: 2016-04-09T04:24:38Z [SPARK-14504] Enable Oracle docker tests --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13600] [MLlib] Use approxQuantile from ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11553#issuecomment-207704973 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55418/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13600] [MLlib] Use approxQuantile from ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11553#issuecomment-207704960 **[Test build #55418 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55418/consoleFull)** for PR 11553 at commit [`5196c2d`](https://github.com/apache/spark/commit/5196c2d0402da946c40bb6b1d320f2cf771bea24). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13600] [MLlib] Use approxQuantile from ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11553#issuecomment-207704971 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13600] [MLlib] Use approxQuantile from ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11553#issuecomment-207704521 **[Test build #55418 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55418/consoleFull)** for PR 11553 at commit [`5196c2d`](https://github.com/apache/spark/commit/5196c2d0402da946c40bb6b1d320f2cf771bea24). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10521][SQL] Utilize Docker for test DB2...
Github user lresende commented on the pull request: https://github.com/apache/spark/pull/9893#issuecomment-207697108 @JoshRosen All good, style regression fixed, all tests passing. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13419] Update SubquerySuite to use chec...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12269#issuecomment-207695691 **[Test build #55416 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55416/consoleFull)** for PR 12269 at commit [`d0652c9`](https://github.com/apache/spark/commit/d0652c93560700ed86b7bf827a0f97aab3886176). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14481] [SQL] Issue Exceptions for All U...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12255#issuecomment-207695693 **[Test build #55417 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55417/consoleFull)** for PR 12255 at commit [`2440869`](https://github.com/apache/spark/commit/244086916ff95aa2b324ea19a044f773ef696431). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13419] Update SubquerySuite to use chec...
GitHub user lresende opened a pull request: https://github.com/apache/spark/pull/12269 [SPARK-13419] Update SubquerySuite to use checkAnswer to validate res… ## What changes were proposed in this pull request? Change SubquerySuite to validate test results utilizing checkAnswer helper method ## How was this patch tested? Existing tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/lresende/spark SPARK-13419 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/12269.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #12269 commit d0652c93560700ed86b7bf827a0f97aab3886176 Author: Luciano Resende Date: 2016-04-09T03:39:59Z [SPARK-13419] Update SubquerySuite to use checkAnswer to validate results --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14481] [SQL] Issue Exceptions for All U...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/12255#discussion_r59109752
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveSqlParser.scala
---
@@ -370,7 +373,8 @@ class HiveSqlAstBuilder extends SparkSqlAstBuilder {
ctx: TableFileFormatContext): CatalogStorageFormat = withOrigin(ctx)
{
import ctx._
if (inDriver != null || outDriver != null) {
- logWarning("INPUTDRIVER ... OUTPUTDRIVER ... clauses are ignored.")
--- End diff --
I will also change it to `Operation not allowed:`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14480][SQL] Simplify CSV parsing proces...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12268#issuecomment-207694215 **[Test build #55415 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55415/consoleFull)** for PR 12268 at commit [`b5a9669`](https://github.com/apache/spark/commit/b5a966962845011d6d56bb45d704d83eb5e06e38). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14480][SQL] Simplify CSV parsing proces...
Github user HyukjinKwon commented on the pull request: https://github.com/apache/spark/pull/12268#issuecomment-207694169 cc @rxin --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14480][SQL] Simplify CSV parsing proces...
Github user HyukjinKwon commented on the pull request: https://github.com/apache/spark/pull/12268#issuecomment-207694129 I have two things to note. - It looks `buildInternalScan()` is not called but just remaining. Just in case I did not remove this and tested this. - It looks the changes are a lot but the logics and filtering are not changed much except that it uses `Iterator` instead of `Reader`. - Performance was tested with this patch. This can be found in https://issues.apache.org/jira/browse/SPARK-14480. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14480][SQL] Simplify CSV parsing proces...
GitHub user HyukjinKwon opened a pull request: https://github.com/apache/spark/pull/12268 [SPARK-14480][SQL] Simplify CSV parsing process with a better performance ## What changes were proposed in this pull request? https://issues.apache.org/jira/browse/SPARK-14480 In `CSVParser.scala`, there is an `Reader` wrapping `Iterator` but there are two problems by this. Firstly, it was actually not faster than processing line by line with Iterator due to additional logics to wrap `Iterator` to `Reader`. Secondly, this brought a bit of complexity because it needs additional logics to allow every line to be read bytes by bytes. So, it was pretty difficult to figure out issues about parsing, (eg. SPARK-14103). This PR removes classes `CSVParser` and introduces new classes `UnivocityParser`, `UnivocityGenerator` and `CSVUtils` to be consistent with JSON data source (`JacksonParser`, `JacksonGenerator` and `JacksonUtils`). Also, `DefaultSource` moves to `CSVRelation` just like `JSONRelation`. To cut in short, this PR includes two changes, - Parse CSV data with `Iterator` not `Reader`. - Refactor CSV data source to be consistent with JSON data source. ## How was this patch tested? Existing tests should cover this. You can merge this pull request into a Git repository by running: $ git pull https://github.com/HyukjinKwon/spark SPARK-14480 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/12268.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #12268 commit 0eff7e7ccf9c8298f0969b28e87532b70ddafc2e Author: hyukjinkwon Date: 2016-04-09T03:27:13Z Simplify CSV parsing process with a better performance commit b5a966962845011d6d56bb45d704d83eb5e06e38 Author: hyukjinkwon Date: 2016-04-09T03:36:17Z Remove unintentionally added test codes --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10521][SQL] Utilize Docker for test DB2...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9893#issuecomment-207693639 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55411/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10521][SQL] Utilize Docker for test DB2...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9893#issuecomment-207693638 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10521][SQL] Utilize Docker for test DB2...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9893#issuecomment-207693594 **[Test build #55411 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55411/consoleFull)** for PR 9893 at commit [`31e1d8b`](https://github.com/apache/spark/commit/31e1d8b5e99a477dcc2e63fa2e192650c2d3680d). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14481] [SQL] Issue Exceptions for All U...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/12255#discussion_r59109505
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveSqlParser.scala
---
@@ -162,14 +162,16 @@ class HiveSqlAstBuilder extends SparkSqlAstBuilder {
// Unsupported clauses.
if (temp) {
-logWarning("TEMPORARY clause is ignored.")
+throw new ParseException(s"Unsupported operation: TEMPORARY
clause.", ctx)
}
if (ctx.bucketSpec != null) {
// TODO add this - we need cluster columns in the CatalogTable for
this to work.
-logWarning("CLUSTERED BY ... [ORDERED BY ...] INTO ... BUCKETS
clause is ignored.")
+throw new ParseException("Unsupported operation: " +
+ "CLUSTERED BY ... [ORDERED BY ...] INTO ... BUCKETS clause.",
ctx)
--- End diff --
I see. Thank you for letting me know it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14481] [SQL] Issue Exceptions for All U...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/12255#discussion_r59109484
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveSqlParser.scala
---
@@ -370,7 +373,8 @@ class HiveSqlAstBuilder extends SparkSqlAstBuilder {
ctx: TableFileFormatContext): CatalogStorageFormat = withOrigin(ctx)
{
import ctx._
if (inDriver != null || outDriver != null) {
- logWarning("INPUTDRIVER ... OUTPUTDRIVER ... clauses are ignored.")
--- End diff --
https://issues.apache.org/jira/browse/HIVE-1546 I checked the discussion.
Based on what Carl Steinbach and Ashutosh Chauhan said,
> Hive doesnt care about INPUTDRIVER and OUTPUTDRIVER
I also searched the code base of hive master. They are not used. Not sure
if we should completely delete them? or keep issuing the exceptions. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14498][ML][PYTHON][SQL] Many cleanups t...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/12266 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14498][ML][PYTHON][SQL] Many cleanups t...
Github user dbtsai commented on the pull request: https://github.com/apache/spark/pull/12266#issuecomment-207692381 Thanks. Merged into master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14455][Streaming] Fix NPE in allocatedE...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12236#issuecomment-207691107 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14455][Streaming] Fix NPE in allocatedE...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12236#issuecomment-207690851 **[Test build #55412 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55412/consoleFull)** for PR 12236 at commit [`4440605`](https://github.com/apache/spark/commit/4440605c9e0b73968f29e305fb95e00bb54151ee). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14455][Streaming] Fix NPE in allocatedE...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12236#issuecomment-207691109 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55412/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14462][ML][MLLIB] add the mllib-local b...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12241#issuecomment-207690112 **[Test build #55414 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55414/consoleFull)** for PR 12241 at commit [`4d6e394`](https://github.com/apache/spark/commit/4d6e394ae36044b97389d47d2b9dfcdbbfefde52). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14462][ML][MLLIB] add the mllib-local b...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12241#issuecomment-207689839 **[Test build #55413 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55413/consoleFull)** for PR 12241 at commit [`eba1e4d`](https://github.com/apache/spark/commit/eba1e4d1624f0be3c5d463b92fecf2b8a8134000). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13597][PySpark][ML] Python API for Gene...
Github user vectorijk commented on the pull request: https://github.com/apache/spark/pull/11468#issuecomment-207689752 @yanboliang Thanks! I have addressed your comments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14354][SQL] Let Expand take name expres...
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/12138#issuecomment-207689521 If my understanding is right, the problem is: when child output changes(e.g. making them null when doing a roll up), the output of `Expand` can't reflect it. I have a simpler idea, when set the output for `Expand`, use a placeholder to reference to a child column, and define a `output` method in `Expand` that replace the placeholder with child attribute. How about it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14354][SQL] Let Expand take name expres...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/12138#discussion_r59109073
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicOperators.scala
---
@@ -512,24 +512,35 @@ private[sql] object Expand {
* a input row.
*
* @param projections to apply
- * @param output of all projections.
* @param child operator.
*/
case class Expand(
-projections: Seq[Seq[Expression]],
-output: Seq[Attribute],
+projections: Seq[Seq[NamedExpression]],
child: LogicalPlan) extends UnaryNode {
+ override def output: Seq[Attribute] = {
+// Take the first projection as output
+val preOutput = projections.head.map(_.toAttribute)
--- End diff --
This seems a waste that we make all projections `Seq[NamedExpression]`, but
only use the first one to produce attribute.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14481] [SQL] Issue Exceptions for All U...
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/12255#discussion_r59109029 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveCommandSuite.scala --- @@ -26,7 +26,7 @@ class HiveCommandSuite extends QueryTest with SQLTestUtils with TestHiveSingleto super.beforeAll() sql( """ -|CREATE EXTERNAL TABLE parquet_tab1 (c1 INT, c2 STRING) --- End diff -- oh, right. I did not notice the USING part. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14481] [SQL] Issue Exceptions for All U...
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/12255#discussion_r59108991 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveCommandSuite.scala --- @@ -26,7 +26,7 @@ class HiveCommandSuite extends QueryTest with SQLTestUtils with TestHiveSingleto super.beforeAll() sql( """ -|CREATE EXTERNAL TABLE parquet_tab1 (c1 INT, c2 STRING) --- End diff -- https://github.com/apache/spark/pull/12255/files#diff-1bb4f7bd5a2656f48bcd3c857167a11bR206 will issue an exception. Thus, we have to remove `EXTERNAL` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14481] [SQL] Issue Exceptions for All U...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/12255#discussion_r59108971
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveSqlParser.scala
---
@@ -162,14 +162,16 @@ class HiveSqlAstBuilder extends SparkSqlAstBuilder {
// Unsupported clauses.
if (temp) {
-logWarning("TEMPORARY clause is ignored.")
+throw new ParseException(s"Unsupported operation: TEMPORARY
clause.", ctx)
}
if (ctx.bucketSpec != null) {
// TODO add this - we need cluster columns in the CatalogTable for
this to work.
-logWarning("CLUSTERED BY ... [ORDERED BY ...] INTO ... BUCKETS
clause is ignored.")
+throw new ParseException("Unsupported operation: " +
+ "CLUSTERED BY ... [ORDERED BY ...] INTO ... BUCKETS clause.",
ctx)
}
if (ctx.skewSpec != null) {
-logWarning("SKEWED BY ... ON ... [STORED AS DIRECTORIES] clause is
ignored.")
+throw new ParseException("Unsupported operation: " +
+ "SKEWED BY ... ON ... [STORED AS DIRECTORIES] clause.", ctx)
--- End diff --
Ok, sure, will do
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14455][Streaming] Fix NPE in allocatedE...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12236#issuecomment-207684879 **[Test build #55412 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55412/consoleFull)** for PR 12236 at commit [`4440605`](https://github.com/apache/spark/commit/4440605c9e0b73968f29e305fb95e00bb54151ee). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14455][Streaming] Fix NPE in allocatedE...
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/12236#discussion_r59108524
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/scheduler/ReceiverTrackerSuite.scala
---
@@ -102,6 +102,31 @@ class ReceiverTrackerSuite extends TestSuiteBase {
}
}
}
+
+ test("get allocated executors") {
+// Test get allocated executors when 1 receiver is registered
+withStreamingContext(new StreamingContext(conf, Milliseconds(100))) {
ssc =>
+ val input = ssc.receiverStream(new TestReceiver)
+ val output = new TestOutputStream(input)
+ output.register()
+ ssc.start()
+ eventually(timeout(10 seconds), interval(10 millis)) {
+assert(ssc.scheduler.receiverTracker.allocatedExecutors().size ===
1)
+ }
+}
+
+// Test get allocated executors when there's no receiver registered
+withStreamingContext(new StreamingContext(conf, Milliseconds(100))) {
ssc =>
+ val rdd = ssc.sc.parallelize(1 to 10)
+ val input = new ConstantInputDStream(ssc, rdd)
+ val output = new TestOutputStream(input)
+ output.register()
+ ssc.start()
+ eventually(timeout(10 seconds), interval(10 millis)) {
+assert(ssc.scheduler.receiverTracker.allocatedExecutors() ===
Map.empty)
--- End diff --
>Before any receiver starts,
ssc.scheduler.receiverTracker.allocatedExecutors will return Map.empty as well.
I see your point, this might cause streaming dynamic allocation to kill the
executor which has a receiver to be started unexpectedly. This might be
happened in some extreme cases.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14487][SQL][WIP] User Defined Type regi...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12259#issuecomment-207680955 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14487][SQL][WIP] User Defined Type regi...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12259#issuecomment-207680956 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55407/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14487][SQL][WIP] User Defined Type regi...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12259#issuecomment-207680872 **[Test build #55407 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55407/consoleFull)** for PR 12259 at commit [`e59f1d5`](https://github.com/apache/spark/commit/e59f1d5757189ad7535a189c4f20064b193c8242). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class UDTRegistrationSuite extends QueryTest with SharedSQLContext with ParquetTest ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14455][Streaming] Fix NPE in allocatedE...
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/12236#discussion_r59107996
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/scheduler/ReceiverTrackerSuite.scala
---
@@ -102,6 +102,31 @@ class ReceiverTrackerSuite extends TestSuiteBase {
}
}
}
+
+ test("get allocated executors") {
+// Test get allocated executors when 1 receiver is registered
+withStreamingContext(new StreamingContext(conf, Milliseconds(100))) {
ssc =>
+ val input = ssc.receiverStream(new TestReceiver)
+ val output = new TestOutputStream(input)
+ output.register()
+ ssc.start()
+ eventually(timeout(10 seconds), interval(10 millis)) {
+assert(ssc.scheduler.receiverTracker.allocatedExecutors().size ===
1)
+ }
+}
+
+// Test get allocated executors when there's no receiver registered
+withStreamingContext(new StreamingContext(conf, Milliseconds(100))) {
ssc =>
+ val rdd = ssc.sc.parallelize(1 to 10)
+ val input = new ConstantInputDStream(ssc, rdd)
+ val output = new TestOutputStream(input)
+ output.register()
+ ssc.start()
+ eventually(timeout(10 seconds), interval(10 millis)) {
+assert(ssc.scheduler.receiverTracker.allocatedExecutors() ===
Map.empty)
--- End diff --
Alright, let me change it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP, DO-NOT-MERGE][SPARK-14473][SQL] Define a...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12246#issuecomment-207678192 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP, DO-NOT-MERGE][SPARK-14473][SQL] Define a...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12246#issuecomment-207678194 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55406/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP, DO-NOT-MERGE][SPARK-14473][SQL] Define a...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12246#issuecomment-207678063 **[Test build #55406 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55406/consoleFull)** for PR 12246 at commit [`bbf9f27`](https://github.com/apache/spark/commit/bbf9f27dbd72de5efef3c22711f7a57cfd04fe17). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10521][SQL] Utilize Docker for test DB2...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9893#issuecomment-207677219 **[Test build #55411 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55411/consoleFull)** for PR 9893 at commit [`31e1d8b`](https://github.com/apache/spark/commit/31e1d8b5e99a477dcc2e63fa2e192650c2d3680d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14462][ML][MLLIB] add the mllib-local b...
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/12241#discussion_r59107331
--- Diff: mllib/pom.xml ---
@@ -63,25 +63,20 @@
${project.version}
- org.scalanlp
--- End diff --
Please keep the breeze dependency in MLlib's pom because we call its APIs
directly. We can move versions to a common place.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10521][SQL] Utilize Docker for test DB2...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9893#issuecomment-207676529 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55410/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10521][SQL] Utilize Docker for test DB2...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9893#issuecomment-207676526 **[Test build #55410 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55410/consoleFull)** for PR 9893 at commit [`9c87286`](https://github.com/apache/spark/commit/9c872862afd1ada303dbbffdc09692b35a999b64). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10521][SQL] Utilize Docker for test DB2...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9893#issuecomment-207676528 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10521][SQL] Utilize Docker for test DB2...
Github user lresende commented on the pull request: https://github.com/apache/spark/pull/9893#issuecomment-207676407 @JoshRosen, thanks for reviewing, I agree with your proposal regarding the repository. Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10521][SQL] Utilize Docker for test DB2...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9893#issuecomment-207676411 **[Test build #55410 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55410/consoleFull)** for PR 9893 at commit [`9c87286`](https://github.com/apache/spark/commit/9c872862afd1ada303dbbffdc09692b35a999b64). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10521][SQL] Utilize Docker for test DB2...
Github user lresende commented on the pull request: https://github.com/apache/spark/pull/9893#issuecomment-207675856 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10521][SQL] Utilize Docker for test DB2...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9893#issuecomment-207675490 **[Test build #55409 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55409/consoleFull)** for PR 9893 at commit [`9c87286`](https://github.com/apache/spark/commit/9c872862afd1ada303dbbffdc09692b35a999b64). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10521][SQL] Utilize Docker for test DB2...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9893#issuecomment-207675496 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55409/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10521][SQL] Utilize Docker for test DB2...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9893#issuecomment-207675495 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10521][SQL] Utilize Docker for test DB2...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9893#issuecomment-207675376 **[Test build #55409 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55409/consoleFull)** for PR 9893 at commit [`9c87286`](https://github.com/apache/spark/commit/9c872862afd1ada303dbbffdc09692b35a999b64). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14454] Better exception handling while ...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/12234#issuecomment-207673039 @sameeragarwal can you create one for 1.6 backport? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
