[GitHub] spark issue #19618: [SPARK-5484][Followup] PeriodicRDDCheckpointer doc clean...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19618 **[Test build #83245 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83245/testReport)** for PR 19618 at commit [`2858cbb`](https://github.com/apache/spark/commit/2858cbb5c8264d7bee592835b56a415961ed1dc4). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19601: [SPARK-22383][SQL] Generate code to directly get value o...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19601 **[Test build #83246 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83246/testReport)** for PR 19601 at commit [`b971506`](https://github.com/apache/spark/commit/b971506f8d5138a2c23e039427d547b736079c13). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16677: [SPARK-19355][SQL] Use map output statistics to improve ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16677 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16677: [SPARK-19355][SQL] Use map output statistics to improve ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16677 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83249/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16677: [SPARK-19355][SQL] Use map output statistics to improve ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16677 **[Test build #83249 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83249/testReport)** for PR 16677 at commit [`e53648e`](https://github.com/apache/spark/commit/e53648e7f58f439bb09a702521c2f84cf2e344bd). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19618: [SPARK-5484][Followup] PeriodicRDDCheckpointer doc clean...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19618 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83245/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19618: [SPARK-5484][Followup] PeriodicRDDCheckpointer doc clean...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19618 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19601: [SPARK-22383][SQL] Generate code to directly get value o...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19601 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19601: [SPARK-22383][SQL] Generate code to directly get value o...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19601 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83246/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19601: [SPARK-22383][SQL] Generate code to directly get value o...
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/19601 Jenkins, retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16677: [SPARK-19355][SQL] Use map output statistics to improve ...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/16677 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19601: [SPARK-22383][SQL] Generate code to directly get value o...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19601 **[Test build #83250 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83250/testReport)** for PR 19601 at commit [`b971506`](https://github.com/apache/spark/commit/b971506f8d5138a2c23e039427d547b736079c13). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16677: [SPARK-19355][SQL] Use map output statistics to improve ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16677 **[Test build #83251 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83251/testReport)** for PR 16677 at commit [`e53648e`](https://github.com/apache/spark/commit/e53648e7f58f439bb09a702521c2f84cf2e344bd). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19531: [SPARK-22310] [SQL] Refactor join estimation to incorpor...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19531 **[Test build #83252 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83252/testReport)** for PR 19531 at commit [`a2dbb8e`](https://github.com/apache/spark/commit/a2dbb8eefacf560ff5e7090baa3796b2353daed2). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15178: [SPARK-17556][SQL] Executor side broadcast for broadcast...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15178 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15178: [SPARK-17556][SQL] Executor side broadcast for broadcast...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15178 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83253/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15178: [SPARK-17556][SQL] Executor side broadcast for broadcast...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/15178 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15178: [SPARK-17556][SQL] Executor side broadcast for broadcast...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15178 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15178: [SPARK-17556][SQL] Executor side broadcast for broadcast...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15178 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83254/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19553: [SPARK-22330][CORE] Linear containsKey operation ...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/19553#discussion_r147918030
--- Diff: core/src/main/scala/org/apache/spark/api/java/JavaUtils.scala ---
@@ -43,10 +43,17 @@ private[spark] object JavaUtils {
override def size: Int = underlying.size
-override def get(key: AnyRef): B = try {
- underlying.getOrElse(key.asInstanceOf[A], null.asInstanceOf[B])
-} catch {
- case ex: ClassCastException => null.asInstanceOf[B]
+// Delegate to implementation because AbstractMap implementation
iterates over whole key set
+override def containsKey(key: AnyRef): Boolean = {
+ underlying.contains(key.asInstanceOf[A])
--- End diff --
No, because Scala Maps do require their key type in contains() and get().
https://www.scala-lang.org/api/current/scala/collection/Map.html . This is
indeed the difference between the APIs that we need to bridge.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15178: [SPARK-17556][SQL] Executor side broadcast for broadcast...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/15178 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19609: [SPARK-21708][Build] update some sbt plugins
Github user srowen commented on the issue: https://github.com/apache/spark/pull/19609 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15178: [SPARK-17556][SQL] Executor side broadcast for broadcast...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15178 **[Test build #83255 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83255/testReport)** for PR 15178 at commit [`a1f2faa`](https://github.com/apache/spark/commit/a1f2faa99f7b6875885820893070c8b039ecddd7). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19609: [SPARK-21708][Build] update some sbt plugins
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19609 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19614: [SPARK-22399][ML] update the location of reference paper
Github user srowen commented on the issue: https://github.com/apache/spark/pull/19614 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19614: [SPARK-22399][ML] update the location of referenc...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19614 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18118: [SPARK-20199][ML] : Provided featureSubsetStrategy to GB...
Github user pralabhkumar commented on the issue: https://github.com/apache/spark/pull/18118 @MLnick @sethah please find some time to look into this --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19580: [SPARK-11334][CORE] Fix bug in Executor allocation manag...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19580 jenkins, retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19580: [SPARK-11334][CORE] Fix bug in Executor allocation manag...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19580 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19580: [SPARK-11334][CORE] Fix bug in Executor allocation manag...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19580 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83256/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19605: [SPARK-22394] [SQL] Remove redundant synchronizat...
Github user wzhfy commented on a diff in the pull request:
https://github.com/apache/spark/pull/19605#discussion_r147929746
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala ---
@@ -89,10 +89,12 @@ private[spark] class HiveExternalCatalog(conf:
SparkConf, hadoopConf: Configurat
}
/**
- * Run some code involving `client` in a [[synchronized]] block and wrap
certain
- * exceptions thrown in the process in [[AnalysisException]].
+ * Run some code involving `client` and wrap certain exceptions thrown
in the process in
+ * [[AnalysisException]]. Thread-safety is guaranteed here because
methods in the `client`
+ * ([[org.apache.spark.sql.hive.client.HiveClientImpl]]) are already
synchronized through
+ * `clientLoader` in the `retryLocked` method.
*/
- private def withClient[T](body: => T): T = synchronized {
+ private def withClient[T](body: => T): T = {
--- End diff --
I went through all methods in `HiveClient` having synchronization (except
`addJar`):
- `getState` is used only in test.
- `setOut`, `setInfo` and `setError` are only used in `SparkSQLEnv.init()`.
- all other methods are called through `HiveExternalCatalog`.
So it seems `addJar` is the only exception.
To make `addJar` also go throught `HiveExternalCatalog`, we can pass
`externalCatalog` instead of `client` at [line46 in
HiveSessionStateBuilder](https://github.com/apache/spark/blob/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionStateBuilder.scala#L46).
But I don't know why we need to call `newSession()` at
[line45](https://github.com/apache/spark/blob/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionStateBuilder.scala#L45),
where a new `HiveClientImpl` instance is created, with the same class loader
and Hive client.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19613: Fixed a typo
Github user srowen commented on the issue: https://github.com/apache/spark/pull/19613 Or in UnsafeArrayData, corrresponding -> corresponding If you just run a spell check of comments in an IDE, I'm sure a quick pass would reveal a lot of little typos. The more you are willing to fix in one pass, the better. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19615: [SPARK-19611][SQL][followup] set dataSchema correctly in...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19615 also cc @dongjoon-hyun @viirya --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19601: [SPARK-22383][SQL] Generate code to directly get value o...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19601 **[Test build #83250 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83250/testReport)** for PR 19601 at commit [`b971506`](https://github.com/apache/spark/commit/b971506f8d5138a2c23e039427d547b736079c13). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19601: [SPARK-22383][SQL] Generate code to directly get value o...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19601 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19601: [SPARK-22383][SQL] Generate code to directly get value o...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19601 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83250/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19531: [SPARK-22310] [SQL] Refactor join estimation to incorpor...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19531 LGTM, merging to master! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19531: [SPARK-22310] [SQL] Refactor join estimation to i...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19531 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19479: [SPARK-17074] [SQL] Generate equi-height histogra...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19479#discussion_r147946241
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala ---
@@ -1032,7 +1032,21 @@ private[spark] class HiveExternalCatalog(conf:
SparkConf, hadoopConf: Configurat
schema.fields.map(f => (f.name, f.dataType)).toMap
stats.colStats.foreach { case (colName, colStat) =>
colStat.toMap(colName, colNameTypeMap(colName)).foreach { case (k,
v) =>
-statsProperties += (columnStatKeyPropName(colName, k) -> v)
+if (k == ColumnStat.KEY_HISTOGRAM) {
+ // In Hive metastore, the length of value in table properties
cannot be larger than 4000,
+ // so we need to split histogram into multiple key-value
properties if it's too long.
+ val maxValueLen = 4000
--- End diff --
use `SCHEMA_STRING_LENGTH_THRESHOLD` instead of hardcode, please follow
`tableMetaToTableProps`.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19479: [SPARK-17074] [SQL] Generate equi-height histogra...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19479#discussion_r147946445
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala ---
@@ -1032,7 +1032,21 @@ private[spark] class HiveExternalCatalog(conf:
SparkConf, hadoopConf: Configurat
schema.fields.map(f => (f.name, f.dataType)).toMap
stats.colStats.foreach { case (colName, colStat) =>
colStat.toMap(colName, colNameTypeMap(colName)).foreach { case (k,
v) =>
-statsProperties += (columnStatKeyPropName(colName, k) -> v)
+if (k == ColumnStat.KEY_HISTOGRAM) {
--- End diff --
don't special-case histogram, we should do it for all stats.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19479: [SPARK-17074] [SQL] Generate equi-height histogra...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19479#discussion_r147947097
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala ---
@@ -1032,7 +1032,21 @@ private[spark] class HiveExternalCatalog(conf:
SparkConf, hadoopConf: Configurat
schema.fields.map(f => (f.name, f.dataType)).toMap
stats.colStats.foreach { case (colName, colStat) =>
colStat.toMap(colName, colNameTypeMap(colName)).foreach { case (k,
v) =>
-statsProperties += (columnStatKeyPropName(colName, k) -> v)
+if (k == ColumnStat.KEY_HISTOGRAM) {
--- End diff --
just make sure if the limitation is not hit, we still generate a simple
key-value.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19531: [SPARK-22310] [SQL] Refactor join estimation to incorpor...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19531 **[Test build #83252 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83252/testReport)** for PR 19531 at commit [`a2dbb8e`](https://github.com/apache/spark/commit/a2dbb8eefacf560ff5e7090baa3796b2353daed2). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19531: [SPARK-22310] [SQL] Refactor join estimation to incorpor...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19531 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83252/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19531: [SPARK-22310] [SQL] Refactor join estimation to incorpor...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19531 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19615: [SPARK-19611][SQL][followup] set dataSchema correctly in...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/19615 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19615: [SPARK-19611][SQL][followup] set dataSchema corre...
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/19615#discussion_r147951101 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala --- @@ -164,13 +164,12 @@ private[hive] class HiveMetastoreCatalog(sparkSession: SparkSession) extends Log } } - val (dataSchema, updatedTable) = -inferIfNeeded(relation, options, fileFormat, Option(fileIndex)) + val updatedTable = inferIfNeeded(relation, options, fileFormat, Option(fileIndex)) --- End diff -- Yeah, we missed this. Actually the variable name is indicating it is data schema... --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19615: [SPARK-19611][SQL][followup] set dataSchema correctly in...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19615 thanks, merging to master/2.2! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19615: [SPARK-19611][SQL][followup] set dataSchema corre...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19615 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16677: [SPARK-19355][SQL] Use map output statistics to improve ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16677 **[Test build #83251 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83251/testReport)** for PR 16677 at commit [`e53648e`](https://github.com/apache/spark/commit/e53648e7f58f439bb09a702521c2f84cf2e344bd). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16677: [SPARK-19355][SQL] Use map output statistics to improve ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16677 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16677: [SPARK-19355][SQL] Use map output statistics to improve ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16677 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83251/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator f...
Github user zhengruifeng commented on a diff in the pull request:
https://github.com/apache/spark/pull/19527#discussion_r147950431
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/feature/OneHotEncoderEstimator.scala
---
@@ -0,0 +1,462 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.feature
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.SparkException
+import org.apache.spark.annotation.Since
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.attribute._
+import org.apache.spark.ml.linalg.Vectors
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasHandleInvalid, HasInputCols,
HasOutputCols}
+import org.apache.spark.ml.util._
+import org.apache.spark.sql.{DataFrame, Dataset}
+import org.apache.spark.sql.expressions.UserDefinedFunction
+import org.apache.spark.sql.functions.{col, lit, udf}
+import org.apache.spark.sql.types.{DoubleType, NumericType, StructField,
StructType}
+
+/** Private trait for params and common methods for OneHotEncoderEstimator
and OneHotEncoderModel */
+private[ml] trait OneHotEncoderBase extends Params with HasHandleInvalid
+with HasInputCols with HasOutputCols {
+
+ /**
+ * Param for how to handle invalid data.
+ * Options are 'keep' (invalid data produces a vector of zeros) or
'error' (throw an error).
+ * Default: "error"
+ * @group param
+ */
+ @Since("2.3.0")
+ override val handleInvalid: Param[String] = new Param[String](this,
"handleInvalid",
+"How to handle invalid data " +
+"Options are 'keep' (invalid data produces a vector of zeros) or error
(throw an error).",
+
ParamValidators.inArray(OneHotEncoderEstimator.supportedHandleInvalids))
+
+ setDefault(handleInvalid, OneHotEncoderEstimator.ERROR_INVALID)
+
+ /**
+ * Whether to drop the last category in the encoded vector (default:
true)
+ * @group param
+ */
+ @Since("2.3.0")
+ final val dropLast: BooleanParam =
+new BooleanParam(this, "dropLast", "whether to drop the last category")
+ setDefault(dropLast -> true)
+
+ /** @group getParam */
+ @Since("2.3.0")
+ def getDropLast: Boolean = $(dropLast)
+
+ protected def checkParamsValidity(schema: StructType): Unit = {
--- End diff --
Can we use a single function `validateAndTransformSchema` instead? like
other algorithms
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator f...
Github user zhengruifeng commented on a diff in the pull request:
https://github.com/apache/spark/pull/19527#discussion_r147950931
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/feature/OneHotEncoderEstimator.scala
---
@@ -0,0 +1,462 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.feature
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.SparkException
+import org.apache.spark.annotation.Since
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.attribute._
+import org.apache.spark.ml.linalg.Vectors
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasHandleInvalid, HasInputCols,
HasOutputCols}
+import org.apache.spark.ml.util._
+import org.apache.spark.sql.{DataFrame, Dataset}
+import org.apache.spark.sql.expressions.UserDefinedFunction
+import org.apache.spark.sql.functions.{col, lit, udf}
+import org.apache.spark.sql.types.{DoubleType, NumericType, StructField,
StructType}
+
+/** Private trait for params and common methods for OneHotEncoderEstimator
and OneHotEncoderModel */
+private[ml] trait OneHotEncoderBase extends Params with HasHandleInvalid
+with HasInputCols with HasOutputCols {
+
+ /**
+ * Param for how to handle invalid data.
+ * Options are 'keep' (invalid data produces a vector of zeros) or
'error' (throw an error).
+ * Default: "error"
+ * @group param
+ */
+ @Since("2.3.0")
+ override val handleInvalid: Param[String] = new Param[String](this,
"handleInvalid",
+"How to handle invalid data " +
+"Options are 'keep' (invalid data produces a vector of zeros) or error
(throw an error).",
+
ParamValidators.inArray(OneHotEncoderEstimator.supportedHandleInvalids))
+
+ setDefault(handleInvalid, OneHotEncoderEstimator.ERROR_INVALID)
+
+ /**
+ * Whether to drop the last category in the encoded vector (default:
true)
+ * @group param
+ */
+ @Since("2.3.0")
+ final val dropLast: BooleanParam =
+new BooleanParam(this, "dropLast", "whether to drop the last category")
+ setDefault(dropLast -> true)
+
+ /** @group getParam */
+ @Since("2.3.0")
+ def getDropLast: Boolean = $(dropLast)
+
+ protected def checkParamsValidity(schema: StructType): Unit = {
+val inputColNames = $(inputCols)
+val outputColNames = $(outputCols)
+val existingFields = schema.fields
+
+require(inputColNames.length == outputColNames.length,
--- End diff --
it's better to add a checking for duplicated columns, like what `Imputer`
and `Interaction` do, IMO
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator f...
Github user zhengruifeng commented on a diff in the pull request:
https://github.com/apache/spark/pull/19527#discussion_r147953998
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/feature/OneHotEncoderEstimator.scala
---
@@ -0,0 +1,462 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.feature
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.SparkException
+import org.apache.spark.annotation.Since
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.attribute._
+import org.apache.spark.ml.linalg.Vectors
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasHandleInvalid, HasInputCols,
HasOutputCols}
+import org.apache.spark.ml.util._
+import org.apache.spark.sql.{DataFrame, Dataset}
+import org.apache.spark.sql.expressions.UserDefinedFunction
+import org.apache.spark.sql.functions.{col, lit, udf}
+import org.apache.spark.sql.types.{DoubleType, NumericType, StructField,
StructType}
+
+/** Private trait for params and common methods for OneHotEncoderEstimator
and OneHotEncoderModel */
+private[ml] trait OneHotEncoderBase extends Params with HasHandleInvalid
+with HasInputCols with HasOutputCols {
+
+ /**
+ * Param for how to handle invalid data.
+ * Options are 'keep' (invalid data produces a vector of zeros) or
'error' (throw an error).
+ * Default: "error"
+ * @group param
+ */
+ @Since("2.3.0")
+ override val handleInvalid: Param[String] = new Param[String](this,
"handleInvalid",
+"How to handle invalid data " +
+"Options are 'keep' (invalid data produces a vector of zeros) or error
(throw an error).",
+
ParamValidators.inArray(OneHotEncoderEstimator.supportedHandleInvalids))
+
+ setDefault(handleInvalid, OneHotEncoderEstimator.ERROR_INVALID)
+
+ /**
+ * Whether to drop the last category in the encoded vector (default:
true)
+ * @group param
+ */
+ @Since("2.3.0")
+ final val dropLast: BooleanParam =
+new BooleanParam(this, "dropLast", "whether to drop the last category")
+ setDefault(dropLast -> true)
+
+ /** @group getParam */
+ @Since("2.3.0")
+ def getDropLast: Boolean = $(dropLast)
+
+ protected def checkParamsValidity(schema: StructType): Unit = {
+val inputColNames = $(inputCols)
+val outputColNames = $(outputCols)
+val existingFields = schema.fields
+
+require(inputColNames.length == outputColNames.length,
+ s"The number of input columns ${inputColNames.length} must be the
same as the number of " +
+s"output columns ${outputColNames.length}.")
+
+inputColNames.zip(outputColNames).map { case (inputColName,
outputColName) =>
+ require(schema(inputColName).dataType.isInstanceOf[NumericType],
+s"Input column must be of type NumericType but got
${schema(inputColName).dataType}")
+ require(!existingFields.exists(_.name == outputColName),
+s"Output column $outputColName already exists.")
+}
+ }
+
+ /** Prepares output columns with proper attributes by examining input
columns. */
+ protected def prepareSchemaWithOutputField(schema: StructType):
StructType = {
+val inputFields = $(inputCols).map(schema(_))
+val outputColNames = $(outputCols)
+
+val outputFields = inputFields.zip(outputColNames).map { case
(inputField, outputColName) =>
+ OneHotEncoderCommon.transformOutputColumnSchema(
+inputField, $(dropLast), outputColName)
+}
+StructType(schema.fields ++ outputFields)
+ }
+}
+
+/**
+ * A one-hot encoder that maps a column of category indices to a column of
binary vectors, with
+ * at most a single one-value per row that indicates the input category
index.
+ * For example with 5 categories, an input value of 2.0 would map to an
output vector of
+ * `[0.0, 0.0, 1.0, 0.0]`.
+ * The l
[GitHub] spark issue #15178: [SPARK-17556][SQL] Executor side broadcast for broadcast...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15178 **[Test build #83255 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83255/testReport)** for PR 15178 at commit [`a1f2faa`](https://github.com/apache/spark/commit/a1f2faa99f7b6875885820893070c8b039ecddd7). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15178: [SPARK-17556][SQL] Executor side broadcast for broadcast...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15178 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83255/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15178: [SPARK-17556][SQL] Executor side broadcast for broadcast...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15178 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator f...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/19527#discussion_r147959976
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/feature/OneHotEncoderEstimator.scala
---
@@ -0,0 +1,462 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.feature
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.SparkException
+import org.apache.spark.annotation.Since
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.attribute._
+import org.apache.spark.ml.linalg.Vectors
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasHandleInvalid, HasInputCols,
HasOutputCols}
+import org.apache.spark.ml.util._
+import org.apache.spark.sql.{DataFrame, Dataset}
+import org.apache.spark.sql.expressions.UserDefinedFunction
+import org.apache.spark.sql.functions.{col, lit, udf}
+import org.apache.spark.sql.types.{DoubleType, NumericType, StructField,
StructType}
+
+/** Private trait for params and common methods for OneHotEncoderEstimator
and OneHotEncoderModel */
+private[ml] trait OneHotEncoderBase extends Params with HasHandleInvalid
+with HasInputCols with HasOutputCols {
+
+ /**
+ * Param for how to handle invalid data.
+ * Options are 'keep' (invalid data produces a vector of zeros) or
'error' (throw an error).
+ * Default: "error"
+ * @group param
+ */
+ @Since("2.3.0")
+ override val handleInvalid: Param[String] = new Param[String](this,
"handleInvalid",
+"How to handle invalid data " +
+"Options are 'keep' (invalid data produces a vector of zeros) or error
(throw an error).",
+
ParamValidators.inArray(OneHotEncoderEstimator.supportedHandleInvalids))
+
+ setDefault(handleInvalid, OneHotEncoderEstimator.ERROR_INVALID)
+
+ /**
+ * Whether to drop the last category in the encoded vector (default:
true)
+ * @group param
+ */
+ @Since("2.3.0")
+ final val dropLast: BooleanParam =
+new BooleanParam(this, "dropLast", "whether to drop the last category")
+ setDefault(dropLast -> true)
+
+ /** @group getParam */
+ @Since("2.3.0")
+ def getDropLast: Boolean = $(dropLast)
+
+ protected def checkParamsValidity(schema: StructType): Unit = {
+val inputColNames = $(inputCols)
+val outputColNames = $(outputCols)
+val existingFields = schema.fields
+
+require(inputColNames.length == outputColNames.length,
+ s"The number of input columns ${inputColNames.length} must be the
same as the number of " +
+s"output columns ${outputColNames.length}.")
+
+inputColNames.zip(outputColNames).map { case (inputColName,
outputColName) =>
+ require(schema(inputColName).dataType.isInstanceOf[NumericType],
+s"Input column must be of type NumericType but got
${schema(inputColName).dataType}")
+ require(!existingFields.exists(_.name == outputColName),
+s"Output column $outputColName already exists.")
+}
+ }
+
+ /** Prepares output columns with proper attributes by examining input
columns. */
+ protected def prepareSchemaWithOutputField(schema: StructType):
StructType = {
+val inputFields = $(inputCols).map(schema(_))
+val outputColNames = $(outputCols)
+
+val outputFields = inputFields.zip(outputColNames).map { case
(inputField, outputColName) =>
+ OneHotEncoderCommon.transformOutputColumnSchema(
+inputField, $(dropLast), outputColName)
+}
+StructType(schema.fields ++ outputFields)
+ }
+}
+
+/**
+ * A one-hot encoder that maps a column of category indices to a column of
binary vectors, with
+ * at most a single one-value per row that indicates the input category
index.
+ * For example with 5 categories, an input value of 2.0 would map to an
output vector of
+ * `[0.0, 0.0, 1.0, 0.0]`.
+ * The last ca
[GitHub] spark pull request #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator f...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/19527#discussion_r147960232
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/feature/OneHotEncoderEstimator.scala
---
@@ -0,0 +1,462 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.feature
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.SparkException
+import org.apache.spark.annotation.Since
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.attribute._
+import org.apache.spark.ml.linalg.Vectors
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasHandleInvalid, HasInputCols,
HasOutputCols}
+import org.apache.spark.ml.util._
+import org.apache.spark.sql.{DataFrame, Dataset}
+import org.apache.spark.sql.expressions.UserDefinedFunction
+import org.apache.spark.sql.functions.{col, lit, udf}
+import org.apache.spark.sql.types.{DoubleType, NumericType, StructField,
StructType}
+
+/** Private trait for params and common methods for OneHotEncoderEstimator
and OneHotEncoderModel */
+private[ml] trait OneHotEncoderBase extends Params with HasHandleInvalid
+with HasInputCols with HasOutputCols {
+
+ /**
+ * Param for how to handle invalid data.
+ * Options are 'keep' (invalid data produces a vector of zeros) or
'error' (throw an error).
+ * Default: "error"
+ * @group param
+ */
+ @Since("2.3.0")
+ override val handleInvalid: Param[String] = new Param[String](this,
"handleInvalid",
+"How to handle invalid data " +
+"Options are 'keep' (invalid data produces a vector of zeros) or error
(throw an error).",
+
ParamValidators.inArray(OneHotEncoderEstimator.supportedHandleInvalids))
+
+ setDefault(handleInvalid, OneHotEncoderEstimator.ERROR_INVALID)
+
+ /**
+ * Whether to drop the last category in the encoded vector (default:
true)
+ * @group param
+ */
+ @Since("2.3.0")
+ final val dropLast: BooleanParam =
+new BooleanParam(this, "dropLast", "whether to drop the last category")
+ setDefault(dropLast -> true)
+
+ /** @group getParam */
+ @Since("2.3.0")
+ def getDropLast: Boolean = $(dropLast)
+
+ protected def checkParamsValidity(schema: StructType): Unit = {
+val inputColNames = $(inputCols)
+val outputColNames = $(outputCols)
+val existingFields = schema.fields
+
+require(inputColNames.length == outputColNames.length,
--- End diff --
Yeah, based on previous comments
https://github.com/apache/spark/pull/19527#discussion_r145911522 and
https://github.com/apache/spark/pull/19527#discussion_r145915847, now I've used
`withColumns` API which checks duplicated output columns.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator f...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/19527#discussion_r147960630
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/feature/OneHotEncoderEstimator.scala
---
@@ -0,0 +1,462 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.feature
+
+import org.apache.hadoop.fs.Path
+
+import org.apache.spark.SparkException
+import org.apache.spark.annotation.Since
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.attribute._
+import org.apache.spark.ml.linalg.Vectors
+import org.apache.spark.ml.param._
+import org.apache.spark.ml.param.shared.{HasHandleInvalid, HasInputCols,
HasOutputCols}
+import org.apache.spark.ml.util._
+import org.apache.spark.sql.{DataFrame, Dataset}
+import org.apache.spark.sql.expressions.UserDefinedFunction
+import org.apache.spark.sql.functions.{col, lit, udf}
+import org.apache.spark.sql.types.{DoubleType, NumericType, StructField,
StructType}
+
+/** Private trait for params and common methods for OneHotEncoderEstimator
and OneHotEncoderModel */
+private[ml] trait OneHotEncoderBase extends Params with HasHandleInvalid
+with HasInputCols with HasOutputCols {
+
+ /**
+ * Param for how to handle invalid data.
+ * Options are 'keep' (invalid data produces a vector of zeros) or
'error' (throw an error).
+ * Default: "error"
+ * @group param
+ */
+ @Since("2.3.0")
+ override val handleInvalid: Param[String] = new Param[String](this,
"handleInvalid",
+"How to handle invalid data " +
+"Options are 'keep' (invalid data produces a vector of zeros) or error
(throw an error).",
+
ParamValidators.inArray(OneHotEncoderEstimator.supportedHandleInvalids))
+
+ setDefault(handleInvalid, OneHotEncoderEstimator.ERROR_INVALID)
+
+ /**
+ * Whether to drop the last category in the encoded vector (default:
true)
+ * @group param
+ */
+ @Since("2.3.0")
+ final val dropLast: BooleanParam =
+new BooleanParam(this, "dropLast", "whether to drop the last category")
+ setDefault(dropLast -> true)
+
+ /** @group getParam */
+ @Since("2.3.0")
+ def getDropLast: Boolean = $(dropLast)
+
+ protected def checkParamsValidity(schema: StructType): Unit = {
--- End diff --
I'm ok to follow it.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/19527 Thanks for reviewing @zhengruifeng. I will update it later. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19565: [SPARK-22111][MLLIB] OnlineLDAOptimizer should filter ou...
Github user akopich commented on the issue: https://github.com/apache/spark/pull/19565 Ping @hhbyyh, @WeichenXu123, @srowen. Seems like the discussion is stuck. Does anybody think that the general approach implemented in this PR should be changed? Currently it is filtering before sampling with no caching. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19617: [SPARK-22347][PySpark][DOC] Add document to notic...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19617#discussion_r147964546 --- Diff: python/pyspark/sql/functions.py --- @@ -2185,6 +2185,12 @@ def udf(f=None, returnType=StringType()): duplicate invocations may be eliminated or the function may even be invoked more times than it is present in the query. +.. note:: The user-defined functions do not support conditional execution by using them with --- End diff -- Hm, should we maybe clarify the output itself is correct if it does not cause the runtime failure by the condition? Maybe I am too much worried but think it might mislead the output is incorrect at all. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19617: [SPARK-22347][PySpark][DOC] Add document to notic...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19617#discussion_r147961260 --- Diff: python/pyspark/sql/functions.py --- @@ -2185,6 +2185,12 @@ def udf(f=None, returnType=StringType()): duplicate invocations may be eliminated or the function may even be invoked more times than it is present in the query. +.. note:: The user-defined functions do not support conditional execution by using them with +SQL conditional expressions such `when` or `if`. The functions still apply on all rows no --- End diff -- Looks a tiny typo `` expressions such `when` or `if`. `` -> `` expressions such as `when` or `if`. ``. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19580: [SPARK-11334][CORE] Fix bug in Executor allocation manag...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19580 Jenkins, retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19580: [SPARK-11334][CORE] Fix bug in Executor allocation manag...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19580 **[Test build #83257 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83257/testReport)** for PR 19580 at commit [`8abaa2c`](https://github.com/apache/spark/commit/8abaa2cbc4c54dd728b7493ddfedabddd8c8ba7b). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19586: [SPARK-22367][WIP][CORE] Separate the serialization of c...
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/19586
@ConeyLiu what about the below example, does your implementation support
this?
```scala
trait Base { val name: String }
case class A(name: String) extends Base
case class B(name: String) extends Base
sc.parallelize(Seq(A("a"), B("b"))).map { i => (i, 1) }.reduceByKey(_ +
_).collect()
```
Here not all the elements have same class type, does your PR here support
such scenario?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19439: [SPARK-21866][ML][PySpark] Adding spark image rea...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/19439#discussion_r147935499
--- Diff: mllib/src/main/scala/org/apache/spark/ml/image/ImageSchema.scala
---
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.image
+
+import java.awt.Color
+import java.awt.color.ColorSpace
+import java.io.ByteArrayInputStream
+import javax.imageio.ImageIO
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.input.PortableDataStream
+import org.apache.spark.sql.{DataFrame, Row, SparkSession}
+import org.apache.spark.sql.types._
+
+@Experimental
+@Since("2.3.0")
+object ImageSchema {
+
+ val undefinedImageType = "Undefined"
+
+ val imageFields: Array[String] = Array("origin", "height", "width",
"nChannels", "mode", "data")
+
+ val ocvTypes: Map[String, Int] = Map(
+undefinedImageType -> -1,
+"CV_8U" -> 0, "CV_8UC1" -> 0, "CV_8UC3" -> 16, "CV_8UC4" -> 24
+ )
+
+ /**
+ * Schema for the image column: Row(String, Int, Int, Int, Int,
Array[Byte])
+ */
+ val columnSchema = StructType(
+StructField(imageFields(0), StringType, true) ::
+StructField(imageFields(1), IntegerType, false) ::
+StructField(imageFields(2), IntegerType, false) ::
+StructField(imageFields(3), IntegerType, false) ::
+// OpenCV-compatible type: CV_8UC3 in most cases
+StructField(imageFields(4), IntegerType, false) ::
+// Bytes in OpenCV-compatible order: row-wise BGR in most cases
+StructField(imageFields(5), BinaryType, false) :: Nil)
+
+ /**
+ * DataFrame with a single column of images named "image" (nullable)
+ */
+ val imageSchema = StructType(StructField("image", columnSchema, true) ::
Nil)
+
+ /**
+ * :: Experimental ::
+ * Gets the origin of the image
+ *
+ * @return The origin of the image
+ */
+ def getOrigin(row: Row): String = row.getString(0)
+
+ /**
+ * :: Experimental ::
+ * Gets the height of the image
+ *
+ * @return The height of the image
+ */
+ def getHeight(row: Row): Int = row.getInt(1)
+
+ /**
+ * :: Experimental ::
+ * Gets the width of the image
+ *
+ * @return The width of the image
+ */
+ def getWidth(row: Row): Int = row.getInt(2)
+
+ /**
+ * :: Experimental ::
+ * Gets the number of channels in the image
+ *
+ * @return The number of channels in the image
+ */
+ def getNChannels(row: Row): Int = row.getInt(3)
+
+ /**
+ * :: Experimental ::
+ * Gets the OpenCV representation as an int
+ *
+ * @return The OpenCV representation as an int
+ */
+ def getMode(row: Row): Int = row.getInt(4)
+
+ /**
+ * :: Experimental ::
+ * Gets the image data
+ *
+ * @return The image data
+ */
+ def getData(row: Row): Array[Byte] = row.getAs[Array[Byte]](5)
+
+ /**
+ * Default values for the invalid image
+ *
+ * @param origin Origin of the invalid image
+ * @return Row with the default values
+ */
+ private def invalidImageRow(origin: String): Row =
+Row(Row(origin, -1, -1, -1, ocvTypes(undefinedImageType),
Array.ofDim[Byte](0)))
+
+ /**
+ * Convert the compressed image (jpeg, png, etc.) into OpenCV
+ * representation and store it in DataFrame Row
+ *
+ * @param origin Arbitrary string that identifies the image
+ * @param bytes Image bytes (for example, jpeg)
+ * @return DataFrame Row or None (if the decompression fails)
+ */
+ private[spark] def decode(origin: String, bytes: Array[Byte]):
Option[Row] = {
+
+val img = ImageIO.read(new ByteArrayInputStream(bytes))
+
+if (img == null) {
+ None
+} else {
+ val isGray = img.getColorModel.getColorSpace.get
[GitHub] spark pull request #19439: [SPARK-21866][ML][PySpark] Adding spark image rea...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19439#discussion_r147986087 --- Diff: python/pyspark/ml/image.py --- @@ -0,0 +1,139 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +from pyspark.ml.util import * +from pyspark.ml.param.shared import * +from pyspark.sql.types import * +from pyspark.sql.types import Row, _create_row +from pyspark.sql import DataFrame, SparkSession, SQLContext +import numpy as np + +undefinedImageType = "Undefined" + +imageFields = ["origin", "height", "width", "nChannels", "mode", "data"] + + +def getOcvTypes(spark=None): +""" +Returns the OpenCV type mapping supported + +:param sparkSession (SparkSession): The current spark session +:rtype dict: The OpenCV type mapping supported + +.. versionadded:: 2.3.0 +""" +spark = spark or SparkSession.builder.getOrCreate() +ctx = spark.sparkContext +return ctx._jvm.org.apache.spark.ml.image.ImageSchema.ocvTypes + + +# DataFrame with a single column of images named "image" (nullable) +def getImageSchema(spark=None): +""" +Returns the image schema + +:param spark (SparkSession): The current spark session +:rtype StructType: The image schema + +.. versionadded:: 2.3.0 +""" +spark = spark or SparkSession.builder.getOrCreate() +ctx = spark.sparkContext +return ctx._jvm.org.apache.spark.ml.image.ImageSchema.imageSchema --- End diff -- Hm, isn't this a JVM object? I think we should do something like: ```python from pyspark.sql.types import _parse_datatype_json_string ... jschema = ctx._jvm.org.apache.spark.ml.image.ImageSchema.imageSchema() return _parse_datatype_json_string(jschema.json()) ``` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19439: [SPARK-21866][ML][PySpark] Adding spark image rea...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19439#discussion_r147981187 --- Diff: python/pyspark/ml/image.py --- @@ -0,0 +1,139 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +from pyspark.ml.util import * +from pyspark.ml.param.shared import * +from pyspark.sql.types import * +from pyspark.sql.types import Row, _create_row +from pyspark.sql import DataFrame, SparkSession, SQLContext +import numpy as np + +undefinedImageType = "Undefined" + +imageFields = ["origin", "height", "width", "nChannels", "mode", "data"] + + +def getOcvTypes(spark=None): +""" +Returns the OpenCV type mapping supported + +:param sparkSession (SparkSession): The current spark session +:rtype dict: The OpenCV type mapping supported + +.. versionadded:: 2.3.0 +""" +spark = spark or SparkSession.builder.getOrCreate() +ctx = spark.sparkContext +return ctx._jvm.org.apache.spark.ml.image.ImageSchema.ocvTypes + + +# DataFrame with a single column of images named "image" (nullable) +def getImageSchema(spark=None): +""" +Returns the image schema + +:param spark (SparkSession): The current spark session +:rtype StructType: The image schema + +.. versionadded:: 2.3.0 +""" +spark = spark or SparkSession.builder.getOrCreate() +ctx = spark.sparkContext +return ctx._jvm.org.apache.spark.ml.image.ImageSchema.imageSchema + + +def toNDArray(image): +""" +Converts an image to a one-dimensional array. + +:param image (object): The image to be converted +:rtype array: The image as a one-dimensional array + +.. versionadded:: 2.3.0 +""" +height = image.height +width = image.width +nChannels = image.nChannels +return np.ndarray( +shape=(height, width, nChannels), +dtype=np.uint8, +buffer=image.data, +strides=(width * nChannels, nChannels, 1)) + + +def toImage(array, origin="", spark=None): +""" +Converts a one-dimensional array to a two-dimensional image. + +:param array (array): The array to convert to image +:param origin (str): Path to the image +:param spark (SparkSession): The current spark session +:rtype object: Two dimensional image + +.. versionadded:: 2.3.0 +""" +spark = spark or SparkSession.builder.getOrCreate() +if array.ndim != 3: +raise +height, width, nChannels = array.shape +ocvTypes = getOcvTypes(spark) +if nChannels == 1: +mode = ocvTypes["CV_8UC1"] +elif nChannels == 3: +mode = ocvTypes["CV_8UC3"] +elif nChannels == 4: +mode = ocvTypes["CV_8UC4"] +else: +raise --- End diff -- ditto. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19439: [SPARK-21866][ML][PySpark] Adding spark image rea...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19439#discussion_r147980830 --- Diff: python/pyspark/ml/image.py --- @@ -0,0 +1,139 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +from pyspark.ml.util import * +from pyspark.ml.param.shared import * +from pyspark.sql.types import * +from pyspark.sql.types import Row, _create_row +from pyspark.sql import DataFrame, SparkSession, SQLContext +import numpy as np + +undefinedImageType = "Undefined" + +imageFields = ["origin", "height", "width", "nChannels", "mode", "data"] + + +def getOcvTypes(spark=None): +""" +Returns the OpenCV type mapping supported + +:param sparkSession (SparkSession): The current spark session +:rtype dict: The OpenCV type mapping supported + +.. versionadded:: 2.3.0 +""" +spark = spark or SparkSession.builder.getOrCreate() +ctx = spark.sparkContext +return ctx._jvm.org.apache.spark.ml.image.ImageSchema.ocvTypes + + +# DataFrame with a single column of images named "image" (nullable) +def getImageSchema(spark=None): +""" +Returns the image schema + +:param spark (SparkSession): The current spark session +:rtype StructType: The image schema + +.. versionadded:: 2.3.0 +""" +spark = spark or SparkSession.builder.getOrCreate() +ctx = spark.sparkContext +return ctx._jvm.org.apache.spark.ml.image.ImageSchema.imageSchema + + +def toNDArray(image): +""" +Converts an image to a one-dimensional array. + +:param image (object): The image to be converted +:rtype array: The image as a one-dimensional array + +.. versionadded:: 2.3.0 +""" +height = image.height +width = image.width +nChannels = image.nChannels +return np.ndarray( +shape=(height, width, nChannels), +dtype=np.uint8, +buffer=image.data, +strides=(width * nChannels, nChannels, 1)) + + +def toImage(array, origin="", spark=None): +""" +Converts a one-dimensional array to a two-dimensional image. + +:param array (array): The array to convert to image +:param origin (str): Path to the image +:param spark (SparkSession): The current spark session +:rtype object: Two dimensional image + +.. versionadded:: 2.3.0 +""" +spark = spark or SparkSession.builder.getOrCreate() +if array.ndim != 3: +raise --- End diff -- Can we throw explicit exception here? Looks an invalid syntax BTW. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19439: [SPARK-21866][ML][PySpark] Adding spark image rea...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19439#discussion_r147988875 --- Diff: python/pyspark/ml/image.py --- @@ -0,0 +1,139 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +from pyspark.ml.util import * +from pyspark.ml.param.shared import * +from pyspark.sql.types import * +from pyspark.sql.types import Row, _create_row +from pyspark.sql import DataFrame, SparkSession, SQLContext +import numpy as np + +undefinedImageType = "Undefined" + +imageFields = ["origin", "height", "width", "nChannels", "mode", "data"] + + +def getOcvTypes(spark=None): +""" +Returns the OpenCV type mapping supported + +:param sparkSession (SparkSession): The current spark session +:rtype dict: The OpenCV type mapping supported + +.. versionadded:: 2.3.0 +""" +spark = spark or SparkSession.builder.getOrCreate() --- End diff -- ditto for not passing `spark`. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19439: [SPARK-21866][ML][PySpark] Adding spark image rea...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19439#discussion_r147988776 --- Diff: python/pyspark/ml/image.py --- @@ -0,0 +1,139 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +from pyspark.ml.util import * +from pyspark.ml.param.shared import * +from pyspark.sql.types import * +from pyspark.sql.types import Row, _create_row +from pyspark.sql import DataFrame, SparkSession, SQLContext +import numpy as np + +undefinedImageType = "Undefined" + +imageFields = ["origin", "height", "width", "nChannels", "mode", "data"] + + +def getOcvTypes(spark=None): +""" +Returns the OpenCV type mapping supported + +:param sparkSession (SparkSession): The current spark session +:rtype dict: The OpenCV type mapping supported + +.. versionadded:: 2.3.0 +""" +spark = spark or SparkSession.builder.getOrCreate() +ctx = spark.sparkContext +return ctx._jvm.org.apache.spark.ml.image.ImageSchema.ocvTypes + + +# DataFrame with a single column of images named "image" (nullable) +def getImageSchema(spark=None): +""" +Returns the image schema + +:param spark (SparkSession): The current spark session +:rtype StructType: The image schema + +.. versionadded:: 2.3.0 +""" +spark = spark or SparkSession.builder.getOrCreate() --- End diff -- Hm.. actually, I think we won't need to pass `spark` in this case? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19439: [SPARK-21866][ML][PySpark] Adding spark image rea...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19439#discussion_r147989197 --- Diff: python/pyspark/ml/image.py --- @@ -0,0 +1,139 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +from pyspark.ml.util import * +from pyspark.ml.param.shared import * +from pyspark.sql.types import * --- End diff -- Would it be difficult to specify what we import here or, are there many instances to import here? I think we generally should avoid wildcard import. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19439: [SPARK-21866][ML][PySpark] Adding spark image rea...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19439#discussion_r147989847 --- Diff: python/pyspark/ml/image.py --- @@ -0,0 +1,139 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +from pyspark.ml.util import * +from pyspark.ml.param.shared import * +from pyspark.sql.types import * +from pyspark.sql.types import Row, _create_row +from pyspark.sql import DataFrame, SparkSession, SQLContext +import numpy as np + +undefinedImageType = "Undefined" + +imageFields = ["origin", "height", "width", "nChannels", "mode", "data"] + + +def getOcvTypes(spark=None): +""" +Returns the OpenCV type mapping supported + +:param sparkSession (SparkSession): The current spark session +:rtype dict: The OpenCV type mapping supported + +.. versionadded:: 2.3.0 +""" +spark = spark or SparkSession.builder.getOrCreate() +ctx = spark.sparkContext +return ctx._jvm.org.apache.spark.ml.image.ImageSchema.ocvTypes + + +# DataFrame with a single column of images named "image" (nullable) +def getImageSchema(spark=None): +""" +Returns the image schema + +:param spark (SparkSession): The current spark session +:rtype StructType: The image schema + +.. versionadded:: 2.3.0 +""" +spark = spark or SparkSession.builder.getOrCreate() +ctx = spark.sparkContext +return ctx._jvm.org.apache.spark.ml.image.ImageSchema.imageSchema + + +def toNDArray(image): +""" +Converts an image to a one-dimensional array. + +:param image (object): The image to be converted +:rtype array: The image as a one-dimensional array + +.. versionadded:: 2.3.0 +""" +height = image.height +width = image.width +nChannels = image.nChannels +return np.ndarray( +shape=(height, width, nChannels), +dtype=np.uint8, +buffer=image.data, +strides=(width * nChannels, nChannels, 1)) + + +def toImage(array, origin="", spark=None): +""" +Converts a one-dimensional array to a two-dimensional image. + +:param array (array): The array to convert to image +:param origin (str): Path to the image +:param spark (SparkSession): The current spark session +:rtype object: Two dimensional image + +.. versionadded:: 2.3.0 +""" +spark = spark or SparkSession.builder.getOrCreate() +if array.ndim != 3: +raise +height, width, nChannels = array.shape +ocvTypes = getOcvTypes(spark) +if nChannels == 1: +mode = ocvTypes["CV_8UC1"] +elif nChannels == 3: +mode = ocvTypes["CV_8UC3"] +elif nChannels == 4: +mode = ocvTypes["CV_8UC4"] +else: +raise +data = bytearray(array.astype(dtype=np.uint8).ravel()) +# Creating new Row with _create_row(), because Row(name = value, ... ) +# orders fields by name, which conflicts with expected schema order +# when the new DataFrame is created by UDF +return _create_row(imageFields, + [origin, height, width, nChannels, mode, data]) + + +def readImages(path, spark=None, recursive=False, numPartitions=0, + dropImageFailures=False, sampleRatio=1.0): +""" +Reads the directory of images from the local or remote source. + +:param path (str): Path to the image directory +:param spark (SparkSession): The current spark session +:param recursive (bool): Recursive search flag +:param numPartitions (int): Number of DataFrame partitions +:param dropImageFailures (bool): Drop the files that are not valid images +:param sampleRatio (double): Fraction of the images loaded +:rtype DataFrame: DataFrame with a single column of "images", + see ImageSchema for details
[GitHub] spark pull request #19439: [SPARK-21866][ML][PySpark] Adding spark image rea...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/19439#discussion_r147935560
--- Diff: mllib/src/main/scala/org/apache/spark/ml/image/ImageSchema.scala
---
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.image
+
+import java.awt.Color
+import java.awt.color.ColorSpace
+import java.io.ByteArrayInputStream
+import javax.imageio.ImageIO
+
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.input.PortableDataStream
+import org.apache.spark.sql.{DataFrame, Row, SparkSession}
+import org.apache.spark.sql.types._
+
+@Experimental
+@Since("2.3.0")
+object ImageSchema {
+
+ val undefinedImageType = "Undefined"
+
+ val imageFields: Array[String] = Array("origin", "height", "width",
"nChannels", "mode", "data")
+
+ val ocvTypes: Map[String, Int] = Map(
+undefinedImageType -> -1,
+"CV_8U" -> 0, "CV_8UC1" -> 0, "CV_8UC3" -> 16, "CV_8UC4" -> 24
+ )
+
+ /**
+ * Schema for the image column: Row(String, Int, Int, Int, Int,
Array[Byte])
+ */
+ val columnSchema = StructType(
+StructField(imageFields(0), StringType, true) ::
+StructField(imageFields(1), IntegerType, false) ::
+StructField(imageFields(2), IntegerType, false) ::
+StructField(imageFields(3), IntegerType, false) ::
+// OpenCV-compatible type: CV_8UC3 in most cases
+StructField(imageFields(4), IntegerType, false) ::
+// Bytes in OpenCV-compatible order: row-wise BGR in most cases
+StructField(imageFields(5), BinaryType, false) :: Nil)
+
+ /**
+ * DataFrame with a single column of images named "image" (nullable)
+ */
+ val imageSchema = StructType(StructField("image", columnSchema, true) ::
Nil)
+
+ /**
+ * :: Experimental ::
+ * Gets the origin of the image
+ *
+ * @return The origin of the image
+ */
+ def getOrigin(row: Row): String = row.getString(0)
+
+ /**
+ * :: Experimental ::
+ * Gets the height of the image
+ *
+ * @return The height of the image
+ */
+ def getHeight(row: Row): Int = row.getInt(1)
+
+ /**
+ * :: Experimental ::
+ * Gets the width of the image
+ *
+ * @return The width of the image
+ */
+ def getWidth(row: Row): Int = row.getInt(2)
+
+ /**
+ * :: Experimental ::
+ * Gets the number of channels in the image
+ *
+ * @return The number of channels in the image
+ */
+ def getNChannels(row: Row): Int = row.getInt(3)
+
+ /**
+ * :: Experimental ::
+ * Gets the OpenCV representation as an int
+ *
+ * @return The OpenCV representation as an int
+ */
+ def getMode(row: Row): Int = row.getInt(4)
+
+ /**
+ * :: Experimental ::
+ * Gets the image data
+ *
+ * @return The image data
+ */
+ def getData(row: Row): Array[Byte] = row.getAs[Array[Byte]](5)
+
+ /**
+ * Default values for the invalid image
+ *
+ * @param origin Origin of the invalid image
+ * @return Row with the default values
+ */
+ private def invalidImageRow(origin: String): Row =
+Row(Row(origin, -1, -1, -1, ocvTypes(undefinedImageType),
Array.ofDim[Byte](0)))
+
+ /**
+ * Convert the compressed image (jpeg, png, etc.) into OpenCV
+ * representation and store it in DataFrame Row
+ *
+ * @param origin Arbitrary string that identifies the image
+ * @param bytes Image bytes (for example, jpeg)
+ * @return DataFrame Row or None (if the decompression fails)
+ */
+ private[spark] def decode(origin: String, bytes: Array[Byte]):
Option[Row] = {
+
+val img = ImageIO.read(new ByteArrayInputStream(bytes))
+
+if (img == null) {
+ None
+} else {
+ val isGray = img.getColorModel.getColorSpace.get
[GitHub] spark pull request #19439: [SPARK-21866][ML][PySpark] Adding spark image rea...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19439#discussion_r147991609 --- Diff: python/pyspark/ml/image.py --- @@ -0,0 +1,139 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +from pyspark.ml.util import * +from pyspark.ml.param.shared import * +from pyspark.sql.types import * +from pyspark.sql.types import Row, _create_row +from pyspark.sql import DataFrame, SparkSession, SQLContext +import numpy as np + +undefinedImageType = "Undefined" + +imageFields = ["origin", "height", "width", "nChannels", "mode", "data"] + + +def getOcvTypes(spark=None): +""" +Returns the OpenCV type mapping supported + +:param sparkSession (SparkSession): The current spark session +:rtype dict: The OpenCV type mapping supported + +.. versionadded:: 2.3.0 +""" +spark = spark or SparkSession.builder.getOrCreate() +ctx = spark.sparkContext +return ctx._jvm.org.apache.spark.ml.image.ImageSchema.ocvTypes --- End diff -- This looks a JVM object too: ```python >>> spark.sparkContext._jvm.org.apache.spark.ml.image.ImageSchema.ocvTypes() JavaObject id=o41 >>> spark.sparkContext._jvm.org.apache.spark.ml.image.ImageSchema.ocvTypes().getClass().toString() u'class scala.collection.immutable.HashMap$HashTrieMap' ``` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19617: [SPARK-22347][PySpark][DOC] Add document to notic...
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/19617#discussion_r147995716 --- Diff: python/pyspark/sql/functions.py --- @@ -2185,6 +2185,12 @@ def udf(f=None, returnType=StringType()): duplicate invocations may be eliminated or the function may even be invoked more times than it is present in the query. +.. note:: The user-defined functions do not support conditional execution by using them with --- End diff -- Yes. I think this is a valid worry. Thanks. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19617: [SPARK-22347][PySpark][DOC] Add document to notic...
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/19617#discussion_r147995905 --- Diff: python/pyspark/sql/functions.py --- @@ -2185,6 +2185,12 @@ def udf(f=None, returnType=StringType()): duplicate invocations may be eliminated or the function may even be invoked more times than it is present in the query. +.. note:: The user-defined functions do not support conditional execution by using them with +SQL conditional expressions such `when` or `if`. The functions still apply on all rows no --- End diff -- Oops! Thanks. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19439: [SPARK-21866][ML][PySpark] Adding spark image rea...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19439#discussion_r147997611 --- Diff: python/pyspark/ml/image.py --- @@ -0,0 +1,139 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +from pyspark.ml.util import * +from pyspark.ml.param.shared import * +from pyspark.sql.types import * +from pyspark.sql.types import Row, _create_row +from pyspark.sql import DataFrame, SparkSession, SQLContext +import numpy as np + +undefinedImageType = "Undefined" + +imageFields = ["origin", "height", "width", "nChannels", "mode", "data"] + + +def getOcvTypes(spark=None): +""" +Returns the OpenCV type mapping supported + +:param sparkSession (SparkSession): The current spark session +:rtype dict: The OpenCV type mapping supported + +.. versionadded:: 2.3.0 +""" +spark = spark or SparkSession.builder.getOrCreate() +ctx = spark.sparkContext +return ctx._jvm.org.apache.spark.ml.image.ImageSchema.ocvTypes --- End diff -- Probably, we should convert this one to `java.util.HashMap()` so that automatically py4j detects this as `dict`. Quick workaround I could think is, somewhere: ```scala import scala.collection.JavaConverters._ val _ocvTypes: util.Map[String, Int] = ImageSchema.ocvTypes.asJava ``` Python side: ```python spark.sparkContext._jvm.org.apache.spark.ml.image.ImageSchema._ocvTypes() ``` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19439: [SPARK-21866][ML][PySpark] Adding spark image rea...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19439#discussion_r147998308 --- Diff: python/pyspark/ml/image.py --- @@ -0,0 +1,139 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +from pyspark.ml.util import * +from pyspark.ml.param.shared import * +from pyspark.sql.types import * +from pyspark.sql.types import Row, _create_row +from pyspark.sql import DataFrame, SparkSession, SQLContext +import numpy as np + +undefinedImageType = "Undefined" + +imageFields = ["origin", "height", "width", "nChannels", "mode", "data"] + + +def getOcvTypes(spark=None): +""" +Returns the OpenCV type mapping supported + +:param sparkSession (SparkSession): The current spark session +:rtype dict: The OpenCV type mapping supported + +.. versionadded:: 2.3.0 +""" +spark = spark or SparkSession.builder.getOrCreate() +ctx = spark.sparkContext +return ctx._jvm.org.apache.spark.ml.image.ImageSchema.ocvTypes + + +# DataFrame with a single column of images named "image" (nullable) +def getImageSchema(spark=None): +""" +Returns the image schema + +:param spark (SparkSession): The current spark session +:rtype StructType: The image schema + +.. versionadded:: 2.3.0 +""" +spark = spark or SparkSession.builder.getOrCreate() +ctx = spark.sparkContext +return ctx._jvm.org.apache.spark.ml.image.ImageSchema.imageSchema --- End diff -- I think we should have simple tests if these are supposed to be public APIs in Python side in order to to check if they can be called properly at least. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19527 **[Test build #83258 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83258/testReport)** for PR 19527 at commit [`4c6cc57`](https://github.com/apache/spark/commit/4c6cc57136a60c577dd0ba2ae90521f7c18ae5d1). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19439: [SPARK-21866][ML][PySpark] Adding spark image rea...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19439#discussion_r147999393 --- Diff: python/pyspark/ml/image.py --- @@ -0,0 +1,139 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +from pyspark.ml.util import * +from pyspark.ml.param.shared import * +from pyspark.sql.types import * +from pyspark.sql.types import Row, _create_row +from pyspark.sql import DataFrame, SparkSession, SQLContext +import numpy as np + +undefinedImageType = "Undefined" + +imageFields = ["origin", "height", "width", "nChannels", "mode", "data"] + + +def getOcvTypes(spark=None): +""" +Returns the OpenCV type mapping supported + +:param sparkSession (SparkSession): The current spark session +:rtype dict: The OpenCV type mapping supported + +.. versionadded:: 2.3.0 +""" +spark = spark or SparkSession.builder.getOrCreate() +ctx = spark.sparkContext +return ctx._jvm.org.apache.spark.ml.image.ImageSchema.ocvTypes --- End diff -- Or.. just simply define a dict in Python side. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19617: [SPARK-22347][PySpark][DOC] Add document to notice users...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19617 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19617: [SPARK-22347][PySpark][DOC] Add document to notice users...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19617 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83259/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19617: [SPARK-22347][PySpark][DOC] Add document to notice users...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/19617 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19617: [SPARK-22347][PySpark][DOC] Add document to notice users...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19617 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83260/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19617: [SPARK-22347][PySpark][DOC] Add document to notice users...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19617 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19617: [SPARK-22347][PySpark][DOC] Add document to notice users...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/19617 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19617: [SPARK-22347][PySpark][DOC] Add document to notice users...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19617 **[Test build #83261 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83261/testReport)** for PR 19617 at commit [`47bb26e`](https://github.com/apache/spark/commit/47bb26e9fe5a9db8cf666ae16a29d82a3e5e6311). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19617: [SPARK-22347][PySpark][DOC] Add document to notice users...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19617 **[Test build #83261 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83261/testReport)** for PR 19617 at commit [`47bb26e`](https://github.com/apache/spark/commit/47bb26e9fe5a9db8cf666ae16a29d82a3e5e6311). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19617: [SPARK-22347][PySpark][DOC] Add document to notice users...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19617 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83261/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19617: [SPARK-22347][PySpark][DOC] Add document to notice users...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19617 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19272: [Spark-21842][Mesos] Support Kerberos ticket rene...
Github user ArtRand commented on a diff in the pull request:
https://github.com/apache/spark/pull/19272#discussion_r148013037
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -233,10 +249,15 @@ class CoarseGrainedSchedulerBackend(scheduler:
TaskSchedulerImpl, val rpcEnv: Rp
context.reply(true)
case RetrieveSparkAppConfig =>
+val tokens = if (renewableDelegationTokens.isDefined) {
+ Some(renewableDelegationTokens.get.credentials)
+} else {
+ None
+}
val reply = SparkAppConfig(
sparkProperties,
SparkEnv.get.securityManager.getIOEncryptionKey(),
- hadoopDelegationCreds)
+ tokens)
--- End diff --
could make `SparkAppConfig` just take a `RenewableDelegationTokens` object?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19272: [Spark-21842][Mesos] Support Kerberos ticket renewal and...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19272 **[Test build #83262 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83262/testReport)** for PR 19272 at commit [`f93c551`](https://github.com/apache/spark/commit/f93c5513178628d6e7c33bd6cc941407484299fb). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19621: [SPARK-11215][ml] Add multiple columns support to...
GitHub user WeichenXu123 opened a pull request: https://github.com/apache/spark/pull/19621 [SPARK-11215][ml] Add multiple columns support to StringIndexer ## What changes were proposed in this pull request? Add multiple columns support to StringIndexer. ## How was this patch tested? UT will add soon You can merge this pull request into a Git repository by running: $ git pull https://github.com/WeichenXu123/spark multi-col-string-indexer Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/19621.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #19621 commit faa839052ebe0949882b7b8c0ca3557f26beb8e8 Author: WeichenXu Date: 2017-10-31T14:45:34Z init pr --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19527 **[Test build #83258 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83258/testReport)** for PR 19527 at commit [`4c6cc57`](https://github.com/apache/spark/commit/4c6cc57136a60c577dd0ba2ae90521f7c18ae5d1). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19527 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83258/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19527 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19621: [SPARK-11215][ML] Add multiple columns support to String...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19621 **[Test build #83263 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83263/testReport)** for PR 19621 at commit [`faa8390`](https://github.com/apache/spark/commit/faa839052ebe0949882b7b8c0ca3557f26beb8e8). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
