[GitHub] spark pull request: [SPARK-13636][SQL] Directly consume UnsafeRow ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11484#issuecomment-191642597 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52377/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13636][SQL] Directly consume UnsafeRow ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11484#issuecomment-191642595 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13036][SPARK-13318][SPARK-13319] Add sa...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11203#issuecomment-191642847 **[Test build #52381 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52381/consoleFull)** for PR 11203 at commit [`ecd1df1`](https://github.com/apache/spark/commit/ecd1df10b68a78ba6da19a3111f3ea4cf09a16a2). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13036][SPARK-13318][SPARK-13319] Add sa...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11203#issuecomment-191643227 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52381/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12720] [SQL] SQL Generation Support for...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11283#issuecomment-191643198 **[Test build #52373 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52373/consoleFull)** for PR 11283 at commit [`749be1b`](https://github.com/apache/spark/commit/749be1b2de07603a7fc9961872954c24a584a1fb). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13036][SPARK-13318][SPARK-13319] Add sa...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11203#issuecomment-191643226 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13495][SQL] Add Null Filters in the que...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11372#issuecomment-191643447 **[Test build #52383 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52383/consoleFull)** for PR 11372 at commit [`2a469e8`](https://github.com/apache/spark/commit/2a469e84a618ffe69580eaab6ca77e2a3ab1b7f7). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12720] [SQL] SQL Generation Support for...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11283#issuecomment-191643566 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52373/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12720] [SQL] SQL Generation Support for...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11283#issuecomment-191643564 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13640][SQL] Synchronize ScalaReflection...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11487#issuecomment-191643706 **[Test build #52375 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52375/consoleFull)** for PR 11487 at commit [`389d644`](https://github.com/apache/spark/commit/389d644f776504ee335cd26c121dc84c1902cf32). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13640][SQL] Synchronize ScalaReflection...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11487#issuecomment-191643939 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13640][SQL] Synchronize ScalaReflection...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11487#issuecomment-191643944 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52375/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13466][SQL] Remove projects that become...
Github user davies commented on the pull request: https://github.com/apache/spark/pull/11341#issuecomment-191644061 LGTM, merging this into master, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10562][SQL] support mixed case partitio...
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/9226#discussion_r54846913 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala --- @@ -218,25 +233,19 @@ private[hive] class HiveMetastoreCatalog(val client: ClientInterface, hive: Hive } } -val metastorePartitionColumns = userSpecifiedSchema.map { schema => - val fields = partitionColumns.map(col => schema(col)) - fields.map { field => -HiveColumn( - name = field.name, - hiveType = HiveMetastoreTypes.toMetastoreType(field.dataType), - comment = "") - }.toSeq -}.getOrElse { - if (partitionColumns.length > 0) { -// The table does not have a specified schema, which means that the schema will be inferred -// when we load the table. So, we are not expecting partition columns and we will discover -// partitions when we load the table. However, if there are specified partition columns, -// we simply ignore them and provide a warning message. -logWarning( - s"The schema and partitions of table $tableIdent will be inferred when it is loaded. " + -s"Specified partition columns (${partitionColumns.mkString(",")}) will be ignored.") +if (userSpecifiedSchema.isDefined && partitionColumns.length > 0) { + tableProperties.put("spark.sql.sources.schema.numPartCols", partitionColumns.length.toString) + partitionColumns.zipWithIndex.foreach { case (partCol, index) => +tableProperties.put(s"spark.sql.sources.schema.partCol.$index", partCol) } - Seq.empty[HiveColumn] +} else { + // The table does not have a specified schema, which means that the schema will be inferred + // when we load the table. So, we are not expecting partition columns and we will discover + // partitions when we load the table. However, if there are specified partition columns, + // we simply ignore them and provide a warning message. + logWarning( +s"The schema and partitions of table $tableIdent will be inferred when it is loaded. " + + s"Specified partition columns (${partitionColumns.mkString(",")}) will be ignored.") --- End diff -- line 293 "partitionColumns = Nil" is after an asset: `assert(partitionColumns.isEmpty)`. We can not write out partitioned table in a hive compatible way, so I think this PR is not the root cause that break your code. But it's surprising that the written table is partitioned at hive side in 1.5.2, can you double check it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13543][SQL] Support for specifying comp...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11464#issuecomment-191644225 **[Test build #52384 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52384/consoleFull)** for PR 11464 at commit [`d2ace37`](https://github.com/apache/spark/commit/d2ace37bea1cd8a4c8e1cfb4b1b7fb2ace67c005). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13543][SQL] Support for specifying comp...
Github user HyukjinKwon commented on the pull request: https://github.com/apache/spark/pull/11464#issuecomment-191644382 test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13466][SQL] Remove projects that become...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/11341 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10562][SQL] support mixed case partitio...
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/9226#discussion_r54847179 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala --- @@ -218,25 +233,19 @@ private[hive] class HiveMetastoreCatalog(val client: ClientInterface, hive: Hive } } -val metastorePartitionColumns = userSpecifiedSchema.map { schema => - val fields = partitionColumns.map(col => schema(col)) - fields.map { field => -HiveColumn( - name = field.name, - hiveType = HiveMetastoreTypes.toMetastoreType(field.dataType), - comment = "") - }.toSeq -}.getOrElse { - if (partitionColumns.length > 0) { -// The table does not have a specified schema, which means that the schema will be inferred -// when we load the table. So, we are not expecting partition columns and we will discover -// partitions when we load the table. However, if there are specified partition columns, -// we simply ignore them and provide a warning message. -logWarning( - s"The schema and partitions of table $tableIdent will be inferred when it is loaded. " + -s"Specified partition columns (${partitionColumns.mkString(",")}) will be ignored.") +if (userSpecifiedSchema.isDefined && partitionColumns.length > 0) { + tableProperties.put("spark.sql.sources.schema.numPartCols", partitionColumns.length.toString) + partitionColumns.zipWithIndex.foreach { case (partCol, index) => +tableProperties.put(s"spark.sql.sources.schema.partCol.$index", partCol) } - Seq.empty[HiveColumn] +} else { + // The table does not have a specified schema, which means that the schema will be inferred + // when we load the table. So, we are not expecting partition columns and we will discover + // partitions when we load the table. However, if there are specified partition columns, + // we simply ignore them and provide a warning message. + logWarning( +s"The schema and partitions of table $tableIdent will be inferred when it is loaded. " + + s"Specified partition columns (${partitionColumns.mkString(",")}) will be ignored.") --- End diff -- The root cause is line 270. Even in non-hive compatible mode, we still keep partition info in hive metadata before. This PR changed it and store partition info as properties. We can't write out partitioned table in a hive-compatible way, so I think it's a safe change, you can't query that table at hive side anyway. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13543][SQL] Support for specifying comp...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11464#issuecomment-191646312 **[Test build #52385 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52385/consoleFull)** for PR 11464 at commit [`e4d4cfb`](https://github.com/apache/spark/commit/e4d4cfbc5a65ad95c4eee31ea5a2f09de7e5934a). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13637][SQL] use more information to sim...
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/11485#issuecomment-191646672 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13637][SQL] use more information to sim...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11485#issuecomment-191648049 **[Test build #52386 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52386/consoleFull)** for PR 11485 at commit [`9db97ec`](https://github.com/apache/spark/commit/9db97ec3f641cba9b069821b456ed8a0dfbd905c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13543][SQL] Support for specifying comp...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11464#issuecomment-191648669 **[Test build #52378 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52378/consoleFull)** for PR 11464 at commit [`2817c9a`](https://github.com/apache/spark/commit/2817c9a774f54c5f79828137bdb5a95c391c90c3). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13543][SQL] Support for specifying comp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11464#issuecomment-191648913 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13543][SQL] Support for specifying comp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11464#issuecomment-191648914 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52378/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13640][SQL] Synchronize ScalaReflection...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/11487#issuecomment-191649365 I assume it still works in 2.11, and is at worst redundant but doesn't hurt much. This is definitely the right lock? Maybe a comment pointing to why that is correct would help since it isn't obvious --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13244][SQL][WIP] Prototyping: migrates ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11443#issuecomment-191649872 **[Test build #52387 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52387/consoleFull)** for PR 11443 at commit [`69695fb`](https://github.com/apache/spark/commit/69695fbe7e90bd78b608e116c483a670715fe916). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13432][SQL] add the source file name an...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11301#issuecomment-191654553 **[Test build #52388 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52388/consoleFull)** for PR 11301 at commit [`c763975`](https://github.com/apache/spark/commit/c763975f7f7a16503610a85ac9f5eb542f76cacf). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13244][SQL][WIP] Prototyping: migrates ...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/11443#issuecomment-191661797 @rxin We don't. As what have been done in this PR, we only replaces `DataFrame` with `Dataset` on Java side. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [HOT-FIX] Recover some deprecations for 2.10 c...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/11488#issuecomment-191664081 OK, I'll merge when tests pass here, thank you. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12154] Upgrade to Jersey 2
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/11223#issuecomment-191669921 This needs a few things -- a rebase of course. Next I'm concerned with dependency harmonization. For example I see differing versions of Jackson dependencies being pulled in. I think I expect to see both 1.x and 2.x dependencies, but not different versions of 1.x or 2.x. Likewise with Jersey, it seems like a few 1.x dependencies are still coming in. Figuring this out is the painful part. We also need to update LICENSE files to reflect new dependencies. More tests can't hurt; I don't know how to test the security aspect. Tests do in general basically exercise the web UI already. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MINOR] Fix typos in comments and testcase nam...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11481#issuecomment-191670172 **[Test build #52369 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52369/consoleFull)** for PR 11481 at commit [`dd43417`](https://github.com/apache/spark/commit/dd434174c1f17bc0f26e0b3acbe9e21120d05265). * This patch **fails from timeout after a configured wait of \`250m\`**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MINOR] Fix typos in comments and testcase nam...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11481#issuecomment-191670305 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MINOR] Fix typos in comments and testcase nam...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11481#issuecomment-191670308 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52369/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MINOR] Fix typos in comments and testcase nam...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/11481#issuecomment-191671137 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MINOR] Fix typos in comments and testcase nam...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11481#issuecomment-191672931 **[Test build #52389 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52389/consoleFull)** for PR 11481 at commit [`dd43417`](https://github.com/apache/spark/commit/dd434174c1f17bc0f26e0b3acbe9e21120d05265). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12925. Improve HiveInspectors.unwrap for...
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/11477#discussion_r54853932 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala --- @@ -320,9 +320,11 @@ private[hive] trait HiveInspectors { case hvoi: HiveCharObjectInspector => UTF8String.fromString(hvoi.getPrimitiveJavaObject(data).getValue) case x: StringObjectInspector if x.preferWritable() => -// Text is in UTF-8 already. No need to convert again via fromString +// Text is in UTF-8 already. No need to convert again via fromString. Copy bytes val wObj = x.getPrimitiveWritableObject(data) -UTF8String.fromBytes(wObj.getBytes, 0, wObj.getLength) +val result = new Array[Byte](wObj.getLength()) --- End diff -- You can clone arrays with `.clone()`. Better yet, `Text` provides a `copyBytes()` method to return a copy of the bytes anyway. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13495][SQL] Add Null Filters in the que...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11372#issuecomment-191678584 **[Test build #52383 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52383/consoleFull)** for PR 11372 at commit [`2a469e8`](https://github.com/apache/spark/commit/2a469e84a618ffe69580eaab6ca77e2a3ab1b7f7). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13495][SQL] Add Null Filters in the que...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11372#issuecomment-191678874 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13495][SQL] Add Null Filters in the que...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11372#issuecomment-191678876 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52383/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13038][SPARK-13388][PySpark] Add load/s...
Github user yanboliang commented on the pull request: https://github.com/apache/spark/pull/11266#issuecomment-191680086 @yinxusen I notice that your change made ```Pipeline``` and ```PipelineModel``` as the wrapper of peer Java object. But I vote to keep the exist behavior, ```Pipeline``` still should be a pure Python object. We do not need to bother a peer Java object for ```Pipeline```. Because most stages(Estimator/Transformer) have already supported save/load, so implementing ```Pipeline``` persistence at Python side is more simple than your change. May be we should here others' thoughts. cc @mengxr @jkbradley --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13637][SQL] use more information to sim...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11485#issuecomment-191680443 **[Test build #52386 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52386/consoleFull)** for PR 11485 at commit [`9db97ec`](https://github.com/apache/spark/commit/9db97ec3f641cba9b069821b456ed8a0dfbd905c). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13637][SQL] use more information to sim...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11485#issuecomment-191680718 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52386/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13637][SQL] use more information to sim...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11485#issuecomment-191680716 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13465] Add a task failure listener to T...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11478#issuecomment-191681417 **[Test build #2605 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2605/consoleFull)** for PR 11478 at commit [`4c6e791`](https://github.com/apache/spark/commit/4c6e791371b29adf86fbfcad1b65fd32310aba2d). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13117] [Web UI] WebUI should use the lo...
GitHub user devaraj-kavali opened a pull request: https://github.com/apache/spark/pull/11490 [SPARK-13117] [Web UI] WebUI should use the local ip not 0.0.0.0 ## What changes were proposed in this pull request? In WebUI, now Jetty Server starts with SPARK_PUBLIC_DNS config value if it is configured otherwise it starts with default value as '0.0.0.0'. It is continuation as per the closed PR https://github.com/apache/spark/pull/11133 for the JIRA SPARK-13117 and discussion in SPARK-13117. ## How was this patch tested? This has been verified using the command 'netstat -tnlp | grep ' to check on which IP/hostname is binding with the below steps. In the below results, mentioned PID in the command is the corresponding process id. Without the patch changes, Web UI(Jetty Server) is not taking the value configured for SPARK_PUBLIC_DNS and it is listening to all the interfaces. ## Master ``` [devaraj@stobdtserver2 sbin]$ netstat -tnlp | grep 3930 tcp6 0 0 :::8080 :::*LISTEN 3930/java ``` ## Worker ``` [devaraj@stobdtserver2 sbin]$ netstat -tnlp | grep 4090 tcp6 0 0 :::8081 :::*LISTEN 4090/java ``` ## History Server Process, ``` [devaraj@stobdtserver2 sbin]$ netstat -tnlp | grep 2471 tcp6 0 0 :::18080:::*LISTEN 2471/java ``` ## Driver ``` [devaraj@stobdtserver2 spark-master]$ netstat -tnlp | grep 6556 tcp6 0 0 :::4040 :::*LISTEN 6556/java ``` With the patch changes # i. With SPARK_PUBLIC_DNS configured If the SPARK_PUBLIC_DNS is configured then all the processes Web UI(Jetty Server) is getting bind to the configured value. ## Master ``` [devaraj@stobdtserver2 sbin]$ netstat -tnlp | grep 1561 tcp6 0 0 x.x.x.x:8080 :::*LISTEN 1561/java ``` ## Worker ``` [devaraj@stobdtserver2 sbin]$ netstat -tnlp | grep 2229 tcp6 0 0 x.x.x.x:8081 :::*LISTEN 2229/java ``` ## History Server ``` [devaraj@stobdtserver2 sbin]$ netstat -tnlp | grep 3747 tcp6 0 0 x.x.x.x:18080 :::*LISTEN 3747/java ``` ## Driver ``` [devaraj@stobdtserver2 spark-master]$ netstat -tnlp | grep 6013 tcp6 0 0 x.x.x.x:4040 :::*LISTEN 6013/java ``` # ii. Without SPARK_PUBLIC_DNS configured If the SPARK_PUBLIC_DNS is not configured then all the processes Web UI(Jetty Server) will start with the '0.0.0.0' as default value. ## Master ``` [devaraj@stobdtserver2 sbin]$ netstat -tnlp | grep 4573 tcp6 0 0 :::8080 :::*LISTEN 4573/java ``` ## Worker ``` [devaraj@stobdtserver2 sbin]$ netstat -tnlp | grep 4703 tcp6 0 0 :::8081 :::*LISTEN 4703/java ``` ## History Server ``` [devaraj@stobdtserver2 sbin]$ netstat -tnlp | grep 4846 tcp6 0 0 :::18080:::*LISTEN 4846/java ``` ## Driver ``` [devaraj@stobdtserver2 sbin]$ netstat -tnlp | grep 5437 tcp6 0 0 :::4040 :::*LISTEN 5437/java ``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/devaraj-kavali/spark SPARK-13117-v1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/11490.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #11490 commit 1d736ff7053b0df7b42b34ae738b7a2873e718a7 Author: Devaraj K Date: 2016-03-03T09:00:24Z [SPARK-13117] [Web UI] WebUI should use the local ip not 0.0.0.0 In WebUI, now Jetty Server starts with SPARK_PUBLIC_DNS config value if it is configured otherwise it starts with default value as '0.0.0.0'. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13117] [Web UI] WebUI should use the lo...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11490#issuecomment-191681962 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [HOT-FIX] Recover some deprecations for 2.10 c...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11488#issuecomment-191683625 **[Test build #2606 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2606/consoleFull)** for PR 11488 at commit [`dd66979`](https://github.com/apache/spark/commit/dd66979d7b3b84e4bc136350dd0c2d85d40f2e73). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13036][SPARK-13318][SPARK-13319] Add sa...
Github user yanboliang commented on the pull request: https://github.com/apache/spark/pull/11203#issuecomment-191683786 LGTM for me, cc @mengxr --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [HOT-FIX] Recover some deprecations for 2.10 c...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/11488#issuecomment-191685036 Merged to master --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13423] [WIP] [CORE] [SQL] [STREAMING] S...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/11292#issuecomment-191686172 Merged to master --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [HOT-FIX] Recover some deprecations for 2.10 c...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/11488 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13423] [WIP] [CORE] [SQL] [STREAMING] S...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/11292 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13543][SQL] Support for specifying comp...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11464#issuecomment-191690696 **[Test build #52385 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52385/consoleFull)** for PR 11464 at commit [`e4d4cfb`](https://github.com/apache/spark/commit/e4d4cfbc5a65ad95c4eee31ea5a2f09de7e5934a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13543][SQL] Support for specifying comp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11464#issuecomment-191691005 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52385/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13543][SQL] Support for specifying comp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11464#issuecomment-191691003 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13646] [MLlib] QuantileDiscretizer coun...
GitHub user eliasah opened a pull request: https://github.com/apache/spark/pull/11491 [SPARK-13646] [MLlib] QuantileDiscretizer counts dataset twice in get… ## What changes were proposed in this pull request? It avoids counting the dataframe twice. You can merge this pull request into a Git repository by running: $ git pull https://github.com/eliasah/spark quantile-discretizer-patch Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/11491.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #11491 commit 0f319edae6a1d0272073b6eb60fa54d5bd765f78 Author: Abou Haydar Elias Date: 2016-03-03T10:05:01Z [SPARK-13646] [MLlib] QuantileDiscretizer counts dataset twice in getSampledInput. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13646] [MLlib] QuantileDiscretizer coun...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11491#issuecomment-191691667 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13244][SQL][WIP] Prototyping: migrates ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11443#issuecomment-191691807 **[Test build #52390 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52390/consoleFull)** for PR 11443 at commit [`2bc521a`](https://github.com/apache/spark/commit/2bc521aa98e6c6ff828a5f75db47048ca6b6baef). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13543][SQL] Support for specifying comp...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11464#issuecomment-191691777 **[Test build #52384 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52384/consoleFull)** for PR 11464 at commit [`d2ace37`](https://github.com/apache/spark/commit/d2ace37bea1cd8a4c8e1cfb4b1b7fb2ace67c005). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13646] [MLlib] QuantileDiscretizer coun...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/11491#issuecomment-191691673 Jenkins test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13543][SQL] Support for specifying comp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11464#issuecomment-191692104 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13543][SQL] Support for specifying comp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11464#issuecomment-191692108 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52384/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13492][MESOS] Configurable Mesos framew...
Github user dragos commented on a diff in the pull request: https://github.com/apache/spark/pull/11369#discussion_r54859540 --- Diff: core/src/main/scala/org/apache/spark/deploy/mesos/MesosClusterDispatcherArguments.scala --- @@ -44,7 +44,7 @@ private[mesos] class MesosClusterDispatcherArguments(args: Array[String], conf: port = value parse(tail) -case ("--webui-port" | "-p") :: IntParam(value) :: tail => --- End diff -- I agree that `-p` would always be matched by the pattern above, `"--port" | "-p"`. So removing it here wouldn't have any user-visible effects. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13647][SQL] also check if numeric value...
GitHub user cloud-fan opened a pull request: https://github.com/apache/spark/pull/11492 [SPARK-13647][SQL] also check if numeric value is within allowed range in _verify_type ## What changes were proposed in this pull request? This PR makes the `_verify_type` in `types.py` more strict, also check if numeric value is within allowed range. ## How was this patch tested? newly added doc test. You can merge this pull request into a Git repository by running: $ git pull https://github.com/cloud-fan/spark py-verify Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/11492.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #11492 commit 23222b6429b1696b3d6690da0904affa27e0416c Author: Wenchen Fan Date: 2016-03-03T09:55:54Z also check if numeric value is within allowed range in _verify_type --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13492][MESOS] Configurable Mesos framew...
Github user dragos commented on the pull request: https://github.com/apache/spark/pull/11369#issuecomment-191692543 LGTM. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13647][SQL] also check if numeric value...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11492#issuecomment-191692842 **[Test build #52391 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52391/consoleFull)** for PR 11492 at commit [`23222b6`](https://github.com/apache/spark/commit/23222b6429b1696b3d6690da0904affa27e0416c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13646] [MLlib] QuantileDiscretizer coun...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11491#issuecomment-191692846 **[Test build #52392 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52392/consoleFull)** for PR 11491 at commit [`0f319ed`](https://github.com/apache/spark/commit/0f319edae6a1d0272073b6eb60fa54d5bd765f78). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13583][CORE][STREAMING] Remove unused i...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/11438#issuecomment-191692936 Merged to master --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13646] [MLlib] QuantileDiscretizer coun...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11491#issuecomment-191693151 **[Test build #52392 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52392/consoleFull)** for PR 11491 at commit [`0f319ed`](https://github.com/apache/spark/commit/0f319edae6a1d0272073b6eb60fa54d5bd765f78). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13646] [MLlib] QuantileDiscretizer coun...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11491#issuecomment-191693158 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52392/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13646] [MLlib] QuantileDiscretizer coun...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11491#issuecomment-191693154 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13423] [WIP] [CORE] [SQL] [STREAMING] S...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/11292#issuecomment-191694216 Nuts. I also just got bitten by 2.10 compatibility in two changes: ``` [error] /home/jenkins/workspace/spark-master-compile-maven-scala-2.10/core/src/main/scala/org/apache/spark/ui/jobs/JobsTab.scala:32: value contains is not a member of Option[org.apache.spark.scheduler.SchedulingMode.SchedulingMode] [error] jobProgresslistener.schedulingMode.contains(SchedulingMode.FAIR) ``` `Option.contains` is 2.11+ only. But really this change should just be backed out. Hotfix coming. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13449] Naive Bayes wrapper in SparkR
Github user yanboliang commented on the pull request: https://github.com/apache/spark/pull/11486#issuecomment-191694969 Labels of ML ```NaiveBayesModel``` are sorted(FYI #7284), so we do not need to store it as member variable. Then it can pass the binary compatibility check. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13432][SQL] add the source file name an...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11301#issuecomment-191696160 **[Test build #52388 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52388/consoleFull)** for PR 11301 at commit [`c763975`](https://github.com/apache/spark/commit/c763975f7f7a16503610a85ac9f5eb542f76cacf). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13432][SQL] add the source file name an...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11301#issuecomment-191696676 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13432][SQL] add the source file name an...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11301#issuecomment-191696679 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52388/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13646] [MLlib] QuantileDiscretizer coun...
Github user eliasah commented on the pull request: https://github.com/apache/spark/pull/11491#issuecomment-191696680 I'm not sure why the *scala style tests" fails. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13423] [HOTFIX] Static analysis fixes f...
GitHub user srowen opened a pull request: https://github.com/apache/spark/pull/11493 [SPARK-13423] [HOTFIX] Static analysis fixes for 2.x / fixed for Scala 2.10 ## What changes were proposed in this pull request? Fixes compile problem due to inadvertent use of `Option.contains`, only in Scala 2.11. The change should have been to replace `Option.exists(_ == x)` with `== Some(x)`. Replacing exists with contains only makes sense for collections. Replacing use of `Option.exists` still makes sense though as it's misleading. ## How was this patch tested? Jenkins tests / compilation (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) You can merge this pull request into a Git repository by running: $ git pull https://github.com/srowen/spark SPARK-13423.2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/11493.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #11493 commit 217dde3483f6f031c5a5a13fcb0809382f89c355 Author: Sean Owen Date: 2016-03-03T10:25:44Z Fixes compile problem due to inadvertent use of Option.contains, only in Scala 2.11 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13640][SQL] Synchronize ScalaReflection...
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/11487#issuecomment-191696955 This PR can resolve the issue mentioned in [SPARK-10719](https://issues.apache.org/jira/browse/SPARK-10719) right? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13646] [MLlib] QuantileDiscretizer coun...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/11491#issuecomment-191697011 It's in the test output -- stray space before the comma :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13640][SQL] Synchronize ScalaReflection...
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/11487#issuecomment-191697316 @zsxwing who reported SPARK-10719. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13423] [HOTFIX] Static analysis fixes f...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11493#issuecomment-191698101 **[Test build #52393 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52393/consoleFull)** for PR 11493 at commit [`217dde3`](https://github.com/apache/spark/commit/217dde3483f6f031c5a5a13fcb0809382f89c355). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13647][SQL] also check if numeric value...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11492#issuecomment-191698467 **[Test build #52391 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52391/consoleFull)** for PR 11492 at commit [`23222b6`](https://github.com/apache/spark/commit/23222b6429b1696b3d6690da0904affa27e0416c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13647][SQL] also check if numeric value...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11492#issuecomment-191698743 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52391/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13244][SQL][WIP] Prototyping: migrates ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11443#issuecomment-191699392 **[Test build #52387 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52387/consoleFull)** for PR 11443 at commit [`69695fb`](https://github.com/apache/spark/commit/69695fbe7e90bd78b608e116c483a670715fe916). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13244][SQL][WIP] Prototyping: migrates ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11443#issuecomment-191699744 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52387/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13244][SQL][WIP] Prototyping: migrates ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11443#issuecomment-191699741 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13646] [MLlib] QuantileDiscretizer coun...
Github user eliasah commented on the pull request: https://github.com/apache/spark/pull/11491#issuecomment-191699932 Ok, sorry this is my first PR. Should I correct, push and pr again ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13646] [MLlib] QuantileDiscretizer coun...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/11491#issuecomment-191703162 Yes just push a tiny commit to the same branch --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13646] [MLlib] QuantileDiscretizer coun...
Github user eliasah commented on the pull request: https://github.com/apache/spark/pull/11491#issuecomment-191703696 Done ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13432][SQL] add the source file name an...
Github user kiszk commented on the pull request: https://github.com/apache/spark/pull/11301#issuecomment-191720188 Addressed all of CRs --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13647][SQL] also check if numeric value...
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/11492#issuecomment-191726473 cc @yhuai @davies --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13647][SQL] also check if numeric value...
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/11492#discussion_r54870946 --- Diff: python/pyspark/sql/types.py --- @@ -1137,7 +1148,28 @@ def _verify_type(obj, dataType): if type(obj) not in _acceptable_types[_type]: raise TypeError("%s can not accept object %r in type %s" % (dataType, obj, type(obj))) -if isinstance(dataType, ArrayType): +if isinstance(dataType, ByteType): +if obj < -128 or obj > 127: +raise ValueError("object of ByteType out of range, got: %s" % obj) + +elif isinstance(dataType, ShortType): +if obj < -32768 or obj > 32767: +raise ValueError("object of ShortType out of range, got: %s" % obj) + +elif isinstance(dataType, IntegerType): +if obj < -2147483648 or obj > 2147483647: +raise ValueError("object of IntegerType out of range, got: %s" % obj) + +elif isinstance(dataType, FloatType): +from math import isinf +from struct import pack, unpack + +if not isinf(obj): +f = unpack("f", pack("f", obj))[0] --- End diff -- Here I try to convert the `obj` to binary string as a `float32`, and then convert it back. If it becomes an `inf`, then it means the `obj` can not fit in `float32`. Is there a better way to check if a python float can fit in `float32`? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13637][SQL] use more information to sim...
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/11485#issuecomment-191726573 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13637][SQL] use more information to sim...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11485#issuecomment-191726855 **[Test build #52394 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52394/consoleFull)** for PR 11485 at commit [`9db97ec`](https://github.com/apache/spark/commit/9db97ec3f641cba9b069821b456ed8a0dfbd905c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13244][SQL][WIP] Prototyping: migrates ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11443#issuecomment-191729769 **[Test build #52390 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52390/consoleFull)** for PR 11443 at commit [`2bc521a`](https://github.com/apache/spark/commit/2bc521aa98e6c6ff828a5f75db47048ca6b6baef). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13244][SQL][WIP] Prototyping: migrates ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11443#issuecomment-191729946 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52390/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13244][SQL][WIP] Prototyping: migrates ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11443#issuecomment-191729943 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13599] [BUILD] remove transitive groovy...
Github user steveloughran commented on the pull request: https://github.com/apache/spark/pull/11449#issuecomment-191741549 The risk is deserialization; Groovy CVE-2015-3253 shows how groovy < 2.4.4 makes it straightforward to use a class in Groovy to run arbitrary shell commands on the destination. This has been show on Java ObjectStream and XStream, so assume Kryo is vulnerable too. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13423] [HOTFIX] Static analysis fixes f...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11493#issuecomment-191743759 **[Test build #52393 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52393/consoleFull)** for PR 11493 at commit [`217dde3`](https://github.com/apache/spark/commit/217dde3483f6f031c5a5a13fcb0809382f89c355). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org