[GitHub] spark issue #22504: [SPARK-25118][Submit] Persist Driver Logs in Yarn Client...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22504 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22504: [SPARK-25118][Submit] Persist Driver Logs in Yarn Client...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22504 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96717/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22458: [SPARK-25459][SQL] Add viewOriginalText back to CatalogT...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22458 Thank you, @zheyuan28 . Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22458: [SPARK-25459][SQL] Add viewOriginalText back to C...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/22458 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22379: [SPARK-25393][SQL] Adding new function from_csv()
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22379 **[Test build #96719 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96719/testReport)** for PR 22379 at commit [`b36d96a`](https://github.com/apache/spark/commit/b36d96af3853701690297e5bd89a37b7209a446e). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22379: [SPARK-25393][SQL] Adding new function from_csv()
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22379 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22379: [SPARK-25393][SQL] Adding new function from_csv()
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22379 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96719/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22570: [SPARK-25553][BUILD] Add EmptyInterpolatedStringChecker ...
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/22570 What do we do next? 1. Change `level` to `warning`and commit all changes. In this case `dev/scalastyle` can pass, but IDEA will have a warning: ![image](https://user-images.githubusercontent.com/5399861/46182341-d3291700-c2fd-11e8-8a02-1f553066b611.png) 2. Remove `EmptyInterpolatedStringChecker` and commit all changes. 3. Just close this PR. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22429: [SPARK-25440][SQL] Dumping query execution info to a fil...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22429 **[Test build #96718 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96718/testReport)** for PR 22429 at commit [`bd331c5`](https://github.com/apache/spark/commit/bd331c5f8b2e3a0642cc9f818f400af9cbcf37ef). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22429: [SPARK-25440][SQL] Dumping query execution info to a fil...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22429 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22429: [SPARK-25440][SQL] Dumping query execution info to a fil...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22429 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96718/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22557: [SPARK-25535][core] Work around bad error handling in co...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22557 **[Test build #96720 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96720/testReport)** for PR 22557 at commit [`e6e5382`](https://github.com/apache/spark/commit/e6e538282d408b5a87d7fd81e1c2344e41854f5c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22557: [SPARK-25535][core] Work around bad error handling in co...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22557 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22557: [SPARK-25535][core] Work around bad error handling in co...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22557 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96720/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22575: [SPARK-24630][SS][WIP] Support SQLStreaming in Sp...
GitHub user stczwd opened a pull request: https://github.com/apache/spark/pull/22575 [SPARK-24630][SS][WIP] Support SQLStreaming in Spark ## What changes were proposed in this pull request? This patch propose new support of SQLStreaming in Spark, Please refer [SPARK-24630](https://issues.apache.org/jira/browse/SPARK-24630) for more details. This patch supports: 1. Support create stream table, which can be used as Source and Sink in SQLStreaming; `create table kafka_sql_test using kafka options( isStreaming = 'true', subscribe = 'topic', kafka.bootstrap.servers = 'localhost:9092')` 2. Add keyword 'STREAM' in sql to support SQLStreaming queries; `select stream * from kafka_sql_test` 3. As for those complex queries, they all can be supported as long as SQL and StructStreaming support. ## How was this patch tested? Some UTs are added to verify sqlstreaming. You can merge this pull request into a Git repository by running: $ git pull https://github.com/stczwd/spark sqlstreaming Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/22575.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #22575 commit af26ea77157a7ff4e0a2c5eecec64c57f73c425d Author: Jackey Lee Date: 2018-09-28T01:04:17Z Support SQLStreaming in Spark: Add keyword 'STREAM'; Support create stream table --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21688 **[Test build #96721 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96721/testReport)** for PR 21688 at commit [`2bd0fcf`](https://github.com/apache/spark/commit/2bd0fcfa5aa89cfd569def82e525bca7cfb469c2). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22519: [SPARK-25505][SQL] The output order of grouping columns ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22519 **[Test build #96723 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96723/testReport)** for PR 22519 at commit [`2aee8b1`](https://github.com/apache/spark/commit/2aee8b137e6dca733b6529e6b96ef2773c5cb2d4). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22570: [SPARK-25553][BUILD] Add EmptyInterpolatedStringChecker ...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22570 For me, this PR is too intrusive to merge at this stage. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22575: [SPARK-24630][SS][WIP] Support SQLStreaming in Spark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22575 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22575: [SPARK-24630][SS][WIP] Support SQLStreaming in Spark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22575 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22519: [SPARK-25505][SQL] The output order of grouping columns ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22519 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22519: [SPARK-25505][SQL] The output order of grouping columns ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22519 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96723/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21688 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22575: [SPARK-24630][SS][WIP] Support SQLStreaming in Spark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22575 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21688 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96721/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22576: [SPARK-25560][SQL] Allow FunctionInjection in Spa...
GitHub user RussellSpitzer opened a pull request: https://github.com/apache/spark/pull/22576 [SPARK-25560][SQL] Allow FunctionInjection in SparkExtensions This allows an implementer of Spark Session Extensions to utilize a method "injectFunction" which will add a new function to the default Spark Session Catalogue. ## What changes were proposed in this pull request? Adds a new function to SparkSessionExtensions def injectFunction(functionDescription: FunctionDescription) Where function description is a new type type FunctionDescription = (FunctionIdentifier, FunctionBuilder) The functions are loaded in BaseSessionBuilder when the function registry does not have a parent function registry to get loaded from. ## How was this patch tested? New unit tests are added for the extension in SparkSessionExtensionSuite You can merge this pull request into a Git repository by running: $ git pull https://github.com/RussellSpitzer/spark SPARK-25560 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/22576.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #22576 commit 5fdca38e21448ccf2f24a68fbadc221c87195581 Author: Russell Spitzer Date: 2018-09-28T01:58:21Z [SPARK-25560][SQL] Allow FunctionInjection in SparkExtensions This allows an implementer of Spark Session Extensions to utilize a method "injectFunction" which will add a new function to the default Spark Session Catalogue. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22576: [SPARK-25560][SQL] Allow FunctionInjection in SparkExten...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22576 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22576: [SPARK-25560][SQL] Allow FunctionInjection in SparkExten...
Github user RussellSpitzer commented on the issue: https://github.com/apache/spark/pull/22576 @hvanhovell Made a full PR for the change we discussed. Also updated the signature to match the new defined types for the registry and Identifier. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22576: [SPARK-25560][SQL] Allow FunctionInjection in SparkExten...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22576 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22576: [SPARK-25560][SQL] Allow FunctionInjection in SparkExten...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22576 **[Test build #96728 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96728/testReport)** for PR 22576 at commit [`5fdca38`](https://github.com/apache/spark/commit/5fdca38e21448ccf2f24a68fbadc221c87195581). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22458: [SPARK-25459][SQL] Add viewOriginalText back to CatalogT...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22458 **[Test build #96722 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96722/testReport)** for PR 22458 at commit [`94488ea`](https://github.com/apache/spark/commit/94488eac182921188502f6331340ea6b23781690). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22458: [SPARK-25459][SQL] Add viewOriginalText back to CatalogT...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22458 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96722/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22458: [SPARK-25459][SQL] Add viewOriginalText back to CatalogT...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22458 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22574: [SPARK-25559][SQL] Just remove the unsupported predicate...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22574 **[Test build #96724 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96724/testReport)** for PR 22574 at commit [`d49d63b`](https://github.com/apache/spark/commit/d49d63bc40a7752990583f9afbd10c68025510b3). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22574: [SPARK-25559][SQL] Just remove the unsupported predicate...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22574 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96724/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22574: [SPARK-25559][SQL] Just remove the unsupported predicate...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22574 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22504: [SPARK-25118][Submit] Persist Driver Logs in Yarn Client...
Github user bersprockets commented on the issue: https://github.com/apache/spark/pull/22504 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22504: [SPARK-25118][Submit] Persist Driver Logs in Yarn Client...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22504 **[Test build #96729 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96729/testReport)** for PR 22504 at commit [`e13d6c6`](https://github.com/apache/spark/commit/e13d6c6744baafa7e79e5d37fc62d1a2fecd6f36). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22574: [SPARK-25559][SQL] Just remove the unsupported pr...
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/22574#discussion_r221127868 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala --- @@ -488,26 +494,25 @@ private[parquet] class ParquetFilters( .map(_(nameToParquetField(name).fieldName, value)) case sources.And(lhs, rhs) => -// At here, it is not safe to just convert one side if we do not understand the -// other side. Here is an example used to explain the reason. -// Let's say we have NOT(a = 2 AND b in ('1')) and we do not understand how to -// convert b in ('1'). If we only convert a = 2, we will end up with a filter -// NOT(a = 2), which will generate wrong results. -// Pushing one side of AND down is only safe to do at the top level. -// You can see ParquetRelation's initializeLocalJobFunc method as an example. -for { - lhsFilter <- createFilter(schema, lhs) - rhsFilter <- createFilter(schema, rhs) -} yield FilterApi.and(lhsFilter, rhsFilter) +// If the unsupported predicate is in the top level `And` condition or in the child +// `And` condition before hitting `Not` or `Or` condition, it can be safely removed. +(createFilterHelper(nameToParquetField, lhs, canRemoveOneSideInAnd = true), + createFilterHelper(nameToParquetField, rhs, canRemoveOneSideInAnd = true)) match { --- End diff -- Thank you for pinging me, @dbtsai . Here, instead of `canRemoveOneSideInAnd = true`, we need to use the received `canRemoveOneSideInAnd`, don't we? Otherwise, this will cause a bug in the nested `And` clauses. So, could you review and merge the test case, https://github.com/dbtsai/spark/pull/5 and fix the PR accordingly? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22573: [SPARK-25558][SQL] Pushdown predicates for nested...
Github user dbtsai commented on a diff in the pull request: https://github.com/apache/spark/pull/22573#discussion_r221128544 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala --- @@ -437,53 +436,65 @@ object DataSourceStrategy { * @return a `Some[Filter]` if the input [[Expression]] is convertible, otherwise a `None`. */ protected[sql] def translateFilter(predicate: Expression): Option[Filter] = { +// Recursively try to find an attribute name from the top level that can be pushed down. +def attrName(e: Expression): Option[String] = e match { + // In Spark and many data sources such as parquet, dots are used as a column path delimiter; + // thus, we don't translate such expressions. + case a: Attribute if !a.name.contains(".") => +Some(a.name) --- End diff -- Do we have any data source currently supporting `dot` in the column name with pushdown? The worst case will be no pushdown for those data sources. I know ORC doesn't work for now. We can have another followup PR to address this. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22466 **[Test build #96730 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96730/testReport)** for PR 22466 at commit [`7d28da0`](https://github.com/apache/spark/commit/7d28da044ca80cd2c06c7d089786bec2657d175f). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22466 **[Test build #96730 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96730/testReport)** for PR 22466 at commit [`7d28da0`](https://github.com/apache/spark/commit/7d28da044ca80cd2c06c7d089786bec2657d175f). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22466 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96730/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22466 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22574: [SPARK-25559][SQL] Just remove the unsupported predicate...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22574 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22574: [SPARK-25559][SQL] Just remove the unsupported predicate...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22574 **[Test build #96731 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96731/testReport)** for PR 22574 at commit [`29a9aa0`](https://github.com/apache/spark/commit/29a9aa0a30bea4d126e8668ade66678bdc2ab5fa). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22574: [SPARK-25559][SQL] Just remove the unsupported predicate...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22574 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3556/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22573: [SPARK-25558][SQL] Pushdown predicates for nested...
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22573#discussion_r221130276 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala --- @@ -437,53 +436,65 @@ object DataSourceStrategy { * @return a `Some[Filter]` if the input [[Expression]] is convertible, otherwise a `None`. */ protected[sql] def translateFilter(predicate: Expression): Option[Filter] = { +// Recursively try to find an attribute name from the top level that can be pushed down. +def attrName(e: Expression): Option[String] = e match { + // In Spark and many data sources such as parquet, dots are used as a column path delimiter; + // thus, we don't translate such expressions. + case a: Attribute if !a.name.contains(".") => +Some(a.name) --- End diff -- JDBC data source seems having no such restrict. So I worry that this change can cause some regressions. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22569: [SPARK-25542][Core][Test] Move flaky test in Open...
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/22569#discussion_r221130737 --- Diff: core/src/test/scala/org/apache/spark/util/collection/OpenHashSetSuite.scala --- @@ -255,4 +255,16 @@ class OpenHashSetSuite extends SparkFunSuite with Matchers { val set = new OpenHashSet[Long](0) assert(set.size === 0) } + + test("support for more than 12M items") { +val cnt = 1200 // 12M +val set = new OpenHashSet[Int](cnt) +for (i <- 0 until cnt) { + set.add(i) + + val pos1 = set.addWithoutResize(i) & OpenHashSet.POSITION_MASK + val pos2 = set.getPos(i) + assert(pos1 == pos2) +} --- End diff -- My intention was to verify whether small values (e.g. `0, 1, 2, 3`) are still valid after several resizings. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22574: [SPARK-25559][SQL] Just remove the unsupported predicate...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22574 **[Test build #96725 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96725/testReport)** for PR 22574 at commit [`8c76e31`](https://github.com/apache/spark/commit/8c76e315bc0e89d9f049b59c82e97d06c8627025). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22574: [SPARK-25559][SQL] Just remove the unsupported predicate...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22574 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96725/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22574: [SPARK-25559][SQL] Just remove the unsupported predicate...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22574 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22569: [SPARK-25542][Core][Test] Move flaky test in Open...
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22569#discussion_r221131588 --- Diff: core/src/test/scala/org/apache/spark/util/collection/OpenHashSetSuite.scala --- @@ -255,4 +255,16 @@ class OpenHashSetSuite extends SparkFunSuite with Matchers { val set = new OpenHashSet[Long](0) assert(set.size === 0) } + + test("support for more than 12M items") { +val cnt = 1200 // 12M +val set = new OpenHashSet[Int](cnt) +for (i <- 0 until cnt) { + set.add(i) + + val pos1 = set.addWithoutResize(i) & OpenHashSet.POSITION_MASK + val pos2 = set.getPos(i) + assert(pos1 == pos2) +} --- End diff -- I see. I think that is not what this wants test and there should be other tests to make sure it works. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22569: [SPARK-25542][Core][Test] Move flaky test in OpenHashMap...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22569 **[Test build #96726 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96726/testReport)** for PR 22569 at commit [`22bfa80`](https://github.com/apache/spark/commit/22bfa8036ad43534db2e2d4736a8b3ccc44b92b1). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22569: [SPARK-25542][Core][Test] Move flaky test in OpenHashMap...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22569 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96726/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22569: [SPARK-25542][Core][Test] Move flaky test in OpenHashMap...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22569 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22577: [CORE][MINOR] Fix obvious error and compiling for...
GitHub user sadhen opened a pull request: https://github.com/apache/spark/pull/22577 [CORE][MINOR] Fix obvious error and compiling for Scala 2.12.7 ## What changes were proposed in this pull request? Fix an obvious error. ## How was this patch tested? Existing tests. You can merge this pull request into a Git repository by running: $ git pull https://github.com/sadhen/spark minor_fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/22577.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #22577 commit 1fa56185d69e5ba9a23d911ccaf844a967880068 Author: Darcy Shen Date: 2018-09-28T03:33:07Z Fix obvious error and compiling for Scala 2.12.7 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22577: [CORE][MINOR] Fix obvious error and compiling for Scala ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22577 **[Test build #96732 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96732/testReport)** for PR 22577 at commit [`1fa5618`](https://github.com/apache/spark/commit/1fa56185d69e5ba9a23d911ccaf844a967880068). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22577: [CORE][MINOR] Fix obvious error and compiling for Scala ...
Github user sadhen commented on the issue: https://github.com/apache/spark/pull/22577 Dear Maintainers, please backport it to branch-2.4 if it is accepted. @cloud-fan @srowen --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22569: [SPARK-25542][Core][Test] Move flaky test in OpenHashMap...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22569 retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22577: [CORE][MINOR] Fix obvious error and compiling for Scala ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22577 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22577: [CORE][MINOR] Fix obvious error and compiling for Scala ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22577 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22569: [SPARK-25542][Core][Test] Move flaky test in OpenHashMap...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22569 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3557/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22569: [SPARK-25542][Core][Test] Move flaky test in OpenHashMap...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22569 **[Test build #96733 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96733/testReport)** for PR 22569 at commit [`22bfa80`](https://github.com/apache/spark/commit/22bfa8036ad43534db2e2d4736a8b3ccc44b92b1). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22569: [SPARK-25542][Core][Test] Move flaky test in OpenHashMap...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22569 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22577: [CORE][MINOR] Fix obvious error and compiling for...
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22577#discussion_r221136204 --- Diff: core/src/main/scala/org/apache/spark/status/api/v1/OneApplicationResource.scala --- @@ -175,7 +175,7 @@ private[v1] class OneApplicationAttemptResource extends AbstractApplicationResou def getAttempt(): ApplicationAttemptInfo = { uiRoot.getApplicationInfo(appId) .flatMap { app => -app.attempts.filter(_.attemptId == attemptId).headOption --- End diff -- hmm, just curious, why this doesn't compile in 2.12.7? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22565: [SPARK-25543][K8S] Print debug message iff execIdsRemove...
Github user ScrapCodes commented on the issue: https://github.com/apache/spark/pull/22565 @dongjoon-hyun Thanks for looking. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22577: [CORE][MINOR] Fix obvious error and compiling for...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22577#discussion_r221137927 --- Diff: core/src/main/scala/org/apache/spark/status/api/v1/OneApplicationResource.scala --- @@ -175,7 +175,7 @@ private[v1] class OneApplicationAttemptResource extends AbstractApplicationResou def getAttempt(): ApplicationAttemptInfo = { uiRoot.getApplicationInfo(appId) .flatMap { app => -app.attempts.filter(_.attemptId == attemptId).headOption --- End diff -- Yea, can you share the error message? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22577: [CORE][MINOR] Fix obvious error and compiling for Scala ...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22577 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22577: [CORE][MINOR] Fix obvious error and compiling for Scala ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22577 **[Test build #96734 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96734/testReport)** for PR 22577 at commit [`1fa5618`](https://github.com/apache/spark/commit/1fa56185d69e5ba9a23d911ccaf844a967880068). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22566: [SPARK-25458][SQL] Support FOR ALL COLUMNS in ANALYZE TA...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22566 **[Test build #96727 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96727/testReport)** for PR 22566 at commit [`cfa157c`](https://github.com/apache/spark/commit/cfa157c3663a74f40a73d749efd55ecd6dae4ffd). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22566: [SPARK-25458][SQL] Support FOR ALL COLUMNS in ANALYZE TA...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22566 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96727/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22566: [SPARK-25458][SQL] Support FOR ALL COLUMNS in ANALYZE TA...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22566 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22567: [SPARK-25550][WEBUI]Environment Page of Spark Job Histor...
Github user shivusondur commented on the issue: https://github.com/apache/spark/pull/22567 @vanzin Please review this patch. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22576: [SPARK-25560][SQL] Allow FunctionInjection in SparkExten...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22576 **[Test build #96728 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96728/testReport)** for PR 22576 at commit [`5fdca38`](https://github.com/apache/spark/commit/5fdca38e21448ccf2f24a68fbadc221c87195581). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22576: [SPARK-25560][SQL] Allow FunctionInjection in SparkExten...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22576 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22576: [SPARK-25560][SQL] Allow FunctionInjection in SparkExten...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22576 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96728/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22577: [CORE][MINOR] Fix obvious error and compiling for Scala ...
Github user sadhen commented on the issue: https://github.com/apache/spark/pull/22577 Error Message: ``` $ dev/change-scala-version.sh 2.12 â spark git:(branch-2.4) â sbt -Dscala.version=2.12.7 sbt (spark)> project core sbt (core)> clean sbt (core)> compile [error] [warn] /Users/rendong/wdi/spark/core/src/main/scala/org/apache/spark/status/api/v1/OneApplicationResource.scala:178: Option[String] and String are unrelated: they will most likely never compare equal [error] [warn] app.attempts.filter(_.attemptId == attemptId).headOption [error] [warn] ``` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22577: [CORE][MINOR] Fix obvious error and compiling for Scala ...
Github user sadhen commented on the issue: https://github.com/apache/spark/pull/22577 @cloud-fan see: https://github.com/scala/scala/blob/2.12.x/src/compiler/scala/tools/nsc/typechecker/RefChecks.scala --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22574: [SPARK-25559][SQL] Just remove the unsupported predicate...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22574 **[Test build #96731 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96731/testReport)** for PR 22574 at commit [`29a9aa0`](https://github.com/apache/spark/commit/29a9aa0a30bea4d126e8668ade66678bdc2ab5fa). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22574: [SPARK-25559][SQL] Just remove the unsupported predicate...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22574 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96731/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22574: [SPARK-25559][SQL] Just remove the unsupported predicate...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22574 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22544: [SPARK-25522][SQL] Improve type promotion for input argu...
Github user dilipbiswal commented on the issue: https://github.com/apache/spark/pull/22544 @cloud-fan Sounds good Wenchen. Thanks for the detailed analysis. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22569: [SPARK-25542][Core][Test] Move flaky test in Open...
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/22569#discussion_r221147152 --- Diff: core/src/test/scala/org/apache/spark/util/collection/OpenHashSetSuite.scala --- @@ -255,4 +255,16 @@ class OpenHashSetSuite extends SparkFunSuite with Matchers { val set = new OpenHashSet[Long](0) assert(set.size === 0) } + + test("support for more than 12M items") { +val cnt = 1200 // 12M +val set = new OpenHashSet[Int](cnt) +for (i <- 0 until cnt) { + set.add(i) + + val pos1 = set.addWithoutResize(i) & OpenHashSet.POSITION_MASK + val pos2 = set.getPos(i) + assert(pos1 == pos2) +} --- End diff -- The original test performed a check by using `map.iterator.count()`. But, this is not a strong preference. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22569: [SPARK-25542][Core][Test] Move flaky test in Open...
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/22569#discussion_r221147850 --- Diff: core/src/test/scala/org/apache/spark/util/collection/OpenHashSetSuite.scala --- @@ -255,4 +255,16 @@ class OpenHashSetSuite extends SparkFunSuite with Matchers { val set = new OpenHashSet[Long](0) assert(set.size === 0) } + + test("support for more than 12M items") { +val cnt = 1200 // 12M +val set = new OpenHashSet[Int](cnt) +for (i <- 0 until cnt) { + set.add(i) + + val pos1 = set.addWithoutResize(i) & OpenHashSet.POSITION_MASK + val pos2 = set.getPos(i) + assert(pos1 == pos2) +} --- End diff -- oh, the original test performed the count check to see how many values are invalid. They are invalid values because the index is wrong due to wrong position mask in OpenHashSet. This rewritten test tests directly the index of OpenHashSet. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22574: [SPARK-25559][SQL] Just remove the unsupported pr...
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22574#discussion_r221151966 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala --- @@ -488,26 +494,25 @@ private[parquet] class ParquetFilters( .map(_(nameToParquetField(name).fieldName, value)) case sources.And(lhs, rhs) => -// At here, it is not safe to just convert one side if we do not understand the -// other side. Here is an example used to explain the reason. -// Let's say we have NOT(a = 2 AND b in ('1')) and we do not understand how to -// convert b in ('1'). If we only convert a = 2, we will end up with a filter -// NOT(a = 2), which will generate wrong results. -// Pushing one side of AND down is only safe to do at the top level. -// You can see ParquetRelation's initializeLocalJobFunc method as an example. -for { - lhsFilter <- createFilter(schema, lhs) - rhsFilter <- createFilter(schema, rhs) -} yield FilterApi.and(lhsFilter, rhsFilter) +// If the unsupported predicate is in the top level `And` condition or in the child +// `And` condition before hitting `Not` or `Or` condition, it can be safely removed. +(createFilterHelper(nameToParquetField, lhs, canRemoveOneSideInAnd = true), + createFilterHelper(nameToParquetField, rhs, canRemoveOneSideInAnd = true)) match { --- End diff -- +1 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22574: [SPARK-25559][SQL] Just remove the unsupported pr...
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22574#discussion_r221152126 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala --- @@ -488,26 +494,25 @@ private[parquet] class ParquetFilters( .map(_(nameToParquetField(name).fieldName, value)) case sources.And(lhs, rhs) => -// At here, it is not safe to just convert one side if we do not understand the -// other side. Here is an example used to explain the reason. -// Let's say we have NOT(a = 2 AND b in ('1')) and we do not understand how to -// convert b in ('1'). If we only convert a = 2, we will end up with a filter -// NOT(a = 2), which will generate wrong results. -// Pushing one side of AND down is only safe to do at the top level. -// You can see ParquetRelation's initializeLocalJobFunc method as an example. -for { - lhsFilter <- createFilter(schema, lhs) - rhsFilter <- createFilter(schema, rhs) -} yield FilterApi.and(lhsFilter, rhsFilter) +// If the unsupported predicate is in the top level `And` condition or in the child +// `And` condition before hitting `Not` or `Or` condition, it can be safely removed. +(createFilterHelper(nameToParquetField, lhs, canRemoveOneSideInAnd = true), --- End diff -- a bit about style ``` val lhs = createFilterHelper... val rhs = createFilterHelper... (lhs, rhs) match { ... } ``` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22574: [SPARK-25559][SQL] Just remove the unsupported pr...
Github user dbtsai commented on a diff in the pull request: https://github.com/apache/spark/pull/22574#discussion_r221152340 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala --- @@ -488,26 +494,25 @@ private[parquet] class ParquetFilters( .map(_(nameToParquetField(name).fieldName, value)) case sources.And(lhs, rhs) => -// At here, it is not safe to just convert one side if we do not understand the -// other side. Here is an example used to explain the reason. -// Let's say we have NOT(a = 2 AND b in ('1')) and we do not understand how to -// convert b in ('1'). If we only convert a = 2, we will end up with a filter -// NOT(a = 2), which will generate wrong results. -// Pushing one side of AND down is only safe to do at the top level. -// You can see ParquetRelation's initializeLocalJobFunc method as an example. -for { - lhsFilter <- createFilter(schema, lhs) - rhsFilter <- createFilter(schema, rhs) -} yield FilterApi.and(lhsFilter, rhsFilter) +// If the unsupported predicate is in the top level `And` condition or in the child +// `And` condition before hitting `Not` or `Or` condition, it can be safely removed. +(createFilterHelper(nameToParquetField, lhs, canRemoveOneSideInAnd = true), + createFilterHelper(nameToParquetField, rhs, canRemoveOneSideInAnd = true)) match { --- End diff -- Thanks for catching this. I just fixed it. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22339: [SPARK-17159][STREAM] Significant speed up for running s...
Github user ScrapCodes commented on the issue: https://github.com/apache/spark/pull/22339 Hi @srowen, would you like to take a look? Is there anything I can do, if this patch is missing something? I have tested it thoroughly against an object store. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22577: [CORE][MINOR] Fix obvious error and compiling for Scala ...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22577 ah so scala 2.12.7 is smarter on finding mistakes. LGTM then. Do you know how long this bug exists? It will decide how far we should backport it. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22504: [SPARK-25118][Submit] Persist Driver Logs in Yarn Client...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22504 **[Test build #96729 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96729/testReport)** for PR 22504 at commit [`e13d6c6`](https://github.com/apache/spark/commit/e13d6c6744baafa7e79e5d37fc62d1a2fecd6f36). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22574: [SPARK-25559][SQL] Just remove the unsupported pr...
Github user dbtsai commented on a diff in the pull request: https://github.com/apache/spark/pull/22574#discussion_r221153414 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala --- @@ -488,26 +494,25 @@ private[parquet] class ParquetFilters( .map(_(nameToParquetField(name).fieldName, value)) case sources.And(lhs, rhs) => -// At here, it is not safe to just convert one side if we do not understand the -// other side. Here is an example used to explain the reason. -// Let's say we have NOT(a = 2 AND b in ('1')) and we do not understand how to -// convert b in ('1'). If we only convert a = 2, we will end up with a filter -// NOT(a = 2), which will generate wrong results. -// Pushing one side of AND down is only safe to do at the top level. -// You can see ParquetRelation's initializeLocalJobFunc method as an example. -for { - lhsFilter <- createFilter(schema, lhs) - rhsFilter <- createFilter(schema, rhs) -} yield FilterApi.and(lhsFilter, rhsFilter) +// If the unsupported predicate is in the top level `And` condition or in the child +// `And` condition before hitting `Not` or `Or` condition, it can be safely removed. +(createFilterHelper(nameToParquetField, lhs, canRemoveOneSideInAnd = true), --- End diff -- Addressed. Thanks. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22574: [SPARK-25559][SQL] Just remove the unsupported predicate...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22574 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22574: [SPARK-25559][SQL] Just remove the unsupported predicate...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22574 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3558/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22504: [SPARK-25118][Submit] Persist Driver Logs in Yarn Client...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22504 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96729/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22574: [SPARK-25559][SQL] Just remove the unsupported predicate...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22574 **[Test build #96735 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96735/testReport)** for PR 22574 at commit [`d37b2e8`](https://github.com/apache/spark/commit/d37b2e8883425d121d1e8d1533f934747c28fff3). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22504: [SPARK-25118][Submit] Persist Driver Logs in Yarn Client...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22504 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22466: [SPARK-25464][SQL]On dropping the Database it will drop ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22466 **[Test build #96736 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96736/testReport)** for PR 22466 at commit [`f05d7c1`](https://github.com/apache/spark/commit/f05d7c14f3942743485c8b78ac40ee7e1d3bafa9). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22577: [CORE][MINOR] Fix obvious error and compiling for Scala ...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22577 BTW, is this the only error related to compiling for Scala 2.12.7? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22568: [SPARK-23401][PYTHON][TESTS] Add more data types ...
Github user AlexanderKoryagin commented on a diff in the pull request: https://github.com/apache/spark/pull/22568#discussion_r221156223 --- Diff: python/pyspark/sql/tests.py --- @@ -5525,32 +5525,73 @@ def data(self): .withColumn("v", explode(col('vs'))).drop('vs') def test_supported_types(self): -from pyspark.sql.functions import pandas_udf, PandasUDFType, array, col -df = self.data.withColumn("arr", array(col("id"))) +from decimal import Decimal +from distutils.version import LooseVersion +import pyarrow as pa +from pyspark.sql.functions import pandas_udf, PandasUDFType -# Different forms of group map pandas UDF, results of these are the same +input_values_with_schema = [ +(1, StructField('id', IntegerType())), +(2, StructField('byte', ByteType())), +(3, StructField('short', ShortType())), +(4, StructField('int', IntegerType())), +(5, StructField('long', LongType())), +(1.1, StructField('float', FloatType())), +(2.2, StructField('double', DoubleType())), +(Decimal(1.123), StructField('decim', DecimalType(10, 3))), +([1, 2, 3], StructField('array', ArrayType(IntegerType(, +(True, StructField('bool', BooleanType())), +('hello', StructField('str', StringType())), +] -output_schema = StructType( -[StructField('id', LongType()), - StructField('v', IntegerType()), - StructField('arr', ArrayType(LongType())), - StructField('v1', DoubleType()), - StructField('v2', LongType())]) +# TODO: Add BinaryType to 'input_values_with_schema' once minimum pyarrow version is 0.10.0 +if LooseVersion(pa.__version__) >= LooseVersion("0.10.0"): +input_values_with_schema.append( +(bytearray([0x01, 0x02]), StructField('bin', BinaryType())) --- End diff -- Thanks for comment! But I think it is better to use native list method `append` here instead of creating a new list with only one value in it and concatenate it with another. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org