With Java 7 / Ubuntu 15, and "-Pyarn -Phadoop-2.6 -Phive -Phive-thriftserver", I still see the Docker tests fail every time. Is anyone else seeing them fail (or running them)?
The Hive CliSuite also fails (stack trace at the bottom). Same deal -- if people are running this test and it's not failing, this is probably just flakiness of some form. There's the aforementioned doc generation issue too. Other than that it compiled and ran all tests for me. JIRA score: 28 issues, of which 11 bugs, of which 5 critical (listed below), of which 0 blockers. OK there. Critical bugs: SPARK-8447 Test external shuffle service with all shuffle managers SPARK-10680 Flaky test: network.RequestTimeoutIntegrationSuite.timeoutInactiveRequests SPARK-11224 Flaky test: o.a.s.ExternalShuffleServiceSuite SPARK-11266 Peak memory tests swallow failures SPARK-11293 Spillable collections leak shuffle memory - Simple commands *** FAILED *** ======================= CliSuite failure output ======================= Spark SQL CLI command line: ../../bin/spark-sql --master local --driver-java-options -Dderby.system.durability=test --conf spark.ui.enabled=false --hiveconf javax.jdo.option.ConnectionURL=jdbc:derby:;databaseName=/home/srowen/spark-1.6.0/sql/hive-thriftserver/target/tmp/spark-240e9e22-8fe8-408b-a116-2a894b3cbf1f;create=true --hiveconf hive.metastore.warehouse.dir=/home/srowen/spark-1.6.0/sql/hive-thriftserver/target/tmp/spark-c336bc67-8e51-4284-b574-e8b79d0d4fce --hiveconf hive.exec.scratchdir=/home/srowen/spark-1.6.0/sql/hive-thriftserver/target/tmp/spark-3a4f9564-d9f1-467f-8016-d4c95389e568 Exception: java.util.concurrent.TimeoutException: Futures timed out after [3 minutes] Executed query 0 "CREATE TABLE hive_test(key INT, val STRING);", But failed to capture expected output "OK" within 3 minutes. 2015-12-14 13:47:23.07 - stderr> SLF4J: Class path contains multiple SLF4J bindings. 2015-12-14 13:47:23.07 - stderr> SLF4J: Found binding in [jar:file:/home/srowen/spark-1.6.0/assembly/target/scala-2.10/spark-assembly-1.6.0-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] 2015-12-14 13:47:23.07 - stderr> SLF4J: Found binding in [jar:file:/home/srowen/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] 2015-12-14 13:47:23.07 - stderr> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 2015-12-14 13:47:23.074 - stderr> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 2015-12-14 13:47:39.36 - stdout> SET spark.sql.hive.version=1.2.1 =========================== End CliSuite failure output =========================== (CliSuite.scala:151) On Sat, Dec 12, 2015 at 5:39 PM, Michael Armbrust <mich...@databricks.com> wrote: > Please vote on releasing the following candidate as Apache Spark version > 1.6.0! > > The vote is open until Tuesday, December 15, 2015 at 6:00 UTC and passes if > a majority of at least 3 +1 PMC votes are cast. > > [ ] +1 Release this package as Apache Spark 1.6.0 > [ ] -1 Do not release this package because ... > > To learn more about Apache Spark, please see http://spark.apache.org/ > > The tag to be voted on is v1.6.0-rc2 > (23f8dfd45187cb8f2216328ab907ddb5fbdffd0b) > > The release files, including signatures, digests, etc. can be found at: > http://people.apache.org/~pwendell/spark-releases/spark-1.6.0-rc2-bin/ > > Release artifacts are signed with the following key: > https://people.apache.org/keys/committer/pwendell.asc > > The staging repository for this release can be found at: > https://repository.apache.org/content/repositories/orgapachespark-1169/ > > The test repository (versioned as v1.6.0-rc2) for this release can be found > at: > https://repository.apache.org/content/repositories/orgapachespark-1168/ > > The documentation corresponding to this release can be found at: > http://people.apache.org/~pwendell/spark-releases/spark-1.6.0-rc2-docs/ > > ======================================= > == How can I help test this release? == > ======================================= > If you are a Spark user, you can help us test this release by taking an > existing Spark workload and running on this release candidate, then > reporting any regressions. > > ================================================ > == What justifies a -1 vote for this release? == > ================================================ > This vote is happening towards the end of the 1.6 QA period, so -1 votes > should only occur for significant regressions from 1.5. Bugs already present > in 1.5, minor regressions, or bugs related to new features will not block > this release. > > =============================================================== > == What should happen to JIRA tickets still targeting 1.6.0? == > =============================================================== > 1. It is OK for documentation patches to target 1.6.0 and still go into > branch-1.6, since documentations will be published separately from the > release. > 2. New features for non-alpha-modules should target 1.7+. > 3. Non-blocker bug fixes should target 1.6.1 or 1.7.0, or drop the target > version. > > > ================================================== > == Major changes to help you focus your testing == > ================================================== > > Spark 1.6.0 Preview > > Notable changes since 1.6 RC1 > > Spark Streaming > > SPARK-2629 trackStateByKey has been renamed to mapWithState > > Spark SQL > > SPARK-12165 SPARK-12189 Fix bugs in eviction of storage memory by execution. > SPARK-12258 correct passing null into ScalaUDF > > Notable Features Since 1.5 > > Spark SQL > > SPARK-11787 Parquet Performance - Improve Parquet scan performance when > using flat schemas. > SPARK-10810 Session Management - Isolated devault database (i.e USE mydb) > even on shared clusters. > SPARK-9999 Dataset API - A type-safe API (similar to RDDs) that performs > many operations on serialized binary data and code generation (i.e. Project > Tungsten). > SPARK-10000 Unified Memory Management - Shared memory for execution and > caching instead of exclusive division of the regions. > SPARK-11197 SQL Queries on Files - Concise syntax for running SQL queries > over files of any supported format without registering a table. > SPARK-11745 Reading non-standard JSON files - Added options to read > non-standard JSON files (e.g. single-quotes, unquoted attributes) > SPARK-10412 Per-operator Metrics for SQL Execution - Display statistics on a > peroperator basis for memory usage and spilled data size. > SPARK-11329 Star (*) expansion for StructTypes - Makes it easier to nest and > unest arbitrary numbers of columns > SPARK-10917, SPARK-11149 In-memory Columnar Cache Performance - Significant > (up to 14x) speed up when caching data that contains complex types in > DataFrames or SQL. > SPARK-11111 Fast null-safe joins - Joins using null-safe equality (<=>) will > now execute using SortMergeJoin instead of computing a cartisian product. > SPARK-11389 SQL Execution Using Off-Heap Memory - Support for configuring > query execution to occur using off-heap memory to avoid GC overhead > SPARK-10978 Datasource API Avoid Double Filter - When implemeting a > datasource with filter pushdown, developers can now tell Spark SQL to avoid > double evaluating a pushed-down filter. > SPARK-4849 Advanced Layout of Cached Data - storing partitioning and > ordering schemes in In-memory table scan, and adding distributeBy and > localSort to DF API > SPARK-9858 Adaptive query execution - Intial support for automatically > selecting the number of reducers for joins and aggregations. > SPARK-9241 Improved query planner for queries having distinct aggregations > - Query plans of distinct aggregations are more robust when distinct columns > have high cardinality. > > Spark Streaming > > API Updates > > SPARK-2629 New improved state management - mapWithState - a DStream > transformation for stateful stream processing, supercedes updateStateByKey > in functionality and performance. > SPARK-11198 Kinesis record deaggregation - Kinesis streams have been > upgraded to use KCL 1.4.0 and supports transparent deaggregation of > KPL-aggregated records. > SPARK-10891 Kinesis message handler function - Allows arbitraray function to > be applied to a Kinesis record in the Kinesis receiver before to customize > what data is to be stored in memory. > SPARK-6328 Python Streamng Listener API - Get streaming statistics > (scheduling delays, batch processing times, etc.) in streaming. > > UI Improvements > > Made failures visible in the streaming tab, in the timelines, batch list, > and batch details page. > Made output operations visible in the streaming tab as progress bars. > > MLlib > > New algorithms/models > > SPARK-8518 Survival analysis - Log-linear model for survival analysis > SPARK-9834 Normal equation for least squares - Normal equation solver, > providing R-like model summary statistics > SPARK-3147 Online hypothesis testing - A/B testing in the Spark Streaming > framework > SPARK-9930 New feature transformers - ChiSqSelector, QuantileDiscretizer, > SQL transformer > SPARK-6517 Bisecting K-Means clustering - Fast top-down clustering variant > of K-Means > > API improvements > > ML Pipelines > > SPARK-6725 Pipeline persistence - Save/load for ML Pipelines, with partial > coverage of spark.ml algorithms > SPARK-5565 LDA in ML Pipelines - API for Latent Dirichlet Allocation in ML > Pipelines > > R API > > SPARK-9836 R-like statistics for GLMs - (Partial) R-like stats for ordinary > least squares via summary(model) > SPARK-9681 Feature interactions in R formula - Interaction operator ":" in > R formula > > Python API - Many improvements to Python API to approach feature parity > > Misc improvements > > SPARK-7685 , SPARK-9642 Instance weights for GLMs - Logistic and Linear > Regression can take instance weights > SPARK-10384, SPARK-10385 Univariate and bivariate statistics in DataFrames - > Variance, stddev, correlations, etc. > SPARK-10117 LIBSVM data source - LIBSVM as a SQL data source > > Documentation improvements > > SPARK-7751 @since versions - Documentation includes initial version when > classes and methods were added > SPARK-11337 Testable example code - Automated testing for code in user guide > examples > > Deprecations > > In spark.mllib.clustering.KMeans, the "runs" parameter has been deprecated. > In spark.ml.classification.LogisticRegressionModel and > spark.ml.regression.LinearRegressionModel, the "weights" field has been > deprecated, in favor of the new name "coefficients." This helps disambiguate > from instance (row) weights given to algorithms. > > Changes of behavior > > spark.mllib.tree.GradientBoostedTrees validationTol has changed semantics in > 1.6. Previously, it was a threshold for absolute change in error. Now, it > resembles the behavior of GradientDescent convergenceTol: For large errors, > it uses relative error (relative to the previous error); for small errors (< > 0.01), it uses absolute error. > spark.ml.feature.RegexTokenizer: Previously, it did not convert strings to > lowercase before tokenizing. Now, it converts to lowercase by default, with > an option not to. This matches the behavior of the simpler Tokenizer > transformer. > Spark SQL's partition discovery has been changed to only discover partition > directories that are children of the given path. (i.e. if > path="/my/data/x=1" then x=1 will no longer be considered a partition but > only children of x=1.) This behavior can be overridden by manually > specifying the basePath that partitioning discovery should start with > (SPARK-11678). > When casting a value of an integral type to timestamp (e.g. casting a long > value to timestamp), the value is treated as being in seconds instead of > milliseconds (SPARK-11724). > With the improved query planner for queries having distinct aggregations > (SPARK-9241), the plan of a query having a single distinct aggregation has > been changed to a more robust version. To switch back to the plan generated > by Spark 1.5's planner, please set > spark.sql.specializeSingleDistinctAggPlanning to true (SPARK-12077). > > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
