I included 26390 as a candidate since it sounded like it bordered on a correctness/expected behaviour fix (eg columpruning rule doing more than column pruning), but if it’s too big a change happy to drop that one.
On Thu, Apr 23, 2020 at 11:43 AM Xiao Li <lix...@databricks.com> wrote: > Hi, Holden, > > We are trying to avoid backporting the improvement/cleanup PRs to the > maintenance releases, especially the core modules, like Spark Core and > SQL. For example, SPARK-26390 is a good example. > > Xiao > > On Thu, Apr 23, 2020 at 11:17 AM Holden Karau <hol...@pigscanfly.ca> > wrote: > >> Tentatively I'm planning on this list to start backporting. If no one >> sees any issues with those I'll start to make backport JIRAs for them for >> tracking this afternoon. >> SPARK-26390 ColumnPruning rule should only do column pruning >> SPARK-25407 Allow nested access for non-existent field for Parquet >> file when nested pruning is enabled >> SPARK-25559 Remove the unsupported predicates in Parquet when >> possible >> SPARK-25860 Replace Literal(null, _) with FalseLiteral whenever >> possible >> SPARK-27514 Skip collapsing windows with empty window expressions >> SPARK-25338 Ensure to call super.beforeAll() and super.afterAll() >> in test cases >> SPARK-27138 Remove AdminUtils calls (fixes deprecation) >> SPARK-27981 Remove `Illegal reflective access` warning for >> `java.nio.Bits.unaligned()` in JDK9+ >> SPARK-26095 Disable parallelization in make-distibution.sh. (Avoid >> build hanging) >> SPARK-25692 Remove static initialization of worker eventLoop >> handling chunk fetch requests within TransportContext. This fixes >> ChunkFetchIntegrationSuite as well >> SPARK-26306 More memory to de-flake SorterSuite >> SPARK-30199 Recover `spark.(ui|blockManager).port` from checkpoint >> SPARK-27676 InMemoryFileIndex should respect >> spark.sql.files.ignoreMissingFiles >> SPARK-31047 Improve file listing for ViewFileSystem >> SPARK-25595 Ignore corrupt Avro file if flag IGNORE_CORRUPT_FILES >> enabled >> >> Maybe: >> SPARK-27801 Delegate to ViewFileSystem during file listing correctly >> >> Not yet merged: >> SPARK-31485 Barrier execution hang if insufficient resources >> >> On Thu, Apr 23, 2020 at 9:13 AM Holden Karau <hol...@pigscanfly.ca> >> wrote: >> >>> >>> >>> On Thu, Apr 23, 2020 at 9:07 AM edeesis <edee...@gmail.com> wrote: >>> >>>> There's other information you can obtain from the Pod metadata on a >>>> describe >>>> than just from the logs, which are typically what's being printed by the >>>> Application itself. >>> >>> Would get pods -w -o yaml do the trick here or is there going to be >>> information that wouldn’t be captured that way? >>> >>>> >>>> >>>> I've also found that Spark has some trouble obtaining the reason for a >>>> K8S >>>> executor death (as evident by the >>>> spark.kubernetes.executor.lostCheck.maxAttempts config property) >>>> >>>> I admittedly don't know what should qualify for a backport, but >>>> considering >>>> 3.0 is a major upgrade (Scala version, et al), is there any room for for >>>> being more generous with backporting to 2.4? >>> >>> I’d like to revisit the conversation around a Spark 2.5 as a >>> transitional release. I know that some people are already effectively >>> maintaining 2.4+ Selective new functionality backports internally. Maybe >>> I’ll kick off that discussion which we can have and that can help inform >>> what we should be putting in 2.4. >>> >>>> >>>> >>>> >>>> >>>> -- >>>> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>> >>>> -- >>> Twitter: https://twitter.com/holdenkarau >>> Books (Learning Spark, High Performance Spark, etc.): >>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>> >> >> >> -- >> Twitter: https://twitter.com/holdenkarau >> Books (Learning Spark, High Performance Spark, etc.): >> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >> > > > -- > <https://databricks.com/sparkaisummit/north-america> > -- Twitter: https://twitter.com/holdenkarau Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
