Hi All, Just circling back to see if there is anything blocking the RC that isn't being tracked in JIRA?
The current in progress list from ((affectedVersion = 2.1.1 AND cf[12310320] is Empty) OR fixVersion = 2.1.1 OR cf[12310320] = "2.1.1") AND project = spark AND resolution = Unresolved ORDER BY priority DESC is only 4 elements: 1. SPARK-19690 <https://issues.apache.org/jira/browse/SPARK-19690> - Join a streaming DataFrame with a batch DataFrame may not work (PR https://github.com/apache/spark/pull/17052 <https://github.com/apache/spark/pull/17052> ) - some discussion around re-targeting exists on the PR 2. 1. 1. SPARK-19522 <https://issues.apache.org/jira/browse/SPARK-19522> - --executor-memory flag doesn't work in local-cluster mode (PR https://github.com/apache/spark/pull/16975 <https://github.com/apache/spark/pull/16975> 2. SPARK-19035 <https://issues.apache.org/jira/browse/SPARK-19035> -rand() function in case when cause failed - no PR exists and it isn't a blocker so I'd suggest we consider re-targetting 1. SPARK-19759 <https://issues.apache.org/jira/browse/SPARK-19759> - ALSModel.predict on Dataframes : potential optimization by not using blas - not explicitly targeted but I'd suggest targeting for 2.3 if people agree Cheers, Holden :) On Tue, Mar 28, 2017 at 2:07 PM, Xiao Li <gatorsm...@gmail.com> wrote: > Hi, Michael, > > Since Daniel Siegmann asked for a bug fix backport in the previous email, > I just merged https://issues.apache.org/jira/browse/SPARK-14536 into > Spark 2.1 branch. > > If this JIRA is not part of Spark 2.1.1 release, could you help me correct > the fix version from 2.1.1. to the next release number. > > Thanks, > > Xiao > > 2017-03-28 8:33 GMT-07:00 Michael Armbrust <mich...@databricks.com>: > >> We just fixed the build yesterday. I'll kick off a new RC today. >> >> On Tue, Mar 28, 2017 at 8:04 AM, Asher Krim <ak...@hubspot.com> wrote: >> >>> Hey Michael, >>> any update on this? We're itching for a 2.1.1 release (specifically >>> SPARK-14804 which is currently blocking us) >>> >>> Thanks, >>> Asher Krim >>> Senior Software Engineer >>> >>> On Wed, Mar 22, 2017 at 7:44 PM, Michael Armbrust < >>> mich...@databricks.com> wrote: >>> >>>> An update: I cut the tag for RC1 last night. Currently fighting with >>>> the release process. Will post RC1 once I get it working. >>>> >>>> On Tue, Mar 21, 2017 at 2:16 PM, Nick Pentreath < >>>> nick.pentre...@gmail.com> wrote: >>>> >>>>> As for SPARK-19759 <https://issues.apache.org/jira/browse/SPARK-19759>, >>>>> I don't think that needs to be targeted for 2.1.1 so we don't need to >>>>> worry >>>>> about it >>>>> >>>>> >>>>> On Tue, 21 Mar 2017 at 13:49 Holden Karau <hol...@pigscanfly.ca> >>>>> wrote: >>>>> >>>>>> I agree with Michael, I think we've got some outstanding issues but >>>>>> none of them seem like regression from 2.1 so we should be good to start >>>>>> the RC process. >>>>>> >>>>>> On Tue, Mar 21, 2017 at 1:41 PM, Michael Armbrust < >>>>>> mich...@databricks.com> wrote: >>>>>> >>>>>> Please speak up if I'm wrong, but none of these seem like critical >>>>>> regressions from 2.1. As such I'll start the RC process later today. >>>>>> >>>>>> On Mon, Mar 20, 2017 at 9:52 PM, Holden Karau <hol...@pigscanfly.ca> >>>>>> wrote: >>>>>> >>>>>> I'm not super sure it should be a blocker for 2.1.1 -- is it a >>>>>> regression? Maybe we can get TDs input on it? >>>>>> >>>>>> On Mon, Mar 20, 2017 at 8:48 PM Nan Zhu <zhunanmcg...@gmail.com> >>>>>> wrote: >>>>>> >>>>>> I think https://issues.apache.org/jira/browse/SPARK-19280 should be >>>>>> a blocker >>>>>> >>>>>> Best, >>>>>> >>>>>> Nan >>>>>> >>>>>> On Mon, Mar 20, 2017 at 8:18 PM, Felix Cheung < >>>>>> felixcheun...@hotmail.com> wrote: >>>>>> >>>>>> I've been scrubbing R and think we are tracking 2 issues >>>>>> >>>>>> https://issues.apache.org/jira/browse/SPARK-19237 >>>>>> >>>>>> https://issues.apache.org/jira/browse/SPARK-19925 >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> ------------------------------ >>>>>> *From:* holden.ka...@gmail.com <holden.ka...@gmail.com> on behalf of >>>>>> Holden Karau <hol...@pigscanfly.ca> >>>>>> *Sent:* Monday, March 20, 2017 3:12:35 PM >>>>>> *To:* dev@spark.apache.org >>>>>> *Subject:* Outstanding Spark 2.1.1 issues >>>>>> >>>>>> Hi Spark Developers! >>>>>> >>>>>> As we start working on the Spark 2.1.1 release I've been looking at >>>>>> our outstanding issues still targeted for it. I've tried to break it down >>>>>> by component so that people in charge of each component can take a quick >>>>>> look and see if any of these things can/should be re-targeted to 2.2 or >>>>>> 2.1.2 & the overall list is pretty short (only 9 items - 5 if we only >>>>>> look >>>>>> at explicitly tagged) :) >>>>>> >>>>>> If your working on something for Spark 2.1.1 and it doesn't show up >>>>>> in this list please speak up now :) We have a lot of issues (including >>>>>> "in >>>>>> progress") that are listed as impacting 2.1.0, but they aren't targeted >>>>>> for >>>>>> 2.1.1 - if there is something you are working in their which should be >>>>>> targeted for 2.1.1 please let us know so it doesn't slip through the >>>>>> cracks. >>>>>> >>>>>> The query string I used for looking at the 2.1.1 open issues is: >>>>>> >>>>>> ((affectedVersion = 2.1.1 AND cf[12310320] is Empty) OR fixVersion = >>>>>> 2.1.1 OR cf[12310320] = "2.1.1") AND project = spark AND resolution = >>>>>> Unresolved ORDER BY priority DESC >>>>>> >>>>>> None of the open issues appear to be a regression from 2.1.0, but >>>>>> those seem more likely to show up during the RC process (thanks in >>>>>> advance >>>>>> to everyone testing their workloads :)) & generally none of them seem to >>>>>> be >>>>>> >>>>>> (Note: the cfs are for Target Version/s field) >>>>>> >>>>>> Critical Issues: >>>>>> SQL: >>>>>> SPARK-19690 <https://issues.apache.org/jira/browse/SPARK-19690> - Join >>>>>> a streaming DataFrame with a batch DataFrame may not work - PR >>>>>> https://github.com/apache/spark/pull/17052 (review in progress by >>>>>> zsxwing, currently failing Jenkins)* >>>>>> >>>>>> Major Issues: >>>>>> SQL: >>>>>> SPARK-19035 <https://issues.apache.org/jira/browse/SPARK-19035> - >>>>>> rand() >>>>>> function in case when cause failed - no outstanding PR (consensus on JIRA >>>>>> seems to be leaning towards it being a real issue but not necessarily >>>>>> everyone agrees just yet - maybe we should slip this?)* >>>>>> Deploy: >>>>>> SPARK-19522 <https://issues.apache.org/jira/browse/SPARK-19522> >>>>>> - --executor-memory flag doesn't work in local-cluster mode - >>>>>> https://github.com/apache/spark/pull/16975 (review in progress by >>>>>> vanzin, but PR currently stalled waiting on response) * >>>>>> Core: >>>>>> SPARK-20025 <https://issues.apache.org/jira/browse/SPARK-20025> - >>>>>> Driver >>>>>> fail over will not work, if SPARK_LOCAL* env is set. - >>>>>> https://github.com/apache/spark/pull/17357 (waiting on review) * >>>>>> PySpark: >>>>>> SPARK-19955 <https://issues.apache.org/jira/browse/SPARK-19955> - >>>>>> Update run-tests to support conda [ Part of Dropping 2.6 support -- which >>>>>> we shouldn't do in a minor release -- but also fixes pip installability >>>>>> tests to run in Jenkins ]- PR failing Jenkins (I need to poke this some >>>>>> more, but seems like 2.7 support works but some other issues. Maybe slip >>>>>> to >>>>>> 2.2?) >>>>>> >>>>>> Minor issues: >>>>>> Tests: >>>>>> SPARK-19612 <https://issues.apache.org/jira/browse/SPARK-19612> - Tests >>>>>> failing with timeout - No PR per-se but it seems unrelated to the 2.1.1 >>>>>> release. It's not targetted for 2.1.1 but listed as affecting 2.1.1 - I'd >>>>>> consider explicitly targeting this for 2.2? >>>>>> PySpark: >>>>>> SPARK-19570 <https://issues.apache.org/jira/browse/SPARK-19570> - Allow >>>>>> to disable hive in pyspark shell - https://github.com/apache/sp >>>>>> ark/pull/16906 PR exists but its difficult to add automated tests >>>>>> for this (although if SPARK-19955 >>>>>> <https://issues.apache.org/jira/browse/SPARK-19955> gets in would >>>>>> make testing this easier) - no reviewers yet. Possible re-target?* >>>>>> Structured Streaming: >>>>>> SPARK-19613 <https://issues.apache.org/jira/browse/SPARK-19613> - Flaky >>>>>> test: StateStoreRDDSuite.versioning and immutability - It's not targetted >>>>>> for 2.1.1 but listed as affecting 2.1.1 - I'd consider explicitly >>>>>> targeting >>>>>> this for 2.2? >>>>>> ML: >>>>>> SPARK-19759 <https://issues.apache.org/jira/browse/SPARK-19759> >>>>>> - ALSModel.predict on Dataframes : potential optimization by not >>>>>> using blas - No PR consider re-targeting unless someone has a PR waiting >>>>>> in >>>>>> the wings? >>>>>> >>>>>> Explicitly targeted issues are marked with a *, the remaining issues >>>>>> are listed as impacting 2.1.1 and don't have a specific target version >>>>>> set. >>>>>> >>>>>> Since 2.1.1 continues the 2.1.0 branch, looking at 2.1.0 shows 1 open >>>>>> blocker in SQL( SPARK-19983 >>>>>> <https://issues.apache.org/jira/browse/SPARK-19983> ), >>>>>> >>>>>> Query string is: >>>>>> >>>>>> affectedVersion = 2.1.0 AND cf[12310320] is EMPTY AND project = spark >>>>>> AND resolution = Unresolved AND priority = targetPriority >>>>>> >>>>>> Continuing on for unresolved 2.1.0 issues in Major there are 163 (76 >>>>>> of them in progress), 65 Minor (26 in progress), and 9 trivial (6 in >>>>>> progress). >>>>>> >>>>>> I'll be going through the 2.1.0 major issues with open PRs that >>>>>> impact the PySpark component and seeing if any of them should be targeted >>>>>> for 2.1.1, if anyone from the other components wants to take a look >>>>>> through >>>>>> we might find some easy wins to be merged. >>>>>> >>>>>> Cheers, >>>>>> >>>>>> Holden :) >>>>>> >>>>>> -- >>>>>> Cell : 425-233-8271 <(425)%20233-8271> >>>>>> Twitter: https://twitter.com/holdenkarau >>>>>> >>>>>> >>>>>> -- >>>>>> Cell : 425-233-8271 <(425)%20233-8271> >>>>>> Twitter: https://twitter.com/holdenkarau >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Cell : 425-233-8271 <(425)%20233-8271> >>>>>> Twitter: https://twitter.com/holdenkarau >>>>>> >>>>> >>>> >>> >> > -- Cell : 425-233-8271 Twitter: https://twitter.com/holdenkarau
