Saisai, There is no blocker now. I ran into some difficulties in publishing the jars into Nexus. The publish task was finished, but Nexus gave me the following error.
*failureMessage Failed to validate the pgp signature of '/org/apache/spark/spark-streaming-flume-assembly_2.11/2.4.1/spark-streaming-flume-assembly_2.11-2.4.1-tests.jar', check the logs.* I am sure my key is in the key server, and the weird thing is that it fails on different jars each time I ran the publish script. Sincerely, DB Tsai ---------------------------------------------------------- Web: https://www.dbtsai.com PGP Key ID: 42E5B25A8F7A82C1 On Wed, Mar 6, 2019 at 6:04 PM Saisai Shao <sai.sai.s...@gmail.com> wrote: > Do we have other block/critical issues for Spark 2.4.1 or waiting > something to be fixed? I roughly searched the JIRA, seems there's no > block/critical issues marked for 2.4.1. > > Thanks > Saisai > > shane knapp <skn...@berkeley.edu> 于2019年3月7日周四 上午4:57写道： > >> i'll be popping in to the sig-big-data meeting on the 20th to talk about >> stuff like this. >> >> On Wed, Mar 6, 2019 at 12:40 PM Stavros Kontopoulos < >> stavros.kontopou...@lightbend.com> wrote: >> >>> Yes its a touch decision and as we discussed today ( >>> https://docs.google.com/document/d/1pnF38NF6N5eM8DlK088XUW85Vms4V2uTsGZvSp8MNIA >>> ) >>> "Kubernetes support window is 9 months, Spark is two years". So we may >>> end up with old client versions on branches still supported like 2.4.x in >>> the future. >>> That gives us no choice but to upgrade, if we want to be on the safe >>> side. We have tested 3.0.0 with 1.11 internally and it works but I dont >>> know what it means to run with old >>> clients. >>> >>> >>> On Wed, Mar 6, 2019 at 7:54 PM Sean Owen <sro...@gmail.com> wrote: >>> >>>> If the old client is basically unusable with the versions of K8S >>>> people mostly use now, and the new client still works with older >>>> versions, I could see including this in 2.4.1. >>>> >>>> Looking at >>>> https://github.com/fabric8io/kubernetes-client#compatibility-matrix >>>> it seems like the 4.1.1 client is needed for 1.10 and above. However >>>> it no longer supports 1.7 and below. >>>> We have 3.0.x, and versions through 4.0.x of the client support the >>>> same K8S versions, so no real middle ground here. >>>> >>>> 1.7.0 came out June 2017, it seems. 1.10 was March 2018. Minor release >>>> branches are maintained for 9 months per >>>> https://kubernetes.io/docs/setup/version-skew-policy/ >>>> >>>> Spark 2.4.0 came in Nov 2018. I suppose we could say it should have >>>> used the newer client from the start as at that point (?) 1.7 and >>>> earlier were already at least 7 months past EOL. >>>> If we update the client in 2.4.1, versions of K8S as recently >>>> 'supported' as a year ago won't work anymore. I'm guessing there are >>>> still 1.7 users out there? That wasn't that long ago but if the >>>> project and users generally move fast, maybe not. >>>> >>>> Normally I'd say, that's what the next minor release of Spark is for; >>>> update if you want later infra. But there is no Spark 2.5. >>>> I presume downstream distros could modify the dependency easily (?) if >>>> needed and maybe already do. It wouldn't necessarily help end users. >>>> >>>> Does the 3.0.x client not work at all with 1.10+ or just unsupported. >>>> If it 'basically works but no guarantees' I'd favor not updating. If >>>> it doesn't work at all, hm. That's tough. I think I'd favor updating >>>> the client but think it's a tough call both ways. >>>> >>>> >>>> >>>> On Wed, Mar 6, 2019 at 11:14 AM Stavros Kontopoulos >>>> <stavros.kontopou...@lightbend.com> wrote: >>>> > >>>> > Yes Shane Knapp has done the work for that already, and also tests >>>> pass, I am working on a PR now, I could submit it for the 2.4 branch . >>>> > I understand that this is a major dependency update, but the problem >>>> I see is that the client version is so old that I dont think it makes >>>> > much sense for current users who are on k8s 1.10, 1.11 etc( >>>> https://github.com/fabric8io/kubernetes-client#compatibility-matrix, >>>> 3.0.0 does not even exist in there). >>>> > I dont know what it means to use that old version with current k8s >>>> clusters in terms of bugs etc. >>>> >>> >>> >>> >> >> -- >> Shane Knapp >> UC Berkeley EECS Research / RISELab Staff Technical Lead >> https://rise.cs.berkeley.edu >> >
