Thanks, Liang-chi. On Thu, 13 Dec 2018, 8:29 am Liang-Chi Hsieh <vii...@gmail.com wrote:
> > Sorry for late. There is a malformed record at CRAN package page again. > I've > already asked CRAN admin for help. It should be fixed soon according to > past > experience. > > Related discussion will be in > https://issues.apache.org/jira/browse/SPARK-24152. I will post here if I > get > reply from CRAN admin. > > Thanks. > > > Liang-Chi Hsieh wrote > > Thanks for letting me know! I will look into it and ask CRAN admin for > > help. > > > > > > Hyukjin Kwon wrote > >> Looks it's happening again. Liang-Chi, do you mind if I ask it again? > >> > >> FYI, R 3.4 is officially deprecated as of > >> https://github.com/apache/spark/pull/23012 > >> We could upgrade R version to 3.4.x in Jenkins, which deals with the > >> malformed(?) responses after 3.0 release. > >> Then, we could get rid of this problem..! > >> > >> 2018년 11월 12일 (월) 오후 1:47, Hyukjin Kwon < > > > >> gurwls223@ > > > >> >님이 작성: > >> > >>> I made a PR to officially drop R prior to version 3.4 ( > >>> https://github.com/apache/spark/pull/23012). > >>> The tests will probably fail for now since it produces warnings for > >>> using > >>> R 3.1.x. > >>> > >>> 2018년 11월 11일 (일) 오전 3:00, Felix Cheung < > > > >> felixcheung_m@ > > > >> >님이 작성: > >>> > >>>> It’s a great point about min R version. From what I see, mostly > because > >>>> of fixes and packages support, most users of R are fairly up to date? > >>>> So > >>>> perhaps 3.4 as min version is reasonable esp. for Spark 3. > >>>> > >>>> Are we getting traction with CRAN sysadmin? It seems like this has > been > >>>> broken a few times. > >>>> > >>>> > >>>> ------------------------------ > >>>> *From:* Liang-Chi Hsieh < > > > >> viirya@ > > > >> > > >>>> *Sent:* Saturday, November 10, 2018 2:32 AM > >>>> *To:* > > > >> dev@.apache > > > >>>> *Subject:* Re: [discuss] SparkR CRAN feasibility check server problem > >>>> > >>>> > >>>> Yeah, thanks Hyukjin Kwon for bringing this up for discussion. > >>>> > >>>> I don't know how higher versions of R are widely used across R > >>>> community. > >>>> If > >>>> R version 3.1.x was not very commonly used, I think we can discuss to > >>>> upgrade minimum R version in next Spark version. > >>>> > >>>> If we ended up with not upgrading, we can discuss with CRAN sysadmin > to > >>>> fix > >>>> it by the service side automatically that prevents malformed R > packages > >>>> info. So we don't need to fix it manually every time. > >>>> > >>>> > >>>> > >>>> Hyukjin Kwon wrote > >>>> >> Can upgrading R able to fix the issue. Is this perhaps not > >>>> necessarily > >>>> > malform but some new format for new versions perhaps? > >>>> > That's my guess. I am not totally sure about it tho. > >>>> > > >>>> >> Anyway we should consider upgrading R version if that fixes the > >>>> problem. > >>>> > Yea, we should. If we should, it should be more them R 3.4. Maybe > >>>> it's > >>>> > good > >>>> > time to start to talk about minimum R version. 3.1.x is too old. > It's > >>>> > released 4.5 years ago. > >>>> > R 3.4.0 is released 1.5 years ago. Considering the timing for Spark > >>>> 3.0, > >>>> > deprecating lower versions, bumping up R to 3.4 might be reasonable > >>>> > option. > >>>> > > >>>> > Adding Shane as well. > >>>> > > >>>> > If we ended up with not upgrading it, I will forward this email to > >>>> CRAN > >>>> > sysadmin to discuss further anyway. > >>>> > > >>>> > > >>>> > > >>>> > 2018년 11월 2일 (금) 오후 12:51, Felix Cheung < > >>>> > >>>> > felixcheung@ > >>>> > >>>> > >님이 작성: > >>>> > > >>>> >> Thanks for being this up and much appreciate with keeping on top of > >>>> this > >>>> >> at all times. > >>>> >> > >>>> >> Can upgrading R able to fix the issue. Is this perhaps not > >>>> necessarily > >>>> >> malform but some new format for new versions perhaps? Anyway we > >>>> should > >>>> >> consider upgrading R version if that fixes the problem. > >>>> >> > >>>> >> As an option we could also disable the repo check in Jenkins but I > >>>> can > >>>> >> see > >>>> >> that could also be problematic. > >>>> >> > >>>> >> > >>>> >> On Thu, Nov 1, 2018 at 7:35 PM Hyukjin Kwon < > >>>> > >>>> > gurwls223@ > >>>> > >>>> > > wrote: > >>>> >> > >>>> >>> Hi all, > >>>> >>> > >>>> >>> I want to raise the CRAN failure issue because it started to block > >>>> Spark > >>>> >>> PRs time to time. Since the number > >>>> >>> of PRs grows hugely in Spark community, this is critical to not > >>>> block > >>>> >>> other PRs. > >>>> >>> > >>>> >>> There has been a problem at CRAN (See > >>>> >>> https://github.com/apache/spark/pull/20005 for analysis). > >>>> >>> To cut it short, the root cause is malformed package info from > >>>> >>> https://cran.r-project.org/src/contrib/PACKAGES > >>>> >>> from server side, and this had to be fixed by requesting it to > CRAN > >>>> >>> sysaadmin's help. > >>>> >>> > >>>> >>> https://issues.apache.org/jira/browse/SPARK-24152 <- newly open. > I > >>>> am > >>>> >>> pretty sure it's the same issue > >>>> >>> https://issues.apache.org/jira/browse/SPARK-25923 <- > >>>> reopen/resolved > >>>> 2 > >>>> >>> times > >>>> >>> https://issues.apache.org/jira/browse/SPARK-22812 > >>>> >>> > >>>> >>> This happened 5 times for roughly about 10 months, causing > blocking > >>>> >>> almost all PRs in Apache Spark. > >>>> >>> Historically, it blocked whole PRs for few days once, and whole > >>>> Spark > >>>> >>> community had to stop working. > >>>> >>> > >>>> >>> I assume this has been not a super big big issue so far for other > >>>> >>> projects or other people because apparently > >>>> >>> higher version of R has some logics to handle this malformed > >>>> documents > >>>> >>> (at least I verified R 3.4.0 works fine). > >>>> >>> > >>>> >>> For our side, Jenkins has low R version (R 3.1.1 if that's not > >>>> updated > >>>> >>> from what I have seen before), > >>>> >>> which is unable to parse the malformed server's response. > >>>> >>> > >>>> >>> So, I want to talk about how we are going to handle this. Possible > >>>> >>> solutions are: > >>>> >>> > >>>> >>> 1. We should start a talk with CRAN sysadmin to permanently > prevent > >>>> this > >>>> >>> issue > >>>> >>> 2. We upgrade R to 3.4.0 in Jenkins (however we will not be able > to > >>>> test > >>>> >>> low R versions) > >>>> >>> 3. ... > >>>> >>> > >>>> >>> If if we fine, I would like to suggest to forward this email to > >>>> CRAN > >>>> >>> sysadmin to discuss further about this. > >>>> >>> > >>>> >>> Adding Liang-Chi Felix and Shivaram who I already talked about > this > >>>> few > >>>> >>> times before. > >>>> >>> > >>>> >>> Thanks all. > >>>> >>> > >>>> >>> > >>>> >>> > >>>> >>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> -- > >>>> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ > >>>> > >>>> --------------------------------------------------------------------- > >>>> To unsubscribe e-mail: > > > >> dev-unsubscribe@.apache > > > >>>> > >>>> > > > > > > > > > > > > -- > > Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ > > > > --------------------------------------------------------------------- > > To unsubscribe e-mail: > > > dev-unsubscribe@.apache > > > > > > -- > Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ > > --------------------------------------------------------------------- > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > >
