Yeah, thanks Hyukjin Kwon for bringing this up for discussion.
I don't know how higher versions of R are widely used across R community. If R version 3.1.x was not very commonly used, I think we can discuss to upgrade minimum R version in next Spark version. If we ended up with not upgrading, we can discuss with CRAN sysadmin to fix it by the service side automatically that prevents malformed R packages info. So we don't need to fix it manually every time. Hyukjin Kwon wrote >> Can upgrading R able to fix the issue. Is this perhaps not necessarily > malform but some new format for new versions perhaps? > That's my guess. I am not totally sure about it tho. > >> Anyway we should consider upgrading R version if that fixes the problem. > Yea, we should. If we should, it should be more them R 3.4. Maybe it's > good > time to start to talk about minimum R version. 3.1.x is too old. It's > released 4.5 years ago. > R 3.4.0 is released 1.5 years ago. Considering the timing for Spark 3.0, > deprecating lower versions, bumping up R to 3.4 might be reasonable > option. > > Adding Shane as well. > > If we ended up with not upgrading it, I will forward this email to CRAN > sysadmin to discuss further anyway. > > > > 2018년 11월 2일 (금) 오후 12:51, Felix Cheung < > felixcheung@ > >님이 작성: > >> Thanks for being this up and much appreciate with keeping on top of this >> at all times. >> >> Can upgrading R able to fix the issue. Is this perhaps not necessarily >> malform but some new format for new versions perhaps? Anyway we should >> consider upgrading R version if that fixes the problem. >> >> As an option we could also disable the repo check in Jenkins but I can >> see >> that could also be problematic. >> >> >> On Thu, Nov 1, 2018 at 7:35 PM Hyukjin Kwon < > gurwls223@ > > wrote: >> >>> Hi all, >>> >>> I want to raise the CRAN failure issue because it started to block Spark >>> PRs time to time. Since the number >>> of PRs grows hugely in Spark community, this is critical to not block >>> other PRs. >>> >>> There has been a problem at CRAN (See >>> https://github.com/apache/spark/pull/20005 for analysis). >>> To cut it short, the root cause is malformed package info from >>> https://cran.r-project.org/src/contrib/PACKAGES >>> from server side, and this had to be fixed by requesting it to CRAN >>> sysaadmin's help. >>> >>> https://issues.apache.org/jira/browse/SPARK-24152 <- newly open. I am >>> pretty sure it's the same issue >>> https://issues.apache.org/jira/browse/SPARK-25923 <- reopen/resolved 2 >>> times >>> https://issues.apache.org/jira/browse/SPARK-22812 >>> >>> This happened 5 times for roughly about 10 months, causing blocking >>> almost all PRs in Apache Spark. >>> Historically, it blocked whole PRs for few days once, and whole Spark >>> community had to stop working. >>> >>> I assume this has been not a super big big issue so far for other >>> projects or other people because apparently >>> higher version of R has some logics to handle this malformed documents >>> (at least I verified R 3.4.0 works fine). >>> >>> For our side, Jenkins has low R version (R 3.1.1 if that's not updated >>> from what I have seen before), >>> which is unable to parse the malformed server's response. >>> >>> So, I want to talk about how we are going to handle this. Possible >>> solutions are: >>> >>> 1. We should start a talk with CRAN sysadmin to permanently prevent this >>> issue >>> 2. We upgrade R to 3.4.0 in Jenkins (however we will not be able to test >>> low R versions) >>> 3. ... >>> >>> If if we fine, I would like to suggest to forward this email to CRAN >>> sysadmin to discuss further about this. >>> >>> Adding Liang-Chi Felix and Shivaram who I already talked about this few >>> times before. >>> >>> Thanks all. >>> >>> >>> >>> -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
