Re: [discuss] SparkR CRAN feasibility check server problem

2018-11-10 Thread Felix Cheung
It’s a great point about min R version. From what I see, mostly because of fixes and packages support, most users of R are fairly up to date? So perhaps 3.4 as min version is reasonable esp. for Spark 3. Are we getting traction with CRAN sysadmin? It seems like this has been broken a few

Re: [discuss] SparkR CRAN feasibility check server problem

2018-11-10 Thread Liang-Chi Hsieh
Yeah, thanks Hyukjin Kwon for bringing this up for discussion. I don't know how higher versions of R are widely used across R community. If R version 3.1.x was not very commonly used, I think we can discuss to upgrade minimum R version in next Spark version. If we ended up with not upgrading,

Re: [discuss] SparkR CRAN feasibility check server problem

2018-11-10 Thread Hyukjin Kwon
> Can upgrading R able to fix the issue. Is this perhaps not necessarily malform but some new format for new versions perhaps? That's my guess. I am not totally sure about it tho. > Anyway we should consider upgrading R version if that fixes the problem. Yea, we should. If we should, it should

Re: Arrow optimization in conversion from R DataFrame to Spark DataFrame

2018-11-10 Thread Hyukjin Kwon
Thanks guys !  2018년 11월 10일 (토) 오전 7:35, Bryan Cutler 님이 작성: > Great work Hyukjin! I'm not too familiar with R, but I'll take a look at > the PR. > > Bryan > > On Fri, Nov 9, 2018 at 9:19 AM Shivaram Venkataraman < > shiva...@eecs.berkeley.edu> wrote: > >> Thanks Hyukjin! Very cool results >>

Re: Spark Utf 8 encoding

2018-11-10 Thread Jörn Franke
Is the original file indeed utf-8? Especially Windows environments tend to mess up the files (E.g. Java on Windows does not use by default UTF-8). However, also the software that processed the data before could have modified it. > Am 10.11.2018 um 02:17 schrieb lsn24 : > > Hello, > > Per the