Re: SVD computation limit

2017-09-19 Thread Mark Bittmann
I've run into this before. The EigenValueDecomposition creates a Java Array with 2*k*n elements. The Java Array is indexed with a native integer type, so 2*k*n cannot exceed Integer.MAX_VALUE values. The array is created here: https://github.com/apache/spark/blob/master/mllib/src/

Character encoding corruption in Spark JDBC connector

2016-09-13 Thread Mark Bittmann
Hello Spark community, I'm reading from a MySQL database into a Spark dataframe using the JDBC connector functionality, and I'm experiencing some character encoding issues. The default encoding for MySQL strings is latin1, but the mysql JDBC connector implementation of "ResultSet.getString()"

Character encoding corruption in Spark JDBC connector

2016-09-13 Thread Mark Bittmann
Hello Spark community, I'm reading from a MySQL database into a Spark dataframe using the JDBC connector functionality, and I'm experiencing some character encoding issues. The default encoding for MySQL stings is latin1, but the mysql JDBC connector implementation of "ResultSet.getString()" will