Re: Does Spark run on Java 10?

It is supported with some limitations on JSR 376 (JPMS) that can cause linker errors. Thanks, Muthu On Sun, Apr 1, 2018 at 11:15 AM, kant kodali wrote: > Hi Muthu, > > "On a side note, if some coming version of Scala 2.11 becomes full Java > 9/10 compliant it could work." > > From the links, yo

OOM when extract big data from MySQL Using JDBC

hi, all, We deploy sparksql in standalone mode without HDFS on 1 machine with 256G RAM and 64 cores. The spark session props like below: SparkSession.builder().appName("MYAPP") > .config("spark.sql.crossJoin.enabled", "true") > .config("spark.executor.memory", th

is there a way of register python UDF using java API?

Hi All, All of our spark code is in Java wondering if there a way to register python UDF's using java API such that the registered UDF's can be used using raw spark SQL. If there is any other way to achieve this goal please suggest! Thanks

Sparse Matrix to Matrix multiplication in Spark

Hi there. Right now Spark MLLib doesn't provide Sparse Matrix to Matrix multiplication. The only thing that I saw is a BlockMatrix being multiplied with a DenseMatrix. Is there any plan (or work going on) in near future to provide Sparse Matrix to Matrix multiplication functionality? Thanks. Rega

Re: [Query] Columnar transformation without Structured Streaming

Hi, as far as I understand, given my limited experience with streaming I may be wrong, DStreams are row based data and in case we want to transform them to columnar based data storage then there is a computation overhead. That may be one of the reasons why its better to avoid. On other hand, I a

Re: Does Spark run on Java 10?

Hi Muthu, "On a side note, if some coming version of Scala 2.11 becomes full Java 9/10 compliant it could work." >From the links, you pointed out. It looks like Scala 2.11.12 is compliant with Java 9/10? Thanks! On Sun, Apr 1, 2018 at 7:50 AM, Muthu Jayakumar wrote: > Short answer may be no

Re: Does Spark run on Java 10?

Short answer may be no. Spark runs on Scala 2.11. Even Scala 2.12 is also not fully Java 9 compliant. For more info... http://docs.scala-lang.org/overviews/jdk-compatibility/overview.html --- check the last section. https://issues.apache.org/jira/browse/SPARK-14220 On a side note, if some coming v

Does Spark run on Java 10?

Hi All, Does anybody got Spark running on Java 10? Thanks!

Re: Re: the issue about the + in column,can we support the string please?

Hi ， I checked the code. It seems it is hard to change the code. Current code, string + int is translated to double + double. If I change the the string + int to string + sting, it will incompatible whit old version. Does anyone have better idea about this issue please? 1427357...@qq.com Fr