Re: IDEA compile fail but sbt test succeed

[Pasha Finkelshteyn]

Dear AlphaBetaGo,

First of all, there are not only guys here, but also women.

Second, you didn't give a context that would allow to understand the 
connection with Spark. From what I see, it's more likely that it's an 
issue in Spark/sbt support in IDEA. Feel free to create an issue in the 
JetBrains YouTrack [1]


[1] https://youtrack.jetbrains.com/newIssue?project=SCL=25-4794862

On 9/9/23 06:24, AlphaBetaGo wrote:



Hi guys



When building Spark Source Code by IDEA an error came while the sbt could 
succeed

no `: _*' annotation allowed here
(such annotations are only allowed in arguments to *-parameters)
private def doReturn(value: Any) = org.mockito.Mockito.doReturn(value, 
Seq.empty: _*)



| |
AlphaBetaGo
|
|
alphabet...@163.com
|



--
Pasha Finkelshteyn
Developer Advocate @ JetBrains

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: Spark Architecture Question

[Pasha Finkelshteyn]

Hi Renganathan,

Not quite. It strongly depends on your usage of UDFs defined in any
manner — as UDF object or just lambdas. If you have ones — they may and
will be called on executors too.

On 21/07/29 05:17, Renganathan Mutthiah wrote:
> Hi,
> 
> I have read in many materials (including from the book: Spark - The
> Definitive Guide) that Spark is a compiler.
> 
> In my understanding, our program is used until the point of DAG generation.
> This portion can be written in any language - Java,Scala,R,Python.
> Post that (executing the DAG), the engine runs in Scala only. This leads to
> Spark being called as a compiler.
> 
> If the above is true, we need to install R / Python only in the driver
> machine. R / Python run time is not needed in worker nodes. Am I correct ?
> 
> Thanks!

-- 
Regards,
Pasha

Big Data Tools @ JetBrains


signature.asc
Description: PGP signature

Re: Apache Spark

[Pasha Finkelshteyn]

Hi Andrey,

It looks like you may contact Databricks for that.
Also it would be easier for non-russian spaekers to respond you if your
name would be written in English.

On 21/01/26 12:41PM, Синий Андрей wrote:
> Hello!
> 
> We plan to use Apache Spark software in our organization, can I purchase paid 
> technical support for this software?
> 
> 
> 
> С уважением,
> Андрей Синий
> Руководитель направления
> Центр управления программным обеспечением
> Филиал ПАО «МТС» в Нижегородской области
> Публичное акционерное общество «Мобильные ТелеСистемы»
> __
> IP: 90096
> mob: +79103801534
> e-mail: avs...@mts.ru
> г. Нижний Новгород, пр. Гагарина, д. 168А, пом. П8, 3, 310
> 
> [cid:image001.png@01D6F3F9.B5D32BD0]
> 



-- 
Regards,
Pasha

Big Data Tools @ JetBrains


signature.asc
Description: PGP signature

Re: Need your help!! (URGENT Code works fine when submitted as java main but part of data missing when running as Spark-Submit)

[Pasha Finkelshteyn]

Hi Rachana,

Couls you please provide us with mre details:

Minimal repro
Spark version
Java version
Scala version


On 20/07/21 08:27AM, Rachana Srivastava wrote:
> I am unable to identify the root cause of why my code is missing data when I 
> run as spark-submit but the code works fine when I run as java main  Any 
> idea

-- 
Regards,
Pasha

Big Data Tools @ JetBrains


signature.asc
Description: PGP signature

Re: Kotlin Spark API

[Pasha Finkelshteyn]

Hi Stephen,

Thank you so much for finding time for looking at our examples! Yes, we've 
tried to implement as clean design of API as possible and are constantly 
looking for ways to make it even more readable, clear and friendly.

And as Maria already stated we welcome any feedback!

On 20/07/14 01:55PM, Stephen Boesch wrote:
> I just looked at the examples.
> https://github.com/JetBrains/kotlin-spark-api/tree/master/examples/src/main/kotlin/org/jetbrains/spark/api/examples
> These look v nice!  V concise yet flexible.  I like the ability to do
> inline *side-effects.  *E.g. caching or printing or showDs()
> 
> package org.jetbrains.spark.api.examples
> import org.apache.spark.sql.Row
> import org.jetbrains.spark.api.*
> 
> fun main() {
> withSpark {
> val sd = dsOf(1, 2, 3)
> sd.createOrReplaceTempView("ds")
> spark.sql("select * from ds")
> .withCached {
> println("asList: ${toList()}")
> println("asArray: ${toArray().contentToString()}")
> this
> }
> .to()
> .withCached {
> println("typed collect: " + (collect() as
> Array).contentToString())
> println("type collectAsList: " + collectAsList())
> }
> 
> dsOf(1, 2, 3)
> .map { c(it, it + 1, it + 2) }
> .to()
> .select("_1")
> .collectAsList()
> .forEach { println(it) }
> }
> }
> 
> 
> So that shows some of the niceness of kotlin: intuitive type conversion
> `to`/`to` and `dsOf( list)`- and also the inlining of the side
> effects. Overall concise and pleasant to read.
> 
> 
> On Tue, 14 Jul 2020 at 12:18, Stephen Boesch  wrote:
> 
> > I started with scala/spark in 2012 and scala has been my go-to language
> > for six years. But I heartily applaud this direction. Kotlin is more like a
> > simplified Scala - with the benefits that brings - than a simplified java.
> > I particularly like the simplified / streamlined collections classes.
> >
> > Really looking forward to this development.
> >
> > On Tue, 14 Jul 2020 at 10:42, Maria Khalusova  wrote:
> >
> >> Hi folks,
> >>
> >> We would love your feedback on the new Kotlin Spark API that we are
> >> working on: https://github.com/JetBrains/kotlin-spark-api.
> >>
> >> Why Kotlin Spark API? Kotlin developers can already use Kotlin with the
> >> existing Apache Spark Java API, however they cannot take full advantage of
> >> Kotlin language features. With Kotlin Spark API, you can use Kotlin data
> >> classes and lambda expressions.
> >>
> >> The API also adds some helpful extension functions. For example, you can
> >> use `withCached` to perform arbitrary transformations on a Dataset and not
> >> worry about the Dataset unpersisting at the end.
> >>
> >> If you like Kotlin and would like to try the API, we've prepared a Quick
> >> Start Guide to help you set up all the needed dependencies in no time using
> >> either Maven or Gradle:
> >> https://github.com/JetBrains/kotlin-spark-api/blob/master/docs/quick-start-guide.md
> >>
> >> In the repo, you’ll also find a few code examples to get an idea of what
> >> the API looks like:
> >> https://github.com/JetBrains/kotlin-spark-api/tree/master/examples/src/main/kotlin/org/jetbrains/spark/api/examples
> >>
> >> We’d love to see your feedback in the project’s GitHub issues:
> >> https://github.com/JetBrains/kotlin-spark-api/issues.
> >>
> >>
> >> Thanks!
> >>
> >>
> >>

-- 
Regards,
Pasha

Big Data Tools @ JetBrains


signature.asc
Description: PGP signature

Re: Running Apache Spark Streaming on the GraalVM Native Image

[Pasha Finkelshteyn]

Hi Ivo, 

I believe there's absolutely no way that Spark will work on GraalVM
Native Image because Spark generates code and loads classes in runtime,
while GraalVM Native Image works only in closed world and has no any way
to load classes which are not present in classpath at compie time.


On 20/07/01 09:56AM, ivo.kn...@t-online.de wrote:
> Hi guys,
>  
> so I want to get Apache Spark to run on the GraalVM Native Image in a 
> simple single-node streaming application, but I get the following error, 
> when trying to build the native image: (check attached file)
>  
> And as I researched online, there seems to be no successful combination of 
> Spark and GraalVM Native Image. Did anyone ever succeed and how?
>  
> Best regards,
>  
> Ivo
>  
> 


> 
> -
> To unsubscribe e-mail: user-unsubscr...@spark.apache.org


-- 
Regards,
Pasha

Big Data Tools @ JetBrains


signature.asc
Description: PGP signature

Re: Different execution results with wholestage codegen on and off

[Pasha Finkelshteyn]

Here[1] it is, please review

[1] https://issues.apache.org/jira/browse/SPARK-31854
On 20/05/27 10:21PM, Xiao Li wrote:
> Thanks for reporting it. Please open a JIRA with a test case.
> 
> Cheers,
> 
> Xiao
> 
> On Wed, May 27, 2020 at 1:42 PM Pasha Finkelshteyn <
> pavel.finkelsht...@gmail.com> wrote:
> 
> > Hi folks,
> >
> > I'm implementing Kotlin bindings for Spark and faced strange problem. In
> > one cornercase Spark works differently when wholestage codegen is on or
> > off.
> >
> > Does it look like bug ot expected behavior?
> > --
> > Regards,
> > Pasha
> >
> > Big Data Tools @ JetBrains
> >
> 
> 
> -- 
> <https://databricks.com/sparkaisummit/north-america>

-- 
Regards,
Pasha

Big Data Tools @ JetBrains


signature.asc
Description: PGP signature

Different execution results with wholestage codegen on and off

[Pasha Finkelshteyn]

Hi folks,

I'm implementing Kotlin bindings for Spark and faced strange problem. In
one cornercase Spark works differently when wholestage codegen is on or
off.

Does it look like bug ot expected behavior?
-- 
Regards,
Pasha

Big Data Tools @ JetBrains


signature.asc
Description: PGP signature