Re: spark structured streaming GroupState returns weird values from sate

Well, the code itself doesn't seem to be OK - you're using ProductStateInformation as the class of State whereas you provide ProductSessionInformation to Encoder for State. On Fri, Mar 27, 2020 at 11:14 PM Jungtaek Lim wrote: > Could you play with Encoders.bean()? You can Encoders.bean() with

Re: OFF TOPIC LIST CRITERIA

OK *user support. user@ is DONE !!!* I reported a work around to an existing bug actually to the experienced user. and "the experienced user" was "not aware" of the setting in the log4j.properties so he learned something new too. Clearly neither were you. Also it may surprise some people but

Re: OFF TOPIC LIST CRITERIA

BCC user, dev, and I encourage others to not reply. I said _dev@_ is not for user support. user@ is. You heard that yesterday, too, and not to cross-post. You actually got answers to several questions, despite their tone, from experienced developers of the project. Messages like yours are, I

OFF TOPIC LIST CRITERIA

Sean Owen says the criteria of these two emailing list is not help to support some body who is new but for people who have been using the software for a long time. He is implying I think that I should only send email when I find bugs so that I can help him in his work. A one way street. He is

Best Practice: Evaluate Expression from Spark DataFrame Column

Hi Spark Users, I want to evaluate expression from dataframe column values on other columns in the same dataframe for each row. Please suggest best approach to deal with this given that not impacting the performance of the job. Thanks Sample code: val sampleDF = Seq( (8, 1, "bat", "NUM IS

Re: what a plava !

That confirms the three technologies are competing for the same space as I suspected but wasn't sure. I can focus on the APIs and not waste any unnecessary time on even looking at mesos and yarn. Backbutton.co.uk ¯\_(ツ)_/¯ ♡۶Java♡۶RMI ♡۶ Make Use Method {MUM} makeuse.org

Re: what a plava !

Spark standalone is a resource manager like YARN and Mesos. It is specific to Spark, and is therefore simpler, as it assumes it can take over whole machines. YARN and Mesos are for mediating resource usage across applications on a cluster, which may be running more than Spark apps. On Fri, Mar

Re: what a plava !

OK, Thanks. issue of load balancing /Clustering: I believe if I setup clustering like so : sbin/start-master.sh sbin/start-slave spark://master:port *another machine* sbin/start-slave spark://master:port Does yarn and mesos do anything different than that ? The spark clustering setup and yarn

Re: what a plava !

- dev@, which is more for project devs to communicate. Cross-posting is discouraged too. The book isn't from the Spark OSS project, so not really the place to give feedback here. I don't quite understand the context of your other questions, but would elaborate them in individual, clear emails

what a plava !

I was very impressed with the amount of material available from https://github.com/databricks/Spark-The-Definitive-Guide/ Over 450+ * megabytes.*

Re: spark structured streaming GroupState returns weird values from sate

Could you play with Encoders.bean()? You can Encoders.bean() with your class, and call .schema() with the return value to see how it transforms to the schema in Spark SQL. The schema must be consistent across multiple JVM runs to make it work properly, but I suspect it doesn't retain the order.

spark structured streaming GroupState returns weird values from sate

I am listening to Kafka topic with a structured streaming application with Java, testing it on my local Mac. When I retrieve back GroupState object with state.get(), it is giving some random values for the fields in the object, some are interchanging some are default and some are junk values.

Re: BUG: take with SparkSession.master[url]

~/spark-3.0.0-preview2-bin-hadoop2.7$ sbin/start-slave.sh spark:// 192.168.0.38:7077 ~/spark-3.0.0-preview2-bin-hadoop2.7$ sbin/start-master.sh Backbutton.co.uk ¯\_(ツ)_/¯ ♡۶Java♡۶RMI ♡۶ Make Use Method {MUM} makeuse.org On Fri, 27 Mar 2020 at 06:12, Zahid Rahman

Re: BUG: take with SparkSession.master[url]

sbin/start-master.sh sbin/start-slave.sh spark://192.168.0.38:7077 Backbutton.co.uk ¯\_(ツ)_/¯ ♡۶Java♡۶RMI ♡۶ Make Use Method {MUM} makeuse.org On Fri, 27 Mar 2020 at 05:59, Wenchen Fan wrote: > Your Spark cluster, spark://192.168.0.38:7077, how is it deployed if

Re: BUG: take with SparkSession.master[url]

Your Spark cluster, spark://192.168.0.38:7077, how is it deployed if you just include Spark dependency in IntelliJ? On Fri, Mar 27, 2020 at 1:54 PM Zahid Rahman wrote: > I have configured in IntelliJ as external jars > spark-3.0.0-preview2-bin-hadoop2.7/jar > > not pulling anything from maven.