ML advice

Hi, I heard some big things about Machine learning and data science. To upgrade my skill set I took a udemy course Python and Spark for Big Data with Spark. It took about week to learn the concepts and the workflow to follow when using each of the Spark APIs. To complete a Machine Learning

Re: Watch "Airbus makes more of the sky with Spark - Jesse Anderson & Hassene Ben Salem" on YouTube

ut Apache Spark.They excluded Spark Streaming > for high latency reasons. > > Why are you posting this video on a Spark mailing list? > > Regards > > J. Strebel > Am 25.04.20 um 05:07 schrieb Zahid Rahman: > > > https://youtu.be/sYlbD_OoHhs > > Backbutton.co.u

Watch "Airbus makes more of the sky with Spark - Jesse Anderson & Hassene Ben Salem" on YouTube

https://youtu.be/sYlbD_OoHhs Backbutton.co.uk ¯\_(ツ)_/¯ ♡۶Java♡۶RMI ♡۶ Make Use Method {MUM} makeuse.org

Re: Can you view thread dumps on spark UI if job finished

I am using history server to see previous UIs. > > However, my question still remains on viewing old thread dumps, as I > cannot see them on the old completed spark UIs, only when spark context is > running. > > On Wed, Apr 8, 2020 at 4:01 PM Zahid Rahman wrote: > >>

Re: Can you view thread dumps on spark UI if job finished

Spark UI is only available while SparkContext is running. However You can get to the Spark UI after your application completes or crashes. To do this Spark includes a tool called the Spark History Server that allows you to reconstruct the Spark UI. You can find up to date information on how

IDE suitable for Spark

When I first logged on I asked if there was a suitable IDE for Spark. I did get a couple of responses. *Thanks.* I did actually find one which is suitable IDE for spark. That is *Apache Zeppelin.* One of many reasons it is suitable for Apache Spark is. The *up and running Stage* which involves

Beautiful Spark Code

me to his book. https://leanpub.com/beautiful-spark https://mungingdata.com/writing-beautiful-apache-spark2-code-with-scala/ On Sat, 28 Mar 2020, 16:49 Zahid Rahman, wrote: > Thanks for the tip! > > But if the first thing you come across > Is somebody using the trim function t

Re: BUG: spark.readStream .schema(staticSchema) not receiving schema information

; Spark, and has been for many years. He, and everyone else participating in > these lists, is doing it voluntarily on their own time. They're not being > paid to handhold you and quickly answer to your every whim. > > On Sat, Mar 28, 2020, 10:46 AM Zahid Rahman wrote: > >> So t

Re: OFF TOPIC LIST CRITERIA

vouch for @Sean Owen 's contribution in the > user group (and especially around CI/CD), your claims and tone in which > those are made are both sadly hilarious. Let us refrain from hurting each > other in this otherwise troubled times. > > Best > > Ayan > > > >

Re: OFF TOPIC LIST CRITERIA

and on-topic, use > user@. > Otherwise we will block you from the lists. > > > Sean > > On Fri, Mar 27, 2020 at 9:46 PM Zahid Rahman wrote: > > > > > > Sean Owen says the criteria of these two emailing list is not help to > support some body > > who is new b

OFF TOPIC LIST CRITERIA

Sean Owen says the criteria of these two emailing list is not help to support some body who is new but for people who have been using the software for a long time. He is implying I think that I should only send email when I find bugs so that I can help him in his work. A one way street. He is

Re: what a plava !

cross applications on > a cluster, which may be running more than Spark apps. > > On Fri, Mar 27, 2020 at 7:30 PM Zahid Rahman wrote: > > > > OK, Thanks. > > > > issue of load balancing /Clustering: > > > > I believe if I setup clustering like so : &g

Re: what a plava !

ease > the chance that someone will answer. > > On Fri, Mar 27, 2020 at 4:49 PM Zahid Rahman wrote: > > > > > > I was very impressed with the amount of material available from > https://github.com/databricks/Spark-The-Definitive-Guide/ > > Over 450+ megabytes. > &

what a plava !

I was very impressed with the amount of material available from https://github.com/databricks/Spark-The-Definitive-Guide/ Over 450+ * megabytes.*

Re: BUG: take with SparkSession.master[url]

~/spark-3.0.0-preview2-bin-hadoop2.7$ sbin/start-slave.sh spark:// 192.168.0.38:7077 ~/spark-3.0.0-preview2-bin-hadoop2.7$ sbin/start-master.sh Backbutton.co.uk ¯\_(ツ)_/¯ ♡۶Java♡۶RMI ♡۶ Make Use Method {MUM} makeuse.org <http://www.backbutton.co.uk> On Fri, 27 Mar 2020 at 06:12, Zahid

Re: BUG: take with SparkSession.master[url]

deployed if you > just include Spark dependency in IntelliJ? > > On Fri, Mar 27, 2020 at 1:54 PM Zahid Rahman wrote: > >> I have configured in IntelliJ as external jars >> spark-3.0.0-preview2-bin-hadoop2.7/jar >> >> not pulling anything from maven. >>

Re: BUG: take with SparkSession.master[url]

ich Spark/Scala version do you use? > > On Fri, Mar 27, 2020 at 1:24 PM Zahid Rahman wrote: > >> >> with the following sparksession configuration >> >> val spark = SparkSession.builder().master("local[*]").appName("Spark Session >> take").

BUG: take with SparkSession.master[url]

with the following sparksession configuration val spark = SparkSession.builder().master("local[*]").appName("Spark Session take").getOrCreate(); this line works flights.filter(flight_row => flight_row.ORIGIN_COUNTRY_NAME != "Canada").map(flight_row => flight_row).take(5) however if change the

Re: results of taken(3) not appearing in console window

he console because the Scala REPL automatically > prints the returned value from take. > > > On Thu, Mar 26, 2020 at 12:15 PM, Zahid Rahman > wrote: > >> I am running the same code with the same libraries but not getting same >> outpu

Re: results of taken(3) not appearing in console window

he results printed to the console because the Scala REPL automatically > prints the returned value from take. > > > On Thu, Mar 26, 2020 at 12:15 PM, Zahid Rahman > wrote: > >> I am running the same code with the same libraries but not getting same >> output. >>

Re: can we all help use our expertise to create an IT solution for Covid-19

"We can then donate this to WHO or others and we can make it very modular though microservices etc." I have no interest because there are 8 million muslims locked up in their home for 8 months by the Hindutwa (Indians) You didn't take any notice of them. Now you are locked up in your home and you

Re: Spark 3 Build Problem!!!

https://youtu.be/iarn1KHeouc You'll find step by step setup guide for linux and windows here. You need to do extra steps for windows. Spark is more user friendly towards *nix. You are better off downloading ubuntu 19. Hopefully ubuntu has the drivers for you PC. If ubuntu do not have the

Re: HDP 3.1 spark Kafka dependency

I have found many library incompatibility issues including JVM headless issues where I had to uninstall headless jvm and install jdk and work through them, anyway This page shows the same error as yours, you may get away with making the changes to your pom.xml as suggested.

yolk -V pyspark

Hi, When I use pip install Pyspark,Version 2.4.5 is picked up automatically. when I checked which versions are available using yolk I also got version 2.4.5 *$ yolk -V pysparkpyspark 2.4.5* Will version *pyspark 3.0.0. preview 2* will be *available* to install using pip soon ? Thanks

sample syntax in spark-env.sh for env.

WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable *I was chasing this warning when I found misinformationfrom SPARK training companies such as eudreka who offer weird and wonderful suggestions*

Re: Hostname :BUG

on developer ). I was thinking of raising a JIRA but I thought I would consult with fellow developers first. :) Please be kind. Backbutton.co.uk ¯\_(ツ)_/¯ ♡۶Java♡۶RMI ♡۶ Make Use Method {MUM} makeuse.org <http://www.backbutton.co.uk> On Mon, 9 Mar 2020 at 08:57, Zahid Rahman wrote: &

Re: Still incompatible schemas

*This issue of has been discussed resolved on this page * *https://issues.apache.org/jira/browse/SPARK-17557 * *It is suggested by one person that by simply reading the parquet file in a different way as illustrated the error may go away. It

Re: Hostname :BUG

() is reading from the etc/hostname and not etc/host file for ip_address. Backbutton.co.uk ¯\_(ツ)_/¯ ♡۶Java♡۶RMI ♡۶ Make Use Method {MUM} makeuse.org <http://www.backbutton.co.uk> On Wed, 4 Mar 2020 at 21:14, Andrew Melo wrote: > Hello Zabid, > > On Wed, Mar 4, 2020 at 1:47 PM Zahi

Re: Hostname :BUG

Talking about copy and paste Larry Tesler The *inventor* of *cut*/*copy* & *paste*, find & replace past away last week age 74. Backbutton.co.uk ¯\_(ツ)_/¯ ♡۶Java♡۶RMI ♡۶ Make Use Method {MUM} makeuse.org <http://www.backbutton.co.uk> On Thu, 5 Mar 2020 at 07:01, Zahid Rahman wr

Re: SPARK Suitable IDE

ood enough/I don't have time/This is the way I am >> <https://edu.51cto.com/sd/aca72> >><https://github.com/TianLangStudio> >> >> -- >> 发件人：Zahid Rahman >> 发送时间：2020年3月3日(星期二) 06:43 >> 收件人：user >> 主 题：

Re: Hostname :BUG

bid, > > On Wed, Mar 4, 2020 at 1:47 PM Zahid Rahman wrote: > >> Hi, >> >> I found the problem was because on my Linux Operating System the >> /etc/hostname was blank. >> >> *STEP 1* >> I searched on google the error message and there was

Hostname :BUG

Hi, I found the problem was because on my Linux Operating System the /etc/hostname was blank. *STEP 1* I searched on google the error message and there was an answer suggesting I should add to /etc/hostname 127.0.0.1 [hostname] localhost. I did that but there was still an error, this

SPARK Suitable IDE

Hi, Can you recommend a suitable IDE for Apache sparks from the list below or if you know a more suitable one ? Codeanywhere goormIDE Koding SourceLair ShiftEdit Browxy repl.it PaizaCloud IDE Eclipse Che Visual Studio Online Gitpod Google Cloud Shell Codio Codepen CodeTasty Glitch JSitor

command line build fail and warnings compare to IDE build success

Hi, When I run the same word count program from the command line and the IDE I get two different logging messages. The command line execution even says BUILD failed when the outcome of the program is success with the production of the directory word count and the resulting output file. I have

Re: Counting streaks in stateful structured streaming

>From this paragraph it appears the answer to your query is YES. page 334 Spark the definitive guide states : "Stream processing is the act of *continuously incorporating new data* to compute a result. In stream processing, the input data is *unbounded *and has *no predetermined beginning or

code changes to SimpleApp

Hi, I have made a number of code changes to make the SimpleApp work. Your error messages are quite good. package co.uk.backbutton.sparksimpleapp; /** * * @author zahid */ /* SimpleApp.java */ import org.apache.spark.SparkConf; import org.apache.spark.api.java.JavaRDD; import

setup pom.xml

Hi, On page https://spark.apache.org/docs/latest/quick-start.html if you change one line in pom.xml for SimpleApp from: spark-sql_2.12 to: spark-sql_2.11 Then the ease of use and simplicity would increase ten fold and the simple app would be even simpler. Backbutton.co.uk ¯\_(ツ)_/¯

Re: configuration error

SOLVED On Sat, 29 Feb 2020 at 17:23, Zahid Rahman wrote: > > Hi, > > I am running it on linux. > Is there a programmatic way to get rid of this security error or is it > configuration file error. > > zahid@localhost > :~/Downloads/apachespark/Apache-Spark-Ex

configuration error

Hi, I am running it on linux. Is there a programmatic way to get rid of this security error or is it configuration file error. zahid@localhost:~/Downloads/apachespark/Apache-Spark-Example/Java-Code-Geek> mvn exec:java -Dexec.mainClass=com.javacodegeek.examples.SparkExampleRDD