- dev@, which is more for project devs to communicate. Cross-posting is discouraged too.
The book isn't from the Spark OSS project, so not really the place to give feedback here. I don't quite understand the context of your other questions, but would elaborate them in individual, clear emails instead to increase the chance that someone will answer. On Fri, Mar 27, 2020 at 4:49 PM Zahid Rahman <zahidr1...@gmail.com> wrote: > > > I was very impressed with the amount of material available from > https://github.com/databricks/Spark-The-Definitive-Guide/ > Over 450+ megabytes. > > I have a corrected the scala code by adding > .sort(desc("sum(total_cost)")) to the code provided on page 34 (see below). > > I have noticed numerous uses of exclamation marks almost over use. > for example: > page 23: Let's specify some more transformatrions ! > page 24: you've read your first explain plan ! > page 26: Notice that these plans compile to the exactsame underlying plan ! > page 29: The last step is our action ! > page 34: The best thing about structured streaming ....rapidly... with > virtually no code > > 1. I have never read a science book with such emotion of frustration. > Is Spark difficult to understand made more complicated with the > proliferation of languages > scala , Java , python SQL R. > > 2. Secondly, Is spark architecture made more complex due to competing > technologies ? > > I have spark cluster setup with master and slave to load balancing heavy > activity like so: > sbin/start-master.sh > sbin/start-slave.sh spark://192.168.0.38:7077 > for load balancing I imagine, conceptually speaking, although I haven't > tried it , I can have as many > slaves(workers) on other physical machines by simply downloading spark zip > file > and running workers from those other physical machine(s) with > sbin/start-slave.sh spark://192.168.0.38:7077. > My question is under the circumstances do I need to bother with mesos or yarn > ? > > Collins dictionary > The exclamation mark is used after exclamations and emphatic expressions. > > I can’t believe it! > Oh, no! Look at this mess! > > The exclamation mark loses its effect if it is overused. It is better to use > a full stop after a sentence expressing mild excitement or humour. > > It was such a beautiful day. > I felt like a perfect banana. > > > import org.apache.spark.sql.SparkSession > import org.apache.spark.sql.functions.{window,column,desc,col} > > object RetailData { > > def main(args: Array[String]): Unit = { > > val spark = > SparkSession.builder().master("spark://192.168.0.38:7077").appName("Retail > Data").getOrCreate(); > > // create a static frame > val staticDataFrame = spark.read.format("csv") > .option ("header","true") > .option("inferschema","true") > .load("/data/retail-data/by-day/*.csv") > > staticDataFrame.createOrReplaceTempView("retail_data") > val staticFrame = staticDataFrame.schema > > staticDataFrame > .selectExpr( > "CustomerId","UnitPrice * Quantity as total_cost", "InvoiceDate") > .groupBy(col("CustomerId"), window(col("InvoiceDate"), "1 day")) > .sum("total_cost") > .sort(desc("sum(total_cost)")) > .show(1) > > } // main > > } // object > > > > Backbutton.co.uk > ¯\_(ツ)_/¯ > ♡۶Java♡۶RMI ♡۶ > Make Use Method {MUM} > makeuse.org --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
