Spark is written in Scala. As a strongly typed functional programming language, Scala makes Spark possible. The API of Spark mirrors Scala collections API, driving adoption through beautifully intuitive data transformations which work both locally and at scale. And where's more inspiration where it came from -- we've seen how Scala and Spark influence each other at the Scala By the Bay developer conference and its own professional training courses for Scala and Spark.
When you learn Scala for Spark, you quickly find out that there’s much more to it than just the language. The REPL is great and you can start playing right away, but when the time comes to run in production, you need to make sure that the code is correct, deployed properly, scalable as needed. But most importantly for the developers, the path from experimenting in the REPL to writing code in an IDE (or vim) to testing to deployment should be as easy but rigorous as coding in Scala itself. It is only natural that Scala developers, running some of the biggest APIs and Big Data setups out there, would think carefully about the right ways to build a Scala-centric development flow. Scala runs on the JVM, which comes with its own ecosystem, toolsets, deployment, monitoring, and so on, which should be adapted to Scala ways selectively. Scala espouses the culture of correctness through strong typing and rigorous test-driven development. In addition to local tools, there are now many cloud-based systemsfor rapid prototyping and deployment, such as Heroku. If you’re building an API, for example, or another kind of service, you can spin VMs or Heroku versions to achieve a much higher fidelity in integration testing with easier setup and teardown. Git has also fundamentally changed the way we go about development in a team. It is an established best practice by now to develop on branches, code-review, and use TDD automation to verify that branches can be merged properly. There are tools to make sure a pull request builds properly before merging. When it comes to deployment, a process must be set up which will package the code and run it through staging to production with an ability to roll back if needed. Overall, a pragmatic programmer needs a tool chain which amplifies his/her language in a real-life flow of repeatable cycle of development, testing, and deployment. The Scala DevFlow workshop is the first professional training structured specifically around the best practices for developers who work in Scala and use agile practices, leading to continuous deployment. http://bythebay.ticketleap.com/scala-devops/ It is taught by Alexy Khrabrov and James Earl Douglas, the first two coding cofounders of Versal and meetup organizers, who drove the all-Scala backend and its development flow and devops processes. The experience led to new ways to develop APIs using TDD in Scala, where API tests are one-liners building the state of the API, with an example at https://github.com/Versal/fireotter. The devflow process was presented at ScalaDays 2013 and Scala By the Bay 2014 — links to the videos are at the workshop site. The workshop will be a collaborative two-day course where developers will program together to master the flow for themselves and their teams. If you’re running a startup on Scala, this is the most valuable focus, as we’ve learned through several years of systems building with it. We’ve structured the training in two days which can be attended together or separately. The first day covers the fundamental development flow, which will quickly align the team. It uses hosted tools. The second day goes into continuous integration/deployment, release management, and in-house setup. The early bird pricing runs through the end of the week, and we're offering a limited number of $100 discounts, please use code DEVFLOW100 (while available). Email any questions to al...@bythebay.io. Just like deeply thought-through functional programming practices make Scala great, properly engineered development flow makes a huge difference. Both sets of knowledge are community-centric -- it's what makes Scala great. Take advantage of this experience, and build more & better companies with it, faster! Hope to code & deploy with you soon. Cheers, A+ --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
