I'm sorry you were offended. I'm not an expert in Python and I wasn't trying to attack you personally. It's just an opinion about what makes a language better or worse, it's not the single source of truth. You don't have to take offense. In the end its about context and what you're trying to achieve under what circumstances.
I know a little about both programming and ETL. To say I know nothing is taking it a bit far. I don't know everything worth to know, that's for sure and goes without saying. It's fine to love Python and good for you being able to write Python programs wiping Java commercial stacks left and right. It's just my opinion that mutable dynamically typed languages encourage/enforce bad habits. The larger the application and team gets, the worse off you are (again just an opinion). Not everyone agrees (just look at Pythons popularity) but it's definitely a relevant aspect when deciding going Spark or Pyspark. br, molotch On Sat, 17 Oct 2020 at 16:40, Sasha Kacanski <skacan...@gmail.com> wrote: > And you are an expert on python! Idiomatic... > Please do everyone a favor and stop commenting on things you have no > idea... > I build ETL systems python that wiped java commercial stacks left and > right. Pyspark was and is and will be a second class citizen in spark > world. That has nothing to do with python. > And as far as scala is concerned good luck with it... > > > > > > On Sat, Oct 17, 2020, 8:53 AM Molotch <ma...@kth.se> wrote: > >> I would say the pros and cons of Python vs Scala is both down to Spark, >> the >> languages in themselves and what kind of data engineer you will get when >> you >> try to hire for the different solutions. >> >> With Pyspark you get less functionality and increased complexity with the >> py4j java interop compared to vanilla Spark. Why would you want that? >> Maybe >> you want the Python ML tools and have a clear use case, then go for it. If >> not, avoid the increased complexity and reduced functionality of Pyspark. >> >> Python vs Scala? Idiomatic Python is a lesson in bad programming >> habits/ideas, there's no other way to put it. Do you really want >> programmers >> enjoying coding i such a language hacking away at your system? >> >> Scala might be far from perfect with the plethora of ways to express >> yourself. But Python < 3.5 is not fit for anything except simple scripting >> IMO. >> >> Doing exploratory data analysis in a Jupiter notebook, Pyspark seems like >> a >> fine idea. Coding an entire ETL library including state management, the >> whole kitchen including the sink, Scala everyday of the week. >> >> >> >> -- >> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ >> >> --------------------------------------------------------------------- >> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >> >>
