There is no good answer to the question “Have I learnt enough”. You can never learn enough. You have to constrantly learn
Practically, if you want to make a career out of using technology XYZ, you only need to learn enough XYZ to get a job doing XYZ. Once you get a job doing XYZ, other people are paying you to learn more XYZ. You want to know if you know Python enough to do PySpark, look at what companies are asking for. Go for interviews. Just speaking from experience, most jobs that call for Python + Spark tend to be data science jobs. These jobs also require you to have a data science background. From: ayan guha <guha.a...@gmail.com> Date: Wednesday, April 14, 2021 at 3:55 PM To: "ashok34...@yahoo.com.INVALID" <ashok34...@yahoo.com.invalid> Cc: "user@spark.apache.org" <user@spark.apache.org> Subject: RE: [EXTERNAL] Python level of knowledge for Spark and PySpark CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. The answer always is "it depends". At the outset it seems you are in pretty good shape and have all the key skills you need. All I can suggest is try to take inherent benefits of the language and hone your coding practices On Thu, 15 Apr 2021 at 2:25 am, ashok34...@yahoo.com.INVALID <ashok34...@yahoo.com.invalid> wrote: Hi gurus, I have knowledge of Java, Scala and good enough knowledge of Spark, Spark SQL and Spark Functional programing with Scala. I have started using Python with Spark PySpark. Wondering, in order to be proficient in PySpark, how much good knowledge of Python programing is needed? I know the answer may be very good knowledge, but in practice how much is good enough. I can write Python in IDE like PyCharm similar to the way Scala works and can run the programs. Does expert knowledge of Python is prerequisite for PySpark? I also know Pandas and am also familiar with plotting routines like matplotlib. Warmest Ashok -- Best Regards, Ayan Guha
