RE: Choice of IDE for Spark

Hello. Any "evolved" code editor allows you to create tasks (or builds, or whatever they are called in the IDE you chose). If you do not find anything that packages by default all you need, you could just create your own tasks. For yarn, one needs to open a terminal and submit from there. You

Re: [Spark-Core] Spark Dry Run

Ok thanks. What is your experience of VS Code (in terms of capabilities ) as it is becoming a standard tool available in Cloud workspaces like Amazon workspace? Mich view my Linkedin profile *Disclaimer:* Use it at your own

Re: [Spark-Core] Spark Dry Run

Not anything specific in my mind. Any IDE which is open to plugins can use it (e.g: VS Code and Jetbrains) to validate execution plans in the background and mark syntax errors based on the result. On Thu, Sep 30, 2021 at 4:40 PM Mich Talebzadeh wrote: > What IDEs do you have in mind? > > > >

Re: [Spark-Core] Spark Dry Run

What IDEs do you have in mind? view my Linkedin profile *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this

Re: [Spark-Core] Spark Dry Run

Yeah it doesn't remove the need of testing on sample data. It would be more of syntax check rather than test. I have witnessed that syntax errors occur a lot. Maybe after having dry-run we will be able to create some automation around basic syntax checking for IDEs too. On Thu, Sep 30, 2021 at

Re: [Spark-Core] Spark Dry Run

If testing, wouldn't you actually want to execute things? even if at a small scale, on a sample of data? On Thu, Sep 30, 2021 at 9:07 AM Ali Behjati wrote: > Hey everyone, > > > By dry run I mean ability to validate the execution plan but not executing > it within the code. I was wondering

[Spark-Core] Spark Dry Run

Hey everyone, By dry run I mean ability to validate the execution plan but not executing it within the code. I was wondering whether this exists in spark or not. I couldn't find it anywhere. If it doesn't exist I want to propose adding such a feature in spark. Why is it useful? 1. Faster

Re: Choice of IDE for Spark

IIRC, you want an IDE for pyspark on yarn ? Mich Talebzadeh 于2021年9月30日周四 下午7:00写道： > Hi, > > This may look like a redundant question but it comes about because of the > advent of Cloud workstation usage like Amazon workspaces and others. > > With IntelliJ you are OK with Spark & Scala. With

Choice of IDE for Spark

Hi, This may look like a redundant question but it comes about because of the advent of Cloud workstation usage like Amazon workspaces and others. With IntelliJ you are OK with Spark & Scala. With PyCharm you are fine with PySpark and the virtual environment. Mind you as far as I know PyCharm