I have a simple question about using Spark, which although most tools usually explain this question explicitly (in important text, such as a specific format or a separate page), I did not find it anywhere. Maybe my search was not enough, but I thought it was good that I ask this question in the hope that maybe the answer will benefit other people as well.
Spark binary is usually downloaded from the following link and installed and configured on the cluster: Download Apache Spark<https://spark.apache.org/downloads.html> If, for example, we use the Java language for programming (although it can be other supported languages), we need the following dependencies to communicate with Spark: <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-core_2.12</artifactId> <version>3.2.0</version> </dependency> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-sql_2.12</artifactId> <version>3.2.0</version> </dependency> As is clear, both the Spark cluster (binary of Spark) and the dependencies used on the application side have a specific version. In my opinion, it is obvious that if the version used is the same on both the application side and the server side, everything will most likely work in its ideal state without any problems. But the question is, what if the two versions are not the same? Is it possible to have compatibility between the server and the application in specific number of conditions (such as not changing major version)? Or, for example, if the client is always ahead, is it not a problem? Or if the server is always ahead, is it not a problem? The argument is that there may be a library that I did not write and it is an old version, but I want to update my cluster (server version). Or it may not be possible for me to update the server version and all the applications version at the same time, so I want to update each one separately. As a result, the application-server version differs in a period of time. (maybe short or long period) I want to know exactly how Spark works in this situation.
