Hi, I have created my own Model Tuner class which I want to use to tune models and return a Model object if the user expects. This Model Tuner is in a file which I would ideally import into another file and call the class and use it.
Outer file {from where I'd be calling the Model Tuner): I am using SparkSession to start my Spark job and then read a file, do some basic operations to prepare it as a Dense Vector and then send it into the Model to tune. While I send it into the model, I get this *error*: *AttributeError: Cannot load _jvm from SparkContext. Is SparkContext initialized?* The *full traceback*: Traceback (most recent call last): File "/home/aakashbasu/PycharmProjects/bb_lite/2.0 backend/model_tuning/aakash_test.py", line 9, in <module> from model_tuning.model_tuning_newer import ModelTuner File "/home/aakashbasu/PycharmProjects/bb_lite/2.0 backend/model_tuning/model_tuning.py", line 13, in <module> labelCol="label", predictionCol="prediction", metricName="accuracy") File "/usr/local/lib/python3.5/dist-packages/pyspark/__init__.py", line 105, in wrapper return func(self, **kwargs) File "/usr/local/lib/python3.5/dist-packages/pyspark/ml/evaluation.py", line 298, in __init__ "org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator", self.uid) File "/usr/local/lib/python3.5/dist-packages/pyspark/ml/wrapper.py", line 59, in _new_java_obj java_obj = _jvm() File "/usr/local/lib/python3.5/dist-packages/pyspark/ml/util.py", line 44, in _jvm raise AttributeError("Cannot load _jvm from SparkContext. Is SparkContext initialized?") AttributeError: Cannot load _jvm from SparkContext. Is SparkContext initialized? How to ensure that the inner file also uses the same Spark Context which is initiated by the outer file using SparkSession builder? Any help? Thanks, Aakash.