Hi Users I have a wheel file , while creating it I have mentioned dependencies in setup.py file. Now I have 2 virtual envs, 1 was already there . another one I created just now.
I have switched to new virtual env, I want spark to download the dependencies while doing spark-submit using wheel. Could you please help me on this? It is not downloading dependencies , instead it is pointing to older version of virtual env and proceeding with the execution of spark job. Please note I have tried setting the env variables also. Also I have tried following options as well in spark submit --conf spark.pyspark.virtualenv.enabled=true --conf spark.pyspark.virtualenv.type=native --conf spark.pyspark.virtualenv.requirements=requirements.txt --conf spark.pyspark.python= /path/to/venv/bin/python3 --conf spark.pyspark.driver.python=/path/to/venv/bin/python3 This did not help too.. Kind Regards, Sachit Murarka
