Re: spark context list_packages()

[Sean Owen]

There is no such method in Spark. I think that's some EMR-specific
modification.

On Wed, Jul 26, 2023 at 11:06 PM second_co...@yahoo.com.INVALID
 wrote:

> I ran the following code
>
> spark.sparkContext.list_packages()
>
> on spark 3.4.1 and i get below error
>
> An error was encountered:
> AttributeError
> [Traceback (most recent call last):
> ,   File "/tmp/spark-3d66c08a-08a3-4d4e-9fdf-45853f65e03d/shell_wrapper.py", 
> line 113, in exec
> self._exec_then_eval(code)
> ,   File "/tmp/spark-3d66c08a-08a3-4d4e-9fdf-45853f65e03d/shell_wrapper.py", 
> line 106, in _exec_then_eval
> exec(compile(last, '', 'single'), self.globals)
> ,   File "", line 1, in 
> , AttributeError: 'SparkContext' object has no attribute 'list_packages'
> ]
>
>
> Is list_packages and install_pypi_package available for vanilla spark or
> only available for AWS services?
>
>
> Thank you
>

spark context list_packages()

[second_co...@yahoo.com.INVALID]

I ran the following code 

spark.sparkContext.list_packages()
on spark 3.4.1 and i get below error
An error was encountered:
AttributeError
[Traceback (most recent call last):
,   File "/tmp/spark-3d66c08a-08a3-4d4e-9fdf-45853f65e03d/shell_wrapper.py", 
line 113, in exec
self._exec_then_eval(code)
,   File "/tmp/spark-3d66c08a-08a3-4d4e-9fdf-45853f65e03d/shell_wrapper.py", 
line 106, in _exec_then_eval
exec(compile(last, '', 'single'), self.globals)
,   File "", line 1, in 
, AttributeError: 'SparkContext' object has no attribute 'list_packages'
]
Is list_packages and install_pypi_package available for vanilla spark or only 
available for AWS services?

Thank you