[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16133565#comment-16133565 ]
Stavros Kontopoulos edited comment on SPARK-21752 at 8/18/17 8:19 PM: ---------------------------------------------------------------------- [~jsnowacki] What I am doing is not manual its just another legitimate way to start jupyter. Btw its far from manual as it works out of the box, but anyway the point here is Spark's config api consistency (since its a public API). I agree the other way to start things is more pythonic and that way is very manual IMHO but its ok since its common practice (that is why I insisted for all the details). Now I did use pip install pyspark, then `jupyter notebook` (or could have just used plain python) and followed your example and I got the following (some results verify what you have already observed): a) setting env variable always works without caring about configuration: {code:java} import pyspark import os os.environ['PYSPARK_SUBMIT_ARGS'] = '--packages org.mongodb.spark:mongo-spark-connector_2.11:2.2.0 pyspark-shell' spark = pyspark.sql.SparkSession.builder\ .appName('test-mongo')\ .master('local[*]')\ .getOrCreate() {code} [I 22:50:58.697 NotebookApp] Adapting to protocol v5.1 for kernel d05897ed-6de4-4ec2-842f-adb094bf0f0d Ivy Default Cache set to: /home/stavros/.ivy2/cache The jars for the packages stored in: /home/stavros/.ivy2/jars :: loading settings :: url = jar:file:/usr/local/lib/python3.5/dist-packages/pyspark/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml org.mongodb.spark#mongo-spark-connector_2.11 added as a dependency :: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0 confs: [default] found org.mongodb.spark#mongo-spark-connector_2.11;2.2.0 in central found org.mongodb#mongo-java-driver;3.4.2 in central :: resolution report :: resolve 160ms :: artifacts dl 3ms :: modules in use: org.mongodb#mongo-java-driver;3.4.2 from central in [default] org.mongodb.spark#mongo-spark-connector_2.11;2.2.0 from central in [default] --------------------------------------------------------------------- | | modules || artifacts | | conf | number| search|dwnlded|evicted|| number|dwnlded| --------------------------------------------------------------------- | default | 2 | 0 | 0 | 0 || 2 | 0 | --------------------------------------------------------------------- :: retrieving :: org.apache.spark#spark-submit-parent confs: [default] 0 artifacts copied, 2 already retrieved (0kB/5ms) Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 17/08/18 22:52:05 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 17/08/18 22:52:05 WARN Utils: Your hostname, universe resolves to a loopback address: 127.0.1.1; using 192.168.2.7 instead (on interface wlp2s0) 17/08/18 22:52:05 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address b) Example 1 for me works without issues: {code:java} import pyspark spark = pyspark.sql.SparkSession.builder\ .appName('test-mongo')\ .master('local[*]')\ .config("spark.jars.packages", "org.mongodb.spark:mongo-spark-connector_2.11:2.2.0")\ .config("spark.mongodb.input.uri", "mongodb://mongo/test.coll") \ .config("spark.mongodb.output.uri", "mongodb://mongo/test.coll") \ .getOrCreate() {code} Output: Creating new notebook in [I 23:03:52.055 NotebookApp] Kernel started: bc93a17a-e7a5-4e83-8a63-df0adba97c79 [W 23:03:52.058 NotebookApp] 404 GET /nbextensions/widgets/notebook/js/extension.js?v=20170818230343 (127.0.0.1) 1.46ms referer=http://localhost:8888/notebooks/Untitled2.ipynb?kernel_name=python3 [I 23:04:21.361 NotebookApp] Adapting to protocol v5.1 for kernel bc93a17a-e7a5-4e83-8a63-df0adba97c79 Ivy Default Cache set to: /home/stavros/.ivy2/cache The jars for the packages stored in: /home/stavros/.ivy2/jars :: loading settings :: url = jar:file:/usr/local/lib/python3.5/dist-packages/pyspark/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml org.mongodb.spark#mongo-spark-connector_2.11 added as a dependency :: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0 confs: [default] found org.mongodb.spark#mongo-spark-connector_2.11;2.2.0 in central found org.mongodb#mongo-java-driver;3.4.2 in central :: resolution report :: resolve 169ms :: artifacts dl 4ms :: modules in use: org.mongodb#mongo-java-driver;3.4.2 from central in [default] org.mongodb.spark#mongo-spark-connector_2.11;2.2.0 from central in [default] --------------------------------------------------------------------- | | modules || artifacts | | conf | number| search|dwnlded|evicted|| number|dwnlded| --------------------------------------------------------------------- | default | 2 | 0 | 0 | 0 || 2 | 0 | --------------------------------------------------------------------- :: retrieving :: org.apache.spark#spark-submit-parent confs: [default] 0 artifacts copied, 2 already retrieved (0kB/5ms) Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 17/08/18 23:04:22 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 17/08/18 23:04:22 WARN Utils: Your hostname, universe resolves to a loopback address: 127.0.1.1; using 192.168.2.7 instead (on interface wlp2s0) 17/08/18 23:04:22 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address c) Example 2 works as expected: {code:java} import pyspark conf = pyspark.SparkConf() conf.set("spark.jars.packages", "org.mongodb.spark:mongo-spark-connector_2.11:2.2.0") conf.set("spark.mongodb.input.uri", "mongodb://mongo/test.coll") conf.set("spark.mongodb.output.uri", "mongodb://mongo/test.coll") spark = pyspark.sql.SparkSession.builder\ .appName('test-mongo')\ .master('local[*]')\ .config(conf=conf)\ .getOrCreate() {code} [I 23:07:13.494 NotebookApp] Creating new notebook in [I 23:07:13.836 NotebookApp] Kernel started: c61a540b-86a2-4b9e-927f-66f977b42b0f [W 23:07:13.840 NotebookApp] 404 GET /nbextensions/widgets/notebook/js/extension.js?v=20170818230708 (127.0.0.1) 2.09ms referer=http://localhost:8888/notebooks/Untitled3.ipynb?kernel_name=python3 [I 23:07:14.353 NotebookApp] Adapting to protocol v5.1 for kernel c61a540b-86a2-4b9e-927f-66f977b42b0f [W 23:07:16.413 NotebookApp] Replacing stale connection: bc93a17a-e7a5-4e83-8a63-df0adba97c79:D7862E2BCD2F4111883A382D5EA7714D Ivy Default Cache set to: /home/stavros/.ivy2/cache The jars for the packages stored in: /home/stavros/.ivy2/jars :: loading settings :: url = jar:file:/usr/local/lib/python3.5/dist-packages/pyspark/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml org.mongodb.spark#mongo-spark-connector_2.11 added as a dependency :: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0 confs: [default] found org.mongodb.spark#mongo-spark-connector_2.11;2.2.0 in central found org.mongodb#mongo-java-driver;3.4.2 in central :: resolution report :: resolve 160ms :: artifacts dl 3ms :: modules in use: org.mongodb#mongo-java-driver;3.4.2 from central in [default] org.mongodb.spark#mongo-spark-connector_2.11;2.2.0 from central in [default] --------------------------------------------------------------------- | | modules || artifacts | | conf | number| search|dwnlded|evicted|| number|dwnlded| --------------------------------------------------------------------- | default | 2 | 0 | 0 | 0 || 2 | 0 | --------------------------------------------------------------------- :: retrieving :: org.apache.spark#spark-submit-parent confs: [default] 0 artifacts copied, 2 already retrieved (0kB/4ms) Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 17/08/18 23:07:18 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 17/08/18 23:07:18 WARN Utils: Your hostname, universe resolves to a loopback address: 127.0.1.1; using 192.168.2.7 instead (on interface wlp2s0) 17/08/18 23:07:18 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address For all cases I started with a simple python kernel and started fresh with a new notebook. Java gateway is always called due to the build of the spark session object and passes config options to the spark submit logic, at least this is what I observed. was (Author: skonto): [~jsnowacki] What I am doing is not manual its just another legitimate way to start jupyter. Btw its far from manual as it works out of the box, but anyway the point here is Spark's config api consistency (since its a public API). I agree the other way to start things is more pythonic and that way is very manual IMHO but its ok since its common practice (that is why I insisted for all the details). Now I did use pip install pyspark, then `jupyter notebook` (or could just use plain python) and followed your example and I got the following (some results verify what you have already observed): a) setting env variable always works without caring about configuration: {code:java} import pyspark import os os.environ['PYSPARK_SUBMIT_ARGS'] = '--packages org.mongodb.spark:mongo-spark-connector_2.11:2.2.0 pyspark-shell' spark = pyspark.sql.SparkSession.builder\ .appName('test-mongo')\ .master('local[*]')\ .getOrCreate() {code} [I 22:50:58.697 NotebookApp] Adapting to protocol v5.1 for kernel d05897ed-6de4-4ec2-842f-adb094bf0f0d Ivy Default Cache set to: /home/stavros/.ivy2/cache The jars for the packages stored in: /home/stavros/.ivy2/jars :: loading settings :: url = jar:file:/usr/local/lib/python3.5/dist-packages/pyspark/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml org.mongodb.spark#mongo-spark-connector_2.11 added as a dependency :: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0 confs: [default] found org.mongodb.spark#mongo-spark-connector_2.11;2.2.0 in central found org.mongodb#mongo-java-driver;3.4.2 in central :: resolution report :: resolve 160ms :: artifacts dl 3ms :: modules in use: org.mongodb#mongo-java-driver;3.4.2 from central in [default] org.mongodb.spark#mongo-spark-connector_2.11;2.2.0 from central in [default] --------------------------------------------------------------------- | | modules || artifacts | | conf | number| search|dwnlded|evicted|| number|dwnlded| --------------------------------------------------------------------- | default | 2 | 0 | 0 | 0 || 2 | 0 | --------------------------------------------------------------------- :: retrieving :: org.apache.spark#spark-submit-parent confs: [default] 0 artifacts copied, 2 already retrieved (0kB/5ms) Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 17/08/18 22:52:05 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 17/08/18 22:52:05 WARN Utils: Your hostname, universe resolves to a loopback address: 127.0.1.1; using 192.168.2.7 instead (on interface wlp2s0) 17/08/18 22:52:05 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address b) Example 1 for me works without issues: {code:java} import pyspark spark = pyspark.sql.SparkSession.builder\ .appName('test-mongo')\ .master('local[*]')\ .config("spark.jars.packages", "org.mongodb.spark:mongo-spark-connector_2.11:2.2.0")\ .config("spark.mongodb.input.uri", "mongodb://mongo/test.coll") \ .config("spark.mongodb.output.uri", "mongodb://mongo/test.coll") \ .getOrCreate() {code} Output: Creating new notebook in [I 23:03:52.055 NotebookApp] Kernel started: bc93a17a-e7a5-4e83-8a63-df0adba97c79 [W 23:03:52.058 NotebookApp] 404 GET /nbextensions/widgets/notebook/js/extension.js?v=20170818230343 (127.0.0.1) 1.46ms referer=http://localhost:8888/notebooks/Untitled2.ipynb?kernel_name=python3 [I 23:04:21.361 NotebookApp] Adapting to protocol v5.1 for kernel bc93a17a-e7a5-4e83-8a63-df0adba97c79 Ivy Default Cache set to: /home/stavros/.ivy2/cache The jars for the packages stored in: /home/stavros/.ivy2/jars :: loading settings :: url = jar:file:/usr/local/lib/python3.5/dist-packages/pyspark/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml org.mongodb.spark#mongo-spark-connector_2.11 added as a dependency :: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0 confs: [default] found org.mongodb.spark#mongo-spark-connector_2.11;2.2.0 in central found org.mongodb#mongo-java-driver;3.4.2 in central :: resolution report :: resolve 169ms :: artifacts dl 4ms :: modules in use: org.mongodb#mongo-java-driver;3.4.2 from central in [default] org.mongodb.spark#mongo-spark-connector_2.11;2.2.0 from central in [default] --------------------------------------------------------------------- | | modules || artifacts | | conf | number| search|dwnlded|evicted|| number|dwnlded| --------------------------------------------------------------------- | default | 2 | 0 | 0 | 0 || 2 | 0 | --------------------------------------------------------------------- :: retrieving :: org.apache.spark#spark-submit-parent confs: [default] 0 artifacts copied, 2 already retrieved (0kB/5ms) Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 17/08/18 23:04:22 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 17/08/18 23:04:22 WARN Utils: Your hostname, universe resolves to a loopback address: 127.0.1.1; using 192.168.2.7 instead (on interface wlp2s0) 17/08/18 23:04:22 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address c) Example 2 works as expected: {code:java} import pyspark conf = pyspark.SparkConf() conf.set("spark.jars.packages", "org.mongodb.spark:mongo-spark-connector_2.11:2.2.0") conf.set("spark.mongodb.input.uri", "mongodb://mongo/test.coll") conf.set("spark.mongodb.output.uri", "mongodb://mongo/test.coll") spark = pyspark.sql.SparkSession.builder\ .appName('test-mongo')\ .master('local[*]')\ .config(conf=conf)\ .getOrCreate() {code} [I 23:07:13.494 NotebookApp] Creating new notebook in [I 23:07:13.836 NotebookApp] Kernel started: c61a540b-86a2-4b9e-927f-66f977b42b0f [W 23:07:13.840 NotebookApp] 404 GET /nbextensions/widgets/notebook/js/extension.js?v=20170818230708 (127.0.0.1) 2.09ms referer=http://localhost:8888/notebooks/Untitled3.ipynb?kernel_name=python3 [I 23:07:14.353 NotebookApp] Adapting to protocol v5.1 for kernel c61a540b-86a2-4b9e-927f-66f977b42b0f [W 23:07:16.413 NotebookApp] Replacing stale connection: bc93a17a-e7a5-4e83-8a63-df0adba97c79:D7862E2BCD2F4111883A382D5EA7714D Ivy Default Cache set to: /home/stavros/.ivy2/cache The jars for the packages stored in: /home/stavros/.ivy2/jars :: loading settings :: url = jar:file:/usr/local/lib/python3.5/dist-packages/pyspark/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml org.mongodb.spark#mongo-spark-connector_2.11 added as a dependency :: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0 confs: [default] found org.mongodb.spark#mongo-spark-connector_2.11;2.2.0 in central found org.mongodb#mongo-java-driver;3.4.2 in central :: resolution report :: resolve 160ms :: artifacts dl 3ms :: modules in use: org.mongodb#mongo-java-driver;3.4.2 from central in [default] org.mongodb.spark#mongo-spark-connector_2.11;2.2.0 from central in [default] --------------------------------------------------------------------- | | modules || artifacts | | conf | number| search|dwnlded|evicted|| number|dwnlded| --------------------------------------------------------------------- | default | 2 | 0 | 0 | 0 || 2 | 0 | --------------------------------------------------------------------- :: retrieving :: org.apache.spark#spark-submit-parent confs: [default] 0 artifacts copied, 2 already retrieved (0kB/4ms) Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 17/08/18 23:07:18 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 17/08/18 23:07:18 WARN Utils: Your hostname, universe resolves to a loopback address: 127.0.1.1; using 192.168.2.7 instead (on interface wlp2s0) 17/08/18 23:07:18 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address For all cases I started with a simple python kernel and started fresh with a new notebook. Java gateway is always called due to the build of the spark session object and passes config options to the spark submit logic, at least this is what I observed. > Config spark.jars.packages is ignored in SparkSession config > ------------------------------------------------------------ > > Key: SPARK-21752 > URL: https://issues.apache.org/jira/browse/SPARK-21752 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.2.0 > Reporter: Jakub Nowacki > > If I put a config key {{spark.jars.packages}} using {{SparkSession}} builder > as follows: > {code} > spark = pyspark.sql.SparkSession.builder\ > .appName('test-mongo')\ > .master('local[*]')\ > .config("spark.jars.packages", > "org.mongodb.spark:mongo-spark-connector_2.11:2.2.0")\ > .config("spark.mongodb.input.uri", "mongodb://mongo/test.coll") \ > .config("spark.mongodb.output.uri", "mongodb://mongo/test.coll") \ > .getOrCreate() > {code} > the SparkSession gets created but there are no package download logs printed, > and if I use the loaded classes, Mongo connector in this case, but it's the > same for other packages, I get {{java.lang.ClassNotFoundException}} for the > missing classes. > If I use the config file {{conf/spark-defaults.comf}}, command line option > {{--packages}}, e.g.: > {code} > import os > os.environ['PYSPARK_SUBMIT_ARGS'] = '--packages > org.mongodb.spark:mongo-spark-connector_2.11:2.2.0 pyspark-shell' > {code} > it works fine. Interestingly, using {{SparkConf}} object works fine as well, > e.g.: > {code} > conf = pyspark.SparkConf() > conf.set("spark.jars.packages", > "org.mongodb.spark:mongo-spark-connector_2.11:2.2.0") > conf.set("spark.mongodb.input.uri", "mongodb://mongo/test.coll") > conf.set("spark.mongodb.output.uri", "mongodb://mongo/test.coll") > spark = pyspark.sql.SparkSession.builder\ > .appName('test-mongo')\ > .master('local[*]')\ > .config(conf=conf)\ > .getOrCreate() > {code} > The above is in Python but I've seen the behavior in other languages, though, > I didn't check R. > I also have seen it in older Spark versions. > It seems that this is the only config key that doesn't work for me via the > {{SparkSession}} builder config. > Note that this is related to creating new {{SparkSession}} as getting new > packages into existing {{SparkSession}} doesn't indeed make sense. Thus this > will only work with bare Python, Scala or Java, and not on {{pyspark}} or > {{spark-shell}} as they create the session automatically; it this case one > would need to use {{--packages}} option. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org