HI Sujeet,
Thinking that might not work
Running this:
#!/usr/bin/env python3
from pyspark_cassandra import CassandraSparkContext, Row
from pyspark import SparkContext, SparkConf
from pyspark.sql import SQLContext
conf =
SparkConf().setAppName("test").setMaster("spark://192.168.23.31:7077").set("spark.cassandra.connection.host",
"192.168.23.31")
sc = CassandraSparkContext(conf=conf)
sqlContext = SQLContext(sc)
df =
sqlContext.read.format("org.apache.spark.sql.cassandra").options(keyspace="lebara_diameter_codes",
table="nl_lebara_diameter_codes").load()
list = df.select("errorcode2001").where("errorcode2001 > 1200").collect()
list2 = df.select("date").collect()
print([i for i in list[0]])
print(type(list[0]))
of course show this error:
py4j.protocol.Py4JJavaError: An error occurred while calling o29.load.
: java.lang.ClassNotFoundException: Failed to find data source:
org.apache.spark.sql.cassandra. Please find packages at
http://spark-packages.org
Is there a way to load up those jar files into the script
Jo
From: sujeet jog [mailto:sujeet....@gmail.com]
Sent: 29 June 2016 14:51
To: Joaquin Alzola <joaquin.alz...@lebara.com>; user <user@spark.apache.org>
Subject: Re: Spark jobs
check if this helps,
from multiprocessing import Process
def training() :
print ("Training Workflow")
cmd = spark/bin/spark-submit ./ml.py & "
os.system(cmd)
w_training = Process(target = training)
On Wed, Jun 29, 2016 at 6:28 PM, Joaquin Alzola
<joaquin.alz...@lebara.com<mailto:joaquin.alz...@lebara.com>> wrote:
Hi,
This is a totally newbie question but I seem not to find the link ….. when I
create a spark-submit python script to be launch …
how should I call it from the main python script with a subprocess.popen?
BR
Joaquin
This email is confidential and may be subject to privilege. If you are not the
intended recipient, please do not copy or disclose its content but contact the
sender immediately upon receipt.
This email is confidential and may be subject to privilege. If you are not the
intended recipient, please do not copy or disclose its content but contact the
sender immediately upon receipt.
