Hi Ted and Saurahb
If I use —conf arguments with pyspark I am able to connect. Any idea how I
can set these values programmatically? (I work on a notebook server and can
not easily reconfigure the server
This works
extraPkgs="--packages com.databricks:spark-csv_2.11:1.3.0 \
--packages datastax:spark-cassandra-connector:1.6.0-M1-s_2.10"
export PYSPARK_PYTHON=python3
export PYSPARK_DRIVER_PYTHON=python3
IPYTHON_OPTS=notebook $SPARK_ROOT/bin/pyspark $extraPkgs --conf
spark.cassandra.connection.host=localhost --conf
spark.cassandra.connection.port=9043 $*
df = sqlContext.read\
.format("org.apache.spark.sql.cassandra")\
.options(table="json_timeseries", keyspace="notification")\
.load()
df.printSchema()
df.show(truncate=False)
I have tried using setContext.setConf() but it does not work. It does not
seem to have any effect
#sqlContext.setConf("spark.cassandra.connection.host","localhost")
#sqlContext.setConf("spark.cassandra.connection.port","9043")
#sqlContext.setConf("connection.host","localhost")
#sqlContext.setConf("connection.port","9043")
sqlContext.setConf("host","localhost")
sqlContext.setConf("port","9043”)
Thanks
Andy
From: Saurabh Bajaj <bajaj.onl...@gmail.com>
Date: Tuesday, March 8, 2016 at 9:13 PM
To: Andrew Davidson <a...@santacruzintegration.com>
Cc: Ted Yu <yuzhih...@gmail.com>, "user @spark" <user@spark.apache.org>
Subject: Re: pyspark spark-cassandra-connector java.io.IOException: Failed
to open native connection to Cassandra at {192.168.1.126}:9042
> Hi Andy,
>
> I believe you need to set the host and port settings separately
> spark.cassandra.connection.host
> spark.cassandra.connection.port
> https://github.com/datastax/spark-cassandra-connector/blob/master/doc/referenc
> e.md#cassandra-connection-parameters
>
> Looking at the logs, it seems your port config is not being set and it's
> falling back to default.
> Let me know if that helps.
>
> Saurabh Bajaj
>
> On Tue, Mar 8, 2016 at 6:25 PM, Andy Davidson <a...@santacruzintegration.com>
> wrote:
>> Hi Ted
>>
>> I believe by default cassandra listens on 9042
>>
>> From: Ted Yu <yuzhih...@gmail.com>
>> Date: Tuesday, March 8, 2016 at 6:11 PM
>> To: Andrew Davidson <a...@santacruzintegration.com>
>> Cc: "user @spark" <user@spark.apache.org>
>> Subject: Re: pyspark spark-cassandra-connector java.io.IOException: Failed
>> to open native connection to Cassandra at {192.168.1.126}:9042
>>
>>> Have you contacted spark-cassandra-connector related mailing list ?
>>>
>>> I wonder where the port 9042 came from.
>>>
>>> Cheers
>>>
>>> On Tue, Mar 8, 2016 at 6:02 PM, Andy Davidson
>>> <a...@santacruzintegration.com> wrote:
>>>>
>>>> I am using spark-1.6.0-bin-hadoop2.6. I am trying to write a python
>>>> notebook that reads a data frame from Cassandra.
>>>>
>>>> I connect to cassadra using an ssh tunnel running on port 9043. CQLSH works
>>>> how ever I can not figure out how to configure my notebook. I have tried
>>>> various hacks any idea what I am doing wrong
>>>>
>>>> : java.io.IOException: Failed to open native connection to Cassandra at
>>>> {192.168.1.126}:9042
>>>>
>>>>
>>>>
>>>> Thanks in advance
>>>>
>>>> Andy
>>>>
>>>>
>>>>
>>>> $ extraPkgs="--packages com.databricks:spark-csv_2.11:1.3.0 \
>>>> --packages datastax:spark-cassandra-connector:1.6.0-M1-s_2.11"
>>>>
>>>> $ export PYSPARK_PYTHON=python3
>>>> $ export PYSPARK_DRIVER_PYTHON=python3
>>>> $ IPYTHON_OPTS=notebook $SPARK_ROOT/bin/pyspark $extraPkgs $*
>>>>
>>>>
>>>>
>>>> In [15]:
>>>> 1
>>>> sqlContext.setConf("spark.cassandra.connection.host”,”127.0.0.1:9043
>>>> <http://127.0.0.1:9043> ")
>>>> 2
>>>> df = sqlContext.read\
>>>> 3
>>>> .format("org.apache.spark.sql.cassandra")\
>>>> 4
>>>> .options(table=“time_series", keyspace="notification")\
>>>> 5
>>>> .load()
>>>> 6
>>>>
>>>> 7
>>>> df.printSchema()
>>>> 8
>>>> df.show()
>>>> ---------------------------------------------------------------------------
>>>> Py4JJavaError Traceback (most recent call last)
>>>> <ipython-input-15-9d8f6dcf210f> in <module>() 1
>>>> sqlContext.setConf("spark.cassandra.connection.host","localhost:9043")---->
>>>> 2 df = sqlContext.read .format("org.apache.spark.sql.cassandra")
>>>> .options(table="kv", keyspace="notification") .load() 3 4
>>>> df.printSchema() 5
>>>> df.show()/Users/andrewdavidson/workSpace/spark/spark-1.6.0-bin-hadoop2.6/py
>>>> thon/pyspark/sql/readwriter.py in load(self, path, format, schema,
>>>> **options) 137 return self._df(self._jreader.load(path))
>>>> 138 else:--> 139 return self._df(self._jreader.load())
>>>> 140 141
>>>> @since(1.4)/Users/andrewdavidson/workSpace/spark/spark-1.6.0-bin-hadoop2.6/
>>>> python/lib/py4j-0.9-src.zip/py4j/java_gateway.py in __call__(self, *args)
>>>> 811 answer = self.gateway_client.send_command(command) 812
>>>> return_value = get_return_value(
>>>> --> 813 answer, self.gateway_client, self.target_id, self.name
>>>> <http://self.name> )
>>>> 814 815 for temp_arg in
>>>> temp_args:/Users/andrewdavidson/workSpace/spark/spark-1.6.0-bin-hadoop2.6/p
>>>> ython/pyspark/sql/utils.py in deco(*a, **kw) 43 def deco(*a, **kw):
>>>> 44 try:---> 45 return f(*a, **kw) 46 except
>>>> py4j.protocol.Py4JJavaError as e: 47 s =
>>>> e.java_exception.toString()/Users/andrewdavidson/workSpace/spark/spark-1.6.
>>>> 0-bin-hadoop2.6/python/lib/py4j-0.9-src.zip/py4j/protocol.py in
>>>> get_return_value(answer, gateway_client, target_id, name) 306
>>>> raise Py4JJavaError(
>>>> 307 "An error occurred while calling
>>>> {0}{1}{2}.\n".--> 308 format(target_id, ".", name),
>>>> value)
>>>> 309 else: 310 raise Py4JError(
>>>>
>>>> Py4JJavaError: An error occurred while calling o280.load.
>>>> : java.io.IOException: Failed to open native connection to Cassandra at
>>>> {192.168.1.126}:9042
>>>> at
>>>> com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$con
>>>> nector$cql$CassandraConnector$$createSession(CassandraConnector.scala:162)
>>>> at
>>>> com.datastax.spark.connector.cql.CassandraConnector$$anonfun$2.apply(Cassan
>>>> draConnector.scala:148)
>>>> at
>>>> com.datastax.spark.connector.cql.CassandraConnector$$anonfun$2.apply(Cassan
>>>> draConnector.scala:148)
>>>> at
>>>> com.datastax.spark.connector.cql.RefCountedCache.createNewValueAndKeys(RefC
>>>> ountedCache.scala:31)
>>>> at
>>>> com.datastax.spark.connector.cql.RefCountedCache.acquire(RefCountedCache.sc
>>>> ala:56)
>>>> at
>>>> com.datastax.spark.connector.cql.CassandraConnector.openSession(CassandraCo
>>>> nnector.scala:81)
>>>> at
>>>> com.datastax.spark.connector.cql.CassandraConnector.withSessionDo(Cassandra
>>>> Connector.scala:109)
>>>> at
>>>> com.datastax.spark.connector.rdd.partitioner.CassandraRDDPartitioner$.getTo
>>>> kenFactory(CassandraRDDPartitioner.scala:184)
>>>> at
>>>> org.apache.spark.sql.cassandra.CassandraSourceRelation$.apply(CassandraSour
>>>> ceRelation.scala:267)
>>>> at
>>>> org.apache.spark.sql.cassandra.DefaultSource.createRelation(DefaultSource.s
>>>> cala:57)
>>>> at
>>>> org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(Resolv
>>>> edDataSource.scala:158)
>>>> at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:119)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:6>>>>
2)
>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImp
>>>> l.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:497)
>>>> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)
>>>> at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:381)
>>>> at py4j.Gateway.invoke(Gateway.java:259)
>>>> at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
>>>> at py4j.commands.CallCommand.execute(CallCommand.java:79)
>>>> at py4j.GatewayConnection.run(GatewayConnection.java:209)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException:
>>>> All host(s) tried for query failed (tried: /192.168.1.126:9042
>>>> <http://192.168.1.126:9042>
>>>> (com.datastax.driver.core.exceptions.TransportException: [/192.168.1.126
>>>> <http://192.168.1.126> ] Cannot connect))
>>>> at
>>>> com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnect
>>>> ion.java:231)
>>>> at
>>>>
com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:7>>>>
7)
>>>> at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1414)
>>>> at com.datastax.driver.core.Cluster.getMetadata(Cluster.java:393)
>>>> at
>>>> com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$con
>>>> nector$cql$CassandraConnector$$createSession(CassandraConnector.scala:155)
>>>> ... 22 more
>>>>
>>>
>
