Re: PostgreSQL JDBC Connections

2017-01-05 Thread t p 
YMMV and I don’t think my approach will work for your use case.

Here is a suggestion based on what I’ve done. In the first paragraph you can 
register tables with code as such.

%spark
val example = sqlContext.read.format("jdbc").options(
Map("url" -> "jdbc:postgresql://localhost:5432/db_name",
"driver" -> "org.postgresql.Driver",
"user" -> “user_name", 
"password" -> “password",
"dbtable" -> “example" )).load()

assets.createOrReplaceTempView(“example")


Subsequently using spark.sql (limited to SQL syntax/semantics) supported by 
Spark. If there are Postgres specific features or some advanced SQL 
functions/syntax, this approach may not work for you.

%spark.sql
select * from example

The rest of the reports/visuals exports to CSV will look identical.



> On Jan 5, 2017, at 5:30 PM, Benjamin Kim  wrote:
> 
> We are using the JDBC interpreter. The business analysts only know SQL and 
> run ad-hoc queries for their report exports to CSV.
> 
> Cheers,
> Ben
> 
> 
>> On Jan 5, 2017, at 2:21 PM, t p  wrote:
>> 
>> Are you using JDBC or the PSQL interpreter? I had encountered something 
>> similar while using the PSQL interpreter and I had to restart Zeppelin. 
>> 
>> My experience using PSQL (Postgresql, HAWK) was not as good as using 
>> spark/scala wrappers (JDBC data source) to connect via JDBC and then 
>> register temp tables. This approach allowed me to work with dynamic forms in 
>> a more meaningful way e.g. use SQL results to create a new drop down to 
>> drive the next page etc…
>> 
>> 
>> 
>>> On Jan 5, 2017, at 12:57 PM, Benjamin Kim  wrote:
>>> 
>>> We are getting “out of shared memory” errors when multiple users are 
>>> running SQL queries against our PostgreSQL DB either simultaneously or 
>>> throughout the day. When this happens, Zeppelin 0.6.0 becomes unresponsive 
>>> for any more SQL queries. It looks like this is being caused by too many 
>>> locks being taken and not released, transactions never closing, and/or 
>>> connections never closing. Has anyone encountered Zeppelin 0.6.0 such an 
>>> issue as this? If so, is there a solution for it?
>>> 
>>> Thanks,
>>> Ben
>> 
>

Re: PostgreSQL JDBC Connections

2017-01-05 Thread Benjamin Kim 
We are using the JDBC interpreter. The business analysts only know SQL and run 
ad-hoc queries for their report exports to CSV.

Cheers,
Ben


> On Jan 5, 2017, at 2:21 PM, t p  wrote:
> 
> Are you using JDBC or the PSQL interpreter? I had encountered something 
> similar while using the PSQL interpreter and I had to restart Zeppelin. 
> 
> My experience using PSQL (Postgresql, HAWK) was not as good as using 
> spark/scala wrappers (JDBC data source) to connect via JDBC and then register 
> temp tables. This approach allowed me to work with dynamic forms in a more 
> meaningful way e.g. use SQL results to create a new drop down to drive the 
> next page etc…
> 
> 
> 
>> On Jan 5, 2017, at 12:57 PM, Benjamin Kim  wrote:
>> 
>> We are getting “out of shared memory” errors when multiple users are running 
>> SQL queries against our PostgreSQL DB either simultaneously or throughout 
>> the day. When this happens, Zeppelin 0.6.0 becomes unresponsive for any more 
>> SQL queries. It looks like this is being caused by too many locks being 
>> taken and not released, transactions never closing, and/or connections never 
>> closing. Has anyone encountered Zeppelin 0.6.0 such an issue as this? If so, 
>> is there a solution for it?
>> 
>> Thanks,
>> Ben
>

Re: PostgreSQL JDBC Connections

2017-01-05 Thread t p 
Are you using JDBC or the PSQL interpreter? I had encountered something similar 
while using the PSQL interpreter and I had to restart Zeppelin. 

My experience using PSQL (Postgresql, HAWK) was not as good as using 
spark/scala wrappers (JDBC data source) to connect via JDBC and then register 
temp tables. This approach allowed me to work with dynamic forms in a more 
meaningful way e.g. use SQL results to create a new drop down to drive the next 
page etc…



> On Jan 5, 2017, at 12:57 PM, Benjamin Kim  wrote:
> 
> We are getting “out of shared memory” errors when multiple users are running 
> SQL queries against our PostgreSQL DB either simultaneously or throughout the 
> day. When this happens, Zeppelin 0.6.0 becomes unresponsive for any more SQL 
> queries. It looks like this is being caused by too many locks being taken and 
> not released, transactions never closing, and/or connections never closing. 
> Has anyone encountered Zeppelin 0.6.0 such an issue as this? If so, is there 
> a solution for it?
> 
> Thanks,
> Ben

pyspark can't run through

2017-01-05 Thread Alec Lee 
Hello, all

I recently come cross good tool - zeppelin, it is easy to use. But I have some 
troubles to make pyspark work in my server. The code below used to work fine, 
but for no reason it pop up errors like permission deny.  

Code
%pyspark
import pandas
## http://blog.csdn.net/lsshlsw/article/details/53768756
## possible error, fix as above
PlateText = sc.textFile("hdfs://node9:54310/user/hadoop/tmp/plate_A216E6.csv")
header = PlateText.first()
#print header
#print PlateText.count()


Error msg ***
Traceback (most recent call last):
  File "/tmp/zeppelin_pyspark-7122867288384299951.py", line 267, in 
raise Exception(traceback.format_exc())
Exception: Traceback (most recent call last):
  File "/tmp/zeppelin_pyspark-7122867288384299951.py", line 265, in 
exec(code)
  File "", line 3, in 
  File "/opt/spark-2.0.2-bin-hadoop2.7/python/pyspark/rdd.py", line 1328, in 
first
rs = self.take(1)
  File "/opt/spark-2.0.2-bin-hadoop2.7/python/pyspark/rdd.py", line 1310, in 
take
res = self.context.runJob(self, takeUpToNumLeft, p)
  File "/opt/spark-2.0.2-bin-hadoop2.7/python/pyspark/context.py", line 933, in 
runJob
port = self._jvm.PythonRDD.runJob(self._jsc.sc(), mappedRDD._jrdd, 
partitions)
  File 
"/opt/spark-2.0.2-bin-hadoop2.7/python/lib/py4j-0.10.3-src.zip/py4j/java_gateway.py",
 line 1133, in __call__
answer, self.gateway_client, self.target_id, self.name)
  File "/opt/spark-2.0.2-bin-hadoop2.7/python/pyspark/sql/utils.py", line 63, 
in deco
return f(*a, **kw)
  File 
"/opt/spark-2.0.2-bin-hadoop2.7/python/lib/py4j-0.10.3-src.zip/py4j/protocol.py",
 line 319, in get_return_value
format(target_id, ".", name), value)
Py4JJavaError: An error occurred while calling 
z:org.apache.spark.api.python.PythonRDD.runJob.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in 
stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 
3, 192.168.168.29): java.io.IOException: Cannot run program 
"/opt/spark-2.0.2-bin-hadoop2.7/python": error=13, Permission denied
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
at 
org.apache.spark.api.python.PythonWorkerFactory.startDaemon(PythonWorkerFactory.scala:163)
at 
org.apache.spark.api.python.PythonWorkerFactory.createThroughDaemon(PythonWorkerFactory.scala:89)
at 
org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:65)
at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:114)
at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:128)
at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:63)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
at org.apache.spark.scheduler.Task.run(Task.scala:86)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: error=13, Permission denied
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.(UNIXProcess.java:248)
at java.lang.ProcessImpl.start(ProcessImpl.java:134)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
... 14 more
Driver stacktrace:
at 
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1454)
at 
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1442)
at 
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1441)
at 
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at 
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1441)
at 
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811)
at 
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811)
at scala.Option.foreach(Option.scala:257)
at 
org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:811)
at 
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1667)
at 
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1622)
at 
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1611)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)