Re: Drill Down

[t p]

Happy to share some thoughts here. I did not find an easy way to do this for 
postgres/psql or other interpreters. Here is what I did using the spark 
interpreter.

I am not doing drill down via html links, but via select/drop drown list. 
Hopefully you can do something for links. 

Here is the angular API for zeppelin:
https://zeppelin.apache.org/docs/0.6.2/displaysystem/back-end-angular.html 

https://zeppelin.apache.org/docs/0.6.2/displaysystem/front-end-angular.html 


In my use case, I have a book with 6 paragraphs, which use angular variables. 
For example in the first paragraph, I have the following:

// Create a select/drop down list of hosts
var hostList: List[(String, String)] = Nil
…..
val selectedHost =  z.select("Select a host for analysis", hostList)

// Bind the selected hosts to an angular variable
z.angularBind("selectedHost", selectedHost)

In the 6th/last one, I have this code which runs the angular watch API, to 
detect and refresh the 4 key paragraphs when the variable changes.

%spark
val selectedOptionContext = z.getInterpreterContext()
def myWatcher0(before:Object, after:Object) {
z.run(1, selectedOptionContext)
z.run(2, selectedOptionContext)
z.run(3, selectedOptionContext)
z.run(4, selectedOptionContext)
}
z.angularWatch("selectedHost", myWatcher0 _)

Hope this helps. There might be better ways of doing this and I’d like hear 
thoughts from others.


> On Mar 31, 2017, at 6:11 PM, murexconsult...@googlemail.com wrote:
> 
> I am trying to create a drill down in Zeppelin on some data. Taking the bank 
> data in the example I have this paragraph:
> 
> %sql
> select concat('%html ',job,''), 
> sum(balance),avg(balance) from bank
> group by job order by job
> 
> Which displays the data I want. But was trying to get the next paragraph to 
> update when I click on each of the links - say retired:
> 
> %sql
> select marital, sum(balance) from bank where job='${type=sevices}'
> group by marital
> 
> 
> i.e. I wanted the variable type be updated and then the paragraph run when 
> someone clicks on each link in the previous paragraph.
> 
> I can see if I use %angular I should be able to do this. But that does not 
> work in the cell of a table. And cannot get the  tag to work - tried a few 
> on-click etc...
> 
> Is this possible?  Guess other approach would be to not return a table but an 
> %angular and do all the formatting with  in that?
> 
> Any tips or clues would be helpful.
> 
> Thanks
> 

Re: Passing variables from %pyspark to %sh

[t p]

Is something like feasible from the front end perspective - i.e the web UI 
(Angular?) - i.e. not matter which process/JVM runs the interpreter, I’d assume 
that a book is executed in the context of a we browser which unifies all the 
pages of the book...

> On Jan 12, 2017, at 9:56 AM, Jeff Zhang <zjf...@gmail.com> wrote:
> 
> 
> Agree to share variables between interpreters. Currently zeppelin launch one 
> JVM for each interpreter group. So it is not possible to share variables 
> between spark and sh. But for some interpreters like sh, md, it is not 
> necessary to create separate JVM for them. We can embed them in spark 
> interpreter JVM.  But we could not do it for all interpreters, because it 
> would cause potential jar conflicts.
> 
>  
> 
> Jongyoul Lee <jongy...@gmail.com <mailto:jongy...@gmail.com>>于2017年1月12日周四 
> 下午10:18写道：
> Yes, many users suggest that feature to share results between paragraphs and 
> different interpreters. I think this would be one of major features in a next 
> release.
> 
> On Thu, Jan 12, 2017 at 10:30 PM, t p <tauis2...@gmail.com 
> <mailto:tauis2...@gmail.com>> wrote:
> Is it possible to have similar support to exchange  checkbox/dropdown 
> variables and can variables be exchanged with other interpreters like PSQL 
> (e.g. variable set by spark/pyspark and accessible in another para which is 
> running PSQL interpreter).
> 
> I’m interested in doing this and I’d like to know if there is a way to 
> accomplish this:
> https://lists.apache.org/thread.html/a1b3530e5a20f983acd70f8fca029f90b6bfe8d0d999597342447e6f@%3Cusers.zeppelin.apache.org%3E
>  
> <https://lists.apache.org/thread.html/a1b3530e5a20f983acd70f8fca029f90b6bfe8d0d999597342447e6f@%3Cusers.zeppelin.apache.org%3E>
> 
> 
>> On Jan 12, 2017, at 2:16 AM, Jongyoul Lee <jongy...@gmail.com 
>> <mailto:jongy...@gmail.com>> wrote:
>> 
>> There's no way to communicate between spark and sh intepreter. It need to 
>> implement it but it doesn't yet. But I agree that it would be helpful for 
>> some cases. Can you create issue?
>> 
>> On Thu, Jan 12, 2017 at 3:32 PM, Ruslan Dautkhanov <dautkha...@gmail.com 
>> <mailto:dautkha...@gmail.com>> wrote:
>> It's possible to exchange variables between Scala and Spark
>> through z.put and z.get.
>> 
>> How to pass a variable to %sh?
>> 
>> In Jupyter it's possible to do for example as
>>   ! hadoop fs -put {localfile} {hdfsfile}
>> 
>> where localfile and and hdfsfile are Python variables.
>> 
>> Can't find any references for something similar in Shell Interpreter
>> https://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/interpreter/shell.html 
>> <https://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/interpreter/shell.html>
>> 
>> In many notebooks we have to pass small variabels 
>> from Zeppelin notes to external scripts as parameters.
>> 
>> It would be awesome to have something like
>> 
>> %sh
>> /path/to/script --param8={var1} --param9={var2}
>> 
>> where var1 and var2 would be implied to be fetched as z.get('var1') 
>> and z.get('var2') respectively.
>> 
>> Other thoughts?
>> 
>> 
>> Thank you,
>> Ruslan Dautkhanov
>> 
>> 
>> 
>> 
>> -- 
>> 이종열, Jongyoul Lee, 李宗烈
>> http://madeng.net <http://madeng.net/>
> 
> 
> 
> 
> -- 
> 이종열, Jongyoul Lee, 李宗烈
> http://madeng.net <http://madeng.net/>

Re: Passing variables from %pyspark to %sh

[t p]

Is it possible to have similar support to exchange  checkbox/dropdown variables 
and can variables be exchanged with other interpreters like PSQL (e.g. variable 
set by spark/pyspark and accessible in another para which is running PSQL 
interpreter).

I’m interested in doing this and I’d like to know if there is a way to 
accomplish this:
https://lists.apache.org/thread.html/a1b3530e5a20f983acd70f8fca029f90b6bfe8d0d999597342447e6f@%3Cusers.zeppelin.apache.org%3E
 



> On Jan 12, 2017, at 2:16 AM, Jongyoul Lee  wrote:
> 
> There's no way to communicate between spark and sh intepreter. It need to 
> implement it but it doesn't yet. But I agree that it would be helpful for 
> some cases. Can you create issue?
> 
> On Thu, Jan 12, 2017 at 3:32 PM, Ruslan Dautkhanov  > wrote:
> It's possible to exchange variables between Scala and Spark
> through z.put and z.get.
> 
> How to pass a variable to %sh?
> 
> In Jupyter it's possible to do for example as
>   ! hadoop fs -put {localfile} {hdfsfile}
> 
> where localfile and and hdfsfile are Python variables.
> 
> Can't find any references for something similar in Shell Interpreter
> https://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/interpreter/shell.html 
> 
> 
> In many notebooks we have to pass small variabels 
> from Zeppelin notes to external scripts as parameters.
> 
> It would be awesome to have something like
> 
> %sh
> /path/to/script --param8={var1} --param9={var2}
> 
> where var1 and var2 would be implied to be fetched as z.get('var1') 
> and z.get('var2') respectively.
> 
> Other thoughts?
> 
> 
> Thank you,
> Ruslan Dautkhanov
> 
> 
> 
> 
> -- 
> 이종열, Jongyoul Lee, 李宗烈
> http://madeng.net 

Re: PostgreSQL JDBC Connections

[t p]

YMMV and I don’t think my approach will work for your use case.

Here is a suggestion based on what I’ve done. In the first paragraph you can 
register tables with code as such.

%spark
val example = sqlContext.read.format("jdbc").options(
Map("url" -> "jdbc:postgresql://localhost:5432/db_name",
"driver" -> "org.postgresql.Driver",
"user" -> “user_name", 
"password" -> “password",
"dbtable" -> “example" )).load()

assets.createOrReplaceTempView(“example")


Subsequently using spark.sql (limited to SQL syntax/semantics) supported by 
Spark. If there are Postgres specific features or some advanced SQL 
functions/syntax, this approach may not work for you.

%spark.sql
select * from example

The rest of the reports/visuals exports to CSV will look identical.



> On Jan 5, 2017, at 5:30 PM, Benjamin Kim <bbuil...@gmail.com> wrote:
> 
> We are using the JDBC interpreter. The business analysts only know SQL and 
> run ad-hoc queries for their report exports to CSV.
> 
> Cheers,
> Ben
> 
> 
>> On Jan 5, 2017, at 2:21 PM, t p <tauis2...@gmail.com> wrote:
>> 
>> Are you using JDBC or the PSQL interpreter? I had encountered something 
>> similar while using the PSQL interpreter and I had to restart Zeppelin. 
>> 
>> My experience using PSQL (Postgresql, HAWK) was not as good as using 
>> spark/scala wrappers (JDBC data source) to connect via JDBC and then 
>> register temp tables. This approach allowed me to work with dynamic forms in 
>> a more meaningful way e.g. use SQL results to create a new drop down to 
>> drive the next page etc…
>> 
>> 
>> 
>>> On Jan 5, 2017, at 12:57 PM, Benjamin Kim <bbuil...@gmail.com> wrote:
>>> 
>>> We are getting “out of shared memory” errors when multiple users are 
>>> running SQL queries against our PostgreSQL DB either simultaneously or 
>>> throughout the day. When this happens, Zeppelin 0.6.0 becomes unresponsive 
>>> for any more SQL queries. It looks like this is being caused by too many 
>>> locks being taken and not released, transactions never closing, and/or 
>>> connections never closing. Has anyone encountered Zeppelin 0.6.0 such an 
>>> issue as this? If so, is there a solution for it?
>>> 
>>> Thanks,
>>> Ben
>> 
> 

Re: PostgreSQL JDBC Connections

[t p]

Are you using JDBC or the PSQL interpreter? I had encountered something similar 
while using the PSQL interpreter and I had to restart Zeppelin. 

My experience using PSQL (Postgresql, HAWK) was not as good as using 
spark/scala wrappers (JDBC data source) to connect via JDBC and then register 
temp tables. This approach allowed me to work with dynamic forms in a more 
meaningful way e.g. use SQL results to create a new drop down to drive the next 
page etc…



> On Jan 5, 2017, at 12:57 PM, Benjamin Kim  wrote:
> 
> We are getting “out of shared memory” errors when multiple users are running 
> SQL queries against our PostgreSQL DB either simultaneously or throughout the 
> day. When this happens, Zeppelin 0.6.0 becomes unresponsive for any more SQL 
> queries. It looks like this is being caused by too many locks being taken and 
> not released, transactions never closing, and/or connections never closing. 
> Has anyone encountered Zeppelin 0.6.0 such an issue as this? If so, is there 
> a solution for it?
> 
> Thanks,
> Ben

line graph after psql query

[t p]

I’m using the plotting features of the table display system. I’m using the 
postgres/psql interpreter. I noticed a few issues and I’m hoping that someone 
can point me in the right direction to resolve these issues:

1) I’m unable to display a value on y-axis (number field) as is (it forces me 
to use a value of min, max, avg, sum, count ) in a plot while using the 
settings drop down
2) I’d like to to use the group by feature and plot a line for each member of 
group for time series data, however I end up getting cluster grouped visuals 
(x-axis time and I get clusters on x-axis for each group).

The only way I can get a graph to work is by using the scatter plot, which 
allows me to plot the actual value, however that is not a connected/smooth 
line. 

Currently using 0.6.1 version.