Calcite example code
After seeing so many people ask for example code to do certain basic things in Calcite, I've been trying to find a good literate programming solution for Java as I like this approach for demoing. I recently came across the IJava (https://github.com/SpencerPark/IJava) kernel for Jupyter notebooks. This is basically just a proof of concept at this point, but here's a simple example https://github.com/michaelmior/calcite-notebooks/blob/master/Query%20parsing.ipynb I'm curious what others think of this approach. If others think it would be useful, I'd be happy to take suggestions on what should be included. Eventually, I'd like to get CI set up for this repository so I can re-run the notebooks at will. I would then aim to check this on every release so we can have a repository of code samples which we know run correctly. -- Michael Mior mm...@apache.org
Re: Calcite example code
Vladimir since on a different thread you asked about playground. As a side note, it might be interesting to have Calcite playground. > It could be like https://rextester.com/l/postgresql_online_compiler but > with Calcite inside. > Michael had setup one with jupyter and binder. Its not exactly the same thing, but definitely allows playing around with Calcite. https://github.com/michaelmior/calcite-notebooks Kevin Risden On Thu, Dec 20, 2018 at 12:48 PM Michael Mior wrote: > Here's another one focused on basic query optimization > > > https://github.com/michaelmior/calcite-notebooks/blob/master/query-optimization.ipynb > > -- > Michael Mior > mm...@apache.org > > > Le mer. 19 déc. 2018 à 18:21, Michael Mior a écrit : > > > Yes the notebook contains the output (it's really just a JSON file). It > > doesn't necessarily have to have output, but it's much more useful if it > > does since it means whoever is viewing doesn't need to execute it. It > would > > certainly be possible to use this for tests although it would require an > > installation of Python. Given how ubiquitous Python is, I don't think > this > > is a huge concern, although we'd need a way of installing a couple Python > > dependencies. > > > > As Kevin mentioned, you can execute these online with Binder. I just had > > to add a Dockerfile so it could run the IJava kernel since it not the > > default. Check out the link below and you can see what the experience is > > like. > > > > https://mybinder.org/v2/gh/michaelmior/calcite-notebooks/master > > > > Just select a notebook once it loads (it may take a couple minutes). The > > experience is basically the same as what you get when running locally. > The > > notebook consists of a series of "cells" which you can run individually > and > > edit as you wish. This would also make it easy for people to play around > > with Calcite a little without having to install anything. > > > > -- > > Michael Mior > > mm...@apache.org > > > > > > Le mer. 19 déc. 2018 à 17:22, Julian Hyde a écrit : > > > >> For old idiots like me, can you explain how the notebook works? The file > >> you checked into GitHub, does it contain the input and output or just > the > >> input? Is there a way to edit or use the notebook interactively? > >> > >> It certainly seems a better way to introduce people to examples than > >> saying “go look at this test”. > >> > >> I think quite a few of our tests could be converted into this format. > >> > >> Julian > >> > >> > >> > On Dec 19, 2018, at 10:52 AM, Michael Mior wrote: > >> > > >> > After seeing so many people ask for example code to do certain basic > >> things > >> > in Calcite, I've been trying to find a good literate programming > >> solution > >> > for Java as I like this approach for demoing. I recently came across > the > >> > IJava (https://github.com/SpencerPark/IJava) kernel for Jupyter > >> notebooks. > >> > > >> > This is basically just a proof of concept at this point, but here's a > >> > simple example > >> > > >> > > >> > https://github.com/michaelmior/calcite-notebooks/blob/master/Query%20parsing.ipynb > >> > > >> > I'm curious what others think of this approach. If others think it > >> would be > >> > useful, I'd be happy to take suggestions on what should be included. > >> > Eventually, I'd like to get CI set up for this repository so I can > >> re-run > >> > the notebooks at will. I would then aim to check this on every release > >> so > >> > we can have a repository of code samples which we know run correctly. > >> > > >> > -- > >> > Michael Mior > >> > mm...@apache.org > >> > >> >
Re: Calcite example code
Michael, I would definitely encourage you and anyone else to participate. That is an awesome idea, especially sometimes there are multiple ways to do the same thing. For example, instantiating a RelNode converter can be done by the planner or by constructing everything by hand. Similarly adding schemas can be done programmatically or by model files. Pointing out the situations of when to use each solution would be a great addition to this. Thanks, Walaa. On Wed, Dec 19, 2018 at 10:53 AM Michael Mior wrote: > After seeing so many people ask for example code to do certain basic things > in Calcite, I've been trying to find a good literate programming solution > for Java as I like this approach for demoing. I recently came across the > IJava (https://github.com/SpencerPark/IJava) kernel for Jupyter notebooks. > > This is basically just a proof of concept at this point, but here's a > simple example > > > https://github.com/michaelmior/calcite-notebooks/blob/master/Query%20parsing.ipynb > > I'm curious what others think of this approach. If others think it would be > useful, I'd be happy to take suggestions on what should be included. > Eventually, I'd like to get CI set up for this repository so I can re-run > the notebooks at will. I would then aim to check this on every release so > we can have a repository of code samples which we know run correctly. > > -- > Michael Mior > mm...@apache.org >
Re: Calcite example code
For old idiots like me, can you explain how the notebook works? The file you checked into GitHub, does it contain the input and output or just the input? Is there a way to edit or use the notebook interactively? It certainly seems a better way to introduce people to examples than saying “go look at this test”. I think quite a few of our tests could be converted into this format. Julian > On Dec 19, 2018, at 10:52 AM, Michael Mior wrote: > > After seeing so many people ask for example code to do certain basic things > in Calcite, I've been trying to find a good literate programming solution > for Java as I like this approach for demoing. I recently came across the > IJava (https://github.com/SpencerPark/IJava) kernel for Jupyter notebooks. > > This is basically just a proof of concept at this point, but here's a > simple example > > https://github.com/michaelmior/calcite-notebooks/blob/master/Query%20parsing.ipynb > > I'm curious what others think of this approach. If others think it would be > useful, I'd be happy to take suggestions on what should be included. > Eventually, I'd like to get CI set up for this repository so I can re-run > the notebooks at will. I would then aim to check this on every release so > we can have a repository of code samples which we know run correctly. > > -- > Michael Mior > mm...@apache.org
Re: Calcite example code
+1 for jupyter notebooks @Julian: I have not used Jupyter notebooks myself, but read about them a few months ago. My understanding is that you can open the ipynb files using the opensource Jupyter web app and change the code interactively using code editors or widgets such as sliders etc. The changes would then be reflected back in the interface in real-time. I suppose it would be similar to documented examples in Go: https://godoc.org/fmt#Stringer (click example to expand it, then click play to open the code in the playground), but with much more interactivity and a better user experience On 20/12/2018 9:22 am, Julian Hyde wrote: For old idiots like me, can you explain how the notebook works? The file you checked into GitHub, does it contain the input and output or just the input? Is there a way to edit or use the notebook interactively? It certainly seems a better way to introduce people to examples than saying “go look at this test”. I think quite a few of our tests could be converted into this format. Julian On Dec 19, 2018, at 10:52 AM, Michael Mior wrote: After seeing so many people ask for example code to do certain basic things in Calcite, I've been trying to find a good literate programming solution for Java as I like this approach for demoing. I recently came across the IJava (https://github.com/SpencerPark/IJava) kernel for Jupyter notebooks. This is basically just a proof of concept at this point, but here's a simple example https://github.com/michaelmior/calcite-notebooks/blob/master/Query%20parsing.ipynb I'm curious what others think of this approach. If others think it would be useful, I'd be happy to take suggestions on what should be included. Eventually, I'd like to get CI set up for this repository so I can re-run the notebooks at will. I would then aim to check this on every release so we can have a repository of code samples which we know run correctly. -- Michael Mior mm...@apache.org
Re: Calcite example code
I like the idea a lot. Notebooks are interactive ways to work with code and results. The notebook typically contains both input and output although you can save it without the output. The notebook can be used interactively with Jupyter or one of the options below. I was going to share this as well since it helps with notebooks on Github: Static viewing * https://nbviewer.jupyter.org/ * https://nbviewer.jupyter.org/github/michaelmior/calcite-notebooks/blob/master/Query%20parsing.ipynb# Dynamic notebook * https://mybinder.org/ * https://mybinder.org/v2/gh/michaelmior/calcite-notebooks/master The dynamic notebook could use a little work to get the Java kernel to work correctly but it is definitely possible. Kevin Risden On Wed, Dec 19, 2018 at 5:22 PM Julian Hyde wrote: > > For old idiots like me, can you explain how the notebook works? The file you > checked into GitHub, does it contain the input and output or just the input? > Is there a way to edit or use the notebook interactively? > > It certainly seems a better way to introduce people to examples than saying > “go look at this test”. > > I think quite a few of our tests could be converted into this format. > > Julian > > > > On Dec 19, 2018, at 10:52 AM, Michael Mior wrote: > > > > After seeing so many people ask for example code to do certain basic things > > in Calcite, I've been trying to find a good literate programming solution > > for Java as I like this approach for demoing. I recently came across the > > IJava (https://github.com/SpencerPark/IJava) kernel for Jupyter notebooks. > > > > This is basically just a proof of concept at this point, but here's a > > simple example > > > > https://github.com/michaelmior/calcite-notebooks/blob/master/Query%20parsing.ipynb > > > > I'm curious what others think of this approach. If others think it would be > > useful, I'd be happy to take suggestions on what should be included. > > Eventually, I'd like to get CI set up for this repository so I can re-run > > the notebooks at will. I would then aim to check this on every release so > > we can have a repository of code samples which we know run correctly. > > > > -- > > Michael Mior > > mm...@apache.org >
Re: Calcite example code
Yes the notebook contains the output (it's really just a JSON file). It doesn't necessarily have to have output, but it's much more useful if it does since it means whoever is viewing doesn't need to execute it. It would certainly be possible to use this for tests although it would require an installation of Python. Given how ubiquitous Python is, I don't think this is a huge concern, although we'd need a way of installing a couple Python dependencies. As Kevin mentioned, you can execute these online with Binder. I just had to add a Dockerfile so it could run the IJava kernel since it not the default. Check out the link below and you can see what the experience is like. https://mybinder.org/v2/gh/michaelmior/calcite-notebooks/master Just select a notebook once it loads (it may take a couple minutes). The experience is basically the same as what you get when running locally. The notebook consists of a series of "cells" which you can run individually and edit as you wish. This would also make it easy for people to play around with Calcite a little without having to install anything. -- Michael Mior mm...@apache.org Le mer. 19 déc. 2018 à 17:22, Julian Hyde a écrit : > For old idiots like me, can you explain how the notebook works? The file > you checked into GitHub, does it contain the input and output or just the > input? Is there a way to edit or use the notebook interactively? > > It certainly seems a better way to introduce people to examples than > saying “go look at this test”. > > I think quite a few of our tests could be converted into this format. > > Julian > > > > On Dec 19, 2018, at 10:52 AM, Michael Mior wrote: > > > > After seeing so many people ask for example code to do certain basic > things > > in Calcite, I've been trying to find a good literate programming solution > > for Java as I like this approach for demoing. I recently came across the > > IJava (https://github.com/SpencerPark/IJava) kernel for Jupyter > notebooks. > > > > This is basically just a proof of concept at this point, but here's a > > simple example > > > > > https://github.com/michaelmior/calcite-notebooks/blob/master/Query%20parsing.ipynb > > > > I'm curious what others think of this approach. If others think it would > be > > useful, I'd be happy to take suggestions on what should be included. > > Eventually, I'd like to get CI set up for this repository so I can re-run > > the notebooks at will. I would then aim to check this on every release so > > we can have a repository of code samples which we know run correctly. > > > > -- > > Michael Mior > > mm...@apache.org > >
Re: Calcite example code
Here's another one focused on basic query optimization https://github.com/michaelmior/calcite-notebooks/blob/master/query-optimization.ipynb -- Michael Mior mm...@apache.org Le mer. 19 déc. 2018 à 18:21, Michael Mior a écrit : > Yes the notebook contains the output (it's really just a JSON file). It > doesn't necessarily have to have output, but it's much more useful if it > does since it means whoever is viewing doesn't need to execute it. It would > certainly be possible to use this for tests although it would require an > installation of Python. Given how ubiquitous Python is, I don't think this > is a huge concern, although we'd need a way of installing a couple Python > dependencies. > > As Kevin mentioned, you can execute these online with Binder. I just had > to add a Dockerfile so it could run the IJava kernel since it not the > default. Check out the link below and you can see what the experience is > like. > > https://mybinder.org/v2/gh/michaelmior/calcite-notebooks/master > > Just select a notebook once it loads (it may take a couple minutes). The > experience is basically the same as what you get when running locally. The > notebook consists of a series of "cells" which you can run individually and > edit as you wish. This would also make it easy for people to play around > with Calcite a little without having to install anything. > > -- > Michael Mior > mm...@apache.org > > > Le mer. 19 déc. 2018 à 17:22, Julian Hyde a écrit : > >> For old idiots like me, can you explain how the notebook works? The file >> you checked into GitHub, does it contain the input and output or just the >> input? Is there a way to edit or use the notebook interactively? >> >> It certainly seems a better way to introduce people to examples than >> saying “go look at this test”. >> >> I think quite a few of our tests could be converted into this format. >> >> Julian >> >> >> > On Dec 19, 2018, at 10:52 AM, Michael Mior wrote: >> > >> > After seeing so many people ask for example code to do certain basic >> things >> > in Calcite, I've been trying to find a good literate programming >> solution >> > for Java as I like this approach for demoing. I recently came across the >> > IJava (https://github.com/SpencerPark/IJava) kernel for Jupyter >> notebooks. >> > >> > This is basically just a proof of concept at this point, but here's a >> > simple example >> > >> > >> https://github.com/michaelmior/calcite-notebooks/blob/master/Query%20parsing.ipynb >> > >> > I'm curious what others think of this approach. If others think it >> would be >> > useful, I'd be happy to take suggestions on what should be included. >> > Eventually, I'd like to get CI set up for this repository so I can >> re-run >> > the notebooks at will. I would then aim to check this on every release >> so >> > we can have a repository of code samples which we know run correctly. >> > >> > -- >> > Michael Mior >> > mm...@apache.org >> >>