Re: Hbase configuration storage without data

[moon soo Lee]

Regarding data in the note.json,

In case of user doesn't want include data in exported note.json, user can
clean the outputs before export, for now.

We might think displaying two export options with / without data when click
export button, if exporting notebook without data is important and need to
user aware everytime they export the notebook.

But please consider many different possible use cases. Some people might
have important information inside of the code (like credentials) but result
of query can be made public, some people might want to restrict access to
the raw data but want to share query result to other people.

Best,
moon

On Tue, Sep 13, 2016 at 10:20 PM Vikash Kumar <vikash.ku...@resilinc.com>
wrote:

> Hi,
>
> But storing the data in a separate file approach will need to maintain the
> link between both files. And also this approach is not preferable when the
> data is obtained on access basis. like in my case data which comes from
> hbase through phoenix is tenant base. So storing that data into note.json
> or in different file is breaking the point of multi tenancy.
>
> So as an approach can we store only configuration and retrieve the data
> when we are loading the note by running the all paragraph for first time
> load.
>
>
>
> But at the same time, i think having data in the note.json helps make
> import/export simple and make notebook render able without run it.
>
>
>
> So for import/export providing the data is it good? Data is always
> confidential and cannot be shared with anyone in form of json. So in this
> approach any one can open the note.json and can access the data.
>
> Thanks & Regards,
>
> *Vikash Kumar*
>
> *From:* Felix Cheung [mailto:felixcheun...@hotmail.com]
> *Sent:* Wednesday, September 14, 2016 6:24 AM
> *To:* users@zeppelin.apache.org; users@zeppelin.apache.org
>
>
> *Subject:* Re: Hbase configuration storage without data
>
>
>
> I like that approach - though you should be able to clear result output
> before exporting the note, if all you want is the config? The should remove
> all output data, keeping it smaller?
>
>
>
>
>
> _____________
> From: Mohit Jaggi <mohitja...@gmail.com>
> Sent: Monday, September 12, 2016 10:38 AM
> Subject: Re: Hbase configuration storage without data
> To: <users@zeppelin.apache.org>
>
>
>
> one option is to keep the data in separate files. notes.json can contain
> the code and the data can be a pointer to /path/to/file. import/export can
> choose to include or exclude the data. when it is included the data files
> are added to a tgz file containing notes.json otherwise you just export
> notes.json
>
>
>
>
>
>
>
> On Mon, Sep 12, 2016 at 10:33 AM, moon soo Lee <m...@apache.org> wrote:
>
> Right big note.json file is a problem.
>
> But at the same time, i think having data in the note.json helps make
> import/export simple and make notebook renderable without run it.
>
>
>
> So far, i didn't see much discussion about this subject on mailing list or
> on the issue tracker.
>
>
>
> If there's an good idea that can handle large data while keeping
> import/export simple and ability to render without run, that would be a
> great starting point of the discussions.
>
>
>
> Thanks,
>
> moon
>
>
>
> On Wed, Sep 7, 2016 at 9:40 PM Vikash Kumar <vikash.ku...@resilinc.com>
> wrote:
>
> Hi moon,
>
> Yes that was the way that I was using. But is there any plan for future
> releases to removing the data from note and storing only configuration?
>
> Because storing the configuration with data when there is no max result
> limit will create a big note.json file.
>
>
>
> Thanks & Regards,
>
> *Vikash Kumar*
>
> *From:* moon soo Lee [mailto:m...@apache.org]
> *Sent:* Wednesday, September 7, 2016 8:39 PM
> *To:* users@zeppelin.apache.org
> *Subject:* Re: Hbase configuration storage without data
>
>
>
> Hi,
>
>
>
> For now, code and result data are mixed in note.json, which is represented
> by 'class Note' [1]. And every Notebook storage layer need to implement
> 'NotebookRepo.get()' [2] to read note.json from underlying storage and
> convert it into 'class Note'.
>
>
>
> As you see the related API and class definition, NotebookRepo actually
> doesn't have any restriction how 'class Note' is serialized and saved in
> the storage.
>
>
>
> So you can event new format, you can exclude result data from saving, and
> so on.
>
>
>
> Hop this helps.
>
>
>
> Thanks,
>
> moon
>
>
>
> [1]
> https://github.com/apache/zeppelin/blob/master/zeppelin-zengine/src/main/java/org/

RE: Hbase configuration storage without data

[Vikash Kumar]

Hi,
But storing the data in a separate file approach will need to maintain the link 
between both files. And also this approach is not preferable when the data is 
obtained on access basis. like in my case data which comes from hbase through 
phoenix is tenant base. So storing that data into note.json or in different 
file is breaking the point of multi tenancy.
So as an approach can we store only configuration and retrieve the data when we 
are loading the note by running the all paragraph for first time load.

But at the same time, i think having data in the note.json helps make 
import/export simple and make notebook render able without run it.

So for import/export providing the data is it good? Data is always confidential 
and cannot be shared with anyone in form of json. So in this approach any one 
can open the note.json and can access the data.
Thanks & Regards,
Vikash Kumar
From: Felix Cheung [mailto:felixcheun...@hotmail.com]
Sent: Wednesday, September 14, 2016 6:24 AM
To: users@zeppelin.apache.org; users@zeppelin.apache.org
Subject: Re: Hbase configuration storage without data

I like that approach - though you should be able to clear result output before 
exporting the note, if all you want is the config? The should remove all output 
data, keeping it smaller?


_
From: Mohit Jaggi <mohitja...@gmail.com<mailto:mohitja...@gmail.com>>
Sent: Monday, September 12, 2016 10:38 AM
Subject: Re: Hbase configuration storage without data
To: <users@zeppelin.apache.org<mailto:users@zeppelin.apache.org>>



one option is to keep the data in separate files. notes.json can contain the 
code and the data can be a pointer to /path/to/file. import/export can choose 
to include or exclude the data. when it is included the data files are added to 
a tgz file containing notes.json otherwise you just export notes.json



On Mon, Sep 12, 2016 at 10:33 AM, moon soo Lee 
<m...@apache.org<mailto:m...@apache.org>> wrote:
Right big note.json file is a problem.
But at the same time, i think having data in the note.json helps make 
import/export simple and make notebook renderable without run it.

So far, i didn't see much discussion about this subject on mailing list or on 
the issue tracker.

If there's an good idea that can handle large data while keeping import/export 
simple and ability to render without run, that would be a great starting point 
of the discussions.

Thanks,
moon

On Wed, Sep 7, 2016 at 9:40 PM Vikash Kumar 
<vikash.ku...@resilinc.com<mailto:vikash.ku...@resilinc.com>> wrote:
Hi moon,
Yes that was the way that I was using. But is there any plan for future 
releases to removing the data from note and storing only configuration?
Because storing the configuration with data when there is no max result limit 
will create a big note.json file.

Thanks & Regards,
Vikash Kumar
From: moon soo Lee [mailto:m...@apache.org<mailto:m...@apache.org>]
Sent: Wednesday, September 7, 2016 8:39 PM
To: users@zeppelin.apache.org<mailto:users@zeppelin.apache.org>
Subject: Re: Hbase configuration storage without data

Hi,

For now, code and result data are mixed in note.json, which is represented by 
'class Note' [1]. And every Notebook storage layer need to implement 
'NotebookRepo.get()' [2] to read note.json from underlying storage and convert 
it into 'class Note'.

As you see the related API and class definition, NotebookRepo actually doesn't 
have any restriction how 'class Note' is serialized and saved in the storage.

So you can event new format, you can exclude result data from saving, and so on.

Hop this helps.

Thanks,
moon

[1] 
https://github.com/apache/zeppelin/blob/master/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java
[2] 
https://github.com/apache/zeppelin/blob/master/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/repo/NotebookRepo.java#L47

On Wed, Sep 7, 2016 at 3:47 AM Vikash Kumar 
<vikash.ku...@resilinc.com<mailto:vikash.ku...@resilinc.com>> wrote:
Hi all,
We are storing the note.json configuration into hbase as it is 
stored into File system. As default behavior in note.json the query data is 
stored along with configuration. But we want to store the configurations only 
and when user loading its note then query should get executed and data 
generated. This feature we are using for phoenix interpreter. So how can we 
remove the data from note.json? Is there any plan for that?


Thanks & Regards,
Vikash Kumar

Re: Hbase configuration storage without data

[Felix Cheung]

I like that approach - though you should be able to clear result output before 
exporting the note, if all you want is the config? The should remove all output 
data, keeping it smaller?


_
From: Mohit Jaggi <mohitja...@gmail.com<mailto:mohitja...@gmail.com>>
Sent: Monday, September 12, 2016 10:38 AM
Subject: Re: Hbase configuration storage without data
To: <users@zeppelin.apache.org<mailto:users@zeppelin.apache.org>>


one option is to keep the data in separate files. notes.json can contain the 
code and the data can be a pointer to /path/to/file. import/export can choose 
to include or exclude the data. when it is included the data files are added to 
a tgz file containing notes.json otherwise you just export notes.json



On Mon, Sep 12, 2016 at 10:33 AM, moon soo Lee 
<m...@apache.org<mailto:m...@apache.org>> wrote:
Right big note.json file is a problem.
But at the same time, i think having data in the note.json helps make 
import/export simple and make notebook renderable without run it.

So far, i didn't see much discussion about this subject on mailing list or on 
the issue tracker.

If there's an good idea that can handle large data while keeping import/export 
simple and ability to render without run, that would be a great starting point 
of the discussions.

Thanks,
moon

On Wed, Sep 7, 2016 at 9:40 PM Vikash Kumar 
<vikash.ku...@resilinc.com<mailto:vikash.ku...@resilinc.com>> wrote:
Hi moon,
Yes that was the way that I was using. But is there any plan for future 
releases to removing the data from note and storing only configuration?
Because storing the configuration with data when there is no max result limit 
will create a big note.json file.

Thanks & Regards,
Vikash Kumar
From: moon soo Lee [mailto:m...@apache.org<mailto:m...@apache.org>]
Sent: Wednesday, September 7, 2016 8:39 PM
To: users@zeppelin.apache.org<mailto:users@zeppelin.apache.org>
Subject: Re: Hbase configuration storage without data

Hi,

For now, code and result data are mixed in note.json, which is represented by 
'class Note' [1]. And every Notebook storage layer need to implement 
'NotebookRepo.get()' [2] to read note.json from underlying storage and convert 
it into 'class Note'.

As you see the related API and class definition, NotebookRepo actually doesn't 
have any restriction how 'class Note' is serialized and saved in the storage.

So you can event new format, you can exclude result data from saving, and so on.

Hop this helps.

Thanks,
moon

[1] 
https://github.com/apache/zeppelin/blob/master/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java
[2] 
https://github.com/apache/zeppelin/blob/master/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/repo/NotebookRepo.java#L47

On Wed, Sep 7, 2016 at 3:47 AM Vikash Kumar 
<vikash.ku...@resilinc.com<mailto:vikash.ku...@resilinc.com>> wrote:
Hi all,
We are storing the note.json configuration into hbase as it is 
stored into File system. As default behavior in note.json the query data is 
stored along with configuration. But we want to store the configurations only 
and when user loading its note then query should get executed and data 
generated. This feature we are using for phoenix interpreter. So how can we 
remove the data from note.json? Is there any plan for that?


Thanks & Regards,
Vikash Kumar

Re: Hbase configuration storage without data

[moon soo Lee]

Right big note.json file is a problem.
But at the same time, i think having data in the note.json helps make
import/export simple and make notebook renderable without run it.

So far, i didn't see much discussion about this subject on mailing list or
on the issue tracker.

If there's an good idea that can handle large data while keeping
import/export simple and ability to render without run, that would be a
great starting point of the discussions.

Thanks,
moon

On Wed, Sep 7, 2016 at 9:40 PM Vikash Kumar <vikash.ku...@resilinc.com>
wrote:

> Hi moon,
>
> Yes that was the way that I was using. But is there any plan for future
> releases to removing the data from note and storing only configuration?
>
> Because storing the configuration with data when there is no max result
> limit will create a big note.json file.
>
>
>
> Thanks & Regards,
>
> *Vikash Kumar*
>
> *From:* moon soo Lee [mailto:m...@apache.org]
> *Sent:* Wednesday, September 7, 2016 8:39 PM
> *To:* users@zeppelin.apache.org
> *Subject:* Re: Hbase configuration storage without data
>
>
>
> Hi,
>
>
>
> For now, code and result data are mixed in note.json, which is represented
> by 'class Note' [1]. And every Notebook storage layer need to implement
> 'NotebookRepo.get()' [2] to read note.json from underlying storage and
> convert it into 'class Note'.
>
>
>
> As you see the related API and class definition, NotebookRepo actually
> doesn't have any restriction how 'class Note' is serialized and saved in
> the storage.
>
>
>
> So you can event new format, you can exclude result data from saving, and
> so on.
>
>
>
> Hop this helps.
>
>
>
> Thanks,
>
> moon
>
>
>
> [1]
> https://github.com/apache/zeppelin/blob/master/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java
>
> [2]
> https://github.com/apache/zeppelin/blob/master/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/repo/NotebookRepo.java#L47
>
>
>
> On Wed, Sep 7, 2016 at 3:47 AM Vikash Kumar <vikash.ku...@resilinc.com>
> wrote:
>
> Hi all,
>
> We are storing the note.json configuration into hbase as
> it is stored into File system. As default behavior in note.json the query
> data is stored along with configuration. But we want to store the
> configurations only and when user loading its note then query should get
> executed and data generated. This feature we are using for phoenix
> interpreter. So how can we remove the data from note.json? Is there any
> plan for that?
>
>
>
>
>
> Thanks & Regards,
>
> *Vikash Kumar*
>
>

Re: Hbase configuration storage without data

[moon soo Lee]

Hi,

For now, code and result data are mixed in note.json, which is represented
by 'class Note' [1]. And every Notebook storage layer need to implement
'NotebookRepo.get()' [2] to read note.json from underlying storage and
convert it into 'class Note'.

As you see the related API and class definition, NotebookRepo actually
doesn't have any restriction how 'class Note' is serialized and saved in
the storage.

So you can event new format, you can exclude result data from saving, and
so on.

Hop this helps.

Thanks,
moon

[1]
https://github.com/apache/zeppelin/blob/master/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java
[2]
https://github.com/apache/zeppelin/blob/master/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/repo/NotebookRepo.java#L47


On Wed, Sep 7, 2016 at 3:47 AM Vikash Kumar 
wrote:

> Hi all,
>
> We are storing the note.json configuration into hbase as
> it is stored into File system. As default behavior in note.json the query
> data is stored along with configuration. But we want to store the
> configurations only and when user loading its note then query should get
> executed and data generated. This feature we are using for phoenix
> interpreter. So how can we remove the data from note.json? Is there any
> plan for that?
>
>
>
>
>
> Thanks & Regards,
>
> *Vikash Kumar*
>

Hbase configuration storage without data

[Vikash Kumar]

Hi all,
We are storing the note.json configuration into hbase as it is 
stored into File system. As default behavior in note.json the query data is 
stored along with configuration. But we want to store the configurations only 
and when user loading its note then query should get executed and data 
generated. This feature we are using for phoenix interpreter. So how can we 
remove the data from note.json? Is there any plan for that?


Thanks & Regards,
Vikash Kumar