Re: deploy dockerized drill cluster

[John Omernik]

So I think part of the problem here is the options on json are stored in
the zookeeper. Are you running the Zookeepers in the docker containers, in
separate docker containers, or completely separate?  For those settings,
I'd boot strap the cluster by itself, get it up and running, and then use
the ALTER SYSTEM set  command, either from SQLLINE or using the Rest API.
Then they will persist int he Zookeeper information.

I know I asked a similar question, because I wanted new clusters to be able
to be setup to use specified admin users. (I.e. I wanted to provide my list
of users that were admins prior to the first drill bit starting, and there
wasn't an easy way to do that at the time I asked).



On Sun, Jul 31, 2016 at 8:07 PM, Jesse Yates <jesse.k.ya...@gmail.com>
wrote:

> Ran into this myself today, so gonna try and close the loop.
>
> Looking at the 1.6 codeline (assuming its the same in 1.7), on startup the
> SystemOptionManager on pulls out the sys.options line from the config and
> uses that to build the PersistentStoreConfig (which get initialized from
> the store in #init()). However, this obviously doesn't take into account
> the config level options, only the startup options since those get pulled
> out to create everything else.
>
> It seems simple enough to wrap the SystemOptionManager into a
> FallbackOptionManager that falls back to the config file, if its not in the
> persistent store (or something like that).
>
> Unless someone already filed a JIRA on this?
>
> On Mon, Jul 25, 2016 at 10:05 AM Scott Kinney <scott.kin...@stem.com>
> wrote:
>
> > it's still not picking up the store.json* config changes
> >
> > The only way I can see to set these is with running ALTER SYSTEM query
> > after drill api is up.
> >
> >
> > 
> > Scott Kinney | DevOps
> > stem   |   m  510.282.1299
> > 100 Rollins Road, Millbrae, California 94030
> >
> > This e-mail and/or any attachments contain Stem, Inc. confidential and
> > proprietary information and material for the sole use of the intended
> > recipient(s). Any review, use or distribution that has not been expressly
> > authorized by Stem, Inc. is strictly prohibited. If you are not the
> > intended recipient, please contact the sender and delete all copies.
> Thank
> > you.
> >
> > ____
> > From: John Omernik <j...@omernik.com>
> > Sent: Monday, July 25, 2016 8:21 AM
> > To: user
> > Subject: Re: deploy dockerized drill cluster
> >
> > Try (for the sake of the conversation here) using host networking, and
> see
> > if it changes how successful your setup is.  (I know bridged is
> preferred,
> > but try the host side and see what happens)
> >
> > John
> >
> > On Mon, Jul 25, 2016 at 10:06 AM, Scott Kinney <scott.kin...@stem.com>
> > wrote:
> >
> > > I'm running the docker in bridged network mode.
> > >
> > >
> > > 
> > > Scott Kinney | DevOps
> > > stem   |   m  510.282.1299
> > > 100 Rollins Road, Millbrae, California 94030
> > >
> > > This e-mail and/or any attachments contain Stem, Inc. confidential and
> > > proprietary information and material for the sole use of the intended
> > > recipient(s). Any review, use or distribution that has not been
> expressly
> > > authorized by Stem, Inc. is strictly prohibited. If you are not the
> > > intended recipient, please contact the sender and delete all copies.
> > Thank
> > > you.
> > >
> > > 
> > > From: John Omernik <j...@omernik.com>
> > > Sent: Sunday, July 24, 2016 8:28 AM
> > > To: user
> > > Subject: Re: deploy dockerized drill cluster
> > >
> > > Are you running Drill in host networking or bridged networking?
> > >
> > > On Sat, Jul 23, 2016 at 1:21 PM, Scott Kinney <scott.kin...@stem.com>
> > > wrote:
> > >
> > > > Hm, i must have set those another way in embeded mode. I can't see
> > where.
> > > > Those settings persist between drill restarts.
> > > >
> > > >
> > > >
> > > > 
> > > > Scott Kinney | DevOps
> > > > stem   |   m  510.282.1299
> > > > 100 Rollins Road, Millbrae, California 94030
> > > >
> > > > This e-mail and/or any attachments contain Stem, Inc. confidential
> and
> > > > proprietary information and material for the sole use of th

Re: deploy dockerized drill cluster

[Scott Kinney]

it's still not picking up the store.json* config changes

The only way I can see to set these is with running ALTER SYSTEM query after 
drill api is up.



Scott Kinney | DevOps
stem   |   m  510.282.1299
100 Rollins Road, Millbrae, California 94030

This e-mail and/or any attachments contain Stem, Inc. confidential and 
proprietary information and material for the sole use of the intended 
recipient(s). Any review, use or distribution that has not been expressly 
authorized by Stem, Inc. is strictly prohibited. If you are not the intended 
recipient, please contact the sender and delete all copies. Thank you.


From: John Omernik <j...@omernik.com>
Sent: Monday, July 25, 2016 8:21 AM
To: user
Subject: Re: deploy dockerized drill cluster

Try (for the sake of the conversation here) using host networking, and see
if it changes how successful your setup is.  (I know bridged is preferred,
but try the host side and see what happens)

John

On Mon, Jul 25, 2016 at 10:06 AM, Scott Kinney <scott.kin...@stem.com>
wrote:

> I'm running the docker in bridged network mode.
>
>
> 
> Scott Kinney | DevOps
> stem   |   m  510.282.1299
> 100 Rollins Road, Millbrae, California 94030
>
> This e-mail and/or any attachments contain Stem, Inc. confidential and
> proprietary information and material for the sole use of the intended
> recipient(s). Any review, use or distribution that has not been expressly
> authorized by Stem, Inc. is strictly prohibited. If you are not the
> intended recipient, please contact the sender and delete all copies. Thank
> you.
>
> 
> From: John Omernik <j...@omernik.com>
> Sent: Sunday, July 24, 2016 8:28 AM
> To: user
> Subject: Re: deploy dockerized drill cluster
>
> Are you running Drill in host networking or bridged networking?
>
> On Sat, Jul 23, 2016 at 1:21 PM, Scott Kinney <scott.kin...@stem.com>
> wrote:
>
> > Hm, i must have set those another way in embeded mode. I can't see where.
> > Those settings persist between drill restarts.
> >
> >
> >
> > 
> > Scott Kinney | DevOps
> > stem   |   m  510.282.1299
> > 100 Rollins Road, Millbrae, California 94030
> >
> > This e-mail and/or any attachments contain Stem, Inc. confidential and
> > proprietary information and material for the sole use of the intended
> > recipient(s). Any review, use or distribution that has not been expressly
> > authorized by Stem, Inc. is strictly prohibited. If you are not the
> > intended recipient, please contact the sender and delete all copies.
> Thank
> > you.
> >
> > 
> > From: Abhishek Girish <abhishek.gir...@gmail.com>
> > Sent: Friday, July 22, 2016 1:57 PM
> > To: Drill User List
> > Subject: Re: deploy dockerized drill cluster
> >
> > You can set boot level start-up options in drill-override.conf [1]. But I
> > don't think we can do the same with the system options. Someone else can
> > comment if there is a workaround.
> >
> > On why it works for you with drill-embedded, is something I'm trying to
> > understand. I attempted this and couldn't manage to get those options to
> > show up in embedded mode.
> >
> > [1] https://drill.apache.org/docs/start-up-options/
> >
> > On Fri, Jul 22, 2016 at 1:20 PM, Scott Kinney <scott.kin...@stem.com>
> > wrote:
> >
> > > I have built a drill docker images very much like
> > > https://github.com/bigstepinc/apache-drill/blob/master/Dockerfile
> > >
> > >
> > > I volume mount a drill-override.conf file that looks like:
> > >
> > > drill.exec:{
> > >   cluster-id: drill1,
> > >   zk.connect: 192.1.1.1:2181",
> > >   sys.store.provider.local.path: "/drill-storage",
> > >   store.json.all_text_mode: True,
> > >   store.json.read_numbers_as_double: True
> > > }
> > >
> > > it seems to be connecting to the zookeeper (otherwise it would fail
> > right?
> > > i should actually confirm this).
> > >
> > > I know it is picking up my s3 plugin that i volume mount inside
> > > /drill-storage but it's not setting the json all_text_mode and
> > > read_numbers_as_double.
> > >
> > > I have a drill-embeded instance that seems to pick these json setting
> > from
> > > the drill-override file just fine.
> > >
> > > Can you see what I'm doing wrong?
> > >
> > >
> > >
> > >
> > >
> > > 
> > > Scott Kinney | DevOps
> > > stem <http://www.stem.com/>   |   m  510.282.1299
> > > 100 Rollins Road, Millbrae, California 94030
> > >
> > > This e-mail and/or any attachments contain Stem, Inc. confidential and
> > > proprietary information and material for the sole use of the intended
> > > recipient(s). Any review, use or distribution that has not been
> expressly
> > > authorized by Stem, Inc. is strictly prohibited. If you are not the
> > > intended recipient, please contact the sender and delete all copies.
> > Thank
> > > you.
> > >
> >
>

Re: deploy dockerized drill cluster

[John Omernik]

Try (for the sake of the conversation here) using host networking, and see
if it changes how successful your setup is.  (I know bridged is preferred,
but try the host side and see what happens)

John

On Mon, Jul 25, 2016 at 10:06 AM, Scott Kinney <scott.kin...@stem.com>
wrote:

> I'm running the docker in bridged network mode.
>
>
> 
> Scott Kinney | DevOps
> stem   |   m  510.282.1299
> 100 Rollins Road, Millbrae, California 94030
>
> This e-mail and/or any attachments contain Stem, Inc. confidential and
> proprietary information and material for the sole use of the intended
> recipient(s). Any review, use or distribution that has not been expressly
> authorized by Stem, Inc. is strictly prohibited. If you are not the
> intended recipient, please contact the sender and delete all copies. Thank
> you.
>
> 
> From: John Omernik <j...@omernik.com>
> Sent: Sunday, July 24, 2016 8:28 AM
> To: user
> Subject: Re: deploy dockerized drill cluster
>
> Are you running Drill in host networking or bridged networking?
>
> On Sat, Jul 23, 2016 at 1:21 PM, Scott Kinney <scott.kin...@stem.com>
> wrote:
>
> > Hm, i must have set those another way in embeded mode. I can't see where.
> > Those settings persist between drill restarts.
> >
> >
> >
> > 
> > Scott Kinney | DevOps
> > stem   |   m  510.282.1299
> > 100 Rollins Road, Millbrae, California 94030
> >
> > This e-mail and/or any attachments contain Stem, Inc. confidential and
> > proprietary information and material for the sole use of the intended
> > recipient(s). Any review, use or distribution that has not been expressly
> > authorized by Stem, Inc. is strictly prohibited. If you are not the
> > intended recipient, please contact the sender and delete all copies.
> Thank
> > you.
> >
> > 
> > From: Abhishek Girish <abhishek.gir...@gmail.com>
> > Sent: Friday, July 22, 2016 1:57 PM
> > To: Drill User List
> > Subject: Re: deploy dockerized drill cluster
> >
> > You can set boot level start-up options in drill-override.conf [1]. But I
> > don't think we can do the same with the system options. Someone else can
> > comment if there is a workaround.
> >
> > On why it works for you with drill-embedded, is something I'm trying to
> > understand. I attempted this and couldn't manage to get those options to
> > show up in embedded mode.
> >
> > [1] https://drill.apache.org/docs/start-up-options/
> >
> > On Fri, Jul 22, 2016 at 1:20 PM, Scott Kinney <scott.kin...@stem.com>
> > wrote:
> >
> > > I have built a drill docker images very much like
> > > https://github.com/bigstepinc/apache-drill/blob/master/Dockerfile
> > >
> > >
> > > I volume mount a drill-override.conf file that looks like:
> > >
> > > drill.exec:{
> > >   cluster-id: drill1,
> > >   zk.connect: 192.1.1.1:2181",
> > >   sys.store.provider.local.path: "/drill-storage",
> > >   store.json.all_text_mode: True,
> > >   store.json.read_numbers_as_double: True
> > > }
> > >
> > > it seems to be connecting to the zookeeper (otherwise it would fail
> > right?
> > > i should actually confirm this).
> > >
> > > I know it is picking up my s3 plugin that i volume mount inside
> > > /drill-storage but it's not setting the json all_text_mode and
> > > read_numbers_as_double.
> > >
> > > I have a drill-embeded instance that seems to pick these json setting
> > from
> > > the drill-override file just fine.
> > >
> > > Can you see what I'm doing wrong?
> > >
> > >
> > >
> > >
> > >
> > > 
> > > Scott Kinney | DevOps
> > > stem <http://www.stem.com/>   |   m  510.282.1299
> > > 100 Rollins Road, Millbrae, California 94030
> > >
> > > This e-mail and/or any attachments contain Stem, Inc. confidential and
> > > proprietary information and material for the sole use of the intended
> > > recipient(s). Any review, use or distribution that has not been
> expressly
> > > authorized by Stem, Inc. is strictly prohibited. If you are not the
> > > intended recipient, please contact the sender and delete all copies.
> > Thank
> > > you.
> > >
> >
>

Re: deploy dockerized drill cluster

[John Omernik]

Are you running Drill in host networking or bridged networking?

On Sat, Jul 23, 2016 at 1:21 PM, Scott Kinney <scott.kin...@stem.com> wrote:

> Hm, i must have set those another way in embeded mode. I can't see where.
> Those settings persist between drill restarts.
>
>
>
> 
> Scott Kinney | DevOps
> stem   |   m  510.282.1299
> 100 Rollins Road, Millbrae, California 94030
>
> This e-mail and/or any attachments contain Stem, Inc. confidential and
> proprietary information and material for the sole use of the intended
> recipient(s). Any review, use or distribution that has not been expressly
> authorized by Stem, Inc. is strictly prohibited. If you are not the
> intended recipient, please contact the sender and delete all copies. Thank
> you.
>
> 
> From: Abhishek Girish <abhishek.gir...@gmail.com>
> Sent: Friday, July 22, 2016 1:57 PM
> To: Drill User List
> Subject: Re: deploy dockerized drill cluster
>
> You can set boot level start-up options in drill-override.conf [1]. But I
> don't think we can do the same with the system options. Someone else can
> comment if there is a workaround.
>
> On why it works for you with drill-embedded, is something I'm trying to
> understand. I attempted this and couldn't manage to get those options to
> show up in embedded mode.
>
> [1] https://drill.apache.org/docs/start-up-options/
>
> On Fri, Jul 22, 2016 at 1:20 PM, Scott Kinney <scott.kin...@stem.com>
> wrote:
>
> > I have built a drill docker images very much like
> > https://github.com/bigstepinc/apache-drill/blob/master/Dockerfile
> >
> >
> > I volume mount a drill-override.conf file that looks like:
> >
> > drill.exec:{
> >   cluster-id: drill1,
> >   zk.connect: 192.1.1.1:2181",
> >   sys.store.provider.local.path: "/drill-storage",
> >   store.json.all_text_mode: True,
> >   store.json.read_numbers_as_double: True
> > }
> >
> > it seems to be connecting to the zookeeper (otherwise it would fail
> right?
> > i should actually confirm this).
> >
> > I know it is picking up my s3 plugin that i volume mount inside
> > /drill-storage but it's not setting the json all_text_mode and
> > read_numbers_as_double.
> >
> > I have a drill-embeded instance that seems to pick these json setting
> from
> > the drill-override file just fine.
> >
> > Can you see what I'm doing wrong?
> >
> >
> >
> >
> >
> > 
> > Scott Kinney | DevOps
> > stem <http://www.stem.com/>   |   m  510.282.1299
> > 100 Rollins Road, Millbrae, California 94030
> >
> > This e-mail and/or any attachments contain Stem, Inc. confidential and
> > proprietary information and material for the sole use of the intended
> > recipient(s). Any review, use or distribution that has not been expressly
> > authorized by Stem, Inc. is strictly prohibited. If you are not the
> > intended recipient, please contact the sender and delete all copies.
> Thank
> > you.
> >
>

Re: deploy dockerized drill cluster

[Scott Kinney]

Hm, i must have set those another way in embeded mode. I can't see where. Those 
settings persist between drill restarts.




Scott Kinney | DevOps
stem   |   m  510.282.1299
100 Rollins Road, Millbrae, California 94030

This e-mail and/or any attachments contain Stem, Inc. confidential and 
proprietary information and material for the sole use of the intended 
recipient(s). Any review, use or distribution that has not been expressly 
authorized by Stem, Inc. is strictly prohibited. If you are not the intended 
recipient, please contact the sender and delete all copies. Thank you.


From: Abhishek Girish <abhishek.gir...@gmail.com>
Sent: Friday, July 22, 2016 1:57 PM
To: Drill User List
Subject: Re: deploy dockerized drill cluster

You can set boot level start-up options in drill-override.conf [1]. But I
don't think we can do the same with the system options. Someone else can
comment if there is a workaround.

On why it works for you with drill-embedded, is something I'm trying to
understand. I attempted this and couldn't manage to get those options to
show up in embedded mode.

[1] https://drill.apache.org/docs/start-up-options/

On Fri, Jul 22, 2016 at 1:20 PM, Scott Kinney <scott.kin...@stem.com> wrote:

> I have built a drill docker images very much like
> https://github.com/bigstepinc/apache-drill/blob/master/Dockerfile
>
>
> I volume mount a drill-override.conf file that looks like:
>
> drill.exec:{
>   cluster-id: drill1,
>   zk.connect: 192.1.1.1:2181",
>   sys.store.provider.local.path: "/drill-storage",
>   store.json.all_text_mode: True,
>   store.json.read_numbers_as_double: True
> }
>
> it seems to be connecting to the zookeeper (otherwise it would fail right?
> i should actually confirm this).
>
> I know it is picking up my s3 plugin that i volume mount inside
> /drill-storage but it's not setting the json all_text_mode and
> read_numbers_as_double.
>
> I have a drill-embeded instance that seems to pick these json setting from
> the drill-override file just fine.
>
> Can you see what I'm doing wrong?
>
>
>
>
>
> 
> Scott Kinney | DevOps
> stem <http://www.stem.com/>   |   m  510.282.1299
> 100 Rollins Road, Millbrae, California 94030
>
> This e-mail and/or any attachments contain Stem, Inc. confidential and
> proprietary information and material for the sole use of the intended
> recipient(s). Any review, use or distribution that has not been expressly
> authorized by Stem, Inc. is strictly prohibited. If you are not the
> intended recipient, please contact the sender and delete all copies. Thank
> you.
>

Re: deploy dockerized drill cluster

[Abhishek Girish]

You can set boot level start-up options in drill-override.conf [1]. But I
don't think we can do the same with the system options. Someone else can
comment if there is a workaround.

On why it works for you with drill-embedded, is something I'm trying to
understand. I attempted this and couldn't manage to get those options to
show up in embedded mode.

[1] https://drill.apache.org/docs/start-up-options/

On Fri, Jul 22, 2016 at 1:20 PM, Scott Kinney  wrote:

> I have built a drill docker images very much like
> https://github.com/bigstepinc/apache-drill/blob/master/Dockerfile
>
>
> I volume mount a drill-override.conf file that looks like:
>
> drill.exec:{
>   cluster-id: drill1,
>   zk.connect: 192.1.1.1:2181",
>   sys.store.provider.local.path: "/drill-storage",
>   store.json.all_text_mode: True,
>   store.json.read_numbers_as_double: True
> }
>
> it seems to be connecting to the zookeeper (otherwise it would fail right?
> i should actually confirm this).
>
> I know it is picking up my s3 plugin that i volume mount inside
> /drill-storage but it's not setting the json all_text_mode and
> read_numbers_as_double.
>
> I have a drill-embeded instance that seems to pick these json setting from
> the drill-override file just fine.
>
> Can you see what I'm doing wrong?
>
>
>
>
>
> 
> Scott Kinney | DevOps
> stem    |   m  510.282.1299
> 100 Rollins Road, Millbrae, California 94030
>
> This e-mail and/or any attachments contain Stem, Inc. confidential and
> proprietary information and material for the sole use of the intended
> recipient(s). Any review, use or distribution that has not been expressly
> authorized by Stem, Inc. is strictly prohibited. If you are not the
> intended recipient, please contact the sender and delete all copies. Thank
> you.
>

deploy dockerized drill cluster

[Scott Kinney]

I have built a drill docker images very much like 
https://github.com/bigstepinc/apache-drill/blob/master/Dockerfile


I volume mount a drill-override.conf file that looks like:

drill.exec:{
  cluster-id: drill1,
  zk.connect: 192.1.1.1:2181",
  sys.store.provider.local.path: "/drill-storage",
  store.json.all_text_mode: True,
  store.json.read_numbers_as_double: True
}

it seems to be connecting to the zookeeper (otherwise it would fail right? i 
should actually confirm this).

I know it is picking up my s3 plugin that i volume mount inside /drill-storage 
but it's not setting the json all_text_mode and read_numbers_as_double.

I have a drill-embeded instance that seems to pick these json setting from the 
drill-override file just fine.

Can you see what I'm doing wrong?






Scott Kinney | DevOps
stem    |   m  510.282.1299
100 Rollins Road, Millbrae, California 94030

This e-mail and/or any attachments contain Stem, Inc. confidential and 
proprietary information and material for the sole use of the intended 
recipient(s). Any review, use or distribution that has not been expressly 
authorized by Stem, Inc. is strictly prohibited. If you are not the intended 
recipient, please contact the sender and delete all copies. Thank you.