Re: When kylin.job.run.as.remote.cmd=true, all hadoop、hive、hbase commands can be in remote cluster?

[Li Yang]

Thanks Albert, good to know!

On Sun, Jan 8, 2017 at 12:11 AM, Alberto Ramón 
wrote:

> To Apache Kylin Dev: This can be useful for you
>
>
> HDFS-9666 
>
> (It Will be in the next HDFS release
>  3CCAMs9kVjMdiw0AufB5u-HY%2B%2B%3D05ijLswZsQ7eyE7yLJxTh8EvsQ%
> 40mail.gmail.com%3E>
> 2.9.0)
>
> 2017-01-05 10:22 GMT+01:00 Alberto Ramón :
>
> > Upppsss,
> > Some times JAR Client and Server are the same file, (this is a
> performance
> > problem)
> > This can be a problem with docker, a lot of dependencies and program
> > runing, breakin the microservice philosophy
> >
> > This can be help you for *future*:
> > https://github.com/apache/hbase/tree/master/hbase-client
> > https://issues.apache.org/jira/browse/HDFS-6200  Finally finished, nice
> ¡¡
> > Artifact org.apache.zookeeper.client (To Find HBase master)
> >
> > *IDEA*: Query nodes, 'only' are used to 'resolve user queries', only need
> > access to HBase / Metatadata (¿?)
> >
> >
> >
> >
> > 2017-01-05 2:33 GMT+01:00 ShaoFeng Shi :
> >
> >> No necessary be hadoop nodes; The only thing is the node that running
> >> Kylin
> >> should have hadoop/hive/hbase clients installed and configured. Docker
> >> container is okay if it matches this.
> >>
> >> 2017-01-04 23:00 GMT+08:00 Alberto Ramón :
> >>
> >> > OK,
> >> >
> >> > I undertand that Kylin cluster mode must be on same nodes that hadoop
> >> > cluster. Is it correct?
> >> > I was thinked on put Kylin cluster mode on Docker Containers (Isolated
> >> from
> >> > Hadoop cluster)
> >> >
> >> > Thanks
> >> >
> >> > 2017-01-04 15:19 GMT+01:00 ShaoFeng Shi :
> >> >
> >> > > Here is the explainations in
> >> > > examples/test_case_data/sandbox/kylin.properties:
> >> > >
> >> > > # If true, job engine will not assume that hadoop CLI reside on the
> >> > > same server as it self
> >> > > # you will have to specify kylin.job.remote-cli-hostname,
> >> > > kylin.job.remote-cli-username and kylin.job.remote-cli-password
> >> > > # It should not be set to "true" unless you're NOT running Kylin.sh
> on
> >> > > a hadoop client machine
> >> > > # (Thus kylin instance has to ssh to another real hadoop client
> >> > > machine to execute hbase,hive,hadoop commands)
> >> > > kylin.job.use-remote-cli=false
> >> > >
> >> > > # Only necessary when kylin.job.use-remote-cli=true
> >> > > kylin.job.remote-cli-hostname=sandbox
> >> > >
> >> > > kylin.job.remote-cli-username=root
> >> > >
> >> > > # Only necessary when kylin.job.use-remote-cli=true
> >> > > kylin.job.remote-cli-password=hadoop
> >> > >
> >> > >
> >> > > Anyway, today Kylin need be running inner a Hadoop client machine.
> >> > > Using a jumpbox is not recommended and may not work, the only
> scenario
> >> > > is for development.
> >> > >
> >> > >
> >> > > 2017-01-04 17:14 GMT+08:00 Alberto Ramón  >:
> >> > >
> >> > > > is it the same remote mode than cluster mode ?
> >> > > > 
> >> > > >
> >> > > >
> >> > > >
> >> > > > 2017-01-04 9:35 GMT+01:00 ShaoFeng Shi :
> >> > > >
> >> > > > > Hi Lei,
> >> > > > >
> >> > > > > The remote mode is not recommended and not supported in real
> >> > > deployment.
> >> > > > > (we only use it in dev environment because local laptop doesn't
> >> have
> >> > > > > hive/hbase/hadoop configured, need ssh to a remote sandbox vm to
> >> run
> >> > > > > commands).
> >> > > > >
> >> > > > > Query doesn't support it;
> >> > > > >
> >> > > > > 2017-01-04 15:18 GMT+08:00 张磊 <121762...@qq.com>:
> >> > > > >
> >> > > > > > Hi ,
> >> > > > > >  I create a aws emr cluster which contains hadoop、hive、hbase
> >> > client,
> >> > > > and
> >> > > > > i
> >> > > > > > install kylin instance in another ec2 instance. Can kylin
> work?
> >> > > > > >
> >> > > > > >
> >> > > > > > In kylin.properties, i find job engine can work in remote
> mode,
> >> but
> >> > > > query
> >> > > > > > engine can also work well?
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > --
> >> > > > > Best regards,
> >> > > > >
> >> > > > > Shaofeng Shi 史少锋
> >> > > > >
> >> > > >
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > > Best regards,
> >> > >
> >> > > Shaofeng Shi 史少锋
> >> > >
> >> >
> >>
> >>
> >>
> >> --
> >> Best regards,
> >>
> >> Shaofeng Shi 史少锋
> >>
> >
> >
>

Re: When kylin.job.run.as.remote.cmd=true, all hadoop、hive、hbase commands can be in remote cluster?

[Alberto Ramón]

To Apache Kylin Dev: This can be useful for you


HDFS-9666 

(It Will be in the next HDFS release

2.9.0)

2017-01-05 10:22 GMT+01:00 Alberto Ramón :

> Upppsss,
> Some times JAR Client and Server are the same file, (this is a performance
> problem)
> This can be a problem with docker, a lot of dependencies and program
> runing, breakin the microservice philosophy
>
> This can be help you for *future*:
> https://github.com/apache/hbase/tree/master/hbase-client
> https://issues.apache.org/jira/browse/HDFS-6200  Finally finished, nice ¡¡
> Artifact org.apache.zookeeper.client (To Find HBase master)
>
> *IDEA*: Query nodes, 'only' are used to 'resolve user queries', only need
> access to HBase / Metatadata (¿?)
>
>
>
>
> 2017-01-05 2:33 GMT+01:00 ShaoFeng Shi :
>
>> No necessary be hadoop nodes; The only thing is the node that running
>> Kylin
>> should have hadoop/hive/hbase clients installed and configured. Docker
>> container is okay if it matches this.
>>
>> 2017-01-04 23:00 GMT+08:00 Alberto Ramón :
>>
>> > OK,
>> >
>> > I undertand that Kylin cluster mode must be on same nodes that hadoop
>> > cluster. Is it correct?
>> > I was thinked on put Kylin cluster mode on Docker Containers (Isolated
>> from
>> > Hadoop cluster)
>> >
>> > Thanks
>> >
>> > 2017-01-04 15:19 GMT+01:00 ShaoFeng Shi :
>> >
>> > > Here is the explainations in
>> > > examples/test_case_data/sandbox/kylin.properties:
>> > >
>> > > # If true, job engine will not assume that hadoop CLI reside on the
>> > > same server as it self
>> > > # you will have to specify kylin.job.remote-cli-hostname,
>> > > kylin.job.remote-cli-username and kylin.job.remote-cli-password
>> > > # It should not be set to "true" unless you're NOT running Kylin.sh on
>> > > a hadoop client machine
>> > > # (Thus kylin instance has to ssh to another real hadoop client
>> > > machine to execute hbase,hive,hadoop commands)
>> > > kylin.job.use-remote-cli=false
>> > >
>> > > # Only necessary when kylin.job.use-remote-cli=true
>> > > kylin.job.remote-cli-hostname=sandbox
>> > >
>> > > kylin.job.remote-cli-username=root
>> > >
>> > > # Only necessary when kylin.job.use-remote-cli=true
>> > > kylin.job.remote-cli-password=hadoop
>> > >
>> > >
>> > > Anyway, today Kylin need be running inner a Hadoop client machine.
>> > > Using a jumpbox is not recommended and may not work, the only scenario
>> > > is for development.
>> > >
>> > >
>> > > 2017-01-04 17:14 GMT+08:00 Alberto Ramón :
>> > >
>> > > > is it the same remote mode than cluster mode ?
>> > > > 
>> > > >
>> > > >
>> > > >
>> > > > 2017-01-04 9:35 GMT+01:00 ShaoFeng Shi :
>> > > >
>> > > > > Hi Lei,
>> > > > >
>> > > > > The remote mode is not recommended and not supported in real
>> > > deployment.
>> > > > > (we only use it in dev environment because local laptop doesn't
>> have
>> > > > > hive/hbase/hadoop configured, need ssh to a remote sandbox vm to
>> run
>> > > > > commands).
>> > > > >
>> > > > > Query doesn't support it;
>> > > > >
>> > > > > 2017-01-04 15:18 GMT+08:00 张磊 <121762...@qq.com>:
>> > > > >
>> > > > > > Hi ,
>> > > > > >  I create a aws emr cluster which contains hadoop、hive、hbase
>> > client,
>> > > > and
>> > > > > i
>> > > > > > install kylin instance in another ec2 instance. Can kylin work?
>> > > > > >
>> > > > > >
>> > > > > > In kylin.properties, i find job engine can work in remote mode,
>> but
>> > > > query
>> > > > > > engine can also work well?
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > Best regards,
>> > > > >
>> > > > > Shaofeng Shi 史少锋
>> > > > >
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > Best regards,
>> > >
>> > > Shaofeng Shi 史少锋
>> > >
>> >
>>
>>
>>
>> --
>> Best regards,
>>
>> Shaofeng Shi 史少锋
>>
>
>

Re: When kylin.job.run.as.remote.cmd=true, all hadoop、hive、hbase commands can be in remote cluster?

[Alberto Ramón]

Upppsss,
Some times JAR Client and Server are the same file, (this is a performance
problem)
This can be a problem with docker, a lot of dependencies and program
runing, breakin the microservice philosophy

This can be help you for *future*:
https://github.com/apache/hbase/tree/master/hbase-client
https://issues.apache.org/jira/browse/HDFS-6200  Finally finished, nice ¡¡
Artifact org.apache.zookeeper.client (To Find HBase master)

*IDEA*: Query nodes, 'only' are used to 'resolve user queries', only need
access to HBase / Metatadata (¿?)




2017-01-05 2:33 GMT+01:00 ShaoFeng Shi :

> No necessary be hadoop nodes; The only thing is the node that running Kylin
> should have hadoop/hive/hbase clients installed and configured. Docker
> container is okay if it matches this.
>
> 2017-01-04 23:00 GMT+08:00 Alberto Ramón :
>
> > OK,
> >
> > I undertand that Kylin cluster mode must be on same nodes that hadoop
> > cluster. Is it correct?
> > I was thinked on put Kylin cluster mode on Docker Containers (Isolated
> from
> > Hadoop cluster)
> >
> > Thanks
> >
> > 2017-01-04 15:19 GMT+01:00 ShaoFeng Shi :
> >
> > > Here is the explainations in
> > > examples/test_case_data/sandbox/kylin.properties:
> > >
> > > # If true, job engine will not assume that hadoop CLI reside on the
> > > same server as it self
> > > # you will have to specify kylin.job.remote-cli-hostname,
> > > kylin.job.remote-cli-username and kylin.job.remote-cli-password
> > > # It should not be set to "true" unless you're NOT running Kylin.sh on
> > > a hadoop client machine
> > > # (Thus kylin instance has to ssh to another real hadoop client
> > > machine to execute hbase,hive,hadoop commands)
> > > kylin.job.use-remote-cli=false
> > >
> > > # Only necessary when kylin.job.use-remote-cli=true
> > > kylin.job.remote-cli-hostname=sandbox
> > >
> > > kylin.job.remote-cli-username=root
> > >
> > > # Only necessary when kylin.job.use-remote-cli=true
> > > kylin.job.remote-cli-password=hadoop
> > >
> > >
> > > Anyway, today Kylin need be running inner a Hadoop client machine.
> > > Using a jumpbox is not recommended and may not work, the only scenario
> > > is for development.
> > >
> > >
> > > 2017-01-04 17:14 GMT+08:00 Alberto Ramón :
> > >
> > > > is it the same remote mode than cluster mode ?
> > > > 
> > > >
> > > >
> > > >
> > > > 2017-01-04 9:35 GMT+01:00 ShaoFeng Shi :
> > > >
> > > > > Hi Lei,
> > > > >
> > > > > The remote mode is not recommended and not supported in real
> > > deployment.
> > > > > (we only use it in dev environment because local laptop doesn't
> have
> > > > > hive/hbase/hadoop configured, need ssh to a remote sandbox vm to
> run
> > > > > commands).
> > > > >
> > > > > Query doesn't support it;
> > > > >
> > > > > 2017-01-04 15:18 GMT+08:00 张磊 <121762...@qq.com>:
> > > > >
> > > > > > Hi ,
> > > > > >  I create a aws emr cluster which contains hadoop、hive、hbase
> > client,
> > > > and
> > > > > i
> > > > > > install kylin instance in another ec2 instance. Can kylin work?
> > > > > >
> > > > > >
> > > > > > In kylin.properties, i find job engine can work in remote mode,
> but
> > > > query
> > > > > > engine can also work well?
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best regards,
> > > > >
> > > > > Shaofeng Shi 史少锋
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > >
> > > Shaofeng Shi 史少锋
> > >
> >
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>

Re: When kylin.job.run.as.remote.cmd=true, all hadoop、hive、hbase commands can be in remote cluster?

[ShaoFeng Shi]

No necessary be hadoop nodes; The only thing is the node that running Kylin
should have hadoop/hive/hbase clients installed and configured. Docker
container is okay if it matches this.

2017-01-04 23:00 GMT+08:00 Alberto Ramón :

> OK,
>
> I undertand that Kylin cluster mode must be on same nodes that hadoop
> cluster. Is it correct?
> I was thinked on put Kylin cluster mode on Docker Containers (Isolated from
> Hadoop cluster)
>
> Thanks
>
> 2017-01-04 15:19 GMT+01:00 ShaoFeng Shi :
>
> > Here is the explainations in
> > examples/test_case_data/sandbox/kylin.properties:
> >
> > # If true, job engine will not assume that hadoop CLI reside on the
> > same server as it self
> > # you will have to specify kylin.job.remote-cli-hostname,
> > kylin.job.remote-cli-username and kylin.job.remote-cli-password
> > # It should not be set to "true" unless you're NOT running Kylin.sh on
> > a hadoop client machine
> > # (Thus kylin instance has to ssh to another real hadoop client
> > machine to execute hbase,hive,hadoop commands)
> > kylin.job.use-remote-cli=false
> >
> > # Only necessary when kylin.job.use-remote-cli=true
> > kylin.job.remote-cli-hostname=sandbox
> >
> > kylin.job.remote-cli-username=root
> >
> > # Only necessary when kylin.job.use-remote-cli=true
> > kylin.job.remote-cli-password=hadoop
> >
> >
> > Anyway, today Kylin need be running inner a Hadoop client machine.
> > Using a jumpbox is not recommended and may not work, the only scenario
> > is for development.
> >
> >
> > 2017-01-04 17:14 GMT+08:00 Alberto Ramón :
> >
> > > is it the same remote mode than cluster mode ?
> > > 
> > >
> > >
> > >
> > > 2017-01-04 9:35 GMT+01:00 ShaoFeng Shi :
> > >
> > > > Hi Lei,
> > > >
> > > > The remote mode is not recommended and not supported in real
> > deployment.
> > > > (we only use it in dev environment because local laptop doesn't have
> > > > hive/hbase/hadoop configured, need ssh to a remote sandbox vm to run
> > > > commands).
> > > >
> > > > Query doesn't support it;
> > > >
> > > > 2017-01-04 15:18 GMT+08:00 张磊 <121762...@qq.com>:
> > > >
> > > > > Hi ,
> > > > >  I create a aws emr cluster which contains hadoop、hive、hbase
> client,
> > > and
> > > > i
> > > > > install kylin instance in another ec2 instance. Can kylin work?
> > > > >
> > > > >
> > > > > In kylin.properties, i find job engine can work in remote mode, but
> > > query
> > > > > engine can also work well?
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > >
> > > > Shaofeng Shi 史少锋
> > > >
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> > Shaofeng Shi 史少锋
> >
>



-- 
Best regards,

Shaofeng Shi 史少锋

Re: When kylin.job.run.as.remote.cmd=true, all hadoop、hive、hbase commands can be in remote cluster?

[Alberto Ramón]

OK,

I undertand that Kylin cluster mode must be on same nodes that hadoop
cluster. Is it correct?
I was thinked on put Kylin cluster mode on Docker Containers (Isolated from
Hadoop cluster)

Thanks

2017-01-04 15:19 GMT+01:00 ShaoFeng Shi :

> Here is the explainations in
> examples/test_case_data/sandbox/kylin.properties:
>
> # If true, job engine will not assume that hadoop CLI reside on the
> same server as it self
> # you will have to specify kylin.job.remote-cli-hostname,
> kylin.job.remote-cli-username and kylin.job.remote-cli-password
> # It should not be set to "true" unless you're NOT running Kylin.sh on
> a hadoop client machine
> # (Thus kylin instance has to ssh to another real hadoop client
> machine to execute hbase,hive,hadoop commands)
> kylin.job.use-remote-cli=false
>
> # Only necessary when kylin.job.use-remote-cli=true
> kylin.job.remote-cli-hostname=sandbox
>
> kylin.job.remote-cli-username=root
>
> # Only necessary when kylin.job.use-remote-cli=true
> kylin.job.remote-cli-password=hadoop
>
>
> Anyway, today Kylin need be running inner a Hadoop client machine.
> Using a jumpbox is not recommended and may not work, the only scenario
> is for development.
>
>
> 2017-01-04 17:14 GMT+08:00 Alberto Ramón :
>
> > is it the same remote mode than cluster mode ?
> > 
> >
> >
> >
> > 2017-01-04 9:35 GMT+01:00 ShaoFeng Shi :
> >
> > > Hi Lei,
> > >
> > > The remote mode is not recommended and not supported in real
> deployment.
> > > (we only use it in dev environment because local laptop doesn't have
> > > hive/hbase/hadoop configured, need ssh to a remote sandbox vm to run
> > > commands).
> > >
> > > Query doesn't support it;
> > >
> > > 2017-01-04 15:18 GMT+08:00 张磊 <121762...@qq.com>:
> > >
> > > > Hi ,
> > > >  I create a aws emr cluster which contains hadoop、hive、hbase client,
> > and
> > > i
> > > > install kylin instance in another ec2 instance. Can kylin work?
> > > >
> > > >
> > > > In kylin.properties, i find job engine can work in remote mode, but
> > query
> > > > engine can also work well?
> > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > >
> > > Shaofeng Shi 史少锋
> > >
> >
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>

Re: When kylin.job.run.as.remote.cmd=true, all hadoop、hive、hbase commands can be in remote cluster?

[ShaoFeng Shi]

Here is the explainations in
examples/test_case_data/sandbox/kylin.properties:

# If true, job engine will not assume that hadoop CLI reside on the
same server as it self
# you will have to specify kylin.job.remote-cli-hostname,
kylin.job.remote-cli-username and kylin.job.remote-cli-password
# It should not be set to "true" unless you're NOT running Kylin.sh on
a hadoop client machine
# (Thus kylin instance has to ssh to another real hadoop client
machine to execute hbase,hive,hadoop commands)
kylin.job.use-remote-cli=false

# Only necessary when kylin.job.use-remote-cli=true
kylin.job.remote-cli-hostname=sandbox

kylin.job.remote-cli-username=root

# Only necessary when kylin.job.use-remote-cli=true
kylin.job.remote-cli-password=hadoop


Anyway, today Kylin need be running inner a Hadoop client machine.
Using a jumpbox is not recommended and may not work, the only scenario
is for development.


2017-01-04 17:14 GMT+08:00 Alberto Ramón :

> is it the same remote mode than cluster mode ?
> 
>
>
>
> 2017-01-04 9:35 GMT+01:00 ShaoFeng Shi :
>
> > Hi Lei,
> >
> > The remote mode is not recommended and not supported in real deployment.
> > (we only use it in dev environment because local laptop doesn't have
> > hive/hbase/hadoop configured, need ssh to a remote sandbox vm to run
> > commands).
> >
> > Query doesn't support it;
> >
> > 2017-01-04 15:18 GMT+08:00 张磊 <121762...@qq.com>:
> >
> > > Hi ,
> > >  I create a aws emr cluster which contains hadoop、hive、hbase client,
> and
> > i
> > > install kylin instance in another ec2 instance. Can kylin work?
> > >
> > >
> > > In kylin.properties, i find job engine can work in remote mode, but
> query
> > > engine can also work well?
> >
> >
> >
> >
> > --
> > Best regards,
> >
> > Shaofeng Shi 史少锋
> >
>



-- 
Best regards,

Shaofeng Shi 史少锋

Re: When kylin.job.run.as.remote.cmd=true, all hadoop、hive、hbase commands can be in remote cluster?

[Alberto Ramón]

is it the same remote mode than cluster mode ?




2017-01-04 9:35 GMT+01:00 ShaoFeng Shi :

> Hi Lei,
>
> The remote mode is not recommended and not supported in real deployment.
> (we only use it in dev environment because local laptop doesn't have
> hive/hbase/hadoop configured, need ssh to a remote sandbox vm to run
> commands).
>
> Query doesn't support it;
>
> 2017-01-04 15:18 GMT+08:00 张磊 <121762...@qq.com>:
>
> > Hi ,
> >  I create a aws emr cluster which contains hadoop、hive、hbase client, and
> i
> > install kylin instance in another ec2 instance. Can kylin work?
> >
> >
> > In kylin.properties, i find job engine can work in remote mode, but query
> > engine can also work well?
>
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>

Re: When kylin.job.run.as.remote.cmd=true, all hadoop、hive、hbase commands can be in remote cluster?

[ShaoFeng Shi]

Hi Lei,

The remote mode is not recommended and not supported in real deployment.
(we only use it in dev environment because local laptop doesn't have
hive/hbase/hadoop configured, need ssh to a remote sandbox vm to run
commands).

Query doesn't support it;

2017-01-04 15:18 GMT+08:00 张磊 <121762...@qq.com>:

> Hi ,
>  I create a aws emr cluster which contains hadoop、hive、hbase client, and i
> install kylin instance in another ec2 instance. Can kylin work?
>
>
> In kylin.properties, i find job engine can work in remote mode, but query
> engine can also work well?




-- 
Best regards,

Shaofeng Shi 史少锋