Re: How to start HDFS on Spark Standalone

[Michael Segel]

Perhaps this is a silly question on my part…. 

Why do you want to start up HDFS on a single node?

You only mention one windows machine in your description of your cluster. 
If this is a learning  experience, why not run Hadoop in a VM (MapR and I think 
the other vendors make linux images that can run in a VM) 

HTH

-Mike

> On Apr 18, 2016, at 10:27 AM, Jörn Franke  wrote:
> 
> I think the easiest would be to use a Hadoop Windows distribution, such as 
> Hortonworks. However, the Linux version of Hortonworks is a little bit more 
> advanced.
> 
> On 18 Apr 2016, at 14:13, My List  > wrote:
> 
>> Deepak,
>> 
>> The following could be a very dumb questions so pardon me for the same.
>> 1) When I download the binary for Spark with a version of Hadoop(Hadoop 2.6) 
>> does it not come in the zip or tar file?
>> 2) If it does not come along,Is there a Apache Hadoop for windows, is it in 
>> binary format or will have to build it?
>> 3) Is there a basic tutorial for Hadoop on windows for the basic needs of 
>> Spark.
>> 
>> Thanks in Advance !
>> 
>> On Mon, Apr 18, 2016 at 5:35 PM, Deepak Sharma > > wrote:
>> Once you download hadoop and format the namenode , you can use start-dfs.sh 
>> to start hdfs.
>> Then use 'jps' to sss if datanode/namenode services are up and running.
>> 
>> Thanks
>> Deepak
>> 
>> On Mon, Apr 18, 2016 at 5:18 PM, My List > > wrote:
>> Hi ,
>> 
>> I am a newbie on Spark.I wanted to know how to start and verify if HDFS has 
>> started on Spark stand alone.
>> 
>> Env - 
>> Windows 7 - 64 bit
>> Spark 1.4.1 With Hadoop 2.6
>> 
>> Using Scala Shell - spark-shell
>> 
>> 
>> -- 
>> Thanks,
>> Harry
>> 
>> 
>> 
>> -- 
>> Thanks
>> Deepak
>> www.bigdatabig.com 
>> www.keosha.net 
>> 
>> 
>> -- 
>> Thanks,
>> Harmeet

Re: How to start HDFS on Spark Standalone

[Jörn Franke]

I think the easiest would be to use a Hadoop Windows distribution, such as 
Hortonworks. However, the Linux version of Hortonworks is a little bit more 
advanced.

> On 18 Apr 2016, at 14:13, My List  wrote:
> 
> Deepak,
> 
> The following could be a very dumb questions so pardon me for the same.
> 1) When I download the binary for Spark with a version of Hadoop(Hadoop 2.6) 
> does it not come in the zip or tar file?
> 2) If it does not come along,Is there a Apache Hadoop for windows, is it in 
> binary format or will have to build it?
> 3) Is there a basic tutorial for Hadoop on windows for the basic needs of 
> Spark.
> 
> Thanks in Advance !
> 
>> On Mon, Apr 18, 2016 at 5:35 PM, Deepak Sharma  wrote:
>> Once you download hadoop and format the namenode , you can use start-dfs.sh 
>> to start hdfs.
>> Then use 'jps' to sss if datanode/namenode services are up and running.
>> 
>> Thanks
>> Deepak
>> 
>>> On Mon, Apr 18, 2016 at 5:18 PM, My List  wrote:
>>> Hi ,
>>> 
>>> I am a newbie on Spark.I wanted to know how to start and verify if HDFS has 
>>> started on Spark stand alone.
>>> 
>>> Env - 
>>> Windows 7 - 64 bit
>>> Spark 1.4.1 With Hadoop 2.6
>>> 
>>> Using Scala Shell - spark-shell
>>> 
>>> 
>>> -- 
>>> Thanks,
>>> Harry
>> 
>> 
>> 
>> -- 
>> Thanks
>> Deepak
>> www.bigdatabig.com
>> www.keosha.net
> 
> 
> 
> -- 
> Thanks,
> Harmeet

Re: How to start HDFS on Spark Standalone

[Steve Loughran]

1. You don't need to start HDFS or anything like that, just set up Spark so 
that it can use the Hadoop APIs for some things; on windows this depends on 
some native libs. This means you don't need to worry about learning it. Focus 
on the Spark APIs, python and/or scala

2. You should be able to find the native windows binaries here: 
https://github.com/steveloughran/winutils

These are either version I lifted out of HDP for windows, or, more recently, 
checking out and building on Windows the same git commit as used/voted in for 
the ASF releases.

3. For performance, you also need the native libs for compression codecs: 
snappy, LZO  I see I've put them in the windows 2.7.1 release, but not the 
others (ASF mvn package doesn't add them): 
https://github.com/steveloughran/winutils/tree/master/hadoop-2.7.1/bin ...

If the version of the hadoop libs aren't working with one of those versions 
I've put up, ping me and I'll build up the relevant binaries out of the ASF




From: My List <mylistt...@gmail.com<mailto:mylistt...@gmail.com>>
Date: Monday, 18 April 2016 at 13:13
To: Deepak Sharma <deepakmc...@gmail.com<mailto:deepakmc...@gmail.com>>
Cc: SparkUser <user@spark.apache.org<mailto:user@spark.apache.org>>
Subject: Re: How to start HDFS on Spark Standalone

Deepak,

The following could be a very dumb questions so pardon me for the same.
1) When I download the binary for Spark with a version of Hadoop(Hadoop 2.6) 
does it not come in the zip or tar file?
2) If it does not come along,Is there a Apache Hadoop for windows, is it in 
binary format or will have to build it?
3) Is there a basic tutorial for Hadoop on windows for the basic needs of Spark.

Thanks in Advance !

On Mon, Apr 18, 2016 at 5:35 PM, Deepak Sharma 
<deepakmc...@gmail.com<mailto:deepakmc...@gmail.com>> wrote:
Once you download hadoop and format the namenode , you can use start-dfs.sh to 
start hdfs.
Then use 'jps' to sss if datanode/namenode services are up and running.

Thanks
Deepak

On Mon, Apr 18, 2016 at 5:18 PM, My List 
<mylistt...@gmail.com<mailto:mylistt...@gmail.com>> wrote:
Hi ,

I am a newbie on Spark.I wanted to know how to start and verify if HDFS has 
started on Spark stand alone.

Env -
Windows 7 - 64 bit
Spark 1.4.1 With Hadoop 2.6

Using Scala Shell - spark-shell


--
Thanks,
Harry



--
Thanks
Deepak
www.bigdatabig.com<http://www.bigdatabig.com>
www.keosha.net<http://www.keosha.net>



--
Thanks,
Harmeet

Re: How to start HDFS on Spark Standalone

[Deepak Sharma]

I prefer using CentOS/SLES/Ubuntu personally.

Thanks
Deepak

On Mon, Apr 18, 2016 at 5:57 PM, My List  wrote:

> Deepak,
>
> I love the unix flavors have been a programmed on them. Just have a
> windows laptop and pc hence haven't moved to unix flavors. Was trying to
> run big data stuff on windows. Have run in so much of issues that I could
> just throw the laptop with windows out.
>
> Your view - Redhat, Ubuntu or Centos.
> Does Redhat give a one year licence on purchase etc?
>
> Thanks
>
> On Mon, Apr 18, 2016 at 5:52 PM, Deepak Sharma 
> wrote:
>
>> It works well with all flavors of Linux.
>> It all depends on your ex with these flavors.
>>
>> Thanks
>> Deepak
>>
>> On Mon, Apr 18, 2016 at 5:51 PM, My List  wrote:
>>
>>> Deepak,
>>>
>>> Would you advice that I use Ubuntu? or Redhat. Cause Windows support etc
>>> and issues are Galore on Spark.
>>> Since I am starting afresh, what would you advice?
>>>
>>> On Mon, Apr 18, 2016 at 5:45 PM, Deepak Sharma 
>>> wrote:
>>>
 Binary for Spark means ts spark built against hadoop 2.6
 It will not have any hadoop executables.
 You'll have to setup hadoop separately.
 I have not used windows version yet but there are some.

 Thanks
 Deepak

 On Mon, Apr 18, 2016 at 5:43 PM, My List  wrote:

> Deepak,
>
> The following could be a very dumb questions so pardon me for the same.
> 1) When I download the binary for Spark with a version of
> Hadoop(Hadoop 2.6) does it not come in the zip or tar file?
> 2) If it does not come along,Is there a Apache Hadoop for windows, is
> it in binary format or will have to build it?
> 3) Is there a basic tutorial for Hadoop on windows for the basic needs
> of Spark.
>
> Thanks in Advance !
>
> On Mon, Apr 18, 2016 at 5:35 PM, Deepak Sharma 
> wrote:
>
>> Once you download hadoop and format the namenode , you can use
>> start-dfs.sh to start hdfs.
>> Then use 'jps' to sss if datanode/namenode services are up and
>> running.
>>
>> Thanks
>> Deepak
>>
>> On Mon, Apr 18, 2016 at 5:18 PM, My List 
>> wrote:
>>
>>> Hi ,
>>>
>>> I am a newbie on Spark.I wanted to know how to start and verify if
>>> HDFS has started on Spark stand alone.
>>>
>>> Env -
>>> Windows 7 - 64 bit
>>> Spark 1.4.1 With Hadoop 2.6
>>>
>>> Using Scala Shell - spark-shell
>>>
>>>
>>> --
>>> Thanks,
>>> Harry
>>>
>>
>>
>>
>> --
>> Thanks
>> Deepak
>> www.bigdatabig.com
>> www.keosha.net
>>
>
>
>
> --
> Thanks,
> Harmeet
>



 --
 Thanks
 Deepak
 www.bigdatabig.com
 www.keosha.net

>>>
>>>
>>>
>>> --
>>> Thanks,
>>> Harmeet
>>>
>>
>>
>>
>> --
>> Thanks
>> Deepak
>> www.bigdatabig.com
>> www.keosha.net
>>
>
>
>
> --
> Thanks,
> Harmeet
>



-- 
Thanks
Deepak
www.bigdatabig.com
www.keosha.net

Re: How to start HDFS on Spark Standalone

[My List]

Deepak,

I love the unix flavors have been a programmed on them. Just have a windows
laptop and pc hence haven't moved to unix flavors. Was trying to run big
data stuff on windows. Have run in so much of issues that I could just
throw the laptop with windows out.

Your view - Redhat, Ubuntu or Centos.
Does Redhat give a one year licence on purchase etc?

Thanks

On Mon, Apr 18, 2016 at 5:52 PM, Deepak Sharma 
wrote:

> It works well with all flavors of Linux.
> It all depends on your ex with these flavors.
>
> Thanks
> Deepak
>
> On Mon, Apr 18, 2016 at 5:51 PM, My List  wrote:
>
>> Deepak,
>>
>> Would you advice that I use Ubuntu? or Redhat. Cause Windows support etc
>> and issues are Galore on Spark.
>> Since I am starting afresh, what would you advice?
>>
>> On Mon, Apr 18, 2016 at 5:45 PM, Deepak Sharma 
>> wrote:
>>
>>> Binary for Spark means ts spark built against hadoop 2.6
>>> It will not have any hadoop executables.
>>> You'll have to setup hadoop separately.
>>> I have not used windows version yet but there are some.
>>>
>>> Thanks
>>> Deepak
>>>
>>> On Mon, Apr 18, 2016 at 5:43 PM, My List  wrote:
>>>
 Deepak,

 The following could be a very dumb questions so pardon me for the same.
 1) When I download the binary for Spark with a version of Hadoop(Hadoop
 2.6) does it not come in the zip or tar file?
 2) If it does not come along,Is there a Apache Hadoop for windows, is
 it in binary format or will have to build it?
 3) Is there a basic tutorial for Hadoop on windows for the basic needs
 of Spark.

 Thanks in Advance !

 On Mon, Apr 18, 2016 at 5:35 PM, Deepak Sharma 
 wrote:

> Once you download hadoop and format the namenode , you can use
> start-dfs.sh to start hdfs.
> Then use 'jps' to sss if datanode/namenode services are up and running.
>
> Thanks
> Deepak
>
> On Mon, Apr 18, 2016 at 5:18 PM, My List  wrote:
>
>> Hi ,
>>
>> I am a newbie on Spark.I wanted to know how to start and verify if
>> HDFS has started on Spark stand alone.
>>
>> Env -
>> Windows 7 - 64 bit
>> Spark 1.4.1 With Hadoop 2.6
>>
>> Using Scala Shell - spark-shell
>>
>>
>> --
>> Thanks,
>> Harry
>>
>
>
>
> --
> Thanks
> Deepak
> www.bigdatabig.com
> www.keosha.net
>



 --
 Thanks,
 Harmeet

>>>
>>>
>>>
>>> --
>>> Thanks
>>> Deepak
>>> www.bigdatabig.com
>>> www.keosha.net
>>>
>>
>>
>>
>> --
>> Thanks,
>> Harmeet
>>
>
>
>
> --
> Thanks
> Deepak
> www.bigdatabig.com
> www.keosha.net
>



-- 
Thanks,
Harmeet

Re: How to start HDFS on Spark Standalone

[Deepak Sharma]

It works well with all flavors of Linux.
It all depends on your ex with these flavors.

Thanks
Deepak

On Mon, Apr 18, 2016 at 5:51 PM, My List  wrote:

> Deepak,
>
> Would you advice that I use Ubuntu? or Redhat. Cause Windows support etc
> and issues are Galore on Spark.
> Since I am starting afresh, what would you advice?
>
> On Mon, Apr 18, 2016 at 5:45 PM, Deepak Sharma 
> wrote:
>
>> Binary for Spark means ts spark built against hadoop 2.6
>> It will not have any hadoop executables.
>> You'll have to setup hadoop separately.
>> I have not used windows version yet but there are some.
>>
>> Thanks
>> Deepak
>>
>> On Mon, Apr 18, 2016 at 5:43 PM, My List  wrote:
>>
>>> Deepak,
>>>
>>> The following could be a very dumb questions so pardon me for the same.
>>> 1) When I download the binary for Spark with a version of Hadoop(Hadoop
>>> 2.6) does it not come in the zip or tar file?
>>> 2) If it does not come along,Is there a Apache Hadoop for windows, is it
>>> in binary format or will have to build it?
>>> 3) Is there a basic tutorial for Hadoop on windows for the basic needs
>>> of Spark.
>>>
>>> Thanks in Advance !
>>>
>>> On Mon, Apr 18, 2016 at 5:35 PM, Deepak Sharma 
>>> wrote:
>>>
 Once you download hadoop and format the namenode , you can use
 start-dfs.sh to start hdfs.
 Then use 'jps' to sss if datanode/namenode services are up and running.

 Thanks
 Deepak

 On Mon, Apr 18, 2016 at 5:18 PM, My List  wrote:

> Hi ,
>
> I am a newbie on Spark.I wanted to know how to start and verify if
> HDFS has started on Spark stand alone.
>
> Env -
> Windows 7 - 64 bit
> Spark 1.4.1 With Hadoop 2.6
>
> Using Scala Shell - spark-shell
>
>
> --
> Thanks,
> Harry
>



 --
 Thanks
 Deepak
 www.bigdatabig.com
 www.keosha.net

>>>
>>>
>>>
>>> --
>>> Thanks,
>>> Harmeet
>>>
>>
>>
>>
>> --
>> Thanks
>> Deepak
>> www.bigdatabig.com
>> www.keosha.net
>>
>
>
>
> --
> Thanks,
> Harmeet
>



-- 
Thanks
Deepak
www.bigdatabig.com
www.keosha.net

Re: How to start HDFS on Spark Standalone

[My List]

Deepak,

Would you advice that I use Ubuntu? or Redhat. Cause Windows support etc
and issues are Galore on Spark.
Since I am starting afresh, what would you advice?

On Mon, Apr 18, 2016 at 5:45 PM, Deepak Sharma 
wrote:

> Binary for Spark means ts spark built against hadoop 2.6
> It will not have any hadoop executables.
> You'll have to setup hadoop separately.
> I have not used windows version yet but there are some.
>
> Thanks
> Deepak
>
> On Mon, Apr 18, 2016 at 5:43 PM, My List  wrote:
>
>> Deepak,
>>
>> The following could be a very dumb questions so pardon me for the same.
>> 1) When I download the binary for Spark with a version of Hadoop(Hadoop
>> 2.6) does it not come in the zip or tar file?
>> 2) If it does not come along,Is there a Apache Hadoop for windows, is it
>> in binary format or will have to build it?
>> 3) Is there a basic tutorial for Hadoop on windows for the basic needs of
>> Spark.
>>
>> Thanks in Advance !
>>
>> On Mon, Apr 18, 2016 at 5:35 PM, Deepak Sharma 
>> wrote:
>>
>>> Once you download hadoop and format the namenode , you can use
>>> start-dfs.sh to start hdfs.
>>> Then use 'jps' to sss if datanode/namenode services are up and running.
>>>
>>> Thanks
>>> Deepak
>>>
>>> On Mon, Apr 18, 2016 at 5:18 PM, My List  wrote:
>>>
 Hi ,

 I am a newbie on Spark.I wanted to know how to start and verify if HDFS
 has started on Spark stand alone.

 Env -
 Windows 7 - 64 bit
 Spark 1.4.1 With Hadoop 2.6

 Using Scala Shell - spark-shell


 --
 Thanks,
 Harry

>>>
>>>
>>>
>>> --
>>> Thanks
>>> Deepak
>>> www.bigdatabig.com
>>> www.keosha.net
>>>
>>
>>
>>
>> --
>> Thanks,
>> Harmeet
>>
>
>
>
> --
> Thanks
> Deepak
> www.bigdatabig.com
> www.keosha.net
>



-- 
Thanks,
Harmeet

Re: How to start HDFS on Spark Standalone

[Deepak Sharma]

Binary for Spark means ts spark built against hadoop 2.6
It will not have any hadoop executables.
You'll have to setup hadoop separately.
I have not used windows version yet but there are some.

Thanks
Deepak

On Mon, Apr 18, 2016 at 5:43 PM, My List  wrote:

> Deepak,
>
> The following could be a very dumb questions so pardon me for the same.
> 1) When I download the binary for Spark with a version of Hadoop(Hadoop
> 2.6) does it not come in the zip or tar file?
> 2) If it does not come along,Is there a Apache Hadoop for windows, is it
> in binary format or will have to build it?
> 3) Is there a basic tutorial for Hadoop on windows for the basic needs of
> Spark.
>
> Thanks in Advance !
>
> On Mon, Apr 18, 2016 at 5:35 PM, Deepak Sharma 
> wrote:
>
>> Once you download hadoop and format the namenode , you can use
>> start-dfs.sh to start hdfs.
>> Then use 'jps' to sss if datanode/namenode services are up and running.
>>
>> Thanks
>> Deepak
>>
>> On Mon, Apr 18, 2016 at 5:18 PM, My List  wrote:
>>
>>> Hi ,
>>>
>>> I am a newbie on Spark.I wanted to know how to start and verify if HDFS
>>> has started on Spark stand alone.
>>>
>>> Env -
>>> Windows 7 - 64 bit
>>> Spark 1.4.1 With Hadoop 2.6
>>>
>>> Using Scala Shell - spark-shell
>>>
>>>
>>> --
>>> Thanks,
>>> Harry
>>>
>>
>>
>>
>> --
>> Thanks
>> Deepak
>> www.bigdatabig.com
>> www.keosha.net
>>
>
>
>
> --
> Thanks,
> Harmeet
>



-- 
Thanks
Deepak
www.bigdatabig.com
www.keosha.net

Re: How to start HDFS on Spark Standalone

[My List]

Deepak,

The following could be a very dumb questions so pardon me for the same.
1) When I download the binary for Spark with a version of Hadoop(Hadoop
2.6) does it not come in the zip or tar file?
2) If it does not come along,Is there a Apache Hadoop for windows, is it in
binary format or will have to build it?
3) Is there a basic tutorial for Hadoop on windows for the basic needs of
Spark.

Thanks in Advance !

On Mon, Apr 18, 2016 at 5:35 PM, Deepak Sharma 
wrote:

> Once you download hadoop and format the namenode , you can use
> start-dfs.sh to start hdfs.
> Then use 'jps' to sss if datanode/namenode services are up and running.
>
> Thanks
> Deepak
>
> On Mon, Apr 18, 2016 at 5:18 PM, My List  wrote:
>
>> Hi ,
>>
>> I am a newbie on Spark.I wanted to know how to start and verify if HDFS
>> has started on Spark stand alone.
>>
>> Env -
>> Windows 7 - 64 bit
>> Spark 1.4.1 With Hadoop 2.6
>>
>> Using Scala Shell - spark-shell
>>
>>
>> --
>> Thanks,
>> Harry
>>
>
>
>
> --
> Thanks
> Deepak
> www.bigdatabig.com
> www.keosha.net
>



-- 
Thanks,
Harmeet

Re: How to start HDFS on Spark Standalone

[Deepak Sharma]

Once you download hadoop and format the namenode , you can use start-dfs.sh
to start hdfs.
Then use 'jps' to sss if datanode/namenode services are up and running.

Thanks
Deepak

On Mon, Apr 18, 2016 at 5:18 PM, My List  wrote:

> Hi ,
>
> I am a newbie on Spark.I wanted to know how to start and verify if HDFS
> has started on Spark stand alone.
>
> Env -
> Windows 7 - 64 bit
> Spark 1.4.1 With Hadoop 2.6
>
> Using Scala Shell - spark-shell
>
>
> --
> Thanks,
> Harry
>



-- 
Thanks
Deepak
www.bigdatabig.com
www.keosha.net