Re: [DISCUSS] Pre-commit build in Windows platform

[Allen Wittenauer]

> On Nov 1, 2016, at 4:00 AM, Brahma Reddy Battula 
>  wrote:

> Thanks for information. Seems to be challenge to get it done. Can we try to 
> get dedicated machine for this..?

We have one.  What we don't have is someone dedicated enough to keep it 
running.  It is not a "one and done". Folks will add new or change dependencies 
and not bother to update or even to tell anyone to update the build servers. 
Even though the Linux/x86 builds have the build setup as part of the source 
tree (dev-support/docker/Dockerfile), people *still* ignore it then complain 
that their new feature isn't getting tested. 

FWIW, there are still bits broken even on Linux/x86.  MAPREDUCE-6743 is 
a great view of the apathetic nature of the committer pool that anyone who 
takes this on will have to work around.   Those native MR tasks tests have been 
known to be broken since qbt was introduced several months ago. (They weren't 
testing in the nightly builds before, so have likely been broken for a very 
long time. qbt just made them visible.) Frankly, Windows builds have zero 
chance if we can't even get the Linux/x86 build stable.
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

RE: [DISCUSS] Pre-commit build in Windows platform

[Brahma Reddy Battula]

Thanks a lot Steve Loughran , Allen Wittenauer ,Ravi Prakash and Mingliang Liu.


[ Steve Loughran Wrote ]
I think a first step would be to have scheduled builds on windows; ideally with 
people caring that they are playing up.

Anyway,

+1 for more windows testing


[ Brahma Reddy ]
May be we add in qbt it self..? Or separate night pre-commit build..?

[ Allen Wittenauer Wrote ]
I actually tried to get Apache Yetus testing Apache Hadoop on the 
hadoop-win box last year.  (This was before qbt mode existed.)  I gave up 
because the components needed to build trunk weren't installed and it looked 
like the box itself was ill. I moved on to the Mac build which, while tricky, 
got it working with a bit of magic.  Then they took the Mac away.  IBM provided 
access to a PowerPC machine, so I moved onto the ppc64le build. After a lot of 
hand wringing, we got it up and running and stable enough to show how broken 
the build is by our usage of leveldbjni.

It should be noted that getting infra support for non-Linux/x86 builds 
is a pretty major exercise in frustration.  They don't really have the time, 
those boxes tend to go down often, and rarely get updated. (The Solaris box is 
running a really old build of Solaris 10.  So old, that when it came out, I was 
still employed by Sun...)  If folks are actually serious about adding more 
platforms, then I'd suggest that some of these big vendors actually cough up 
some cash to move the precommit infrastructure off of the ASF and onto 
something more reliable with more platform diversity.

[ Brahma Reddy ]
Thanks for information. Seems to be challenge to get it done. Can we try to get 
dedicated machine for this..?

[ Mingliang Liu wrote ]
I also like the idea of having a precommit build in Windows. If only the 
pre-commit infrastructure was reliable. If this is a pain, I see little value 
making Windows unit test failures release blockers.

[ Brahma Reddy ]
Yes, I think we can do like this, if nobody object on this.



--Brahma Reddy Battula

-Original Message-
From: Mingliang Liu [mailto:lium...@gmail.com] 
Sent: 31 October 2016 03:49
To: common-dev@hadoop.apache.org
Subject: Re: [DISCUSS] Pre-commit build in Windows platform

I also like the idea of having a precommit build in Windows. If only the 
pre-commit infrastructure was reliable. If this is a pain, I see little value 
making Windows unit test failures release blockers.

Thanks,

Sent from my iPhone

> On Oct 28, 2016, at 8:16 AM, Allen Wittenauer  
> wrote:
> 
> 
>> On Oct 27, 2016, at 8:20 PM, Brahma Reddy Battula 
>>  wrote:
>> 
>> As we supporting the Hadoop in windows, I feel, we should have pre-commit 
>> build in windows( atleast in qbt).
> 
> 
>I actually tried to get Apache Yetus testing Apache Hadoop on the 
> hadoop-win box last year.  (This was before qbt mode existed.)  I gave up 
> because the components needed to build trunk weren't installed and it looked 
> like the box itself was ill. I moved on to the Mac build which, while tricky, 
> got it working with a bit of magic.  Then they took the Mac away.  IBM 
> provided access to a PowerPC machine, so I moved onto the ppc64le build. 
> After a lot of hand wringing, we got it up and running and stable enough to 
> show how broken the build is by our usage of leveldbjni.
> 
>It should be noted that getting infra support for non-Linux/x86 builds is 
> a pretty major exercise in frustration.  They don't really have the time, 
> those boxes tend to go down often, and rarely get updated. (The Solaris box 
> is running a really old build of Solaris 10.  So old, that when it came out, 
> I was still employed by Sun...)  If folks are actually serious about adding 
> more platforms, then I'd suggest that some of these big vendors actually 
> cough up some cash to move the precommit infrastructure off of the ASF and 
> onto something more reliable with more platform diversity. 
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> 

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [DISCUSS] Pre-commit build in Windows platform

[Mingliang Liu]

I also like the idea of having a precommit build in Windows. If only the 
pre-commit infrastructure was reliable. If this is a pain, I see little value 
making Windows unit test failures release blockers.

Thanks,

Sent from my iPhone

> On Oct 28, 2016, at 8:16 AM, Allen Wittenauer  
> wrote:
> 
> 
>> On Oct 27, 2016, at 8:20 PM, Brahma Reddy Battula 
>>  wrote:
>> 
>> As we supporting the Hadoop in windows, I feel, we should have pre-commit 
>> build in windows( atleast in qbt).
> 
> 
>I actually tried to get Apache Yetus testing Apache Hadoop on the 
> hadoop-win box last year.  (This was before qbt mode existed.)  I gave up 
> because the components needed to build trunk weren't installed and it looked 
> like the box itself was ill. I moved on to the Mac build which, while tricky, 
> got it working with a bit of magic.  Then they took the Mac away.  IBM 
> provided access to a PowerPC machine, so I moved onto the ppc64le build. 
> After a lot of hand wringing, we got it up and running and stable enough to 
> show how broken the build is by our usage of leveldbjni.
> 
>It should be noted that getting infra support for non-Linux/x86 builds is 
> a pretty major exercise in frustration.  They don't really have the time, 
> those boxes tend to go down often, and rarely get updated. (The Solaris box 
> is running a really old build of Solaris 10.  So old, that when it came out, 
> I was still employed by Sun...)  If folks are actually serious about adding 
> more platforms, then I'd suggest that some of these big vendors actually 
> cough up some cash to move the precommit infrastructure off of the ASF and 
> onto something more reliable with more platform diversity. 
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> 

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [DISCUSS] Pre-commit build in Windows platform

[Ravi Prakash]

Was anybody ever successful in getting Windows(tm) installed in a VM from
the developer accounts that Microsoft affords Apache?

On Fri, Oct 28, 2016 at 4:44 AM, Steve Loughran 
wrote:

>
> On 28 Oct 2016, at 04:20, Brahma Reddy Battula <
> brahmareddy.batt...@huawei.com>
> wrote:
>
> Hi All
>
> As we supporting the Hadoop in windows, I feel, we should have pre-commit
> build in windows( atleast in qbt).
>
> We've never had it work too well with Windows precommit & Jenkins; maybe
> someone could volunteer to try again. What's important is not to overwhelm
> the JIRAs with noise that gets ignored.
>
>
>
> Background:
>
> As of now pre-commit will not run on windows, we may end up with following
>
>
> 1)  Test cases  can fail in windows
>
> 2)  Compilation Error (now it's failing, after HADOOP-10075)
>
>
> I've not had branch-3 building for a while, though maybe it's my Windows
> Server VM that's a mess. Maybe I should rm it and grab the limited-lifespan
> dev box, then add the extra Hadoop bits https://developer.microsoft.
> com/en-us/windows/downloads/virtual-machines
>
> 3)  Some feature might not work
>
> It will be good, if we catch all these at time of commit. (Prevention is
> better than cure..:) )
>
> I think a first step would be to have scheduled builds on windows; ideally
> with people caring that they are playing up.
>
> Anyway,
>
> +1 for more windows testing
>

Re: [DISCUSS] Pre-commit build in Windows platform

[Allen Wittenauer]

> On Oct 27, 2016, at 8:20 PM, Brahma Reddy Battula 
>  wrote:
> 
> As we supporting the Hadoop in windows, I feel, we should have pre-commit 
> build in windows( atleast in qbt).


I actually tried to get Apache Yetus testing Apache Hadoop on the 
hadoop-win box last year.  (This was before qbt mode existed.)  I gave up 
because the components needed to build trunk weren't installed and it looked 
like the box itself was ill. I moved on to the Mac build which, while tricky, 
got it working with a bit of magic.  Then they took the Mac away.  IBM provided 
access to a PowerPC machine, so I moved onto the ppc64le build. After a lot of 
hand wringing, we got it up and running and stable enough to show how broken 
the build is by our usage of leveldbjni.

It should be noted that getting infra support for non-Linux/x86 builds 
is a pretty major exercise in frustration.  They don't really have the time, 
those boxes tend to go down often, and rarely get updated. (The Solaris box is 
running a really old build of Solaris 10.  So old, that when it came out, I was 
still employed by Sun...)  If folks are actually serious about adding more 
platforms, then I'd suggest that some of these big vendors actually cough up 
some cash to move the precommit infrastructure off of the ASF and onto 
something more reliable with more platform diversity. 
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [DISCUSS] Pre-commit build in Windows platform

[Steve Loughran]

On 28 Oct 2016, at 04:20, Brahma Reddy Battula 
mailto:brahmareddy.batt...@huawei.com>> wrote:

Hi All

As we supporting the Hadoop in windows, I feel, we should have pre-commit build 
in windows( atleast in qbt).

We've never had it work too well with Windows precommit & Jenkins; maybe 
someone could volunteer to try again. What's important is not to overwhelm the 
JIRAs with noise that gets ignored.



Background:

As of now pre-commit will not run on windows, we may end up with following


1)  Test cases  can fail in windows

2)  Compilation Error (now it's failing, after HADOOP-10075)


I've not had branch-3 building for a while, though maybe it's my Windows Server 
VM that's a mess. Maybe I should rm it and grab the limited-lifespan dev box, 
then add the extra Hadoop bits 
https://developer.microsoft.com/en-us/windows/downloads/virtual-machines

3)  Some feature might not work

It will be good, if we catch all these at time of commit. (Prevention is better 
than cure..:) )

I think a first step would be to have scheduled builds on windows; ideally with 
people caring that they are playing up.

Anyway,

+1 for more windows testing

[DISCUSS] Pre-commit build in Windows platform

[Brahma Reddy Battula]

Hi All

As we supporting the Hadoop in windows, I feel, we should have pre-commit build 
in windows( atleast in qbt).

Background:

As of now pre-commit will not run on windows, we may end up with following


1)  Test cases  can fail in windows

2)  Compilation Error (now it's failing, after HADOOP-10075)

3)  Some feature might not work

It will be good, if we catch all these at time of commit. (Prevention is better 
than cure..:) )


If it's already discussed and concluded, Please ignore me and let us know the 
conclusion.



--Brahma Reddy Battula