Re: N.America apachcon & bigtop meetup

[Antonio Rosales]

Myself and/or Kevin are working on attendance. Be great to have a BigTop meetup.

-Antonio

On Mon, Mar 27, 2017 at 7:47 PM, Jonathan Kelly  wrote:
> I will not be there, but I'm pretty sure one or more of my teammates from
> EMR will be there. I'll let you know when I find out.
>
> ~ Jonathan
> On Mon, Mar 27, 2017 at 5:52 PM MrAsanjar .  wrote:
>
>> is there any plans for Bigtop meetups during NA ApacheCon conference in
>> Miami?
>>



-- 
Antonio Rosales
Ecosystem Engineering
Canonical

[jira] [Created] (BIGTOP-2724) putting a final nail in Java's 7 coffin

[Roman Shaposhnik (JIRA)]

Roman Shaposhnik created BIGTOP-2724:


 Summary: putting a final nail in Java's 7 coffin
 Key: BIGTOP-2724
 URL: https://issues.apache.org/jira/browse/BIGTOP-2724
 Project: Bigtop
  Issue Type: Bug
Affects Versions: 1.1.0
Reporter: Roman Shaposhnik
Assignee: Roman Shaposhnik
 Fix For: 1.2.0


As Olaf is saying -- we're done with Java 7. Really done.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Releasing 1.2 for non-x86 architectures

[Roman Shaposhnik]

On Mon, Mar 27, 2017 at 5:49 PM, MrAsanjar .  wrote:
> +1 on Roman proposal..
> I ran multiple small smoke tests with x86 zeppelin, spark 1.6.x and tez on
> ppc64le and they looked fine. Hadoop however failed when native libraries
> were included, as expected.
> I would do more extensive testing when repo becomes available..

Great! Thanks for the testing. Btw, the following now has the non-x86 repos
laid out according to the proposal:
https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-repos/

It would be great if ARM folks can test as well.

Thanks,
Roman.

[GitHub] bigtop issue #175: BIGTOP-2675. [Puppet] Kafka can only be deployed on maste...

[rvs]

Github user rvs commented on the issue:

https://github.com/apache/bigtop/pull/175
  
LGTM! Please commit ASAP!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] bigtop pull request #191: BIGTOP-2672. Update gradle wrapper for Docker Prov...

[evans-ye]

GitHub user evans-ye opened a pull request:

https://github.com/apache/bigtop/pull/191

BIGTOP-2672. Update gradle wrapper for Docker Provisioner



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/evans-ye/bigtop BIGTOP-2672

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/bigtop/pull/191.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #191


commit 381418a48f264d414497d359710de026a6d96988
Author: Evans Ye 
Date:   2017-03-28T03:08:18Z

BIGTOP-2672. Update gradle wrapper for Docker Provisioner




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: N.America apachcon & bigtop meetup

[Jonathan Kelly]

I will not be there, but I'm pretty sure one or more of my teammates from
EMR will be there. I'll let you know when I find out.

~ Jonathan
On Mon, Mar 27, 2017 at 5:52 PM MrAsanjar .  wrote:

> is there any plans for Bigtop meetups during NA ApacheCon conference in
> Miami?
>

N.America apachcon & bigtop meetup

[MrAsanjar .]

is there any plans for Bigtop meetups during NA ApacheCon conference in
Miami?

Re: Releasing 1.2 for non-x86 architectures

[MrAsanjar .]

+1 on Roman proposal..
I ran multiple small smoke tests with x86 zeppelin, spark 1.6.x and tez on
ppc64le and they looked fine. Hadoop however failed when native libraries
were included, as expected.
I would do more extensive testing when repo becomes available..

On Mon, Mar 27, 2017 at 12:12 PM, Roman Shaposhnik 
wrote:

> On Mon, Mar 27, 2017 at 10:05 AM, MrAsanjar .  wrote:
> > we need to be careful here, some components have build dependencies to
> > other native components. Casing point, Zeppelin, Tez, and Amabri have
> > dependencies to frontend-maven-plugin that pulls in native NodeJS binary
> > based on the build system.
>
> Those still are built-time dependencies. If you know anything that makes
> a difference during the runtime -- please let us know.
>
> Thanks,
> Roman.
>

[jira] [Created] (BIGTOP-2723) Fix asciidoctor-maven-plugin for HBase build on AArch64

[Ganesh Raju (JIRA)]

Ganesh Raju created BIGTOP-2723:
---

 Summary: Fix asciidoctor-maven-plugin for HBase build on AArch64
 Key: BIGTOP-2723
 URL: https://issues.apache.org/jira/browse/BIGTOP-2723
 Project: Bigtop
  Issue Type: Bug
Reporter: Ganesh Raju


HBase fails building on AArch64 due to asciidoctor-maven-plugin



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] bigtop issue #190: BIGTOP-2722. Remove workaround allowing build by root in ...

[ejono]

Github user ejono commented on the issue:

https://github.com/apache/bigtop/pull/190
  
LGTM, thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: what happened to ppc64le build?

[Evans Ye]

My bad. I should have you noticed.
I created a new job to build all packages by jenkins instead of root.
And then I have the old job deleted to prevent confusion.

2017-03-28 1:33 GMT+08:00 MrAsanjar . :

> thanks Roman
>
> On Mon, Mar 27, 2017 at 12:18 PM, Roman Shaposhnik 
> wrote:
>
> > We're using this URL now:
> >https://ci.bigtop.apache.org/view/Packages/job/Bigtop-
> > trunk-packages-by-jenkins/
> >
> > On Mon, Mar 27, 2017 at 10:08 AM, MrAsanjar . 
> wrote:
> > > HI team,
> > > I just realized the path to Bigtop builds for ppc64le is no longer
> valid.
> > > https://ci.bigtop.apache.org/job/Bigtop-trunk-packages
> >
>

[GitHub] bigtop pull request #190: BIGTOP-2722. Remove workaround allowing build by r...

[evans-ye]

GitHub user evans-ye opened a pull request:

https://github.com/apache/bigtop/pull/190

BIGTOP-2722. Remove workaround allowing build by root in Tez



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/evans-ye/bigtop BIGTOP-2722

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/bigtop/pull/190.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #190


commit 9f41a8f5c3cd991dbce192ad83fca4d15a77eb55
Author: Evans Ye 
Date:   2017-03-27T19:08:14Z

BIGTOP-2722. Remove workaround allowing build by root in Tez




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: what happened to ppc64le build?

[MrAsanjar .]

thanks Roman

On Mon, Mar 27, 2017 at 12:18 PM, Roman Shaposhnik 
wrote:

> We're using this URL now:
>https://ci.bigtop.apache.org/view/Packages/job/Bigtop-
> trunk-packages-by-jenkins/
>
> On Mon, Mar 27, 2017 at 10:08 AM, MrAsanjar .  wrote:
> > HI team,
> > I just realized the path to Bigtop builds for ppc64le is no longer valid.
> > https://ci.bigtop.apache.org/job/Bigtop-trunk-packages
>

Re: what happened to ppc64le build?

[Roman Shaposhnik]

We're using this URL now:
   
https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages-by-jenkins/

On Mon, Mar 27, 2017 at 10:08 AM, MrAsanjar .  wrote:
> HI team,
> I just realized the path to Bigtop builds for ppc64le is no longer valid.
> https://ci.bigtop.apache.org/job/Bigtop-trunk-packages

Re: Releasing 1.2 for non-x86 architectures

[Roman Shaposhnik]

On Mon, Mar 27, 2017 at 10:05 AM, MrAsanjar .  wrote:
> we need to be careful here, some components have build dependencies to
> other native components. Casing point, Zeppelin, Tez, and Amabri have
> dependencies to frontend-maven-plugin that pulls in native NodeJS binary
> based on the build system.

Those still are built-time dependencies. If you know anything that makes
a difference during the runtime -- please let us know.

Thanks,
Roman.

what happened to ppc64le build?

[MrAsanjar .]

HI team,
I just realized the path to Bigtop builds for ppc64le is no longer valid.
https://ci.bigtop.apache.org/job/Bigtop-trunk-packages

Re: Releasing 1.2 for non-x86 architectures

[MrAsanjar .]

we need to be careful here, some components have build dependencies to
other native components. Casing point, Zeppelin, Tez, and Amabri have
dependencies to frontend-maven-plugin that pulls in native NodeJS binary
based on the build system.

On Sat, Mar 25, 2017 at 2:04 PM, Roman Shaposhnik  wrote:

> Those workaround only matter during build time. Once the package is
> built it is truly
> arch independent. IOW, we can build on x86 and use it anywhere.
>
> This is a bit confusing since node itself is NOT arch independent.
> However, none of
> these packages actually need node -- they all use it to
> preprocess/bundle JS during
> compile time.
>
> Makes sense?
>
> Thanks,
> Roman.
>
> On Sat, Mar 25, 2017 at 6:26 AM, Olaf Flebbe  wrote:
> > Hi Roman,
> >
> > BTW: There are _arch dependent_ package workarounds in _arch independent_
> > packages ! I think we should mark the package as arch dependent in order
> to
> > make sure.
> >
> > The one which comes to mind is tez:
> >
> > See src/common/tez/do-component-build ;-(
> >
> > Will file a JIRA.
> >
> > Olaf
> >
> >
> > Am 25.03.2017 um 13:57 schrieb Olaf Flebbe :
> >
> > Hi,
> >
> > On the debian side I was assuming that reprepro will handle this for you.
> > Simply add different arch specific packages (which can be found by name).
> >
> > I am not sure about the yum side of things.
> >
> >
> > Olaf
> >
> >
> > Am 24.03.2017 um 01:42 schrieb Roman Shaposhnik :
> >
> > Hi!
> >
> > recent issues surrounding builds on two of
> > our two non-x86 platforms got me thinking
> > that while it is fun to build on them (to iron out
> > kinds in JDK and what not) we don't actually
> > have to release the packages for most of the
> > projects since all those are arch independent.
> > The only ones that ARE arch dependent would be:
> >  bigtop-jsvc
> >  gpdb
> >  hadoop
> >  hue
> >  qfs
> >  tajo
> >  zookeeper
> >
> > The question then becomes how do you structure
> > the repos. Do you have on repo for arch independent
> > stuff and one for each arch? Can you have all the
> > packages in a single repo?
> >
> > What are the good practices around this?
> >
> > I'm especially looking for advice from Canonical guys
> > and Linaro folks.
> >
> > Thanks,
> > Roman.
> >
> >
> >
>

[GitHub] bigtop pull request #188: BIGTOP-2706 Apex package does not run successfully...

[tweise]

Github user tweise commented on a diff in the pull request:

https://github.com/apache/bigtop/pull/188#discussion_r108193720
  
--- Diff: bigtop-packages/src/common/apex/do-component-build ---
@@ -26,10 +26,10 @@ command -v mvn > /dev/null 2>&1 || {
 }
 
 # Build apex
-mvn clean package -DskipTests "$@"
+mvn clean package -DskipTests -Dhadoop.version=$HADOOP_VERSION "$@"
 
 # Find all the dependencies of apex
-mvn dependency:copy-dependencies -DoutputDirectory=`pwd`/all-deps 
-DincludeScope=runtime -DexcludeScope=test
+mvn dependency:copy-dependencies -DoutputDirectory=`pwd`/all-deps 
-DincludeScope=runtime -DexcludeScope=test -Dhadoop.version=$HADOOP_VERSION
--- End diff --

@chinmaykolhatkar agreed, that should be taken up separately.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BIGTOP-2721) libsnappy reference path is not correct for hadoop/ubuntu16 build

[Jun He (JIRA)]

Jun He created BIGTOP-2721:
--

 Summary: libsnappy reference path is not correct for 
hadoop/ubuntu16 build
 Key: BIGTOP-2721
 URL: https://issues.apache.org/jira/browse/BIGTOP-2721
 Project: Bigtop
  Issue Type: Bug
  Components: build, ci
Affects Versions: 1.1.0
 Environment: Ubuntu 16.04
Reporter: Jun He
Assignee: Jun He


When libsnappy-dev is installed on Ubuntu 16.04, the destination is 
/usr/lib/${HOSTTYPE}-linux-gnu. This path is not configured in hadoop's 
do-component-build. So the hadoop built out on Ubuntu 16.04 docker will not 
include snappy support.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] bigtop pull request #188: BIGTOP-2706 Apex package does not run successfully...

[chinmaykolhatkar]

Github user chinmaykolhatkar commented on a diff in the pull request:

https://github.com/apache/bigtop/pull/188#discussion_r108097463
  
--- Diff: bigtop-packages/src/common/apex/do-component-build ---
@@ -26,10 +26,10 @@ command -v mvn > /dev/null 2>&1 || {
 }
 
 # Build apex
-mvn clean package -DskipTests "$@"
+mvn clean package -DskipTests -Dhadoop.version=$HADOOP_VERSION "$@"
 
 # Find all the dependencies of apex
-mvn dependency:copy-dependencies -DoutputDirectory=`pwd`/all-deps 
-DincludeScope=runtime -DexcludeScope=test
+mvn dependency:copy-dependencies -DoutputDirectory=`pwd`/all-deps 
-DincludeScope=runtime -DexcludeScope=test -Dhadoop.version=$HADOOP_VERSION
--- End diff --

Thanks @rvs  Fastest way is to check if the deb/rpm package that there is 
no hadoop jar present. I'm writing package tests after for this for another 
jira (BIGTOP-2375).. That would make it even easier.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] bigtop pull request #188: BIGTOP-2706 Apex package does not run successfully...

[rvs]

Github user rvs commented on a diff in the pull request:

https://github.com/apache/bigtop/pull/188#discussion_r108095450
  
--- Diff: bigtop-packages/src/common/apex/do-component-build ---
@@ -26,10 +26,10 @@ command -v mvn > /dev/null 2>&1 || {
 }
 
 # Build apex
-mvn clean package -DskipTests "$@"
+mvn clean package -DskipTests -Dhadoop.version=$HADOOP_VERSION "$@"
 
 # Find all the dependencies of apex
-mvn dependency:copy-dependencies -DoutputDirectory=`pwd`/all-deps 
-DincludeScope=runtime -DexcludeScope=test
+mvn dependency:copy-dependencies -DoutputDirectory=`pwd`/all-deps 
-DincludeScope=runtime -DexcludeScope=test -Dhadoop.version=$HADOOP_VERSION
--- End diff --

@chinmaykolhatkar I've committed your change. I would like to make sure 
that we fixed the problem. What's the easiest way to run a test for this change?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---