from:"Konstantin Boudnik"


Looks like some sort of trouble with one of the dependencies of httpfs:


Downloading from apache.snapshots.https: 
https://repository.apache.org/content/repositories/snapshots/org/apache/maven/skins/maven-default-skin/1.0/maven-default-skin-1.0.jar 

Downloading from repository.jboss.org: 
http://repository.jboss.org/nexus/content/groups/public/org/apache/maven/skins/maven-default-skin/1.0/maven-default-skin-1.0.jar 

[WARNING] Checksum validation failed, expected  but is 
da39a3ee5e6b4b0d3255bfef95601890afd80709 from repository.jboss.
org for 
http://repository.jboss.org/nexus/content/groups/public/org/apache/maven/skins/maven-default-skin/1.0/maven-default-skin-1.0.jar 

[WARNING] Could not validate integrity of download from 
http://repository.jboss.org/nexus/content/groups/public/org/apache/maven/skins/maven-default-skin/1.0/maven-default-skin-1.0.jar: 
Checksum validation failed, expected  but is 
da39a3ee5e6b4b0d3255bfef95601890afd80709
[WARNING] Checksum validation failed, expected  but is 
da39a3ee5e6b4b0d3255bfef95601890afd80709 from repository.jboss.org for 
http://repository.jboss.org/nexus/content/groups/public/org/apache/maven/skins/maven-default-skin/1.0/maven-default-skin-1.0.jar 

Downloaded from repository.jboss.org: 
http://repository.jboss.org/nexus/content/groups/public/org/apache/maven/skins/maven-default-skin/1.0/maven-default-skin-1.0.jar 
(0 B at 0 B/s)


And then build fails right at that module. Doesn't look like a Bigtop 
issue, methink. I am on the road right now with somewhat limited 
internet, will be back over the weekend and try to look more. Perhaps in 
the meanwhile someone on the list would have any other ideas.


--
  Cos

On 1/23/20 11:40 PM, Konstantin Boudnik wrote:
Well, I am not familiar with Gradle enterprise - looks like it has 
everything indeed, thanks! Let me dig and see what I can figure out.


--
  Cos

On 1/23/20 11:28 PM, Alexander Frolushkin wrote:

Thank you for responding, Konstantin.

Honestly, I was thinking --scan gradle will publish all necessary
information about its execution, which may be required to 
inverstigate the

problem.

Anyway, I'm ready to provide maximum data for this dicsussion. Firstly,
here is a complete console output of entire hadoop-pkg-ind task till the
error -
https://drive.google.com/file/d/16zmFTj485RQ44BFMh5zlhB4cyUzmyaNR/view?usp=drivesdk 



If I need to collect additional data, please direct me, what exactly 
I need

to do to get it for processing here.

I really want to build hadoop repo by bigtop-1.4.0.

On Thu, Jan 23, 2020, 22:07 Konstantin Boudnik  wrote:


Thanks, Alexander, for bringing this up.

I don't see anything wrong with the way you're starting the build,
except that official CI is now using "hadoop-pkg-ind" target and build
then pulls in a correct container automatically. But I see you already
tried that one.

As a side note, I am looking at [1] and see the following error (which
seems to be there for some time):

Plugin [id: 'de.undercouch.download', version: '3.2.0'] was not 
found in

any of the following sources:

- Gradle Core Plugins (plugin is not in 'org.gradle' namespace)
- Plugin Repositories (could not resolve plugin artifact
'de.undercouch.download:de.undercouch.download.gradle.plugin:3.2.0')
    Searched in the following repositories:
  Gradle Central Plugin Repository

Unfortunately, it is hard to tell what kind of error you're getting in
your build. Any chance you can publish a snippet of the actual logs,
rather than a screenshot?

--
Thanks,
    Cos

[1]

https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/COMPONENTS=hadoop,OS=centos-7/567/console 



On 1/23/20 4:50 AM, Alexander Frolushkin wrote:

Hello everyone!

Sorry for repeating this in dev list, but I really need help in this
situation.

On bigtop-1.4.0 build fails on hadoop-hdfs-httpfs with error:

Failed to execute goal

org.apache.maven.plugins:maven-project-info-reports-plugin:2.7:dependencies 


(default) on project hadoop-hdfs-httpfs: An error has occurred in
Dependencies report generation.: zip file is empty -> [Help 1]

Launched in clean bigtop-1.4.0 dir, with command docker run -v 
`pwd`:/ws

bigtop/slaves:1.4.0-centos-7 bash -l -c 'cd /ws;gradle hadoop-pkg

I made a --scan for this attempt - https://gradle.com/s/7iu4z3fsbrh62

Maybe I'm using not a valid way to get it perform correctly?

Or this can be bug and I need to fill a jira ticket for this?

Also I tryed to use direct building by ./gradlew hadoop-rpm and via

docker
./gradlew hadoop-pkg-ind, got the same error. If I use 
bigtop-trunk, this
stage (hadoop-hdfs-httpfs) passed without error, but as I've been 
told,

currently hadoop is unbuildable in master branch.

Plus, the same error I get on bigtop-1.3.0.

Cleaning the local maven repository does not fix it.

So, likely, the problem is in building environment. How I can 
configure

it

correctly for successful hadoop building in bigtop?

On Wed, Jan 22, 2020, 16:20 Evans Ye  wrote

Re: Bigtop error on Centos 7

Well, I am not familiar with Gradle enterprise - looks like it has
everything indeed, thanks! Let me dig and see what I can figure out.

--
Cos

On 1/23/20 11:28 PM, Alexander Frolushkin wrote:

Thank you for responding, Konstantin.

Honestly, I was thinking --scan gradle will publish all necessary
information about its execution, which may be required to inverstigate the
problem.

Anyway, I'm ready to provide maximum data for this dicsussion. Firstly,
here is a complete console output of entire hadoop-pkg-ind task till the
error -
https://drive.google.com/file/d/16zmFTj485RQ44BFMh5zlhB4cyUzmyaNR/view?usp=drivesdk

If I need to collect additional data, please direct me, what exactly I need
to do to get it for processing here.

I really want to build hadoop repo by bigtop-1.4.0.

On Thu, Jan 23, 2020, 22:07 Konstantin Boudnik wrote:

Thanks, Alexander, for bringing this up.

I don't see anything wrong with the way you're starting the build,
except that official CI is now using "hadoop-pkg-ind" target and build
then pulls in a correct container automatically. But I see you already
tried that one.

As a side note, I am looking at [1] and see the following error (which
seems to be there for some time):

Plugin [id: 'de.undercouch.download', version: '3.2.0'] was not found in
any of the following sources:

- Gradle Core Plugins (plugin is not in 'org.gradle' namespace)
- Plugin Repositories (could not resolve plugin artifact
'de.undercouch.download:de.undercouch.download.gradle.plugin:3.2.0')
Searched in the following repositories:
Gradle Central Plugin Repository

Unfortunately, it is hard to tell what kind of error you're getting in
your build. Any chance you can publish a snippet of the actual logs,
rather than a screenshot?

--
Thanks,
Cos

[1]

https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/COMPONENTS=hadoop,OS=centos-7/567/console

On 1/23/20 4:50 AM, Alexander Frolushkin wrote:

Hello everyone!

Sorry for repeating this in dev list, but I really need help in this
situation.

On bigtop-1.4.0 build fails on hadoop-hdfs-httpfs with error:

Failed to execute goal

org.apache.maven.plugins:maven-project-info-reports-plugin:2.7:dependencies

(default) on project hadoop-hdfs-httpfs: An error has occurred in
Dependencies report generation.: zip file is empty -> [Help 1]

Launched in clean bigtop-1.4.0 dir, with command docker run -v `pwd`:/ws
bigtop/slaves:1.4.0-centos-7 bash -l -c 'cd /ws;gradle hadoop-pkg

I made a --scan for this attempt - https://gradle.com/s/7iu4z3fsbrh62

Maybe I'm using not a valid way to get it perform correctly?

Or this can be bug and I need to fill a jira ticket for this?

Also I tryed to use direct building by ./gradlew hadoop-rpm and via

docker

./gradlew hadoop-pkg-ind, got the same error. If I use bigtop-trunk, this
stage (hadoop-hdfs-httpfs) passed without error, but as I've been told,
currently hadoop is unbuildable in master branch.

Plus, the same error I get on bigtop-1.3.0.

Cleaning the local maven repository does not fix it.

So, likely, the problem is in building environment. How I can configure

correctly for successful hadoop building in bigtop?

On Wed, Jan 22, 2020, 16:20 Evans Ye wrote:

Currently building Hadoop across all platform on master branch is

failing:

https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/

I'm not sure whether the master code can be built successfully since

we're

in transition of upgrading zookeeper.

If you just want to use bigtop, please get our latest release 1.4.0.

If you are willing to contribute, you can report a bug on our jira first
and we ca work on a fix together.

Let us know if any question.

Evan's

Yuqi Gu 於 2020年1月22日 週三 16:31 寫道：

Hi Alexander,

It seems bigtop_toolchain didn't deploy correctly.
Please try to build bigtop components in docker with bigtop/slaves

docker

images <https://hub.docker.com/r/bigtop/slaves/tags?page=1=centos

Run:
cd BIGTOP_SRC_DIR
docker run -v `pwd`:/ws bigtop/slaves:1.4.0-centos-7 bash -l -c 'cd

/ws ;

gradle hadoop-pkg'

BR,
Yuqi

On Wed, 22 Jan 2020 at 15:05, Alexander Frolushkin
I have this error on bigtop-trunk. In webtop-1.4.0 it fails with other
error.

Also, in webtop-trunk this particular error appeard while building

hadoop

in docker (trunk-centos-7) image (by ./gradlew hadoop-pkg-ind) and

without

it (by ./gradlew hadoop-rpm).

Performed steps:

I was trying to follow the available guides, do a clean centos 7

install,

upgrade to latest official rpms, install openjdk 7 and unzip from

repo.

Then puppetize.sh from bigtop_toolchain/bin and puppet apply
--modulepath=`pwd`:/etc/puppet/modules -e "include
bigtop_toolchain::installer".

After this run ./gradlew hadoop-pkg-ind or ./gradlew hadoop-rpm and

got

this error.

On Wed, Jan 22, 2020, 12:56 Youngwoo Kim (김영우)

wrote:

Hi Alexander,

Which version or branch of Bigtop do you use? and also can you

provide

the steps

Re: Bigtop error on Centos 7

Thanks, Alexander, for bringing this up.

As a side note, I am looking at [1] and see the following error (which
seems to be there for some time):

Plugin [id: 'de.undercouch.download', version: '3.2.0'] was not found in
any of the following sources:

- Gradle Core Plugins (plugin is not in 'org.gradle' namespace)
- Plugin Repositories (could not resolve plugin artifact
'de.undercouch.download:de.undercouch.download.gradle.plugin:3.2.0')

Searched in the following repositories:
Gradle Central Plugin Repository

Unfortunately, it is hard to tell what kind of error you're getting in
your build. Any chance you can publish a snippet of the actual logs,
rather than a screenshot?

--
Thanks,
Cos

[1]
https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/COMPONENTS=hadoop,OS=centos-7/567/console

On 1/23/20 4:50 AM, Alexander Frolushkin wrote:

Hello everyone!

Sorry for repeating this in dev list, but I really need help in this
situation.

On bigtop-1.4.0 build fails on hadoop-hdfs-httpfs with error:

Failed to execute goal
org.apache.maven.plugins:maven-project-info-reports-plugin:2.7:dependencies
(default) on project hadoop-hdfs-httpfs: An error has occurred in
Dependencies report generation.: zip file is empty -> [Help 1]

Launched in clean bigtop-1.4.0 dir, with command docker run -v `pwd`:/ws
bigtop/slaves:1.4.0-centos-7 bash -l -c 'cd /ws;gradle hadoop-pkg

I made a --scan for this attempt - https://gradle.com/s/7iu4z3fsbrh62

Maybe I'm using not a valid way to get it perform correctly?

Or this can be bug and I need to fill a jira ticket for this?

Also I tryed to use direct building by ./gradlew hadoop-rpm and via docker
./gradlew hadoop-pkg-ind, got the same error. If I use bigtop-trunk, this
stage (hadoop-hdfs-httpfs) passed without error, but as I've been told,
currently hadoop is unbuildable in master branch.

Plus, the same error I get on bigtop-1.3.0.

Cleaning the local maven repository does not fix it.

So, likely, the problem is in building environment. How I can configure it
correctly for successful hadoop building in bigtop?

On Wed, Jan 22, 2020, 16:20 Evans Ye wrote:

Currently building Hadoop across all platform on master branch is failing:
https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/

I'm not sure whether the master code can be built successfully since we're
in transition of upgrading zookeeper.

If you just want to use bigtop, please get our latest release 1.4.0.

If you are willing to contribute, you can report a bug on our jira first
and we ca work on a fix together.

Let us know if any question.

Evan's

Yuqi Gu 於 2020年1月22日 週三 16:31 寫道：

Hi Alexander,

It seems bigtop_toolchain didn't deploy correctly.
Please try to build bigtop components in docker with bigtop/slaves docker
images .

Run:
cd BIGTOP_SRC_DIR
docker run -v `pwd`:/ws bigtop/slaves:1.4.0-centos-7 bash -l -c 'cd /ws ;
gradle hadoop-pkg'

BR,
Yuqi

On Wed, 22 Jan 2020 at 15:05, Alexander Frolushkin
wrote:

I have this error on bigtop-trunk. In webtop-1.4.0 it fails with other
error.

Also, in webtop-trunk this particular error appeard while building

hadoop

in docker (trunk-centos-7) image (by ./gradlew hadoop-pkg-ind) and

without

it (by ./gradlew hadoop-rpm).

Performed steps:

I was trying to follow the available guides, do a clean centos 7

install,

upgrade to latest official rpms, install openjdk 7 and unzip from repo.

Then puppetize.sh from bigtop_toolchain/bin and puppet apply
--modulepath=`pwd`:/etc/puppet/modules -e "include
bigtop_toolchain::installer".

After this run ./gradlew hadoop-pkg-ind or ./gradlew hadoop-rpm and got
this error.

On Wed, Jan 22, 2020, 12:56 Youngwoo Kim (김영우)

wrote:

Hi Alexander,

Which version or branch of Bigtop do you use? and also can you

provide

the steps you did?

Thanks,
Youngwoo

On Wed, Jan 22, 2020 at 2:00 PM Alexander Frolushkin <

iamho...@gmail.com

wrote:

Hello everyone.

Looks like Bigtop mailing lists is not very active, but I still

hope

someone could help me.

We are trying to employ BigTop to build own hadoop repository

(./gradlew

hadoop-rpm), but after all efforts still getting error on zookeeper

version

dependency:

-org.apache.hadoop:hadoop-yarn-server-resourcemanager:2.8.5

it wants -org.apache.zookeeper:zookeeper:3.4.6

while all other conponents depends on
-org.apache.zookeeper:zookeeper:3.4.13

so we got:

[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-enforcer-plugin:1.3.1:enforce

(depcheck)

project hadoop-yarn-server-tests

Please advise me some solution for this show stopper, if possible.

Thanks!

Re: Bigtop error on Centos 7

Thanks, Alexander, for bringing this up.

As a side note, I am looking at [1] and see the following error (which
seems to be there for some time):

Plugin [id: 'de.undercouch.download', version: '3.2.0'] was not found in
any of the following sources:

- Gradle Core Plugins (plugin is not in 'org.gradle' namespace)
- Plugin Repositories (could not resolve plugin artifact
'de.undercouch.download:de.undercouch.download.gradle.plugin:3.2.0')

Searched in the following repositories:
Gradle Central Plugin Repository

Unfortunately, it is hard to tell what kind of error you're getting in
your build. Any chance you can publish a snippet of the actual logs,
rather than a screenshot?

Thanks,Cos

[1]
https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/COMPONENTS=hadoop,OS=centos-7/567/console

On 1/23/20 4:50 AM, Alexander Frolushkin wrote:

Hello everyone!

Sorry for repeating this in dev list, but I really need help in this
situation.

On bigtop-1.4.0 build fails on hadoop-hdfs-httpfs with error:

Launched in clean bigtop-1.4.0 dir, with command docker run -v `pwd`:/ws
bigtop/slaves:1.4.0-centos-7 bash -l -c 'cd /ws;gradle hadoop-pkg

I made a --scan for this attempt - https://gradle.com/s/7iu4z3fsbrh62

Maybe I'm using not a valid way to get it perform correctly?

Or this can be bug and I need to fill a jira ticket for this?

Plus, the same error I get on bigtop-1.3.0.

Cleaning the local maven repository does not fix it.

So, likely, the problem is in building environment. How I can configure it
correctly for successful hadoop building in bigtop?

On Wed, Jan 22, 2020, 16:20 Evans Ye wrote:

Currently building Hadoop across all platform on master branch is failing:
https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/

I'm not sure whether the master code can be built successfully since we're
in transition of upgrading zookeeper.

If you just want to use bigtop, please get our latest release 1.4.0.

If you are willing to contribute, you can report a bug on our jira first
and we ca work on a fix together.

Let us know if any question.

Evan's

Yuqi Gu 於 2020年1月22日 週三 16:31 寫道：

Hi Alexander,

It seems bigtop_toolchain didn't deploy correctly.
Please try to build bigtop components in docker with bigtop/slaves docker
images .

Run:
cd BIGTOP_SRC_DIR
docker run -v `pwd`:/ws bigtop/slaves:1.4.0-centos-7 bash -l -c 'cd /ws ;
gradle hadoop-pkg'

BR,
Yuqi

On Wed, 22 Jan 2020 at 15:05, Alexander Frolushkin
wrote:

I have this error on bigtop-trunk. In webtop-1.4.0 it fails with other
error.

Also, in webtop-trunk this particular error appeard while building

hadoop

in docker (trunk-centos-7) image (by ./gradlew hadoop-pkg-ind) and

without

it (by ./gradlew hadoop-rpm).

Performed steps:

I was trying to follow the available guides, do a clean centos 7

install,

upgrade to latest official rpms, install openjdk 7 and unzip from repo.

Then puppetize.sh from bigtop_toolchain/bin and puppet apply
--modulepath=`pwd`:/etc/puppet/modules -e "include
bigtop_toolchain::installer".

After this run ./gradlew hadoop-pkg-ind or ./gradlew hadoop-rpm and got
this error.

On Wed, Jan 22, 2020, 12:56 Youngwoo Kim (김영우)

wrote:

Hi Alexander,

Which version or branch of Bigtop do you use? and also can you

provide

the steps you did?

Thanks,
Youngwoo

On Wed, Jan 22, 2020 at 2:00 PM Alexander Frolushkin <

iamho...@gmail.com

wrote:

Hello everyone.

Looks like Bigtop mailing lists is not very active, but I still

hope

someone could help me.

We are trying to employ BigTop to build own hadoop repository

(./gradlew

hadoop-rpm), but after all efforts still getting error on zookeeper

version

dependency:

-org.apache.hadoop:hadoop-yarn-server-resourcemanager:2.8.5

it wants -org.apache.zookeeper:zookeeper:3.4.6

while all other conponents depends on
-org.apache.zookeeper:zookeeper:3.4.13

so we got:

[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-enforcer-plugin:1.3.1:enforce

(depcheck)

project hadoop-yarn-server-tests

Please advise me some solution for this show stopper, if possible.

Thanks!

Re: Happy new year!

2019-12-31 Thread Konstantin Boudnik

Happy New Year, yall! It is great to see this project coming to its 10th year
being strong as ever and innovative as always!

This group is great and it feels s good to be a psrt of it!
Good luck and best wishes, everyone!

Cos

On Wed, Jan 01, 2020 at 04:34AM, Evans Ye wrote:
> Hi folks,
> 
> Another great year at Apache Bigtop. I'm glad for being one of the
> community member.
> 
> This year we have a big initiative: cloud native bigtop. I'm very excited
> to see the upcoming first release of it. In the meantime, we also have
> other folks keep advancing the core bigtop distribution. Thank you all &
> happy new year!
> 
> Evans

Re: Minio to copresent , quorum for next gen bigtop ?

2019-09-12 Thread Konstantin Boudnik

Well, here's an interesting piece of news:

https://www.theregister.co.uk/2019/09/10/google_cloud_lights_spark_for_kubernetes/

Mind you, these nice fellas [/sarc] promised on two different occasions that
they will be contributing back to Apache Bigtop - the foundation of the
Dataproc - and yet they never did.  But I guess this is fine ... not the first
company taking this kind of advantage :)

--
  Cos

On Wed, Sep 04, 2019 at 05:54AM, Jay Vyas wrote:
> Hi folks.  Couple of updateys  in the next gen brainstorming we’ve been 
> doing...
> 
> Sid Mani from minio is going to copresent with me at our bigtop talk at
> apachecon next week, around the internals of minio integrations etc. 
> 
> Spoke with cos and sounds like we definetly have a quorum for hacking on this 
> next gen implementation.
> 
> Whose in on hacking on it,  and who will be in Vegas on the 12th?
> 
> I’ll help w the k8s stuff so don’t be intimidated if that’s your weak spot.

signature.asc
Description: Digital signature

Re: Shall tajo

2019-08-30 Thread Konstantin Boudnik

+1 on both counts. Perhaps 'branch-1.x' would be informative.
Also, let's wait a couple of days to let others on the list to express their
opinion, perhaps different and/or complimentary.

And Jay - thanks for sharing your work: I owe you a look into this, but life
last month wasn't overly relaxed in terms of spare time.

Thanks,
  cos

On Fri, Aug 30, 2019 at 11:30AM, Youngwoo Kim (김영우) wrote:
> Sounds great!
> 
> I like the idea that reshaping big data architecture at cloud native
> deployments.
> What about creating 'branch-1' from current status for existing users and
> traditional architectures? So, we can make progress from master branch for
> the future Bigtop releases.
> 
> Jay,
> I feel like we should create a JIRA issue or shared doc to keep discussions
> on detailed plan and design docs for Bigtop NG.
> 
> Thanks,
> Youngwoo
> 
> On Fri, Aug 30, 2019 at 10:28 AM Jun HE  wrote:
> 
> > Super +1 for Bigtop 2.0 with modern components and K8S support! I cannot
> > wait to see Bigtop running on K8S...
> > So can someone help to create a branch-2.0 now? :P
> >
> >
> > Konstantin Boudnik  于2019年8月30日周五 上午2:24写道：
> >
> > > Aah, k8s - thanks Jay!
> > >
> > > BTW, I believe we can do this in a less formal stuff (ie no VOTE): we
> > have
> > > a
> > > discussion going on the version of Bigtop. How about we make it 2.0 and
> > > trim
> > > the BOM into the needed shape and form? If there's enough drive in the
> > > community to continue with 1.x line (1.4 and so on), it can be done as a
> > > parallel effort.
> > >
> > > Thoughts?
> > >   Cos
> > >
> > > On Fri, Aug 30, 2019 at 01:26AM, Evans Ye wrote:
> > > > I'm super +1 with K8S stuff and the core components idea!
> > > >
> > > > I'm with Cos and Jun. Removing those stuffs can ease our CI resource
> > > > utilization and yield more stable CI results. This is what I'm thinking
> > > how
> > > > to proceed based on the previous feedback that a drop of supporting
> > > > component should be carefully discussed
> > > >
> > > > 1. Spin up a vote to drop project XXX (if confident enough, put
> > multiple
> > > > components here)
> > > > 2. If +3 binding votes with no -1 votes, it passes.
> > > > 3. Put up the PR for code removing
> > > > 4. Refine our CI settings (I can definitely help with this part)
> > > >
> > > > Best,
> > > > Evans
> > > >
> > > >
> > > >
> > > >
> > > > Jay Vyas  於 2019年8月29日 週四 下午7:26寫道：
> > > >
> > > > > First of all I’m all for dropping the old stuff.
> > > > >
> > > > > 1) My opinion - to further cos point - I think people running old
> > stuff
> > > > > don’t really need version updates, or if they do, they don’t
> > > constitute a
> > > > > large audience , or should contribute them ok their own.
> > > > >
> > > > > 2) surprise surprise :):) here’s my k8s native. View - we should move
> > > the
> > > > > old components into sustaining mode, and rebuild a new bigtop focus
> > > solely
> > > > > on spark , NiFi , Presto (with a lot of attention to minimal
> > standalone
> > > > > Hadoop w/ a Hive connector ) and target it at kubernetes native
> > > deployments
> > > > > :).
> > > > >
> > > > > I can make it so the k8s part is an impl detail so users can be
> > mostly
> > > > > decoupled from it .
> > > > >
> > > > > I’ve been integration testing various things in the bigdata ecosystem
> > > on
> > > > > k8s and there is a lot of demand without anyone owning the
> > integration.
> > > > >
> > > > > > On Aug 29, 2019, at 2:39 AM, Konstantin Boudnik 
> > > wrote:
> > > > > >
> > > > > > I am not sure about Giraph, Tajo and some others, but Sqoop seems
> > to
> > > be
> > > > > user
> > > > > > around by people. So, if it isn't much of the burden for us - and
> > it
> > > > > seems
> > > > > > pretty stable at the moment - I'd leave it.
> > > > > >
> > > > > > What I would think would makes sense to spend some of our efforts
> > on
> > > is
> > > > > on
> > > > > > adding modern tooling like

Re: Shall tajo

2019-08-29 Thread Konstantin Boudnik

Aah, k8s - thanks Jay!

BTW, I believe we can do this in a less formal stuff (ie no VOTE): we have a
discussion going on the version of Bigtop. How about we make it 2.0 and trim
the BOM into the needed shape and form? If there's enough drive in the
community to continue with 1.x line (1.4 and so on), it can be done as a
parallel effort.

Thoughts?
  Cos

On Fri, Aug 30, 2019 at 01:26AM, Evans Ye wrote:
> I'm super +1 with K8S stuff and the core components idea!
> 
> I'm with Cos and Jun. Removing those stuffs can ease our CI resource
> utilization and yield more stable CI results. This is what I'm thinking how
> to proceed based on the previous feedback that a drop of supporting
> component should be carefully discussed
> 
> 1. Spin up a vote to drop project XXX (if confident enough, put multiple
> components here)
> 2. If +3 binding votes with no -1 votes, it passes.
> 3. Put up the PR for code removing
> 4. Refine our CI settings (I can definitely help with this part)
> 
> Best,
> Evans
> 
> 
> 
> 
> Jay Vyas  於 2019年8月29日 週四 下午7:26寫道：
> 
> > First of all I’m all for dropping the old stuff.
> >
> > 1) My opinion - to further cos point - I think people running old stuff
> > don’t really need version updates, or if they do, they don’t constitute a
> > large audience , or should contribute them ok their own.
> >
> > 2) surprise surprise :):) here’s my k8s native. View - we should move the
> > old components into sustaining mode, and rebuild a new bigtop focus solely
> > on spark , NiFi , Presto (with a lot of attention to minimal standalone
> > Hadoop w/ a Hive connector ) and target it at kubernetes native deployments
> > :).
> >
> > I can make it so the k8s part is an impl detail so users can be mostly
> > decoupled from it .
> >
> > I’ve been integration testing various things in the bigdata ecosystem on
> > k8s and there is a lot of demand without anyone owning the integration.
> >
> > > On Aug 29, 2019, at 2:39 AM, Konstantin Boudnik  wrote:
> > >
> > > I am not sure about Giraph, Tajo and some others, but Sqoop seems to be
> > user
> > > around by people. So, if it isn't much of the burden for us - and it
> > seems
> > > pretty stable at the moment - I'd leave it.
> > >
> > > What I would think would makes sense to spend some of our efforts on is
> > on
> > > adding modern tooling like Nifi/Airflow into the mix. As you said -
> > things
> > > move forward pretty fast, and we seem to be sticking to the some of the
> > old
> > > stuff.
> > >
> > > Thanks for starting this discussion!
> > >  Cos
> > >
> > >> On Wed, Aug 28, 2019 at 04:12PM, Jun HE wrote:
> > >> Hi, folks,
> > >>
> > >> I went through current components Bigtop is supporting, and I noticed
> > that
> > >> these upstream projects haven't been released for quite a while:
> > >> Apache Tajo:
> > >>last release: release-0.11.3-rc0, May 11, 2016
> > >> Apache Apex:
> > >>last release: v3.7.0, Apr 27, 2018
> > >> Apache Giraph:
> > >>last release: rel/1.2.0-RC1, Oct 13 2016
> > >> Apache Hama:
> > >>last release: 0.7.1-RC2, Mar 12, 2016
> > >> Apache Sqoop:
> > >>last release: release-1.4.7-rc0, Dec 6, 2017; release-1.99.7-rc1, Jul
> > >> 20, 2016
> > >>
> > >> And some of them seem to be in slow development:
> > >> Apache Tajo:
> > >>last commit: Jul 13, 2018
> > >> Apache Apex:
> > >>last commit: Jun 20, 2018
> > >> Apache Hama:
> > >>last commit: Jul 30, 2018
> > >>
> > >> So I'm wondering whether we should continue support for these components
> > >> (or part of them) in next/future releases.
> > >>
> > >> I understand that similar topics were discussed before. But, you know,
> > this
> > >> is an quickly evolving world, maybe it worth another revisit now? ;)
> > >>
> > >> Regards,
> > >>
> > >> Jun
> >


signature.asc
Description: Digital signature

Re: Shall tajo

2019-08-29 Thread Konstantin Boudnik

I am not sure about Giraph, Tajo and some others, but Sqoop seems to be user
around by people. So, if it isn't much of the burden for us - and it seems
pretty stable at the moment - I'd leave it.

What I would think would makes sense to spend some of our efforts on is on
adding modern tooling like Nifi/Airflow into the mix. As you said - things
move forward pretty fast, and we seem to be sticking to the some of the old
stuff.

Thanks for starting this discussion!
  Cos

On Wed, Aug 28, 2019 at 04:12PM, Jun HE wrote:
> Hi, folks,
> 
> I went through current components Bigtop is supporting, and I noticed that
> these upstream projects haven't been released for quite a while:
> Apache Tajo:
> last release: release-0.11.3-rc0, May 11, 2016
> Apache Apex:
> last release: v3.7.0, Apr 27, 2018
> Apache Giraph:
> last release: rel/1.2.0-RC1, Oct 13 2016
> Apache Hama:
> last release: 0.7.1-RC2, Mar 12, 2016
> Apache Sqoop:
> last release: release-1.4.7-rc0, Dec 6, 2017; release-1.99.7-rc1, Jul
> 20, 2016
> 
> And some of them seem to be in slow development:
> Apache Tajo:
> last commit: Jul 13, 2018
> Apache Apex:
> last commit: Jun 20, 2018
> Apache Hama:
> last commit: Jul 30, 2018
> 
> So I'm wondering whether we should continue support for these components
> (or part of them) in next/future releases.
> 
> I understand that similar topics were discussed before. But, you know, this
> is an quickly evolving world, maybe it worth another revisit now? ;)
> 
> Regards,
> 
> Jun

signature.asc
Description: PGP signature

Re: [DISCUSS] Roadmap for the next release.1.5? 2.0?

2019-07-25 Thread Konstantin Boudnik

Ugly... I wonder if it means that Hadoop development has decided to go
full-parcel (and its irks) and make Hadoop a boutique software?

At any rate, I believe this will slower the adoption of it to a point of where
it is only available through a single vendor left in the commercial market.
But that, fortunately, not my problem to resolve ;)

Cos
 
On Thu, Jul 25, 2019 at 05:27AM, Olaf Flebbe wrote:
> Hi
> 
> hadoop changed their scripts to break when not absolutely everything is in 
> place.
> 
> The start scripts for yarn for example expect to all jars for hdfs, yarn and
> mapreduce to be present. They renamed (to be precise deprecated the old
> name) of environment variables in a way it looks more convenient, but it
> will mix up hadoop subprojects. we cannot use independent /run and /var dirs
> as the debian build rules and some other unwritten rules of linux packaging
> demand in order to be installed independently.
> 
> So hadoop has tied everything together. without rewriting their build
> scripts it is not possible to have yarn without hdfs or hdfs without yarn to
> be installed independently. 
> 
> Olaf
> 
> ps: The way pid files are written confuses systemd (or only me). effectively
> systemd somehow doesnt properly detect the state of the daemons.
> 
> Von meinem iPad gesendet
> 
> > Am 25.07.2019 um 02:11 schrieb Konstantin Boudnik :
> > 
> > Olaf, thanks for trying... this could be exhausting, for sure ;(
> > 
> > Just to clarify: when you say "monolithic chunk" doesn't it mean that it
> > relies heavily on relative paths or something of the sort and couldn't be
> > broken into pieces because... well, it is so broken?
> > 
> > Thanks,
> >  Cos
> > 
> >> On Mon, Jul 22, 2019 at 10:34PM, Olaf Flebbe wrote:
> >> Hi,
> >> 
> >> If I would have only known that google document before ... I am
> >> asking myself if we should link it from confluence.
> >> 
> >> + hadoop 3.
> >> As you may know, I worked on it (bigtop-alpha branch), got it
> >> compiled and packaged. However, while testing, I discovered that
> >> Hadoop project built on top of some assumptions which do not hold
> >> true for current Bigtop. One of them is that Hadoop 3 is to be
> >> installed as a monolithic chunk on a filesystem. This is not what I
> >> understand as integrated into a Linux distribution. Other point is
> >> that hadoop's installation proposal does not follow the LSB:
> >> platform specfic libs are in "share" directories. I am exhausted,
> >> this is so broken.
> >> 
> >> I will stop here ... for a reason.
> >> 
> >> Olaf
> >> 
> >> 
> >> 
> >> 
> >> 
> >>> Am 17.07.19 um 09:09 schrieb Youngwoo Kim (김영우):
> >>> Hi folks,
> >>> 
> >>> After 1.4.0 release, there is no discussion for the next release yet. so I
> >>> believe we need to share the ideas and prioritize the items for
> >>> development.
> >>> 
> >>> And also https://issues.apache.org/jira/browse/BIGTOP-3123 and
> >>> https://docs.google.com/document/d/1F2Gxu8GARQDZXgqHn12LKkQ5wCV_AF4b_tVmjYB6YfA/edit
> >>> are good starting point for this discussion.
> >>> 
> >>> My personal preferences are:
> >>> - Distribution based on Hadoop 3
> >>> - Up-to-date BigPetStore
> >>> - Software stacks and framework for streaming data & Machine Learning
> >>> - Containers and Cloud Native: What? How?
> >>> 
> >>> It would be great to hear your thoughts.
> >>> 
> >>> Thanks,
> >>> Youngwoo
> >>> 


signature.asc
Description: Digital signature

Re: [DISCUSS] Roadmap for the next release.1.5? 2.0?

2019-07-24 Thread Konstantin Boudnik

Olaf, thanks for trying... this could be exhausting, for sure ;(

Just to clarify: when you say "monolithic chunk" doesn't it mean that it
relies heavily on relative paths or something of the sort and couldn't be
broken into pieces because... well, it is so broken?

Thanks,
  Cos

On Mon, Jul 22, 2019 at 10:34PM, Olaf Flebbe wrote:
> Hi,
> 
> If I would have only known that google document before ... I am
> asking myself if we should link it from confluence.
> 
> + hadoop 3.
> As you may know, I worked on it (bigtop-alpha branch), got it
> compiled and packaged. However, while testing, I discovered that
> Hadoop project built on top of some assumptions which do not hold
> true for current Bigtop. One of them is that Hadoop 3 is to be
> installed as a monolithic chunk on a filesystem. This is not what I
> understand as integrated into a Linux distribution. Other point is
> that hadoop's installation proposal does not follow the LSB:
> platform specfic libs are in "share" directories. I am exhausted,
> this is so broken.
> 
> I will stop here ... for a reason.
> 
> Olaf
> 
> 
> 
> 
> 
> Am 17.07.19 um 09:09 schrieb Youngwoo Kim (김영우):
> >Hi folks,
> >
> >After 1.4.0 release, there is no discussion for the next release yet. so I
> >believe we need to share the ideas and prioritize the items for
> >development.
> >
> >And also https://issues.apache.org/jira/browse/BIGTOP-3123 and
> >https://docs.google.com/document/d/1F2Gxu8GARQDZXgqHn12LKkQ5wCV_AF4b_tVmjYB6YfA/edit
> >are good starting point for this discussion.
> >
> >My personal preferences are:
> >  - Distribution based on Hadoop 3
> >  - Up-to-date BigPetStore
> >  - Software stacks and framework for streaming data & Machine Learning
> >  - Containers and Cloud Native: What? How?
> >
> >It would be great to hear your thoughts.
> >
> >Thanks,
> >Youngwoo
> >


signature.asc
Description: Digital signature

Re: RFC: Building .tar.gz binary packaging for all components in Bigtop

2019-07-19 Thread Konstantin Boudnik

+1 on both Olaf's and Evans' points. Tarballs are messy stuff and are hard to
control in operations: I believe the main reason for the existence of them in
the official component releases is the simplicity of the media and lesser
pressure on the community to integrate the software into a target stack (to
Olaf's point).

To the ARM's folks issue: aren't they able to use structured packages for some
reason? Funny, this conversation happens every 2-3 years, like a clock ;)

Thanks,
  Cos

On Sat, Jul 20, 2019 at 01:06AM, Evans Ye wrote:
> From technical perspective Olaf's points are reasonable. However I'd like
> to bridge the gap here so that we can strike a balance of what arm folks
> need.
> 
> Guodong let's back the story with some technical discussions. For example,
> is it because your customer already implemented their deployment tool with
> tar.gz? If so that's a strong reason from their perspective.
> To me tar.gz has no much pros for production. For Apache projects, all the
> release are source code hence tar.gz is the most simple way. Furthermore,
> most of the projects will provide compiled binary tarball just for
> convenient.
> 
> Olaf Flebbe  於 2019年7月19日 週五 上午3:40寫道：
> 
> > Hi
> >
> > Maybe we have different target groups: For educational users, trying to
> > figure out how everything works together, the single computer cluster is
> > great.
> >
> > This Tar.gz artefact were never ever been sufficient to run in production.
> > They did not contain 64bit libs for instance. They do not provide start
> > scripts, error prone when you accidently using the wrong java, do not place
> > logfiles in suitable places and so on.
> >
> > This tar artefacts and instructions are POC quality, that's it.
> >
> > Please be aware that Bigtop changes directory layout, directory
> > permissions, configuration and startup of the original code in order to
> > support large scale installations and automation.
> >
> > We have a much better alternative: Distribution integrated repositories.
> > Youll only have to apt/yum install hadoop-hdfs-datanode and everything is
> > already setup, including java, runscripts, directory layout and users. Then
> > you can concentrate on configuring that beast. You do not even need no
> > special instruction, since it is distribution native -- well, almost. If
> > you feel that bigtops runscripts / layout is not suitable , you are very
> > welcome to contribute.
> >
> > If i look at the instructions [1] . they contain tons of settings you
> > should do. With our packages you can actually do exactly this, without
> > bothering to install and configure all the dependencies.
> >
> > Olaf
> >
> >
> >
> >
> >
> > Von meinem iPad gesendet
> >
> > > Am 18.07.2019 um 08:57 schrieb Guodong Xu :
> > >
> > > Hi, Evans
> > >
> > > Comments in below.
> > >
> > >>  On Tue, Jul 16, 2019 at 10:46 PM Evans Ye 
> > wrote:
> > >>
> > >> I'm not objecting this, but I'd love to have more discussion to figure
> > out
> > >> whether this is the right thing to do. What I get from your proposal is
> > >> users want to do things which RPM/DEB don't do well while tar.gz is
> > good at
> > >> it. However is it able to do it in another way which is far more
> > beneficial
> > >> in more scenarios? In general, it's like when users are asking for a
> > faster
> > >> horses, can we come up with cars?
> > >>
> > >> How about we start with the first step which is to elaborate why users
> > >> choose to go for tar.gz?
> > >
> > >
> > > Right, agree. Here is what I learned from users of Arm servers:
> > >
> > > One background for this is, currently, CDH and Hortonworks both have no
> > > official release for Arm server yet. So, Bigtop is the only available and
> > > verified distribution to them. As you know, with effort from the
> > community
> > > and Arm Inc., Linaro, Bigtop now has officially supported Arm64.
> > >
> > > To these users, before they start to use Bigtop on Arm, they are already
> > > familiar with each component's individual installation and usage on x86.
> > > Most of them are released in .tar.gz format. (i.e. Most apache big data
> > > component doesn't release in deb/rpm. So, if we tell users that the only
> > > available format in Bigtop is deb/rpm, this just hesitates them).
> > >
> > > Eg. For Hadoop, the official site for installation is here [1] and here
> > > [2], their release format is .tar.gz.
> > >
> > > So,
> > > 1. using .tar.gz is the minimum effort route for them to start their
> > touch
> > > with Bigtop (if we can support .tar.gz).
> > > 2. Bigtop has all components tested. That provides very big confidence to
> > > the users. So they like to use the binary built from Bigtop on Arm64
> > > servers.
> > >
> > > [1]
> > >
> > https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html
> > > [2] https://www.apache.org/dyn/closer.cgi/hadoop/common/
> > >
> > > -Guodong
> > >
> > >
> > >>
> > >> Guodong Xu  於 2019年7月16日 週二 下午12:24寫道：
> > >>
> > >>>

Re: [ANNOUNCE] Apache Bigtop 1.4.0 released

2019-06-19 Thread Konstantin Boudnik

Quite an achievement, thanks for driving this, Evans!
--
  With regards,
Konstantin (Cos) Boudnik

On Tue, Jun 18, 2019 at 7:07 PM Evans Ye  wrote:
>
> On behalf of the Apache Bigtop team, I'd love to announce the general
> availability of the Bigtop 1.4.0 release.
>
> The release is available here:
> https://bigtop.apache.org/download.html#releases
>
> A few highlights of this release include:
> - Integration Test Framework 2.0: one-stop integrated build and test
>   framework at a single entry: ./gradlew [1]
> - Newly developed Smoke Test CI Matrix to guard the quality of releases [2]
> - Hadoop 2.8.5, Spark 2.2.3, Flink 1.6.0, Alluxio 1.8.1 and more [3]
> - 100+ JIRAs are resolved in this release
>
> With Bigtop 1.4.0 the community continues to deliver the most advanced big
> data stack to date. More details about 1.4.0 release are here:
> https://bigtop.apache.org/release-notes.html
>
> Deploying Bigtop is easy: grab the repo/list file for your favorite Linux
> distribution:
>   https://www.apache.org/dyn/closer.lua/bigtop/bigtop-1.4.0/repos/
> and you'll be running your very own big data cluster in no time!
>
> We welcome your help and feedback. For more information on how to report
> problems, and to get involved, visit the project website at:
>   https://bigtop.apache.org
>
> Lastly, I want to emphasize that this is a collaborative work done by
> project
> contributors and other communities, who continue to devote time to make
> Bigtop a better software. Thank you all for making this release possible!
>
> Thanks,
> Evans Ye, Release Manager
>
> [1]
> https://cwiki.apache.org/confluence/display/BIGTOP/Quickstart+Guide%3A+Bigtop+Integration+Test+Framework+2.0
> [2]
> https://ci.bigtop.apache.org/view/Test/job/Bigtop-trunk-smoke-tests-1.4.0/
> [3] https://issues.apache.org/jira/browse/BIGTOP-3162

Re: [VOTE] Release Bigtop version 1.4.0 (Release Candidate 1)

2019-06-12 Thread Konstantin Boudnik

Thanks, was about the send this message...

+1 [binding] 

I have done some basic chores - it is releasable, thanks!

Thanks!
  Cos

On Wed, Jun 12, 2019 at 05:16PM, Evans Ye wrote:
> Hey Cos, I'll close the vote now and process the release. If you have spot
> any issues let's fix it in master ;)
> 
> Evans Ye  於 2019年6月10日 週一 下午10:48寫道：
> 
> > No hurries. Actually I’d flavor more community involvement :) So please
> > explore the RC with your feedback.
> >
> > Jun HE 於 2019年6月10日 週一，上午10:17寫道：
> >
> >> +1.
> >>
> >> Tested Ubuntu and Debian, and basically works.
> >>
> >> Konstantin Boudnik  于2019年6月10日周一 上午4:55写道：
> >>
> >> > Fellas, I just got back from a weekend trip and I didn't have time to do
> >> > any release testing. I see there's enough votes already, but if it'd
> >> make
> >> > sense to have an extra pair of eyes to look at it - I would be happy
> >> to. I
> >> > need an extra day though.
> >> >
> >> > Thanks!
> >> > --
> >> > With regards,
> >> >   Konstantin
> >> >
> >> > On June 5, 2019 1:29:00 AM GMT+03:00, Evans Ye 
> >> wrote:
> >> > >Let's extend the vote to *Sunday, June 9th, 2019 at noon PDT*.
> >> > >Let me know if it's still not enough time for you to evaluate the
> >> > >release :)
> >> > >
> >> > >Youngwoo Kim (김영우)  於 2019年6月3日 週一 上午3:23寫道：
> >> > >
> >> > >> Sorry for the late, Evans.
> >> > >>
> >> > >> We need to extend time window for this vote. just a couple of days?
> >> > >> I would like to test the release candidate when I come back to my
> >> > >office.
> >> > >>
> >> > >> Thanks,
> >> > >> Youngwoo
> >> > >>
> >> > >> On Mon, Jun 3, 2019 at 8:02 AM Evans Ye  wrote:
> >> > >>
> >> > >> > Hi folks,
> >> > >> >
> >> > >> > Any info I can provide and help you to evaluate the release
> >> > >candidate?
> >> > >> >
> >> > >> > Evans Ye  於 2019年5月28日 週二 下午8:37寫道：
> >> > >> >
> >> > >> > > Hi folks,
> >> > >> > >
> >> > >> > > If you have voted RC0, this is pretty much the same except for
> >> > >the fix
> >> > >> to
> >> > >> > > change docker image name from trunk to 1.4.0 :)
> >> > >> > >
> >> > >> > > Evans Ye 於 2019年5月23日 週四，下午4:12寫道：
> >> > >> > >
> >> > >> > >> BTW, binary artifacts for centos7, debian9, opensuse42.3 on PPC
> >> > >are
> >> > >> all
> >> > >> > >> uploaded to S#. However only debian 9 seems good. Others
> >> > >encountered
> >> > >> > >> dependency issues when installing packages. We shall only mark
> >> > >those
> >> > >> > >> working Distros as supported. I'll make this clear in the
> >> > >release
> >> > >> note.
> >> > >> > >>
> >> > >> > >> Evans Ye  於 2019年5月23日 週四 下午8:59寫道：
> >> > >> > >>
> >> > >> > >>> Hi folks,
> >> > >> > >>>
> >> > >> > >>> This is the vote to release 1.4.0 of Apache Bigtop with Release
> >> > >> > >>> Candidate 1.
> >> > >> > >>>
> >> > >> > >>> It fixes the following issues:
> >> > >> > >>> *
> >> > >> > >>>
> >> > >> >
> >> > >>
> >> > >
> >> >
> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12344113=12311420
> >> > >> > >>>
> >> > >> > >>> The vote will be going for at least 72 hours and will be closed
> >> > >on
> >> > >> > >>> Sunday,
> >> > >> > >>> *May 26th, 2019 at noon PDT*.  Please download, test and vote
> >> > >with
> >> > >> > >>>
> >> > >> > >>> [ ] +1, accept RC1 as the official 1.4.0 release of Apache
> >> > >Bigtop
> >> > >> > >>> [ ] +0, I don't care either way,
> >> > >> > >>> [ ] -1, do not accept RC1 as the official 1.4.0 release of
> >> > >Apache
> >> > >> > >>> Bigtop, because...
> >> > >> > >>>
> >> > >> > >>> Change(s) compared to RC0:
> >> > >> > >>> * Switch to 1.4.0 docker images for provisioner:
> >> > >> > >>>
> >> > >> >
> >> > >>
> >> > >
> >> >
> >> https://github.com/apache/bigtop/commit/4921ebd737bacb48ad22c7e8b572185b572ad7b4
> >> > >> > >>>
> >> > >> > >>> Source and binary files:
> >> > >> > >>> *
> >> > >https://dist.apache.org/repos/dist/dev/bigtop/bigtop-1.4.0-RC1
> >> > >> > >>>
> >> > >> > >>> Maven staging repo:
> >> > >> > >>> *
> >> > >> > >>>
> >> > >> >
> >> > >
> >> https://repository.apache.org/content/repositories/orgapachebigtop-1023
> >> > >> > >>>
> >> > >> > >>> The git tag to be voted upon is release-1.4.0-RC1
> >> > >> > >>>
> >> > >> > >>> Bigtop's KEYS file containing PGP keys we use to sign the
> >> > >release:
> >> > >> > >>> * https://dist.apache.org/repos/dist/release/bigtop/KEYS
> >> > >> > >>>
> >> > >> > >>> Bigtop 1.4.0 CI result:
> >> > >> > >>> * Packaging:
> >> > >> > https://ci.bigtop.apache.org/view/Releases/job/Bigtop-1.4.0
> >> > >> > >>> * Deploy & Test:
> >> > >> > >>>
> >> > >> >
> >> > >>
> >> > >
> >> https://ci.bigtop.apache.org/view/Test/job/Bigtop-trunk-smoke-tests-1.4.0
> >> > >> > >>>
> >> > >> > >>>
> >> > >> > >>> Evans Ye, RM
> >> > >> > >>>
> >> > >> > >>
> >> > >> >
> >> > >>
> >> >
> >>
> >


signature.asc
Description: PGP signature

Re: [VOTE] Release Bigtop version 1.4.0 (Release Candidate 1)

2019-06-09 Thread Konstantin Boudnik

Fellas, I just got back from a weekend trip and I didn't have time to do any 
release testing. I see there's enough votes already, but if it'd make sense to 
have an extra pair of eyes to look at it - I would be happy to. I need an extra 
day though.

Thanks! 
-- 
With regards,
  Konstantin

On June 5, 2019 1:29:00 AM GMT+03:00, Evans Ye  wrote:
>Let's extend the vote to *Sunday, June 9th, 2019 at noon PDT*.
>Let me know if it's still not enough time for you to evaluate the
>release :)
>
>Youngwoo Kim (김영우)  於 2019年6月3日 週一 上午3:23寫道：
>
>> Sorry for the late, Evans.
>>
>> We need to extend time window for this vote. just a couple of days?
>> I would like to test the release candidate when I come back to my
>office.
>>
>> Thanks,
>> Youngwoo
>>
>> On Mon, Jun 3, 2019 at 8:02 AM Evans Ye  wrote:
>>
>> > Hi folks,
>> >
>> > Any info I can provide and help you to evaluate the release
>candidate?
>> >
>> > Evans Ye  於 2019年5月28日 週二 下午8:37寫道：
>> >
>> > > Hi folks,
>> > >
>> > > If you have voted RC0, this is pretty much the same except for
>the fix
>> to
>> > > change docker image name from trunk to 1.4.0 :)
>> > >
>> > > Evans Ye 於 2019年5月23日 週四，下午4:12寫道：
>> > >
>> > >> BTW, binary artifacts for centos7, debian9, opensuse42.3 on PPC
>are
>> all
>> > >> uploaded to S#. However only debian 9 seems good. Others
>encountered
>> > >> dependency issues when installing packages. We shall only mark
>those
>> > >> working Distros as supported. I'll make this clear in the
>release
>> note.
>> > >>
>> > >> Evans Ye  於 2019年5月23日 週四 下午8:59寫道：
>> > >>
>> > >>> Hi folks,
>> > >>>
>> > >>> This is the vote to release 1.4.0 of Apache Bigtop with Release
>> > >>> Candidate 1.
>> > >>>
>> > >>> It fixes the following issues:
>> > >>> *
>> > >>>
>> >
>>
>https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12344113=12311420
>> > >>>
>> > >>> The vote will be going for at least 72 hours and will be closed
>on
>> > >>> Sunday,
>> > >>> *May 26th, 2019 at noon PDT*.  Please download, test and vote
>with
>> > >>>
>> > >>> [ ] +1, accept RC1 as the official 1.4.0 release of Apache
>Bigtop
>> > >>> [ ] +0, I don't care either way,
>> > >>> [ ] -1, do not accept RC1 as the official 1.4.0 release of
>Apache
>> > >>> Bigtop, because...
>> > >>>
>> > >>> Change(s) compared to RC0:
>> > >>> * Switch to 1.4.0 docker images for provisioner:
>> > >>>
>> >
>>
>https://github.com/apache/bigtop/commit/4921ebd737bacb48ad22c7e8b572185b572ad7b4
>> > >>>
>> > >>> Source and binary files:
>> > >>> *
>https://dist.apache.org/repos/dist/dev/bigtop/bigtop-1.4.0-RC1
>> > >>>
>> > >>> Maven staging repo:
>> > >>> *
>> > >>>
>> >
>https://repository.apache.org/content/repositories/orgapachebigtop-1023
>> > >>>
>> > >>> The git tag to be voted upon is release-1.4.0-RC1
>> > >>>
>> > >>> Bigtop's KEYS file containing PGP keys we use to sign the
>release:
>> > >>> * https://dist.apache.org/repos/dist/release/bigtop/KEYS
>> > >>>
>> > >>> Bigtop 1.4.0 CI result:
>> > >>> * Packaging:
>> > https://ci.bigtop.apache.org/view/Releases/job/Bigtop-1.4.0
>> > >>> * Deploy & Test:
>> > >>>
>> >
>>
>https://ci.bigtop.apache.org/view/Test/job/Bigtop-trunk-smoke-tests-1.4.0
>> > >>>
>> > >>>
>> > >>> Evans Ye, RM
>> > >>>
>> > >>
>> >
>>

Re: Use of ruby in docker-hadoop

2019-05-28 Thread Konstantin Boudnik

On Tue, May 28, 2019 at 08:34PM, Evans Ye wrote:
> I wrote it with ruby just because it works. I’d favor to reduce any package
> required to be installed, if possible.

Yup, I totally understand and appreciate it - no critique here ;)

> Just wondering what’s the problem you are facing?  Is Python most likely to
> be available out of the box?

Well, funny enough it was the case with this box. The Python was right there.
But nonetheless, I think the reliance on two (or more) things instead of one
just adds the complexity and could be reduced. If there's no objection I will
take a shot at it.

And once again - I am so glad the docker provisioner is here! Makes my life
easier...!

Thanks,
  Cos

> Konstantin Boudnik 於 2019年5月28日 週二，下午8:21寫道：
> 
> > Hey fellas.
> >
> > I am preparing for a small demo, using a brand new install Linux
> > installation. And a couple of things came to my attention:
> >
> > -- python is missing 'six' package
> > -- the docker-hadoop.sh is using Ruby
> >
> > The first one is a small env issue, but I wonder why do we need need Ruby
> > to
> > (if I am not mistaken) parse a yaml config? It seems like an old code going
> > back to 2016. Would anyone mind if I supply a patch to replace it with some
> > tiny python hack?
> >
> > Thoughts?
> >   Cos
> >


signature.asc
Description: PGP signature

Use of ruby in docker-hadoop

2019-05-28 Thread Konstantin Boudnik

Hey fellas.

I am preparing for a small demo, using a brand new install Linux
installation. And a couple of things came to my attention:

-- python is missing 'six' package
-- the docker-hadoop.sh is using Ruby

The first one is a small env issue, but I wonder why do we need need Ruby to
(if I am not mistaken) parse a yaml config? It seems like an old code going
back to 2016. Would anyone mind if I supply a patch to replace it with some
tiny python hack?

Thoughts?
  Cos


signature.asc
Description: PGP signature

Re: R.I.P. mrdocs (1963–2019)

2019-04-01 Thread Konstantin Boudnik

Was going through an old photo-archive and came across this picture of
Peter. If I am not mistaken, it was taken at the Suse conference in
2013.

https://photos.app.goo.gl/ytDgo85SS39iXFca9

--
  With regards,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622

Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.

On Sun, Mar 31, 2019 at 11:13 PM Sean Mackrory  wrote:
>
> This is sad news, indeed. Peter and I never overlapped at Cloudera but he
> went out of his way to find me at conferences and introduce me to people. I
> very much appreciated his mentorship and friendship. If anyone does know of
> an address or is planning to send a token to his family, I would like to
> contribute.
>
> On Thu, Mar 28, 2019 at 4:01 PM Bruno Mahé  wrote:
>
> > RIP Peter.
> >
> > It's a very sad news.
> >
> >
> > On 3/22/19 5:20 PM, Roman Shaposhnik wrote:
> > > It is with incredibly heavy heart that I have to tell you all about this:
> > >  https://www.scribus.net/r-i-p-mrdocs-1963-2019/
> > >
> > > A lot of us Bigtop folks knew Peter in person and he was every bit
> > > the kind of man that made Open Source what it is today. I'm rattled
> > > and saddened to hear of his passing. My condolences to his family
> > > and everyone around here who knew him.
> > >
> > > Thanks,
> > > Roman.
> >
> > --
> > Thanks,
> > Bruno
> >
> >

Re: [DISCUSS] Remove Crunch in Bigtop

2019-03-24 Thread Konstantin Boudnik

Thanks Evans! 
--
Regards,
  Cos

On March 24, 2019 12:40:23 AM PDT, Evans Ye  wrote:
>Since there's no objection and we got at least 2 votes(binding) for
>removing it. I'll proceed with the removal by applying BIGTOP-3196
>tonight.
>
>Evans Ye  於 2019年3月19日 週二 下午2:19寫道：
>
>> Hi folks,
>>
>> To bring back the discussion ongoing in BIGTOP-3192
>> . We've 3 options
>to
>> deal with Crunch build failure with spark 2:
>>
>> 1. Keep spark1
>> 2. Remove Spark1 and Crunch
>> 3. Fix the problem by ourself (either pulling from upstream or
>develop one)
>>
>> Please read the discussion there if you wanna participate. Note that
>> Crunch haven't do a release since Feb. 2017.
>>
>> Evans
>>
>>
>>

Re: R.I.P. mrdocs (1963–2019)

2019-03-22 Thread Konstantin Boudnik

Rest in peace, Peter - we are going to remember you laughing and
lighthearted, and a good friend you were!
--
  With regards,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622

Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.

On Fri, Mar 22, 2019 at 5:20 PM Roman Shaposhnik  wrote:
>
> It is with incredibly heavy heart that I have to tell you all about this:
> https://www.scribus.net/r-i-p-mrdocs-1963-2019/
>
> A lot of us Bigtop folks knew Peter in person and he was every bit
> the kind of man that made Open Source what it is today. I'm rattled
> and saddened to hear of his passing. My condolences to his family
> and everyone around here who knew him.
>
> Thanks,
> Roman.

Re: [DISCUSS] Jump to 1.4.0 release directly instead of 1.3.1

2019-03-17 Thread Konstantin Boudnik

First draft of the patch is here
  https://issues.apache.org/jira/browse/BIGTOP-3192
Would appreciate any comments. Thanks!
--
  With regards,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622

Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.

On Sun, Mar 17, 2019 at 12:27 PM Evans Ye  wrote:
>
> That'd be great!
>
> Konstantin Boudnik  於 2019年3月17日 週日 下午9:23寫道：
>
> > Yup, indeed I do ;) Will give it a try tonight...
> > --
> >   With regards,
> > Konstantin (Cos) Boudnik
> > 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
> >
> > Disclaimer: Opinions expressed in this email are those of the author,
> > and do not necessarily represent the views of any company the author
> > might be affiliated with at the moment of writing.
> >
> > On Sun, Mar 17, 2019 at 9:15 AM Olaf Flebbe  wrote:
> > >
> > > Hi Cos,
> > >
> > > if you are answering to
> > > my question regarding removal of spark1: yes please.
> > >
> > > Olaf
> > >
> > > > Am 17.03.2019 um 13:10 schrieb Konstantin Boudnik :
> > > >
> > > > Honestly, I think it is long overdue... I don't think we should waste
> > > > any resources on it. I can do a quick patch to get rid of it in
> > > > master, if there's no objections.
> > > > --
> > > >  With regards,
> > > > Konstantin (Cos) Boudnik
> > > > 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
> > > >
> > > > Disclaimer: Opinions expressed in this email are those of the author,
> > > > and do not necessarily represent the views of any company the author
> > > > might be affiliated with at the moment of writing.
> > > >
> > > >> On Wed, Mar 13, 2019 at 1:48 AM Olaf Flebbe  wrote:
> > > >>
> > > >> Hi,
> > > >>
> > > >> We learned the hard way spark1 vs spark2 . Can we drop spark1 now?
> > What do you think?
> > > >>
> > > >> olaf
> > > >>
> > > >>> Am 13.03.2019 um 06:16 schrieb Konstantin Boudnik :
> > > >>>
> > > >>> While I am all for the latest and greatest, I learned to be cautious
> > > >>> when it gets to the software. Especially, when we look at something
> > > >>> considering itself on the "top of the stack". Hence I welcome your
> > the
> > > >>> conservative approach!
> > > >>> --
> > > >>> With regards,
> > > >>> Konstantin (Cos) Boudnik
> > > >>> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
> > > >>>
> > > >>> Disclaimer: Opinions expressed in this email are those of the author,
> > > >>> and do not necessarily represent the views of any company the author
> > > >>> might be affiliated with at the moment of writing.
> > > >>>
> > > >>>> On Wed, Mar 13, 2019 at 5:58 AM Evans Ye 
> > wrote:
> > > >>>>
> > > >>>> Thank you all for the support!
> > > >>>>
> > > >>>> Until now, I was dealing with various compatibility issue for Spark.
> > > >>>> However there're too many incompatible components with spark 2.4.0:
> > > >>>>
> > > >>>>
> > https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/
> > > >>>>
> > > >>>> Therefore I'd like to revoke the 2.4.0 upgrade and just slightly
> > bump it up
> > > >>>> to 2.2.3.
> > > >>>> The new release date is targeted at the end of March.
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> Jun HE  於 2019年3月6日 週三 下午9:28寫道：
> > > >>>>
> > > >>>>> Awesome, and huge +1 to this!
> > > >>>>>
> > > >>>>> Konstantin Boudnik  于2019年3月6日周三 下午9:11写道：
> > > >>>>>
> > > >>>>>> Oh my, that's amazing! Thank you man!
> > > >>>>>> +1 on moving directly to 1.4
> > > >>>>>> --
> > > >>>>>> With regards,
> > > >>>>>> Konstantin
> > > >>>>>>
> > >

[jira] [Created] (BIGTOP-3192) Remove Spark 1.6 from the stack

2019-03-17 Thread Konstantin Boudnik (JIRA)

Konstantin Boudnik created BIGTOP-3192:
--

 Summary: Remove Spark 1.6 from the stack
 Key: BIGTOP-3192
 URL: https://issues.apache.org/jira/browse/BIGTOP-3192
 Project: Bigtop
  Issue Type: Sub-task
  Components: spark
Affects Versions: 1.3.0
Reporter: Konstantin Boudnik
Assignee: Konstantin Boudnik
 Fix For: 1.4.0


Let's get rid of the outdated version of Spark. There's not much need to 
support it going forward. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] Jump to 1.4.0 release directly instead of 1.3.1

2019-03-17 Thread Konstantin Boudnik

Yup, indeed I do ;) Will give it a try tonight...
--
  With regards,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622

Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.

On Sun, Mar 17, 2019 at 9:15 AM Olaf Flebbe  wrote:
>
> Hi Cos,
>
> if you are answering to
> my question regarding removal of spark1: yes please.
>
> Olaf
>
> > Am 17.03.2019 um 13:10 schrieb Konstantin Boudnik :
> >
> > Honestly, I think it is long overdue... I don't think we should waste
> > any resources on it. I can do a quick patch to get rid of it in
> > master, if there's no objections.
> > --
> >  With regards,
> > Konstantin (Cos) Boudnik
> > 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
> >
> > Disclaimer: Opinions expressed in this email are those of the author,
> > and do not necessarily represent the views of any company the author
> > might be affiliated with at the moment of writing.
> >
> >> On Wed, Mar 13, 2019 at 1:48 AM Olaf Flebbe  wrote:
> >>
> >> Hi,
> >>
> >> We learned the hard way spark1 vs spark2 . Can we drop spark1 now? What do 
> >> you think?
> >>
> >> olaf
> >>
> >>> Am 13.03.2019 um 06:16 schrieb Konstantin Boudnik :
> >>>
> >>> While I am all for the latest and greatest, I learned to be cautious
> >>> when it gets to the software. Especially, when we look at something
> >>> considering itself on the "top of the stack". Hence I welcome your the
> >>> conservative approach!
> >>> --
> >>> With regards,
> >>> Konstantin (Cos) Boudnik
> >>> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
> >>>
> >>> Disclaimer: Opinions expressed in this email are those of the author,
> >>> and do not necessarily represent the views of any company the author
> >>> might be affiliated with at the moment of writing.
> >>>
> >>>> On Wed, Mar 13, 2019 at 5:58 AM Evans Ye  wrote:
> >>>>
> >>>> Thank you all for the support!
> >>>>
> >>>> Until now, I was dealing with various compatibility issue for Spark.
> >>>> However there're too many incompatible components with spark 2.4.0:
> >>>>
> >>>> https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/
> >>>>
> >>>> Therefore I'd like to revoke the 2.4.0 upgrade and just slightly bump it 
> >>>> up
> >>>> to 2.2.3.
> >>>> The new release date is targeted at the end of March.
> >>>>
> >>>>
> >>>>
> >>>> Jun HE  於 2019年3月6日 週三 下午9:28寫道：
> >>>>
> >>>>> Awesome, and huge +1 to this!
> >>>>>
> >>>>> Konstantin Boudnik  于2019年3月6日周三 下午9:11写道：
> >>>>>
> >>>>>> Oh my, that's amazing! Thank you man!
> >>>>>> +1 on moving directly to 1.4
> >>>>>> --
> >>>>>> With regards,
> >>>>>> Konstantin
> >>>>>>
> >>>>>> On March 5, 2019 12:58:44 AM GMT+03:00, Olaf Flebbe 
> >>>>> wrote:
> >>>>>>> Hi Evans,
> >>>>>>>
> >>>>>>> You found and fixed a lot of bugs and made the smoke tests work, so a
> >>>>>>> big +1 .
> >>>>>>>
> >>>>>>> Olaf
> >>>>>>>
> >>>>>>>> Am 04.03.2019 um 08:12 schrieb Evans Ye :
> >>>>>>>>
> >>>>>>>> -1 welcomed as well !!!
> >>>>>>>>
> >>>>>>>> Evans Ye  於 2019年3月4日 週一 下午3:11寫道：
> >>>>>>>>
> >>>>>>>>> Hi all,
> >>>>>>>>>
> >>>>>>>>> The overhaul as well as development is almost completed. With
> >>>>>>> several new
> >>>>>>>>> features added, I'd like to propose to release Bigtop 1.4.0 directly
> >>>>>>>>> instead of doing 1.3.1 release(https://s.apache.org/2a8K). This
> >>>>>>> helps to
> >>>>>>>>> promote these new features:
> >>>>>>>>>
> >>>>>>>>> 1. Full support to build and test inside docker with one stop
> >>>>>>> seamlessly
> >>>>>>>>> integration.
> >>>>>>>>> Now just one command to build and test components:
> >>>>>>>>> Example:
> >>>>>>>>> $ ./gradlew spark-ind repo-ind docker-provisioner -POS=ubuntu-16.04
> >>>>>>>>> -Pnexus -Penable_local_repo -Pstack=spark-standalone
> >>>>>>> -Psmoke-tests=spark
> >>>>>>>>>
> >>>>>>>>> 2. Support to build from git commit hash
> >>>>>>>>> Example:
> >>>>>>>>> $ ./gradlew kafka-pkg-ind
> >>>>>>> -Pgit_repo=https://github.com/apache/kafka.git
> >>>>>>>>> -Pgit_ref=1.1 -Pgit_sha1=4dae083af486eaedd27c69c973c74605bffd416b
> >>>>>>>>> -Pbase_version=1.1.1
> >>>>>>>>>
> >>>>>>>>> 3. Version bumps and tests
> >>>>>>>>> Hadoop 2.8.5, Kafka 1.1.1, Spark 2.4.0, Alluxio 1.8.1
> >>>>>>>>> New smoke tests: Flink, Giraph (Crunch)
> >>>>>>>>>
> >>>>>>>>> 4. Lts of bug fixes!
> >>>>>>>>>
> >>>>>>>>> If agreed, please +1 and I'll still serve as the RM for 1.4 release,
> >>>>>>> if no
> >>>>>>>>> preemption ;)
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Best,
> >>>>>>>>> Evans
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>>

Re: [DISCUSS] Jump to 1.4.0 release directly instead of 1.3.1

2019-03-17 Thread Konstantin Boudnik

Honestly, I think it is long overdue... I don't think we should waste
any resources on it. I can do a quick patch to get rid of it in
master, if there's no objections.
--
  With regards,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622

Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.

On Wed, Mar 13, 2019 at 1:48 AM Olaf Flebbe  wrote:
>
> Hi,
>
> We learned the hard way spark1 vs spark2 . Can we drop spark1 now? What do 
> you think?
>
> olaf
>
> > Am 13.03.2019 um 06:16 schrieb Konstantin Boudnik :
> >
> > While I am all for the latest and greatest, I learned to be cautious
> > when it gets to the software. Especially, when we look at something
> > considering itself on the "top of the stack". Hence I welcome your the
> > conservative approach!
> > --
> >  With regards,
> > Konstantin (Cos) Boudnik
> > 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
> >
> > Disclaimer: Opinions expressed in this email are those of the author,
> > and do not necessarily represent the views of any company the author
> > might be affiliated with at the moment of writing.
> >
> >> On Wed, Mar 13, 2019 at 5:58 AM Evans Ye  wrote:
> >>
> >> Thank you all for the support!
> >>
> >> Until now, I was dealing with various compatibility issue for Spark.
> >> However there're too many incompatible components with spark 2.4.0:
> >>
> >> https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/
> >>
> >> Therefore I'd like to revoke the 2.4.0 upgrade and just slightly bump it up
> >> to 2.2.3.
> >> The new release date is targeted at the end of March.
> >>
> >>
> >>
> >> Jun HE  於 2019年3月6日 週三 下午9:28寫道：
> >>
> >>> Awesome, and huge +1 to this!
> >>>
> >>> Konstantin Boudnik  于2019年3月6日周三 下午9:11写道：
> >>>
> >>>> Oh my, that's amazing! Thank you man!
> >>>> +1 on moving directly to 1.4
> >>>> --
> >>>> With regards,
> >>>>  Konstantin
> >>>>
> >>>> On March 5, 2019 12:58:44 AM GMT+03:00, Olaf Flebbe 
> >>> wrote:
> >>>>> Hi Evans,
> >>>>>
> >>>>> You found and fixed a lot of bugs and made the smoke tests work, so a
> >>>>> big +1 .
> >>>>>
> >>>>> Olaf
> >>>>>
> >>>>>> Am 04.03.2019 um 08:12 schrieb Evans Ye :
> >>>>>>
> >>>>>> -1 welcomed as well !!!
> >>>>>>
> >>>>>> Evans Ye  於 2019年3月4日 週一 下午3:11寫道：
> >>>>>>
> >>>>>>> Hi all,
> >>>>>>>
> >>>>>>> The overhaul as well as development is almost completed. With
> >>>>> several new
> >>>>>>> features added, I'd like to propose to release Bigtop 1.4.0 directly
> >>>>>>> instead of doing 1.3.1 release(https://s.apache.org/2a8K). This
> >>>>> helps to
> >>>>>>> promote these new features:
> >>>>>>>
> >>>>>>> 1. Full support to build and test inside docker with one stop
> >>>>> seamlessly
> >>>>>>> integration.
> >>>>>>> Now just one command to build and test components:
> >>>>>>> Example:
> >>>>>>> $ ./gradlew spark-ind repo-ind docker-provisioner -POS=ubuntu-16.04
> >>>>>>> -Pnexus -Penable_local_repo -Pstack=spark-standalone
> >>>>> -Psmoke-tests=spark
> >>>>>>>
> >>>>>>> 2. Support to build from git commit hash
> >>>>>>> Example:
> >>>>>>> $ ./gradlew kafka-pkg-ind
> >>>>> -Pgit_repo=https://github.com/apache/kafka.git
> >>>>>>> -Pgit_ref=1.1 -Pgit_sha1=4dae083af486eaedd27c69c973c74605bffd416b
> >>>>>>> -Pbase_version=1.1.1
> >>>>>>>
> >>>>>>> 3. Version bumps and tests
> >>>>>>> Hadoop 2.8.5, Kafka 1.1.1, Spark 2.4.0, Alluxio 1.8.1
> >>>>>>> New smoke tests: Flink, Giraph (Crunch)
> >>>>>>>
> >>>>>>> 4. Lts of bug fixes!
> >>>>>>>
> >>>>>>> If agreed, please +1 and I'll still serve as the RM for 1.4 release,
> >>>>> if no
> >>>>>>> preemption ;)
> >>>>>>>
> >>>>>>>
> >>>>>>> Best,
> >>>>>>> Evans
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>
> >>>

Re: [DISCUSS] Jump to 1.4.0 release directly instead of 1.3.1

2019-03-12 Thread Konstantin Boudnik

While I am all for the latest and greatest, I learned to be cautious
when it gets to the software. Especially, when we look at something
considering itself on the "top of the stack". Hence I welcome your the
conservative approach!
--
  With regards,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622

Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.

On Wed, Mar 13, 2019 at 5:58 AM Evans Ye  wrote:
>
> Thank you all for the support!
>
> Until now, I was dealing with various compatibility issue for Spark.
> However there're too many incompatible components with spark 2.4.0:
>
> https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/
>
> Therefore I'd like to revoke the 2.4.0 upgrade and just slightly bump it up
> to 2.2.3.
> The new release date is targeted at the end of March.
>
>
>
> Jun HE  於 2019年3月6日 週三 下午9:28寫道：
>
> > Awesome, and huge +1 to this!
> >
> > Konstantin Boudnik  于2019年3月6日周三 下午9:11写道：
> >
> > > Oh my, that's amazing! Thank you man!
> > > +1 on moving directly to 1.4
> > > --
> > > With regards,
> > >   Konstantin
> > >
> > > On March 5, 2019 12:58:44 AM GMT+03:00, Olaf Flebbe 
> > wrote:
> > > >Hi Evans,
> > > >
> > > >You found and fixed a lot of bugs and made the smoke tests work, so a
> > > >big +1 .
> > > >
> > > >Olaf
> > > >
> > > >> Am 04.03.2019 um 08:12 schrieb Evans Ye :
> > > >>
> > > >> -1 welcomed as well !!!
> > > >>
> > > >> Evans Ye  於 2019年3月4日 週一 下午3:11寫道：
> > > >>
> > > >>> Hi all,
> > > >>>
> > > >>> The overhaul as well as development is almost completed. With
> > > >several new
> > > >>> features added, I'd like to propose to release Bigtop 1.4.0 directly
> > > >>> instead of doing 1.3.1 release(https://s.apache.org/2a8K). This
> > > >helps to
> > > >>> promote these new features:
> > > >>>
> > > >>> 1. Full support to build and test inside docker with one stop
> > > >seamlessly
> > > >>> integration.
> > > >>> Now just one command to build and test components:
> > > >>> Example:
> > > >>> $ ./gradlew spark-ind repo-ind docker-provisioner -POS=ubuntu-16.04
> > > >>> -Pnexus -Penable_local_repo -Pstack=spark-standalone
> > > >-Psmoke-tests=spark
> > > >>>
> > > >>> 2. Support to build from git commit hash
> > > >>> Example:
> > > >>> $ ./gradlew kafka-pkg-ind
> > > >-Pgit_repo=https://github.com/apache/kafka.git
> > > >>> -Pgit_ref=1.1 -Pgit_sha1=4dae083af486eaedd27c69c973c74605bffd416b
> > > >>> -Pbase_version=1.1.1
> > > >>>
> > > >>> 3. Version bumps and tests
> > > >>> Hadoop 2.8.5, Kafka 1.1.1, Spark 2.4.0, Alluxio 1.8.1
> > > >>> New smoke tests: Flink, Giraph (Crunch)
> > > >>>
> > > >>> 4. Lts of bug fixes!
> > > >>>
> > > >>> If agreed, please +1 and I'll still serve as the RM for 1.4 release,
> > > >if no
> > > >>> preemption ;)
> > > >>>
> > > >>>
> > > >>> Best,
> > > >>> Evans
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > >
> >

Re: [DISCUSS] Jump to 1.4.0 release directly instead of 1.3.1

2019-03-06 Thread Konstantin Boudnik

Oh my, that's amazing! Thank you man!
+1 on moving directly to 1.4
-- 
With regards,
  Konstantin

On March 5, 2019 12:58:44 AM GMT+03:00, Olaf Flebbe  wrote:
>Hi Evans,
>
>You found and fixed a lot of bugs and made the smoke tests work, so a
>big +1 . 
>
>Olaf
>
>> Am 04.03.2019 um 08:12 schrieb Evans Ye :
>> 
>> -1 welcomed as well !!!
>> 
>> Evans Ye  於 2019年3月4日 週一 下午3:11寫道：
>> 
>>> Hi all,
>>> 
>>> The overhaul as well as development is almost completed. With
>several new
>>> features added, I'd like to propose to release Bigtop 1.4.0 directly
>>> instead of doing 1.3.1 release(https://s.apache.org/2a8K). This
>helps to
>>> promote these new features:
>>> 
>>> 1. Full support to build and test inside docker with one stop
>seamlessly
>>> integration.
>>> Now just one command to build and test components:
>>> Example:
>>> $ ./gradlew spark-ind repo-ind docker-provisioner -POS=ubuntu-16.04
>>> -Pnexus -Penable_local_repo -Pstack=spark-standalone
>-Psmoke-tests=spark
>>> 
>>> 2. Support to build from git commit hash
>>> Example:
>>> $ ./gradlew kafka-pkg-ind
>-Pgit_repo=https://github.com/apache/kafka.git
>>> -Pgit_ref=1.1 -Pgit_sha1=4dae083af486eaedd27c69c973c74605bffd416b
>>> -Pbase_version=1.1.1
>>> 
>>> 3. Version bumps and tests
>>> Hadoop 2.8.5, Kafka 1.1.1, Spark 2.4.0, Alluxio 1.8.1
>>> New smoke tests: Flink, Giraph (Crunch)
>>> 
>>> 4. Lts of bug fixes!
>>> 
>>> If agreed, please +1 and I'll still serve as the RM for 1.4 release,
>if no
>>> preemption ;)
>>> 
>>> 
>>> Best,
>>> Evans
>>> 
>>> 
>>> 
>>>

Re: [DISCUSS] Upgrade to Puppet 5.X

2019-02-22 Thread Konstantin Boudnik

Here's a provocative idea (and believe me I am not offering it
lightly): perhaps there's another deployment system that would have a
better distro representation and will be easier to maintain?
--
  With regards,
Konstantin (Cos) Boudnik

On Sat, Feb 16, 2019 at 10:38 PM Evans Ye  wrote:
>
> I can recall you said not having puppet on suse. So agree if we need to
> drop suse. Instead of having things broken, I prefer to have small set of
> features but working perfectly.
>
> Upgrade to Puppet 5.X seems to be too aggressive. I'll drop the proposal.
>
> Olaf Flebbe  於 2019年2月16日 週六 下午4:44寫道：
>
> > Hi all,
> >
> > some may recall we had this discussion before:
> >
> > I voted to stay at the distro provided packages and modules, since
> > puppetlabs distro does not support our core target of being
> > arch-independent.
> >
> > However, the downside of puppet in general is that we lose recent opensuse
> > leap ( no support from puppetlabs  nor distro)
> >
> > There are other / additional options, though:
> >
> > 3) remove distros without proper puppet support. If someone complains, ask
> > him/her to go for options 4
> >
> > My vote was to remove opensuse from our build matrix.
> >
> > 4a) Building ruby and puppet ourselfs and providing packages for all
> > platforms.
> >
> > 4b) Work with distro upstream to support a decent version (see opensuse
> > build service as an example)
> >
> > 5) switch from puppet to a different configmanagment/automation tool: This
> > has to be carefully planned, since we will likely have the same
> > version/arch/distro provided problem anyway.
> >
> >
> > my 2cents
> >
> > olaf
> >
> >
> > > Am 14.02.2019 um 09:15 schrieb Evans Ye :
> > >
> > > Hi folks,
> > >
> > > Recently I spend some effort trying to upgrade to Puppet 5.X[1]. However
> > as
> > > what Olaf pointed out: Puppet 5 is not available for AARCH64 or PPC64LE
> > > except for EL7.  And it seems that Puppet have not intention to add cross
> > > platform support.
> > > *
> > >
> > https://puppet.com/docs/puppet/5.5/system_requirements.html#platforms-without-packages
> > > * https://tickets.puppetlabs.com/browse/PA-1330
> > >
> > > I think here we have two options:
> > > 1.  Upgrade to Puppet to 5.X as much as we can, as what we did for
> > others.
> > > For example, we don't have full set of Distros supported on AARCH64. The
> > > same idea goes here. We don't have Puppet 5 on all the supported archs.
> > But
> > > we'll ensure what we provide still works well. For example Puppet 3.X or
> > > 4.X still works fine with Bigtop on AARCH64.
> > > 2. Discard this upgrade and relies on Distro's version of puppet. Not a
> > > good sign in software though. When the gap become too big. We'll spend a
> > > lot of effort to catch up...
> > >
> > > I'd like to take option 2. What do you think?
> > >
> > > - Evans
> >

Re: [DISCUSS] Next minor Release of Apache Bigtop

2019-02-05 Thread Konstantin Boudnik

Great idea, I'll try to help as much as my time permits. Thanks for 
volunteering! 
--
Regards,
  Cos

On February 3, 2019 2:04:34 PM GMT+03:00, Evans Ye  wrote:
>Hi all,
>
>I'd like to propose a next minor release of Apache Bigtop, which is
>1.3.1.
>
>Although Bigtop 1.3.0 was released back in Nov. 2018. It's a good sign
>for
>a project to release early, release often.
>
>The release will contain the following features:
>1. Overhaul the deployment and testing modules
>2. Project Frontier: Bigtop Integration Test Framework 2.0
>3. Minor upgrade to components such as: Hadoop, Spark, Kafka, etc
>4. Other bug fixes
>
>This minor release is more focus on the quality end instead of catching
>up
>to latest version of components.
>The plan is to have the RC out around mid February, 2019.
>
>If the community is up to the release, I volunteer to be the RM.
>However I'll yield if someone would like to take the role :)
>
>
>- Evans

Re: [VOTE] Move to gitbox

2019-01-04 Thread Konstantin Boudnik

+1
--
Regards,
  Cos

On January 3, 2019 4:24:50 PM AST, Olaf Flebbe  wrote:
>Hi,
>
>let see if we have consensus to move to gitbox early.
>
>More information on thread 
>https://lists.apache.org/thread.html/b3868092e86d3ffa3abec7f97dbc8272518c63e57232e5386c103f19@%3Cdev.bigtop.apache.org%3E
>
>Please vote by reply to this email until  Sun., 6th of January 9pm CET 
>
>+1: Let's move to gitbox early
>-1: Let's wait
>0 : don't care
>
>
>Olaf Flebbe

Re: [VOTE] Release Bigtop version 1.3.0

2018-11-22 Thread Konstantin Boudnik

Good stuff! Thanks a lot!
--
  With regards,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622

Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.
On Tue, Nov 20, 2018 at 9:23 AM Jun HE  wrote:
>
> Hi, folks,
>
> Just give a quick update here.
>
> With Evans's help, maven artifacts, source packages and repos are in
> position now. Tag (rel/1.3.0) is added.
> Now I'm waiting for all the mirrors to sync. Hopefully I'll deploy site and
> send out release announcment tomorrow.
>
> This would not happen without your kind help and support during the whole
> process. Really appreciated!
>
> Regards,
>
> Jun
>
> Jun HE  于2018年11月15日周四 下午12:48写道：
>
> > Thanks Evans for clarify this. :)
> > Here is my vote:
> > +1 [binding]: Checked signature, checksum, smoke tests fo basic components
> > with several distros.
> >
> > So, here is the vote result:
> > With four binding +1s, no non-binding +1, and no -1, or +-0 votes.
> > The vote for 1.3.0 release PASSES.
> >
> > Binding +1s:
> >   Olaf Flebbe
> >   Konstantin Boudnik
> >   Evans Ye
> >   Jun He
> >
> >
> > Thank you all for voting, testing, and giving feedback to make this
> > release happen.
> > I'll start to prepare the formal release and roll it out shortly.
> >
> > Regards,
> >
> > Jun
> >
> > Evans Ye  于2018年11月15日周四 上午10:20写道：
> >
> >> Hi Jun,
> >>
> >> I guess you are referring your own vote for 3+1 votes. Yes, your vote is
> >> also a binding one.
> >> Please wrap up the vote formally as any vote we have done previously.
> >> Thanks!
> >>
> >> Best,
> >> Evans
> >>
> >> Jun HE  於 2018年11月14日 週三 下午7:10寫道：
> >>
> >> > Hi, folks,
> >> >
> >> > Thanks for your prompt response and info. I think I missed Olaf's vote.
> >> > Sorry.
> >> > Yes, we do have 3 +1 now and I'll star to proceed formal release. Hope
> >> it
> >> > can be finished in this week.
> >> >
> >> > Thanks again!
> >> >
> >> > Regards,
> >> >
> >> > Jun
> >> >
> >> > Konstantin Boudnik  于2018年11月14日周三 下午4:38写道：
> >> >
> >> > > Huge +1 here! More people are looking into the testing of the bits
> >> > > we're offering to our users - better the quality will be (speak like
> >> > > Yoda I do ;)
> >> > > --
> >> > >   With regards,
> >> > > Konstantin (Cos) Boudnik
> >> > > 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
> >> > >
> >> > > Disclaimer: Opinions expressed in this email are those of the author,
> >> > > and do not necessarily represent the views of any company the author
> >> > > might be affiliated with at the moment of writing.
> >> > >
> >> > >
> >> > > On Wed, Nov 14, 2018 at 9:38 AM, Evans Ye  wrote:
> >> > > > Yes. I also encourage not only PMC, but also committers and users to
> >> > test
> >> > > > out the release candidate.
> >> > > > No matter binding or non-binding, the input are valuable.
> >> > > >
> >> > > > BTW, I checked the rule[1]. We've reached the ASF criterial for a
> >> > > release,
> >> > > > which is at least three votes from PMC.
> >> > > > So, if you as the RM want to hear more from the community, we can
> >> set
> >> > > > another deadline.
> >> > > > Afterwards we can proceed to the official release and the
> >> announcement.
> >> > > >
> >> > > > [1]
> >> > http://www.apache.org/legal/release-policy.html#approving-a-release
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > Jun HE  於 2018年11月14日 週三 下午1:09寫道：
> >> > > >
> >> > > >> Thanks for testing and voting, Cos and Evans.
> >> > > >>
> >> > > >> So far this release still didn't get enough votes. Could others pls
> >> > > help to
> >> > > >> check and vote? Thanks a lot for your kind support!
> >> > > >>
> >> > > >> Regards,
> >&

Re: [ANNOUNCE] YoungWoo Kim is now the chair of Apache Bigtop

2018-11-22 Thread Konstantin Boudnik

Congratulations YoungWoo!  There's a nice pile of things to do and now
you have a shovel ! ;) Good luck!
And we are here to help, of course!
--
  With regards,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622

Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.
On Thu, Nov 22, 2018 at 9:01 PM Evans Ye  wrote:
>
> Hi all,
>
> With ASF board's approval on November's meeting, YoungWoo Kim is now 
> officially appointed as the chair of Apache Bigtop. Please join me in 
> congratulating YoungWoo. Meanwhile, thanks to our previous chair Peter's 
> service which makes the community better than ever.
>
> Evans

Re: Returned post for annou...@apache.org

2018-11-22 Thread Konstantin Boudnik

Agree.

--
  With regards,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622

Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.

On Thu, Nov 22, 2018 at 9:20 PM Evans Ye  wrote:
>
> Let me add this to the dev list cause I see this can be public :)
>
> Yes I think updating download page as what you referred is the best choice.
> More specifically, having something similar to Hadoop[1] and slightly
> simplified is enough.
> I don't think we need to list all the releases from the very beginning of
> Bigtop.
> We can list available releases started from 1.1.0.
>
> [1] https://hadoop.apache.org/releases.htm
>
> Jun HE  於 2018年11月22日 週四 上午11:03寫道：
>
> > Hi, folks,
> >
> > Seems Apache release process has been updated, resulted in download page (
> > http://bigtop.apache.org/download.html#releases) needs to be updated to
> > comply with these requirements.
> > For 1.3.0 I'd like to modify
> > https://github.com/apache/bigtop/blob/branch-1.3/src/site/xdoc/download.xml
> > directly with all required fields. Meanwhile we can discuss how to reflect
> > these mandatory requirements to master branch.
> > What's your thought?
> >
> > Regards,
> >
> > Jun
> >
> > -- Forwarded message -
> > From: 
> > Date: 2018年11月21日周三 下午7:06
> > Subject: Returned post for annou...@apache.org
> > To: 
> >
> >
> >
> > Hi! This is the ezmlm program. I'm managing the
> > annou...@apache.org mailing list.
> >
> > I'm sorry, your message (enclosed) was not accepted by the moderator.
> > If the moderator has made any comments, they are shown below.
> >
> > >  >
> > This announcement was rejected because it does not conform to standards.
> >
> > The announcement should contain a link to the bigtop download page, not to
> > the dyn/closer direct download.
> > The download page should contain links to all current and archived
> > releases along with links to KEYS, checksums, and signatures for all
> > releases.
> >
> > Once the download page has been fixed and the announcement changed to
> > refer to the download page, please resubmit the announcement.
> > <  <
> >
> >
> >
> >
> > -- Forwarded message --
> > From: Jun HE 
> > To: annou...@apache.org, u...@bigtop.apache.org, dev@bigtop.apache.org
> > Cc:
> > Bcc:
> > Date: Wed, 21 Nov 2018 12:58:17 +0800
> > Subject: [ANNOUNCE] Apache Bigtop 1.3.0 released
> > On behalf of the Apache Bigtop team, I'd love to announce the general
> > availability of the Bigtop 1.3.0 release.
> >
> > The release is available here:
> > https://www.apache.org/dyn/closer.lua/bigtop/bigtop-1.3.0/
> >
> > A few highlights of this release include:
> >
> >- AArch64 architecture is added to the support matrix
> >- updated distributions (centos-7, fedora-26, debian-9, ubuntu-16.04,
> >opensuse-42.3)
> >- new docker images are available for distributions on different
> >architectures
> >- upgraded toolchain to include new tools and latest versions updates
> >- many upgrades to the latest versions of the ecosystem projects
> >(Hadoop, HBase, Hive, Spark, Solr, Flume, Kafka, GPDB and many others)
> >- improved multi-arch support in several components (hadoop, hbase,
> >qfs, ignite-hadoop and etc)
> >- improvements in the deployment for unattended cluster
> >- improvements in the smoke-tests coverage
> >
> > With Bigtop 1.3.0 the community continues to deliver the most advanced big
> > data stack to date. More details about 1.3.0 release are here:
> > http://bigtop.apache.org/release-notes.html
> >
> > Deploying Bigtop is easy: grab the repo/list file for your favorite Linux
> > distribution:
> >   https://www.apache.org/dyn/closer.lua/bigtop/bigtop-1.3.0/repos/
> > and you'll be running your very own bigdata cluster in no time!
> >
> > We welcome your help and feedback. For more information on how to report
> > problems, and to get involved, visit the project website at:
> > http://bigtop.apache.org
> >
> > I want to emphasize that this is a collaborative work done by project
> > contributors and other communities, who continue to devote time to make
> > Bigtop a better software. Thank you all for making this release possible!
> >
> > Thanks,
> >
> > Jun He (Bigtop 1.3.0 Release Manager)
> >

Re: [VOTE] Release Bigtop version 1.3.0

2018-11-14 Thread Konstantin Boudnik

Huge +1 here! More people are looking into the testing of the bits
we're offering to our users - better the quality will be (speak like
Yoda I do ;)
--
  With regards,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622

Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.


On Wed, Nov 14, 2018 at 9:38 AM, Evans Ye  wrote:
> Yes. I also encourage not only PMC, but also committers and users to test
> out the release candidate.
> No matter binding or non-binding, the input are valuable.
>
> BTW, I checked the rule[1]. We've reached the ASF criterial for a release,
> which is at least three votes from PMC.
> So, if you as the RM want to hear more from the community, we can set
> another deadline.
> Afterwards we can proceed to the official release and the announcement.
>
> [1] http://www.apache.org/legal/release-policy.html#approving-a-release
>
>
>
>
> Jun HE  於 2018年11月14日 週三 下午1:09寫道：
>
>> Thanks for testing and voting, Cos and Evans.
>>
>> So far this release still didn't get enough votes. Could others pls help to
>> check and vote? Thanks a lot for your kind support!
>>
>> Regards,
>>
>> Jun
>>
>> Konstantin Boudnik 于2018年11月12日 周一02:36写道：
>>
>> > Thanks for waiting...
>> >
>> > +1 [binding]
>> >
>> > Did basic consistency checks  - everything looks ok.
>> > --
>> >   With regards,
>> > Konstantin (Cos) Boudnik
>> > 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
>> >
>> > Disclaimer: Opinions expressed in this email are those of the author,
>> > and do not necessarily represent the views of any company the author
>> > might be affiliated with at the moment of writing.
>> >
>> >
>> > On Tue, Oct 30, 2018 at 10:31 AM, Jun HE  wrote:
>> > > This is the vote for release 1.3.0 of Apache Bigtop.
>> > >
>> > > It fixes the following issues:
>> > >
>> > >
>> >
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12338976=12311420
>> > >
>> > > The vote will be going for at least 72 hours and will be closed on
>> > Saturday,
>> > > November 3, 2018 at noon PDT.  Please download, test and vote with
>> > >
>> > > [ ] +1, accept RC2 as the official 1.3.0 release of Apache Bigtop
>> > > [ ] +0, I don't care either way,
>> > > [ ] -1, do not accept RC2 as the official 1.3.0 release of Apache
>> Bigtop,
>> > > because...
>> > >
>> > > Source and binary files:
>> > >
>> https://dist.apache.org/repos/dist/dev/bigtop/bigtop-1.3.0-RC2/
>> > >
>> > > Maven staging repo:
>> > >
>> > >
>> https://repository.apache.org/content/repositories/orgapachebigtop-1020
>> > >
>> > > The git tag to be voted upon is release-1.3.0
>> > >
>> > > Bigtop's KEYS file containing PGP keys we use to sign the release:
>> > > https://dist.apache.org/repos/dist/release/bigtop/KEYS
>> >
>>

Re: Packaging things to Docker

2018-11-12 Thread Konstantin Boudnik

well, just to close the loop

I've dived deeper into this whole Hadoop on containers story. And
unfortunately, I have came to the conclusion that it doesn't make that
much sense. Building separate images per a component and orchestrating
it through k8s or Swarm doesn't solve anything, but adds a lot of
hassle. Using this approach as a sort of packaging technique also
doesn't add much to a developer or an admin.

We already have this mechanism where one can create a docker image
with an arbitrary set of components and deploy a cluster using
different images like this. It is good enough for most of the cases
where it makes sense to deploy Hadoop stack from containers.

Hence, I decided to pull off of this project. But it someone else can
thing of a better way doing this sort of things, I would be happy to
join hands.

--
  With regards,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622

Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.


On Thu, Oct 18, 2018 at 2:26 PM, Konstantin Boudnik  wrote:
> Indeed, to be heard you don't need to be a committer: they aren't some sort of
> privileged class here ;)
>
> Anyway, back to this discussion and answering some of the concerns by Evans.
> Tarballs aren't a key requirement for this approach: I was using tarballs
> built from debs to cut some corners and not to change any of the Puppet
> recipes while I am certain my experiment shows something viable. My first
> intention was, of course, to use our packages. But I couldn't think of any
> clever way to avoid pulling all install-time dependencies without a massive
> rewrite of the packages. E.g. it won't be possible to install just Spark
> without pulling in YARN or HDFS dependencies. And will undermine the idea of
> component-specific images (or layers, as I've called them in the OP).
>
> In fact, you know my stance on the whole tarball thing: I've been pushing back
> on parcel-like approach since I can remember. I still think it's a horrible
> idea to produce tarballs as a first-class artifacts. There's plenty of reasons
> for this which are out of the scope of this conversation.
>
> Speaking of use-cases: as both Mikhail and pointed out, it is intended for
> something like Swarm or K8S (basically, anything that can orchestrate
> containers into something meaningful at scale).
>
> Much like Mikhail suggested, mixing base-layers would achieve my idea of
> piling up components on top of each other in order to create different
> special purpose or function roles. I guess it is much like sandbox you've
> mentioned, but without the hassle of creating whole stack for each new
> combination of the components. I will look more closely to the Swarm thing in
> the next day or so.
>
> Thanks guys!
>   Cos
>
>
> On Tue, Oct 16, 2018 at 01:38AM, Evans Ye wrote:
>> To Mikhail:
>> It never has to be committer to join the discuss. Welcome to share any idea
>> you have :)
>>
>> To reply in all:
>> This might be tangential, but I just want to bring in more information.
>> Currently we have Docker Provisioner and Docker Sandbox(experimental)
>> features inside Bigtop.
>> Which:
>> 1. Provisioner: install RPM/DEB via Puppet on the fly when creating cluster
>> 2. Sandbox: pre-install RPM/DEB via Puppet as special purposed stack(say
>> HDFS+SPARK) and save as an images
>>
>> Non of the above go for tarball because they're built to tale around bigtop
>> RPM/DEB packages, which might be the most valuable thing we produce. I
>> don't mean when can't ditch packages, but we have to come up with
>> considerations to cover the whole picture, say:
>>
>> 1. Where does the tarball from? Is it from upstream directly or produced by
>> Bigtop with self-patches for compatibility fixes?
>> 2. If we'd like to support install from tarball, how will the orchestration
>> tool(Puppet) being shaped? Is it going to support both RPM/DEB and tarball
>> or just the new one?
>> 3. What's the purpose of producing docker images in this new way? If we can
>> made it supported to run on K8S, that's a perfect use case!
>>
>> Overall, I champion to have this new feature in Bigtop, but just want to
>> bring up something more for discussion :)
>>
>> Evans
>>
>> Mikhail Epikhin  於 2018年10月15日 週一 下午5:04寫道：
>>
>> > Looks very interesting!
>> >
>> > Sorry for breaking into discussion, i'm not a commiter, just yet another
>> > user, but..
>> >
>> > As you wrote, docker doesn't fit this well.
>> > The problem is that you tried to push all c

Re: [VOTE] Release Bigtop version 1.3.0

2018-11-11 Thread Konstantin Boudnik

Thanks for waiting...

+1 [binding]

Did basic consistency checks  - everything looks ok.
--
  With regards,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622

Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.


On Tue, Oct 30, 2018 at 10:31 AM, Jun HE  wrote:
> This is the vote for release 1.3.0 of Apache Bigtop.
>
> It fixes the following issues:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12338976=12311420
>
> The vote will be going for at least 72 hours and will be closed on Saturday,
> November 3, 2018 at noon PDT.  Please download, test and vote with
>
> [ ] +1, accept RC2 as the official 1.3.0 release of Apache Bigtop
> [ ] +0, I don't care either way,
> [ ] -1, do not accept RC2 as the official 1.3.0 release of Apache Bigtop,
> because...
>
> Source and binary files:
> https://dist.apache.org/repos/dist/dev/bigtop/bigtop-1.3.0-RC2/
>
> Maven staging repo:
>
> https://repository.apache.org/content/repositories/orgapachebigtop-1020
>
> The git tag to be voted upon is release-1.3.0
>
> Bigtop's KEYS file containing PGP keys we use to sign the release:
> https://dist.apache.org/repos/dist/release/bigtop/KEYS

Re: [VOTE] Release Bigtop version 1.3.0

2018-11-04 Thread Konstantin Boudnik

That's be perfect! Thank you guys!
--
  With regards,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622

Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.


On Sun, Nov 4, 2018 at 4:31 PM, Jun HE  wrote:
> Hi, Cos,
>
> Sure. So far we didn't receive enough votes yet. Would it work for you if
> the vote is extended to Nov 10?
>
> Regards,
>
> Jun
>
> Konstantin Boudnik  于2018年11月4日周日 下午2:04写道：
>
>> Can we give it a couple more days? I am away during the weekend, and won't
>> be
>> able to look at the release.
>>
>> Thanks!
>>   Cos
>>
>> On Tue, Oct 30, 2018 at 03:31PM, Jun HE wrote:
>> > This is the vote for release 1.3.0 of Apache Bigtop.
>> >
>> > It fixes the following issues:
>> >
>> >
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12338976=12311420
>> >
>> > The vote will be going for at least 72 hours and will be closed on
>> Saturday,
>> > November 3, 2018 at noon PDT.  Please download, test and vote with
>> >
>> > [ ] +1, accept RC2 as the official 1.3.0 release of Apache Bigtop
>> > [ ] +0, I don't care either way,
>> > [ ] -1, do not accept RC2 as the official 1.3.0 release of Apache Bigtop,
>> > because...
>> >
>> > Source and binary files:
>> > https://dist.apache.org/repos/dist/dev/bigtop/bigtop-1.3.0-RC2/
>> >
>> > Maven staging repo:
>> >
>> > https://repository.apache.org/content/repositories/orgapachebigtop-1020
>> >
>> > The git tag to be voted upon is release-1.3.0
>> >
>> > Bigtop's KEYS file containing PGP keys we use to sign the release:
>> > https://dist.apache.org/repos/dist/release/bigtop/KEYS
>>

Re: [VOTE] Release Bigtop version 1.3.0

2018-11-04 Thread Konstantin Boudnik

Can we give it a couple more days? I am away during the weekend, and won't be
able to look at the release.

Thanks!
  Cos

On Tue, Oct 30, 2018 at 03:31PM, Jun HE wrote:
> This is the vote for release 1.3.0 of Apache Bigtop.
> 
> It fixes the following issues:
> 
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12338976=12311420
> 
> The vote will be going for at least 72 hours and will be closed on Saturday,
> November 3, 2018 at noon PDT.  Please download, test and vote with
> 
> [ ] +1, accept RC2 as the official 1.3.0 release of Apache Bigtop
> [ ] +0, I don't care either way,
> [ ] -1, do not accept RC2 as the official 1.3.0 release of Apache Bigtop,
> because...
> 
> Source and binary files:
> https://dist.apache.org/repos/dist/dev/bigtop/bigtop-1.3.0-RC2/
> 
> Maven staging repo:
> 
> https://repository.apache.org/content/repositories/orgapachebigtop-1020
> 
> The git tag to be voted upon is release-1.3.0
> 
> Bigtop's KEYS file containing PGP keys we use to sign the release:
> https://dist.apache.org/repos/dist/release/bigtop/KEYS


signature.asc
Description: Digital signature

Re: Packaging things to Docker

2018-10-18 Thread Konstantin Boudnik

Indeed, to be heard you don't need to be a committer: they aren't some sort of
privileged class here ;)

Anyway, back to this discussion and answering some of the concerns by Evans.
Tarballs aren't a key requirement for this approach: I was using tarballs
built from debs to cut some corners and not to change any of the Puppet
recipes while I am certain my experiment shows something viable. My first
intention was, of course, to use our packages. But I couldn't think of any
clever way to avoid pulling all install-time dependencies without a massive
rewrite of the packages. E.g. it won't be possible to install just Spark
without pulling in YARN or HDFS dependencies. And will undermine the idea of
component-specific images (or layers, as I've called them in the OP).

In fact, you know my stance on the whole tarball thing: I've been pushing back
on parcel-like approach since I can remember. I still think it's a horrible
idea to produce tarballs as a first-class artifacts. There's plenty of reasons
for this which are out of the scope of this conversation.

Speaking of use-cases: as both Mikhail and pointed out, it is intended for
something like Swarm or K8S (basically, anything that can orchestrate
containers into something meaningful at scale). 

Much like Mikhail suggested, mixing base-layers would achieve my idea of
piling up components on top of each other in order to create different
special purpose or function roles. I guess it is much like sandbox you've
mentioned, but without the hassle of creating whole stack for each new
combination of the components. I will look more closely to the Swarm thing in
the next day or so. 

Thanks guys!
  Cos

On Tue, Oct 16, 2018 at 01:38AM, Evans Ye wrote:
> To Mikhail:
> It never has to be committer to join the discuss. Welcome to share any idea
> you have :)
> 
> To reply in all:
> This might be tangential, but I just want to bring in more information.
> Currently we have Docker Provisioner and Docker Sandbox(experimental)
> features inside Bigtop.
> Which:
> 1. Provisioner: install RPM/DEB via Puppet on the fly when creating cluster
> 2. Sandbox: pre-install RPM/DEB via Puppet as special purposed stack(say
> HDFS+SPARK) and save as an images
> 
> Non of the above go for tarball because they're built to tale around bigtop
> RPM/DEB packages, which might be the most valuable thing we produce. I
> don't mean when can't ditch packages, but we have to come up with
> considerations to cover the whole picture, say:
> 
> 1. Where does the tarball from? Is it from upstream directly or produced by
> Bigtop with self-patches for compatibility fixes?
> 2. If we'd like to support install from tarball, how will the orchestration
> tool(Puppet) being shaped? Is it going to support both RPM/DEB and tarball
> or just the new one?
> 3. What's the purpose of producing docker images in this new way? If we can
> made it supported to run on K8S, that's a perfect use case!
> 
> Overall, I champion to have this new feature in Bigtop, but just want to
> bring up something more for discussion :)
> 
> Evans
> 
> Mikhail Epikhin  於 2018年10月15日 週一 下午5:04寫道：
> 
> > Looks very interesting!
> >
> > Sorry for breaking into discussion, i'm not a commiter, just yet another
> > user, but..
> >
> > As you wrote, docker doesn't fit this well.
> > The problem is that you tried to push all components into one container,
> > and you lost immutability of image.
> > I fully agree and understand this way for production, for more local
> > connectivity, but docker containers doesn't have big difference for run
> > this hive, spark, hdfs on one docker container, or on many different.
> > Anyway, their using network for connectivity, and you compare connectivity
> > inside one container and connectivity between many containers on one local
> > machine.
> >
> > They all run on one single machine, and if you create own container for
> > each component hdfs, hive, spark, hbase, yarn its good fitting to docker
> > model.
> >
> > Futher, you can create environment using docker-compose for mixing this
> > base layers [hdfs, hive, spark, hhbase, ignite] as you wish.
> >
> > Just create a set of base images and templating script for creating
> > docker-compose.yml for connect their.
> >
> > Futher, if you want to simulate many-nodes cluster -- you can do it just
> > writting new docker-compose.yaml. You can test High Availability, HDFS
> > decommission, or anything you want just write your own docker-compose.yaml.
> >
> > --
> > Mikhail Epikhin
> >
> > On Thu, Oct 11, 2018, at 20:25, Konstantin Boudnik wrote:
> > > Well, finally I came around and started working on the long-awaiting
> > feature
> >

Re: Why the dependencies on bigtop-* packages are not reflected in bigtop.bom?

2018-10-17 Thread Konstantin Boudnik

Yes, you can but you'll end up with dependencies built elsewhere by
someone you might not trust in an environment you don't control.
That's the big problem right there. And we want Bigtop stack to be
consistent.

--
  With regards,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622

Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.


On Wed, Oct 17, 2018 at 8:24 PM, Dagang Wei  wrote:
> Thanks Olaf! But how is build time dependencies reflected in Bigtop? e.g.,
> how does HBase depends on Hadoop at build time? If I clone a HBase git
> repo, I can build it without depending on Hadoop source, Hadoop JARs will
> be downloaded from Maven central.
>
> On Tue, Oct 16, 2018 at 10:29 PM Olaf Flebbe  wrote:
>
>> Hi
>>
>> in bigtop.bom we note build time dependencies.
>>
>> package dependencies are for install time and runtime.
>>
>> best regards,
>> olaf
>>
>>
>> > Am 16.10.2018 um 22:57 schrieb Dagang Wei :
>> >
>> > Hi folks,
>> >
>> > I noticed that hadoop depends on bigtop-utils [1]:
>> >
>> >  Depends: ${shlibs:Depends}, ${misc:Depends}, adduser, bigtop-utils (>=
>> > 0.7), zookeeper (>= 3.4.0), psmisc, netcat-openbsd
>> >
>> > But it is not reflected in bigtop.bom [2]:
>> >
>> >  dependencies = [
>> >zookeeper:['hadoop', 'hbase'],
>> >hadoop:['ignite-hadoop', 'hbase', 'crunch', 'hive', 'tez', 'sqoop',
>> > 'sqoop2',
>> >  'oozie', 'mahout', 'flume', 'giraph', 'solr', 'spark','spark1',
>> >  'phoenix', 'alluxio', 'kafka', 'ycsb', 'hama', 'zeppelin',
>> >  'tajo', 'apex'
>> >],
>> >hbase:['phoenix','giraph','ycsb','hive'],
>> >hive:['oozie', 'zeppelin'],
>> >'ignite-hadoop':['zeppelin'],
>> >spark:['zeppelin']
>> >  ]
>> >
>> > Why is that? Should we add it?
>> >
>> > [1]:
>> >
>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/deb/hadoop/control#L26
>> > [2]: https://github.com/apache/bigtop/blob/master/bigtop.bom#L111
>>

Packaging things to Docker

2018-10-11 Thread Konstantin Boudnik

Well, finally I came around and started working on the long-awaiting feature
for Bigtop, where one would be able to quickly build a container with an
arbitrary set of components in it for further orchestration.

The ideas was to have components in different layers, so they could be put
combined together for desired effect. Say there are layers with:
  1 hdfs
  2 hive
  3 spark
  4 hbase
  5 ignite
  6 yarn
and so on

If one wants to assemble a spark only cluster there would be a way to layer up
3 and 1 (ideally, 3's dependency to 1 would be automatically calculated) and
boom - there's an image, which would be put to use. The number of combination
might be greater, of course. E.g. 3-6-1, or 4-2-1-6 and so forth.

Turned out, that I can't "prebuild" those layers as Docker won't allow you to
combine separate images to one ;( However, there's still a way to achieve a
similar effect. All I need to do is to create a set of tar-ball containing all
bits of particular components, i.e. all bits of spark or hive. When an image
needs to be build, these tarballs would be used to layer the software on top
of the base image and each other. In the above example, Dockerfile would look
something like

FROM ubuntu:16.04
ADD hdfs-all.tar /tmp
RUN tar xf /tmp/hdfs-all.tar 
ADD spark-all.tar /tmp
RUN tar xf /tmp/spark-all.tar 

Once the images is generated, the orchestration and configuration phases will
kick in. At which point a docker-based cluster would be all ready to go.

Do you guys see any value in this approach comparing to the current
package-based way of managing things? 

Appreciate any thoughts!
--
  Cos

P.S. BTW, I guess I have a decent answer to all those asking for tar-ball
installation artifacts. It is as easy as running 
dpkg-deb -xv
on all packages and then tar'ing up the resulted set of files.



signature.asc
Description: Digital signature

Re: Bringing AVRO as a dependecy...

2018-10-11 Thread Konstantin Boudnik

Thanks guys, something along these lines were forming in the back of
my head. Good to see sketchy thinking is sensible.
The use-case is pretty straight-forward: let the applications on top
of the stack to read/write Avro format to HBase (or else) without a
need to bundle the serde dependency in each and every case.
--
  With regards,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622

Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.


On Mon, Oct 8, 2018 at 11:39 AM, Roman Shaposhnik  wrote:
> On Sun, Oct 7, 2018 at 11:37 PM Konstantin Boudnik  wrote:
>>
>> Guys,
>>
>> I want to bounce a question off the team: what would be the best way to bring
>> Avro dependency into the stack? I don't think it deserves to be a separate
>> component per se - after all it serves a niche purpose and doesn't stand on
>> its own.
>>
>> However, it makes sense to carry around in case an app needs such a
>> serialization. I've been thinking of perhaps of packing it up as a Hadoop
>> or HBase dependency, but it doesn't seem like the most straight forward idea,
>> to be honest. Or may be we should introduce a separate helper package with
>> serializers and such?
>>
>> What would be a preferred way from your stand point?
>
> The way I see this is two-fold:
>   1. it could be useful to have a standalone pacakge with all the jar
> dependencies
>   so that any component of Bigtop stack to use that instead of
> whatever comes through
>   Maven dependencies. This will be similar to how we standardize on
> zookeper deps
>   and such.
>
>   2. it could be useful to provide binary scripts in that package that
> would automate/facilitate
>   some of these kinds of steps:
>   
> http://avro.apache.org/docs/current/gettingstartedjava.html#Compiling+the+schema
>   http://avro.apache.org/docs/current/gettingstartedpython.html
>   http://avro.apache.org/docs/current/idl.html
>   and anything like that.
>
> Do you have any other usecases in mind, Cos?
>
> Thanks,
> Roman.

Re: Ubuntu 18.04 support

Piling up on Evans' points: doing the work right away might be a good idea.
However, considering the span and potential instability of the changes we
should do it on a branch and only put an essential work on master, that might
improve the overall state of the project. Say, upgrading to Puppet 5 might not
be taken well by CentOS7. And while I don't particularly care about what would
happen to it ;), it is a part of our supported matrix.

Once we think the branch is ready to get merged, we'll bite the bullet and
move to the new "platform". 

Any comments?
Cos

On Sat, Oct 06, 2018 at 01:46AM, Evans Ye wrote:
> Hi Mikhail,
> 
> Thank you for sharing your survey of supporting Ubuntu 18.04 in bigtop. I’d
> like to take a step back and ask some really dumb questions if you dont
> mind ;)
> 
> 1. What’s the trigger that you wanna have support for Ubuntu 18.04? Is it
> easier to make a shift to Ubuntu 18.04 if we wait for it to be more mature?
> 2. The refactor is a total package. So if we upgrade the whole build system
> for Ubuntu 18.04. Does the Debian side or even the RPM side gonna be
> compatible for those OS specific tools?
> 
> Evans
> 
> Mikhail Epikhin 於 2018年10月5日 週五，下午2:03寫道：
> 
> > Hi folks!
> >
> > I wrote this post on github https://github.com/apache/bigtop/pull/374,
> > but, it will be better to move this discussion to mail.
> >
> > TL;DR
> > Just before support ubuntu 18.04 we should do some major things, and
> > before doing it we should reach an agreement:
> >
> > * Support puppet 5, current version is 3.
> > * Rewrite templating and code generation of initrd scripts and support
> > systemd services.
> > * Use aptly insteadof reprepro for debian-based distros?
> >
> > Okey, now longread:
> >
> > Ubuntu 18.04 supports only puppet 5, but in bigtop we are using puppet 3.
> > I've tried to using manifests from third version, but puppet fifth have new
> > keyword like site, and some configs clashes with puppet 5. I fixed that,
> > but, i don't know, maybe some problems are still there. Living with
> > manifests for two versions is not good idea, so we need to choice one of
> > them and migrate all code to newest deploy system.
> >
> > Ubuntu 18.04 have magic converting tool from sysvinit to systemd services,
> > but it doesn't work with current initd scritps. I successfully tried to
> > write some of systemd services, but this way have a couple of shades:
> > a) We should generate this services as we generate initd services, but the
> > current way is a bit of weird. A lot of bash templating, and current "bash
> > templates" a not usable for new systemd services, just because the same
> > things in both ways are very different. Working with locks, pids,
> > controlling aliveness of a process, controlling user and group. I've tried
> > to change current bash templates, but the code looks ugly.
> > b) A lot of components. And each components have a lot of services. I just
> > can't write a templating script for all of this scripts, because every
> > service have own specific things in bash methods, and u cannot convert it
> > automatically, only by reading your eyes.
> > c) Current initrd scripts have custom features like rollingupgrade and
> > init in hdfs-namenode. Systemd doesn't support that, and the right ways is
> > -- create new systemd service only for that! For each custom feature and
> > action -- new systemd service. A lot of services.
> >
> > This is not blocker issue, but the execution of ./gradlew apt on builded
> > components doesn't support *.ddeb artifacts. Newest reprepro doesn't
> > support this files, and the problem looks like this
> > https://github.com/ros-infrastructure/buildfarm_deployment/issues/186 and
> > we have three ways:
> > a) skip these artifacts, weird way;
> > b) use new version of reprepro not from ubuntu repositories, which is also
> > weird;
> > c) use tool like https://www.aptly.info, which is more supportable and
> > powerful in features.
> >
> > If anyone have a time for support 18.04 it would be great for doing it
> > together with splitting this pull request to sub tasks and discussion it.
> >
> > So, anyone?:)
> >
> > ---
> > Mikhail Epikhin
> >


signature.asc
Description: Digital signature

Re: [VOTE] Release Bigtop version 1.3.0

So, looks like the vote needs to be stopped until RC2 is ready, right?
Otherwise, things will become confusing...

Cos

On Wed, Oct 03, 2018 at 07:07PM, Jun HE wrote:
> I've tested CentOS-7 with 4.12.0 puppetlabs-stdlib, will submit patch soon.
> For Debian, I'll update the config file as you suggested, and centos/fedora
> config as well.
> They should be ready in today for review.
> 
> Evans Ye  于2018年10月3日周三 下午1:48写道：
> 
> > For provisioner part, I tested CentOS, Ubuntu, and Debian.
> > Currently only Ubuntu works.
> > * CentOS: Need to pin down the puppetlabs-stdlib to 4.12.0 as Jun He
> > suggested
> > * Debian: Just need to update config file from debian 8 to debian 9.
> >
> > Jun are you already working on this or you need my help fixing them?
> >
> >
> > Jun HE  於 2018年9月30日 週日 下午2:30寫道：
> >
> > > Hi Olaf,
> > >
> > > For debian provision test, use bigtop/puppet:1.3.0-debian-9 should work.
> > > As for issue in centos-7, it should be the same of previous puppet-stdlib
> > > dependency (See: https://tickets.puppetlabs.com/browse/MODULES-3962).
> > Pin
> > > puppet-stdlib to 4.12.0 should fix this problem.
> > > I'll do test and submit fix if local verification is OK.
> > >
> > >
> > > Olaf Flebbe  于2018年9月30日周日 上午12:03写道：
> > >
> > > > Hi,
> > > >
> > > > I tried the docker provisioner and it failed for debian because
> > > /sbin/init
> > > > not present in bigtop/puppet:trunk-debian-9
> > > >
> > > > --- config --
> > > >
> > > > docker:
> > > > memory_limit: "4g"
> > > > image: "bigtop/puppet:trunk-debian-9"
> > > >
> > > > repo: "http://repos.bigtop.apache.org/releases/1.3.0/debian/9/x86_64;
> > > > distro: debian
> > > > components: [hdfs, yarn, mapreduce]
> > > > enable_local_repo: false
> > > > smoke_test_components: [hdfs, yarn, mapreduce]
> > > > -
> > > > I can live with it, since docker provisioner had many issues with
> > systemd
> > > > in the past:
> > > >
> > > > But it failed for centos-7  too.
> > > >
> > > > + docker exec
> > > > 8d3cf8113318d6b695a85742b985b65d6e146152a7019994dfb8fd8cf0017297 bash
> > -c
> > > > 'puppet apply --parser future
> > > >
> > >
> > --modulepath=/bigtop-home/bigtop-deploy/puppet/modules:/etc/puppet/modules:/usr/share/puppet/modules
> > > > /bigtop-home/bigtop-deploy/puppet/manifests'
> > > > + future='--parser future'
> > > > + docker exec
> > > > c18b89c4cfd0847d71798b0bf5a86db01c6f3becb42b8ba293343227a71a8717 bash
> > -c
> > > > 'puppet apply --parser future
> > > >
> > >
> > --modulepath=/bigtop-home/bigtop-deploy/puppet/modules:/etc/puppet/modules:/usr/share/puppet/modules
> > > > /bigtop-home/bigtop-deploy/puppet/manifests'
> > > > + future='--parser future'
> > > > + docker exec
> > > > 79b0a9ce1ae5d25f25cbf35c3e3b8e26131ca556a2b046eca56737020ffbd31e bash
> > -c
> > > > 'puppet apply --parser future
> > > >
> > >
> > --modulepath=/bigtop-home/bigtop-deploy/puppet/modules:/etc/puppet/modules:/usr/share/puppet/modules
> > > > /bigtop-home/bigtop-deploy/puppet/manifests'
> > > > Warning: Config file /etc/puppet/hiera.yaml not found, using Hiera
> > > defaults
> > > > Error: Could not find data item bigtop::hadoop_head_node in any Hiera
> > > data
> > > > file and no default supplied on node 8d3cf8113318.bigtop.apache.org
> > > > Error: Could not find data item bigtop::hadoop_head_node in any Hiera
> > > data
> > > > file and no default supplied on node 8d3cf8113318.bigtop.apache.org
> > > > Warning: Config file /etc/puppet/hiera.yaml not found, using Hiera
> > > defaults
> > > > Warning: Config file /etc/puppet/hiera.yaml not found, using Hiera
> > > defaults
> > > > Error: Could not find data item bigtop::hadoop_head_node in any Hiera
> > > data
> > > > file and no default supplied on node c18b89c4cfd0.bigtop.apache.org
> > > > Error: Could not find data item bigtop::hadoop_head_node in any Hiera
> > > data
> > > > file and no default supplied on node c18b89c4cfd0.bigtop.apache.org
> > > > Error: Could not find data item bigtop::hadoop_head_node in any Hiera
> > > data
> > > > file and no default supplied on node 79b0a9ce1ae5.bigtop.apache.org
> > > > Error: Could not find data item bigtop::hadoop_head_node in any Hiera
> > > data
> > > > file and no default supplied on node 79b0a9ce1ae5.bigtop.apache.org
> > > >
> > > > 
> > > >
> > > > Evans, what do you think: Can you reproduce issues ?
> > > >
> > > > Cheers,
> > > >
> > > > Olaf
> > > >
> > > >
> > > > > Am 29.09.2018 um 12:11 schrieb Jun HE :
> > > > >
> > > > > Already updated.
> > > > >
> > > > > Jun HE  于2018年9月29日周六 下午3:20写道：
> > > > >
> > > > >> Sure. Will update them today.
> > > > >>
> > > > >>
> > > > >> Evans Ye  于2018年9月29日周六 下午2:32写道：
> > > > >>
> > > > >>> I need some extra time to perform evaluation and give my +1 out.
> > > > >>> I'll be back to Taiwan tomorrow which I'll have full network
> > > > >>> accessibility.
> > > > >>> However, if you got enough vote for the release, go ahead w/o
> > waiting
> > > > me.
> > > > >>>
> > > > >>> BTW, I poked

Bringing AVRO as a dependecy...

Guys,

I want to bounce a question off the team: what would be the best way to bring
Avro dependency into the stack? I don't think it deserves to be a separate
component per se - after all it serves a niche purpose and doesn't stand on
its own.

However, it makes sense to carry around in case an app needs such a
serialization. I've been thinking of perhaps of packing it up as a Hadoop
or HBase dependency, but it doesn't seem like the most straight forward idea,
to be honest. Or may be we should introduce a separate helper package with
serializers and such?

What would be a preferred way from your stand point?
--
  With regards,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622

Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.

Re: [VOTE] Release Bigtop version 1.3.0

I will do some testing in a next a couple of days, absolutely.
BTW, looks like my emails aren't getting through sometimes, as I don't
see my earlier email suggesting that we need to cancel this vote, as
RC1 didn't pass.
Thanks
--
  With regards,
Konstantin (Cos) Boudnik


On Sun, Oct 7, 2018 at 9:52 AM, Evans Ye  wrote:
> Oh yes. You're right!
> I was testing provisioner part against the branch instead of the RC...
> So at least we need RC2 for provisioner stuffs being fixed.
>
> Jun,
> can you work out RC2 for vote?
>
> Cos,
> Any part you'd like to evaluate?
>
>
> Konstantin Boudnik  於 2018年10月8日 週一 上午12:44寫道：
>
>> Guys,
>>
>> am I reading the list wrong and there weren't any issues that required
>> to be fixed to make RC1 fully working?
>> --
>>   With regards,
>> Konstantin (Cos) Boudnik
>> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
>>
>> Disclaimer: Opinions expressed in this email are those of the author,
>> and do not necessarily represent the views of any company the author
>> might be affiliated with at the moment of writing.
>>
>>
>> On Sun, Oct 7, 2018 at 9:39 AM, Evans Ye  wrote:
>> > +1
>> >
>> > * signatures are evaluated
>> > * Provisioner tested
>> > * Random builds manually tested
>> >
>> > Thanks for pushing the release through, Jun.
>> > Everyone, please help to evaluate or cast your doubts to the release.
>> > Thanks!
>> >
>> > Jun HE  於 2018年10月3日 週三 下午7:07寫道：
>> >
>> >> I've tested CentOS-7 with 4.12.0 puppetlabs-stdlib, will submit patch
>> soon.
>> >> For Debian, I'll update the config file as you suggested, and
>> centos/fedora
>> >> config as well.
>> >> They should be ready in today for review.
>> >>
>> >> Evans Ye  于2018年10月3日周三 下午1:48写道：
>> >>
>> >> > For provisioner part, I tested CentOS, Ubuntu, and Debian.
>> >> > Currently only Ubuntu works.
>> >> > * CentOS: Need to pin down the puppetlabs-stdlib to 4.12.0 as Jun He
>> >> > suggested
>> >> > * Debian: Just need to update config file from debian 8 to debian 9.
>> >> >
>> >> > Jun are you already working on this or you need my help fixing them?
>> >> >
>> >> >
>> >> > Jun HE  於 2018年9月30日 週日 下午2:30寫道：
>> >> >
>> >> > > Hi Olaf,
>> >> > >
>> >> > > For debian provision test, use bigtop/puppet:1.3.0-debian-9 should
>> >> work.
>> >> > > As for issue in centos-7, it should be the same of previous
>> >> puppet-stdlib
>> >> > > dependency (See: https://tickets.puppetlabs.com/browse/MODULES-3962
>> ).
>> >> > Pin
>> >> > > puppet-stdlib to 4.12.0 should fix this problem.
>> >> > > I'll do test and submit fix if local verification is OK.
>> >> > >
>> >> > >
>> >> > > Olaf Flebbe  于2018年9月30日周日 上午12:03写道：
>> >> > >
>> >> > > > Hi,
>> >> > > >
>> >> > > > I tried the docker provisioner and it failed for debian because
>> >> > > /sbin/init
>> >> > > > not present in bigtop/puppet:trunk-debian-9
>> >> > > >
>> >> > > > --- config --
>> >> > > >
>> >> > > > docker:
>> >> > > > memory_limit: "4g"
>> >> > > > image: "bigtop/puppet:trunk-debian-9"
>> >> > > >
>> >> > > > repo: "
>> http://repos.bigtop.apache.org/releases/1.3.0/debian/9/x86_64
>> >> "
>> >> > > > distro: debian
>> >> > > > components: [hdfs, yarn, mapreduce]
>> >> > > > enable_local_repo: false
>> >> > > > smoke_test_components: [hdfs, yarn, mapreduce]
>> >> > > > -
>> >> > > > I can live with it, since docker provisioner had many issues with
>> >> > systemd
>> >> > > > in the past:
>> >> > > >
>> >> > > > But it failed for centos-7  too.
>> >> > > >
>> >> > > > + docker exec
>> >> > > > 8d3cf8113318d6b695a85742b985b65d6e146152a7019994dfb8fd8cf0017297
>> bash
>> >> > -c
>> >> > > > 'puppet apply -

Re: [VOTE] Release Bigtop version 1.3.0

Guys,

am I reading the list wrong and there weren't any issues that required
to be fixed to make RC1 fully working?
--
  With regards,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622

Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.


On Sun, Oct 7, 2018 at 9:39 AM, Evans Ye  wrote:
> +1
>
> * signatures are evaluated
> * Provisioner tested
> * Random builds manually tested
>
> Thanks for pushing the release through, Jun.
> Everyone, please help to evaluate or cast your doubts to the release.
> Thanks!
>
> Jun HE  於 2018年10月3日 週三 下午7:07寫道：
>
>> I've tested CentOS-7 with 4.12.0 puppetlabs-stdlib, will submit patch soon.
>> For Debian, I'll update the config file as you suggested, and centos/fedora
>> config as well.
>> They should be ready in today for review.
>>
>> Evans Ye  于2018年10月3日周三 下午1:48写道：
>>
>> > For provisioner part, I tested CentOS, Ubuntu, and Debian.
>> > Currently only Ubuntu works.
>> > * CentOS: Need to pin down the puppetlabs-stdlib to 4.12.0 as Jun He
>> > suggested
>> > * Debian: Just need to update config file from debian 8 to debian 9.
>> >
>> > Jun are you already working on this or you need my help fixing them?
>> >
>> >
>> > Jun HE  於 2018年9月30日 週日 下午2:30寫道：
>> >
>> > > Hi Olaf,
>> > >
>> > > For debian provision test, use bigtop/puppet:1.3.0-debian-9 should
>> work.
>> > > As for issue in centos-7, it should be the same of previous
>> puppet-stdlib
>> > > dependency (See: https://tickets.puppetlabs.com/browse/MODULES-3962).
>> > Pin
>> > > puppet-stdlib to 4.12.0 should fix this problem.
>> > > I'll do test and submit fix if local verification is OK.
>> > >
>> > >
>> > > Olaf Flebbe  于2018年9月30日周日 上午12:03写道：
>> > >
>> > > > Hi,
>> > > >
>> > > > I tried the docker provisioner and it failed for debian because
>> > > /sbin/init
>> > > > not present in bigtop/puppet:trunk-debian-9
>> > > >
>> > > > --- config --
>> > > >
>> > > > docker:
>> > > > memory_limit: "4g"
>> > > > image: "bigtop/puppet:trunk-debian-9"
>> > > >
>> > > > repo: "http://repos.bigtop.apache.org/releases/1.3.0/debian/9/x86_64
>> "
>> > > > distro: debian
>> > > > components: [hdfs, yarn, mapreduce]
>> > > > enable_local_repo: false
>> > > > smoke_test_components: [hdfs, yarn, mapreduce]
>> > > > -
>> > > > I can live with it, since docker provisioner had many issues with
>> > systemd
>> > > > in the past:
>> > > >
>> > > > But it failed for centos-7  too.
>> > > >
>> > > > + docker exec
>> > > > 8d3cf8113318d6b695a85742b985b65d6e146152a7019994dfb8fd8cf0017297 bash
>> > -c
>> > > > 'puppet apply --parser future
>> > > >
>> > >
>> >
>> --modulepath=/bigtop-home/bigtop-deploy/puppet/modules:/etc/puppet/modules:/usr/share/puppet/modules
>> > > > /bigtop-home/bigtop-deploy/puppet/manifests'
>> > > > + future='--parser future'
>> > > > + docker exec
>> > > > c18b89c4cfd0847d71798b0bf5a86db01c6f3becb42b8ba293343227a71a8717 bash
>> > -c
>> > > > 'puppet apply --parser future
>> > > >
>> > >
>> >
>> --modulepath=/bigtop-home/bigtop-deploy/puppet/modules:/etc/puppet/modules:/usr/share/puppet/modules
>> > > > /bigtop-home/bigtop-deploy/puppet/manifests'
>> > > > + future='--parser future'
>> > > > + docker exec
>> > > > 79b0a9ce1ae5d25f25cbf35c3e3b8e26131ca556a2b046eca56737020ffbd31e bash
>> > -c
>> > > > 'puppet apply --parser future
>> > > >
>> > >
>> >
>> --modulepath=/bigtop-home/bigtop-deploy/puppet/modules:/etc/puppet/modules:/usr/share/puppet/modules
>> > > > /bigtop-home/bigtop-deploy/puppet/manifests'
>> > > > Warning: Config file /etc/puppet/hiera.yaml not found, using Hiera
>> > > defaults
>> > > > Error: Could not find data item bigtop::hadoop_head_node in any Hiera
>> > > data
>> > > > file and no default supplied on node 8d3cf8113318.bigtop.apache.org
>> > > > Error: Could not find data item bigtop::hadoop_head_node in any Hiera
>> > > data
>> > > > file and no default supplied on node 8d3cf8113318.bigtop.apache.org
>> > > > Warning: Config file /etc/puppet/hiera.yaml not found, using Hiera
>> > > defaults
>> > > > Warning: Config file /etc/puppet/hiera.yaml not found, using Hiera
>> > > defaults
>> > > > Error: Could not find data item bigtop::hadoop_head_node in any Hiera
>> > > data
>> > > > file and no default supplied on node c18b89c4cfd0.bigtop.apache.org
>> > > > Error: Could not find data item bigtop::hadoop_head_node in any Hiera
>> > > data
>> > > > file and no default supplied on node c18b89c4cfd0.bigtop.apache.org
>> > > > Error: Could not find data item bigtop::hadoop_head_node in any Hiera
>> > > data
>> > > > file and no default supplied on node 79b0a9ce1ae5.bigtop.apache.org
>> > > > Error: Could not find data item bigtop::hadoop_head_node in any Hiera
>> > > data
>> > > > file and no default supplied on node 79b0a9ce1ae5.bigtop.apache.org
>> > > >
>> > > > 
>> > > >
>> > > > Evans, what do you

Re: Coding Guidelines:

2018-10-02 Thread Konstantin Boudnik

Huge +1!
So far, CTR worked well for us, let's not break the good flow. Thanks!
--
  With regards,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622

Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.


On Tue, Oct 2, 2018 at 9:56 PM, Olaf Flebbe  wrote:
> Hi,
>
> I encountered two commits to master in the last month not having the JIRA ID 
> in the subject.
>
> May I ask to stick to convention "start commit message with JIRA ID on the 
> first line", please ?
>
> Best
> Olaf
>
>

Re: [ANNOUNCE] New Bigtop PMC member: Jun He

2018-09-29 Thread Konstantin Boudnik

Welcome on board, dude!
--
  With regards,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622

Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.


On Thu, Sep 27, 2018 at 12:09 PM, Jun HE  wrote:
> Thanks a lot for your kind words, folks! Really appreciate your confidence
> in me.
>
> I learned a lot since I joined Bigtop community, and I'm looking for
> contributing more in the future.
>
> 김영우 (YoungWoo Kim)  于2018年9月27日周四 下午1:04写道：
>
>> Congratulations and welcome Jun!
>>
>> 2018년 9월 27일 (목) 오전 1:19, Evans Ye 님이 작성:
>>
>> > On behalf of the Apache Bigtop Project Management Committee, I am pleased
>> > to announce that Jun He has accepted our invitation to join the
>> > Bigtop PMC.
>> >
>> > Please join me in congratulating Jun!
>> >
>>

Re: Hello world example for adding a new package

2018-09-20 Thread Konstantin Boudnik

Brilliant idea, thank you!
I believe we do have some wiki instructions for developers, but not something 
in a form of a ready to go code template.

--
Regards,
  Cos

On September 20, 2018 12:07:13 PM PDT, Dagang Wei  wrote:
>Hi folks,
>
>I am wondering is there a step by step guide to create a hello-world
>package (a package with only a hello-world.sh script) in Bigtop? That
>would
>help beginners to learn how the Bigtop works.
>
>If we don't have now, and that makes sense, I can file a JIRA issue and
>work on it. Thanks!
>
>Dagang

Re: Why there is no do-component-build and install_bigtop_utils.sh for bigtop-utils?

2018-09-19 Thread Konstantin Boudnik

Yup
--
Regards,
  Cos

On September 19, 2018 5:59:27 PM PDT, Jun HE  wrote:
>Hi, Dagang,
>
>My understanding is that bigtop_utils provides "helper" tools by
>script,
>say bigtop-detect-javahome. That's why there is no need to build them
>and
>install them.
>See:
>https://github.com/apache/bigtop/tree/master/bigtop-packages/src/common/bigtop-utils
>
>These helper scripts will be installed by rules defined
>in bigtop-packages/src/deb/bigtop-utils/rules.b
>
>
>Dagang Wei  于2018年9月20日周四 上午4:41写道：
>
>> Hi folks,
>>
>> I am trying to understand the package bigtop-utils, one interesting
>thing I
>> noticed is that unlike other packages, there is no do-component-build
>and
>> install_bigtop_utils.sh under
>bigtop-packages/src/common/bigtop-utils. Why
>> is that? And how that works?
>>
>> Thanks!
>> Dagang
>>

Re: Package versioning with patches

2018-09-17 Thread Konstantin Boudnik

Yes, you can.

If you want to only reflect the patch version for Hadoop component,
you can change the either the pkg version or release number in the
hadoop section of bigtop.bom. Look for the line

  version { base = '2.8.4'; pkg = base; release = 1 }

Alternatively, set this sys.env. BIGTOP_BUILD_STAMP to some integer.
Alternatively, you can specify 'buildstamp' in the bigtop.bom file. It
has been intended to automatically increment the versions when
packages are produced as a part of the CI/CD process and would have an
effect on all packages you build - a very different from you want, I
guess.
--
  With regards,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622

Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.


On Mon, Sep 17, 2018 at 8:19 PM, Dagang Wei  wrote:
> Hi folks,
>
> Seems after applying a patch, the package versioning will not change. For
> example, if I add a patch to Hadoop 2.8.4 in Bigtop, the result package
> still gets the version 2.8.4, so it is not able to tell from the version
> number whether a package contains a patch or not.
>
> Is there a way I can add additional information to reflect the patch in the
> version number? Thanks!

Re: Speed up Bigtop build through parallelism?

2018-07-22 Thread Konstantin Boudnik

Check this out

https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/
--
  With regards,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622

Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.


On Sun, Jul 22, 2018 at 8:33 PM, Dagang Wei  wrote:
> Hi Cos,
>
> Is there a guide or could you point me to the source code of "splitting the
> build by components"?
>
> The problem I am experience now is that, sometimes component build (e.g.,
> hbase) is flaky (due to network issue, etc), then I need to retry the whole
> build. I am thinking about splitting package build with apt repository
> build, apt repository build depends on debs, so that instead of retry the
> whole build for a single component failure, only retry the failed component
> build.
>
> Thanks!
>
> On Sun, Jun 17, 2018 at 10:51 AM functicons  wrote:
>>
>> Thanks Cos!
>>
>> On Sat, Jun 16, 2018 at 12:58 AM Konstantin Boudnik 
>> wrote:
>>>
>>> What we do is splitting the build by components (remember that some of
>>> them are dependent on each other: HBase-Hadoop, etc.) and then execute
>>> component builds at different agents.
>>> That's what we do with ci.bigtop.apache.org
>>>
>>> There is a tricky step of gathering all the artifacts together, but it is
>>> relatively easy to accomplish.
>>>
>>> Hope it helps.
>>> --
>>> Regards,
>>>   Cos
>>>
>>> On June 16, 2018 7:16:21 AM GMT+03:00, functicons 
>>> wrote:
>>> >Hi folks,
>>> >
>>> >We are building packages from our fork of Bigtop, but the build takes
>>> >really long. So I'm wondering is it possible to build in parallel? All
>>> >suggestions are appreciated. Thanks!
>>> >
>>> >Dagang

Re: Speed up Bigtop build through parallelism?

2018-06-16 Thread Konstantin Boudnik

What we do is splitting the build by components (remember that some of them are 
dependent on each other: HBase-Hadoop, etc.) and then execute component builds 
at different agents.
That's what we do with ci.bigtop.apache.org

There is a tricky step of gathering all the artifacts together, but it is 
relatively easy to accomplish.

Hope it helps.
--
Regards,
  Cos

On June 16, 2018 7:16:21 AM GMT+03:00, functicons  wrote:
>Hi folks,
>
>We are building packages from our fork of Bigtop, but the build takes
>really long. So I'm wondering is it possible to build in parallel? All
>suggestions are appreciated. Thanks!
>
>Dagang

Re: Fwd: ASF Board Meeting Summary - May 16, 2018

2018-05-17 Thread Konstantin Boudnik

God speed, Peter!
--
Regards,
  Cos

On May 17, 2018 8:43:30 AM GMT+03:00, Roman Shaposhnik  
wrote:
>Major congrats to Peter! Well deserved!
>
>Thanks,
>Roman.
>
>
>-- Forwarded message --
>From: Phil Steitz 
>Date: Wed, May 16, 2018 at 5:15 PM
>Subject: ASF Board Meeting Summary - May 16, 2018
>To: committ...@apache.org
>
>
>The May board meeting took place on the 16th.
>
>The April minutes were approved.
>Minutes will be posted to
>http://www.apache.org/foundation/records/minutes/
>
>All of the received reports to the board were approved.
>
>The following resolutions were passed unanimously:
>
>  A. Change the Apache MINA Project Chair (Guillaume Nodet, VP)
>  B. Change the Apache Bigtop Project Chair (Peter Linnell, VP)
>  C. Change the Apache Hadoop Project Chair (Vinod Kumar
>Vavilapalli, VP)
>  D. Establish the Apache Traffic Control Project (David Neuman, VP)
>  E. Change the Apache Giraph Project Chair (Dionysios Logothetis, VP)
>  F. Change the Apache Camel Project Chair (Andrea Cosentino, VP)
>  G. Change the Apache Incubator Project Chair (Justin Mclean, VP)
>
>The next board meeting will be on the 20th of June.

Re: [DISCUSS] Remove hue from bigtop

2018-04-10 Thread Konstantin Boudnik

About time.
+1
--
  With regards,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622

Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.


On Tue, Apr 10, 2018 at 9:57 PM, Olaf Flebbe  wrote:
> Hi,
>
> Since Infrastructure is now in sane state again, showing we are in a
> not-so-ugly state, I like to propose to remove "hue" from bigtop.
>
> See the matrix https://ci.bigtop.apache.org/job/Bigtop-trunk-packages/ (hue
> does not build anywhere right now, due to issues mentioned below)
>
> Reasons:
>
> * The application does not work with recent linux distributions due to
> incompatibilites with openssl version 1.1 default for debian-9, fedora-26
> and AFAIK opensuse.
>A bugreport https://github.com/cloudera/hue/issues/629 has been issued in
> november and not even answered upstream.
>
> * The application is a contains a lot of outdated python dependencies hidden
> in desktop/core/ext-py . Trying to fix that opened a can of worms.
>   BIGTOP-2960, BIGTOP-2952, BIGTOP-2946
>
> * I am withdrawing from maintainer state of hue.
>   https://issues.apache.org/jira/browse/BIGTOP-3021
>
> Cheers,
> Olaf
>
>

Re: Something weird is going on with our puppet

Unfortunately, no. When I started using "--parser future" it still
gives me the same errors.

Just to confirm: I am using official bigtop/puppet container for
Ubuntu 16.04 and the command line looks like this

puppet apply  --parser=future -d
--modulepath=/bigtop/bigtop-deploy/puppet/modules:/etc/puppet/modules
/bigtop/bigtop-deploy/puppet/manifests/site.pp

I understand that the commit in question has nothing to do with it -
it's just the last one for the particular file. And the problem is
clearly elsewhere, but I just don't see where it comes from. The
execution is happening according to what e.g. deploy-hadoop.sh is
doing. Yes, clearly, something is off... I keep digging.
--
  With regards,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622

Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.


On Wed, Feb 28, 2018 at 5:06 PM, Kevin Monroe
<kevin.mon...@canonical.com> wrote:
> Cos, are you resolved?
>
> We use "'--parser=future" (probably the same as --future) in the puppet
> apply args for puppet3 on ubuntu-16.04.  The commit you referenced is to
> ensure the jdk class is available even if the jdk has been pre-installed:
>
> https://github.com/apache/bigtop/commit/40e796bd693a8bc6243ee39a6540357e1d559ea8
>
> If there's an alternate way to define the jdk class to be more puppet 3 and
> 4 compliant, i'm all ears. Otherwise, don't you dare break it :)
>
> -Kevin
>
> On Wed, Feb 28, 2018 at 12:11 PM, Konstantin Boudnik <c...@apache.org> wrote:
>
>> Thanks for the hints, Olaf. I recollect the conversion about --future
>> stuff now.
>> And I am using bigtop/puppet:ubuntu-16.04, which has puppet 3.8.5 on it.
>> --
>>   With regards,
>> Konstantin (Cos) Boudnik
>> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
>>
>> Disclaimer: Opinions expressed in this email are those of the author,
>> and do not necessarily represent the views of any company the author
>> might be affiliated with at the moment of writing.
>>
>>
>> On Wed, Feb 28, 2018 at 10:03 AM, Olaf Flebbe <o...@oflebbe.de> wrote:
>> > BTW,
>> >
>> > Bigtop/puppet is versioned now as well . bigtop/puppet:trunk-centos-7 is
>> the
>> > most recent one.
>> >
>> > Olaf
>> >
>> >
>> >
>> >
>> > Am 28.02.2018 um 19:00 schrieb Olaf Flebbe <o...@oflebbe.de>:
>> >
>> > Hi Cos,
>> >
>> > At least your command line is not quite up to date. It should read
>> >
>> > puppet apply $future
>> > --modulepath=/bigtop-puppet/modules:/etc/puppet/modules:/
>> usr/share/puppet/modules
>> > /bigtop-puppet/manifests/site.pp
>> >
>> > $future == --future for puppet 3
>> > and without for puppet version 4
>> >
>> > It is not possible to have one major version of puppet in bigtop, see
>> > archive of dev.
>> > /usr/share/puppet/modules is used used for more recent versions of puppet
>> > because of the filesystem hierarchy standard .
>> >
>> > You may want to check if you have recent bigtop/puppet container.
>> >
>> > Olaf
>> >
>> >
>> > puppet apply -d
>> > --modulepath=bigtop-deploy/puppet/modules:/etc/puppet/modules
>> > bigtop-deploy/puppet/manifests/site.pp
>> >
>> >
>> >
>> > Am 28.02.2018 um 17:18 schrieb Konstantin Boudnik <c...@apache.org>:
>> >
>> > I'm doing this in our official puppet container, so the version should be
>> > aligned. Will dig more into this in the afternoon and report back.
>> Thanks!
>> > --
>> > Regards,
>> >  Cos
>> >
>> > On February 28, 2018 2:17:41 AM PST, Olaf Flebbe <o...@oflebbe.de> wrote:
>> >
>> > This may relate to the puppet version change. The gradle command should
>> > work.
>> >
>> > Am 28.02.2018 um 10:35 schrieb Evans Ye <evan...@apache.org>:
>> >
>> > Cos,
>> >
>> > Not sure what's the problem you encountered but form
>> >
>> > https://ci.bigtop.apache.org/view/Provisioner/job/Bigtop-
>> trunk-deployments/
>> >
>> > I can see the Puppet deployment is doing well.
>> >
>> > Can you share more info? Env, config, command, etc.
>> >
>> >
>> > 2018-02-28 14:34 GMT+08:00 Konstantin Boudnik <c...@apache.org>:
>> >
>> > Guys,
>> >
>> >

Re: Something weird is going on with our puppet

Thanks for the hints, Olaf. I recollect the conversion about --future stuff now.
And I am using bigtop/puppet:ubuntu-16.04, which has puppet 3.8.5 on it.
--
  With regards,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622

Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.


On Wed, Feb 28, 2018 at 10:03 AM, Olaf Flebbe <o...@oflebbe.de> wrote:
> BTW,
>
> Bigtop/puppet is versioned now as well . bigtop/puppet:trunk-centos-7 is the
> most recent one.
>
> Olaf
>
>
>
>
> Am 28.02.2018 um 19:00 schrieb Olaf Flebbe <o...@oflebbe.de>:
>
> Hi Cos,
>
> At least your command line is not quite up to date. It should read
>
> puppet apply $future
> --modulepath=/bigtop-puppet/modules:/etc/puppet/modules:/usr/share/puppet/modules
> /bigtop-puppet/manifests/site.pp
>
> $future == --future for puppet 3
> and without for puppet version 4
>
> It is not possible to have one major version of puppet in bigtop, see
> archive of dev.
> /usr/share/puppet/modules is used used for more recent versions of puppet
> because of the filesystem hierarchy standard .
>
> You may want to check if you have recent bigtop/puppet container.
>
> Olaf
>
>
> puppet apply -d
> --modulepath=bigtop-deploy/puppet/modules:/etc/puppet/modules
> bigtop-deploy/puppet/manifests/site.pp
>
>
>
> Am 28.02.2018 um 17:18 schrieb Konstantin Boudnik <c...@apache.org>:
>
> I'm doing this in our official puppet container, so the version should be
> aligned. Will dig more into this in the afternoon and report back. Thanks!
> --
> Regards,
>  Cos
>
> On February 28, 2018 2:17:41 AM PST, Olaf Flebbe <o...@oflebbe.de> wrote:
>
> This may relate to the puppet version change. The gradle command should
> work.
>
> Am 28.02.2018 um 10:35 schrieb Evans Ye <evan...@apache.org>:
>
> Cos,
>
> Not sure what's the problem you encountered but form
>
> https://ci.bigtop.apache.org/view/Provisioner/job/Bigtop-trunk-deployments/
>
> I can see the Puppet deployment is doing well.
>
> Can you share more info? Env, config, command, etc.
>
>
> 2018-02-28 14:34 GMT+08:00 Konstantin Boudnik <c...@apache.org>:
>
> Guys,
>
> I am trying to run the following command:
>
> puppet apply -d
> --modulepath=bigtop-deploy/puppet/modules:/etc/puppet/modules
> bigtop-deploy/puppet/manifests/site.pp
>
> from the top-level directory of Bigtop source tree.
>
> to deploy some cluster configuration in a container. All I am
>
> getting
>
> though is this error message:
>
> Debug: Caching environment 'production' (ttl = 0 sec)
> Error: Could not find class jdk for c41f94213de4 on node
>
> c41f94213de4
>
>
> Looks like class jdk under bigtop-deploy/puppet/manifests could not
>
> be
>
> found. The latest change related to this has happened in hash
>
> 40e796b
>
> (made by Kevin), but I am kinda sure I was able to deploy cluster
> after that.
>
> Did something got changed which I missed and their new rules of
>
> using
>
> the recipes? Thanks for any leads!
> --
> With regards,
> Konstantin (Cos) Boudnik
> 2CAC 8312 4870 D885 8616  6115 220F 6980 1F27 E622
>
> Disclaimer: Opinions expressed in this email are those of the
>
> author,
>
> and do not necessarily represent the views of any company the author
> might be affiliated with at the moment of writing.
>
>
>

Re: CI Infra

Yeah, I was trying to relaunch the agent on that slave, but didn't succeed. Now 
I see why :) Thanks for fixing it!
-- 
Regards,
  Cos

On February 28, 2018 2:20:31 AM PST, Evans Ye  wrote:
>I still can't get into slave07. And reboot seems no help. So I've
>recreated
>slave07.
>Now it has been attached to our Jenkins.
>
>Meanwhile, I've added two experimental CI jobs to test new features:
>*
>https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages-BIGTOP-2993/
>*
>https://ci.bigtop.apache.org/view/Provisioner/job/Bigtop-trunk-deployments-BIGTOP-2996/
>
>Will remove them once test finished.
>
>Best,
>Evans
>
>
>2018-02-13 11:19 GMT+08:00 Evans Ye :
>
>> Thanks, Olaf!
>>
>> 2018-02-13 4:54 GMT+08:00 Olaf Flebbe :
>>
>>> Hi,
>>>
>>> slave07 was stuck, had to stop the instance. It has a new IP Adress
>now!
>>>
>>> Updated & rebooted slave06, slave07, slave02, master.
>>> Updated jenkins, had to enable CSRF protection, removed obsolete
>plugins,
>>> updated webstart bindings.
>>>
>>>
>>> Olaf
>>>
>>>
>>>
>>

Re: Something weird is going on with our puppet