Re: Official Apache Slack Channel for Hadoop projects

[Yufei Gu]

Thanks Wei-Chiu. Just join both hdfs and yarn channel. Yes, there is a yarn
channel. There are only 3 members in the yarn channel.

Best,

Yufei

`This is not a contribution`


On Fri, Oct 11, 2019 at 4:35 PM Wei-Chiu Chuang  wrote:

> Hi Hadoop devs,
>
> In case you don't know, there is an official ASF slack, and there's a HDFS
> channel in it. This is the slack workplace managed by Apache Infra.
>
> Please see this wiki to get invite:
> https://cwiki.apache.org/confluence/display/INFRA/Slack+Guest+Invites or
> DM
> me to get an invite.
>
> Once you get access to the ASF workplace, search for #hdfs channel. There
> is also an #ozone channel, #hadoop, #submarine-dev, #submarine-user. I
> don't see a #yarn channel, but I can create one (not sure if who is
> eligible for creating channels, PMC or committers or any one?)
>
> We will not use Slack channel to vote on project decisions/vote, but it
> might be an easier way to find me. Right now the channels are quite dry.
> Let's see if we can revive them.
>
> Weichiu
>

Re: [DISCUSS] A unified and open Hadoop community sync up schedule?

[Yufei Gu]

+1 for this idea. Thanks Wangda for bringing this up.

Some comments to share:

   - Agenda needed to be posted ahead of meeting and welcome any interested
   party to contribute to topics.
   - We should encourage more people to attend. That's whole point of the
   meeting.
   - Hopefully, this can mitigate the situation that some patches are
   waiting for review for ever, which turns away new contributors.
   - 30m per session sounds a little bit short, we can try it out and see
   if extension is needed.

Best,

Yufei

`This is not a contribution`


On Fri, Jun 7, 2019 at 4:39 PM Wangda Tan  wrote:

> Hi Hadoop-devs,
>
> Previous we have regular YARN community sync up (1 hr, biweekly, but not
> open to public). Recently because of changes in our schedules, Less folks
> showed up in the sync up for the last several months.
>
> I saw the K8s community did a pretty good job to run their sig meetings,
> there's regular meetings for different topics, notes, agenda, etc. Such as
>
> https://docs.google.com/document/d/13mwye7nvrmV11q9_Eg77z-1w3X7Q1GTbslpml4J7F3A/edit
>
>
> For Hadoop community, there are less such regular meetings open to the
> public except for Ozone project and offline meetups or Bird-of-Features in
> Hadoop/DataWorks Summit. Recently we have a few folks joined DataWorks
> Summit at Washington DC and Barcelona, and lots (50+) of folks join the
> Ozone/Hadoop/YARN BoF, ask (good) questions and roadmaps. I think it is
> important to open such conversations to the public and let more
> folk/companies join.
>
> Discussed a small group of community members and wrote a short proposal
> about the form, time and topic of the community sync up, thanks for
> everybody who have contributed to the proposal! Please feel free to add
> your thoughts to the Proposal Google doc
> <
> https://docs.google.com/document/d/1GfNpYKhNUERAEH7m3yx6OfleoF3MqoQk3nJ7xqHD9nY/edit#
> >
> .
>
> Especially for the following parts:
> - If you have interests to run any of the community sync-ups, please put
> your name to the table inside the proposal. We need more volunteers to help
> run the sync-ups in different timezones.
> - Please add suggestions to the time, frequency and themes and feel free to
> share your thoughts if we should do sync ups for other topics which are not
> covered by the proposal.
>
> Link to the Proposal Google doc
> <
> https://docs.google.com/document/d/1GfNpYKhNUERAEH7m3yx6OfleoF3MqoQk3nJ7xqHD9nY/edit#
> >
>
> Thanks,
> Wangda Tan
>

Re: YARN SLS improving idea

[Yufei Gu]

+YANG WEIWEI 

Make sense to me from SLS perspective, but I am not familiar with Placement
Constraints. Add WeiWei.

Best,

Yufei

`This is not a contribution`


On Sat, Aug 11, 2018 at 8:57 AM Daniel Templeton 
wrote:

> Yufei, Wangda, Sunil, any comments?
>
> Daniel
>
> On 8/11/18 8:48 AM, Sichen Zhao wrote:
> > Hi,
> > Is there anyone who can reply my ideas?
> >
> > Best Regards
> > Sichen Zhao
> >
> > 
> > From: Sichen Zhao 
> > Sent: Friday, August 10, 2018 11:10
> > To: Hadoop Common
> > Subject: YARN SLS improving idea
> >
> > Hi,
> > I am a developer from AliBaBa China, i recently used SLS for scheduling
> simulation, SLS currently supports multidimensional resource input(CPU, mem
> , other resources: disk). But SLS can't take scheduling request, which is
> currently widely used in YARN, as input, so Placement Constraints and
> attributes are not supported.
> >
> > So what i wanna improve the SLS: Add scheduling emulation for scheduling
> request resource format.
> >
> > The specific work is as follows:
> > 1. Add input support for the scheduling request format.
> > 2. Add support for scheduling request resource format in NMSim.
> > 3. Adding scheduling request support for the Capacity Scheduler(maybe it
> is already done in current version).
> >
> > What do you think about my ideas?
> >
> >
> > Best Regards
> > Sichen Zhao
> >
> > -
> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >
> >
> > -
> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >
>
>

Re: [VOTE] Release Apache Hadoop 3.0.2 (RC0)

[Yufei Gu]

Thanks Lei for working on this!

+1 (non-binding)

   - Downloaded the binary tarball and verified the checksum.
   - Started a pseudo cluster inside one docker container
   - Run Resource Manager with Fair Scheduler
   - Verified distributed shell
   - Verified mapreduce pi job
   - Sanity checked RM WebUI

Best,

Yufei

On Fri, Apr 6, 2018 at 11:16 AM, Lei Xu  wrote:

> Hi, All
>
> I've created release candidate RC-0 for Apache Hadoop 3.0.2.
>
> Please note: this is an amendment for Apache Hadoop 3.0.1 release to
> fix shaded jars in apache maven repository. The codebase of 3.0.2
> release is the same as 3.0.1.  New bug fixes will be included in
> Apache Hadoop 3.0.3 instead.
>
> The release page is:
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0+Release
>
> New RC is available at: http://home.apache.org/~lei/hadoop-3.0.2-RC0/
>
> The git tag is release-3.0.2-RC0, and the latest commit is
> 5c141f7c0f24c12cb8704a6ccc1ff8ec991f41ee
>
> The maven artifacts are available at
> https://repository.apache.org/content/repositories/orgapachehadoop-1096/
>
> Please try the release, especially, *verify the maven artifacts*, and vote.
>
> The vote will run 5 days, ending 4/11/2018.
>
> Thanks for everyone who helped to spot the error and proposed fixes!
>
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)

[Yufei Gu]

 Thanks Wangda for working on this!

+1 (non-binding)

   - Downloaded the binary tarball and verified the checksum.
   - Started a pseudo cluster inside one docker container
   - Run Resource Manager with Fair Scheduler
   - Verified distributed shell
   - Verified mapreduce pi job
   - Sanity checked RM WebUI

Best,

Yufei

On Thu, Mar 29, 2018 at 9:15 PM, Wangda Tan  wrote:

> Hi folks,
>
> Thanks to the many who helped with this release since Dec 2017 [1]. We've
> created RC1 for Apache Hadoop 3.1.0. The artifacts are available here:
>
> http://people.apache.org/~wangda/hadoop-3.1.0-RC1
>
> The RC tag in git is release-3.1.0-RC1. Last git commit SHA is
> 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d
>
> The maven artifacts are available via repository.apache.org at
> https://repository.apache.org/content/repositories/orgapachehadoop-1090/
> This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific.
>
> 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions
> include the first class GPU/FPGA support on YARN, Native services, Support
> rich placement constraints in YARN, S3-related enhancements, allow HDFS
> block replicas to be provided by an external storage system, etc.
>
> For 3.1.0 RC0 vote discussion, please see [3].
>
> We’d like to use this as a starting release for 3.1.x [1], depending on how
> it goes, get it stabilized and potentially use a 3.1.1 in several weeks as
> the stable release.
>
> We have done testing with a pseudo cluster:
> - Ran distributed job.
> - GPU scheduling/isolation.
> - Placement constraints (intra-application anti-affinity) by using
> distributed shell.
>
> My +1 to start.
>
> Best,
> Wangda/Vinod
>
> [1]
> https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b
> c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0)
> AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY
> fixVersion ASC
> [3]
> https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7
> 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E
>

Re: [VOTE] Release Apache Hadoop 3.0.1 (RC1)

[Yufei Gu]

Thanks Eddy!

+1 (non-binding)

   - Downloaded the hadoop-3.0.1.tar.gz from
http://home.apache.org/~lei/hadoop-3.0.1-RC1/
   - Started a pseudo cluster inside one docker container
   - Verified distributed shell
   - Verified mapreduce pi job
   - Sanity check RM WebUI

Best,

Yufei

On Tue, Mar 20, 2018 at 9:32 AM, Eric Payne 
wrote:

>  Thanks for working on this release!
> +1 (binding)
> I tested the following:
> - yarn distributed shell job
>
> - yarn streaming job
>
> - inter-queue preemption
>
> - compared behavior of fair and fifo ordering policy
>
> - both userlimit_first mode and priority_first mode of intra-queue
> preemption
>
> Eric Payne
>
>
>
> On Saturday, March 17, 2018, 11:11:32 PM CDT, Lei Xu 
> wrote:
>
>  Hi, all
>
> I've created release candidate RC-1 for Apache Hadoop 3.0.1
>
> Apache Hadoop 3.0.1 will be the first bug fix release for Apache
> Hadoop 3.0 release. It includes 49 bug fixes and security fixes, which
> include 12
> blockers and 17 are critical.
>
> Please note:
> * HDFS-12990. Change default NameNode RPC port back to 8020. It makes
> incompatible changes to Hadoop 3.0.0.  After 3.0.1 releases, Apache
> Hadoop 3.0.0 will be deprecated due to this change.
>
> The release page is:
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0+Release
>
> New RC is available at: http://home.apache.org/~lei/hadoop-3.0.1-RC1/
>
> The git tag is release-3.0.1-RC1, and the latest commit is
> 496dc57cc2e4f4da117f7a8e3840aaeac0c1d2d0
>
> The maven artifacts are available at:
> https://repository.apache.org/content/repositories/orgapachehadoop-1081/
>
> Please try the release and vote; the vote will run for the usual 5
> days, ending on 3/22/2017 6pm PST time.
>
> Thanks!
>
> -
> To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
>
>

[jira] [Created] (HADOOP-14926) Reconsider the default value of RPC timeout and document it

[Yufei Gu (JIRA)]

Yufei Gu created HADOOP-14926:
-

 Summary: Reconsider the default value of RPC timeout and document 
it
 Key: HADOOP-14926
 URL: https://issues.apache.org/jira/browse/HADOOP-14926
 Project: Hadoop Common
  Issue Type: Improvement
  Components: common
Affects Versions: 3.0.0-alpha3, 2.8.1
Reporter: Yufei Gu


We use zero as the default value of RPC timeout, which mean we don't enforce 
any timeout aka infinity timeout. I think that *zero means infinite* is 
counter-intuitive and error-prone through some Java libs(e.g. Socket 
#setSoTimeout()) do that as well. Zero could be considered as a valid timeout 
value, while negative one isn't. If we use zero to represent infinite, which 
number could be used to represent zero timeout? I suggest use -1 as the default 
value to indicate infinite. 
We also need to document the default value and it means infinite timeout. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14914) Change to a safely casting long to int.

[Yufei Gu (JIRA)]

Yufei Gu created HADOOP-14914:
-

 Summary: Change to a safely casting long to int. 
 Key: HADOOP-14914
 URL: https://issues.apache.org/jira/browse/HADOOP-14914
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 3.1.0
Reporter: Yufei Gu


There are bunches of casting long to int like this:
{code}
long l = 123
int i = (int) l;
{code}
This is not a safe cast. if l is greater than Integer.MAX_VALUE, i would be 
negative, which is an unexpected behavior. We probably at least want to throw 
an exception in that case. I suggest to use {{Math.toIntExact(longValue)}} to 
replace them, which throws an exception if the value overflows an int. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [DISCUSS] official docker image(s) for hadoop

[Yufei Gu]

It would be very helpful for testing the RC. To vote a RC, committers and
PMCs usually spend lots of time to compile, deploy the RC, do several
sanity tests, then +1 for the RC. The docker image potentially saves the
compilation and deployment time, and people can do more tests.

Best,

Yufei

On Wed, Sep 13, 2017 at 11:19 AM, Wangda Tan  wrote:

> +1 to add Hadoop docker image for easier testing / prototyping, it gonna be
> super helpful!
>
> Thanks,
> Wangda
>
> On Wed, Sep 13, 2017 at 10:48 AM, Miklos Szegedi <
> miklos.szeg...@cloudera.com> wrote:
>
> > Marton, thank you for working on this. I think Official Docker images for
> > Hadoop would be very useful for a lot of reasons. I think that it is
> better
> > to have a coordinated effort with production ready base images with
> > dependent images for prototyping. Does anyone else have an opinion about
> > this?
> >
> > Thank you,
> > Miklos
> >
> > On Fri, Sep 8, 2017 at 5:45 AM, Marton, Elek  wrote:
> >
> > >
> > > TL;DR: I propose to create official hadoop images and upload them to
> the
> > > dockerhub.
> > >
> > > GOAL/SCOPE: I would like improve the existing documentation with
> > > easy-to-use docker based recipes to start hadoop clusters with various
> > > configuration.
> > >
> > > The images also could be used to test experimental features. For
> example
> > > ozone could be tested easily with these compose file and configuration:
> > >
> > > https://gist.github.com/elek/1676a97b98f4ba561c9f51fce2ab2ea6
> > >
> > > Or even the configuration could be included in the compose file:
> > >
> > > https://github.com/elek/hadoop/blob/docker-2.8.0/example/doc
> > > ker-compose.yaml
> > >
> > > I would like to create separated example compose files for federation,
> > ha,
> > > metrics usage, etc. to make it easier to try out and understand the
> > > features.
> > >
> > > CONTEXT: There is an existing Jira https://issues.apache.org/jira
> > > /browse/HADOOP-13397
> > > But it’s about a tool to generate production quality docker images
> > > (multiple types, in a flexible way). If no objections, I will create a
> > > separated issue to create simplified docker images for rapid
> prototyping
> > > and investigating new features. And register the branch to the
> dockerhub
> > to
> > > create the images automatically.
> > >
> > > MY BACKGROUND: I am working with docker based hadoop/spark clusters
> quite
> > > a while and run them succesfully in different environments (kubernetes,
> > > docker-swarm, nomad-based scheduling, etc.) My work is available from
> > here:
> > > https://github.com/flokkr but they could handle more complex use cases
> > > (eg. instrumenting java processes with btrace, or read/reload
> > configuration
> > > from consul).
> > >  And IMHO in the official hadoop documentation it’s better to suggest
> to
> > > use official apache docker images and not external ones (which could be
> > > changed).
> > >
> > > Please let me know if you have any comments.
> > >
> > > Marton
> > >
> > > -
> > > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> > > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> > >
> > >
> >
>

[jira] [Created] (HADOOP-13279) Fix all Bad Practices flagged in Fortify

[Yufei Gu (JIRA)]

Yufei Gu created HADOOP-13279:
-

 Summary: Fix all Bad Practices flagged in Fortify
 Key: HADOOP-13279
 URL: https://issues.apache.org/jira/browse/HADOOP-13279
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.9.0
Reporter: Yufei Gu
Assignee: Yufei Gu


The following code contain potential problems:
{code}
Unreleased Resource: StreamsTopCLI.java:738
Unreleased Resource: StreamsGraph.java:189
Unreleased Resource: StreamsCgroupsLCEResourcesHandler.java:291
Unreleased Resource: StreamsUnmanagedAMLauncher.java:195
Unreleased Resource: StreamsCGroupsHandlerImpl.java:319
Unreleased Resource: StreamsTrafficController.java:629
Portability Flaw: Locale Dependent Comparison   TimelineWebServices.java:421
Null DereferenceApplicationImpl.java:465
Null DereferenceVisualizeStateMachine.java:52
Null DereferenceContainerImpl.java:1089
Null DereferenceQueueManager.java:219
Null DereferenceQueueManager.java:232
Null DereferenceResourceLocalizationService.java:1016
Null DereferenceResourceLocalizationService.java:1023
Null DereferenceResourceLocalizationService.java:1040
Null DereferenceResourceLocalizationService.java:1052
Null DereferenceProcfsBasedProcessTree.java:802
Null DereferenceTimelineClientImpl.java:639
Null DereferenceLocalizedResource.java:206
Code Correctness: Double-Checked LockingResourceHandlerModule.java:142
Code Correctness: Double-Checked LockingRMPolicyProvider.java:51
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-13279) Fix all Bad Practices flagged in Fortify

[Yufei Gu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yufei Gu resolved HADOOP-13279.
---
Resolution: Duplicate

> Fix all Bad Practices flagged in Fortify
> 
>
> Key: HADOOP-13279
> URL: https://issues.apache.org/jira/browse/HADOOP-13279
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 2.9.0
>    Reporter: Yufei Gu
>Assignee: Yufei Gu
>
> The following code contain potential problems:
> {code}
> Unreleased Resource: Streams  TopCLI.java:738
> Unreleased Resource: Streams  Graph.java:189
> Unreleased Resource: Streams  CgroupsLCEResourcesHandler.java:291
> Unreleased Resource: Streams  UnmanagedAMLauncher.java:195
> Unreleased Resource: Streams  CGroupsHandlerImpl.java:319
> Unreleased Resource: Streams  TrafficController.java:629
> Portability Flaw: Locale Dependent Comparison TimelineWebServices.java:421
> Null Dereference  ApplicationImpl.java:465
> Null Dereference  VisualizeStateMachine.java:52
> Null Dereference  ContainerImpl.java:1089
> Null Dereference  QueueManager.java:219
> Null Dereference  QueueManager.java:232
> Null Dereference  ResourceLocalizationService.java:1016
> Null Dereference  ResourceLocalizationService.java:1023
> Null Dereference  ResourceLocalizationService.java:1040
> Null Dereference  ResourceLocalizationService.java:1052
> Null Dereference  ProcfsBasedProcessTree.java:802
> Null Dereference  TimelineClientImpl.java:639
> Null Dereference  LocalizedResource.java:206
> Code Correctness: Double-Checked Locking  ResourceHandlerModule.java:142
> Code Correctness: Double-Checked Locking  RMPolicyProvider.java:51
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13254) Make Diskchecker Pluggable

[Yufei Gu (JIRA)]

Yufei Gu created HADOOP-13254:
-

 Summary: Make Diskchecker Pluggable
 Key: HADOOP-13254
 URL: https://issues.apache.org/jira/browse/HADOOP-13254
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Yufei Gu






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org