Re: [DISCUSS] EOL 2.8 or another 2.8.x release?

2019-07-25 Thread Akira Ajisaka
I'm +1 for 1 more release in 2.8.x and declare that 2.8 is EoL.

> would be even happier if we could move people to 2.9.x
Agreed.

-Akira

On Thu, Jul 25, 2019 at 10:59 PM Steve Loughran
 wrote:
>
> I'm in favour of 1 more release (it fixes the off-by 1 bug in
> S3AInputStream HADOOP-16109), but would be even happier if we could move
> people to 2.9.x
>
> maybe do a 2.9.x release and declare that 2.8 is EOL?
>
>
> On Thu, Jul 25, 2019 at 2:08 PM Wei-Chiu Chuang  wrote:
>
> > My bad -- Didn't realize I was looking at the old Hadoop page.
> > Here's the correct list of releases.
> > https://hadoop.apache.org/releases.html
> >
> > On Thu, Jul 25, 2019 at 12:49 AM 张铎(Duo Zhang) 
> > wrote:
> >
> > > IIRC we have a 2.8.5 release?
> > >
> > > On the download page:
> > >
> > > 2.8.5 2018 Sep 15
> > >
> > > Wei-Chiu Chuang  于2019年7月25日周四 上午9:39写道:
> > >
> > > > The last 2.8 release (2.8.4) was made in the last May, more than a year
> > > > ago. https://hadoop.apache.org/old/releases.html
> > > >
> > > > How do folks feel about the fate of branch-2.8? During the last
> > community
> > > > meetup in June, it sounds like most users are still on 2.8 or even 2.7,
> > > so
> > > > I don't think we want to abandon 2.8 just yet.
> > > >
> > > > I would personally want to urge folks to move up to 3.x, so I can stop
> > > > cherrypicking stuff all the way down into 2.8. But it's not up to me
> > > along
> > > > to decide :)
> > > >
> > > > How do people feel about having another 2.8 release or two? I am not
> > > saying
> > > > I want to drive it, but I want to raise the awareness that folks are
> > > still
> > > > on 2.8 and there's not been an update for over a year.
> > > >
> > > > Thoughts?
> > > >
> > >
> >

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org



Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2019-07-25 Thread Apache Jenkins Server
For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1208/

[Jul 24, 2019 3:23:06 AM] (github) HDDS-1848. Fix TestOzoneManagerHA and 
TestOzoneManagerSnapShotProvider.
[Jul 24, 2019 5:11:43 PM] (stevel) HADOOP-16433. S3Guard: Filter expired 
entries and tombstones when
[Jul 24, 2019 6:32:28 PM] (github) HDDS-1853. Fix failing blockade test-cases. 
(#1151)
[Jul 24, 2019 9:03:40 PM] (github) HDDS-1819. Implement S3 Commit MPU request 
to use Cache and
[Jul 24, 2019 10:59:30 PM] (aengineer) HDDS-1817. GetKey fails with 
IllegalArgumentException.
[Jul 25, 2019 12:25:25 AM] (weichiu) HADOOP-16451. Update jackson-databind to 
2.9.9.1. Contributed by Siyao
[Jul 25, 2019 1:21:07 AM] (ayushsaxena) HDFS-14647. NPE during secure namenode 
startup. Contributed by Fengnan




-1 overall


The following subsystems voted -1:
asflicense findbugs hadolint pathlen unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml
 

FindBugs :

   
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-mawo/hadoop-yarn-applications-mawo-core
 
   Class org.apache.hadoop.applications.mawo.server.common.TaskStatus 
implements Cloneable but does not define or use clone method At 
TaskStatus.java:does not define or use clone method At TaskStatus.java:[lines 
39-346] 
   Equals method for 
org.apache.hadoop.applications.mawo.server.worker.WorkerId assumes the argument 
is of type WorkerId At WorkerId.java:the argument is of type WorkerId At 
WorkerId.java:[line 114] 
   
org.apache.hadoop.applications.mawo.server.worker.WorkerId.equals(Object) does 
not check for null argument At WorkerId.java:null argument At 
WorkerId.java:[lines 114-115] 

FindBugs :

   module:hadoop-tools/hadoop-aws 
   Inconsistent synchronization of 
org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore.ttlTimeProvider; locked 75% 
of time Unsynchronized access at LocalMetadataStore.java:75% of time 
Unsynchronized access at LocalMetadataStore.java:[line 623] 

Failed junit tests :

   hadoop.hdfs.server.datanode.TestBPOfferService 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.hdfs.server.datanode.TestDirectoryScanner 
   hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks 
   hadoop.fs.http.client.TestHttpFSFWithSWebhdfsFileSystem 
   hadoop.hdfs.server.federation.router.TestRouterWithSecureStartup 
   hadoop.hdfs.server.federation.security.TestRouterHttpDelegationToken 
   hadoop.yarn.server.nodemanager.amrmproxy.TestFederationInterceptor 
   hadoop.yarn.sls.appmaster.TestAMSimulator 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1208/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1208/artifact/out/diff-compile-javac-root.txt
  [332K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1208/artifact/out/diff-checkstyle-root.txt
  [17M]

   hadolint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1208/artifact/out/diff-patch-hadolint.txt
  [4.0K]

   pathlen:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1208/artifact/out/pathlen.txt
  [12K]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1208/artifact/out/diff-patch-pylint.txt
  [216K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1208/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1208/artifact/out/diff-patch-shelldocs.txt
  [44K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1208/artifact/out/whitespace-eol.txt
  [9.6M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1208/artifact/out/whitespace-tabs.txt
  [1.1M]

   xml:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1208/artifact/out/xml.txt
  [16K]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/120

LocatedFileStatusFetcher.getFileStatuses failing intermittently with s3

2019-07-25 Thread Steve Loughran
FYI, HADOOP-16458 : LocatedFileStatusFetcher.getFileStatuses failing
intermittently with s3

This is inevitably something up with S3A, but I'm going to be making
changes to the LocatedFileStatusFetcher code as well as o.a.h.fs.Globber to
help diagnose this, so it's stepping into MAPREDUCE land.

Two questions.

-there are no explicit unit tests of LocatedFileStatusFetcher doing scans
of object stores or filesystems. Is there anything I've not seen?
- the FileSystem globber has code which, if it does a listStatus(path) gets
a single entry, calls getFileStatus to get some more information, which the
docs say "needed to handle symlinks"

I don't know where we are with symlinks right now, because they aren't in
any object store, and disabled for HDFS.

What do people think if I actually removed that secondary check?

I may play with some subclassing games and just remove it for S3A, so it's
lower risk, while improving perf slightly. ABFS could copy.

Any thoughts?


Re: [DISCUSS] EOL 2.8 or another 2.8.x release?

2019-07-25 Thread Steve Loughran
I'm in favour of 1 more release (it fixes the off-by 1 bug in
S3AInputStream HADOOP-16109), but would be even happier if we could move
people to 2.9.x

maybe do a 2.9.x release and declare that 2.8 is EOL?


On Thu, Jul 25, 2019 at 2:08 PM Wei-Chiu Chuang  wrote:

> My bad -- Didn't realize I was looking at the old Hadoop page.
> Here's the correct list of releases.
> https://hadoop.apache.org/releases.html
>
> On Thu, Jul 25, 2019 at 12:49 AM 张铎(Duo Zhang) 
> wrote:
>
> > IIRC we have a 2.8.5 release?
> >
> > On the download page:
> >
> > 2.8.5 2018 Sep 15
> >
> > Wei-Chiu Chuang  于2019年7月25日周四 上午9:39写道:
> >
> > > The last 2.8 release (2.8.4) was made in the last May, more than a year
> > > ago. https://hadoop.apache.org/old/releases.html
> > >
> > > How do folks feel about the fate of branch-2.8? During the last
> community
> > > meetup in June, it sounds like most users are still on 2.8 or even 2.7,
> > so
> > > I don't think we want to abandon 2.8 just yet.
> > >
> > > I would personally want to urge folks to move up to 3.x, so I can stop
> > > cherrypicking stuff all the way down into 2.8. But it's not up to me
> > along
> > > to decide :)
> > >
> > > How do people feel about having another 2.8 release or two? I am not
> > saying
> > > I want to drive it, but I want to raise the awareness that folks are
> > still
> > > on 2.8 and there's not been an update for over a year.
> > >
> > > Thoughts?
> > >
> >
>


Apache Hadoop qbt Report: branch2+JDK7 on Linux/x86

2019-07-25 Thread Apache Jenkins Server
For more details, see 
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/393/

[Jul 24, 2019 4:21:29 PM] (jeagles) YARN-9563. Resource report REST API could 
return NaN or Inf (Ahmed




-1 overall


The following subsystems voted -1:
asflicense findbugs hadolint pathlen unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/empty-configuration.xml
 
   hadoop-tools/hadoop-azure/src/config/checkstyle-suppressions.xml 
   hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/public/crossdomain.xml 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/public/crossdomain.xml
 

FindBugs :

   
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client
 
   Boxed value is unboxed and then immediately reboxed in 
org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result,
 byte[], byte[], KeyConverter, ValueConverter, boolean) At 
ColumnRWHelper.java:then immediately reboxed in 
org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result,
 byte[], byte[], KeyConverter, ValueConverter, boolean) At 
ColumnRWHelper.java:[line 335] 

Failed junit tests :

   hadoop.ha.TestZKFailoverController 
   hadoop.contrib.bkjournal.TestBookKeeperJournalManager 
   hadoop.hdfs.shortcircuit.TestShortCircuitCache 
   hadoop.hdfs.tools.TestDFSAdminWithHA 
   hadoop.hdfs.TestDFSInotifyEventInputStream 
   hadoop.hdfs.TestDFSPermission 
   hadoop.hdfs.server.datanode.TestDirectoryScanner 
   hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys 
   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.hdfs.server.balancer.TestBalancerRPCDelay 
   hadoop.hdfs.TestHDFSFileSystemContract 
   hadoop.hdfs.TestDFSRollback 
   hadoop.hdfs.client.impl.TestBlockReaderRemote2 
   hadoop.hdfs.TestMissingBlocksAlert 
   hadoop.hdfs.TestFileCreationDelete 
   hadoop.contrib.bkjournal.TestBookKeeperJournalManager 
   hadoop.registry.secure.TestSecureLogins 
   hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/393/artifact/out/diff-compile-cc-root-jdk1.7.0_95.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/393/artifact/out/diff-compile-javac-root-jdk1.7.0_95.txt
  [328K]

   cc:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/393/artifact/out/diff-compile-cc-root-jdk1.8.0_212.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/393/artifact/out/diff-compile-javac-root-jdk1.8.0_212.txt
  [308K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/393/artifact/out/diff-checkstyle-root.txt
  [16M]

   hadolint:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/393/artifact/out/diff-patch-hadolint.txt
  [4.0K]

   pathlen:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/393/artifact/out/pathlen.txt
  [12K]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/393/artifact/out/diff-patch-pylint.txt
  [24K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/393/artifact/out/diff-patch-shellcheck.txt
  [72K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/393/artifact/out/diff-patch-shelldocs.txt
  [8.0K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/393/artifact/out/whitespace-eol.txt
  [12M]
   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/393/artifact/out/whitespace-tabs.txt
  [1.2M]

   xml:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/393/artifact/out/xml.txt
  [12K]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/393/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/393/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/393/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [4.0K]
   
https://builds.apache.org/job/ha

Re: [DISCUSS] EOL 2.8 or another 2.8.x release?

2019-07-25 Thread Wei-Chiu Chuang
My bad -- Didn't realize I was looking at the old Hadoop page.
Here's the correct list of releases. https://hadoop.apache.org/releases.html

On Thu, Jul 25, 2019 at 12:49 AM 张铎(Duo Zhang) 
wrote:

> IIRC we have a 2.8.5 release?
>
> On the download page:
>
> 2.8.5 2018 Sep 15
>
> Wei-Chiu Chuang  于2019年7月25日周四 上午9:39写道:
>
> > The last 2.8 release (2.8.4) was made in the last May, more than a year
> > ago. https://hadoop.apache.org/old/releases.html
> >
> > How do folks feel about the fate of branch-2.8? During the last community
> > meetup in June, it sounds like most users are still on 2.8 or even 2.7,
> so
> > I don't think we want to abandon 2.8 just yet.
> >
> > I would personally want to urge folks to move up to 3.x, so I can stop
> > cherrypicking stuff all the way down into 2.8. But it's not up to me
> along
> > to decide :)
> >
> > How do people feel about having another 2.8 release or two? I am not
> saying
> > I want to drive it, but I want to raise the awareness that folks are
> still
> > on 2.8 and there's not been an update for over a year.
> >
> > Thoughts?
> >
>


Re: Any thoughts making Submarine a separate Apache project?

2019-07-25 Thread Wangda Tan
Thanks everybody for sharing your thoughts. I saw positive feedbacks from
20+ contributors!

So I think we should move it forward, any suggestions about what we should
do?

Best,
Wangda

On Mon, Jul 22, 2019 at 5:36 PM neo  wrote:

> +1, This is neo from TiDB & TiKV community.
> Thanks Xun for bring this up.
>
> Our CNCF project's open source distributed KV storage system TiKV,
> Hadoop submarine's machine learning engine helps us to optimize data
> storage,
> helping us solve some problems in data hotspots and data shuffers.
>
> We are ready to improve the performance of TiDB in our open source
> distributed relational database TiDB and also using the hadoop submarine
> machine learning engine.
>
> I think if submarine can be independent, it will develop faster and better.
> Thanks to the hadoop community for developing submarine!
>
> Best Regards,
> neo
> www.pingcap.com / https://github.com/pingcap/tidb /
> https://github.com/tikv
>
> Xun Liu  于2019年7月22日周一 下午4:07写道:
>
> > @adam.antal
> >
> > The submarine development team has completed the following preparations:
> > 1. Established a temporary test repository on Github.
> > 2. Change the package name of hadoop submarine from org.hadoop.submarine
> to
> > org.submarine
> > 3. Combine the Linkedin/TonY code into the Hadoop submarine module;
> > 4. On the Github docked travis-ci system, all test cases have been
> tested;
> > 5. Several Hadoop submarine users completed the system test using the
> code
> > in this repository.
> >
> > 赵欣  于2019年7月22日周一 上午9:38写道:
> >
> > > Hi
> > >
> > > I am a teacher at Southeast University (https://www.seu.edu.cn/). We
> are
> > > a major in electrical engineering. Our teaching teams and students use
> > > bigoop submarine for big data analysis and automation control of
> > electrical
> > > equipment.
> > >
> > > Many thanks to the hadoop community for providing us with machine
> > learning
> > > tools like submarine.
> > >
> > > I wish hadoop submarine is getting better and better.
> > >
> > >
> > > ==
> > > 赵欣
> > > 东南大学电气工程学院
> > >
> > > -
> > >
> > > Zhao XIN
> > >
> > > School of Electrical Engineering
> > >
> > > ==
> > > 2019-07-18
> > >
> > >
> > > *From:* Xun Liu 
> > > *Date:* 2019-07-18 09:46
> > > *To:* xinzhao 
> > > *Subject:* Fwd: Re: Any thoughts making Submarine a separate Apache
> > > project?
> > >
> > >
> > > -- Forwarded message -
> > > 发件人: dashuiguailu...@gmail.com 
> > > Date: 2019年7月17日周三 下午3:17
> > > Subject: Re: Re: Any thoughts making Submarine a separate Apache
> project?
> > > To: Szilard Nemeth , runlin zhang <
> > > runlin...@gmail.com>
> > > Cc: Xun Liu , common-dev <
> > common-...@hadoop.apache.org>,
> > > yarn-dev , hdfs-dev <
> > > hdfs-...@hadoop.apache.org>, mapreduce-dev <
> > > mapreduce-dev@hadoop.apache.org>, submarine-dev <
> > > submarine-...@hadoop.apache.org>
> > >
> > >
> > > +1 ,Good idea, we are very much looking forward to it.
> > >
> > > --
> > > dashuiguailu...@gmail.com
> > >
> > >
> > > *From:* Szilard Nemeth 
> > > *Date:* 2019-07-17 14:55
> > > *To:* runlin zhang 
> > > *CC:* Xun Liu ; Hadoop Common
> > > ; yarn-dev ;
> > > Hdfs-dev ; mapreduce-dev
> > > ; submarine-dev
> > > 
> > > *Subject:* Re: Any thoughts making Submarine a separate Apache project?
> > > +1, this is a very great idea.
> > > As Hadoop repository has already grown huge and contains many
> projects, I
> > > think in general it's a good idea to separate projects in the early
> > phase.
> > >
> > >
> > > On Wed, Jul 17, 2019, 08:50 runlin zhang  wrote:
> > >
> > > > +1 ,That will be great !
> > > >
> > > > > 在 2019年7月10日,下午3:34,Xun Liu  写道:
> > > > >
> > > > > Hi all,
> > > > >
> > > > > This is Xun Liu contributing to the Submarine project for deep
> > learning
> > > > > workloads running with big data workloads together on Hadoop
> > clusters.
> > > > >
> > > > > There are a bunch of integrations of Submarine to other projects
> are
> > > > > finished or going on, such as Apache Zeppelin, TonY, Azkaban. The
> > next
> > > > step
> > > > > of Submarine is going to integrate with more projects like Apache
> > > Arrow,
> > > > > Redis, MLflow, etc. & be able to handle end-to-end machine learning
> > use
> > > > > cases like model serving, notebook management, advanced training
> > > > > optimizations (like auto parameter tuning, memory cache
> optimizations
> > > for
> > > > > large datasets for training, etc.), and make it run on other
> > platforms
> > > > like
> > > > > Kubernetes or natively on Cloud. LinkedIn also wants to donate TonY
> > > > project
> > > > > to Apache so we can put Submarine and TonY together to the same
> > > codebase
> > > > > (Page #30.
> > > > >
> > > >
> > >
> >
> https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond#30
> > > > > ).
> > > > >
> > > > > This expands the scope of the ori

Re: [DISCUSS] EOL 2.8 or another 2.8.x release?

2019-07-25 Thread Duo Zhang
IIRC we have a 2.8.5 release?

On the download page:

2.8.5 2018 Sep 15

Wei-Chiu Chuang  于2019年7月25日周四 上午9:39写道:

> The last 2.8 release (2.8.4) was made in the last May, more than a year
> ago. https://hadoop.apache.org/old/releases.html
>
> How do folks feel about the fate of branch-2.8? During the last community
> meetup in June, it sounds like most users are still on 2.8 or even 2.7, so
> I don't think we want to abandon 2.8 just yet.
>
> I would personally want to urge folks to move up to 3.x, so I can stop
> cherrypicking stuff all the way down into 2.8. But it's not up to me along
> to decide :)
>
> How do people feel about having another 2.8 release or two? I am not saying
> I want to drive it, but I want to raise the awareness that folks are still
> on 2.8 and there's not been an update for over a year.
>
> Thoughts?
>


Re: YARN+MR APAC Sync Up Meeting Minutes 07/25

2019-07-25 Thread Wanqiang Ji
Sorry for the previous mail with wrong format, I had fixed it.
—
Participants:
Cloudera, Tencent, Taiwan University, Xiaomi, ByteDance, JD, Didi, Alibaba

ByteDance:
   Version: 2.6.0
   1000+ patches
   Fair scheduler
   Improvements: like CS global scheduling
   Scale: 40,000 hosts last year!!
   Flink + Spark + MR + other computations
   Not server side federation: client side to route apps to clusters
   Largest cluster 10,000 nodes

Alibaba:
   Version: 3.1.0 (upgraded 1.5 year before)
   CS + Global scheduling
   Overcommitment with opportunistic containers -> increase utilization
   Scheduler activities (share more details in upcoming meetup)

Xiaomi:
   2.6.0 -> 3.1.0 (upgrading in progress)
   Migrate FS to CS
   Investigating federation
   TODO: Open umbrella for migration

Hadoop Meetup 8/10 discussion:
   Location TBD, most likely will be a one day event
   Setting up a review-board, we are glad to have 1 or 2 contract-point
from each company in this board, please nominate to junping...@apache.org
   Call for more talks!!!


Wanqiang Ji

Wanqiang Ji 于2019年7月25日 周四14:49写道:

> July 25, 2019
>
> Participants: Cloudera, Tencent, Taiwan University, Xiaomi, ByteDance, JD,
> Didi, Alibaba
>
> ByteDance
>
>-
>
>Version: 2.6.0
>-
>
>1000+ patches
>-
>
>Fair scheduler
>-
>
>   Improvements: like CS global scheduling
>   -
>
>Scale: 40,000 hosts last year!!
>-
>
>Flink + Spark + MR + other computations
>-
>
>Not server side federation: client side to route apps to clusters
>-
>
>Largest cluster 10,000 nodes
>
> Alibaba
>
>-
>
>Version: 3.1.0 (upgraded 1.5 year before)
>-
>
>CS + Global scheduling
>-
>
>Overcommitment with opportunistic containers -> increase utilization
>-
>
>Scheduler activities (share more details in upcoming meetup)
>
> Xiaomi
>
>-
>
>2.6.0 -> 3.1.0 (upgrading in progress)
>-
>
>Migrate FS to CS
>-
>
>Investigating federation
>-
>
>TODO: Open umbrella for migration
>
> Hadoop Meetup 8/10 discussion
>
>-
>
>Location TBD, most likely will be a one day event
>-
>
>Setting up a review-board, we are glad to have 1 or 2 contract-point
>from each company in this board, please nominate to
>junping...@apache.org
>-
>
>Call for more talks!!!
>
>
>
> FYI
> Wanqiang Ji
>
> 俊平堵 于2019年7月25日 周四11:53写道:
>
>> Hi Folks,
>>
>>  Kindly remind that we have YARN+MR APAC sync today, and you are
>> welcome to join:
>>
>>
>> Time and Date:07/25 1:00 pm (CST Time)
>>
>> Zoom link:Zoom | https://cloudera.zoom.us/j/880548968
>>
>> Summary:
>>
>> https://docs.google.com/document/d/1GY55sXrekVd-aDyRY7uzaX0hMDPyh3T-AL1kUY2TI5M
>>
>>
>> Thanks,
>>
>>
>> Junping
>>
>>
>>
>> Wangda Tan  于2019年6月28日周五 上午2:57写道:
>>
>> > Hi folks,
>> >
>> > Here's the Hadoop Community Sync Up proposal/schedule:
>> >
>> https://docs.google.com/document/d/1GfNpYKhNUERAEH7m3yx6OfleoF3MqoQk3nJ7xqHD9nY/edit#heading=h.xh4zfwj8ppmn
>> >
>> > And here's calendar file:
>> >
>> >
>> >
>> https://calendar.google.com/calendar/ical/hadoop.community.sync.up%40gmail.com/public/basic.ics
>> >
>> > We gave it a try this week for YARN+MR and Submarine sync, feedbacks
>> from
>> > participants seems pretty good, lots of new information shared during
>> sync
>> > up, and companies are using/developing Hadoop can better know each
>> other.
>> >
>> > Next week there're 4 community sync-ups (Two Submarine for different
>> > timezones, one YARN+MR, one storage), please join to whichever you're
>> > interested:
>> >
>> > [image: image.png]
>> >
>> > Zoom info and notes can be found in the Google calendar invitation.
>> >
>> > Thanks,
>> > Wangda
>> >
>>
>