Re: [VOTE] Release Apache Hadoop 3.2.2 - RC2

I suggest we wait for HDFS-15680 (If I understand it correctly, it's a blocker) I wish to update jetty too in this release, but it's turning out to be more involved than i had anticipated. On Fri, Nov 13, 2020 at 3:22 PM Chao Sun wrote: > +1

Re: This week's Hadoop Storage online meetup (& next one after)

On Wed, Oct 14, 2020 at 8:51 AM Wei-Chiu Chuang wrote: > Just a quick reminder the Dynamic disk-level tiering feature talk is > happening in about an hour! > > Join Zoom Meeting > > https://cloudera.zoom.us/j/880548968 > > One tap mobile > > +16465588656,,880548968#

Re: Mandarin Hadoop Storage Online Meetup this week (APAC Mandarin)

Thanks for organizing it, Xiaoqiao! Here's the recording: https://cloudera.zoom .us/rec/share/mkBZkURfl0A5-8i4jD5GbRggyy-WCjIe2EUCnZH73YNJJHvn1fRQkYKnVczZLh8P.Bkzdz89moGWvnABM Passcode: FXvGv5.B I wasn't able to join, but I'll watch the recording. On Tue, Nov 10, 2020 at 7:37 PM Xiaoqiao He

[jira] [Resolved] (HDFS-15485) Fix outdated properties of JournalNode when performing rollback

[ https://issues.apache.org/jira/browse/HDFS-15485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-15485. Fix Version/s: 3.2.3 3.1.5 3.3.1 Resolution

No Hadoop Storage Online Meetup this week.

Wednesday is Veterans Day in the US when many businesses will take the day off, so I'm not arranging a call. But feel free to suggest a topic for the next one! Wei-Chiu

Committers please subscribe to security@hadoop

The security@hadoop mailing list is restricted to Hadoop committers. Subscribing to the mailing list is not an automatic process. If you are a committer, please subscribe by sending an email to security-subscr...@hadoop.apache.org so we can discuss vulnerabilities in private. Thanks, Wei-Chiu

Fwd: Heads-up, Apache Yetus looking for feedback on RC with lots of changes

Forwarding to the Hadoop devs -- Forwarded message - From: Sean Busbey Date: Tue, Nov 3, 2020 at 9:51 AM Subject: Heads-up, Apache Yetus looking for feedback on RC with lots of changes To: dev Hi folks! FYI, Apache Yetus is doing RCs for their 0.13.0 release and it's got a

Re: This week's Hadoop Storage online meetup : IOStatistics API

Thanks Steve for today's discussion. For those who were able to join, here's the recording: https://cloudera.zoom.us/rec/share/TBbCfsg44BEgyBpFwZCe_uDX_uh-qPh8Wa2oP4IVR6lu7KGSviZfdR-YVdFbEoNN.JX2mPnDm913SDCt_ Passcode: u^a0^ On Wed, Oct 28, 2020 at 9:50 AM Wei-Chiu Chuang wrote: > Hi gen

Re: This week's Hadoop Storage online meetup : IOStatistics API

Hi gentle reminder: this is happening in 10 minutes. See you later! On Tue, Oct 27, 2020 at 11:20 AM Wei-Chiu Chuang wrote: > Hi! > > Here's the reminder for this week's call tomorrow. Steve will discuss the > new IOStatistics API (HADOOP-16830). Useful for developing Hadoop &g

Re: For review: HADOOP-17313. FileSystem.get to support slow-to-instantiate FS clients

I just reconfirmed with Steven. It's going to happen at 10am pacific time (10 minutes later) On Wed, Oct 28, 2020 at 9:10 AM Clay Baenziger (BLOOMBERG/ 919 3RD A) < cbaenzi...@bloomberg.net> wrote: > Oh I think I see the issue. Steve, you mean 11am PDT today (or 11am in the > Pacific Timezone)?

This week's Hadoop Storage online meetup : IOStatistics API

Hi! Here's the reminder for this week's call tomorrow. Steve will discuss the new IOStatistics API (HADOOP-16830). Useful for developing Hadoop applications. Time: Wednesday 10am pacific time; 5pm UTC See you then! Join Zoom Meeting https://cloudera.zoom.us/j/880548968 One tap mobile

Re: Fixing flaky tests in Apache Hadoop

:03 PM Wei-Chiu Chuang wrote: > Thanks for raising the issue, Akira and Ahmed, > > Fixing flaky tests is a thankless job so I want to take this opportunity > to recognize the time and effort. > > We will always have flaky tests due to bad tests or simply infra issues. > Fixing

Re: Fixing flaky tests in Apache Hadoop

Thanks for raising the issue, Akira and Ahmed, Fixing flaky tests is a thankless job so I want to take this opportunity to recognize the time and effort. We will always have flaky tests due to bad tests or simply infra issues. Fixing flaky tests will take time but if they are not addressed it

JIRA privilege for committers/PMCs

Hi, Is there any one who is a Hadoop committer or PMC, but who does not have the committer/administrator privilege (e.g. assign/resolve jiras)? JIRA projects like HADOOP has "hadoop" and "hadoop-pmc" groups added as administrators. [image: Screen Shot 2020-10-14 at 3.20.59 PM.png] So I would

Re: This week's Hadoop Storage online meetup (& next one after)

968> Dial by your location +1 646 558 8656 US (New York) +1 720 707 2699 US 877 853 5257 US Toll-free 888 475 4499 US Toll-free Meeting ID: 880 548 968 Find your local number: https://zoom.us/u/acaGRDfMVl On Mon, Oct 12, 2020 at 10:31 AM Wei-Chiu Chuang wrote:

Re: Wire compatibility between Hadoop 3.x client and 2.x server

Thanks Jialiang for reporting the issue. That sounds bad and should've not happened. Could you file a HDFS jira and fill in more details? On Mon, Oct 12, 2020 at 8:59 PM Wu,Jianliang(vip.com) < jianliang...@vipshop.com> wrote: > In our case, when nn has upgraded to 3.1.3 and dn’s version was

This week's Hadoop Storage online meetup (& next one after)

Hi! Here's the reminder for this week's call. We will have Leon and Ekanth from Uber talking about a new feature HDFS-15547 (Dynamic disk-level tiering) on Wednesday. Also Steve will be talking about the new IOStatistics API on Oct 28. In parallel, Xiaomi's Jinglun will talk about HDFS

Re: [E] Re: Increased DN heap usage during Hadoop 3 upgrade

ts? > > Kihwal > > On Mon, Oct 5, 2020 at 1:40 PM Wei-Chiu Chuang > wrote: > >> We experienced this issue on CDH6 and HDP3, so roughly Hadoop 3.0.x and >> 3.1.x. >> Hermanth experienced the same issue on Hadoop 3.1.1 as well (H

Re: Increased DN heap usage during Hadoop 3 upgrade

o you use? > > On Mon, Oct 5, 2020 at 10:03 AM Wei-Chiu Chuang > wrote: > >> I have anecdotally learned of multiple data points where during the >> upgrading from Hadoop 2 to Hadoop 3, DN heap usage increases to the point >> where it goes OOM. >> >> Don't have

Increased DN heap usage during Hadoop 3 upgrade

I have anecdotally learned of multiple data points where during the upgrading from Hadoop 2 to Hadoop 3, DN heap usage increases to the point where it goes OOM. Don't have much logs for this issue, but I suspect it's caused by the layout change added in Hadoop 2.8.0. Does anyone else observe the

Next Hadoop Storage Community Online Meetup

Hello, I'm happy to invite Uber's Leo and Ekanth to discuss a recent proposal HDFS-15547 (Dynamic disk-level tiering) in the next call on October 14th. Moving forward, I'll put the agenda of future meetups in this wiki: https://cwiki.apache.org/confluence/display/HADOOP/Meetup+agenda (where

Re: Hadoop 3.2.2 Release Code Freeze Plan

Thanks Xiaoqiao! Glad to see this is moving along. I noticed your dashboard has private filters and therefore the results are not visible publicly. On Tue, Sep 29, 2020 at 10:02 PM Xiaoqiao He wrote: > Hi All, > > Plan to code frozen for Hadoop-3.2.2 release at 2020/10/15. From now on, > most

Re: [VOTE] Moving Ozone to a separated Apache project

+1 Look forward to it. On Tue, Sep 29, 2020 at 4:48 PM Konstantin Shvachko wrote: > +1 > > Stay safe, > --Konstantin > > On Thu, Sep 24, 2020 at 10:59 PM Elek, Marton wrote: > > > Hi all, > > > > Thank you for all the feedback and requests, > > > > As we discussed in the previous thread(s)

Recent Performance Improvements in HDFS

People asked me what are the recent performance improvements in HDFS. I used to maintain a spreadsheet of these improvements, and I just updated it to incorporate the recent one. If you're interested, feel free to hover to

No Hadoop Storage Community Online Meetup this week & ApacheCon@Home

I should have sent out earlier. Given that this week's ApacheCon@Home conference, and there are a lot of great talks going to happen this week anyways, I did not arrange a call this week. https://apachecon.com/acah2020/tracks/bigdata-1.html https://apachecon.com/acah2020/tracks/bigdata-2.html

[ANNOUNCE] Hui Fei is a new Apache Hadoop Committer

y Sync Online > > https://docs.google.com/document/d/1jXM5Ujvf-zhcyw_5kiQVx6g-HeKe-YGnFS_1-qFXomI/edit#heading=h.irqxw1iy16zo > where >he discussed the Hadoop 3 rolling upgrade support > > Let's congratulate Hui for this new role! Cheers, Wei-Chiu Chuang (on behalf of the Apache Hadoop PMC)

[ANNOUNCE] Lisheng Sun is a new Apache Hadoop Committer

ssues.apache.org/jira/browse/HDFS-13694> and HDFS-13693 <https://issues.apache.org/jira/browse/HDFS-13693>. Code wise, he resolved 57 Hadoop jiras. Let's congratulate Lisheng for this new role! Cheers, Wei-Chiu Chuang (on behalf of the Apache Hadoop PMC)

Hadoop Storage Online Meetup in the Wiki

Hi! We've been running this call for over a year but I just realized we never managed to publish the information in a searchable location. So here it is in our wiki: https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Storage+Online+Meetup By the way, does anyone know of a good solution

Skipping this week’s APAC Hadoop storage online meetup

The Chinese Hadoop Meetup will take place this Saturday. So a call is not planned this week. If you are interested, feee free to sign up at https://www.slidestalk.com/m/290 (The event is in Mandarin) Thanks Xiaoqiao for organizing the next few calls & agenda.

Re: [VOTE] Release Apache Hadoop 2.10.1 (RC0)

+1 (binding) I did a security scan for the 2.10.1 RC0 and it looks fine to me. Checked recent critical/blocker HDFS issues that are not in 2.10.1. It looks mostly fine. Most of them are Hadoop 3.x features (EC, ... etc) but there is one worth attention: 1. HDFS-14674

Two more binding votes required for Apache Hadoop 2.10.1 RC0

Masatake is doing a great job rolling out 2.10.1 RC0. Let's give him a final push to get it out. Thanks, Wei-Chiu

Re: Sep. Y2020 Hadoop Meetup in China

This is great news! Can you also share this with the user@ and user-zh@ mailing lists? We need to get the message out. On Tue, Sep 15, 2020 at 10:36 PM 俊平堵 wrote: > Hi, > Last year Aug, we had a very successful meetup in Beijing ( > https://blogs.apache.org/hadoop) which help to promote

Today's Hadoop storage community meetup

Hello community, I was away on leave the past few months. I just came back last week and started to gear up. Don't really have topics in mind to discuss this week. But thought I should open up the bridge in case anyone want to join and chat. Let me know if you have a topic in mind you want to

Re: This Week's Hadoop Storage Online Meetup (APAC Mandarin)

gh as 6000ms. After the change, the latency reduced to less than 100ms. Future work: The implementation was based on 2.7. Need to forward port to newer releases for the community. Load balance between multiple disks. The slides will be shared and translated later. On Tue, Sep 8, 2020 at 7:08 AM Wei

This Week's Hadoop Storage Online Meetup (APAC Mandarin)

We are happy to have 李铭翔 leading the discussion on DataNode拆锁优化 (DataNode locking optimization) this week. Time/Date: 9/9 10PM (US west coast PST) 9/10 1PM (Beijing, China CST) Join Zoom Meeting https://cloudera.zoom.us/j/880548968

[ANNOUNCEMENT] Apache Hadoop 2.9.x release line end of life

to newer release lines: 2.10.0 / 3.1.4 / 3.2.1 / 3.3.0. Please check out our Release EOL wiki for details: https://cwiki.apache.org/confluence/display/HADOOP/EOL+(End-of-life)+Release+Branches Best Regards, Wei-Chiu Chuang (On Behalf of the Apache Hadoop PMC)

Re: [VOTE] End of Life Hadoop 2.9

; > binding > > > > On Mon, 31 Aug 2020 at 20:09, Wei-Chiu Chuang > wrote: > > > > > Dear fellow Hadoop developers, > > > > > > Given the overwhelming feedback from the discussion thread > > > https://s.apache.org/hadoop2.9eold, I'd like

Hadoop & ApacheCon

Hello, This year's ApacheCon will take place online between 9/29 and 10/1. There are lots of sessions made by our fellow Hadoop developers: https://apachecon.com/acah2020/tracks/bigdata-1.html https://apachecon.com/acah2020/tracks/bigdata-2.html In case you didn't realize, the registration is

[DISCUSS] Hadoop 3.2.2 release

Hi folks, I was reminded by Xiaoqiao that Hadoop 3.2.1 was made almost a year ago (released on September 22, 2019) and we're overdue for a follow-up. @Rohith Sharma K S you were the RM for Hadoop 3.2.1. Are we planning to make the 3.2.2 releases soon? Xiaoqiao wants to help with the release.

[DISCUSS] Hadoop 2.10.1 release

Hello, I see that Masatake graciously agreed to volunteer with the Hadoop 2.10.1 release work in the 2.9 branch EOL discussion thread https://s.apache.org/hadoop2.9eold Anyone else likes to contribute also? Thanks

[VOTE] End of Life Hadoop 2.9

Dear fellow Hadoop developers, Given the overwhelming feedback from the discussion thread https://s.apache.org/hadoop2.9eold, I'd like to start an official vote thread for the community to vote and start the 2.9 EOL process. What this entails: (1) an official announcement that no further

Re: [DISCUSS] fate of branch-2.9

, Weichiu On Fri, Mar 6, 2020 at 5:47 PM Wei-Chiu Chuang wrote: > I think that's a great suggestion. > Currently, we make 1 minor release per year, and within each minor release > we bring up 1 thousand to 2 thousand commits in it compared with the > previous one. > I can tot

Re: [Virtual MEETUP]: Migration to Hadoop 3

Thanks Brahma, Eric, do you have a target Hadoop 3 release line in mind? The "unofficial" plan here at Cloudera is to rebase our current dev codebase from Hadoop 3.1.1 to 3.3 some time later. The Hadoop 3.1 code line will approach its 3rd anniversary by this year's end so perhaps we can start to

Re: Mandarin Hadoop online sync this week

can port Ozone features into Hadoop Common. For example, Java-based configuration is a power feature which can benefit Hadoop as well. On Tue, Aug 25, 2020 at 9:47 AM Wei-Chiu Chuang wrote: > Hello, > > There hasn't been a Mandarin online sync for quite some time. I'd like to >

Re: Mandarin Hadoop online sync this week

, Aug 25, 2020 at 9:58 AM Brahma Reddy Battula wrote: > HI, > > what you are planning for this week? > > On Tue, Aug 25, 2020 at 10:18 PM Wei-Chiu Chuang > wrote: > >> Hello, >> >> There hasn't been a Mandarin online sync for quite some time. I'd like to >

Mandarin Hadoop online sync this week

Hello, There hasn't been a Mandarin online sync for quite some time. I'd like to call for one this week: Date/time: 8/27 Thursday Beijing Time 1PM 8/26 Wednesday US Pacific Time 10PM Link: https://cloudera.zoom.us/j/880548968 Past sync summary:

Re: ZStandard compression crashes

input size is bigger than > output size. By manually setting buffer size to a small value, input size > will be smaller than recommended output size to keep the system working. > By returning ZTD_CStreamOutSize() in getSteramSize, it may enable the > system to work without a predefined default.

[ANNOUNCE] New Apache Hadoop Committer - He Xiaoqiao

In bcc: general@ It's my pleasure to announce that He Xiaoqiao has been elected as a committer on the Apache Hadoop project recognizing his continued contributions to the project. Please join me in congratulating him. Hearty Congratulations & Welcome aboard Xiaoqiao! Wei-Chiu Chuang (On be

[jira] [Resolved] (HDFS-13639) SlotReleaser is not fast enough

[ https://issues.apache.org/jira/browse/HDFS-13639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-13639. Fix Version/s: 3.4.0 Resolution: Fixed > SlotReleaser is not fast eno

TestBalancerWithNodeGroup is failing due to HDFS-13183

Hi if you are curious why TestBalancerWithNodeGroup is failing in trunk and branch-3.3, it's because of HDFS-13181. Xiaoqiao is working on a fix now. Hoping to bring the branches back to normal soon.

[jira] [Resolved] (HDFS-15202) HDFS-client: boost ShortCircuit Cache

[ https://issues.apache.org/jira/browse/HDFS-15202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-15202. Fix Version/s: 3.4.0 Resolution: Fixed > HDFS-client: boost ShortCircuit Ca

Re: [DISCUSS] making Ozone a separate Apache project

+1 On Wed, May 13, 2020 at 8:32 AM anu engineer wrote: > +1 > —Anu > > > On May 13, 2020, at 12:53 AM, Elek, Marton wrote: > > > >  > > > > I would like to start a discussion to make a separate Apache project for > Ozone > > > > > > > > ### HISTORY [1] > > > > * Apache Hadoop Ozone

Re: Changing the default Datanode Volume Choosing policy

I don't think we have I/O-based balancing. That would surely make a great research project but it doesn't seem trivial to me. Also worth noting the implementation doesn't try to achieve very fine grained balance in space. As long as all volumes has the available space within a threshold (10GB by

Re: ZStandard compression crashes

gt; > -Ayush > > > On 11-May-2020, at 10:11 PM, Wei-Chiu Chuang wrote: > > > > Hadoop devs, > > > > A colleague of mine recently hit a strange issue where zstd compression > > codec crashes. >

ZStandard compression crashes

Hadoop devs, A colleague of mine recently hit a strange issue where zstd compression codec crashes. Caused by: java.lang.InternalError: Error (generic) at org.apache.hadoop.io.compress.zstd.ZStandardCompressor.deflateBytesDirect(Native Method) at

[jira] [Resolved] (HDFS-15334) INodeAttributeProvider's new API checkPermissionWithContext not getting called in for authorization

[ https://issues.apache.org/jira/browse/HDFS-15334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-15334. Fix Version/s: 3.3.0 Resolution: Fixed PR is merged. Cherrypicked to branch-3.3

[jira] [Created] (HDFS-15334) INodeAttributeProvider's new API checkPermissionWithContext not getting called in for authorization

Wei-Chiu Chuang created HDFS-15334: -- Summary: INodeAttributeProvider's new API checkPermissionWithContext not getting called in for authorization Key: HDFS-15334 URL: https://issues.apache.org/jira/browse/HDFS

Re: [VOTE] Release Apache Hadoop 3.1.4 (RC0)

Gabor, I'm sorry there's a test failure in branch-3.1 HDFS-14599 1. 1. I just cherrypicked the fix to branch-3.2 branch-3.1. It's a test only fix so technically I could live with it. But it would be best to add the fix to

[jira] [Resolved] (HDFS-15270) Account for *env == NULL in hdfsThreadDestructor

[ https://issues.apache.org/jira/browse/HDFS-15270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-15270. Fix Version/s: 3.4.0 Resolution: Fixed Thanks [~babsingh] this is in the trunk. Do

[jira] [Created] (HDFS-15317) Fix libhdfspp warnings

Wei-Chiu Chuang created HDFS-15317: -- Summary: Fix libhdfspp warnings Key: HDFS-15317 URL: https://issues.apache.org/jira/browse/HDFS-15317 Project: Hadoop HDFS Issue Type: Bug

Re: Next week's Hadoop storage community call: HDFS-15289

/vJx8deD5rG43SNySuQSDBPV9W9S0K_qs2ihLrPUIxBm3UHcKY1Ogb7YWMOPp1lMTvnpe4vlA6DI2Icfn?continueMode=true Password: 1i!?# Uma’s slides https://drive.google.com/open?id=1HGODu5KF6Djn0fup1WAl1U4rbcgQpcvQ On Wed, Apr 29, 2020 at 9:02 AM Wei-Chiu Chuang wrote: > Reminder: this call is going to happen in ab

Re: Next week's Hadoop storage community call: HDFS-15289

Reminder: this call is going to happen in about an hour. I see that LinkedIn is working on something similar. It would be great if the team can join the call and share your use cases so we can iron out potential problems. On Thu, Apr 23, 2020 at 11:56 AM Wei-Chiu Chuang wrote: > Hi! >

Re: [NOTICE] Removal of protobuf classes from Hadoop Token's public APIs' signature

in classpath. So downstream depending > on 2.5.0 version of protobuf still be able to access them. > > -vinay > > On Wed, 29 Apr 2020, 11:17 am Wei-Chiu Chuang, > wrote: > >> I'm sorry for coming to this late. I missed this message. It should have >> been a DISCUSS thread

Re: [NOTICE] Removal of protobuf classes from Hadoop Token's public APIs' signature

I'm sorry for coming to this late. I missed this message. It should have been a DISCUSS thread rather than NOTICE. Looks like this is inevitable. But we should make the downstream developers aware & make the update easier. As long as it is stated clearly how to update the code to support Hadoop

Re: Changing the default Datanode Volume Choosing policy

+1 to switch it on in Hadoop 3.4.0 (1) it doesn't break any existing applications I am aware of. (2) No noticeable performance regression in any cases observed. I feel compelled to make a feature the default if it is strictly better. Hopefully we can make Hadoop easier to use in this way too.

Re: Hadoop 3.1.4 Release Code Freeze

Gabor, can you also create the dot release branch so folks can start cherrypicking commits for 3.1.5? Thank you On Wed, Apr 22, 2020 at 3:54 AM Gabor Bota wrote: > Hi All, > > The code freeze for Hadoop 3.1.4 release is planned for today. There > some issues still open for that version. > >

Re: [Hadoop-3.3 Release update]- branch-3.3 has created

nks and regards, >>> Akira >>> >>> On Fri, Apr 17, 2020 at 11:11 AM Brahma Reddy Battula >>> wrote: >>> >>>> Hi All, >>>> >>>> we are down to two blockers issues now (YARN-10194 and YARN-9848) which >>>> a

Next week's Hadoop storage community call: HDFS-15289

Hi! I am inviting @Uma Maheswara Rao Gangumalla to talk about a new feature proposal HDFS-15289 and invite the community to discuss it. This feature is built upon ViewFS and overcomes its limitations. It is intended to bridge Ozone and HDFS,

[jira] [Resolved] (HDFS-15269) NameNode should check the authorization API version only once during initialization

[ https://issues.apache.org/jira/browse/HDFS-15269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-15269. Fix Version/s: 3.3.0 Resolution: Fixed PR is merged in trunk, and cherrypicked

[jira] [Created] (HDFS-15269) NameNode should check the authorization API version only once during initialization

Wei-Chiu Chuang created HDFS-15269: -- Summary: NameNode should check the authorization API version only once during initialization Key: HDFS-15269 URL: https://issues.apache.org/jira/browse/HDFS-15269

Re: [DISCUSS] Shade guava into hadoop-thirdparty

t APIs that will require recompiling all client > applications because they need to use shaded Guava classes? > > On Sat, Apr 4, 2020 at 12:13 PM Wei-Chiu Chuang > wrote: > >> Hi Hadoop devs, >> >> I spent a good part of the past 7 months working with a dozen of >> colleagues

[DISCUSS] Shade guava into hadoop-thirdparty

Hi Hadoop devs, I spent a good part of the past 7 months working with a dozen of colleagues to update the guava version in Cloudera's software (that includes Hadoop, HBase, Spark, Hive, Cloudera Manager ... more than 20+ projects) After 7 months, I finally came to a conclusion: Update to Hadoop

Re: Hadoop Storage community call

Here's the slides: https://www.slideshare.net/GaborBota/hadoop-330-s3a-community-call-202004 On Wed, Apr 1, 2020 at 11:43 AM Wei-Chiu Chuang wrote: > Thanks Gabor for today's discussion. > > Please find the recording here: > https://cloudera.zoom.us/rec/play/6ccrJOiuqD43SdeUsQSDVvQv

Re: Hadoop Storage community call

delete throttling to avoid thundering herd effect, S3Guard Fsck Future: IOStatistics, Vectored IO On Tue, Mar 24, 2020 at 5:44 PM Wei-Chiu Chuang wrote: > Hi! > > For the bi-weekly Hadoop Storage community next week, we'll do something > different this time: > > Gabor Bot

Re: Hadoop Storage community call

Reminder -- this call is happening in about 2 hours. Stay safe! Weichiu On Tue, Mar 24, 2020 at 5:44 PM Wei-Chiu Chuang wrote: > Hi! > > For the bi-weekly Hadoop Storage community next week, we'll do something > different this time: > > Gabor Bota is going to talk about &quo

Hadoop Storage community call

Hi! For the bi-weekly Hadoop Storage community next week, we'll do something different this time: Gabor Bota is going to talk about "404 Not Found" -- New and old issues with S3A; Current state and what's next. Stay tuned. Looking forward to seeing more cloud connector topics in the future.

2.10.1 release?

The 2.10.0 release went GA roughly 5 months ago. A bunch of issues were fixed. If any one's interested in being the Release Manager, I've made your work slightly easier: Here's a dashboard for 2.10.1 issues. I granted edit permission to all committers.

[jira] [Resolved] (HDFS-14587) Support fail fast when client wait ACK by pipeline over threshold

[ https://issues.apache.org/jira/browse/HDFS-14587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-14587. Resolution: Duplicate I believe this is a dup of HDFS-8311, so I"ll resolve this one.

[jira] [Reopened] (HDFS-15113) Missing IBR when NameNode restart if open processCommand async feature

[ https://issues.apache.org/jira/browse/HDFS-15113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang reopened HDFS-15113: Reopen to have the addendum tested. > Missing IBR when NameNode restart if o

Re: [ANNOUNCE] New Apache Hadoop Committer - Siyao Meng

Congrats! On 2020/03/20 17:24:16, Xiaoyu Yao wrote: > It's my pleasure to announce that Siyao Meng has been elected as committer > on the Apache Hadoop project recognizing his continued contributions to the > project. > > Please join me in congratulating him. > > Congratulations & Welcome

[jira] [Created] (HDFS-15230) Sanity check should not assume key base name can be derived from version name

Wei-Chiu Chuang created HDFS-15230: -- Summary: Sanity check should not assume key base name can be derived from version name Key: HDFS-15230 URL: https://issues.apache.org/jira/browse/HDFS-15230

Re: Next week's Hadoop storage community call: Consistent Read from Standby

Chuang wrote: > Just a reminder this is happening in about 2 hours. > > On Fri, Mar 13, 2020 at 4:20 PM Wei-Chiu Chuang > wrote: > >> Hi! >> >> Consistent Read from Standby is one of the major features that's going to >> land in the upcoming Hadoop 3.3

[jira] [Resolved] (HDFS-15208) Suppress bogus AbstractWadlGeneratorGrammarGenerator in KMS stderr in hdfs

[ https://issues.apache.org/jira/browse/HDFS-15208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-15208. Fix Version/s: 3.2.2 3.1.4 Resolution: Fixed > Suppress bo

Re: Next week's Hadoop storage community call: Consistent Read from Standby

Just a reminder this is happening in about 2 hours. On Fri, Mar 13, 2020 at 4:20 PM Wei-Chiu Chuang wrote: > Hi! > > Consistent Read from Standby is one of the major features that's going to > land in the upcoming Hadoop 3.3.0 release. > > I'm happy to announce that Konstantin

[jira] [Resolved] (HDFS-14743) Enhance INodeAttributeProvider/ AccessControlEnforcer Interface in HDFS to support Authorization of mkdir, rm, rmdir, copy, move etc...

[ https://issues.apache.org/jira/browse/HDFS-14743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-14743. Fix Version/s: 3.3.0 Resolution: Fixed Thanks [~xyao] and [~ste...@apache.org

Next week's Hadoop storage community call: Consistent Read from Standby

Hi! Consistent Read from Standby is one of the major features that's going to land in the upcoming Hadoop 3.3.0 release. I'm happy to announce that Konstantin graciously agreed to talk about the production experience with this feature. Please note that we will start this call at 11am pacific

Re: [DISCUSS] Accelerate Hadoop dependency updates

nor > CDH release? Or this is in preparation for the next major/minor release of > CDH? > > On Wed, Mar 11, 2020 at 5:45 PM Wei-Chiu Chuang > wrote: > >> FWIW we are updating guava in Spark and Hive at Cloudera. Don't know which >> Apache version are they going to

Re: [DISCUSS] Accelerate Hadoop dependency updates

r Hadoop 3.3.0 release than 3.2.1. > > On Tue, Mar 10, 2020 at 3:03 PM Wei-Chiu Chuang > wrote: > >> I'm not hearing any feedback so far, but I want to suggest: >> >> use hadoop-thirdparty repository to host any dependencies that are known >> to >> break compatib

Re: data node reads blocks sequentially from disk

Hi David, We talked a bit about a similar topic on DataNode sockets a while back. Any feedback on the DataNode disk access? On Thu, Mar 5, 2020 at 4:16 PM Mania Abdi wrote: > Hello everyone > > I have a question regarding HDFS, data node code version 2.7.2. I have > posted my question as Jira

This week's APAC Hadoop storage community online sync

Hi! Gentle reminder: Tomorrow's the APAC Hadoop storage community online sync. Date/time: March 12th 1PM (China) / 2PM (Japan) / 10:30AM (India) March 11th 10PM (US West Coast) Zoom link: https://cloudera.zoom.us/j/880548968 Past meeting minutes:

[NOTICE] branch HADOOP-15566-OpenTracing created

Hi devs, I forked a branch HADOOP-15566-OpenTracing from trunk. This branch is used to develop HADOOP-15566 (OpenTracing support in Hadoop)

Re: [DISCUSS] Accelerate Hadoop dependency updates

ava is in the classpath of every application. Thoughts? On Sat, Mar 7, 2020 at 9:31 AM Wei-Chiu Chuang wrote: > Hi Hadoop devs, > > I the past, Hadoop tends to be pretty far behind the latest versions of > dependencies. Part of that is due to the fear of the breaking changes > brought in by the

[ANNOUNCEMENT] Apache Hadoop 2.8.x release line end of life

to newer release lines: 2.10.0 / 3.1.3 / 3.2.1. Please check out our Release EOL wiki for details: https://cwiki.apache.org/confluence/display/HADOOP/EOL+(End-of-life)+Release+Branches Best Regards, Wei-Chiu Chuang (On Behalf of the Apache Hadoop PMC)

Re: [VOTE] EOL Hadoop branch-2.8

This is to conclude the vote result. with 10 +1 binding votes and 0 -1 votes. This vote is passed. I'll proceed with a formal announcement. On Mon, Mar 9, 2020 at 2:22 AM Sammi Chen wrote: > +1. > Thanks, Wei-Chiu. > > BR, > Sammi > > On Tue, Mar 3, 2020 at 2:48 AM W

[DISCUSS] Accelerate Hadoop dependency updates

Hi Hadoop devs, I the past, Hadoop tends to be pretty far behind the latest versions of dependencies. Part of that is due to the fear of the breaking changes brought in by the dependency updates. However, things have changed dramatically over the past few years. With more focus on security

[jira] [Created] (HDFS-15208) Supress bogus AbstractWadlGeneratorGrammarGenerator in KMS stderr in hdfs

Wei-Chiu Chuang created HDFS-15208: -- Summary: Supress bogus AbstractWadlGeneratorGrammarGenerator in KMS stderr in hdfs Key: HDFS-15208 URL: https://issues.apache.org/jira/browse/HDFS-15208 Project

Re: This week's Hadoop storage community call

. We also discussed the branch-2.9 EOL discussion. Brahma suggested to invite user@ mailing list to join the discussion. On Mon, Mar 2, 2020 at 1:59 PM Wei-Chiu Chuang wrote: > Hi! > > I'd like to use this week's community call as an opportunity to drive the > releases forward

Re: [DISCUSS] fate of branch-2.9

On Mon, Mar 2, 2020 at 5:12 PM, Wei-Chiu Chuang > wrote: Hi, > > Following the discussion to end branch-2.8, I want to start a discussion > around what's next with branch-2.9. I am hesitant to use the word "end of > life" but consider these facts: > > * 2.9.0 was re

Re: This week's Hadoop storage community call

Just.a reminder this call is happening in about an hour. On Mon, Mar 2, 2020 at 1:59 PM Wei-Chiu Chuang wrote: > Hi! > > I'd like to use this week's community call as an opportunity to drive the > releases forward. RMs: Gabor and Brahma, would you be able to join the call? &

[ANNOUNCE] New Apache Hadoop Committer - Stephen O'Donnell

In bcc: general@ It's my pleasure to announce that Stephen O'Donnell has been elected as committer on the Apache Hadoop project recognizing his continued contributions to the project. Please join me in congratulating him. Hearty Congratulations & Welcome aboard Stephen! Wei-Chiu Ch

<    1   2   3   4   5   6   7   8   >