Thanks Yiqun for sharing with us this morning. The following is my note taken today. Feel free to update the Google doc in case I missed something.
https://docs.google.com/document/d/1jXM5Ujvf-zhcyw_5kiQVx6g-HeKe-YGnFS_1-qFXomI/edit?usp=sharing 11/30/2019 10/30 Attendee: Yiqun, Weichiu, Chao, Matt, Craig, Pifta Yiqun presented the configurations and liras that help optimize and stabilize large scale clusters at eBay. HDFS-13183 was brought up. It runs well at eBay. Even though community think Consistent Read from Standby would supersede this improvement, there’s still value to add it since there can be users who do not use CRFS or are using older releases. eBay: Queue time < 10 ms Uber avg Queue time > 100 ms eBay is looking into consistent read from standby soon. RBF or Ozone appears to be a big change so not considered. Additionally, federation is running well at eBay so not looking into RBF now. Looking into upgrading to Hadoop 2.9 or Hadoop 3 Ozone tested by Pinduoduo and JD in China. Talked about recent upstream Hadoop development: Namenode fine grained locking, OpenTracing, JDK11 On Wed, Oct 30, 2019 at 9:30 AM Wei-Chiu Chuang <weic...@apache.org> wrote: > Gentle reminder. Yiqun will present in 30 minutes! > > On Mon, Oct 28, 2019 at 7:41 PM Wei-Chiu Chuang <weic...@apache.org> > wrote: > >> Hello, I am super stoked to have Yiqun Lin with us this Wednesday morning >> Oct 30 US Pacific 10am/CET (Budapest) 6pm/ IST (Banglore) 10:30pm/ CST >> (Beijing) Oct 31 1am / JST (Tokyo) 2am to talk about “HDFS Cluster >> Optimization in eBay” — Yiqun happens to be in the bay area this week and >> this is the same talk that he is going to present Tuesday night at Yahoo >> this week. >> >> HDFS Cluster Optimization in eBay >> >> >> >> Yiqun Lin, Hadoop Team, eBay + Apache Hadoop Committer / PMC member >>> On eBay, we have many large HDFS clusters with thousands of nodes. We >>> face many stability/data availability problems in our cluster. Today we >>> want to share some optimizations we did in the system layer or HDFS level >>> to improve our clusters. Besides, that makes our cluster more stable than >>> before. >> >> >> Past meeting notes and zoom link: >> >> https://docs.google.com/document/d/1jXM5Ujvf-zhcyw_5kiQVx6g-HeKe-YGnFS_1-qFXomI/edit?usp=sharing >> >> Best, >> Weichiu >> >
