Re: [DISCUSS] Migrate hadoop from log4j1 to log4j2

mitigating the security issues will prefer that. > On Jan 20, 2022, at 8:59 AM, Andrew Purtell wrote: > > Reload4J has fixed all of those CVEs without requiring an upgrade. > >> On Jan 20, 2022, at 5:56 AM, Duo Zhang wrote: >> >> There are 3 new CVEs for log4j1 re

Re: [DISCUSS] Migrate hadoop from log4j1 to log4j2

Reload4J has fixed all of those CVEs without requiring an upgrade. > On Jan 20, 2022, at 5:56 AM, Duo Zhang wrote: > > There are 3 new CVEs for log4j1 reported recently[1][2][3]. So I think it > is time to speed up the migration to log4j2 work[4] now. > > You can see the discussion on the

[jira] [Created] (HDFS-13933) [JDK 11] SWebhdfsFileSystem related tests fail with hostname verification problems for "localhost"

Andrew Purtell created HDFS-13933: - Summary: [JDK 11] SWebhdfsFileSystem related tests fail with hostname verification problems for "localhost" Key: HDFS-13933 URL: https://issues.apache.org/jira/b

[jira] [Created] (HDFS-13932) [JDK 11] Casts to BlockStoragePolicy[] in unit tests raise ClassCastExceptions

Andrew Purtell created HDFS-13932: - Summary: [JDK 11] Casts to BlockStoragePolicy[] in unit tests raise ClassCastExceptions Key: HDFS-13932 URL: https://issues.apache.org/jira/browse/HDFS-13932

Re: [DISCUSS] A final minor release off branch-2?

> From recent classpath isolation work, I was surprised to find out that many of our downstream projects (HBase, Tez, etc.) are still consuming many non-public, server side APIs of Hadoop, not saying the projects/products outside of hadoop ecosystem. Our API compatibility test does not (and should

[jira] [Created] (HDFS-11357) Secure Delete

Andrew Purtell created HDFS-11357: - Summary: Secure Delete Key: HDFS-11357 URL: https://issues.apache.org/jira/browse/HDFS-11357 Project: Hadoop HDFS Issue Type: New Feature

Re: [VOTE] Release Apache Hadoop 2.7.3 RC1

major release line. > > We could benefit from getting a patch on the compatibility doc that > addresses the HDFS audit log specifically. > > --Chris Nauroth > > On 8/18/16, 8:47 AM, "Andrew Purtell" <andrew.purt...@gmail.com> wrote: > > An incompat

Re: [VOTE] Release Apache Hadoop 2.7.3 RC1

An incompatible APIs change is developer unfriendly. An incompatible behavioral change is operator unfriendly. Historically, one dimension of incompatibility has had a lot more mindshare than the other. It's great that this might be changing for the better. Where I work when we move from one

Re: 2.7.3 release plan

As a downstream consumer of Apache Hadoop 2.7.x releases, I expect we would patch the release to revert HDFS-8791 before pushing it out to production. For what it's worth. On Fri, Apr 1, 2016 at 11:23 AM, Andrew Wang wrote: > One other thing I wanted to bring up

Re: Erasure coding in branch-2 [Was Re: [VOTE] Merge HDFS-7285 (erasure coding) branch to trunk]

Yes we can mostly likely help you. Please come over to dev@bigtop. > On Nov 4, 2015, at 12:40 PM, Andrew Wang wrote: > > We used to get help from Bigtop when it comes to integration testing. Do we > think that's possible for 2.8? > > On Wed, Nov 4, 2015 at 10:08 AM,

Fwd: Disk space leak when using HBase and HDFS ShortCircuit

Forwarded -- Forwarded message -- From: Vladimir Rodionov vrodio...@carrieriq.com Date: Wed, Jun 25, 2014 at 12:03 PM Subject: RE: Disk space leak when using HBase and HDFS ShortCircuit To: u...@hbase.apache.org u...@hbase.apache.org Apparently those file descriptors were

Re: Hadoop Summit EU

I'm curious what part of the below communication should restrict it to private lists? Anyway, use of dev@hbase for this was fine in my opinion. We try to discourage discussion on HBase private lists which should really be done on the dev lists according to the Apache Way. On Mon, Apr 7, 2014 at

Re: Issue with my username on my company provided dev box? (Was: ...)

The Apache Software Foundation takes branding seriously, we all know this. Making an inquiry about a possible, and I believe unintended, mis-branding issue involving Apache Hadoop artifacts is not a personal assault. The hysterical responses here have been unprofessional and disgraceful, and only

[jira] [Created] (HDFS-5578) [JDK8] Fix Javadoc errors caused by incorrect or illegal tags in doc comments

Andrew Purtell created HDFS-5578: Summary: [JDK8] Fix Javadoc errors caused by incorrect or illegal tags in doc comments Key: HDFS-5578 URL: https://issues.apache.org/jira/browse/HDFS-5578 Project

Re: [Proposal] Pluggable Namespace

Hi Doug, I recognize some of what we recently experienced on a HDFS matter in what Milind wrote even if this was not the appropriate forum for it. Odd mention of conspiracy theories aside, for people who may come to this thread later, perhaps you can recommend an appropriate public Apache

Re: Upgrade to protobuf 2.5.0 for the 2.1.0 release, HADOOP-9845

On Thu, Aug 8, 2013 at 3:34 PM, Roman Shaposhnik r...@apache.org wrote: I'd be huge +1 on a flag day migrating us to 2.5.x. We can run a set of validation tests in Bigtop on all components that could be affected by it. Hadoop and HBase are the obvious suspects. What else in the ecosystem

[jira] [Resolved] (HDFS-4672) Support tiered storage policies

[ https://issues.apache.org/jira/browse/HDFS-4672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HDFS-4672. -- Resolution: Later HDFS-2832 and new subtasks have picked up some ideas from here, we might

[jira] [Created] (HDFS-4744) TestBlockRecovery should bind ephemeral ports redux

Andrew Purtell created HDFS-4744: Summary: TestBlockRecovery should bind ephemeral ports redux Key: HDFS-4744 URL: https://issues.apache.org/jira/browse/HDFS-4744 Project: Hadoop HDFS Issue

[jira] [Created] (HDFS-4723) Occasional failure in TestDFSClientRetries#testGetFileChecksum because the number of available xcievers is set too low

Andrew Purtell created HDFS-4723: Summary: Occasional failure in TestDFSClientRetries#testGetFileChecksum because the number of available xcievers is set too low Key: HDFS-4723 URL: https://issues.apache.org

[jira] [Created] (HDFS-4718) TestHDFSCLI regexps reject valid user names

Andrew Purtell created HDFS-4718: Summary: TestHDFSCLI regexps reject valid user names Key: HDFS-4718 URL: https://issues.apache.org/jira/browse/HDFS-4718 Project: Hadoop HDFS Issue Type

Re: collision in the naming of '.snapshot' directory between hdfs snapshot and hbase snapshot

Thanks for the consideration but we've just committed a change to address this as HBASE-8352 On Wednesday, April 17, 2013, Harsh J wrote: Pardon my late inquisition here but since HBase already shipped out with a name .snapshots/, why do we force them to change it, and not rename HDFS'

Re: [VOTE] Release Apache Hadoop 2.0.4-alpha

I find that branch-2.0.4-alpha won't compile for me. o.a.h.yarn.server.resourcemanager.schduler.fifo.TestFifoScheduler is missing an import for ResourceRequest or ResourceRequest is not available on the branch. On Thu, Apr 11, 2013 at 4:27 PM, Vinod Kumar Vavilapalli vino...@hortonworks.com

Re: [VOTE] Release Apache Hadoop 2.0.4-alpha

Thanks Roman I'll use the tarball. On Friday, April 12, 2013, Roman Shaposhnik wrote: On Fri, Apr 12, 2013 at 12:32 PM, Andrew Purtell apurt...@apache.orgjavascript:; wrote: I find that branch-2.0.4-alpha won't compile for me

[jira] [Created] (HDFS-4672) Support tiered storage policies

Andrew Purtell created HDFS-4672: Summary: Support tiered storage policies Key: HDFS-4672 URL: https://issues.apache.org/jira/browse/HDFS-4672 Project: Hadoop HDFS Issue Type: New Feature

Re: [Vote] Merge branch-trunk-win to trunk

to branch-2? New features on a branch should be voted first, no? Thanks, --Konstantin On Mon, Mar 25, 2013 at 1:36 PM, Andrew Purtell apurt...@apache.org wrote: Noticed this too. Simply a 'public' modifier is missing, but it's unclear how this could not have been caught prior to check

Re: VOTE: HDFS-347 merge

Only once someone adds support for doing HDFS-347 style local reads which work on Windows will we consider merging HDFS-347 to branch-2. There's no chance of having both HDFS-347 and HDFS-2246 style local reads coexisting in branch-2? It would be nice if HDFS-347 was in branch-2 sooner rather

Re: VOTE: HDFS-347 merge

+1 (non binding) If this gets in, a backport to branch-2 would be most appreciated. On Sun, Feb 17, 2013 at 1:48 PM, Colin McCabe cmcc...@alumni.cmu.eduwrote: Hi all, I would like to merge the HDFS-347 branch back to trunk. It's been under intensive review and testing for several months.

Re: Release numbering for branch-2 releases

On Fri, Feb 1, 2013 at 2:34 AM, Tom White t...@cloudera.com wrote: Possibly the reason for Stack's consternation is that this is a Hadoop-specific versioning scheme, rather than a standard one like Semantic Versioning (http://semver.org/) which is more widely understood. If I can offer an

[jira] [Created] (HDFS-4394) QJM client tests require a bit more time in some environments

Andrew Purtell created HDFS-4394: Summary: QJM client tests require a bit more time in some environments Key: HDFS-4394 URL: https://issues.apache.org/jira/browse/HDFS-4394 Project: Hadoop HDFS

[jira] [Resolved] (HDFS-4394) QJM client tests require a bit more time in some environments

[ https://issues.apache.org/jira/browse/HDFS-4394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HDFS-4394. -- Resolution: Invalid Closing as invalid. Sorry for the noise. I didn't realize these tests

[jira] [Created] (HDFS-4392) Use NetUtils#getFreeSocketPort in MiniDFSCluster

Andrew Purtell created HDFS-4392: Summary: Use NetUtils#getFreeSocketPort in MiniDFSCluster Key: HDFS-4392 URL: https://issues.apache.org/jira/browse/HDFS-4392 Project: Hadoop HDFS Issue

Re: [VOTE] Merge HDFS-3077 (QuorumJournalManager) branch to trunk

Our position on the QJM is we've already taken delivery from the feature branch and will maintain a private HDFS fork of branch-2 if necessary, i.e. we don't have a significant stake in this discussion except at a meta level as potential contributors. This is a case study of why feature branch

Re: [VOTE] Merge HDFS-3077 (QuorumJournalManager) branch to trunk

Sorry, just to be clear our and we below refer to my employer, nothing to do with HBase. Please pardon any confusion. On Tuesday, October 9, 2012, Andrew Purtell wrote: Our position on the QJM is we've already taken delivery from the feature branch and will maintain a private HDFS fork

Re: [VOTE] Merge HDFS-3077 (QuorumJournalManager) branch to trunk

to the design from discussions with Suresh, Sanjay, Henry Robinson, Patrick Hunt, Ivan Kelly, Andrew Purtell, Flavio Junqueira, Ben Reed, Nicholas, Bikas, Brandon, and others. Additionally, special thanks to Andrew Purtell and Stephen Chu -- Best regards, - Andy Problems worthy of attack prove

Re: [VOTE] Merge HDFS-3077 (QuorumJournalManager) branch to trunk

I don't follow. For example the QJM and HA NameNode configuration are designed together to eliminate the SPOF in HDFS, within HDFS. I don't see how they make sense separately. On Thursday, September 27, 2012, Konstantin Shvachko wrote: The SPOF is in HDFS. This project is about shared storage

Re: [VOTE] Merge HDFS-3077 (QuorumJournalManager) branch to trunk

AM, Andrew Purtell apurt...@apache.orgjavascript:; wrote: Speaking as an Apache Hadoop user who must do something with the NameNode single point of failure this year, I don't subscribe to the view that moving that SPOF from the NameNode to a NFS filer is reasonable to ask of those

Re: Heads up: merge for QJM branch soon

We have been backporting Todd's HDFS-3077 branch changes on top of branch-2 for a while now, and testing the result in small clusters (5-10 nodes). Although we certainly have not had the test coverage Todd describes below that is their internal testing, we can add the datapoint that the QJM, in

testing HDFS-3077 on branch-2

Hello HDFS devs, Just a heads up we are trying out HDFS-3077 changes on branch-2. It appears that only HDFS-3049, HDFS-3190, HDFS-3571, and HDFS-3573 are needed on top of today's branch-2 head ahead of HDFS-3077 for the unit tests to pass. On to cluster tests next. -- Best regards, - Andy

Re: hsync in hadoop 2.0?

+1 for 2.0 For those trying to stabilize that for deployment early next year, commit of this only to trunk won't help as much, though a private backport could happen in that case. - Andy On Friday, June 1, 2012, lars hofhansl wrote: HDFS-744 adds support for true durable sync for

[jira] [Created] (HDFS-1795) Port 0.20-append changes onto 0.20-security-203

Port 0.20-append changes onto 0.20-security-203 --- Key: HDFS-1795 URL: https://issues.apache.org/jira/browse/HDFS-1795 Project: Hadoop HDFS Issue Type: Task Reporter: Andrew Purtell

Re: That append patch in 0.20

From: Arun C Murthy a...@yahoo-inc.com [...] Folks are welcome to port it on top of branch-0.20-security too. I believe that has been done by some hbase folks too. We ported 0.20-append patches on top of Y! 0.20.100.3: https://github.com/trendmicro/hadoop-common but stopped at HDFS-1346.

Re: Please join me in welcoming the following people as committers to the Hadoop project

Congratulations to all! Best regards, - Andy --- On Wed, 1/5/11, Ian Holsman had...@holsman.net wrote: From: Ian Holsman had...@holsman.net Subject: Please join me in welcoming the following people as committers to the Hadoop project To: mapreduce-...@hadoop.apache.org,

Re: Rolling a Hadoop 0.20.2

So the list of HDFS issues for next 0.20.x or 0.21 most relevant to HBase stability I have is: 127 200 (well, hflush) 630 793 Sound about right? Anything important I'm missing? - Andy - Original Message From: Todd Lipcon t...@cloudera.com To: