[jira] [Commented] (HDFS-916) Rewrite DFSOutputStream to use a single thread with NIO

2019-08-12 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16905664#comment-16905664 ] Todd Lipcon commented on HDFS-916: -- I doubt anyone is working on this at this point (I certainly am not,

[jira] [Commented] (HDFS-12046) Hadoop CRC implementation using Intel ISA-L library

2019-06-18 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-12046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866284#comment-16866284 ] Todd Lipcon commented on HDFS-12046: What does the assembly look like for the ISAL variiant? Is it

[jira] [Commented] (HDFS-14531) Datanode's ScanInfo requires excessive memory

2019-06-11 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16861176#comment-16861176 ] Todd Lipcon commented on HDFS-14531: It seems the DirectoryScanner serves two purposes: (1) detect

[jira] [Resolved] (HDFS-14535) The default 8KB buffer in requestFileDescriptors#BufferedOutputStream is causing lots of heap allocation in HBase when using short-circut read

2019-06-04 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon resolved HDFS-14535. Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.3.0 > The default 8KB buffer

[jira] [Commented] (HDFS-14535) The default 8KB buffer in requestFileDescriptors#BufferedOutputStream is causing lots of heap allocation in HBase when using short-circut read

2019-06-04 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16855848#comment-16855848 ] Todd Lipcon commented on HDFS-14535: bq. OK, I considered something wrong here. It's a big HFile and

[jira] [Commented] (HDFS-14535) The default 8KB buffer in requestFileDescriptors#BufferedOutputStream is causing lots of heap allocation in HBase when using short-circut read

2019-06-03 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16855018#comment-16855018 ] Todd Lipcon commented on HDFS-14535: +1, Seems like an improvement to me. One thing I wonder, though

[jira] [Updated] (HDFS-14482) Crash when using libhdfs with bad classpath

2019-05-14 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-14482: --- Affects Version/s: 3.3.0 Fix Version/s: 3.3.0 > Crash when using libhdfs with bad classpath

[jira] [Resolved] (HDFS-14482) Crash when using libhdfs with bad classpath

2019-05-14 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon resolved HDFS-14482. Resolution: Fixed > Crash when using libhdfs with bad classpath >

[jira] [Commented] (HDFS-14482) Crash when using libhdfs with bad classpath

2019-05-08 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836076#comment-16836076 ] Todd Lipcon commented on HDFS-14482: Seems the 'if (!state->env)' check just needs to be moved up

[jira] [Created] (HDFS-14482) Crash when using libhdfs with bad classpath

2019-05-08 Thread Todd Lipcon (JIRA)
Todd Lipcon created HDFS-14482: -- Summary: Crash when using libhdfs with bad classpath Key: HDFS-14482 URL: https://issues.apache.org/jira/browse/HDFS-14482 Project: Hadoop HDFS Issue Type: Bug

[jira] [Commented] (HDFS-12628) libhdfs crashes on thread exit for JNI+libhdfs applications

2019-05-07 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-12628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16835112#comment-16835112 ] Todd Lipcon commented on HDFS-12628: [~joemcdonnell] I tried running your test program against Hadoop

[jira] [Updated] (HDFS-14417) CryptoInputStream decrypt methods can avoid usage of intermediate buffers

2019-04-09 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-14417: --- Component/s: performance > CryptoInputStream decrypt methods can avoid usage of intermediate buffers

[jira] [Updated] (HDFS-14394) Add -std=c99 / -std=gnu99 to libhdfs compile flags

2019-04-03 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-14394: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.3.0 Status: Resolved

[jira] [Commented] (HDFS-14394) Add -std=c99 / -std=gnu99 to libhdfs compile flags

2019-04-02 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16808180#comment-16808180 ] Todd Lipcon commented on HDFS-14394: One more piece of related info here is that the concern about

[jira] [Commented] (HDFS-14394) Add -std=c99 / -std=gnu99 to libhdfs compile flags

2019-04-02 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16808173#comment-16808173 ] Todd Lipcon commented on HDFS-14394: BTW I should have said I'm +1 on this patch. I'll commit unless

[jira] [Commented] (HDFS-14394) Add -std=c99 / -std=gnu99 to libhdfs compile flags

2019-04-02 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16808172#comment-16808172 ] Todd Lipcon commented on HDFS-14394: I'm pretty certain that depending on GNU extensions to the C

[jira] [Commented] (HDFS-14394) Add -std=c99 / -std=gnu99 to libhdfs compile flags

2019-03-28 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16804190#comment-16804190 ] Todd Lipcon commented on HDFS-14394: yea, looks like GCC5 turns on C99 mode by default. Tested gcc

[jira] [Updated] (HDFS-14304) High lock contention on hdfsHashMutex in libhdfs

2019-03-26 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-14304: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.3.0 Status: Resolved

[jira] [Updated] (HDFS-14348) Fix JNI exception handling issues in libhdfs

2019-03-26 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-14348: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.3.0 Status: Resolved

[jira] [Commented] (HDFS-13616) Batch listing of multiple directories

2019-03-13 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16791908#comment-16791908 ] Todd Lipcon commented on HDFS-13616: Another data point here: this would be very useful for the Hive

[jira] [Commented] (HDFS-14111) hdfsOpenFile on HDFS causes unnecessary IO from file offset 0

2019-02-26 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778955#comment-16778955 ] Todd Lipcon commented on HDFS-14111: BTW does this also end up fixing HDFS-14083 along the way? It

[jira] [Commented] (HDFS-14111) hdfsOpenFile on HDFS causes unnecessary IO from file offset 0

2019-02-25 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16777375#comment-16777375 ] Todd Lipcon commented on HDFS-14111: {quote} jthr = newJavaStr(env, "in:readbytebuffer", );

[jira] [Updated] (HDFS-14083) libhdfs logs errors when opened FS doesn't support ByteBufferReadable

2019-02-11 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-14083: --- Component/s: (was: test) libhdfs > libhdfs logs errors when opened FS doesn't

[jira] [Commented] (HDFS-14083) libhdfs logs errors when opened FS doesn't support ByteBufferReadable

2019-02-11 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765296#comment-16765296 ] Todd Lipcon commented on HDFS-14083: {quote} if (errno != 0) { file->flags &=

[jira] [Commented] (HDFS-14111) hdfsOpenFile on HDFS causes unnecessary IO from file offset 0

2018-12-11 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718160#comment-16718160 ] Todd Lipcon commented on HDFS-14111: Thanks for the links to those other initiatives, Steve. I think,

[jira] [Commented] (HDFS-14111) hdfsOpenFile on HDFS causes unnecessary IO from file offset 0

2018-11-28 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702237#comment-16702237 ] Todd Lipcon commented on HDFS-14111: [~ste...@apache.org] would be curious on your take on this.

[jira] [Created] (HDFS-14111) hdfsOpenFile on HDFS causes unnecessary IO from file offset 0

2018-11-28 Thread Todd Lipcon (JIRA)
Todd Lipcon created HDFS-14111: -- Summary: hdfsOpenFile on HDFS causes unnecessary IO from file offset 0 Key: HDFS-14111 URL: https://issues.apache.org/jira/browse/HDFS-14111 Project: Hadoop HDFS

[jira] [Resolved] (HDFS-10369) hdfsread crash when reading data reaches to 128M

2018-11-28 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon resolved HDFS-10369. Resolution: Invalid You're mallocing a buffer of 5 bytes here, seems your C code is just broken.

[jira] [Commented] (HDFS-13826) Add a hidden configuration for NameNode to generate fake block locations

2018-08-15 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16580755#comment-16580755 ] Todd Lipcon commented on HDFS-13826: Thanks for pointing me at Dynamometer. That's quite interesting

[jira] [Created] (HDFS-13826) Add a hidden configuration for NameNode to generate fake block locations

2018-08-14 Thread Todd Lipcon (JIRA)
Todd Lipcon created HDFS-13826: -- Summary: Add a hidden configuration for NameNode to generate fake block locations Key: HDFS-13826 URL: https://issues.apache.org/jira/browse/HDFS-13826 Project: Hadoop

[jira] [Updated] (HDFS-13747) Statistic for list_located_status is incremented incorrectly by listStatusIterator

2018-07-18 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-13747: --- Tags: (was: newbie) > Statistic for list_located_status is incremented incorrectly by >

[jira] [Updated] (HDFS-13747) Statistic for list_located_status is incremented incorrectly by listStatusIterator

2018-07-18 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-13747: --- Description: The DirListingIterator constructor calls

[jira] [Updated] (HDFS-13747) Statistic for list_located_status is incremented incorrectly by listStatusIterator

2018-07-18 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-13747: --- Labels: newbie (was: ) > Statistic for list_located_status is incremented incorrectly by >

[jira] [Created] (HDFS-13747) Statistic for list_located_status is incremented incorrectly by listStatusIterator

2018-07-18 Thread Todd Lipcon (JIRA)
Todd Lipcon created HDFS-13747: -- Summary: Statistic for list_located_status is incremented incorrectly by listStatusIterator Key: HDFS-13747 URL: https://issues.apache.org/jira/browse/HDFS-13747

[jira] [Commented] (HDFS-13703) Avoid allocation of CorruptedBlocks hashmap when no corrupted blocks are hit

2018-06-27 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525822#comment-16525822 ] Todd Lipcon commented on HDFS-13703: Different set of random tests failed this time. I am guessing

[jira] [Commented] (HDFS-13702) HTrace hooks taking 10-15% CPU in DFS client when disabled

2018-06-27 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525820#comment-16525820 ] Todd Lipcon commented on HDFS-13702: The test failure seems unrelated. Given I have a +1 on this from

[jira] [Commented] (HDFS-13702) HTrace hooks taking 10-15% CPU in DFS client when disabled

2018-06-27 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525569#comment-16525569 ] Todd Lipcon commented on HDFS-13702: {quote}I do want a trace layer in there; I do want it broader

[jira] [Commented] (HDFS-13702) HTrace hooks taking 10-15% CPU in DFS client when disabled

2018-06-27 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525536#comment-16525536 ] Todd Lipcon commented on HDFS-13702: Perhaps using '-nsu' on all the patch builds? Would be nice if

[jira] [Commented] (HDFS-13702) HTrace hooks taking 10-15% CPU in DFS client when disabled

2018-06-27 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525524#comment-16525524 ] Todd Lipcon commented on HDFS-13702: Looked into this a bit more and found the issue: the build

[jira] [Commented] (HDFS-13703) Avoid allocation of CorruptedBlocks hashmap when no corrupted blocks are hit

2018-06-27 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16525498#comment-16525498 ] Todd Lipcon commented on HDFS-13703: Reattached the same patch to re-run Jenkins > Avoid allocation

[jira] [Updated] (HDFS-13703) Avoid allocation of CorruptedBlocks hashmap when no corrupted blocks are hit

2018-06-27 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-13703: --- Attachment: hdfs-13703.patch > Avoid allocation of CorruptedBlocks hashmap when no corrupted blocks

[jira] [Commented] (HDFS-13702) HTrace hooks taking 10-15% CPU in DFS client when disabled

2018-06-26 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16524376#comment-16524376 ] Todd Lipcon commented on HDFS-13702: Not sure what's up with the above precommit. It appears like it

[jira] [Commented] (HDFS-13703) Avoid allocation of CorruptedBlocks hashmap when no corrupted blocks are hit

2018-06-26 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16524351#comment-16524351 ] Todd Lipcon commented on HDFS-13703: Anyone know if the failed tests are known flakes? Doesn't seem

[jira] [Updated] (HDFS-13702) HTrace hooks taking 10-15% CPU in DFS client when disabled

2018-06-26 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-13702: --- Attachment: hdfs-13702.patch > HTrace hooks taking 10-15% CPU in DFS client when disabled >

[jira] [Commented] (HDFS-13703) Avoid allocation of CorruptedBlocks hashmap when no corrupted blocks are hit

2018-06-26 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16524182#comment-16524182 ] Todd Lipcon commented on HDFS-13703: BTW would be better if we could entirely avoid allocating the

[jira] [Commented] (HDFS-13703) Avoid allocation of CorruptedBlocks hashmap when no corrupted blocks are hit

2018-06-26 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16524176#comment-16524176 ] Todd Lipcon commented on HDFS-13703: Before/after perf stat results from teravalidate running on LJR

[jira] [Updated] (HDFS-13702) HTrace hooks taking 10-15% CPU in DFS client when disabled

2018-06-26 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-13702: --- Attachment: hdfs-13702.patch > HTrace hooks taking 10-15% CPU in DFS client when disabled >

[jira] [Updated] (HDFS-13703) Avoid allocation of CorruptedBlocks hashmap when no corrupted blocks are hit

2018-06-26 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-13703: --- Attachment: hdfs-13703.patch > Avoid allocation of CorruptedBlocks hashmap when no corrupted blocks

[jira] [Updated] (HDFS-13703) Avoid allocation of CorruptedBlocks hashmap when no corrupted blocks are hit

2018-06-26 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-13703: --- Status: Patch Available (was: Open) > Avoid allocation of CorruptedBlocks hashmap when no corrupted

[jira] [Commented] (HDFS-13702) HTrace hooks taking 10-15% CPU in DFS client when disabled

2018-06-26 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16524147#comment-16524147 ] Todd Lipcon commented on HDFS-13702: Here are some perf results based on a 180GB teravalidate on a

[jira] [Updated] (HDFS-13702) HTrace hooks taking 10-15% CPU in DFS client when disabled

2018-06-26 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-13702: --- Status: Patch Available (was: Open) > HTrace hooks taking 10-15% CPU in DFS client when disabled >

[jira] [Assigned] (HDFS-13702) HTrace hooks taking 10-15% CPU in DFS client when disabled

2018-06-26 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon reassigned HDFS-13702: -- Assignee: Todd Lipcon > HTrace hooks taking 10-15% CPU in DFS client when disabled >

[jira] [Updated] (HDFS-13702) HTrace hooks taking 10-15% CPU in DFS client when disabled

2018-06-26 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-13702: --- Attachment: hdfs-13702.patch > HTrace hooks taking 10-15% CPU in DFS client when disabled >

[jira] [Created] (HDFS-13703) Avoid allocation of CorruptedBlocks hashmap when no corrupted blocks are hit

2018-06-26 Thread Todd Lipcon (JIRA)
Todd Lipcon created HDFS-13703: -- Summary: Avoid allocation of CorruptedBlocks hashmap when no corrupted blocks are hit Key: HDFS-13703 URL: https://issues.apache.org/jira/browse/HDFS-13703 Project:

[jira] [Commented] (HDFS-13702) HTrace hooks taking 10-15% CPU in DFS client when disabled

2018-06-26 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16524128#comment-16524128 ] Todd Lipcon commented on HDFS-13702: Removing all the HTrace stuff from the block reader path

[jira] [Created] (HDFS-13702) HTrace hooks taking 10-15% CPU in DFS client when disabled

2018-06-26 Thread Todd Lipcon (JIRA)
Todd Lipcon created HDFS-13702: -- Summary: HTrace hooks taking 10-15% CPU in DFS client when disabled Key: HDFS-13702 URL: https://issues.apache.org/jira/browse/HDFS-13702 Project: Hadoop HDFS

[jira] [Created] (HDFS-13701) Removal of logging guards regressed performance

2018-06-26 Thread Todd Lipcon (JIRA)
Todd Lipcon created HDFS-13701: -- Summary: Removal of logging guards regressed performance Key: HDFS-13701 URL: https://issues.apache.org/jira/browse/HDFS-13701 Project: Hadoop HDFS Issue Type:

[jira] [Commented] (HDFS-13616) Batch listing of multiple directories

2018-05-29 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494327#comment-16494327 ] Todd Lipcon commented on HDFS-13616: bq. In batch mode, user can submit any file system calls. All

[jira] [Commented] (HDFS-13616) Batch listing of multiple directories

2018-05-29 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493863#comment-16493863 ] Todd Lipcon commented on HDFS-13616: bq. maybe getSourcePath? getListedPath? Yea, I think

[jira] [Commented] (HDFS-13616) Batch listing of multiple directories

2018-05-25 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491009#comment-16491009 ] Todd Lipcon commented on HDFS-13616: bq. Why not just RemoteIterator? In an earlier prototype of this

[jira] [Comment Edited] (HDFS-13616) Batch listing of multiple directories

2018-05-24 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489981#comment-16489981 ] Todd Lipcon edited comment on HDFS-13616 at 5/24/18 11:32 PM: -- Actually

[jira] [Commented] (HDFS-13616) Batch listing of multiple directories

2018-05-24 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489981#comment-16489981 ] Todd Lipcon commented on HDFS-13616: Actually collecting that was easier than I thought. I found a

[jira] [Commented] (HDFS-13616) Batch listing of multiple directories

2018-05-24 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489969#comment-16489969 ] Todd Lipcon commented on HDFS-13616: [~zhz] My feeling on these sorts of APIs is that a user who wants

[jira] [Commented] (HDFS-13616) Batch listing of multiple directories

2018-05-24 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489530#comment-16489530 ] Todd Lipcon commented on HDFS-13616: One other performance point: in addition to reducing "wait time"

[jira] [Commented] (HDFS-13611) Unsafe use of Text as a ConcurrentHashMap key in PBHelperClient

2018-05-23 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-13611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16487867#comment-16487867 ] Todd Lipcon commented on HDFS-13611: +1, looks good. While we're at it, we could reduce one extra

[jira] [Resolved] (HDFS-3653) 1.x: Add a retention period for purged edit logs

2018-04-23 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-3653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon resolved HDFS-3653. --- Resolution: Won't Fix > 1.x: Add a retention period for purged edit logs >

[jira] [Assigned] (HDFS-3041) DFSOutputStream.close doesn't properly handle interruption

2018-04-23 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon reassigned HDFS-3041: - Assignee: (was: Todd Lipcon) > DFSOutputStream.close doesn't properly handle interruption >

[jira] [Resolved] (HDFS-3069) If an edits file has more edits in it than expected by its name, should trigger an error

2018-04-23 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-3069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon resolved HDFS-3069. --- Resolution: Won't Fix Target Version/s: (was: ) > If an edits file has more edits in it

[jira] [Assigned] (HDFS-3447) StandbyException should not be logged at ERROR level on server

2018-04-23 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-3447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon reassigned HDFS-3447: - Assignee: (was: Todd Lipcon) > StandbyException should not be logged at ERROR level on server

[jira] [Updated] (HDFS-5058) QJM should validate startLogSegment() more strictly

2018-04-23 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-5058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-5058: -- Resolution: Won't Fix Status: Resolved (was: Patch Available) > QJM should validate

[jira] [Commented] (HDFS-12737) Thousands of sockets lingering in TIME_WAIT state due to frequent file open operations

2017-11-13 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-12737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250627#comment-16250627 ] Todd Lipcon commented on HDFS-12737: Not following what you mean by "implement multiplexing in the

[jira] [Commented] (HDFS-12737) Thousands of sockets lingering in TIME_WAIT state due to frequent file open operations

2017-11-01 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-12737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235221#comment-16235221 ] Todd Lipcon commented on HDFS-12737: In the data transfer protocol we just pass tokens with each

[jira] [Commented] (HDFS-2970) Contending block synchronizations can result in scary log messages

2017-04-27 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-2970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15987793#comment-15987793 ] Todd Lipcon commented on HDFS-2970: --- Yep, if my memory from 5 years ago serves me correctly, we can just

[jira] [Commented] (HDFS-2173) saveNamespace should not throw IOE when only one storage directory fails to write VERSION file

2016-05-03 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-2173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269962#comment-15269962 ] Todd Lipcon commented on HDFS-2173: --- Hrm, the original test/bug was filed >4.5 years ago so I'm a bit

[jira] [Assigned] (HDFS-916) Rewrite DFSOutputStream to use a single thread with NIO

2016-04-29 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon reassigned HDFS-916: Assignee: (was: Todd Lipcon) > Rewrite DFSOutputStream to use a single thread with NIO >

[jira] [Commented] (HDFS-7661) [umbrella] support hflush and hsync for erasure coded files

2016-04-06 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15229490#comment-15229490 ] Todd Lipcon commented on HDFS-7661: --- I'm not super active on either HDFS or HBase anymore, but Kai asked

[jira] [Assigned] (HDFS-3743) QJM: improve formatting behavior for JNs

2015-06-16 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-3743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon reassigned HDFS-3743: - Assignee: (was: Todd Lipcon) No progress from my side. This didn't turn out to be a highly

[jira] [Commented] (HDFS-7966) New Data Transfer Protocol via HTTP/2

2015-06-16 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14588913#comment-14588913 ] Todd Lipcon commented on HDFS-7966: --- +1 to what Stack said. We already have many access

[jira] [Commented] (HDFS-3743) QJM: improve formatting behavior for JNs

2015-06-16 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-3743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14589097#comment-14589097 ] Todd Lipcon commented on HDFS-3743: --- See my comment above on a proposed design. I'd

[jira] [Commented] (HDFS-8132) Namenode Startup Failing When we add Jcarder.jar in class Path

2015-04-14 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14495504#comment-14495504 ] Todd Lipcon commented on HDFS-8132: --- Yea, I'm guessing the same issue -- probably getting

[jira] [Commented] (HDFS-8132) Namenode Startup Failing When we add Jcarder.jar in class Path

2015-04-14 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14495698#comment-14495698 ] Todd Lipcon commented on HDFS-8132: --- Nope. Feel free to send me a pull request for

[jira] [Commented] (HDFS-5522) Datanode disk error check may be incorrectly skipped

2015-03-16 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-5522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14364604#comment-14364604 ] Todd Lipcon commented on HDFS-5522: --- I know this has been closed for a while, but wanted

[jira] [Commented] (HDFS-6658) Namenode memory optimization - Block replicas list

2015-03-03 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14345572#comment-14345572 ] Todd Lipcon commented on HDFS-6658: --- {quote} 1 the balancer 2 datanode manager start/stop

[jira] [Comment Edited] (HDFS-6658) Namenode memory optimization - Block replicas list

2015-03-03 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14345572#comment-14345572 ] Todd Lipcon edited comment on HDFS-6658 at 3/3/15 7:26 PM: ---

[jira] [Commented] (HDFS-7845) Compress block reports

2015-03-02 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14343879#comment-14343879 ] Todd Lipcon commented on HDFS-7845: --- Agreed we should run various simulations before

[jira] [Commented] (HDFS-6658) Namenode memory optimization - Block replicas list

2015-03-02 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14343530#comment-14343530 ] Todd Lipcon commented on HDFS-6658: --- Hey Daryn... jumping in unsolicited with a random

[jira] [Commented] (HDFS-7845) Compress block reports

2015-02-28 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341828#comment-14341828 ] Todd Lipcon commented on HDFS-7845: --- fwiw I ran a quick test on a random DN I found in

[jira] [Commented] (HDFS-7845) Compress block reports

2015-02-28 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341822#comment-14341822 ] Todd Lipcon commented on HDFS-7845: --- One quick hint here that may help: for a series of

[jira] [Commented] (HDFS-1148) Convert FSDataset to ReadWriteLock

2015-02-13 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14321180#comment-14321180 ] Todd Lipcon commented on HDFS-1148: --- I don't think this is nearly as useful anymore now

[jira] [Commented] (HDFS-957) FSImage layout version should be only once file is complete

2015-02-11 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316883#comment-14316883 ] Todd Lipcon commented on HDFS-957: -- I don't think this is necessary anymore since HDFS-1073

[jira] [Resolved] (HDFS-957) FSImage layout version should be only once file is complete

2015-02-11 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon resolved HDFS-957. -- Resolution: Won't Fix FSImage layout version should be only once file is complete

[jira] [Commented] (HDFS-2825) Add test hook to turn off the writer preferring its local DN

2014-12-11 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14243637#comment-14243637 ] Todd Lipcon commented on HDFS-2825: --- [~nroberts] - have you seen HDFS-3702? It's related

[jira] [Commented] (HDFS-3107) HDFS truncate

2014-10-17 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-3107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14175835#comment-14175835 ] Todd Lipcon commented on HDFS-3107: --- bq. Dhruba, Indeed. Lack of concurrent writes to a

[jira] [Commented] (HDFS-7122) Use of ThreadLocalRandom results in poor block placement

2014-09-30 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14153684#comment-14153684 ] Todd Lipcon commented on HDFS-7122: --- bq. I might be wrong – It did seem that the author

[jira] [Commented] (HDFS-6940) Initial refactoring to allow ConsensusNode implementation

2014-09-09 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14127725#comment-14127725 ] Todd Lipcon commented on HDFS-6940: --- bq. This issue is being rejected based on an

[jira] [Commented] (HDFS-6940) Initial refactoring to allow ConsensusNode implementation

2014-09-07 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125224#comment-14125224 ] Todd Lipcon commented on HDFS-6940: --- I'll only comment on the technical issue at hand

[jira] [Commented] (HDFS-6469) Coordinated replication of the namespace using ConsensusNode

2014-09-03 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14120753#comment-14120753 ] Todd Lipcon commented on HDFS-6469: --- bq. Total of 12 nodes, but with only one being

[jira] [Commented] (HDFS-3529) Use direct buffers for data in write path

2014-08-28 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113917#comment-14113917 ] Todd Lipcon commented on HDFS-3529: --- Hey Yi. Please note that [~james.thomas] has been

[jira] [Commented] (HDFS-6865) Byte array native checksumming on client side (HDFS changes)

2014-08-28 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114602#comment-14114602 ] Todd Lipcon commented on HDFS-6865: --- Sweet. You the man, Stack! I'll commit this

[jira] [Updated] (HDFS-6865) Byte array native checksumming on client side (HDFS changes)

2014-08-28 Thread Todd Lipcon (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-6865: -- Resolution: Fixed Fix Version/s: 2.6.0 Hadoop Flags: Reviewed Status: Resolved

  1   2   3   4   5   6   7   8   9   10   >