[ https://issues.apache.org/jira/browse/HADOOP-12057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15263293#comment-15263293 ]
Andres Perez commented on HADOOP-12057: --------------------------------------- I used a Bluemix Object Store, where you don't have access to the swift.conf file to change the default maximum object size limit. I think this is good because it separate the dependency between needing to change something in swift for Hadoop to work. This just uses the [Large Object Direct API|http://docs.openstack.org/developer/swift/overview_large_objects.html#additional-notes] of Swift, which in theory will allow to store 1TB files in several segments without merging them, but still providing a single file to download and combine all the pieces together. The only issues if that is you hdfs dfs -ls swift://container.store/ you will see the file displaying with size 0, which is also expected from the documentation linked above. > swiftfs rename on partitioned file attempts to consolidate partitions > --------------------------------------------------------------------- > > Key: HADOOP-12057 > URL: https://issues.apache.org/jira/browse/HADOOP-12057 > Project: Hadoop Common > Issue Type: Bug > Components: fs/swift > Reporter: David Dobbins > Assignee: David Dobbins > Attachments: HADOOP-12057-006.patch, HADOOP-12057-008.patch, > HADOOP-12057.007.patch, HADOOP-12057.patch, HADOOP-12057.patch, > HADOOP-12057.patch, HADOOP-12057.patch, HADOOP-12057.patch > > > In the swift filesystem for openstack, a rename operation on a partitioned > file uses the swift COPY operation, which attempts to consolidate all of the > partitions into a single object. This causes the rename to fail when the > total size of all the partitions exceeds the maximum object size for swift. > Since partitioned files are primarily created to allow a file to exceed the > maximum object size, this bug makes writing to swift extremely unreliable. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org