[jira] [Commented] (HADOOP-15607) AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream
[ https://issues.apache.org/jira/browse/HADOOP-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16566638#comment-16566638 ] wujinhu commented on HADOOP-15607: -- Thanks [~Sammi] > AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream > - > > Key: HADOOP-15607 > URL: https://issues.apache.org/jira/browse/HADOOP-15607 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.1.0, 2.10.0, 2.9.1, 3.2.0, 3.0.3 >Reporter: wujinhu >Assignee: wujinhu >Priority: Critical > Fix For: 2.10.0, 3.2.0, 2.9.2, 3.0.4, 3.1.2 > > Attachments: HADOOP-15607-branch-2.001.patch, HADOOP-15607.001.patch, > HADOOP-15607.002.patch, HADOOP-15607.003.patch, HADOOP-15607.004.patch > > > When I generated data with hive-tpcds tool, I got exception below: > 2018-07-16 14:50:43,680 INFO mapreduce.Job: Task Id : > attempt_1531723399698_0001_m_52_0, Status : FAILED > Error: com.aliyun.oss.OSSException: The list of parts was not in ascending > order. Parts list must specified in order by part number. > [ErrorCode]: InvalidPartOrder > [RequestId]: 5B4C40425FCC208D79D1EAF5 > [HostId]: 100.103.0.137 > [ResponseError]: > > > InvalidPartOrder > The list of parts was not in ascending order. Parts list must > specified in order by part number. > 5B4C40425FCC208D79D1EAF5 > xx.xx.xx.xx > current PartNumber 3, you given part number 3is not in > ascending order > > at > com.aliyun.oss.common.utils.ExceptionFactory.createOSSException(ExceptionFactory.java:99) > at > com.aliyun.oss.internal.OSSErrorResponseHandler.handle(OSSErrorResponseHandler.java:69) > at > com.aliyun.oss.common.comm.ServiceClient.handleResponse(ServiceClient.java:248) > at > com.aliyun.oss.common.comm.ServiceClient.sendRequestImpl(ServiceClient.java:130) > at > com.aliyun.oss.common.comm.ServiceClient.sendRequest(ServiceClient.java:68) > at com.aliyun.oss.internal.OSSOperation.send(OSSOperation.java:94) > at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:149) > at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:113) > at > com.aliyun.oss.internal.OSSMultipartOperation.completeMultipartUpload(OSSMultipartOperation.java:185) > at com.aliyun.oss.OSSClient.completeMultipartUpload(OSSClient.java:790) > at > org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystemStore.completeMultipartUpload(AliyunOSSFileSystemStore.java:643) > at > org.apache.hadoop.fs.aliyun.oss.AliyunOSSBlockOutputStream.close(AliyunOSSBlockOutputStream.java:120) > at > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72) > at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101) > at > org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWriter.close(TextOutputFormat.java:106) > at > org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.close(MultipleOutputs.java:574) > at org.notmysock.tpcds.GenTable$DSDGen.cleanup(GenTable.java:169) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:149) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1686) > > I reviewed code below, > {code:java} > blockId {code} > has thread synchronization problem > {code:java} > // code placeholder > private void uploadCurrentPart() throws IOException { > blockFiles.add(blockFile); > blockStream.flush(); > blockStream.close(); > if (blockId == 0) { > uploadId = store.getUploadId(key); > } > ListenableFuture partETagFuture = > executorService.submit(() -> { > PartETag partETag = store.uploadPart(blockFile, key, uploadId, > blockId + 1); > return partETag; > }); > partETagsFutures.add(partETagFuture); > blockFile = newBlockFile(); > blockId++; > blockStream = new BufferedOutputStream(new FileOutputStream(blockFile)); > } > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15607) AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream
[ https://issues.apache.org/jira/browse/HADOOP-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16564760#comment-16564760 ] Sammi Chen commented on HADOOP-15607: - I fired YETUS-652 to track the build issue. Will commit the patch to branch-2 and branch-2.9 soon. > AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream > - > > Key: HADOOP-15607 > URL: https://issues.apache.org/jira/browse/HADOOP-15607 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.1.0, 2.10.0, 2.9.1, 3.2.0, 3.0.3 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Fix For: 3.2.0, 3.0.4, 3.1.2 > > Attachments: HADOOP-15607-branch-2.001.patch, HADOOP-15607.001.patch, > HADOOP-15607.002.patch, HADOOP-15607.003.patch, HADOOP-15607.004.patch > > > When I generated data with hive-tpcds tool, I got exception below: > 2018-07-16 14:50:43,680 INFO mapreduce.Job: Task Id : > attempt_1531723399698_0001_m_52_0, Status : FAILED > Error: com.aliyun.oss.OSSException: The list of parts was not in ascending > order. Parts list must specified in order by part number. > [ErrorCode]: InvalidPartOrder > [RequestId]: 5B4C40425FCC208D79D1EAF5 > [HostId]: 100.103.0.137 > [ResponseError]: > > > InvalidPartOrder > The list of parts was not in ascending order. Parts list must > specified in order by part number. > 5B4C40425FCC208D79D1EAF5 > xx.xx.xx.xx > current PartNumber 3, you given part number 3is not in > ascending order > > at > com.aliyun.oss.common.utils.ExceptionFactory.createOSSException(ExceptionFactory.java:99) > at > com.aliyun.oss.internal.OSSErrorResponseHandler.handle(OSSErrorResponseHandler.java:69) > at > com.aliyun.oss.common.comm.ServiceClient.handleResponse(ServiceClient.java:248) > at > com.aliyun.oss.common.comm.ServiceClient.sendRequestImpl(ServiceClient.java:130) > at > com.aliyun.oss.common.comm.ServiceClient.sendRequest(ServiceClient.java:68) > at com.aliyun.oss.internal.OSSOperation.send(OSSOperation.java:94) > at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:149) > at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:113) > at > com.aliyun.oss.internal.OSSMultipartOperation.completeMultipartUpload(OSSMultipartOperation.java:185) > at com.aliyun.oss.OSSClient.completeMultipartUpload(OSSClient.java:790) > at > org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystemStore.completeMultipartUpload(AliyunOSSFileSystemStore.java:643) > at > org.apache.hadoop.fs.aliyun.oss.AliyunOSSBlockOutputStream.close(AliyunOSSBlockOutputStream.java:120) > at > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72) > at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101) > at > org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWriter.close(TextOutputFormat.java:106) > at > org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.close(MultipleOutputs.java:574) > at org.notmysock.tpcds.GenTable$DSDGen.cleanup(GenTable.java:169) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:149) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1686) > > I reviewed code below, > {code:java} > blockId {code} > has thread synchronization problem > {code:java} > // code placeholder > private void uploadCurrentPart() throws IOException { > blockFiles.add(blockFile); > blockStream.flush(); > blockStream.close(); > if (blockId == 0) { > uploadId = store.getUploadId(key); > } > ListenableFuture partETagFuture = > executorService.submit(() -> { > PartETag partETag = store.uploadPart(blockFile, key, uploadId, > blockId + 1); > return partETag; > }); > partETagsFutures.add(partETagFuture); > blockFile = newBlockFile(); > blockId++; > blockStream = new BufferedOutputStream(new FileOutputStream(blockFile)); > } > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15607) AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream
[ https://issues.apache.org/jira/browse/HADOOP-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16563531#comment-16563531 ] wujinhu commented on HADOOP-15607: -- [~Sammi] I have built in my local environment successfully. Can we get some logs from remote servers? > AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream > - > > Key: HADOOP-15607 > URL: https://issues.apache.org/jira/browse/HADOOP-15607 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.1.0, 2.10.0, 2.9.1, 3.2.0, 3.0.3 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Fix For: 3.2.0, 3.0.4, 3.1.2 > > Attachments: HADOOP-15607-branch-2.001.patch, HADOOP-15607.001.patch, > HADOOP-15607.002.patch, HADOOP-15607.003.patch, HADOOP-15607.004.patch > > > When I generated data with hive-tpcds tool, I got exception below: > 2018-07-16 14:50:43,680 INFO mapreduce.Job: Task Id : > attempt_1531723399698_0001_m_52_0, Status : FAILED > Error: com.aliyun.oss.OSSException: The list of parts was not in ascending > order. Parts list must specified in order by part number. > [ErrorCode]: InvalidPartOrder > [RequestId]: 5B4C40425FCC208D79D1EAF5 > [HostId]: 100.103.0.137 > [ResponseError]: > > > InvalidPartOrder > The list of parts was not in ascending order. Parts list must > specified in order by part number. > 5B4C40425FCC208D79D1EAF5 > xx.xx.xx.xx > current PartNumber 3, you given part number 3is not in > ascending order > > at > com.aliyun.oss.common.utils.ExceptionFactory.createOSSException(ExceptionFactory.java:99) > at > com.aliyun.oss.internal.OSSErrorResponseHandler.handle(OSSErrorResponseHandler.java:69) > at > com.aliyun.oss.common.comm.ServiceClient.handleResponse(ServiceClient.java:248) > at > com.aliyun.oss.common.comm.ServiceClient.sendRequestImpl(ServiceClient.java:130) > at > com.aliyun.oss.common.comm.ServiceClient.sendRequest(ServiceClient.java:68) > at com.aliyun.oss.internal.OSSOperation.send(OSSOperation.java:94) > at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:149) > at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:113) > at > com.aliyun.oss.internal.OSSMultipartOperation.completeMultipartUpload(OSSMultipartOperation.java:185) > at com.aliyun.oss.OSSClient.completeMultipartUpload(OSSClient.java:790) > at > org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystemStore.completeMultipartUpload(AliyunOSSFileSystemStore.java:643) > at > org.apache.hadoop.fs.aliyun.oss.AliyunOSSBlockOutputStream.close(AliyunOSSBlockOutputStream.java:120) > at > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72) > at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101) > at > org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWriter.close(TextOutputFormat.java:106) > at > org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.close(MultipleOutputs.java:574) > at org.notmysock.tpcds.GenTable$DSDGen.cleanup(GenTable.java:169) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:149) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1686) > > I reviewed code below, > {code:java} > blockId {code} > has thread synchronization problem > {code:java} > // code placeholder > private void uploadCurrentPart() throws IOException { > blockFiles.add(blockFile); > blockStream.flush(); > blockStream.close(); > if (blockId == 0) { > uploadId = store.getUploadId(key); > } > ListenableFuture partETagFuture = > executorService.submit(() -> { > PartETag partETag = store.uploadPart(blockFile, key, uploadId, > blockId + 1); > return partETag; > }); > partETagsFutures.add(partETagFuture); > blockFile = newBlockFile(); > blockId++; > blockStream = new BufferedOutputStream(new FileOutputStream(blockFile)); > } > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15607) AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream
[ https://issues.apache.org/jira/browse/HADOOP-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16563125#comment-16563125 ] SammiChen commented on HADOOP-15607: [~wujinhu], have you tried building the branch-2 locally with the patch applied? It seems the build system has some issues on branch-2. > AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream > - > > Key: HADOOP-15607 > URL: https://issues.apache.org/jira/browse/HADOOP-15607 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.1.0, 2.10.0, 2.9.1, 3.2.0, 3.0.3 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Fix For: 3.2.0, 3.0.4, 3.1.2 > > Attachments: HADOOP-15607-branch-2.001.patch, HADOOP-15607.001.patch, > HADOOP-15607.002.patch, HADOOP-15607.003.patch, HADOOP-15607.004.patch > > > When I generated data with hive-tpcds tool, I got exception below: > 2018-07-16 14:50:43,680 INFO mapreduce.Job: Task Id : > attempt_1531723399698_0001_m_52_0, Status : FAILED > Error: com.aliyun.oss.OSSException: The list of parts was not in ascending > order. Parts list must specified in order by part number. > [ErrorCode]: InvalidPartOrder > [RequestId]: 5B4C40425FCC208D79D1EAF5 > [HostId]: 100.103.0.137 > [ResponseError]: > > > InvalidPartOrder > The list of parts was not in ascending order. Parts list must > specified in order by part number. > 5B4C40425FCC208D79D1EAF5 > xx.xx.xx.xx > current PartNumber 3, you given part number 3is not in > ascending order > > at > com.aliyun.oss.common.utils.ExceptionFactory.createOSSException(ExceptionFactory.java:99) > at > com.aliyun.oss.internal.OSSErrorResponseHandler.handle(OSSErrorResponseHandler.java:69) > at > com.aliyun.oss.common.comm.ServiceClient.handleResponse(ServiceClient.java:248) > at > com.aliyun.oss.common.comm.ServiceClient.sendRequestImpl(ServiceClient.java:130) > at > com.aliyun.oss.common.comm.ServiceClient.sendRequest(ServiceClient.java:68) > at com.aliyun.oss.internal.OSSOperation.send(OSSOperation.java:94) > at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:149) > at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:113) > at > com.aliyun.oss.internal.OSSMultipartOperation.completeMultipartUpload(OSSMultipartOperation.java:185) > at com.aliyun.oss.OSSClient.completeMultipartUpload(OSSClient.java:790) > at > org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystemStore.completeMultipartUpload(AliyunOSSFileSystemStore.java:643) > at > org.apache.hadoop.fs.aliyun.oss.AliyunOSSBlockOutputStream.close(AliyunOSSBlockOutputStream.java:120) > at > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72) > at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101) > at > org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWriter.close(TextOutputFormat.java:106) > at > org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.close(MultipleOutputs.java:574) > at org.notmysock.tpcds.GenTable$DSDGen.cleanup(GenTable.java:169) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:149) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1686) > > I reviewed code below, > {code:java} > blockId {code} > has thread synchronization problem > {code:java} > // code placeholder > private void uploadCurrentPart() throws IOException { > blockFiles.add(blockFile); > blockStream.flush(); > blockStream.close(); > if (blockId == 0) { > uploadId = store.getUploadId(key); > } > ListenableFuture partETagFuture = > executorService.submit(() -> { > PartETag partETag = store.uploadPart(blockFile, key, uploadId, > blockId + 1); > return partETag; > }); > partETagsFutures.add(partETagFuture); > blockFile = newBlockFile(); > blockId++; > blockStream = new BufferedOutputStream(new FileOutputStream(blockFile)); > } > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15607) AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream
[ https://issues.apache.org/jira/browse/HADOOP-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16562989#comment-16562989 ] genericqa commented on HADOOP-15607: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} docker {color} | {color:red} 0m 5s{color} | {color:red} Docker failed to build yetus/hadoop:f667ef1. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | HADOOP-15607 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12933564/HADOOP-15607-branch-2.001.patch | | Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/14966/console | | Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream > - > > Key: HADOOP-15607 > URL: https://issues.apache.org/jira/browse/HADOOP-15607 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.1.0, 2.10.0, 2.9.1, 3.2.0, 3.0.3 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Fix For: 3.2.0, 3.0.4, 3.1.2 > > Attachments: HADOOP-15607-branch-2.001.patch, HADOOP-15607.001.patch, > HADOOP-15607.002.patch, HADOOP-15607.003.patch, HADOOP-15607.004.patch > > > When I generated data with hive-tpcds tool, I got exception below: > 2018-07-16 14:50:43,680 INFO mapreduce.Job: Task Id : > attempt_1531723399698_0001_m_52_0, Status : FAILED > Error: com.aliyun.oss.OSSException: The list of parts was not in ascending > order. Parts list must specified in order by part number. > [ErrorCode]: InvalidPartOrder > [RequestId]: 5B4C40425FCC208D79D1EAF5 > [HostId]: 100.103.0.137 > [ResponseError]: > > > InvalidPartOrder > The list of parts was not in ascending order. Parts list must > specified in order by part number. > 5B4C40425FCC208D79D1EAF5 > xx.xx.xx.xx > current PartNumber 3, you given part number 3is not in > ascending order > > at > com.aliyun.oss.common.utils.ExceptionFactory.createOSSException(ExceptionFactory.java:99) > at > com.aliyun.oss.internal.OSSErrorResponseHandler.handle(OSSErrorResponseHandler.java:69) > at > com.aliyun.oss.common.comm.ServiceClient.handleResponse(ServiceClient.java:248) > at > com.aliyun.oss.common.comm.ServiceClient.sendRequestImpl(ServiceClient.java:130) > at > com.aliyun.oss.common.comm.ServiceClient.sendRequest(ServiceClient.java:68) > at com.aliyun.oss.internal.OSSOperation.send(OSSOperation.java:94) > at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:149) > at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:113) > at > com.aliyun.oss.internal.OSSMultipartOperation.completeMultipartUpload(OSSMultipartOperation.java:185) > at com.aliyun.oss.OSSClient.completeMultipartUpload(OSSClient.java:790) > at > org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystemStore.completeMultipartUpload(AliyunOSSFileSystemStore.java:643) > at > org.apache.hadoop.fs.aliyun.oss.AliyunOSSBlockOutputStream.close(AliyunOSSBlockOutputStream.java:120) > at > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72) > at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101) > at > org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWriter.close(TextOutputFormat.java:106) > at > org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.close(MultipleOutputs.java:574) > at org.notmysock.tpcds.GenTable$DSDGen.cleanup(GenTable.java:169) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:149) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1686) > > I reviewed code below, > {code:java} > blockId {code} > has thread synchronization problem > {code:java} > // code placeholder > private void uploadCurrentPart() throws IOException { > blockFiles.add(blockFile); > blockStream.flush(); > blockStream.close(); > if (blockId == 0) { > uploadId = store.getUploadId(key); > } > ListenableFuture partETagFuture = > executorService.submit(() -> { > PartETag partETag = store.uploadPart(blockFile, key, uploadId, > blockId + 1); > return partETag; > }); > pa
[jira] [Commented] (HADOOP-15607) AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream
[ https://issues.apache.org/jira/browse/HADOOP-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16561689#comment-16561689 ] genericqa commented on HADOOP-15607: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} docker {color} | {color:red} 11m 48s{color} | {color:red} Docker failed to build yetus/hadoop:f667ef1. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | HADOOP-15607 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12933564/HADOOP-15607-branch-2.001.patch | | Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/14961/console | | Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream > - > > Key: HADOOP-15607 > URL: https://issues.apache.org/jira/browse/HADOOP-15607 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.1.0, 2.10.0, 2.9.1, 3.2.0, 3.0.3 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Fix For: 3.2.0, 3.0.4, 3.1.2 > > Attachments: HADOOP-15607-branch-2.001.patch, HADOOP-15607.001.patch, > HADOOP-15607.002.patch, HADOOP-15607.003.patch, HADOOP-15607.004.patch > > > When I generated data with hive-tpcds tool, I got exception below: > 2018-07-16 14:50:43,680 INFO mapreduce.Job: Task Id : > attempt_1531723399698_0001_m_52_0, Status : FAILED > Error: com.aliyun.oss.OSSException: The list of parts was not in ascending > order. Parts list must specified in order by part number. > [ErrorCode]: InvalidPartOrder > [RequestId]: 5B4C40425FCC208D79D1EAF5 > [HostId]: 100.103.0.137 > [ResponseError]: > > > InvalidPartOrder > The list of parts was not in ascending order. Parts list must > specified in order by part number. > 5B4C40425FCC208D79D1EAF5 > xx.xx.xx.xx > current PartNumber 3, you given part number 3is not in > ascending order > > at > com.aliyun.oss.common.utils.ExceptionFactory.createOSSException(ExceptionFactory.java:99) > at > com.aliyun.oss.internal.OSSErrorResponseHandler.handle(OSSErrorResponseHandler.java:69) > at > com.aliyun.oss.common.comm.ServiceClient.handleResponse(ServiceClient.java:248) > at > com.aliyun.oss.common.comm.ServiceClient.sendRequestImpl(ServiceClient.java:130) > at > com.aliyun.oss.common.comm.ServiceClient.sendRequest(ServiceClient.java:68) > at com.aliyun.oss.internal.OSSOperation.send(OSSOperation.java:94) > at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:149) > at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:113) > at > com.aliyun.oss.internal.OSSMultipartOperation.completeMultipartUpload(OSSMultipartOperation.java:185) > at com.aliyun.oss.OSSClient.completeMultipartUpload(OSSClient.java:790) > at > org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystemStore.completeMultipartUpload(AliyunOSSFileSystemStore.java:643) > at > org.apache.hadoop.fs.aliyun.oss.AliyunOSSBlockOutputStream.close(AliyunOSSBlockOutputStream.java:120) > at > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72) > at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101) > at > org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWriter.close(TextOutputFormat.java:106) > at > org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.close(MultipleOutputs.java:574) > at org.notmysock.tpcds.GenTable$DSDGen.cleanup(GenTable.java:169) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:149) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1686) > > I reviewed code below, > {code:java} > blockId {code} > has thread synchronization problem > {code:java} > // code placeholder > private void uploadCurrentPart() throws IOException { > blockFiles.add(blockFile); > blockStream.flush(); > blockStream.close(); > if (blockId == 0) { > uploadId = store.getUploadId(key); > } > ListenableFuture partETagFuture = > executorService.submit(() -> { > PartETag partETag = store.uploadPart(blockFile, key, uploadId, > blockId + 1); > return partETag; > }); > pa
[jira] [Commented] (HADOOP-15607) AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream
[ https://issues.apache.org/jira/browse/HADOOP-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16561567#comment-16561567 ] wujinhu commented on HADOOP-15607: -- Thanks [~Sammi] I have attached a patch for branch-2.:) > AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream > - > > Key: HADOOP-15607 > URL: https://issues.apache.org/jira/browse/HADOOP-15607 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.1.0, 2.10.0, 2.9.1, 3.2.0, 3.0.3 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Fix For: 3.2.0, 3.0.4, 3.1.2 > > Attachments: HADOOP-15607-branch-2.001.patch, HADOOP-15607.001.patch, > HADOOP-15607.002.patch, HADOOP-15607.003.patch, HADOOP-15607.004.patch > > > When I generated data with hive-tpcds tool, I got exception below: > 2018-07-16 14:50:43,680 INFO mapreduce.Job: Task Id : > attempt_1531723399698_0001_m_52_0, Status : FAILED > Error: com.aliyun.oss.OSSException: The list of parts was not in ascending > order. Parts list must specified in order by part number. > [ErrorCode]: InvalidPartOrder > [RequestId]: 5B4C40425FCC208D79D1EAF5 > [HostId]: 100.103.0.137 > [ResponseError]: > > > InvalidPartOrder > The list of parts was not in ascending order. Parts list must > specified in order by part number. > 5B4C40425FCC208D79D1EAF5 > xx.xx.xx.xx > current PartNumber 3, you given part number 3is not in > ascending order > > at > com.aliyun.oss.common.utils.ExceptionFactory.createOSSException(ExceptionFactory.java:99) > at > com.aliyun.oss.internal.OSSErrorResponseHandler.handle(OSSErrorResponseHandler.java:69) > at > com.aliyun.oss.common.comm.ServiceClient.handleResponse(ServiceClient.java:248) > at > com.aliyun.oss.common.comm.ServiceClient.sendRequestImpl(ServiceClient.java:130) > at > com.aliyun.oss.common.comm.ServiceClient.sendRequest(ServiceClient.java:68) > at com.aliyun.oss.internal.OSSOperation.send(OSSOperation.java:94) > at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:149) > at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:113) > at > com.aliyun.oss.internal.OSSMultipartOperation.completeMultipartUpload(OSSMultipartOperation.java:185) > at com.aliyun.oss.OSSClient.completeMultipartUpload(OSSClient.java:790) > at > org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystemStore.completeMultipartUpload(AliyunOSSFileSystemStore.java:643) > at > org.apache.hadoop.fs.aliyun.oss.AliyunOSSBlockOutputStream.close(AliyunOSSBlockOutputStream.java:120) > at > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72) > at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101) > at > org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWriter.close(TextOutputFormat.java:106) > at > org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.close(MultipleOutputs.java:574) > at org.notmysock.tpcds.GenTable$DSDGen.cleanup(GenTable.java:169) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:149) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1686) > > I reviewed code below, > {code:java} > blockId {code} > has thread synchronization problem > {code:java} > // code placeholder > private void uploadCurrentPart() throws IOException { > blockFiles.add(blockFile); > blockStream.flush(); > blockStream.close(); > if (blockId == 0) { > uploadId = store.getUploadId(key); > } > ListenableFuture partETagFuture = > executorService.submit(() -> { > PartETag partETag = store.uploadPart(blockFile, key, uploadId, > blockId + 1); > return partETag; > }); > partETagsFutures.add(partETagFuture); > blockFile = newBlockFile(); > blockId++; > blockStream = new BufferedOutputStream(new FileOutputStream(blockFile)); > } > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15607) AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream
[ https://issues.apache.org/jira/browse/HADOOP-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16561554#comment-16561554 ] genericqa commented on HADOOP-15607: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} docker {color} | {color:red} 16m 35s{color} | {color:red} Docker failed to build yetus/hadoop:f667ef1. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | HADOOP-15607 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12933551/HADOOP-15607-branch-2.001.patch | | Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/14960/console | | Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream > - > > Key: HADOOP-15607 > URL: https://issues.apache.org/jira/browse/HADOOP-15607 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.1.0, 2.10.0, 2.9.1, 3.2.0, 3.0.3 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Fix For: 3.2.0, 3.0.4, 3.1.2 > > Attachments: HADOOP-15607-branch-2.001.patch, HADOOP-15607.001.patch, > HADOOP-15607.002.patch, HADOOP-15607.003.patch, HADOOP-15607.004.patch > > > When I generated data with hive-tpcds tool, I got exception below: > 2018-07-16 14:50:43,680 INFO mapreduce.Job: Task Id : > attempt_1531723399698_0001_m_52_0, Status : FAILED > Error: com.aliyun.oss.OSSException: The list of parts was not in ascending > order. Parts list must specified in order by part number. > [ErrorCode]: InvalidPartOrder > [RequestId]: 5B4C40425FCC208D79D1EAF5 > [HostId]: 100.103.0.137 > [ResponseError]: > > > InvalidPartOrder > The list of parts was not in ascending order. Parts list must > specified in order by part number. > 5B4C40425FCC208D79D1EAF5 > xx.xx.xx.xx > current PartNumber 3, you given part number 3is not in > ascending order > > at > com.aliyun.oss.common.utils.ExceptionFactory.createOSSException(ExceptionFactory.java:99) > at > com.aliyun.oss.internal.OSSErrorResponseHandler.handle(OSSErrorResponseHandler.java:69) > at > com.aliyun.oss.common.comm.ServiceClient.handleResponse(ServiceClient.java:248) > at > com.aliyun.oss.common.comm.ServiceClient.sendRequestImpl(ServiceClient.java:130) > at > com.aliyun.oss.common.comm.ServiceClient.sendRequest(ServiceClient.java:68) > at com.aliyun.oss.internal.OSSOperation.send(OSSOperation.java:94) > at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:149) > at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:113) > at > com.aliyun.oss.internal.OSSMultipartOperation.completeMultipartUpload(OSSMultipartOperation.java:185) > at com.aliyun.oss.OSSClient.completeMultipartUpload(OSSClient.java:790) > at > org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystemStore.completeMultipartUpload(AliyunOSSFileSystemStore.java:643) > at > org.apache.hadoop.fs.aliyun.oss.AliyunOSSBlockOutputStream.close(AliyunOSSBlockOutputStream.java:120) > at > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72) > at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101) > at > org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWriter.close(TextOutputFormat.java:106) > at > org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.close(MultipleOutputs.java:574) > at org.notmysock.tpcds.GenTable$DSDGen.cleanup(GenTable.java:169) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:149) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1686) > > I reviewed code below, > {code:java} > blockId {code} > has thread synchronization problem > {code:java} > // code placeholder > private void uploadCurrentPart() throws IOException { > blockFiles.add(blockFile); > blockStream.flush(); > blockStream.close(); > if (blockId == 0) { > uploadId = store.getUploadId(key); > } > ListenableFuture partETagFuture = > executorService.submit(() -> { > PartETag partETag = store.uploadPart(blockFile, key, uploadId, > blockId + 1); > return partETag; > }); > pa
[jira] [Commented] (HADOOP-15607) AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream
[ https://issues.apache.org/jira/browse/HADOOP-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16561384#comment-16561384 ] Hudson commented on HADOOP-15607: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14665 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14665/]) HADOOP-15607. AliyunOSS: fix duplicated partNumber issue in (sammi.chen: rev 0857f116b754d83d3c540cd6f989087af24fef27) * (edit) hadoop-tools/hadoop-aliyun/src/main/java/org/apache/hadoop/fs/aliyun/oss/AliyunOSSFileSystemStore.java * (edit) hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunOSSBlockOutputStream.java * (edit) hadoop-tools/hadoop-aliyun/src/main/java/org/apache/hadoop/fs/aliyun/oss/AliyunOSSBlockOutputStream.java > AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream > - > > Key: HADOOP-15607 > URL: https://issues.apache.org/jira/browse/HADOOP-15607 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.1.0, 2.10.0, 2.9.1, 3.2.0, 3.0.3 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Fix For: 3.2.0, 3.0.4, 3.1.2 > > Attachments: HADOOP-15607.001.patch, HADOOP-15607.002.patch, > HADOOP-15607.003.patch, HADOOP-15607.004.patch > > > When I generated data with hive-tpcds tool, I got exception below: > 2018-07-16 14:50:43,680 INFO mapreduce.Job: Task Id : > attempt_1531723399698_0001_m_52_0, Status : FAILED > Error: com.aliyun.oss.OSSException: The list of parts was not in ascending > order. Parts list must specified in order by part number. > [ErrorCode]: InvalidPartOrder > [RequestId]: 5B4C40425FCC208D79D1EAF5 > [HostId]: 100.103.0.137 > [ResponseError]: > > > InvalidPartOrder > The list of parts was not in ascending order. Parts list must > specified in order by part number. > 5B4C40425FCC208D79D1EAF5 > xx.xx.xx.xx > current PartNumber 3, you given part number 3is not in > ascending order > > at > com.aliyun.oss.common.utils.ExceptionFactory.createOSSException(ExceptionFactory.java:99) > at > com.aliyun.oss.internal.OSSErrorResponseHandler.handle(OSSErrorResponseHandler.java:69) > at > com.aliyun.oss.common.comm.ServiceClient.handleResponse(ServiceClient.java:248) > at > com.aliyun.oss.common.comm.ServiceClient.sendRequestImpl(ServiceClient.java:130) > at > com.aliyun.oss.common.comm.ServiceClient.sendRequest(ServiceClient.java:68) > at com.aliyun.oss.internal.OSSOperation.send(OSSOperation.java:94) > at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:149) > at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:113) > at > com.aliyun.oss.internal.OSSMultipartOperation.completeMultipartUpload(OSSMultipartOperation.java:185) > at com.aliyun.oss.OSSClient.completeMultipartUpload(OSSClient.java:790) > at > org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystemStore.completeMultipartUpload(AliyunOSSFileSystemStore.java:643) > at > org.apache.hadoop.fs.aliyun.oss.AliyunOSSBlockOutputStream.close(AliyunOSSBlockOutputStream.java:120) > at > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72) > at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101) > at > org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWriter.close(TextOutputFormat.java:106) > at > org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.close(MultipleOutputs.java:574) > at org.notmysock.tpcds.GenTable$DSDGen.cleanup(GenTable.java:169) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:149) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1686) > > I reviewed code below, > {code:java} > blockId {code} > has thread synchronization problem > {code:java} > // code placeholder > private void uploadCurrentPart() throws IOException { > blockFiles.add(blockFile); > blockStream.flush(); > blockStream.close(); > if (blockId == 0) { > uploadId = store.getUploadId(key); > } > ListenableFuture partETagFuture = > executorService.submit(() -> { > PartETag partETag = store.uploadPart(blockFile, key, uploadId, > blockId + 1); > return partETag; > }); > partETagsFutures.add(partETagFuture); > blockFile = newBlockFile(); > blockId++; > blockStream = new BufferedOutputStream(new FileOutputStream(blockFile)); > } > {code} -- This message was sent b
[jira] [Commented] (HADOOP-15607) AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream
[ https://issues.apache.org/jira/browse/HADOOP-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16561380#comment-16561380 ] SammiChen commented on HADOOP-15607: Hi [~wujinhu], would you provide a patch for branch-2. It reports conflict when I apply the current patch to branch-2. > AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream > - > > Key: HADOOP-15607 > URL: https://issues.apache.org/jira/browse/HADOOP-15607 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.1.0, 2.10.0, 2.9.1, 3.2.0, 3.0.3 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Attachments: HADOOP-15607.001.patch, HADOOP-15607.002.patch, > HADOOP-15607.003.patch, HADOOP-15607.004.patch > > > When I generated data with hive-tpcds tool, I got exception below: > 2018-07-16 14:50:43,680 INFO mapreduce.Job: Task Id : > attempt_1531723399698_0001_m_52_0, Status : FAILED > Error: com.aliyun.oss.OSSException: The list of parts was not in ascending > order. Parts list must specified in order by part number. > [ErrorCode]: InvalidPartOrder > [RequestId]: 5B4C40425FCC208D79D1EAF5 > [HostId]: 100.103.0.137 > [ResponseError]: > > > InvalidPartOrder > The list of parts was not in ascending order. Parts list must > specified in order by part number. > 5B4C40425FCC208D79D1EAF5 > xx.xx.xx.xx > current PartNumber 3, you given part number 3is not in > ascending order > > at > com.aliyun.oss.common.utils.ExceptionFactory.createOSSException(ExceptionFactory.java:99) > at > com.aliyun.oss.internal.OSSErrorResponseHandler.handle(OSSErrorResponseHandler.java:69) > at > com.aliyun.oss.common.comm.ServiceClient.handleResponse(ServiceClient.java:248) > at > com.aliyun.oss.common.comm.ServiceClient.sendRequestImpl(ServiceClient.java:130) > at > com.aliyun.oss.common.comm.ServiceClient.sendRequest(ServiceClient.java:68) > at com.aliyun.oss.internal.OSSOperation.send(OSSOperation.java:94) > at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:149) > at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:113) > at > com.aliyun.oss.internal.OSSMultipartOperation.completeMultipartUpload(OSSMultipartOperation.java:185) > at com.aliyun.oss.OSSClient.completeMultipartUpload(OSSClient.java:790) > at > org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystemStore.completeMultipartUpload(AliyunOSSFileSystemStore.java:643) > at > org.apache.hadoop.fs.aliyun.oss.AliyunOSSBlockOutputStream.close(AliyunOSSBlockOutputStream.java:120) > at > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72) > at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101) > at > org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWriter.close(TextOutputFormat.java:106) > at > org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.close(MultipleOutputs.java:574) > at org.notmysock.tpcds.GenTable$DSDGen.cleanup(GenTable.java:169) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:149) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1686) > > I reviewed code below, > {code:java} > blockId {code} > has thread synchronization problem > {code:java} > // code placeholder > private void uploadCurrentPart() throws IOException { > blockFiles.add(blockFile); > blockStream.flush(); > blockStream.close(); > if (blockId == 0) { > uploadId = store.getUploadId(key); > } > ListenableFuture partETagFuture = > executorService.submit(() -> { > PartETag partETag = store.uploadPart(blockFile, key, uploadId, > blockId + 1); > return partETag; > }); > partETagsFutures.add(partETagFuture); > blockFile = newBlockFile(); > blockId++; > blockStream = new BufferedOutputStream(new FileOutputStream(blockFile)); > } > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15607) AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream
[ https://issues.apache.org/jira/browse/HADOOP-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16559460#comment-16559460 ] wujinhu commented on HADOOP-15607: -- Build result has already came out. You can see result above. :) > AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream > - > > Key: HADOOP-15607 > URL: https://issues.apache.org/jira/browse/HADOOP-15607 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.1.0, 2.10.0, 2.9.1, 3.2.0, 3.0.3 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Attachments: HADOOP-15607.001.patch, HADOOP-15607.002.patch, > HADOOP-15607.003.patch, HADOOP-15607.004.patch > > > When I generated data with hive-tpcds tool, I got exception below: > 2018-07-16 14:50:43,680 INFO mapreduce.Job: Task Id : > attempt_1531723399698_0001_m_52_0, Status : FAILED > Error: com.aliyun.oss.OSSException: The list of parts was not in ascending > order. Parts list must specified in order by part number. > [ErrorCode]: InvalidPartOrder > [RequestId]: 5B4C40425FCC208D79D1EAF5 > [HostId]: 100.103.0.137 > [ResponseError]: > > > InvalidPartOrder > The list of parts was not in ascending order. Parts list must > specified in order by part number. > 5B4C40425FCC208D79D1EAF5 > xx.xx.xx.xx > current PartNumber 3, you given part number 3is not in > ascending order > > at > com.aliyun.oss.common.utils.ExceptionFactory.createOSSException(ExceptionFactory.java:99) > at > com.aliyun.oss.internal.OSSErrorResponseHandler.handle(OSSErrorResponseHandler.java:69) > at > com.aliyun.oss.common.comm.ServiceClient.handleResponse(ServiceClient.java:248) > at > com.aliyun.oss.common.comm.ServiceClient.sendRequestImpl(ServiceClient.java:130) > at > com.aliyun.oss.common.comm.ServiceClient.sendRequest(ServiceClient.java:68) > at com.aliyun.oss.internal.OSSOperation.send(OSSOperation.java:94) > at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:149) > at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:113) > at > com.aliyun.oss.internal.OSSMultipartOperation.completeMultipartUpload(OSSMultipartOperation.java:185) > at com.aliyun.oss.OSSClient.completeMultipartUpload(OSSClient.java:790) > at > org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystemStore.completeMultipartUpload(AliyunOSSFileSystemStore.java:643) > at > org.apache.hadoop.fs.aliyun.oss.AliyunOSSBlockOutputStream.close(AliyunOSSBlockOutputStream.java:120) > at > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72) > at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101) > at > org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWriter.close(TextOutputFormat.java:106) > at > org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.close(MultipleOutputs.java:574) > at org.notmysock.tpcds.GenTable$DSDGen.cleanup(GenTable.java:169) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:149) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1686) > > I reviewed code below, > {code:java} > blockId {code} > has thread synchronization problem > {code:java} > // code placeholder > private void uploadCurrentPart() throws IOException { > blockFiles.add(blockFile); > blockStream.flush(); > blockStream.close(); > if (blockId == 0) { > uploadId = store.getUploadId(key); > } > ListenableFuture partETagFuture = > executorService.submit(() -> { > PartETag partETag = store.uploadPart(blockFile, key, uploadId, > blockId + 1); > return partETag; > }); > partETagsFutures.add(partETagFuture); > blockFile = newBlockFile(); > blockId++; > blockStream = new BufferedOutputStream(new FileOutputStream(blockFile)); > } > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15607) AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream
[ https://issues.apache.org/jira/browse/HADOOP-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16559456#comment-16559456 ] SammiChen commented on HADOOP-15607: +1. Thanks [~wujinhu] for the contribution. I will commit after the build comes out. > AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream > - > > Key: HADOOP-15607 > URL: https://issues.apache.org/jira/browse/HADOOP-15607 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.1.0, 2.10.0, 2.9.1, 3.2.0, 3.0.3 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Attachments: HADOOP-15607.001.patch, HADOOP-15607.002.patch, > HADOOP-15607.003.patch, HADOOP-15607.004.patch > > > When I generated data with hive-tpcds tool, I got exception below: > 2018-07-16 14:50:43,680 INFO mapreduce.Job: Task Id : > attempt_1531723399698_0001_m_52_0, Status : FAILED > Error: com.aliyun.oss.OSSException: The list of parts was not in ascending > order. Parts list must specified in order by part number. > [ErrorCode]: InvalidPartOrder > [RequestId]: 5B4C40425FCC208D79D1EAF5 > [HostId]: 100.103.0.137 > [ResponseError]: > > > InvalidPartOrder > The list of parts was not in ascending order. Parts list must > specified in order by part number. > 5B4C40425FCC208D79D1EAF5 > xx.xx.xx.xx > current PartNumber 3, you given part number 3is not in > ascending order > > at > com.aliyun.oss.common.utils.ExceptionFactory.createOSSException(ExceptionFactory.java:99) > at > com.aliyun.oss.internal.OSSErrorResponseHandler.handle(OSSErrorResponseHandler.java:69) > at > com.aliyun.oss.common.comm.ServiceClient.handleResponse(ServiceClient.java:248) > at > com.aliyun.oss.common.comm.ServiceClient.sendRequestImpl(ServiceClient.java:130) > at > com.aliyun.oss.common.comm.ServiceClient.sendRequest(ServiceClient.java:68) > at com.aliyun.oss.internal.OSSOperation.send(OSSOperation.java:94) > at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:149) > at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:113) > at > com.aliyun.oss.internal.OSSMultipartOperation.completeMultipartUpload(OSSMultipartOperation.java:185) > at com.aliyun.oss.OSSClient.completeMultipartUpload(OSSClient.java:790) > at > org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystemStore.completeMultipartUpload(AliyunOSSFileSystemStore.java:643) > at > org.apache.hadoop.fs.aliyun.oss.AliyunOSSBlockOutputStream.close(AliyunOSSBlockOutputStream.java:120) > at > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72) > at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101) > at > org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWriter.close(TextOutputFormat.java:106) > at > org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.close(MultipleOutputs.java:574) > at org.notmysock.tpcds.GenTable$DSDGen.cleanup(GenTable.java:169) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:149) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1686) > > I reviewed code below, > {code:java} > blockId {code} > has thread synchronization problem > {code:java} > // code placeholder > private void uploadCurrentPart() throws IOException { > blockFiles.add(blockFile); > blockStream.flush(); > blockStream.close(); > if (blockId == 0) { > uploadId = store.getUploadId(key); > } > ListenableFuture partETagFuture = > executorService.submit(() -> { > PartETag partETag = store.uploadPart(blockFile, key, uploadId, > blockId + 1); > return partETag; > }); > partETagsFutures.add(partETagFuture); > blockFile = newBlockFile(); > blockId++; > blockStream = new BufferedOutputStream(new FileOutputStream(blockFile)); > } > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15607) AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream
[ https://issues.apache.org/jira/browse/HADOOP-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16559386#comment-16559386 ] wujinhu commented on HADOOP-15607: -- Thanks [~Sammi]. I have fixed those code style issues. > AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream > - > > Key: HADOOP-15607 > URL: https://issues.apache.org/jira/browse/HADOOP-15607 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.1.0, 2.10.0, 2.9.1, 3.2.0, 3.0.3 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Attachments: HADOOP-15607.001.patch, HADOOP-15607.002.patch, > HADOOP-15607.003.patch, HADOOP-15607.004.patch > > > When I generated data with hive-tpcds tool, I got exception below: > 2018-07-16 14:50:43,680 INFO mapreduce.Job: Task Id : > attempt_1531723399698_0001_m_52_0, Status : FAILED > Error: com.aliyun.oss.OSSException: The list of parts was not in ascending > order. Parts list must specified in order by part number. > [ErrorCode]: InvalidPartOrder > [RequestId]: 5B4C40425FCC208D79D1EAF5 > [HostId]: 100.103.0.137 > [ResponseError]: > > > InvalidPartOrder > The list of parts was not in ascending order. Parts list must > specified in order by part number. > 5B4C40425FCC208D79D1EAF5 > xx.xx.xx.xx > current PartNumber 3, you given part number 3is not in > ascending order > > at > com.aliyun.oss.common.utils.ExceptionFactory.createOSSException(ExceptionFactory.java:99) > at > com.aliyun.oss.internal.OSSErrorResponseHandler.handle(OSSErrorResponseHandler.java:69) > at > com.aliyun.oss.common.comm.ServiceClient.handleResponse(ServiceClient.java:248) > at > com.aliyun.oss.common.comm.ServiceClient.sendRequestImpl(ServiceClient.java:130) > at > com.aliyun.oss.common.comm.ServiceClient.sendRequest(ServiceClient.java:68) > at com.aliyun.oss.internal.OSSOperation.send(OSSOperation.java:94) > at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:149) > at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:113) > at > com.aliyun.oss.internal.OSSMultipartOperation.completeMultipartUpload(OSSMultipartOperation.java:185) > at com.aliyun.oss.OSSClient.completeMultipartUpload(OSSClient.java:790) > at > org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystemStore.completeMultipartUpload(AliyunOSSFileSystemStore.java:643) > at > org.apache.hadoop.fs.aliyun.oss.AliyunOSSBlockOutputStream.close(AliyunOSSBlockOutputStream.java:120) > at > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72) > at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101) > at > org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWriter.close(TextOutputFormat.java:106) > at > org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.close(MultipleOutputs.java:574) > at org.notmysock.tpcds.GenTable$DSDGen.cleanup(GenTable.java:169) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:149) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1686) > > I reviewed code below, > {code:java} > blockId {code} > has thread synchronization problem > {code:java} > // code placeholder > private void uploadCurrentPart() throws IOException { > blockFiles.add(blockFile); > blockStream.flush(); > blockStream.close(); > if (blockId == 0) { > uploadId = store.getUploadId(key); > } > ListenableFuture partETagFuture = > executorService.submit(() -> { > PartETag partETag = store.uploadPart(blockFile, key, uploadId, > blockId + 1); > return partETag; > }); > partETagsFutures.add(partETagFuture); > blockFile = newBlockFile(); > blockId++; > blockStream = new BufferedOutputStream(new FileOutputStream(blockFile)); > } > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15607) AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream
[ https://issues.apache.org/jira/browse/HADOOP-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16559378#comment-16559378 ] genericqa commented on HADOOP-15607: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m 22s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 14s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 22s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 46s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 29s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 19s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 16s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 19s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 27s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 35s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 19s{color} | {color:green} hadoop-aliyun in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 23s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 55m 46s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 | | JIRA Issue | HADOOP-15607 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12933319/HADOOP-15607.004.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 2719c77de48c 3.13.0-144-generic #193-Ubuntu SMP Thu Mar 15 17:03:53 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 8d3c068 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_171 | | findbugs | v3.1.0-RC1 | | Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/14954/testReport/ | | Max. process+thread count | 336 (vs. ulimit of 1) | | modules | C: hadoop-tools/hadoop-aliyun U: hadoop-tools/hadoop-aliyun | | Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/14954/console | | Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream > - > >
[jira] [Commented] (HADOOP-15607) AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream
[ https://issues.apache.org/jira/browse/HADOOP-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16559315#comment-16559315 ] SammiChen commented on HADOOP-15607: Hi [~wujinhu], the 003 patch looks overall good. A few minor issues, # testMultiPartUploadConcurrent, cross line indent is 4 space, not 8 space # conf.setInt("fs.oss.upload.active.blocks", 20); Use the Constants field # conf.setInt(IO_CHUNK_BUFFER_SIZE, conf.getInt(Constants.MULTIPART_UPLOAD_PART_SIZE_KEY, 0)); line exceeds 80 characters # removePartFiles is also called in uploadCurrentPart. It can help to remove the temp files as soon as possible. But if a temp file is deleted, it's corresponding partETagFuture is not removed from the partETagsFutures. You will find many misleading "Failed to delete temporary file {}" messages later in the log file. The newly added UT really helps verify the issues is fixed. > AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream > - > > Key: HADOOP-15607 > URL: https://issues.apache.org/jira/browse/HADOOP-15607 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.10.0, 2.9.1, 3.2.0, 3.0.3 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Attachments: HADOOP-15607.001.patch, HADOOP-15607.002.patch, > HADOOP-15607.003.patch > > > When I generated data with hive-tpcds tool, I got exception below: > 2018-07-16 14:50:43,680 INFO mapreduce.Job: Task Id : > attempt_1531723399698_0001_m_52_0, Status : FAILED > Error: com.aliyun.oss.OSSException: The list of parts was not in ascending > order. Parts list must specified in order by part number. > [ErrorCode]: InvalidPartOrder > [RequestId]: 5B4C40425FCC208D79D1EAF5 > [HostId]: 100.103.0.137 > [ResponseError]: > > > InvalidPartOrder > The list of parts was not in ascending order. Parts list must > specified in order by part number. > 5B4C40425FCC208D79D1EAF5 > xx.xx.xx.xx > current PartNumber 3, you given part number 3is not in > ascending order > > at > com.aliyun.oss.common.utils.ExceptionFactory.createOSSException(ExceptionFactory.java:99) > at > com.aliyun.oss.internal.OSSErrorResponseHandler.handle(OSSErrorResponseHandler.java:69) > at > com.aliyun.oss.common.comm.ServiceClient.handleResponse(ServiceClient.java:248) > at > com.aliyun.oss.common.comm.ServiceClient.sendRequestImpl(ServiceClient.java:130) > at > com.aliyun.oss.common.comm.ServiceClient.sendRequest(ServiceClient.java:68) > at com.aliyun.oss.internal.OSSOperation.send(OSSOperation.java:94) > at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:149) > at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:113) > at > com.aliyun.oss.internal.OSSMultipartOperation.completeMultipartUpload(OSSMultipartOperation.java:185) > at com.aliyun.oss.OSSClient.completeMultipartUpload(OSSClient.java:790) > at > org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystemStore.completeMultipartUpload(AliyunOSSFileSystemStore.java:643) > at > org.apache.hadoop.fs.aliyun.oss.AliyunOSSBlockOutputStream.close(AliyunOSSBlockOutputStream.java:120) > at > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72) > at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101) > at > org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWriter.close(TextOutputFormat.java:106) > at > org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.close(MultipleOutputs.java:574) > at org.notmysock.tpcds.GenTable$DSDGen.cleanup(GenTable.java:169) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:149) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1686) > > I reviewed code below, > {code:java} > blockId {code} > has thread synchronization problem > {code:java} > // code placeholder > private void uploadCurrentPart() throws IOException { > blockFiles.add(blockFile); > blockStream.flush(); > blockStream.close(); > if (blockId == 0) { > uploadId = store.getUploadId(key); > } > ListenableFuture partETagFuture = > executorService.submit(() -> { > PartETag partETag = store.uploadPart(blockFile, key, uploadId, > blockId + 1); > return partETag; > }); > partETagsFutures.add(partETagFuture); > blockFile = newBlockFile(); > blockId++; > blockStream = new BufferedOutputStream(new File
[jira] [Commented] (HADOOP-15607) AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream
[ https://issues.apache.org/jira/browse/HADOOP-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16554234#comment-16554234 ] genericqa commented on HADOOP-15607: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m 50s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 15s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 24s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 41s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 32s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 21s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 17s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 9s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 19s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 26s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 19s{color} | {color:green} hadoop-aliyun in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 24s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 56m 14s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 | | JIRA Issue | HADOOP-15607 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12932879/HADOOP-15607.003.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 00f2aae744e9 3.13.0-144-generic #193-Ubuntu SMP Thu Mar 15 17:03:53 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / ff7c2ed | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_171 | | findbugs | v3.1.0-RC1 | | Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/14936/testReport/ | | Max. process+thread count | 336 (vs. ulimit of 1) | | modules | C: hadoop-tools/hadoop-aliyun U: hadoop-tools/hadoop-aliyun | | Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/14936/console | | Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream > - > >
[jira] [Commented] (HADOOP-15607) AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream
[ https://issues.apache.org/jira/browse/HADOOP-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16554209#comment-16554209 ] wujinhu commented on HADOOP-15607: -- [~Sammi] Finally, I reproduced this issue by changing some configurations in my local environment successfully, please help to review again, thanks. > AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream > - > > Key: HADOOP-15607 > URL: https://issues.apache.org/jira/browse/HADOOP-15607 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.10.0, 2.9.1, 3.2.0, 3.1.1, 3.0.3 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Attachments: HADOOP-15607.001.patch, HADOOP-15607.002.patch, > HADOOP-15607.003.patch > > > When I generated data with hive-tpcds tool, I got exception below: > 2018-07-16 14:50:43,680 INFO mapreduce.Job: Task Id : > attempt_1531723399698_0001_m_52_0, Status : FAILED > Error: com.aliyun.oss.OSSException: The list of parts was not in ascending > order. Parts list must specified in order by part number. > [ErrorCode]: InvalidPartOrder > [RequestId]: 5B4C40425FCC208D79D1EAF5 > [HostId]: 100.103.0.137 > [ResponseError]: > > > InvalidPartOrder > The list of parts was not in ascending order. Parts list must > specified in order by part number. > 5B4C40425FCC208D79D1EAF5 > xx.xx.xx.xx > current PartNumber 3, you given part number 3is not in > ascending order > > at > com.aliyun.oss.common.utils.ExceptionFactory.createOSSException(ExceptionFactory.java:99) > at > com.aliyun.oss.internal.OSSErrorResponseHandler.handle(OSSErrorResponseHandler.java:69) > at > com.aliyun.oss.common.comm.ServiceClient.handleResponse(ServiceClient.java:248) > at > com.aliyun.oss.common.comm.ServiceClient.sendRequestImpl(ServiceClient.java:130) > at > com.aliyun.oss.common.comm.ServiceClient.sendRequest(ServiceClient.java:68) > at com.aliyun.oss.internal.OSSOperation.send(OSSOperation.java:94) > at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:149) > at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:113) > at > com.aliyun.oss.internal.OSSMultipartOperation.completeMultipartUpload(OSSMultipartOperation.java:185) > at com.aliyun.oss.OSSClient.completeMultipartUpload(OSSClient.java:790) > at > org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystemStore.completeMultipartUpload(AliyunOSSFileSystemStore.java:643) > at > org.apache.hadoop.fs.aliyun.oss.AliyunOSSBlockOutputStream.close(AliyunOSSBlockOutputStream.java:120) > at > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72) > at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101) > at > org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWriter.close(TextOutputFormat.java:106) > at > org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.close(MultipleOutputs.java:574) > at org.notmysock.tpcds.GenTable$DSDGen.cleanup(GenTable.java:169) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:149) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1686) > > I reviewed code below, > {code:java} > blockId {code} > has thread synchronization problem > {code:java} > // code placeholder > private void uploadCurrentPart() throws IOException { > blockFiles.add(blockFile); > blockStream.flush(); > blockStream.close(); > if (blockId == 0) { > uploadId = store.getUploadId(key); > } > ListenableFuture partETagFuture = > executorService.submit(() -> { > PartETag partETag = store.uploadPart(blockFile, key, uploadId, > blockId + 1); > return partETag; > }); > partETagsFutures.add(partETagFuture); > blockFile = newBlockFile(); > blockId++; > blockStream = new BufferedOutputStream(new FileOutputStream(blockFile)); > } > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15607) AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream
[ https://issues.apache.org/jira/browse/HADOOP-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16552455#comment-16552455 ] wujinhu commented on HADOOP-15607: -- Thanks [~Sammi] [~uncleGen] for your comments. As the code shows, uploadCurrentPart() executes blockId++ when submit tasks to the thread pool. If task executes later than the main thread, then the blockId has been changed. Changing blockFiles type is to fix another problem(we cannot delete files in write(), because they may be used by upload threads), so we can track upload files by blockId. I have tried to add unit test to reproduce this issue in my mac environment, but failed. It seems difficult to reproduce this in local environment only when I debug old code sometimes. > AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream > - > > Key: HADOOP-15607 > URL: https://issues.apache.org/jira/browse/HADOOP-15607 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.10.0, 2.9.1, 3.2.0, 3.1.1, 3.0.3 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Attachments: HADOOP-15607.001.patch, HADOOP-15607.002.patch > > > When I generated data with hive-tpcds tool, I got exception below: > 2018-07-16 14:50:43,680 INFO mapreduce.Job: Task Id : > attempt_1531723399698_0001_m_52_0, Status : FAILED > Error: com.aliyun.oss.OSSException: The list of parts was not in ascending > order. Parts list must specified in order by part number. > [ErrorCode]: InvalidPartOrder > [RequestId]: 5B4C40425FCC208D79D1EAF5 > [HostId]: 100.103.0.137 > [ResponseError]: > > > InvalidPartOrder > The list of parts was not in ascending order. Parts list must > specified in order by part number. > 5B4C40425FCC208D79D1EAF5 > xx.xx.xx.xx > current PartNumber 3, you given part number 3is not in > ascending order > > at > com.aliyun.oss.common.utils.ExceptionFactory.createOSSException(ExceptionFactory.java:99) > at > com.aliyun.oss.internal.OSSErrorResponseHandler.handle(OSSErrorResponseHandler.java:69) > at > com.aliyun.oss.common.comm.ServiceClient.handleResponse(ServiceClient.java:248) > at > com.aliyun.oss.common.comm.ServiceClient.sendRequestImpl(ServiceClient.java:130) > at > com.aliyun.oss.common.comm.ServiceClient.sendRequest(ServiceClient.java:68) > at com.aliyun.oss.internal.OSSOperation.send(OSSOperation.java:94) > at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:149) > at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:113) > at > com.aliyun.oss.internal.OSSMultipartOperation.completeMultipartUpload(OSSMultipartOperation.java:185) > at com.aliyun.oss.OSSClient.completeMultipartUpload(OSSClient.java:790) > at > org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystemStore.completeMultipartUpload(AliyunOSSFileSystemStore.java:643) > at > org.apache.hadoop.fs.aliyun.oss.AliyunOSSBlockOutputStream.close(AliyunOSSBlockOutputStream.java:120) > at > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72) > at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101) > at > org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWriter.close(TextOutputFormat.java:106) > at > org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.close(MultipleOutputs.java:574) > at org.notmysock.tpcds.GenTable$DSDGen.cleanup(GenTable.java:169) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:149) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1686) > > I reviewed code below, > {code:java} > blockId {code} > has thread synchronization problem > {code:java} > // code placeholder > private void uploadCurrentPart() throws IOException { > blockFiles.add(blockFile); > blockStream.flush(); > blockStream.close(); > if (blockId == 0) { > uploadId = store.getUploadId(key); > } > ListenableFuture partETagFuture = > executorService.submit(() -> { > PartETag partETag = store.uploadPart(blockFile, key, uploadId, > blockId + 1); > return partETag; > }); > partETagsFutures.add(partETagFuture); > blockFile = newBlockFile(); > blockId++; > blockStream = new BufferedOutputStream(new FileOutputStream(blockFile)); > } > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HADOOP-15607) AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream
[ https://issues.apache.org/jira/browse/HADOOP-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16550151#comment-16550151 ] SammiChen commented on HADOOP-15607: Thanks [~wujinhu] for reporting and working on it. Can you explain more detail about how the duplicate part numbers are generated. Is the blockFiles type change from List to Map to guarantee no duplicate part number? A unit test case is preferred to verify the issue. > AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream > - > > Key: HADOOP-15607 > URL: https://issues.apache.org/jira/browse/HADOOP-15607 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.10.0, 2.9.1, 3.2.0, 3.1.1, 3.0.3 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Attachments: HADOOP-15607.001.patch, HADOOP-15607.002.patch > > > When I generated data with hive-tpcds tool, I got exception below: > 2018-07-16 14:50:43,680 INFO mapreduce.Job: Task Id : > attempt_1531723399698_0001_m_52_0, Status : FAILED > Error: com.aliyun.oss.OSSException: The list of parts was not in ascending > order. Parts list must specified in order by part number. > [ErrorCode]: InvalidPartOrder > [RequestId]: 5B4C40425FCC208D79D1EAF5 > [HostId]: 100.103.0.137 > [ResponseError]: > > > InvalidPartOrder > The list of parts was not in ascending order. Parts list must > specified in order by part number. > 5B4C40425FCC208D79D1EAF5 > 100.103.0.137 > current PartNumber 3, you given part number 3is not in > ascending order > > at > com.aliyun.oss.common.utils.ExceptionFactory.createOSSException(ExceptionFactory.java:99) > at > com.aliyun.oss.internal.OSSErrorResponseHandler.handle(OSSErrorResponseHandler.java:69) > at > com.aliyun.oss.common.comm.ServiceClient.handleResponse(ServiceClient.java:248) > at > com.aliyun.oss.common.comm.ServiceClient.sendRequestImpl(ServiceClient.java:130) > at > com.aliyun.oss.common.comm.ServiceClient.sendRequest(ServiceClient.java:68) > at com.aliyun.oss.internal.OSSOperation.send(OSSOperation.java:94) > at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:149) > at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:113) > at > com.aliyun.oss.internal.OSSMultipartOperation.completeMultipartUpload(OSSMultipartOperation.java:185) > at com.aliyun.oss.OSSClient.completeMultipartUpload(OSSClient.java:790) > at > org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystemStore.completeMultipartUpload(AliyunOSSFileSystemStore.java:643) > at > org.apache.hadoop.fs.aliyun.oss.AliyunOSSBlockOutputStream.close(AliyunOSSBlockOutputStream.java:120) > at > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72) > at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101) > at > org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWriter.close(TextOutputFormat.java:106) > at > org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.close(MultipleOutputs.java:574) > at org.notmysock.tpcds.GenTable$DSDGen.cleanup(GenTable.java:169) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:149) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1686) > > I reviewed code below, > {code:java} > blockId {code} > has thread synchronization problem > {code:java} > // code placeholder > private void uploadCurrentPart() throws IOException { > blockFiles.add(blockFile); > blockStream.flush(); > blockStream.close(); > if (blockId == 0) { > uploadId = store.getUploadId(key); > } > ListenableFuture partETagFuture = > executorService.submit(() -> { > PartETag partETag = store.uploadPart(blockFile, key, uploadId, > blockId + 1); > return partETag; > }); > partETagsFutures.add(partETagFuture); > blockFile = newBlockFile(); > blockId++; > blockStream = new BufferedOutputStream(new FileOutputStream(blockFile)); > } > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15607) AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream
[ https://issues.apache.org/jira/browse/HADOOP-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546125#comment-16546125 ] genericqa commented on HADOOP-15607: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} docker {color} | {color:red} 7m 47s{color} | {color:red} Docker failed to build yetus/hadoop:abb62dd. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | HADOOP-15607 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12931891/HADOOP-15607.002.patch | | Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/14894/console | | Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream > - > > Key: HADOOP-15607 > URL: https://issues.apache.org/jira/browse/HADOOP-15607 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.10.0, 2.9.1, 3.2.0, 3.1.1, 3.0.3 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Attachments: HADOOP-15607.001.patch, HADOOP-15607.002.patch > > > When I generated data with hive-tpcds tool, I got exception below: > 2018-07-16 14:50:43,680 INFO mapreduce.Job: Task Id : > attempt_1531723399698_0001_m_52_0, Status : FAILED > Error: com.aliyun.oss.OSSException: The list of parts was not in ascending > order. Parts list must specified in order by part number. > [ErrorCode]: InvalidPartOrder > [RequestId]: 5B4C40425FCC208D79D1EAF5 > [HostId]: 100.103.0.137 > [ResponseError]: > > > InvalidPartOrder > The list of parts was not in ascending order. Parts list must > specified in order by part number. > 5B4C40425FCC208D79D1EAF5 > 100.103.0.137 > current PartNumber 3, you given part number 3is not in > ascending order > > at > com.aliyun.oss.common.utils.ExceptionFactory.createOSSException(ExceptionFactory.java:99) > at > com.aliyun.oss.internal.OSSErrorResponseHandler.handle(OSSErrorResponseHandler.java:69) > at > com.aliyun.oss.common.comm.ServiceClient.handleResponse(ServiceClient.java:248) > at > com.aliyun.oss.common.comm.ServiceClient.sendRequestImpl(ServiceClient.java:130) > at > com.aliyun.oss.common.comm.ServiceClient.sendRequest(ServiceClient.java:68) > at com.aliyun.oss.internal.OSSOperation.send(OSSOperation.java:94) > at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:149) > at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:113) > at > com.aliyun.oss.internal.OSSMultipartOperation.completeMultipartUpload(OSSMultipartOperation.java:185) > at com.aliyun.oss.OSSClient.completeMultipartUpload(OSSClient.java:790) > at > org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystemStore.completeMultipartUpload(AliyunOSSFileSystemStore.java:643) > at > org.apache.hadoop.fs.aliyun.oss.AliyunOSSBlockOutputStream.close(AliyunOSSBlockOutputStream.java:120) > at > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72) > at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101) > at > org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWriter.close(TextOutputFormat.java:106) > at > org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.close(MultipleOutputs.java:574) > at org.notmysock.tpcds.GenTable$DSDGen.cleanup(GenTable.java:169) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:149) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1686) > > I reviewed code below, > {code:java} > blockId {code} > has thread synchronization problem > {code:java} > // code placeholder > private void uploadCurrentPart() throws IOException { > blockFiles.add(blockFile); > blockStream.flush(); > blockStream.close(); > if (blockId == 0) { > uploadId = store.getUploadId(key); > } > ListenableFuture partETagFuture = > executorService.submit(() -> { > PartETag partETag = store.uploadPart(blockFile, key, uploadId, > blockId + 1); > return partETag; > }); > partETagsFutures.add(partETagFuture); > blockFile = newBlockFile(); > blockId++; > blockStream = new BufferedOutputStream(new FileOutputStre
[jira] [Commented] (HADOOP-15607) AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream
[ https://issues.apache.org/jira/browse/HADOOP-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16545952#comment-16545952 ] Genmao Yu commented on HADOOP-15607: [~wujinhu] Overall LGTM,but maybe we should use one intermediate variable for blockId to pass to upload task, like: {code:java} int bid = block++ PartETag partETag = store.uploadPart(currentFile, key, uploadId, bid); ... {code} > AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream > - > > Key: HADOOP-15607 > URL: https://issues.apache.org/jira/browse/HADOOP-15607 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.10.0, 2.9.1, 3.2.0, 3.1.1, 3.0.3 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Attachments: HADOOP-15607.001.patch > > > When I generated data with hive-tpcds tool, I got exception below: > 2018-07-16 14:50:43,680 INFO mapreduce.Job: Task Id : > attempt_1531723399698_0001_m_52_0, Status : FAILED > Error: com.aliyun.oss.OSSException: The list of parts was not in ascending > order. Parts list must specified in order by part number. > [ErrorCode]: InvalidPartOrder > [RequestId]: 5B4C40425FCC208D79D1EAF5 > [HostId]: 100.103.0.137 > [ResponseError]: > > > InvalidPartOrder > The list of parts was not in ascending order. Parts list must > specified in order by part number. > 5B4C40425FCC208D79D1EAF5 > 100.103.0.137 > current PartNumber 3, you given part number 3is not in > ascending order > > at > com.aliyun.oss.common.utils.ExceptionFactory.createOSSException(ExceptionFactory.java:99) > at > com.aliyun.oss.internal.OSSErrorResponseHandler.handle(OSSErrorResponseHandler.java:69) > at > com.aliyun.oss.common.comm.ServiceClient.handleResponse(ServiceClient.java:248) > at > com.aliyun.oss.common.comm.ServiceClient.sendRequestImpl(ServiceClient.java:130) > at > com.aliyun.oss.common.comm.ServiceClient.sendRequest(ServiceClient.java:68) > at com.aliyun.oss.internal.OSSOperation.send(OSSOperation.java:94) > at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:149) > at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:113) > at > com.aliyun.oss.internal.OSSMultipartOperation.completeMultipartUpload(OSSMultipartOperation.java:185) > at com.aliyun.oss.OSSClient.completeMultipartUpload(OSSClient.java:790) > at > org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystemStore.completeMultipartUpload(AliyunOSSFileSystemStore.java:643) > at > org.apache.hadoop.fs.aliyun.oss.AliyunOSSBlockOutputStream.close(AliyunOSSBlockOutputStream.java:120) > at > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72) > at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101) > at > org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWriter.close(TextOutputFormat.java:106) > at > org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.close(MultipleOutputs.java:574) > at org.notmysock.tpcds.GenTable$DSDGen.cleanup(GenTable.java:169) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:149) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1686) > > I reviewed code below, > {code:java} > blockId {code} > has thread synchronization problem > {code:java} > // code placeholder > private void uploadCurrentPart() throws IOException { > blockFiles.add(blockFile); > blockStream.flush(); > blockStream.close(); > if (blockId == 0) { > uploadId = store.getUploadId(key); > } > ListenableFuture partETagFuture = > executorService.submit(() -> { > PartETag partETag = store.uploadPart(blockFile, key, uploadId, > blockId + 1); > return partETag; > }); > partETagsFutures.add(partETagFuture); > blockFile = newBlockFile(); > blockId++; > blockStream = new BufferedOutputStream(new FileOutputStream(blockFile)); > } > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15607) AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream
[ https://issues.apache.org/jira/browse/HADOOP-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16545913#comment-16545913 ] wujinhu commented on HADOOP-15607: -- [~uncleGen] [~Sammi] Please help to review this patch, thanks! > AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream > - > > Key: HADOOP-15607 > URL: https://issues.apache.org/jira/browse/HADOOP-15607 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.10.0, 2.9.1, 3.2.0, 3.1.1, 3.0.3 >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Attachments: HADOOP-15607.001.patch > > > When I generated data with hive-tpcds tool, I got exception below: > 2018-07-16 14:50:43,680 INFO mapreduce.Job: Task Id : > attempt_1531723399698_0001_m_52_0, Status : FAILED > Error: com.aliyun.oss.OSSException: The list of parts was not in ascending > order. Parts list must specified in order by part number. > [ErrorCode]: InvalidPartOrder > [RequestId]: 5B4C40425FCC208D79D1EAF5 > [HostId]: 100.103.0.137 > [ResponseError]: > > > InvalidPartOrder > The list of parts was not in ascending order. Parts list must > specified in order by part number. > 5B4C40425FCC208D79D1EAF5 > 100.103.0.137 > current PartNumber 3, you given part number 3is not in > ascending order > > at > com.aliyun.oss.common.utils.ExceptionFactory.createOSSException(ExceptionFactory.java:99) > at > com.aliyun.oss.internal.OSSErrorResponseHandler.handle(OSSErrorResponseHandler.java:69) > at > com.aliyun.oss.common.comm.ServiceClient.handleResponse(ServiceClient.java:248) > at > com.aliyun.oss.common.comm.ServiceClient.sendRequestImpl(ServiceClient.java:130) > at > com.aliyun.oss.common.comm.ServiceClient.sendRequest(ServiceClient.java:68) > at com.aliyun.oss.internal.OSSOperation.send(OSSOperation.java:94) > at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:149) > at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:113) > at > com.aliyun.oss.internal.OSSMultipartOperation.completeMultipartUpload(OSSMultipartOperation.java:185) > at com.aliyun.oss.OSSClient.completeMultipartUpload(OSSClient.java:790) > at > org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystemStore.completeMultipartUpload(AliyunOSSFileSystemStore.java:643) > at > org.apache.hadoop.fs.aliyun.oss.AliyunOSSBlockOutputStream.close(AliyunOSSBlockOutputStream.java:120) > at > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72) > at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101) > at > org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWriter.close(TextOutputFormat.java:106) > at > org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.close(MultipleOutputs.java:574) > at org.notmysock.tpcds.GenTable$DSDGen.cleanup(GenTable.java:169) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:149) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1686) > > I reviewed code below, > {code:java} > blockId {code} > has thread synchronization problem > {code:java} > // code placeholder > private void uploadCurrentPart() throws IOException { > blockFiles.add(blockFile); > blockStream.flush(); > blockStream.close(); > if (blockId == 0) { > uploadId = store.getUploadId(key); > } > ListenableFuture partETagFuture = > executorService.submit(() -> { > PartETag partETag = store.uploadPart(blockFile, key, uploadId, > blockId + 1); > return partETag; > }); > partETagsFutures.add(partETagFuture); > blockFile = newBlockFile(); > blockId++; > blockStream = new BufferedOutputStream(new FileOutputStream(blockFile)); > } > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15607) AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream
[ https://issues.apache.org/jira/browse/HADOOP-15607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16545201#comment-16545201 ] genericqa commented on HADOOP-15607: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 23s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m 41s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 21s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 15s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 23s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 39s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 31s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 19s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 17s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 19s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 30s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 19s{color} | {color:green} hadoop-aliyun in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 25s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 56m 9s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd | | JIRA Issue | HADOOP-15607 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12931784/HADOOP-15607.001.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux d762ce6a1d98 3.13.0-144-generic #193-Ubuntu SMP Thu Mar 15 17:03:53 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 5074ca9 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_171 | | findbugs | v3.1.0-RC1 | | Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/14890/testReport/ | | Max. process+thread count | 335 (vs. ulimit of 1) | | modules | C: hadoop-tools/hadoop-aliyun U: hadoop-tools/hadoop-aliyun | | Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/14890/console | | Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > AliyunOSS: fix duplicated part