[jira] [Commented] (HADOOP-18161) [WASB] Retry not getting implemented when using wasb scheme in hadoop-azure 2.7.4
[
https://issues.apache.org/jira/browse/HADOOP-18161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17509811#comment-17509811
]
Aryan commented on HADOOP-18161:
[~ste...@apache.org]
Do we have IO retry currently implemented for WASB protocol as we have for ABFS
protocol with hadoop-azure?
I came across this
doc:[https://hadoop.apache.org/docs/stable/hadoop-azure/abfs.html]
which has different properties for IO retry for ABFS protocol..but din't get
any such doc for WASB protocol?
Can you please confirm?
> [WASB] Retry not getting implemented when using wasb scheme in hadoop-azure
> 2.7.4
> --
>
> Key: HADOOP-18161
> URL: https://issues.apache.org/jira/browse/HADOOP-18161
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 2.7.4
>Reporter: Aryan
>Priority: Minor
>
> I am using prestodb to read data from blob.
> Presto is using hadoop-azure-2.7.4 jar.
> I'm using *wasb* scheme to query the data on blob. I'm afraid for some reason
> the hadoop-azure library is not retrying when getting IO exception.
> Attaching the stack trace below,
> {code:java}
> com.facebook.presto.spi.PrestoException: Error reading from
> wasb://oemdpv3prd...@oemdpv3prd.blob.core.windows.net/data/pipelines/hudi/kafka/telemetrics_v2/dp.hmi.quectel.bms.data.packet.v2/dt=2022-01-15/e576abc3-942a-434d-be02-6899798258eb-0_5-13327-290407_20220115211203.parquet
> at position 65924529
> at
> com.facebook.presto.hive.parquet.HdfsParquetDataSource.readInternal(HdfsParquetDataSource.java:66)
> at
> com.facebook.presto.parquet.AbstractParquetDataSource.readFully(AbstractParquetDataSource.java:60)
> at
> com.facebook.presto.parquet.AbstractParquetDataSource.readFully(AbstractParquetDataSource.java:51)
> at
> com.facebook.presto.parquet.reader.ParquetReader.readPrimitive(ParquetReader.java:247)
> at
> com.facebook.presto.parquet.reader.ParquetReader.readColumnChunk(ParquetReader.java:330)
> at
> com.facebook.presto.parquet.reader.ParquetReader.readBlock(ParquetReader.java:313)
> at
> com.facebook.presto.hive.parquet.ParquetPageSource$ParquetBlockLoader.load(ParquetPageSource.java:182)
> at
> com.facebook.presto.hive.parquet.ParquetPageSource$ParquetBlockLoader.load(ParquetPageSource.java:160)
> at
> com.facebook.presto.common.block.LazyBlock.assureLoaded(LazyBlock.java:291)
> at
> com.facebook.presto.common.block.LazyBlock.getLoadedBlock(LazyBlock.java:282)
> at
> com.facebook.presto.operator.ScanFilterAndProjectOperator$RecordingLazyBlockLoader.load(ScanFilterAndProjectOperator.java:314)
> at
> com.facebook.presto.operator.ScanFilterAndProjectOperator$RecordingLazyBlockLoader.load(ScanFilterAndProjectOperator.java:300)
> at
> com.facebook.presto.common.block.LazyBlock.assureLoaded(LazyBlock.java:291)
> at
> com.facebook.presto.common.block.LazyBlock.getLoadedBlock(LazyBlock.java:282)
> at
> com.facebook.presto.operator.project.InputPageProjection.project(InputPageProjection.java:69)
> at
> com.facebook.presto.operator.project.PageProjectionWithOutputs.project(PageProjectionWithOutputs.java:56)
> at
> com.facebook.presto.operator.project.PageProcessor$ProjectSelectedPositions.processBatch(PageProcessor.java:323)
> at
> com.facebook.presto.operator.project.PageProcessor$ProjectSelectedPositions.process(PageProcessor.java:197)
> at
> com.facebook.presto.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:315)
> at
> com.facebook.presto.operator.WorkProcessorUtils$YieldingIterator.computeNext(WorkProcessorUtils.java:79)
> at
> com.facebook.presto.operator.WorkProcessorUtils$YieldingIterator.computeNext(WorkProcessorUtils.java:65)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136)
> at
> com.facebook.presto.operator.project.MergingPageOutput.getOutput(MergingPageOutput.java:113)
> at
> com.facebook.presto.operator.ScanFilterAndProjectOperator.processPageSource(ScanFilterAndProjectOperator.java:295)
> at
> com.facebook.presto.operator.ScanFilterAndProjectOperator.getOutput(ScanFilterAndProjectOperator.java:242)
> at com.facebook.presto.operator.Driver.processInternal(Driver.java:418)
> at
> com.facebook.presto.operator.Driver.lambda$processFor$9(Driver.java:301)
> at com.facebook.presto.operator.Driver.tryWithLock(Driver.java:722)
> at com.facebook.presto.operator.Driver.processFor(Driver.java:294)
> at
> com.facebook.presto.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1077)
> at
> com.facebook.presto.execution.executor.P
[jira] [Commented] (HADOOP-18161) [WASB] Retry not getting implemented when using wasb scheme in hadoop-azure 2.7.4
[
https://issues.apache.org/jira/browse/HADOOP-18161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17507765#comment-17507765
]
Steve Loughran commented on HADOOP-18161:
-
hadoop 2.7 is six years old
please upgrade to hadoop 3.3, retest and, if the problem still exists, we will
take patches on the branch-3.3 line.
downgrading to minor
> [WASB] Retry not getting implemented when using wasb scheme in hadoop-azure
> 2.7.4
> --
>
> Key: HADOOP-18161
> URL: https://issues.apache.org/jira/browse/HADOOP-18161
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure, hadoop-thirdparty
>Affects Versions: 2.7.4
>Reporter: Aryan
>Priority: Major
>
> I am using prestodb to read data from blob.
> Presto is using hadoop-azure-2.7.4 jar.
> I'm using *wasb* scheme to query the data on blob. I'm afraid for some reason
> the hadoop-azure library is not retrying when getting IO exception.
> Attaching the stack trace below,
> {code:java}
> com.facebook.presto.spi.PrestoException: Error reading from
> wasb://oemdpv3prd...@oemdpv3prd.blob.core.windows.net/data/pipelines/hudi/kafka/telemetrics_v2/dp.hmi.quectel.bms.data.packet.v2/dt=2022-01-15/e576abc3-942a-434d-be02-6899798258eb-0_5-13327-290407_20220115211203.parquet
> at position 65924529
> at
> com.facebook.presto.hive.parquet.HdfsParquetDataSource.readInternal(HdfsParquetDataSource.java:66)
> at
> com.facebook.presto.parquet.AbstractParquetDataSource.readFully(AbstractParquetDataSource.java:60)
> at
> com.facebook.presto.parquet.AbstractParquetDataSource.readFully(AbstractParquetDataSource.java:51)
> at
> com.facebook.presto.parquet.reader.ParquetReader.readPrimitive(ParquetReader.java:247)
> at
> com.facebook.presto.parquet.reader.ParquetReader.readColumnChunk(ParquetReader.java:330)
> at
> com.facebook.presto.parquet.reader.ParquetReader.readBlock(ParquetReader.java:313)
> at
> com.facebook.presto.hive.parquet.ParquetPageSource$ParquetBlockLoader.load(ParquetPageSource.java:182)
> at
> com.facebook.presto.hive.parquet.ParquetPageSource$ParquetBlockLoader.load(ParquetPageSource.java:160)
> at
> com.facebook.presto.common.block.LazyBlock.assureLoaded(LazyBlock.java:291)
> at
> com.facebook.presto.common.block.LazyBlock.getLoadedBlock(LazyBlock.java:282)
> at
> com.facebook.presto.operator.ScanFilterAndProjectOperator$RecordingLazyBlockLoader.load(ScanFilterAndProjectOperator.java:314)
> at
> com.facebook.presto.operator.ScanFilterAndProjectOperator$RecordingLazyBlockLoader.load(ScanFilterAndProjectOperator.java:300)
> at
> com.facebook.presto.common.block.LazyBlock.assureLoaded(LazyBlock.java:291)
> at
> com.facebook.presto.common.block.LazyBlock.getLoadedBlock(LazyBlock.java:282)
> at
> com.facebook.presto.operator.project.InputPageProjection.project(InputPageProjection.java:69)
> at
> com.facebook.presto.operator.project.PageProjectionWithOutputs.project(PageProjectionWithOutputs.java:56)
> at
> com.facebook.presto.operator.project.PageProcessor$ProjectSelectedPositions.processBatch(PageProcessor.java:323)
> at
> com.facebook.presto.operator.project.PageProcessor$ProjectSelectedPositions.process(PageProcessor.java:197)
> at
> com.facebook.presto.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:315)
> at
> com.facebook.presto.operator.WorkProcessorUtils$YieldingIterator.computeNext(WorkProcessorUtils.java:79)
> at
> com.facebook.presto.operator.WorkProcessorUtils$YieldingIterator.computeNext(WorkProcessorUtils.java:65)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136)
> at
> com.facebook.presto.operator.project.MergingPageOutput.getOutput(MergingPageOutput.java:113)
> at
> com.facebook.presto.operator.ScanFilterAndProjectOperator.processPageSource(ScanFilterAndProjectOperator.java:295)
> at
> com.facebook.presto.operator.ScanFilterAndProjectOperator.getOutput(ScanFilterAndProjectOperator.java:242)
> at com.facebook.presto.operator.Driver.processInternal(Driver.java:418)
> at
> com.facebook.presto.operator.Driver.lambda$processFor$9(Driver.java:301)
> at com.facebook.presto.operator.Driver.tryWithLock(Driver.java:722)
> at com.facebook.presto.operator.Driver.processFor(Driver.java:294)
> at
> com.facebook.presto.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1077)
> at
> com.facebook.presto.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:162)
> at
> com.facebook.presto.execution.executor.TaskExecutor$TaskRunner.run(TaskEx
