[
https://issues.apache.org/jira/browse/HADOOP-17454?focusedWorklogId=531148&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-531148
]
ASF GitHub Bot logged work on HADOOP-17454:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 05/Jan/21 12:11
Start Date: 05/Jan/21 12:11
Worklog Time Spent: 10m
Work Description: bgaborg commented on pull request #2593:
URL: https://github.com/apache/hadoop/pull/2593#issuecomment-754598161
Testing: I had one failing test during the CLI run, but it was ok after a
rerun:
```
[ERROR]
ITestS3ADeleteCost.testDeleteSingleFileInDir:110->AbstractS3ACostTest.verifyMetrics:360->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88
operation returning after fs.delete(simpleFile) action_executor_acquired
starting=0 current=0 diff=0, action_http_get_request starting=0 current=0
diff=0, action_http_head_request starting=4 current=5 diff=1,
committer_bytes_committed starting=0 current=0 diff=0, committer_bytes_uploaded
starting=0 current=0 diff=0, committer_commit_job starting=0 current=0 diff=0,
committer_commits.failures starting=0 current=0 diff=0,
committer_commits_aborted starting=0 current=0 diff=0,
committer_commits_completed starting=0 current=0 diff=0,
committer_commits_created starting=0 current=0 diff=0,
committer_commits_reverted starting=0 current=0 diff=0,
committer_jobs_completed starting=0 current=0 diff=0, committer_jobs_failed
starting=0 current=0 diff=0, committer_magic_files_created starting=0 current=0
diff=0, committer_materialize_file starting=0 current=0 diff=0,
committer_stage_file_upload starting=0 current=0 diff=0,
committer_tasks_completed starting=0 current=0 diff=0, committer_tasks_failed
starting=0 current=0 diff=0, delegation_token_issued starting=0 current=0
diff=0, directories_created starting=2 current=3 diff=1, directories_deleted
starting=0 current=0 diff=0, fake_directories_created starting=0 current=0
diff=0, fake_directories_deleted starting=6 current=8 diff=2, files_copied
starting=0 current=0 diff=0, files_copied_bytes starting=0 current=0 diff=0,
files_created starting=1 current=1 diff=0, files_delete_rejected starting=0
current=0 diff=0, files_deleted starting=0 current=1 diff=1, ignored_errors
starting=0 current=0 diff=0, multipart_instantiated starting=0 current=0
diff=0, multipart_upload_abort_under_path_invoked starting=0 current=0 diff=0,
multipart_upload_aborted starting=0 current=0 diff=0,
multipart_upload_completed starting=0 current=0 diff=0,
multipart_upload_part_put starting=0 current=0 diff=0,
multipart_upload_part_put_bytes starting=0 current=0 diff=0,
multipart_upload_started starting=0 current=0 diff=0,
object_bulk_delete_request starting=3 current=4 diff=1,
object_continue_list_request starting=0 current=0 diff=0, object_copy_requests
starting=0 current=0 diff=0, object_delete_objects starting=6 current=9 diff=3,
object_delete_request starting=0 current=1 diff=1, object_list_request
starting=5 current=6 diff=1, object_metadata_request starting=4 current=5
diff=1, object_multipart_aborted starting=0 current=0 diff=0,
object_multipart_initiated starting=0 current=0 diff=0, object_put_bytes
starting=0 current=0 diff=0, object_put_request starting=3 current=4 diff=1,
object_put_request_completed starting=3 current=4 diff=1,
object_select_requests starting=0 current=0 diff=0, op_copy_from_local_file
starting=0 current=0 diff=0, op_create starting=1 current=1 diff=0,
op_create_non_recursive starting=0 current=0 diff=0, op_delete starting=0
current=1 diff=1, op_exists starting=0 current=0 diff=0,
op_get_delegation_token starting=0 current=0 diff=0, op_get_file_checksum
starting=0 current=0 diff=0, op_get_file_status starting=2 current=2 diff=0,
op_glob_status starting=0 current=0 diff=0, op_is_directory starting=0
current=0 diff=0, op_is_file starting=0 current=0 diff=0, op_list_files
starting=0 current=0 diff=0, op_list_located_status starting=0 current=0
diff=0, op_list_status starting=0 current=0 diff=0, op_mkdirs starting=2
current=2 diff=0, op_open starting=0 current=0 diff=0, op_rename starting=0
current=0 diff=0, s3guard_metadatastore_authoritative_directories_updated
starting=0 current=0 diff=0, s3guard_metadatastore_initialization starting=0
current=0 diff=0, s3guard_metadatastore_put_path_request starting=0 current=0
diff=0, s3guard_metadatastore_record_deletes starting=0 current=0 diff=0,
s3guard_metadatastore_record_reads starting=0 current=0 diff=0,
s3guard_metadatastore_record_writes starting=0 current=0 diff=0,
s3guard_metadatastore_retry starting=0 current=0 diff=0,
s3guard_metadatastore_throttled starting=0 current=0 diff=0, store_io_request
starting=0 current=0 diff=0, store_io_retry starting=0 current=0 diff=0,
store_io_throttled starting=0 current=0 diff=0, stream_aborted starting=0
current=0 diff=0, stream_read_bytes starting=0 current=0 diff=0,
stream_read_bytes_backwards_on_seek starting=0 current=0 diff=0,
stream_read_bytes_discarded_in_abort starting=0 current=0 diff=0,
stream_read_bytes_discarded_in_close starting=0 current=0 diff=0,
stream_read_close_operations starting=0 current=0 diff=0, stream_read_closed
starting=0 current=0 diff=0, stream_read_exceptions starting=0 current=0
diff=0, stream_read_fully_operations starting=0 current=0 diff=0,
stream_read_opened starting=0 current=0 diff=0, stream_read_operations
starting=0 current=0 diff=0, stream_read_operations_incomplete starting=0
current=0 diff=0, stream_read_seek_backward_operations starting=0 current=0
diff=0, stream_read_seek_bytes_discarded starting=0 current=0 diff=0,
stream_read_seek_bytes_skipped starting=0 current=0 diff=0,
stream_read_seek_forward_operations starting=0 current=0 diff=0,
stream_read_seek_operations starting=0 current=0 diff=0,
stream_read_seek_policy_changed starting=0 current=0 diff=0,
stream_read_total_bytes starting=0 current=0 diff=0,
stream_read_version_mismatches starting=0 current=0 diff=0,
stream_write_block_uploads starting=0 current=0 diff=0,
stream_write_block_uploads_aborted starting=0 current=0 diff=0,
stream_write_block_uploads_committed starting=0 current=0 diff=0,
stream_write_bytes starting=0 current=0 diff=0, stream_write_exceptions
starting=0 current=0 diff=0, stream_write_exceptions_completing_upload
starting=0 current=0 diff=0, stream_write_queue_duration starting=0 current=0
diff=0, stream_write_total_data starting=0 current=0 diff=0,
stream_write_total_time starting=0 current=0 diff=0: object_delete_objects
expected:<2> but was:<3>
```
does not seem related
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 531148)
Time Spent: 20m (was: 10m)
> [s3a] Disable bucket existence check - set fs.s3a.bucket.probe to 0
> -------------------------------------------------------------------
>
> Key: HADOOP-17454
> URL: https://issues.apache.org/jira/browse/HADOOP-17454
> Project: Hadoop Common
> Issue Type: Improvement
> Affects Versions: 3.3.0
> Reporter: Gabor Bota
> Assignee: Gabor Bota
> Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Set the value of fs.s3a.bucket.probe to 0 by default.
> Bucket existence checks are done in the initialization phase of the
> S3AFileSystem. It's not required to run this check: the operation itself will
> fail if the bucket does not exist instead of the check.
> Some points on why do we want to set this to 0:
> * When it's set to 0, bucket existence checks won't be done during
> initialization thus making it faster.
> * Avoid the additional one or two requests on the bucket root, so the user
> does not need rights to read or list that folder.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
