ivankelly commented on a change in pull request #2152: GCS offload support(4): add documentations for GCS URL: https://github.com/apache/incubator-pulsar/pull/2152#discussion_r205820379
########## File path: site/docs/latest/cookbooks/tiered-storage.md ########## @@ -86,6 +93,67 @@ If you are running in EC2 you can also use instance profile credentials, provide {% include admonition.html type="warning" content="The broker must be rebooted for credentials specified in pulsar_env to take effect." %} +#### Configuring the size of block read/write + +Pulsar also provides some knobs to configure the size of requests sent to AWS S3. + +- ```s3ManagedLedgerOffloadMaxBlockSizeInBytes``` configures the maximum size of a "part" sent during a multipart upload. This cannot be smaller than 5MB. Default is 64MB. +- ```s3ManagedLedgerOffloadReadBufferSizeInBytes``` configures the block size for each individual read when reading back data from AWS S3. Default is 1MB. + +In both cases, these should not be touched unless you know what you are doing. + + +### Configuring for "google-cloud-storage" driver + +#### Configuring the Bucket + +On the broker, the administrator must configure the bucket and credentials for the cloud storage service. The configured bucket and credentials must exist before attempting to offload. If it does not exist, the offload operation will fail. + +Regarding driver type "google-cloud-storage", the administrator should configure `gcsManagedLedgerOffloadBucket`. + +```conf +gcsManagedLedgerOffloadBucket=pulsar-topic-offload +``` + +#### Configuring the Bucket Region + +Bucket Region is the region where bucket located. + +Regarding GCS, buckets are default created in the `us multi-regional location`, page [Bucket Locations](https://cloud.google.com/storage/docs/bucket-locations) contains more information. + +- GCS Region example: + +```conf +gcsManagedLedgerOffloadRegion=europe-west3 +``` + +#### Configuring the Authenticating + +The administrator need configure `gcsManagedLedgerOffloadServiceAccountKeyFile` in `broker.conf` to get GCS service available. It is a Json file, which contains GCS credentials of service account key. Review comment: The administrator needs to configure `gcsManagedLedgerOffloadServiceAccountKeyFile` in `broker.conf` for the broker to be able to access the GCS service. `gcsManagedLedgerOffloadServiceAccountKeyFile` is a Json file, containing the GCS credentials of a service account. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services