ivankelly commented on a change in pull request #2152: GCS offload support(4): add documentations for GCS URL: https://github.com/apache/incubator-pulsar/pull/2152#discussion_r205820083
########## File path: site/docs/latest/cookbooks/tiered-storage.md ########## @@ -17,44 +19,50 @@ A topic in Pulsar is backed by a log, known as a managed ledger. This log is com The Tiered Storage offloading mechanism takes advantage of this segment oriented architecture. When offloading is requested, the segments of the log are copied, one-by-one, to tiered storage. All segments of the log, apart from the segment currently being written to can be offloaded. -## Amazon S3 - -Tiered storage currently supports S3 for long term storage. On the broker, the administrator must configure a S3 bucket and the AWS region where the bucket exists. Offloaded data will be placed into this bucket. +On the broker, the administrator must configure the bucket or credentials for the cloud storage service. The configured bucket must exist before attempting to offload. If it does not exist, the offload operation will fail. -The configured S3 bucket must exist before attempting to offload. If it does not exist, the offload operation will fail. +Pulsar uses multi-part objects to upload the segment data. It is possible that a broker could crash while uploading the data. We recommend you add a life cycle rule your bucket to expire incomplete multi-part upload after a day or two to avoid getting charged for incomplete uploads. -Pulsar users multipart objects to update the segment data. It is possible that a broker could crash while uploading the data. We recommend you add a lifecycle rule your S3 bucket to expire incomplete multipart upload after a day or two to avoid getting charged for incomplete uploads. - -### Configuring the broker +## Configuring the driver for "aws-s3" or "google-cloud-storage" in the broker Offloading is configured in ```broker.conf```. -At a minimum, the user must configure the driver, the region and the bucket. +At a minimum, the administrator must configure the driver, the bucket and the authenticating. There is also some other knobs to configure, like the bucket regions, the max block size in backed storage, etc. + +Currently we support driver of types: { "aws-s3", "google-cloud-storage" }, +{% include admonition.html type="warning" content="Driver names are case-insensitive for driver's name. "s3" and "aws-s3" are similar, with "aws-s3" you just don't need to define the url of the endpoint because it is aligned with region, and default is `s3.amazonaws.com`; while with s3, you must provide the endpoint url by `s3ManagedLedgerOffloadServiceEndpoint`." %} ```conf -managedLedgerOffloadDriver=S3 -s3ManagedLedgerOffloadRegion=eu-west-3 -s3ManagedLedgerOffloadBucket=pulsar-topic-offload +managedLedgerOffloadDriver=aws-s3 ``` -It is also possible to specify the s3 endpoint directly, using ```s3ManagedLedgerOffloadServiceEndpoint```. This is useful if you are using a non-AWS storage service which provides an S3 compatible API. +### Configuring for "aws-s3" driver -{% include admonition.html type="warning" content="If the endpoint is specified directly, then the region must _not_ be set." %} +#### Configuring the Bucket -{% include admonition.html type="warning" content="The broker.conf of all brokers must have the same configuration for driver, region and bucket for offload to avoid data becoming unavailable as topics move from one broker to another." %} +On the broker, the administrator must configure the bucket and credentials for the cloud storage service. The configured bucket and credentials must exist before attempting to offload. If it does not exist, the offload operation will fail. -Pulsar also provides some knobs to configure the size of requests sent to S3. +Regarding AWS S3, the administrator should configure `s3ManagedLedgerOffloadBucket`. -- ```s3ManagedLedgerOffloadMaxBlockSizeInBytes``` configures the maximum size of a "part" sent during a multipart upload. This cannot be smaller than 5MB. Default is 64MB. -- ```s3ManagedLedgerOffloadReadBufferSizeInBytes``` configures the block size for each individual read when reading back data from S3. Default is 1MB. +```conf +s3ManagedLedgerOffloadBucket=pulsar-topic-offload +``` -In both cases, these should not be touched unless you know what you are doing. +#### Configuring the Bucket Region -{% include admonition.html type="warning" content="The broker must be rebooted for any changes in the configuration to take effect." %} +Bucket Region is the region where bucket located. -### Authenticating with S3 +Regarding AWS S3, the default region is `US East (N. Virginia)`. Page [AWS Regions and Endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html) contains more information. -To be able to access S3, you need to authenticate with S3. Pulsar does not provide any direct means of configuring authentication for S3, but relies on the mechanisms supported by the [DefaultAWSCredentialsProviderChain](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html). +- AWS S3 Region example: + +```conf +s3ManagedLedgerOffloadRegion=eu-west-3 +``` + +#### Configuring the Authenticating Review comment: Just "Authentication" is enough. Or "Authentication with AWS" ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services