This is an automated email from the ASF dual-hosted git repository. acosentino pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/camel-kamelets-examples.git
The following commit(s) were added to refs/heads/main by this push: new 2bb1641 Added a little example for Google Storage CDC Source Kamelet 2bb1641 is described below commit 2bb1641543a8eb20d5e5a1c363ab04ba416229c1 Author: Andrea Cosentino <anco...@gmail.com> AuthorDate: Wed Dec 14 13:52:51 2022 +0100 Added a little example for Google Storage CDC Source Kamelet Signed-off-by: Andrea Cosentino <anco...@gmail.com> --- jbang/google-storage-blob-cdc/README.adoc | 206 +++++++++++++++++++++ .../google-storage-cdc.yaml | 32 ++++ 2 files changed, 238 insertions(+) diff --git a/jbang/google-storage-blob-cdc/README.adoc b/jbang/google-storage-blob-cdc/README.adoc new file mode 100644 index 0000000..e1a5364 --- /dev/null +++ b/jbang/google-storage-blob-cdc/README.adoc @@ -0,0 +1,206 @@ +== Google Storage Blob CDC Example + +In this sample you'll use the Google Storage Blob CDC Source Kamelet. + +Through the usage of Google Pubsub and Google Storage Services you'll be able to consume events from specific bucket. + +=== Install JBang + +First install JBang according to https://www.jbang.dev + +When JBang is installed then you should be able to run from a shell: + +[source,sh] +---- +$ jbang --version +---- + +This will output the version of JBang. + +To run this example you can either install Camel on JBang via: + +[source,sh] +---- +$ jbang app install camel@apache/camel +---- + +Which allows to run CamelJBang with `camel` as shown below. + +=== Setup the Google Services and infrastructure through gcloud CLI + +First we need to create a project + +[source,sh] +---- +gcloud projects create gs-storage-cdc-test +---- + +We need to associate the project with a billing account + +[source,sh] +---- +gcloud alpha billing accounts projects link gs-storage-cdc-test --billing-account=XXXXXX-XXXXXX-XXXXXX +---- + +Now we could set the project as default, so we won't have to specify it in each command. + +[source,sh] +---- +gcloud config set project gs-storage-cdc-test +---- + +Let's create the Google Storage Bucket named gs-storage-cdc-test + +[source,sh] +---- +gcloud storage buckets create gs://gs-storage-cdc-test +---- + +Let's create the Google Pubsub Topic named gs-storage-cdc-test + +[source,sh] +---- +gcloud pubsub topics create gs-storage-cdc-test +---- + +Let's subscribe the pubsub topic to the event coming from the Google Storage bucket + +[source,sh] +---- +gcloud storage buckets notifications create gs://gs-storage-cdc-test --topic=gs-storage-cdc-test +---- + +Let's create a subscription to consume from the Google Pubsub topic + +[source,sh] +---- +gcloud pubsub subscriptions create gs-storage-cdc-test-sub --topic=gs-storage-cdc-test +---- + +Now it's time to create the Service Account to be used: + +[source,sh] +---- +gcloud iam service-accounts create gs-storage-cdc-test-sa --description="GCP Storage CDC Test" +---- + +Let's give to the SA a owner role + +[source,sh] +---- +gcloud projects add-iam-policy-binding gs-storage-cdc-test --member="serviceAccount:gs-storage-cdc-test...@gs-storage-cdc-test.iam.gserviceaccount.com" --role="roles/owner" +---- + +We need to create a service account key and then transform it to a base64 encoded string. + +[source,sh] +---- +gcloud iam service-accounts keys create gs-storage-cdc-test-sa.json --iam-account=gs-storage-cdc-test...@gs-storage-cdc-test.iam.gserviceaccount.com +---- + +Now we need to encode the value as base64 + +[source,sh] +---- +cat gs-storage-cdc-test-sa.json | base64 -w 0 +---- + +Place the value in the google-storage-cdc.yaml file as serviceAccountKey parameter. + +=== How to run + +Then you can run this example using: + +[source,sh] +---- +$ camel run --local-kamelet-dir=<local-kamelets-dir> google-storage-cdc.yaml +---- + +Replace the local kamelet dir with your local directory. + +Or run with JBang using the longer command line (without installing camel as app in JBang): + +[source,sh] +---- +$ jbang run camel@apache/camel run --local-kamelet-dir=<local-kamelets-dir> google-storage-cdc.yaml +---- + +Replace the local kamelet dir with your local directory. + +=== Developer Web Console + +You can enable the developer console via `--console` flag as show: + +[source,sh] +---- +$ camel run --local-kamelet-dir=<local-kamelets-dir> google-storage-cdc.yaml --console +---- + +Then you can browse: http://localhost:8080/q/dev to introspect the running Camel applicaton. + +=== Create and delete an object + +While the integration is running you can run the following commands: + +[source,sh] +---- +gcloud storage cp timer-mongodb.yaml gs://gs-storage-cdc-test +Copying file://timer-mongodb.yaml to gs://gs-storage-cdc-test/timer-mongodb.yaml + Completed files 1/1 | 1.4kiB/1.4kiB +gcloud storage rm gs://gs-storage-cdc-test/timer-mongodb.yaml +Removing objects: +Removing gs://gs-storage-cdc-test/timer-mongodb.yaml... + Completed 1/1 +---- + +You should see at first the content of the uploaded file and in the second message the deletion of the blob from the container. + +[source,sh] +---- +2022-12-14 12:04:52.347 INFO 91156 --- [ main] org.apache.camel.main.MainSupport : Apache Camel (JBang) 3.20.0-SNAPSHOT is starting +2022-12-14 12:04:52.542 INFO 91156 --- [ main] org.apache.camel.main.MainSupport : Using Java 11.0.16.1 with PID 91156. Started by oscerd in /home/oscerd/workspace/apache-camel/camel-kamelets-examples/jbang +2022-12-14 12:04:52.559 INFO 91156 --- [ main] he.camel.cli.connector.LocalCliConnector : Camel CLI enabled (local) +2022-12-14 12:04:55.173 INFO 91156 --- [ main] e.camel.impl.engine.AbstractCamelContext : Apache Camel 3.20.0-SNAPSHOT (pubsub-test) is starting +2022-12-14 12:04:55.339 INFO 91156 --- [ main] onent.google.pubsub.GooglePubsubConsumer : Starting Google PubSub consumer for gs-storage-cdc-test/gs-storage-cdc-test-sub +2022-12-14 12:04:55.378 INFO 91156 --- [ main] org.apache.camel.main.BaseMainSupport : Property-placeholders summary +2022-12-14 12:04:55.378 INFO 91156 --- [ main] org.apache.camel.main.BaseMainSupport : [rage-cdc-source.kamelet.yaml] projectId=gs-storage-cdc-test +2022-12-14 12:04:55.379 INFO 91156 --- [ main] org.apache.camel.main.BaseMainSupport : [rage-cdc-source.kamelet.yaml] subscriptionName=gs-storage-cdc-test-sub +2022-12-14 12:04:55.379 INFO 91156 --- [ main] org.apache.camel.main.BaseMainSupport : [rage-cdc-source.kamelet.yaml] serviceAccountKey=xxxxx +2022-12-14 12:04:55.379 INFO 91156 --- [ main] org.apache.camel.main.BaseMainSupport : [rage-cdc-source.kamelet.yaml] getObject=true +2022-12-14 12:04:55.379 INFO 91156 --- [ main] org.apache.camel.main.BaseMainSupport : [rage-cdc-source.kamelet.yaml] bucketNameOrArn=gs-storage-cdc-test +2022-12-14 12:04:55.379 INFO 91156 --- [ main] org.apache.camel.main.BaseMainSupport : [log-sink.kamelet.yaml] showHeaders=true +2022-12-14 12:04:55.379 INFO 91156 --- [ main] org.apache.camel.main.BaseMainSupport : [log-sink.kamelet.yaml] showStreams=true +2022-12-14 12:04:55.400 INFO 91156 --- [ main] e.camel.impl.engine.AbstractCamelContext : Routes startup (started:3) +2022-12-14 12:04:55.400 INFO 91156 --- [ main] e.camel.impl.engine.AbstractCamelContext : Started route1 (kamelet://google-storage-cdc-source) +2022-12-14 12:04:55.400 INFO 91156 --- [ main] e.camel.impl.engine.AbstractCamelContext : Started google-storage-cdc-source-1 (google-pubsub://gs-storage-cdc-test:gs-storage-cdc-test-sub) +2022-12-14 12:04:55.400 INFO 91156 --- [ main] e.camel.impl.engine.AbstractCamelContext : Started log-sink-2 (kamelet://source) +2022-12-14 12:04:55.400 INFO 91156 --- [ main] e.camel.impl.engine.AbstractCamelContext : Apache Camel 3.20.0-SNAPSHOT (pubsub-test) started in 1s799ms (build:98ms init:1s474ms start:227ms JVM-uptime:3s) +2022-12-14 12:06:04.398 INFO 91156 --- [ Gax-1] log-sink : Exchange[ExchangePattern: InOnly, Headers: {CamelGoogleCloudStorageBlobId=BlobId{bucket=gs-storage-cdc-test, name=timer-mongodb.yaml, generation=1671015960548591}, CamelGoogleCloudStorageCacheControl=null, CamelGoogleCloudStorageComponentCount=null, CamelGoogleCloudStorageContentDisposition=null, CamelGoogleCloudStorageContentEncoding=null, CamelGoogleCloudStorageContentLanguage=null, Cam [...] +nanos: 788000000 +}, BodyType: byte[], Body: ## ---------------------------------------------------------------------------## Licensed to the Apache Software Foundation (ASF) under one or more## contributor license agreements. See the NOTICE file distributed with## this work for additional information regarding copyright ownership.## The ASF licenses this file to You under the Apache License, Version 2.0## (the "License"); you may not use this file except in compliance with## the License. You may obtain [...] +2022-12-14 12:07:24.102 INFO 91156 --- [ Gax-1] log-sink : Exchange[ExchangePattern: InOnly, Headers: {CamelGooglePubsubAttributes={objectId=timer-mongodb.yaml, notificationConfig=projects/_/buckets/gs-storage-cdc-test/notificationConfigs/1, payloadFormat=JSON_API_V1, objectGeneration=1671015960548591, eventTime=2022-12-14T11:07:21.431184Z, bucketId=gs-storage-cdc-test, eventType=OBJECT_DELETE}, CamelGooglePubsubMessageId=6438927518282147, CamelG [...] +nanos: 554000000 +}, BodyType: byte[], Body: { "kind": "storage#object", "id": "gs-storage-cdc-test/timer-mongodb.yaml/1671015960548591", "selfLink": "https://www.googleapis.com/storage/v1/b/gs-storage-cdc-test/o/timer-mongodb.yaml", "name": "timer-mongodb.yaml", "bucket": "gs-storage-cdc-test", "generation": "1671015960548591", "metageneration": "1", "contentType": "application/octet-stream", "timeCreated": "2022-12-14T11:06:00.635Z", "updated": "2022-12-14T11:06:00.635Z", "storageClass": "STA [...] + +---- + +=== Cleanup Google Services + +You'll need to cleanup everything from Gcloud CLI. + +[source,sh] +---- +gcloud projects delete gs-storage-cdc-test +---- + +and answer yes. + +=== Help and contributions + +If you hit any problem using Camel or have some feedback, then please +https://camel.apache.org/community/support/[let us know]. + +We also love contributors, so +https://camel.apache.org/community/contributing/[get involved] :-) + +The Camel riders! diff --git a/jbang/google-storage-blob-cdc/google-storage-cdc.yaml b/jbang/google-storage-blob-cdc/google-storage-cdc.yaml new file mode 100644 index 0000000..84fca2e --- /dev/null +++ b/jbang/google-storage-blob-cdc/google-storage-cdc.yaml @@ -0,0 +1,32 @@ +## --------------------------------------------------------------------------- +## Licensed to the Apache Software Foundation (ASF) under one or more +## contributor license agreements. See the NOTICE file distributed with +## this work for additional information regarding copyright ownership. +## The ASF licenses this file to You under the Apache License, Version 2.0 +## (the "License"); you may not use this file except in compliance with +## the License. You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## --------------------------------------------------------------------------- + +- route: + from: + uri: "kamelet:google-storage-cdc-source" + parameters: + projectId: "gs-storage-cdc-test" + subscriptionName: "gs-storage-cdc-test-sub" + bucketNameOrArn: "gs-storage-cdc-test" + getObject: true + serviceAccountKey: <serviceAccountkey_base64> + steps: + - to: + uri: "kamelet:log-sink" + parameters: + showStreams: "true" + showHeaders: "true"