2019-03-16 11:00:38 UTC - 张琳: @张琳 has joined the channel ---- 2019-03-16 11:05:27 UTC - SuyambuGanesh: There is also an apache project called Ratis, which implements the RAFT protocol and it is available as library so that it can be used in any systems <https://github.com/apache/incubator-ratis> @Sijie Guo @Kev Jackson ---- 2019-03-16 11:32:56 UTC - Sijie Guo: @SuyambuGanesh I know one of the engineers who built ratis :slightly_smiling_face: slightly_smiling_face : SuyambuGanesh ---- 2019-03-16 11:33:24 UTC - Sijie Guo: you might also check out jraft (recently open sourced by Ant Financial) : <https://github.com/alipay/sofa-jraft> ---- 2019-03-16 14:00:46 UTC - Karthik Ramasamy: Streamlio launched managed pulsar service bananadance : Yuvaraj Loganathan, Shivji Kumar Jha, Ali Ahmed clap : Yuvaraj Loganathan, Ezequiel Lovelle, SuyambuGanesh, Ali Ahmed ---- 2019-03-16 14:00:54 UTC - Karthik Ramasamy: <https://siliconangle.com/2019/03/13/streamlio-debuts-cloud-native-version-apache-pulsar-based-pub-sub-platform/> ---- 2019-03-16 19:37:01 UTC - SuyambuGanesh: @Sijie Guo thanks ---- 2019-03-16 21:57:09 UTC - Jon Bock: @naga: As mentioned by @Karthik Ramasamy, Streamlio announced our managed service for Pulsar on AWS earlier this week: <https://streaml.io/product/cloud>
Availability expected early in Q2. +1 : Shivji Kumar Jha ---- 2019-03-16 23:09:55 UTC - JAYARAM NAGARAJAN: @David Kjerrumgaard yes only one value for managedLedgerOffloadDriver , i am following instrucitons in <http://pulsar.apache.org/docs/en/cookbooks-tiered-storage/> THis time i used a different namespace as previous one was giving me issues, this time it seems to have worked, but i am unable to see in s3 where the offload is written to... Can you please let me know how the s3 offloaded file names should look like? Where to find it in the bucket? 1) broker.conf: # Driver to use to offload old data to long term storage (Possible values: S3, aws-s3, google-cloud-storage) # When using google-cloud-storage, Make sure both Google Cloud Storage and Google Cloud Storage JSON API are enabled for # the project (check from Developers Console -> Api&auth -> APIs). managedLedgerOffloadDriver=aws-s3 # Maximum number of thread pool threads for ledger offloading managedLedgerOffloadMaxThreads=2 # For Amazon S3 ledger offload, AWS region s3ManagedLedgerOffloadRegion=us-east-1 # For Amazon S3 ledger offload, Bucket to place offloaded ledger into s3ManagedLedgerOffloadBucket=<our_bucket> 2) Able to access the s3 bucket with IAM role: aws s3 cp ./README s3://<our_bucket>/temp/pulsar/ --sse upload: ./README to <s3://cof-qa-coaf-lake/temp/pulsar/README> 3) Manually Triggered offload: bin/pulsar-admin topics offload --size-threshold 2M nest/testNamespace/sampleTopic Offload triggered for <persistent://nest/testNamespace/sampleTopic> for messages before 328:0:-1 18:05:41.516 [pulsar-web-55-3] INFO org.eclipse.jetty.server.RequestLog - 127.0.0.1 - - [16/Mar/2019:18:05:41 +0000] "GET /admin/v2/persistent/nest/testNamespace/sampleTopic/internalStats HTTP/1.1" 200 753 "-" "Jersey/2.27 (HttpUrlConnection 1.8.0_111)" 3 18:05:41.675 [pulsar-web-55-3] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [nest/testNamespace/persistent/sampleTopic] Start ledgersOffload. ledgers=[328] totalSize=28482873 18:05:41.675 [pulsar-web-55-3] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [nest/testNamespace/persistent/sampleTopic] No ledgers to offload 18:05:41.676 [pulsar-web-55-3] INFO org.eclipse.jetty.server.RequestLog - 127.0.0.1 - - [16/Mar/2019:18:05:41 +0000] "PUT /admin/v2/persistent/nest/testNamespace/sampleTopic/offload HTTP/1.1" 204 0 "-" "Jersey/2.27 (HttpUrlConnection 1.8.0_111)" 3 18:05:43.476 [pulsar-web-55-1] INFO org.eclipse.jetty.server.RequestLog - 127.0.0.1 - - [16/Mar/2019:18:05:43 +0000] "GET /admin/v2/persistent/public/functions/coordinate/stats HTTP/1.1" 200 824 "-" "Jersey/2.27 (HttpUrlConnection 1.8.0_111)" 4 [root@ip-10-207-192-140 bin]# ./pulsar-admin topics offload-status nest/testNamespace/sampleTopic Offload was a success ---- 2019-03-16 23:41:21 UTC - David Kjerrumgaard: @JAYARAM NAGARAJAN the output you shared shows "No ledgers to offload", so I am not sure if it worked. ---- 2019-03-16 23:42:34 UTC - David Kjerrumgaard: @JAYARAM NAGARAJAN I think that since you are running the broker in standalone mode, you need to add the tiered storage properties to the to the `conf/standalone.conf` file, as described in Step 2 here: <https://streaml.io/blog/configuring-apache-pulsar-tiered-storage-with-amazon-s3> ---- 2019-03-16 23:50:38 UTC - David Kjerrumgaard: offloaded segments in S3 appear as follows ---- 2019-03-17 00:17:10 UTC - Alexandre DUVAL: Any news? Good week-end :stuck_out_tongue:. ---- 2019-03-17 03:12:27 UTC - naga: @Karthik Ramasamy thanks for sharing. ---- 2019-03-17 04:04:21 UTC - JAYARAM NAGARAJAN: @David Kjerrumgaard Thanks for sharing this David, this time i followed the streamlio blog exactly you shared.. this time getting new error. ---- 2019-03-17 04:08:57 UTC - David Kjerrumgaard: @JAYARAM NAGARAJAN Actually it is the same error as before, the managed ledger offloader property isn't getting read, which leads to you using the NullLedgerOffloader. Line 99: "org.apache.bookkeeper.mledger.impl.NullLedgerOffloader.offload(NullLedgerOffloader.java:44) ~[org.apache.pulsar-managed-ledger-original-2.3.0.jar:2.3.0]" ---- 2019-03-17 04:09:32 UTC - David Kjerrumgaard: @JAYARAM NAGARAJAN Can you try using "s3" instead of "aws-s3" for that property? ---- 2019-03-17 04:25:39 UTC - JAYARAM NAGARAJAN: @David Kjerrumgaard yes tried with all possible "s3", "S3", and "aws-s3" Still getting the same error ---- 2019-03-17 04:27:29 UTC - David Kjerrumgaard: @JAYARAM NAGARAJAN and you tried setting these values in both the `broker.conf` and `standalone.conf` files? ---- 2019-03-17 04:33:35 UTC - JAYARAM NAGARAJAN: @David Kjerrumgaard Yes followed the same in both files.. ---- 2019-03-17 04:52:13 UTC - David Kjerrumgaard: @JAYARAM NAGARAJAN Can you check the value of your "OffloadersDirectory" property in your broker config file? and verify that the directory exists on your broker node, is readable by the user you are running pulsar as, etc? ---- 2019-03-17 04:58:04 UTC - JAYARAM NAGARAJAN: Yes it seems to have correct info, also all directories have 755 permissions ----
