2019-03-31 10:23:30 UTC - David Tinker: The producer could use the Pulsar admin interface to check on the backlog from time to time. Its also possible to check message rates for each subscription. ---- 2019-03-31 11:09:43 UTC - Ben S: I think TTL won‘t work. I need the most recent message to be persisted on the server where the current state is shared between failover instances. It should be persisted in case the whole cluster goes offline for maintenance or else. Using a TTL for the client would work. ---- 2019-03-31 12:00:26 UTC - m.makaveeva: Hi everyone, I am trying to analyze the possibilities to use Pulsar for offline communication. What I mean is, I imagine a scenario in which a producer publishes messages then goes offline, then a consumer joins and subscribes to the topic. From what I understood in the documentation, as long as the produced messages have not been acknowledged they will be stored in a managed ledger and dispatched from there. Unfortunately, just using a standard setup, I cannot reproduce this situation. If the consumer and producer were not online at the same time, the messages don't reach the consumer. Do you know if this feature is supported? Also, could someone tell me how to enable it?
Currently I am running bin/pulsar standalone and a regular command line: pulsar-client produce/consume topic options Thanks in advance for your help! ---- 2019-03-31 12:25:56 UTC - jia zhai: @m.makaveeva By default, consumer is consume from the latest position in ledger. So it will not receive the messages before the time it is created. In other client, like java, there is a parameter to set SubscriptionInitialPosition, it could be set as earliest/latest. ---- 2019-03-31 13:24:00 UTC - m.makaveeva: @jia zhai Thanks a lot, I just tried it and works perfect! ---- 2019-03-31 13:24:28 UTC - jia zhai: you are welcome ---- 2019-03-31 13:36:48 UTC - m.makaveeva: Sorry, do you also know if we could tweak the system so that all messages are sent to all consumers (and not round robin)? ---- 2019-03-31 13:37:44 UTC - jia zhai: by default, it is share mode subscribe. ---- 2019-03-31 13:37:58 UTC - jia zhai: If you want each consumer get all message ---- 2019-03-31 13:38:36 UTC - jia zhai: You could try each subscribe has only one consumer ---- 2019-03-31 13:39:49 UTC - jia zhai: one topic could have several subscribes, each of subscribe could get all messages ---- 2019-03-31 13:44:47 UTC - m.makaveeva: ok, thank you :slightly_smiling_face: ---- 2019-03-31 14:19:44 UTC - Ben L: Thanks for the detailed responses! It does look like I'm going to have to use some degree of multiplexing to reach the number of 'topics' I need. ---- 2019-03-31 21:15:16 UTC - Remco Verhoef: Hi Sree, thanks for your invite. I found Pulsar when I was looking for a queuing alternative that would allow us to create thousands or millions of queues. Very happy to have found Pulsar, and we build a streaming data platform (named Raven) on top of Pulsar, Docker and some other technologies. This platform manages data flows (companies have many of them, like security companies of industry and most of them are ad-hoc), and automatically scales containers to process. Data processing can be enrichment, filtering or machine learning models algorithms. Because the usage of Docker and containers we’re language agnostic, machine learning models can be run in Python, as other transformations in different languages. We’re not yet complete sure about licensing, as combining opensource and commercial licenses is causing lots of discussions. +1 : Karthik Ramasamy ---- 2019-03-31 21:40:32 UTC - Ali Ahmed: @Remco Verhoef hav you also used or looked at pulsar functions ? ---- 2019-04-01 01:38:42 UTC - Sree Vaddi: Thank you for the great introduction, @Remco Verhoef. It is a great effort and I personally, highly appreciate your effort. Feel free to talk to us for the pricing model. @Karthik Ramasamy @Matteo Merli . ---- 2019-04-01 01:40:03 UTC - Sree Vaddi: <https://twitter.com/arun_kejariwal/status/1110660631890321409?s=21> ---- 2019-04-01 01:40:19 UTC - Sree Vaddi: full house :clap: +1 : Ali Ahmed ---- 2019-04-01 03:53:49 UTC - Samuel Sun: :+1: ---- 2019-04-01 04:47:51 UTC - Yuvaraj Loganathan: <https://pulsar.apache.org/docs/en/cookbooks-retention-expiry/#set-retention-policy> ---- 2019-04-01 05:49:56 UTC - bhagesharora: Hello everyone, /apache-pulsar-2.3.0$ ./bin/pulsar-admin source create --tenant test-tenant --namespace test-namespace --name generator --destinationTopicName generator_test --source-type data-generator Tenant does not exist Reason: Tenant does not exist I an getting follwing error "Tenant does not exist", when trying to execute source command In pulsar SQL steps following ref url - <https://pulsar.apache.org/docs/en/sql-getting-started/> what would be the appropriate solution ? ---- 2019-04-01 05:53:00 UTC - Ali Ahmed: @bhagesharora you need to create the tenant ---- 2019-04-01 05:56:26 UTC - bhagesharora: @Ali Ahmed Is there any command to create a tenant or any reference URL, which can help me ?? ---- 2019-04-01 05:57:17 UTC - Ali Ahmed: <https://pulsar.apache.org/docs/latest/admin-api/tenants/> ---- 2019-04-01 06:07:54 UTC - bhagesharora: @Ali Ahmed Thanks for your help, I am able to create tenant, see below. b..................@pulsar-setup-2-3-0-n1-standalone:~/apache-pulsar-2.3.0$ bin/pulsar-admin tenants list public sample test-tenant b..................@pulsar-setup-2-3-0-n1-standalone:~/apache-pulsar-2.3.0$ ./bin/pulsar-admin source create --tenant test-tenant --namespace test-namespace --name generator --destinationTopicName generator_test --source-type data-generator null Reason: javax.ws.rs.ProcessingException: java.net.SocketTimeoutException: Read timed out But in next step again, some "SocketTimeoutException" is coming ---- 2019-04-01 07:24:30 UTC - bhagesharora: @Sijie Guo I am following same document, I am able to create a Test.java in my machine, and its successfully executed. ---- 2019-04-01 07:24:50 UTC - bhagesharora: bhagesharora93@pulsar-setup-2-3-0-n1-standalone:~/apache-pulsar-2.3.0/pulsar_sql_code/target$ java -cp pulsar_sql_code-0.0.2-SNAPSHOT-jar-with-dependencies.jar pulsar_sql_code.Test [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConnectionPool - [[id: 0xde7b09e5, L:/127.0.0.1:39630 - R:localhost/127.0.0.1:6650]] Connected to server [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ProducerStatsRecorderImpl - Starting Pulsar producer perf with config: { "topicName" : "test_topic", "producerName" : null, "sendTimeoutMs" : 30000, "blockIfQueueFull" : false, "maxPendingMessages" : 1000, "maxPendingMessagesAcrossPartitions" : 50000, "messageRoutingMode" : "RoundRobinPartition", "hashingScheme" : "JavaStringHash", "cryptoFailureAction" : "FAIL", "batchingMaxPublishDelayMicros" : 1000, "batchingMaxMessages" : 1000, "batchingEnabled" : true, "compressionType" : "NONE", "initialSequenceId" : null, "autoUpdatePartitions" : true, "properties" : { } } [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ProducerStatsRecorderImpl - Pulsar client config: { "serviceUrl" : "<pulsar://localhost:6650>", "operationTimeoutMs" : 30000, "statsIntervalSeconds" : 60, "numIoThreads" : 1, "numListenerThreads" : 1, "connectionsPerBroker" : 1, "useTcpNoDelay" : true, "useTls" : false, "tlsTrustCertsFilePath" : "", ---- 2019-04-01 07:24:52 UTC - bhagesharora: "tlsHostnameVerificationEnable" : false, "concurrentLookupRequest" : 5000, "maxLookupRequest" : 50000, "maxNumberOfRejectedRequestPerConnection" : 50, "keepAliveIntervalSeconds" : 30, "connectionTimeoutMs" : 10000 } [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConnectionPool - [[id: 0x934758e9, L:/127.0.0.1:39632 - R:localhost/127.0.0.1:6650]] Connected to server [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ClientCnx - [id: 0x934758e9, L:/127.0.0.1:39632 - R:localhost/127.0.0.1:6650] Connected through proxy to target broker at pulsar-setup-2-3-0-n1-standalone.us-east1-b.c.secret-envoy-226512.internal:6650 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ProducerImpl - [test_topic] [null] Creating producer on cnx [id: 0x934758e9, L:/127.0.0.1:39632 - R:localhost/127.0.0.1:6650] [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ProducerImpl - [test_topic] [standalone-5-2] Created producer on cnx [id: 0x934758e9, L:/127.0.0.1:39632 - R:localhost/127.0.0.1:6650] [pulsar-timer-4-1] INFO com.scurrilous.circe.checksum.Crc32cIntChecksum - SSE4.2 CRC32C provider initialized [main] INFO org.apache.pulsar.client.impl.ProducerImpl - [test_topic] [standalone-5-2] Closed Producer [main] INFO org.apache.pulsar.client.impl.PulsarClientImpl - Client closing. URL: <pulsar://localhost:6650> [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ClientCnx - [id: 0xde7b09e5, L:/127.0.0.1:39630 ! R:localhost/127.0.0.1:6650] Disconnected [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ClientCnx - [id: 0x934758e9, L:/127.0.0.1:39632 ! R:localhost/127.0.0.1:6650] Disconnected ---- 2019-04-01 07:27:06 UTC - bhagesharora: @Sijie Guo But in document mentioned like.. Afterwards, you should be able query the data you just wrote. so when I am trying to see the schema and table nothing is coming, so any step I missed or what wouls be the appropriate way?? reference URL - <https://pulsar.apache.org/docs/en/sql-getting-started/> ----
