2020-08-26 14:03:17 UTC - Mark Marijnissen: *Schema compatibility* I don't understand Forward & Backward. What happen with the `?` below?
```FORWARD_TRANSITIVE: P1 P2 P3 C1 ? ? Y C2 ? ? Y C3 ? ? Y BACKWARD_TRANSITIVE: P1 P2 P3 C1 ? ? ? C2 ? ? ? C3 Y Y Y Where P1,P2,P3 = producer v1, v2, v3 and C1,C2,C3 = consumer v1, v2, v3``` ---- 2020-08-26 14:03:58 UTC - Dmitri Zamysloff: @Dmitri Zamysloff has joined the channel ---- 2020-08-26 14:43:48 UTC - Alexei Grigoriev: @Alexei Grigoriev has joined the channel ---- 2020-08-26 15:04:04 UTC - Evan Furman: Hi! Is there documentation on getting pulsar logs into grafana via loki in non-k8s environments? We are running on EC2. ---- 2020-08-26 15:27:27 UTC - Dmitri Zamysloff: Hi All, I am pretty new to Pulsar. Started testing 2.6.1 as possible basis for CQRS implementation. Have written small producer and consumer in java (two spring-boot applications). 1. What I cannot understand why presto does not see date (or instant) fields? I have tried with JSON schema and enabled JSR310 in schema definition. Messages are comming by consumer consistently, but in selects of presto I cannot see date/time fields. Also over JDBC presto driver they are not visible. 2. Another point is that I cannot see last message over presto/presto-JDBC. Is there any way to see it? 3. As I red the description of Pulsar I was like WOW and I thought Pulsar can take over creation of Aggregates and event registry. Like I will be able to see somehow aggregated objects. And on other hand is event storage (i can only see possibility to set retention to some size) which I can then export and replay. But I have found, that over presto I can just see messages in topic and retention can be limited for namespace over admin api, which I have done. So is pulsar only a smart messaging platform or it can do Aggregates of domain objects out of the box? Or I have to build the aggregated states of domain objects myself in some separate DB like Mongo? Shall I also take care of event registry myself? ---- 2020-08-26 15:28:13 UTC - Dmitri Zamysloff: Or maybe there any blueprint for cqrs implementation with pulsar? ---- 2020-08-26 15:28:28 UTC - Dmitri Zamysloff: Thank you in advance. ---- 2020-08-26 15:49:30 UTC - Addison Higham: 1. here is a mapping of how pulsar maps to schemas types: <https://pulsar.apache.org/docs/en/schema-understand/#primitive-type> but can you provide an example schema definition? 2. This is a known issue with a workaround, see this thread for details: <https://github.com/apache/pulsar/issues/4976>. Setting `bookkeeperExplicitLacIntervalInMills` in broker.conf to something greater than 0 (maybe like 5-50ms) will cause this last message to be available for pulsar SQL 3. I am not sure I completely understand your question, but it doesn't automatically *create* aggregates, but it still maybe a reasonable place to track, send, and store aggregates For example, suppose you have multiple different types of events and you transform those with a flink application or a pulsar functions window function. You can create a new schema representing the aggregate event, write the results to a topic, use compaction to keep only the most recent projection of that aggregation, and then use Pulsar SQL to query for aggregates for analytical use cases, or use a pulsar sink to a JDBC/other database for the query layer ---- 2020-08-26 15:52:28 UTC - Chris Hansen: I think Apache Flink can be used for this on top of Pulsar to keep track of the batch state. Something to look into. ---- 2020-08-26 18:15:10 UTC - Matt Mitchell: what would cause this producer problem? A slow consumer? ```org.apache.pulsar.client.api.PulsarClientException$TimeoutException: The producer my-producer can not send message to the topic <persistent://public/default/my-topic> within given timeout``` ---- 2020-08-26 18:15:24 UTC - Matt Mitchell: Or is that an issue with the broker? ---- 2020-08-26 19:17:05 UTC - Erik Jansen: I have started a standalone cluster and produced several millions of messages to different topics. No retention set. After a while the topics are removed due to inactivity. When I look into the data directory there are however still some huge files ( 3 of them with total size of about 6Gbyte) with the extension: .txn. Can someone explain why these files still exist and what they are used for? ---- 2020-08-26 19:29:25 UTC - Josh Janssen: @Josh Janssen has joined the channel ---- 2020-08-26 23:09:18 UTC - Alexander Ursu: Hi, I recently started using the Postgres JDBC sink connector, and am having troubles with `ERROR: permission denied for table ...` , even after trying to grant all privileges to the user on the schema, table, etc. For anyone that's used it before, is there some standard setup to permissions for the built-in sink connector that has worked for you? Also to note that this is on 2.6.1 ---- 2020-08-27 03:48:29 UTC - Maksym Domarev: Is there a way to TTL topic, e.g. automatically delete it if it’s not in use for let’s day 2 days? ---- 2020-08-27 05:04:54 UTC - Raghav: Not at topic level but I think there are some settings at broker level for this ---- 2020-08-27 08:10:42 UTC - aloyszhang: TTL can be set at namespace level using Admin tool. Like `bin/pulsar-admin namespaces set-message-ttl tenant/namespace -ttl timetInSeconds` ---- 2020-08-27 08:12:02 UTC - aloyszhang: see <https://pulsar.apache.org/docs/en/2.6.1/cookbooks-retention-expiry/#set-the-ttl-for-a-namespace|here> ----
