2020-08-06 12:07:42 UTC - Alan Hoffmeister: is pulsar recommended as a
permanent event storage? I'm trying to setup some event sourcing structure but
I would like to avoid storing my event log inside mysql
----
2020-08-06 14:31:18 UTC - Pramiti Guha: @Pramiti Guha has joined the channel
----
2020-08-06 15:07:03 UTC - David Kjerrumgaard: @Alan Hoffmeister I would
definitely say that Pulsar is a good candidate for permanent event storage,
particularly with its tiered-storage capabilities.
+1 : Alan Hoffmeister, Rich Adams, Chris Hansen
----
2020-08-06 15:09:45 UTC - David Kjerrumgaard: If you have a scenario where you
have older noBK nodes re-joining the cluster, then you will end up with some
over-replicated segments (which isn't bad) and more storage capacity. I can't
envision a scenario where this would result in unwanted behavior. Are you
experiencing any?
----
2020-08-06 15:20:22 UTC - Feanil: Hey all, I'm trying to wrap my head around
pulsar schema evolution with JSON schemas(in python). I can't seem to find
good docs on what the format of the schema is, that is actually sent to the
pulsar registry. Is it an implementation of the `json-schema` spec? Is any
JSON schema is valid? or is it something more restrictive? In particular it
seems like the python bindings don't seem to generate the schema with a default
value, is this on purpose?
----
2020-08-06 15:32:30 UTC - Bre Gielissen: Quick question: can you clarify the
error "Cannot delete non empty namespace"? I was assuming that it meant that
there were topics underneath but that either means the documentation is wrong
in your REST API documentation or I actually need to do something else to truly
"empty" the namespace. Thanks!
----
2020-08-06 16:03:01 UTC - David Kjerrumgaard: @Feanil I spent a lot of time
focusing on Schemas in my upcoming book about Pulsar because i found the public
documentation to be lacking. I gave an example using Avro, and found it to be
quite easy to use. There is also avro support for Python
----
2020-08-06 16:22:54 UTC - Addison Higham: are you using async or sync APIs? for
performance, the biggest thing you can do first is make sure you are using
async producer and consumer and to enable batching
+1 : charles
----
2020-08-06 16:23:03 UTC - Addison Higham: @charles ^^
----
2020-08-06 16:24:48 UTC - Alan Broddle: Has anyone tried to use the Prometheus
metrics in DataDog? Our corporate preferred monitoring environment is DataDog.
We have enabled the DataDog Zookeeper and DataDog OpenMetrics integrations
from our Pulsar cluster, and the data is seen by DataDog. The issue is DataDog
does not ‘like’ the complexity of the metrics Prometheus/Pulsar creates. DD
does not understand metrics that are formatted with comma separated values in
the metric conditionals.
pulsar_rate_in{cluster=“Test”,namespace=“public/default”,topic=“<persistent://public/default/my-topic2>”}
99.999 1596728053329
More details if needed…
Basically anted to know if anyone has tried this yet?
----
2020-08-06 16:29:28 UTC - Addison Higham: @Alan Broddle I have used datadog in
the past, datadog with the openmetrics integration should do fine with the
labels. What we had to do though was add annotations for metrics that don't
have type metadata. In most promotheus scrapers, if a metric doesn't have type
info, it just gets interpreted as a gauge, but datadog is a bit more picky and
only will consumer metrics that it has type info for, either by the type
annotations provided by the endpoint OR if you add your own annotations, as an
example:
```<http://ad.datadoghq.com/broker.instances|ad.datadoghq.com/broker.instances>:
'[{"prometheus_url":"<http://%%host%>%:8080/metrics","namespace":"pulsar","metrics":["*"],"type_overrides":{"pulsar_topics_count":"gauge","pulsar_subscriptions_count":"gauge"}}'```
----
2020-08-06 16:30:04 UTC - Addison Higham: :thinking_face: I don't think that
doc is correct
----
2020-08-06 16:51:09 UTC - Vil: Ok, thanks for your reply @Addison Higham. Not
sure I understand everything in your reply (what to do with acks etc.), so I
use my own words to summarize what you said. Let me know if that’s correct.
1. I can attempt to make effectively once work with pulsar functions but it
requires extra work on my end (and here I do not know exactly what I would have
to do).
2. Better option is to not use pulsar functions for effectively once but
instead use flink.
Is this a good summary?
If it is my question is are there any plans to make pulsar functions ‘just
work’ for effectively once? I mean, roadmap?
----
2020-08-06 16:53:44 UTC - Addison Higham: for sources/sinks particularity,
pulsar functions by themselves, when you use just a simple input and output
topic already "just work"
----
2020-08-06 16:55:29 UTC - Vil: to use an example: input topic > pulsar
function > output topic, would just work? and if yes (nice!) I do not yet
understand in which case it would not work :slightly_smiling_face:
----
2020-08-06 16:56:06 UTC - Vil: or do you mean input topic > pulsar functions
work, and pulsar function > output topic works, but not the full line of
input > pulsar function > output ?
----
2020-08-06 16:56:47 UTC - Addison Higham: the first one
----
2020-08-06 16:58:04 UTC - Addison Higham: So yes, with a simple input >
function > output, you get effectively once. But, for that, they can't
handle as much throughput, as we can't use batch any other techniques and must
ack per message
----
2020-08-06 17:29:11 UTC - Brandon Whitehead: can someone help me actually send
a message on the localhost:6650?
----
2020-08-06 17:29:33 UTC - Brandon Whitehead: I'm trying to make a go program
that sends a simple message on the actual localhost
----
2020-08-06 17:29:56 UTC - Brandon Whitehead: but despite it seeming to work in
the backend, I'm not sure how to send a message on the frontend
----
2020-08-06 17:30:32 UTC - Feanil: @Brandon Whitehead Are you following the
client
example(<https://pulsar.apache.org/docs/en/next/client-libraries-go/#producers>)
or something else?
----
2020-08-06 17:31:43 UTC - Brandon Whitehead: I am
----
2020-08-06 17:32:32 UTC - Feanil: Do you see any output, the python client is
pretty verbose, I imagine the Go one is similar
----
2020-08-06 17:32:44 UTC - Brandon Whitehead: this is the code I have
----
2020-08-06 17:32:53 UTC - Brandon Whitehead: ```package main;
import (
"log"
"fmt"
"time"
"context"
"<http://github.com/apache/pulsar-client-go/pulsar|github.com/apache/pulsar-client-go/pulsar>"
)
func main() {
client, err := pulsar.NewClient(pulsar.ClientOptions{
URL: "<pulsar://localhost:6650>",
OperationTimeout: 60 * time.Second,
ConnectionTimeout: 60 * time.Second,
})
if err != nil {
log.Fatalf("Could not instantiate Pulsar client: %v", err)
} else {
log.Printf("Success!")
}
defer client.Close()
producer, err1 := client.CreateProducer(pulsar.ProducerOptions{
Topic: "my-topic",
Name: "test-producer",
})
if err1 != nil {
log.Fatalf("Could not connect to client: %v", err1)
} else {
log.Printf("Success!")
}
_, err2 := producer.Send(context.Background(), &pulsar.ProducerMessage{
Payload: []byte("hello"),
})
defer producer.Close()
if err2 != nil {
fmt.Println("Failed to publish message", err2)
} else {
log.Printf("Success!")
}
fmt.Println("Published message")
consumer, err3 := client.Subscribe(pulsar.ConsumerOptions{
Topic: "topic-1",
SubscriptionName: "my-sub",
Type: pulsar.Shared,
})
if err3 != nil {
log.Fatal(err3)
}
defer consumer.Close()
for i := 0; i < 10; i++ {
msg, err4 := consumer.Receive(context.Background())
if err4 != nil {
log.Fatal(err4)
} else {
log.Printf("Success!")
}
log.Printf("Received message msgId: %#v -- content: '%s'\n",
msg.ID(), string(msg.Payload()))
consumer.Ack(msg)
}
if err5 := consumer.Unsubscribe(); err5 != nil {
log.Fatal(err5)
}
}```
----
2020-08-06 17:32:59 UTC - Feanil: Also, have you confirmed that the pulsar
server is accepting connections?
----
2020-08-06 17:33:18 UTC - Brandon Whitehead: from what I can see, the go
program connected
----
2020-08-06 17:33:22 UTC - Feanil: `telnet localhost 6650` and seeing if that
successfully establishes a connection
----
2020-08-06 17:33:31 UTC - Brandon Whitehead: where do I enter that?
----
2020-08-06 17:33:36 UTC - Brandon Whitehead: I'm using docker
----
2020-08-06 17:33:37 UTC - Vil: ok, thank you again @Addison Higham
----
2020-08-06 17:34:45 UTC - Feanil: did you bind the port to your localhost? How
are you using docker(with docker-compose, or manually bringing up multiple
containers?)
----
2020-08-06 17:35:04 UTC - Brandon Whitehead: I'm just using the tutorial on the
website
----
2020-08-06 17:35:26 UTC - Brandon Whitehead: so if I got the terminology right,
I think it's only one container?
----
2020-08-06 17:36:22 UTC - Bre Gielissen: I was wondering. That is unfortunate.
That would make it so much easier.
----
2020-08-06 17:36:28 UTC - Brandon Whitehead:
<https://pulsar.apache.org/docs/en/next/standalone-docker/>
----
2020-08-06 17:43:03 UTC - Brandon Whitehead: so in my go program, I make a
client, then a producer
----
2020-08-06 17:43:25 UTC - Brandon Whitehead: how do I get the localhost to be a
consumer so they get the producer's message?
----
2020-08-06 17:44:16 UTC - Feanil: gotcha so you should be able to run that
telnet command on your local machine in a shell
----
2020-08-06 17:44:24 UTC - Feanil: you shouldn't need to run it in the container
----
2020-08-06 17:44:47 UTC - Brandon Whitehead: the telnet command doesn't work
----
2020-08-06 17:44:51 UTC - Brandon Whitehead: in powershell
----
2020-08-06 17:45:03 UTC - Brandon Whitehead: doesn't recognize the telnet
command at all
----
2020-08-06 17:45:19 UTC - Feanil: @David Kjerrumgaard do you know what the
format of the schema you upload is supposed to be? I'm trying to upload the
schema but it's not exactly clear what the format of the file is supposed to be
----
2020-08-06 17:45:49 UTC - Ebere Abanonu: @Brandon Whitehead if the pulsar is
running and you cannot connect then try 127.0.0.1
----
2020-08-06 17:46:20 UTC - Feanil: ah didn't realize you were on windows
----
2020-08-06 17:46:24 UTC - Brandon Whitehead: I think pulsar is running though
----
2020-08-06 17:46:29 UTC - Brandon Whitehead: docker says it connected
----
2020-08-06 17:46:39 UTC - Feanil: try:
```Test-NetConnection -ComputerName localhost -Port 6650```
----
2020-08-06 17:46:44 UTC - Brandon Whitehead: and the /metrics page for the
localhost:8080 says it did too
----
2020-08-06 17:47:45 UTC - Brandon Whitehead: where do I figure out my
computername?
----
2020-08-06 17:48:07 UTC - Feanil: it's just localhost `-ComputerName` is the
name of the parameter
----
2020-08-06 17:48:21 UTC - Brandon Whitehead: whoops
----
2020-08-06 17:48:33 UTC - Feanil: you can also try `127.0.0.1` if localhost
doesn't connect
----
2020-08-06 17:48:46 UTC - Brandon Whitehead: okay I got this
----
2020-08-06 17:48:58 UTC - Brandon Whitehead: ```ComputerName : localhost
RemoteAddress : ::1
RemotePort : 6650
InterfaceAlias : Loopback Pseudo-Interface 1
SourceAddress : ::1
TcpTestSucceeded : True```
----
2020-08-06 17:49:20 UTC - Brandon Whitehead: what exactly does this mean?
----
2020-08-06 17:49:33 UTC - Feanil: ok, so it looks like connection is fine, in
that case maybe start with the simple example code from the client docs
----
2020-08-06 17:49:59 UTC - Brandon Whitehead: and what exactly do I use to send
messages to that localhost so I can actually see them on my computer when I
reach the page?
----
2020-08-06 17:50:28 UTC - Brandon Whitehead: I'm new with this whole pulsar
thing and I'm looking through the architecture and it's confusing me about what
actually happens
----
2020-08-06 17:50:54 UTC - Brandon Whitehead: so I make a client in my go
program that is tied to the localhost, then what?
----
2020-08-06 17:51:16 UTC - Brandon Whitehead: do I make a producer to make a
message, and then make a consumer to get that message? or do I use a reader?
----
2020-08-06 17:52:19 UTC - Brandon Whitehead: cause the go client tutorial is
not clear about what order this code needs to be in, or if it needs to be
multiple different files running concurrently
----
2020-08-06 17:52:20 UTC - Feanil: you need a producer in order to make the
messages and then you can consume the messages with a consumer
----
2020-08-06 17:52:40 UTC - Brandon Whitehead: and when I consume the messages,
shouldn't they show up on the localhost?
----
2020-08-06 17:53:04 UTC - Feanil: what do you mean by "show up" on localhost?
----
2020-08-06 17:53:37 UTC - Brandon Whitehead: like when I actually go to the
actual page on my browser, currently nothing shows up
----
2020-08-06 17:53:39 UTC - Feanil: producers put messages into pulsar, and then
consumers take messages out, once the consumer takes the message out, they can
print it to screen or proccess it in some other way
----
2020-08-06 17:53:50 UTC - Feanil: which page are you going to?
----
2020-08-06 17:54:12 UTC - Brandon Whitehead: `<http://localhost:6650/>`
----
2020-08-06 17:54:32 UTC - Feanil: right, that's not an HTTP endpoint, that's
the endpoint your producer/consumer talk to
----
2020-08-06 17:54:44 UTC - Brandon Whitehead: okay?
----
2020-08-06 17:54:50 UTC - Brandon Whitehead: what does that mean exactly?
----
2020-08-06 17:54:56 UTC - Feanil: if you want to get a message back out of
pulsar, you need a consumer process
----
2020-08-06 17:55:16 UTC - Brandon Whitehead: well I made a consumer in my go
program like the tutorial said
----
2020-08-06 17:55:42 UTC - Brandon Whitehead: and it's supposed to receive the
message from the topic the producer made, and then display it
----
2020-08-06 17:55:50 UTC - Brandon Whitehead: at least if I'm reading the code
right
----
2020-08-06 17:56:26 UTC - Brandon Whitehead: here is the code
----
2020-08-06 17:56:30 UTC - Brandon Whitehead: ```package main;
import (
"log"
"fmt"
"time"
"context"
"<http://github.com/apache/pulsar-client-go/pulsar|github.com/apache/pulsar-client-go/pulsar>"
)
func main() {
client, err := pulsar.NewClient(pulsar.ClientOptions{
URL: "<pulsar://localhost:6650>",
OperationTimeout: 60 * time.Second,
ConnectionTimeout: 60 * time.Second,
})
if err != nil {
log.Fatalf("Could not instantiate Pulsar client: %v", err)
} else {
log.Printf("Success!")
}
defer client.Close()
producer, err1 := client.CreateProducer(pulsar.ProducerOptions{
Topic: "my-topic",
Name: "test-producer",
})
if err1 != nil {
log.Fatalf("Could not connect to client: %v", err1)
} else {
log.Printf("Success!")
}
_, err2 := producer.Send(context.Background(), &pulsar.ProducerMessage{
Payload: []byte("hello"),
})
defer producer.Close()
if err2 != nil {
fmt.Println("Failed to publish message", err2)
} else {
log.Printf("Success!")
}
fmt.Println("Published message")
consumer, err3 := client.Subscribe(pulsar.ConsumerOptions{
Topic: "my-topic",
SubscriptionName: "my-sub",
Type: pulsar.Shared,
})
if err3 != nil {
log.Fatal(err3)
}
defer consumer.Close()
for i := 0; i < 10; i++ {
msg, err4 := consumer.Receive(context.Background())
if err4 != nil {
log.Fatal(err4)
} else {
log.Printf("Success!")
}
fmt.Printf("Received message msgId: %#v -- content: '%s'\n",
msg.ID(), string(msg.Payload()))
consumer.Ack(msg)
}
if err5 := consumer.Unsubscribe(); err5 != nil {
log.Fatal(err5)
}
}```
----
2020-08-06 17:56:30 UTC - Feanil: it seems like you don't have a clear
undrestanding of TCP vs HTTP, is that right?
----
2020-08-06 17:56:39 UTC - Brandon Whitehead: I guess not
----
2020-08-06 17:56:54 UTC - Brandon Whitehead: I'm sorry, but I'm really lost
----
2020-08-06 17:57:08 UTC - Feanil:
<https://www.extrahop.com/company/blog/2018/tcp-vs-http-differences-explained/>
----
2020-08-06 17:57:16 UTC - Brandon Whitehead: all I want is to show a simply
text line saying "hello" on the localhost
----
2020-08-06 17:57:45 UTC - Feanil: if you're trying to understand how the
messaging system works you'll probably want a good handle on the basics of how
the communication is happening
----
2020-08-06 17:58:09 UTC - Feanil: in general though, message buses are not
super useful if you are in the same process
----
2020-08-06 17:58:19 UTC - Feanil: they become more useful when trying to
communicate across processes
----
2020-08-06 17:59:11 UTC - Brandon Whitehead: so TCP is about sending out data,
and HTTP is about reading that data?
----
2020-08-06 17:59:27 UTC - Brandon Whitehead: is that the basic gist?
----
2020-08-06 18:01:03 UTC - Brandon Whitehead: if that's the case, then the
localhost should be trying to receive and read the data from my go program
right?
----
2020-08-06 18:22:16 UTC - Evan Furman: Does anyone have prometheus/grafana
configs they can share that are NOT for kubernetes? Looking for an example of
how to setup configs running on ec2
----
2020-08-06 18:26:16 UTC - Prashanth Tirupachur Vasanthakrishnan:
<http://pulsar.apache.org/docs/en/reference-metrics/#monitor> The
“apache-pulsar-grafana-dashboard” link has grafana dashboards for k8 &
on-prem.
----
2020-08-06 18:26:55 UTC - Brandon Whitehead: so I haven't really gotten any
closer to getting a message on the localhost that I can see on the browser
----
2020-08-06 18:27:42 UTC - Brandon Whitehead: is there something, I should be
doing within the go program?
----
2020-08-06 18:28:23 UTC - Brandon Whitehead: the topic I created with the
producer within the go program works fine
----
2020-08-06 18:28:59 UTC - Brandon Whitehead: and the consumer says it's
received the message via the powershell log, but I'm not sure how to translate
that Payload from the Producer onto the screen
----
2020-08-06 18:42:55 UTC - David Kjerrumgaard: There is a tool for manually
downloading and uploading schemas directly to Pulsar, along with some examples
here. <https://pulsar.apache.org/docs/en/schema-manage/#upload-a-schema>
----
2020-08-06 18:49:58 UTC - Evan Furman: Thanks @Prashanth Tirupachur
Vasanthakrishnan! I got it going but seeing some really odd stuff. I have 4
bookies…
----
2020-08-06 19:02:57 UTC - Addison Higham: I agree, I think there was some
discussion on making an API for that...
----
2020-08-06 19:16:16 UTC - Rattanjot Singh: ```19:15:22.336 [BookieJournal-3181]
ERROR org.apache.bookkeeper.bookie.BookieCriticalThread - Uncaught exception in
thread BookieJournal-3181 and is exiting!
io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216
byte(s) of direct memory (used: 4294967296, max: 4294967296)
at
io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:726)
~[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at
io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:681)
~[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at
io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:758)
~[io.netty-netty-buffer-4.1.43.Final.jar:4.1.43.Final]
at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:734)
~[io.netty-netty-buffer-4.1.43.Final.jar:4.1.43.Final]
at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:245)
~[io.netty-netty-buffer-4.1.43.Final.jar:4.1.43.Final]
at io.netty.buffer.PoolArena.allocate(PoolArena.java:227)
~[io.netty-netty-buffer-4.1.43.Final.jar:4.1.43.Final]
at io.netty.buffer.PoolArena.allocate(PoolArena.java:147)
~[io.netty-netty-buffer-4.1.43.Final.jar:4.1.43.Final]
at
io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:342)
~[io.netty-netty-buffer-4.1.43.Final.jar:4.1.43.Final]
at
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
~[io.netty-netty-buffer-4.1.43.Final.jar:4.1.43.Final]
at
org.apache.bookkeeper.common.allocator.impl.ByteBufAllocatorImpl.newDirectBuffer(ByteBufAllocatorImpl.java:164)
~[org.apache.bookkeeper-bookkeeper-common-allocator-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.common.allocator.impl.ByteBufAllocatorImpl.newDirectBuffer(ByteBufAllocatorImpl.java:158)
~[org.apache.bookkeeper-bookkeeper-common-allocator-4.10.0.jar:4.10.0]
at
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
~[io.netty-netty-buffer-4.1.43.Final.jar:4.1.43.Final]
at
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
~[io.netty-netty-buffer-4.1.43.Final.jar:4.1.43.Final]
at
org.apache.bookkeeper.bookie.BufferedChannel.<init>(BufferedChannel.java:86)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.bookie.BufferedChannel.<init>(BufferedChannel.java:77)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.bookie.BufferedChannel.<init>(BufferedChannel.java:71)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.bookie.Journal.lambda$getBufferedChannelBuilder$0(Journal.java:1129)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.bookie.JournalChannel.<init>(JournalChannel.java:175)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.bookie.JournalChannel.<init>(JournalChannel.java:114)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.bookkeeper.bookie.Journal.run(Journal.java:947)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
4.386: Total time for which application threads were stopped: 0.0001696
seconds, Stopping threads took: 0.0000834 seconds
4.387: Total time for which application threads were stopped: 0.0001677
seconds, Stopping threads took: 0.0001105 seconds
4.387: Total time for which application threads were stopped: 0.0001555
seconds, Stopping threads took: 0.0000935 seconds
19:15:22.338 [component-shutdown-thread] INFO
org.apache.bookkeeper.replication.AutoRecoveryMain - Shutting down auto
recovery: 0
19:15:22.338 [component-shutdown-thread] INFO
org.apache.bookkeeper.replication.AutoRecoveryMain - Shutting down AutoRecovery
Heap```
getting this error in bookeeper
----
2020-08-06 19:17:26 UTC - Caleb Epstein: Need more memory for the JVM to run
----
2020-08-06 19:18:05 UTC - Rattanjot Singh: I am running on kubernetes. where to
set that?
----
2020-08-06 19:18:50 UTC - Caleb Epstein: Not sure - I run on bare metal and its
in conf/bkenv.sh, see BOOKIE_MEM
----
2020-08-06 19:19:55 UTC - Caleb Epstein: If you can pass in environment
variables, you can set that and it should be honored by the script
----
2020-08-06 19:20:11 UTC - Caleb Epstein: I changed from 4G to 8G when I ran
into the same failure
----
2020-08-06 19:20:37 UTC - Caleb Epstein:
`BOOKIE_MEM=${BOOKIE_MEM:-${PULSAR_MEM:-"-Xms8g -Xmx8g
-XX:MaxDirectMemorySize=8g"}}`
----
2020-08-06 19:21:14 UTC - Rattanjot Singh: let me try this
----
2020-08-06 19:21:15 UTC - julien: @julien has joined the channel
----
2020-08-06 19:29:01 UTC - julien: Hello, I am setting up bookkeeper with pulsar
for production use, I am looking for filesystem recommendation, do you know if
there is a documentation about that?
----
2020-08-06 19:54:26 UTC - Addison Higham: as in which type of filesystem and
settings? XFS is what is used in the helm chart and i know it is also used in
some of the larger bookkeeper installs, it helps with latencies. I have not
seen much about additional tuning (like block size, etc)
The most important thing for performance in bookkeeper is fast journal disk
----
2020-08-06 19:56:27 UTC - julien: ok, thank you Addison
----
2020-08-06 20:11:53 UTC - Rattanjot Singh: yes it worked thanks
+1 : Caleb Epstein
----
2020-08-06 20:38:50 UTC - Caleb Epstein: Wow, 64 bookies. Curious how much
traffic you're handling at that size.
----
2020-08-06 20:39:26 UTC - Caleb Epstein: 96% of 64 doesn't sound like 51 tho
:slightly_smiling_face:
----
2020-08-06 21:00:44 UTC - Brandon Whitehead: so I have to wonder
----
2020-08-06 21:01:08 UTC - Brandon Whitehead: does the localhost that is made
via docker connecting with pulsar actually for frontend use?
----
2020-08-06 21:02:36 UTC - Brandon Whitehead: cause I still run the go program,
and I get no actual message from my consumer object in the program, yet I do
get confirmation that the topic is running, it connected to the topic, and that
it got A message
----
2020-08-06 21:02:48 UTC - Brandon Whitehead: but I have no way to show said
message anywhere
----
2020-08-06 21:02:54 UTC - Brandon Whitehead: where does it get sent to?
----
2020-08-06 21:13:31 UTC - Brandon Whitehead: I'd just like to know how to look
at a payload's message at all
----
2020-08-06 21:13:52 UTC - Brandon Whitehead: maybe then I can figure out how to
transfer that to some simplistic HTML format for the localhost or something
----
2020-08-06 21:38:12 UTC - Axel Sirota: @Sijie Guo do you have any clues? I got
this example from your tutorial :slightly_smiling_face:
----
2020-08-06 21:40:27 UTC - Sijie Guo: I am not sure if that example still works
or not. The demo is 3-years old :slightly_smiling_face:
----
2020-08-06 21:40:36 UTC - Sijie Guo: I also need to try it to verify
----
2020-08-06 21:40:52 UTC - Axel Sirota: haha indeed! I couldn’t make it work
:joy: is the client adapter still a thing??
----
2020-08-06 21:41:01 UTC - Axel Sirota: thanks for following!
----
2020-08-06 21:43:33 UTC - Axel Sirota: For reference, my app trying to use the
adapter is in :
<https://github.com/axel-sirota/handling-streaming-apache-pulsar/blob/master/src/main/resources/log4j2.xml>
I am trying to make a video course on Pulsar and how it makes a revolution in
the pub-sub space
----
2020-08-06 21:43:42 UTC - Axel Sirota: So a video showing how to migrate will
be super useful
----
2020-08-06 22:15:48 UTC - Shilpy: @Shilpy has joined the channel
----
2020-08-07 00:20:31 UTC - Brandon Whitehead: how do you use pulsarctl to force
a consumer to unsubscribe?
----
2020-08-07 00:21:41 UTC - julien: Hello, is there a specific mechanism to
backup data in bookkeeper? Do I need backup also metadata in Zookeeper or it is
it possible to reconstruct it?
----
2020-08-07 00:24:34 UTC - Brandon Whitehead: right now, I'm trying to test my
one consumer on my one topic by unsubscribing them from the topic, but using
pulsarctl, I can't unsubscribe it because the consumer is still using the
subscription
----
2020-08-07 00:41:53 UTC - Shilpy: Hi, i am getting below error in my service
after the service has run for quite few hours :
org.apache.pulsar.client.api.PulsarClientException$TimeoutException: The
producer standalone-0-71 can not send message to the topic <TOPICNAME>
with given timeout
----
2020-08-07 00:42:18 UTC - Shilpy: Can anybody help in this.
----
2020-08-07 03:21:25 UTC - Yuvaraj Loganathan: This is used by delayed message
delivery. Do you publish messages with very huge delay ?
----
2020-08-07 05:22:35 UTC - Takahiro Hozumi: I skimmed the source code of
compaction and found that the whole keys are held on memory using regular
HashMap during compaction.
This implementation requires more memory than the total size of the key.
<https://github.com/apache/pulsar/blob/df19f261def8e66de213bff310a8423ae133ef58/pulsar-broker/src/main/java/org/apache/pulsar/compaction/TwoPhaseCompactor.java#L80>
----
2020-08-07 08:19:27 UTC - charles: @Addison Higham: I used batched sync
:face_with_hand_over_mouth: .... switched to async now ... and the big numbers
start reporting (seeing 100k+ msg/s) thanks for this reminder :+1:
----
2020-08-07 08:48:09 UTC - Arek R.: @Arek R. has joined the channel
----
2020-08-07 08:49:32 UTC - Arek R.: Hello, is it possible to load the data into
pulsar when no consumer is running?
----
