2020-10-22 09:20:42 UTC - Johannes Wienke: What is the issue with SubRecords in
Python? Can't get this code working:
```from enum import Enum
import pulsar
from pulsar.schema import AvroSchema, String, Record
class TypeEnum(Enum):
TYPE_1 = "type_1"
TYPE_2 = "type_2"
class TestType(Record):
value = String()
class Composite(Record):
the_type = TypeEnum
first = TestType()
client = pulsar.Client("<pulsar://localhost:6650>")
producer = client.create_producer("schema-test-2", schema=AvroSchema(Composite))
for i in range(10):
producer.send(Composite(the_type=TypeEnum.TYPE_1,
first=TestType(value="test")))
client.close()```
This always gives
> ValueError: <__main__.TestType object at 0x7efc7b09eeb0> (type
<class '__main__.TestType'>) do not match ['null', {'type': 'record',
'name': 'TestType', 'fields': [{'name': 'value', 'type': ['null', 'string']}]}]
on field first
----
2020-10-22 11:45:59 UTC - VanderChen: Thank you very much. I'll have a try.
----
2020-10-22 12:15:08 UTC - Joshua Decosta: I appreciate the response, no worries
as i know this isn’t your main responsibility. I’d be down for a call. I’ve
been looking at the source code to see what’s going on. I noticed in September
there was an issue that seems related to what I’m seeing. It was closed with a
recent PR. Perhaps that’s not in the current version I’m using.
----
2020-10-22 12:16:43 UTC - Joshua Decosta: Did you configure Prometheus to use
the same auth? If not it can’t get the metrics unless you’ve turned off the
auth on the metrics endpoint
----
2020-10-22 16:37:19 UTC - Aaron Stockton: Am I reading this correctly, that
with batching enabled there’s no way to get a count of unconsumed messages,
only unconsumed batches?
<https://pulsar.apache.org/docs/en/concepts-messaging/#batching>
----
2020-10-22 16:40:30 UTC - Lari Hotari: this is a real gotcha in Pulsar. I now
realize that I have this wrong for the namespace where the 100% CPU bug
occured. I checked the docs and it's only in the javadocs where it's documented
clearly.
<https://pulsar.apache.org/docs/en/2.6.1/cookbooks-retention-expiry/#set-retention-policy>
doesn't explicitly document the meaning of 0 and -1. There's an example, but
no explicit documentation like there is in the Javadoc at
<https://github.com/apache/pulsar/blob/22b8923a251c3356401090dacb8ff97fd37d14c1/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedLedgerConfig.java#L415-L417>
. I could create a Github issue for documenting this in a better way.
I think that setting the retention policy to something where only either one of
the values is non zero doesn't make sense at all. That should fail already in
validation. I think that deserves it's own Github issue. WDYT?
----
2020-10-22 16:43:28 UTC - Lari Hotari: I take my claim back. I now see a
sentence "It is also possible to set _unlimited_ retention time or size by
setting `-1` for either time or size retention."
<https://pulsar.apache.org/docs/en/2.6.1/cookbooks-retention-expiry/#retention-policies>
. However it's really easy to miss that and I think that a config where one is
non zero and other is zero doesn't make sense and should be rejected to prevent
mistakes in configuration.
----
2020-10-22 16:44:08 UTC - Lari Hotari: Well that sentence is a bit misleading.
since it says "by setting -1 for either time or size retention".
----
2020-10-22 16:55:11 UTC - Sijie Guo: Nice catch. Are you going to fix the
documentation? Or shall I loop @Jennifer Huang on this?
----
2020-10-22 17:01:10 UTC - Lari Hotari: I'll create a GH issue now and I can
possibly send a PR. I'm currently filing 2 GH issues, one for the docs and the
other for the validation of the policy when it's set.
----
2020-10-22 17:05:09 UTC - otman: Hi i face this error in my client that try to
connect to remote broker: - 2020-10-22 16:51:57.091 [pulsar-client-io-923-1]
WARN org.apache.pulsar.client.impl.ClientCnx - [id: 0x106d4fd0,
L:/10.42.10.2:57588 - R:pulsar.testenv.local/127.0.0.158:30590] Received error
from server: Namespace bundle for topic
(<persistent://tenant/ns/action-7015e80f-60dc-485f-9c6f-b092bf5644d9>) not
served by this instance. Please redo the lookup. Request is denied:
namespace=tenant/ns. - Also i have several broker running in server side and
50000 clients that try to connect to broker. Each client have 4 topics. When i
try to list all available non partitioned topics from CLI, i face 500 internal
error and when i did a lookup on a specific topic, it return to me a specific
broker (for ex broker2). Is it possible that a topics is handle by a specific
instance ?
----
2020-10-22 17:12:03 UTC - Lari Hotari:
<https://github.com/apache/pulsar/issues/8345> for the documentation issue
+1 : Jennifer Huang
----
2020-10-22 17:14:51 UTC - Addison Higham: I agree with having either one of
those values be zero is going to cause issues and we should validate against
that, bit of a truth table on all the combinations:
both greater than zero: valid
both -1: valid
both 0: valid
one greater than, one -1: valid
one 0, the other greater than 0 or -1: invalid
I think that makes sense?
+1 : Lari Hotari
----
2020-10-22 17:20:08 UTC - Lari Hotari: yes, makes sense. I created a separate
issue for the validation <https://github.com/apache/pulsar/issues/8346>
----
2020-10-22 17:20:56 UTC - Lari Hotari: @Addison Higham please add your truth
table suggestion on that issue
----
2020-10-22 17:22:08 UTC - Addison Higham: done
+1 : Lari Hotari
----
2020-10-22 17:40:26 UTC - Guillaume: I don't think so. Where and how can I
configure Prometheus to use the same auth? Thank you.
----
2020-10-22 17:41:52 UTC - Joshua Decosta: You would need to google Prometheus
authentication. (Sorry I’m on my phone) there are some docs floating around
that will show you what you need to do.
----
2020-10-22 17:42:40 UTC - Joshua Decosta: I know there is a token setup
floating around somewhere if that’s what you’re using
----
2020-10-22 18:27:36 UTC - Milos Matijasevic: here is added async producer send,
(<https://gist.github.com/milos-matijasevic/e29d00e279b5d1a540552864c7ce1321|new-code>)
but i have problem that for example 10k messages are processed and acked but
in new topic only 10 messages are inserted and for solving this, i put
`DisableBatching` to true in producersOptions, but i don't understand why this
is problem ?
----
2020-10-22 21:47:23 UTC - Milos Matijasevic: i understood, all messages are
there just they are batched inside entries thats why there is 10 entries for
10k messages because 1 batch contains 1k messages
----
2020-10-22 21:57:41 UTC - Kenny Sargent: Hey Pulsar friends, any idea why
RedeliveryCount not being incremented in Exclusive and Failover subscriptions?
When we nack a message and it retries (a minute later) we see RedeliveryCount
stay at zero. For Shared and KeyShared subscription types (with or without dead
letter queue) we see the RedeliveryCount incrementing. Is this by design,
something we’re doing wrong, or a pulsar bug? Any thoughts?
----
2020-10-23 00:58:27 UTC - Jennifer Huang: :+1:
----
2020-10-23 02:28:12 UTC - テテナン: @テテナン has joined the channel
----
2020-10-23 02:29:50 UTC - テテナン: Hi, I'm using Python with Pulsar-Client, but
cannot make it through the company proxy. Any idea ?
----
2020-10-23 05:07:30 UTC - Sankararao Routhu: Hi, is it possible to have two
bookies(part of the Ensemble) down at the same time with out impacting consumer
with Ensemble=3, Qw=2, Qa=2..It is causing the consumers to stall until I bring
back one of the bookies back..I tried resetting the cursor to a recent
messageId and then consumer is able to resume with recent messageId. But
consumer is missing the old unconsumed messages. If I bring back the bookies
and reset the cursor back to old msgId then consumer started receiving old
messages along with recent messages(which are already consumed and
acknowledged). Is there any better to handle this case?
----
2020-10-23 05:13:47 UTC - Addison Higham: I believe some metrics properly break
it down, but in general that is correct
----
2020-10-23 05:16:53 UTC - Addison Higham: interesting... not sure why that
would be happening, but are you trying to use a dead letter queue?
----
2020-10-23 05:22:08 UTC - Addison Higham: is the consumer caught up when it
stalls? because when Qa=2 and you have two brokers down, producers won't be
able to produce.
----
2020-10-23 05:39:06 UTC - Sankararao Routhu: @Addison Higham I brought down 2
bookies, and topic owner broker is still up. Producer is still able to publish
when two bookies down because we have 5 bookies in a region and Ensemble(3)
formed with two other bookies
----
2020-10-23 05:39:36 UTC - Sankararao Routhu: Only consumer is stalled because
it is waiting for a message which is in down bookies
----
2020-10-23 05:45:59 UTC - Addison Higham: I see, consumers should be able to
read from any copy of the bookie, @Sijie Guo you have any idea on that one
----
2020-10-23 05:47:23 UTC - Sijie Guo: Because Qw is 2
----
2020-10-23 05:47:34 UTC - Sijie Guo: So when you lose 2 bookies, there is no
copy to read
----
2020-10-23 05:48:15 UTC - Addison Higham: oops! didn't notice that
----
2020-10-23 05:48:52 UTC - Sankararao Routhu: yes 2 copies might be in the down
bookies
----
2020-10-23 05:49:45 UTC - Sankararao Routhu: is there a way to recover from
this immediately without impacting the consumer until we bring back the bookies?
----
2020-10-23 05:50:00 UTC - Addison Higham: yes, so the benefit of having more
members in ensemble that in write quorum is striping but it does come at cost
of reads, so usually you would want 3/3/2
----
2020-10-23 05:50:42 UTC - Sankararao Routhu: you mean write quorum should be
same as Ensemble?
----
2020-10-23 05:51:33 UTC - Addison Higham: but if you are sure you want just 2
copies, then you can use `autoSkipNonRecoverableData`
----
2020-10-23 05:51:58 UTC - Sankararao Routhu: 2 copies are enough for us
----
2020-10-23 05:52:50 UTC - David Tinker: Hi Guys. I have a simple pulsar setup
with 3 machines each running zk, bookie and broker. I want to use a separate zk
cluster for the configuration store in preparation for adding another pulsar
cluster. How can do this while keeping everything live? Tx.
----
2020-10-23 05:52:54 UTC - Sankararao Routhu: will `autoSkipNonRecoverableData`
create any message loss to consumer?
----
2020-10-23 05:53:28 UTC - Sankararao Routhu: I mean what ever messages that
were skipped, can I go back and consume when bookies are brought back?
----
2020-10-23 05:56:33 UTC - Addison Higham: if the ledger can't currently `be
read, then once the bookeis come back up, you would need to rewind the consumer
----
2020-10-23 05:57:08 UTC - Addison Higham: but in answer to you question,
usually keeping your ensemble size and quorum at the same value is preferred
----
2020-10-23 06:03:29 UTC - Sankararao Routhu: thanks @Addison Higham I will test
with `autoSkipNonRecoverableData` enabled. I have used Ensemble=3 just to
achieve striping for high performance, if E and Qw should be same then I will
set E=2, Qw=2
----
2020-10-23 06:04:13 UTC - Sankararao Routhu: can I set Qa=1 in this case? will
it cause any data loss if one bookie goes down with E=2, Qw=2,Qa=1?
----
2020-10-23 08:03:33 UTC - Lari Hotari: I pushed the PR to improve the
documentation of retention policy: <https://github.com/apache/pulsar/pull/8356>
. Please review
----
