2020-07-22 13:10:38 UTC - Joshua Decosta: Is there a guaranteed order for
authentication and authorization? Can I expect once I’ve entered the Authorize
methods that I’ve been authenticated?
----
2020-07-22 13:10:47 UTC - anbutech17: Hello All!Could you please someone help
me to setup the pulsar python setup in the pycharm IDE.i have google it and
didn't get any information on this.
it will be very helpful, if anyone share the steps to install the pulsar python
libraries in a IDE.
----
2020-07-22 13:11:14 UTC - Joshua Decosta: Do you have your brokerClient
configurations in the standalone conf?
----
2020-07-22 13:11:35 UTC - Joshua Decosta: Are you just using Authentication or
did you enable authorization as well?
----
2020-07-22 13:11:48 UTC - Joshua Decosta: Did you configure the client conf for
pulsar-admin?
----
2020-07-22 13:12:48 UTC - Rattanjot Singh: What are the brokerClient
configurations that we need to pass in standalone conf
----
2020-07-22 13:13:40 UTC - rwaweber: I was checking the metric from the brokers
too, after your suggestion earlier — sorry! I should’ve followed up with that
----
2020-07-22 13:13:43 UTC - Joshua Decosta: The broker client configs are for the
broker to broker communications and function workers as well in standalone
----
2020-07-22 13:14:08 UTC - Joshua Decosta: There is reference to them in the
docs but I’m on my phone at the moment and can’t copy the link
----
2020-07-22 13:14:09 UTC - Rattanjot Singh: But in standalone do we need to pass
those?
----
2020-07-22 13:14:13 UTC - Joshua Decosta: Yes
----
2020-07-22 13:14:22 UTC - Joshua Decosta: The standalone setup relies on them
----
2020-07-22 13:22:28 UTC - Rattanjot Singh: Can you give a reference to them?
----
2020-07-22 13:23:12 UTC - Joshua Decosta: If you look in the broker
configuration you’ll see them mentioned. They start with brokerClient
----
2020-07-22 13:23:51 UTC - Joshua Decosta: It’s the client that the broker
itself uses to communicate with the cluster. Similar to how the admin client is
used by you to speak to the cluster. It’s for authenticating the actual pulsar
components
----
2020-07-22 13:27:42 UTC - Joshua Decosta: This is a reference to it in the JWT
authentication section.
----
2020-07-22 13:36:53 UTC - Ebere Abanonu: Currently testing KeyShared, I have
two different consumers having same orderingkey, the test is failing. All
message have same key
----
2020-07-22 15:11:37 UTC - Roberto D'Arco: @Roberto D'Arco has joined the channel
----
2020-07-22 15:47:05 UTC - Sijie Guo: Yes. Authorization happens after
Authentication.
+1 : Joshua Decosta
----
2020-07-22 15:47:53 UTC - Sijie Guo: How did you do the test?
----
2020-07-22 16:29:00 UTC - Kenny Sargent: @Kenny Sargent has joined the channel
----
2020-07-22 16:52:56 UTC - Ebere Abanonu: ```private void
ReceiveAndCheckDistribution(IList<(IActorRef consr, string name)>
consumers)
{
// Add a key so that we know this key was already
assigned to one consumer
IDictionary<string, IActorRef> keyToConsumer = new
Dictionary<string, IActorRef>();
IDictionary<IActorRef, int> messagesPerConsumer = new
Dictionary<IActorRef, int>();
var totalMessages = 0;
foreach (var c in consumers)
{
var messagesForThisConsumer = 0;
while (true)
{
var msg =
_common.PulsarSystem.Receive(c.name, 100);
if (msg == null)
{
// Go to next consumer
messagesPerConsumer[c.consr] = messagesForThisConsumer;
break;
}
++totalMessages;
++messagesForThisConsumer;
_common.PulsarSystem.Acknowledge(msg);
if (msg.Message.HasKey() || msg.Message.HasOrderingKey())
{
var orderingKey = msg.Message.HasOrderingKey() ?
((byte[]) (object) msg.Message?.OrderingKey).GetString(): string.Empty;
string key =
msg.Message.HasOrderingKey() ? orderingKey : msg.Message.Key;
if (!keyToConsumer.TryGetValue(key, out var
assignedConsumer))
{
// This is a new key
keyToConsumer[key] = c.consr;
}
else
{
// The consumer should be the same
_output.WriteLine($"The
consumer should be the same: [{assignedConsumer}] > [{c.consr}]
(OrderingKey: {orderingKey}, Key: {msg.Message.Key})");
Assert.Equal(c.consr, assignedConsumer);
}
}
}
}```
----
2020-07-22 16:55:30 UTC - Ebere Abanonu: `Assert.Equal(c.consr,
assignedConsumer);` fails when it moves to consumer2 first message
----
2020-07-22 16:57:12 UTC - Ebere Abanonu: I ported
`TestOrderingKeyWithHashRangeAutoSplitStickyKeyConsumerSelectorBatch` to c#
----
2020-07-22 18:01:00 UTC - Joshua Eric: Not exactly sure what you're needing
here, but there are steps here on how to get setup:
<https://pulsar.apache.org/docs/en/client-libraries-python/>
----
2020-07-22 18:04:30 UTC - Joshua Eric: In PyCharm, just setup a virtual
environment and then the steps at the above link should get you started.
Here is information on how to get Pulsar running locally so that you can
interact with it from inside of PyCharm:
<http://pulsar.apache.org/docs/en/2.6.0/standalone/>
I suggest docker, as it is much easier:
<http://pulsar.apache.org/docs/en/2.6.0/standalone-docker/>
Then you could use something like this:
<https://www.jetbrains.com/help/pycharm/docker.html>
----
2020-07-22 18:04:58 UTC - Rattanjot Singh: I have disabled authorization
still getting the same error
----
2020-07-22 18:09:03 UTC - Joshua Eric: The included proxy for pulsar is
relatively lightweight, but can be extended.
See PIP-60 here:
<https://pulsar.apache.org/blog/2020/06/18/Apache-Pulsar-2-6-0/>
You can use Envoy as the proxy, and then you can add a lot of functionality
there. If you're wanting to inspect the payload or do things around
authentication, you may be limited in that manner due to encryption. But you
can use your own authentication method, or worst case, extend the pulsar code
base itself.
The cool thing is once you have this information, you can use Pulsar itself to
provide an audit log by sending all of this data to a topic!
slightly_smiling_face : alex kurtser
+1 : alex kurtser
----
2020-07-22 18:15:56 UTC - Joshua Eric: I can't answer all of your questions,
but we have observed similar behavior.
There is caching behavior that could be the problem. Additionally, there are
default settings around topics and messages that could cause a topic to get
deleted. Your first call might hit the cache, which then gets flushed,
affecting the second call.
I've also noticed that there are other settings that affect how different parts
of the system report their data, how data is aggregated, and the such.
Pulsar has a ton of configurable options, so I would suggest reading through
the code for that endpoint or playing with different settings like you were.
You could also setup Prometheus and see how that data compares.
----
2020-07-22 19:12:21 UTC - Joshua Decosta: You haven’t configure authentication
correctly then. Without seeing your conf files i can’t really tell you what you
need to change
----
2020-07-22 23:28:45 UTC - Luke Stephenson: Is there a timeline on 2.6.1
release? I'm after this fix <https://github.com/apache/pulsar/pull/7389>
----
2020-07-23 00:30:17 UTC - James Wilson: @James Wilson has joined the channel
----
2020-07-23 04:41:12 UTC - Sijie Guo: @xiaolong.ran is working on 2.6.1 release.
should be coming out in the next 2 weeks.
----
2020-07-23 05:02:20 UTC - Rattanjot Singh: ```ERROR
org.apache.pulsar.client.impl.ProducerImpl -
[<persistent://public/functions/assignments>]
[c-standalone-fw-localhost-8080-scheduler-manager] Failed to create producer:
java.lang.IllegalStateException: TopicOperation is not supported by the
Authorization provider you are using.
07:55:42.571 [main] ERROR org.apache.pulsar.functions.worker.SchedulerManager -
Exception while at creating producer to topic
<persistent://public/functions/assignments>
java.util.concurrent.ExecutionException:
org.apache.pulsar.client.api.PulsarClientException$AuthorizationException:
java.lang.IllegalStateException: TopicOperation is not supported by the
Authorization provider you are using.```
isn't a superUser allowed to do this?
----
2020-07-23 05:53:06 UTC - Sijie Guo: Are you using customized authorization
plugin?
----
2020-07-23 05:53:13 UTC - Rattanjot Singh: yes
----
2020-07-23 05:54:49 UTC - Sijie Guo: Okay. The Authorization Interface is
refactored in 2.6.0 to provide finer granularity access control. This refactor
works for the default implementation. But it might break customized
implementation. If you are upgrading to 2.6.0, you might need to update your
authorization plugin to implement the newly introduced methods.
----
2020-07-23 07:11:15 UTC - Odette Caruana: @Odette Caruana has joined the channel
----
2020-07-23 07:38:04 UTC - Takahiro Hozumi: Hi, I tried to make a subscription
with KeyShared mode and `readCompacted=true` and noticed the following
restriction.
<https://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ConsumerBuilder.html#readCompacted-boolean->
> readCompacted can only be enabled subscriptions to persistent topics,
which have a single active consumer (i.e. failure or exclusive subscriptions).
I'm bit confused as this restriction is not mentioned in the document.
<https://pulsar.apache.org/docs/ja/concepts-topic-compaction/>
<https://pulsar.apache.org/docs/ja/cookbooks-compaction/>
Is the restriction specific to java client or imposed by pulsar?
----
2020-07-23 08:33:55 UTC - Arthur: Thanks. In fact using Prometheus could be a
good compromise.
I endeavor to find in broker config a way to get metrics more quickly, actually
I have to wait several minutes and it does not fit my use case.
----
