2019-07-11 11:54:39 UTC - Venugopal Doddi: np... @Sijie Guo
----
2019-07-11 13:00:22 UTC - Enrico Olivelli: Hello, I am trying to upgrade from
2.3.0 to 2.4.0 but I get this error "Policies not found for idc2/ns namespace"
----
2019-07-11 13:01:26 UTC - Enrico Olivelli: I am creating the namespace with the
Java API: admin.namespaces().createNamespace(namespace, new
HashSet<>(Arrays.asList(CLUSTER_NAME))); and it works in 2.3.0
I found also that in 2.4.0 I have to explicitly "create" the "cluster"
----
2019-07-11 13:02:53 UTC - Enrico Olivelli: @jia zhai do you have any idea ? I
did not find notes in <https://pulsar.apache.org/release-notes/#2.4.0> about
these changes
----
2019-07-11 13:04:56 UTC - Enrico Olivelli: this is the doc of the REST API
<https://pulsar.apache.org/admin-rest-api/#operation/createNamespace>
----
2019-07-11 13:15:29 UTC - Devin G. Bost: @Enrico Olivelli Did you create the
tenant? Typically, that error appears when something didn't go right with
creating the namespace. The namespace policies won't exist if the namespace
wasn't actually created successfully.
----
2019-07-11 13:16:52 UTC - Enrico Olivelli: in 2.3.0 I am doing:
TenantInfo tenantInfo = new TenantInfo();
tenantInfo.setAllowedClusters(new HashSet<>(Arrays.asList(CLUSTER_NAME)));
admin.tenants().createTenant(tenantId, tenantInfo);
admin.namespaces().createNamespace(namespace, new
HashSet<>(Arrays.asList(CLUSTER_NAME)));
and the namespace is: tenantid+"/ns"
----
2019-07-11 13:17:34 UTC - Enrico Olivelli: "createTenant" returns with success
and also createNamespace
----
2019-07-11 13:17:58 UTC - Enrico Olivelli: which is the good way in 2.4.0 ?
(thank you @Devin G. Bost for taking a look)
----
2019-07-11 13:28:40 UTC - Devin G. Bost: Was your code working in 2.3.0? I
haven't tried running mine in 2.4.0 yet.
----
2019-07-11 13:29:14 UTC - Devin G. Bost: Also, I'd recommend that you try
getting the namespace with the name you're providing (e.g. in a unit test) to
ensure that it's coming back as expected.
----
2019-07-11 13:29:51 UTC - Devin G. Bost: In your test, I'd recommend providing
the name as an explicit string just in case the name is drifting somehow.
----
2019-07-11 13:31:00 UTC - Devin G. Bost: You can check for an existing tenant
like this:
```
@Override
public boolean existsInPulsar() {
PulsarAdmin pulsarAdmin = this.getPulsarAdmin();
List<String> existingTenants = null;
try {
existingTenants = pulsarAdmin.tenants().getTenants();
} catch (PulsarAdminException e) {
e.printStackTrace();
}
List<String> tenants = existingTenants;
String name = this.getName();
return existingTenants.contains(this.getName());
}
```
You can check for an existing namespace like this:
```
@Override
public boolean existsInPulsar() {
PulsarAdmin pulsarAdmin = this.getPulsarAdmin();
List<String> namespaces = null;
try {
namespaces =
pulsarAdmin.namespaces().getNamespaces(this.getTenant());
} catch (PulsarAdminException e) {
e.printStackTrace();
}
return namespaces.contains(this.getFullyQualifiedName());
}
```
----
2019-07-11 13:32:32 UTC - Devin G. Bost: Creating your tenant looks like this:
```
try {
Tenant tenantObj = new Tenant();
tenantObj.setName(tenant);
tenantObj.setPulsarServiceURL(pulsarAdmin.getServiceUrl());
if (!tenantObj.existsInPulsar()){
pulsarAdmin.tenants().createTenant(tenant,
tenantObj.getDefaultTenantInfo());
}
} catch (PulsarAdminException e) {
e.printStackTrace();
}
```
In my code, my `.getDefaultTenantInfo()` looks like this:
```
public TenantInfo getDefaultTenantInfo(){
fastDeployConfig = StaticUtils.getFastDeployConfig();
Set<String> adminRoleSet =
fastDeployConfig.getFastdeployAuthzAdminRoleSetClean();
Set<String> clusterSet =
fastDeployConfig.getFastdeployAllowedClusterSetClean();
TenantInfo tenantInfo = new TenantInfo(adminRoleSet, clusterSet);
return tenantInfo;
}
```
----
2019-07-11 13:33:16 UTC - Devin G. Bost: Then, creating a namespace looks like
this:
```
try {
System.out.println("Creating namespace: " +
this.getFullyQualifiedName());
pulsarAdmin.namespaces().createNamespace(this.getFullyQualifiedName());
} catch (PulsarAdminException e) {
e.printStackTrace();
}
```
----
2019-07-11 13:33:27 UTC - Devin G. Bost: @Enrico Olivelli Please let me know if
this helps.
----
2019-07-11 13:42:03 UTC - Penghui Li: @Enrico Olivelli I have test it on my mac:
```
PulsarAdmin admin =
PulsarAdmin.builder().serviceHttpUrl("<http://localhost:8080>").build();
TenantInfo tenantInfo = new TenantInfo();
tenantInfo.setAllowedClusters(Sets.newHashSet("standalone"));
admin.tenants().createTenant("my-tenant1", tenantInfo);
admin.namespaces().createNamespace("my-tenant1/my-namespace1");
System.out.println(admin.namespaces().getPolicies("my-tenant1/my-namespace1"));
output:
Policies{auth_policies=org.apache.pulsar.common.policies.data.AuthPolicies@745f,
replication_clusters=[standalone], bundles=BundlesData{numBundles=4,
boundaries=[0x00000000, 0x40000000, 0x80000000, 0xc0000000, 0xffffffff]},
backlog_quota_map={destination_storage=BacklogQuota{limit=10737418240,
policy=producer_request_hold}}, persistence=null, deduplicationEnabled=null,
topicDispatchRate={standalone=DispatchRate{dispatchThrottlingRateInMsg=0,
dispatchThrottlingRateInByte=0, ratePeriodInSecond=1}},
subscriptionDispatchRate={standalone=DispatchRate{dispatchThrottlingRateInMsg=0,
dispatchThrottlingRateInByte=0, ratePeriodInSecond=1}},
replicatorDispatchRate={},
clusterSubscribeRate={standalone=SubscribeRate{subscribeThrottlingRatePerConsumer=0,
ratePeriodInSecond=30}}, latency_stats_sample_rate={}, antiAffinityGroup=null,
message_ttl_in_seconds=0, retention_policies=null, deleted=false,
encryption_required=false, subscription_auth_mode=None,
max_producers_per_topic=0, max_consumers_per_topic=0,
max_consumers_per_subscription=0, compaction_threshold=0, offload_threshold=-1,
offload_deletion_lag_ms=null, schema_auto_update_compatibility_strategy=Full,
schema_validation_enforced=false}
```
----
2019-07-11 14:42:22 UTC - Tim Farber-Newman: @Tim Farber-Newman has joined the
channel
----
2019-07-11 14:42:58 UTC - Enrico Olivelli: Thank you so much @Penghui Li and
@Devin G. Bost
I have reproduced the correct creating in unit tests and all works as expected.
and then I found my problem. When Pulsar returns this error "Policies not found
for idc2/ns namespace" it is saying "the name space does not exist" . I had a
typo in my client code and actually the code that was creating the namespace
wasn't called and more. :sweat_smile:
+1 : David Kjerrumgaard, Matteo Merli
----
2019-07-11 14:43:16 UTC - Enrico Olivelli: unit tests were actually working but
integration tests not
----
2019-07-11 14:43:37 UTC - Devin G. Bost: :slightly_smiling_face:
----
2019-07-11 16:05:47 UTC - Ryan Samo: Hey guys,
I have a Pulsar cluster stood up with functions workers outside of the brokers
as their own cluster. When we create functions, they work fine, no issues. If
we create a sink, it creates but then never subscribes the source topic. Also,
if you try to delete the sink you get a 401 unauthorized exception. Any idea
why functions would work fine but sinks do not? We can create, update, get,
list, you name it. Just can’t delete or get the sink to subscribe
----
2019-07-11 16:07:23 UTC - Ryan Samo: The sink logs show it picks up the nar
file, never a subscription or anything though
----
2019-07-11 16:08:39 UTC - Ryan Samo: Are sinks even allowed to run on function
workers outside the brokers?
----
2019-07-11 16:09:11 UTC - Jerry Peng: Yes
+1 : Ryan Samo
----
2019-07-11 16:09:17 UTC - Ryan Samo: TLS enabled cluster, using pulsar proxy as
well
----
2019-07-11 16:10:41 UTC - Jerry Peng: Is authorization turned on as well?
----
2019-07-11 16:11:14 UTC - Ryan Samo: In the functions_worker.yml or?
----
2019-07-11 16:11:51 UTC - Jerry Peng: Yes
----
2019-07-11 16:12:24 UTC - Ryan Samo: authorizationEnabled: true
----
2019-07-11 16:13:41 UTC - Jerry Peng: Try sending the delete command to
different function workers to see if you get the same result
----
2019-07-11 16:14:05 UTC - Jerry Peng: If you are able to create or update you
should be able to delete
----
2019-07-11 16:15:15 UTC - Ryan Samo: So I can create, update, and delete
functions just fine. I can create and update sinks but cannot delete them
----
2019-07-11 16:15:35 UTC - Ryan Samo: Tried multiple times on multiple different
sinks
----
2019-07-11 16:15:58 UTC - Ryan Samo: Using the provided sinks, not custom
----
2019-07-11 16:17:06 UTC - Ryan Samo: Getting the error, one second
----
2019-07-11 16:25:39 UTC - Ryan Samo: Here you go
----
2019-07-11 16:26:09 UTC - Ryan Samo: And the cli times out with a null response
----
2019-07-11 16:31:15 UTC - Jerry Peng: did you also configure:
```
# the authentication plugin to be used by the pulsar client used in worker
service
clientAuthenticationPlugin:
# the authentication parameter to be used by the pulsar client used in worker
service clientAuthenticationParameters:
```
----
2019-07-11 16:33:28 UTC - Ryan Samo: # the authentication plugin to be used by
the pulsar client used in worker service
clientAuthenticationPlugin: org.apache.pulsar.client.impl.auth.AuthenticationTls
# the authentication parameter to be used by the pulsar client used in worker
service
clientAuthenticationParameters:
tlsCertFile:/opt/pulsar/certificates/admin.cert.pem,tlsKeyFile:/opt/pulsar/certificates/admin.key-pk8.pem
----
2019-07-11 16:42:13 UTC - Jerry Peng: you made sure those keys are valid?
----
2019-07-11 16:42:50 UTC - Jerry Peng: Are those keys signed by a local CA?
----
2019-07-11 16:43:37 UTC - Ryan Samo: Yeah they are created from the same CA
that created the broker certs etc.
----
2019-07-11 16:44:00 UTC - Ryan Samo: And they seem to work fine for other admin
commands
----
2019-07-11 16:44:35 UTC - Ryan Samo: Is there a good way to validate them that
you know of?
----
2019-07-11 16:47:34 UTC - Jerry Peng: On one of the nodes your are running the
function worker, try using the pulsar-admin CLI with those keys to see if it
works
----
2019-07-11 16:48:24 UTC - Fiodar Zboichyk: @Fiodar Zboichyk has joined the
channel
----
2019-07-11 16:49:40 UTC - Jerry Peng: you also set superUserRoles in the
broker.conf on your brokers to contain the subject CN of the certs you are
using for your function worker?
----
2019-07-11 16:51:13 UTC - Ryan Samo: Admin CLI works when you run it on a
function worker as long as you pass the —admin-url flag and target the
functions workers url
----
2019-07-11 16:52:22 UTC - Ryan Samo: The broker config has the superusers set
as “admin,proxy” and the CN for both the admin and proxy certs are a match to
that
----
2019-07-11 16:54:39 UTC - Ryan Samo: Normal admin cli commands work as well
like listing the topics etc. I only have to use the —admin-url flag when
executing functions commands as the proxy does not forward them even though I
have the urls set for that in the proxy.conf
----
2019-07-11 16:56:02 UTC - Ryan Samo: Otherwise I will get “Function worker
service is not done initializing”
----
2019-07-11 17:16:08 UTC - Addison Higham: hrm... is there any API for doing a
"raw" list of topics in a namespace that would include both partitioned topics
(including the "internal" topic name) as well as non-partitioned topics?
----
2019-07-11 17:18:14 UTC - Ryan Samo: $PULSAR_HOME/bin/pulsar-admin topics list
public/default
----
2019-07-11 17:18:22 UTC - Ryan Samo: Gives both
----
2019-07-11 17:18:42 UTC - Jerry Peng: @Ryan Samo target the broker from the
function worker node
----
2019-07-11 17:18:51 UTC - Jerry Peng: i.e. set --admin-url to a broker
----
2019-07-11 17:18:56 UTC - Jerry Peng: and use those certs
----
2019-07-11 17:19:14 UTC - Jerry Peng: to simulate function workers
communicating with brokers
----
2019-07-11 17:19:29 UTC - Addison Higham: oh? hrm, do you actually have to
write records first then? because it doesn't seem to be showing, but will try
again
----
2019-07-11 17:19:56 UTC - Ryan Samo: @Jerry Peng will do
----
2019-07-11 17:21:38 UTC - Ryan Samo: @Addison Higham if you created the topic
via producing to it or even created via the admin-cli it should show up there
----
2019-07-11 17:24:45 UTC - Ryan Samo: @Jerry Peng it works, I just ran a topics
list from a functions worker to a broker targeting the broker
----
2019-07-11 17:28:29 UTC - Jerry Peng: @Ryan Samo can you try getting the stats
of one of the topics to make sure that is working
----
2019-07-11 17:28:51 UTC - Jerry Peng: e.g. ./bin/pulsar-admin topics stats
topic-name
----
2019-07-11 17:30:11 UTC - Ryan Samo: @Jerry Peng works as well
----
2019-07-11 17:30:31 UTC - Jerry Peng: weird
----
2019-07-11 17:31:02 UTC - Ryan Samo: Yeah it’s strange for sure, I’m puzzled
----
2019-07-11 17:31:13 UTC - Ryan Samo: v2.3.2
----
2019-07-11 17:35:10 UTC - Jerry Peng: is your CA a public one? if not you also
need to specify tlsTrustCertsFilePath
----
2019-07-11 17:35:27 UTC - Jerry Peng: or one Java knows
----
2019-07-11 17:36:34 UTC - Ryan Samo: I have tlsTrustCertsFilePath set to our ca
on all nodes
----
2019-07-11 17:38:33 UTC - Jerry Peng: For the exceptions you posted before, how
often do they occur? On every node running function worker?
----
2019-07-11 17:40:33 UTC - Addison Higham: I am getting different behavior, at
least with pulsar standalone:
```
$ pulsar-admin --admin-url <http://localhost:8080> topics create
<persistent://public/default/my-single-topic>
$ pulsar-admin --admin-url <http://localhost:8080> topics
create-partitioned-topic -p 5 <persistent://public/default/my-topic>
$ pulsar-admin --admin-url <http://localhost:8080> topics list public/default
<persistent://public/default/my-single-topic>
$ pulsar-admin --admin-url <http://localhost:8080> topics
list-partitioned-topics public/default
<persistent://public/default/my-topic>
```
----
2019-07-11 17:40:44 UTC - Addison Higham: going to try it on a real cluster and
see if I get something different
----
2019-07-11 17:40:52 UTC - Ryan Samo: @Jerry Peng let me try to run it parallel
and see
----
2019-07-11 17:50:25 UTC - Addison Higham: onto something somewhat different...
when doing an upgrade using the helm chart on k8s, it tries to change the
`zookeeper-metadata` job, which is immutable. It doesn't look like the helm
chart has a way to disable creation of that resource, so it seems like the
easiest thing to do would be to delete the job, but that only seems safe is
`initialize-cluster-metadata` is either idempotent or guards against anything
----
2019-07-11 18:02:45 UTC - Tarek Shaar: Hello quick question. Doing some testing
on pulsar Java consumers. It seems that if you make several consumers listening
on the same topic, then each message is always dispatch on the same thread. I
thought each consumer will get its own thread. Am i missing something
----
2019-07-11 18:04:48 UTC - Tarek Shaar:
----
2019-07-11 18:08:14 UTC - Tarek Shaar: It is always
Thread[pulsar-external-listener-3-1,5,main]. This is the case whether I used
shared or exclusive subs
----
2019-07-11 18:08:35 UTC - Tarek Shaar:
----
2019-07-11 18:15:37 UTC - Ryan Samo: @Jerry Peng no it looks to only do that
on 1 worker, depending on the DNS round robin
----
2019-07-11 18:25:13 UTC - Aaron Langford: @Addison Higham, on standalone, looks
like you have to send messages first before those partitioned topics show up...
----
2019-07-11 18:26:19 UTC - Aaron Langford: Then they will show up when you list
all topics in a namespace, but if you list partitioned topics, you will still
only get the single logical topic.
----
2019-07-11 18:28:15 UTC - Aaron Langford: Partitioned topics are of the form:
```
"persistent://{tenant}/{namespace}/{topic}-partition-{partition-number}",
```
Where partition number starts at 0.
----
2019-07-11 18:33:06 UTC - Ryan Samo: @Jerry Peng do the Pulsar brokers know
about the functions workers at all if they are running separately, or do the
functions workers request from the brokers like clients?
----
2019-07-11 18:33:45 UTC - Aaron Langford: I'm not sure if this is the case for
a full blown cluster, but I would speculate that it is so.
----
2019-07-11 18:33:50 UTC - Ryan Samo: Trying to figure out the difference
between why Pulsar functions would work fine but Pulsar IO wouldn’t
----
2019-07-11 18:50:31 UTC - Jerry Peng: @Ryan Samo when running separately,
function workers request from brokers like clients
----
2019-07-11 18:51:42 UTC - Ryan Samo: Ok so when creating a sink, it should
reach out to the brokers and subscribe to the topic then correct? Like starting
a consumer
----
2019-07-11 18:51:59 UTC - Jerry Peng: The sink instance will
----
2019-07-11 18:54:00 UTC - Ryan Samo: Ok and the sink will use whatever certs
are in the client.conf or the functions_worker.yml?
----
2019-07-11 18:55:58 UTC - Jerry Peng: in functions_worker.yml
----
2019-07-11 18:56:39 UTC - Ryan Samo: :+1:
----
2019-07-11 18:59:47 UTC - Aaron: @Matteo Merli I get the same issue when
testing the loop by itself - 13+ mb of data allocated from a 1 hour ackTimeout
value.
----
2019-07-11 19:04:07 UTC - Matteo Merli: Do you have a heap dump? How many
message ids are in the UnAckedMessageTracker?
----
2019-07-11 19:14:38 UTC - Addison Higham: I read a chunk of that code somewhat
recently, it seemed to me that the way the client was constructed had branching
code paths based on IO/functions, so it is quite possible it is just something
a bit different there
----
2019-07-11 19:33:37 UTC - Aaron: None - this is in the initialization of the
unackedmessagetracker
----
2019-07-11 19:36:32 UTC - Ryan Samo: @Addison Higham yeah something is weird
for sure
----
2019-07-11 19:37:36 UTC - Jerry Peng: the pulsar admin client created within a
function worker is the same between pulsar functions and sources and sinks
----
2019-07-11 19:39:29 UTC - Jerry Peng: There are two admin clients created in a
function worker. One for communicating with other function workers via function
endpoints and one for communicating with brokers
----
2019-07-11 19:40:03 UTC - Jerry Peng: Auth settings for the two are the same
----
2019-07-11 19:40:42 UTC - Jerry Peng: The exceptions @Ryan Samo posted earlier
seem to indicate a problem with the function worker communicating with brokers
----
2019-07-11 19:42:30 UTC - Ryan Samo: @Jerry Peng I agree, it does seem to be an
issue from functions worker to broker. What settings specifically are targeting
that communication path? Would it help to attach my functions_worker.yml?
----
2019-07-11 19:42:53 UTC - Jerry Peng: sure
----
2019-07-11 19:43:03 UTC - Ryan Samo: Ok one sec
----
2019-07-11 19:43:23 UTC - Jerry Peng: usually these configs:
```
pulsarServiceUrl: <pulsar://localhost:6650>
pulsarWebServiceUrl: <http://localhost:8080>
# the authentication plugin to be used by the pulsar client used in worker
service
clientAuthenticationPlugin:
# the authentication parameter to be used by the pulsar client used in worker
service
clientAuthenticationParameters:
```
----
2019-07-11 19:46:29 UTC - Jerry Peng: are you using the same set of certs on
both brokers and function workers?
----
2019-07-11 19:47:09 UTC - Ryan Samo:
----
2019-07-11 19:47:52 UTC - Ryan Samo: Certs on the brokers are broker.cert.pem
----
2019-07-11 19:48:28 UTC - Ryan Samo: I’m using the admin certs for the
functions
----
2019-07-11 19:48:57 UTC - Ryan Samo: Which are the same that is used in the
client.conf for the admin cli
----
2019-07-11 19:49:55 UTC - Ryan Samo: Maybe my client auth plugin is incorrect?
----
2019-07-11 19:51:44 UTC - Jerry Peng: most likely reason
----
2019-07-11 19:52:14 UTC - Jerry Peng: The subject CN for
/opt/pulsar/certificates/admin.cert.pem is listed as a super user role in
broker.conf on all brokers?
----
2019-07-11 19:52:44 UTC - Jerry Peng: all the certs on both functions workers
and brokers are signed by the same CA?
----
2019-07-11 19:53:22 UTC - Ryan Samo: Yes, CN for the admin.cert.pem is admin
----
2019-07-11 19:53:37 UTC - Ryan Samo: And it is set as superuser on all brokers
----
2019-07-11 19:54:32 UTC - Ryan Samo: I also added the proxy for good measure as
a superuser, not sure that’s required
----
2019-07-11 19:55:13 UTC - Ryan Samo: And the functions workers are targeting
the brokers, not the proxies
----
2019-07-11 20:01:40 UTC - Jerry Peng: You also successfully have used the
pulsar admin to connect to the broker with
tlsCertFile:/opt/pulsar/certificates/admin.cert.pem,tlsKeyFile:/opt/pulsar/certificates/admin.key-pk8.pem
right?
----
2019-07-11 20:01:55 UTC - Jerry Peng: the code for creating pulsar admins in
the worker is here:
<https://github.com/apache/pulsar/blob/master/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/WorkerService.java#L127>
----
2019-07-11 20:02:20 UTC - Ryan Samo: Double checking for good measure
----
2019-07-11 20:02:37 UTC - Jerry Peng: There is nothing obviously wrong in your
functions_worker.yaml
----
2019-07-11 20:03:06 UTC - Ryan Samo: That’s good at least lol
----
2019-07-11 20:29:54 UTC - Ryan Samo: Yup checks out
----
2019-07-11 22:16:25 UTC - Ryan Samo: @Jerry Peng hey man I just wanted to let
you know that I wiped the cluster configs and rebuilt it, now it’s working
perfectly. Not sure what got messed up or how but back in action. Thanks for
all your help!
+1 : David Kjerrumgaard, Ali Ahmed
----
2019-07-12 01:44:29 UTC - tuteng: This issue has been addressed, the search is
already working properly.
----
2019-07-12 01:46:06 UTC - tuteng: This issue has been addressed, the search is
already working properly.
clap : David Kjerrumgaard
----
2019-07-12 02:36:14 UTC - Santiago Del Campo: I think finally got it. Just like
you said, made sure there is persistent storage for each bookie and zookeeper
pod in each node. It was possible making use of Kubernetes Persistent volumes.
Definitely the cluster is waaaaay more stable now and with that i can make
further tests with other features Pulsar has to offer... i really appreciate
your help so far :slightly_smiling_face:.
I just wanted to point out that this might be some important modification that
should be done with the Yaml files (at least for the generic ones, which i am
using).... instead of using simple volumes, where the information is lost in
every redeploy, add comments inside the file about using persistent volumes to
maintain cluster consistency (for zookeeper and bookie deployments, of course,
the user has to create the Persistent Volume and Claim, but i think it should
be pointed out).
Another point is the init container inside de Bookie Yaml.... before the pods
are deployed... a *bin/bookkeeper shell metaformat --nonInteractive || true;*
is executed, erasing the metadata information and producing an Invalid Cookie
Exception. I just commented it, and combined with the persistent volume
approach.. i get a more stable cluster if i reboot the servers, or redeploy all
the pods... or simply if the cluster crashes for some reason for a short period
of time.
----
2019-07-12 05:27:26 UTC - takuto: @takuto has joined the channel
----
