2020-03-19 09:33:18 UTC - Poul Henriksen: I'm trying to increase the number of
partitions for a topic, but it fails, claiming it is not partitioned (it is
partitioned, but currently only with one partition):
``` root@tgtg-pulsar-bastion-7bb8c9b8f9-f86bg:/pulsar# bin/pulsar-admin
topics get-partitioned-topic-metadata
<persistent://prod/local/PulsarHighLoadMessage>
{
"partitions" : 1
}
root@tgtg-pulsar-bastion-7bb8c9b8f9-f86bg:/pulsar# bin/pulsar-admin
topics update-partitioned-topic <persistent://prod/local/PulsarHighLoadMessage>
-p 3
09:21:44.863 [AsyncHttpClient-5-1] WARN
org.apache.pulsar.client.admin.internal.BaseResource -
[<http://tgtg-pulsar-broker:8080/admin/v2/persistent/prod/local/PulsarHighLoadMessage/partitions?updateLocalTopicOnly=false>]
Failed to perform http post request: javax.ws.rs.ClientErrorException: HTTP
409 Conflict
Topic is not partitioned topic
Reason: Topic is not partitioned topic```
----
2020-03-19 09:44:51 UTC - Geetish Sanjeeb Nayak: Hello Everyone. I am Geetish
Nayak and exploring Pulsar for Cross Data Centre replication use-case for our
product. I wanted to ask if there is a setup guide to configure running Pulsar
with Geo-Replication. I wanted to basically know what all changes are required
to be done to make sure that Pulsar clusters deployed in different regions are
aware of each other. Any pointers/documents would be of great help. Thanks!
----
2020-03-19 09:47:27 UTC - Ali Ahmed: @Geetish Sanjeeb Nayak are you trying to
do Asynchronous or Synchronous replication ?
----
2020-03-19 09:47:57 UTC - Geetish Sanjeeb Nayak: @Ali Ahmed Async replication
----
2020-03-19 09:49:00 UTC - Ali Ahmed: they are is specific info here
<https://medium.com/@pckeyan/apache-pulsar-geo-replication-ad4f0ca3224b>
----
2020-03-19 09:49:26 UTC - Ali Ahmed: more generation info is here
<https://www.splunk.com/en_us/blog/it/geo-replication-in-apache-pulsar-part-1-concepts-and-features.html>
<https://www.splunk.com/en_us/blog/it/geo-replication-in-apache-pulsar-part-2-patterns-and-practices.html>
----
2020-03-19 09:50:05 UTC - Geetish Sanjeeb Nayak: Thanks @Ali Ahmed. This is a
good start. Appreciate the help.
----
2020-03-19 13:10:13 UTC - Poul Henriksen: Just tried this after changing
`defaultNumPartitions` from 1 to 2, then I can increase them correctly. Seems
to be a bug in the logic determining whether a topic is partitioned or not.
I'll report a bug.
----
2020-03-19 13:30:23 UTC - Pradeesh: @tuteng still the same issue even after
running the init sql statements
----
2020-03-19 13:30:31 UTC - Pradeesh: these are the logs
----
2020-03-19 13:30:35 UTC - Pradeesh: ```[07:29][pnair200@COSML-1686681:~]$ k
logs pulsar-manager-879745749-sp2kk -n kube-system
Starting PostGreSQL Server
The files belonging to this database system will be owned by user "pulsar".
This user must also own the server process.
The database cluster will be initialized with locales
COLLATE: C
CTYPE: C.UTF-8
MESSAGES: C
MONETARY: C
NUMERIC: C
TIME: C
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /data/postgresql ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default timezone ... UTC
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
sh: locale: not found
2020-03-19 13:14:37.368 UTC [23] WARNING: no usable system locales were found
performing post-bootstrap initialization ... ok
WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
syncing data to disk ... ok
Success.
waiting for server to start....2020-03-19 13:14:38.207 UTC [27] LOG: listening
on IPv6 address "::1", port 5432
2020-03-19 13:14:38.207 UTC [27] LOG: listening on IPv4 address "127.0.0.1",
port 5432
2020-03-19 13:14:38.214 UTC [27] LOG: listening on Unix socket
"/run/postgresql/.s.PGSQL.5432"
2020-03-19 13:14:38.220 UTC [27] LOG: listening on Unix socket
"/tmp/.s.PGSQL.5432"
2020-03-19 13:14:38.238 UTC [28] LOG: database system was shut down at
2020-03-19 13:14:38 UTC
2020-03-19 13:14:38.241 UTC [27] LOG: database system is ready to accept
connections
done
server started
ALTER ROLE
CREATE DATABASE
GRANT
You are now connected to database "pulsar_manager" as user "pulsar".
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
CREATE TABLE
Starting Pulsar Manager Front end
Starting Pulsar Manager Back end
Start servie no enable JWT.
2020-03-19 13:14:40,592 CRIT Supervisor is running as root. Privileges were
not dropped because no user is specified in the config file. If you intend to
run as root, you can set user=root in the config file to avoid this message.
2020-03-19 13:14:40,612 INFO supervisord started with pid 50
2020-03-19 13:14:41,625 INFO spawned: 'pulsar-manager-backend' with pid 54
2020-03-19 13:14:42,627 INFO success: pulsar-manager-backend entered RUNNING
state, process has stayed up for > than 1 seconds (startsecs)
2020-03-19 13:14:48.049 UTC [88] FATAL: role "postgres" does not exist
2020-03-19 13:14:49.050 UTC [93] FATAL: role "postgres" does not exist
2020-03-19 13:14:50.039 UTC [94] FATAL: role "postgres" does not exist
2020-03-19 13:14:51.032 UTC [95] FATAL: role "postgres" does not exist
2020-03-19 13:14:52.032 UTC [117] FATAL: role "postgres" does not exist
2020-03-19 13:14:53.029 UTC [118] FATAL: role "postgres" does not exist
2020-03-19 13:14:54.031 UTC [124] FATAL: role "postgres" does not exist
2020-03-19 13:14:55.037 UTC [125] FATAL: role "postgres" does not exist```
----
2020-03-19 13:32:39 UTC - Pradeesh: does it need a postgres role ? I don’t see
any currently
----
2020-03-19 13:32:44 UTC - Pradeesh: ```pulsar=# \du+
List of roles
Role name | Attributes |
Member of | Description
-----------+------------------------------------------------------------+-----------+-------------
pulsar | Superuser, Create role, Create DB, Replication, Bypass RLS | {}
|```
----
2020-03-19 13:35:55 UTC - tuteng: Do you use your own database or a
pre-installed database in the docker image?
----
2020-03-19 13:39:04 UTC - tuteng: Can you check log file pulsar-manager.log?
----
2020-03-19 13:46:36 UTC - tuteng: This is pulsar manager's k8s yaml file
<https://github.com/apache/pulsar/blob/master/deployment/kubernetes/helm/pulsar/values.yaml#L483>
----
2020-03-19 14:03:28 UTC - Poul Henriksen:
<https://github.com/apache/pulsar/issues/6564>
----
2020-03-19 16:34:02 UTC - Pradeesh: pre-installed database with the docker
image : 0.1.0
----
2020-03-19 16:37:21 UTC - Pradeesh: this is the spec I’m using
```apiVersion: extensions/v1beta1
kind: Deployment
metadata:
labels:
app: pulsar
component: pulsar-manager
name: pulsar-manager
namespace: kube-system
spec:
replicas: 1
selector:
matchLabels:
app: pulsar
component: pulsar-manager
template:
metadata:
labels:
app: pulsar
component: pulsar-manager
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: <http://rdei.io/sec-zone-green|rdei.io/sec-zone-green>
operator: In
values:
- "true"
containers:
- env:
- name: REDIRECT_HOST
value: 10.54.213.138
- name: REDIRECT_PORT
value: "9527"
- name: DRIVER_CLASS_NAME
value: org.postgresql.Driver
- name: URL
value: jdbc:<postgresql://127.0.0.1:5432/pulsar_manager>
- name: USERNAME
value: pulsar
- name: PASSWORD
value: pulsar
- name: LOG_LEVEL
value: DEBUG
image:
<http://registry.vipertv.net/viper/pulsar-manager:v0.1.0|registry.vipertv.net/viper/pulsar-manager:v0.1.0>
imagePullPolicy: IfNotPresent
name: pulsar-manager
resources:
limits:
cpu: "1"
memory: 2Gi
requests:
cpu: "1"
memory: 2Gi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /data
name: pulsar-manager-data
dnsPolicy: ClusterFirst
hostNetwork: true
nodeName: 10.54.213.138
restartPolicy: Always
terminationGracePeriodSeconds: 30
volumes:
- emptyDir: {}
name: pulsar-manager-data```
----
2020-03-19 16:39:23 UTC - Pradeesh: pulsar-manager.log
```2020-03-19 16:38:27.054 DEBUG 54 --- [pool-2-thread-1]
org.apache.http.headers : http-outgoing-659 <<
Content-Length: 3467
2020-03-19 16:38:27.054 DEBUG 54 --- [pool-2-thread-1] org.apache.http.headers
: http-outgoing-659 << Connection: close
2020-03-19 16:38:27.054 DEBUG 54 --- [pool-2-thread-1] org.apache.http.headers
: http-outgoing-659 << Server: Jetty(9.4.20.v20190813)
2020-03-19 16:38:27.054 DEBUG 54 --- [pool-2-thread-1]
o.a.http.impl.execchain.MainClientExec : Cancelling request execution
2020-03-19 16:38:27.054 DEBUG 54 --- [pool-2-thread-1]
h.i.c.DefaultManagedHttpClientConnection : http-outgoing-659: Shutdown
connection
2020-03-19 16:38:27.054 DEBUG 54 --- [pool-2-thread-1]
o.a.http.impl.execchain.MainClientExec : Connection discarded
2020-03-19 16:38:27.054 DEBUG 54 --- [pool-2-thread-1]
h.i.c.PoolingHttpClientConnectionManager : Connection released: [id:
659][route: {}-><http://172.27.223.166:8080>][total kept alive: 0; route
allocated: 0 of 10; total allocated: 0 of 100]
2020-03-19 16:38:27.055 ERROR 54 --- [pool-2-thread-1]
o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in
scheduled task.
java.lang.NullPointerException: null
at
org.apache.pulsar.manager.service.impl.ClustersServiceImpl.getClustersList(ClustersServiceImpl.java:62)
~[pulsar-manager.jar:na]
at
org.apache.pulsar.manager.service.impl.BrokerStatsServiceImpl.scheduleCollectStats(BrokerStatsServiceImpl.java:128)
~[pulsar-manager.jar:na]
at sun.reflect.GeneratedMethodAccessor134.invoke(Unknown Source)
~[na:na]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[na:1.8.0_212]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_212]
at
org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)
~[spring-context-5.0.6.RELEASE.jar:5.0.6.RELEASE]
at
org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
~[spring-context-5.0.6.RELEASE.jar:5.0.6.RELEASE]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_212]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[na:1.8.0_212]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_212]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[na:1.8.0_212]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_212]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_212]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_212]
2020-03-19 16:38:31.898 DEBUG 54 ---
[SimpleHostRoutingFilter.connectionManagerTimer]
h.i.c.PoolingHttpClientConnectionManager : Closing expired connections```
----
2020-03-19 18:26:40 UTC - Aaron Stockton: @Penghui Li so “unackedMessages” for
Exclusive subscriptions is always 0 now? Is it not possible to track how many
messages have been consumed, but not acked?
----
2020-03-19 18:35:44 UTC - Pradeesh: @tuteng ^^
----
2020-03-19 18:38:21 UTC - Roman Popenov: try restarting the service?
----
2020-03-19 18:38:50 UTC - Pradeesh: restarting the pod ? done that several times
----
2020-03-19 18:39:04 UTC - Pradeesh: curious about the null pointer exception
----
2020-03-19 19:49:26 UTC - Chris: @Sijie Guo that log file might have to do.
pulsar-perf gets super slow with the `-m` argument it seems
----
2020-03-19 20:00:40 UTC - Sijie Guo: do you have grafana dashboard set up for
bookkeeper? Can I see the bookkeeper metrics?
----
2020-03-19 20:01:54 UTC - Chris: I do. What do you want to see?
----
2020-03-19 20:05:28 UTC - Chris: Here's the main stats from me testing last
night. This is good pulsar perf behavior.
----
2020-03-19 20:11:04 UTC - Chris: Gonna need overnight or so to get bookie stats
for the slow reads. There's little to no extra bookie activity, like they're
barely even trying.
----
2020-03-19 20:39:24 UTC - Sijie Guo: @Chris: okay. Can you try increase
`dispatcherMaxReadBatchSize=100` in conf/broker.conf to `1000` and test
pulsar-perf with your data？
----
2020-03-19 20:42:38 UTC - Chris: Will do. Still waiting on the backlog to grow
so I'll get back to you tomorrow.
----
2020-03-19 20:52:08 UTC - Sijie Guo: okay
----
2020-03-19 21:33:00 UTC - Pradeesh: @Sijie Guo ^^ can you look at the logs and
guide us …have been stuck on this for a week now
----
2020-03-20 02:17:03 UTC - tuteng: How is your environment configured? is
environment's service_url connected to pulsar-manager?
----
2020-03-20 02:18:29 UTC - tuteng: ```ClustersServiceImpl.getClustersList```
will get cluster information by calling service_url
----
2020-03-20 02:29:58 UTC - Pradeesh: yes it is connected using the serviceURL
----
2020-03-20 02:30:44 UTC - Pradeesh: this is what is configured in the UI
----
2020-03-20 02:30:47 UTC - Pradeesh: ```root@pulsar-admin:/pulsar#
/pulsar/bin/pulsar-admin clusters get pulsar-dev-dev
{
"serviceUrl" : "<http://172.27.223.166:8080/>",
"brokerServiceUrl" : "<pulsar://172.27.223.166:6650/>"
}```
----
2020-03-20 02:32:00 UTC - Pradeesh: but if I curl the serviceURL it returns a
404 ..is that normal ?
----
2020-03-20 03:26:41 UTC - Lin Yiming: @Lin Yiming has joined the channel
----
2020-03-20 05:32:50 UTC - tuteng: Is there any result from this command?
```curl -v <http://172.27.223.166:8080/metrics/>```
----
2020-03-20 05:58:12 UTC - Ken Huang: If pulsar has 2 brokers, and only has 1
topic 1 partition, in this case, one broker will always idle?
----
2020-03-20 07:32:09 UTC - Sijie Guo: yes
----
2020-03-20 07:32:23 UTC - Sijie Guo: one partition is served by one broker
----
